PDA

View Full Version : Phil's Library plugin, and LMS patch



Philip Meyer
2015-07-10, 17:08
I wrote a plugin to provide a couple of library views related to track ratings (rated via Erland's TrackStat plugin).

Not Rated: only include music that is not rated
Rated High: only show music that is rated 4* or 5*

Works well for browsing in normal LMS menus.

However, I didn't like that having played some music and rated it, the libraries would not refresh. The only way to refresh the libraries was to rescan. This was annoying as the rebuild library phase completes in less than a second.

So I modified LMS to optionally allow a library view to refresh when the library view is selected. By default it will not rebuild, but can be enabled per library when the library is registered.

If anyone is interested, i've included a patch and my library plugin that uses this rebuildOnSelect option.

Phil

mherger
2015-07-13, 03:56
Thanks for that patch, Phil. I think the idea to have a library
optionally updated on some events is totally reasonable. Not all of them
will need this.

But I doubt triggering whenever somebody selects a library won't be good
enough. Eg. if the library is based on some rating or counts or
whatever, the content might get changed while the library itself is
active. I think we should rather trigger an update whenever a change to
the database happens. Which in turn might become rather expensive when
the data gets updated often... do you know whether plugins like
trackstat would signal an event when they update the db?

--

Michael

Philip Meyer
2015-07-13, 17:06
>But I doubt triggering whenever somebody selects a library won't be good
>enough. Eg. if the library is based on some rating or counts or
>whatever, the content might get changed while the library itself is
>active.
>
I thought about that, but a rebuild after each event could be problemmatic. eg, if I rate the currently playing song, it may no longer belong in the current library view. It may then fail to play, would not be browseable any more, etc.

I think it's best that the user is aware that such views are static; a snapshot in time, but has control to force a quick rebuild when convenient.

>I think we should rather trigger an update whenever a change to
>the database happens. Which in turn might become rather expensive when
>the data gets updated often... do you know whether plugins like
>trackstat would signal an event when they update the db?
I'm not sure.

mherger
2015-07-13, 21:27
> I thought about that, but a rebuild after each event could be problemmatic. eg, if I rate the currently playing song, it may no longer belong in the current library view. It may then fail to play, would not be browseable any more, etc.

Agreed. But...

> I think it's best that the user is aware that such views are static; a snapshot in time, but has control to force a quick rebuild when convenient.

I don't think the user knows that. Nor is it really static. I'd even bet
he would love to see the behaviour be dynamic when it comes to inclusion
of stuff. Eg. he listened to a track on a different player which didn't
have the current view applied. Therefore he would expect that track to
show up in the play count based view active on an other player. Or he
would even expect a track in a ratings based view to disappear if he
changed the rating on the other player etc.

IMHO views should be as dynamic and accurate as possible, as this is the
way _I_ would expect users to expect them to behave. The case where
stuff would disappear most likely is less current than the addition.

--

Michael

mherger
2015-07-15, 08:46
Ok, here's my suggestion: a trigger which could cause a library re-build whenever a table in our library is changed. See the attached patch. With this patch a library can be registered with an "rebuildOnUpdate" parameter. This must be a regex matching the table you want to monitor. Whenever a change to the database happens, LMS would check what table and database this happened in, and whether there's a library registered to be re-built on such changes. If there is a match, then the library will be re-built.

I've added some code which hopefully would allow us to re-build even largish libraries without breaking playback. I tested this with a library view of tracks which had not been played before. A Random Mix would trigger an update on every track change. Worked perfectly fine.

In your own plugin you'd have to replace


rebuildOnSelect => 1

with something like


rebuildOnUpdate => qr/track_statistics/

Could you please give this a try and let me know what you think about it? The database.virtuallibraries debugging flag would give you some more information about what's going on, in case you encountered any issue.

Philip Meyer
2015-07-15, 18:38
>Ok, here's my suggestion: a trigger which could cause a library re-build
>whenever a table in our library is changed. See the attached patch. With
>this patch a library can be registered with an "rebuildOnUpdate"
>parameter. This must be a regex matching the table you want to monitor.
>Whenever a change to the database happens, LMS would check what table
>and database this happened in, and whether there's a library registered
>to be re-built on such changes. If there is a match, then the library
>will be re-built.
>
Okay, sounds good. I'll give it a go.

Unfortunately, I can't seem to apply the patch.
I've pulled latest 7.9, and using Tortoice Git, tried to apply patch. I runs and reports the following:

git.exe am --3way --ignore-space-change --keep-cr "D:\Downloads\library_updates.diff"
Patch format detection failed.

tried from a git shell too.

Philip Meyer
2015-07-15, 19:09
>I don't think the user knows that. Nor is it really static.
I would love to be able to view the library with dynamic updates, but worried about:

1. Performance rebuilding the whole library (or several libraries) whilst browsing/playing music,etc.
2. Problems due to items disappearing from the library. For example, play an unrated album whilst browsing "unrated music" library. Rate the playing song. Song no longer belongs in the library. What effects could this cause? Pause song, can't resume? etc.

On my server, rebuilding all libraries (I only have 3) is very quick, but not sure what it would be like for others.

>IMHO views should be as dynamic and accurate as possible, as this is the
>way _I_ would expect users to expect them to behave. The case where
>stuff would disappear most likely is less current than the addition.

I more frequently browse unrated music, as it is more likely this is music I haven't heard at least recently. I try to rate each track that plays, to fill my library with rating stats.
In trackstat, I also play "not recently played" and "music recently added". Creating library views like this is likely to lose content if rebuilt dynamically.

If playback is protected, and browsing does something sensible if the item isn't in the library any more, then dynamic library updates is better.

Note, I also plan to build a library view to show the content of "Favourites", so favourites can be browsed in better way, eg. it would be possible to create additional browse modes to browse favourite artists and favourite rock albums. I plan to iterate through all sub-folders in Favourites, finding songs, artists or albums to build the library content in code. I imagine this could potentially be a lot slower to rebuild than executing a sql statement. To make it dynamic would also require a library plugin registering for different events (Favourites Changed event?).

mherger
2015-07-15, 22:29
>> I don't think the user knows that. Nor is it really static.
> I would love to be able to view the library with dynamic updates, but worried about:
>
> 1. Performance rebuilding the whole library (or several libraries) whilst browsing/playing music,etc.

That's an important point. If you look at the patch then you'll see that
a lot of the code changes are to deal with this problem. It tries to
deal with this in various ways:

- only handle libraries this way which explicitly ask for it. It's
important that a library plugin dev does use this with care, eg. making
sure the regex only matches a minimum changes to the db to minimize the
number of updates. You can't define an auto-updating library using Adv.
Search. Some dev being aware of this problem has to implement it.

- SQLite allows to provide a hook which is called regularly on long
running queries. I'm using this to allow LMS to handle the streaming
every now and then (remember LMS is single-threaded: we have to give
control back to the streaming code every now and then in many places)

- library updates are being buffered. Should there be repeated updates
within a few seconds, only one update (per library) would be executed

- I personally see very few library views which would require this kind
of update. Changes to the database outside a scan are very rare.
Playcount probably is the most obvious (and often called). Ratings might
be another candidate, but even less regular.

> 2. Problems due to items disappearing from the library. For example, play an unrated album whilst browsing "unrated music" library. Rate the playing song. Song no longer belongs in the library. What effects could this cause? Pause song, can't resume? etc.

Library _views_ are only views. Tracks don't disappear from the
database. In your case queued tracks should still be played, though they
can't be found by browsing that view any more. Yes, this could
admittedly be confusing. But I've had complaints of the other behaviour
("played a track and it's still in my un-played library") before. We'll
see what causes more complaints. Once there actually are plugins out
there taking advantage of this new feature...

> On my server, rebuilding all libraries (I only have 3) is very quick, but not sure what it would be like for others.

Same here: two libraries (Kids' music, my music), takes less than 2s
(about 20k tracks, some 3yrs old Atom CPU <2GHz, 4GB, CentOS6). What's
your library size and hardware spec?

> I more frequently browse unrated music, as it is more likely this is music I haven't heard at least recently. I try to rate each track that plays, to fill my library with rating stats.
> In trackstat, I also play "not recently played" and "music recently added". Creating library views like this is likely to lose content if rebuilt dynamically.

What do you mean by lose content? If you have a library of untracked
music and you play it, I'd expect it to disappear. The same applies to
"not recently played" and similar definitions (good idea, btw :-)).

> Note, I also plan to build a library view to show the content of "Favourites", so favourites can be browsed in better way, eg. it would be possible to create additional browse modes to browse favourite artists and favourite rock albums. I plan to iterate through all sub-folders in Favourites, finding songs, artists or albums to build the library content in code. I imagine this could potentially be a lot slower to rebuild than executing a sql statement. To make it dynamic would also require a library plugin registering for different events (Favourites Changed event?).

You can always implement code to monitor whatever event you'd like to
trigger an update, then run that update from your code. I only tried to
cover the "how can we detect library updates to re-build our library
views" use case.

--

Michael

mherger
2015-07-15, 22:40
Ok, maybe I ruined the patch when I removed some unnecessary changes manually... here's the full patch, including the "non-played tracks" library definition.

Philip Meyer
2015-07-16, 12:52
>> 1. Performance rebuilding the whole library (or several libraries) whilst browsing/playing music,etc.
>
Sounds like you've got it all figured out!

>- I personally see very few library views which would require this kind
>of update. Changes to the database outside a scan are very rare.
>Playcount probably is the most obvious (and often called). Ratings might
>be another candidate, but even less regular.
>
Time based: "Not played recently", "Played within the last day", etc.

>> 2. Problems due to items disappearing from the library. For example, play an unrated album whilst browsing "unrated music" library. Rate the playing song. Song no longer belongs in the library. What effects could this cause? Pause song, can't resume? etc.
>
>Library _views_ are only views. Tracks don't disappear from the
>database. In your case queued tracks should still be played, though they
>can't be found by browsing that view any more.
Not so worried about the effects of browsing a library, but the effects of the current playlist, if tracks are no longer part of the current library view?
So once a track is added to the current playlist, if I change library and that track is no longer contained in the current library, the track will remain playable?
If I click to browse it, will it show details or an error?
I should try this really, when I get home.

>Yes, this could
>admittedly be confusing. But I've had complaints of the other behaviour
>("played a track and it's still in my un-played library") before. We'll
>see what causes more complaints. Once there actually are plugins out
>there taking advantage of this new feature...
>
I can see benefits both ways.

>Same here: two libraries (Kids' music, my music), takes less than 2s
>(about 20k tracks, some 3yrs old Atom CPU <2GHz, 4GB, CentOS6).
>What's your library size and hardware spec?
>
Windows 7 64bit
Windows Experience Index score 7.7 (out of 8)
Intel i7-3770K 4 core Hyperthreaded @ 3.50GH
8GB
LMS running on SSD
Music Library on a 3TB WD Red HDD

>> I more frequently browse unrated music, as it is more likely this is music I haven't heard at least recently. I try to rate each track that plays, to fill my library with rating stats.
>> In trackstat, I also play "not recently played" and "music recently added". Creating library views like this is likely to lose content if rebuilt dynamically.
>
>What do you mean by lose content?
Yeah, I meant the effect of doing something (playing or rating a track) dynamically causing the track to be removed from the library view.
That's good when browsing in normal ways, but I wonder what all effects of this could be. Play a track then try to view the song and it says it doesn't exist.

>> Note, I also plan to build a library view to show the content of "Favourites", so favourites can be browsed in better way, eg. it would be possible to create additional browse modes to browse favourite artists and favourite rock albums. I plan to iterate through all sub-folders in Favourites, finding songs, artists or albums to build the library content in code. I imagine this could potentially be a lot slower to rebuild than executing a sql statement. To make it dynamic would also require a library plugin registering for different events (Favourites Changed event?).
>
>You can always implement code to monitor whatever event you'd like to
>trigger an update, then run that update from your code. I only tried to
>cover the "how can we detect library updates to re-build our library
>views" use case.
I have little idea how best make that dynamic, but thats way off in the future. Just get the library to work first!

Philip Meyer
2015-07-16, 13:11
>Ok, maybe I ruined the patch when I removed some unnecessary changes
>manually... here's the full patch, including the "non-played tracks"
>library definition.

Doesn't like how I am trying to apply it.

git.exe am --3way --ignore-space-change --keep-cr "D:\Downloads\library_updates.diff"
Patch format detection failed.

Fail

mherger
2015-07-17, 02:35
> git.exe am --3way --ignore-space-change --keep-cr "D:\Downloads\library_updates.diff"
> Patch format detection failed.

I don't think "git am" is what you'd need:

"git-am - Apply a series of patches from a mailbox: Splits mail messages
in a mailbox into commit log message, authorship information and
patches, and applies them to the current branch."

Try a patch tool instead. eg.
http://stackoverflow.com/questions/517257/how-do-i-apply-a-diff-patch-on-windows

--

Michael

Philip Meyer
2015-07-17, 19:08
>I don't think "git am" is what you'd need:
>
>"git-am - Apply a series of patches from a mailbox: Splits mail messages
>in a mailbox into commit log message, authorship information and
>patches, and applies them to the current branch."
>
>Try a patch tool instead. eg.
>http://stackoverflow.com/questions/517257/how-do-i-apply-a-diff-patch-on-windows

Ah, I tried using "Apply Patch Serial" within TortoiseGit as it was the only patch option available. I have TortoiseGitMerge, and invoked it maually to apply the patch.

I re-enabled the sub-library demo, and upon restarting the server, all the libraries defined in that plugin were showing 0 tracks. A scan was necessary to populate them. In my library plugin, I put a call in to rebuild each library in the initPlugin function. I suggest you add that to the sub-library demo too, especially if the library is meant to be dynamically updated.

For some reason, it doesn't seem to mark an unplayed track as played after playing it at the moment for me. I'll investigate tomorrow, and try applying rebuildOnUpdate into my libraries.

Phil

erland
2015-07-17, 22:29
For some reason, it doesn't seem to mark an unplayed track as played after playing it at the moment for me.

If it's TrackStat that isn't marking the track as played there are two reason this could happen:

- TrackStat doesn't write to the database if scanning is in progress, so make sure LMS scanner process isn't running for some reason. If it's running TrackStat will queue up the database write and do it after scanning is finished.

- TrackStat doesn't mark tracks as played until you have played a certain amount of the track. LMS marks it as played immediately but TrackStat have logic so it doesn't mark a track as played if you for example quickly skip to the next track.

mherger
2015-07-17, 23:35
>I don't think "git am" is what you'd need:[color=blue]
I re-enabled the sub-library demo, and upon restarting the server, all the libraries defined in that plugin were showing 0 tracks. A scan was necessary to populate them. In my library plugin, I put a call in to rebuild each library in the initPlugin function. I suggest you add that to the sub-library demo too, especially if the library is meant to be dynamically updated.

I think that's something the virtual library manager should handle. Therefore my latest incarnation has some code to do this: when a library registers and doesn't exist in the database yet, it's being built.

BTW: I'm currently testing this with my 20k tracks library on a ReadyNAS Duo v2 with 256MB RAM. It's re-building two library views (as defined in the LibraryDemo plugin) on every track played on my Transporter, and it yet has to skip a beat. It would be interesting to see how this mechanism behaves with a 100k tracks library :-). Unless you report really bad behavior I'm going to commit this change after the weekend. Thanks for testing!

Philip Meyer
2015-07-18, 05:03
>- TrackStat doesn't mark tracks as played until you have played a
>certain amount of the track. LMS marks it as played immediately but
>TrackStat have logic so it doesn't mark a track as played if you for
>example quickly skip to the next track.

The track was about 7 minutes long. I played about a minute at the start, then skipped to near the end. Thought this would still be detected as a play if I played the end of it.

Philip Meyer
2015-07-18, 05:29
>Unless you report really bad behavior I'm going to commit this change after the weekend.
>Thanks for testing!
>
Testing with this latest patch, using my "Not Rated Tracks" library view rebuildOnUpdate.

Seems to be working well.

The only thing that I'm confused by is the web UI not refreshing the content of an album correctly.

Steps:
1. Select "Not Rated Tracks" library view
2. View an album. Play the album.
3. Rate the currently playing song (first song on album).
4. Wait a few secs.
5. View the album again.
6. The song is still listed on the album, even though it is rated.
7. Click on song to see details; shows 2nd song on album.
8. Force a refresh of web ui, still shows incorrectly.

I viewed the album on Touch UI, and the song is not included on the album, so I think it works, just web UI refresh/cache issue.

Philip Meyer
2015-07-20, 17:59
>Unless you report really bad behavior I'm going to commit this change after the weekend.
>Thanks for testing!

I find my SB3 slow to respond to remote control actions when transiting between songs. I've had a couple of stutters, and sometimes music just stops instead of transiting to play the next track.

The log doesn't reveal much, but there are many warnings, along the lines of:

[01:02:09.1580] Slim::Utils::Misc::msg (1233) Warning: [01:02:09.1577] Use of uninitialized value in subroutine entry at C:/Squeezebox/LMS/Beta/server/Slim/Music/VirtualLibraries.pm line 320.
[01:02:09.2382] Slim::Utils::Misc::msg (1233) Warning: [01:02:09.2380] Use of uninitialized value in subroutine entry at C:/Squeezebox/LMS/Beta/server/Slim/Music/VirtualLibraries.pm line 329.
[01:02:09.5487] Slim::Utils::Misc::msg (1233) Warning: [01:02:09.5485] Use of uninitialized value in subroutine entry at C:/Squeezebox/LMS/Beta/server/Slim/Music/VirtualLibraries.pm line 344.
[01:02:09.5646] Slim::Utils::Misc::msg (1233) Warning: [01:02:09.5644] Use of uninitialized value in subroutine entry at C:/Squeezebox/LMS/Beta/server/Slim/Music/VirtualLibraries.pm line 356.

I got 35 warnings within 1/2 second.

These relate to statements like:
$delete_sth->execute($id);

mherger
2015-07-21, 02:03
I find my SB3 slow to respond to remote control actions when transiting between songs. I've had a couple of stutters, and sometimes music just stops instead of transiting to play the next track.

The log doesn't reveal much, but there are many warnings, along the lines of:

[01:02:09.1580] Slim::Utils::Misc::msg (1233) Warning: [01:02:09.1577] Use of uninitialized value in subroutine entry at C:/Squeezebox/LMS/Beta/server/Slim/Music/VirtualLibraries.pm line 320.
[01:02:09.2382] Slim::Utils::Misc::msg (1233) Warning: [01:02:09.2380] Use of uninitialized value in subroutine entry at C:/Squeezebox/LMS/Beta/server/Slim/Music/VirtualLibraries.pm line 329.


Hmm... you shouldn't even reach those lines without a valid ID (which it's complaining about). I've committed my changes (with an additional tweak to hopefully better deal with the stuttering). Could you please test it? And if you still run into this problem, could you please run LMS with the databases.virtuallibraries debug flag, then provide server.log and your plugin code? Thanks!

Philip Meyer
2015-07-21, 16:12
>Hmm... you shouldn't even reach those lines without a valid ID (which
>it's complaining about). I've committed my changes (with an additional
>tweak to hopefully better deal with the stuttering). Could you please
>test it? And if you still run into this problem, could you please run
>LMS with the databases.virtuallibraries debug flag, then provide
>server.log and your plugin code? Thanks!

Okay, I'll give it a go. Note, I am currently running with your library demo plugin, and my library plugin, so between them there are potentially up to 4 libraries to refresh dynamically.
My library is currently about 54k songs.

mherger
2015-07-21, 21:18
> Okay, I'll give it a go. Note, I am currently running with your library demo plugin, and my library plugin, so between them there are potentially up to 4 libraries to refresh dynamically.
> My library is currently about 54k songs.

Ooh... that'll be a good test :-). Do you seriously use any of the
LibraryDemo libraries?

--

Michael

Philip Meyer
2015-07-22, 04:59
>Ooh... that'll be a good test :-). Do you seriously use any of the
>LibraryDemo libraries?
I wasn't, but enabled it for testing, and found browsing "New Music" within "Never Heard Tracks" library view quite useful.

Ashrob
2015-09-03, 07:12
I've had a problem with the plugins screen in server settings constantly asking me to restart it and after some messing about I've got a couple of plugins in the 'updated plugins available' menu that won't update. They are both by Philip Meyer. Is this a coincidence or something else? Many thanks to the unofficial devs for all your work on LMS, it's very much appreciated.

Edit: I have also noticed that if I turn on automatic updating of the plugins then another problem I am trying to fix, i.e. choosing a plugin but it not installing, returns. If I turn it off I can install them and manually update them except Philip Meyer's, which simply stay in the 'updated plugins available' menu.

Philip Meyer
2015-09-06, 17:19
>I've had a problem with the plugins screen in server settings constantly
>asking me to restart it and after some messing about I've got a couple
>of plugins in the updated plugins available menu that won't update. They
>are both by Philip Meyer. Is this a coincidence or something else? Many
>thanks to the unofficial devs for all your work on LMS, it's very much
>appreciated.
>
What are the two plugins?

ugh_bough
2016-08-08, 03:31
Hi,

is this patch/feature included in current releases of 7.9?

Cheers
ugh_bough

mherger
2016-08-08, 04:21
> is this patch/feature included in current releases of 7.9?

The patch yes, has been included in 7.9. But it doesn't provide any new
features. The library views Phil mentioned would need to be implemented
in his plugin - which I haven't found in this thread.

--

Michael

ugh_bough
2016-08-08, 04:47
> is this patch/feature included in current releases of 7.9?

The patch yes, has been included in 7.9. But it doesn't provide any new
features. The library views Phil mentioned would need to be implemented
in his plugin - which I haven't found in this thread.

--

Michael

Hi Mr. Herger,

and, yet again, thanks so much for the quick and helpful response!

I was just about to report back that I found that this great feature works just like it should :)

Regards
ugh_bough