PDA

View Full Version : An alternative to "Look for new and changed music"...?



egd
2008-04-25, 00:30
I don't know about other SC users, but one of the most annoying things about the whole Squeezebox/Transporter/digital audio library thing, that I'm faced with every time I purchase music (which is clearly too frequently) is having to go through the damned "Look for new and changed music" routine.

Now that SC7's been delivered, is there any chance that some effort can be put into an alternative routine that allows people to point to a folder and say "Scan/ Rescan this folder and its contents only", thereby adding a new album or updating data for an existing album that's been changed (including coverart etc.) without having to traverse the entire audio library. Whilst "Look for new and changed music" clearly has its place it must be possible to implement a routine that does the same but only traverses that part of a tree you point it to. I'd much rather spend my time listening to music than waiting for a full traverse of my library to complete its run for the sake of adding one or two, or even 10 albums.

Would you like to see SC7 enhanced to include "Scan/ Rescan this folder and its contents only" functionality?

peter
2008-04-25, 00:48
egd wrote:

> Would you like to see SC7 enhanced to include "Scan/ Rescan this folder
> and its contents only" functionality?
>

That already exists. Just go to "Music folder" and browse to the folder
where you have the new stuff. It will be automatically added to the
database. I *never* use "Look for new and changed music".

Regards,
Peter

egd
2008-04-25, 01:04
That already exists. Just go to "Music folder" and browse to the folder where you have the new stuff. It will be automatically added to the
database. I *never* use "Look for new and changed music".That option is well known and understood by most, however, it doesn't 1) add album art, 2) detect changes to tags.

I should have clarified this in my first post.

kdf
2008-04-25, 01:11
On 25-Apr-08, at 1:04 AM, egd wrote:

>
> ------------------------------------------------------------------------
> A poll associated with this post was created, to vote and see the
> results, please visit http://forums.slimdevices.com/showthread.php?t=46795
> ------------------------------------------------------------------------
> Question: Would you like to see SC7 enhanced to include "Scan/
> Rescan this folder and its conte
>
> - Yes, I'd love to see this enhancement implemented
> - No, don't see any point, I watch TV between updates
> ------------------------------------------------------------------------
>
> peter;295386 Wrote:
>> That already exists. Just go to "Music folder" and browse to the
>> folder
>> where you have the new stuff. It will be automatically added to the
>> database. I *never* use "Look for new and changed music".That
>> option is well know and understood by most, however, it doesn't 1)
> add album art, 2) detect changes to tags.

1) it does, granted not perfectly for post-scan related issues
(compilations). 2) it does (if you properly set the tagger to
correctly update the 'last modified' time).
-k

peter
2008-04-25, 01:14
egd wrote:
> ------------------------------------------------------------------------
> A poll associated with this post was created, to vote and see the
> results, please visit http://forums.slimdevices.com/showthread.php?t=46795
> ------------------------------------------------------------------------
> Question: Would you like to see SC7 enhanced to include "Scan/
> Rescan this folder and its conte
>
> - Yes, I'd love to see this enhancement implemented
> - No, don't see any point, I watch TV between updates
> ------------------------------------------------------------------------
>
> peter;295386 Wrote:
>
>> That already exists. Just go to "Music folder" and browse to the folder
>> where you have the new stuff. It will be automatically added to the
>> database. I *never* use "Look for new and changed music".That option is well know and understood by most, however, it doesn't 1)
>>
> add album art, 2) detect changes to tags.
>
> I should have clarified this in my first post.
>

I see, well, perhaps that should be the improvement then.
Add a 'rescan folder' button to the web interface as you suggest.

Still, I'm one of these users who use 'music folder' exclusively (my
tags are a mess and I rarely change them) and I do seem to recall that
whenever I change album art it's updated the next time I play the album
(via music folder).

Regards,
Peter

egd
2008-04-25, 01:18
I see, well, perhaps that should be the improvement then.
Add a 'rescan folder' button to the web interface as you suggest.

Still, I'm one of these users who use 'music folder' exclusively (my
tags are a mess and I rarely change them) and I do seem to recall that
whenever I change album art it's updated the next time I play the album
(via music folder). Yep, the album art appears when you browse to the folders as you've indicated you do, however, it definitely won't appear if browsing the artist or album list and would likely not appear on a Controller either, hence the need (for those of us that use tags and browse using these means) to carry out a rescan looking for changes.

egd
2008-04-25, 01:41
1) it does, granted not perfectly for post-scan related issues
(compilations). yet to see this work on any album, compilation or otherwise on any version of SS/SC. There anything you do to make this work, even for non-compilations?


2) it does (if you properly set the tagger to
correctly update the 'last modified' time).-k will give that a spin, not seen that work, but I may have mp3tag set not to change date/timestamps.

Just completed a rescan looking for updates after updating my MiP cache:
Total Tracks: 88,963
Total Albums: 5,813
MusicIP Import (88963 of 88963) Complete 01:31:35
SqueezeCenter has finished scanning your music collection.01:57:26

Two hours to deal with the addition of perhaps 5 albums?

hickinbottoms
2008-04-25, 01:49
Would you like to see SC7 enhanced to include "Scan/ Rescan this folder and its contents only" functionality?

I think a better alternative is for the server to monitor and automatically spot changes to the music folder and to rescan it as and when it sees those changes - you shouldn't need to manually poke it all IMO.

Modern OS's include robust functions to monitor folders for changes that don't rely on applications changing timestamps etc - I know at least MAC OSX, Linux, BSD and Windows include this. Unfortunately the mechanisms to implement this differ from OS to OS.

I've worked up a beta plugin to do this on Linux ('AutoRescan' - check the plugins on the wiki if you're interested), and I'm planning on widening it to support Windows as well when I get time.

Stuart

kdf
2008-04-25, 01:51
On 25-Apr-08, at 1:41 AM, egd wrote:

> SqueezeCenter has finished scanning your music collection.01:57:26
>
> Two hours to deal with the addition of perhaps 5 albums?
>

All of which can be done while you sleep. The only thing you need to
do immediately is play the music, which takes only a few seconds using
BMF.

Ideally, the final goal is to never even have to do a wipe and rescan,
but the ability to do that cross platform just isn't there yet.

You are also including MusicIP, which means you really need to be
bring their devs in too. For that matter, that probably means you'll
need to involve Apple too, as many users get their data from iTunes.

-k

egd
2008-04-25, 02:01
All of which can be done while you sleep. The only thing you need to do immediately is play the music, which takes only a few seconds using BMF.Don't think BMF is available if you've got MiP integration going, at least not from the remote - perhaps I was tired.


You are also including MusicIP, which means you really need to be bring their devs in too. For that matter, that probably means you'll
need to involve Apple too, as many users get their data from iTunes.MiP now has an SDK so MiP could theoretically become built-in. iTunes produces an XML file, why involve their dev's?

Craig
2008-04-25, 02:04
I think a better alternative is for the server to monitor and automatically spot changes to the music folder and to rescan it as and when it sees those changes - you shouldn't need to manually poke it all IMO.

Stuart

I agree with you, it should be automatic. I like the sound of your plugin though, give me a shout when you have a windows version built.

did you ever get anywhere with lazysearch for jive?

Craig

egd
2008-04-25, 02:04
I think a better alternative is for the server to monitor and automatically spot changes to the music folder and to rescan it as and when it sees those changes - you shouldn't need to manually poke it all IMO. Agreed, this would be the ideal approach, but difficult to achieve.


I've worked up a beta plugin to do this on Linux ('AutoRescan' - check the plugins on the wiki if you're interested), and I'm planning on widening it to support Windows as well when I get time.The Linux flavour will be all I need, thx - I'll give it a spin. BTW, I Love LazySearch, and use it all the time. Thanks for making it.

kdf
2008-04-25, 02:17
On 25-Apr-08, at 2:01 AM, egd wrote:
>
> kdf;295409 Wrote:
>> All of which can be done while you sleep. The only thing you need to
>> do immediately is play the music, which takes only a few seconds
>> using
>> BMF.Don't think BMF is available if you've got MiP integration
>> going, at
> least not from the remote - perhaps I was tired.
It's merged data so they are not exclusive. Set up a folder that gets
you to the new music.
>
>
> kdf;295409 Wrote:
>> You are also including MusicIP, which means you really need to be
>> bring
>> their devs in too. For that matter, that probably means you'll
>> need to involve Apple too, as many users get their data from
>> iTunes.MiP now has an SDK so MiP could theoretically become built-in.

I think you overestimate how many people are involved in SC
development. It's a tall order to sideline a large portion of that in
order to integrate someone else's software.
I know I won't be writing it, despite being a big fan of MIP and
having been involved in a large portion of the existing integration.
I just don't have that kind of time. Even with direct access to the
m3lib file, it would need to be imported into the SC database one way
or another and that takes time.

>> iTunes
> produces an XML file, why involve their dev's?
because they break the format with every new release.

slimkid
2008-04-25, 08:17
On 25-Apr-08, at 1:04 AM, egd wrote:
[color=blue]
...2) it does (if you properly set the tagger to
correctly update the 'last modified' time).
-k

And, if one properly sets the tagger no to update the 'last modified' date, then it doesn't work. Updating the last modified every time tags are changed defeats the purpose of another feature - 'New Music' .

K

egd
2008-04-25, 15:29
And, if one properly sets the tagger no to update the 'last modified' date, then it doesn't work. Updating the last modified every time tags are changed defeats the purpose of another feature - 'New Music' .

K

Thx Slimkid, I knew there was something I was forgetting, but it was late and I was in need of sleep. :)

JJZolx
2008-04-25, 16:12
Nobody is forced to run SqueezeCenter on slow hardware. If a "new and changed" scan is taking 5 hours on your server, and if that time is unacceptable to you, then I would suggest you "need" a faster server. On my P4 3GHz WinXP server it takes less than 3 minutes to run this type of scan for 20,300 tracks.

egd
2008-04-25, 16:17
Nobody is forced to run SqueezeCenter on slow hardware. If a "new and changed" scan is taking 5 hours on your server, and if that time is unacceptable to you, then I would suggest you "need" a faster server. On my P4 3GHz WinXP server it takes less than 3 minutes to run this type of scan for 20,300 tracks.

Agreed. My server isn't that slow, it was just MiP integration that wasn't working properly. The rescan (running off my NAS) only takes a fraction of that time. If you force a rebuild of the MiP cache file then SC rebuilds off the back of that too in effect doing a full rescan.

DVB
2008-04-25, 16:59
Browsing through "Music Folder" wouldn't be of any use if the folder's name consisted of non-latin characters, as nothing meaningful would be displayed.

/DVB

Pale Blue Ego
2008-04-25, 20:12
I would like to have the option to "clear and rescan this folder and its subfolders". Maybe it could be built into the "browse music folder" interface. This would make for much quicker scans because you would usually only want to scan a much smaller subset of the total library. You could have a fully updated library in a minute instead of an hour.

schiegl
2008-04-26, 01:55
I think a better alternative is for the server to monitor and automatically spot changes to the music folder and to rescan it as and when it sees those changes - you shouldn't need to manually poke it all IMO.

Modern OS's include robust functions to monitor folders for changes that don't rely on applications changing timestamps etc - I know at least MAC OSX, Linux, BSD and Windows include this. Unfortunately the mechanisms to implement this differ from OS to OS.

I've worked up a beta plugin to do this on Linux ('AutoRescan' - check the plugins on the wiki if you're interested), and I'm planning on widening it to support Windows as well when I get time.

Stuart

Stuart,

i recently tested your AutoRescan Plugin. For new/modified files it works wonderful! Would it be (even theoretically) possible to remove files from the SC-Database this way?

The server.log indicates the plugin knows what happened (first a new file has been copied to .../Incoming, a few minutes later i removed it again)



[08-04-25 13:49:14.0938] Plugins::AutoRescan::Plugin::watchCallback (289) Directory detected as modified by file modification: /.../Incoming
[08-04-25 13:49:19.1166] Plugins::AutoRescan::Plugin::inotifyPoller (232) Triggering RESCAN of folder: /.../Incoming
...
[08-04-25 13:55:07.8648] Plugins::AutoRescan::Plugin::watchCallback (309) Directory detected as modified by move out: /.../Incoming
[08-04-25 13:55:12.8877] Plugins::AutoRescan::Plugin::inotifyPoller (232) Triggering RESCAN of folder: /.../Incoming


I know the offical way to remove files from the DB is a wipe & scan procedure.

kind regards,
Markus

hickinbottoms
2008-04-26, 04:24
I did look into that recently. I trigger the rescan when that happens
and, tracing through the SqueezeCenter code, it looks as though removed
files are supposed to be removed from the database if they can't be
found during that rescan. However, the files aren't actually removed,
which I think might be a bug.

I think there's a couple of known enhancement request in this area:
http://bugs.slimdevices.com/show_bug.cgi?id=3928
http://bugs.slimdevices.com/show_bug.cgi?id=5158

When I get some time I might try tracing through bit more thoroughly to
try to understand what's going on. Like I said, I *think* it's supposed
to work already judging by the code I found.

Given the complexities of the database I think it would be quite messy
for a plugin such as mine to start removing data from the database so I
think I'm reliant on the core SqueezeCenter functionality to do that,
unfortunately.

Stuart

schiegl wrote:
> ------------------------------------------------------------------------
> A poll associated with this post was created, to vote and see the
> results, please visit http://forums.slimdevices.com/showthread.php?t=46795
> ------------------------------------------------------------------------
> Question: Would you like to see SC7 enhanced to include "Scan/
> Rescan this folder and its conte
>
> - Yes, I'd love to see this enhancement implemented
> - No, don't see any point, I watch TV between updates
> ------------------------------------------------------------------------
>
> hickinbottoms;295403 Wrote:
>
>> I think a better alternative is for the server to monitor and
>> automatically spot changes to the music folder and to rescan it as and
>> when it sees those changes - you shouldn't need to manually poke it all
>> IMO.
>>
>> Modern OS's include robust functions to monitor folders for changes
>> that don't rely on applications changing timestamps etc - I know at
>> least MAC OSX, Linux, BSD and Windows include this. Unfortunately the
>> mechanisms to implement this differ from OS to OS.
>>
>> I've worked up a beta plugin to do this on Linux ('AutoRescan' - check
>> the plugins on the wiki if you're interested), and I'm planning on
>> widening it to support Windows as well when I get time.
>>
>> Stuart
>>
>
> Stuart,
>
> i recently tested your AutoRescan Plugin. For new/modified files it
> works wonderful! Would it be (even theoretically) possible to remove
> files from the SC-Database this way?
>
> The server.log indicates the plugin knows what happened (first a new
> file has been copied to .../Incoming, a few minutes later i removed it
> again)
>
>
> Code:
> --------------------
>
> [08-04-25 13:49:14.0938] Plugins::AutoRescan::Plugin::watchCallback (289) Directory detected as modified by file modification: /.../Incoming
> [08-04-25 13:49:19.1166] Plugins::AutoRescan::Plugin::inotifyPoller (232) Triggering RESCAN of folder: /.../Incoming
> ...
> [08-04-25 13:55:07.8648] Plugins::AutoRescan::Plugin::watchCallback (309) Directory detected as modified by move out: /.../Incoming
> [08-04-25 13:55:12.8877] Plugins::AutoRescan::Plugin::inotifyPoller (232) Triggering RESCAN of folder: /.../Incoming
>
> --------------------
>
>
> I know the offical way to remove files from the DB is a wipe & scan
> procedure.
>
> kind regards,
> Markus
>
>
>

schiegl
2008-04-26, 06:52
I did look into that recently. I trigger the rescan when that happens
and, tracing through the SqueezeCenter code, it looks as though removed
files are supposed to be removed from the database if they can't be
found during that rescan. However, the files aren't actually removed,
which I think might be a bug.

I think there's a couple of known enhancement request in this area:
http://bugs.slimdevices.com/show_bug.cgi?id=3928
http://bugs.slimdevices.com/show_bug.cgi?id=5158

When I get some time I might try tracing through bit more thoroughly to
try to understand what's going on. Like I said, I *think* it's supposed
to work already judging by the code I found.

this somehow confirms what i've experienced so far. I have removed the track from the first directory and retagged the file (some attributes only). After a MusicIP-Import/SC-Rescan the new file (in another directory) showed up in the DB as expected and the old DB-entry has been purged (verified via mysql/track-table) - without a clear&rescan!

further investigations need on my side...

kind regards,
Markus

Craig
2008-04-26, 12:59
And, if one properly sets the tagger no to update the 'last modified' date, then it doesn't work. Updating the last modified every time tags are changed defeats the purpose of another feature - 'New Music' .

K


Thx Slimkid, I knew there was something I was forgetting, but it was late and I was in need of sleep. :)

My file date changer, available from the link below (scroll to the bottom of the page) was designed to over come this exact problem.

Craig

slimkid
2008-04-27, 08:32
My file date changer, available from the link below (scroll to the bottom of the page) was designed to over come this exact problem.

Craig

I'm afraid we are not talking about the same things:

- if updating the tags don't change last update (configurable in mp3tag or using the script to change them back to original), then 'New music' works fine, but the scanner won't pick up the track on partial re-scan.

- if updating the tags do change the last update date, then partial scan picks it up, but 'New music' shows them also, which is not what is desired (by me),

The solution is either to make scanner recognize changed files in some other way than using 'changed date' or enable 'full scan' on the subset of the library or make some kind of real time monitor for any activity on the files in library or ...


K