PDA

View Full Version : Windows Users: Run a full library scan following time changes



JJZolx
2008-03-11, 19:35
There's an odd Windows "anomaly" where after a system time change, all of file date/times reported by the operating system will have changed. If you pay attention this can even be seen in Windows Explorer, where file modification times will shift one hour following a daylight savings time change.

Whenever SlimServer/SqueeeCenter accesses any music file on the server it checks the file modification timestamp against what it has stored in its database. If it sees a difference, it will rescan the file and update its database. The effect this has is to cause a lot of unnecessary overhead when browsing and playing the music library. Whether or not you see a slowdown in response times of either the remote, web, or controller interfaces will probably depend on the speed of your server.

So, IMO, it's a good idea to run a full clear/rescan following a Daylight Savings time change if you're running SS or SC on a Windows server.

MrC
2008-03-11, 19:46
Hmmmm.

Is UTC time stored, or local time ?

Window's isn't changing the time of the files - a given application (like explorer) gets timezone adjusted dates based upon the user's timezone settings.

JJZolx
2008-03-11, 19:57
Hmmmm.

Is UTC time stored, or local time ?

Window's isn't changing the time of the files - a given application (like explorer) gets timezone adjusted dates based upon the user's timezone settings.

No, Windows isn't modifying the times of files. In NTFS file systems UTC is stored. But what happens is that the timezone offset is changed when DST begins and ends.

Here's an article that explains the problem, and even the reasons why Microsoft chose to implement it as they have.

http://www.codeproject.com/KB/datetime/dstbugs.aspx

I'm not sure if SqueezeCenter could work around the issue. I think it would depend upon whether ActiveState Perl can return pure UTC file times and then SC store those instead of local times.

Ben Sandee
2008-03-11, 20:15
On Tue, Mar 11, 2008 at 9:57 PM, JJZolx <
JJZolx.3652pz1205290801 (AT) no-mx (DOT) forums.slimdevices.com> wrote:

> Here's an article that explains the problem, and even the reasons why
> Microsoft chose to implement it as they have.
>
> http://www.codeproject.com/KB/datetime/dstbugs.aspx
>

That is a really funny article...

MrC
2008-03-11, 20:35
Indeed, thanks for sharing.

haunyack
2008-03-11, 20:38
Would this apply if NTFS Timestamp function disabled in WXP?

.

JJZolx
2008-03-11, 20:50
Would this apply if NTFS Timestamp function disabled in WXP?

Yes.

The NtfsDisableLastAccessUpdate setting is supposed to be a performance tweak, correct?

http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/regentry/46656.mspx?mfr=true

This only disables updating of last-access times (and I think it only applies to directories, not files). SqueezeCenter stores file last-modified timestamps in its database, not last-access.

haunyack
2008-03-11, 21:36
Yes.The NtfsDisableLastAccessUpdate setting is supposed to be a performance tweak, correct?

Yes, that is what is promulgated but I have not seen any quantifiable data to prove that out, nor do I notice any performance gain.
The improvement is probably measured by nano/pico seconds saved and claimed as performance increase.


This only disables updating of last-access times (and I think it only applies to directories, not files). SqueezeCenter stores file last-modified timestamps in its database, not last-access.

Good to know...thanks for the info as I noticed that my usual re-scan took about 45 minutes longer than expected the night of the time-change.

.

JJZolx
2008-03-11, 22:03
I noticed that my usual re-scan took about 45 minutes longer than expected the night of the time-change.

If you were running a "new and changed" scan then it probably ended up rescanning every file in the library.

haunyack
2008-03-11, 22:50
If you were running a "new and changed" scan then it probably ended up rescanning every file in the library.


Yes, I add music regularly by using the My Music\<folder\album> method.
After that I initiate a scan for MIP integration.
I noticed that when I executed a backup next day, all my previous music files reported date & time change.
Subsequently, I may have accidentally avoided the slowdown you mentioned by enduring the extra time needed to update the db.
What luck!

.

JJZolx
2012-03-11, 10:57
Reminder to those running the server under Windows:

You should run a full "clear & rescan everything" today. Before you run your next "new & changed" scan, or else you'll probably see that new & changed scan rescan every file in the library due to the time change. Which is a lot slower than just running a full rescan.

castalla
2012-03-11, 11:43
What time change would that be?

Mnyb
2012-03-11, 11:48
What time change would that be?

DST , daylight saving .

http://www.timeanddate.com/time/dst/2012.html

Swedis windows and LMS user should pay attention 25/3

Atlantic
2012-03-11, 11:56
What time change would that be?

Perhaps this:
http://www.bbc.co.uk/news/science-environment-16597191

Except that they postponed the decision, again. Until 2015.

regards, Atlantic

castalla
2012-03-11, 12:00
Does not compute!

-quote

If this does happen, the ITU says that leap seconds would be eliminated from 1 January 2018.

-unquote

castalla
2012-03-11, 12:11
DST today only applies to parts of Canada, USA, Haiti, Bahamas & Bermuda and part of Greenland.

Everybody else where DST operates can relax until next week - apart from the odd-balls who change on their own eccentric dates!

Southern Hemisphere users - you are on your own timetable!

jimzak
2012-03-11, 12:27
And I might add (from hard experience), if you don't run a complete rescan, LMS keeps running a prolonged version of the new & changed scan each time it runs thereafter.

This thread didn't exist this AM when I first experienced the problem.

LMS has succeeded in wasting another hour or so of my already shortened weekend :(

aubuti
2012-03-11, 12:31
This thread didn't exist this AM when I first experienced the problem.
Sure it did, although it's also natural that a 4 year-old thread wouldn't be the first thing (or 2nd, or 3rd...) on most people's mind this morning. Thanks to Jim for bumping it.

Atlantic
2012-03-11, 12:45
Does not compute!

-quote

If this does happen, the ITU says that leap seconds would be eliminated from 1 January 2018.

-unquote

Yes, they said that IF it happened, then there'd be 6 years' notice.

Time enough, you would think, 6 years. But Time does not seem to be what we think it is. It does (or doesn't) match the Earth's rotation. It gets delayed by some old guys in Paris who decide whether to add a second now and then. It gets shifted around, so that midday - in our neck of the woods - occurs at a quarter past one for half the year; our sundial is never right.

Beats me how anything works, really.

Anyway, that decision was postponed until 2015:

http://www.bbc.co.uk/news/science-environment-16625614

regards, Atlantic

pomatomus
2012-03-12, 10:16
Is this advice applicable to all versions of Windows?

Benefactor
2012-03-13, 16:37
Thanks for the heads-up.

I could not for the life of me figure out what was going on with the "new and changed" scan rescanning the whole library.

JJZolx
2013-11-03, 00:46
Bump. A reminder for anyone running LMS under Windows:

Run a full "clear library & rescan everything" following a time change to or from Daylight Savings Time. Otherwise, the next new & changed scan may rescan the whole library, which is typically much slower than doing a full scan.

reinholdk
2013-11-03, 15:50
Been there, done that. :)

Any ideas why new and changed scans following the DST switch always find half the number of modified tracks compared to the previous scan? (http://forums.slimdevices.com/showthread.php?98276-I-Got-Them-Eastern-Standard-Time-Blues&p=760952&viewfull=1#post760952)

RonM
2013-11-03, 19:43
DST today only applies to parts of Canada, USA, Haiti, Bahamas & Bermuda and part of Greenland.

Everybody else where DST operates can relax until next week - apart from the odd-balls who change on their own eccentric dates!

Southern Hemisphere users - you are on your own timetable!

Actually, the UK and western Europe (at least) changed to standard time last week. For a week we here in Toronto were only 4 hours earlier than London. Back to the normal 5 now.

R

MeSue
2013-11-04, 09:37
Bump. A reminder for anyone running LMS under Windows:

Run a full "clear library & rescan everything" following a time change to or from Daylight Savings Time. Otherwise, the next new & changed scan may rescan the whole library, which is typically much slower than doing a full scan.

I didn't see this in time. My scheduled new and changed scan took 1.5 hours last night. Normally 10 minutes or less. Will it really continue to be slow until I clear and rescan as one post above indicates?

Makes no sense to me, but I will go ahead and change my next scheduled scan to clear and rescan.

garym
2013-11-04, 14:54
I didn't see this in time. My scheduled new and changed scan took 1.5 hours last night. Normally 10 minutes or less. Will it really continue to be slow until I clear and rescan as one post above indicates?

Makes no sense to me, but I will go ahead and change my next scheduled scan to clear and rescan.

yes. bite the bullet and do a clear and rescan. :D

JJZolx
2013-11-04, 15:47
If the new & change scan rescanned every file in the library, then it shouldn't rescan those files again, because the new timestamps would be recorded in the library database. One the other hand, if there's some bug is causing only 1/2 of the files to be scanned each time, then it would keep doing it until it finishes the whole library.

I recall a couple of similar bugs where only half of the tracks expected to be scanned actually do get scanned. Don't know if they're related.

http://bugs.slimdevices.com/show_bug.cgi?id=17438

IMO, it's good to run a full rescan periodically anyway.

MeSue
2013-11-04, 18:30
I changed the scheduler to run clear and rescan tomorrow and then I have a reminder so I don't forget to switch it back.

JJZolx
2014-03-09, 16:05
Reminder bump.

bwaldron
2014-03-10, 14:16
Reminder bump.

Thanks...I usually remember to do this, but had neglected this time.

dasmueller
2014-03-10, 14:37
I forgot as well.

Thanks

dasmueller
2015-03-06, 07:18
Daylight savings is coming up. I believe we still need to do a full rescan.

reinholdk
2015-03-06, 13:33
You're certainly right, still three weeks to go for me.

Just wondered if there's a command line argument for LMS to start a full scan, so that one could create a scheduled task.
But a command line arg is not needed. This should do the trick: create a batch file to stop the LMS service (net stop "Logitech Media Server"), delete the db files and then restart the service (net start "Logitech Media Server"). Then create a scheduled task that runs under an admin account triggered after the two DST switching times of the year...

Edit: this works for me:


net stop "Logitech Media Server"
if errorlevel 1 exit /b %errorlevel%
copy /y %ALLUSERSPROFILE%\Squeezebox\Cache\*.db %ALLUSERSPROFILE%\Squeezebox\Cache\*.db.bak
del %ALLUSERSPROFILE%\Squeezebox\Cache\*.db
net start "Logitech Media Server"

mherger
2015-03-12, 07:17
I've committed a bunch of changes which try to deal with the DST change on Windows issue. If you're still in winter time (Europe), would you please give the latest 7.9 nightly build a try?

http://downloads.slimdevices.com/nightly/?ver=7.9

I'd really like to know whether this improves the situation or not. Thanks!

kidstypike
2015-03-12, 07:36
I've committed a bunch of changes which try to deal with the DST change on Windows issue. If you're still in winter time (Europe), would you please give the latest 7.9 nightly build a try?

http://downloads.slimdevices.com/nightly/?ver=7.9

I'd really like to know whether this improves the situation or not. Thanks!

Michael

Anything particular needs doing with this version, or just wait until 29th March?

mherger
2015-03-12, 08:12
> Anything particular needs doing with this version, or just wait until
> 21st March?

Nothing to do. Except check progress when you do your first scan for new
& changed after the dst switch. It should no longer remove all tracks
and re-scan them all.

--

Michael

kidstypike
2015-03-14, 10:27
> Anything particular needs doing with this version, or just wait until
> 21st March?

Nothing to do. Except check progress when you do your first scan for new
& changed after the dst switch. It should no longer remove all tracks
and re-scan them all.

--

Michael

Thanks, I've only just seen your reply, DST is of course 29th not 21st. (you don't see edits).

kidstypike
2015-04-03, 15:24
> Anything particular needs doing with this version, or just wait until
> 21st March?

Nothing to do. Except check progress when you do your first scan for new
& changed after the dst switch. It should no longer remove all tracks
and re-scan them all.

--

Michael

I'd forgotten I would be away on the 29th, just got back, I see there are no replies to this thread.

Just to let you know, I just did a scan for new and changed and scanner did not remove all tracks and re-scan them all - so looks like a fix?

JJZolx
2015-04-03, 15:33
That's great news. Michael, what was the workaround?

mherger
2015-04-04, 04:07
> That's great news. Michael, what was the workaround?

That would indeed be great news.

How it works? After every scan I'm setting a flag whether the scan
happened in DST or not. And before every new & changed scan it would
check that flag against the current DST state. If it changed, then it
runs an update on the tracks table to modify it according to the change
and sets the flag to the new status. (and all of this in Windows only)

I decided to do this one time update rather than add more logic to all
of the scanner code spread across multiple source files. Seems to be
good enough. Keep fingers crossed. There will be a next DST change in
the opposite sense :-).

--

Michael

kesey
2015-04-04, 06:51
Thanks Michael. All appears well on a "Look for new and changed media files" scan.

Discovering files/directories: /mnt/disk1/Tunes_Lacie (15479 of 15479) Complete 00:00:37

Removing deleted files: /mnt/disk1/Tunes_Lacie (1 of 1) Complete 00:00:02

Discovering playlists: /mnt/disk1/Tunes_Lacie (1609 of 1609) Complete 00:00:17

Scanning new playlists: /mnt/disk1/Tunes_Lacie (1 of 1) Complete 00:00:05

Building full text index (7 of 7) Complete 00:00:29

Find updated coverart files (373 of 373) Complete 00:00:01

Pre-caching Artwork (46 of 46) Complete 00:00:06

Database Optimize (2 of 2) Complete 00:00:43

The server has finished scanning your media library.

dasmueller
2016-03-11, 15:07
Reminder to do a complete rescan when the time changes

mherger
2016-03-11, 15:33
> Reminder to do a complete rescan when the time changes

Hopefully not. I applied a change which should deal with it a year ago.
When is the time change up?

--

Michael

kidstypike
2016-03-11, 15:35
Reminder to do a complete rescan when the time changes

I thought this was fixed?

JJZolx
2016-03-11, 15:40
> Reminder to do a complete rescan when the time changes

Hopefully not. I applied a change which should deal with it a year ago.
When is the time change up?

This coming Saturday night. I'm not sure I ever really tested the fix, but I've disabled my nightly auto-rescan, so I'll be able to know for sure on Sunday.

No comments yet in the bug report.

http://bugs.slimdevices.com/show_bug.cgi?id=18078

dasmueller
2016-03-11, 15:48
> Reminder to do a complete rescan when the time changes

Hopefully not. I applied a change which should deal with it a year ago.
When is the time change up?

--

Michael

What version(s) of LMS would this apply to ?

mherger
2016-03-11, 16:03
> What version(s) of LMS would this apply to ?

I think it's 7.9 only. Heh... at the time I thought I'd back-port it if
it turned out to be ok.

--

Michael

JJZolx
2016-03-13, 08:52
It looks like a New & Changed scan did NOT see any music files as changed following the time change. But, it did see all of the covers as having changed and resized and re-cached all of them.

Discovering files/directories: D:\Flac (48390 of 48390) Complete 00:00:40
Discovering files/directories: C:\ProgramData\Squeezebox\playlists (2 of 2) Complete 00:00:00
Find updated coverart files (3774 of 3774) Complete 00:00:04
Pre-caching Artwork (3580 of 3580) Complete 00:03:00

The server has finished scanning your media library.
Total Time: 00:03:44 (Sunday, March 13, 2016 / 9:48 AM)

ftlight
2016-03-13, 13:20
Win7, LMS 7.9, clear and rescan:

Discovering files/directories: D:\Audio\CD (13939 of 13939) Complete 00:00:30

Scanning new music files: D:\Audio\CD (12643 of 12643) Complete 00:03:42

Discovering playlists: D:\Audio\Playlists (35 of 35) Complete 00:00:00

Scanning new playlists: D:\Audio\Playlists (34 of 34) Complete 00:00:01

Building full text index (7 of 7) Complete 00:00:04

Pre-caching Artwork (875 of 875) Complete 00:00:38

Database Optimize (2 of 2) Complete 00:00:05

mherger
2016-03-14, 01:35
> It looks like a New & Changed scan did NOT see any music files as
> changed following the time change. But, it did see all of the covers as
> having changed and resized and re-cached all of them.

Hmm... did I investigate this before? I'll check again. Thanks!

--

Michael

mherger
2016-03-15, 00:49
> It looks like a New & Changed scan did NOT see any music files as
> changed following the time change. But, it did see all of the covers as
> having changed and resized and re-cached all of them.

Are you using all standalone artwork (eg. cover.jpg) or mostly embedded?
I looked into this, and I guess if I looked into it before, I probably
came to the same conclusion as I've come today: things are complicated
:-). It only applies to cases where you're using an external artwork
file, was its timestamp would be reported incorrectly too. Unfortunately
the artwork cache isn't a nice table where the timestamp is its own
column. Therefore we can't just bump that timestamp as we do for the
tracks. Which means we can't easily fix this. Not easily enough to make
it worth the hassle and risk imho. After all that stage doesn't have the
same negative impact as does a full rescan of the tracks table.

--

Michael

JJZolx
2016-03-15, 00:57
Yeah, they're all cover.jpg files.

So, you're able to reliably detect that a DST time change has occurred since the last scan?

mherger
2016-03-15, 01:06
> So, you're able to reliably detect that a DST time change has occurred
> since the last scan?

LMS does set a flag in the database whether DST is on or off. Before
running the scan it checks that flag against the current DST state and
updates the timestamps column if needed (on Windows only).

--

Michael

JJZolx
2016-03-15, 01:49
Haven't tested the "fall back" situation, but so long as you're adjusting the timestamps by the opposite amount when it occurs, I would call it fixed.

Thanks!

jimzak
2016-11-06, 05:16
Is running a full scan still necessary? I'm on 7.9 beta.

dasmueller
2016-11-06, 06:37
I am running-Logitech Media Server Version: 7.9.0 - 1472633108 @ Wed Aug 31 08:54:44 CUT 2016

all appears OK wo a full rescan

JJZolx
2016-11-06, 12:07
Is running a full scan still necessary? I'm on 7.9 beta.

My scheduled new & changed scan ran a little slower than normal this morning, but it did not see all the files as changed. Looks like this is fixed and a full scan following a time change is no longer necessary on Windows. :)