? new version of 7.9 much slower to do a "new & changed" scan

[swayzak]

Hi

I recently installed 7.90 Weds 22nd Oct nightly build.

I had thought that recent versions of LMS were pretty slick at rescanning library for new / changed files (I have 40000+ music files in 500GB collection).

However since installing this version this type of scan seems VERY slow (over 1/2 hour) when it had, I'm sure, been quicker than this.

Has something changed ?

Thanks

[JJZolx]

If LMS is running on Windows and you've recently gone through a Daylight Savings Time change, you're probably seeing the effects of a Windows oddity following such a time change. The last modified time of every file will appear to have changed by one hour, so LMS will rescan every file in the library the first time you do a new & changed scan. Because of how slow that is, it's best to run a full clear & rescan after a time change.

[mherger]

? new version of 7.9 much slower to do a"new & changed" scan

> I had thought that recent versions of LMS were pretty slick at
> rescanning library for new / changed files (I have 40000+ music files in
> 500GB collection).
>
> However since installing this version this type of scan seems VERY slow
> (over 1/2 hour) when it had, I'm sure, been quicker than this.

In addition to what jjzolx said (which is a pain...) there's a new full
text index build part of the scanner. But it shouldn't take that long.
If you can reproduce the long duration of the scan, then please run the
scanner with scan.* all set to debugging (Settings/Advanced/Logging) and
provide the log file. Thanks!

BTW: what was the scan time before?

--

Michael

[swayzak]

> I had thought that recent versions of LMS were pretty slick at
> rescanning library for new / changed files (I have 40000+ music files in
> 500GB collection).
>
> However since installing this version this type of scan seems VERY slow
> (over 1/2 hour) when it had, I'm sure, been quicker than this.

In addition to what jjzolx said (which is a pain...) there's a new full
text index build part of the scanner. But it shouldn't take that long.
If you can reproduce the long duration of the scan, then please run the
scanner with scan.* all set to debugging (Settings/Advanced/Logging) and
provide the log file. Thanks!

BTW: what was the scan time before?

--

Michael

Thanks (both)

I will attempt to replicate this & get the log file.

Only a few days ago I was forced to run a complete re-scan (due to the new/changed scan appearing to get "stuck"), so this should have sorted the Daylight Saving issue anyway shouldn't it ?

[JJZolx]

Only a few days ago I was forced to run a complete re-scan (due to the new/changed scan appearing to get "stuck"), so this should have sorted the Daylight Saving issue anyway shouldn't it ?

If it was run after the time change, yes.

[swayzak]

I'm running a new/changed scan tonight. Where do I find the log file afterwards ?

[JJZolx]

I'm running a new/changed scan tonight. Where do I find the log file afterwards ?

In the web interface, look under Settings > Information, at the bottom of the page. It should show the full file path of the server.log file.

[swayzak]

I have a 15MB scan log which won't upload to Attachments Manager. It shows 100% upload but then nothing appears.

Here it is...

https://drive.google.com/file/d/0B_0x4GphMbDLUWVFMHBkTVAxMEE/view?usp=sharing

Scan took 30 mins +

I should add that this is on a 2 year old netbook (Atom cpu) - but it still seems longer than before...

[JJZolx]

A new & changed scan is actually logged to the server.log file, while only a full clear & rescan is logged in the scanner.log file. If you can zip the log file, it should compress considerably. Looks like the forums will permit up to a 488 KB zip file to be attached.

[JJZolx]

I have a 15MB scan log which won't upload to Attachments Manager. It shows 100% upload but then nothing appears.

Here it is...

https://drive.google.com/file/d/0B_0x4GphMbDLUWVFMHBkTVAxMEE/view?usp=sharing

Scan took 30 mins +

I should add that this is on a 2 year old netbook (Atom cpu) - but it still seems longer than before...

Michael normally accesses the forums via email gateway, so he may not see edits like this.

Can you cut and paste the info shown on that same Information page, under 'Media Scan Details'? It would be interesting just to see the rough numbers. On my system the fulltext scan increased a typical new & changed scan's time by about 50%, but that amount is fairly incidental given that a scan used to take only 30 seconds.

For example:

Discovering files/directories: E:\Flac (47040 of 47040) Complete 00:00:23
Discovering playlists: C:\ProgramData\Squeeze\70\playlists (1 of 1) Complete 00:00:00
Building full text index (7 of 7) Complete 00:00:14
Find updated coverart files (3674 of 3674) Complete 00:00:01
Database Optimize (2 of 2) Complete 00:00:08

The server has finished scanning your media library.
Total Time: 00:00:46 (Thursday, November 6, 2014 / 4:10 AM)

[mherger]

? new version of 7.9 much slower to do a"new & changed" scan

Thanks for the log (thanks JJZolx for the update ;-)).

Here's my breakdown from the log file. Does this (roughly) match with
what is reported in Settings/Information?

File discovery: 10 Mins
New: a few seconds
Changed: 24 mins
Playlists: secs
SmartMix: 25 secs
Fulltext: 3 mins
Artwork: 5 mins

So it's still scanning for changed files. Lot's of them (9.5k?). I
wonder whether the following log entry which is reported for every
"changed" item might be a hint:

[14-11-06 22:51:07.4154] Carp::Clan::__ANON__ (216) Warning:
DBIx::Class::ResultSet::single(): Query returned more than one row. SQL
that returns multiple rows is DEPRECATED for ->find and ->single at
/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema.pm line 2777

I would suggest you run a full wipe & rescan, then check the scanner.log
file again.

> I should add that this is on a 2 year old netbook (Atom cpu) - but it
> still seems longer than before...

3 minutes for the fulltext search index of 40k files doesn't seem all
that much off on this kind of machine. I take it you're running Windows
with only 2GB of RAM? What's LMS' memory setting in Advanced/Performance?

FWIW: I'm running my main LMS on an Atom of similar age, too. On Linux
without GUI though.

--

Michael

[mherger]

? new version of 7.9 much slower to do a&quot;new &amp; changed&quot; scan

> A new & changed scan is actually logged to the server.log file, while

Not always: if you're using one of the importers implemented in a plugin
or virtual libraries it might always be run in the standalone scanner,
reporting to scanner.log. I think the FTS plugin would already cause
this. Doesn't it?

> only a full clear & rescan is logged in the scanner.log file. If you can
> zip the log file, it should compress considerably. Looks like the forums
> will permit up to a 488 KB zip file to be attached.

Yes, please always zip log files. And I've upped the limit for zip files
to 1.5MB.

--

Michael

[JJZolx]

So it's still scanning for changed files. Lot's of them (9.5k?). I
wonder whether the following log entry which is reported for every
"changed" item might be a hint:

[14-11-06 22:51:07.4154] Carp::Clan::__ANON__ (216) Warning:
DBIx::Class::ResultSet::single(): Query returned more than one row. SQL
that returns multiple rows is DEPRECATED for ->find and ->single at
/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema.pm line 2777

Are you talking about the multiple rows deprecated warning? People have been seeing those in the logs for years.

[JJZolx]

> A new & changed scan is actually logged to the server.log file, while

Not always: if you're using one of the importers implemented in a plugin
or virtual libraries it might always be run in the standalone scanner,
reporting to scanner.log. I think the FTS plugin would already cause
this. Doesn't it?

Ah, I keep forgetting this. I hadn't noticed that the fulltext plugin caused the standalone scanner to be run. Yes, I see that new & changed scans are now being logged to scanner.log.

[mherger]

? new version of 7.9 much slower to do a&quot;new &amp; changed&quot; scan

>> [14-11-06 22:51:07.4154] Carp::Clan::__ANON__ (216) Warning:
>> DBIx::Class::ResultSet::single(): Query returned more than one row. SQL
>> that returns multiple rows is DEPRECATED for ->find and ->single at
>> /<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema.pm line 2777
>
> Are you talking about the multiple rows deprecated warning? People have
> been seeing those in the logs for years.

That's true, but I don't remember having seen them in the scanner.log.

--

Michael