PDA

View Full Version : Extremely long startup and rescan times



Mandarb
2012-10-04, 15:34
I am currently running Logitech Media Server (v7.7.3, 1336728576) on a
Debian/Unstable box, and starting the server, or rescanning the library
takes just over 2 hours. I have ~20k tracks, and a bunch of erland's
plugins installed.

The slow part seems to be when cache/persist.db gets (very slowly)
copied into cache/persist.db-wal. My persist.db is 187M, and it copies
around 20KB/sec. During this entire time one CPU is pegged at 100%.

strace shows kazillions of lines like:
lseek(21, 61262992, SEEK_SET) = 61262992
read(21, "\n\0\0\0\25\0\315\0\3=\3d\3\213\3\262\3\331\0\315\ 0\364\1\33\1B\1i\1\220\1\267"..., 1024) = 1024
lseek(20, 176803840, SEEK_SET) = 176803840
read(20, "\2\0\0\0\2\2y\0\0\2\177\360\3C\2y\0\0\0\0\0\0\0\0\ 0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(21, 35200280, SEEK_SET) = 35200280
read(21, "\2\0\0\0\4\0\233\0\0\2\177\357\0\233\2C\1Y\3\26\0x \0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(20, 155569152, SEEK_SET) = 155569152
read(20, "\2\0\0\0\4\0X\0\0\2K;\0X\1B\2,\3\26\2\206\0\0\0\0\ 0\0\0\0\0\0"..., 1024) = 1024
lseek(21, 62461880, SEEK_SET) = 62461880
write(21, "\0\1\355\224\0\0\0\0iP\327\35\3634\304\357$\270\22 1\301\213\372\30\262", 24) = 24
lseek(21, 62461904, SEEK_SET) = 62461904
write(21, "\n\0\0\0\6\0\232\0\1+\1\274\2M\2\336\3o\0\232\3\22 3\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(21, 56047096, SEEK_SET) = 56047096
read(21, "\2\0\0\0\3\1B\0\0\2>\200\1B\2,\3\26\3\26\0\340\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024

file id 20 is /var/tmp/squeezeboxserver/cache/persist.db, and 21 is
/var/tmp/squeezeboxserver/cache/persist.db-wal. This mirrors what an ls
of cache/persist* shows.

I tried moving the cache directory into a RAM disk to eliminate disk
issues, and it did not speed it up at all. Once it is up and running
performance is good, but the startup and rescan is a killer.

Anyone have any idea how to speed this up? Is MySQL still a viable
option given I rely on erland's plugins? I was thinking of tweaking
/usr/share/perl5/Slim/Utils/SQLiteHelper.pm to seriously increase
cache_size, or maybe just set page_size to a larger value (like 16k).

Anyone else see anything like this?

Thanks,
Omen

--
Confidence is the feeling you have before you understand the situation.

erland
2012-10-04, 21:27
If it's related to my plugins, you will find some ideas in the following thread:
http://forums.slimdevices.com/showthread.php?96124-Erland-s-Plug-ins-Large-Collections-and-Hardware-Requirements

Mandarb
2012-10-05, 21:49
Quoting erland <erland.5jy2sn (AT) no-mx (DOT) forums.slimdevices.com> on Thu, Oct 04 21:27:
>
>
> If it's related to my plugins, you will find some ideas in the following
> thread:
> http://forums.slimdevices.com/showthread.php?96124-Erland-s-Plug-ins-Large-Collections-and-Hardware-Requirements

I knew I forgot something important; the hardware is a beefy, lightly
loaded home server. It is a Xeon E31275 @ 3.40GHz (so 4 cores, 8 threads)
with 16GB of RAM. My primary disk, the one that stores the cache is an
Intel 510 SSD. I was thinking I was I/O bound so I moved the cache to a
RAM disk but that did not speed it up at all, so it is not an I/O issue.

I'll dig through that thread and post the debug info when I get a chance.
It sounds like turning off `musicbrainz tags' and maybe `Startup refresh'
and `Rescan refresh' would solve the issue. I don't often re-arrange
my music, but I would hate to lose the ratings if/when I do.

Do you still support MySQL, or it it SQLite only now?

Has anyone had ever set the page_size PRAGMA for SQLite? I am tempted
to bump that to 16k, which with cache_size = 20000 would give 312MB of
cache. I have the RAM, might as well use it.

It it worth moving from 7.7.3 to 7.8? The changelog looks fairly
small.

Thanks

Mnyb
2012-10-05, 21:56
Quoting erland <erland.5jy2sn (AT) no-mx (DOT) forums.slimdevices.com> on Thu, Oct 04 21:27:
>
>
> If it's related to my plugins, you will find some ideas in the following
> thread:
> http://forums.slimdevices.com/showthread.php?96124-Erland-s-Plug-ins-Large-Collections-and-Hardware-Requirements

I knew I forgot something important; the hardware is a beefy, lightly
loaded home server. It is a Xeon E31275 @ 3.40GHz (so 4 cores, 8 threads)
with 16GB of RAM. My primary disk, the one that stores the cache is an
Intel 510 SSD. I was thinking I was I/O bound so I moved the cache to a
RAM disk but that did not speed it up at all, so it is not an I/O issue.

I'll dig through that thread and post the debug info when I get a chance.
It sounds like turning off `musicbrainz tags' and maybe `Startup refresh'
and `Rescan refresh' would solve the issue. I don't often re-arrange
my music, but I would hate to lose the ratings if/when I do.

Do you still support MySQL, or it it SQLite only now?

Has anyone had ever set the page_size PRAGMA for SQLite? I am tempted
to bump that to 16k, which with cache_size = 20000 would give 312MB of
cache. I have the RAM, might as well use it.

It it worth moving from 7.7.3 to 7.8? The changelog looks fairly
small.

Thanks

Setting higher cache pragma is exactly what the " high mem" setting in LMS does ( look in the advanced / performance settings ).

Mandarb
2012-10-05, 22:06
Quoting Mnyb <Mnyb.5jzyun (AT) no-mx (DOT) forums.slimdevices.com> on Fri, Oct 05 21:56:
>
> Setting higher cache pragma is exactly what the " high mem" setting in
> LMS does ( look in the advanced / performance settings ).

I know. The "high mem" setting (which I have enabled) bumps the cache
from 2,000 pages to 20,000 pages. At 1KB page size (the default) that is
still only 20MB. Given that the I/O I posted from strace shows 1024 byte
reads and writes, bumping the page_size to 16KB would make there be 16x
fewer I/O operations. Leaving the "high mem" setting enabled with 16KB
page_size would give ~315MB of cache, significantly more than the 20MB
currently being used. The down side is that to change the page_size I
have to hack the code, and then completely wipe and regenerate the
databases. Still, it might be worth it.

Mnyb
2012-10-05, 22:38
Quoting Mnyb <Mnyb.5jzyun (AT) no-mx (DOT) forums.slimdevices.com> on Fri, Oct 05 21:56:
>
> Setting higher cache pragma is exactly what the " high mem" setting in
> LMS does ( look in the advanced / performance settings ).

I know. The "high mem" setting (which I have enabled) bumps the cache
from 2,000 pages to 20,000 pages. At 1KB page size (the default) that is
still only 20MB. Given that the I/O I posted from strace shows 1024 byte
reads and writes, bumping the page_size to 16KB would make there be 16x
fewer I/O operations. Leaving the "high mem" setting enabled with 16KB
page_size would give ~315MB of cache, significantly more than the 20MB
currently being used. The down side is that to change the page_size I
have to hack the code, and then completely wipe and regenerate the
databases. Still, it might be worth it.

Yes you have too, but the code hack is only one line I think it is in SQLiteHelper.pm or ? Or something like it ( not at home can't peek into my server rigth now ) this will be wiped at every upgrade .
But do you really need to rescan ? Just restart LMS and see if it helps .

pallfreeman
2012-10-06, 02:07
Yes you have too, but the code hack is only one line I think it is in SQLiteHelper.pm or ?

Oh, FFS. Groundhog Day.

The only way to fix this is have cache size as a pref, because we'd never agree on a default. Yeah?

Mnyb
2012-10-06, 14:40
Oh, FFS. Groundhog Day.

The only way to fix this is have cache size as a pref, because we'd never agree on a default. Yeah?

Yes , but for the latest years it has been an absolute no no to introduce more prefs or settings to the system :)

A sure fire way to kill any enhancement request has been to suggest that it should have a preference or settings :)

My pow you can have an astonishing amount of settings , but have sane defaults and hid the settings in advanced menus ,win win

Mandarb
2012-10-06, 17:10
I'll dig through that thread and post the debug info when I get a chance.
It sounds like turning off `musicbrainz tags' and maybe `Startup refresh'
and `Rescan refresh' would solve the issue. I don't often re-arrange
my music, but I would hate to lose the ratings if/when I do.


Definitely musicbrainz:



[12-10-05 22:18:40.1197] Plugins::TrackStat::Storage::refreshTracks (1236) TrackStat: Synchronizing TrackStat data, please wait...
[12-10-05 22:18:40.1200] Plugins::TrackStat::Storage::refreshTracks (1286) Starting to update urls in statistic data based on musicbrainz ids
[12-10-06 04:05:41.1529] Plugins::TrackStat::Storage::refreshTracks (1365) Finished updating urls in statistic data based on musicbrainz ids, updated 179564 items : It took 20821.032768 seconds


The funny thing is that I only have "1049 albums with 21109 songs by 327 artists", but I have 179564 musicbrainz ids.

I have bumped the cache_size to 400,000 pages, to see if that would speed it up, but it does not look like it. The persist.db-wal file is growing at a rate of ~25KB/sec, so no real improvement. Time to bump the page_size to see if that helps.

erland
2012-10-06, 22:03
Definitely musicbrainz:



[12-10-05 22:18:40.1197] Plugins::TrackStat::Storage::refreshTracks (1236) TrackStat: Synchronizing TrackStat data, please wait...
[12-10-05 22:18:40.1200] Plugins::TrackStat::Storage::refreshTracks (1286) Starting to update urls in statistic data based on musicbrainz ids
[12-10-06 04:05:41.1529] Plugins::TrackStat::Storage::refreshTracks (1365) Finished updating urls in statistic data based on musicbrainz ids, updated 179564 items : It took 20821.032768 seconds


The funny thing is that I only have "1049 albums with 21109 songs by 327 artists", but I have 179564 musicbrainz ids.

I have bumped the cache_size to 400,000 pages, to see if that would speed it up, but it does not look like it. The persist.db-wal file is growing at a rate of ~25KB/sec, so no real improvement. Time to bump the page_size to see if that helps.

Try installing the free Database Query plugin and run the "TrackStat inconsistentcy/problems" query and the "Squeezebox Server Inconsistency/Problems", it sounds like the "Duplicate entries in TrackStat statistics" and "Duplicate musicbrainz song tags" entries are going to indicate issues in your setup.

Then if you have a lot of duplicate musicbrainz ids/TrackStat entries, which I suspect you do, follow the instructions in the following post:
http://forums.slimdevices.com/showthread.php?96124-Erland-s-Plug-ins-Large-Collections-and-Hardware-Requirements&p=717268&viewfull=1#post717268

Basically it's caused by a change musicbrainz did 2010 which results in that musicbrainz ids aren't unique any more, the track on a normal album have the same identity as the same track on a compilation album. After this change the TrackStat musicbrainz logic can't be used because it will result in that the duplicated musicbrainz entries will be duplicated another time each time you perform a rescan or restart the server and this will eventually make the problem even worse. To get rid of the duplicated entries you unfortunately need to make a backup/clear/restore operation as described in the above post and after the restore you can disable the musicbrainz support in TrackStat settings page.

Mandarb
2012-10-07, 12:44
Quoting erland <erland.5k1tsn (AT) no-mx (DOT) forums.slimdevices.com> on Sat, Oct 06 22:03:
>
> Try installing the free Database Query plugin and run the "TrackStat
> inconsistentcy/problems" query and the "Squeezebox Server
> Inconsistency/Problems", it sounds like the "Duplicate entries in
> TrackStat statistics" and "Duplicate musicbrainz song tags" entries are
> going to indicate issues in your setup.

That was indeed the problem. I turned off `Enable musicbrainz tags' and
dropped and restored the stats and startup speed is now 9 seconds rather
than 2-4 hours. It is a shame to lose musicbrainz support, but I would
rather live without it than live with the long startup/rescan times.

Thank you for your help!

Omen

Bradley
2012-10-10, 10:40
I was experiencing this problem and was looking for anyone else experiencing the same thing.

Your fix works perfectly! Thanks for posting it!

*B

th80
2012-10-11, 10:53
I was experiencing this problem and was looking for anyone else experiencing the same thing.

Your fix works perfectly! Thanks for posting it!

*B
Thanks Bradley for pointing me to this thread. I had and posted the same issue in thread http://forums.slimdevices.com/showthread.php?96659-LMS-problems-on-daily-rescan-with-TrackStat

The fix worked for me too!

Thanks
Thomas