PDA

View Full Version : how do i debug logitechmediaserver-7.9.1 on CentOS 7



ezekieldas
2017-11-12, 21:34
At startup I see these three items in /var/log/squeezeboxserver/server.log. Meanwhile the squeezeboxserver proc is burning through CPU and consumes all swap until the proc crashes. web interface never becomes available. The only fix I have in getting LMS up is to remove the contents of /var/lib/squeezeboxserver/cache and start over.

How can I debug / fix this? wiki notes for troubleshooting are obselete.

[17-11-12 20:21:52.7302] main::init (387) Starting Logitech Media Server (v7.9.1, 1509357021, Mon Oct 30 10:09:05 UTC 2017) perl 5.016003 - x86_64-linux-thread-multi
[17-11-12 20:21:52.9551] Slim::Utils::SQLiteHelper::postConnect (374) Optimizing DB because of missing or empty sqlite_stat1 table
[17-11-12 20:22:35.7074] Slim::Schema::forceCommit (2149) Warning: Trying to commit transactions before DB is initialized!

mherger
2017-11-12, 22:58
> At startup I see these three items in
> /var/log/squeezeboxserver/server.log. Meanwhile the squeezeboxserver
> proc is burning through CPU and consumes all swap until the proc
> crashes. web interface never becomes available. The only fix I have in
> getting LMS up is to remove the contents of
> /var/lib/squeezeboxserver/cache and start over.

Are you using any 3rd party plugins?

You could modify /etc/sysconfig/squeezeboxserver to add --d_startup to
SQUEEZEBOX_ARGS. This would give some more information in the server.log
file.

When you do wipe the cache folder and restart, what files would be there?

--

Michael

ezekieldas
2017-11-13, 11:21
Hi Michael. Thanks for responding.

1) plugins: I have spotty (thank you very very much btw), MusicWalkWithMe, NPR, BandCamp plugins installed.
2) --d_startup: I added this to SQUEEZEBOX_ARGS yet verbosity of log output remains the same. Is order important? I was able to get the proc running again so I pasted the running value below.
3) cache: Previously I was just removing the entire contents of the cache folder. What offered success this time around was more limited:

# /var/lib/squeezeboxserver/cache>find . -name "*.db" -exec ls -lh {} \;
-rw-r--r--. 1 squeezeboxserver squeezeboxserver 3.1M Nov 12 20:18 ./cache.db
-rw-r--r--. 1 squeezeboxserver squeezeboxserver 9.3G Nov 12 20:18 ./library.db
-rw-r--r--. 1 squeezeboxserver squeezeboxserver 13K Nov 12 20:18 ./persist.db
-rw-r--r--. 1 squeezeboxserver squeezeboxserver 235K Nov 12 20:18 ./artwork.db
-rw-r--r--. 1 squeezeboxserver squeezeboxserver 4.0K Nov 12 08:24 ./imgproxy.db
-rw-r--r--. 1 squeezeboxserver squeezeboxserver 175K Nov 12 20:18 ./bandcamp.db

# /var/lib/squeezeboxserver/cache>rm library.db

# /etc/init.d/squeezeboxserver start

# ps auwx | grep squeeze

squeeze+ 28891 0.5 1.4 393156 113232 ? Ss 09:58 0:01 /usr/bin/perl /usr/libexec/squeezeboxserver --d_startup --daemon --prefsdir=/var/lib/squeezeboxserver/prefs --logdir=/var/log/squeezeboxserver --cachedir=/var/lib/squeezeboxserver/cache --charset=utf8

-----

I know very little about sqllite but it looks like it's having a really difficult time loading a 9Gb table. Does it attempt to load library.db in entirety, or does it just reference that when necessary? It would seem to me the latter. Anyhow, here's a summary of what's going on. I can reproduce this consistently and I could probably do it across systems.

1. Start LMS, point to media folder /opt/mp3 (450Gb)
2. Media Library Scan creates library.db of considerable size (I have previous cache folders of 4, 9, 12Gb)
3. I attempt to restart LMS. I see the log messages posted previously + kswapd goes crazy, swap is consumed in entirety. The squeezeboxserver proc dies quietly.
4. I can repeat 3. until I remove library.db

Roland0
2017-11-13, 12:07
I know very little about sqllite but it looks like it's having a really difficult time loading a 9Gb table. Does it attempt to load library.db in entirety, or does it just reference that when necessary?

While your assumption is correct, your library DB size is extremely large, and this quite likely is the issue (for comparison, my library.db is ~250MB (~120MB without full text search enabled), and my music library is probably larger (judging from on-disk size of your media folder))
While you could try to get LMS to work with it (e.g. change the memory settings in the Performance section of the server settings), you may first want to try to find out what causes this.
If you delete your library.db and do a rescan, is it already that large?
Then, try again without any plugins active (by adding --failsafe to the startup parameters)

mherger
2017-11-13, 12:38
Thanks Roland0. You just saved me a lot of typing :-).

ezekieldas - what hardware are you running your LMS on? Could there be
symlinks in your music library?
--

Michael

ezekieldas
2017-11-13, 13:43
My hardware is just commodity x86 stuff. It would seem to me the source of the issue is the processing and/or representation of the music library. Why would 450Gb be a problem? There may be a few links in that tree but nothing excessive. Can this not handle symlinks? I'd like to do this: "change the memory settings in the Performance section of the server settings" Can you tell me where to do that so I don't get lost digging?

Looks like library.db hasn't been written to for the past 30 min and there's no scanning proc running. library.db is at 11Gb. I'm tempted to restart squeezeboxserver to see what happens at this point but I'll wait to see if you have any suggestions?

mherger
2017-11-13, 14:08
> My hardware is just commodity x86 stuff. It would seem to me the source
> of the issue is the processing and/or representation of the music
> library. Why would 450Gb be a problem?

The size in GB doesn't really matter. What is important is the number of
files.


> There may be a few links in that
> tree but nothing excessive. Can this not handle symlinks?

It should. But maybe they're creating loops back, triggering a bug in
LMS which would result in a never ending scan?


> I'd like to
> do this: "change the memory settings in the Performance section of the
> server settings" Can you tell me where to do that so I don't get lost
> digging?

Shut down LMS. Edit server.prefs (likely in /var/lib/squeezebox/prefs).
Set the value for dbhighmem: 2 (assuming you have 1GB of RAM or more).


--

Michael

Roland0
2017-11-13, 14:13
Why would 450Gb be a problem?
It's not - mine is larger, and others much larger.



There may be a few links in that tree but nothing excessive. Can this not handle symlinks?

It can - just make sure there's no loop (use find -L /path to check)



I'd like to do this: "change the memory settings in the Performance section of the server settings" Can you tell me where to do that so I don't get lost digging?

WebUI/settings/advanced/Performance/Database Memory Config

ezekieldas
2017-11-13, 14:52
Hello and thanks. I modified dbhighmem to reflect 2 but this had no positive effect. Same behavior as described above. While I understand sym and hard links I don't entirely understand what 'find -L' is doing. The manual page isn't making sense to me. I'm unsure if these examples are telling me everything under /opt/mp3 is a link?


#/opt/mp3>find -L /opt/mp3 | wc -l
57985
#/opt/mp3>find /opt/mp3 | wc -l
57985
#/opt/mp3>find /opt/mp3 -type f | wc -l
52911
#/opt/mp3>find /opt/mp3 -type d | wc -l
5072

Roland0
2017-11-13, 15:12
While I understand sym and hard links I don't entirely understand what 'find -L' is doing. The manual page isn't making sense to me.

See here (https://serverfault.com/questions/265598/how-do-i-find-circular-symbolic-links) and here (https://unix.stackexchange.com/questions/99159/is-there-an-algorithm-to-decide-if-a-symlink-loops)



I'm unsure if these examples are telling me everything under /opt/mp3 is a link?

No, this would: find /opt/mp3 -type l | wc -l
but isn't really relevant for the purpose of finding loops

Jeff07971
2017-11-13, 15:19
Have you got SELinux enabled ? Anything in the audit log ?

If you have got SELinux enabled try disabling with "setenforce 0" and try restarting LMS

I had some troubles with SElinux and LMS (an LOTS of other things) was not LMS itself but where the media was

ezekieldas
2017-11-13, 17:33
This worked best for me in finding loops 'find . -follow -printf ""' I was able to isolate some files and moved them out of /opt/mp3 (see below). I then restarted squeezeboxserver and library.db appears to remain at a manageable size:

-rw-r--r--. 1 squeezeboxserver squeezeboxserver 112M Nov 13 15:01 library.db

Before closing this thread I restarted squeezeboxserver one last time. It stops/starts as expected now.


# /opt/mp3.TRASH>find . -follow -printf ""
[ ... ]
find: File system loop detected; ‘./Brian Eno/Brian Eno - Another Day On Earth (FLAC)/05 - Caught Between.flac/proc/self/task/28068/cwd’ is part of the same file system loop as ‘.’.
find: File system loop detected; ‘./Brian Eno/Brian Eno - Another Day On Earth (FLAC)/05 - Caught Between.flac/proc/self/task/28068/root’ is part of the same file system loop as ‘./Brian Eno/Brian Eno - Another Day On Earth (FLAC)/05 - Caught Between.flac’.
find: File system loop detected; ‘./Brian Eno/Brian Eno - Another Day On Earth (FLAC)/05 - Caught Between.flac/proc/self/fd/3’ is part of the same file system loop as ‘.’.
[ ... ]

# /opt/mp3>find . -follow -printf ""

(returns nothing)

:-)

Thanks for the attention on this.