PDA

View Full Version : slimserver slowness and bloating w/ large number of files



R Neill
2004-03-31, 06:54
i love my sqeezebox to bits, however i have one large problem. i have
approximately 45k mp3s (my own collection, not downloads) and the slimserver
seems to be quite stressed. i am also pointing to itunes which may exacerbate
the situation.

symptoms:
* massive slimserver process (approaches 200mb)
* constant grinding of itunes library file
* playlists not showing up until end or parsing (this can take over an hour)
* cpu hog
* anytime the itunes file changes, file is scanned again and playlists drop off
until completed

my first approach was to throw hardware at the problem, upgraded from 512mb to
1gb of ram and upgraded cpu from athlon xp 1800 to 2600. this helped with some
latency when swithing between applications and when running several other ram
intensive applications, but seemed to do little to improve the performance of
the server.

to me it appears that the server is very inefficient in both parsing and
storing the data from the itunes file. with the the itunes file being 50mb in
size, a 200mb process size for slimserver seems a bit out of whack.

thoughts:
* could the parsing be adjusted so that playlists are parsed first
* on a reparse, could the default behavior be to only parse playlists (full
parses to require a specific command from user)
* re-evaluation of the xml parsing algo to improve performance
* re-evaluation of the servers data structures for keeping track of the files
to reduce memory footprint


does anyone think the following would help me at all:
* set up a defragger that would keep the itunes db file continuously defragged
* upgrade c: hd from wd 7200rpm to a raptor 10k rpm
* some sort of ram drive configuration that could load the itunes file


comments? suggestions?
-rolf




__________________________________
Do you Yahoo!?
Yahoo! Finance Tax Center - File online. File on time.
http://taxes.yahoo.com/filing.html

dean
2004-03-31, 08:47
Hi Rolf,

If you are playing music with iTunes at the same time that SlimServer
is running, then it's likely that SlimServer is rescanning your music
library every minute or so. If your iTunes Music Library is large (and
45k tracks is large for iTunes), then this can bog down the server as
it's probably constantly reloading. Try quitting iTunes and see if
things settle down after the scan is completed.

Another solution would be to click off "use iTunes" in the server
settings and have SlimServer scan the files directly. This means that
you'd need to manually choose to rescan and you'd lose access to your
iTunes playlists.

The next pre-release version of SlimServer will have an Additional
Server Setting to adjust the amount of time the server waits before
reloading a changed iTunes Music Library, this might also help your
situation.

Also, 200MB is way too high for the memory usage. How big is your
iTunes Music Library.xml file?

-dean



On Mar 31, 2004, at 5:54 AM, R Neill wrote:

> i love my sqeezebox to bits, however i have one large problem. i have
> approximately 45k mp3s (my own collection, not downloads) and the
> slimserver
> seems to be quite stressed. i am also pointing to itunes which may
> exacerbate
> the situation.
>
> symptoms:
> * massive slimserver process (approaches 200mb)
> * constant grinding of itunes library file
> * playlists not showing up until end or parsing (this can take over an
> hour)
> * cpu hog
> * anytime the itunes file changes, file is scanned again and playlists
> drop off
> until completed
>
> my first approach was to throw hardware at the problem, upgraded from
> 512mb to
> 1gb of ram and upgraded cpu from athlon xp 1800 to 2600. this helped
> with some
> latency when swithing between applications and when running several
> other ram
> intensive applications, but seemed to do little to improve the
> performance of
> the server.
>
> to me it appears that the server is very inefficient in both parsing
> and
> storing the data from the itunes file. with the the itunes file being
> 50mb in
> size, a 200mb process size for slimserver seems a bit out of whack.
>
> thoughts:
> * could the parsing be adjusted so that playlists are parsed first
> * on a reparse, could the default behavior be to only parse playlists
> (full
> parses to require a specific command from user)
> * re-evaluation of the xml parsing algo to improve performance
> * re-evaluation of the servers data structures for keeping track of
> the files
> to reduce memory footprint
>
>
> does anyone think the following would help me at all:
> * set up a defragger that would keep the itunes db file continuously
> defragged
> * upgrade c: hd from wd 7200rpm to a raptor 10k rpm
> * some sort of ram drive configuration that could load the itunes file
>
>
> comments? suggestions?
> -rolf
>
>
>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Finance Tax Center - File online. File on time.
> http://taxes.yahoo.com/filing.html
>

ethan@audio-crusade.com
2004-03-31, 09:37
Well I won't know more until my wife resets the box but something bad
happened to it after I left for work this morning. Then only thing in my
env that changed is I had slimserver.pl running. I got my sqeezebox last
night and set it up this morning.

I'm running the 2.6.4 kernel and perl 5.8.3.

Has anyone else experienced a lock up or hung network? The NIC in the box
is a RTL8139 and my router is a wrt54g with the latest firmware. It's not
the router though cause my wife's machine's wireless connection is fine.
The problem is localized to the linux box running slimserver.pl.

-e

kdf
2004-03-31, 10:54
Quoting R Neill <stuff212 (AT) yahoo (DOT) com>:

> i love my sqeezebox to bits, however i have one large problem. i have
> approximately 45k mp3s (my own collection, not downloads) and the slimserver
> seems to be quite stressed. i am also pointing to itunes which may
> exacerbate
> the situation.
>
> symptoms:
> * massive slimserver process (approaches 200mb)
> * constant grinding of itunes library file
> * playlists not showing up until end or parsing (this can take over an hour)
> * cpu hog
> * anytime the itunes file changes, file is scanned again and playlists drop
> off
> until completed
>
> my first approach was to throw hardware at the problem, upgraded from 512mb
> to
> 1gb of ram and upgraded cpu from athlon xp 1800 to 2600. this helped with
> some
> latency when swithing between applications and when running several other
> ram
> intensive applications, but seemed to do little to improve the performance
> of
> the server.
>
> to me it appears that the server is very inefficient in both parsing and
> storing the data from the itunes file. with the the itunes file being 50mb
> in
> size, a 200mb process size for slimserver seems a bit out of whack.

the iTunes library isn't the only thing cached in memory. It is simply parsed
for what information is has. This is then used to grab more information from
the files, and to organise the data into what slimserver needs.

> thoughts:
> * could the parsing be adjusted so that playlists are parsed first
> * on a reparse, could the default behavior be to only parse playlists (full
> parses to require a specific command from user)
> * re-evaluation of the xml parsing algo to improve performance

I tried this once, using an off-the-shelf parser: XML::Simple. Overall load in
was faster, but since it leaves no room for other processes, the players end up
being dumped. I could find no way to schedule it like the built-in server scans.

> * re-evaluation of the servers data structures for keeping track of the
> files
> to reduce memory footprint

That's on the list. Sadly, new features end up requiring even more data to be
cached in memory.

> does anyone think the following would help me at all:
> * set up a defragger that would keep the itunes db file continuously
> defragged
> * upgrade c: hd from wd 7200rpm to a raptor 10k rpm
> * some sort of ram drive configuration that could load the itunes file
>

What version of the server are you using? Prior to 5.1.1, the artwork loading
was causing the iTunes scan to be very slow. This has improved by about a
factor of 8 in 5.1.1. If you run slimserver with iTunes shut down, does the
performance improve?

-kdf

Rob Funk
2004-03-31, 13:56
kdf wrote:
>> * re-evaluation of the xml parsing algo to improve performance
>
> I tried this once, using an off-the-shelf
parser: XML::Simple. Overall load in
> was faster, but since it leaves no room for other processes, the players
end up
> being
dumped. I could find no way to schedule it like t he built-in server scans.

Maybe use something like the xml2perl script I posted for the Television
plugin: have the server fork/exec that preprocessor, and when the
preprocessor is done have the server load the result.

I may be misunderstanding the problem since I don't use iTunes, but it seems
to me that parsing XML is best done outside of the server.

--
==============================| "A slice of life isn't the whole cake
Rob Funk <rfunk (AT) funknet (DOT) net> | One tooth will never make a full grin"
http://www.funknet.net/rfunk | -- Chris Mars, "Stuck in Rewind"

Ethan Alpert
2004-03-31, 21:34
Turns out it was the firmware upgrade to my wrt54g, which I might add I only
did cause the FAQ recommended it. My linux box had enjoyed the same DHCP
assigned IP address for nearly 4 months of uptime. Since I moved into my
current house. Not once did it not get the IP it had previously been
assigned. Suddenly this morning at 9am it's lease expired and out of the
blue the router sends it a new one.

Anyhow slimserver.pl had nothing directly to do with me being unable to
reach my box all day.

Sorry for the false alarm.

-ethan