PDA

View Full Version : scaling a slimserver to 40+ squeezeboxes



bglad
2005-11-11, 09:19
do you know anyone who has run 40 squeezeboxes on one server? how about 200 s'boxes - is that feasible on one server? i'm assuming that users won't use the web interface, just the remote control; that all files are mp3; that all sboxes will be running simultaneously; and that they're all on a LAN

i'm looking for an idea of the server and network specs. any info or suggestions much appreciated

thanks
ben

fuzzyT
2005-11-11, 10:23
bglad wrote:
> do you know anyone who has run 40 squeezeboxes on one server?

synch'd, unsynch'd or mixed?

seanadams
2005-11-11, 10:24
I don't think 40 would be a problem, but at some point approaching 200 units, the streaming workload would probably need to be managed differently. The solution might be to hand that off to apache, possibly on another machine. This used to be a fairly easy hack in slimserver but I'm not sure what it takes to do it now - will let the server gurus comment. Also there are load balancing techniques for running multiple SlimServer processes on one machine. If they're streaming internet radio that will take significant load off, as this data comes straight from the internet.

Anything is possible, but it's hard to say what specific issues might come up, as I don't know of a customer installation of more than 40 units. If you run into any problems bringing up this installation, we can help.

bglad
2005-11-11, 10:36
fuzzyT - unsynched. It's a hotel, the idea is eventually to have one in every room

thanks sean - presumably one could also run two or more separate slimserver boxes sharing a nas drive?

seanadams
2005-11-11, 10:38
fuzzyT - unsynched. It's a hotel, the idea is eventually to have one in every room

thanks sean - presumably one could also run two or more separate slimserver boxes sharing a nas drive?

Wouldn't recommend low-end NAS for this - much slower than a PC.

radish
2005-11-11, 10:59
I'd think you'd want a number of server boxes with their own local drives mirroring the data. Disk throughput will be an issue and any NAS solution vaguely affordable will be slower than local disks.

MrC
2005-11-11, 11:12
Indeed - get enough hardware. If you care about your guests, you wouldn't want them to suffer dropouts while someone searches the library, for example.

Curious, is broadcasting music this way in a commercial environment legal where you live? Sure wouldn't want the music-police to visit your place and drop a suit on you!

ceejay
2005-11-11, 11:15
I'd think you'd want a number of server boxes with their own local drives mirroring the data. Disk throughput will be an issue and any NAS solution vaguely affordable will be slower than local disks.

True, but what we mean by "affordable" might change if you're buying 200 SB3s! Without speculating about the kind of price break Slim might give for this quantity, lets say $25k for the SBs - in which case $10k - $15k on the servers and storage might seem sensible - especially if you're looking for a robust environment (hotel guests hate seeing gadgets that don't work, its much worse than not having them at all).

In which case, why not configure 3 or 4 Linux blades linked up to a shared storage infrastructure?

And where is this hotel, and can we all get a discount?

Ceejay.

Jeff Coffler
2005-11-11, 12:09
Hi bglad,

> fuzzyT - unsynched. It's a hotel, the idea is eventually to have one in
> every room

I'd probably have separate subnets per every x rooms (perhaps per floor or
something), with a separate SlimServer per subnet. This has a host of
advantages, the most significant being: A broadcast storm would be
contained, and wouldn't bring your entire network to it's heals. You also
have easy, discrete control over what client connects to what server
(without mucking with it at boot time).

As others have said, NAS that's anything remotely affordable is slower than
local drives. So I'd just use rsync to syncronize the data between systems.
Thus, if you add something or change something on a "master" system, it
would just get copied over to the others at night or something.

-- Jeff

oreillymj
2005-11-11, 12:35
I'm assuming that your going to require at least 100mb switched ethernet.

How many 192kbs mp3 streams can a 100mbs card handle with a 30%-40% overhead to avoid maxing out the card.

The network also needs to have 20%-30% in reserve to avoid saturation from retries if remember correctly.

fuzzyT
2005-11-11, 12:47
Jeff Coffler wrote:

> As others have said, NAS that's anything remotely affordable is slower
> than local drives. So I'd just use rsync to syncronize the data between
> systems. Thus, if you add something or change something on a "master"
> system, it would just get copied over to the others at night or something.

You might want to take this approach to the entire server config and not
just the music files, (with distinct sync operations for the two of
course).

Especially with identical hardware, you should be able to get an image
of your OS and software install and use it for all of the servers. At
upgrade time, you'd get to a new stable configuration and then
distribute that image. Even if you didn't want to do this for the OS,
it might make sense for the SS setup and config.

And if you don't want to roll your own, take a look at the LiveCD
approach, "SlimCD".

<http://www.herger.net/slim/detail.php?nr=763>

--rt

meyergru
2005-11-11, 14:24
This seems like a very large load for just one server:

Network:

Let's first assume that you use the MP3 maximum bitrate of 320 kbit/s. This is equivalent to 40.000 bytes per second. With all 200 clients running, you have 8.000.000 bytes/s throughput, so that the network interface is nearly saturated, if you do not choose to use a gigabit interface, which would by a logical choice, since the backbone switch should be gigabit as well. Regarding your overall investment, this is not that much more expensive. Whatever, network throughput should be no problem.

Harddisk:

While the throughput of 8 MByte/s does not seem much at first sight with current harddisks running at 30 MByte/s at least (inner tracks), you have to remember, it is 200 clients causing that load, each probably playing a different song.

This means that with buffering at - say - 1 second, there would be 200 random seeks per second, each taking ~12ms for a current harddisk including latency and read. This would take 2.4 seconds, so it would not work. You have to make sure that buffering is at least 3 seconds, which equals around 128 KBytes. With a Linux server, you can control this read-ahead, regardless of application buffering.

However, with real-life assumptions, maybe it would work out just fine, since you don't have to use maximum bitrate and in a hotel, you'd rarely see more than half of the clients actually working.

Processing:

This does not mean there is no other K.O. criterium here, I don't know if the processor load caused by slimserver may be to high - after all, it's interpreted PERL...

At least with a large music database, I can not even play uninterrupted music for nearly one hour when the database is scanned on slimserver startup...

Maybe someone with more than a few clients can give figures about their CPU load factors?

seanadams
2005-11-11, 14:56
Keep in mind Squeezenetwork runs several hundred clients on each server PC - I don't have the number off the top of my head but I think it's over a thousand. This is almost entirely the same code base as SlimServer - the core performance considerations needed to make this work are already present in SlimServer.

Serving throughput is the first concern, but this should be relatively simple to alleviate.

Searching and building huge playlists is the other concern - to alleviate those you'd need to either load balance to multiple processes or disable searching perhaps.

I'm not worried at all about ethernet throughput. A gigabit port would be good for thousands of clients.

At any rate we won't know for sure until you start building it, but it is definitely possible. These are the most likely bottlenecks though.

meyergru
2005-11-11, 16:58
Keep in mind Squeezenetwork runs several hundred clients on each server PC - I don't have the number off the top of my head but I think it's over a thousand. This is almost entirely the same code base as SlimServer - the core performance considerations needed to make this work are already present in SlimServer.


But with Squeezenetwork, only the metainformation delivery part is used, not serving to the clients.



At any rate we won't know for sure until you start building it, but it is definitely possible. These are the most likely bottlenecks though.

Yep. And while a worst-case scenario may overload the setup, real-world usage with 200 potential clients should be no problem, even with searching, playlists and so on. Ought to be a real server with some punch, though, and not a NAS.

dean
2005-11-11, 17:27
On Nov 11, 2005, at 3:58 PM, meyergru wrote:
> seanadams Wrote:
>> Keep in mind Squeezenetwork runs several hundred clients on each
>> server
>> PC - I don't have the number off the top of my head but I think it's
>> over a thousand. This is almost entirely the same code base as
>> SlimServer - the core performance considerations needed to make this
>> work are already present in SlimServer.
>>
>
> But this is only the metainformation delivery part, not serving to the
> clients.
True. Luckily, the data serving part of SlimServer is quite
efficient (compared to the screen rendering, database access, etc.)
Even so, you will need more CPU power.

The thing I would worry about most is the single-threaded nature of
SlimServer for a large number of players.

One thing we do on the SqueezeNetwork is run multiple SlimServers on
a given machine and use a load balancer to assign players to server
processes.

A simpler solution for a fixed installation would be to have the
server configured with multiple IP addresses and bind server
processes to separate IP addresses via the command line options.
Then have your players configured to connect to specific server IP
addresses.

Finally, consider using multiple inexpensive server machines each
running a single instance of slimserver. A Redundant Array of
Inexpensive SlimServers.

-dean

bglad
2005-11-12, 02:19
thanks to all for the good thoughts. the one thing i didn't mention is that the hotel's in a fairly remote location so reliability, simplicity and remote admin are important. i lean towards dean's idea of a RAIS, but i'll throw all this at the guys who're doing the networking etc and see what fits

one thing concerns me whatever the architecture - rescanning for new music flogs the server... they'll want a 24/7 music service and to frequently add the latest cds - is there any way to lessen the load of rescans?

thanks again
ben

ceejay
2005-11-12, 02:45
Ok, so this may be a silly thought, but for this instance (and possibly for us more "normal" users too) ... would it be possible somehow to script a process to force slimserver to add specific items (ones you know you've just ripped and added to the filestore) to the database? Doesn't this happen if you browse through the music folders to get to new music?

That would avoid having to do a full rescan just to add that new CD you've just loaded.

If it can't be done with current commands and options, could we have one added? (eg a command line command to read a specific folder and add its contents to the database)

Ceejay

chrisla
2005-11-12, 09:07
Are the databases instance specific? Else you could:

- have a master server that no customer box calls
- copy new tracks to that
- scan on this box
- sync new tracks to slave servers
- sync DB files to new servers in a temp directory
- backup the old slave DB files to a temp file
- shutdown / restart to insert new DB file per requirements
- come back up with the new DB

If this could be made to work you could just have a brief blip in the
middle of the night when your slaves re-start. Seems it may work if
all of the paths are the same. Seems quite redundant to let all of the
box's CPU's churn on the same task.


-Chris

On 11/12/05, bglad <bglad.1ydmzb (AT) no-mx (DOT) forums.slimdevices.com> wrote:
>
> thanks to all for the good thoughts. the one thing i didn't mention is
> that the hotel's in a fairly remote location so reliability, simplicity
> and remote admin are important. i lean towards dean's idea of a RAIS,
> but i'll throw all this at the guys who're doing the networking etc and
> see what fits
>
> one thing concerns me whatever the architecture - rescanning for new
> music flogs the server... they'll want a 24/7 music service and to
> frequently add the latest cds - is there any way to lessen the load of
> rescans?
>
> thanks again
> ben
>
>
> --
> bglad
> ------------------------------------------------------------------------
> bglad's Profile: http://forums.slimdevices.com/member.php?userid=1052
> View this thread: http://forums.slimdevices.com/showthread.php?t=18098
>
>