Performance measurements ?

**mherger** · 2005-10-13, 14:03

Performance measurements ?

> This data has to be taken for what it is: a very imprecise, but
> possibly useful ballpark figure about how fast SlimServer responds.

Quite true. I got interesting results (connecting to my server 150km
further southwest using SSH :-)):

[0.375] Artists: 433
[0.032] Albums : 544
[0.031] Songs : 6543
[0.031] Genres : 50
[20.891] titles 0 10
[16.235] titles 0 100
[29.329] titles 0 1000
[114.393] titles 0 10000

Interestingly "titles 0 10" was slower than "titles 0 100" three out of
four times I run the test. (BTW: what does that command actually do?)

This is on a Via C3/1GHz/512MB running SME Linux.

> The output of this file on my laptop (the starting value is the time in
> seconds it took to execute the command):

Maybe you should give some numbers about you configuration (CPU, RAM).

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
SlimString Translation Helper (http://www.herger.net/slim/)

**Niek Jongerius** · 2005-10-13, 16:11

Performance measurements ?

> Quite true. I got interesting results (connecting to my server 150km
> further southwest using SSH :-)):
>
> [0.375] Artists: 433
> [0.032] Albums : 544
> [0.031] Songs : 6543
> [0.031] Genres : 50
> [20.891] titles 0 10
> [16.235] titles 0 100
> [29.329] titles 0 1000
> [114.393] titles 0 10000
>
> Interestingly "titles 0 10" was slower than "titles 0 100" three out of
> four times I run the test. (BTW: what does that command actually do?)

According to the CLI tech docs, it returns songs (with the first number
as the start number, and the second one the maximum number of titles
returned, so "titles 0 10" gives you the first 10 songs).

> This is on a Via C3/1GHz/512MB running SME Linux.
>
>> The output of this file on my laptop (the starting value is the time in
>> seconds it took to execute the command):
>
> Maybe you should give some numbers about you configuration (CPU, RAM).

This was on a P4 2.8 GHz with 1 GB RAM, running 5 instances of SlimServer.
The songs themselves are on an iPod, but that probably doesn't matter for
the times reported, as they interact with the database.

Niek.

**Bill Burns** · 2005-10-13, 16:22

Performance measurements ?

Took me a minute to find out that the CLI port is 9090!

Here's my results over local network. Slimserver on P4/3GHz/1GB/WinXP,
streaming an internet radio station to an SB1 during these tests with
nothing else running on the server.

[0.109] Artists: 503
[0.000] Albums : 818
[0.000] Songs : 7068
[0.000] Genres : 93
[47.063] titles 0 10
[12.500] titles 0 100
[21.453] titles 0 1000
[58.562] titles 0 10000

Second pass:

[9.875] titles 0 10
[10.609] titles 0 100
[27.500] titles 0 1000
[55.391] titles 0 10000

Third pass:

[10.047] titles 0 10
[14.328] titles 0 100
[20.125] titles 0 1000
[66.390] titles 0 10000

Here "titles 0 10" was slower only on the first pass.

--
Bill Burns
Long Island NY USA
mailto:billb (AT) ftldesign (DOT) com

**MrC** · 2005-10-13, 16:51

Some food for though...

Second pass:
[55.391] titles 0 10000

Third pass:
[66.390] titles 0 10000

With a 20% variance, we can see that the testing methodology and environment is very rough.

And, with Niek running a P4/2.8 getting

[24.201] titles 0 10000

and Bill running a P4/3 getting an average of

[60.114] titles 0 10000

There's an almost 2.5x difference in times running on hardware where hardware specs alone would account roughly for only 10% difference.

Hopefully nobody will look at these data points and attempt to draw unwarranted conclusions.

**wr420** · 2005-10-13, 17:03

Performance measurements ?

Can you post the source? Need to run on Solaris.
Linux version errors. Maybe I'm doing something wrong?
I renamed settime.bin to settime.
Set execute permissions
ssettime is in my path
slimserver.pl is in my path as slimserver.pl and as slimserver
user can read input file.
If I were to run it from a windows or linux box and point it at my slim
server across the network would that effect the results?

-bash-3.00# sstime 192.168.1.35 9090 sstime.txt
-bash: /usr/bin/sstime: Invalid argument

Thanks

The information contained in this e-mail is strictly confidential and for the
intended use of the addressee only; it may also be legally privileged and/or
price sensitive. Notice is hereby given that any disclosure, use or copying
of the information by anyone other than the intended recipient is prohibited
and may be illegal. If you have received this message in error, please
notify the sender immediately by return e-mail. All e-mail sent to this
address will be received by Acacia Pacific Holding's e-mail system and is
subjected to archiving and review by someone other than the recipient.

Acacia Pacific Holdings has taken every reasonable precaution to ensure that
any attachment to this e-mail has been swept for viruses. We accept no
liability for any damage sustained as a result of software viruses and
advise you carry out your own virus checks before opening any attachment.

**Philip Meyer** · 2005-10-13, 19:34

Performance measurements ?

> sstime <SlimServer_IP> <CLI_port> <inputfile>
>
I can't get this to work under XP.

I get the following output:

[0.000] Artists:
Error reading from SlimServer!

I'm running 6.2 trunk.

Phil

**mherger** · 2005-10-13, 19:37

Performance measurements ?

>> sstime <SlimServer_IP> <CLI_port> <inputfile>
>>
> I can't get this to work under XP.

You'll have to open port 9090 on the server's firewall

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

**Philip Meyer** · 2005-10-13, 20:19

Performance measurements ?

>You'll have to open port 9090 on the server's firewall
Windows firewall is disabled. I am running Sygate Personal Firewall, which lists port 9090 as allowed for perl.exe.

I can connect via telnet to localhost 9090, but as soon as I type anything and press return, I get a disconnect with no output.

Phil

**mherger** · 2005-10-13, 20:48

Performance measurements ?

>> You'll have to open port 9090 on the server's firewall
> Windows firewall is disabled. I am running Sygate Personal Firewall,
> which lists port 9090 as allowed for perl.exe.

It's not perl.exe which is accessing port 9090, but sstime.exe. Put the
firewall in learning mode (or disable it entirely to be sure).

> I can connect via telnet to localhost 9090, but as soon as I type
> anything and press return, I get a disconnect with no output.

I'd really say this is the firewall blocking access to this port.

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

**Michaelwagner** · 2005-10-13, 23:27

Yeah, you really need to be careful about methodology here.

If you want to select typical things and get typical response times, you probably need to carefully think out the typical things people do at the user interface and mimic them in the CLI and run them on many different configurations.

I doubt many people list the top thousand songs when they're at the remote.

If you want to benchmark the code to do before and after studies of code improvements, you need to have one typical machine, benchmark it accurately AND THEN FREEZE IT AND DON'T CHANGE IT. That almost means dedicate it to the benchmarking task and making it a reference system, because you never know when installing Microsoft Office 2007 (heaven help us) or IE 17.2 won't change the way I/O works or how much background activity there is cluttering up the disk.

**Niek Jongerius** · 2005-10-14, 07:00

Re: Performance measurements ?

>> Second pass:
>> [55.391] titles 0 10000
>>
>> Third pass:
>> [66.390] titles 0 10000
>>
> With a 20% variance, we can see that the testing methodology and
> environment is very rough.
>
> And, with Niek running a P4/2.8 getting
>
> [24.201] titles 0 10000
>
> and Bill running a P4/3 getting an average of
>
> [60.114] titles 0 10000
>
> There's an almost 2.5x difference in times running on hardware where
> hardware specs alone would account roughly for only 10% difference.
>
> Hopefully nobody will look at these data points and attempt to draw
> unwarranted conclusions.

Agreed. There are too many variables in the way machines are set up to
readily compare output numbers. CPU and RAM are by no means the only
variables here. OS, procs running, procs priority, intermediate network
(if test prog is run over a network) etc.

But bottom line still is that it takes that reported amount of time for
the SlimServer to cough up the data requested (assuming the CLI uses
comparable ways in getting the data). If Bill is getting 2.5 times worse
performance in the same tests as I get, I would assume his setup performs
about that factor worse than mine when serving a SqueezeBox. The proggy
does nothing fancy (I'll post the source in a minute on my site), it
just times the start and end of the CLI command.

I have not been very inventive in the queries I posed in my sample input
file. It could be that my example commands are somehow not representable
for gauging performance. Someone with a better understanding of what
actually are reasonable queries could maybe give a few. It's just a matter
of editing the input file to test other CLI commands.

Niek.

**Niek Jongerius** · 2005-10-14, 07:08

Performance measurements ?

> Can you post the source? Need to run on Solaris.

Wil do in a mo.

> Linux version errors. Maybe I'm doing something wrong?
> I renamed settime.bin to settime.
> Set execute permissions
> ssettime is in my path
> slimserver.pl is in my path as slimserver.pl and as slimserver
> user can read input file.
> If I were to run it from a windows or linux box and point it at my slim
> server across the network would that effect the results?

Well, it could affect times reported if the network is dog slow, but the
logics of the program should work. I've tested the Windows version over
a VMWare "network" on my machine.

> -bash-3.00# sstime 192.168.1.35 9090 sstime.txt
> -bash: /usr/bin/sstime: Invalid argument

The version you have has been compiled on a SuSE 9.3. It probably compiled
against dynamic libraries, so it assumes some version of runtime libs.
You could try running like so:

strace sstime 192.168.1.35 9090 sstime.txt

This may give an indication what went wrong. I'll post the source on my
site in a minute.

Niek.

**Niek Jongerius** · 2005-10-14, 07:11

Performance measurements ?

>>You'll have to open port 9090 on the server's firewall
> Windows firewall is disabled. I am running Sygate Personal Firewall,
> which lists port 9090 as allowed for perl.exe.
>
> I can connect via telnet to localhost 9090, but as soon as I type anything
> and press return, I get a disconnect with no output.

This is the reason sstime will not work either. It does exactly that.
If you cannot get a manual telnet to 9090 to work, the test program will
fail as well.

Niek.

**Niek Jongerius** · 2005-10-14, 07:20

Re: Performance measurements ?

>
> Yeah, you really need to be careful about methodology here.
>
> If you want to select typical things and get typical response times,
> you probably need to carefully think out the typical things people do
> at the user interface and mimic them in the CLI and run them on many
> different configurations.
>
> I doubt many people list the top thousand songs when they're at the
> remote.
>
> If you want to benchmark the code to do before and after studies of
> code improvements, you need to have one typical machine, benchmark it
> accurately AND THEN FREEZE IT AND DON'T CHANGE IT. That almost means
> dedicate it to the benchmarking task and making it a reference system,
> because you never know when installing Microsoft Office 2007 (heaven
> help us) or IE 17.2 won't change the way I/O works or how much
> background activity there is cluttering up the disk.

Note that I did not intend this to be a benchmark tool. In a lot of posts
on this list people said the performance of their install was <insert your
favourite speed indication here>, which was a very subjective indication.
This program simply times a request to the CLI. It should give some idea
about what a user would see if using a real SqueezeBox (assuming we use a
reasonable set of CLI queries, which I probably don't).

There are even some of us desparately switching OSes and tweaking stuff
on the same machine and discussing whether ActiveState is faster than
compiled Windows or CygWin or whatever. This tool then is able to give
_some_ numbers. Again, you cannot compare them 1 on 1 to other installs,
but IMHO if one install does someting in 20 seconds, and another one in
just 2, there is going to be the same user experience when connecting and
using a SqueezeBox.

Niek.

Performance measurements ?

Performance measurements ?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment