PDA

View Full Version : Slimserver footprint, time and space, and an architectural suggestion



Michaelwagner
2005-05-28, 11:33
I'm not sure the developers forum is the right place for this, but ... no other place seemed more suitable, so ... here goes.

I'm trying to understand why the slimserver has such a large footprint in time and space. By space I mean RAM, by time I mostly mean scan time.

I can read tags for my set of about 8500 songs in about 2 minutes. That's using FOXPRO (an interpreted language, like perl) on a 1.8GHz Pentium with 500MB ram. I did some experiments in a moderately compiled language (Visual Basic 6, the last one before .net). It was 2 orders of magnitude faster. So 12 seconds would be a good expectation for a commercial product.

The memory footprint for the Foxpro compiled code is about 55MB including disk caching, but disk caching seems to be about 40MB of it, so I'm guessing the program footprint itself is about 15MB. As a database program, it builds rapid lookup indexes and everything in that time and space. After it's finished building, I can do keystroke by keystroke interactive song lookup when I'm searching for that song whose title contains the phrase "s'wonderful" or "robin" or whatever.

The machine where slimserver runs is a 600MHz machine with 128MB RAM. It's so slow when it scans the tags, I can have not just a coffee but a shower and a shave while it's doing it. At a third the speed, I would expect about 6 minutes if it's all done in perl, at least an order of magnitude less with some optimization.

The memory footprint seems to be basically everything available on the 128MB machine (just to head off the question, each machine has it's own copy of the song database on a disk local to the machine, so there's no networking differences here).

Slim doesn't report the scan time, so I have accurate numbers, just a feeling that it's slightly longer than forever.

My direct motivation for asking is, I admit, a somewhat unusual application (I DJ with the squeezebox), but the issues raised seem to touch on other peoples experience too, as I found when I tried a search for the keywords "slow startup".

It seems to me the slimserver has two fundamentally different tasks to perform - feeding data to the hardware and managing a user interface.

As it turns out, I don't need the user interface stuff, the database, etc and don't use it (except for debugging & testing). But that's just me.

When I'm DJing, I shut off the rebranded web browser (I think this is the slimserver process).

But can I shut off the whole user interface? The song database, the scanning, everything?

Clearly, the server has to scan the tags of the song it's about to play, in order to put the name up on the screen, to answer CLI calls for what song are we playing, etc. Does it in fact do it that way, or does it go to the database?

And if it goes to the database currently, can that be changed?

It seems to me that the current slim server code should really be 2 modules: hardware driver and user interface.

Feeding the hardware is time critical - if you don't feed it fast enough, you get dropouts.

If they are on the same machine (the "vanilla" installation), they should contend for operating system resources at different priorities and communicate over an interface.

One could also chose to run them on different machines. The hardware driver on a NAS box like the buffalo or linksys, close to the disk data, and the user interface, with it's larger resource needs but less time critical, on some other box "closer" to the user.

In my searching, I found a number of people who wrote about something in the user interface part of the code stopping the hardware interface. And I've done this myself. So it is a concern.

In summary, my questions:

1. Why is tag scanning so resource intensive?

2. I thought the switch to a database system in 6 was supposed to address this problem ... my experience was that it was worse (I switched back to 5, for other reasons, so I can't check at the moment).

3. Given that it is so resource intensive (and potentially, even if you improved it, it's a large problem ... for every improvement in code efficiency, someone out there will have a larger library of songs ... ) how about these solutions:

4. Can the scanning and database service be turned off in the current code (something like scan never, flush cache)? Does anything I care about (song information like title and duration) break?

5. Can the user interface be separated from the "hardware feeding"
a. in the current code?
b. as a future enhancement?

Thanks in advance for your time and considered opinions.

Michael

danaronson
2005-05-28, 20:15
One possible (but not necessarily recommended) way to solve your problem would be to write a special purpose server. Since the protocols are openly documented, you (or someone) could probably write a server in a compiled language that might have a very
small footprint that could do exactly what you want.

--dan

Michaelwagner
2005-05-31, 07:17
True. But I think the perl code is fast enough for driving the hardware, and there are tremendous advantages to sticking to the mainstream codebase.

Thanks for the offer, though.

Michael

Robin Bowes
2005-05-31, 11:07
Michaelwagner wrote:

> It seems to me the slimserver has two fundamentally different tasks to
> perform - feeding data to the hardware and managing a user interface.

Exactly right.

I've been saying (at every available opportunity) that audio streaming
should be broken out into a separate process/thread so it can be
prioritised by the O/S.

I've looked into doing this myself a few times but the slimserver code
is pretty monolithic and it's not a task I feel I'm up to.

What's required is to remove the modules that deal with audio streaming
from slimserver.pl and start them up in a separate process, and decide
on a mechanism for the audio streaming process and main slimserver
process to communicate. It wouldn't require a particularly complex
protocol - all the audio streaming has to do is to send audio files from
a playlist to a player until it runs out. It should also communicate
status back to the slimserver process for the player display updates,
but this must be a non-blocking operation.

One other possibility is that a stand-alone streaming process could be
written to support upnp and thus open up streaming to the SB from
Windows Media Connect, etc.

I remain hopeful that someone will bite the bullet one of these days...

R.
--
http://robinbowes.com

JJZolx
2005-05-31, 14:02
5. Can the user interface be separated from the "hardware feeding"
a. in the current code?
b. as a future enhancement?


This is something I've been wondering as well. It would seem to be a lot more flexible having a lightweight service / daemon / device-driver that just drives the Squeezebox hardware and doesn't require the HTTP server and database server to function. Having all that stuff loaded into memory just to use an interface like the CLI doesn't make a lot of sense.
________
smoke weed every day (http://smokeweedeveryday.org)

JJZolx
2005-05-31, 14:21
The machine where slimserver runs is a 600MHz machine with 128MB RAM. It's so slow when it scans the tags, I can have not just a coffee but a shower and a shave while it's doing it. At a third the speed, I would expect about 6 minutes if it's all done in perl, at least an order of magnitude less with some optimization.

BTW, it sounds like you're running a Windows system. Which version of Windows are you running in 128MB of RAM? You can run XP in 256MB, but I'd consider 512MB to be the minimum unless performance is a secondary consideration. Running XP in 128MB is nothing short of painfully slow.
________
teen vid (http://teenvid.org)

Michaelwagner
2005-05-31, 15:23
Which version of Windows are you running in 128MB of RAM?
W98SE

(BTW, on my first attempt, that's all I wrote, and the forum software told me my reply was too short and I had to write at least 10 characters. Really? Why?)

JJZolx
2005-05-31, 17:29
W98SE

Doable in 128MB, but with SlimServer taking up 50-75MB, not very comfortable. I understand, though, that was one of the original points of this thread. :-)

I'm curious - How do you use the Squeezebox for DJing? Given that you have to run a computer to function as a server to drive the Squeezebox, why not use something like a USB DAC (such as the M-Audio Audiophile USB) and then eliminate SlimServer? You'd have the added flexibility of using different software packages to control the music - foobar, Winamp, iTunes, etc. The SlimServer/Squeezebox combo is handy for a lot of scenarios, but I'm not seeing how DJing is one of them.
________
BMW K75C (http://www.cyclechaos.com/wiki/BMW_K75C)

Michaelwagner
2005-05-31, 19:09
I'm curious - How do you use the Squeezebox for DJing?
It's fairly straightforward. I have a database of songs, criteria like speed, suitableness for dancing, what style of dance goes with the song, etc, and I select the next few songs I want to play and it watches them play out and sends the next one over.

It was important that I maintain the queue and not the MP3 player, because I sometimes change my mind (that is to say, the floor changes, the people change, and what was a suitable song 2 songs ago is no longer suitable) and I reserve the right to change my mind about the next song until about 10 seconds before the end of the current one.


Given that you have to run a computer to function as a server to drive the Squeezebox, why not use something like a USB DAC (such as the M-Audio Audiophile USB) and then eliminate SlimServer?
Several reasons:
1. I started with an Audiotron, and the Audiotron did all of the byte fetching itself, in response to (relatively) high level commands like "play this next song".
2. When I had to abandon the Audiotron (it was heavy, and it was discontinued), I looked around for something similar, ironically, because I didn't want to write the byte dribbling level of software. I assumed that when I bought the slim package, I was getting someone else's code that already did this.


You'd have the added flexibility of using different software packages to control the music - foobar, Winamp, iTunes, etc.
I'm not seeing an advantage there. I don't generally switch CD heads in the middle of a dance, and I don't want to switch to a different software package either. I wrote my software because nothing I saw out there (when I first started the package, 2 years ago ... situation may be different now) did what I as a DJ want. Sure, all sorts of packages do what people do at house parties, but not real stuff.

And no, I don't do beat mixing (hate it, actually). The style of DJing I do believes that the artist put the tip and the tail of the song there for a reason and they should be left there. Can you imagine reading a series of books without reading the first and last chapters, just the stuff in between?


The SlimServer/Squeezebox combo is handy for a lot of scenarios, but I'm not seeing how DJing is one of them.
Well, I was hoping for a "slimmer" slim server, but even now, it's not bad. My app. is written in high level terms (queue this song now) and I leave the bytes to someone else.

DJing is a hobby. I do it to earn enough money to pay for my CDs (I like paying for my music - it keeps blues and blues artists alive). I don't particularly have time to write low level hardware driving software, and didn't much want to.

Looks like I might have to anyways. Or at least learn how to prune a little perl :-)

Michael

JJZolx
2005-05-31, 20:17
It's fairly straightforward. I have a database of songs, criteria like speed, suitableness for dancing, what style of dance goes with the song, etc, and I select the next few songs I want to play and it watches them play out and sends the next one over.

I don't particularly have time to write low level hardware driving software, and didn't much want to.

I see. You have your own custom application that needs a simple means to play music tracks that you queue. Currently it uses the CLI to tell SlimServer which files to feed a Squeezebox.

So what you really need is an interface similar to the CLI to talk to some piece of playback software and then you could use a computer with either an internal or external sound card. There must be more than a couple ways to do this.

For one idea, take a look at foobar2000 and the available command line parameters to foobar2000.exe. You have things like /add /play /pause. And it's under 4MB in memory. Your program would shell out and issue a command such as:

"C:\Program Files\foobar2000\foobar2000.exe" /play "D:\Music\Cal Tjader\Soul Sauce\01 Soul Sauce.mp3"
________
70 (http://www.cyclechaos.com/wiki/Honda_70)

Michaelwagner
2005-06-01, 05:44
The sound hardware in most laptops is abysmal. I wanted the slim (and before that the audiotron) to produce high quality sound. If I just wanted an MP3 player to play out the lousy sound hardware of the laptop, there are a number of solutions to that problem.

Michaelwagner
2005-06-01, 05:49
oh, but thank you for the suggestion of foobar2000. It might be a good alternative for monitoring (the DJ often wants to listen to a song before playing it, to see if it fits the current mood, and laptop hardware is up to that task).

Michael

JJZolx
2005-06-01, 10:59
The sound hardware in most laptops is abysmal. I wanted the slim (and before that the audiotron) to produce high quality sound. If I just wanted an MP3 player to play out the lousy sound hardware of the laptop, there are a number of solutions to that problem.

That's why I suggested an external soundcard - I figured you were using a laptop. It would eliminate the Squeezebox and divorce you from being stuck using SlimServer. If the laptop had S/PDIF out then you might also go from the laptop to an external DAC of your choice. In all three hardware scenarios you have a single external piece of gear, but only the setup with the SqueezeBox limits you to working with the SlimServer software and its mega-footprint.
________
video reviews (http://videoreviews.org)

Michaelwagner
2005-06-01, 19:52
JJZolx:

I appreciate your comments and will check them out in due time.

In the mean time, I already own 2 squeezeboxes (squeezeboxen?) and don't have a lot of money to buy more things. I'd rather find a way to use the hardware I already have (provided that's possible and practical).

The point of posting on this forum was to find a squeezebox solution, not another one. I'm not the first one to find the squeezebox "resource footprint" large.

Back to the original questions: any more answers?

1. Why is (MP3) tag scanning so resource intensive?

2. I thought the switch to a database system in 6 was supposed to address this problem.

3. Given that it is so resource intensive (and for every improvement in code efficiency, someone out there will have a larger library of songs ... ) how about these solutions:

4. Can the scanning and database service be turned off in the current code (something like flush cache, scan never)? Does anything I care about (song information like title and duration) break?

5. Can the user interface be separated from the "hardware feeding"
a. in the current code?
b. as a future enhancement?

I partially answered one of my own questions - I found that if you start a rescan and then shut down the server, it clears the cache and doesn't restart the scan when you restart the server. After that, if you request songs via the web interface, it scans only those songs you request (to get the title, artist, etc, info, I guess). That gives me hope that commenting out the global scan in code (once I download active state perl) will yield at least a crude solution for my specific case.

Michael

mherger
2005-06-01, 23:26
Just to give at least one answer :-)

> 1. Why is (MP3) tag scanning so resource intensive?

Don't know.

> 2. I thought the switch to a database system in 6 was supposed to
> address this problem.

Nope. It's not intended to address resource intensive scanning directly.
The database should offer a more stable mechanism to manage the
information found during the scan. And it should considerably lower memory
consumption with large collections. In slimserver <6 everything was kept
in memory. Now it's written to a database.

> 4. Can the scanning and database service be turned off in the current
> code (something like flush cache, scan never)? Does anything I care
> about (song information like title and duration) break?

No scan, no music information. But normally you scan your collection once
in a while. It should not have an impact in everyday usage.

> I partially answered one of my own questions - I found that if you
> start a rescan and then shut down the server, it clears the cache and
> doesn't restart the scan when you restart the server. After that, if
> you request songs via the web interface, it scans only those songs you
> request (to get the title, artist, etc, info, I guess). That gives me
> hope that commenting out the global scan in code (once I download
> active state perl) will yield at least a crude solution for my specific
> case.

Do whatever you want :-). But I really don't get what you want. What do
you need slimserver for if you want neither user interface nor the
information?!?

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

Michaelwagner
2005-06-02, 04:27
Just to give at least one answer :-)
I appreciate the effort.


> 2. I thought the switch to a database system in 6 was supposed to address this problem.

Nope. It's not intended to address resource intensive scanning directly. The database should offer a more stable mechanism to manage the information found during the scan. And it should considerably lower memory consumption with large collections. In slimserver <6 everything was kept in memory. Now it's written to a database.
OK, I must admit, I didn't spend enough time benchmarking 6. Maybe the memory footprint is smaller when re-starting after a full scan.


No scan, no music information. But normally you scan your collection once in a while. It should not have an impact in everyday usage.
First part: slim will scan songs that are queued for playback to get the song information even if they aren't in the database.
Second part: For household use, yes. For DJ use, not necessarily true. One might bring different collections to a gig and have to switch them. Or have a hardware failure and have to switch to a backup system.


I really don't get what you want. What do you need slimserver for if you want neither user interface nor the information?!?
To drive the hardware (economically). The thing everyone who looks at the pubs thinks the software does.

I think it's quite clear there are 2 different purposes being served by the slimserver software - drive the hardware and support the information database. These are fundamentally different things, have different "punctuality" requirements, etc.

Driving the hardware is a real-time operation. The info database, no one cares if it's 200ms late.

It seems to me they should be separate processes (in the operating system sense). The fact that I don't happen to need one of them is the back door whereby I came to this conclusion, but I think the architecture is sound regardless.

Michael

mherger
2005-06-02, 04:42
>> No scan, no music information. But normally you scan your collection
>> once in a while. It should not have an impact in everyday usage.
> First part: slim will scan songs that are queued for playback to get
> the song information even if they aren't in the database.

....or _only_ if they aren't in the database? I don't know how you tested
this. But could your repeat the test after a complete scan?

> Second part: For household use, yes. For DJ use, not necessarily true.
> One might bring different collections to a gig and have to switch them.
> Or have a hardware failure and have to switch to a backup system.

Hardware failure might be an exception :-).

>> I really don't get what you want. What do you need slimserver for if you
>> want neither user interface nor the information?!?
> To drive the hardware (economically). The thing everyone who looks at
> the pubs thinks the software does.

....and you're feeding the playlist, the songs etc. all yourself by the
means of your own application?

> I think it's quite clear there are 2 different purposes being served by
> the slimserver software - drive the hardware and support the information
> database. These are fundamentally different things, have different
> "punctuality" requirements, etc.

But you'll have to admit that there won't be many users who use a SB as a
luxury soundcard. I've never heard of anyone else using slimserver as a
hardware driver _only_ before...

Honestly: in my eyes your application is too esoteric, to expect
developpers putting much effort in supporting it (but this is only my
opinion). I assume the current architecture - while it's technically not
the most sophisticated - will satisfy close to 1100% of the users. It just
works for most of us.

What's your main concern: latency or memory foodprint? I was able to
reduce memory usage of a stock 6.0 trunk copy from about 43MB to 36MB
within an hour (remove plugins, setup, XML::Simple). Next victim would
have been the web interface. But I'm not sure if this is worth the effort.

Have you ever thought of writing your own application to use slimproto and
feed the device directly?

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

mherger
2005-06-02, 09:35
>> It just works for most of us.
>
> That sentence can be read in at least two ways; I'm sure you meant "It
> just *works* ..." but to my mind, "It *just* works". I've noticed

You're right, of course, it *works* :-)

>> Have you ever thought of writing your own application to use slimproto
>> and feed the device directly?
>
> I have, several times, but each time I get anywhere near the slimserver
> code I think of something more important I have to do... :)

So might do Dean, Dan & co. when reading about splitting the code ;-).

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

Robin Bowes
2005-06-02, 09:39
Michael Herger wrote:
> But you'll have to admit that there won't be many users who use a SB as
> a luxury soundcard. I've never heard of anyone else using slimserver as
> a hardware driver _only_ before...

Ah, but that's because you can't. This is essentially the purpose of
Microsoft's Media Centre Connect - it allows music from PCs to be
streamed to compatible hardware devices.

Personally, I'd love to be able to do that. Whilst I find slimserver
adequate for managing what I play on my SB, I'd much prefer to have the
freedom to use other software to do this and to stream the audio to a
"slimmed" down slimserver process that just deals with the audio (and
IR, and display updating).

> It just works for most of us.

That sentence can be read in at least two ways; I'm sure you meant "It
just *works* ..." but to my mind, "It *just* works". I've noticed
occasional dropouts playing back flac over a wired connection. I'm sure
this wouldn't happen if the streaming process was separated from the
playlist management.

> Have you ever thought of writing your own application to use slimproto
> and feed the device directly?

I have, several times, but each time I get anywhere near the slimserver
code I think of something more important I have to do... :)

R.

--
http://robinbowes.com

Robin Bowes
2005-06-02, 11:48
Michael Herger wrote:

>> I have, several times, but each time I get anywhere near the
>> slimserver code I think of something more important I have to do... :)
>
>
> So might do Dean, Dan & co. when reading about splitting the code ;-).

It's not that big a job, *if* you understand the existing code. All the
code's there, it just needs architecting at a high level and
re-organising. I'm sure Dean/Dan/Vidur/ et al. are more than up to the task.

R.

--
http://robinbowes.com

Dan Sully
2005-06-02, 11:50
* Robin Bowes shaped the electrons to say...

>>>I have, several times, but each time I get anywhere near the
>>>slimserver code I think of something more important I have to do... :)
>>
>>So might do Dean, Dan & co. when reading about splitting the code ;-).
>
>It's not that big a job, *if* you understand the existing code. All the
>code's there, it just needs architecting at a high level and
>re-organising. I'm sure Dean/Dan/Vidur/ et al. are more than up to the task.

Actually it is. Two words: database locking.

-D
--
vacation (n) : an extended trip away from home in search of inconvenient ways to connect to the Internet.

fuzzyT
2005-06-02, 12:44
Dan Sully wrote:

> Actually it is. Two words: database locking.

from <http://www.sqlite.org/whentouse.html>

"SQLite uses reader/writer locks on the entire database file. That means
if any process is reading from any part of the database, all other
processes are prevented from writing any other part of the database."

Not table, page or row level locking, but DATABASE level locking. Whew.
This is OK for multiple concurrent reads, but writes are gonna be a
little challenged.

Given that, webUI and playback threads should play nice with others.
But the scan/updateDB thread is gonna be a bully.

So, what's the workaround? An update thread that works in chunks?
Different engine? Alternating Live/Standby SS database file instances?
Hmmm.

--rt

JJZolx
2005-06-02, 13:10
>It's not that big a job, *if* you understand the existing code. All the
>code's there, it just needs architecting at a high level and
>re-organising. I'm sure Dean/Dan/Vidur/ et al. are more than up to the task.[/color]

Actually it is. Two words: database locking.

Dan, please explain what you mean by this. I'm unclear why database interaction needs to be tied so closely to feeding the device an audio stream, or sending it display data, or querying it for status info.

I can certainly understand how the current SlimServer became (or remained) this monolithic music/HTTP/database/telnet server. And I can understand that it might be far from trivial to break out the device-interaction into a separate process, but I think the benefits would be worth the effort. It would probably encourage more software developers to make their programs interact with a Squeezebox if they don't have to get down and work with SlimProto. Running a program as large as SlimServer, 99% of which would go unused, is nothing short of crazy just to be able to play music though a Squeezebox.
________
drugtest (http://drugtestingkit.org)

mherger
2005-06-02, 13:53
>> Actually it is. Two words: database locking.
>
> Dan, please explain what you mean by this. I'm unclear why database
> interaction needs to be tied so closely to feeding the device an audio
> stream, or sending it display data, or querying it for status info.

Why don't you accept that it obviously isn't as simple as some might
think? I'm sure it would have been done long before, if it was an easy
task. Everybody's so smart... I'm sure, patches are most welcome :-)

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

Robin Bowes
2005-06-02, 15:36
Michael Herger wrote:
>>> Actually it is. Two words: database locking.
>>
>>
>> Dan, please explain what you mean by this. I'm unclear why database
>> interaction needs to be tied so closely to feeding the device an audio
>> stream, or sending it display data, or querying it for status info.
>
>
> Why don't you accept that it obviously isn't as simple as some might
> think? I'm sure it would have been done long before, if it was an easy
> task. Everybody's so smart... I'm sure, patches are most welcome :-)

Because it is relatively simple - there are other forces at work here...
(priorities, etc.)

R.

--
http://robinbowes.com

Robert Moser
2005-06-02, 15:50
Robin Bowes wrote:
> Because it is relatively simple - there are other forces at work here...
> (priorities, etc.)
> R.

What makes you think that this task is simple?

Robin Bowes
2005-06-02, 16:42
Robert Moser wrote:
> Robin Bowes wrote:
>
>> Because it is relatively simple - there are other forces at work
>> here... (priorities, etc.)
>> R.
>
>
> What makes you think that this task is simple?

I have my reasons...

Let's see what happens, eh?

R.

--
http://robinbowes.com

Michaelwagner
2005-06-02, 19:18
I was sloppy - I'm sorry.

I should have written:
If a song is not in the database, slim will scan the song when it is queued for playback to get the song information.


Hardware failure might be an exception :-).
DJs are held to higher standards. No one wants to hear that their wedding music was cancelled due to a hardware failure. I travel with 2 slims, a backup hard disk, etc. All DJs have 2 head CD players, so that if one fails .... :-)


you're feeding the playlist, the songs etc. all yourself by the means of your own application?
Yes.


Honestly: in my eyes your application is too esoteric, to expect developpers putting much effort in supporting it
I said before, I realize my application is unusual. But the requirement (dropout-free audio) is not. The slim architecture makes this unachievable during a scan. Read the other threads -others have mentioned this, not just me.


the current architecture will satisfy close to 1100% of the users. It just works for most of us.
You like dropouts?


What's your main concern: latency or memory foodprint?
If you're asking me to prioritize, latency.
In a previous life I was a performance analyst. I find the memory footprint amazing on an intellectual level. But I don't really care about memory. I can buy memory. I can't buy time.

My production system has plenty of memory. It's my test system that's a little low on the memory side. :-}


Have you ever thought of writing your own application to use slimproto and feed the device directly?
Yes, of course. But since the effort would only be slightly less than "fixing" the architectural problem with slim, I thought I'd make the suggestion.

If no one likes my suggestion, I'll go back into my hole and hack the code on my own.

Michael

Michaelwagner
2005-06-02, 20:22
Check out http://forums.slimdevices.com/showthread.php?t=14534

mherger
2005-06-02, 23:34
> Check out http://forums.slimdevices.com/showthread.php?t=14534

Or http://forums.slimdevices.com/showthread.php?t=14199 - other user,
different priority. And everybody thinks he's with the majority.

I hope you've read Robin's statements, too.
http://forums.slimdevices.com/showthread.php?t=14453&page=2&pp=20

I take these as an announcement ;-)

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

robinbowes
2005-06-03, 02:21
I hope you've read Robin's statements, too.
http://forums.slimdevices.com/showthread.php?t=14453&page=2&pp=20

I take these as an announcement ;-)


I wouldn't hold your breath...

R.

Michaelwagner
2005-06-03, 07:20
I hope you've read Robin's statements, too.
http://forums.slimdevices.com/showthread.php?t=14453&page=2&pp=20

I take these as an announcement ;-)
Of course I've read it. It's in the same thread with my comments. I read all the postings carefully.

If it's an announcement, it's a pretty obscure one.

The stuff about database locking mostly says that people didn't understand my suggestion.

While it's sad that the database chosen cannot lock at the record level, that also isn't a requirement in what I was proposing. Cross-process communication could get the one title my proposal would need every 3 minutes and still be effective.

Michael

mherger
2005-06-03, 07:34
>> I hope you've read Robin's statements, too.
>> http://forums.slimdevices.com/showthread.php?t=14453&page=2&pp=20
>>
>> I take these as an announcement ;-)
>> Of course I've read it. It's in the same thread with my comments. I read
> all the postings carefully.
>
> If it's an announcement, it's a pretty obscure one.

It isn't an announcement. It was only what seems to be a bad joke of mine:
Robin was so sure it was easy, and he had his reasons to know, that I
wanted him to be working on it.

> The stuff about database locking mostly says that people didn't
> understand my suggestion.

They do, I'm sure (hey, even I do!). But they don't have the capacity to
implement everything people think is most important right now or
yesterday. That's what I wanted to say with my last post. You want an
independant hardware driver (just to give it a name). Someone else wants
rhapsody. (don't even think about the p-word) And for all of you it's most
important.

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

Dan Sully
2005-06-03, 07:39
* Malcolm Wotton shaped the electrons to say...

>This isn't strictly a slim thing, but I thought the community here sould
>like to know.
>
>I just got the Activestate version of Audio::FLAC and was using it to retag
>some of my FLAC files. Unfortunately it has a bug in the write command and
>corrupts the FLAC file.

Malcolm - you should be using Audio::FLAC::Header instead. Audio::FLAC was
split into ::Header and ::Decoder

-D
--
Off the record, on the QT, and very hush-hush.

Robin Bowes
2005-06-03, 08:00
Michael Herger wrote:
>>> I hope you've read Robin's statements, too.
>>> http://forums.slimdevices.com/showthread.php?t=14453&page=2&pp=20
>>>
>>> I take these as an announcement ;-)
>>> Of course I've read it. It's in the same thread with my comments. I read
>>
>> all the postings carefully.
>>
>> If it's an announcement, it's a pretty obscure one.

It wasn't an announcement.

>
> It isn't an announcement. It was only what seems to be a bad joke of
> mine: Robin was so sure it was easy, and he had his reasons to know,
> that I wanted him to be working on it.

:) I'm afraid I'm not.

>> The stuff about database locking mostly says that people didn't
>> understand my suggestion.
>
>
> They do, I'm sure (hey, even I do!). But they don't have the capacity
> to implement everything people think is most important right now or
> yesterday. That's what I wanted to say with my last post. You want an
> independant hardware driver (just to give it a name). Someone else
> wants rhapsody. (don't even think about the p-word) And for all of you
> it's most important.

That just about hit the nail on the head.

It's all about priorities.

I'll shut up now...

R.
--
http://robinbowes.com

Michaelwagner
2005-06-03, 09:57
It isn't an announcement. It was only what seems to be a bad joke of mine
OK. Got it now. I didn't realize you were joking.

But they don't have the capacity to implement everything people think is most important right now or yesterday. That's what I wanted to say with my last post.
Ah. So let me refocus the discussion.

Forget that I DJ with the slim. That's a red herring.

Look at the slim web site: On this page
http://www.slimdevices.com/pi_overview.html

It says:

Easy to use and set upóIt takes just a few minutes to install: simply load SlimServer, our powerful and free Open Source software, onto your computer and connect the player to your network. Squeezebox2 automatically configures itself and is ready to use.


Find Songs FasteróBrowse and search 100,000 song libraries in seconds to find the perfect song for the perfect moment. Squeezebox2's user-friendly interface allows you to browse quickly through your whole music collection via custom remote controls or a web browser.

These are not my priorities. They are the priorities of the people who prepared the sales literature. And judging by what I read here, and by my own experiences, the software is not performing to that level.

My suggestion was a relatively straightforward one (albeit it, nontrivial work) to implement what has already been advertised as being there.

I'm not talking about adding new features for me. I'm talking about getting the features working that have already been advertised and people have already been sold boxes on the basis that this stuff works.

Michael

kdf
2005-06-04, 03:32
Quoting Michaelwagner <Michaelwagner.1q28a0 (AT) no-mx (DOT) forums.slimdevices.com>:

> It says:
> > Easy to use and set upóIt takes just a few minutes to install: simply
> > load SlimServer, our powerful and free Open Source software, onto your
> > computer and connect the player to your network. Squeezebox2
> > automatically configures itself and is ready to use.

hundredds, if not, thousands of units have worked exactly this way. I have 3
boxen, and all were plug and play. This doesn't mean that there are not
specific cases where problems do come into play. They are just not as
widespread as you may feel at this point. A vast majority of postings on
forums like this are from only those who are needing solutions to their
problems.

>
> > Find Songs FasteróBrowse and search 100,000 song libraries in seconds to
> > find the perfect song for the perfect moment. Squeezebox2's
> > user-friendly interface allows you to browse quickly through your whole
> > music collection via custom remote controls or a web browser.

ditto above. 'find faster' is marketing-speak anyway. faster than what? its a
HUGE lot faster than the old system. 'in seconds'...right, how many. clearly
its never enough for some. simple to fix, according to others.

Sadly, from all that i've read of this increasingly annoying thread is that you
are likely to find little in the way of an immediate solution. Those who have
proven themselves time and time again as ready and able to work their butts off
to fix code recognise that it is needed, but not simple enough to fix quickly
or without a lot of thought. Others proclaim its easy, seemingly implying that
those prior workhorse types are, in fact, simply lying or lazy. Strangely,
those claiming the simplicity of the task always say that they have better
thing to do.

The problem is that your needs differ slightly from those with home systems.
Slimserver 6.0 is transitional in some ways, since it has to deal with
data/playlists created from 5.x releases. It also must deal with users who
have music from many different sources, not all of which are designed to play
nicely with others. This means the server has to keep wasting time with sanity
checks (db lookups in the current playlist, for example). To put my opinion
plainly, this sucks. I'd prefer a setup where the server simply demanded that
the user put in the effort and just used slimserver (instead of competing
demands that it work more like itunes vs more like WMP). however, braver souls
than myself have ddecided that slimserver will support as many formats and user
styles as possible. This means that some users will simply have to be patient
over some features. A difficult task, it appears, for some.

Now, to address your issues with current playlist db lookup, try Dan's patch for
playlists in the db (or wait a little longer until it makes the nightly
builds). I'm sure this will help with some of the problems you are seeing.

-kdf

Michaelwagner
2005-06-04, 08:50
'find faster' is marketing-speak anyway. faster than what? its a HUGE lot faster than the old system.
Well, I come from the Audiotron world, so I have something to compare to. For about 6000 songs that I had at the time, it took about 2-3 minutes *and* it was fetching them over the network, and it was running on a much slower processor than slim is running on. Slim has them on a local hard disk and takes an order of magnitude longer. Now, the audiotron couldn't play until it finished scanning, but it was fast. And once it finished scanning, it was reliable - no dropouts.


from all that i've read of this increasingly annoying thread
I'm sorry. Annoying you or anyone else was not my intent.

you are likely to find little in the way of an immediate solution.
I suspect for myself, downloading activeperl or whatever it's called so I can fiddle the source myself and commenting out the places where it calls for long scans will be sufficient. I won't have a database I don't need, it still scans for songs I queue so the display will be right (not that that was truly necessary for me) and I'll be happy as a clam.

My motive in making the architectural comments was altruistic. This was supposed to be an open source community and (I thought) suggestions were welcome.

Instead, what I got when I made a suggestion was derision, scorn and irritation that I'm not prepared to jump in and implement the suggestion as well. Well, I'm not. I am a programmer by trade, but I don't do work I can't be proud of. My lack of understanding of perl is such that I couldn't submit a 2 line fix with any confidence.

I can, however, figure out how to comment things out from the online perl tutorial, so that may be enough to fix *my* immediate problem. The rest of the comments were intended to be of help to the community as a whole.


Those who have proven themselves time and time again as ready and able to work their butts off to fix code recognise that it is needed, but not simple enough to fix quickly or without a lot of thought. Others proclaim its easy, seemingly implying that those prior workhorse types are, in fact, simply lying or lazy. Strangely, those claiming the simplicity of the task always say that they have better thing to do.

For the first part, I haven't seen anyone say this re-architecting is needed. If I had, I wouldn't have made the suggestion. For the second part, I said none of those things. The people who did can answer for themselves.


The problem is that your needs differ slightly from those with home systems.
As I said to Michael H. yesterday, the fact that I DJ from the system is a red herring. The important thing is that people experience dropouts, and that it got worse with 6.


Slimserver 6.0 [...] has to keep wasting time with sanity
checks (db lookups in the current playlist, for example). To put my opinion plainly, this sucks.
You're making my point for me. The main code of the server could be any kind of pig it wanted to be and no one would care, provided the hardware driving code ran in another process (or thread in Windoz).

In marketing terms, all those features you alude to can be added, at no performance penalty (read - no audio dropouts) if the two fundamentally different tasks are split and put in different processes.


braver souls than myself have decided that slimserver will support as many formats and user styles as possible. This means that some users will simply have to be patient over some features.
Or customers could merely go elsewhere. Wouldn't it be smarter for all concerned to adapt the technology to be less brittle and attract and keep more customers? Audio dropouts when using the advertised features are not a small problem.

Imagine that, when you adjust the toaster knob to make the bread darker, the toaster stoped toasting for 10 seconds because it couldn't pay attention to the darkness adjustment and toasting at the same time. Who would accept such an appliance?

Or the MP3 player in your car stops when you step on the gas, because the computer that does the spark plug delay timing also runs the player and there isn't enough time for both and they weren't prioritized properly.

(or worse, when you search for a new MP3, the car backfires :-) )


to address your issues with current playlist db lookup try Dan's patch for playlists in the db
I don't submit playlists, I submit individual songs. I'm not sure how his patch applies. I ignored it when I saw it. Was I wrong? Does it do more than it sounds like?

Michael

John A. Tamplin
2005-06-04, 08:53
On Fri, 3 Jun 2005, Michaelwagner wrote:

> The stuff about database locking mostly says that people didn't
> understand my suggestion.

If you have multiple processes all needing access to the same database,
locking does enter into it. If your streaming process gets blocked by a
lock held by another process, that is no better than the current
single-threaded architecture where the streaming is blocked by other
CPU requirements.

> While it's sad that the database chosen cannot lock at the record
> level, that also isn't a requirement in what I was proposing.
> Cross-process communication could get the one title my proposal would
> need every 3 minutes and still be effective.

IMHO, if you start having to architect around the limitations of the
database, you are better off going with a different database backend that
doesn't have that problem.

Also, for those who say it is trivial to split a large single-threaded
program into multiple threads of control, I can only say you haven't done
it before. Correctly writing multithreaded code is not easy, and it is
especially not easy to take code not written to be multithreaded and make
it so.

For the company to be successful, they have to make it where the average
user can buy it, plug it in, and it just works. The average user does not
need to carry a Squeezebox with them to be a DJ, nor do they need to tie
into a pre-existing Informix database (in case you haven't followed the
list for a long time, that was my requirement).

I am not the average user, nor are most people on this list. For us, we
can either make do with what is provided, work to make modifications that
we want without degrading what the general public needs, or since it is
open source we can take it and make it do exactly what we want. If you
don't have the capability or time to do that yourself, I am sure there are
consultants available to do the development for you.

--
John A. Tamplin jat (AT) jaet (DOT) org
770/436-5387 HOME 4116 Manson Ave
Smyrna, GA 30082-3723

Michaelwagner
2005-06-04, 09:19
If you have multiple processes all needing access to the same database, locking does enter into it.
Of course. But that's a big if. Do you truly have such a requirement?

What part of the audio hardware driving process needs access to the database information?

At most, if you put the screen driving software in the same process, you need to ask, once every three minutes or so, for the title, artist, genre, album and duration of a song. One request every 3 minutes or so. Not so hard.

*And* you could make a more costly cross-process call to the database process to find out the name of the song, once every three minutes, and avoid cross-process database work altogether. Just like now I can ask the CLI what is the artist of this song, so could the hardware driving process.

A better architecture would be to put the screen driver not in the same process with the audio driver (put it in the database process). If the screen is a second late, no one will notice. Most of the time, no one is even looking at it. And if they are anyways searching, they already expect it to be slow.

But if the audio is 50ms late, everyone will hear it.


If your streaming process gets blocked by a lock held by another process
But it wouldn't because it has no need for that information.


IMHO, if you start having to architect around the limitations of the database, you are better off going with a different database backend that doesn't have that problem.
They might indeed have chosen a different database. I don't know what went into that decision. And I'm not trying to second-guess them. But the decision is not fatal, not even very important, at this point, since the requirements from the second process are trivial in the extreme.


for those who say it is trivial to split a large single-threaded program into multiple threads of control, I can only say you haven't done it before.
Before my current job I did operating systems maintainance for a living. So I do have a clue whereof I speak.

I'm not saying it's trivial. Only necessary.

But I'm sure the code already falls more or less into 2 categories: stuff that drives hardware and stuff that does other stuff. The directory structure certainly indicates that people were thinking along those lines when they laid out the files.

Has anyone analyzed a call tree to figure out how hard this really would be?


For the company to be successful, they have to make it where the average user can buy it, plug it in, and it just works.
Agreed. Do you think that happens?


The average user does not need to carry a Squeezebox with them to be a DJ
I'm sorry I ever answered Michael's question about why I DJ with the thing. Please, people, get past the fact that I happen to DJ with it some times. I'm also a regular user of the box and it drops out when I'm playing music at home. It is a total red herring that I DJ with it. I also listen to it at home and I get audio dropouts when it's scanning at home too.

Michael

John A. Tamplin
2005-06-04, 09:39
On Sat, 4 Jun 2005, Michaelwagner wrote:

> What part of the audio hardware driving process needs access to the
> database information?

If you store the song metadata in the database, you might well need to
fetch that metadata for the streaming. For example, getting sample size,
bit order, encoding, etc -- all needed to decide how to setup the hardware
and what sorts of conversion need to be done before sending it over the
wire.

> At most, if you put the screen driving software in the same process,
> you need to ask, once every three minutes or so, for the title, artist,
> genre, album and duration of a song. One request every 3 minutes or so.
> Not so hard.

I agree that unless you are actually storing the actual sample data in the
database (which is not done by anyone at present to my knowlege), you only
need to access the database once per song. However, when you need to
access it you still do not want it blocked by some long-running
transaction in some other process/thread.

> *And* you could make a more costly cross-process call to the database
> process to find out the name of the song, once every three minutes, and
> avoid cross-process database work altogether. Just like now I can ask
> the CLI what is the artist of this song, so could the hardware driving
> process.

Ok, so now we have a single database process that manages all access to
the database. Unless it is multithreaded (and most databases I know of,
including commercial ones, have limited multithreading support -- usually
you need a separate process for each connection for best results), you are
now in a worse situation in that any long-running query will block the
streaming.

No matter where you put it, if you have a single funnel point for database
access you will still have a blocking problem. Either you need to have a
database that supports multiple concurrent access paths or you have to do
something fancier of caching the data you need outside the database (and
the hassles of excess resource usage and keeping it up to date), which
defeats much of the purpose of switching to the database in the first
place (and you are reimplementing part of the job of the database).

> They might indeed have chosen a different database. I don't know what
> went into that decision. And I'm not trying to second-guess them. But
> the decision is not fatal, not even very important, at this point,
> since the requirements from the second process are trivial in the
> extreme.

Look in the discussion archives for a long and at times heated discussion
of that topic. The choice goes back to my main point -- Slim made that
decision based on what was needed for the typical user. The rest of us
are free to do things beyond the typical requirements, but we can't expect
Slim to expend limited resources pleasing a tiny fraction of the user
base.

> I'm not saying it's trivial. Only necessary.

Necessary for some needs. I agree it is a desirable goal, but while it
would be great to have a deadline-driven realtime scheduler for the audio,
the needs do not justify the effort required. If the average user starts
running into problems with the single-threaded nature of the current code
(and they might), then it will become for of a necessity.

> But I'm sure the code already falls more or less into 2 categories:
> stuff that drives hardware and stuff that does other stuff. The
> directory structure certainly indicates that people were thinking along
> those lines when they laid out the files.
>
> Has anyone analyzed a call tree to figure out how hard this really
> would be?

I haven't since I am running a heavily hacked version from about two years
ago to meet my database requirements. I hope to be able to use the new
database version and hook it into my database, but I am going to let it
settle down a bit more before I undertake that project. Perl DBI is
rather portable between databases, so I hope with a few views to make the
schema compatible I will be able to start running the current code again.

>> For the company to be successful, they have to make it where the average
>> user can buy it, plug it in, and it just works.
> Agreed. Do you think that happens?

I have a couple of decidedly non-technical friends who use it without any
difficulty.

> I'm sorry I ever answered Michael's question about why I DJ with the
> thing. Please, people, get past the fact that I happen to DJ with it
> some times. I'm also a regular user of the box and it drops out when
> I'm playing music at home. It is a total red herring that I DJ with it.
> I also listen to it at home and I get audio dropouts when it's scanning
> at home too.

The point remains that people with special requirements, including huge
song databases, are not the typical customer and therefore can't be the
focus of development efforts.

--
John A. Tamplin jat (AT) jaet (DOT) org
770/436-5387 HOME 4116 Manson Ave
Smyrna, GA 30082-3723

Michaelwagner
2005-06-04, 12:20
The point remains that people with special requirements, including huge song databases, are not the typical customer and therefore can't be the focus of development efforts.
My song database is currently 8000 songs. Is that atypical? Is that a special requirement? It worked well on 5, less well on 6. Sounds like a step backwards to me.

Michael

John A. Tamplin
2005-06-04, 12:56
On Sat, 4 Jun 2005, Michaelwagner wrote:

> My song database is currently 8000 songs. Is that atypical? Is that a
> special requirement? It worked well on 5, less well on 6. Sounds like a
> step backwards to me.

Yes, I would agree that is a problem that needs fixing. However, it isn't
clear that dividing the server into two threads is required to do this,
since version 5 was also a single-threaded server.

I haven't used 6 yet because of the customizations I am running, but if
the performance has degraded I would think that needs to be fixed. It is
possible new features that have been added may necessitate such an
approach, but I suspect it is more of an optimization problem.

I know that when I personally switched to keeping all the metadata in a
database rather than reading it from the MP3s at startup and caching it in
RAM greatly reduced the RAM footprint, dropped startup time dramatically,
and enabled new features such as per-playlist weighted random play. I
haven't ever had any dropout problems even with other things running on
the server, other than when backups are running (which can only be solved
by a faster I/O subsystem or massive prebuffering). I can't imagine other
changes to use a database would not offer similar improvements.

One of the things I did was to incrementally compute the next random song
to play (which requires going through all the songs in the playlist and
computing a non-trivial function involving previous play timestamp, user
preference, etc) -- once the song started to play, I would start computing
the next one, and only process a few fetches from the database each time.
I don't know if there are similar things in the current code that might
benefit from a similar treatment.

Clearly, if you were starting from scratch you would consider a
multi-threaded architecture (you may not choose it for portability or
other issues -- in particular I don't know how portably you can do
multi-threading under Windows in perl). However, this isn't starting from
scratch and you can't just ignore the effort required to do a huge project
of making it multithreaded versus the likely smaller effort required to
make the existing code work acceptably well.

--
John A. Tamplin jat (AT) jaet (DOT) org
770/436-5387 HOME 4116 Manson Ave
Smyrna, GA 30082-3723

kdf
2005-06-04, 17:18
Quoting Michaelwagner <Michaelwagner.1q49nn (AT) no-mx (DOT) forums.slimdevices.com>:

>
> John A. Tamplin Wrote:
> > The point remains that people with special requirements, including huge
> > song databases, are not the typical customer and therefore can't be the
> > focus of development efforts.
> My song database is currently 8000 songs. Is that atypical? Is that a
> special requirement? It worked well on 5, less well on 6. Sounds like a
> step backwards to me.

well...how?

less well...how?

if you want this to stay on topic, stop helping the btich crowd and start
sticking to your own details so that people can understand better exactly what
you actually ARE after.

-kdf

Michaelwagner
2005-06-04, 17:58
if you want this to stay on topic, stop helping the btich crowd and start sticking to your own details so that people can understand better exactly what you actually ARE after.
I think I somehow unwittingly stepped into a place where everyone wants to yell at everyone else, and helpful suggestions made at moderate volume are neither appreciated nor, frankly, even read.

I am not trying to help the bitch crowd. I had no idea there was a bitch crowd. If I have inadvertantly fed into someones agenda, that was not my intention.

I have said now several times what I want, both for myself personally and for the software. I am not going to repeat myself yet again. Read my initial posting and the set of questions there. They're all there in the forum thread, and the gist was repeated later.

For an open source community, this certainly doesn't seem to be a very open community.

I will go away now and solve my problem locally, using the supplied documentation and the helpful suggestions made by some here.

If, on my journey, I find some information that is generally applicable, and if it doesn't require enduring another flamestorm, I will post my results here.

Michael

stinkingpig
2005-06-04, 20:02
Michaelwagner wrote:
....
> For an open source community, this certainly doesn't seem to be a very
> open community.
>

Sorry to disappoint, but this is perfectly normal for the open source
community. You should check out the LKML archives sometime :)

> I will go away now and solve my problem locally, using the supplied
> documentation and the helpful suggestions made by some here.
>
> If, on my journey, I find some information that is generally
> applicable, and if it doesn't require enduring another flamestorm, I
> will post my results here.
>
> Michael
>


Please do -- I'd like to see the issues you describe solved as well, but
I just don't have the time to work on it myself. For future reference,
the issue that is causing the flamestorm hasn't got anything to do with
you personally.

--
Jack at Monkeynoodle dot Org: It's a Scientific Venture...
Riding the Emergency Third Rail Power Trip since 1996!

Caleb Epstein
2005-06-05, 05:07
On Sat, Jun 04, 2005 at 08:50:56AM -0700, Michaelwagner wrote:

>
> kdf Wrote:
> >
> > 'find faster' is marketing-speak anyway. faster than what? its a HUGE
> > lot faster than the old system.
> Well, I come from the Audiotron world, so I have something to compare
> to. For about 6000 songs that I had at the time, it took about 2-3
> minutes *and* it was fetching them over the network, and it was running
> on a much slower processor than slim is running on. Slim has them on a
> local hard disk and takes an order of magnitude longer. Now, the
> audiotron couldn't play until it finished scanning, but it was fast.
> And once it finished scanning, it was reliable - no dropouts.

Try using MySQL as a back end. SQLite is garbage. It hardly
qualifies as a database at all. All the data is stored as
pipe-delimited strings and not as native types. The fact that
every row is a variable size and nothing is stored natively
explains its atrocious performance.

--
Caleb Epstein | bklyn . org | The person you rejected yesterday could make
cae at | Brooklyn Dust | you happy, if you say yes.
bklyn dot org | Bunny Mfg. |

Marc Sherman
2005-06-05, 06:00
Michaelwagner wrote:
>
> For an open source community, this certainly doesn't seem to be a
> very open community.

It seems like a perfectly typical open source community to me; it's very
open to suggestions accompanied by patches, and expects submitters of
patchless design proposals to be open to requests for patches.

Look, it's obvious to anyone who's ever done any significant amount of
software design that your proposal (at least at the big picture level)
is Correct. That doesn't make it easy or expedient, though, and work
has to be prioritized. If it's that important to you, you know how to
submit a patch.

- Marc

John A. Tamplin
2005-06-05, 06:48
On Sun, 5 Jun 2005, Caleb Epstein wrote:

> Try using MySQL as a back end. SQLite is garbage. It hardly
> qualifies as a database at all. All the data is stored as
> pipe-delimited strings and not as native types. The fact that
> every row is a variable size and nothing is stored natively
> explains its atrocious performance.

Yes, SQLite has its limitations, but based on other messages being posted
here it sounds like the time is not in the database query but rather
building the set of perl objects from the query results. If that is
correct, switching databases will at best improve the part that takes less
than 15% of the time.

Frankly for the size databases in use here, the database doesn't have to
be that fancy to get acceptable performance. The database is only going
to be an issue with very complicated queries where different query
execution plans can have orders of magnitude difference or with huge
databases (100k rows in a database is not huge).

--
John A. Tamplin jat (AT) jaet (DOT) org
770/436-5387 HOME 4116 Manson Ave
Smyrna, GA 30082-3723

Jeff Coffler
2005-06-06, 09:10
"John A. Tamplin" wrote:

>> *And* you could make a more costly cross-process call to the database
>> process to find out the name of the song, once every three minutes, and
>> avoid cross-process database work altogether. Just like now I can ask
>> the CLI what is the artist of this song, so could the hardware driving
>> process.
>
> [...] Unless it is multithreaded (and most databases I know of, including
> commercial ones, have limited multithreading support -- usually you need a
> separate process for each connection for best results), you are now in a
> worse situation in that any long-running query will block the streaming.

This is total hogwash. I work with both free and commercial databases all
day long in my job, developing a carrier-grade product that has hundreds (if
not thousands) of concurrent threads.

We work with both Berkeley DB (free) from Sleepycat Corporation and Oracle
(commercial). Both have excellent multi-threaded support. We generally
have *ONE* process, and we have no problems having many many threads
accessing the database concurrently. I'll grant you that things weren't so
rosy with Oracle in this regard, but that was - what - 10 years ago? There
have been many many releases where, quite simply, Oracle has no problems
with multithreaded access. And Berkeley DB has never had problems with
multithreaded access.

Obviously, the database backend chosen by Slim Devices isn't as well behaved
in this context, but then they didn't need it to be given their short-term
needs. But there's no question to me that backend databases exist that
behave quite well with multi-threaded access.

> No matter where you put it, if you have a single funnel point for database
> access you will still have a blocking problem. Either you need to have a
> database that supports multiple concurrent access paths or you have to do
> something fancier of caching the data you need outside the database (and
> the hassles of excess resource usage and keeping it up to date), which
> defeats much of the purpose of switching to the database in the first
> place (and you are reimplementing part of the job of the database).

Not sure I totally agree here. Obviously, you can't tolerate database
blocking here. But then an argument can be made that if there are two
processes (one being a streamer and one being a UI interface), the UI
interface should give all data it needs to the streaming process (via IPC or
some other mechanism). After all, if you're after performance, having the
UI interface look up a bunch of stuff from the DB, only to have the
streaming process do the same - that just seems like substandard performance
(compared to the "cost" for an IPC call).

>> I'm not saying it's trivial. Only necessary.
>
> Necessary for some needs. I agree it is a desirable goal, but while it
> would be great to have a deadline-driven realtime scheduler for the audio,
> the needs do not justify the effort required. If the average user starts
> running into problems with the single-threaded nature of the current code
> (and they might), then it will become for of a necessity.

Yeah, but this is more cloudy. With the SB2, I *NEVER* have dropouts
anymore (due to the much larger buffer). So this situation is really
user-specific *AND* hardware specific. I did have occasional dropouts with
the older SB1, but only when my UNIX system became quite busy and began
thrashing the disks. Disk thrashing was far more likely to cause dropouts
than CPU loading, but that's user-specific as well, I'd expect.

> The point remains that people with special requirements, including huge
> song databases, are not the typical customer and therefore can't be the
> focus of development efforts.

I'm not one to dictate or command Slim Devices to do anything. I ask
nicely, and perhaps they do it. Or I just do it myself and ask them to
commit my changes. Or, if it's an out and out "bug", then I post a bug
report. Those are my choices.

But here, we totally agree. It only makes sense for Slim (or any company)
to focus on average customers, which is the lion's share of where they make
their money from.

-- Jeff

Maditude
2005-06-06, 13:30
I've been following this thread for a while, and it seems to me that there's a pretty simple-to-code way around it, without any serious re-architecting. I'd be interested to hear what some of your Perl gurus think:

Make the database-rebuild fork a new process to create a NEW database (leaving the existing one alone). Then, when it completes, send a message to the main slimserver process, informing it that there's a newly built database. Then, slimserver could just close it's db-connection(s), discard the current database, rename the new database, and re-connect to it. It might still be slow enough that really low-spec servers would still have dropouts, and the extra disk-space consumed could be a concern, but it seems like it oughta work, and put an end to a whole lotta griping here.

Dan Sully
2005-06-06, 13:55
* Maditude shaped the electrons to say...

>Make the database-rebuild fork a new process to create a NEW database
>(leaving the existing one alone). Then, when it completes, send a
>message to the main slimserver process, informing it that there's a
>newly built database. Then, slimserver could just close it's
>db-connection(s), discard the current database, rename the new
>database, and re-connect to it. It might still be slow enough that
>really low-spec servers would still have dropouts, and the extra
>disk-space consumed could be a concern, but it seems like it oughta
>work, and put an end to a whole lotta griping here.

That's a good stop gap (or even full gap!) suggestion. Thanks.

The disk usage isn't really a concern, especially in relation to the size of
people's music collections. :)

-D
--
<iNoah> all your base class are belong to us

Dan Sully
2005-06-06, 14:05
* John A. Tamplin shaped the electrons to say...

>That might work ok when you know how the underlying database is stored,
>which might be feasible for SQLite. However, in general you have no
>information about what the underlying database does with its data, you
>probably don't have permission to move it around, would require
>cooperation with potentially other processes accessing the database, and
>it may reside on another machine in raw partitions rather than disk files.
>Even if you did it all through SQL, it would not be terribly faster to
>read from the newly built database and insert into the current database
>than just doing it there to start with, and you have other issues with
>locking to prevent updates from stepping on each other.

Well, right now we have some concurrency issues with SQLite. I'll try to
explain more, with some thoughts a bit later in the day.

-D
--
<iNoah> my pdp goes to 11.

Michaelwagner
2005-06-06, 14:26
Make the database-rebuild fork a new process to create a NEW database (leaving the existing one alone). Then, when it completes, send a message to the main slimserver process, informing it that there's a newly built database. Then, slimserver could just close it's db-connection(s), discard the current database, rename the new database, and re-connect to it. It might still be slow enough that really low-spec servers would still have dropouts, and the extra disk-space consumed could be a concern [....]

My 2 cents ... this assumes database concurrency during scan/rebuild is the problem to be solved. From what I've seen on my system and what I've read here, that isn't clear.

And for those where memory is the limiting resource (not my case, however, may be for others), this would probably double the memory footprint during a rescan, making life for the fork with real-time hardware responsibility worse rather than better.

Michael

Michaelwagner
2005-06-06, 14:33
if there are two processes (one being a streamer and one being a UI interface), the UI interface should give all data it needs to the streaming process (via IPC or some other mechanism). After all, if you're after performance, having the UI interface look up a bunch of stuff from the DB, only to have the streaming process do the same - that just seems like substandard performance (compared to the "cost" for an IPC call).
Exactly. Hence there are no cross-process database issues.

The UI tells the streamer everything it conceivably needs to know, title, album, data stream details if need be (why these would be stored in the db I have no idea, but if they are, them too) in one cross-process message to start the song.

This is what I tried to say in a previous posting. Jeff said it better than I did.

Michael

Michaelwagner
2005-06-09, 19:27
I said I would check in here from time to time with any results that seem generally applicable.

Over the last few nights I have spent some time sprinkling debug statements hither and yon and have a few interesting results to share.

Reading the MP3 tags cost about 5 ms.

Reading the MP3 info (stream type, frequency, time in seconds, etc) cost about 120-150 to as much as 210ms. (!!!!)

These first 2 occur in the CPAN MP3 code.

Then there is some more (slim) code in the wrapper function that goes fishing around for various audio frames, to correct errors in the CPAN module (if the (sparse) comments are correct). This part takes about 15 to 25 ms.

So in fact, it isn't the MP3 tag reading that is the CPU pig, it's the other two, especially the stream info stuff.

More info when I know more.

Michael

Michaelwagner
2005-10-11, 07:45
I'd like to follow up on this thread with a needs analysis question:

Much of the time during scan seems to be spent on things that were important to the CPAN writers, but perhaps not so much to the SLIM user.

In particular, the mp3 analysis code seems to spend a lot of resources on determining the length of the song, to a high degree of accuracy. For a human, the nearest second (or perhaps 10th) would be more than accurate, and faster techniques exist to estimate that (at least for CBR, don't know much about VBR). For me, at least, everything I do is CBR, so it would be a big savings to use a lower overhead technique.

So here's my needs analysis question: does the decoder (or some other code component) need a highly accurate song duration? How accurate? and why?

Because if it doesn't need to be so accurate, there's a lot of time and I/O resource to be saved there.

I don't mean to be rude, but I'm looking for an answer with some authority. So tell me if you *know* because you've been through a place in the code where it matters. Please don't speculate, or at least label your speculation as such. What I'll be proposing, if it turns out no-one and nothing needs this accuracy, is at least a half order of magnitude speedup for scanning mp3s, more if your I/O is far away (over a network, on a slug, on an external hard disk, etc) but it relies critically on knowing the lesser accuracy doesn't break anything.

radish
2005-10-11, 10:51
I can't see how you'd have any hope of getting gapless transitions to work if you didn't know exactly how long the track is. But seeing as we don't have gapless mp3 now anyway, it's not a loss.

Michaelwagner
2005-10-11, 11:01
I can't see how you'd have any hope of getting gapless transitions to work if you didn't know exactly how long the track is.
Yes, gapless is one of the concerns I had.
Surely someone/some part of the code has to know how long the track is, but who and when is the issue. Does the info in the database feed the decoder and tell it when to shut off? Seems unlikely, to my eyes anyways. As I read the code, the streamer stops sending data to the decoder, and waits for the decoder to stop. The streamer doesn't (seem to) tell the decoder when to stop. So the decoder must be finding some kind of end-of-stream mark in the datastream, not relying on a length.


But seeing as we don't have gapless mp3 now anyway, it's not a loss.
Fair enough. But no one wants to make a performance improvement that impedes further quality improvements down the line.

dean
2005-10-11, 15:01
The original motivation (I wrote that code) was to know exactly where
the frame boundaries were at the beginning and end of the file.
Squeezebox1 would drop out its S/PDIF output if it lost frame sync,
which means that in the case that the MP3 frames didn't end and begin
exactly after and before the ID3 tags, there'd be an audible glitch.

I asked Dan how hard it would be to rewrite the portion of the
MP3::Info module in C that accounts for 90% of the time in scanning a
given set of MP3 files. Dan?


-dean


On Oct 11, 2005, at 11:01 AM, Michaelwagner wrote:

>
> radish Wrote:
>
>> I can't see how you'd have any hope of getting gapless transitions to
>> work if you didn't know exactly how long the track is.
>>
> Yes, gapless is one of the concerns I had.
> Surely someone/some part of the code has to know how long the track
> is,
> but who and when is the issue. Does the info in the database feed the
> decoder and tell it when to shut off? Seems unlikely, to my eyes
> anyways. As I read the code, the streamer stops sending data to the
> decoder, and waits for the decoder to stop. The streamer doesn't (seem
> to) tell the decoder when to stop. So the decoder must be finding some
> kind of end-of-stream mark in the datastream, not relying on a length.
>
>
>> But seeing as we don't have gapless mp3 now anyway, it's not a loss.
>>
> Fair enough. But no one wants to make a performance improvement that
> impedes further quality improvements down the line.
>
>
> --
> Michaelwagner
>

Michaelwagner
2005-10-11, 15:17
I asked Dan how hard it would be to rewrite the portion of the MP3::Info module in C that accounts for 90% of the time in scanning a given set of MP3 files.
It's been a while since I looked at that code (about 2 months, I think), but I seem to recall the problem wasn't CPU it was I/O. It seeks about a lot. Not sure if C would help (or help enough) but it might be worth a try.

A better thing yet to consider is whether it needs to be done ahead of time and stored in the database. Slim is going to read every byte as it dribbles it out to the device ... at which point the bytes will already all be in memory ... isn't that soon enough?

Michael

dean
2005-10-11, 15:41
On Oct 11, 2005, at 3:17 PM, Michaelwagner wrote:
> dean Wrote:
>
>> I asked Dan how hard it would be to rewrite the portion of the
>> MP3::Info
>> module in C that accounts for 90% of the time in scanning a given
>> set of
>> MP3 files.
>>
> It's been a while since I looked at that code (about 2 months, I
> think), but I seem to recall the problem wasn't CPU it was I/O. It
> seeks about a lot. Not sure if C would help (or help enough) but it
> might be worth a try.
>
> A better thing yet to consider is whether it needs to be done ahead of
> time and stored in the database.
It already is in the form of offset and length, but to get that
information, we have to parse the MP3 data to find the frames when a
song is scanned.

Michaelwagner
2005-10-11, 16:04
I wrote:
A better thing yet to consider is whether it needs to be done ahead of time and stored in the database.


It already is in the form of offset and length, but to get that information, we have to parse the MP3 data to find the frames when a song is scanned.
Sorry, Dean. I was doing too many things at once and wrote something that could be interpreted ambiguously.

So let me try again.

I am suggesting that we NOT parse the MP3 data at song scan time to find the starting and ending frame, but rather only do that as we are actually playing the song. When playing the song, we already have it in storage. It might be easier and less costly to hunt around in it then.