PDA

View Full Version : Database Integration



2003-12-03, 11:23
I am curious what the status of the "Database Integration" currently
underway is.

-- ardent

dean
2003-12-03, 13:38
Just checked in Richard's patch to cache the Music Library
meta-information.

Developers: please use and abuse this so we can make it really solid
for our customers.

-dean


On Dec 3, 2003, at 10:23 AM, <ardent-slim (AT) ahmeni (DOT) org> wrote:

> I am curious what the status of the "Database Integration" currently
> underway is.
>
> -- ardent
>
>

Kevin Deane-Freeman
2003-12-03, 14:20
Quoting dean blackketter <dean (AT) slimdevices (DOT) com>:

> Just checked in Richard's patch to cache the Music Library
> meta-information.
>
> Developers: please use and abuse this so we can make it really solid
> for our customers.
>
> -dean
yer ugly and yer mother dresses you funny!

er..oh, you mean the code. Yup, looking forward to it. :)

Richard Purdie
2003-12-03, 17:00
> Just checked in Richard's patch to cache the Music Library
> meta-information.

Excellent! I'm having a look over it and will produce a patch to tidy up a
few things (mostly cosmetic).

I'd love some feedback on speed improvements...

> Developers: please use and abuse this so we can make it really solid
> for our customers.

:)

One thing on a related but slightly different note. In Info.pm there is a
line that sets the 'AGE' tag to the file system timestamp. I thought AGE was
meant to be a guide to how long the data had been cached for? It is equated
to the return value of Time::HiRes::time() elsewhere so I'm guessing this is
a bug?

I think someone had intended to age entries in the cache using this tag and
my caching code should preserve that but this line is wrong in that case.
Can anyone confirm my thoughts one way or the other?

RP

dean
2003-12-03, 17:12
The age is intended to be the time that the data was last updated.
Usually this is time stamp on a file or folder, but if the data is
extracted from, say, iTunes, that could be just the current time.

I'm not sure if the code's correct (I suspect that it's not) but that's
the intent.


On Dec 3, 2003, at 4:00 PM, Richard Purdie wrote:
>> Developers: please use and abuse this so we can make it really solid
>> for our customers.
>
> :)
>
> One thing on a related but slightly different note. In Info.pm there
> is a
> line that sets the 'AGE' tag to the file system timestamp. I thought
> AGE was
> meant to be a guide to how long the data had been cached for? It is
> equated
> to the return value of Time::HiRes::time() elsewhere so I'm guessing
> this is
> a bug?
>
> I think someone had intended to age entries in the cache using this
> tag and
> my caching code should preserve that but this line is wrong in that
> case.
> Can anyone confirm my thoughts one way or the other?

Richard Purdie
2003-12-03, 17:21
> The age is intended to be the time that the data was last updated.
> Usually this is time stamp on a file or folder, but if the data is
> extracted from, say, iTunes, that could be just the current time.
>
> I'm not sure if the code's correct (I suspect that it's not) but that's
> the intent.

That's fine. We just have two tags with the same info in :).

I'll merge mine into yours...

RP

Richard Purdie
2003-12-03, 18:08
I've had a hard look at that database code and checked a few things I was
worried about. It's all fine except someone added a bit of code I hadn't
noticed until now with a check of:

exists($infoCache{$file})

which you mustn't do!

isCached($file)

is much more friendly as it checked infoCacheDB as well...

The above error meant the caching code wasn't working at all because of
where it was. I thought it wasn't as fast as it should be... :)

The attached patch tidies up the code and sorts the above problem. It also
merged the AGE and TIMESTAMP tags as they were duplications (when I
originally implemented it there was good reason to keep them separate but
the code has since changed a lot).

I've enabled scanning of timestamps for directories. I'm not too sure if
that will work under windows but I'd like to see if it does. It can easily
be removed if it doesn't (and the code *should* dump cached data if it is at
all unsure about it's validity...).

RP

Richard Purdie
2003-12-03, 18:54
A further patch to make doubly sure we don't use bad cache data (to be
applied on top of the other one).

The code is a bit ugly but I don't see an easier way...

RP

Kevin Deane-Freeman
2003-12-03, 22:27
I tried this out, with the latest patches. CPU usage 99.97%. Mandrake Linux
9.1, Athlon 2000+, 512M Ram, 10k songs.

This was while streaming from shoutcast to squeezebox. Stopping the stream and
playing a local song dropped the usage down. Streaming did not cause 99% cpu
usage until this update. Startup also rails at 99% for some time.

Last note: $dbname should be .slimserver.db or SLIMSERVER.DB (slimp3 reference
is outdated)

-kdf

Quoting Richard Purdie <rpurdie (AT) rpsys (DOT) net>:

> A further patch to make doubly sure we don't use bad cache data (to be
> applied on top of the other one).
>
> The code is a bit ugly but I don't see an easier way...
>
> RP
>

dean
2003-12-03, 23:38
Kevin,
Do you think that the new cache code is causing CPU hogging on your
machine?
-dean

On Dec 3, 2003, at 9:27 PM, Kevin Deane-Freeman wrote:

> I tried this out, with the latest patches. CPU usage 99.97%.
> Mandrake Linux
> 9.1, Athlon 2000+, 512M Ram, 10k songs.
>
> This was while streaming from shoutcast to squeezebox. Stopping the
> stream and
> playing a local song dropped the usage down. Streaming did not cause
> 99% cpu
> usage until this update. Startup also rails at 99% for some time.
>
> Last note: $dbname should be .slimserver.db or SLIMSERVER.DB (slimp3
> reference
> is outdated)
>
> -kdf
>
> Quoting Richard Purdie <rpurdie (AT) rpsys (DOT) net>:
>
>> A further patch to make doubly sure we don't use bad cache data (to be
>> applied on top of the other one).
>>
>> The code is a bit ugly but I don't see an easier way...
>>
>> RP
>>
>
>
>
>

Robert Moser II
2003-12-03, 23:43
I'm seeing something similar, but I'm not entirely sure that it is the
fault of the Info changes. Try turning on d_remotestream. When I do
this I get a ton of garbage thrown up posing as metadata. I'm thinking
that there is something wrong in RemoteStream.

Kevin Deane-Freeman blurted out:

> I tried this out, with the latest patches. CPU usage 99.97%. Mandrake Linux
> 9.1, Athlon 2000+, 512M Ram, 10k songs.
>
> This was while streaming from shoutcast to squeezebox. Stopping the stream and
> playing a local song dropped the usage down. Streaming did not cause 99% cpu
> usage until this update. Startup also rails at 99% for some time.
>
> Last note: $dbname should be .slimserver.db or SLIMSERVER.DB (slimp3 reference
> is outdated)
>
> -kdf
>
> Quoting Richard Purdie <rpurdie (AT) rpsys (DOT) net>:
>
>
>>A further patch to make doubly sure we don't use bad cache data (to be
>>applied on top of the other one).
>>
>>The code is a bit ugly but I don't see an easier way...
>>
>>RP
>>
>
>
>
>
>

Kevin Deane-Freeman
2003-12-04, 01:27
Quoting dean blackketter <dean (AT) slimdevices (DOT) com>:

> Kevin,
> Do you think that the new cache code is causing CPU hogging on your
> machine?
> -dean

I went back and ran a version just before the database, and it was still 99% for
streaming, but playing back just fine. After a minute or so it started to
stutter. I stopped and ran the server again, started a stream and cpu usage was
only 3%. Good news is, I dont think its the database. Startup cpu usage was
99% for both versions. I didn't time how long. Memory usage seems about the
same, maybe a bit less using the database.

-kdf


> On Dec 3, 2003, at 9:27 PM, Kevin Deane-Freeman wrote:
>
> > I tried this out, with the latest patches. CPU usage 99.97%.
> > Mandrake Linux
> > 9.1, Athlon 2000+, 512M Ram, 10k songs.
> >
> > This was while streaming from shoutcast to squeezebox. Stopping the
> > stream and
> > playing a local song dropped the usage down. Streaming did not cause
> > 99% cpu
> > usage until this update. Startup also rails at 99% for some time.
> >
> > Last note: $dbname should be .slimserver.db or SLIMSERVER.DB (slimp3
> > reference
> > is outdated)
> >
> > -kdf
> >
> > Quoting Richard Purdie <rpurdie (AT) rpsys (DOT) net>:
> >
> >> A further patch to make doubly sure we don't use bad cache data (to be
> >> applied on top of the other one).
> >>
> >> The code is a bit ugly but I don't see an easier way...
> >>
> >> RP
> >>
> >
> >
> >
> >

Kevin Deane-Freeman
2003-12-04, 01:42
Quoting Robert Moser II <rlmoser (AT) earthlink (DOT) net>:

> I'm seeing something similar, but I'm not entirely sure that it is the
> fault of the Info changes. Try turning on d_remotestream. When I do
> this I get a ton of garbage thrown up posing as metadata. I'm thinking
> that there is something wrong in RemoteStream.
>
Using Slimp3, I get a lot of:
2003-12-04 00:39:24 metadata size: 0
about once a second.

using squeeze, I get the same thing but a lot more...
it was nonprintable characters, and it locked my console. Its something like
the same thing that happens when you try to cat a binary file (dunno what this
might end up like for some. sorry if its horrible):

2003-12-04 00:39:51 metadata: M=`sb( `2
*itj޿X+Ud\ħB
.7$pEæ ~o
R ʦzj0k:o(mrϹqq`'$4I7 0OT)@)c]MZ