Tag Info Storing over restarts

[Richard Purdie]

Attached are updated copies of the patches I use to enable caching of the
tag database between restarts.

It results in a slightly longer start-up time before you can use the server
but it doesn't have to read all the files from scratch so it stops the
server having to churn your disks up after each restart. To enable set the
usetagdatabase option in your prefs file to 1. It needs Storable which is
included in perl 5.8.

Dean: There's a reference to Slimp3::Misc::msg in Slim/Music/Info.pm which
has escaped and this patch corrects. This patch does nothing unless you
enable usetagdatabase so could we talk about including it in CVS please? If
I knew it was going to be used, I can enhance it further as well.

RP

[dean]

Thanks, Richard.

The current plan is to use the same XML format as the iTunes XML
database we already scan at startup, rather than tying to a DBM file.
This lets us start up immediately and also offers the possibility of
having other applications parse the XML data, so I'm going to hold off
on merging your patch.

-dean


On Nov 25, 2003, at 4:35 AM, Richard Purdie wrote:

> Attached are updated copies of the patches I use to enable caching of
> the
> tag database between restarts.
>
> It results in a slightly longer start-up time before you can use the
> server
> but it doesn't have to read all the files from scratch so it stops the
> server having to churn your disks up after each restart. To enable set
> the
> usetagdatabase option in your prefs file to 1. It needs Storable which
> is
> included in perl 5.8.
>
> Dean: There's a reference to Slimp3::Misc::msg in Slim/Music/Info.pm
> which
> has escaped and this patch corrects. This patch does nothing unless you
> enable usetagdatabase so could we talk about including it in CVS
> please? If
> I knew it was going to be used, I can enhance it further as well.
>
> RP
> <rp.diff><rp2.diff>

[Richard Purdie]

> Dean:
> The current plan is to use the same XML format as the iTunes XML
> database we already scan at startup, rather than tying to a DBM file.

My patch doesn't tie to a DBM file. It uses storable to dump the contents of
a hash to disk. Upon startup it can be instantly reloaded.

> This lets us start up immediately and also offers the possibility of
> having other applications parse the XML data,

The above approach also loads instantly. The "slight" delay is because I put
a parser in to check the timestamp of the data against the timestamps of
files on disk. I did this to stop dead data staying in the cache and it's a
fairly quick check as you don't need to access the files themselves, just
the FAT. I'd be happy to discuss alternative ways of doing this. (I'd
appreciate someone with a large file library commenting on the speed as
well).

I agree with creating a function to dump the data to an XML file in the same
form as iTunes. Using is as the store of the cached data doesn't make 100%
sense to me however as there is much more overhead involved in XML parsing
compared to Storable.

For reference, the patch I've submitted tidies up the currently badly broken
caching code (which uses MLDBM) and will make any future system easier to
implement. Even if you do go down the iTunes route, this patch does help...

RP

[dean]

Fair enough. I'll take a look after 5.0.1 goes out.

Thanks.

-dean

On Nov 25, 2003, at 8:16 AM, Richard Purdie wrote:

>> Dean:
>> The current plan is to use the same XML format as the iTunes XML
>> database we already scan at startup, rather than tying to a DBM file.
>
> My patch doesn't tie to a DBM file. It uses storable to dump the
> contents of
> a hash to disk. Upon startup it can be instantly reloaded.
>
>> This lets us start up immediately and also offers the possibility of
>> having other applications parse the XML data,
>
> The above approach also loads instantly. The "slight" delay is because
> I put
> a parser in to check the timestamp of the data against the timestamps
> of
> files on disk. I did this to stop dead data staying in the cache and
> it's a
> fairly quick check as you don't need to access the files themselves,
> just
> the FAT. I'd be happy to discuss alternative ways of doing this. (I'd
> appreciate someone with a large file library commenting on the speed as
> well).
>
> I agree with creating a function to dump the data to an XML file in
> the same
> form as iTunes. Using is as the store of the cached data doesn't make
> 100%
> sense to me however as there is much more overhead involved in XML
> parsing
> compared to Storable.
>
> For reference, the patch I've submitted tidies up the currently badly
> broken
> caching code (which uses MLDBM) and will make any future system easier
> to
> implement. Even if you do go down the iTunes route, this patch does
> help...
>
> RP
>
>

[Robert Moser II]

>
> For reference, the patch I've submitted tidies up the currently badly broken
> caching code (which uses MLDBM) and will make any future system easier to
> implement. Even if you do go down the iTunes route, this patch does help...
>
> RP
>

I loaded up your patch and things seem to work ok. I haven't done any
timing to see how much time it saves over a full scan. One thing I did
notice though, when I tried it on a standard install on Debian (Woody),
Storable was not available. apt-get install libstorable-perl took care
of that problem, but it made me wonder about how prevalent Storable is.

Robert

[Dan Sully]

* Jason Holtzapple <jasonholtzapple (AT) yahoo (DOT) com> shaped the electrons to say...

> --- Robert Moser II <rlmoser (AT) earthlink (DOT) net> wrote:
> > ... but it made me wonder about how prevalent Storable is.
>
> As far as I know Storable was added to the standard perl
> install for v5.8.

That is correct. 5.8.0+ includes it by default.

-D
--
Dammit, my hovercraft's full of eels again.

[Richard Purdie]

Robert Moser II:
> I loaded up your patch and things seem to work ok. I haven't done any
> timing to see how much time it saves over a full scan. One thing I did
> notice though, when I tried it on a standard install on Debian (Woody),
> Storable was not available. apt-get install libstorable-perl took care
> of that problem, but it made me wonder about how prevalent Storable is.

Out of all the options, Storable is probably the best choice as it's a
standard part of perl 5.8. What I'd like to see would be the non functional
MLDBM code replaced with the Storable code so that anyone with Storable can
use the caching but leave it off by default for now. I'd imagine some of the
people here on the dev list would be happy to give it a try and report on
the usefulness of it.

If it proves to be stable (which I don't see why it shouldn't - I've been
using the code for the last year?), when ready it can be enabled by default.
I'm not sure how the windows builds are done these days - is support for
Storable available in them? If so, it just leaves the mac platform (does
this use perl 5.8?) and linux where you can add the Storable modules if
necessary - we can even have the server exit with an error explaining what
to do in those cases.

RP