PDA

View Full Version : svn 2111 - corrupt database produced



Triode
2005-02-18, 11:44
Dan,

I seem to have a problem with 2111 (and 2110) producing a corrupt database. I've just tried 2111 now and it did the same thing as
2110 yesterday. [NB not applied your patch for speeding up scanning]

I deleted the database and let it scan, the resulting data base file is smaller than normal. However the critical thing is that
when you browse entries in the menus (web or player) no actual tracks are found. Its also possible to crash the server by trying to
play the empty play list.

Is there a way to test the consitency of the database?

Happy to produce appropriate debugs - I've just started looking at this.

Adrian

Dan Sully
2005-02-18, 11:57
* Triode shaped the electrons to say...

>I seem to have a problem with 2111 (and 2110) producing a corrupt database.
>I've just tried 2111 now and it did the same thing as 2110 yesterday. [NB
>not applied your patch for speeding up scanning]
>
>I deleted the database and let it scan, the resulting data base file is
>smaller than normal. However the critical thing is that when you browse
>entries in the menus (web or player) no actual tracks are found. Its also
>possible to crash the server by trying to play the empty play list.
>
>Is there a way to test the consitency of the database?
>
>Happy to produce appropriate debugs - I've just started looking at this.

The best thing to do is look at the database with the sqlite3 command tool.

It can be downloaded from http://www.sqlite.org/download.html

-D
--
<nil> It sucks to discover that you are the foremost authority on some set of things when you've got a problem.

Triode
2005-02-18, 11:58
Is the following normal when you browse by album (results in empty display)? Not sure whether it is building the database or
searching it where the problem is:

2005-02-18 18:53:42.0902 Running SQL query: [SELECT DISTINCT albums.id AS id,albums.title AS title,albums.titlesort AS
titlesort,albums.contributors AS contributors,albums.artwork_path AS artwork_path,albums.disc AS disc,albums.discc AS
discc,albums.musicmagic_mixable AS musicmagic_mixable FROM albums ORDER BY albums.titlesort, albums.disc]
2005-02-18 18:53:42.1630 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:42.1654 Start and End node: [album:album]
2005-02-18 18:53:42.1678 Backtrace:

frame 0: Slim::DataStores::DBI::DataModel::find (/usr/local/slimserver/Slim/DataStores/DBI/DBIStore.pm line 253)
frame 1: Slim::DataStores::DBI::DBIStore::find (/usr/local/slimserver/Slim/Buttons/BrowseID3.pm line 708)
frame 2: Slim::Buttons::BrowseID3::lines (/usr/local/slimserver/Slim/Display/Display.pm line 120)
frame 3: Slim::Display::Display::curLines (/usr/local/slimserver/Slim/Buttons/Common.pm line 1193)
frame 4: Slim::Buttons::Common::pushModeLeft (/usr/local/slimserver/Slim/Buttons/Home.pm line 347)
frame 5: Slim::Buttons::Home::homeExitHandler (/usr/local/slimserver/Slim/Buttons/Input/List.pm line 279)
frame 6: Slim::Buttons::Input::List::exitInput (/usr/local/slimserver/Slim/Buttons/Input/List.pm line 63)
frame 7: Slim::Buttons::Input::List::__ANON__ (/usr/local/slimserver/Slim/Hardware/IR.pm line 613)
frame 8: Slim::Hardware::IR::executeButton (/usr/local/slimserver/Slim/Control/Command.pm line 572)
frame 9: Slim::Control::Command::execute (/usr/local/slimserver/Slim/Hardware/IR.pm line 637)
frame 10: Slim::Hardware::IR::processCode (/usr/local/slimserver/Slim/Hardware/IR.pm line 455)
frame 11: Slim::Hardware::IR::processIR (/usr/local/slimserver/Slim/Control/Command.pm line 572)
frame 12: Slim::Control::Command::execute (/usr/local/slimserver/Slim/Hardware/IR.pm line 87)
frame 13: Slim::Hardware::IR::idle (./slimserver.pl line 539)
frame 14: main::idle (./slimserver.pl line 505)
frame 15: main::main (./slimserver.pl line 1025)

2005-02-18 18:53:42.1689 Running SQL query: [SELECT DISTINCT albums.id AS id,albums.title AS title,albums.titlesort AS
titlesort,albums.contributors AS contributors,albums.artwork_path AS artwork_path,albums.disc AS disc,albums.discc AS
discc,albums.musicmagic_mixable AS musicmagic_mixable FROM tracks, albums WHERE albums.id = tracks.album AND ( tracks.album = ? )]
2005-02-18 18:53:42.1695 Bind arguments: [2]

2005-02-18 18:53:43.1841 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:43.1852 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:44.1847 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:44.1858 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:45.1855 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:45.1866 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:46.1863 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:46.1874 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:47.1869 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:47.1880 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:48.1877 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:48.1888 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:49.1886 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:49.1897 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:50.1893 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:50.1904 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:51.1900 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:51.1911 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:52.1909 Generated findKey: [album:album:10000 Maniacs MTV unplugged::::]
2005-02-18 18:53:52.1920 Used previous results for findKey: [album:album:10000 Maniacs MTV unplugged::::]

----- Original Message -----
From: "Triode" <triode1 (AT) btinternet (DOT) com>
To: "Slim Devices Developers" <developers (AT) lists (DOT) slimdevices.com>
Sent: Friday, February 18, 2005 6:44 PM
Subject: [Developers] svn 2111 - corrupt database produced


> Dan,
>
> I seem to have a problem with 2111 (and 2110) producing a corrupt database. I've just tried 2111 now and it did the same thing as
> 2110 yesterday. [NB not applied your patch for speeding up scanning]
>
> I deleted the database and let it scan, the resulting data base file is smaller than normal. However the critical thing is that
> when you browse entries in the menus (web or player) no actual tracks are found. Its also possible to crash the server by trying
> to play the empty play list.
>
> Is there a way to test the consitency of the database?
>
> Happy to produce appropriate debugs - I've just started looking at this.
>
> Adrian
>

Dan Sully
2005-02-18, 12:00
* Triode shaped the electrons to say...

>Is the following normal when you browse by album (results in empty
>display)? Not sure whether it is building the database or searching it
>where the problem is:
>
>2005-02-18 18:53:42.0902 Running SQL query: [SELECT DISTINCT albums.id AS
>id,albums.title AS title,albums.titlesort AS titlesort,albums.contributors
>AS contributors,albums.artwork_path AS artwork_path,albums.disc AS
>disc,albums.discc AS discc,albums.musicmagic_mixable AS musicmagic_mixable
>FROM albums ORDER BY albums.titlesort, albums.disc]

It seems to be missing a WHERE clause.

-D
--
<nil> It sucks to discover that you are the foremost authority on some set of things when you've got a problem.

kdf
2005-02-18, 12:05
Quoting Triode <triode1 (AT) btinternet (DOT) com>:

> Is the following normal when you browse by album (results in empty display)?
> Not sure whether it is building the database or
> searching it where the problem is:

--d_info when starting from a blank database will show you each NewTrack and
what attributes are added for each. That will help determine if the data being
read makes sense. sqlite3 can check that the data actual writes to the db
correctly.

testing the output is a bit trickier. I usually end up putting in temporary
markers and messages depending on what and where the problem is.

I assume you are using default skin. Is this just folder scan or are you using
any of the importers? I haven't noted problems like this one, with the
exception of browse playlist, which was showing all blanks for a rev or two.

Dan's optimisation is really great, btw. It took me a while to actually grasp
what it is doing, once I did I can say its an interesting idea. I'd recommend
trying it out. Hopefully that can get committed to the trunk soon :)

-kdf

Triode
2005-02-18, 13:58
Dan,

Has the database changed between 2077 and 2011 radically (checkin logs does not seem to say so). 2077 is my prevous working copy.

For a single folder scanned with 2077, I get a database which gives:

sqlite> select * from tracks;
1|file:///home/samba/My%20Music/10000%20Maniacs|10000 Maniacs|10000 MANIACS|1||dir|1|1097323184||||||||||||||||||||||| |||
2|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged|||1||||110505332 2||||||||||||||||||||||||||
3|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged/01%20These%20are%20days%20Maniac.flac|These are
days Maniac|THESE ARE DAYS
MANIAC|2|1|flc|1|1089922112|35276316|35276316|4844 |1993|294.293333333333||||||958811.308445089|44100 ||2||||||||||||
4|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged/02%20Eat%20for%20Two%20Maniacs.flac|Eat for Two
Maniacs|EAT FOR TWO
MANIACS|2|2|flc|1|1089922112|26679134|26679134|478 8|1993|262.506666666667||||||812911.804144657|4410 0||2||||||||||||
5|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged/03%20Candy%20everybody%20wants%20Maniac.flac|Candy
everybody wants Maniac|CANDY EVERYBODY WANTS
MANIAC|2|3|flc|1|1089922112|24361862|24361862|4671 |1993|199.6||||||976240.120240481|44100||2|||||||| ||||
6|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged/04%20I%20am%20not%20the%20man%20Maniac.flac|I am
not the man Maniac|I AM NOT THE MAN
MANIAC|2|4|flc|1|1089922112|23086272|23086272|4720 |1993|226.36||||||815746.669022796|44100||2||||||| |||||
.....

For the same folders scanned with 2011, I get a database with less entries in the table:
sqlite> select * from tracks;
1|file:///home/samba/My%20Music/10000%20Maniacs|||1||||1097323184||||||||||||||||| |||||||||
2|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged|||1||||110505332 2||||||||||||||||||||||||||
3|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged/01%20These%20are%20days%20Maniac.flac||||||||||||| ||||||||||||||||||||
4|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged/02%20Eat%20for%20Two%20Maniacs.flac||||||||||||||| ||||||||||||||||||
5|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged/03%20Candy%20everybody%20wants%20Maniac.flac|||||| |||||||||||||||||||||||||||
6|file:///home/samba/My%20Music/10000%20Maniacs/10000%20Maniacs%20MTV%20unplugged/04%20I%20am%20not%20the%20man%20Maniac.flac||||||| ||||||||||||||||||||||||||

My database knowledge is very rusty, but this seems to suggest that 2011 is putting less fields into the database? This is backed up
by the database being a few % smaller.

I am scanning a folder [I've changed to only scan one directory]

Adrian

Triode
2005-02-18, 15:59
Kdf,

Problem appears to be change for subversion 2102:

"Set up Importers to NOT readTags" in DBIStore.pm

I changed:
$track = $self->newTrack($url, $attributeHash, undef, !$noTags);

Back to
$track = $self->newTrack($url, $attributeHash, undef, 1);

And I get the right database again!

Adrian



----- Original Message -----
From: "Triode" <triode1 (AT) btinternet (DOT) com>
To: "Slim Devices Developers" <developers (AT) lists (DOT) slimdevices.com>
Sent: Friday, February 18, 2005 8:58 PM
Subject: Re: [Developers] svn 2111 - corrupt database produced


> Dan,
>
> Has the database changed between 2077 and 2011 radically (checkin logs does not seem to say so). 2077 is my prevous working copy.
>
> For a single folder scanned with 2077, I get a database which gives:
>

kdf
2005-02-18, 16:10
Quoting Triode <triode1 (AT) btinternet (DOT) com>:

> Kdf,
>
> Problem appears to be change for subversion 2102:
>
> "Set up Importers to NOT readTags" in DBIStore.pm
>
> I changed:
> $track = $self->newTrack($url, $attributeHash, undef, !$noTags);
>
> Back to
> $track = $self->newTrack($url, $attributeHash, undef, 1);
>
> And I get the right database again!

argh. ok, well, maybe that was one step ahead. it will most certainly be
required after dan's patch as it is a complete waste to read tags again for
each importer. bloody pain these things are, and I dont even use them myself
:)

-kdf

Triode
2005-02-18, 16:44
OK - I've now applied Dan's patch and it works as previous!!

I suggest Dan's code gets comitted or we regress the tag thing below.

Adrian
----- Original Message -----
From: "kdf" <slim-mail (AT) deane-freeman (DOT) com>
To: "Slim Devices Developers" <developers (AT) lists (DOT) slimdevices.com>
Sent: Friday, February 18, 2005 11:10 PM
Subject: Re: [Developers] svn 2111 - corrupt database produced


> Quoting Triode <triode1 (AT) btinternet (DOT) com>:
>
>> Kdf,
>>
>> Problem appears to be change for subversion 2102:
>>
>> "Set up Importers to NOT readTags" in DBIStore.pm
>>
>> I changed:
>> $track = $self->newTrack($url, $attributeHash, undef, !$noTags);
>>
>> Back to
>> $track = $self->newTrack($url, $attributeHash, undef, 1);
>>
>> And I get the right database again!
>
> argh. ok, well, maybe that was one step ahead. it will most certainly be
> required after dan's patch as it is a complete waste to read tags again for
> each importer. bloody pain these things are, and I dont even use them myself
> :)
>
> -kdf
>

Dan Sully
2005-02-18, 16:47
* Triode shaped the electrons to say...

>OK - I've now applied Dan's patch and it works as previous!!
>
>I suggest Dan's code gets comitted or we regress the tag thing below.

I'm doing some final testing, and it'll get checked in.

-D
--
It does not do to leave a live Dragon out of your calculations..