Why does a playlist-only scan require a full database cleanup?

[smr888]

I do not understand why a playlist-only scan should require a cleanup of the database.

I have been asked not to discuss this in the bug tracker; apparently extremely long, potentially unnecessary scanning is not considered a bug. (Possibly it's a feature?)

My comments from bug 6308 are listed below:

*****

I have four playlists. The scan of those playlists finishes in 28 seconds.

I am then forced to sit through:

"Merge Various Artists" (1547 of them), takes 5:04

"Database Cleanup #1" (31782), takes 1:34

"Database Cleanup #2", might have finished in 2 minutes or so, hard to tell.

So basically 9 minutes to scan my four playlists, with a lot of scans I didn't ask for.

*****

At the risk of trying your patience, can you explain to me why
doing the playlist checks you mention should require [the above listed scans] . . .

From the outside, I would have thought that a user-initiated playlist-only scan would have meant something like this:

"I have created this playlist. Please add it to your list of playlists."


It apparently also seems to mean:

"Please make sure every song in the playlist is present in the database so you can play it."


which is fine, but I would not have thought it _also_ meant

"and while you're at it, conduct an eight minute cleanup of my database."


and I'm not completely sure why it should have to mean that. Especially when those other database cleanup scans take place AFTER the playlist scan (if the messages to the user are to be believed), meaning that the playlist scan doesn't even benefit from scanning a newly clean database, if that was supposed to be the point.

*****

Any help appreciated. I cannot see how these additional database scans and cleanups could under any condition be considered necessary. If you want a full database cleanup, then do a full scan! If you've added new music, do a full scan! (Or a "new music scan".)

But if I just want to create a playlist and add it to my list of playlists, that process should not require anything beyond checking to make sure that the playlist entries are in the database -- should it?

[andyg]

If a playlist contains items which are not in your database, the playlist scan adds them to the database. This is probably why a cleanup is required.

[smr888]

Thanks Andy. I will give that a try, and see if that is what is causing this behavior.

[smr888]

Nope, that is not it. I took an existing playlist, whose entries were already all in the database, chopped it down to four songs, saved it under a different name, and requested a playlist-only scan. The playlist portion, as usual, took about 30 seconds.

Now it is running "Merge Various Artists" -- 1:38

Now it is doing the database cleanups, which will take another five minutes or so.

Is this really necessary?

[Siduhe]

Andy, I have exactly the same experience as the OP. My system does a full clear and rescan at 4am and I have not added any music today. I use MusicIP, but no changes to the database/cache there either. I also get 2 database cleanups on a playlist only scan as follows (c/p):

Playlist Scan ( of ) Complete 00:00:03
Merge Various Artists (0 of 0) Complete 00:00:13
Database Cleanup #1 (0 of 0) Complete 00:00:20
Database Cleanup #2 (0 of ) Complete 00:00:17
Database Optimize (0 of ) Complete 00:00:14
SqueezeCenter has finished scanning your music collection.Total Time:00:01:07

It's not an issue for me because the scans are done pretty quickly (my playlists are short), but I don't think it can be as a result of new music needing to be added to the database.

[kdf]

Why does a playlist-only scan require a full databasecleanup?

On 21-Jan-08, at 8:25 AM, smr888 wrote:
>
> Is this really necessary?

for the time being, yes it is.
-kdf

[smr888]

>>for the time being, yes it is.<<

This is the second unhelpful response you've given me, kdf. Would you please not respond to me if you don't have something useful to say? I'd appreciate it. Your responses make me want to stop reporting bugs, and if that is your intention, you are succeeding admirably.

[kdf]

Why does a playlist-only scan require a full databasecleanup?

>
>>>for the time being, yes it is.<<
>
> This is the second unhelpful response you've given me, kdf.

I'm sorry you feel that way. However, it was intended as a very simple
response to a simple question. The process is there for many reasons.
Every check through playlists involves gathering new data (even if the
data is already in the database), which then needs post-scan processing to
sort out where it fits into the database. In time, we may work out ways
to get rid of post scan, but for now it's necessary. It isn't always about
NEW data, but doing as good a job as possible to match the playlist
information with all that has been done to organise the metadata for the
tracks referenced.

In many aspects, it's similar to a "check new and changed" scan, but in
this case only the files listed in playlists and the playlists themselves,
instead of the whole library.

-kdf

[smr888]

Thank you. Obviously there is more to the process than is dreamt of in my philosophy.

[kdf]

Why does a playlist-only scan require a full databasecleanup?

>
> Thank you. Obviously there is more to the process than is dreamt of in
> my philosophy.

If it helps, I believe the idea is that it should make db access cleaner,
faster and more stable later on.

What I don't know offhand is what the specifics are of phase '1' and '2'.

-kdf

[JJZolx]

Those aren't additional scans. They're phases of the scanning process.

Are they necessary when no new tracks have been added to the database? Probably not all of them. That could be raised as a bug or enhancement request. Lots of room for optimization.

[kdf]

Why does a playlist-only scan require a full databasecleanup?

>
> Lots of room for optimization.
>
Optimisation is an ingoing process, and not applicable as a bug report
unless there is a specific benchmark to be achieved. A goal without a
specific, measurable result is of dubious value.

The optimise process is simply a call for mysql to "optimise" the tables.
Even with no 'new data', there is manipulation. I would also be curious
as to how much activity the db is seeing not from scans, but from general
use: ratings updates, play counts, this-party plugin activity etc. All of
these would have an effect on the db and may play into the time taken for
mysql to
"OPTIMIZE TABLE tracks;"

However, as a fun investigation, try making schema_optimize.sql an empty
file and see what happens. It should certainly save time, but the result
on the db would probably depend on the contents of the db. This sql file
not only optimises tables, but rebuilds the contributor_albums in response
to bug 4882.

-kdf

[smr888]

I tried to raise this as a bug (that is, to discuss it within an already reported bug, 6308, that was similar to what I was commenting on) and was told to discuss enhancements and the implementation out here in the forums . . .

[JJZolx]

Optimisation is an ingoing process, and not applicable as a bug report
unless there is a specific benchmark to be achieved. A goal without a
specific, measurable result is of dubious value.

Not really, but you're entitled to your opinion.

If the server is doing A and B and C and D, while B, C and D are unnecessary in some cases, and taking 80% of the time required, then requesting that those processes be eliminated would seem to be a logical request.

[kdf]

Why does a playlist-only scan require a full databasecleanup?

>

> If the server is doing A and B and C and D, while B, C and D are
> unnecessary in some cases, and taking 80% of the time required, then
> requesting that those processes be eliminated would seem to be a
> logical request.

Eliminating isn't 'optimising'. Removing an entire process is a specific
goal, "make it faster' isn't.

If you want it removed, you can try this your self as I have described.
Feel free to post your results to the already mentioned bug report if you
like.

-kdf