Slow scanning -- probably a FAQ

[Rutt]

With 6.3.0, the library scans seem very slow compared to my memory of older versions. My library is almost 6k albums and 80k songs. It used to take about 15 minutes to scan it. Now it's more like several hours, on much better newer hardware.

I see that the development version uses a separate process to scan. Is this much faster? Can someone describe it, please?

During the scan, the server is near 100% cpu usage, so I'm thinking this is not an I/O or memory issue, but something intrinsic to the way the code is written, perhaps some nonlinear behavior.

[Dan Sully]

* Rutt shaped the electrons to say...

>With 6.3.0, the library scans seem very slow compared to my memory of
>older versions. My library is almost 6k albums and 80k songs. It used
>to take about 15 minutes to scan it. Now it's more like several hours,
>on much better newer hardware.

Older versions didn't use a database backend, they kept it all in main
memory. Which is faster, but does not scale. Newer versions also fetch a lot
more data from the audio files, and do things like find artwork, etc.

>I see that the development version uses a separate process to scan. Is
>this much faster? Can someone describe it, please?

I've seen it be about 40% faster vs the 6.3.x scanning process.

When you scan in 6.5, we fork off a scanner process, which trolls through
your library in a linear fashion. Because it's in a separate process, the CPU
can context switch between it and other processes (specifically the main
slimserver), so interactive response is much better. The scanning process can
also run as a different priority, or even on a different machine if you so
choose.

-D
--
<dmercer> Because that is what our industry does.
Churns out useless shit. Followed by inferior re-implementations of useless shit.

[Rutt]

Older versions didn't use a database backend, they kept it all in main
memory. Which is faster, but does not scale. Newer versions also fetch a lot
more data from the audio files, and do things like find artwork, etc.

Here's what I find mysterious: during the scan the server is cpu bound. This means it isn't waiting for the database disk accesses or the reads of the music files in the library. It's something it's actually doing in the cpu. My music files don't have any artwork or fancy stuff. And I turned the artwork scanning off anyway.

I have some experience with sqlite and it was pretty good. Somehow I don't think it's causing this kind of overhead. What it can do if not used carefully is linearize disk writes. But then the server wouldn't show full cpu utilization while scanning.

So what am I missing? Is it understood? I'm willing to spend a little time on it, but I don't want to duplicate work that's already been done.

[stinkingpig]

On 7/7/06, Rutt <Rutt.2al6tn1152294901 (AT) no-mx (DOT) forums.slimdevices.com> wrote:
>
>
> Dan Sully Wrote:
> >
> > Older versions didn't use a database backend, they kept it all in main
> > memory. Which is faster, but does not scale. Newer versions also fetch
> > a lot
> > more data from the audio files, and do things like find artwork, etc.
> >
>
> Here's what I find mysterious: during the scan the server is cpu bound.
> This means it isn't waiting for the database disk accesses or the reads
> of the music files in the library. It's something it's actually doing
> in the cpu. My music files don't have any artwork or fancy stuff. And
> I turned the artwork scanning off anyway.
>
> I have some experience with sqlite and it was pretty good. Somehow I
> don't think it's causing this kind of overhead. What it can do if not
> used carefully is linearize disk writes. But then the server wouldn't
> show full cpu utilization while scanning.
>
> So what am I missing? Is it understood? I'm willing to spend a little
> time on it, but I don't want to duplicate work that's already been done.
>
>
Just turn on the scanning debugging and you'll get an idea of what it's
doing.
--
"I spent all me tin with the ladies drinking gin,
So across the Western ocean I must wander" -- traditional

[Fred]

Just turn on the scanning debugging and you'll get an idea of what it's doing.

I've recently updated it to really help you understand what it is doing. Given it is a different process, you need to launch it manually with --d_info.

I was even thinking maybe we could save this info with each track, visible from the web interface (if some pref is enabled)? Dan, Dean, any confirmation keeping this would make users life better?

Fred

[Dan Sully]

* Fred shaped the electrons to say...

>I was even thinking maybe we could save this info with each track,
>visible from the web interface (if some pref is enabled)? Dan, Dean,
>any confirmation keeping this would make users life better?

I don't understand what the benefit would be..

-D
--
There was supposed to be a big kaboom.

[Fred]

I don't understand what the benefit would be..

Well, some untechnical user could understand (a) setting a debug flag in the web interface and (b) rescanning its library, then (c) checking the songs he has trouble with to check (and maybe post) the info gathered by the scan process.

As an example, the debug messages makes it very clear what COMPILATION value Slimserver reads from the track. We all know how critical this info is and cannot deny the amount of forum bytes devoted to this very detail.
Or I am sure the info "same album as last track" could also explain a lot of strange reports.

Of course, the technical user would have used perl -d scanner.pl and understood it all by himself from the code. The point of the messages I added were to prevent that, but they still require launching scanner.pl manually from terminal, and redirecting the output to a file to get the info on a particular set of track when rescanning a large library.
If the server can keep the "logic" for each track and display it, the whole thing is a lot easier.

I am just suggesting making it easier for non techies to grasp the complex logic by which SlimServer understands their music library. I am certain YOU don't need it :-)

Fred