PDA

View Full Version : Library Scanning engine



ob_kook
2007-09-27, 20:28
Has the Library Scanning Engine changed much over the past couple of major Slimserver releases?

I recall there was a time when there was a multitude of threads on this forum with questions and complaints about long scanning times for large libraries, but I don't seem to see that many now.

The reason I ask is that I also noticed extremely long scan times in the past and I addressed the problem by installing a self learning predictive I/O scheduler and cache engine (which my company had in beta and which an engineer was kind enough to provide to me).

It certainly seemed to help the situation, but now I am not sure if there were other external factors (such as the version changes, hardware improvements, etc) which came into play as well.

Does anyone know?

MrSinatra
2007-09-27, 23:05
what are you talking about? what exactly is it that your engineer pal provided you with?

ob_kook
2007-09-27, 23:38
Sorry if I was being vague.

The beta software was a performance acceleration tool which dedicates a portion of the system memory for I/O cache together with a predictive algorithm for doing prefetching of data. The net effect is that in most databases and other I/O intensive applications, a 3-4x performance improvement is often observed. (It is a well known fact that in most applications, the performance bottleneck lies at the disk.)

Since mileage varies depending on systems and applications (for example, there would not be any improvement at all if the disk was *not* the bottleneck), I was curious if the slow library scans would be affected or not. And as a matter of fact, I did install the tool and saw huge improvments in many applications, but I did not do any specific testing against the library scans - just a subjective seat-of-the-pants feel of "yup, that's faster..."

This was all done quite some time ago, and I forgot about the tool and left it installed. Yesterday we announced release of the product, and that jogged my memory, so I did a search on the forums, and did not see as many posts about slow scans as I was expecting. I was curious if this is an issue that has been "fixed" or improved.

I plan on running tests again collecting empirical data, but I'm travelling for the next 3 weeks and therefore am left to muse.

MrSinatra
2007-09-27, 23:43
interesting, sounds like a great idea for a plugin.

i actually am not a good person to answer this, but in another thread i was trying to find out if setting exclusions for ss exe files in my avast AV will speed up library scans.

but i have no answers...

ob_kook
2007-09-27, 23:57
If you are the tinkering type, I see we put up a trial version on the website that can be downloaded. And here is a link to the press release if you're interested (it explains a lot better about the technology than I did!)

http://datacore.com/pressroom/pr_070926.asp

ob_kook
2007-10-01, 13:33
I am still travelling and have not been able to re-run my own tests, but InfoWorld magazine seems to have downloaded the trial and gave it a spin with some interesting reults.

Too interesting actually, they found a 15x performance increase on reads which they felt was too high to be real. That's benchmarks for you. I am looking forward to running real world scans when I get back from this business trip.

Article can be found here:
http://www.infoworld.com/article/07/09/28/39OPstorinside-datacore-uptempo_1.html

aubuti
2007-10-01, 15:39
Has the Library Scanning Engine changed much over the past couple of major Slimserver releases?
<snip>
Does anyone know?
Versions 6.5.x of slimserver and the 7.0alpha use MySQL for the database. Prior to that it used SQLite (I don't know how far back, but at least 6.3 and 6.2). Version 6.5 also split off scanning as a separate process, so that scanning doesn't prevent slimserver from doing other things, such as streaming music.

How much speedup in scanning you get from your tool versus the change to MySQL is, as they say, an empirical question.