PDA

View Full Version : split-scanner branch todo



Dan Sully
2006-03-21, 18:55
I've started a Todo list on the Wiki:

http://wiki.slimdevices.com/index.cgi?SplitScannerTodo

Feedback / updates welcome.

-D
--
<dr.pox> do they call it 'gq' because it makes your text fashionable?

dean
2006-03-21, 23:01
I'll post here to generate some discussion rather than edit immediately.

Installing:

For Mac and Windows installs, we'll definitely need to install pre-
built binaries for the MySQL as part of the slimserver installer and
run it separately from any other installed version of mysql on the
system. We can't afford for new, naive users to have conflicts with
existing installs and/or incompatibilities. For other unix installs,
building in dependencies for other mysql packages is perfectly
reasonable.

Launching:

Windows & Mac: We'll start up mysql when the server starts up (or
even have the server start the mysql engine)
Other Unix: let the system start it up.

User/db creation:

Installers on all platforms should do this, right?

SlimServer experience:

Let slimserver initiate a rescan by exec'ing a scanner process, in
response to commands. That process can set flags in the db as it's
scanning that the main server can watch for. Will need failsafe
mechanism to detect unexpected death of scanner process.






On Mar 21, 2006, at 5:55 PM, Dan Sully wrote:

> I've started a Todo list on the Wiki:
>
> http://wiki.slimdevices.com/index.cgi?SplitScannerTodo
>
> Feedback / updates welcome.
>
> -D
> --
> <dr.pox> do they call it 'gq' because it makes your text fashionable?
>

JJZolx
2006-03-22, 03:00
I'll post here to generate some discussion rather than edit immediately.

Installing:

For Mac and Windows installs, we'll definitely need to install pre-
built binaries for the MySQL as part of the slimserver installer and
run it separately from any other installed version of mysql on the
system. We can't afford for new, naive users to have conflicts with
existing installs and/or incompatibilities. For other unix installs,
building in dependencies for other mysql packages is perfectly
reasonable.
At the very least, on the Win and Mac platforms, SlimServer needs to be _able_ to run with a non-slim installed MySQL server. Even if that means experienced users have to go through some detailed step-by-step process of removing the installed MySQL server and pointing SlimServer at another, that would be acceptable.

It doesn't have to be easy for the experienced user, just doable.

I'm also hoping that it won't be asking too much that SlimServer be able to work with MySQL installed on another machine, rather than just localhost.



Launching:

Windows & Mac: We'll start up mysql when the server starts up (or
even have the server start the mysql engine)
Other Unix: let the system start it up.
Keep in mind the above. If SlimServer can run with a non-Slim installed MySQL server, then you can't assume you own and control the db server. Maybe just a basic config setting that tells SlimServer whether or not it it should control a local MySQL server's startup and shutdown.



SlimServer experience:

Let slimserver initiate a rescan by exec'ing a scanner process, in
response to commands. That process can set flags in the db as it's
scanning that the main server can watch for. Will need failsafe
mechanism to detect unexpected death of scanner process.

Wouldn't it be desirable to be able to launch the scanner from outside of SlimServer, say from a scheduled job? Maybe even without SlimServer itself running.

I think it would also be beneficial if the scanner were capable of being run on a machine separate from the SlimServer host. For instance, someone runs the server on a low powered server or NAS that needs many hours to scan a large library. They could run the scanner on their fast desktop PC and cut down the scanning time considerably. Again, this wouldn't need to be something the casual user could do easily.

mherger
2006-03-22, 03:40
I agree with all you say about the MySQL.

Having a separate DB server would allow for slimserver to run on a low
power NAS, mainly caring about streaming, while the DB is sitting in the
datacenter of your web hosting company... I guess that the delay in
communication might be lower than the slow response time of a low power
device. (I set up a system like this yesterday to check a compatibility
issue with different MySQL versions)

> I think it would also be beneficial if the scanner were capable of
> being run on a machine separate from the SlimServer host. For
> instance, someone runs the server on a low powered server or NAS that
> needs many hours to scan a large library. They could run the scanner
> on their fast desktop PC and cut down the scanning time considerably.

Keep in mind that these two devices don't see the same: while the NAS is
seeing local files, your computer sees a file on a share, where the server
might already have screwed up file names, hidden some files due to file
system limitations etc.

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
SlimString Translation Helper (http://www.herger.net/slim/)

Dan Sully
2006-03-22, 09:31
* JJZolx shaped the electrons to say...

>At the very least, on the Win and Mac platforms, SlimServer needs to be
>_able_ to run with a non-slim installed MySQL server. Even if that means
>experienced users have to go through some detailed step-by-step process
>of removing the installed MySQL server and pointing SlimServer at
>another, that would be acceptable.
>
>It doesn't have to be easy for the experienced user, just doable.
>
>I'm also hoping that it won't be asking too much that SlimServer be
>able to work with MySQL installed on another machine, rather than just
>localhost.

Yes to all of the above. We are certainly not planning on crippling the
install, and existing MySQL installations (mine included!) is a
consideration.

>Keep in mind the above. If SlimServer can run with a non-Slim installed
>MySQL server, then you can't assume you own and control the db server.
>Maybe just a basic config setting that tells SlimServer whether or not
>it it should control a local MySQL server's startup and shutdown.

Very good point. Thanks.

>Wouldn't it be desirable to be able to launch the scanner from outside
>of SlimServer, say from a scheduled job? Maybe even without SlimServer
>itself running.

It is desirable - for people like you and me. Again, not doing anything to
stop that functionality. Run it from cron:

0 */4 * * * /usr/local/slimserver/scanner.pl --rescan

No problem.

The 'launching' refers to the usage by 97% of the users who will go through
the Settings UI, and click rescan.

>I think it would also be beneficial if the scanner were capable of
>being run on a machine separate from the SlimServer host. For

Again - this should work just fine, assuming the mount paths to your music
library are the same.

-D
--
This knob controls the thing that changes when you turn it. - noah

Robin Bowes
2006-03-22, 13:10
dean blackketter wrote:
> I'll post here to generate some discussion rather than edit immediately.
>
> Installing:
>
> For Mac and Windows installs, we'll definitely need to install pre-
> built binaries for the MySQL as part of the slimserver installer and
> run it separately from any other installed version of mysql on the
> system. We can't afford for new, naive users to have conflicts with
> existing installs and/or incompatibilities. For other unix installs,
> building in dependencies for other mysql packages is perfectly reasonable.

It's worth mentioning that you can considerably reduce the MySQL memory
requirements by adjusting the config settings.

When I did my XBox install, I found that, having tuned the setup, MySQL
uses less memory [1] than SQLite and performs better.

[1] resident size

Resident size of MySQL is 3M - virtual size is 28M.

R.

JJZolx
2006-03-22, 13:54
That all sounds great. Should be the best of both worlds, once the details are worked out. I would think the scanner will need access to the slimserver pref file, so if run on another machine, a command-line switch to designate the prefs file might be necessary.

To continue the user experience angle: A very basic high-level scan log suitable for inexperienced users. Something viewable in the web UI server settings area.



2006-03-23 04:00:34am Begin full library scan, purge all existing data
2006-03-23 04:08:05am Begin 2nd pass, scanning artwork
2006-03-23 04:11:16am End scan, total time (hh:mm:ss) 00:10:42
And whatever other info might help a user see whether a scan did or did not complete successfully, and the time it took. Definitely add major errors here for troubleshooting purposes. If the scanning process dies, this is where the user needs to be able to go to find out what happened, which file was being processed when the error occurred. It would be nice if something like this could help people get past problems without having to turn on debugging options and dig up difficult to read, highly detailed logs.

A better, native scanning scheduler would be welcome. One in which all possible scanning options can be designated. The rescan buttons, scheduler settings, and access to the high-level scan log viewer should all be on the same page in server settings.