Home of the Squeezebox™ & Transporter® network music players.
Results 1 to 10 of 47

Hybrid View

  1. #1
    Senior Member
    Join Date
    Mar 2011
    Posts
    177

    Working on a plugin - Scanning question

    Hi,

    i'm working on a project where the library needs to be scanned. There are two scanning modes:
    1.) within main process of LMS
    2.) within the scanning module (import module)

    how do i trigger the 1. and the second programatically?

    Thanks

    mamema

  2. #2
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,538
    Hi mamema

    I moved your thread to the dev forum, as the DIY is more for the hardware tinkerers.

    > i'm working on a project where the library needs to be scanned. There
    > are two scanning modes:
    > 1.) within main process of LMS

    The scanner will always run in a separate process if a plugin is involved. You can ignore this use case. It's been this way for years (7.7?) now. I'd concentrate on LMS8 going forward.

    > 2.) within the scanning module (import module)

    You need to define an "importmodule" item in the install.xml file (see eg. https://github.com/michaelherger/Spo...nstall.xml#L10). This tells LMS which module to use in the importer.

    In your importer module, you'd define an initPlugin() function which would register the importer:

    Code:
    	
    	Slim::Music::Import->addImporter($class, {
    		'type'         => 'post',
    		'weight'       => 85,
    		'use'          => 1,
    	});
    The type defines what kind of importer this is. There basically are three: file (import some files into the library), post (post processing after the file scan has been done), artwork. They're executed in this order. Most likely you'd be using the post type, as you're not dealing with importing the files themselves, nor artwork, but want to manipulate the data that was read during the file scan.

    Weight helps the scanner to define an order in which the scanners are executed.

    The startScan() would be what is being called by the scanner when it's time to execute your code. Do whatever you need to do in there.

    Once you got this basically running, we can start to look into nice to haves, like reporting progress etc.
    Michael

    "It doesn't work - what shall I do?" - "Please check your server.log and/or scanner.log file!"
    (LMS: Settings/Information)

  3. #3
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    11,290
    Quote Originally Posted by mherger View Post

    The type defines what kind of importer this is. There basically are three: file (import some files into the library), post (post processing after the file scan has been done), artwork. They're executed in this order. Most likely you'd be using the post type, as you're not dealing with importing the files themselves, nor artwork, but want to manipulate the data that was read during the file scan.

    Weight helps the scanner to define an order in which the scanners are executed.

    The startScan() would be what is being called by the scanner when it's time to execute your code. Do whatever you need to do in there.
    Sorry for interrupting the discussion with a related question.

    Michael, would it be appropriate to implement a potentially long running operation that needs to be executed at LMS startup as an importer ?
    If so, how would the importer be triggered to run from initPlugin ? Should one just initiate a new/changed files rescan in initPlugin or is it a better way to do it ?

    Asking because I know some of my plugins does things at LMS startup that potentially can take a bit of time. The plugin doesn’t work properly before the operation is finished but it has always felt bad to risk hanging whole LMS. I use main::idleStreams() where it’s possible and I’ve also experimented a bit with Slim::Utils::Scheduler::add_task but for these to work good the plugin needs to be able to divide the work into smaller task which isn’t always easy.
    Last edited by erland; 2021-02-25 at 12:32.
    Erland Isaksson (My homepage)
    Developer of many plugins/applets
    Starting with LMS 8.0 I no longer support my plugins/applets (see here for more information )

  4. #4
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,538

    Working on a plugin - Scanning question

    > Michael, would it be appropriate to implement a potentially long running
    > operation that needs to be executed at LMS startup as an importer ?


    Yes, definitely.

    But why would you need to run this on every restart?

    > If so, how would the importer be triggered to run from pluginInit ?
    > Should one just initiate a new/changed files rescan in initPlugin or is
    > it a better way to do it ?


    Be nice and give LMS a few seconds to start up. Then run a rescan:

    Slim::Control::Request::executeRequest(undef, ['rescan']);

  5. #5
    Senior Member
    Join Date
    Mar 2011
    Posts
    177
    Quote Originally Posted by mherger View Post
    > Michael, would it be appropriate to implement a potentially long running
    > operation that needs to be executed at LMS startup as an importer ?


    Yes, definitely.

    But why would you need to run this on every restart?

    > If so, how would the importer be triggered to run from pluginInit ?
    > Should one just initiate a new/changed files rescan in initPlugin or is
    > it a better way to do it ?


    Be nice and give LMS a few seconds to start up. Then run a rescan:

    Slim::Control::Request::executeRequest(undef, ['rescan']);
    It's not only startup related, it is more the point that some of Erlands plugins are scanning stuff which brings LMS web server to an halt. Doesn't matter on startup or during runtime. Especially for people like me, with large libraries, this could take hours, so the idea is, to use a sort of scanning, which doesn't interfere with the LMS web server at all, or as Erland suggests, divide this into chunks.....

    What is your best design propose to get a scanning done, with 80.000 files and still be able to use LMS?

    For example:
    [21-02-25 01:45:44.0012] Plugins::TrackStat::Storage::refreshTracks (1371) Finished updating urls in statistic data based on musicbrainz ids, updated 108090 items : It took 12190.793819 seconds

    3 hours, 38 min :-)
    Last edited by mamema; 2021-02-26 at 02:20.

  6. #6
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,538

    Working on a plugin - Scanning question

    > What is your best design propose to get a scanning done, with 80.000
    > files and still be able to use LMS?


    I'd go the "run external scanner" route. It's much simpler than trying
    to do split up a large chunk of work in to smaller pieces, and get them
    done without blocking the single threaded server. And most systems
    nowadays have multiple cores. Let's take advantage of them!

    But first of all I'd try to understand what is taking so long, and why.
    Some understanding of the underlying SQL can help a lot. Are you
    repeatedly running the same queries? Could some results be cached? Is
    the performance CPU bound? Or disk IO?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •