Home of the Squeezebox™ & Transporter® network music players.
Page 1 of 2 12 LastLast
Results 1 to 10 of 13
  1. #1
    Senior Member
    Join Date
    Jun 2009
    Posts
    402

    Scanner MUCH quicker in r1529096694

    Did You change anything with the scanner / crawler.

    I recognize the "New & changed"-scanning being MUCH quicker in this release - maybe twice to three times quicker than in the previous versions. It scans my 600k database now in about 6 minutes (before it took about 15 minutes).

    Can You confirm this - would the "clean" scan be that much quicker too (in my db it usually took 6 hours,,,)

  2. #2
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,261

    Scanner MUCH quicker in r1529096694

    > Did You change anything with the scanner / crawler.

    Since which build?

    There was a change specific to scanning very large connections about two
    weeks ago:

    https://github.com/Logitech/slimserv...c34f75550b76a1

    BUT I would have expected this change to actually slow down the process,
    rather than speed it up...

    Can you tell me about your server system (OS, memory, disk system where
    cache lives)?

    --

    Michael

  3. #3
    Senior Member
    Join Date
    Jun 2009
    Posts
    402
    Quote Originally Posted by mherger View Post
    > Since which build?

    Just since THIS sepcific build from last friday r1529096694.
    I try every nightly and do a "new & changed"-scan every day (sometimes twice a day), so I would have recognized if this had happened earlier

    Quote Originally Posted by mherger View Post
    > There was a change specific to scanning very large connections about two
    weeks ago:
    BUT I would have expected this change to actually slow down the process,
    rather than speed it up...
    Yes, I've recognized this, too, some one or two builds ago - it slowed down the server VERY much (a new&scan lasted 2-3 hours instead of 15-20 minutes, but it happened only a few times, normally the scan was in the "15-20 minutes window")



    Quote Originally Posted by mherger View Post
    >Can you tell me about your server system (OS, memory, disk system where
    cache lives)?
    Windows 10 Pro
    intelCore i7-2600, 3,4 Hz
    8 GB RAM
    Cache on local HDD (2TB, NTFS)

  4. #4
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,261

    Scanner MUCH quicker in r1529096694

    >>> Since which build?
    >
    > Just since THIS sepcific build from last friday r1529096694.
    > I try every nightly and do a "new & changed"-scan every day (sometimes
    > twice a day), so I would have recognized if this had happened earlier


    Ok, then that's interesting: the last change I did was specific to
    Windows and how it tries to handle long filenames. Can you do a BMF and
    see whether file names are still correct?

    Would you happen to have files with non-latin characters (Chinese,
    Cyrillic, Hebrew etc.)?

    See https://github.com/Logitech/slimserver/pull/206

    > Yes, I've recognized this, too, some one or two builds ago - it slowed
    > down the server VERY much (a new&scan lasted 2-3 hours instead of 15-20
    > minutes, but it happened only a few times, normally the scan was in the
    > "15-20 minutes window")


    Oh, that's rough... I don't know whether you had seen the discussion
    which lead to this change: there was a user reporting crashes with his
    collection even larger than yours. And it was crashing running out of
    memory during the vacuum stage of the scan. That's where the whole
    database would be re-written to optimize disk space allocation.
    Previously this would have been run in memory, and only be written to
    disk when done.

    How large is your library.db?

    --

    Michael

  5. #5
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,261

    Scanner MUCH quicker in r1529096694

    Could you please update to the latest again? That previous change
    introduced a problem with the filename expansion in case of Windows
    short filenames (abcdef~1.mp3).

    Would you have many files without metadata? I tried to figure out where
    that routine was even used during a scan. And FWICT this should only be
    the case when there was no title available for a file. In that case LMS
    will try to guess it from the filename.

    --

    Michael

  6. #6
    Senior Member
    Join Date
    Jun 2009
    Posts
    402
    Quote Originally Posted by mherger View Post
    Can you do a BMF and
    see whether file names are still correct?
    I browsed for a while but could see now different behaviour from before

    Quote Originally Posted by mherger View Post
    Would you happen to have files with non-latin characters (Chinese,
    Cyrillic, Hebrew etc.)?
    I hope: NONE, so I'm quite sure 99,9% None. I convert the filenames into latin charachters (even converting the german "Umlaute" ń, ÷, Ř, ▀)

    Quote Originally Posted by mherger View Post
    That's where the whole
    database would be re-written to optimize disk space allocation.
    Previously this would have been run in memory, and only be written to
    disk when done.

    How large is your library.db?
    ok, I never saw this problem. My library.db is about 984 MB (the artwork.db btw about 1,5 GB)

    Quote Originally Posted by mherger View Post
    Could you please update to the latest again? That previous change
    introduced a problem with the filename expansion in case of Windows
    short filenames (abcdef~1.mp3). Would you have many files without metadata?
    I will update just in a few seconds. With metadata it's the same as with non-latin characters, for I try to re-tag every file there should be meta-data and 'correct' filenames to 99,9% of my DB,

    btw.
    Many thanks again for keeping up the great work!

  7. #7
    Senior Member
    Join Date
    Jun 2009
    Posts
    402
    Quote Originally Posted by mherger View Post
    Could you please update to the latest again?
    Now it's slower than a "clean scan": in 22 minutes (!) "new&changed" scanned just 866 tracks. So I could assume it will take .... about 240 hours or 10 days to do the "new & changed" - I will interrupt this... ;-)

  8. #8
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,261

    Scanner MUCH quicker in r1529096694

    > mherger wrote:
    >> Could you please update to the latest again?

    >
    > Now it's slower than a "clean scan": in 22 minutes (!) "new&changed"
    > scanned just 866 tracks. So I could assume it will take .... about 240
    > hours or 10 days to do the "new & changed" - I will interrupt this...


    That doesn't make sense... because the changed code should only ever be
    used if a file didn't have any title metadata. And as you said that
    should not be the case.

    Could you please re-run a scan with database.info set to INFO?

    --

    Michael

  9. #9
    Senior Member
    Join Date
    Jan 2009
    Posts
    131
    Same here. New & changed takes now 1h10m. Before it was 10m. Setup: max2play, files on NAS.

  10. #10
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,261

    Scanner MUCH quicker in r1529096694

    > Same here. New & changed takes now 1h10m. Before it was 10m. Setup:
    > max2play, files on NAS.


    That change was Windows only
    (https://github.com/Logitech/slimserv...0c60407a4cfb2a).
    There must be something else.

    How large is your collection?

    --

    Michael

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •