How does new&changed scanner detect what files to scan?

[frank1969]

To see if there is (still) a bug I wanna know:
How does new&changed scanner detect what files to scan?

I changed 131 files - 2/3 have been recognized and changed, 1/3 didn't.
I guessed, it takes the "file change" date (from windows) - but this had been changed with all files...

Description:
I renamed all tracks with artist "Younotus" => "YouNotUs" (usinge MediaMonkey5 - all in the same way).

After a new&change scan
2/3 had "YouNotUs" as (track) artist
1/3 still had "Younotus" as (track) artist.

I then looked in the contributors.. and tracks tables => all files still inked to "Younotus" had no new "updated_time" so I guess, scanner didn't see them as "changed" and didn't handle them.
Checked manually the windows "file change time" - that had been updated and was the same as with the file, it "saw" and handled.

So I wonder, what is the criteria for scanner to detect a file in "new&changed"?


Btw:
ALL files that were linked to "Younotus" as ALBUM artist now had "Younotus" AND "YouNotUs" as ALBUM artist (sounds like a slight case sensivity bug again...?)

And btw2:
After scanning files 2 more times with "new&changed" it detected all files (with the 2nd run about 15 more, withe the 3rd run the remaining...)

[mherger]

> To see if there is (still) a bug I wanna know:
> How does new&changed scanner detect what files to scan?

Timestamp and file size.

> I guessed, it takes the "file change" date (from windows) - but this had
> been changed with all files...

Yes, that should be used (mtime equivalent on Windows).

> I then looked in the contributors.. and tracks tables => all files still
> inked to "Younotus" had no new "updated_time" so I guess, scanner didn't
> see them as "changed" and didn't handle them.

Did they not get recognized in the initial "looking for changed files"
already, or just not updated during the actuall update step?

> Checked manually the windows "file change time" - that had been updated
> and was the same as with the file, it "saw" and handled.

Are you saying that when you changed it for the second time, those files
were picked up?

As just posted in the other thread: please keep log and database file
around for analysis whenever you report such an issue. Without them it's
all guess work.

[frank1969]

>
Did they not get recognized in the initial "looking for changed files"
already, or just not updated during the actuall update step?

No they were recognized in the initial scan but they weren't handled/changed in the 1st new&changed) ( added_time was the same as updated_time though mtime had changed).
scanner.log said nothing about them (maybe I have to change my logging options from "standard" though?)

>
Are you saying that when you changed it for the second time, those files
were picked up?

Not exactly:
I did the 2nd n&c with UNCHANGED files - and scanner picked some of the files (changed before the 1st (!) n&c), some it missed again.
Then I "touched" some (! not all!) of those files (all in different directories, btw), some were still unchanged (compared to 1st run) - and now it found ALL...

Just doin' the same with HUGEL => Hugel and have a close look.

[mherger]

> Just doin' the same with HUGEL => Hugel and have a close look.

I'm sure if you zipped the resulting database file it would shrink to
"only" a few hundred MBs :-D

https://www.dropbox.com/request/T3RctyzGgNg0oFDubq6a

[frank1969]

This should have been the scanner.log from 2nd run:

[22-02-05 21:55:49.4085] main::main (206) Starting Logitech Media Server scanner (v8.3.0, 1644040271, Sat Feb 5 07:17:19 WEST 2022) perl 5.014001
[22-02-05 21:55:49.4986] Carp::Clan::__ANON__ (216) Warning: DBIx::Class::ResultSet::update_or_create(): Query returned more than one row. SQL that returns multiple rows is DEPRECATED for ->find and ->single at /<C:\PROGRA~2\SQUEEZ~1\server\scanner.exe>Slim/Schema.pm line 1905
[22-02-05 21:55:49.9735] Slim::Music::Import::runImporter (578) Starting Slim::Media::MediaFolderScan scan
[22-02-05 21:55:49.9738] Slim::Utils::Scanner::Local::rescan (179) Discovering audio files in M:
[22-02-05 22:06:37.8643] Slim::Utils::Scanner::Local::__ANON__ (191) Start processing found tracks
[22-02-05 22:06:37.8649] Slim::Utils::Scanner::Local::__ANON__ (199) Connect do DB
[22-02-05 22:06:37.8651] Slim::Utils::Scanner::Local::__ANON__ (202) Get latest ID
[22-02-05 22:06:37.8654] Slim::Utils::Scanner::Local::__ANON__ (224) Delete temporary table if exists
[22-02-05 22:06:37.8657] Slim::Utils::Scanner::Local::__ANON__ (227) Re-build temporary table
[22-02-05 22:06:45.0755] Slim::Utils::Scanner::Local::__ANON__ (276) Get deleted tracks count
[22-02-05 22:06:51.3208] Slim::Utils::Scanner::Local::__ANON__ (283) Get new tracks count
[22-02-05 22:06:51.3212] Slim::Utils::Scanner::Local::__ANON__ (288) Get changed tracks count
[22-02-05 22:06:56.4989] Slim::Utils::Scanner::Local::deleteTracks (364) Removing deleted audio files (29)
[22-02-05 22:07:22.1622] Slim::Utils::Scanner::Local::updateTracks (531) Rescanning changed audio files (70)
[22-02-05 22:07:31.6498] Slim::Utils::Scanner::Local::addTracks (447) Scanning new audio files (50)
[22-02-05 22:07:33.2814] Slim::Music::Import::endImporter (710) Completed Slim::Media::MediaFolderScan Scan in 703.308 seconds.
[22-02-05 22:07:33.2820] Slim::Music::Import::runImporter (578) Starting Slim::Music::PlaylistFolderScan scan
[22-02-05 22:07:33.2823] Slim::Utils::Scanner::Local::rescan (179) Discovering audio files in D:\__Playlists
[22-02-05 22:07:34.7944] Slim::Utils::Scanner::Local::__ANON__ (191) Start processing found tracks
[22-02-05 22:07:34.7947] Slim::Utils::Scanner::Local::__ANON__ (199) Connect do DB
[22-02-05 22:07:34.7949] Slim::Utils::Scanner::Local::__ANON__ (202) Get latest ID
[22-02-05 22:07:34.7952] Slim::Utils::Scanner::Local::__ANON__ (224) Delete temporary table if exists
[22-02-05 22:07:34.7955] Slim::Utils::Scanner::Local::__ANON__ (227) Re-build temporary table
[22-02-05 22:07:41.5580] Slim::Utils::Scanner::Local::__ANON__ (276) Get deleted tracks count
[22-02-05 22:07:47.7942] Slim::Utils::Scanner::Local::__ANON__ (283) Get new tracks count
[22-02-05 22:07:47.7946] Slim::Utils::Scanner::Local::__ANON__ (288) Get changed tracks count
[22-02-05 22:07:47.7983] Slim::Utils::Scanner::Local::deleteTracks (364) Removing deleted audio files (2)
[22-02-05 22:07:54.6661] Slim::Utils::Scanner::Local::updateTracks (531) Rescanning changed audio files (2)
[22-02-05 22:07:54.6695] Slim::Utils::Scanner::Local::changed (1105) Handling changed playlist file:///D:/__Playlists/0%20ESC%202022%20-%20Estland.m3u
[22-02-05 22:07:54.6765] Slim::Utils::Scanner::Local::new (916) Handling new playlist file:///D:/__Playlists/0%20ESC%202022%20-%20Estland.m3u
[22-02-05 22:07:54.7162] Slim::Utils::Scanner::Local::changed (1105) Handling changed playlist file:///D:/__Playlists/0%20ESC%202022%20-%20Norwegen.m3u
[22-02-05 22:07:54.7184] Slim::Utils::Scanner::Local::new (916) Handling new playlist file:///D:/__Playlists/0%20ESC%202022%20-%20Norwegen.m3u
[22-02-05 22:07:54.7416] Slim::Utils::Scanner::Local::addTracks (447) Scanning new audio files (1)
[22-02-05 22:07:54.7452] Slim::Utils::Scanner::Local::new (916) Handling new playlist file:///D:/__Playlists/ESC%202022%20-%20Estland%20-%202.%20Semifinale.m3u
[22-02-05 22:07:54.7766] Slim::Music::Import::endImporter (710) Completed Slim::Music::PlaylistFolderScan Scan in 21.495 seconds.
[22-02-05 22:07:54.7773] Slim::Music::Import::runImporter (578) Starting Slim::Plugin::FullTextSearch::Plugin scan
[22-02-05 22:07:54.7777] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (475) Starting fulltext index build
[22-02-05 22:07:54.7779] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (479) Initialize fulltext table
[22-02-05 22:07:55.6770] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (492) Create fulltext index for tracks
[22-02-05 22:12:51.5468] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (502) Create fulltext index for albums
[22-02-05 22:16:09.0784] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (511) Create fulltext index for contributors
[22-02-05 22:16:13.0939] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (521) Create fulltext index for playlists
[22-02-05 22:16:37.5409] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (536) Optimize fulltext index
[22-02-05 22:16:42.1325] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (551) Fulltext index build done!
[22-02-05 22:16:42.1327] Slim::Music::Import::endImporter (710) Completed Slim::Plugin::FullTextSearch::Plugin Scan in 527.355 seconds.
[22-02-05 22:16:42.1332] Slim::Music::Import::runImporter (578) Starting Slim::Plugin::ExtendedBrowseModes::Libraries scan
[22-02-05 22:16:42.1334] Slim::Music::Import::endImporter (710) Completed Slim::Plugin::ExtendedBrowseModes::Libraries Scan in 0.000 seconds.
[22-02-05 22:16:42.1339] Slim::Music::Import::runImporter (578) Starting Slim::Plugin::OnlineLibrary::Importer::VirtualLibr ariesCleanup scan
[22-02-05 22:16:42.1487] Slim::Music::Import::endImporter (710) Completed Slim::Plugin::OnlineLibrary::Importer::VirtualLibr ariesCleanup Scan in 0.015 seconds.
[22-02-05 22:16:43.8956] Slim::Music::Artwork::updateStandaloneArtwork (234) Starting updateStandaloneArtwork for 16120 albums
[22-02-05 22:17:36.6133] Slim::Music::Import::endImporter (710) Completed updateStandaloneArtwork Scan in 54.464 seconds.
[22-02-05 22:17:37.3392] Slim::Music::Artwork:recacheAllArtwork (657) Starting precacheArtwork for 36 albums
[22-02-05 22:17:39.6383] Image::Scale::new (44) Warning: buffer_get_ret: trying to get more bytes 2 than in buffer 0 at /<C:\PROGRA~2\SQUEEZ~1\server\scanner.exe>Image/Scale.pm line 44.
[22-02-05 22:17:39.7399] Slim::Music::Artwork::__ANON__ (808) precacheArtwork finished in 2.39795517921448
[22-02-05 22:17:39.7401] Slim::Music::Import::endImporter (710) Completed precacheArtwork Scan in 3.119 seconds.
[22-02-05 22:17:39.7429] Slim::Music::Import::runScanPostProcessing (480) Starting Database optimization.
[22-02-05 22:18:23.4822] Slim::Music::Import::endImporter (710) Completed dbOptimize Scan in 43.739 seconds.

[mherger]

> [22-02-05 22:06:56.4989] Slim::Utils::Scanner::Local::deleteTracks (364)
> Removing deleted audio files (29)
> [22-02-05 22:07:22.1622] Slim::Utils::Scanner::Local::updateTracks (531)
> Rescanning changed audio files (70)
> [22-02-05 22:07:31.6498] Slim::Utils::Scanner::Local::addTracks (447)
> Scanning new audio files (50)

Has renaming (folder or file) been part of this update? 29 items
previously scanned seem to be missing, 50 are new (not scanned before).
The 70 updates might +/- be the 2/3 recognized?

The "scanned_files" table would have the list of all files found. The
scan then would compare this with what there is in the library. The
corresponding queries can be found in
https://github.com/Logitech/slimserv.../Local.pm#L212
downwards.

I'm wondering whether this is a newly introduced issue after I changed
the order in which tracks are being deleted/updated/added...

[slartibartfast]

In my experience tagging changes aren't always picked up properly in a new and changed scan anyway. I always remove and replace the affected albums doing a new and changed scan twice.

Sent from my Pixel 3a using Tapatalk

[mherger]

> In my experience tagging changes aren't always picked up properly in a
> new and changed scan anyway. I always remove and replace the affected
> albums doing a new and changed scan twice.

I've fixed a few of those issues in the past week. Please give it
another try.

[frank1969]

> [22-02-05 22:06:56.4989]

Has renaming (folder or file) been part of this update? 29 items
previously scanned seem to be missing, 50 are new (not scanned before).
The 70 updates might +/- be the 2/3 recognized?

The "scanned_files" table would have the list of all files found. The
scan then would compare this with what there is in the library. The
corresponding queries can be found in
https://github.com/Logitech/slimserv.../Local.pm#L212
downwards.

I'm wondering whether this is a newly introduced issue after I changed
the order in which tracks are being deleted/updated/added...

Yes, there were a few files changed besides the Younotus => YouNotUs thing, so it is not an 1:1 scan - but I only analyzed those "YouNotUs" files.

Meanwhile I did 2 new update scans:
a) HUGEL => Hugel
b) Atb => ATB with fixing a few things like "ATB feat. xxx" or "ATB & xxx" => "ATB; xxx"

a) was fine (about 30 tracks)
b) was 63 tracks
Most thing worked finde, too (honest 60 of 63, what is about 90 %, which is great )
... but 3 ones failed and I can't think why:
- 2 have not been renamed - they must have been seen by scanner (I meanwhile checked the scanner log, saying
[22-02-06 17:49:52.2772] Slim::Utils::Scanner::Local::updateTracks (531) Rescanning changed audio files (63)"

) but they were not changed and got no "today's timestamp" - I can't see any modifications but I'll compare with the old version of library.db, as You suggested
- 1 has been scanned and got a new timestamp, but not changed - there I modified the artist "ATB: Topic" to "ATB; Topic"

Please understand me right:
I don't wanna have You to do "guessing work" for my collection - I try myself, too, but it's only if You have an idea, what could cause this, for I know my collection, but You know the changed to scanner .

Nevertheless, it is 90 % better than 2 weeks ago :-)))

.. and next thing is to do a "second run" for the "ATB"-thing for I guess too, it has something to do with scanning order?!

[frank1969]

So, 2 more scans - the last one with r1644170574 - it looks that the "You & Me" => "You&me" issue is back again (the "you & me" artist stays in db).
Could this be?!

Nevertheless, I do another scan with r1644170574 with unchanged files to see what happens...