How does new&changed scanner detect what files to scan?

[frank1969]

> It's 7.317 (so for the first 4 seem to be headlines (see below), it
> makes sense):

Not surprisingly that's a lot. Does the fulltext search still provide
some value to you? All those 7k popular terms would only be searched for
in titles, but eg. not in comments.

I'm not saying "turn off FTS" to work around that crash. But I believe
some of the FTS plugin implementation doesn't scale well with larger
libraries.

[frank1969]

How does new&changed scanner detect what files to scan?

To see if there is (still) a bug I wanna know:
How does new&changed scanner detect what files to scan?

I changed 131 files - 2/3 have been recognized and changed, 1/3 didn't.
I guessed, it takes the "file change" date (from windows) - but this had been changed with all files...

Description:
I renamed all tracks with artist "Younotus" => "YouNotUs" (usinge MediaMonkey5 - all in the same way).

After a new&change scan
2/3 had "YouNotUs" as (track) artist
1/3 still had "Younotus" as (track) artist.

I then looked in the contributors.. and tracks tables => all files still inked to "Younotus" had no new "updated_time" so I guess, scanner didn't see them as "changed" and didn't handle them.
Checked manually the windows "file change time" - that had been updated and was the same as with the file, it "saw" and handled.

So I wonder, what is the criteria for scanner to detect a file in "new&changed"?


Btw:
ALL files that were linked to "Younotus" as ALBUM artist now had "Younotus" AND "YouNotUs" as ALBUM artist (sounds like a slight case sensivity bug again...?)

And btw2:
After scanning files 2 more times with "new&changed" it detected all files (with the 2nd run about 15 more, withe the 3rd run the remaining...)

[mherger]

How does new&changed scanner detect whatfiles to scan?

> To see if there is (still) a bug I wanna know:
> How does new&changed scanner detect what files to scan?

Timestamp and file size.

> I guessed, it takes the "file change" date (from windows) - but this had
> been changed with all files...

Yes, that should be used (mtime equivalent on Windows).

> I then looked in the contributors.. and tracks tables => all files still
> inked to "Younotus" had no new "updated_time" so I guess, scanner didn't
> see them as "changed" and didn't handle them.

Did they not get recognized in the initial "looking for changed files"
already, or just not updated during the actuall update step?

> Checked manually the windows "file change time" - that had been updated
> and was the same as with the file, it "saw" and handled.

Are you saying that when you changed it for the second time, those files
were picked up?

As just posted in the other thread: please keep log and database file
around for analysis whenever you report such an issue. Without them it's
all guess work.

[frank1969]

>
Did they not get recognized in the initial "looking for changed files"
already, or just not updated during the actuall update step?

No they were recognized in the initial scan but they weren't handled/changed in the 1st new&changed) ( added_time was the same as updated_time though mtime had changed).
scanner.log said nothing about them (maybe I have to change my logging options from "standard" though?)

>
Are you saying that when you changed it for the second time, those files
were picked up?

Not exactly:
I did the 2nd n&c with UNCHANGED files - and scanner picked some of the files (changed before the 1st (!) n&c), some it missed again.
Then I "touched" some (! not all!) of those files (all in different directories, btw), some were still unchanged (compared to 1st run) - and now it found ALL...

Just doin' the same with HUGEL => Hugel and have a close look.

[mherger]

How does new&changed scanner detect whatfiles to scan?

> Just doin' the same with HUGEL => Hugel and have a close look.

I'm sure if you zipped the resulting database file it would shrink to
"only" a few hundred MBs :-D

https://www.dropbox.com/request/T3RctyzGgNg0oFDubq6a

[frank1969]

This should have been the scanner.log from 2nd run:

[22-02-05 21:55:49.4085] main::main (206) Starting Logitech Media Server scanner (v8.3.0, 1644040271, Sat Feb 5 07:17:19 WEST 2022) perl 5.014001
[22-02-05 21:55:49.4986] Carp::Clan::__ANON__ (216) Warning: DBIx::Class::ResultSet::update_or_create(): Query returned more than one row. SQL that returns multiple rows is DEPRECATED for ->find and ->single at /<C:\PROGRA~2\SQUEEZ~1\server\scanner.exe>Slim/Schema.pm line 1905
[22-02-05 21:55:49.9735] Slim::Music::Import::runImporter (578) Starting Slim::Media::MediaFolderScan scan
[22-02-05 21:55:49.9738] Slim::Utils::Scanner::Local::rescan (179) Discovering audio files in M:
[22-02-05 22:06:37.8643] Slim::Utils::Scanner::Local::__ANON__ (191) Start processing found tracks
[22-02-05 22:06:37.8649] Slim::Utils::Scanner::Local::__ANON__ (199) Connect do DB
[22-02-05 22:06:37.8651] Slim::Utils::Scanner::Local::__ANON__ (202) Get latest ID
[22-02-05 22:06:37.8654] Slim::Utils::Scanner::Local::__ANON__ (224) Delete temporary table if exists
[22-02-05 22:06:37.8657] Slim::Utils::Scanner::Local::__ANON__ (227) Re-build temporary table
[22-02-05 22:06:45.0755] Slim::Utils::Scanner::Local::__ANON__ (276) Get deleted tracks count
[22-02-05 22:06:51.3208] Slim::Utils::Scanner::Local::__ANON__ (283) Get new tracks count
[22-02-05 22:06:51.3212] Slim::Utils::Scanner::Local::__ANON__ (288) Get changed tracks count
[22-02-05 22:06:56.4989] Slim::Utils::Scanner::Local::deleteTracks (364) Removing deleted audio files (29)
[22-02-05 22:07:22.1622] Slim::Utils::Scanner::Local::updateTracks (531) Rescanning changed audio files (70)
[22-02-05 22:07:31.6498] Slim::Utils::Scanner::Local::addTracks (447) Scanning new audio files (50)
[22-02-05 22:07:33.2814] Slim::Music::Import::endImporter (710) Completed Slim::Media::MediaFolderScan Scan in 703.308 seconds.
[22-02-05 22:07:33.2820] Slim::Music::Import::runImporter (578) Starting Slim::Music::PlaylistFolderScan scan
[22-02-05 22:07:33.2823] Slim::Utils::Scanner::Local::rescan (179) Discovering audio files in D:\__Playlists
[22-02-05 22:07:34.7944] Slim::Utils::Scanner::Local::__ANON__ (191) Start processing found tracks
[22-02-05 22:07:34.7947] Slim::Utils::Scanner::Local::__ANON__ (199) Connect do DB
[22-02-05 22:07:34.7949] Slim::Utils::Scanner::Local::__ANON__ (202) Get latest ID
[22-02-05 22:07:34.7952] Slim::Utils::Scanner::Local::__ANON__ (224) Delete temporary table if exists
[22-02-05 22:07:34.7955] Slim::Utils::Scanner::Local::__ANON__ (227) Re-build temporary table
[22-02-05 22:07:41.5580] Slim::Utils::Scanner::Local::__ANON__ (276) Get deleted tracks count
[22-02-05 22:07:47.7942] Slim::Utils::Scanner::Local::__ANON__ (283) Get new tracks count
[22-02-05 22:07:47.7946] Slim::Utils::Scanner::Local::__ANON__ (288) Get changed tracks count
[22-02-05 22:07:47.7983] Slim::Utils::Scanner::Local::deleteTracks (364) Removing deleted audio files (2)
[22-02-05 22:07:54.6661] Slim::Utils::Scanner::Local::updateTracks (531) Rescanning changed audio files (2)
[22-02-05 22:07:54.6695] Slim::Utils::Scanner::Local::changed (1105) Handling changed playlist file:///D:/__Playlists/0%20ESC%202022%20-%20Estland.m3u
[22-02-05 22:07:54.6765] Slim::Utils::Scanner::Local::new (916) Handling new playlist file:///D:/__Playlists/0%20ESC%202022%20-%20Estland.m3u
[22-02-05 22:07:54.7162] Slim::Utils::Scanner::Local::changed (1105) Handling changed playlist file:///D:/__Playlists/0%20ESC%202022%20-%20Norwegen.m3u
[22-02-05 22:07:54.7184] Slim::Utils::Scanner::Local::new (916) Handling new playlist file:///D:/__Playlists/0%20ESC%202022%20-%20Norwegen.m3u
[22-02-05 22:07:54.7416] Slim::Utils::Scanner::Local::addTracks (447) Scanning new audio files (1)
[22-02-05 22:07:54.7452] Slim::Utils::Scanner::Local::new (916) Handling new playlist file:///D:/__Playlists/ESC%202022%20-%20Estland%20-%202.%20Semifinale.m3u
[22-02-05 22:07:54.7766] Slim::Music::Import::endImporter (710) Completed Slim::Music::PlaylistFolderScan Scan in 21.495 seconds.
[22-02-05 22:07:54.7773] Slim::Music::Import::runImporter (578) Starting Slim::Plugin::FullTextSearch::Plugin scan
[22-02-05 22:07:54.7777] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (475) Starting fulltext index build
[22-02-05 22:07:54.7779] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (479) Initialize fulltext table
[22-02-05 22:07:55.6770] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (492) Create fulltext index for tracks
[22-02-05 22:12:51.5468] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (502) Create fulltext index for albums
[22-02-05 22:16:09.0784] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (511) Create fulltext index for contributors
[22-02-05 22:16:13.0939] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (521) Create fulltext index for playlists
[22-02-05 22:16:37.5409] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (536) Optimize fulltext index
[22-02-05 22:16:42.1325] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (551) Fulltext index build done!
[22-02-05 22:16:42.1327] Slim::Music::Import::endImporter (710) Completed Slim::Plugin::FullTextSearch::Plugin Scan in 527.355 seconds.
[22-02-05 22:16:42.1332] Slim::Music::Import::runImporter (578) Starting Slim::Plugin::ExtendedBrowseModes::Libraries scan
[22-02-05 22:16:42.1334] Slim::Music::Import::endImporter (710) Completed Slim::Plugin::ExtendedBrowseModes::Libraries Scan in 0.000 seconds.
[22-02-05 22:16:42.1339] Slim::Music::Import::runImporter (578) Starting Slim::Plugin::OnlineLibrary::Importer::VirtualLibr ariesCleanup scan
[22-02-05 22:16:42.1487] Slim::Music::Import::endImporter (710) Completed Slim::Plugin::OnlineLibrary::Importer::VirtualLibr ariesCleanup Scan in 0.015 seconds.
[22-02-05 22:16:43.8956] Slim::Music::Artwork::updateStandaloneArtwork (234) Starting updateStandaloneArtwork for 16120 albums
[22-02-05 22:17:36.6133] Slim::Music::Import::endImporter (710) Completed updateStandaloneArtwork Scan in 54.464 seconds.
[22-02-05 22:17:37.3392] Slim::Music::Artwork:recacheAllArtwork (657) Starting precacheArtwork for 36 albums
[22-02-05 22:17:39.6383] Image::Scale::new (44) Warning: buffer_get_ret: trying to get more bytes 2 than in buffer 0 at /<C:\PROGRA~2\SQUEEZ~1\server\scanner.exe>Image/Scale.pm line 44.
[22-02-05 22:17:39.7399] Slim::Music::Artwork::__ANON__ (808) precacheArtwork finished in 2.39795517921448
[22-02-05 22:17:39.7401] Slim::Music::Import::endImporter (710) Completed precacheArtwork Scan in 3.119 seconds.
[22-02-05 22:17:39.7429] Slim::Music::Import::runScanPostProcessing (480) Starting Database optimization.
[22-02-05 22:18:23.4822] Slim::Music::Import::endImporter (710) Completed dbOptimize Scan in 43.739 seconds.

[mherger]

How does new&amp;changed scanner detect whatfiles to scan?

> [22-02-05 22:06:56.4989] Slim::Utils::Scanner::Local::deleteTracks (364)
> Removing deleted audio files (29)
> [22-02-05 22:07:22.1622] Slim::Utils::Scanner::Local::updateTracks (531)
> Rescanning changed audio files (70)
> [22-02-05 22:07:31.6498] Slim::Utils::Scanner::Local::addTracks (447)
> Scanning new audio files (50)

Has renaming (folder or file) been part of this update? 29 items
previously scanned seem to be missing, 50 are new (not scanned before).
The 70 updates might +/- be the 2/3 recognized?

The "scanned_files" table would have the list of all files found. The
scan then would compare this with what there is in the library. The
corresponding queries can be found in

https://github.com/Logitech/slimserver/blob/5b6438c2f28dfce5c703e9452eb2c7f3ac0f8b43/Slim/Utils/Scanner/Local.pm#L212

downwards.

I'm wondering whether this is a newly introduced issue after I changed
the order in which tracks are being deleted/updated/added...

[slartibartfast]

In my experience tagging changes aren't always picked up properly in a new and changed scan anyway. I always remove and replace the affected albums doing a new and changed scan twice.

Sent from my Pixel 3a using Tapatalk

[mherger]

How does new&amp;changed scanner detect whatfiles to scan?

> In my experience tagging changes aren't always picked up properly in a
> new and changed scan anyway. I always remove and replace the affected
> albums doing a new and changed scan twice.

I've fixed a few of those issues in the past week. Please give it
another try.

[frank1969]

> [22-02-05 22:06:56.4989]

Has renaming (folder or file) been part of this update? 29 items
previously scanned seem to be missing, 50 are new (not scanned before).
The 70 updates might +/- be the 2/3 recognized?

The "scanned_files" table would have the list of all files found. The
scan then would compare this with what there is in the library. The
corresponding queries can be found in

https://github.com/Logitech/slimserver/blob/5b6438c2f28dfce5c703e9452eb2c7f3ac0f8b43/Slim/Utils/Scanner/Local.pm#L212

downwards.

I'm wondering whether this is a newly introduced issue after I changed
the order in which tracks are being deleted/updated/added...

Yes, there were a few files changed besides the Younotus => YouNotUs thing, so it is not an 1:1 scan - but I only analyzed those "YouNotUs" files.

Meanwhile I did 2 new update scans:
a) HUGEL => Hugel
b) Atb => ATB with fixing a few things like "ATB feat. xxx" or "ATB & xxx" => "ATB; xxx"

a) was fine (about 30 tracks)
b) was 63 tracks
Most thing worked finde, too (honest 60 of 63, what is about 90 %, which is great )
... but 3 ones failed and I can't think why:
- 2 have not been renamed - they must have been seen by scanner (I meanwhile checked the scanner log, saying
[22-02-06 17:49:52.2772] Slim::Utils::Scanner::Local::updateTracks (531) Rescanning changed audio files (63)"

) but they were not changed and got no "today's timestamp" - I can't see any modifications but I'll compare with the old version of library.db, as You suggested
- 1 has been scanned and got a new timestamp, but not changed - there I modified the artist "ATB: Topic" to "ATB; Topic"

Please understand me right:
I don't wanna have You to do "guessing work" for my collection - I try myself, too, but it's only if You have an idea, what could cause this, for I know my collection, but You know the changed to scanner .

Nevertheless, it is 90 % better than 2 weeks ago :-)))

.. and next thing is to do a "second run" for the "ATB"-thing for I guess too, it has something to do with scanning order?!

[frank1969]

So, 2 more scans - the last one with r1644170574 - it looks that the "You & Me" => "You&me" issue is back again (the "you & me" artist stays in db).
Could this be?!

Nevertheless, I do another scan with r1644170574 with unchanged files to see what happens...

[frank1969]

ok, 4 scans further with different sets: The "you&me" issue seems to be back - and there seems to be another issue with casesensivity (Atb => ATB, just tried A-ha => a-ha and R3HAB => R3hab)

Next test will be tommorow set I set back my library.db to an old backup (I've one from 02.02., that should be just after my initial scan) and make a n&c scan with a nightly from 04.02. and/or 05.02. - I will report...

[mherger]

How does new&amp;changed scanner detect whatfiles to scan?

> ok, 4 scans further with different sets: The "you&me" issue seems to be
> back - and there seems to be another issue with casesensivity (Atb =>
> ATB, just tried A-ha => a-ha and R3HAB => R3hab)

Can't reproduce the case sensitive case: test with and without file name
change. In both cases the change would be picked up as expected.

Are you sure you're running the latest buid? And it would still be great
to get access to the log and database files.

> Next test will be tommorow set I set back my library.db to an old backup

Keep in mind that one of the changes around case senstitivity require
the file the be created anew. Restoring an older file will indeed
re-introduce the problem.

> (I've one from 02.02., that should be just after my initial scan) and
> make a n&c scan with a nightly from 04.02. and/or 05.02. - I will
> report...

Please double check scanned_files.url is case insensitive ("collate
nocase").

[frank1969]

Are you sure you're running the latest buid?

Yes, I'm always running the last nightly (check it before every scan ).
I just today did a regression to 02.02. nightly to see if it shows anoter behaviour - it didn't.

Keep in mind that one of the changes around case senstitivity require
the file the be created anew. Restoring an older file will indeed
re-introduce the problem.

Yapp, I'm aware, I tested with a backup made after the initial scan (with Your database changes)

Please double check scanned_files.url is case insensitive ("collate
nocase").

And once again Yes, I checked it, it is "text" "NOT NULL COLLATE NOCASE"

And Yes, I see, if You can't reproduce I have to do some more tests maybe with a special test set (usually I do several changes everyday in my huge db , but I only track those "name changes".
I will take the time to do nothing than those "name changes" and watch the logfiles. Now changed all 3 scanner logging-options to "debug"...

[frank1969]

@mherger:

So, here are my results and logs and "what obviously could have happened".

In both cases it were misspelled artsist.
Both were the only entries from those artists a) "ATB: Topic" b) "Deejay Ötzi"

a) with renaming: changed "ATB: Topic" => "ATB; Topic" (as album artist and track artist) AND renamed the file
b) without renaming: I changed "Deejay Ötzi => DJ Ötzi" (it's an album by "Various Artists" with comp=1 ) and DIDN't rename the file

Important:
I did it this weekend AFTER my initial re-scan using the (at that point) last nightly (should have been r1643836091 or 1644040271). From this "1st run" I don't have the logs (resp. only the "short log" with no details).

Result:
a) Track was in db with artists "ATB" and "Topic" - but the "ATB: Topic" artist stayed in with an "empty" entry,
b) Track was in db with "Deejay Ötzi" AND "DJ Ötzi"

After changing my logging options, I "touched" them (so that scanner sees them as new) and scanned again (with r1644170574.exe)
(the first track results from another thing I did, changing the artist "Voxxclub" => "voxxclub" - THIS one worked and it sais "Removing unused contributor".

My examples a) and b) he also found as "changed" - but didn't obviously do anything with them.

(...)
[22-02-07 17:17:44.8926] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/V/voXXclub/Donnawedda Volksmusik/18 - voxxclub - Haeddi dadi wari.mp3
[22-02-07 17:17:45.1879] Slim::Schema::Contributor::rescan (244) Removing unused contributor: 18303
[22-02-07 17:17:45.9323] Slim::Utils::SQLiteHelper::updateProgress (442) Notify to server: [
"progress:1644250658.81424||importer||M:|directory _changed||23||24||",
]
[22-02-07 17:17:45.9353] Slim::Utils::SQLiteHelper::updateProgress (466) Notify to server OK
[22-02-07 17:17:45.9357] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/_Various/B/Bravo%20Hits%20030%20-%20CD%202/18%20-%20Deejay%20Oetzi%20-%20Hey%20Baby%20-%20Deejay%20Oetzi%20-%20Hey%20Baby.mp3
[22-02-07 17:17:48.4268] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/A/ATB/2021%20-%20Your%20Love%20(Single)/01%20-%20ATB%3B%20Topic%3B%20A7S%20-%20Your%20Love%20(9PM).mp3
[22-02-07 17:17:54.2329] Slim::Utils::SQLiteHelper::updateProgress (442) Notify to server: [
"progress:1644250658||importer||M:|directory_chang ed||24||24||1644250674.23192",
]

So I looked in my library.db:

in "contributors":
both misspelled artists are still "in"
a) ATB: Topic = 101413
b) Deejay Ötzi = 68185

in "contributor_album":
a) 10143 is linked with "role 5" and "role 6" (NOT role 1) to one album (41226)
b) 68185 is linked with "role 1" to one album (30954)

in "contributor_track":
a + b) both contributors DOESN'T exist.

in "albums":
a) album 41226 shows as contributor 10143
b) album 30954 shows as contributor 3 ("Various Artists")

in "tracks":
a) album 41226 shows no result
b) album 30954 shows track 439161 with primary_artist 5883 (which IS "DJ Ötzi", the correct contributor!)

once again looking in "contributor_track"
a) - (for there is no track)
b) track 43961 has contributor 5883 (the correct "DJ Ötzi") with role 1 and contributor 3 ("Various Artists") with role 5

So IMHO I guess (!) this is what happens:
a) scanner recognized the RENAMED file,
- added the new file to teh db "tracks" (which is there, track #1160846 with correct album and correct artists)
- deleted the old file from "tracks" (I also CAN'T find it searching for the old URL),
but did - not see, it was the only track from the album and did not delete the "unused album".
- not see, the contributor has no more entries than this "unused album" and so did not delete the "unused contributor"

b) scanner recognized the CHANGED file,
- added the new (corrected) contributor to "contributor_track" and deleted the "wrong" contributor from "contributor_track"
- added the new (corrected) contributor to "tracks" as "primary_artist"
- did not add the new (corrected) contribuotr to "contributor_album" and did not remove the "old" contributor from "contributor_album"
- did not see that the "old" contributor is "unused" (which scanner couldn't see, for it is still in use in "contributor_album")

Would any more information be helpful I could look up for You?

Maybe this gives You a hint - otherwise I will start another test with a similar case - and this time log "everything" again...