How does new&changed scanner detect what files to scan?

[mherger]

How does new&changed scanner detect whatfiles to scan?

> In both cases it were misspelled artsist.
> Both were the only entries from those artists a) "ATB: Topic" b) "Deejay
> Ötzi"
>
> a) with renaming: changed "ATB: Topic" => "ATB; Topic" (as album artist
> and track artist) AND renamed the file
> b) without renaming: I changed "Deejay Ötzi => DJ Ötzi" (it's an album
> by "Various Artists" with comp=1 ) and DIDN't rename the file

With the information I have I can't reproduce the issue. I tried to
follow both. But I'm missing pieces:

- what't the compilation's album artist (literally)?
- what are your settings in Settings/My Music?

I think it has something to do with the artist roles. Therefore the last
point could be crucial, as I don't include composers, conductors etc. in
my collection.

> So IMHO I guess (!) this is what happens:
> a) scanner recognized the RENAMED file,
> - added the new file to teh db "tracks" (which is there, track #1160846
> with correct album and correct artists)
> - deleted the old file from "tracks" (I also CAN'T find it searching for
> the old URL),

Deletion happens before the addition.

[frank1969]

>- what't the compilation's album artist (literally)?

It's named "Various Artists" (see below)

- what are your settings in Settings/My Music?

- Use singe, configurable list of artists
- Composer, band... NOTHING is checked
- List compilation albums under each artist
- Treat TPE2 MP3 tag as Album Artist
- When compilation albums are grouped together, they appear under "Various Artists" by default. You can change that name below. (I typed Various Artists again, though I know it's repeated unnecessarily)
- Show all albums and tracks for an artist
- Only show tracks or albums matching the selected role for an artist
- Search withing words
- The El La Los Las Le Les Der Die Das
- ;
- 200
- Treat multiple disc-sets as multiple albums
- Remember playlist
- Leave in same order
- Save playlists in order songs were added

I think it has something to do with the artist roles. Therefore the last
point could be crucial, as I don't include composers, conductors etc. in
my collection.

This is what I do - I guess... (see above)

Deletion happens before the addition.

OK, mentioned it in random order for I didn't know...

[mherger]

How does new&changed scanner detect whatfiles to scan?

> a) with renaming: changed "ATB: Topic" => "ATB; Topic" (as album artist
> and track artist) AND renamed the file
> b) without renaming: I changed "Deejay Ötzi => DJ Ötzi" (it's an album
> by "Various Artists" with comp=1 ) and DIDN't rename the file

Thanks for the additional information! I believe I now got the "DJ Ötzi"
case covered, too. Hopefully it doesn't break too many things... I'm
still struggling to reproduce the case which renames files/folders. This
could be a Windows problem. I'll have to see whether I can fire up some
VM to reproduce the issue.

New build is on its way.

[frank1969]

Thanks for the build.

Just a note:
I tested by "touching" the files again (changing mtime) so they were discovered by scanner.
It did but it did no changes to the db.
Am I right that this is "expected behaviour" for scanner didn't see that the old "Deejay Ötzi" artist is as orphaned entry still connected to the album in contributor_album?

If then I would try to create a new "Deejay Ötzi => DJ Ötzi"

[mherger]

How does new&changed scanner detect whatfiles to scan?

> I tested by "touching" the files again (changing mtime) so they were
> discovered by scanner.
> It did but it did no changes to the db.
> Am I right that this is "expected behaviour" for scanner didn't see that
> the old "Deejay Ötzi" artist is as orphaned entry still connected to the
> album in contributor_album?

Yes, I believe this could be true: the cleanup happens when the content
changes. But despite the updated timestamp, the scanner would not have
seen an updated artist.

[frank1969]

>Yes, I believe this could be true: the cleanup happens when the content
changes. But despite the updated timestamp, the scanner would not have
seen an updated artist.

OK, thanks - then I will now turn the files back in "broken" status => scan => correct the tags => scan ...

[mherger]

How does new&changed scanner detect whatfiles to scan?

> OK, thanks - then I will now turn the files back in "broken" status =>
> scan => correct the tags => scan ...

You don't have new files to clean up today?... :-D

[frank1969]

>You don't have new files to clean up today?... :-D

I'll propably will
But before I try to do a scan only with those 2 or 3 files changed to see, what logfiles and db tell...

[frank1969]

So a few scans later: GOOD NEWS!

I once again "broke" the tags, scanned, fixed them and scanned again with r1644324995.

4 constellations - in every constellation the misspelled artist was with only one track in db

a)
Changing "ATB: Topic" => "ATB; Topic" (the old "you&me" problem WITH renaming the file)

- scanner "detected" the old/now-missing file and deleted it from db - it also deleted the orphaned "ATB: Topic" artist
[22-02-08 15:31:38.7309] Slim::Utils::Scanner::Local::deleteTracks (364) Removing deleted audio files (1)
[22-02-08 15:31:45.6420] Slim::Utils::SQLiteHelper::updateProgress (442) Notify to server: [
"progress:1644330698.73126||importer||M:|directory _deleted||1||1||",
]
[22-02-08 15:31:45.6449] Slim::Utils::SQLiteHelper::updateProgress (466) Notify to server OK
[22-02-08 15:31:45.6452] Slim::Utils::Scanner::Local::deleted (638) Handling deleted audio file file:///M:/A/ATB/2021 - Your Love (Single)/01 - ATB Topic%3B A7S - Your Love (9PM).mp3
[22-02-08 15:31:45.6619] Slim::Schema::Contributor::rescan (244) Removing unused contributor: 132058
[22-02-08 15:31:45.6646] Slim::Schema::Album::rescan (273) Removing unused album: 89466
[22-02-08 15:31:53.5483] Slim::Utils::SQLiteHelper::updateProgress (442) Notify to server: [
"progress:1644330698||importer||M:|directory_delet ed||1||1||1644330713.54737",

- scanner detected the new file and added it correctly
[22-02-08 15:32:03.5401] Slim::Utils::Scanner::Local::addTracks (447) Scanning new audio files (1)
[22-02-08 15:32:03.5412] Slim::Utils::SQLiteHelper::updateProgress (442) Notify to server: [
"progress:1644330723.54035||importer||M:|directory _new||1||1||",
]
[22-02-08 15:32:03.5441] Slim::Utils::SQLiteHelper::updateProgress (466) Notify to server OK
[22-02-08 15:32:03.5446] Slim::Utils::Scanner::Local::new (878) Handling new audio track file:///M:/A/ATB/2021 - Your Love (Single)/01 - ATB%3B Topic%3B A7S - Your Love (9PM).mp3
[22-02-08 15:32:03.5665] Slim::Utils::SQLiteHelper::updateProgress (442) Notify to server: [
"progress:1644330723||importer||M:|directory_new|| 1||1||1644330723.5658",
]
[22-02-08 15:32:03.5704] Slim::Utils::SQLiteHelper::updateProgress (466) Notify to server OK

LOOKS GOOD!

b)
Changing "Billy Idol with Generation X" => "Bill Idol; Generation X" (the old "you&me" problem WITHOUT renaming the file)

- scanner found the file and added "Billy Idol" and "Generation X" for this track (and album) and removed "Billy Idol with Generation X
[22-02-08 21:19:31.1362] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/_Various/0/101 Floorfillers/12. Billy Idol with Generation X - Dancing With Myself.mp3
[22-02-08 21:19:31.1806] Slim::Schema::Contributor::rescan (244) Removing unused contributor: 16178

LOOKS GOOD!


c)
Changing "Deejay Ötzi" => "DJ Ötzi"

- scanner found the file and added "DJ Ötzi" for this track (and album) and removed "Deejay Ötzi"
[22-02-08 21:19:32.7800] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/_Various/B/Bravo Hits 030 - CD 2/18 - Deejay Oetzi - Hey Baby - Deejay Oetzi - Hey Baby.mp3
[22-02-08 21:19:32.8177] Slim::Schema::Contributor::rescan (244) Removing unused contributor: 68185

LOOKS GOOD!

d)
Changing "Paul Mccartney" => "Paul McCartney" (case sensivity)
[22-02-08 21:19:35.9607] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/_Various/F/Foute Kerst CD (2006)/06 - Paul Mccartney%3B Frog Chorus - We All Stand Together.mp3

- scanner found the file (as "changed") and...
- added "Paul McCartney" to track / album
- removed "Paul Mccartney" from track / album

BUT ... didn't remove "Paul Mccartney" as unused from contributors...

So I looked into the database and saw:
1. "old" artist ("Paul Mccartney") is still in contributors (# 69120)
2. contributor #69120 is in contributor_album linked to 3 albums (role 2!)
3. contributor #69120 is in contributor_track linked to 3 albums (role 2!)
BUT
If I do a (FTS) search, the artist "Paul Mccartney" wasn't found

So the reason was clear, LMS doesn't delete it, for it is in some other tracks as composer (as You know, I don't have them "checked" as contributors, but I keep the data that are in bought mp3 as "nice-to-have-information")

I thinks, this should be "wanted behaviour"

I took another try to correct all the "Paul Mccartney" => "Paul McCartney" in tag composer as well

.. scanned again and the contributor "Paul Mccartney" was gone

[22-02-08 23:12:55.4882] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/_Various/C/Christmas%20Rules/02%20-%20Shins%20-%20Wonderful%20Christmastime.mp3
[22-02-08 23:12:55.7537] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/B/Bert%20Kaempfert/1971%20-%20Bert%20Kaempfert%20Now!/08%20-%20Bert%20Kaempfert%20-%20Oh%20Woman,%20Oh%20Why.mp3
[22-02-08 23:12:56.0720] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/C/Cilla%20Black/1971%20-%20Images/02%20-%20Cilla%20Black%20-%20Junk.mp3
[22-02-08 23:12:56.0790] Slim::Schema::Contributor::rescan (244) Removing unused contributor: 69120

So it looks very good ... - though I'm careful to call it "solved forever" after one try ...

... a BIG THANK YOU!

[mherger]

How does new&changed scanner detect whatfiles to scan?

> So it looks very good ... - though I'm careful to call it "solved
> forever" after one try ...

Forever is a long time out :-). Thank you very much for your extensive
testing!

[frank1969]

Forever lasted a day ... or I saw a "spooky" phenomenon.

I once again did some changes to my db yesterday and some of them I tracked.
especially those ones
"Nalin & Kane" => "Nalin + Kane"
"Charly Lownoise & Mental Theo" => "Charly Lownoise + Mental Theo"
"Simon & Garfunkel" => "Simon + Garfunkel"
(You see the principle)

I did a "new&changed" on 10 Feb, 00:30 - obviously nothing changed. The old "&" artistst were still "in".

I looked into scanner.log - some tracks were handled ("Handling changed audio track") but not all of them.
I looked into library.db - and some tracks were still "unchanged" with the old "&" and role 1...

Then I did another scan this morning 10 Feb, 09:30.
Once again - some tracks had been changed - some not.
I concentrated on 3 tracks to find them in scanner.log - they were not even there.
I looked intro library.db - they were still there and not changed.

Then I did a 3rd scan today 10 Feb, 10:30.
I was already wondering, for scanner progress showed 99 changed files - and I had done NO changes!
(I only copied the scanner.log and library.db after 2nd scan and started the 3rd scan)
And to my surprise and delight:
The old artists were gone.
And I looked into scanner.log - those files were recognized and handled...

To be sure I once again checked the mtime of my *.mp3-files (though I think, I KNOW what I do and when I do) and it confessed: Last change of all files was on 09 Feb about 23:00 - this means: ALL were changed before the 1st scan already, but only seen and handled with 3rd scan?!

btw I did all 3 scans with last nightly r1644324995

Maybe You remember me mentioning this a few days of one of my testings that scanner found and changed files just at the 3rd scan after they were changed...

No I'm thinking if this might be a "timing problem" between my windows time and the time scanner sees?!

Maybe another observation to this point:
When You post a new update, I often can see it on http://downloads.slimdevices.com/nightly/ but when I check LMS for new updates it says "There's no updated Logitech Media Server version available."
When I install an update, it shows another time than the update was published. e.g r1644324995 is shown as published on "February 08 2022 13:31"
After installation, LMS says "Logitech Media Server Version: 8.3.0 - 1644324995 @ Tue Feb 8 14:17:32 WEST 2022" (and this is not the time of installation but of the "version")

But after scanning, LMS shows the "real" time:
"The server has finished scanning your media library.
Total Time: 00:37:30 (Thursday, 10. February 2022 / 11.13.42)"

Is there a "time" setting withing LMS/sqlite?
Or could there be a link to the language I use?
Usually I use LMS in German - but for "analysis" I sometimes switch to english so I can give You the exact "response" from LMS?!

I'm using windows 11 in my "usual" timezone (Germany Berlin).

Mysterious...

[frank1969]

.. I found another proof:
I did another new&changed scan and it found 44 changed tracks - though I knew exactly I did nothing than "touch" (change mtime) 1 (!!!) file.

So I looked into scanner.log - all files that were found now - 10.02.2022, 18:30 - were changed by me on 08.02.2022 between 23:00 and 23:59.

I guess this explains, why it "sometimes works", "sometimes not"...

Do You have any explanation for this?
Double changed my region settings in windows,
Time: UCT +01:00 Amsterdam, Berlin...
Language: German (Germany)
Land / Region: Germany
Regional/territorial format (don't know what there is in english windows): recommended

[mherger]

How does new&changed scanner detect whatfiles to scan?

> So I looked into scanner.log - all files that were found now -
> 10.02.2022, 18:30 - were changed by me on 08.02.2022 between 23:00 and
> 23:59.

That's just odd. I have no idea why some changes would require multiple
scans to be be recognized as modified files. It just doesn't make sense,
unless the scanner crashes before all files are handled. But you'd see
that in the log and in the scanner progress information.

> I guess this explains, why it "sometimes works", "sometimes not"...

I don't understand what is explained...

Unfortunately without access to your library.db it's hard for me to
understand whether the search for new files is failing, or the
processing of whatever that search found. I'm still offering my inbox:

https://www.dropbox.com/request/T3RctyzGgNg0oFDubq6a

If you zip up the file it should be reasonably small :-)

[frank1969]

> I guess this explains, why it "sometimes works", "sometimes not"...

I don't understand what is explained...

OK, I hoped, my English was better
In German "Das erklärt, warum es einmal geht und einmal nicht (obwohl nichts am Code oder an den Dateien verändert wurde)"

I once again tracked one file - this time it worked on "2nd scan" already (but not on 1st as it should):

I changed the file "12. Nalin & Kane" - Beachball.mp3 on 22-02-09 23:14
(changed "Nalin I.N.C." => "Nalin INC" - and just checked my files, it is STILL with mtime 22-02-09 23:14)

AFTER that I started a scan [22-02-10 00:02:45.8662]

The scan found the file
[22-02-10 00:43:08.4509] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found M:\[...]\12 - Nalin & Kane - Beachball.mp3

but it was NOT handled - this is the only log-entry for this file

I looked in "tracks" (library.db) there it was unchanged with added_time = 1643710345 and updated_time = 1643710345 (which is Tue Feb 01 2022 11:12:25 GMT+0100 - the day of my initial scan )

So scanner didn't see it as "changed" though it was changed more than 1 hour before it was scanned...

I started a 2nd scan on [22-02-10 09:21:59.9377]

NOTICE:
- I didn't do anything change to the file (it is with mtime = 22-02-09 till NOW ;-)) and there had been no scanner/LMS updates

It was once again FOUND:
[22-02-10 10:03:58.0321] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found M:[...]\12 - Nalin & Kane - Beachball.mp3

and this time (!) it was also HANDLED:
[22-02-10 10:09:01.6794] Slim::Utils::Scanner::Local::changed (969) Handling changed audio track file:///M:/[...]/12 - %20Nalin%20&%20Kane%20-%20Beachball.mp3

But I ask myself (and maybe You):
Why didn't the scanner on "1st scan" ( [22-02-10 00:43:08.4509] ) not detect a file from [22-02-09 23:14] as handled - but does it on [22-02-10 10:03:58.0321] ??

I watched this with many files now, that CHANGED files are not detected on 1st scan but on 2nd or even just 3rd scan, while NEW and DELETED files (as far as I see) ALWAYS are detected on 1st scan...?!

By the way: I think You could not see it from my db anywhere (and for it's a business db not even my girlfriend gets access ;-)) for as I described - the problem is, scanner doesn't detect the files and so db doesn't show any changes...

I guess it's a windows/server/sql "timing" problem or something like that...?!

[frank1969]

... and I also took care about those files, I changed on 22-02-08 that scanner didn't see as changed before 22-02-10 (though here have been 5 scans inbetween).

I checked library.db carefully, copied all lines about those files to Excel and searched for any changes - there have been no changes.
I browsed all logfiles - the files have been "found" any time but not "handled" till 2 days after changing...

I meanwhile don't think the problem is the db / scanner itself - the problem results from "when" scanner sees a file as "changed" and maybe which timestamp scanner has to compare with the files timestamp.
Or could it even be, that there is a difference between windows mtime and the "Unix" timestamp, server sees?!

I think I'm not as bad in analysing db's - but I have very little knowledge about server structure...