Scan taking hours instead of minutes on recent nightlies

[kbeast]

>
It's hard to give advice when I lack any information...

I know this quote wasn't for me directly (and its also taken out of context a little), but I still owe you a db file -- I'm still getting an error when scanning. I finally had time to copy the library.db file over, but I'm a little hesitant to upload it to dropbox--

Building full text index   (3 of 7)   Complete  00:01:47
Media scan terminated unexpectedly (Scanning Progress)

Seems like you were making headway, but I'm assuming you still need mine?

[mherger]

I know this quote wasn't for me directly (and its also taken out of context a little), but I still owe you a db file -- I'm still getting an error when scanning. I finally had time to copy the library.db file over, but I'm a little hesitant to upload it to dropbox--

Building full text index   (3 of 7)   Complete  00:01:47
Media scan terminated unexpectedly (Scanning Progress)

Seems like you were making headway, but I'm assuming you still need mine?

Did you see these failures with the latest build? Other reporters of this issue have confirmed it was fixed (for them...).

[kbeast]

Did you see these failures with the latest build? Other reporters of this issue have confirmed it was fixed (for them...).

I'm pretty sure I'm up to date, there's nothing screaming at me that it needs to be. and the scans are still stopping unexpectedly.

[mherger]

> I'm pretty sure I'm up to date, there's nothing screaming at me that it
> needs to be. and the scans are still stopping unexpectedly.

Just post your scanner.log and we'll know...

--

Michael

[kbeast]

Just post your scanner.log and we'll know...

Logitech Media Server Version: 8.0.0 - 1601380107 @ Tue Sep 29 14:20:56 CEST 2020
Hostname: odroid
Server IP Address: xxxxxx
Server HTTP Port Number: 9000
Operating system: Debian - EN - utf8
Platform Architecture: armv7l-linux
Perl Version: 5.26.1 - arm-linux-gnueabihf-thread-multi-64int
Audio::Scan: 0.95
IO::Socket::SSL: 2.060
Database Version: DBD::SQLite 1.34_01 (sqlite 3.7.7.1)
Total Players Recognized: 1

[20-09-30 15:33:27.8827] main::main (206) Starting Logitech Media Server scanner (v8.0.0, 1601380107, Tue Sep 29 14:20:56 CEST 2020) perl 5.026001
[20-09-30 15:33:28.1342] Slim::Schema::forceCommit (2157) Warning: Trying to commit transactions before DB is initialized!
[20-09-30 15:33:28.3237] Slim::Music::Import::runImporter (577) Starting Slim::Media::MediaFolderScan scan
[20-09-30 15:33:28.3243] Slim::Utils::Scanner::Local::rescan (179) Discovering audio files in /mnt/mother/mp3
[20-09-30 15:41:07.6788] Slim::Utils::Scanner::Local::__ANON__ (191) Start processing found tracks
[20-09-30 15:41:07.6796] Slim::Utils::Scanner::Local::__ANON__ (199) Connect do DB
[20-09-30 15:41:07.6799] Slim::Utils::Scanner::Local::__ANON__ (202) Get latest ID
[20-09-30 15:41:07.6804] Slim::Utils::Scanner::Local::__ANON__ (224) Delete temporary table if exists
[20-09-30 15:41:07.6808] Slim::Utils::Scanner::Local::__ANON__ (227) Re-build temporary table
[20-09-30 15:41:11.9542] Slim::Utils::Scanner::Local::__ANON__ (276) Get deleted tracks count
[20-09-30 15:41:16.5410] Slim::Utils::Scanner::Local::__ANON__ (283) Get new tracks count
[20-09-30 15:41:16.5418] Slim::Utils::Scanner::Local::__ANON__ (288) Get changed tracks count
[20-09-30 15:41:22.5568] Slim::Utils::Scanner::Local::deleteTracks (519) Removing deleted audio files (0)
[20-09-30 15:41:22.5573] Slim::Utils::Scanner::Local::__ANON__ (301) Scanning new audio files (1)
[20-09-30 15:41:22.5954] Slim::Utils::Scanner::Local::__ANON__ (381) Rescanning changed audio files (0)
[20-09-30 15:41:22.5964] Slim::Music::Import::endImporter (710) Completed Slim::Media::MediaFolderScan Scan in 474 seconds.
[20-09-30 15:41:22.5972] Slim::Music::Import::runImporter (577) Starting Slim::Music::PlaylistFolderScan scan
[20-09-30 15:41:22.6002] Slim::Utils::Scanner::Local::rescan (179) Discovering audio files in /mnt/mother/! Playlists
[20-09-30 15:41:23.6902] Slim::Utils::Scanner::Local::__ANON__ (191) Start processing found tracks
[20-09-30 15:41:23.6909] Slim::Utils::Scanner::Local::__ANON__ (199) Connect do DB
[20-09-30 15:41:23.6913] Slim::Utils::Scanner::Local::__ANON__ (202) Get latest ID
[20-09-30 15:41:23.6919] Slim::Utils::Scanner::Local::__ANON__ (224) Delete temporary table if exists
[20-09-30 15:41:23.6924] Slim::Utils::Scanner::Local::__ANON__ (227) Re-build temporary table
[20-09-30 15:41:27.8180] Slim::Utils::Scanner::Local::__ANON__ (276) Get deleted tracks count
[20-09-30 15:41:30.3941] Slim::Utils::Scanner::Local::__ANON__ (283) Get new tracks count
[20-09-30 15:41:30.3947] Slim::Utils::Scanner::Local::__ANON__ (288) Get changed tracks count
[20-09-30 15:41:30.3970] Slim::Utils::Scanner::Local::deleteTracks (519) Removing deleted audio files (0)
[20-09-30 15:41:30.3973] Slim::Utils::Scanner::Local::__ANON__ (301) Scanning new audio files (0)
[20-09-30 15:41:30.3976] Slim::Utils::Scanner::Local::__ANON__ (381) Rescanning changed audio files (0)
[20-09-30 15:41:30.3981] Slim::Music::Import::endImporter (710) Completed Slim::Music::PlaylistFolderScan Scan in 7 seconds.
[20-09-30 15:41:30.4025] Slim::Music::Import::runImporter (577) Starting Slim::Plugin::ExtendedBrowseModes::Libraries scan
[20-09-30 15:41:30.4029] Slim::Music::Import::runImporter (577) Starting Slim::Plugin::FullTextSearch::Plugin scan
[20-09-30 15:41:30.4040] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (445) Starting fulltext index build
[20-09-30 15:41:30.4043] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (449) Initialize fulltext table
[20-09-30 15:41:36.3298] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (462) Create fulltext index for tracks
[20-09-30 15:42:47.6769] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (472) Create fulltext index for albums
[20-09-30 15:43:08.6206] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (481) Create fulltext index for contributors
[20-09-30 15:43:10.5347] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (491) Create fulltext index for playlists
[20-09-30 15:43:10.7835] Slim::Schema::Storage::throw_exception (121) Error: DBI Exception: DBD::SQLite::db do failed: SQL logic error or missing database
unable to use function MATCH in the requested context [for Statement "
	INSERT  INTO fulltext (id, type, w10, w5, w3, w1)
		-- w10: title, w3: url, w1: track metadata
		SELECT playlist_track.playlist, 'playlist', ?, '', ?, UNIQUE_TOKENS(GROUP_CONCAT(w10 || ' ' || w5 || ' ' || w3 || ' ' || w1))
		FROM playlist_track
			LEFT JOIN tracks ON tracks.url = playlist_track.track
			LEFT JOIN fulltext ON fulltext MATCH IGNORE_PUNCTUATION(REPLACE(REPLACE(playlist_track.track, '%20', ' '), 'file://', '')) AND type = 'track'
		WHERE playlist_track.playlist = ?
"]
[20-09-30 15:43:10.7842] Slim::Schema::Storage::throw_exception (121) Backtrace:

   frame 0: Slim::Utils::Log::logBacktrace (/usr/share/perl5/Slim/Schema/Storage.pm line 121)
   frame 1: Slim::Schema::Storage::throw_exception (/usr/share/squeezeboxserver/CPAN/DBIx/Class/Storage/DBI.pm line 1007)
   frame 2: DBIx::Class::Storage::DBI::__ANON__ (/usr/share/perl5/Slim/Plugin/FullTextSearch/Plugin.pm line 532)
   frame 3: Slim::Plugin::FullTextSearch::Plugin::_createPlaylistItem (/usr/share/perl5/Slim/Plugin/FullTextSearch/Plugin.pm line 500)
   frame 4: Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (/usr/share/perl5/Slim/Plugin/FullTextSearch/Plugin.pm line 158)
   frame 5: Slim::Plugin::FullTextSearch::Plugin::startScan (/usr/share/perl5/Slim/Music/Import.pm line 579)
   frame 6: Slim::Music::Import::runImporter (/usr/share/perl5/Slim/Music/Import.pm line 455)
   frame 7: Slim::Music::Import::runScanPostProcessing (/usr/sbin/squeezeboxserver-scanner line 344)
   frame 8: (eval) (/usr/sbin/squeezeboxserver-scanner line 344)
   frame 9: main::main (/usr/sbin/squeezeboxserver-scanner line 499)

[20-09-30 15:43:10.7865] main::main (348) Error: Failed when running scan post-process: [Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::db do failed: SQL logic error or missing database
unable to use function MATCH in the requested context [for Statement "
	INSERT  INTO fulltext (id, type, w10, w5, w3, w1)
		-- w10: title, w3: url, w1: track metadata
		SELECT playlist_track.playlist, 'playlist', ?, '', ?, UNIQUE_TOKENS(GROUP_CONCAT(w10 || ' ' || w5 || ' ' || w3 || ' ' || w1))
		FROM playlist_track
			LEFT JOIN tracks ON tracks.url = playlist_track.track
			LEFT JOIN fulltext ON fulltext MATCH IGNORE_PUNCTUATION(REPLACE(REPLACE(playlist_track.track, '%20', ' '), 'file://', '')) AND type = 'track'
		WHERE playlist_track.playlist = ?
"] at /usr/share/perl5/Slim/Schema/Storage.pm line 125
]
[20-09-30 15:43:10.7868] main::main (349) Error: Not updating lastRescanTime!

[mherger]

> [20-09-30 15:43:10.7835] Slim::Schema::Storage::throw_exception (121) Error: DBI Exception: DBD::SQLite::db do failed: SQL logic error or missing database
> unable to use function MATCH in the requested context [for Statement "

I thought I had that fixed... would you mind sharing your library.db
with me?

https://www.dropbox.com/request/T3RctyzGgNg0oFDubq6a


--

Michael

[kbeast]

I thought I had that fixed... would you mind sharing your library.db
with me?

yes, will send over.

Looking at the error, anything to do with possibly saving youtube in a playlist?

youtube://

I notice there's a LEFT JOIN that is truncating/replacing file:// and %20 stuff.

[carp]

It seems that the playlist issue also applies to CUE files.

Btw: There is no iTunes with me.

Logitech Media Server Version: 8.0.0 - 1600144351 @ Tue Sep 15 06:55:13 CEST 2020

[Eric Seaberg]

Michael, I'm wondering if something changed in 7.9.3 as well? After all the issues I went through with Catalina on Mac MINI, I reverted back to High Sierra from a TimeMachine backup, but installed 7.9.3 and did a complete rescan. NO iTUNES, just my audio drives. There's a little over 40,000 songs and it took almost 2.5 hours to complete the scan. When I've done it before, a new rescan would take 5-10 minutes at the most.

I most likely had a 'nightly' in the TimeMachine restore but wouldn't know which one. TimeMachine date was Sept 9, and I'm usually pretty good about keeping up with the nightlies if that gives you an idea.


Thanks again for all of the help. I'm stable again and will avoid upgrading the OS for a while.



Eric