PDA

View Full Version : Unable to perform full rescan



jimzak
2019-08-29, 07:31
I have over 400k files on two 8 TB hard drives.

Recent complete rescans have stalled at the point you see below:

Logitech Media Server Version: 7.9.2 - 1565967976 @ Fri Aug 16 17:08:05 WEDT 2019
Hostname: musicserver2019
Server IP Address: 192.168.0.xxx
Server HTTP Port Number: 9000
Operating system: Windows 10 - EN - cp1252
Platform Architecture: 8664
Perl Version: 5.14.1 - MSWin32-x86-multi-thread
Audio::Scan: 0.95
IO::Socket::SSL: 2.024
Database Version: DBD::SQLite 1.58 (sqlite 3.22.0)
Total Players Recognized: 7

Media Scan Details
Discovering files/directories: J: (231441 of 231441) Complete 00:05:00

Scanning new music files: J: (206239 of 206239) Complete 00:46:43

Discovering files/directories: K: (272071 of 272071) Complete 00:06:43

Scanning new music files: K: (244656 of 244656) Complete 01:09:26

Discovering files/directories: K:\V-Z\Various (1 of 1) Complete 00:00:00

Discovering playlists: C:\Users\user\Music\Playlists (1 of 1) Complete 00:00:00

Building full text index (1 of 7) Running 01:37:20

Scanner log:

[19-08-29 07:47:11.8381] Slim::Utils::Scanner::Local::__ANON__ (229) Re-build temporary table
[19-08-29 07:47:13.4713] Slim::Utils::Scanner::Local::__ANON__ (278) Get deleted tracks count
[19-08-29 07:47:13.4716] Slim::Utils::Scanner::Local::__ANON__ (285) Get new tracks count
[19-08-29 07:47:13.4719] Slim::Utils::Scanner::Local::__ANON__ (290) Get changed tracks count
[19-08-29 07:47:13.4720] Slim::Utils::Scanner::Local::__ANON__ (296) Removing deleted audio files (0)
[19-08-29 07:47:13.4722] Slim::Utils::Scanner::Local::__ANON__ (374) Scanning new audio files (0)
[19-08-29 07:47:13.4723] Slim::Utils::Scanner::Local::__ANON__ (454) Rescanning changed audio files (0)
[19-08-29 07:47:13.4725] Slim::Music::Import::endImporter (643) Completed Slim::Music::PlaylistFolderScan Scan in 1 seconds.
[19-08-29 07:47:13.4739] Slim::Music::Import::runImporter (511) Starting Slim::Plugin::FullTextSearch::Plugin scan
[19-08-29 07:47:13.4744] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (413) Starting fulltext index build
[19-08-29 07:47:13.4746] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (417) Initialize fulltext table
[19-08-29 07:47:13.4760] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (430) Create fulltext index for tracks

Any diagnostic or therapeutic help would be appreciated.

mherger
2019-08-29, 08:12
> I have over 400k files on two 8 TB hard drives.

How large are your library.db* files? How much free disk space do you
have on the drive where your LMS cache is?

--

Michael

jimzak
2019-08-29, 16:17
> I have over 400k files on two 8 TB hard drives.

How large are your library.db* files? How much free disk space do you
have on the drive where your LMS cache is?

--

Michael

Thanks for replying.

The cache is 3.16 GB on an 8 GB RAM drive that I have been using for several years. Full rescans have worked well over the years until now.

However, I found a workaround.

After the full scan stalled, I aborted it. I then ran a "new and changed media" scan which completed.

Total Tracks: 452,925
Total Albums: 45,690
Total Artists: 66,298
Total Genres: 2,236
Total Playing Time: 49108:12:58

Media Scan Details
Discovering files/directories: J: (231441 of 231441) Complete 00:06:05

Scanning changed files: J: (206192 of 206192) Complete 00:52:33

Discovering files/directories: K: (272071 of 272071) Complete 00:06:59

Scanning new music files: K: (1 of 1) Complete 00:00:00

Scanning changed files: K: (244634 of 244634) Complete 01:20:27

Discovering files/directories: K:\V-Z\Various (1 of 1) Complete 00:00:00

Discovering playlists: C:\Users\user\Music\Playlists (1 of 1) Complete 00:00:00

Building full text index (7 of 7) Complete 00:01:55

Find updated coverart files (45636 of 45636) Complete 00:05:35

Pre-caching Artwork (47037 of 47037) Complete 00:34:19

Database Optimize (2 of 2) Complete 00:00:42

-----------

Unclear why I had to do this.

mherger
2019-08-29, 21:16
> The cache is 3.16 GB on an 6 GB RAM drive that I have been using for
> several years. Full rescans have worked well over the years until now.

That's a large file, and the RAM drive probably not large enough. At
some stages the temporary files created by the scanner can become as
large as the main library. Could you try to increase the RAM disk's
size, or put the cache back on your main drive? Or at least monitor disk
usage while in the critical phase of the scan?

--

Michael

mherger
2019-08-30, 00:35
>>> I have over 400k files on two 8 TB hard drives.
....
>
> The cache is 3.16 GB on an 6 GB RAM drive that I have been using for

Thinking about this again... 3GB is HUGE, even for 400k tracks. I've got
20k, library.db is about 40M. Assuming 20x the tracks generate 20x the
data, this should still be less than 1GB. That's >8k of information per track.
Even with Lyrics this seems like a lot (artwork is stored separately).

Next time you plan to do a full wipe & scan I'd suggest you shut down
LMS and delete library.db*.

--

Michael

zordaz
2019-08-30, 02:06
This looks familiar to me: https://forums.slimdevices.com/showthread.php?109660-LMS-rescan-problems&highlight=

My .db files were not that big, recently I tried a full rescan after removal all of them, but it still didn't work.
My workaround is to choose "Looking for new and changed media files" when rescanning.

mherger
2019-08-30, 02:14
> This looks familiar to me:
> https://forums.slimdevices.com/showthread.php?109660-LMS-rescan-problems&highlight=

You're posting too many references to other threads... that thread you
link here is pointing to yet another one, which in turn would link back
to yours... Maybe you should be more specific.

--

Michael

jimzak
2019-08-30, 05:22
>>> I have over 400k files on two 8 TB hard drives.
....
>
> The cache is 3.16 GB on an 6 GB RAM drive that I have been using for

Thinking about this again... 3GB is HUGE, even for 400k tracks. I've got
20k, library.db is about 40M. Assuming 20x the tracks generate 20x the
data, this should still be less than 1GB. That's >8k of information per track.
Even with Lyrics this seems like a lot (artwork is stored separately).

Next time you plan to do a full wipe & scan I'd suggest you shut down
LMS and delete library.db*.

--

Michael

Very interesting.

I have displayed the contents of my cache below. Let me know if something sticks out as being strange. My RAM drive is 8 GB (6 was a typo). And deleting library.db is exactly how I initiated the failed rescans.

28065

mherger
2019-08-30, 05:45
> I have displayed the contents of my cache below. Let me know if
> something sticks out as being strange. My RAM drive is 8 GB (6 was a
> typo). And deleting library.db is exactly how I initiated the failed
> rescans.

Ah, ok. That 3.16GB file is not library.db, but artwork.db. That makes
sense (as does the size of library.db).

But still I'd monitor disk space next time you run a full scan.
Optimizing those files can double the disk space requirement temporarily.

--

Michael

jimzak
2019-09-15, 14:56
I took the RAM disk offline and did a complete resinstall of the server, latest beta.

The scan still does not complete.

It seems to be choking on damaged jpeg image files.

The reason for damaged files relates to an old hard drive enclosure that I was previously using. Apparently the USB controller was damaged or just became ill and started damaging any files that passed through it, likely on the way in from my network.

I didn't discover the problem for while, because, I think, very few people verify that files are copied correctly. When I finally did, I replaced the hardware, but the damage was done.

Not only was the damage done, but the backups were damaged too.

I have the original files, which I am slowly restoring from 25 GB Blu-ray recordables.

However, in the meantime, it would be nice if I could get the server to ignore the damaged jpegs and complete the scan.

Any help would be appreciated.

Addendum: the Squeezesonic plugin at least lets me access my collection while we are sorting this out.

-------------------------

Total Tracks: 464,440
Total Albums: 46,302
Total Artists: 66,431
Total Genres: 2,257
Total Playing Time: 52416:35:56

Media Scan Details
Discovering files/directories: J: (235549 of 235549) Complete 00:05:22

Scanning new music files: J: (210062 of 210062) Complete 00:42:56

Discovering files/directories: K: (280101 of 280101) Complete 00:07:25

Scanning new music files: K: (252086 of 252086) Complete 00:53:31

Discovering files/directories: K:\V-Z\Various (1 of 1) Complete 00:00:00

Discovering playlists: C:\Users\user\Music\Playlists (1 of 1) Complete 00:00:00

Building full text index (7 of 7) Complete 00:01:16

Pre-caching Artwork (16132 of 47561) Complete 00:08:35

Media scan terminated unexpectedly (Scanning Progress)
Total Time: 01:59:05 (Sunday, September 15, 2019 / 4:23 PM)

Excerpt from scanner log:

[19-09-15 16:16:35.2738] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2739] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2741] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2742] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2744] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2746] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2747] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2749] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2750] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2752] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2753] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2755] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2756] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2758] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2759] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Ignoring bad adaptive filter type (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2764] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Extra compressed data (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:35.2765] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libpng warning: Extra compression data (K:\Various A-L\Hercules and Love Affair - DJ-Kicks\10. In Flagranti - Magojiro (Original Mix).mp3)
[19-09-15 16:16:40.4238] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libjpeg error: Corrupt JPEG data: premature end of data segment (J:\Misc\Dance\Porterhouse\Deetron - Re-Creation Remixes Compiled-\01 - Deetron, Justin Chapman - Let's Get Over It Feat. Justin Chapman (Deetron Unreleased Remix).flac)
[19-09-15 16:16:42.4925] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libjpeg error: Corrupt JPEG data: bad Huffman code (K:\T\The Threshold HouseBoys Choir - The Threshold Houseboys Choir Amulet Edition\CD B\02_-_Threshold_HouseBoys_Choir,_The_-_As_X_Is_To_Geff.flac)
[19-09-15 16:16:43.3792] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libjpeg error: Corrupt JPEG data: 2 extraneous bytes before marker 0xd9 (K:\T\Two Lone Swordsmen - Wrong Meeting (2007) (FLAC)\09 - Two Lone Swordsmen - Get Out Of My Kingdom.flac)
[19-09-15 16:16:46.6143] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libjpeg error: Corrupt JPEG data: 53 extraneous bytes before marker 0xd9 (K:\T\Tally Hall - Good & Evil\14_-_Tally_Hall_-_Fate_Of_The_Stars.flac)
[19-09-15 16:16:54.0967] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libjpeg error: Corrupt JPEG data: 227 extraneous bytes before marker 0xd9 (K:\T\Thanatos - Blisters\11_-_Thanatos_-_Dry_Gray_Whisper.flac)

Roland0
2019-09-15, 19:04
It seems to be choking on damaged jpeg image files.
However, in the meantime, it would be nice if I could get the server to ignore the damaged jpegs and complete the scan.

Excerpt from scanner log:
[19-09-15 16:16:43.3792] Slim::Utils::GDResizer::resize (205) Warning: Image::Scale libjpeg error: Corrupt JPEG data: 2 extraneous bytes before marker 0xd9 (K:\T\Two Lone Swordsmen - Wrong Meeting (2007) (FLAC)\09 - Two Lone Swordsmen - Get Out Of My Kingdom.flac)


It seems that the damaged images are embedded in the music files. If that's the case, it's quite likely the music files are corrupt as well.
Run e.g.

flac -t K:\T\Two Lone Swordsmen - Wrong Meeting (2007) (FLAC)\09 - Two Lone Swordsmen - Get Out Of My Kingdom.flac

to check

Then, try


metaflac --export-picture-to=<some filename> K:\T\Two Lone Swordsmen - Wrong Meeting (2007) (FLAC)\09 - Two Lone Swordsmen - Get Out Of My Kingdom.flac

and verify the exported picture

jimzak
2019-09-18, 16:56
Undoubtedly you are correct. But the damaged music files play (with glitches); however, I cannot get the scanner to complete scanning so I can play them.

Can you suggest a file scanner so that I can identify the damaged ones from my library of 400k+ files?

jimzak
2019-09-19, 04:05
So, in the absence of a better short term plan, I looked at the scanner log and simply removed the embedded artwork from the entries that seemed to choke the scanner.

I then performed a rescan for new and changed files, and the scan was at least able to complete.

My long term plan is to find a music file scanner and remove all the corrupted files but at least for now, I can play the glitchy ones.

-----------------------


Discovering files/directories: J: (235549 of 235549) Complete 00:02:31

Scanning changed files: J: (25 of 25) Complete 00:00:02

Discovering files/directories: K: (280089 of 280089) Complete 00:02:51

Removing deleted files: K: (20 of 20) Complete 00:00:08

Scanning new music files: K: (10 of 10) Complete 00:00:00

Scanning changed files: K: (23 of 23) Complete 00:00:02

Discovering files/directories: K:\V-Z\Various (1 of 1) Complete 00:00:00

Discovering playlists: C:\Users\user\Music\Playlists (1 of 1) Complete 00:00:00

Building full text index (7 of 7) Complete 00:01:20

Find updated coverart files (46441 of 46441) Complete 00:00:16

Pre-caching Artwork (31398 of 31398) Complete 00:12:21

Database Optimize (2 of 2) Complete 00:01:23

The server has finished scanning your media library.
Total Time: 00:20:54 (Thursday, September 19, 2019 / 5:54 AM)

dolodobendan
2019-09-19, 05:38
Very interesting.

I have displayed the contents of my cache below. Let me know if something sticks out as being strange. My RAM drive is 8 GB (6 was a typo). And deleting library.db is exactly how I initiated the failed rescans.

28065


> I have displayed the contents of my cache below. Let me know if
> something sticks out as being strange. My RAM drive is 8 GB (6 was a
> typo). And deleting library.db is exactly how I initiated the failed
> rescans.

Ah, ok. That 3.16GB file is not library.db, but artwork.db. That makes
sense (as does the size of library.db).

But still I'd monitor disk space next time you run a full scan.
Optimizing those files can double the disk space requirement temporarily.

--

Michael

Do I have to worry about my 1GB persist.db? I don't have 400k tracks...

dolodobendan
2019-09-19, 05:45
My long term plan is to find a music file scanner and remove all the corrupted files but at least for now, I can play the glitchy ones.

You could give AudioTester (http://www.vuplayer.com/other.php) a try.

mherger
2019-09-19, 05:48
> Do I have to worry about my 1GB persist.db? I don't have 400k tracks...

That sounds like an awful lot. Run

select count(1) from tracks_persistent;

to see how many records there are. It shouldn't be much more than the
number of tracks you have.

But then I don't know what eg. some of Erland's plugins do. They could
be blowing it up.

--

Michael

garym
2019-09-19, 05:52
You could give AudioTester (http://www.vuplayer.com/other.php) a try.

or foobar2000 with the "file integrity verifier" component.
https://www.foobar2000.org/download
https://www.foobar2000.org/components/view/foo_verifier

dolodobendan
2019-09-19, 06:36
That sounds like an awful lot. Run

select count(1) from tracks_persistent;

to see how many records there are. It shouldn't be much more than the
number of tracks you have.


I don't know where to run that, but I opened the file and had a look at the table "tracks_persistent". It has the same number of entries that LMS reports as total track count.


But then I don't know what eg. some of Erland's plugins do. They could
be blowing it up.


There's also a table "Customscan_track_attributes" with 1.3 million entries. So, yeah.

Thank you. :)

Roland0
2019-09-19, 15:44
My long term plan is to find a music file scanner and remove all the corrupted files but at least for now, I can play the glitchy ones.


While the suggested tools may be simple to use, depending on the number of corrupt files they report, it might become quite tedious to manually remove all files.
Here's a simple bash script to find corrupt flacs:


IFS=$'\n'; for i in $(find . -name \*.flac); do flac -t --totally-silent "$i" || echo "$i"; done
The output can be redirected to a text file.
Substituting mv or rm for echo would move or delete the file, automating the whole process.

jimzak
2019-09-20, 16:49
Now that the scans complete, I am impressed by the scanning speed, especially for a "new or changed music scan".

It's lightning fast without a RAM drive: all my files in about 7 minutes.

Next is the arduous process of corrupted file surgery.

Thanks for the tips above.

zordaz
2019-11-11, 02:36
> This looks familiar to me:
> https://forums.slimdevices.com/showthread.php?109660-LMS-rescan-problems&highlight=

You're posting too many references to other threads... that thread you
link here is pointing to yet another one, which in turn would link back
to yours... Maybe you should be more specific.

--

Michael

You're right Michael. The thing is I don't know what to analyze further.
Today I again tried a 'clear librabry and rescan everything' with a recent LMS build.
The scan seems to freeze/stop after the first to steps:

Discovering files/directories: /srv/data/music (37104 of 37104) Complete 00:00:13
Scanning new music files: /srv/data/music (33169 of 33169) Complete 00:12:19

Latest entry in scanner.log:
[19-11-11 10:14:59.8102] Slim::Utils::Scanner::Local::__ANON__ (454) Rescanning changed audio files (0)

Latest entry in server.log
[19-11-11 08:51:14.4117] Slim::Utils::Scanner::Local::__ANON__ (454) Rescanning changed audio files (0)

And that's about it, nothing more is happening. The only solution is to send a kill -9 to all LMS processes, do a manual restart of LMS
and start a scan via 'look for new and changed media files', which works fine.

Any suggestions?