New in 7.9: Fulltext search

[prob]

more general question, how to clean up metadata

I am back running LMS after serveral years and after having sold my Logitech hardware. My very large music library is collected from all over. I have been tidying up the metadata and paying attention to messages from the scanner.

Is there a preferred way of tidying up a library to avoid scanner / server crashes?

Regards.

[mherger]

New in 7.9: Fulltext search

> It's running Windows on a small Intel NUC (Celeron) with about 4 gig of
> RAM (maybe 8 gig, will check once I get home).

Could you please check memory usage while the scanner is running? In
particular during the Fulltext Search indexing?

> I have the DB memory setting at maximum in the
> Settings/Advanced/Performance.

Would the scanner succeed if you set it to "High"?

> Is there a setting to make the server ignore certain playlists?

Put them in a folder which isn't scanned? Or maybe using my "Ignore
files and folders" plugin (which I haven't used in a long time...).

[prob]

Thanks for your prompt response.

It's running Windows on a small Intel NUC (Celeron) with about 4 gig of RAM (maybe 8 gig, will check once I get home).

I have the DB memory setting at maximum in the Settings/Advanced/Performance.

Is there a setting to make the server ignore certain playlists?

Regards.

[mherger]

New in 7.9: Fulltext search

> It looks like full text search is choking because of the name of a
> playlist. I have nearly 900 playlists so I would help to know what I
> should be looking for. Any suggestions?

It's not the name, but likely just the number of tracks overall: the
scanner is running out of memory. What hardware are you running your LMS
on? What is the DB memory setting in Settings/Advanced/Performance?

[prob]

Error because of name of playlist?

It looks like full text search is choking because of the name of a playlist. I have nearly 900 playlists so I would help to know what I should be looking for. Any suggestions?

Thanks in advance.

>>
[22-01-16 03:58:08.3316] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (458) Starting fulltext index build
[22-01-16 03:58:08.3323] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (462) Initialize fulltext table
[22-01-16 03:58:20.8305] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (475) Create fulltext index for tracks
[22-01-16 04:01:11.3224] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (485) Create fulltext index for albums
[22-01-16 04:03:16.8128] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (494) Create fulltext index for contributors
[22-01-16 04:03:21.0851] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (504) Create fulltext index for playlists
[22-01-16 06:49:47.7155] Slim::Schema::Storage::throw_exception (121) Error: DBI Exception: DBD::SQLite::db do failed: out of memory [for Statement "
INSERT INTO fulltext (id, type, w10, w5, w3, w1)
-- w10: title, w3: url, w1: track metadata
SELECT playlist_track.playlist, 'playlist', ?, '', ?, UNIQUE_TOKENS(GROUP_CONCAT(w10 || ' ' || w5 || ' ' || w3 || ' ' || w1))
FROM playlist_track
LEFT JOIN tracks ON tracks.url = playlist_track.track
LEFT JOIN fulltext ON fulltext MATCH 'type:track ' || IGNORE_PUNCTUATION(REPLACE(REPLACE(playlist_track. track, '%20', ' '), 'file://', ''))
WHERE playlist_track.playlist = ?
"]
[22-01-16 06:49:47.7209] Slim::Schema::Storage::throw_exception (121) Backtrace:

frame 0: Slim::Utils::Log::logBacktrace (/Slim/Schema/Storage.pm line 121)
frame 1: Slim::Schema::Storage::throw_exception (/DBIx/Class/Storage/DBI.pm line 1006)
frame 2: DBIx::Class::Storage:BI::__ANON__ (C:/PROGRA~2/SQUEEZ~1/server/Slim/Plugin/FullTextSearch/Plugin.pm line 545)
frame 3: Slim::Plugin::FullTextSearch::Plugin::_createPlayl istItem (C:/PROGRA~2/SQUEEZ~1/server/Slim/Plugin/FullTextSearch/Plugin.pm line 513)
frame 4: Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (C:/PROGRA~2/SQUEEZ~1/server/Slim/Plugin/FullTextSearch/Plugin.pm line 171)
frame 5: Slim::Plugin::FullTextSearch::Plugin::startScan (/Slim/Music/Import.pm line 580)
frame 6: Slim::Music::Import::runImporter (/Slim/Music/Import.pm line 455)
frame 7: Slim::Music::Import::runScanPostProcessing (scanner.pl line 344)
frame 8: (eval) (scanner.pl line 344)
frame 9: main::main (scanner.pl line 499)

[22-01-16 06:49:47.7558] main::main (348) Error: Failed when running scan post-process: [Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::db do failed: out of memory [for Statement "
INSERT INTO fulltext (id, type, w10, w5, w3, w1)
-- w10: title, w3: url, w1: track metadata
SELECT playlist_track.playlist, 'playlist', ?, '', ?, UNIQUE_TOKENS(GROUP_CONCAT(w10 || ' ' || w5 || ' ' || w3 || ' ' || w1))
FROM playlist_track
LEFT JOIN tracks ON tracks.url = playlist_track.track
LEFT JOIN fulltext ON fulltext MATCH 'type:track ' || IGNORE_PUNCTUATION(REPLACE(REPLACE(playlist_track. track, '%20', ' '), 'file://', ''))
WHERE playlist_track.playlist = ?
"] at /Slim/Schema/Storage.pm line 125
]
[22-01-16 06:49:47.7573] main::main (349) Error: Not updating lastRescanTime!


PS

>>

Version: 8.2.1 - 1639114554 @ Fri Dec 10 07:58:57 WEST 2021
Hostname: Home-Office-PC
Server IP Address: 192.168.214.179
Operating system: Windows 10 - EN - cp1252
Platform Architecture: 8664
Perl Version: 5.14.1 - MSWin32-x86-multi-thread
Audio::Scan: 1.02
IO::Socket::SSL: 2.068
Database Version: DBD::SQLite 1.58 (sqlite 3.22.0)

[slartibartfast]

> Does high work OK with 1GB of RAM? How many tracks do you have and what
> size is your swap space?

It should, unless you run a lot of other services on the same Pi. I've
been using "max" on a pCP based server for years (Pi 3B+). It handles
about 21k tracks smoothly.

MusicIP uses a lot.

Sent from my Pixel 3a using Tapatalk

[mherger]

New in 7.9: Fulltext search

> Does high work OK with 1GB of RAM? How many tracks do you have and what
> size is your swap space?

It should, unless you run a lot of other services on the same Pi. I've
been using "max" on a pCP based server for years (Pi 3B+). It handles
about 21k tracks smoothly.

[macpac]

Does high work OK with 1GB of RAM? How many tracks do you have and what size is your swap space?

Sent from my Pixel 3a using Tapatalk

I have 12,342 tracks and scanning took some hours (maybe also due to other plugins). I set the maximum swap size to 4096.

[slartibartfast]

I also set it to 'high' after Michael's hint. It was on 'normal' before.

Does high work OK with 1GB of RAM? How many tracks do you have and what size is your swap space?

Sent from my Pixel 3a using Tapatalk

[macpac]

What is your "Database memory config" setting in settings/advanced/performance?

Sent from my Pixel 3a using Tapatalk

I also set it to 'high' after Michael's hint. It was on 'normal' before.

[slartibartfast]

Thanks for the hint regarding the cache. I changed the swap partition according to this thread and then it worked.

What is your "Database memory config" setting in settings/advanced/performance?

Edit. When I was using a Pi3B with 1GB RAM I had to use an enormous swap partition to accommodate MusicIP. A Pi4 4GB helped a lot.



Sent from my Pixel 3a using Tapatalk

[macpac]

>> What Pi? How much RAM? How large is your data partition where the LMS
>> cache lives?
>
> Raspberry PI 4B 1GB
> Do you mean dphys-swapfile ?

Where does TMPDIR live? What does "df -h" say? Can you check memory and
free disk space while the scanner is running? Anything in scanner.log?

Thanks for the hint regarding the cache. I changed the swap partition according to this thread and then it worked.

[mherger]

New in 7.9: Fulltext search

>> What Pi? How much RAM? How large is your data partition where the LMS
>> cache lives?
>
> Raspberry PI 4B 1GB
> Do you mean dphys-swapfile ?

Where does TMPDIR live? What does "df -h" say? Can you check memory and
free disk space while the scanner is running? Anything in scanner.log?

[macpac]

> Obviously, you're right. I'm on max2play with DBD::SQLite 1.58 (sqlite

What Pi? How much RAM? How large is your data partition where the LMS
cache lives?

Raspberry PI 4B 1GB
Do you mean dphys-swapfile ?

[mherger]

New in 7.9: Fulltext search

> Obviously, you're right. I'm on max2play with DBD::SQLite 1.58 (sqlite

What Pi? How much RAM? How large is your data partition where the LMS
cache lives?