FTS crashes with Out Of Memory error (using 1,6 GB of RAM)

[slartibartfast]

> Or did you forget to reset some logging you enabled at
some point?

Yup that was it [emoji2357]. I was investigating a playlist issue three weeks ago.


Sent from my Pixel 3a using Tapatalk

[slartibartfast]

> Unfortunately I'm currently facing an issue with the build pipeline. The
> latest build hasn't been... ahm... built yet :-). I'll have to see
> what's wrong with it once I'm in the office.

There's a problem with our Windows signing host. So... I've disabled
signing for this one build :-(. Hopefully it'll get through and not
cause any additional issues for Windows users.

Slightly off topic but I notice recent LMS updates have lots of logging enabled by default for playlists. Is that intentional?

Sent from my Pixel 3a using Tapatalk

[mherger]

FTS crashes with Out Of Memory error (using1, 6 GB of RAM)

> Put the other thing into an issue:
> https://github.com/Logitech/slimserver/issues/787

Thanks! I haven't found time to look into it yet, though.

[frank1969]

> Unfortunately I'm currently facing an issue with the build pipeline. The
> latest build hasn't been... ahm... built yet :-). I'll have to see
> what's wrong with it once I'm in the office.

There's a problem with our Windows signing host. So... I've disabled
signing for this one build :-(. Hopefully it'll get through and not
cause any additional issues for Windows users.

OK, just installed r1651382409

Put the other thing into an issue:

https://github.com/Logitech/slimserver/issues/787

[mherger]

FTS crashes with Out Of Memory error (using1, 6 GB of RAM)

> Unfortunately I'm currently facing an issue with the build pipeline. The
> latest build hasn't been... ahm... built yet :-). I'll have to see
> what's wrong with it once I'm in the office.

There's a problem with our Windows signing host. So... I've disabled
signing for this one build :-(. Hopefully it'll get through and not
cause any additional issues for Windows users.

[mherger]

FTS crashes with Out Of Memory error (using1, 6 GB of RAM)

>>> I've applied a change which _should_ address this case (if this
>> configuration difference indeed was the reason for the crash). So no
>> need to change the db memory setting if you tested with the latest LMS
>> 8.3 build.
>
> Yes, I tested with latest nightly ( r1651071091 ).

Unfortunately I'm currently facing an issue with the build pipeline. The
latest build hasn't been... ahm... built yet :-). I'll have to see
what's wrong with it once I'm in the office.

[frank1969]

I reported the "not-deleted artist-thing" in an issue

https://github.com/Logitech/slimserver/issues/787

[frank1969]

> I've applied a change which _should_ address this case (if this
configuration difference indeed was the reason for the crash). So no
need to change the db memory setting if you tested with the latest LMS
8.3 build.

Yes, I tested with latest nightly ( r1651071091 ).

Did some test scans to loacate the other bug (not-deleting deleted artists on a track), will try to analyze later.

[mherger]

FTS crashes with Out Of Memory error (using1, 6 GB of RAM)

> You could probably test this by setting the db memory to high instead of
> max, wipe the "popularTerms" and restart LMS to launch a re-build in the
> server. If my theory was right, that would run successfully, whereas
> "max" would make LMS crash.

I've applied a change which _should_ address this case (if this
configuration difference indeed was the reason for the crash). So no
need to change the db memory setting if you tested with the latest LMS
8.3 build.

[mherger]

FTS crashes with Out Of Memory error (using1, 6 GB of RAM)

> So
> - on scenario 1 (building FTS on startup), server uses more and more RAM
> till it crashes with an out of memory error.
> - on scenario 2 (building FTS while n&c scanning), server uses just a
> little more RAM and scanner's RAM use stays constantly...

Now that's interesting. The database indeed is configured differently in
the server (mostly read-only) vs. the scanner (lots of writing going
on). Cache and buffer sizes are different in these two environments.

You could probably test this by setting the db memory to high instead of
max, wipe the "popularTerms" and restart LMS to launch a re-build in the
server. If my theory was right, that would run successfully, whereas
"max" would make LMS crash.

Your setup exposes quite some weaknesses and poor assumptions in LMS :-).

> This time the FTS is in scanner.log - and the step, that let it crash
> while startup before
> (Slim::Plugin::FullTextSearch::Plugin::_rebuildInd ex (536) Optimize
> fulltext index) is only taking 5 sec ...)

Here I don't have any idea why that would be.

[frank1969]

Another observation.
library.db after initial scan had been 1,57 GB (as I told before)
now - one "new&changed" with FTS later - it is 2,27 GB again...

And:
Another old "bastard" is back

=> I had a file with "Jeanette" and "Jeanette Biedermann" as (track) artist.
=> I deleted "Jeanette" as artist
=> did a new&changed
=> browsing artists it STILL appears calling artist "Jeanette" AND "Jeanette Biedermann" - but when I call it from "Jeanette" it is ... "empty" (just the cover and artist name appearing).
I will examine this further the next days.
(unfortunately I obviously have set back the logging/debugging options so I have nothing about this change/handling in scanner.log... - will change logging-setting back)

[frank1969]

Is LMS crashing due to that indexing? Can you check
prefs/plugin/fulltext.prefs to see whether there are any values after

popularTerms:
- in
- interpret
- mp3
- music
- of

or similar? LMS should _not_ trigger the scan on startup if that list
existed (the scanner would). So if you can make sure that list exists
_before_ starting LMS you should be able to start up LMS, then launch a
scan. Would that scan succeed?

OK, thanks for Your hints, I could fix it with this - I'll describe what I did and what happened, maybe it gives You a hint, why LMS crashes on initial scan.

1. I Checked the content of my fulltext.prefs. It was:
---
_version: 1

(Remark: After FTS crashed, I deactivated it again, so maybe therefore it was "empty").

2. I modified the fulltext.prefs (using some terms of my old installation like):
---
_ts_popularTerms: 1650471674
_version: 1
popularTerms:
- '16'
- '2021'
- '2022'
- '256000'
- 256kbps
- '320000'
- 320kbps
- '44'
- '44100'
- album
- albumartist
- artist
- artists
- compilations
- composer

3. Started LMS - this time it started without crashing

4. Started a "new&scan" - "new&scan" run fine (and created a 71 kb sized fulltext.prefs again)

What I find interesting is, what I watched concerning RAM usage:

- Usually server uses 0,9 - 1 GB of RAM.

- When LMS tried to build the FTS on server start, usage from server grew up to 1,7 GB RAM - and then crashed

- When I did the "new&changed", scanner more or less constantly used 0,3 GB RAM

- server used from 0,9 up to 1,2 GB RAM during the new&changed scan.

So
- on scenario 1 (building FTS on startup), server uses more and more RAM till it crashes with an out of memory error.
- on scenario 2 (building FTS while n&c scanning), server uses just a little more RAM and scanner's RAM use stays constantly...

btw:
This time the FTS is in scanner.log - and the step, that let it crash while startup before (Slim::Plugin::FullTextSearch::Plugin::_rebuildInd ex (536) Optimize fulltext index) is only taking 5 sec ...)

[22-04-30 15:27:25.4391] Slim::Music::Import::runImporter (579) Starting Slim::Plugin::FullTextSearch::Plugin scan
[22-04-30 15:27:25.4397] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (475) Starting fulltext index build
[22-04-30 15:27:25.4400] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (479) Initialize fulltext table
[22-04-30 15:27:25.4415] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (492) Create fulltext index for tracks
[22-04-30 15:33:11.7830] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (502) Create fulltext index for albums
[22-04-30 15:37:13.4179] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (511) Create fulltext index for contributors
[22-04-30 15:37:20.0594] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (521) Create fulltext index for playlists
[22-04-30 15:38:08.3578] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (536) Optimize fulltext index
[22-04-30 15:38:13.9458] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (551) Fulltext index build done!
[22-04-30 15:38:13.9462] Slim::Music::Import::endImporter (711) Completed Slim::Plugin::FullTextSearch::Plugin Scan in 648.507 seconds.

[mherger]

FTS crashes with Out Of Memory error (using1, 6 GB of RAM)

> If I understand right, FTS index is completely re-built with every
> new&changed, right?

Yes, there's no incremental update or something.

> Is there any difference between building the FTS index at the end of a
> new&changed-scan and the end of a initial scan?

No.

Is LMS crashing due to that indexing? Can you check
prefs/plugin/fulltext.prefs to see whether there are any values after

popularTerms:
- in
- interpret
- mp3
- music
- of

or similar? LMS should _not_ trigger the scan on startup if that list
existed (the scanner would). So if you can make sure that list exists
_before_ starting LMS you should be able to start up LMS, then launch a
scan. Would that scan succeed?

[frank1969]

>How old is this? As you're saying LMS version from March would show the
same behaviour?

The database had an initial scan from february and was updated (new&changed) nearly daily.
I tested with r1647754697 (march 20th, I guess)

What about disk space on your C: drive? What is the 1.7 GB RAM number
based on?

Free disk space on C:\ is around 300 GB.
The 1,7 GB RAM is taken from windows task manager used by process from SqueezeSvr - it "usually" uses abou 0,9 - 1 GB RAM and it grows up to 1,7 during starting process while building FTS index.

If I understand right, FTS index is completely re-built with every new&changed, right?
Is there any difference between building the FTS index at the end of a new&changed-scan and the end of a initial scan?

[mherger]

FTS crashes with Out Of Memory error (using1, 6 GB of RAM)

> If I switch back to my old database (even 2,27 GB), grown over weeks or
> months, FTS runs fine.

How old is this? As you're saying LMS version from March would show the
same behaviour?

> Is there any workaround? I don't see that spending 1,7 GB RAM is "too
> much" on a 16 GB RAM system with 10 GB free at this point...?

What about disk space on your C: drive? What is the 1.7 GB RAM number
based on?