Memory Leak in Perl Engine on piCorePlayer?

[mvordeme]

For a few days, now, I have been running a piCorePlayer 6.1.0 with LMS on a Pi 4 B with 2G of memory. I mainly use the Pi to transcode AAC streams from Tidal. There are no additional plug-ins installed.

I observe the following behaviour: After the LMS is started, it will run without complaints for about a day, and then the connected players will suddenly start to re-buffer all the time and eventually stop playing altogether.

I have been looking at CPU and memory consumption on the Pi (I don't quite understand how the in-memory file system fits into it) and noticed that the main Perl process seems to be hogging memory. When a certain limit is exceeded, the transcoding processes suddenly will not have as much memory as they used to, and the re-buffering will start. I have monitored the system with top for a day and copied the output.

file system:

Filesystem                Size      Used Available Use% Mounted on
tmpfs                     1.7G     17.8M      1.7G   1% /
tmpfs                   954.3M         0    954.3M   0% /dev/shm
/dev/mmcblk0p2           29.1G    175.0M     28.3G   1% /mnt/mmcblk0p2

LMS restarted:

Mem: 388012K used, 1566340K free, 14432K shrd, 31648K buff, 163864K cached
CPU:  0.2% usr  0.1% sys  0.0% nic 99.5% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.14 0.08 0.08 2/139 12551
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
12463     1 tc       S     100m  5.2   0  0.3 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff

playing and scanning:

Mem: 453652K used, 1500700K free, 14652K shrd, 31648K buff, 178824K cached
CPU: 25.2% usr  0.5% sys  0.0% nic 74.1% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.37 0.12 0.09 2/144 12573
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
12463     1 tc       S     103m  5.4   1  0.8 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
12572 12463 tc       R    62140  3.1   1 24.5 /usr/local/bin/perl /usr/local/slimserver/scanner.pl --logconfig=/usr/local/slimserver/prefs/log.conf --novideo --onlinelibrary --prefsdir=/usr/local
12571 12569 tc       S     5404  0.2   1  0.1 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
12569 12463 tc       S     3200  0.1   2  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
12570 12569 tc       S     2780  0.1   0  0.2 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -

playing ...:

Mem: 478556K used, 1475796K free, 18952K shrd, 31652K buff, 207356K cached
CPU:  1.3% usr  0.1% sys  0.0% nic 98.4% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.25 0.11 0.03 3/140 13831
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
12463     1 tc       S     140m  7.3   2  0.4 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
13782 13780 tc       S     5404  0.2   2  0.3 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
13780 12463 tc       S     3200  0.1   2  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
13781 13780 tc       S     2780  0.1   2  0.7 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -

out of memory:

Mem: 536364K used, 1417988K free, 21760K shrd, 31656K buff, 210196K cached
CPU:  0.9% usr  0.4% sys  0.0% nic 98.4% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.07 0.08 0.03 1/140 15290
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
12463     1 tc       S     198m 10.3   0  1.0 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
15290 15288 tc       S     5272  0.2   2  0.0 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
15288 12463 tc       S     3200  0.1   0  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
15289 15288 tc       S     2748  0.1   1  0.0 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -

The Perl process is constantly growing from 100m to about 200m. The faad and flac processes run with 5404 and 2780 k of memory consistently all day until the re-buffering starts. I can't make anything of it since the O/S still seems to have plenty of memory to spend on caching, but maybe somebody else can.

[mherger]

> For a few days, now, I have been running a piCorePlayer 6.1.0 with LMS
> on a Pi 4 B with 2G of memory. I mainly use the Pi to transcode AAC
> streams from Tidal. There are no additional plug-ins installed.
>
> I observe the following behaviour: After the LMS is started, it will run
> without complaints for about a day, and then the connected players will
> suddenly start to re-buffer all the time and eventually stop playing
> altogether.

First of all: LMS will grow in memory consumption over the first time of
use, as more dependencies get initialized, data gets cached etc. 200MB
isn't unusual if the DB memory cache is set to hight or max (which is by
default on a 2GB system).

> out of memory:
>
> Mem: 536364K used, 1417988K free, 21760K shrd, 31656K buff, 210196K cached

Don't know why you'd say "out of memory", when almost 1.5GB are still free?

> The Perl process is constantly growing from 100m to about 200m. The faad
> and flac processes run with 5404 and 2780 k of memory consistently all
> day until the re-buffering starts.

Would these be the same processes over a long period of time, are these
new processes for each track played?

--

Michael

[bpa]

The handling of streaming MP4 (i.e. AAC in MPEG-4) is a new chunk of code in LMS. When processing MPEG-4, the audio data atom could be one chunk (if not in streamed format) which has to be broken up into smaller ADTS frames which are then passed to player or faad.

I'm not sure how the Tidal audio data is sent but if the audio data atoms are handled in large chunks - it would explain increased memory usage.

Also the addition of MP4 handling code carries the risk of a memory leak somewhere. However most of the code is not brand new but ported over from other plugins.

[paul-]

Definitely not an out of memory situation from that info. And the cpu load drops when the problem occurs. More likely a network issue. What are the connected clients? Are you trying to play them synchronized? What is the network connectivity of the server and the clients?

[mvordeme]

What I don't understand about the memory: The file system looks like there are 2.6 G of virtual file system in memory, which is not possible, and which would not leave 1.9 G of memory for the processes, so in some way, the numbers need interpretation. I wrote "out of memory" because that is my interpretation. I should have added a "?".

So far, there are no music files on the server, so the library consists of my Tidal favourites only. I don't do much with it, just go to Tidal > My Favourites > Artists on the Controller, pick an artist and album and play. I would expect the process size to increase at these points in time when it caches the album art, but it actually grows all the time while playing.

The server starts new faad and flac processes for every track. For a few seconds, they have an increased CPU usage, and they die a few seconds before the track ends, so there is a buffer somewhere. I actually did not check whether the server still spawns new processes once the players get stuck.

So far, I have not synchronized any players.

It actually took me some time to notice that the problem is on the server because I am mainly listening with my Receiver which has its own set of problems. The server is connected to the router over 100 MBit/s Ethernet, but all players use WiFi. In case of the Receiver, there is even a WiFi repeater involved. In order to minimize my problems with the Receiver, I have moved the WiFi Repeater very close to the Receiver. It shows a signal strength of more than 90 % most of the time. The WiFi router displays connection speeds of > 200 MBit/s between router and repeater and > 50 MBit/s between repeater and Receiver. I can play an entire 24 bit 96 kHz FLAC album over that connection without any problem when the Receiver is in a good mood.

When the problem occurred for the first time, I then moved to the Boom in the kitchen, and it showed the same behaviour. There is no repeater in the Boom's network path, and it has never shown any problems during playback, so far. When the problem persisted, I updated the LMS, and the problem was gone, but the next night it was back. This time, I just restarted the LMS and everything was fine again. That's when I started to monitor the processes on the server.

I don't think that it is a network problem because restarting the LMS consistently helps every time. If you have an idea how I could debug the network, I can of course try. I already gave the piCore real-time priority on the router to make the inbound stream more reliable.

While I am writing, I had to restart the LMS again. This time, it had only reached a size of 154 M, and the processes for transcoding were still looking good memory-wise. The CPU load does drop when the player re-buffers, but the problem appears to be on the server.

re-buffering:

Mem: 491628K used, 1462724K free, 19816K shrd, 31664K buff, 208536K cached
CPU:  0.8% usr  0.2% sys  0.0% nic 98.8% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.07 0.07 0.02 2/143 31491
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
15428     1 tc       S     154m  8.1   0  0.7 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
31420 31418 tc       S     5404  0.2   0  0.0 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
31418 15428 tc       S     3200  0.1   1  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
31419 31418 tc       S     2780  0.1   0  0.1 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -

fixed:

Mem: 394588K used, 1559764K free, 14484K shrd, 31664K buff, 163784K cached
CPU:  1.2% usr  0.1% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.17 0.10 0.03 3/144 31726
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
31625     1 tc       R     100m  5.2   0  0.4 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
31716 31714 tc       S     5404  0.2   0  0.2 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
31714 31625 tc       S     3200  0.1   1  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
31715 31714 tc       S     2780  0.1   0  0.6 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -

You can see that the shared and cached memory sizes drop when the LMS is restarted. Both of these sizes do also grow constantly while the LMS is playing. I'd think that it is okay as long as there is enough free memory to go around, but the fact that they drop instantly when the LMS is restarted could hint at some kind of resource leakage like unreleased file handles or something of that sort. Mind that I am entirely in the dark here and just poking around.

[paul-]

Unless you see Free memory go to zero(or close to it), its not out of memory. You still have well over 1GB of free memory. As for virtual memory, there is compressed swap in ram, which could easily account for that.

Anyway, let me ask to verify.....

Playing local FLAC music to the Boom, will play without issues?

The issue only occurs when playing Tidal AAC streams to the Boom?

Do you see problems playing Tidal to devices other than the Boom? i.e. do you have any squeezelite players that should handle AAC natively?

[mvordeme]

Currently, my FLAC library is on the other server, but I'll copy a few albums and test.

Right now, I only have a Touch which used to be able to play Tidal natively. There have been no problems with that, as far as I know. "Seeking" within a track didn't work any more, last time I checked, but was fixed when connected to the Pi. I can install a software player on a PC and test.

When the problem first occurred on the Receiver, I gave up on Tidal and played web radio, instead. There were no problems with that. Back then, I still suspected network problems like you do, so I started rummaging through my router's logs instead of testing the LMS, which is why I don't know much about other music sources.

[mvordeme]

A few results. When Tidal streaming got stuck again, today, I checked a few things.

• The faad and flac processes had the same memory footprint as usual.
• The LMS would still spawn new faad and flac processes when skipping tracks.
• Local FLAC files (16 bit / 44.1 kHz) would play fine without down-sampling.
• Local FLAC files (24 bit / 96 kHz) would play equally fine with down-sampling (to 48 kHz).
• Deezer Smart Radio would play fine.
• Connecting players to the LMS seemed a bit slow but worked.

Unfortunately, squeezelite would not connect to the server as it had last night, so I didn't test direct streaming from Tidal, but I will, next time. In a vain hope that it would help squeezelite to connect, I restarted the LMS, and Tidal streaming was fixed.

[paul-]

So appears to be a tidal problem. Which version of LMS are you using?

[mherger]

> So appears to be a tidal problem. Which version of LMS are you using?

Is it really buffering, or rather stuttering? The latter is a known
issue with some TIDAL tracks.

--

Michael