Home of the Squeezebox™ & Transporter® network music players.
Page 1 of 2 12 LastLast
Results 1 to 10 of 17
  1. #1
    Senior Member mvordeme's Avatar
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    345

    Memory Leak in Perl Engine on piCorePlayer?

    For a few days, now, I have been running a piCorePlayer 6.1.0 with LMS on a Pi 4 B with 2G of memory. I mainly use the Pi to transcode AAC streams from Tidal. There are no additional plug-ins installed.

    I observe the following behaviour: After the LMS is started, it will run without complaints for about a day, and then the connected players will suddenly start to re-buffer all the time and eventually stop playing altogether.

    I have been looking at CPU and memory consumption on the Pi (I don't quite understand how the in-memory file system fits into it) and noticed that the main Perl process seems to be hogging memory. When a certain limit is exceeded, the transcoding processes suddenly will not have as much memory as they used to, and the re-buffering will start. I have monitored the system with top for a day and copied the output.

    file system:
    Code:
    Filesystem                Size      Used Available Use% Mounted on
    tmpfs                     1.7G     17.8M      1.7G   1% /
    tmpfs                   954.3M         0    954.3M   0% /dev/shm
    /dev/mmcblk0p2           29.1G    175.0M     28.3G   1% /mnt/mmcblk0p2
    LMS restarted:
    Code:
    Mem: 388012K used, 1566340K free, 14432K shrd, 31648K buff, 163864K cached
    CPU:  0.2% usr  0.1% sys  0.0% nic 99.5% idle  0.0% io  0.0% irq  0.0% sirq
    Load average: 0.14 0.08 0.08 2/139 12551
      PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    12463     1 tc       S     100m  5.2   0  0.3 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
    playing and scanning:
    Code:
    Mem: 453652K used, 1500700K free, 14652K shrd, 31648K buff, 178824K cached
    CPU: 25.2% usr  0.5% sys  0.0% nic 74.1% idle  0.0% io  0.0% irq  0.0% sirq
    Load average: 0.37 0.12 0.09 2/144 12573
      PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    12463     1 tc       S     103m  5.4   1  0.8 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
    12572 12463 tc       R    62140  3.1   1 24.5 /usr/local/bin/perl /usr/local/slimserver/scanner.pl --logconfig=/usr/local/slimserver/prefs/log.conf --novideo --onlinelibrary --prefsdir=/usr/local
    12571 12569 tc       S     5404  0.2   1  0.1 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
    12569 12463 tc       S     3200  0.1   2  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
    12570 12569 tc       S     2780  0.1   0  0.2 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -
    playing ...:
    Code:
    Mem: 478556K used, 1475796K free, 18952K shrd, 31652K buff, 207356K cached
    CPU:  1.3% usr  0.1% sys  0.0% nic 98.4% idle  0.0% io  0.0% irq  0.0% sirq
    Load average: 0.25 0.11 0.03 3/140 13831
      PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    12463     1 tc       S     140m  7.3   2  0.4 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
    13782 13780 tc       S     5404  0.2   2  0.3 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
    13780 12463 tc       S     3200  0.1   2  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
    13781 13780 tc       S     2780  0.1   2  0.7 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -
    out of memory:
    Code:
    Mem: 536364K used, 1417988K free, 21760K shrd, 31656K buff, 210196K cached
    CPU:  0.9% usr  0.4% sys  0.0% nic 98.4% idle  0.0% io  0.0% irq  0.0% sirq
    Load average: 0.07 0.08 0.03 1/140 15290
      PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    12463     1 tc       S     198m 10.3   0  1.0 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
    15290 15288 tc       S     5272  0.2   2  0.0 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
    15288 12463 tc       S     3200  0.1   0  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
    15289 15288 tc       S     2748  0.1   1  0.0 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -
    The Perl process is constantly growing from 100m to about 200m. The faad and flac processes run with 5404 and 2780 k of memory consistently all day until the re-buffering starts. I can't make anything of it since the O/S still seems to have plenty of memory to spend on caching, but maybe somebody else can.
    SCALEO Home Server 2105 & piCorePlayer 6.1.0 | Logitech Media Server 8.0.1 | Server Power Control 20120716.103808 | Transporter & Duet & Touch & Boom & Radio | Rotel RC-995 & RMB-100 | Nubert NuVero 140

  2. #2
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,480

    Memory Leak in Perl Engine on piCorePlayer?

    > For a few days, now, I have been running a piCorePlayer 6.1.0 with LMS
    > on a Pi 4 B with 2G of memory. I mainly use the Pi to transcode AAC
    > streams from Tidal. There are no additional plug-ins installed.
    >
    > I observe the following behaviour: After the LMS is started, it will run
    > without complaints for about a day, and then the connected players will
    > suddenly start to re-buffer all the time and eventually stop playing
    > altogether.


    First of all: LMS will grow in memory consumption over the first time of
    use, as more dependencies get initialized, data gets cached etc. 200MB
    isn't unusual if the DB memory cache is set to hight or max (which is by
    default on a 2GB system).

    > out of memory:
    >
    > Mem: 536364K used, 1417988K free, 21760K shrd, 31656K buff, 210196K cached


    Don't know why you'd say "out of memory", when almost 1.5GB are still free?

    > The Perl process is constantly growing from 100m to about 200m. The faad
    > and flac processes run with 5404 and 2780 k of memory consistently all
    > day until the re-buffering starts.


    Would these be the same processes over a long period of time, are these
    new processes for each track played?

    --

    Michael

  3. #3
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    19,619
    The handling of streaming MP4 (i.e. AAC in MPEG-4) is a new chunk of code in LMS. When processing MPEG-4, the audio data atom could be one chunk (if not in streamed format) which has to be broken up into smaller ADTS frames which are then passed to player or faad.

    I'm not sure how the Tidal audio data is sent but if the audio data atoms are handled in large chunks - it would explain increased memory usage.

    Also the addition of MP4 handling code carries the risk of a memory leak somewhere. However most of the code is not brand new but ported over from other plugins.

  4. #4
    Senior Member paul-'s Avatar
    Join Date
    Jan 2013
    Posts
    3,561
    Definitely not an out of memory situation from that info. And the cpu load drops when the problem occurs. More likely a network issue. What are the connected clients? Are you trying to play them synchronized? What is the network connectivity of the server and the clients?
    Last edited by paul-; 2020-11-28 at 06:57.
    piCorePlayer a small player for the Raspberry Pi in RAM.
    Homepage: https://www.picoreplayer.org

    Please donate if you like the piCorePlayer

  5. #5
    Senior Member mvordeme's Avatar
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    345
    What I don't understand about the memory: The file system looks like there are 2.6 G of virtual file system in memory, which is not possible, and which would not leave 1.9 G of memory for the processes, so in some way, the numbers need interpretation. I wrote "out of memory" because that is my interpretation. I should have added a "?".

    So far, there are no music files on the server, so the library consists of my Tidal favourites only. I don't do much with it, just go to Tidal > My Favourites > Artists on the Controller, pick an artist and album and play. I would expect the process size to increase at these points in time when it caches the album art, but it actually grows all the time while playing.

    The server starts new faad and flac processes for every track. For a few seconds, they have an increased CPU usage, and they die a few seconds before the track ends, so there is a buffer somewhere. I actually did not check whether the server still spawns new processes once the players get stuck.

    So far, I have not synchronized any players.

    It actually took me some time to notice that the problem is on the server because I am mainly listening with my Receiver which has its own set of problems. The server is connected to the router over 100 MBit/s Ethernet, but all players use WiFi. In case of the Receiver, there is even a WiFi repeater involved. In order to minimize my problems with the Receiver, I have moved the WiFi Repeater very close to the Receiver. It shows a signal strength of more than 90 % most of the time. The WiFi router displays connection speeds of > 200 MBit/s between router and repeater and > 50 MBit/s between repeater and Receiver. I can play an entire 24 bit 96 kHz FLAC album over that connection without any problem when the Receiver is in a good mood.

    When the problem occurred for the first time, I then moved to the Boom in the kitchen, and it showed the same behaviour. There is no repeater in the Boom's network path, and it has never shown any problems during playback, so far. When the problem persisted, I updated the LMS, and the problem was gone, but the next night it was back. This time, I just restarted the LMS and everything was fine again. That's when I started to monitor the processes on the server.

    I don't think that it is a network problem because restarting the LMS consistently helps every time. If you have an idea how I could debug the network, I can of course try. I already gave the piCore real-time priority on the router to make the inbound stream more reliable.

    While I am writing, I had to restart the LMS again. This time, it had only reached a size of 154 M, and the processes for transcoding were still looking good memory-wise. The CPU load does drop when the player re-buffers, but the problem appears to be on the server.

    re-buffering:
    Code:
    Mem: 491628K used, 1462724K free, 19816K shrd, 31664K buff, 208536K cached
    CPU:  0.8% usr  0.2% sys  0.0% nic 98.8% idle  0.0% io  0.0% irq  0.0% sirq
    Load average: 0.07 0.07 0.02 2/143 31491
      PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    15428     1 tc       S     154m  8.1   0  0.7 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
    31420 31418 tc       S     5404  0.2   0  0.0 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
    31418 15428 tc       S     3200  0.1   1  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
    31419 31418 tc       S     2780  0.1   0  0.1 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -
    fixed:
    Code:
    Mem: 394588K used, 1559764K free, 14484K shrd, 31664K buff, 163784K cached
    CPU:  1.2% usr  0.1% sys  0.0% nic 98.5% idle  0.0% io  0.0% irq  0.0% sirq
    Load average: 0.17 0.10 0.03 3/144 31726
      PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
    31625     1 tc       R     100m  5.2   0  0.4 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon --user tc --group staff
    31716 31714 tc       S     5404  0.2   0  0.2 /usr/local/slimserver/Bin/armhf-linux/flac -cs --totally-silent --compression-level-0 --ignore-chunk-sizes -
    31714 31625 tc       S     3200  0.1   1  0.0 sh -c "/usr/local/slimserver/Bin/armhf-linux/faad" -q -w -f 1 - | "/usr/local/slimserver/Bin/armhf-linux/flac" -cs --totally-silent --compression-lev
    31715 31714 tc       S     2780  0.1   0  0.6 /usr/local/slimserver/Bin/armhf-linux/faad -q -w -f 1 -
    You can see that the shared and cached memory sizes drop when the LMS is restarted. Both of these sizes do also grow constantly while the LMS is playing. I'd think that it is okay as long as there is enough free memory to go around, but the fact that they drop instantly when the LMS is restarted could hint at some kind of resource leakage like unreleased file handles or something of that sort. Mind that I am entirely in the dark here and just poking around.
    Last edited by mvordeme; 2020-11-28 at 08:31.
    SCALEO Home Server 2105 & piCorePlayer 6.1.0 | Logitech Media Server 8.0.1 | Server Power Control 20120716.103808 | Transporter & Duet & Touch & Boom & Radio | Rotel RC-995 & RMB-100 | Nubert NuVero 140

  6. #6
    Senior Member paul-'s Avatar
    Join Date
    Jan 2013
    Posts
    3,561
    Unless you see Free memory go to zero(or close to it), its not out of memory. You still have well over 1GB of free memory. As for virtual memory, there is compressed swap in ram, which could easily account for that.

    Anyway, let me ask to verify.....

    Playing local FLAC music to the Boom, will play without issues?

    The issue only occurs when playing Tidal AAC streams to the Boom?

    Do you see problems playing Tidal to devices other than the Boom? i.e. do you have any squeezelite players that should handle AAC natively?
    piCorePlayer a small player for the Raspberry Pi in RAM.
    Homepage: https://www.picoreplayer.org

    Please donate if you like the piCorePlayer

  7. #7
    Senior Member mvordeme's Avatar
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    345
    Currently, my FLAC library is on the other server, but I'll copy a few albums and test.

    Right now, I only have a Touch which used to be able to play Tidal natively. There have been no problems with that, as far as I know. "Seeking" within a track didn't work any more, last time I checked, but was fixed when connected to the Pi. I can install a software player on a PC and test.

    When the problem first occurred on the Receiver, I gave up on Tidal and played web radio, instead. There were no problems with that. Back then, I still suspected network problems like you do, so I started rummaging through my router's logs instead of testing the LMS, which is why I don't know much about other music sources.
    SCALEO Home Server 2105 & piCorePlayer 6.1.0 | Logitech Media Server 8.0.1 | Server Power Control 20120716.103808 | Transporter & Duet & Touch & Boom & Radio | Rotel RC-995 & RMB-100 | Nubert NuVero 140

  8. #8
    Senior Member mvordeme's Avatar
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    345
    A few results. When Tidal streaming got stuck again, today, I checked a few things.

    • The faad and flac processes had the same memory footprint as usual.
    • The LMS would still spawn new faad and flac processes when skipping tracks.
    • Local FLAC files (16 bit / 44.1 kHz) would play fine without down-sampling.
    • Local FLAC files (24 bit / 96 kHz) would play equally fine with down-sampling (to 48 kHz).
    • Deezer Smart Radio would play fine.
    • Connecting players to the LMS seemed a bit slow but worked.


    Unfortunately, squeezelite would not connect to the server as it had last night, so I didn't test direct streaming from Tidal, but I will, next time. In a vain hope that it would help squeezelite to connect, I restarted the LMS, and Tidal streaming was fixed.
    SCALEO Home Server 2105 & piCorePlayer 6.1.0 | Logitech Media Server 8.0.1 | Server Power Control 20120716.103808 | Transporter & Duet & Touch & Boom & Radio | Rotel RC-995 & RMB-100 | Nubert NuVero 140

  9. #9
    Senior Member paul-'s Avatar
    Join Date
    Jan 2013
    Posts
    3,561
    So appears to be a tidal problem. Which version of LMS are you using?
    piCorePlayer a small player for the Raspberry Pi in RAM.
    Homepage: https://www.picoreplayer.org

    Please donate if you like the piCorePlayer

  10. #10
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,480

    Memory Leak in Perl Engine on piCorePlayer?

    > So appears to be a tidal problem. Which version of LMS are you using?

    Is it really buffering, or rather stuttering? The latter is a known
    issue with some TIDAL tracks.

    --

    Michael

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •