Home of the Squeezebox™ & Transporter® network music players.
Page 17 of 18 FirstFirst ... 715161718 LastLast
Results 161 to 170 of 172
  1. #161
    Senior Member mvordeme's Avatar
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    402
    That is a fact which makes investigating the problem difficult. When there are 500 open file handles, it is difficult to keep track of what is happening.

    I understand from the code that with a memory configuration of high, the process should not use up more than 500 MB of memory. Is that correct?
    SCALEO Home Server 2105 & piCorePlayer 6.1.0 | Logitech Media Server 8.2.0 | Server Power Control 20120716.103808 | Transporter & Duet & Touch & Boom & Radio | Rotel RC-995 & RMB-100 | Nubert NuVero 140

  2. #162
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    7,099
    Quote Originally Posted by mvordeme View Post
    That is a fact which makes investigating the problem difficult. When there are 500 open file handles, it is difficult to keep track of what is happening.

    I understand from the code that with a memory configuration of high, the process should not use up more than 500 MB of memory. Is that correct?
    No, AFAIK, 500 is the number of cached tracks. After that, they should be closed but I'm not sure that works as expected. I've reduced that to 10 and traced DESTROY it's very unclear to me when it happens, except when you terminate the slimserver process. The code is in /usr/share/perl5/Slim (from memory).

    In Slim/Utils/Scanner/Remote.pm you'll find the part that get the headers and the parsers are in Slim/Formats. Each format can offer a parse() method, here is the flac one for example

    Code:
    sub parseStream {
    	my ( $class, $dataref, $args, $length ) = @_;
    
    	$args->{_scanbuf} .= $$dataref;
    	return -1 if length $args->{_scanbuf} < 32*1024;
    
    	my $fh = File::Temp->new();
    	$fh->write($args->{_scanbuf});
    	$fh->seek(0, 0);
    
    	my $info = Audio::Scan->scan_fh( flac => $fh )->{info};
    	return undef unless $info->{samplerate};
    
    	$info->{fh} = $fh;
    	$info->{avg_bitrate} = int(8*1000 * ($length - $info->{audio_offset}) / $info->{song_length_ms}) if $info->{song_length_ms} && $length;
    
    	return $info;
    }
    Now, if you look in Remote.pm, the code calling flac parser is

    Code:
    sub parseFlacHeader {
    	my ( $http, $dataref, $track, $args ) = @_;
    
    	Slim::Formats->loadTagFormatForType('flc');
    	my $formatClass = Slim::Formats->classForFormat('flc');
    	my $info = $formatClass->parseStream($dataref, $args, $http->response->content_length);
    	
    	return 1 if ref $info ne 'HASH' && $info;
    
    	$track->content_type('flc');
    	
    	if ($info) {
    		# don't set audio_offset & audio_size as these are not reliable here
    		$track->samplerate( $info->{samplerate} );
    		$track->samplesize( $info->{bits_per_sample} );
    		$track->channels( $info->{channels} );	
    		# if no bitrate, swag it to allow seek
    		my $bitrate = $info->{avg_bitrate} || int($info->{samplerate} * $info->{bits_per_sample} * $info->{channels} * 0.6);	
    		Slim::Music::Info::setBitrate( $track, $bitrate );
    		Slim::Music::Info::setDuration( $track, $info->{song_length_ms} / 1000 );
    
    		# we have valid header, means there will be no alignment unless we seek
    		$track->processors('flc', Slim::Schema::RemoteTrack::INITIAL_BLOCK_ONSEEK, \&Slim::Formats::FLAC::initiateFrameAlign);
    	} else {
    		# if we don't have an header, need to always process
    		$track->processors('flc', Slim::Schema::RemoteTrack::INITIAL_BLOCK_ALWAYS, \&Slim::Formats::FLAC::initiateFrameAlign );
    	}	
    
    	# All done
    	$args->{cb}->( $track, undef, @{$args->{pt} || []} );
    	return 0;
    }
    So there is no reference kept to $info, so I would think that it goes out of scope and Perl removes the {fh} object which closes & deletes the temp file. Obviously that does not happen. I can explicitly close the fh here, but there are cases where they are supposed to last as long as the $track object is kept. Normally, there is a DESTROY in Slim/Schema/RemoteTrack.pm that explicitly closes it
    Code:
    sub DESTROY {
    	my $self = shift;
    
    	if (my $fh = $self->initial_block_fh) {
    		$fh->close;
    		unlink $fh;
    	}
    	
    	$self->SUPER::DESTROY();
    }
    But it's that part that seems not called (even with 10 tracks max in the cache) - unless slimserver.pl terminates which proves the DESTROY is at least at the right place

    I've tried that basic code
    Code:
    use strict;
    
    use File::Temp;
    
    sub parseStream {
    	my $fh = File::Temp->new();
    	$fh->write("something");
    	$fh->seek(0, 0);
    	my $info;
    	$info->{fh} = $fh;
    	$info->{key} = "key";
    	print "$info->{key} ", $info->{fh}->filename, "\n";
    	<>;
    }
    
    print "hello\n";
    parseStream();
    <>;
    And when $info goes out of scope of parseStream, the file is destroyed, so I don't know...
    Last edited by philippe_44; 2021-03-22 at 19:33.
    LMS 8.2 on Odroid-C4 - SqueezeAMP!, 5xRadio, 5xBoom, 2xDuet, 1xTouch, 1xSB3. Sonos PLAY:3, PLAY:5, Marantz NR1603, Foobar2000, ShairPortW, 2xChromecast Audio, Chromecast v1 and v2, Squeezelite on Pi, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5, RivaArena 1 & 3

  3. #163
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,543

    Memory Leak in Perl Engine on piCorePlayer?

    > No, AFAIK, 500 is the number of cached tracks. After that, they should
    > be closed but I'm not sure that works as expected. I've reduced that to
    > 10 and traced DESTROY it's very unclear to me when it happens, except
    > when you terminate the slimserver process. The code is in
    > /usr/share/perl5/Slim (from memory)


    We discussed some concerns about this back on August 27 on Gitter. I'd
    have loved to send you a link, but I didn't find a way to do so :-)

  4. #164
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    7,099

    Memory Leak in Perl Engine on piCorePlayer?

    Quote Originally Posted by mherger View Post
    &gt; No, AFAIK, 500 is the number of cached tracks. After that, they should
    &gt; be closed but I'm not sure that works as expected. I've reduced that to
    &gt; 10 and traced DESTROY it's very unclear to me when it happens, except
    &gt; when you terminate the slimserver process. The code is in
    &gt; /usr/share/perl5/Slim (from memory)


    We discussed some concerns about this back on August 27 on Gitter. I'd
    have loved to send you a link, but I didn't find a way to do so :-)
    I do remember very well the discussion. I have made a few additional tests and in fact the DESTROY happens as expected when the cache is set to 10 but it need to be created with 10, not modified later at runtime as I was doing before.

    Still, that does not solve the issue of open handles. I guess we can tolerate a large number of files in temp, but we canít have them opened. According to what I have read, File:.Temp unlinks the file when it is closed, so I would need to do something different like having a temp file that I only unlink in DESTROY but is closed unless the header is needed ... crap, that means I cannot rely on File:Temp for the random path/name generation and I need to memorize the name and reopen/close it all the time. I was hoping that the cache of $tracks would be decorrelated from playlist length and that LMS would re-read the ones that went out of LRU cache. Un
    LMS 8.2 on Odroid-C4 - SqueezeAMP!, 5xRadio, 5xBoom, 2xDuet, 1xTouch, 1xSB3. Sonos PLAY:3, PLAY:5, Marantz NR1603, Foobar2000, ShairPortW, 2xChromecast Audio, Chromecast v1 and v2, Squeezelite on Pi, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5, RivaArena 1 & 3

  5. #165
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    7,099

    Memory Leak in Perl Engine on piCorePlayer?

    Quote Originally Posted by philippe_44 View Post
    I do remember very well the discussion. I have made a few additional tests and in fact the DESTROY happens as expected when the cache is set to 10 but it need to be created with 10, not modified later at runtime as I was doing before.

    Still, that does not solve the issue of open handles. I guess we can tolerate a large number of files in temp, but we canít have them opened. According to what I have read, File:.Temp unlinks the file when it is closed, so I would need to do something different like having a temp file that I only unlink in DESTROY but is closed unless the header is needed ... crap, that means I cannot rely on File:Temp for the random path/name generation and I need to memorize the name and reopen/close it all the time. I was hoping that the cache of $tracks would be decorrelated from playlist length and that LMS would re-read the ones that went out of LRU cache. Un
    I know you donít see edits so Iíll answer my own post. It seems that the new of File::Temp has an UNLINK option to prevent deletion on close. Iíve looked at the code and itís likely easy to reopen/re-read the content where I need it as it is very located in the code.

    So likely good news and a PR in the coming days. Do you want that in 8.2 or in 8.1.2? I think itís a fix so Iíd rather put it in 8.1.2, but Iíll do as you like.
    LMS 8.2 on Odroid-C4 - SqueezeAMP!, 5xRadio, 5xBoom, 2xDuet, 1xTouch, 1xSB3. Sonos PLAY:3, PLAY:5, Marantz NR1603, Foobar2000, ShairPortW, 2xChromecast Audio, Chromecast v1 and v2, Squeezelite on Pi, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5, RivaArena 1 & 3

  6. #166
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    7,099

    Memory Leak in Perl Engine on piCorePlayer?

    See https://github.com/Logitech/slimserver/pull/553 if you want to give it a try. I'm checking files with
    Code:
    lsof -r +D /tmp -a  -p &lt;your_pid&gt;
    And I can confirm that nothing is left open with AAC and with FLAC files are removed automatically. At exit, everything is cleaned up. I've not verified that at very long term (500 or 2000 files) they start to be removed one by one but I think they will be.
    Last edited by philippe_44; 2021-03-23 at 14:04.
    LMS 8.2 on Odroid-C4 - SqueezeAMP!, 5xRadio, 5xBoom, 2xDuet, 1xTouch, 1xSB3. Sonos PLAY:3, PLAY:5, Marantz NR1603, Foobar2000, ShairPortW, 2xChromecast Audio, Chromecast v1 and v2, Squeezelite on Pi, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5, RivaArena 1 & 3

  7. #167
    Senior Member mvordeme's Avatar
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    402
    I installed tonight's development build and will give it a spin. Thanks a lot for your help.
    SCALEO Home Server 2105 & piCorePlayer 6.1.0 | Logitech Media Server 8.2.0 | Server Power Control 20120716.103808 | Transporter & Duet & Touch & Boom & Radio | Rotel RC-995 & RMB-100 | Nubert NuVero 140

  8. #168
    Senior Member mvordeme's Avatar
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    402
    I can confirm that no open file handles are left behind, only the number of files in /tmp/ is growing. I'll keep an eye on the files and on the memory consumption.
    Quote Originally Posted by philippe_44 View Post
    I'm checking files with
    Code:
    lsof -r +D /tmp -a  -p &lt;your_pid&gt;
    lsof on the BusyBox doesn't seem to care at all which options I supply.
    SCALEO Home Server 2105 & piCorePlayer 6.1.0 | Logitech Media Server 8.2.0 | Server Power Control 20120716.103808 | Transporter & Duet & Touch & Boom & Radio | Rotel RC-995 & RMB-100 | Nubert NuVero 140

  9. #169
    Senior Member mvordeme's Avatar
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    402
    Today, the server is dead again. The difference is that now, there are no open file handles. Memory consumption is at 617 MB, and there are 958 temp files around:
    Code:
    Mem: 1911440K used, 42912K free, 127048K shrd, 5900K buff, 1157512K cached
    CPU:  0.2% usr  0.0% sys  0.0% nic 99.6% idle  0.0% io  0.0% irq  0.0% sirq
    Load average: 0.02 0.02 0.00 2/138 407
      PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
     2706     1 tc       S     617m 32.3   0  0.2 {slimserver.pl} /usr/bin/perl /usr/local/slimserver/slimserver.pl --daemon
    
    ~$ sudo lsof -p 2706 | grep 2706 | cut -f 3 | grep /tmp/ | wc -l
    1
    ~$ find /tmp -type f -maxdepth 1 -user tc -perm 600 | wc -l
    958
    The Transporter keeps switching off and on again and does not play a single note. Since it lasted a little longer last time, I am going to reboot the server itself.
    SCALEO Home Server 2105 & piCorePlayer 6.1.0 | Logitech Media Server 8.2.0 | Server Power Control 20120716.103808 | Transporter & Duet & Touch & Boom & Radio | Rotel RC-995 & RMB-100 | Nubert NuVero 140

  10. #170
    Senior Member mvordeme's Avatar
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    402
    This time, a large number of temp files had been left behind after I had shut down the LMS.
    Code:
    ~$ find /tmp -type f -maxdepth 1 -user tc -perm 600 | wc -l
    612
    Edit: Seems like the current version of LMS does not write any temp files at all.
    Last edited by mvordeme; 2021-04-17 at 16:26.
    SCALEO Home Server 2105 & piCorePlayer 6.1.0 | Logitech Media Server 8.2.0 | Server Power Control 20120716.103808 | Transporter & Duet & Touch & Boom & Radio | Rotel RC-995 & RMB-100 | Nubert NuVero 140

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •