Home of the Squeezebox™ & Transporter® network music players.
Results 1 to 10 of 10
  1. #1
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    5,718

    Slim::Networking::Async::HTTP (onStream)

    In my venture to try to fix the RP server connection, I tried a lot of different options to handle the HTTP connection using 1.1, persistent connection, cutting the data in chunks and using different methods of LMS classes. Basically, what was needed is the possibility to "monitor" the streaming socket from within the protocol handler's sysread and re-open it if it fails/stalls.

    Initially, I divided the stream into small chunks so that I would get a full body everytime, being called back with the "onBody" option, then get the next full body and so on (hence I needed persistent connections). Still it was not great and was causing performances issues.

    Then, I found that the class Slim::Networking::Async::HTTP has an "onStream" parameter that causes a callback to happen every time there is data to read on the associated socket, instead of waiting for the whole body. I thought that would be perfect because then I would not have to divide downloads in chunks (I'm not talking about chunk-encoding).

    Unfortunately, the onStream callback is actually invoked everytime there is data on the socket and you *have to* consume it. There does not seem to be a way to pace if down and you cannot block in Perl/LMS. So I was using up all available memory in an interim buffer. If false is returned by the onStream callee, then it stops but the socket is closed, which is not what I wanted either.

    In summary, you either get a callback with "onBody" and get the whole body at once, or one with "onStream" and you get data as it arrives but can't regulate it, or with "onHeaders" but then the class does not listen to the socket anymore (does not close it though) and it's up to you to manually get the body (which is what I'm doing now).

    So the reason for this long post is simply to get opinions, if anybody is interested, and to know if I'm missing something in the way Slim::Networking::Async:HTTP is supposed to work (especially onStream). Or shall we add methods like streamSuspend/streamResume to the Slim::Networking::Async::HTTP class so that the onStream callback can be more usable in case the whole data cannot be consumed immediately. This is probably rethorical discussion, but ...
    Last edited by philippe_44; 2019-12-17 at 22:27.
    LMS 7.7, 7.8 and 7.9 - 5xRadio, 3xBoom, 4xDuet, 1xTouch, 1 SB2. Sonos PLAY:3, PLAY:5, Marantz NR1603, JBL OnBeat, XBoxOne, XBMC, Foobar2000, ShairPortW, JRiver 21, 2xChromecast Audio, Chromecast v1 and v2, , Pi B3, B2, Pi B+, 2xPi A+, Odroid-C1, Odroid-C2, Cubie2, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5

  2. #2
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,422

    Slim::Networking::Async::HTTP (onStream)

    I'm not sure I understand your posting... Are you asking for an
    improvement which might simplify what you've done for the RP plugin, or
    are you looking into improving LMS' code, in order to address the issues
    some people are experiencing with podcasts?

    --

    Michael

  3. #3
    Senior Member
    Join Date
    Apr 2015
    Posts
    687
    Quote Originally Posted by philippe_44 View Post
    In my venture to try to fix the RP server connection, I tried a lot of different options to handle the HTTP connection using 1.1, persistent connection, cutting the data in chunks and using different methods of LMS classes. Basically, what was needed is the possibility to "monitor" the streaming socket from within the protocol handler's sysread and re-open it if it fails/stalls.

    Initially, I divided the stream into small chunks so that I would get a full body everytime, being called back with the "onBody" option, then get the next full body and so on (hence I needed persistent connections). Still it was not great and was causing performances issues.

    Then, I found that the class Slim::Networking::Async::HTTP has an "onStream" parameter that causes a callback to happen every time there is data to read on the associated socket, instead of waiting for the whole body. I thought that would be perfect because then I would not have to divide downloads in chunks (I'm not talking about chunk-encoding).

    Unfortunately, the onStream callback is actually invoked everytime there is data on the socket and you *have to* consume it. There does not seem to be a way to pace if down and you cannot block in Perl/LMS. So I was using up all available memory in an interim buffer. If false is returned by the onStream callee, then it stops but the socket is closed, which is not what I wanted either.

    In summary, you either get a callback with "onBody" and get the whole body at once, or one with "onStream" and you get data as it arrives but can't regulate it, or with "onHeaders" but then the class does not listen to the socket anymore (does not close it though) and it's up to you to manually get the body (which is what I'm doing now).

    So the reason for this long post is simply to get opinions, if anybody is interested, and to know if I'm missing something in the way Slim::Networking::Async:HTTP is supposed to work (especially onStream). Or shall we add methods like streamSuspend/streamResume to the Slim::Networking::Async::HTTP class so that the onStream callback can be more usable in case the whole data cannot be consumed immediately. This is probably rethorical discussion, but ...
    He edited his post. Naughty one!

  4. #4
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    18,605
    I'll look at the is issue but when I needed chunked http 1.1 for BBCiPlayer & PlayHLS plugins (for HLS and DASH) - as I needed a solution that could work with older LMS (I didn't want to force users to upgrade LMS and also minimise my work) I used AnyEvent::HTTP which already some of the functionality you seem to be looking for.

    I'll have a look but IIRC onStream was used by LMS just for scanning remote stream - usually getting the header (e.g. MPEG4) before handing over to a protocol handler for full playing by reopen the stream.

    WHat is the use case for your enhanced onStream ?

  5. #5
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    5,718

    Slim::Networking::Async::HTTP (onStream)

    Quote Originally Posted by mherger View Post
    I'm not sure I understand your posting... Are you asking for an
    improvement which might simplify what you've done for the RP plugin, or
    are you looking into improving LMS' code, in order to address the issues
    some people are experiencing with podcasts?

    --

    Michael
    I could use it in the RP plugin changes I’m proposing but I don’t have to. It was more a general discussion about that change to be added in LMS. It’s a minor change I think and I can make a proposal, but I was wondering if others could see the benefit or if there was already a solution.

    Basically, you could start a big or infinite download and use onStream to receive data as it comes but suspend and resume it when you want, instead of having to use chunks
    Last edited by philippe_44; 2019-12-18 at 10:22.
    LMS 7.7, 7.8 and 7.9 - 5xRadio, 3xBoom, 4xDuet, 1xTouch, 1 SB2. Sonos PLAY:3, PLAY:5, Marantz NR1603, JBL OnBeat, XBoxOne, XBMC, Foobar2000, ShairPortW, JRiver 21, 2xChromecast Audio, Chromecast v1 and v2, , Pi B3, B2, Pi B+, 2xPi A+, Odroid-C1, Odroid-C2, Cubie2, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5

  6. #6
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    5,718
    I looked at the modification in LMS and that's really nothing I think, I've submitted a small PR and the usage is better explained with an example

    Code:
    my $data;
    my $session;
    my $streamUrl;
    
    sub startStreaming {
    	$session = Slim::Networking::Async::HTTP->new;
    	my $request = HTTP::Request->new( GET => $streamUrl ); 
    
    	$session->send_request( {
    			request     => $request,
    			onStream => sub {
    				my ($self, $buffer) = @_;
    				$log->warn("got data ", length $$buffer);
    				$data .= $$buffer;
    				# assumption is that $data is being consumed by another part of the code
    				if ( length $data > 128*1024 ) {
    					$log->warn("too much data, need to pause");
    					$self->suspendStream;
    					Slim::Utils::Timers::setTimer( $self, time() + 10, \&emptyBuffer);
    				}
    				return 1;
    			},	
    			onError  => sub { 
    				$log->error("some error happened $_[1]");
    			},
    	} );
    }
    
    sub emptyBuffer {
    	my $self = shift;
    	$data = '';
    	$log->warn("time to empty buffer and resume");
    	$self->resumeStream;
    }
    Last edited by philippe_44; 2019-12-18 at 20:30.
    LMS 7.7, 7.8 and 7.9 - 5xRadio, 3xBoom, 4xDuet, 1xTouch, 1 SB2. Sonos PLAY:3, PLAY:5, Marantz NR1603, JBL OnBeat, XBoxOne, XBMC, Foobar2000, ShairPortW, JRiver 21, 2xChromecast Audio, Chromecast v1 and v2, , Pi B3, B2, Pi B+, 2xPi A+, Odroid-C1, Odroid-C2, Cubie2, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5

  7. #7
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    5,718
    Quote Originally Posted by bpa View Post
    I'll look at the is issue but when I needed chunked http 1.1 for BBCiPlayer & PlayHLS plugins (for HLS and DASH) - as I needed a solution that could work with older LMS (I didn't want to force users to upgrade LMS and also minimise my work) I used AnyEvent::HTTP which already some of the functionality you seem to be looking for.

    I'll have a look but IIRC onStream was used by LMS just for scanning remote stream - usually getting the header (e.g. MPEG4) before handing over to a protocol handler for full playing by reopen the stream.

    WHat is the use case for your enhanced onStream ?
    Wouldn't that be "onHeaders" instead?

    "onStream" means that you're still delegating the streaming to LMS, but want to handle the data as it comes I think. I would have you that in my various plugins (YouTube, Pluzz, LCI, Canal+) that have a protocol handler that opens its own socket and wants to serve sysread() with that socket, not the socket opended by a HTTP subclass. I'm sure there are other cases where you want to handle data by chunks but not want to use ranges or chunked-encoding (and you can't just wait for the whole body). I did not do that before because I did not realize onStream existed, but even though I fail to understand how to use onStream if you have can't pace it down.
    LMS 7.7, 7.8 and 7.9 - 5xRadio, 3xBoom, 4xDuet, 1xTouch, 1 SB2. Sonos PLAY:3, PLAY:5, Marantz NR1603, JBL OnBeat, XBoxOne, XBMC, Foobar2000, ShairPortW, JRiver 21, 2xChromecast Audio, Chromecast v1 and v2, , Pi B3, B2, Pi B+, 2xPi A+, Odroid-C1, Odroid-C2, Cubie2, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5

  8. #8
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    5,718

    Slim::Networking::Async::HTTP (onStream)

    Quote Originally Posted by karlek View Post
    He edited his post. Naughty one!
    Bad habit, I know
    Last edited by philippe_44; 2019-12-19 at 00:04.
    LMS 7.7, 7.8 and 7.9 - 5xRadio, 3xBoom, 4xDuet, 1xTouch, 1 SB2. Sonos PLAY:3, PLAY:5, Marantz NR1603, JBL OnBeat, XBoxOne, XBMC, Foobar2000, ShairPortW, JRiver 21, 2xChromecast Audio, Chromecast v1 and v2, , Pi B3, B2, Pi B+, 2xPi A+, Odroid-C1, Odroid-C2, Cubie2, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5

  9. #9
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    18,605
    I have wandered through this code before when dealing with MPEG4 - I found it a bit confusing as there seems to be overlaps. The following is my interpretation.

    Quote Originally Posted by philippe_44 View Post
    Wouldn't that be "onHeaders" instead?
    Not the way LMS currently uses it. If type/bitrate cannot be identified (e.g. not wma, ogg, aac or no ICY stuff) - LMS falls into using onStream. See Slim/Utils/Scanner/Remote.pm routine streamAudioData which is set up in Slim/Utils/Scanner/Remote.pm by "onStream => \&streamAudioData". Routine saves 128k of first bytes of a stream so that Scanner can determine bit rate etc. (in case transcoding is necessary) before playing the stream. After 128k has been read - connection is closed. The playing stream re-opens the URL.

    onHeaders is also used in Slim/Utils/Scanner/Remote.pm. When type is identified ( e.g. stream is wma, ogg, aac or has ICY stuff) routine readRemoteHeaders but it is used to read the HTTP headers and URL suffix first and later "scanXXXHeader" routine is used to scan the "body" for bitrate etc. but it doesn't close the stream. So I think it would be used for "STDIN" in transcoding

    "onStream" means that you're still delegating the streaming to LMS, but want to handle the data as it comes I think. I would have you that in my various plugins (YouTube, Pluzz, LCI, Canal+) that have a protocol handler that opens its own socket and wants to serve sysread() with that socket, not the socket opended by a HTTP subclass. I'm sure there are other cases where you want to handle data by chunks but not want to use ranges or chunked-encoding (and you can't just wait for the whole body). I did not do that before because I did not realize onStream existed, but even though I fail to understand how to use onStream if you have can't pace it down.
    I have not looked into your plugins YouTube, Pluzz etc.

    I found I had to be careful not to mix up Async onStream with the registered Protocol Handler onStream

  10. #10
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    5,718
    Quote Originally Posted by bpa View Post
    Not the way LMS currently uses it. If type/bitrate cannot be identified (e.g. not wma, ogg, aac or no ICY stuff) - LMS falls into using onStream. See Slim/Utils/Scanner/Remote.pm routine streamAudioData which is set up in Slim/Utils/Scanner/Remote.pm by "onStream => \&streamAudioData". Routine saves 128k of first bytes of a stream so that Scanner can determine bit rate etc. (in case transcoding is necessary) before playing the stream. After 128k has been read - connection is closed. The playing stream re-opens the URL.

    onHeaders is also used in Slim/Utils/Scanner/Remote.pm. When type is identified ( e.g. stream is wma, ogg, aac or has ICY stuff) routine readRemoteHeaders but it is used to read the HTTP headers and URL suffix first and later "scanXXXHeader" routine is used to scan the "body" for bitrate etc. but it doesn't close the stream. So I think it would be used for "STDIN" in transcoding
    Oh I understand the confusion now. I was thinking HTTP headers when what is needed in some cases are the file's format headers as in the body (like MPEG4 headers or so). So, in that case I can see the use of "onStream" until you have what you need, then you can return false and re-open the url for actual audio data.

    That works well if you need a small amount of data from the stream, but if you really want the whole set, then (unless you save it to a file), it could not be used as download speed far exceeds playback speed.

    The small extension I submitted allows pacing onStream callbacks to whatever the application wants.
    I found I had to be careful not to mix up Async onStream with the registered Protocol Handler onStream
    My main usage would be to use onStream in an Async separated session
    LMS 7.7, 7.8 and 7.9 - 5xRadio, 3xBoom, 4xDuet, 1xTouch, 1 SB2. Sonos PLAY:3, PLAY:5, Marantz NR1603, JBL OnBeat, XBoxOne, XBMC, Foobar2000, ShairPortW, JRiver 21, 2xChromecast Audio, Chromecast v1 and v2, , Pi B3, B2, Pi B+, 2xPi A+, Odroid-C1, Odroid-C2, Cubie2, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •