Radio Paradise hickups still ongoing, solution?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • paul-
    Senior Member
    • Jan 2013
    • 5675

    #16
    Anyone running pCP/squeezelite having this issue, please try to update openssl.

    Open an ssh session and run "pcp-update openssl" then reboot when done. I just upgraded opensll from 1.1.0h -> 1.1.0l and that seemed to make the problem much less. Interesting to note the 1.1.0l is the last version in that series. 1.1.0 is now depreciated.

    For those with older players, turning on proxying will probably help, however your LMS server might need openssl upgraded.
    piCorePlayer a small player for the Raspberry Pi in RAM.
    Homepage: https://www.picoreplayer.org

    Please donate if you like the piCorePlayer

    Comment

    • BenB
      Junior Member
      • Dec 2015
      • 11

      #17
      Originally posted by paul-
      Anyone running pCP/squeezelite having this issue, please try to update openssl.
      Thank you! I've got two pCP players, a Duet and several Booms and this problem has been going on since the beginning of the FLAC streams. I have tried most things to no avail, but upgrading openssl on the two pCP players has solved the issue for me. (My Debian server was already on the latest version).

      Again, a huge thanks to everyone involved in keeping our systems sounding great.

      Edit: Spoke too soon. It's much improved, but still re-buffers every few hours.
      Last edited by BenB; 2019-11-16, 17:53.

      Comment

      • Ron F.
        Senior Member
        • May 2006
        • 746

        #18
        RP FLAC Interactive failing to stay connected...

        Regarding the issue of the FLAC Interactive streams stopping abruptly, and often restarting many seconds later, and going backwards three or four song to play all over again...

        Using Wireshark, I have seen different occurrences in which a running connection will be closed and the LMS RP plugin will have to initiate a new connection to RP:
        1. RP server, without warning, simply ends the connection; I assume this might have something to do with load balancing between the various RP servers.

        2. RP server simply stops sending data to LMS, leaving the client starved for data, LMS eventually closes the connection when this happens, but it can take much longer than 30 seconds before LMS starts a new connection.

        3. RP server sends a new HTTP /1.1 200 OK (audio/flac) packet, I do no understand the reason for this is, but LMS responds by closing the connection; I don't know if that is necessary.

        4. LMS server will send two or three [TCP Window Update] packets, over a period of 3 to 5 seconds, after which no further data is received from the RP server. I have seen as many as 80 seconds go by before the LMS server then sends a [FIN, ACK] packet to abruptly close the existing TCP connection and start over with sending a [SYN] to restart the stream. LMS sends a TCP packet that looks like "HTTP 394 GET /blocks/chan/1/4/280705-280708.flac HTTP/1.0" and we are off and running again - often on a different server than before. What is interesting is that I do then sometimes see packets received from the old server again - I am not sure the old connection was terminated correctly. Question: is the LMS RP plugin attempting to keep multiple TCP connections open with RP servers?

        In all of these cases the result is the same - it is the responsibility of LMS to initiate the new connection, and this usually takes too long to happen. When faced with a closure of the existing TCP connection, it sometimes takes LMS more than 30 seconds to start a new connection with a TCP [SYN] packet sent to Radio Paradise. The buffer runs out during this time, and the music stops. I don't know if this delay between connections is intentional, or a bug either in LMS or possibly the Eth stack?

        I don't know much about load balancing, but I imagine once connected to a server the client should remain there for a long time? Should new clients be sent to the least loaded server, and then remain there? Is there a possibility that the RP servers are being attacked, causing some of these problems?

        My first take on all this is that the RP stream is stopped or dies and restarted too frequently. Before the buffer runs out however, is it possible for LMS to take more aggressive action and close the existing dead connection, and open a new one? Certainly waiting more than a minute is too long. I am wondering if LMS restarted the connection sooner, possibly we would not go backwards several songs?

        Once a new connection is established, the LMS server ACKs a bunch of packets individually for a short while, and then TCP windowing takes over ... and then after a while the problems return.

        Edit: I note that I see my connection to RP being tossed about between three servers, when playing one of their FLAC Interactive streams. I am located in California, and I believe the RP servers are in Florida - I don't know if distance has anything to do with the issue at hand.

        Edit: I am also wondering if we could get the new connection going sooner, we might stay on the same server used in the previous session, and not go backwards several songs, but continue where we left off?
        Last edited by Ron F.; 2019-11-11, 19:20.
        Living Room: SB Touch + DIY PSU > CI Audio VDA.2 DAC + VAC.1 PSU > VRX.1 cables > Emotiva XSP-1 Gen 2 preamp + XPA-DR2 amp > Blue Jeans cables > B&W 804 speakers
        Laptop: System76 Galago + Ubuntu 18.04 + Squeezelite + Epiphany/Material Skin > Emotiva Little Ego DAC > Grado PS500 headphones
        Bedroom: RPi Zero W + Squeezelite > miniBOSS DAC HAT > Bose SoundLink Revolve
        Phone: Pixel 6a + Termux/Squeezelite + Material APK > Senn IE80 earbuds
        Server: System76 Meerkat + Pop!_OS 22.04 + LMS 8.5

        Comment

        • philippe_44
          Senior Member
          • May 2008
          • 9197

          #19
          Originally posted by Ron F.
          Regarding the issue of the FLAC Interactive streams stopping abruptly, and often restarting many seconds later, and going backwards three or four song to play all over again...

          Using Wireshark, I have seen different occurrences in which a running connection will be closed and the LMS RP plugin will have to initiate a new connection to RP:
          1. RP server, without warning, simply ends the connection; I assume this might have something to do with load balancing between the various RP servers.

          2. RP server simply stops sending data to LMS, leaving the client starved for data, LMS eventually closes the connection when this happens, but it can take much longer than 30 seconds before LMS starts a new connection.

          3. RP server sends a new HTTP /1.1 200 OK (audio/flac) packet, I do no understand the reason for this is, but LMS responds by closing the connection; I don't know if that is necessary.

          4. LMS server will send two or three [TCP Window Update] packets, over a period of 3 to 5 seconds, after which no further data is received from the RP server. I have seen as many as 80 seconds go by before the LMS server then sends a [FIN, ACK] packet to abruptly close the existing TCP connection and start over with sending a [SYN] to restart the stream. LMS sends a TCP packet that looks like "HTTP 394 GET /blocks/chan/1/4/280705-280708.flac HTTP/1.0" and we are off and running again - often on a different server than before. What is interesting is that I do then sometimes see packets received from the old server again - I am not sure the old connection was terminated correctly. Question: is the LMS RP plugin attempting to keep multiple TCP connections open with RP servers?

          In all of these cases the result is the same - it is the responsibility of LMS to initiate the new connection, and this usually takes too long to happen. When faced with a closure of the existing TCP connection, it sometimes takes LMS more than 30 seconds to start a new connection with a TCP [SYN] packet sent to Radio Paradise. The buffer runs out during this time, and the music stops. I don't know if this delay between connections is intentional, or a bug either in LMS or possibly the Eth stack?

          I don't know much about load balancing, but I imagine once connected to a server the client should remain there for a long time? Should new clients be sent to the least loaded server, and then remain there? Is there a possibility that the RP servers are being attacked, causing some of these problems?

          My first take on all this is that the RP stream is stopped or dies and restarted too frequently. Before the buffer runs out however, is it possible for LMS to take more aggressive action and close the existing dead connection, and open a new one? Certainly waiting more than a minute is too long. I am wondering if LMS restarted the connection sooner, possibly we would not go backwards several songs?

          Once a new connection is established, the LMS server ACKs a bunch of packets individually for a short while, and then TCP windowing takes over ... and then after a while the problems return.

          Edit: I note that I see my connection to RP being tossed about between three servers, when playing one of their FLAC Interactive streams. I am located in California, and I believe the RP servers are in Florida - I don't know if distance has anything to do with the issue at hand.

          Edit: I am also wondering if we could get the new connection going sooner, we might stay on the same server used in the previous session, and not go backwards several songs, but continue where we left off?
          BTW on that topic, what I was able to establish is that the RP side closed the SSL session and it seems to be on a crypto error (I have to find my logs). It was linked to a pause made by LMS in requesting data. I even went all the way to write a very simple app in C and Perl getting the same URL from RP server and was able to demonstrate that if these few lines of code pause by ~5s from time to time (just sleep), then in most of the cases RP would end up the connection after a few pauses. When no "long" pause is made (say pause <5s), all downloads were successful. What happens as well is that with different openssl clients, the "long pause" effect was more or less problematic (say from 100% to 20% failed downloads)
          LMS 8.2 on Odroid-C4 - SqueezeAMP!, 5xRadio, 5xBoom, 2xDuet, 1xTouch, 1xSB3. Sonos PLAY:3, PLAY:5, Marantz NR1603, Foobar2000, ShairPortW, 2xChromecast Audio, Chromecast v1 and v2, Squeezelite on Pi, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5, RivaArena 1 & 3

          Comment

          • Ron F.
            Senior Member
            • May 2006
            • 746

            #20
            Originally posted by philippe_44
            BTW on that topic, what I was able to establish is that the RP side closed the SSL session and it seems to be on a crypto error (I have to find my logs). It was linked to a pause made by LMS in requesting data. I even went all the way to write a very simple app in C and Perl getting the same URL from RP server and was able to demonstrate that if these few lines of code pause by ~5s from time to time (just sleep), then in most of the cases RP would end up the connection after a few pauses. When no "long" pause is made (say pause <5s), all downloads were successful. What happens as well is that with different openssl clients, the "long pause" effect was more or less problematic (say from 100% to 20% failed downloads)
            OK, but then I do not understand why this issue does not occur with the Regular FLAC stream, only the Interactive stream?
            Living Room: SB Touch + DIY PSU > CI Audio VDA.2 DAC + VAC.1 PSU > VRX.1 cables > Emotiva XSP-1 Gen 2 preamp + XPA-DR2 amp > Blue Jeans cables > B&W 804 speakers
            Laptop: System76 Galago + Ubuntu 18.04 + Squeezelite + Epiphany/Material Skin > Emotiva Little Ego DAC > Grado PS500 headphones
            Bedroom: RPi Zero W + Squeezelite > miniBOSS DAC HAT > Bose SoundLink Revolve
            Phone: Pixel 6a + Termux/Squeezelite + Material APK > Senn IE80 earbuds
            Server: System76 Meerkat + Pop!_OS 22.04 + LMS 8.5

            Comment

            • philippe_44
              Senior Member
              • May 2008
              • 9197

              #21
              Originally posted by Ron F.
              OK, but then I do not understand why this issue does not occur with the Regular FLAC stream, only the Interactive stream?
              I can only guess at that time that the regular stream are infinite streams, so the HTTP server might behave differently when there is no content-length.

              I also tried to think of a resume solution, but the way LMS and plugins are architectured, it seems very difficult
              LMS 8.2 on Odroid-C4 - SqueezeAMP!, 5xRadio, 5xBoom, 2xDuet, 1xTouch, 1xSB3. Sonos PLAY:3, PLAY:5, Marantz NR1603, Foobar2000, ShairPortW, 2xChromecast Audio, Chromecast v1 and v2, Squeezelite on Pi, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5, RivaArena 1 & 3

              Comment

              • Ron F.
                Senior Member
                • May 2006
                • 746

                #22
                Originally posted by philippe_44
                I can only guess at that time that the regular stream are infinite streams, so the HTTP server might behave differently when there is no content-length.

                I also tried to think of a resume solution, but the way LMS and plugins are architectured, it seems very difficult
                Question: is LMS using the system's installed version of openssl, or is it using a copy that it brings with it?
                Living Room: SB Touch + DIY PSU > CI Audio VDA.2 DAC + VAC.1 PSU > VRX.1 cables > Emotiva XSP-1 Gen 2 preamp + XPA-DR2 amp > Blue Jeans cables > B&W 804 speakers
                Laptop: System76 Galago + Ubuntu 18.04 + Squeezelite + Epiphany/Material Skin > Emotiva Little Ego DAC > Grado PS500 headphones
                Bedroom: RPi Zero W + Squeezelite > miniBOSS DAC HAT > Bose SoundLink Revolve
                Phone: Pixel 6a + Termux/Squeezelite + Material APK > Senn IE80 earbuds
                Server: System76 Meerkat + Pop!_OS 22.04 + LMS 8.5

                Comment

                • philippe_44
                  Senior Member
                  • May 2008
                  • 9197

                  #23
                  Originally posted by Ron F.
                  Question: is LMS using the system's installed version of openssl, or is it using a copy that it brings with it?
                  AFAIK, it's using the system's openSSL
                  LMS 8.2 on Odroid-C4 - SqueezeAMP!, 5xRadio, 5xBoom, 2xDuet, 1xTouch, 1xSB3. Sonos PLAY:3, PLAY:5, Marantz NR1603, Foobar2000, ShairPortW, 2xChromecast Audio, Chromecast v1 and v2, Squeezelite on Pi, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5, RivaArena 1 & 3

                  Comment

                  • Ron F.
                    Senior Member
                    • May 2006
                    • 746

                    #24
                    Originally posted by philippe_44
                    AFAIK, it's using the system's openSSL
                    OK - thank you. My home server is running Ubuntu 18.04.3. I re-installed openssl-1.1.1, released last Sept 11, 2018, to no effect - the issue persisted. I then installed openssl-1.1.1d, the latest stable version released this past Sept 10, 2019 - the issue persists. I need to stream one of the RP FLAC Interactive streams much longer to know whether there is any improvement, but I have already experienced a long delay (~80 seconds,) followed by a restart that set me back several songs again.

                    I suppose I could build the openssl master branch, which is going to be 1.1.1e when released, and try that, but I have no confidence it will make any difference.

                    Nothing I have done has ever made any difference whatsoever. If there is a crypto error occurring in openssl, then people smarter than me are going to have to deal with it.

                    Edit: regardless of the error, I just wish the occasional long delays could be eliminated - that by itself would take care of the issue for the most part, from a user's perspective.
                    Last edited by Ron F.; 2019-11-12, 05:20.
                    Living Room: SB Touch + DIY PSU > CI Audio VDA.2 DAC + VAC.1 PSU > VRX.1 cables > Emotiva XSP-1 Gen 2 preamp + XPA-DR2 amp > Blue Jeans cables > B&W 804 speakers
                    Laptop: System76 Galago + Ubuntu 18.04 + Squeezelite + Epiphany/Material Skin > Emotiva Little Ego DAC > Grado PS500 headphones
                    Bedroom: RPi Zero W + Squeezelite > miniBOSS DAC HAT > Bose SoundLink Revolve
                    Phone: Pixel 6a + Termux/Squeezelite + Material APK > Senn IE80 earbuds
                    Server: System76 Meerkat + Pop!_OS 22.04 + LMS 8.5

                    Comment

                    • slartibartfast
                      Senior Member
                      • Jan 2010
                      • 13173

                      #25
                      Originally posted by Ron F.
                      OK - thank you. My home server is running Ubuntu 18.04.3. I re-installed openssl-1.1.1, released last Sept 11, 2018, to no effect - the issue persisted. I then installed openssl-1.1.1d, the latest stable version released this past Sept 10, 2019 - the issue persists. I need to stream one of the RP FLAC Interactive streams much longer to know whether there is any improvement, but I have already experienced a long delay (~80 seconds,) followed by a restart that set me back several songs again.

                      I suppose I could build the openssl master branch, which is going to be 1.1.1e when released, and try that, but I have no confidence it will make any difference.

                      Nothing I have done has ever made any difference whatsoever. If there is a crypto error occurring in openssl, then people smarter than me are going to have to deal with it.

                      Edit: regardless of the error, I just wish the occasional long delays could be eliminated - that by itself would take care of the issue for the most part, from a user's perspective.
                      Does anyone know why these interruptions started happening? For me the interactive stream seemed to play properly until a few months ago when I first mentioned it in this thread. It seems odd that it would suddenly break. Did something change?

                      Sent from my SM-G900F using Tapatalk
                      Living Room: Touch or Squeezelite (Pi3B) > Topping E30 > Audiolab 8000A > Monitor Audio S5 + BK200-XLS DF
                      Bedroom: Radio
                      Bathroom: Radio

                      Comment

                      • philippe_44
                        Senior Member
                        • May 2008
                        • 9197

                        #26
                        Originally posted by slartibartfast
                        Does anyone know why these interruptions started happening? For me the interactive stream seemed to play properly until a few months ago when I first mentioned it in this thread. It seems odd that it would suddenly break. Did something change?

                        Sent from my SM-G900F using Tapatalk
                        I don’t think anything changed on our side. But as said above, a few month ago when I made these test programs, I had exactly the same problems as soon as I introduced the ‘pause’ and these tests have no link at all with LMS. Something might have changed on RP’s plumbing side. So far the best would have been to avoid the pauses but I did not find a convenient way to do that. Nothing bitrate based was reliable
                        LMS 8.2 on Odroid-C4 - SqueezeAMP!, 5xRadio, 5xBoom, 2xDuet, 1xTouch, 1xSB3. Sonos PLAY:3, PLAY:5, Marantz NR1603, Foobar2000, ShairPortW, 2xChromecast Audio, Chromecast v1 and v2, Squeezelite on Pi, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5, RivaArena 1 & 3

                        Comment

                        • d6jg
                          Senior Member
                          • Feb 2011
                          • 8652

                          #27
                          You can't use this with Michael's plugin but you you can force the regular streams to use a particular server rather than the load balancer by replacing "stream" in the URL with "icy-X"

                          This from an old RP Forum post
                          "Wasn't aware of any issues with the load balancer. But your solution on that is indeed correct. icy-4 and icy-5 will directly call our servers in London, icy-6 or icy-8 in Texas, icy-7 in Virginia."

                          There are also icy-1, 2 & 3 but I do not know where they are

                          Click image for larger version

Name:	2019-11-12 12_21_43-Command Prompt - nslookup.png
Views:	1
Size:	13.3 KB
ID:	1567351
                          Jim



                          VB2.4 storage QNAP TS419p (NFS)
                          Living Room Joggler & Pi4/Khadas -> Onkyo TXNR686 -> Celestion F20s
                          Office Joggler & Pi3 -> Onkyo CRN775 -> Celestion F10s
                          Dining Room SB Radio
                          Bedroom (Bedside) Pi Zero+DAC ->ToppingTP21 ->AKG Headphones
                          Bedroom (TV) & Bathroom SB Touch ->Denon AVR ->Mordaunt Short M10s + Kef ceiling speakers
                          Guest Room Joggler > Denon RCFN8 -> Wharfedale Modus Cubes

                          Comment

                          • paul-
                            Senior Member
                            • Jan 2013
                            • 5675

                            #28
                            Originally posted by Ron F.
                            Question: is LMS using the system's installed version of openssl, or is it using a copy that it brings with it?
                            LMS uses perl IO::Socket::SSL, which then uses the system openssl for connections (I wonder if the perl module version makes a difference), however LMS is only used if you are using proxied streaming. If the player supports SSL, then the players SSL is what matters. If the player does not support SSL, then LMS will automatically proxy. However, I'm not sure how SSL plays into this. I was doing a test over the weekend, and the connections to RP were not secured. The music was streaming over plain HTTP. (Both in the interactive and continuous stream)

                            When playing the continuous stream, data is fed to the player at a nice controlled pace from the server. When the interactive stream is played, the server tries to send the whole file all at once, so the file fills the player and network receive buffers. I wonder what happens if you increase your player buffer to hold more data.
                            piCorePlayer a small player for the Raspberry Pi in RAM.
                            Homepage: https://www.picoreplayer.org

                            Please donate if you like the piCorePlayer

                            Comment

                            • philippe_44
                              Senior Member
                              • May 2008
                              • 9197

                              #29
                              Originally posted by paul-
                              When playing the continuous stream, data is fed to the player at a nice controlled pace from the server. When the interactive stream is played, the server tries to send the whole file all at once, so the file fills the player and network receive buffers. I wonder what happens if you increase your player buffer to hold more data.
                              That makes sense. The continuous is treated as a live stream so data is produced piece by piece where the interactive are files (blocks of 2 to 5 tracks) so it can be grabbed as fast as possible. That corroborates what I’ve seen.
                              LMS 8.2 on Odroid-C4 - SqueezeAMP!, 5xRadio, 5xBoom, 2xDuet, 1xTouch, 1xSB3. Sonos PLAY:3, PLAY:5, Marantz NR1603, Foobar2000, ShairPortW, 2xChromecast Audio, Chromecast v1 and v2, Squeezelite on Pi, Yamaha WX-010, AppleTV 4, Airport Express, GGMM E5, RivaArena 1 & 3

                              Comment

                              • Ron F.
                                Senior Member
                                • May 2006
                                • 746

                                #30
                                Originally posted by paul-
                                LMS uses perl IO::Socket::SSL, which then uses the system openssl for connections (I wonder if the perl module version makes a difference), however LMS is only used if you are using proxied streaming. If the player supports SSL, then the players SSL is what matters. If the player does not support SSL, then LMS will automatically proxy. However, I'm not sure how SSL plays into this. I was doing a test over the weekend, and the connections to RP were not secured. The music was streaming over plain HTTP. (Both in the interactive and continuous stream)

                                When playing the continuous stream, data is fed to the player at a nice controlled pace from the server. When the interactive stream is played, the server tries to send the whole file all at once, so the file fills the player and network receive buffers. I wonder what happens if you increase your player buffer to hold more data.
                                Thank you for this. I do not understand how SSL works; I have some reading to do. From my use of Wireshark this past weekend and watching RP FLAC packets on their way to LMS, I did not see anything related to SSL - but I did not understand the implication of that at the time. Playing with different releases of openssl was not going to make any difference, and indeed - it did not.

                                I have tried playing with the LMS network settings in the past, and setting "Radio Station Buffer Seconds" to it's max: 30, has not make any difference. Enabling "Proxied Streaming" never made much if any difference either. Does Proxied Streaming cause LMS to create an additional buffer on the server? Regardless, if the interactive stream is attempting to download an entire file all at once, then this sounds like an ftp download! If every time Bill does a station announcement represents the start of a new file, then we could be talking about a GByte of data. Ouch - I am surprised this works as well as it does! There are all kinds of timeouts that might occur causing this to fail, and the RP server attempts to restart the download ... from the beginning.

                                When streaming Interactive, wouldn't the RP plugin then need to offload all this data into a temporary file? A buffer in memory big enough to solve the issue would be impractical. The TCP Rx buffer is going to fill and I imagine timeout if the plugin does not read out the data and store it somewhere. Is the LMS RP plugin relying on the network stack to control the connection, to close the connection for instance when there is a problem?

                                If LSM RP plugin is only reading sufficient data out of the network stack to fill a 30 second buffer, then I doubt that buffer size is going to matter if the RP server is trying to shove a file's worth of data to LMS. It also seems likely that increasing the size of the TCP Rx Buffer in the LMS server network stack is not going to help either. I imagine then the only solution is that the RP plugin needs to be able to store a LOT of data into a temporary FIFO file on the server's HDD.

                                While writing this, I have experienced two long delays, followed by restarts in the RP Main FLAC Interactive stream.
                                Last edited by Ron F.; 2019-11-12, 18:12.
                                Living Room: SB Touch + DIY PSU > CI Audio VDA.2 DAC + VAC.1 PSU > VRX.1 cables > Emotiva XSP-1 Gen 2 preamp + XPA-DR2 amp > Blue Jeans cables > B&W 804 speakers
                                Laptop: System76 Galago + Ubuntu 18.04 + Squeezelite + Epiphany/Material Skin > Emotiva Little Ego DAC > Grado PS500 headphones
                                Bedroom: RPi Zero W + Squeezelite > miniBOSS DAC HAT > Bose SoundLink Revolve
                                Phone: Pixel 6a + Termux/Squeezelite + Material APK > Senn IE80 earbuds
                                Server: System76 Meerkat + Pop!_OS 22.04 + LMS 8.5

                                Comment

                                Working...