Home of the Squeezebox™ & Transporter® network music players.
Page 1 of 3 123 LastLast
Results 1 to 10 of 25
  1. #1
    Junior Member
    Join Date
    Sep 2019
    Posts
    18

    help me unravel deep networking problem with classic squeezebox

    Bear with me, this is messy and weird.

    Basic problem: stuttering; and it's worse with some songs than others. Only the actual SB Classics stutter - I just set up squeezelite on a piCorePlayer and it doesn't stutter. On the Classics, the stutter is worse on some units than others. Donn't prejudge the problem: it's not network bandwidth, and the Classic that's on wifi stutters the least.

    I want to keep the Classics going as long as I can, because my wife is used to using the remote and the menus on the display.

    Now for the many weird aspects. First of all, all the Classics are on rev 81 of the firmware. Why so old? Because I'm not using LMS. I'm using a custom server that understands the protocol used in version 81, and for complicated reasons I'll skip over unless asked, that aspect won't change. I'm very wedded to the custom server, and my wife is very wedded to the completely custom way it displays menus and other info on the classic. PiCore and Squeezelite won't be an option for her, at least until I have a way to display a facsimile of those menus.

    Second, when I ran my server on a raspberry pi, there were no stutters, in about 10 years of use. I recently moved it to a NUC, a much faster system; that's when the stuttering started - very rarely for some players and some some songs, but horribly often for one particular song and a couple of players.

    Since it's my own server, I can instrument it however I like, so I measured the timing for reading chunks from the music file, and how long it took to send() each chunk over the network.

    Reading the file isn't a problem: Reading 64k at a time from an internal SSD disk, the NUC is spending between 28 to maybe 60 microseconds to load each buffer, with very occasional spikes as high as 1.3 milliseconds. The average is around 40 microseconds. Reading is fine.

    Writing over the socket is where the problem is. When I use the piCore player, I get exactly what I expect: when a song first starts loading to the piCore, socket writes happen quickly, with a peak of maybe 3 milliseconds to write each 64k, usually rather less. But the squeezelite fills its internal buffer, and reads less often, so the TCP stream backs up as expected. Socket writes in the server start blocking, waiting for the TCP stream to unjam. So I'll get a write time like 1093 milliseconds or 786 milliseconds, followed by something much faster for the next 64k chunk, like 108 microseconds. This is just what I'd expect - as squeezelite uses up the buffer, it gets more interested in doing TCP reads, which unblocks the server to quickly ship enough to top off squeezelite again. All is perfect.

    But when I move that same ethernet cable to a classic (running firmware 81), the socket writes are a completely different story.

    There is an initial inrush of buffers, though it's slower, at around 24ms each. (Except for some songs where it's slower). But thereafter, there's no alternating between slow and fast send times. Every send is in the hundreds of milliseconds. It's as if the Classic isn't bothering to read from the socket very often, so it stalls on its own buffer being empty. And it's simple to test that theory: if I just pause the music, the server keeps sending buffers, so when I unpause there's a period of unstuttering play.

    I could believe the classic just isn't getting around to reading the socket often enough, but when the server ran on a raspberry pi this never happened.

    And the weirdest aspect: this is sensitive to which song is playing. George Winston's Carol of the Bells doesn't stutter, or at least I haven't caught it at it. The Strawbs' Benedictus stutters like mad, towards the end of the song and on every repeat. (Both are flac). CSNY's Carry On starts stuttering only partway through.

    All I can come up with is that there's some difference between the TCP stack of a raspberry pi and this NUC (linux mint) and somehow the classic is sensitive to it (and picore isn't), but that seems like a fringe theory. Note that I've tried this with different buffers sizes, and with nagle enabled and disabled. Still stutters.

    I know the next step is to move the server back to a pi and see if the problem goes away again. Or move all the classics up to a more recent firmware and see if something changed.

    Does anyone have any insight into what could cause this?
    Last edited by ScottAM; 2019-09-18 at 07:08. Reason: details

  2. #2
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    17,866
    Quote Originally Posted by ScottAM View Post
    Basic problem: stuttering; and it's worse with some songs than others. Only the actual SB Classics stutter - I just set up squeezelite on a piCorePlayer and it doesn't stutter. On the Classics, the stutter is worse on some units than others. Donn't prejudge the problem: it's not network bandwidth, and the Classic that's on wifi stutters the least.
    Start with basics
    What OS
    What version of LMS ?
    What processor etc basic system details.
    What format is the audio being sent to Classic - codec, sample size, sample rate
    Are the players synced ?
    What is the maximum 100% packet rate you can get from Classic using network test ?

  3. #3
    Senior Member
    Join Date
    Apr 2005
    Location
    UK/London
    Posts
    1,849
    FYI bpa ... no LMS ... this is a homegrown alternative


    A quirk of different frame sizes?

    Risk firmware update on one Classic and see if it still works with homegrown server and if yes ... maybe it will play without stuttering
    Paul Webster
    http://dabdig.blogspot.com
    Author Radio France (FIP etc) plugin

  4. #4
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    17,866
    Quote Originally Posted by Paul Webster View Post
    FYI bpa ... no LMS ... this is a homegrown alternative
    Audio format of source is still important - if say original file is AAC and is transcoded into WAV - it will be a bigger load and smaller time in buffer.

    What does the Classic say in its STMx responses about buffer fullness ?
    Has the player been started before enough of internal buffer has been filled ? (I.e. what threshold has been set)

  5. #5
    Junior Member
    Join Date
    Sep 2019
    Posts
    18
    Quote Originally Posted by Paul Webster View Post
    FYI bpa ... no LMS ... this is a homegrown alternative


    A quirk of different frame sizes?

    Risk firmware update on one Classic and see if it still works with homegrown server and if yes ... maybe it will play without stuttering
    This is likely where I'm headed. I already know my server doesn't work with firmware 131 - but the issues are to do with UDP discovery (how that works changed sometime after version 81). With a little reverse engineering I can probably get my server to do everything the new way. By bypassing UDP discovery, I got squeezelite to work with it, and that emulates the newer protocol after all, so I know my server is doing most protocol things right.

    I hadn't remembered that the response to STMx commands report buffer fullness. What I'd expect, based on behaviour, is that the buffer is generally starving, but it would be interesting to have the classic's word on that.

    Thanks.

  6. #6
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    17,866
    Quote Originally Posted by ScottAM View Post
    I hadn't remembered that the response to STMx commands report buffer fullness. What I'd expect, based on behaviour, is that the buffer is generally starving, but it would be interesting to have the classic's word on that.
    Buffer fuillness is reported in nearly all STMx so you can tell if server is slow to replenish but STMu is actual underrun - do you get any STMu's ?

  7. #7
    Junior Member
    Join Date
    Sep 2019
    Posts
    18
    Quote Originally Posted by bpa View Post
    Buffer fuillness is reported in nearly all STMx so you can tell if server is slow to replenish but STMu is actual underrun - do you get any STMu's ?
    ...

    .....

    There are no STMu's. Not even at the end of the song, which according to the comments in my code is expected to happen.

    But the ratio of fullness to total, from STAT messages, calculated this way:
    unsigned int outputBufferFullness = uintFrom4(data + 33);
    unsigned int outputBufferTotal = uintFrom4(data + 29);
    clientBufferFullness_ = (float)outputBufferFullness / outputBufferTotal;

    ...accurately tells the true story - near the song beginning it gets as high as 0.6 or so, and then slowly drifts down. When it reports 0.0 I hear stuttering. If I pause the music and let a bunch of sends happen, and then unpause, the ratio is over .9 - and stays there for a bit but then drifts back down to 0, and stutters again.

    I'm having trouble believing that the thread in my server that's delivering buffers is lagging - it's using the real time scheduler with a priority of 97, and nothing else runs that high. CPU use is trivially low. Nagle is off. Nothing else but the trivial file read in that loop is blocking. send() is just returning very late, and only when it's pointed at a Classic.

    What the heck?

    Edit: to make it more interesting, on the last run I watched the numbers throughout. The classic took a bunch of buffers initially and the fullness got above .99. And it stayed there for a long while, consistently above .99, which is what I'd expect. But then it abruptly started drifting down, and fairly quickly, even though sends were still happening - though the send time was gradually creeping up to 1.5s.

    I would swear this is just a bad squeezebox - I just got it - but I have another squeezebox that's worked for years and now it stutters too - though nowhere near as badly or often. Were Classics ever known to have really weird implementations of TCP?
    Last edited by ScottAM; 2019-09-18 at 11:11. Reason: more surprises

  8. #8
    Senior Member
    Join Date
    Apr 2005
    Location
    UK/London
    Posts
    1,849
    Can you watch it with Wireshark or similar?
    Paul Webster
    http://dabdig.blogspot.com
    Author Radio France (FIP etc) plugin

  9. #9
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    17,866
    Quote Originally Posted by ScottAM View Post
    ...accurately tells the true story - near the song beginning it gets as high as 0.6 or so, and then slowly drifts down. When it reports 0.0 I hear stuttering. If I pause the music and let a bunch of sends happen, and then unpause, the ratio is over .9 - and stays there for a bit but then drifts back down to 0, and stutters again.
    Why are you letting the buffer gets so empty ?
    When the buffer gets low - the server should be topping it up.

    I'm having trouble believing that the thread in my server that's delivering buffers is lagging - it's using the real time scheduler with a priority of 97, and nothing else runs that high. CPU use is trivially low. Nagle is off. Nothing else but the trivial file read in that loop is blocking. send() is just returning very late, and only when it's pointed at a Classic.
    So is it TCP window edge and acknowledgements ?
    Without a trace or somehting hard info - all we have is your interpretation.
    AFAICT you still have said what is the audio format you are sending to player ? PCM, , Flac, MP3, ?


    Edit: to make it more interesting, on the last run I watched the numbers throughout. The classic took a bunch of buffers initially and the fullness got above .99. And it stayed there for a long while, consistently above .99, which is what I'd expect. But then it abruptly started drifting down, and fairly quickly, even though sends were still happening - though the send time was gradually creeping up to 1.5s.
    It would be nice to have a trace so data exchange and relative timings can be examined.

    I would swear this is just a bad squeezebox - I just got it - but I have another squeezebox that's worked for years and now it stutters too - though nowhere near as badly or often. Were Classics ever known to have really weird implementations of TCP?
    No TCP implementation is good but you need to be careful of TCP window edge moving - IIRC the edge is only moved once audio is played not when data is received by player.

  10. #10
    Junior Member
    Join Date
    Sep 2019
    Posts
    18
    Quote Originally Posted by Paul Webster View Post
    Can you watch it with Wireshark or similar?
    I installed wireshark but I'm having problems getting it to report what I want. I'll keep banging on that.

    I recorded and graphed the time it takes to send each buffer (16K in this case) against the buffer-fullness of the squeezebox. The send time jumps all over the place, starting out very fast and then staggers between a very long time and fairly fast, presumably because the squeezebox is frequently quenching the flow until it wants more. The x axis is milliseconds, the y is nanoseconds.
    Name:  jaggedSendTimesms-ns.png
Views: 135
Size:  45.7 KB

    Smoothing 10 values on each side, a pattern of fast-to-start, then leveling out is visible.
    Name:  smoothedns-ms.png
Views: 134
Size:  15.0 KB
    The main point here is that things don't get dramatically worse at 19000ms. Which is interesting, because here's the graph of the squeezebox's report of the buffer fullness, measured at intervals (hence the steps):
    Name:  fullness.png
Views: 133
Size:  15.3 KB

    It looks to me as if the squeezebox is in fact reading the TCP stream more or less steadily (once the buffer is full) - but at a certain point is just stops adding what it reads to the squeezebox's play buffer in any sort of timely fashion, and what it has just gets used up in a more or less linear fashion. Before that point, the server is clearly having no problem keeping the squeezebox fed. Once it starts stuttering, the fullness bounces around between 0.0 and 0.04, if that.

    These graphs are fairly typical, but I've seen at least one case where the problem, whatever it is, happens right near the start, and the buffer gets maybe 0.07 full and then rapidly slides down to 0. When the stuttering starts varies, in other words, but near the beginning and towards the end seem the typical cases.

    Insight welcome. This is slowly driving me mad. Edit: the steady decline in the fullness was not the end of the song. I killed the test when the stuttering started, and the server hadn't announced "ok, I sent everything in this song" yet.
    Last edited by ScottAM; 2019-09-19 at 18:02.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •