Thoughts about further synchronization improvements

[awy]

A number of other bugs and requests in the forums have dealt with issues other than maintaining synchronization when playing with multiple synchronized players. This thread is to discuss their relative merits and solicit input on priorities, problems, solutions, etc.

1. Gapless
   
   The current sync design uses start-all-players-one-after-another-as-quickly-as-possible at the start of each track as a way of achieving synchronized playback. The start (unpause) commands are sent out serially and really need to reach each player within a window of around 10ms to get good sync playback and sometimes this does not happen. Some proposed firmware changes (unpauseAt) may be able to improve on that.
   
   But waiting for all players to be ready at the start of each new track almost inevitably introduces an audible delay between tracks. In fact the delay would be even greater if a larger syncBufferThreshold were used, and reliable sync-startup without an immediate underrun really does require a much larger threshold than the current default of 128 bytes. We are talking of around 100-250ms here and so this is only really relevant to gapless playback.
   
   If the sync-maintenance algorithms are sufficiently reliable then it no longer becomes necessary to wait for all players to have finished playing out the previous track before starting the next one. For an SB1, however, a change in format (mp3, wav, etc.) still requires playout to complete and only then open a new stream.
   
   The current design is reflected in code scattered over many parts of the server and will require careful work to unwind it.
   
   
2. Adding a player to an existing sync-group without restart
   
   The main thing needed for this is maintenance of a relationship between stream bytes (at frame boundaries) and play-time represented at the same frame-boundary. This is needed in any case if SliMP3s or SB1s are part of the sync-group but would not otherwise be required for SB2+ players. Also, because of the larger buffers in the SB2+ players, the range of the stream for which this data is needed is much larger. The cost of maintaining this data is neither trivial nor excessive. It seems a pity to have to maintain it when not required – that is, when there is no intention to dynamically add a player to an existing sync-group mid-track without restart. Nonetheless, it is doable.
   
   There are a number of other issues which raise problems with respect to this functionality.
   
   
   1. stream-format
      
      The current design use a lowest-common-supported model for the stream format across the set of synced players. A single source stream is then used to feed the data stream to each player. This is clearly a problem if one tries to add a player to a sync-group where the new player does not support the current stream format.
      
      An option might be to abandon the lowest-common-supported format model, and use multiple streams with (potentially) different formats, but there would be a danger that a transcoded stream might not have a tractable relationship with respect to playing time - for example, if the transcoding involved a change between 48000 and 44100 frame rates. Also, multiple parallel transcoding operations could become very CPU intensive.
      
      
   2. buffered-data
      
      Modern players (SB2/3, Transporter) buffer a significant amount of encoded audio data, possibly representing more than 5 minutes of audio playback, and easily more than 3 minutes (for 128kbps MP3). If a player were simply spliced into the stream at the current point, then it could be an unacceptably long time before the new player actually started playback.
      
      For local files, where it can be sure that rewinding or reopening the stream will result in the same byte-stream, then this is an option. For local data with a transcoded stream there is the danger that the CPU-intensive transcoding might take too long to get to the same point in the stream as is currently playing on the existing players. For remote streams then rewinding is simply not an option; restarting may be an option for live streams but one needs a way to know that it is live.
      
   
   
   
3. Syncing a player with a standalone player w/o restart
   
   This is really just a slightly more extreme version of adding a client to an existing sync-group.
   
   In the case of a locally-streamed file then it should simply be possible to grab the file (and any waiting chunks) from Slim::Web:HTTP.
   
   In the case that the existing player is streaming directly from a remote source then there seems to be little possibility of syncing-in another player.
   
   The question then arises as to what to do when:
   1. adding player to existing sync-group mid-track is not possible;
   2. adding player will result in substantial delay before new player starts.
   
   
   Options are:
   1. restart current track across all players in group (current behaviour);
   2. add new player only at start of next track (or after substantial delay).

Alan.

[cjk32]

Thoughts about further synchronization improvements

Alan,

Is there a write up anywhere of all the issues with synched players.
I've got lots of 'Can't you just...?' type questions, but know that it's
never that simple, and would like to understand the problem better first.

Regards,

Chris

[awy]

Is there a write up anywhere of all the issues with synched players.
I've got lots of 'Can't you just...?' type questions, but know that it's
never that simple, and would like to understand the problem better first.

Not that I know of. I have posted a couple of threads on the issues, here (see https://forums.slimdevices.com/node/34854) and in the General Discussion forum (see https://forums.slimdevices.com/node/34414). Also see https://forums.slimdevices.com/node/35718 You can also search the bug database.

It would be great however if you could post your list here. I have done a lot of work on the re-sync stuff and it would be good to cover off as many issues as I reasonably can.

Alan.

[cjk32]

Thoughts about further synchronization improvements

awy wrote:
> Christopher Key;209930 Wrote:
>
>> Is there a write up anywhere of all the issues with synched players.
>> I've got lots of 'Can't you just...?' type questions, but know that
>> it's
>> never that simple, and would like to understand the problem better
>> first.
>>
>
> Not that I know of. I have posted a couple of threads on the issues,
> here (see 34535) and in the General Discussion forum
> (see 34133). Also see 35427 You can
> also search the bug database.
>
> It would be great however if you could post your list here. I have done
> a lot of work on the re-sync stuff and it would be good to cover off as
> many issues as I reasonably can.
>
> Alan.
>
>
I've had a quick read, will go back for a more thorough read later.
I've only a peripheral knowledge of the workings of Slimserver, and the
following are really only idle musings.

My thoughts only relate to SB2/3 as they'd need a firmware update to
work. Essentially, what I was thinking of was to put some of the onus
onto the Squeezeboxes themselves for staying in sync. Each would keep a
local clock synchronised to the servers clock through SNTP / NTP to suit
the required accuracy.

The strm commands sent to the clients would include a startAt time.
This start time would typically be sufficiently into the future to allow
clients to fill their buffer. However, if a client failed to start in
time e.g. late receipt of the strm message, it should skip an
appropriate amount into the stream. In order to support gapless
playback, it might be an idea to include a flag forcing the Squeezebox
to 'snap' to the end of the previous track.

This obviously doesn't address drift. Is there anyway that the
Squeezebox can deal with this itself, keeping its output in line the
server clock? One option would be to insert or delete samples during
periods of silence to keep in line, although this could cause problems
for non-pcm data. I know that AC3 and DTS typically have padding
between frames but don't know what the effect would be of adjusting the
length of this padding. Is there any way that the Squeezebox can
dynamically adjust its output rate, eg 44101Hz, or is it limited to
discrete frequencies.

Finally, is there any merit in adding a startOffset parameter. This
would be intended to allow a client to synchronise in the middle of a
stream. It would also prevent the necessity of 'transcoding' whole
albums files to extract the necessary section, and hence allow FF / RW
within them.


Regards,

Chris

[awy]

The strm commands sent to the clients would include a startAt time. This start time would typically be sufficiently into the future to allow clients to fill their buffer. However, if a client failed to start in time e.g. late receipt of the strm message, it should skip an appropriate amount into the stream.

This is exactly what the new firmware and SS code will do, at least on current thinking. It seems to work well.

In order to support gapless playback, it might be an idea to include a flag forcing the Squeezebox to 'snap' to the end of the previous track.

I do not think that you can support gapless with the resync-at-track-start mechanism. The pre-start buffering period needs to be too long, especially in wireless environments where network congestion can be a significant factor. I admit that gapless is lower on my priority list than other types of synced playback but I have not forgotten it.

This obviously doesn't address drift. Is there anyway that the Squeezebox can deal with this itself, keeping its output in line the server clock? One option would be to insert or delete samples during periods of silence to keep in line, although this could cause problems for non-pcm data. I know that AC3 and DTS typically have padding between frames but don't know what the effect would be of adjusting the length of this padding. Is there any way that the Squeezebox can dynamically adjust its output rate, eg 44101Hz, or is it limited to discrete frequencies.

Inserting and deleting samples is easy enough. The new firmware would do this in after decoding the encoded audio so that it can do it with discrete samples. Making the player itself responsible for dealing with drift is an interesting idea. The current code I'm working on has the server do all this work, sending skip-ahead and pause-for messages to the players as necessary. I have been careful to make the code also work with SliMP3s and SB1s where there is little alternative to having the server manage things but this does not necessarily rule out pushing more responsibility onto the players that are capable of handling it.

Finally, is there any merit in adding a startOffset parameter. This would be intended to allow a client to synchronise in the middle of a stream. It would also prevent the necessity of 'transcoding' whole albums files to extract the necessary section, and hence allow FF / RW within them.

Again, an interesting idea. I can imaging however that the network bandwidth and processing on the player would be an issue in getting to 80% of a long track. I suspect that doing this on the server would be more practical. Remember, the players are supposed to be 'slim' devices.

Alan.

[cjk32]

Thoughts about further synchronization improvements

awy wrote:
> Christopher Key;210070 Wrote:
>
>> The strm commands sent to the clients would include a startAt time.
>> This start time would typically be sufficiently into the future to
>> allow clients to fill their buffer. However, if a client failed to
>> start in time e.g. late receipt of the strm message, it should skip an
>> appropriate amount into the stream.
>>
>
> This is exactly what the new firmware and SS code will do, at least on
> current thinking. It seems to work well.
>
>
Is this is principle, or in actuality? My understanding was that the
players would all start buffering, then receive an 'unpause at'
command. I was just wondering if this would be a little cleaner. As
you say though, this isn't much use for the SB1 though, which would
presumably still need to start off on an unpause.
>> In order to support gapless playback, it might be an idea to include a
>> flag forcing the Squeezebox to 'snap' to the end of the previous track.
>>
>
> I do not think that you can support gapless with the
> resync-at-track-start mechanism. The pre-start buffering period needs
> to be too long, especially in wireless environments where network
> congestion can be a significant factor. I admit that gapless is lower
> on my priority list than other types of synced playback but I have not
> forgotten it.
>
>
I agree, gapless is probably a pretty low priority. How do you mean,
'The pre-start buffering period needs to be too long'? Why, when you
are playing back via synced players, can you not start streaming to them
as early as required. I was hoping that if the players could keep track
of drift themselves, then you wouldn't need to explicitly resync at the
start of each track. Each track has a precise length, and each player
has a synchronised time (via ntp / sntp), hence, in theory, they should
all be able to stay in sync. In practice, drift will mean that they
won't play back the tracks at precisely the correct rate, and would
hence get further and further apart without the resyncing at the start
of tracks, hence the desire to have the players correct for this themselves.


>> This obviously doesn't address drift. Is there anyway that the
>> Squeezebox can deal with this itself, keeping its output in line the
>> server clock? One option would be to insert or delete samples during
>> periods of silence to keep in line, although this could cause problems
>> for non-pcm data. I know that AC3 and DTS typically have padding
>> between frames but don't know what the effect would be of adjusting the
>> length of this padding. Is there any way that the Squeezebox can
>> dynamically adjust its output rate, eg 44101Hz, or is it limited to
>> discrete frequencies.
>>
>
> Inserting and deleting samples is easy enough. The new firmware would
> do this in after decoding the encoded audio so that it can do it with
> discrete samples. Making the player itself responsible for dealing with
> drift is an interesting idea. The current code I'm working on has the
> server do all this work, sending skip-ahead and pause-for messages to
> the players as necessary. I have been careful to make the code also
> work with SliMP3s and SB1s where there is little alternative to having
> the server manage things but this does not necessarily rule out pushing
> more responsibility onto the players that are capable of handling it.
>
>
Having the player do it makes sense for a few reasons. This is where
the 'problem' is occurring, and fixing it here makes conceptual sense.
The drift rate is presumably more simply calculated in the client, given
that it has direct access to the relevant information. It should also
be able to do a more transparent job of fixing it. If it has a local
estimate of drift, it can perform much smaller, more frequent
adjustments, adding or removing single samples as required. It might
also be easier to make this more transparent, as it could add samples
during periods of silence, or during zero crossings . This would have
to be done carefully though to avoid corrupting non PCM streams. I'm
still thinking that SlimServer should support the concept of non PCM
audio, flagging it as such during scanning, so that things like replay
gain, volume adjustment and crossfade can be disabled.
>> Finally, is there any merit in adding a startOffset parameter. This
>> would be intended to allow a client to synchronise in the middle of a
>> stream. It would also prevent the necessity of 'transcoding' whole
>> albums files to extract the necessary section, and hence allow FF / RW
>> within them.
>>
>
> Again, an interesting idea. I can imaging however that the network
> bandwidth and processing on the player would be an issue in getting to
> 80% of a long track. I suspect that doing this on the server would be
> more practical. Remember, the players are supposed to be 'slim'
> devices.
>
Is this any differing to scanning through a track on the Squeezebox.
This only works for tracks that are not being transcoded on the server,
so I assume that the Squeezebox is issuing further HTTP requests to
acquire the relevant sections of the file. If it's easier on the
server, then that's clearly the best place to put it. It just look like
implementing something on the client side required about ten times the
code on the server side to achieve the same thing.

Chris

[awy]

>> Finally, is there any merit in adding a startOffset parameter. This
>> would be intended to allow a client to synchronise in the middle of a
>> stream. It would also prevent the necessity of 'transcoding' whole
>> albums files to extract the necessary section, and hence allow FF / RW
>> within them.
>>
>
> Again, an interesting idea. I can imaging however that the network
> bandwidth and processing on the player would be an issue in getting to
> 80% of a long track. I suspect that doing this on the server would be
> more practical. Remember, the players are supposed to be 'slim'
> devices.
>
Is this any differing to scanning through a track on the Squeezebox.
This only works for tracks that are not being transcoded on the server,
so I assume that the Squeezebox is issuing further HTTP requests to
acquire the relevant sections of the file. If it's easier on the
server, then that's clearly the best place to put it. It just look like
implementing something on the client side required about ten times the
code on the server side to achieve the same thing.

A bit of a delayed response ...

I think that this does not really work for remote streams and, as you suggest, is not much help with server-transcoded ones either.

Overall, I find your ideas of pushing some of the responsibility for maintaining sync out to the players quite attractive. I suspect however, that the set of exceptions to the circumstances under which they would be of much help may argue against this. I am about to embark on another round of design and I'll certainly keep these ideas in mind.

Alan.

[wcattey]

Bit the bullet: Deal with streams and transcodes.

I've been watching the sync issue for some time myself.

I think that if one is going to go to the effort of making sync work then the design should account for things like transcoded and streaming content.

The unpause thing is just not workable. (I found that softsqueeze and a squeezebox would end up up to FIVE SECONDS out of sync with each other using this mechanism.)

Instead, synchronized time, and a periodic "play at time XX" sync heartbeat for members of the sync group seems best. This would neatly cover the case of adding a new member to the sync group, as well as drift. Streamed content would work, whereas that's only synced once, at the start-up of the stream.

It might be possible to keep a table of latencies for members of the sync group to help decide how best to deal with drift. Alas, the latencies would change if there were client-side transcoding being done.

[andyg]

Thoughts about further synchronization improvements

On Sep 6, 2007, at 12:31 PM, wcattey wrote:

>
> I've been watching the sync issue for some time myself.
>
> I think that if one is going to go to the effort of making sync work
> then the design should account for things like transcoded and
> streaming
> content.
>
> The unpause thing is just not workable. (I found that softsqueeze and
> a squeezebox would end up up to FIVE SECONDS out of sync with each
> other using this mechanism.)
>
> Instead, synchronized time, and a periodic "play at time XX" sync
> heartbeat for members of the sync group seems best. This would
> neatly
> cover the case of adding a new member to the sync group, as well as
> drift. Streamed content would work, whereas that's only synced once,
> at the start-up of the stream.
>
> It might be possible to keep a table of latencies for members of the
> sync group to help decide how best to deal with drift. Alas, the
> latencies would change if there were client-side transcoding being
> done.

Have you tried SlimServer 7 lately? This is exactly how it works.

[wcattey]

I'll download the nightly and try it out straight away!

[plympton]

On Sep 6, 2007, at 12:31 PM, wcattey wrote:

> It might be possible to keep a table of latencies for members of the
> sync group to help decide how best to deal with drift. Alas, the
> latencies would change if there were client-side transcoding being
> done.

Have you tried SlimServer 7 lately? This is exactly how it works.

Is there a modifiable table in SS 7? I have some clients that are always 200ms, or 768ms, or 2.3s, off (say), so can I tell my SB to have a 200ms delay over these others (actually a -200ms "undelay" or my other clients, but that's not possible with the current time/space relationship of Newtonian physics)

-Dan

[andyg]

Thoughts about further synchronization improvements

On Sep 6, 2007, at 3:08 PM, plympton wrote:

>
> Andy Grundman;225584 Wrote:
>> On Sep 6, 2007, at 12:31 PM, wcattey wrote:
>>
>>> It might be possible to keep a table of latencies for members of the
>>> sync group to help decide how best to deal with drift. Alas, the
>>> latencies would change if there were client-side transcoding being
>>> done.
>>
>> Have you tried SlimServer 7 lately? This is exactly how it works.
>
> Is there a modifiable table in SS 7? I have some clients that are
> always 200ms, or 768ms, or 2.3s, off (say), so can I tell my SB to
> have
> a 200ms delay over these others (actually a -200ms "undelay" or my
> other
> clients, but that's not possible with the current time/space
> relationship of Newtonian physics)

Alan and I talked about this as a way to get Softsqueeze in sync on
an OS that doesn't report the time properly. So it's not in there
yet but Alan may add it.

You shouldn't have this issue with hardware players.

[wcattey]

I'd advocate for adding it. Legacy SB2 SB1 and that other brand of players that emulates SB1 might benefit from my being able to drop in a latency by hand.

[andyg]

Thoughts about further synchronization improvements

On Sep 6, 2007, at 3:40 PM, wcattey wrote:

>
> I'd advocate for adding it. Legacy SB2 SB1 and that other brand of
> players that emulates SB1 might benefit from my being able to drop
> in a
> latency by hand.

I'll let Alan comment more on the issue.

[plympton]

On Sep 6, 2007, at 3:08 PM, plympton wrote:

>
> Andy Grundman;225584 Wrote:
>> On Sep 6, 2007, at 12:31 PM, wcattey wrote:
>>
>>> It might be possible to keep a table of latencies for members of the
>>> sync group to help decide how best to deal with drift. Alas, the
>>> latencies would change if there were client-side transcoding being
>>> done.
>>
>> Have you tried SlimServer 7 lately? This is exactly how it works.
>
> Is there a modifiable table in SS 7? I have some clients that are
> always 200ms, or 768ms, or 2.3s, off (say), so can I tell my SB to
> have
> a 200ms delay over these others (actually a -200ms "undelay" or my
> other
> clients, but that's not possible with the current time/space
> relationship of Newtonian physics)

Alan and I talked about this as a way to get Softsqueeze in sync on
an OS that doesn't report the time properly. So it's not in there
yet but Alan may add it.

You shouldn't have this issue with hardware players.

My hardware is taking a stream and feeding it to madplay on Linux - it's reporting itself as a Slimp3 to Slimserver, though.

It's a pretty far corner case, however...

-Dan