PDA

View Full Version : Synchronization should use fine-grained network clock



tgoldstone
2005-12-27, 07:48
This is regarding bug # 259. What about using IP multicasting to synchronize squeezeboxes? It appears that perl supports it, and it would be a great way to have squeezeboxes synchronize. They would just need to add themselves to a particular multicast group to receive the data stream.
Is anyone familiar with multicast streaming?

Michaelwagner
2005-12-27, 09:48
I am sure I only understand some of the issues involved, but the problem is complex.

At the moment, synchronization occurs at every new song.

The various squeezeboxen (which can be of different generations) all buffer up and wait for a "go" signal. The go signal is probably sent multicast, but I actually don't know that for sure. After the go signal, they all buffer and stream out music to the end of the song, then all buffer up and wait for the next song to start.

Within one song, they rely on the accuracy of their own clocks to keep in sync with each other.

In the software roadmap
http://wiki.slimdevices.com/index.cgi?SoftwareRoadmap
there is a line item for using a network clock for synchronization. I'm not sure what implementation that would take and whether multicast would play a role, but it seems a likely route. The goal, though, would not be to multicast the content (which is very difficult when sending to multiple generations with different buffering and streaming requirements) but merely to multicast a synchronization clock.

Even still, I don't know what accuracy you can get with multicast.

Across a hub, propegation is essentially instantaneous.

Across a switch, there is a very tiny delay and the potential of a longer one if there is interfering traffic on the other side of the switch. I don't know if there is a promise by the protocol to limit the delay to a level where it would be insignificant for this use.

Across a wireless access point? I expect all bets are off. I'm not sure multicast will offer any advantages there ...

mikeb
2005-12-27, 11:44
inline.

Michaelwagner wrote:
> I am sure I only understand some of the issues involved, but the problem
> is complex.
>
> At the moment, synchronization occurs at every new song.
>
> The various squeezeboxen (which can be of different generations) all
> buffer up and wait for a "go" signal. The go signal is probably sent
> multicast, but I actually don't know that for sure. After the go

The signal is sent unicast to each client. No multicast traffic is
being generated.

> signal, they all buffer and stream out music to the end of the song,
> then all buffer up and wait for the next song to start.
>
> Within one song, they rely on the accuracy of their own clocks to keep
> in sync with each other.
>
> In the software roadmap
> http://wiki.slimdevices.com/index.cgi?SoftwareRoadmap
> there is a line item for using a network clock for synchronization. I'm
> not sure what implementation that would take and whether multicast would
> play a role, but it seems a likely route. The goal, though, would not be
> to multicast the content (which is very difficult when sending to
> multiple generations with different buffering and streaming
> requirements) but merely to multicast a synchronization clock.
>
> Even still, I don't know what accuracy you can get with multicast.
>
> Across a hub, propegation is essentially instantaneous.
>
> Across a switch, there is a very tiny delay and the potential of a
> longer one if there is interfering traffic on the other side of the
> switch. I don't know if there is a promise by the protocol to limit the
> delay to a level where it would be insignificant for this use.
>
> Across a wireless access point? I expect all bets are off. I'm not sure
> multicast will offer any advantages there ...

As you've said, the benefit in multicast to the multiple client
synchronization would be minimal (if nonexistent). The benefit to
multicast would be lower bandwidth utilization in a situation where
many clients exist. A heart-beat sent to the server by each client
with a frame number and ntp time synched time stamp would allow the
server to send a signal to the client to advance the track 3 frames to
catch up with the other clients. Of course, if there are only two
players there would need to be logic in the server to decide who was
more accurate on a consistent basis. If done every second, the effect
would be near perfect synchronization. From reading about Sonos, this
network heartbeat is how they synchronize accurately.

Multicast could be used to send the heartbeat to a group address, but
since the logic needs to be controlled at a central location (if 4 of
your squeezeboxes all decide the one fast is the most accurate you get
degraded sound on 4 instead of 1, there needs to be central control of
the logic), it makes the most sense to only unicast it to the server.
This continues to follow the client/server model Slim Devices has
incorporated to date as well.

Now, for the actual music stream, I send 192k VBR MP3 to the
squeezeboxes in my house. I can imagine the remote possibility of say
7 of them, and on a high bitrate song, that comes out at about
1.8mbit. My 54mbit network is hardly going to even hiccup at that.
Personally I prefer the rudimentary network capabilities of a unicast
network that would allow me to easily setup remote routed nodes for my
squeezeboxes, be it either in the house or over the Internet. Keep in
mind I also work on a few hundred node network using Multicast in
multiple capacities for a living, and I don't want Multicast because
of it's complexities, watch out for your standard user..

My suggestion for anything with a HUGE number of squeezeboxes is the
development of a method to bounce streams off another slimserver, so
that 30 squeezeboxes on floors 1 and 2 of the hotel can be on
slimserver 1, and the 30 squeezeboxes on floors 3 and 4 can be on the
second one. While both slimservers synchronize themselves in the
background. But even with 60 squeezeboxes running at 256kbit, you're
still only talking 15mbit. Find APs, switches, and routers that can
understand 802.1p QoS and set the priorities accordingly so you can
differentiate your music over the web browsing on the network and you
should be good.

Hell, with 60 rooms and a 100mbit network you could have every room
listen to music, talk on a VoIP phone, and browse the web with very
little degradation to their web surfing, and absolutely none to their
phone calls and music if CoS is enabled properly.. Since not every
room will do everything at the same time, a cheap 100mbit switched
network could provide the entertainment and communications for a 60
room hotel.. pretty impressive.

Now, if you're scaling a setup with 100s of squeezeboxes I would
suggest you talk to Slim Devices before going forward, I bet they
could dedicate some development time to figuring out how to meet your
needs =) (hey, money talks right?). If Multicast is the only answer,
they may be able to do it in the squeezebox firmware, but just be sure
to leave the rest of us our simple unicast world!

-- Mike

Michaelwagner
2005-12-27, 13:07
A heart-beat sent to the server by each client
with a frame number and ntp time synched time stamp would allow the server to send a signal to the client to advance the track 3 frames to catch up with the other clients.

Personally, that's not a route I would prefer.

Assuming the drift of the timebase (crystal?) in the squeezeboxen is fairly low but there is some individual variation amongst them, a better correction would be, at startup time, establish error in the crystal and correct for it low in the timing chain.

Dropping frames, as you propose, would be audible. Correcting the clock errors could be done much more fine-grained.

Michael

mikeb
2005-12-27, 13:35
Michaelwagner wrote:
> mikeb Wrote:
>> A heart-beat sent to the server by each client
>> with a frame number and ntp time synched time stamp would allow the
>> server to send a signal to the client to advance the track 3 frames to
>> catch up with the other clients.
>
> Personally, that's not a route I would prefer.
>
> Assuming the drift of the timebase (crystal?) in the squeezeboxen is
> fairly low but there is some individual variation amongst them, a
> better correction would be, at startup time, establish error in the
> crystal and correct for it low in the timing chain.
>
> Dropping frames, as you propose, would be audible. Correcting the clock
> errors could be done much more fine-grained.

Slimserver could even instruct the player to play the next ten frames
at a 2% quicker speed to make up the sub-second variation. You're
right dropping would be suboptimal..

-- Mike

> Michael
>
>

stinkingpig
2005-12-28, 15:00
>
> mikeb Wrote:
>> A heart-beat sent to the server by each client
>> with a frame number and ntp time synched time stamp would allow the
>> server to send a signal to the client to advance the track 3 frames to
>> catch up with the other clients.
>
> Personally, that's not a route I would prefer.
>
> Assuming the drift of the timebase (crystal?) in the squeezeboxen is
> fairly low but there is some individual variation amongst them, a
> better correction would be, at startup time, establish error in the
> crystal and correct for it low in the timing chain.
>
> Dropping frames, as you propose, would be audible. Correcting the clock
> errors could be done much more fine-grained.
>
> Michael
>
>

either would be a vast improvement over the current situation though. I
tend to prefer the idea of dropping frames because I don't assume that the
clock is the only source of error. In fact, if I had to guess what causes
the synchronization issues that are seen now, I'd blame the TCP stack.

Anyway, as others have noted, multicast doesn't like wireless.

--
Jack At Monkeynoodle.Org: It's A Scientific Venture...
"Believe what you're told; there'd be chaos if everyone thought for
themselves." -- Top Dog hotdog stand, Berkeley, CA

mikeb
2005-12-28, 15:17
Jack Coates wrote:
>
> Anyway, as others have noted, multicast doesn't like wireless.

There may be issues with layer 2 ethernet multicast over certain
vendors' APs, but the standard layer 3 routed multicast packets (that
I at least was speaking of?) will work fine regardless of the layer 2
device underneath.. Wireless won't make a difference as long as the
device is acting as a bridge and not a router.

-- Mike

stinkingpig
2005-12-28, 16:03
> Jack Coates wrote:
>>
>> Anyway, as others have noted, multicast doesn't like wireless.
>
> There may be issues with layer 2 ethernet multicast over certain
> vendors' APs, but the standard layer 3 routed multicast packets (that
> I at least was speaking of?) will work fine regardless of the layer 2
> device underneath.. Wireless won't make a difference as long as the
> device is acting as a bridge and not a router.
>
> -- Mike

Theoretically, you are right. In practice, multicast does not like
wireless. Granted, it may be able to struggle along well enough to serve
the low-bandwidth needs of a Squeezebox stream, but since there are plenty
of people unable to get that working now, I have my doubts.

On a side note, try typing "Theory" into Google Earth :)

Anyway, multicast (much like Ethernet itself) rather requires that all
nodes in the group are able to operate at a lowest common denominator
speed. WAPs deal with this situation by either breaking the bus down into
bridged zones based on the connection speeds attained by the connected
nodes, or by forcing all nodes to the speed attained by the slowest node.
Therefore an IP multicast group may contain members who are at 1.1 mbps
half-duplex and members who are at 11 mbps half-duplex. A WAP that's
bridging into a real Ethernet is even worse, as now the fast end of the
pool is 100 Mbps full-duplex.

End result is that your entire multicast group bumbles along at 50% of 1.1
mbps in order to cater to the slowest node, who is still complaining
mightily that it's going too fast and that he's missing half the packets.
By doing unicast streams instead, most nodes perform fine and the slow
node is the only one who sucks. Sure, you could write your multicast so
that slow complainers get dropped, but why would SDI go to that effort for
this application?

--
Jack At Monkeynoodle.Org: It's A Scientific Venture...
"Believe what you're told; there'd be chaos if everyone thought for
themselves." -- Top Dog hotdog stand, Berkeley, CA

mikew
2006-01-07, 07:50
The 802.11 MAC requires that all broadcast and multicast packets be sent at rates only in the Basic Service Set, which is generally the 1 or 2Mbps rate, even under 802.11g. Also, these packets do not receive the reliable (ACKed) transmission sevice, so drops go undetected and uncorrected at the link layer. This behavior is at the root of a lot of wireless network behavior oddities. It is sometimes possible to extend the BSS with good driver software, but I have never seen consumer-grade equipment offering this feature. One would certainly not attempt to suggest it to the public at large as a solution.

The true benefit of multicast for synchronization is that if the server sends a single multicast sync packet, all squeezeboxes should receive it simultaneously (aside from propagation variations, about 1ns per foot difference in distance from the host access point, which is negligible for our purposes) This approach eliminates the delay difference in the multiple unicast approach, even if the 802.11 MAC transmits the sync packet at a low rate in an unreliable fashion. If you send one sync packet per second, even if one gets lost there is another one coming real soon.

Don't try to sync all squeezeboxes to a single squeezebox on the network, sync them all to the server. Buffering and streaming do not need to be changed; just signpost to a single common clock and all should be well.

Michaelwagner
2006-01-07, 09:57
You might be able to come up with something like this for SB2s and 3s.

I'm not sure how you would support legacy systems. Their microcode is now effectively closed.

seanadams
2006-01-07, 12:02
Across a hub, propegation is essentially instantaneous.

Across a switch, there is a very tiny delay and the potential of a longer one if there is interfering traffic on the other side of the switch. I don't know if there is a promise by the protocol to limit the delay to a level where it would be insignificant for this use.

Across a wireless access point? I expect all bets are off. I'm not sure multicast will offer any advantages there ...

You need to reconsider your definition of instantaneous. :)

Sound only travels at measely 1.1 feet per millisecond, so even if your synchronization were "only" accurate to within a millisecond, you would still be synced to just a small fraction of the distance between your speakers.

The thing is NTP can even do a lot better than 1ms... how accurate you can get is mostly a function of variance over long periods time in the latency of the network/OS (and this is very low for Squeezeboxes on a wireless lan), NOT the absolute timing between hosts.

i.e. it doesn't matter if one machine is directly connected and the other is on wireless, you can compensate for this with a high degree of accuracy, provided you can make reasonable assumptions about the latency being consistent and symmetric for trasmit vs receive (which lucky for us it is).

When you consider the speed of sound (30,000 uS between rooms) vs the speed of 802.11g (600 uS to go the same distance) it makes sense that even our simple "go now!" signal should work in most cases (usually within 2-5ms). But with NTP (or similar protocol) it could be a lot more accurate, and there are other sync improvements that would be possible once a solid continuous time base is available, eg using multicast to save bandwidth, and drifting instead of rebuffering to align clocks.

mikew
2006-01-08, 20:14
In order to maintain sub-ms clock accuracy, ntp requires significant support from the host operating system, which the squeezeboxen may not have in place. Under linux, there is an entire kernel process devoted to system time, with a corresponding API for slewing the time in a gapless fashion to maintain a highly accurate system time (which is NOT based on the RTC, but the processor clocks while the system is running--the RTC is read/set only on system startup/shutdown). ntp may have impose too much overhead for small embedded devices; not positive though.