PDA

View Full Version : synchronization problems - and getting worse



dcote
2009-07-22, 14:34
after spending half the day researching this forum, the bugzilla database and the forum over at qnap.com and not finding anything conclusive, i give up and post my own thread now.

first, my symptoms:
-> on the odd occasion i _can_ get my players synced, it holds for anywhere between 5 to 50 minutes, then breaks
-> usually, sync just slowly drifts apart. if i let it run, it will usually drift apart by about 2-3 seconds after about 30 minutes or so
-> i can aggravate it by loading a new playlist while synched. my playlists are large, ~1000 tracks. that breaks sync almost immediately.
-> a song transition (i have crossfade active) also seems to provoke the issue
-> once broken, the only way i have found to re-sync is to cold-boot one or both players and/or restart SC.
-> after an SC restart, sync seems to work for a while before the problems start over.
-> the situation seems to be deteriorating continuously. currently, it is almost impossible for me to acheive any reasonable synching at all.
-> the problem seems to be independent of the load on my qnap's processor
-> according to a recommendation in this forum, i have tried to increase the "player start delay" to 500 ms, as well as increasing the "minimum synchronization adjustment" to 200 ms and decreasing it to 50 ms.

now, my environment:
Version: 7.3.2 - 24695 @ Mon Jan 19 18:36:25 PST 2009
Hostname: Turbonas
Server IP Address: 192.168.0.102
Server HTTP Port Number: 9001
Operating system: Linux - EN - iso-8859-1
Platform Architecture: armv5tejl-linux
Perl Version: 5.8.8 - armv5tejl-linux-thread-multi
MySQL Version: 5.0.27
Total Players Recognized: 2
-> both players are SB3/classics, both run FW 123
-> running on a qnap turbonas 409, FW 2.1.4 Build 0318T, with SSTOS 3.15.
-> my library is about 50% vorbis (native), 45% mp3 (native) and 5% AAC (transcoded)
-> my playlists are typically ~500-1000 tracks long, but the problem also occurs with shorter playlists ~40 tracks.
-> control is exclusively done by the remote control
-> my qnap is configured to update its system clock every hour by ntp. i have verified this actually works.
-> my players are connected by wifi. the network test comes through up to 4000 kbit @ 100% for at least 10 minutes.

does anyone have any suggestions for me? please keep in mind that i only have rudimentary linux knowledge...

thanks VERY much in advance!

CatBus
2009-07-23, 21:29
Sounds like you've already done much of the troubleshooting, so I'm not left with many good suggestions.

Have you checked to see if the server clock has noticeable drift from correct time? Have you noticed any difference in behavior on transcoded vs non-transcoded tracks? If you set up a test server on a regular PC and do the same thing, does the problem go away?

If the answer to the last question is yes, then it's something about the NAS. Maybe I/O if CPU utilization is low.

dcote
2009-07-24, 03:05
hi catbus!

no worries - right now, i am happy for ANY suggestions. ;-)

yes, i have tried much already. my daytime job is a hardware/tech/IT trainer&consultant, so tried a few things which seem smart from a generic tech point of view.

what is "noticeable" drift? using all the interfaces i have at my disposal, i can not detect any drift. but that isnt saying much, since i dont know how to to a professionally accurate measurment (with ms precision!) of my 409's clock.

how would i got about that?

better yet, how can i skip that step and install an NTP daemon? according to some threads i hve read here, that could solve the problem. unfortunately, they dont describe how to got about that...

jo-wie
2009-07-24, 05:12
better yet, how can i skip that step and install an NTP daemon? according to some threads i hve read here, that could solve the problem. unfortunately, they dont describe how to got about that...

It's in german, but I guess that's not a problem for you ;-)

http://forum.qnapclub.de/viewtopic.php?f=80&t=3194

I hope it's also valid for your system, I do not have a QNAP.

aubuti
2009-07-24, 06:55
I don't have any great insights because sync has been working fine for me, but here are a couple ideas. Do you still get the problems if the playlist doesn't have any transcoded tracks (AAC in your case)? When the players are drifting apart, what happens on track transitions? Is there any stuttering/re-starting as SC tries to re-sync the players, or do they just carry on, with one player making the song transition smoothly 2-3 seconds before the other makes a smooth transition? (I'm assuming here that your individual tracks are not 30 minutes long -- please let me know if that assumption is wrong!)

CatBus
2009-07-24, 09:14
When I say noticeable drift, I mean it'd be correct one day, 10 minutes slow the next day, 20 minutes slow two days later, etc. A little normal clock drift, that you wouldn't notice except over very long periods of time, should be perfectly fine.

I think your best bet is to set up a test server on a full-powered PC. If the problem goes away, we know it's something with the QNAP. If it DOESN'T go away, one of your SB3's may have a hardware problem, and you could call support.

dcote
2009-07-27, 02:41
yeah, i guess i'll just have to hunker down, find (lots of!) time, and do some experiments...

bpa
2009-07-27, 02:49
Don't forget when doing experiments enable the player.sync to DEBUG so that what you see can be tracked back to what SC is seeing/doing.

dcote
2009-08-12, 06:40
this is going to be a LONG one...

after some LENGTHY experimentation i have found more details.

preparation:
- upgraded to SC 7.3.3. the rest, such as perl etc. remains the same.
- wired up my two SB3s to eliminate the WLAN as a problem
- fixed up some files encoded with different vorbis encoders: GTune3b2 (my preference), aoTuV and xiph reference.

results:

to make sync break BOTH the following conditions must be met:
A. i am using the _remote_ to skip tracks, scan tracks or load playlists
B. i am playing or trying to play a _vorbis_ file, or the first file of a playlist is a vorbis file.

under these circumstances it seems i can not break sync:
A. using an MP3 and/or FLAC only playlist - even with the remote
B. exclusively using the web UI to control the sync group ie. skipping and scanning tracks and loading playlists - even with vorbis tracks in them

the vorbis encoder used does not seem to have any effect on this.

i can actually make sync come back if i fiddle around enough by:
A. skipping a couple of tracks _not_ through the remote, but through the web UI
B. pointing the remote at the _other_ (!) SB3 and restarting and/or skipping to the next track
C. manually starting an MP3 track or loading an MP3 only playlist

i have set the player.sync log to debug. if someone can tell me where to find the log and what to do with it i'd be grateful!

here's my problem: about 60% of our music collection is vorbis and especially when we have guests roaming the house we sync our SBs. but as soon as we change playlists, skip a track or even change the volume, sync seems to freak out. these findings are more or less in line with our past experiences, which were that if we sync up with a nice long playlist it would sometimes work, sometimes not. the more we left the SBs alone, the longer they would stay synced.

so how do i get vorbis to work properly when synced?

dsdreamer
2009-08-12, 07:49
to make sync break BOTH the following conditions must be met:
A. i am using the _remote_ to skip tracks, scan tracks or load playlists
B. i am playing or trying to play a _vorbis_ file, or the first file of a playlist is a vorbis file.



Vorbis decoding on the Squeezebox was not working until quite recently, and so most people had to use transcoding on the SC to get it to function. Are you still doing that? If so, switch to native. And if you are already using the native Vorbis decoder on the SB, switch to having it transcoded if the QNAP has sufficient CPU power to do it.

I admire the systematic troubleshooting you did. Such information would be excellent input to a new bug report (hint, hint).

Best regards,

dcote
2009-08-12, 07:56
my vorbis is playing native. when you say "recently", you mean like for the last three years now, right? ;-)
at least, thats how long i have been playing vorbis natively.

the qnap probably has enough power to transcode vorbis, but according to logitech's specs of the SB, that should not be necessary and it is supported natively. which is why i bought the SBs in the first place. ;-)

dcote
2009-08-12, 08:46
ADDENDUM:

this sucks - the drifting effect i described in my first post is also still there.

but once again, it seems that this only happens on vorbis files. when an MP3 comes along in the playlist, the players sync up perfectly. then, after 2-3 subsequent vorbis tracks, it starts drifting once more.

VERY odd...

dcote
2009-08-12, 09:58
this is the recent log of my SC. is this what would help to troubleshoot? if not, please let me know of other options! :-)

dcote
2009-08-13, 01:09
confirmed behavior by playing over night.

this problem is too consistent for my taste. hence:

https://bugs.slimdevices.com/show_bug.cgi?id=13380

dcote
2009-08-13, 01:22
no sooner have i filed a bug report to the effect that i _thought_ there was a vorbis/sync problem than a purely mp3 playlist also started drifting apart. this is the first time today, yesterday the mp3 playlist stayed rock solid synced for over two hours.

probably means i can stick my bug report where the sun don't shine... :-(

dcote
2009-08-13, 07:46
OK, more testing done:

1. deactivated crossfade -> no effect
2. installed SC 7.3.3. on a high-powered windows vista machine -> no problems so far with any means of operation or file type. will leave the sync-group playing over night to see if anything wierd happens.
i guess this means that my vorbis files are OK then...

i know this is starting to look like a NAS problem. but please bear in mind:
A. the effect is so much stronger with vorbis files. why is that?
B. why does my using the remote on one or the other SB or not using it at all influence sync reliability?
C. am i the only one with sync problems on a QNAP?

the next step for troubleshooting is to stop spamming bugzilla (which is what i have done) but to ask this forum:

what part of the SC syncing algorithm could be causing this? rather: what part of my setup could be throwing the sync off?

i think i need to start systematically eliminating potential sync problems...

tcutting
2009-08-13, 09:36
Is the time on the NAS accurate? I believe I've seen other threads on synchronization issues/drifting, and recall that often there was also issues with the Squeezecenter computer's clock (in your case, your NAS) having issues keeping time. Might be worth searching through previous forum threads.

dsdreamer
2009-08-13, 20:29
my vorbis is playing native. when you say "recently", you mean like for the last three years now, right? ;-)
at least, thats how long i have been playing vorbis natively.

the qnap probably has enough power to transcode vorbis, but according to logitech's specs of the SB, that should not be necessary and it is supported natively. which is why i bought the SBs in the first place. ;-)

https://bugs.slimdevices.com/show_bug.cgi?id=4418

and

https://bugs.slimdevices.com/show_bug.cgi?id=7857

Also see the change log for 7.3.3

http://svn.slimdevices.com/repos/slim/7.3/trunk/server/Changelog7.html

So if you've been playing vorbis files natively for years without trouble, I don't know how you did it!

Anyway, if you have the flexibility to transcode or not, it might be instructive to see if it makes a difference.

dcote
2009-08-13, 23:42
hehe - yeah i voted for those bugs (especially 4418) WAY back when they were first reported. ;-)

apparently, they are fixed now, which was one of the reasons i upgraded to 7.3.3. (apart from other bugs, where playback will randomly stop)
thing is, those bugs were "only" major for me. this bug (if it is one) is CRITICAL to me.
how nice that upgrading to 7.3.3. broke my aac playback. :-( seems to be a known issue in this forum though...

you are right tho - transcoding is one of the few things i haven't tried yet. hope it works, because the qnap's processor is not too powerful and vorbis decoding uses lots of cpu cycles. besides, i am not expecting any miracles from that, since sometimes even MP3 seems to drift. (see an earlier post of mine).

i could also try playing some pure WAV and see how that behaves...

is there someone out there who would mind explaining the sync mechanism to me and perhaps be willing to discuss what could disrupt it so badly?

dcote
2009-08-13, 23:56
Is the time on the NAS accurate? I believe I've seen other threads on synchronization issues/drifting, and recall that often there was also issues with the Squeezecenter computer's clock (in your case, your NAS) having issues keeping time. Might be worth searching through previous forum threads.

thanks for the suggestion. :-)

well, i've followed those threads as well.

firstly, my NAS NTP updates its system clock once per hour. as far as the logs tell, that is working just fine.

secondly, there are suggestions to install an NTP daemon, which supposedly will make the clock more accurate(?). however, over at the qnap forum it is felt that that will not do the trick. a linux friend of mine has confirmed that. not sure why it seemed to help others, tho...
the qnap forum also said it should be highly unlikely to see the qnap's clock drift by more than a couple of seconds per day, which is about right for a hardware-based RTC.

thirdly, logic tells me that shouldn't be an issue. if the qnap's clock is the identical and concurrent reference for ALL SBs at the same time, a drift in the qnap's clock should cause ALL the SBs to drift at the same time, right? so the worst case would be that ALL SBs skip forward at the same time, and all by the same amount. OTOH, any differences between the SBs themselves would still be seen, because of their varying time shift _relative_ to the qnap.
but as i said in my previous post, i am not sure how the sync mechanism actually works, and this chain of logic is based on pure assumption.

dcote
2009-08-16, 07:15
news: after a 1/2 day's battle, i managed to do a clean re-install of SSOTS and SC. now i am running SSOTS 3.18 (after discovering that the 3.17 provided by qnap is broken) and SC 7.3.3.

indeed, sync is greatly improved! :-)

however, after two hours of synced playing, there still were two instances where sync broke on a vorbis file. and again, it came back as soon as an MP3 file came up. i didnt even need to do anything about it...

strange. as if SC sometimes "forgets" to keep sync when playing a vorbis file and suddenly "remembers" when an MP3 comes up.

dsdreamer
2009-08-16, 07:51
news: after a 1/2 day's battle, i managed to do a clean re-install of SSOTS and SC. now i am running SSOTS 3.18 (after discovering that the 3.17 provided by qnap is broken) and SC 7.3.3.

indeed, sync is greatly improved! :-)

however, after two hours of synced playing, there still were two instances where sync broke on a vorbis file. and again, it came back as soon as an MP3 file came up. i didnt even need to do anything about it...

strange. as if SC sometimes "forgets" to keep sync when playing a vorbis file and suddenly "remembers" when an MP3 comes up.

Probably the least CPU-intensive think you can try is Ogg/vorbis -> Wav. This avoids having to do simultaneous decoding and encoding. You say you have plenty of bandwidth and Wave files are constant bit rate (CBR), which may just possibly help with synchronization.

I was wondering whether CBR vs. VBR had anything to do with this issue? Vorbis is inherently a VBR codec, so there is a small probability that this issue occurs when Slimproto uses the average bit rate to determine how many samples to skip and gets it wrong. Do the MP3 files that seem to reset the synch issue have CBR encoding, by the way?

I would also mention that your original bug report is valid.

dcote
2009-08-16, 09:38
no THAT might explain A LOT.

i use the GTune3b2 vorbis encoder, whose bitrate envelope is quite different from the reference decoder's. GTune3b2 will allocate MANY more bits/s on transients and high-frequency passages than the reference. if all the assumptions sync does are based on the reference - no wonder.

i will see if the sync breaks are reproduceable on certain tracks. if yes, i will try encoding them with the reference encoder and see if the problem goes away.

dcote
2009-08-28, 03:01
i havent had much time to do any additional intensive testing, but i have been seeing a pattern - i _think_.

if sync breaks (much rarer now), then it does so when BOTH the following conditions are met:
-> transitioning TO a vorbis file. higher bitrates seem to increase the likelyhood of it happening.
-> my qnap's processor is _very_ busy doing something else, such as accepting a large SMB transfer (~GB) from the network.

if i load the processor in the middle of a vorbis track, sync also starts drifting audibly (i had 2 SBs right next to me on my desk for testing). but it returns once the load drops again. however, if a song transitions meanwile... see above.

the odd thing is, it still seems as if an MP3 file is required for sync to return - even after the CPU load has dropped back to normal.