[Announce] Community Firmware for Squeezebox Radio/Touch/Controller and LMS 8

[dzenc]

If you're comfortable using git, feel free to create a pull request at https://github.com/ralph-irving/squeezeos
If there's a bug in the squeezeplay interface, the repo is at https://github.com/ralph-irving/squeezeos-squeezeplay

Thanks. Sounds like a good plan. I had been hopeful that we were close to getting this checked in, but there is clearly still more work to be done...

We should discuss rebuilding the tools in the driver, should we or just use the current working ones?
Turns out that the source for the tools was not in the original SW83 files you discovered, I added them to the tar file used for the recipe.

I think either way is ok. If someone wants to jump in and take a look at getting them built, that would be great. Otherwise we can stick with the binaries for now. I'm gonna stay focused on the driver, at least until I nail this.

[staresy]

Another update.

Ok rebooted radio running dev FW, set it playing a long playlist of local FLAC.

It ran flawlessly for 24 hrs 18mins, then exhibited the "stuck at end of track" error, ie. it won't progress to next track in playlist.

No loss of WIFI, radio still responsive, still controllable via web GUIs, no "connection reset by local host message". Just manually skipped to next track, now replaying OK

Not sure if this is a red herring as mentioned before but anyway, this time I could SSH in, so have captured the data you requested previously (attached).

[dzenc]

Thanks. I had a typo in my instructions (should have said ifconfig not ipconfig, now fixed) but that data isn’t needed for now anyway. The only thing I can observe in your log is that, towards the end of the log, it notes that it reconnected to wifi. Although it doesn’t explicitly log a disconnect, the implication is that it previously disconnected. This could be expected behavior if there is a burst of noise (e.g. microwave turned on) or other temporary interference. The key point is that after the interference, the wifi successfully reconnected which is obviously good.

I’m not sure if that’s related to your playback issue or not.

[dzenc]

Since I last provided an update, I am still waiting for one of my new-driver radios to disconnect so I can capture more data. Unfortunately (or fortunately?) it can take multiple days. Since the new driver has source code, that’s where I’m able to do the additional logging, so that’s where I need to see it fail again.

At the same time, the old-driver radio continues to die every couple of hours.

So… while I wait… I’m experimenting with ways to binary-patch the old driver to see if I can hack some logging into it . Since the failure happens much more frequently on the old driver, that would be a sneaky way to help speed up debugging. We’ll see…

[ralphy]

Been thinking more about this. The problem with leaving the message enabled in DEBUG mode is that it STILL floods the logs when DEBUG mode is enabled, whereas the rest of DEBUG output is pretty measured and has been very useful. So it makes DEBUG mode less helpful. I think the right approach may be to leave it in as you suggest, but to also change it to a 2nd order DEBUG message, which isn't printed in DEBUG mode unless you actively raise the dynamic debug level. There is support for this already in the driver, so it's just a matter of changing which PRINT function you call.

I agree that raising the debug print level for that message is the better option.

[ralphy]

I think either way is ok. If someone wants to jump in and take a look at getting them built, that would be great. Otherwise we can stick with the binaries for now. I'm gonna stay focused on the driver, at least until I nail this.

Looked through the commit history for the radio wifi tools and I've only ever included the prebuilt binaries from logitech.
I'll see if I can get them patched and built, I only have the tools source code from SW63. I'll check the original file you found.

[staresy]

I’m not sure if that’s related to your playback issue or not.

Interestingly, it seems that after suffering the initial "playback stuck" issue (resolved by skipping to next track and selecting play), it seems to occur a lot more frequently (every couple of hours or so, cw 24 hrs initially without a problem).

However, agree that this maybe nothing to do with what you are chasing, so I only mention it for completeness.

So as not to cloud the issue, I won't send any further logs unless you want them (or unless I get another "host connection reset" message).

[dzenc]

Interestingly, it seems that after suffering the initial "playback stuck" issue (resolved by skipping to next track and selecting play), it seems to occur a lot more frequently (every couple of hours or so, cw 24 hrs initially without a problem).

This is suspicious to me because I see something somewhat similar on my 7.7.3 radio after it goes red. When the I use the UI to reconnect to wifi (rather than e.g. power cycling), the next disconnect tends to happen quicker than the previous one. Its like something is getting mucked up in its internal state and reconnecting gets it back connected but doesn’t fix the underlying issue…

If you fully turn off and then back on the radio, does it go back to its 24 hour playing cycle?

[staresy]

This is suspicious to me because I see something somewhat similar on my 7.7.3 radio after it goes red. When the I use the UI to reconnect to wifi (rather than e.g. power cycling), the next disconnect tends to happen quicker than the previous one. Its like something is getting mucked up in its internal state and reconnecting gets it back connected but doesn’t fix the underlying issue…

If you fully turn off and then back on the radio, does it go back to its 24 hour playing cycle?

OK, as if by magic, I just caught it in the act, playback stuck, no audio, "Connection reset by remote host" message.

Logs attached. I WAS actually using the microwave at the time, which is quite close (adjacent room, opp side of wall), so that could well have been the spurt of 2.45GHz RF that upset it.

Anyway, skipping to next track, pressing play a few times has resumed operation.

In answer to your question, yes a reboot seems to make it last considerably longer (between 16-24hrs), once it starts getting poorly it seems to get worse and worse until I put it out of it's misery with a reboot.

[dzenc]

One thing I see in your second set of logs is that the Loopback adapter has seen 62 packets transferred, which in my experience is a tell-tail sign that the system lost wifi at some point (though seemed to recover).

(I had tcpdump running on the Loopback interface at one point while my failure was occurring and I caught this in action: I saw that the system was basically telling itself via ICMP messages that it had no route to the LMS server, and these ICMP messages, which it was sending from itself to itself, were traveling, somewhat logically, over the Loopback interface.)

[dzenc]

Since I last provided an update, I am still waiting for one of my new-driver radios to disconnect so I can capture more data. Unfortunately (or fortunately?) it can take multiple days. Since the new driver has source code, that’s where I’m able to do the additional logging, so that’s where I need to see it fail again.

At the same time, the old-driver radio continues to die every couple of hours.

So… while I wait… I’m experimenting with ways to binary-patch the old driver to see if I can hack some logging into it . Since the failure happens much more frequently on the old driver, that would be a sneaky way to help speed up debugging. We’ll see…

It’s been a couple of days and my testing new-driver radio has not yet turned red again. (and I believe none of my other new-driver radios have ever turned red). I suppose this is good news but for me trying to reproduce this failure it’s a bit of a pain…

In the mean time I was successful in my attempt to binary patch some logging into the 7.7.3 old driver(!!)

And I caught the old driver failing. I was able to tap somewhere into the middle of the packet receive function in the driver. I am logging ARP packets because they are frequent but not log overwhelming and during a failure, the system is regularly trying to send and receive them.

I can confirm that when the failure occurs on the old driver radio, and tcpdump is no longer seeing received packets, that, at least at that place in the driver, the old driver is also no longer seeing ARP packets.

So the two things are consistent, but perhaps not good news, at least as far as the old driver is concerned.

I have a similar test set up on a new-driver radio and I’m just waiting for it to go red.

The reason I am still hopeful about the new driver is that, during my last new driver test, though I wasn’t yet logging ARP packets, I did have another log message in for a different kind of packet and contrary to what the old driver test just showed, on the new driver, I DID see packets continuing to arrive in the driver even AFTER the failure occurred. Which is actually quite promising if the same holds up for ARP packets…

It will be interesting to see what the ARP logging shows, if/when the new driver radio eventually goes red again.

[yosherl]

...

It may be too early to hand this out, but since it's been working well here, and seems to have resolved the known issues, I'm going to be adventurous. If you are also feeling adventurous, please give it a try.

Files are here (Already renamed to custom.baby.*) - https://github.com/dzenc/SqueezeBuil...ain/testv1.zip
...

dzenc after updating my "Baby" to the test version, there was no WiFi at all anymore. Only LAN is working...
Did someone else have this behaviour as well? I'll try another factory reset :-)

Update: Yes, everybody. Check your scripts before when playing around... issue is the wifi boot script looking for the just - deleted loadAR-83 script

[dzenc]

Just to confirm, this boot script change was something you had done yourself beforehand correct? Nothing in the test firmware I posted should be referencing a loadAR-83 script…

Is it working for you now?

[yosherl]

Okay, separate comment after fixing my own issue :-D

Napster 320kbit playback via LAN is perfect, no beginning congestion.
Napster 320kbit playback via WIFI is 99% perfect now as well. Will monitor this. I had 1-2 files after the boot that had minimal pre-buffer issues, but now looks good.

[yosherl]

Just to confirm, this boot script change was something you had done yourself beforehand correct? Nothing in the test firmware I posted should be referencing a loadAR-83 script…

Is it working for you now?

I was a bit puzzled last week on where to put which file (last week, before the two release binarys). But fixed it now back. One week holiday and you forget what you've done a week ago ...