[Announce] Community Firmware for Squeezebox Radio/Touch/Controller and LMS 8

[ralphy]

ralphy - planning ahead for when we've collected enough beta feedback, how do you think we should coordinate checking these changes in? Happy to discuss in PM if that's easier.

If you're comfortable using git, feel free to create a pull request at https://github.com/ralph-irving/squeezeos
If there's a bug in the squeezeplay interface, the repo is at https://github.com/ralph-irving/squeezeos-squeezeplay
Or you can send the files to me and I'll commit them.
We should discuss rebuilding the tools in the driver, should we or just use the current working ones?
Turns out that the source for the tools was not in the original SW83 files you discovered, I added them to the tar file used for the recipe.

[ralphy]

Upon reflection, I think ralphy 's fix for "AR6000 updating target stats" is probably the better one, since it eliminates the spamming even when DEBUG is enabled (and having DEBUG enabled is super helpful).

Might it be better to apply the patch that moves the message to the debug output only instead of removing it completely for the firmware builds? I prefer to try to keep as close as possible to the original firmware.
If there are issues in the future, we are going to have to rebuild the driver with debug enabled anyway, we can modify the recipe locally to use the removal patch instead. Just keep both patch files in the folder.

[dzenc]

I caught it in the act! And this time both me and the radio were well prepared to debug this type of issue (said issue which feels different than e.g. the kernel crash, assert, or wifi drops I had observed on previous builds)..

Interestingly, I went and looked at my one Radio still running 7.7.3-r16676, the Radio that still pretty reliably turns RED every few hours. And, IT TOO, is in a similar DHCP failure state and it too, is continuing to send and receive wifi traffic that has a bad IP address. So this bug seems to have existed since the shipping firmware. I've now instrumented this Radio with logging. It's been very reliably turning RED on a regular basis up until now. Let's hope it keeps that up, at least until I fully diagnose this.

-Dan

[mrw]

dzenc

A failure to renew the DHCP lease would have the effects that you have seen (i.e. link local IP, etc). For why is another matter.

[dzenc]

For sure. The catch is that I’m seeing both the dhcp request and successful response when monitoring with tcpdump on the router. So it sure seems like it’s getting a successful response, it’s just that for whatever reason it’s not acting like it is.

(And this repeats multiple times, so it’s not like a single lost packet or something. Something is getting mis-configured in a way that seems to stop it from “accepting” the proper response.)

[dzenc]

The fact that the Loopback adapter starts to see unusual traffic when this occurs makes me suspect it’s some kind of routing issue, either a mis-configuration or bug.

I‘m trying some things out…

[mrw]

For sure. The catch is that I’m seeing both the dhcp request and successful response when monitoring with tcpdump on the router. So it sure seems like it’s getting a successful response, it’s just that for whatever reason it’s not acting like it is.

(And this repeats multiple times, so it’s not like a single lost packet or something. Something is getting mis-configured in a way that seems to stop it from “accepting” the proper response.)

I have, once, seen the same sort of thing. Wireless traffic is sent to the Radio, but it acts like it didn’t receive it. In this particular case, association responses. So the Radio kept sending association requests, and ignoring the response. I know the responses were sent. So I figured it must be getting stuck in the firmware, or in the driver. I think someone else reached a similar conclusion, that incoming wireless traffic suddenly stops being seen.

[dzenc]

Yeah, as frustrating as it is, I’m pretty sure I’m still just finding out the same things that have already been learned by others in the past while analyzing this problem. (…many Bothans died to bring us this information...)

Though I’m happy to take suggestions/hints if anyone has any…

We’ll see if I get any further. In the words of Princess Lea, I only hope that when the data's analyzed, a weakness can be found. 😉

[gegen]

Yeah, as frustrating as it is, I’m pretty sure I’m still just finding out the same things that have already been learned by others in the past while analyzing this problem. (…many Bothans died to bring us this information...)

Though I’m happy to take suggestions/hints if anyone has any…

We’ll see if I get any further. In the words of Princess Lea, I only hope that when the data's analyzed, a weakness can be found. 😉

"dzenc, the Force will be with you. Always." - Obi-Wan Kenobi, Star Wars Episode IV: A New Hope

[dzenc]

So I figured it must be getting stuck in the firmware, or in the driver. I think someone else reached a similar conclusion, that incoming wireless traffic suddenly stops being seen.

I just reproduced this (again) on version 7.7.3. This time I had tcpdump capturing both on the Radio and the router. Comparing the two captures, I can confirm that, at the point of failure, tcpdump on the Radio stops seeing incoming packets, indicating that something in the Radio's network stack is preventing them from getting to it; as you said, it's most likely stuck in the wifi firmware or wifi driver.

I've now set up my most-likely-to-fail new-driver Radio with similar instrumentation. I will attempt to catch it in the act again with the new driver, and see if the new driver behaves the same way. It may take a while to reproduce...

[mrw]

One possible approach that occurred to me could be, conceivably, to organize matters to “notice” that incoming WiFi packets have inadvertently stopped. This noticing taking place before IP gets involved. Currently available stats are all, I think, limited to IP, so I put the thought to one side.

Just an idle thought really, but if we are able to modify a kernel driver, then perhaps offering a more reliable approach than than monitoring IP connectivity. But there are many other matters to be quietly understood and explored.

Note to self: I vaguely recall that a patchlet may be needed to build tcpdump. Review, and offer up if necessary.

[gegen]

Am using this version of the driver and firmware:

Files are here (Already renamed to custom.baby.*) - https://github.com/dzenc/SqueezeBuil...ain/testv1.zip

and with WLANPOKE uninstalled.

Radio has now been playing, with no wifi issues, for 30 hours, a mixture of local flacs and remote spotify tracks.
The UI is also still responsive.

This radio is in a 'good' location which hasn't had the 'WiFi6 interference' problems as far as I could know,
so only testing stability.
Have attached the outputs of dmesg, iwconfig and route, hope they are of some use.

I'm now going to move the radio to a place that does seem to have the interference problem. At this location the radio
would be fine on the old driver/firmware for 3 or 4 days and then suddenly there would be 2 or 3 disconnects
a day for a few days, very unpredictable. WLANPOKE did improve this but not completely, still occasional disconnects.

I'll test over a longer period to see how it goes. Thanks for all the effort that you are all putting into this.

radio logs.zip

[dzenc]

😃 Thank you for testing. I look forward to hearing what happens in the new location.

[dzenc]

One possible approach that occurred to me could be, conceivably, to organize matters to “notice” that incoming WiFi packets have inadvertently stopped.

Agreed. I'm still hopeful we can do better, but it's possible that this is the best answer we end up with. In the back of my mind I have been wondering if the difference between this wifi firmware - which I've yet to see "assert" - and the newer wifi firmware, which I've seen intermittently assert - is that Atheros had the same thinking. For whatever reason, they weren't able to find/fix the actual bug in the firmware (maybe they just didn't want to spend the time working on an "old" chip), so they added a controlled crash to the firmware when they detect the condition. Which could then be further detected on the host and used to reset the wifi stack.

Per the release notes from that enterprise product that also used this wifi chip:

- (reliability) The "qtperiodic" watchdog task now recognizes radio "ar6000_target_failure" events and restarts the radio whenever such a fatal error occurs. If this takes place, the system uptime (in seconds) is saved to the file "/var/tmp/wlan_reconfig_initiated".

It may or may not be the case that this assert is the same as our issue. It's also possible Atheros simply introduced a NEW bug in the new firmware and ours is something completely different. But it's suspicious. Still TBD.

[dzenc]

Might it be better to apply the patch that moves the message to the debug output only instead of removing it completely for the firmware builds? I prefer to try to keep as close as possible to the original firmware.
If there are issues in the future, we are going to have to rebuild the driver with debug enabled anyway, we can modify the recipe locally to use the removal patch instead. Just keep both patch files in the folder.

Been thinking more about this. The problem with leaving the message enabled in DEBUG mode is that it STILL floods the logs when DEBUG mode is enabled, whereas the rest of DEBUG output is pretty measured and has been very useful. So it makes DEBUG mode less helpful. I think the right approach may be to leave it in as you suggest, but to also change it to a 2nd order DEBUG message, which isn't printed in DEBUG mode unless you actively raise the dynamic debug level. There is support for this already in the driver, so it's just a matter of changing which PRINT function you call.