Home of the Squeezebox™ & Transporter® network music players.
Page 1 of 27 12311 ... LastLast
Results 1 to 10 of 264
  1. #1
    Senior Member
    Join Date
    Dec 2009
    Posts
    137

    Lightbulb Radio alarm issues [was: Radio architecture question for Slim developers]

    I admit to being a novice here at the Squeezebox forums, so I may not be entirely up to speed on the proper etiquette for addressing the Logitech development organization. I apologize in advance if this is the wrong place for this post.

    Having said that, it would help me greatly (and perhaps many other Radio adopters) if you guys at Logitech could answer a specific question about how the Radio operates under the covers...

    Some history: I've spent the day testing and examining the AlarmSnooze applet and now understand why it is seemingly so unstable (I also see what are likely to be instabilities in the newest version of the applet, which I also tested). The underlying reasons for the instability of this applet may also bear a relationship to why the Radio seems to exhibit instability in other respects.

    Without going into too many specific details on the AlarmSnooze applet (so as not obscure the forest for the trees), I see what appears to be a glaring hole in the operation of Radio applets in general. Now, it may be that since I don't fully understand the underlying Radio infrastructure at this point I am missing something. I've only been studying Radio applet operation for one day now (albeit all day), but I'm very surprised at the general nature of the applets I've reviewed (in particular, AlarmSnooze).

    Please keep in mind that I really don't mean to appear presumptuous...

    with that said, it appears to me that the Radio is using Lua in a multithreaded manner (seemingly using LOOP packages). I gleaned that asynchronous event notifications can be delivered to applets; for example I see that the playerConnected and playerAlarmState notifications are registered by the AlarmSnooze applet. It also appears that timers provide for a separate callback mechanism when they expire and subsequently invoke the applicable registered callback function(s). I looked long and hard for either a critical section directive, or a semaphore or mutex provision but was unable to find one. I searched at both the Jive level and lower at the Lua proper level. I did locate some locking mechanisms specified in the Lua documentation amongst the Lua language extensions, but I don't see any use of, or provision for, them on the Radio after extensive searching.

    I haven't had the time yet to explore SqueezeOS in depth, but I'm curious if you guys are using a run-to-completion, timesliced, or preemptive threading model on the Radio...? Or are you forcing some other manner of simulated single threaded operation by funneling all disparate looking thread contexts through a single monolithic executive thread of some type...?

    In a nutshell, are you running a truly multithreaded environment? And if so, how do you have the scheduler configured?

    If there are really disparate threads (for example, the notification callback subsystem and the timer subsystem) that can invoke applet functionality from their own distinct contexts then there either needs to be forcible synchronization/serialization of applet threads by the underlying OS scheduler (perhaps run-to-completion) or there needs to be some critical section, semaphore, or mutex type operation(s) available for applets to explicitly provide for code synchronization where applet resources are shared amongst functions. I see specific spots in the AlarmSnooze applet (which I'm certain can be extrapolated to many other pieces of software on the Radio) where disparate underlying system thread contexts can crash into (preempt) each other because there is no overt means of locking a particular section of code against said preemption...

    As I'm guessing you guys know, multithreaded embedded systems typically use a variety of synchronization mechanisms to prevent these types of problems when information needs to be coherently shared amongst more than one thread of execution.

    If you guys have a critical section mechanism (or mechanisms), or OS scheduler forced thread serialization model (like run-to-completion), under the covers somewhere then can you please point it/them out...?

    Thanks very much, and kudos on all your hard work on the very cool Squeezebox products.

    Marc


    p.s. I believe I have hardened up the latest release of the AlarmSnooze applet, but until I understand how the underlying scheduler and threading model works I do not believe that the applet (and perhaps others) can be properly modified to be entirely stable.
    Last edited by Marc; 2009-12-23 at 22:11. Reason: spelling/grammar, 12/23 changed title to Radio alarms

  2. #2
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,352

    Radio architecture question for Slimdevelopers

    Hi Marc,

    let me tell you I don't know nothing about Radio's multithreading.

    > p.s. I believe I have hardened up the latest release of the
    > AlarmSnooze applet, but until I understand how the underlying scheduler
    > and threading model works I do not believe that the applet (and perhaps
    > others) can be properly modified to be entirely stable.


    What are your changes? Do you have a diff available? You'd have many friends if you could help us fix the alarm asap.

    --

    Michael

  3. #3
    Senior Member
    Join Date
    Dec 2009
    Posts
    137
    I've only been looking for a day, Michael. I definitely see problems, though, and have already corrected some (though more testing is needed before distribution). I'm convinced the problems can likely be corrected in relatively short order...

    Again, though, no truly stable operation can be ensured prior to understanding the threading model of the Radio and then coordinating the applet code with the operation of the underlying OS. The applet can't be properly fixed in a vacuum, without that broader system knowledge.

    The disparate threads either need to be guaranteed they'll run to completion respectively, or there *must* be some manner of synchronization provided by the OS which can be explicitly called by the applets to prevent re-entrancy during critical sections of code. There really is no middle ground...

    The bigger picture is that this issue transcends the AlarmSnooze applet problems. It applies to virtually all software running on the Radio. That is, if it's an issue at all (read: SqueezeOS may yet be forcing serialized applet execution through a single executive thread - I haven't had time yet to reverse engineer the underlying operation). Let's see what Logitech has to say about their threading model and/or critical section locking provisions...

    Marc

  4. #4
    Senior Member bluegaspode's Avatar
    Join Date
    Jul 2009
    Location
    Berlin, Germany
    Posts
    3,229
    I haven't had the time yet to explore SqueezeOS in depth, but I'm curious if you guys are using a run-to-completion, timesliced, or preemptive threading model on the Radio...? Or are you forcing some other manner of simulated single threaded operation by funneling all disparate looking thread contexts through a single monolithic executive thread of some type...?
    I'm no Logitech-Developer so what I'm writing might be wrong.
    I stumbled about the same things when I wrote an applet that used the Timer-Framework a lot and I wondered about the same things

    As far as I remember there is a (Single-Threaded) Main-Event-Loop which 'fires' all events (including Timer-Events). I'll try to find the corresponding classes in the evening (I am at work right now) I think I remember them lying around under the 'jive' package.

    I guess/hope that external events are queued as well so you won't have 'real' multithreading in the LUA-Applets themselfes.

    As I said I found these places when following the Timer-System (was it window.addTimer ??).
    As the HTTP Framework also works asynchronious (receiving the responses) you might also start there. The Flickr-Applet does HTTP-Requests so its a good starting point for further investigations as well.
    Did you know: SqueezePlayer will stream all your music to your Android device. Take your music everywhere!
    Remote Control + Streaming to your iPad? Squeezebox + iPad = SqueezePad
    Want to see a Weather Forecast on your Radio/Touch/Controller ? => why not try my Weather Forecast Applet
    Want to use the Headphones with your Controller ? => why not try my Headphone Switcher Applet

  5. #5
    Senior Member
    Join Date
    Dec 2009
    Posts
    137
    Quote Originally Posted by bluegaspode View Post
    I'm no Logitech-Developer so what I'm writing might be wrong.
    I stumbled about the same things when I wrote an applet that used the Timer-Framework a lot and I wondered about the same things

    As far as I remember there is a (Single-Threaded) Main-Event-Loop which 'fires' all events (including Timer-Events). I'll try to find the corresponding classes in the evening (I am at work right now) I think I remember them lying around under the 'jive' package.

    I guess/hope that external events are queued as well so you won't have 'real' multithreading in the LUA-Applets themselfes.

    As I said I found these places when following the Timer-System (was it window.addTimer ??).
    As the HTTP Framework also works asynchronious (receiving the responses) you might also start there. The Flickr-Applet does HTTP-Requests so its a good starting point for further investigations as well.

    Ok, I think I see how the system works, in general, now that I've looked further at Lua coroutines. An explanation (albeit, a bit confusing) is here: http://lua-users.org/wiki/CoroutinesTutorial

    I do see sporadic use of coroutine mechanisms in some of the jive and lua code on the Radio, though the explicit use of these coroutine mechanisms is not very widespread.

    So, it looks like the system is using cooperative scheduling in a run-to-completion model. That is, it appears each thread runs either to completion (without interruption) or until it explicitly yields to other threads in the system by calling coroutine.yield(...)

    Now, where an applet does not explicitly yield (AlarmSnooze does not do so anywhere in the applet) the question that remains is this: where, if anywhere, does the underlying jive (or lua proper) code implicitly yield during a system call of some kind that is made from the applet? The point is that the applet can still yield to other threads when it makes another system call (which is defined elsewhere) which itself calls coroutine.yield(...) I don't think this behavior (where system calls themselves yield under the covers on behalf of the calling thread) is widespread, since I don't see too much use of yield in the jive foundation classes. I'm going to presume for now that this implicit yielding does not occur...

    Ok, I've poked around some more and see how both the timer and notification callbacks work. The main handling loop is defined in the UI framework and both notifications (via processEvents()) and timer callbacks are handled serially.

    So, everything does indeed look good in terms of thread-safety. I should never have doubted the Logitech guys in the context of system thread safety!

    Now I feel comfortable continuing on in the hardening of the AlarmSnooze applet. I've got some stuff to do today, but should be able to work on it again later tonight. Is there any licensing issue with modifying and then redistributing a Logitech provided applet? I don't see any reference to licensing in the AlarmSnooze applet code itself...

    I'll report back once I've had a chance to further harden and test the applet. I'll likely need some people who are willing to do their own testing with it as well (once it's ready for that), since it's very unlikely that I'll be able to test all the code paths in my own environment. I will be certain to heavily instrument the first distribution for troubleshooting purposes so that when logging is turned up for the applet there will be little doubt about what has transpired in problem cases...

    Marc

  6. #6
    Ne'er-do-well, Vagabond bklaas's Avatar
    Join Date
    Apr 2005
    Location
    Minneapolis, MN
    Posts
    2,033
    hi Marc,

    a couple of notes...

    Just at the end of last week I finally have had the opportunity to start a rework of the AlarmSnooze applet. I'm doing this on the 7.4 branch and made a checkin late last week to start that work.

    http://svn.slimdevices.com/jive?view...&revision=8237

    I plan to be making more checkins on alarm issues this week.

    I am interested in what you're doing to harden this applet, but I'd ask that you also keep up to date with what I'm doing.

    My two cents is that you're diving way too low in the code in evaluating problems with Alarm. I may turn out to be wrong, but I believe that the current issues being seen by some customers are going to be solved at a higher level than what you've been assessing thus far.

    cheers,
    #!/ben
    Former Logitech Developer: Squeezeplay/SqueezeOS/SqueezeboxController/SqueezeCenter
    Community Developer: Nokia770Skin (r.i.p.)

    http://www.last.fm/user/bklaas/
    KHAAAN!...BUNNIES!

  7. #7
    Senior Member
    Join Date
    Dec 2009
    Posts
    137
    Quote Originally Posted by bklaas View Post
    hi Marc,

    a couple of notes...

    Just at the end of last week I finally have had the opportunity to start a rework of the AlarmSnooze applet. I'm doing this on the 7.4 branch and made a checkin late last week to start that work.

    http://svn.slimdevices.com/jive?view...&revision=8237

    I plan to be making more checkins on alarm issues this week.

    I am interested in what you're doing to harden this applet, but I'd ask that you also keep up to date with what I'm doing.

    My two cents is that you're diving way too low in the code in evaluating problems with Alarm. I may turn out to be wrong, but I believe that the current issues being seen by some customers are going to be solved at a higher level than what you've been assessing thus far.

    cheers,
    #!/ben

    Hi Ben,

    Thanks for the feedback. I did start with your recently modified AlarmSnooze applet (it was quickly included in the SqueezePlay distribution after your checkin).

    I absolutely agree with your two cents as to how the alarm problems likely reside in the applet itself, and not the underlying Radio infrastructure. Remember, though, that before yesterday I had never done any work with SqueezeOS, lua, or any Logitech/Slim devices. Thus, I didn't have a good understanding about how the system operated. Once I recognized that the applet can be entered through a number of different means (timer expiration, system notifications, initialization), it would have been remiss to start working on the applet issues without first understanding the threading model underneath. Now that I understand the threading model and realize there's no need to worry about re-entrancy concerns at the applet level, it makes sense to focus directly on the applet logic.

    I'm glad to work together on this, or even just let you have at it yourself if you're dedicating some cycles to the issue now. Let me know how you'd like to proceed...

    Marc

  8. #8
    Ne'er-do-well, Vagabond bklaas's Avatar
    Join Date
    Apr 2005
    Location
    Minneapolis, MN
    Posts
    2,033
    hi Marc-

    I'm happy to get your input on alarm fixes. The first thing I'd like to do is move this thread to the developers forum. Let me see about doing that first and then we'll continue the conversation.

    cheers,
    #!/ben
    Former Logitech Developer: Squeezeplay/SqueezeOS/SqueezeboxController/SqueezeCenter
    Community Developer: Nokia770Skin (r.i.p.)

    http://www.last.fm/user/bklaas/
    KHAAAN!...BUNNIES!

  9. #9
    Wait, wait, let's keep talking architecture!

    I thought SqueezePlay, like Squeezebox Server, was deliberately single-threaded and applet developers did not need to worry about thread safety. Are you saying that a smart developer could actually use threads (e.g. I could write an RS232 amp interface for Touch that used a separate thread to prevent messing up the SqueezePlay UI responsiveness while waiting a half a second for serial port data -- instead of moving the RS232 stuff into a separate process on SqueezeOS)?
    owner of the stuff at https://tuxreborn.netlify.com/
    (which used to reside at www.tux.org/~peterw/)
    Note: The best way to reach me is email or PM, as I don't spend much time on the forums.
    Free plugins: AllQuiet Auto Dim/AutoDisplay BlankSaver ContextMenu DenonSerial
    FuzzyTime KidsPlay KitchenTimer PlayLog PowerCenter/BottleRocket SaverSwitcher
    SettingsManager SleepFade StatusFirst SyncOptions VolumeLock

  10. #10
    Former Squeezebox Guy andyg's Avatar
    Join Date
    Jan 2006
    Location
    Pittsburgh, PA
    Posts
    7,396
    In the early days of SP a separate thread was used for the networking code. But this turned out to have much worse performance than doing it single-threaded. So now it's single-threaded. But yes, there is nothing stopping an applet from using threads, you just need to be careful and know what you are doing.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •