PDA

View Full Version : Device manager Dwite @github



Mysingen
2009-07-08, 12:19
What: A pre-alpha release of the SqueezeBox device manager Dwite.
How: "git clone git://github.com/Mysingen/dwite.git"
License: GPLv3

Enjoy!

erland
2009-07-08, 22:08
What: A pre-alpha release of the SqueezeBox device manager Dwite.
How: "git clone git://github.com/Mysingen/dwite.git"
License: GPLv3

Enjoy!

What functionality does this pre-alpha version contain ?

Do you feel it's safe to run it towards real Squeezebox devices or is there any risk destroying something ?

Which hardware types have you tested it against ?

Do you want help testing something specific ?

Mysingen
2009-07-09, 00:39
Hello

Running against real hardware should be safe. I don't change the firmware and I don't change the in-hardware settings. I run it against an SB3 of SlimDevices fame.

I went public because I'm hoping someone might be interested in collaborating with me. A solid test suite would be golden, if anyone wants to write one :) The reason I wrote Dwite in the first place is I don't care much for media managers that track my entire collection, so SqueezeCenter was never for me. I want Dwite to work well with a "large" CM, but my own needs are better served by something that XMMS can interact with.

A brief feature rundown:

* Streaming of MP3's work, but only "play" and "stop" are implemented in STRM.

* Volume control works (preamp + new-style gain).

* Brightness control works.

* The remote's IR codes are mapped up and you can use it to change the volume, navigate the device manager's menu system, start playback and change brightness. Essentially, the framework for remote control is done, but most of the features you'd want to control with it are not.

* There is a screen rendering framework and what little I do with it works very well (show TTF font strings and automatically scroll too long lines). The rendering is driven by a central tick() function that has to be called at least 10 times per second to get smooth animations.

* There is a menu system that is built from stock items. Now there are only stock items for file system directories and files, and Dwite knows how to populate both itself. The plan is to introduce some other stock items like "playlist", "now playing", et.c., and interact with a content manager to populate container items with meta data. This lets the device manager run ls() on a container whenever it needs to step in or out of a container (because the user navigated so with the remote) and have the ls() handled by an external program. This way the device manager never needs to allocate more RAM for the menu system than is needed to populate the currently active container.

Right now I'm working on being able to restart the various subsystems independently. This is needed to gracefully handle e.g. device disconnects/reconnects.

BR / Klas



What functionality does this pre-alpha version contain ?

Do you feel it's safe to run it towards real Squeezebox devices or is there any risk destroying something ?

Which hardware types have you tested it against ?

Do you want help testing something specific ?

Mysingen
2009-08-04, 20:41
I have recently implemented seeking, added basic playlist functionality (i.e. the ADD button adds or removes stuff from a special menu directory) and, perhaps more importantly, fixed a bunch of issues that prevented Dwite from running outside OS X. At least now it works just as well in Linux. I have no idea how it behaves under Windows, but maybe someone will tell me? :-)

To clone: "git clone git://github.com/Mysingen/dwite.git"

To run: "./dwite"

Dwite listens to the same port as SqueezeboxServer, so don't run then both at the same time.

BR / Klas

MrSinatra
2009-08-04, 23:33
please forgive my dummy questions...

but is this a plugin or what? like is this something that could let me use winamp to power my SB?

i don't get what this is, (obviously ;)

eLR!C
2009-08-05, 03:37
./dwite

Traceback (most recent call last):
File "core.py", line 17, in <module>
from device import Classic
File "/home/user/bin/dwite/device.py", line 20, in <module>
from resource import Playable, Playlist
ImportError: cannot import name Playable
Goodbye


Seems I miss the "resource.py" file ...

By the way, there's a link in the git archive : is it supposed to be a link to the music folder ?

(I'm impatient to try it out anyway :)

Mysingen
2009-08-05, 09:45
please forgive my dummy questions...

but is this a plugin or what? like is this something that could let me use winamp to power my SB?

i don't get what this is, (obviously ;)

Right, no problem. Dwite is a stab at a completely new server implementation. The goal is to separate the device management from the content management. Dwite still contains a very simple streamer and content manager to let me have some content to play while I'm developing it, but I'm actually fairly close to be able to move that out into separate programs.

Doing all this will let me accomplish two things:
1: Because the device manager does not handle content, it will be able to run on a very limited system. A modern cell phone would be overkill. All content can be served off some Apache box in my basement.
2: It will let me write a plugin for e.g. iTunes to use a squeezebox for output.

Some things are out of scope for me: I don't need multiple device sync and I only have a SqueezeBox Classic, so if you have something else, you might not be able to use Dwite.

BR / Klas

Mysingen
2009-08-05, 09:49
./dwite

Traceback (most recent call last):
File "core.py", line 17, in <module>
from device import Classic
File "/home/user/bin/dwite/device.py", line 20, in <module>
from resource import Playable, Playlist
ImportError: cannot import name Playable
Goodbye


Seems I miss the "resource.py" file ...

By the way, there's a link in the git archive : is it supposed to be a link to the music folder ?

(I'm impatient to try it out anyway :)

Cool... My first bug report :-)

I was clearly a bit too eager to get it out (don't create releases when you're late for work). Anyway I've fixed it and also made sure you can run dwite from any directory (so you don't have to put in symlinks to your media collection like I did).

BR / Klas

MrSinatra
2009-08-05, 12:53
Right, no problem. Dwite is a stab at a completely new server implementation. The goal is to separate the device management from the content management. Dwite still contains a very simple streamer and content manager to let me have some content to play while I'm developing it, but I'm actually fairly close to be able to move that out into separate programs.

Doing all this will let me accomplish two things:
1: Because the device manager does not handle content, it will be able to run on a very limited system. A modern cell phone would be overkill. All content can be served off some Apache box in my basement.
2: It will let me write a plugin for e.g. iTunes to use a squeezebox for output.

Some things are out of scope for me: I don't need multiple device sync and I only have a SqueezeBox Classic, so if you have something else, you might not be able to use Dwite.

BR / Klas

how does itunes use it if it is on an apache box? you have a separate plugin for itunes which finds it on there and routes the music that way?

i very much want a plugin for winamp that works on the same box winamp is on, and all it would do is take the audio as handled by winamp, and power/route it to the SB2 i have.

do you think your project is close to that?

Mysingen
2009-08-05, 21:31
how does itunes use it if it is on an apache box? you have a separate plugin for itunes which finds it on there and routes the music that way?

i very much want a plugin for winamp that works on the same box winamp is on, and all it would do is take the audio as handled by winamp, and power/route it to the SB2 i have.

do you think your project is close to that?

Those would be separate usage scenarios, but they could be supported simultaneously: If Winamp connects, a "Winamp media library" could show up in the menu system on the device. The simplest form of support though would be to let Winamp completely take over the device.

My next few steps include defining the JSON and RPC interfaces to the device manager so that the content manager and streamer can be moved to fully separate programs. After that, it should be possible to write a plugin for Winamp, although it would have to be considered experimental until the exact interface mechanism can be settled.

BR / Klas

Mysingen
2009-08-05, 22:21
how does itunes use it if it is on an apache box? you have a separate plugin for itunes which finds it on there and routes the music that way?

i very much want a plugin for winamp that works on the same box winamp is on, and all it would do is take the audio as handled by winamp, and power/route it to the SB2 i have.

do you think your project is close to that?

Those would be separate usage scenarios, but they could be supported simultaneously: If Winamp connects, a "Winamp media library" could show up in the menu system on the device. The simplest form of support though would be to let Winamp completely take over the device.

My next few steps include defining the JSON and RPC interfaces to the device manager so that the content manager and streamer can be moved to fully separate programs. After that, it should be possible to write a plugin for Winamp, although it would have to be considered experimental until the exact interface mechanism can be settled.

BR / Klas

Dan Sully
2009-08-06, 06:07
* Mysingen shaped the electrons to say...

>My next few steps include defining the JSON and RPC interfaces to the
>device manager so that the content manager and streamer can be moved to
>fully separate programs. After that, it should be possible to write a
>plugin for Winamp, although it would have to be considered experimental
>until the exact interface mechanism can be settled.

Have you considered using Thrift or ProtocolBuffers as the RPC layer?

-D
--
<dsully> please describe web 2.0 to me in 2 sentences or less.
<jwb> you make all the content. they keep all the revenue.

Mysingen
2009-08-06, 16:55
Have you considered using Thrift or ProtocolBuffers as the RPC layer?


I wasn't aware of Thrift before. Protobuf would be more efficient (and I kind of like it for other reasons too), but I'm leaning towards JSON because it has Python standard library support and because I want to be able to make a rewrite in plain C some day. Of course, if I run into performance problems with JSON, I might reconsider. I don't expect to do that though; it will probably take more time for a content manager to query an SQL database for just about anything than it will take to transfer the results to the device manager.

BT / Klas

steve9000
2009-10-29, 17:18
What: A pre-alpha release of the SqueezeBox device manager Dwite.
How: "git clone git://github.com/Mysingen/dwite.git"
Enjoy!

This is fantastic! I wanted to to start this project several months ago
but don't really have the time do complete it.

Your code is pretty readable, I'm amazed that it can do so much with so little and how primitive the player must be to have the server do so much work.

I hacked the code to recognise the boom (ID 10) and could use the remote to browse a folder, but attempting to play produced a really loud bang! No permanent damage luckily.

One thing I found the threading a little tricky to discover the flow
control compared to a single select loop. Perhaps that becomes clearer
when the streaming is broken out?
Also, what's the idea behind overriding __new__? Is it simply to get new
style classes?

Stephen.

MrSinatra
2009-10-29, 17:49
i'll pray daily you guys take pity on non-itunes users and develop a winamp plugin (which i think works with media monkey too)

...on my knees... ;)

Mysingen
2011-06-03, 13:39
I hacked the code to recognise the boom (ID 10) and could use the remote to browse a folder, but attempting to play produced a really loud bang! No permanent damage luckily.

There is definitely something wrong with volume control, also on the regular SqueezeBox. I will look into it of course, but I'm not planning on buying more hardware, so please consider sending in patches.

BR / Klas

Mysingen
2011-06-03, 13:43
Hello everyone. Sorry for simply disappearing for 18 months. "Life is what happens while you make other plans" (John Lennon)

Anyway I am going to work on Dwite this summer. That's all for now, but a push to Github is imminent. The main change is a split of device and content management into fully separate processes that use TCP sockets to communicate.

I also started a project plan here: https://chaff.se/items.html?store=dwite

BR / Klas

Mysingen
2011-06-05, 17:15
* Fixed volume control on the Classic device. (Does it work better on the Boom now?)
* Implemented a graphical volume meter.
* Threw out the Banshee DB content management backend (it never worked properly and was just used for experimentation anyway) and replaced it with a plain file system backend.
* Bugfixes and various quality improvements.

To run:
cd dwite
./dwite &
./cleo
Both must typically restarted if either is stopped.

Mysingen
2011-06-06, 07:50
* Pause/unpause works.
* Documented the STAT message as well as I can. The SBS source comments are clearly wrong in a couple of cases. A few mysteries remain, but don't seem to matter in practice.

Mysingen
2011-06-09, 09:38
Dwite now detects if the device becomes unreachable on the network and allows it to connect again.

Detection is done by polling the device with the "stat" command without any parameters. It will cause the device to answer with a regular STAT message where .event="stat".

Mysingen
2011-06-11, 11:46
The DM now detects lost CM connections and starts waiting for a new one so that the CM can be restarted at will. The menu system also reflects this by removing CM entries and forcing a redraw if the currently focused item is owned by a disconnected CM.

Mysingen
2011-06-19, 02:44
* The CM got a name change. The full package is now Dwite and the Conman.

* Made use of libmagic in Conman to detect file types. The mutagen.File class does not handle non-audio formats correctly and will loop forever on e.g. broken ASF videos. Mutagen is still used for tag and metadata extraction.

* Improved error handling in various places in the Conman.

Mysingen
2011-06-19, 15:25
Implemented support for FLAC files, among other things. Here's a pick of git log entries since last report:

* Made use of libmagic in Conman to detect file types. mutagen.File does not handle non-audio formats correctly and will loop forever on e.g. broken ASF videos.

* The device usually does not send STMo to indicate that the track is running out. Instead the server has to track the device's buffer fill to figure out when it's time to skip to the next track.

* Jumping to arbitrary byte offsets is not allowed as the device gets confused by offsets that are not frame aligned. To get the frame byte offsets, libFLAC was used. (Clone it from https://github.com/dsully/pyflac.git if it is not included in your distribution.)

* The device gets confused if you send strm(start) too quickly after strm(stop). This never happens with MP3 files, but about once in ten for FLAC files. Solved by sleeping 0.1 seconds between the commands.

* The HTTP GET to be passed to the streamer was changed to include the seek value in the URL. This is cleaner and allows for seek points in bookmarks (if I ever get that far).

* Fixed the buggy code that picked the next track to play when the last one played to finish.

* Increased seeking intervals from 1 to 5 seconds. It was just too slow before (from a human interaction point of view).

* Bound the numeric keys on the remote to send the strm(status) command. This makes device degugging a little easier as you can get a STAT message anytime you want.

Mysingen
2011-07-13, 05:01
Some progress got done again:

* Recognize more MP3 magic strings.
* Refactored file recognition.
* Fixed unicode bug in coding/decoding of HTTP GET part of 'strm' command.

git pull git://github.com/Mysingen/dwite.git

Mysingen
2011-07-13, 07:34
Cleaned up the playlist behavior. Before, skipping to next/previous track when playing from the playlist would choose a track from the "real" container item of the track instead of from the playlist. Now it chooses the next/previous playlist item.

erland
2011-07-13, 11:40
The goal is to separate the device management from the content management. Dwite still contains a very simple streamer and content manager to let me have some content to play while I'm developing it, but I'm actually fairly close to be able to move that out into separate programs.

Is there any integration with a content manager yet ?
Is there any API's defined regarding the device management/content management integration ?

Mysingen
2011-07-13, 19:01
Is there any integration with a content manager yet ?
Is there any API's defined regarding the device management/content management integration ?

Sure. Dwite and Conman speak a simple JSON formatted RPC. Conman is just a directory navigator that recognizes MP3 and FLAC files. It answers to 'ls' and HTTP GET calls from the device. It doesn't need to understand much more than that.

Dwite has an RPC function to get keyword dictionaries sent to it. There's also a Search "widget" in the menu root, but it's not quite finished. I'll probably do that one next.

All functionality that is needed from e.g. an Amarok plugin exists but Dwite's RPC wrappers for play/pause/fwd/rwd/seek are missing. The plugin would run the same HTTP server as Conman but implement its own backend to create valid HTTP GET identifiers from the host app's playlist or library.

erland
2011-07-13, 21:52
Sure. Dwite and Conman speak a simple JSON formatted RPC. Conman is just a directory navigator that recognizes MP3 and FLAC files. It answers to 'ls' and HTTP GET calls from the device. It doesn't need to understand much more than that.

The HACKING file refers to a cleo/cleo.py, I'm guessing this is old information ?
The one I should be looking into is backend_fs.py, right ?
Or is there anything more than backend_fs.py and backend.py included in the content manager part ?



Dwite has an RPC function to get keyword dictionaries sent to it. There's also a Search "widget" in the menu root, but it's not quite finished. I'll probably do that one next.

Is this some communication that happens between device manager and content manager ?
Which file is it implemented in ?



All functionality that is needed from e.g. an Amarok plugin exists but Dwite's RPC wrappers for play/pause/fwd/rwd/seek are missing. The plugin would run the same HTTP server as Conman but implement its own backend to create valid HTTP GET identifiers from the host app's playlist or library.
Let's say I'd like to implement my own content manager which is based around my own database.

Note, I realize that your needs are probably a lot more simple than mine, so it might not be suitable to use Dwite for my needs, I'm just asking these questions to be sure. So if you feel something is out of scope and doesn't fit the Dwite design philosophy, just say so, I don't want to be the one that turns it into bigger if you want it to be small. If you are ok with that it gets bigger, I'd of course be willing to help with the work implementing the additions.

Some questions regarding a third party content manager (I'm not speaking python fluently, yet, so sorry if some questions are stupid).

1.
Would I have to do it in Python or can it be any language where I can implement a server communicating via JSON over a socket ?

2.
If I can do it in any language, is there any way to see how the JSON messages actually looks like ?

3.
Does it just communicate over a socket where raw JSON messages are sent or are we talking via HTTP or something similar ?

4.
Is it only the "get_item" and "handle" methods defined in backend.py that have to be implemented ?

5.
Is there or will there be some kind of playlist concept ? In that case, will the current playlist be managed by the device manager or content manager ? As an example how will the process look like when it should switch from the first file to the second file in the playlist automatically after the first file has played to the end.

6.
Does it only support third party content managers that provide file url's today or does it also support a content manager that provides streaming urls from a online streaming service ?

7.
As I understand it only supports the old Squeezebox models today (Classic, Boom, Transporter) but not the new ones (Touch, Radio), is that still correct ?
Are you willing to support all if we assume someone is willing to contribute code and testing work or would you prefer that it stays small and only support the old models ?
I think the new models might communicate completely differently as I don't think they use the SlimProto protocol, not completely sure though.

8.
Is there or will there be some concept to retrieve and show extra metadata provided by the content manager, like information about artist, composer, conductor, release year and similar things ?

9.
Are the functions available on an item decided by the device manager or can the content manager add extra functionality ?
For example, let's say I've browsed to an artist and want to find "Similar artists" or "Find artist on Spotify", would or will that be possible to implement only in the content manager or would the device manager have to implement each of these operations and just talk to content manager to get the actual data ?

10.
Is it possible or will it be possible to have multiple content managers simultaneously ? For example one for local music library, another one for mp3music.com storage, a third one for Spotify, making it possible to browse all libraries by accessing three different menus.

Mysingen
2011-07-14, 04:52
The HACKING file refers to a cleo/cleo.py, I'm guessing this is old information ?
The one I should be looking into is backend_fs.py, right ?

Correct



Is this some communication that happens between device manager and content manager ?
Which file is it implemented in ?

Several:
* protocol.py contains all message definitions. The JsonMessage class is the base for all JSON messaging between DM and CM (both directions). To add a new JSON API, you'd cook a new class based on JsonMessage and handle it in parse_json(). parse_json() is given a raw data blob, calls json.loads() to turn it into a dicationary object and then calls the constructor for a message class depending on what message header it finds.
* The wire.py:JsonWire class is a socket select()-er that is used by the CM and DM to set up JSON RPC. It will call protocol.py: parse_body() (which eventually calls protocol.py: parse_json()) and put the resulting Message instance on a queue. You don't have to change anything in wire.py to add a new JSON API.
* The other end of the queue is handled by the DM and CM main loops. For the CM, that's conman.py:Conman:run(). For the DM, that's device.py:Classic:run(). You would have to do something to recognize new message classes here and act on them. Conman does very little and just passes the Ls message on to the backend (from the DM wire), and passes Listing messages on to the DM wire (from the backend).

If you write your own backend to be used from Conman, you *must* implement it in its own thread (just copy the structure of backend_fs.py). This is to make sure that the CM main loop stays responsive. There's quite a bit of threading in Dwite and Conman; it's all there to make sure they will be able to gracefully handle the long RPC roundtrips that are typical for really huge backend databases.


Let's say I'd like to implement my own content manager which is based around my own database.

Note, I realize that your needs are probably a lot more simple than mine, so it might not be suitable to use Dwite for my needs, I'm just asking these questions to be sure. So if you feel something is out of scope and doesn't fit the Dwite design philosophy, just say so, I don't want to be the one that turns it into bigger if you want it to be small. If you are ok with that it gets bigger, I'd of course be willing to help with the work implementing the additions.

Your modesty is noted : )

I'd be fine with Dwite implementing a large API to support all kinds of CM functionality as long as it's useful and makes sense. I try to write tight code, but the big/small nomenclature is really about limiting the responsibilities of the major components (DM & CM).



1.
Would I have to do it in Python or can it be any language where I can implement a server communicating via JSON over a socket ?

2.
If I can do it in any language, is there any way to see how the JSON messages actually looks like ?


See protocol discussion above. You can certainly implement a CM in any language you like, but of course then you can't share protocol.py and wire.py with Dwite. What's your preferred language?



3.
Does it just communicate over a socket where raw JSON messages are sent or are we talking via HTTP or something similar ?


Raw sockets. The only quirk is that the messages are prefixed by a string "JSON%i" where %i is the size of the message body expressed as a small-endian uint32. Devices use the same header structure, so it was easier that way. I'm perhaps not entirely happy about it now, but it's so simple it doesn't matter much. protocol.py:JsonMessage(Message):serialize() is all you need to see the format of the complete message you will get from the DM.

Take an extra look at protocol.py:Hail(JsonMessage). It is used by the CM to register itself with the DM. Without it, the DM does not know the name of the CM or it's streamer's address/port.



4.
Is it only the "get_item" and "handle" methods defined in backend.py that have to be implemented ?

Yes. handle() is supposed to act on JsonMessage objects and post something back on the outgoing queue. get_item() is only used by the streamer to translate GUID's (seen in HTTP GET requests) to local file paths that can be opened, read and streamed to the device.

The GUID's can be anything on string format. The pattern is that DM asks CM to list containers and gets lists of GUID/label objects back (which can be used to render the menu system). If the user presses "play" on an item, the GUID for that item will be used in the HTTP GET request sent by the DM to the device (which passes it on to the CM). This way the DM doesn't have to know anything about the internal structure of a CM's library.



5.
Is there or will there be some kind of playlist concept ? In that case, will the current playlist be managed by the device manager or content manager ? As an example how will the process look like when it should switch from the first file to the second file in the playlist automatically after the first file has played to the end.


The DM owns the global playlist. Entries in it can point into any number of CM's. There's no RPC API to manipulate it from a CM. Some planning would be good before changing that.



6.
Does it only support third party content managers that provide file url's today or does it also support a content manager that provides streaming urls from a online streaming service ?


To support remote streamers, you'd write a CM that merely tells the DM about address/port and GUID's to put in the HTTP request. You could probably make it play something with very little effort. Handling metadata from the streamer would be more work, for sure. I haven't done anything yet to help with that.



7.
As I understand it only supports the old Squeezebox models today (Classic, Boom, Transporter) but not the new ones (Touch, Radio), is that still correct ?
Are you willing to support all if we assume someone is willing to contribute code and testing work or would you prefer that it stays small and only support the old models ?
I think the new models might communicate completely differently as I don't think they use the SlimProto protocol, not completely sure though.


Dwite will focus on devices that can be controlled through SlimProto. My understanding is the newer models still interface with SqueezeBox Server over SlimProto but I have no idea how that communication is initiated. My personal interest is only with the older models.



8.
Is there or will there be some concept to retrieve and show extra metadata provided by the content manager, like information about artist, composer, conductor, release year and similar things ?


Yes, there is already. The idea is that the DM treats anything listed by the CM as a container for more entries. Currently you can't press RIGHT on the remote when an MP3 is focused (only PLAY and ADD work), but if you could it would then ask the CM for a metadata listing for the track and use the data to populate a new menu directory under the track. Similar to how SqueezeBox Server works.

The DM can have any number of specialty "widgets" that list containers by GUID and use the results to draw or populate the widget in various ways. menu.py:Playlist(Tree) and menu.py:Searcher(Tree) are examples of that. Please discuss your implementation strategy with me before you begin (if you want to add a new widget).



9.
Are the functions available on an item decided by the device manager or can the content manager add extra functionality ?
For example, let's say I've browsed to an artist and want to find "Similar artists" or "Find artist on Spotify", would or will that be possible to implement only in the content manager or would the device manager have to implement each of these operations and just talk to content manager to get the actual data ?


The CM could include actions like you describe in e.g. the metadata listing for a track. Entering the "similar artists" container from the DM should then cause the CM to produce the result as a Listing object.

Listing objects can currently classify items as directories, plain files and audio tracks. E.g. {'kind':'mp3'}. Look at backend_fs.py:FileSystem(Backend):_get_children() for the structure.

If richer CM API's are needed, you should think about:
* New 'kind' values to use in Listing objects,
* Default DM behavior for combinations of item kinds and remote buttons.
* New protocol classes to express new behavior.
* If it is reasonable to require *all* CM's to implement the new API.

The last bullet is especially important because CM's will never be allowed to inject arbitrary code in the DM to dynamically load new functionality.



10.
Is it possible or will it be possible to have multiple content managers simultaneously ? For example one for local music library, another one for mp3music.com storage, a third one for Spotify, making it possible to browse all libraries by accessing three different menus.

Yes, but I don't know if it works properly. So far I've had enough to do with just one CM to work against, but what you describe is exactly how it's supposed to work.

Mysingen
2011-07-14, 06:39
You also asked about searching but I ended up writing something about general communication instead. Here's about searchable keyword dictionaries:

There is a DM RPC that CM's can use to publish keyword dictionaries. These are consumed by the menu.py:Searcher class which also provides a root menu item. When the item is entered, the user is presented with a special widget that lets you type in T9 style and be presented with possible completions from the published keyword dictionary. The user can compose a sentence this way.

The CM does not implement a search(sentence) RPC yet, but you can probably see where this is going. The reason for doing it through keyword dictionaries is that it is a lot faster to type in T9 than lettering out each word more precisely. The CM is of course able to trivially know all usable search terms, so the search is much more likely to yield a result if the user can only choose from published keywords.

Searching is high on my todo list, so expect some progress there fairly soon.

erland
2011-07-14, 10:24
You can certainly implement a CM in any language you like, but of course then you can't share protocol.py and wire.py with Dwite. What's your preferred language?

The content manager in question is written in Java but I'll have to look into it a bit more, it might make sense to have a thin Python wrapper around it.

Before doing that I have to look into all your answers as see if Dwite fit my needs, thanks for all the answers. Dwite sounds interesting because I'm really focused at the content manager part and would like to use an existing device manager. At the moment, we are using SBS as device manager but it contains a lot of other things which we really don't need.

However, the critical issue is that the solution I'm working on must have support for the new players (Radio and Touch), so I have have to dig a bit and see how that fits into Dwite. I'm not sure the new players use SlimProto, at least I know for sure they don't use SlimProto for menu navigation as that uses completely separate interface (JSON/CLI) on the Touch/Radio.

Additionally, I also need support for third party controllers which will be used to do the browsing, how would that fit into Dwite ? Does Dwite assume that all menu navigation/control is done through the device manager part via a IR remote ?

Mysingen
2011-07-14, 10:25
Pressing ADD on a directory works: All playable tracks from the directory are added recursively to the playlist.

Mysingen
2011-07-14, 18:27
I also need support for third party controllers which will be used to do the browsing, how would that fit into Dwite ? Does Dwite assume that all menu navigation/control is done through the device manager part via a IR remote ?

The DM controls the device screen and receives all IR messages. I.e. it owns all contact with the actual device, except that CM's answer HTTP GET requests.

On the other hand it isn't hard to pass IR messages on to an external UI thing, or to add RPC wrappers for most DM functionality that might be needed in same UI thing.

If you mean third party *remotes*, then you'd just map the new IR codes to existing messages. Check tactile.py

Mysingen
2011-07-14, 18:46
However, the critical issue is that the solution I'm working on must have support for the new players (Radio and Touch), so I have have to dig a bit and see how that fits into Dwite. I'm not sure the new players use SlimProto, at least I know for sure they don't use SlimProto for menu navigation as that uses completely separate interface (JSON/CLI) on the Touch/Radio.


I could be completely wrong, BUT my understanding is this:

* There is still an old-style SlimProto device that only does playback and sound settings in the newer models. This works just like before, including the HTTP GET part.
* The UI parts of the newer models use some JSON RPC API that I know nothing about, except SBS doesn't appear to be using it. Maybe the calls are made in the other direction? I.e. some calls to the JSON API will trigger further calls to SBS.

Let me know when you find out :-)

erland
2011-07-14, 20:56
The DM controls the device screen and receives all IR messages. I.e. it owns all contact with the actual device, except that CM's answer HTTP GET requests.

On the other hand it isn't hard to pass IR messages on to an external UI thing, or to add RPC wrappers for most DM functionality that might be needed in same UI thing.

If you mean third party *remotes*, then you'd just map the new IR codes to existing messages. Check tactile.py

I'm not talking about IR remotes, I'm taking about smart remotes which have a display of their own and communicates with the server over Wifi using JSON, for example an iPhone or Android app or even a Squeezebox Controller (the remote in the Duet package).

I guess they could kind of be seen as a complete device (with or without playback), the difference is just that they are able to also control other devices besides themselves but show the menus on their own screen. This is the thing that makes the older models different from the newer Squeezebox models so I can understand why Dwite isn't prepared for it at the moment.

Is it possible for an external application to send a PLAY or ADD command to the device manager and pass a parameter specifying a guid/url with the command ?
If that's possible, it should be possible for these smart remotes to communicate with the content manager and just pass PLAY/ADD commands with a guid/url to the device manager when they want to trigger something to be played.
If that isn't possible and you first have to navigate the device manager to the right item in the menu structure to select a guid/url and then send a PLAY/ADD command to it, I'm not sure it fits the newer Squeezebox models.

Mysingen
2011-07-15, 06:01
I'm not talking about IR remotes, I'm taking about smart remotes which have a display of their own and communicates with the server over Wifi using JSON, for example an iPhone or Android app or even a Squeezebox Controller (the remote in the Duet package).

Is it possible for an external application to send a PLAY or ADD command to the device manager and pass a parameter specifying a guid/url with the command ?


PLAY and ADD expect an object which is trivial to create if you have the following information to put in:

* guid (an arbitrary but unique string)
* label (a string to print on the display)
* kind ('mp3' or 'flac' for playback. 'dir' can also be ADDed)
* size (full payload size in bytes)
* duration (track length in milliseconds)

Dwite then checks in it's CM registry for the address/port of the streamer to complete the request. This isn't perfectly prepared for URL based PLAY/ADD, but very little is missing. I could add an RPC for PLAY with the following prototype (ADD would be very similar):

{
"method":"play",
"parameters":{"url":<string>, "kind":<string>, "pretty":{"banner":<string>, "artist":<string>, "album":<string>, "track":<string>, et.c. ...}, "size":<integer>, "duration":<integer>, "seek":<integer>}}
}

Only "url" and "kind" would be mandatory to specify.

Two forms of the "url" parameter can be accepted without too much work:
* The URL specifies "address : port / path" of the stream, or
* The URL is a GUID prefixed with an internal scheme and the name of the CM, e.g. "guid://<registered CM name>/<GUID>".

Today, Dwite does not perform URL parsing to find the address/port of the streamer so the second form is less work me to implement. For internet radio it may be necessary to support both forms.

erland
2011-07-15, 09:15
Today, Dwite does not perform URL parsing to find the address/port of the streamer so the second form is less work me to implement. For internet radio it may be necessary to support both forms.

I'll get back to you after I've taken a closer look at Dwite.

Mysingen
2011-07-15, 10:37
Changed the implementation details of all JsonMessage based classes to make the protocol more expressive. This had some far reaching effects on other code, but now all messages have trackable GUID's and the format is able to handle chunking of payloads into smaller messages.

The change was also necessary to enable RPC wrappers around Dwite functionality such as ADD and PLAY.

The format is final. Additional parameters may be added in the future, but no existing ones will change.

Mysingen
2011-07-16, 14:58
Added a DM RPC to control playback, plus a skeleton CLI to prove that it works.

Example use:

./cli play "cm://File system/101017/download-helicopter-2009/07-Radio_Silence.flac" 3000

The last parameter is the seek value in milliseconds. I.e. playback starts 3 seconds into the track.

Mysingen
2011-07-16, 20:37
Cleanup of CM (un)registration. It now works to have any number of CM's and user interfaces attached to Dwite.

Mysingen
2011-07-17, 15:03
* Lots of stability fixes.

* Reduced hardwired logging in a number of places.

* Implemented JsonMessage.respond() that automatically finds the right
wire to send a JsonResult message on. Used this newfound elegance in
lots of places.

* Added sub-command "add" to the CLI. It adds single items and whole
directories to the playlist, just like the remote's + button.

* Found a reliable way to see if a socket connection is broken: Zero
bytes handled in send()/revc() even though select() indicates that
writing/reading is possible.

* Collected all message handler callbacks in one place: The Message-
Register class in dwite.py. One has to be careful when executing
the callbacks though to not touch object members that belong to a
diffferent thread. On the plus side, the callbacks lets me do all
the various forms of message handling in the right place, instead
of cramming it all into the Device classes, where it were collecting
before.

* Closed a couple of bugs in the (un)registration of DM's, CM's and
UI's.

* Removed all direct CM interaction from menu.py because it got too
hard to do correctly with message callbacks. It probably wasn't the
best idea to begin with anyway.

* Added a new state STOPPING to the Wire class. Just shutting it down
hard would frequently prevent queued up messages from being sent.
There is still a way to shut down the hard way.

Mysingen
2011-07-18, 17:47
* Persistent settings for device managers based on device MAC address.
The Volume and Display classes have methods to dump default and actual
settings while the Device classes take care of loading and saving.

* Stability improvements.

Settings are saved under ~/.dwite

Mysingen
2011-07-19, 09:49
Load/save the global playlist on DM start/shutdown.

The items are loaded without checking if the owning CM exists. The items simply won't play if the CM still has not registered with the DM in time for playback.

Mysingen
2011-07-21, 07:22
* Files with names that are neither ASCII nor UTF8 can now be played correctly. The filename is converted to an escaped format and then un-escaped by the streamer.

* Fixed bugs in Render classes and set all classes to inherit from object.

* NOW PLAYING refocuses menu on currently playing item and activates a mode that causes the menu to refocus when the track plays out and a new one starts. Broswing away from the item deactivates the mode.

* Changed CM response to Ls to include the listed item, and added a parameter to list the parent of the specified GUID instead of the GUID's actual item. This is necessary to implement browsing from menu leaf nodes when the menu is refocused without going through items from the menu root. I.e. when RPC is used to start playback, followed by pressing NOW PLAYING. The focused item in the menu will then not have its parent set and we can't *guess* the parent's GUID because the CM is free to compose the GUID's in completely arbitrary ways. The only recourse is then to ask the CM for the parent of a GUID that you *do* know; in this case the currently playing item.

* If the DM host machine is suspended while the DM is running, the TCP connection with the device will be reset when the host machine resumes. Then stopping the DM and spawning a new one (instead of restarting it in accept() mode as before) allows the DM and device to negotiate a new connection. I.e. the system recovers from host suspension.

* Stability improvements.

* Unicode bugfixes.

Mysingen
2011-07-21, 07:53
Here: http://forums.slimdevices.com/showthread.php?t=88922

I just realized that the developers' forum is supposed to be about Squeezebox Server and that my little project fits better in the DIY forum. I sure as **** won't *leave* though as I will have more questions about device interaction from time to time. All the same, a big thank you to SlimDevices and Logitech for making an open product that is fun to work with.