How to decode CLI escaped strings?

[AndrewFG]

The CLI documentation states that the CLI delivers strings in "percent escaped Utf-8" format. But I am having trouble to decode some of those strings.

Particulary taxing is the songinfo url tag that seems to have double encoding on it. For example one url in my library is returned with a string in it %25%E9 which _should_ be the UTF-8 encoded version of the é (accented e acute) character. It looks like the CLI is trying to deliver the $E9 character; but $E9 is actually the ANSI code for é, and the UTF-8 code for é is actually $C3 $A9.

Is the CLI escaping broken? (So it seems to me...) Or alternatively can somebody please provide with the algorithm for un-escaping URLs delivered by the CLI ??

[fredgoodman]

Just out of curiousity --- do your results depend on the version of SBS?

See http://bugs.slimdevices.com/show_bug.cgi?id=16319.

[AndrewFG]

Just out of curiousity --- do your results depend on the version of SBS?

See http://bugs.slimdevices.com/show_bug.cgi?id=16319.

I did want to change SBS versions, so I have not tried that.

However, it is possible that my issue is the same as the one in the bug.

It looks as if the urls are encoded as (something like) "double percent escaped ANSI" strings rather than "percent escaped UTF-8" strings that they should be according to the specifications...

[pippin]

What OS are you on? On Linux I believe it depends on whether you use a locale that supports utf8, may be the same under Windows.

[fredgoodman]

On further thought, I think my bug report is probably irrelevant, although it does indicate that the SBS developers got themselves into a muddle with unicode support. I have seen other indications in these forums that unicode support is problematic and that it is being worked on.

Your results do not necessarily indicate something wrong. The CLI request for a url gives output that depends on the OS being used, not necessarily utf-8. Different OS's encode unicode differently.

For example, OS X on mac, which I use, employs the "normalized D form". In this form, e with acute accent is given as 65 cc 81, where 65 is the code for ordinary e, and the sequence cc 81 encodes the acute accent. For me, a CLI path request gives "e%25CC%2581" for e with acute accent.

You can check if you are getting the correct % escape encoded url by feeding it back to the CLI with

<player> playlist play THE_URL

If it is correct, you can probably see by looking at enough examples how to decode it, assuming you just have to deal with a few accented characters and umlauts.

It is possible, however, that your work could be undone by the developers' eventual fix for unicode support.

[peterw]

What OS are you on? On Linux I believe it depends on whether you use a locale that supports utf8, may be the same under Windows.

So how do you detect the encoding being used by SBS? Is there some CLI command that tells you explicitly? Or does iPeng just use the JSON API so the HTTP Response headers tell you what to expect?

[pippin]

It doesn't, there's no way.
You need to make sure the server's OS and locale supports UTF8 or you'll get garbage in any case.

There's a section on encodings in the CLI doc.

[AndrewFG]

The CLI documentation says the following:

Extended queries accepting the "charset" parameter normally return field values in utf8, Perl's lenient version of UTF-8. The value of this parameter is sent as is to Perl's Encode plumbing. As a consequence, any encoding known to Perl is valid, for example "iso-8859-1" (the web standard), "cp437" (MS-DOS), etc. More exotic choices like "UTF-16" or "shiftjis" are possible. Note that the server will crash if the encoding is unknown.

And:

All paths and URLs, for example the ones returned by the query "songinfo" are double URL escaped. To get a useable path (that you can use with your file system), you will need to unescape the field twice. Also note the URLs are not translated into Unicode (or into any other charset requested).

(So clearly the URL is _not_ UTF-8 encoded...)

However, it is not exactly clear to me what "double URL escaped" means, and what algorithm one should use to convert it to a normal (unicode) string. => Any help would be appreciated.

[pippin]

The CLI documentation says the following:


And:


(So clearly the URL is _not_ UTF-8 encoded...)

However, it is not exactly clear to me what "double URL escaped" means, and what algorithm one should use to convert it to a normal (unicode) string. => Any help would be appreciated.

URL encoded means that anything that's not allowed in a url is replaced by some "%xx" code.

That has nothing to do with the UTF-8 encoding. UTF-8 is a character encoding. Character encoding is something else, it tells you how you find a numerical representation for a certain character. Your characters are ALWAYS encoded in some way, be it ASCII, ANSI with codepage whathever or UTF-8. SBS tries to use UTF-8.

[AndrewFG]

That has nothing to do with the UTF-8 encoding. UTF-8 is a character encoding. Character encoding is something else, it tells you how you find a numerical representation for a certain character. Your characters are ALWAYS encoded in some way, be it ASCII, ANSI with codepage whathever or UTF-8.

We hold these truths to be self evident...

But thanks anyway...