PDA

View Full Version : Mensa Test - SC encoding - Foreign Chars and some MIP :s



cparker
2008-07-23, 07:34
Hi guys,

My confusion continues with foreign characters and SC's handling of them.
So I thought I would document what I've tried and hopefully someone can crack it and we can all learn from it. :)

Below is my example; *Running on XP*

The actual Filename and Path; (Latin1?)
C:\Documents and Settings\u\My Documents\My Music\720į PresÍnts Two Revolutions (Disc 1 of 2)\08-Seba _ The Lost Key.mp3

My return from MIP arrives as; (utf8 encoded?)
C:\Documents and Settings\u\My Documents\My Music\720¬į Pres√™nts Two Revolutions (Disc 1 of 2)\08-Seba _ The Lost Key.mp3

I have tried pushing this to SC and it fails, so tried;

Run these two lines of code;
my $enc = Slim::Utils::Unicode::encodingFromString($element) ;
$element = Slim::Utils::Unicode::utf8decode_guess($element, $enc);

Which gives me my original filename/path; (so we are Latin1?)
C:\Documents and Settings\u\My Documents\My Music\720į PresÍnts Two Revolutions (Disc 1 of 2)\08-Seba _ The Lost Key.mp3

Now from my understanding this is now Latin1 format, but if I push this to SC via the $client->execute(["playlist", "add", $element]); command it fails (though confusingly it does fire a playlist add/done return msg)

Now hand coded like this and it works;
$element=('file:///C:/Documents%20and%20Settings/u/My%20Documents/My%20Music/720%B0%20Pres%EAnts%20Two%20Revolutions%20(Disc%20 1%20of%202)/08-Seba%20_%20The%20Lost%20Key.mp3');
$client->execute(["playlist", "add", $element]);

So what exactly is SC looking for here? The hand coded line seems to be URL encoded?? I would have thought that straight utf8 would have worked fine?

Thanks in advance

radish
2008-07-23, 15:34
The CLI docs state that parameters must all be URL encoded, so yes, you need to URL encode them :)

cparker
2008-07-24, 15:16
Yeah I appreciate that however look at this;

URL encoded into scalar $element - fails to load using this command;
$client->execute(['playlist', 'append', $element]); # Our new track add to SC
C:/Documents%20and%20Settings/u/My%20Documents/My%20Music/Bj%F6rk/Debut/04-There%27s%20More%20To%20Life%20Than%20This.mp3

[08-07-24 23:03:38.9645] Plugins::SugarCube::Plugin::addncleanup (613) #################
[08-07-24 23:03:38.9758] Plugins::SugarCube::Plugin::addncleanup (614) file:///C:/Documents%20and%20Settings/u/My%20Documents/My%20Music/Bj%F6rk/Debut/04-There%27s%20More%20To%20Life%20Than%20This.mp3
[08-07-24 23:03:39.2135] Plugins::SugarCube::Plugin::addncleanup (616) #################
[08-07-24 23:03:39.4316] Plugins::SugarCube::Plugin::commandCallback (333) SugarCube: received command: playlist append
[08-07-24 23:03:39.5527] Plugins::SugarCube::Plugin::commandCallback (333) SugarCube: received command: playlist load_done
[08-07-24 23:03:55.6181] Slim::Utils::Misc::msg (1350) Warning: [23:03:55.6101] Terminating on signal SIGHUP(1)

Fails so stop and copy out of the log! what failed into a scalar like so; (NOTE file:/// gets added by the routine)
$element =('C:/Documents%20and%20Settings/u/My%20Documents/My%20Music/Bj%F6rk/Debut/04-There%27s%20More%20To%20Life%20Than%20This.mp3');

[08-07-24 23:05:59.7750] Plugins::SugarCube::Plugin::addncleanup (614) #################
[08-07-24 23:05:59.7871] Plugins::SugarCube::Plugin::addncleanup (615) file:///C:/Documents%20and%20Settings/u/My%20Documents/My%20Music/Bj%F6rk/Debut/04-There%27s%20More%20To%20Life%20Than%20This.mp3
[08-07-24 23:06:02.6480] Plugins::SugarCube::Plugin::addncleanup (618) #################
[08-07-24 23:06:02.9564] Plugins::SugarCube::Plugin::commandCallback (333) SugarCube: received command: playlist append
[08-07-24 23:06:03.0700] Plugins::SugarCube::Plugin::commandCallback (333) SugarCube: received command: playlist load_done
[08-07-24 23:06:12.6821] Slim::Utils::Misc::msg (1350) Warning: [23:06:12.6723] Terminating on signal SIGHUP(1)

This does load!

Like what is the difference? I've tried, whitespace trimming, chomp any newlines, even appending a newline.. Tried playlist add and append, No difference, just wont load!

What makes the client->execute work correctly?

Right fed up :(

Fred
2008-07-25, 06:26
In a nutshell, due to the multi-platform-nesss of SqueezeCenter, Perl smarts, and maybe sloppy programming, there is serious voodoo surrounding PATHs in SqueezeCenter. It's just useless to try to invent the way they are encoded. In particular, they are not utf8, it kind of depends on the underlying file system.

The only way it works reliably is to feed back to SC a path it has given you. Sorry.

Fred

cparker
2008-07-25, 09:22
Hi Fred,

Yeah it certainly does seem like it, I dropped back to SC7.01 which made no difference. There definately seems to be an issue with the playlist add command when trying to add foreign characters that have been URL encoded.

I even wrote my own encoding routine and double-checked its output against a URL encoding website before injecting SC to no avail.

I do wonder why SC doesnt just support UTF8???

Seems this isnt actually fixable at the moment, so I will just have to drop support for foreign characters until such time as it does work or someone has an idea, I noticed there are a few threads around foreign character encoding issues.

Hey ho... :s

Thanks for your comments, glad its not just me going mad... well..

tamanaco
2008-09-16, 19:05
Is anyone in the SC development side looking at the foreign characters handling issued raised by cparker? His SugarCube plugin is a very well received enhancement to the SC-SB music listening experience. It is a pity the SugarCube developer is dropping support for foreign characters because SC lacks a function that "appears" to be very fundamental. I'm unable to take advantage of this wonderful plugin just because my music library is tagged using foreign characters. Can someone take a second look this?

Thanks...

mherger
2008-09-16, 23:35
> Is anyone in the SC development side looking at the foreign characters
> handling issued raised by cparker?

The issue must be in the plugins. Mixing using the MusicIP plugin is
working fine for me, whether a file or path has non-latin characters or
not (Linux, OSX, Windows).

Michael

tamanaco
2008-09-17, 05:24
> Is anyone in the SC development side looking at the foreign characters
> handling issued raised by cparker?

The issue must be in the plugins. Mixing using the MusicIP plugin is
working fine for me, whether a file or path has non-latin characters or
not (Linux, OSX, Windows).

Michael

Michael, can you clarify something for me... I'm not that sharp? It "appears" to me that when MusicIP submits a mix to SC it actually submits a "Playlist" to be played... Correct? Is SugarCube also submitting multiple Playlists each with a "single" track to be played next or is it submitting a different type of request? It's bit odd that it fails when it submits tracks with foreign characters, but works just fine when tracks have none. Again, I'm "assuming" that it's using the same submission method in both cases. Something is not Kosher here. The ball can not be on both sides of the fence... Baseball analogy... The Mets are killing me.

tamanaco
2008-09-18, 12:24
Michael, a follow up question, is it possible for you to share with the SugarCube developer the character string (message/command) submitted to SC that represent a MusicIP mix that works and that contains foreign characters? It "could" be something trivial that is being overlooked.

Thanks...

mherger
2008-09-23, 00:28
> Michael, a follow up question, is it possible for you to share with the
> SugarCube developer the character string (message/command) submitted to
> SC that represent of MusicIP mix that works and that contains foreign
> characters? It "could" be something trivial that is being overlooked.

Just enable debugging for plugin.musicip and you'll get not only the urls called, but also the answers by MIP:

http://localhost:10002/api/mix?song%3D%2FUsers%2Fmh%2FMusic%2FiTunes%2FiTunes %20Music%2FAdrian%20Stern%2FS'Blaue%20vom%20Himmel %2F10%20Ma%CC%88ngisch.mp3&variety=2&rejectsize=0&mixgenre=0&style=0&sizetype=tracks&size=12

The interesting part is the filename: Ma%CC%88ngisch.mp3 (Mšngisch.mp3)

This returns (among others) "..../09 Hey Novšmber.mp3"

Tested on OSX 10.5, MIP 1.8

--

Michael

cparker
2008-10-07, 06:20
Hi Michael

I fixed the issue with calling MIP with foreign characters, however SC seems to not like being fed path/filenames here is some POC code which shows how the Add Track fails when feeding SC a path and filename with foreign characters.

Code;

my $client = shift;
my $element = shift;

$log->debug("\n\n\nAddTrack-$element\n\n\n");
my $url = Slim::Utils::Misc::fileURLFromPath($element);
$log->debug("\n\n\nAddTrack-$url\n\n\n");
my $track = Slim::Schema->objectForUrl({'url' => $url});
$log->debug("\n\n\nTrack-$track\n\n\n");

$client->execute(["playlist", "add", $track]); # Our new track add to SC

.....

Here is the debug output when Track has a foreign character;

AddTrack-\\Oldskool\charlie\Dev-Music\My Music\720¬į Pres√™nts Two Revolutions (Disc 1 of 2)\Tim√ī Mass - To G√ęt Down.mp3

[08-10-07 14:02:51.7494] Plugins::SugarCube::Plugin::addncleanup (713)

AddTrack-file:////Oldskool/charlie/Dev-Music/My%20Music/720%C2%B0%20Pres%C3%AAnts%20Two%20Revolutions%20(D isc%201%20of%202)/Tim%C3%B4%20Mass%20-%20To%20G%C3%ABt%20Down.mp3

[08-10-07 14:02:51.7845] Plugins::SugarCube::Plugin::addncleanup (715)

Track-

(TRACK output is empty and nothing happens)

######################
Here is the debug output when Track DOES NOT have a foreign character;

AddTrack-\\Oldskool\charlie\Dev-Music\My Music\Cruel And Unusual Disc 1\05-Hideous (Bse and Noisia).mp3

[08-10-07 14:03:57.0269] Plugins::SugarCube::Plugin::addncleanup (713)

AddTrack-file:////Oldskool/charlie/Dev-Music/My%20Music/Cruel%20And%20Unusual%20Disc%201/05-Hideous%20(Bse%20and%20Noisia).mp3

[08-10-07 14:03:57.0935] Plugins::SugarCube::Plugin::addncleanup (715)

Track-Slim::Schema::Track=HASH(0x8d671cc)

Also receive command callbacks indicating that the track was added.

[08-10-07 14:03:57.5497] Plugins::SugarCube::Plugin::commandCallback (372) SugarCube: received command: playlist add
[08-10-07 14:03:58.2022] Plugins::SugarCube::Plugin::commandCallback (372) SugarCube: received command: playlist addtracks


Any ideas or suggestions?

Thanks

mherger
2008-10-08, 00:13
> I fixed the issue with calling MIP with foreign characters, however SC
> seems to not like being fed path/filenames

?!? Did you fix it or not?

> my $url = Slim::Utils::Misc::fileURLFromPath($element);
> $log->debug("\n\n\nAddTrack-$url\n\n\n");

Did you have a play with the various methods in Slim::Utils::Unicode
(utf8decode_locale() etc.)?

Michael

cparker
2008-10-09, 10:08
> I fixed the issue with calling MIP with foreign characters, however SC seems to not like being fed path/filenames

?!? Did you fix it or not?

Yep, I can take the current playing track encode it and pass it to MIP (including foreign chars) and get a valid response back from MIP.
The $element in the code is the track I now want to give SC to add to the playlist.

My initial thoughts were that its already in utf8 format and I wouldnt need any conversion and that I could just pass it to SC, but it doesnt seem to work when passed via $client->execute(["playlist", "add", $track]);




> my $url = Slim::Utils::Misc::fileURLFromPath($element);
> $log->debug("\n\n\nAddTrack-$url\n\n\n");

Did you have a play with the various methods in Slim::Utils::Unicode
(utf8decode_locale() etc.)?

Michael

Yeah trust me, I've been through every combination I could think of without success, including writing my own encoding routine!

I'm wondering if its possible to use $client->execute(["playlist", "add", $track]); with foreign characters, surely it must be...

Any Proof of Concept code gratefully received!

Confused as always ;)

tamanaco
2008-10-13, 16:36
Glad to see that there is "some" noise in this thread. Now that SugarCube works in conjuction with erland's plugins, it would be nice to see it working with my Music Libary and not feel left out of the fun.