PDA

View Full Version : Problem with Accented Characters



IPGuy
2005-07-07, 08:04
I have a SB2 and am running the July 6th evening build of 6.1b slimserver on a Windows XP SP2 machine. I also have installed AlienBBC 0.97 and MPlayer 1.0pre7. I am using the iTunes library as the source of music & playlists.

While it has taken some time I have gotten ALMOST everything working great, I can play MP3s and AACs from the iTunes Library, I can see and use the iTunes playlists, I can play internet radio selected from the iTunes Radio stations, I can play internet streams of MP3, ASX and real audio.

My last problem is the following:

Any audio file that has an accented character (for example á, é or ä) anywhere in the path (i.e., "D:\My_Music\iTunes\Smetana_Dvorak\Smetana_ Má vlast_Dvorak_ Symphony No.4") shows up as "EMPTY" when browsing or searching; and crashes the server with a "Problem: Can't open file" if the song is selected from a playlist.

It would be a huge hassle to go through all 5000 songs to find any accented characters and then change the names and then reload the iTunes and SlimServer music lists.

Searching the forum suggests others have seen this problem. Is there a solution short of renaming all the folders/files? Is Slim Devices aware of this problem? Is this something that could be fixed in one of the upcoming evening releases or is there a bigger problem here?

Thanks for you help!

Dan Sully
2005-07-07, 08:51
* IPGuy shaped the electrons to say...

>My last problem is the following:
>
>Any audio file that has an accented character (for example �, � or �)
>anywhere in the path (i.e., "D:\My_Music\iTunes\Smetana_Dvorak\Smetana_
>M� vlast_Dvorak_ Symphony No.4") shows up as "EMPTY" when browsing or
>searching; and crashes the server with a "Problem: Can't open file" if
>the song is selected from a playlist.
>
>It would be a huge hassle to go through all 5000 songs to find any
>accented characters and then change the names and then reload the
>iTunes and SlimServer music lists.
>
>Searching the forum suggests others have seen this problem. Is there a
>solution short of renaming all the folders/files? Is Slim Devices aware
>of this problem? Is this something that could be fixed in one of the
>upcoming evening releases or is there a bigger problem here?

IPGuy - can you wipe & rescan your database?

What method are you using for browsing? 'Browse Music Folder', or one of the
other methods?

Thanks.

-D
--
<iNoah> my pdp goes to 11.

IPGuy
2005-07-07, 09:03
I actually uninstalled the 6.02 build, deleted the program folder, installed the 7/6 build clean, rescaned (including the erase existing data option) the db four times while resolving other issues.

Dan Sully
2005-07-07, 09:08
* IPGuy shaped the electrons to say...

>I actually uninstalled the 6.02 build, deleted the program folder,
>installed the 7/6 build clean, rescaned (including the erase existing
>data option) the db four times while resolving other issues.

Ok - could you please open a bug, and attach your iTunes.xml file (zipped),
and one of the problem audio files (also zipped to preserve the accented
filename / directory).

http://bugs.slimdevices.com/

Did you use iTunes to originally rip one of the problem CDs?

Thanks.

-D
--
Ya gotta love UNIX, where else do you wonder whether
you can kill a zombie spawned by a daemon's fork?

IPGuy
2005-07-07, 09:26
I will send what you requested from home tonight.

As for how they were ripped, some were ripped with iTunes and others with MusicMatch.

mikerob
2005-07-08, 02:08
oops, I replied to this message in the General Forum and didn't notice it was covered in the Beta forum.

To repeat my reply - I had a similar problem and raised bug http://bugs.slimdevices.com/show_bug.cgi?id=1709

I tried the 2005-07-07 nightly yesterday and this seems to have fixed the problem with the exception of one particular album title which is still garbled on SB and browser. I will investigate this further.

IPGuy
2005-07-08, 09:43
MikeRob, glad to here it is mostly fixed. Let me know what you find out about your last non-working file so I can troubleshoot any albums of mine that may have similar characters.

mikerob
2005-07-08, 17:36
I think I discovered what the problem is.

I had a playlist (.m3u) file in my Playlists directory however the file path to one of the songs had been changed after the playlist was created.

For some reason, this song's album appeared on the browser and SB menu in addition to the correct album. The album's title had accents in it and were displayed garbled.

The garbled album wasn't displayed when I removed the playlist file from the directory or edited the playlist file so the track had the correct file path.

IPGuy
2005-07-08, 21:04
I tried the 7/8 build and my problem was not resolved. As requested I opened a bug report and attached my iTunes.xml and an example song. Hopefully this will be resolved shortly.

Thanks,
Dan

mherger
2005-07-09, 14:01
> I had a playlist (.m3u) file in my Playlists directory however the file
> path to one of the songs had been changed after the playlist was
> created.

This might be bug 1735 (http://bugs.slimdevices.com/show_bug.cgi?id=1735)

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

oiler1fan
2005-07-11, 13:07
I reported this problem a while back - it occurs in 6.0.2, 6.1 and I'm sad to say 6.1 beta as well. You can fix it by removing the server version (exe) and running from the zip version. Why we are still fighting this issue is a mystery

Dan Sully
2005-07-11, 13:22
* oiler1fan shaped the electrons to say...

>I reported this problem a while back - it occurs in 6.0.2, 6.1 and I'm
>sad to say 6.1 beta as well. You can fix it by removing the server
>version (exe) and running from the zip version. Why we are still
>fighting this issue is a mystery

Because there are at least 3 different platforms, N number of character sets,
random user configs, incorrect encodings vs specified locales, and people to deal with.

Patches welcome if you think you can magically fix it.

-D
--
"Hey, careful, man, there's a beverage here!"

dean
2005-07-11, 15:20
On Jul 11, 2005, at 1:07 PM, oiler1fan wrote:
> I reported this problem a while back - it occurs in 6.0.2, 6.1 and I'm
> sad to say 6.1 beta as well. You can fix it by removing the server
> version (exe) and running from the zip version. Why we are still
> fighting this issue is a mystery

Because we don't understand what's going on.

Is there a specific bug <http://bugs.slimdevices.com> open with the
details of your issue? that's the best way to communicate specific
issues, track them and make sure that they are fixed.

MikeGilpin
2005-07-15, 05:40
I finally realized that a problem I'd been having with blank entries showing up in my playlists after they were scanned in from iTunes (where they look fine) was this. And it's not just accented characters. It's any time certain "illegal" characters appear in the pathname to the (in my case) mp3 file, that the scanned-in version of that track shows up with blank track name and album name in the SlimServer library - and any playlists.

And then when it tries to play back this playlist, when it gets to that track, it hangs up, generating lots of interesting messages into the log (I have samples if you want).

The illegal characters include comma, the "degrees" symbol (as in Alex Bugnon's 107 "degrees" in the shade album), and various special characters that appear in Spanish-language cuts such as in Linda Ronstadt's "Mas Canciones" album, Buena Vista Social Club, and so on. Not all "special characters" (in programmer terminology) are a problem - parentheses and the "plus" symbol are OK - but dashes (the minus symbol) are not.

Let me know if you want my logs, and I'll email you the ZIP. I have also previously emailed these files to your support email address, to Kevin Pearsall.

MikeGilpin
2005-07-15, 20:29
Kevin Pearsall suggested I post this from an email thread:

It is an absolutely 100% correlation that it’s these tracks with filenames containing certain special characters that are causing the problem. I can post a log if that would be helpful - tell me where to do that.

By the way, another potentially useful piece of information: several of these tracks are things that I recall hearing played by an earlier version of the SlimServer software. So it does seem to be a problem with the beta, and that the original “freezing up” problem that I first started tracking several weeks ago with 6.0.2 was a different problem, that appears to have been fixed in the beta. That was also different in that it would hang the machine, whereas these tracks that hang with special characters as the reason come “unstuck” when I click on Skip.

Note that when the special character causing the problem is in the album name, then *none* of the cuts in that album show up in the SlimServer library (that is, they are all blank entries). All the tracks in “107 degrees in the shade,” for example.

My theory about the cause is that there’s some string-handling function in the code that has a restricted view of what characters are legal to appear in a filename, which is more restrictive than is actually supported by the Windows and Mac filesystems. An earlier version of the code worked fine, so there’s no reason it can’t be made to work again…

Dan Sully
2005-07-15, 21:44
* MikeGilpin shaped the electrons to say...

>It is an absolutely 100% correlation that it�s these tracks with
>filenames containing certain special characters that are causing the
>problem. I can post a log if that would be helpful - tell me where to
>do that.

Thanks Mike - this is a lot of information that I've heard before, but have
never been able to reproduce myself. Could you tell us a little bit about
your system? OS? Where does your music reside? Local disk? Network Share?

>My theory about the cause is that there�s some string-handling function
>in the code that has a restricted view of what characters are legal to
>appear in a filename, which is more restrictive than is actually
>supported by the Windows and Mac filesystems. An earlier version of the
>code worked fine, so there�s no reason it can�t be made to work again�

That's not actually the case - we don't restrict any high-bit characters -
the issue is somewhere between determining what your 'locale' is, and
translating what the filesystem gives us when we do a 'readdir()'

Thanks.

-D
--
Do not panic, do not panic! We are trained professionals!
Now, stay calm. We are going around the leaf.

Timbo
2005-07-16, 05:32
Hi - sorry to butt in, but is this the same problem I am having with poor old Björk's Greatest Hits - couldn't get it to play - slimserver found everything all right (via iTunes database) but sb2 just said 'empty' in file list for that album (although showed artisit as Bjork?) then when I went into iTunes and renamed the info for all the tracks (WAV files so don't know where iTunes keeps this info?) to Bjork losing the accented 'ö' it plays no probs :-)

Won't raise a bug fo this if it's the same issue :-)

PS sorry - I also have build of 6.1b2 slimserver on a Windows XP SP2 machine

/

MikeGilpin
2005-07-16, 06:30
Info requested by Dan Sully:

1) Music (320K VBR MP3s ripped by the Slim ripping service) stored on a Linksys EFG250 Network Attached Storage device, hard-wired network connection via 1GBps network (switches, everything). It runs Linux but was designed to look like a Windows NT Server file-system - what it looks like to Mac or PC.

2) Server runs on a Mac Mini running original Jaguar (not updated to Tiger), 512MB RAM, fully patched up to current level. This is the same config I was running when, on an earlier release, some of these same tracks would play.

3) The music files are therefore a network share of an "SMB" filesystem, from the Mac point of view. Note that this same arrangement is what I've been using to run SlimServer for about the last five months, since before I got my SB2s (I have three). I definitely recall hearing some of the tracks that now won't play, due to this special-character issue, since I got the Mac to be a dedicated music server. For example "Scootcha-Booty" by Al Jarreau - which now won't play because of the dash in the name.

MikeGilpin
2005-07-16, 09:36
Dan Sully: I also created a bug, 1829, to which I attached a ZIP file containing an export of my iTunes.xml file, and some log entries you may find useful in running this down. Good luck! (and thanks for your help)

IPGuy
2005-07-26, 07:19
Any progress on this bug yet? Seems that almost daily someone posts in one of the forums with similar problems. I view this as a high priority fix before I buy any more SB2's (I was planning on buying 5, but until I can get the first one working I won't be buying any more).

Thanks,
Dan

dean
2005-07-26, 08:28
Dan,
We're investigating that bug for the 6.1.2 release, along with a few
others.

Stay tuned...


On Jul 26, 2005, at 7:19 AM, IPGuy wrote:

>
> Any progress on this bug yet? Seems that almost daily someone
> posts in
> one of the forums with similar problems. I view this as a high
> priority fix before I buy any more SB2's (I was planning on buying 5,
> but until I can get the first one working I won't be buying any more).
>
> Thanks,
> Dan
>
>
> --
> IPGuy
>

Dan Sully
2005-07-26, 10:22
* IPGuy shaped the electrons to say...

>Any progress on this bug yet? Seems that almost daily someone posts in
>one of the forums with similar problems. I view this as a high
>priority fix before I buy any more SB2's (I was planning on buying 5,
>but until I can get the first one working I won't be buying any more).

Are you running on Windows?

-D
--
<iNoah> my pdp goes to 11.

IPGuy
2005-07-26, 15:28
Yes running XP Pro SP2

Dan Sully
2005-07-26, 16:12
* IPGuy shaped the electrons to say...

>Yes running XP Pro SP2

iTunes?

And the latest 6.1.1 release?

Thanks.

-D
--
"Hey, careful, man, there's a beverage here!"

IPGuy
2005-07-29, 12:21
I have a SB2 and am running the July 6th evening build of 6.1b slimserver on a Windows XP SP2 machine. I also have installed AlienBBC 0.97 and MPlayer 1.0pre7. I am using the iTunes library as the source of music & playlists.

dip
2005-07-30, 07:38
I was running a lot of nightly builds of 6.1 (on Windows XP) and had never any problems with accented characters. But yesterday I switched to the nightly build of July 29 (of 6.2) and after a complete rescan only about 2900 of my 5600 songs were shown. In addition a lot of artists having accented characters in their names were shown incomplete. For example when browsing by artist there was an entry "Herbert Gr" in addition to a further entry "Herbert Grönemeyer" (the last one is correct). When klicking on "Herbert Gr" there was not shown an album but the rest of the last name but the "ö" was replaced by "fx6" (if I remember correctly) so there was shown "fx6nemeyer". When I klicked on the entry I saw the songs of an album of Herbert Grönemeyer but there was shown "no artist" and "no album" after the song names.
I then tried to reinstall previous versions and also the present versions of the 6.1.2 tree but the only improvement was that the number of songs were restored to about 5600 (after a rescan). The umlaut problem still was there.
I even made a complete uninstall but that did not help.
Only when I made a complete uninstall and reinstalled the nightly build of July 17, 2005 of the 6.1b2 version after a complete rescan everything is fine again. So it seems that anywhere between July 17 and today there have been inserted problems with accented characters (maybe two different, one relating to the wrong songs number and one related to the wront display of umlauts).
Dieter

Dan Sully
2005-07-30, 10:44
* dip shaped the electrons to say...

>I was running a lot of nightly builds of 6.1 (on Windows XP) and had
>never any problems with accented characters. But yesterday I switched
>to the nightly build of July 29 (of 6.2) and after a complete rescan

Dieter - I just made some significant changes yesterday, that are in the July
30th nightly builds. Could you give that a try?

>of July 17, 2005 of the 6.1b2 version after a complete rescan everything
>is fine again. So it seems that anywhere between July 17 and today there
>have been inserted problems with accented characters (maybe two
>different, one relating to the wrong songs number and one related to
>the wront display of umlauts).

Thanks - if my recent changes don't fix it, I'll isolate the problem between
those dates.

-D
--
This knob controls the thing that changes when you turn it. - noah

dip
2005-07-30, 11:29
Shall I simply install over my present installation or first make a complete uninstall? And shall I make a rescan with clear cache after the installation?
Dieter

Dan Sully
2005-07-30, 11:33
* dip shaped the electrons to say...

>Shall I simply install over my present installation or first make a
>complete uninstall? And shall I make a rescan with clear cache after
>the installation?

Yes on both accounts, just to be sure.

-D
--
<dr.pox> NO, NETBSD IS NOT REALLY BUILT WITH ELITE FORTRAN77!!@$#$

dip
2005-07-30, 13:16
I uninstalled the July 17 version of 6.1b2 and installed the July 30 version of 6.2b1
During installation I confirmed the default for my music folder to be D:\Eigene Dateien\Eigene Musik (=D:\My Files\My Music) and changed the playlist folder to D:\Eigene Dateien\Playlists (D:\My Files\Playlists).

SlimServer than started automatically to scan my music directory. This usually takes about 5-10 minutes. But after 30 minutes slim.exe still took 99% of my CPU power. I finally found that SlimSever not only scans my music folder but the whole D: partition. There are a lot of directories shown in browse music folder which are directly located in D:\
There are also a lot of wav-files shown which are outside of my music directory.
But in server settings the music folder is correctly shown as D:\Eigene Dateien\Eigene Musik

So I just stopped SlimServer, restarted it and started a manual rescan with clearing the cache. I will report as soon as I see what happen.

Dieter

dip
2005-07-30, 13:28
Same problem again. And I just saw that this problem has been described today already in this thread.

MikeGilpin
2005-08-05, 07:32
Dan, any word on this? Any more recent nightly builds that contain anything I should be testing? Need any more info from me? If you're stumped, let me know where to look in the code and I'll see what I can figure out - although I don't know Perl, I have programmed in several other languages...

Dan Sully
2005-08-05, 08:38
* MikeGilpin shaped the electrons to say...

>Dan, any word on this? Any more recent nightly builds that contain
>anything I should be testing? Need any more info from me? If you're
>stumped, let me know where to look in the code and I'll see what I can
>figure out - although I don't know Perl, I have programmed in several
>other languages...

Mike - the latest 6.2 nightlies contain incremental fixes - I do suggest you give them a try.

-D
--
<fuz> deregulation will lead to greater competition, consumer choice, and lower prices.
my name is elmer fudd. I own a mansion and a yacht.

MikeGilpin
2005-08-06, 11:55
Using the 8-5 nightly build, it seems to be working. As usual I rescanned with the "delete first" option turned on, and now when I look through the playlists that were previously showing blank entries, those entries are no longer blank - the troublesome cuts are now showing up correctly.

When I tried to play one of the lists, shuffled, it acted a little weird at first, but after restarting the software and all the players and making sure the synch modes were all set the way I wanted them, it now is proceeding OK through a shuffled list.

I'll let you know if there are any further problems on this one.

MikeGilpin
2005-08-07, 11:06
Progress report - 20 hours or so of playing later, all working fine.

Those pesky error messages that were showing up in the log are gone, too (hundreds of them were appearing complaining about a problem with an unitialized string value, in various places).

Woo hoo!

dip
2005-08-08, 15:02
It works now perfectly for me (with the August 7 version)! Thanks a lot.

Renyao
2005-08-12, 07:48
I tried out Slimserver v6.0.2,v6.1.2 and v6.2.1b.The server
produced web pages are consisted of characters of mixed encodings.Some are encoded with cp936 locale,some are in the form of utf8 of cp936, and the largest part are wrongly encoded as utf8 of latin-1.They can't be displayed correctly in web browser.


When I run perl slimserver.pl,a long list of "Malformed UTF-8
character ..." will display on the screen.

The mixed encodings in web page are caused by successful/unsuccessful encoding/decoding,concatenation of utf8 character string with
Chinese byte string, and carelessness about the utf8 on/off flags.

I corrected some bugs.But at last I decided to solve the problem
in a package.

I wrote a subroutine that convert a byte string with mixed encodings into "one-encoding-only" string(utf8 character string of locale or byte string of locale).

I inserted
$output = Slim::Utils::HtmlAdjust::htmlReform($output);
just before
return \$output;
in the if ($type eq 'fill') block in subroutine _generateContentFromFile in Slim/Web/HTTP.pm

Now the Chinese characters display well in browses.

Yesterday,I upgraded to V6.2.1.I added a statment
$string = Slim::Utils::HtmlAdjust::htmlReform($string);
in subroutine string in Slim/Display/graphics.pm

Now the Chinese characters display well in SB2.

Of course,"use Slim::Utils::HtmlAdjust;" should be added
where neccessary.

BTW,the "_" character is replaced with " " by slim programs for
some reason and it is the second byte of many Chinese characters.

e.g. Chinese character "\xE9\x5F" will become "\xE9\x20" leading to many errors.Chinese character recovery is made in
Slim::Utils::HtmlAdjust."\xE9\x20" is recovered to "\xE9\x5F"

This program may be useful in CJK windows.
Where and how to post it?

dean
2005-08-12, 08:20
Renyao,

Thanks, this is great work. Please post your code to the developers
forum.

Thanks again,

dean

On Aug 12, 2005, at 7:48 AM, Renyao wrote:

>
> I tried out Slimserver v6.0.2,v6.1.2 and v6.2.1b.The server
> produced web pages are consisted of characters of mixed encodings.Some
> are encoded with cp936 locale,some are in the form of utf8 of cp936,
> and the largest part are wrongly encoded as utf8 of latin-1.They can't
> be displayed correctly in web browser.
>
>
> When I run perl slimserver.pl,a long list of "Malformed UTF-8
> character ..." will display on the screen.
>
> The mixed encodings in web page are caused by successful/successful
> encoding/decoding,concatenation of utf8 character string with
> Chinese byte string, and carelessness about the utf8 on/off flags.
>
> I corrected some bugs.But at last I decided to solve the problem
> in a package.
>
> I wrote a subroutine that convert a byte string with mixed encodings
> into "one-encoding-only" string(utf8 character string of locale or
> byte
> string of locale).
>
> I inserted
> $output = Slim::Utils::HtmlAdjust::htmlReform($output);
> just before
> return \$output;
> in the if ($type eq 'fill') block in subroutine
> _generateContentFromFile in Slim/Web/HTTP.pm
>
> Now the Chinese characters display well in browses.
>
> Yesterday,I upgraded to V6.2.1.I added a statment
> $string = Slim::Utils::HtmlAdjust::htmlReform($string);
> in subroutine string in Slim/Display/graphics.pm
>
> Now the Chinese characters display well in SB2.
>
> Of course,"use Slim::Utils::HtmlAdjust;" should be added
> where neccessary.
>
> BTW,the "_" character is replaced with " " by slim programs for
> some reason and it is the second byte of many Chinese characters.
>
> e.g. Chinese character "\xE9\x5F" will become "\xE9x\20" leading to
> many errors.Chinese character recovery is made in
> Slim::Utils::HtmlAdjust."\xE9x\20" is recovered to "\xE9x\5F"
>
> This program may be useful in CJK windows.
> Where and how to post it?
>
>
> --
> Renyao
>