PDA

View Full Version : missing tracks after scan



Matt McKinnon
2005-03-28, 08:18
i just picked up a new CD. after encoding and transferring to my
computer (linux), and rescanning the hard drive for new music,
slimserver fails to read any of the tracks with non english characters.
here is the --d_scan output for the problematic tracks. let me know if
you need any more debugging done:


2005-03-28 10:09:00.0949
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/01%20-%20Sikertelens%EF%BF%BDg.mp3
2005-03-28 10:09:00.0960
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/02%20-%20Szerencs%EF%BF%BDtlen.mp3
2005-03-28 10:09:00.0972
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/03%20-%20%EF%BF%BDngyilkos%20Vas%EF%BF%BDrnap.mp3
2005-03-28 10:09:00.0983
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/04%20-%20Felbomlasztott%20Ment%EF%BF%BDkocsi.mp3
2005-03-28 10:09:00.0993
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/05%20-%20Hajnal.mp3
2005-03-28 10:09:00.1005
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/06%20-%20Galamb%20Egyed%EF%BF%BDl.mp3
2005-03-28 10:09:00.1018
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/07%20-%20M%EF%BF%BDsodik%20Galamb.mp3
2005-03-28 10:09:00.1030
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/08%20-%20Szam%EF%BF%BDr%20Mad%EF%BF%BDr.mp3
2005-03-28 10:09:00.1041
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/09%20-%20Hisz%EF%BF%BDkeny.mp3
2005-03-28 10:09:00.1053
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/10%20-%20K%EF%BF%BDtsark%EF%BF%BD%20Mozgalom.mp3
2005-03-28 10:09:00.1063
file:///music/signals/Venetian%20Snares/Rossz%20Csillag%20Allat%20Szuletett/11%20-%20Senki%20Dala.mp3

Dan Sully
2005-03-28, 11:30
* Matt McKinnon shaped the electrons to say...

>i just picked up a new CD. after encoding and transferring to my
>computer (linux), and rescanning the hard drive for new music,
>slimserver fails to read any of the tracks with non english characters.
>here is the --d_scan output for the problematic tracks. let me know if
>you need any more debugging done:

Matt - what is your LC_CTYPE environment variable set to?

-D
--
"This basis of our government being the opinion of the people, the very first object should be to keep that right;
and were it left to me to decided whether we should have a government without newspapers, or newspapers without a
government, I should not hesitate a moment to prefer the latter." - Thomas Jefferson

Matt McKinnon
2005-03-28, 11:38
Dan Sully wrote:

> * Matt McKinnon shaped the electrons to say...
>
>> i just picked up a new CD. after encoding and transferring to my
>> computer (linux), and rescanning the hard drive for new music,
>> slimserver fails to read any of the tracks with non english
>> characters. here is the --d_scan output for the problematic tracks.
>> let me know if you need any more debugging done:
>
>
> Matt - what is your LC_CTYPE environment variable set to?
>
> -D

# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Dan Sully
2005-03-28, 11:41
* Matt McKinnon shaped the electrons to say...

>>Matt - what is your LC_CTYPE environment variable set to?
>>
># locale
>LANG=en_US.UTF-8
>LC_CTYPE="en_US.UTF-8"

And your ripping program respects that LC_CTYPE? IE: it didn't write out ISO-8859-1 instead?

-D
--
( ( ( [ ] ) ) )
In Stereo Where
Available

Matt McKinnon
2005-03-28, 12:10
Dan Sully wrote:

> * Matt McKinnon shaped the electrons to say...
>
>>> Matt - what is your LC_CTYPE environment variable set to?
>>>
>> # locale
>> LANG=en_US.UTF-8
>> LC_CTYPE="en_US.UTF-8"
>
>
> And your ripping program respects that LC_CTYPE? IE: it didn't write
> out ISO-8859-1 instead?
>
> -D

not sure. quite possible... using CDex on a windows box before
transfering files to the linux box. thanks for the hint.

-m

Dan Sully
2005-03-28, 12:12
* Matt McKinnon shaped the electrons to say...

>>And your ripping program respects that LC_CTYPE? IE: it didn't write
>>out ISO-8859-1 instead?
>
>not sure. quite possible... using CDex on a windows box before
>transfering files to the linux box. thanks for the hint.

That's definitely it then - your windows box is likely in cp1252 aka iso-8859-1.

-D
--
Welcome to hell. Here's your accordion.

Gordon Harris
2005-03-28, 13:53
Dan Sully <dan@...> writes:


> That's definitely it then - your windows box is likely in cp1252 aka iso-8859-
1.
>
> -D



I'm having the same problem with the 3-28 v6 nightly on my Windows XP box. The
file "Flute Music - Aur├ęole Ensemble.flac" doesn't make it into the scan. If I
change the name to "Flute Music - Aureole Ensemble.flac" it scans just fine.

Gordon Harris
2005-03-28, 17:20
Dan Sully <dan@...> writes:
>
> That's definitely it then - your windows box is likely in cp1252 aka iso-8859-
1.

Dan: is there a bug # for this? I'd like to follow the progress for a fix for
this problem. I expect my squeezebox2 to arrive tomorrow, but, as it is, about
1/2 of my flac music collection won't be scanned by slimserver 6 because of
this bug.

Dan Sully
2005-03-28, 18:13
* Gordon Harris shaped the electrons to say...

>> That's definitely it then - your windows box is likely in cp1252 aka iso-8859-1.
>
>Dan: is there a bug # for this? I'd like to follow the progress for a fix for
>this problem. I expect my squeezebox2 to arrive tomorrow, but, as it is, about
>1/2 of my flac music collection won't be scanned by slimserver 6 because of this bug.

Gordon - it's not really a bug. If you are doing some type of file sharing -
the character sets / code pages on the machines need to be the same, or some
translation needs to be done on one of the sides.

The issue here is that your Linux charset (LC_CTYPE) is set to UTF-8 in
/etc/sysconfig/i18n (or similar), which is fine in and of itself. But when
mounting the remote Windows share, that charset is cp1252 in windows speak,
iso-8859-1 / latin1 to the rest of us. There's no way for SlimServer to know
that however, so the names get munged, thinking they are UTF-8 when they aren't.

The fix here should be adding the iocharset and codepage options to the smb mount.

From the smbmount(1) man page:

iocharset=<arg>

sets the charset used by the Linux side for codepage to charset translations
(NLS). Argument should be the name of a charset, like iso8859-1.
(Note: only kernel 2.4.0 or later)

codepage=<arg>

sets the codepage the server uses. See the iocharset option. Example value
cp850. (Note: only kernel 2.4.0 or later)

So the command would look something like:

mount -t smbfs -o iocharset=utf8,codepage=cp1252 remote\\path /mnt


-D
--
It is dark. You are likely to be eaten by a grue.

Gordon Harris
2005-03-28, 20:55
Dan Sully <dan@...> writes:


> Gordon - it's not really a bug. If you are doing some type of file sharing -
> the character sets / code pages on the machines need to be the same, or some
> translation needs to be done on one of the sides.
>
> The issue here is that your Linux charset (LC_CTYPE) is set to UTF-8 in
> /etc/sysconfig/i18n (or similar...

I'm not running Linux, nor am I having Slimserver fetch flacs from any sort of
share. I do rip/encode/tag on a different machine, but it's running WinXP too,
and all the XP machines are running with the same "Regional and Language
Options." After I rip/encode/tag, I copy the flacs to a raid array running on
the Slimserver box.

Gordon Harris
2005-03-28, 21:26
Dan Sully <dan@...> writes:

> Gordon - it's not really a bug. If you are doing some type of file sharing -
> the character sets / code pages on the machines need to be the same...

Again, just to be clear, ALL my machines are running WinXP, and ALL of them
return "437" (United States) from a "mode con codepage /status" query. None of
the flac files with diacritic characters in their filenames were ripped on or
stored on a linux box or anything with a UTF8-aware file system. All the
filenames are straight Win32 ANSI.

Also, I'm only noticing this problem with whole-album flac files. Individual
track MP3 files with diacritic names seem to scan just fine, though they don't
always display right.
E.g.: "C:\Recordings\Music\l_Modern_Central_European\Jan├ ícek, L\Orchestral
Works -- CSP, Serebrier\06 - La┼ísk├ę tance (1924) - 1 Starod├ívn├Ż I.mp3" scans
and plays just fine, though Slimserver doesn't like to display the "š".

If I take an otherwise good flac file and rename it to that same "La┼ísk├ę tance"
filename (but with a .flac extension) it becomes invisible to slimserver, even
after forcing a rescan via stopping the service and deleting the .db file from
the cache dir.

Also, it seems like the problem only crops up if the diacritic character is in
the flac filename itself. Flac files with out diacritics in the filename, but
with diacritcs in directory names seem to scan ok.

So, unless I'm doing something really inane here, I do think this is a bug.