PDA

View Full Version : SC7 doesn't recognize all MP3 files



shein
2008-07-19, 10:30
Hi all,

I have an extensive MP3 library (about 27,000 tracks) which is organized with iTunes. I have recently acquired a QNAP TS-101, installed SC7, and moved the library there. The problem is that SC7 only includes about 19,000 of the tracks in its own library, even though SC7 does see all 27,000 tracks.

I'm not sure whether this is purely due to SC7 or due to some interaction with the specific hardware. But there was no response in the "Third Party Hardware" forum, so I'm trying here. I've seen some references to tags being the problem, but even if that's the case, I don't know how to fix the problem.

There seems to be a correlation between missed tracks and having "international" characters in the filenames or tags, but this is not a 1-to-1 correlation.

Here are the details.

TS-101 (2.3.0 Build 0618T)

Total Tracks: 19,618
Total Albums: 2,353
Total Artists: 859
Total Genres: 22
Total Playing Time: 1414:32:47

Music Scan Details
iTunes Import (27132 of 27132) Complete 01:13:15

SqueezeCenter Version: 7.0.1 - 19705 @ Wed May 14 19:53:43 PDT 2008 - Linux - EN - utf8
Server IP address: 192.168.178.25
Perl Version: 5.8.8 ppc-linux-thread-multi
MySQL Version: 5.0.37
Platform Architecture: ppc-linux
Hostname: QNAP101
Server Port Number: 9001
Total Players Recognized: 1

There are samples from the iTunes XML file, the scanner.log file and the corresponding directory content in the posting over in the hardware forum, if that's helpful. I won't copy the files here, but they're at http://forums.slimdevices.com/showthread.php?t=49775.

Many thanks for any help,
Soren

MrSinatra
2008-07-19, 10:47
there could be lots of reasons for this.

look here:

http://bugs.slimdevices.com/show_bug.cgi?id=8380

and here:

http://bugs.slimdevices.com/show_bug.cgi?id=8764

for just a start. might it also be an issue with the mp3 header?

do you use itunes integration? did SC7 used to see all the stuff b4?

what exactly do you mean SC7 "sees" it all but only has some of it?

shein
2008-07-20, 08:07
Thanks, MrSinatra. I'm guessing there is some overlap with your bug report 8764. My problem is shown in the scan results in my original post,

Total Tracks: 19,618
[...]
Music Scan Details
iTunes Import (27132 of 27132) Complete 01:13:15

So SC7 only shows 19,618 of the 27,132 tracks, although it does see them well enough at least to count them when it imports from the iTunes XML file. Somehow it skips the others.

It could well be to do with the MP3 headers, and I initially thought it was due to funny international characters either in the file name, XML Location tag or somewhere else. This may still be the case, but I've not found a simple rule that covers it. Rescanning the library completely takes 16 hours (as it sits on my QNAP TS-101), so I'm not very keen on narrowing it down incrementally. Sometimes entire albums are missing, sometimes it's individual songs from an album. In all cases, everything was ripped in iTunes, and each album was ripped in one go. So it's not clear to me why some tracks would be treated differently.

I wasn't actually using SC7 before -- I was using the previous version running on a PC, and if there were tracks missing, I didn't notice it then.

Any ideas for narrowing it down would be welcome. Any pointers to programs I can use to play with my MP3 tags -- likewise.

Soren

Siduhe
2008-07-20, 08:53
When you say "The problem is that SC7 only includes about 19,000 of the tracks in its own library, even though SC7 does see all 27,000 tracks." - do you mean that SC sees them via Browse Music folder? No chance the missing tracks have DRM (which SC won't be able to play)?

MrSinatra
2008-07-20, 11:35
i don't use itunes integration or itunes, so i'm not much help. but i have read many issues on here about things not showing up, including even cases where one install of SC7 finds it, while another does not! in that case it did turn out to be unusual character sets, but it didn't explain why one install found it while another didn't.

i'm confused b/c my understanding was if u used itunes integration it simply let itunes do all the work for the ML. in any case, i would try calling the support number.

shein
2008-07-20, 14:23
Siduhe,

Thanks for your question. I tell SC7 (running on the QNAP TS-101) to import the iTunes library from the iTunes XML file. As shown in another of my posts in this thread, the iTunes import routine does show the right number of tracks being "considered" / "seen" (I don't know how else to describe it), but in SC7's own library, the number of tracks is considerably lower. No, there is no DRM on any of the tracks.

Thanks for any further pointers. I'm happy to do further work myself, but I need some direction...

Soren

Nonreality
2008-07-20, 18:16
Siduhe,

Thanks for your question. I tell SC7 (running on the QNAP TS-101) to import the iTunes library from the iTunes XML file. As shown in another of my posts in this thread, the iTunes import routine does show the right number of tracks being "considered" / "seen" (I don't know how else to describe it), but in SC7's own library, the number of tracks is considerably lower. No, there is no DRM on any of the tracks.

Thanks for any further pointers. I'm happy to do further work myself, but I need some direction...

Soren
It's got to be in your tags I would think. Take a good look at the ones that make it and the ones that don't. It may help figure it out. If you can load them into mp3tag and see if the problem ones have a different tag like ID3v2.4 or ape that would explain the difference.

moley6knipe
2008-07-21, 08:28
iTunes is a lot less tolerant of filename and path length than SqueezeCenter, I think. I've fallen foul of this - if I scan the .xml using iTunes integration in SC I get less tracks than if I scan the music folder directly. Reason being, as far as I can see, if a filename/path combo exceeds a certain length iTunes won't import it. Not sure what that length is, mind!

I know if you rip a CD with iTunes the resulting filename will never exceed 36 characters excluding extension. So now I've set dBpoweramp to truncate filenames to 30 chars to be on the safe side.

But as Nonreality said, your tags could also be in need of examination. PITA finding which ones, though.

moley6knipe
2008-07-21, 08:36
Are they all mp3, or are there any AAC or Apple Lossless files? Seem to recall there's a bug with mov123.exe/QuickTime (used by SC for transcoding these files to a format SqueezeBoxen can play) which means files with a filename of 63 chars of more won't work. A forum search for "faad" normally finds this one.

shein
2008-07-21, 12:21
Thanks, Nonreality and moley6knipe.

All tags are of the same "kind" (ID3v2.4, APE etc).

The issue does not appear to be related to the length of the filename/tag.

All files are MP3.

I wasn't sure what to make of the reference that "iTunes is a lot less tolerant of filename and path length than SqueezeCenter, I think". iTunes itself is able to read all the files in is XML library. If you mean that the iTunes import routine within SC7 is less tolerant than the general routine that scans directories looking for music files, then: This is possible. I've not tried letting SC7 loose on the directories in that way.

However, I do have a theory. I'm not sure this explains 100% of the issues, but it might. It seems that the tracks that are rejected have a different capitalization of the track name than either the XML Location tag (in the iTunes XML library) or the MP3 tag. (In the examples that I've found, the XML Location tag and the MP3 tag have the same capitalization, so I haven't been able to find out if it's one or the other.)

So a file might be called "Popmusikerens Vise.mp3" while the XML tag might contain the title "Popmusikerens vise" and the Location tag might be "...Popmusikerens%20vise.mp3" (note the lower-case "v"; I've left out the path).

Presumably iTunes is case-insensitive in this regard, while SC7 isn't. Perhaps this is a result of running both on PCs and Macs... The longer the title, the more likely I am to have fiddled with at least one letter, and also I'm more likely to have done it for international titles (which I generally put in the capitalization of the actual language).

How did this happen? I'm quite particular about my track titles as they appear in iTunes, and often I correct the capitalization after ripping the CD. Perhaps this leads to the observed behavior which you might call a bug in iTunes. iTunes shouldn't change the Location tag just because the Name tag changes (or it should change the filename, too). By the way, I do the same with album names, but here iTunes doesn't seem to fix the Location tag (at least I haven't noticed so far).

Does anybody know the SC7 code well enough to say whether I'm right?

If yes, one solution might be for SC7 to check a lower-case version of the Location tag against lower-case versions of the directory content (at least optionally). I realize this could be slow if people don't keep the music sorted by artist and album, but keep everything in one great big directory. If people can point me to the right place in the right script, I might just do this for myself. (I don't know how to make it properly available as an option in the web interface.)

Alternatively, I suppose I might write a script to fix the capitalization of the Location tags vs the Name tags in the XML file, once and for all. I'm not quite sure whether iTunes will let this stand, though. I know there's both an XML file an an itl file with some database representation. Perhaps it would be better to fix the actual filenames.

Or maybe there's a better way yet? I've not tried mp3tag yet, as I'm doing this from a Mac and mp3tag doesn't seem to exist in a Mac version. But I could do it from a Windows PC as well.

Soren

Laz
2008-09-21, 00:03
Shein,

This is a long standing issue - check http://bugs.slimdevices.com/show_bug.cgi?id=4892

(I've just run into again, and now I can't even get the scanner to tell me which files are causing the problem - grump).

Larry

Nonreality
2008-09-21, 00:30
Thanks, Nonreality and moley6knipe.

All tags are of the same "kind" (ID3v2.4, APE etc).

The issue does not appear to be related to the length of the filename/tag.

All files are MP3.

I wasn't sure what to make of the reference that "iTunes is a lot less tolerant of filename and path length than SqueezeCenter, I think". iTunes itself is able to read all the files in is XML library. If you mean that the iTunes import routine within SC7 is less tolerant than the general routine that scans directories looking for music files, then: This is possible. I've not tried letting SC7 loose on the directories in that way.

However, I do have a theory. I'm not sure this explains 100% of the issues, but it might. It seems that the tracks that are rejected have a different capitalization of the track name than either the XML Location tag (in the iTunes XML library) or the MP3 tag. (In the examples that I've found, the XML Location tag and the MP3 tag have the same capitalization, so I haven't been able to find out if it's one or the other.)

So a file might be called "Popmusikerens Vise.mp3" while the XML tag might contain the title "Popmusikerens vise" and the Location tag might be "...Popmusikerens%20vise.mp3" (note the lower-case "v"; I've left out the path).

Presumably iTunes is case-insensitive in this regard, while SC7 isn't. Perhaps this is a result of running both on PCs and Macs... The longer the title, the more likely I am to have fiddled with at least one letter, and also I'm more likely to have done it for international titles (which I generally put in the capitalization of the actual language).

How did this happen? I'm quite particular about my track titles as they appear in iTunes, and often I correct the capitalization after ripping the CD. Perhaps this leads to the observed behavior which you might call a bug in iTunes. iTunes shouldn't change the Location tag just because the Name tag changes (or it should change the filename, too). By the way, I do the same with album names, but here iTunes doesn't seem to fix the Location tag (at least I haven't noticed so far).

Does anybody know the SC7 code well enough to say whether I'm right?

If yes, one solution might be for SC7 to check a lower-case version of the Location tag against lower-case versions of the directory content (at least optionally). I realize this could be slow if people don't keep the music sorted by artist and album, but keep everything in one great big directory. If people can point me to the right place in the right script, I might just do this for myself. (I don't know how to make it properly available as an option in the web interface.)

Alternatively, I suppose I might write a script to fix the capitalization of the Location tags vs the Name tags in the XML file, once and for all. I'm not quite sure whether iTunes will let this stand, though. I know there's both an XML file an an itl file with some database representation. Perhaps it would be better to fix the actual filenames.

Or maybe there's a better way yet? I've not tried mp3tag yet, as I'm doing this from a Mac and mp3tag doesn't seem to exist in a Mac version. But I could do it from a Windows PC as well.

SorenJust as a trial, find an album that is not showing up, or a few to experiment on. Use mp3tag to get rid of any tags that are not ID3v2.3. There are good instructions in the wiki and on the mp3tag site for this. Then rescan and see if it shows up. This is basically what you need to do, eliminate any possible tag conflicts. Mp3tag is free and a very important program to fix tags and do much more. Ape tags sometimes take over preference in programs and sometimes they have very little info. If you use mp3gain to do your replay gain this could be the problem. It writes the replay gain tags to v2.4 so a lot of programs only see the replaygain tags and not the album artist, etc. that are written to the ID3v2.3 or the id3.1 tags that contain all that stuff. I've see it happen a lot. Mp3tag will show you all the types of tags that you have in a album and let you get rid of the ones you don't need. This is what I would recommend starting with to find the reason. It's not that hard and once you find what is causing it, if it is tags, you can then fix your whole library quite easily.

Mark Lanctot
2008-09-23, 06:27
Ape tags sometimes take over preference in programs and sometimes they have very little info. If you use mp3gain to do your replay gain this could be the problem. It writes the replay gain tags to v2.4 so a lot of programs only see the replaygain tags and not the album artist, etc. that are written to the ID3v2.3 or the id3.1 tags that contain all that stuff.

Actually MP3Gain writes RG info to APEv2 tags.

This isn't normally a problem except if you also have other data already in APEv2 tags and need to correct that data.

Provided that you start fresh (i.e. Mp3tag doesn't indicate the presence of an APEv2 tag), then you use MP3Gain you shouldn't have a problem.

You can set Mp3tag to read APEv2 tags but not write them - this can confirm the presence of other data in the APEv2 tags.