TrackStat/Custom Scan/LMS Â/¬/URL problem

[prabbit]

Yes, the song title is Dominion/Mother Russia. The / is part of the title. It's for this reason (and other "odd" characters that I use ¬ character, because I know for certain that ¬ does not appear in any artist name or album or song title). Anyway, the queries worked and the results (CSV) are in the attached zip files.

db_query_trackstat_results.zip

[erland]

Yes, the song title is Dominion/Mother Russia. The / is part of the title. It's for this reason (and other "odd" characters that I use ¬ character, because I know for certain that ¬ does not appear in any artist name or album or song title). Anyway, the queries worked and the results (CSV) are in the attached zip files.

db_query_trackstat_results.zip

Ok, the / character is definitely the issue because it doesn't exist in the TrackStat files and this track doesn't seem to have any musicbrainz id in the tracks table and due to this the TrackStat backup isn't able to restore the rating. Do you know if there is a reason why the musicbrainz data is missing ? Have you removed it from this track or does the TrackStat backup (which have a musicbrainz id) refer to another music file ?

I'll try in my own setup if I can reproduce the issue, but before I do that, could you have renamed this file and added the / characters since the rating was last changed in TrackStat ?
Does the other tracks which have problems also have a / character in the file name ?

[prabbit]

Ok, the / character is definitely the issue because it doesn't exist in the TrackStat files and this track doesn't seem to have any musicbrainz id in the tracks table and due to this the TrackStat backup isn't able to restore the rating. Do you know if there is a reason why the musicbrainz data is missing ?

To be honest, I don't know. It's possible that this file hasn't yet gotten a musicbrainz ID. Well I just ran this query and it returned thousands of tracks, none of which have a musicbrainz ID.

select * from tracks where url like '%.m4a'

Then I ran the same query but for '%.mp3'.

None of the m4a files have musicbrainz IDs and only 1955 mp3 tracks have musicbrainz IDs, which matches pretty closely to the ~2200 mp3 tracks that have a rating after a TrackStat restore.

Have you removed it from this track or does the TrackStat backup (which have a musicbrainz id) refer to another music file ?

This has been a complicated thread to follow, but here's a quick recap of what I have done.

1. I had .flac files with musicbrainz IDs (well, not all of them, but the vast majority).
2. Using XLD on a Mac I converted .flac to Apple Lossless (.m4a).

My guess is that in the process of conversion I lost the musicbrainz ID.

I'll try in my own setup if I can reproduce the issue, but before I do that, could you have renamed this file and added the / characters since the rating was last changed in TrackStat ?
Does the other tracks which have problems also have a / character in the file name ?

I really think the '/' issue is a red herring and you shouldn't chase it right now. It may be an issue, but I suspect it's a minor one.

Since most of my music files have lost their musicbrainz ID, let me spend a day or two getting the musicbrainz IDs back into my files. After they're back, I'll run the TrackStat restore again and report back.

[erland]

This has been a complicated thread to follow, but here's a quick recap of what I have done.

1. I had .flac files with musicbrainz IDs (well, not all of them, but the vast majority).
2. Using XLD on a Mac I converted .flac to Apple Lossless (.m4a).

My guess is that in the process of conversion I lost the musicbrainz ID.

Ok, this will make the TrackStat restore fail, because changing the file name from flac to mp4 will make the restore stop working unless:
1. You have musicbrainz identities
or
2. You have the "Find alternative files" option active during the restore operation (which will generally make the restore take a lot longer time)

However, for the track we have just looked at, alternative 2 with the "Find alternative files" option doesn't work because it's not just the file extension that differs but also the extra / in the file name. So the only way to make the restore work would be through musicbrainz id's which exists in the TrackStat backup file (which refers to .flac file).

The "Find alternative files" option will try to find a matching track by skipping the file extension and skipping the first part of the path (typically drive letter or directory where the music folder has been mounted). It's main purpose is to make it possible to reconnect ratings if you have changed drive. So if the extra '/' character in this particular file isn't a general problem, it might be worth to run the restore with "Find alternative files" option enabled as it will be able to handle the .flac to .m4a file name change. However, getting musicbrainz identities into all the files would of course be preferred as this makes it possible to run without the "Find alternative files" option and this makes the restore operation run a lot faster.

I really think the '/' issue is a red herring and you shouldn't chase it right now. It may be an issue, but I suspect it's a minor one.

Since most of my music files have lost their musicbrainz ID, let me spend a day or two getting the musicbrainz IDs back into my files. After they're back, I'll run the TrackStat restore again and report back.

Ok sounds good, after you have got the musicbrainz id tags back remember that you need to do a standard rescan in SBS/LMS to get them into the "tracks" table before you start the TrackStat restore.

Let me know if the problems remain after you have tried this.

[prabbit]

Before I attempt to restore my TrackStat ratings, I am making sure I have everything in place. To do that, I

1. Got the musicbrainz IDs back into my files. I used Jaikoz on a Mac to do it. According to Jaikoz 19,246 of my 20,910 songs have musicbrainz IDs.
2. Deleted my LMS cache and preferences.
3. Rescanned my files.
4. Reinstalled TrackStat and Database Query plugins.
5. Have NOT yet tried to restore my ratings.

Database Query question...

After doing the above, when I look at Song Info and then click View Tags, I see the musicbrainz IDs are there. Below is the details for the Dominion/Mother Russia track we've been looking at. (It's hard to read, but you can see there are values for the musicbrainz are there. This is the track ID: b8bb131f-5428-4067-bb96-31412c9a48d0.)



This is what Database Query returns for my LMS stats (notice how Database Query thinks only 6496 tracks have a musicbrainz ID):

Number of contributors: 1555
Number of artists: 1298
Number of track artists: 433
Number of album artists: 424
Number of bands: 0
Number of conductors: 24
Number of composers: 49
Number of compilation albums: 273
Number of artist albums: 224
Number of genres: 227
Number of songs: 20910
Number of played unique songs: 0
Number of totally played songs:
Number of unplayed songs: 20806
Number of unplayed albums: 2036
Number of unplayed artists: 1554
Number of rated songs: 0
Number of unrated songs: 20806
Number of songs with musicbrainz tags: 6496
Number of songs without musicbrainz tags: 14414
Number of artists with musicbrainz tags: 157
Number of artists without musicbrainz tags: 1398
Number of albums with musicbrainz tags: 691
Number of albums without musicbrainz tags: 1346
Number of lossless compressed or uncompressed songs: 13363
Number of lossy compressed songs: 7547
Number of lossless compressed or uncompressed albums: 1161
Number of lossy compressed albums: 876
Number of alc songs: 13363
Number of mp3 songs: 7522
Number of mp4 songs: 25
Number of songs with comment tags: 97
Number of songs without comment tags: 20813
Number of albums with album art: 1998
Number of albums without album art: 39
Number of albums with album gain tags: 873
Number of albums without album gain tags: 1164
Number of songs with replay gain tags: 7538
Number of songs without replay gain tags: 13372
Number of MusicIP mixable songs: 0
Number of not MusicIP mixable songs: 20910
Number of songs with lyrics: 14836
Number of songs without lyrics: 6074
Number of local songs: 20910
Number of remote songs: 0
Number of songs tagged with ID3v2.3.0: 7440
Number of songs tagged with ID3v2.3.0, ID3v1.1: 82
Number of songs with sample rate 22050 Hz: 110
Number of songs with sample rate 32000 Hz: 2
Number of songs with sample rate 44100 Hz: 20782
Number of songs with sample rate 48000 Hz: 16
Total file size of local songs: 443 GB
Total length of local songs: 67.4 days

This is what Database Query returns when I query specifically for the Dominion/Mother Russia track (no musicbrainz ID).

url: file:///Volumes/MediaFiles/alac_mp3/alac/lief/The%20Sisters%20Of%20Mercy/Floodland/1987%20%C2%AC%2001-01%20%C2%AC%20Dominion%EF%BC%8Fmother%20Russia.m4a
musicbrainz_id:
playCount:
added: 1335883820
lastPlayed:
rating:
urlmd5: 1c7c0779158ebd4d1b99e1635b58d639

Why the discrepancy between Jaikoz and Database Query for the number of tracks with a musicbrainz ID? Why does Jaikoz tell me that my Dominion/Mother Russia file has a musicbrainz ID? Why does viewing track info show me the musicbrainz ID for this same file? But when I run a database query, it says the file does not have a musicbrainz ID?

It seems like I should figure this out before attempting to restore TrackStat data.

[erland]

Why the discrepancy between Jaikoz and Database Query for the number of tracks with a musicbrainz ID? Why does Jaikoz tell me that my Dominion/Mother Russia file has a musicbrainz ID? Why does viewing track info show me the musicbrainz ID for this same file? But when I run a database query, it says the file does not have a musicbrainz ID?

It seems like I should figure this out before attempting to restore TrackStat data.

Yes, this definitely looks strange.
Can you check what the "tracks" table shows for the track with Database Query plugin ?

select * from tracks where url like '%Sisters%Floodland%Dominion%mother%'

TrackStat will copy information from "tracks" table, so if the information hasn't been scanned by standard SBS/LMS scanner it means that tracks.musicbrainz_id is empty and then when TrackStat copy it the track_statistics.musicbrainz_id will also be empty. If track_statistics.musicbrainz_id is empty, it means that the TrackStat restore won't be able to use musicbrainz id during restore.

The "View Tags" page shows what's in the file, it doesn't show what's in the database, so even if it is in the file there might be some reason why the standard SBS/LMS scanner isn't able to scan it, which is very strange.

I'm not sure which tagging format that's used for m4a files, looking at the SBS/LMS code I suspect it either expects a tag:
MusicBrainz Track Id
or
UFID
or possibly
MUSICBRAINZ_ID

FLAC files uses MUSICBRAINZ_TRACKID (or possibly MUSICBRAINZ_ID)

And none of these seems to exist in your file when using the "View Tags" function. So assuming Jaikoz does the right thing, I suspect you might have found a bug in SBS/LMS scanner related to m4a files.

[prabbit]

Results from the query. As you can see, there is no musicbrainz ID.

id: 33126
url: file:///Volumes/MediaFiles/alac_mp3/alac/lief/The%20Sisters%20Of%20Mercy/Floodland/1987%20%C2%AC%2001-01%20%C2%AC%20Dominion%EF%BC%8Fmother%20Russia.m4a
title: Dominion/mother Russia
titlesort: DOMINION MOTHER RUSSIA
titlesearch: DOMINION MOTHER RUSSIA
customsearch:
album: 3096
tracknum: 1
content_type: alc
timestamp: 1335836402
filesize: 50273273
audio_size: 50273273
audio_offset: 0
year: 1987
secs: 421
cover: 141419
vbr_scale: 1
bitrate: 951834
samplerate: 44100
samplesize: 16
channels: 2
block_alignment:
endian:
bpm:
tagversion:
drm: 0
disc: 1
audio: 1
remote: 0
lossless: 1
lyrics:
musicbrainz_id:
musicmagic_mixable:
replay_gain:
replay_peak:
extid:
primary_artist: 2647
urlmd5: 1c7c0779158ebd4d1b99e1635b58d639
coverid: 097c907d
cover_cached: 1
virtual:
added_time: 1335885057
updated_time: 1335885057
dlna_profile:
hash:

This is what I see when I look at the extended tags in mp3tag. The m4a file appears to be using 'MusicBrainz Track Id' (at least that's what I see in mp3tag).



So, do I understand you correctly, that this appears to be more and more likely an LMS scanning issue with m4a tracks?

[erland]

So, do I understand you correctly, that this appears to be more and more likely an LMS scanning issue with m4a tracks?

Since the tracks.musicbrainz_id column is empty it means that the issue is either:
- Jaikoz using incorrect tag names
or
- SBS/LMS scanner reading incorrect tag names

The Picard (Musicbrainz official tagger) uses the following mapping:
http://wiki.musicbrainz.org/Picard_Tag_Mapping

The iTunes MP4 column says: ----:com.apple.iTunes:MusicBrainz Track Id
And SBS/LMS uses: MusicBrainz Track Id

So possibly it might be a problem with Jaikoz using capital letters in the tag name.
Maybe you can create a copy of a file and try to tag it with Picard and see if it tags it differently ?

[prabbit]

Maybe you can create a copy of a file and try to tag it with Picard and see if it tags it differently ?

I'm not seeing anything different using Picard. By the way, the screenshot in my previous post showed MP3TAG, not Jaikoz. I wanted to show that multiple editors/taggers were showing the same thing.

I've set all scanner logging options to debug and cleared my library. I am scanning now and will look at the log file for anything interesting. If I see anything, I'll post it.

How would I go about confirming the issue is an LMS scanning bug?

[erland]

How would I go about confirming the issue is an LMS scanning bug?

I would say that if the scanner doesn't pickup the Musicbrainz tag from a file that has been tagged with Picard (the official Musicbrainz tagger), it can be considered to be a LMS scanner issue, in this case please register a bug for it at http://bugs.slimdevices.com unless there already is one registered.