Weird Genpuid Problem

[slamhound]

In my setup I use genpuid to analyze and tag files before loading them into the SC library. That has all worked fine for quite some time, but recently, I started having a weird problem with genpuid.

I usually move new albums into a special directory and run genpuid on all files within that directory, then move the files to my music directory for the MusicMagic server and SC to find them. However, when I tried running this yesterday, genpuid would go through and fingerprint every file and then start the analysis. It would get through some songs (though never actually write the tags, which I think happens when it finishes everything) but then just stop working. The command didn't finish (in other words, I didn't get back to the command prompt), but the mipcore process was no longer showing in my list of processes, and the cpu usage went back down to nothing (this happens on a 64-bit arch linux machine).

Here's the strange thing: I tried running file by file and noticed that genpuid works for any file that is less than 8MB in size, but for anything larger it just stops running without completing the analysis. I've confirmed this with three different albums: Anything less than 8MB gets scanned, anything more doesn't. There are no error messages or anything.

These are MP3s downloaded from Amazon, and I've never had a problem with this before. I also often use flac, which are obviously often much larger than 8MB, and they always worked fine (though I haven't tried one recently).

Finally, I copied one directory with 9 files that were less than 8MB (and appropriately analyzed) and 4 files that were more than 8MB (which couldn't be analyzed) over to my Music directory. I added the files through the MusicMagic Server web interface, and it correctly identified the 9 files as being analyzed and the 4 needing validation. I clicked on "start validation," but it just never stopped. So it seems that it is not just a genpuid problem.

Any suggestions?

[audiomuze]

Gracenote has acquired and killed MusicIP technology. If you want to analyze new files you need to do so offline using MusicIP mixer.

[slamhound]

Thanks for the response; I've been looking at other threads, so I know that all the work has to be done locally; but I thought that at least while a server was available for genpuid to contact, it (rather than MusicIP) could do the work for me.

But here's the weird part--the problem is selective. I can analyze some of the songs that I've recently purchased and they work fine, but others don't (in the way described above). They are all new songs from new albums, so they certainly weren't analyzed when MusicIP was still providing the analysis.

In addition, when I use MusicIP to do the analysis (that's what "validate" does on the web interface, right?), it also gets stuck with these songs. For instance, if I add a 12 song album that has been analyzed in genpuid, but with 4 songs for which the process didn't work, then MusicIP will recognize all 12 new songs but then say that 4 need to be validated. When I click "validate", I just get the message "validating songs" but nothing ever happens. So genpuid (and MusicIP) still work perfectly for me for most songs--but there are just a few that it doesn't.

And just as an update, there does not seem to be anything magical about 8MB as a file size, though it still looks like smaller files do better.

[Phil Leigh]

Validation has been very slow for the last week or so...

[bookemdano]

I posted a similar experience in the other genpuid thread here. genpuid binaries for linux and macos seem to be having major issues.

If you have access to a windows machine, try using the windows version of genpuid. It is working for me--with several quirks.

1. If I use it to scan a single file it usually fetches the analysis off the server and the PUID to go along with it (assuming those tracks have had their analysis previously done and stored on the server).

2. If I use it to scan a single album, consisting of, say 10-15 tracks, it sometimes retrieves analysis for all tracks, sometimes just a few. The rest it performs local analysis on.

3. If I use it to scan a huge folder full of albums (with the -r flag) it may retrieve analysis on a couple of songs from the entire group, and analyzes everything else locally.

Makes me wonder if it is experiencing DDOS attacks or they've throttled it somehow to drop larger queries. Haven't figured it out yet.

But at a minimum the windows version of genpuid at least doesn't hang like the linux and mac versions do. I find genpuid better to use than the full Mixer app because I can specify multiple threads and make use of all my processor's cores--which saves a tremendous amount of time considering it's performing local analysis on just about everything.

Keep in mind if MusicIP/Gracenote ever takes the server completely down then genpuid will cease to work (the very first thing it does when you run it is check to see that the server is up and running. It's more sophisticated than a simple ping, but not sure exactly). Someone more experienced than I should wireshark it and see what can be discovered in the communications exchanged back and forth. Perhaps the server could be spoofed somehow.

I am scanning all FLACs, by the way, so 99.9% of my files are larger than 8MB.

[slamhound]

Thanks for the reply. Unfortunately, trying it on Windows has the same effect. It can scan some files, but it gets stuck on others. I've done more testing, and here's what I've found.

First, the problem is with specific files, not with analyzing in general. In other words, if I take an album of files and scan file by file, some files will be analyzed appropriately (doing local analysis) others will not. It is always the same files that don't get analyzed. In other words, if I copy a clean version of the files and try to re-analyze them, the ones that could be analyzed previously can again be analyzed again; the ones that previously failed will still fail.

Second, I've now tried this on my main Arch system, an Ubuntu 11.04 system, my previous Ubuntu 10.04 system that has always worked, and on a Windows XP system. I've also tried both genpuid and the MusicIP interface (from the headless server). The problem is exactly the same. All these systems will scan some files, but the files that don't get analyzed on one system also never get analyzed on the other. The way it fails is exactly the same: The log file says that the file is fingerprinted, and then for files that can't be analyzed, mipcore just disappears from the process list. I don't get the "analyzing" line in the log file for those files. In other words, if I analyze three files successfully and then it hits one that fails, the log will state "analyzing" and then "analyzed" for the first three songs, and then the log just ends with no other message and no note that it is analyzing the fourth song.

I checked, and this problem occurs for files that have successfully been tagged in the past. I copy a clean file (with no MusicIP tags) and only some can be analyzed. So it is not some change that Amazon did with their new files they are distributing.

I have also tried copying my mp3 files to flac, just to check. Files that can be analyzed as mp3s can also be analyzed as flac; files that cannot be analyzed as mp3s also cannot be analyzed as flac. So it is clearly not file size, as all are much larger as flacs; but it is the case that the longer songs tend not to be analyzed, and short songs are almost always analyzed.

I've tried removing all tags before analyzing (don't think that should matter) and the problem persists.

I'm stuck and don't know what else to try!