Weird Genpuid Problem

[slamhound]

In my setup I use genpuid to analyze and tag files before loading them into the SC library. That has all worked fine for quite some time, but recently, I started having a weird problem with genpuid.

I usually move new albums into a special directory and run genpuid on all files within that directory, then move the files to my music directory for the MusicMagic server and SC to find them. However, when I tried running this yesterday, genpuid would go through and fingerprint every file and then start the analysis. It would get through some songs (though never actually write the tags, which I think happens when it finishes everything) but then just stop working. The command didn't finish (in other words, I didn't get back to the command prompt), but the mipcore process was no longer showing in my list of processes, and the cpu usage went back down to nothing (this happens on a 64-bit arch linux machine).

Here's the strange thing: I tried running file by file and noticed that genpuid works for any file that is less than 8MB in size, but for anything larger it just stops running without completing the analysis. I've confirmed this with three different albums: Anything less than 8MB gets scanned, anything more doesn't. There are no error messages or anything.

These are MP3s downloaded from Amazon, and I've never had a problem with this before. I also often use flac, which are obviously often much larger than 8MB, and they always worked fine (though I haven't tried one recently).

Finally, I copied one directory with 9 files that were less than 8MB (and appropriately analyzed) and 4 files that were more than 8MB (which couldn't be analyzed) over to my Music directory. I added the files through the MusicMagic Server web interface, and it correctly identified the 9 files as being analyzed and the 4 needing validation. I clicked on "start validation," but it just never stopped. So it seems that it is not just a genpuid problem.

Any suggestions?

[audiomuze]

Gracenote has acquired and killed MusicIP technology. If you want to analyze new files you need to do so offline using MusicIP mixer.

[slamhound]

Thanks for the response; I've been looking at other threads, so I know that all the work has to be done locally; but I thought that at least while a server was available for genpuid to contact, it (rather than MusicIP) could do the work for me.

But here's the weird part--the problem is selective. I can analyze some of the songs that I've recently purchased and they work fine, but others don't (in the way described above). They are all new songs from new albums, so they certainly weren't analyzed when MusicIP was still providing the analysis.

In addition, when I use MusicIP to do the analysis (that's what "validate" does on the web interface, right?), it also gets stuck with these songs. For instance, if I add a 12 song album that has been analyzed in genpuid, but with 4 songs for which the process didn't work, then MusicIP will recognize all 12 new songs but then say that 4 need to be validated. When I click "validate", I just get the message "validating songs" but nothing ever happens. So genpuid (and MusicIP) still work perfectly for me for most songs--but there are just a few that it doesn't.

And just as an update, there does not seem to be anything magical about 8MB as a file size, though it still looks like smaller files do better.

[Phil Leigh]

Validation has been very slow for the last week or so...

[bookemdano]

I posted a similar experience in the other genpuid thread here. genpuid binaries for linux and macos seem to be having major issues.

If you have access to a windows machine, try using the windows version of genpuid. It is working for me--with several quirks.

1. If I use it to scan a single file it usually fetches the analysis off the server and the PUID to go along with it (assuming those tracks have had their analysis previously done and stored on the server).

2. If I use it to scan a single album, consisting of, say 10-15 tracks, it sometimes retrieves analysis for all tracks, sometimes just a few. The rest it performs local analysis on.

3. If I use it to scan a huge folder full of albums (with the -r flag) it may retrieve analysis on a couple of songs from the entire group, and analyzes everything else locally.

Makes me wonder if it is experiencing DDOS attacks or they've throttled it somehow to drop larger queries. Haven't figured it out yet.

But at a minimum the windows version of genpuid at least doesn't hang like the linux and mac versions do. I find genpuid better to use than the full Mixer app because I can specify multiple threads and make use of all my processor's cores--which saves a tremendous amount of time considering it's performing local analysis on just about everything.

Keep in mind if MusicIP/Gracenote ever takes the server completely down then genpuid will cease to work (the very first thing it does when you run it is check to see that the server is up and running. It's more sophisticated than a simple ping, but not sure exactly). Someone more experienced than I should wireshark it and see what can be discovered in the communications exchanged back and forth. Perhaps the server could be spoofed somehow.

I am scanning all FLACs, by the way, so 99.9% of my files are larger than 8MB.

[slamhound]

Thanks for the reply. Unfortunately, trying it on Windows has the same effect. It can scan some files, but it gets stuck on others. I've done more testing, and here's what I've found.

First, the problem is with specific files, not with analyzing in general. In other words, if I take an album of files and scan file by file, some files will be analyzed appropriately (doing local analysis) others will not. It is always the same files that don't get analyzed. In other words, if I copy a clean version of the files and try to re-analyze them, the ones that could be analyzed previously can again be analyzed again; the ones that previously failed will still fail.

Second, I've now tried this on my main Arch system, an Ubuntu 11.04 system, my previous Ubuntu 10.04 system that has always worked, and on a Windows XP system. I've also tried both genpuid and the MusicIP interface (from the headless server). The problem is exactly the same. All these systems will scan some files, but the files that don't get analyzed on one system also never get analyzed on the other. The way it fails is exactly the same: The log file says that the file is fingerprinted, and then for files that can't be analyzed, mipcore just disappears from the process list. I don't get the "analyzing" line in the log file for those files. In other words, if I analyze three files successfully and then it hits one that fails, the log will state "analyzing" and then "analyzed" for the first three songs, and then the log just ends with no other message and no note that it is analyzing the fourth song.

I checked, and this problem occurs for files that have successfully been tagged in the past. I copy a clean file (with no MusicIP tags) and only some can be analyzed. So it is not some change that Amazon did with their new files they are distributing.

I have also tried copying my mp3 files to flac, just to check. Files that can be analyzed as mp3s can also be analyzed as flac; files that cannot be analyzed as mp3s also cannot be analyzed as flac. So it is clearly not file size, as all are much larger as flacs; but it is the case that the longer songs tend not to be analyzed, and short songs are almost always analyzed.

I've tried removing all tags before analyzing (don't think that should matter) and the problem persists.

I'm stuck and don't know what else to try!

[bookemdano]

Odd, it's still working for me as of this evening.

What version of genpuid are you using on Windows? What command line options are you using? Here's the way I'm doing it:

genpuid (dns key) -archive -data -logex -print -r -threads=4 -xml=genpuid.xml "c:\flac"

It never hangs at all, even feeding it hundreds of files (via the -r switch).

The only times it can't generate or download analysis is if the file is too short (usually short "interlude"-type tracks or silent tracks) or if the file name contains illegal windows characters (? / \ * < > : etc.) or if the filename is too long. But in all of those cases genpuid simply skips over the file and proceeds with the others.

I think cparker mentioned in the other thread that sometimes genpuid/mipcore can choke on weirdly formatted tags too. So another thing you could try would be removing all ID3/vorbiscomment tags from your files and try again.

Perhaps another difference is I am not caching the analysis in a m3.lib file. My sole goal here is to get the analysis tag in all my flac files. Once that's done I will allow MusicIP Mixer to create its m3.lib file by reading all those tags.

Hopefully that gives you some more stuff to try. Please do report back the version(s) of genpuid you're using though--that could be it.

[slamhound]

Thanks for the suggestions. I'm using version 1.4 (downloaded from here: http://ftp.musicbrainz.org/pub/musicbrainz/genpuid/). The linux version is also 1.4, but I downloaded a long time ago, so I might have got it directly from MusicIP.

I use similar options as you, but I've also tried just about every combination of options that are available with the same result.

And I've tried removing all the tags before analyzing and has the same result.

To me, the strangest things are: (a) specific files cause the problem and these files cannot be analyzed, no matter how I try them (different formats, different computers, different OSs, different tags, different genpuid options); and (b) files that I have analyzed in the past can no longer be analyzed, even though these files have not changed and the software has not changed. Very strange.

[bookemdano]

Thanks for the suggestions. I'm using version 1.4 (downloaded from here: http://ftp.musicbrainz.org/pub/musicbrainz/genpuid/). The linux version is also 1.4, but I downloaded a long time ago, so I might have got it directly from MusicIP.

I use similar options as you, but I've also tried just about every combination of options that are available with the same result.

And I've tried removing all the tags before analyzing and has the same result.

To me, the strangest things are: (a) specific files cause the problem and these files cannot be analyzed, no matter how I try them (different formats, different computers, different OSs, different tags, different genpuid options); and (b) files that I have analyzed in the past can no longer be analyzed, even though these files have not changed and the software has not changed. Very strange.

Yeah I am using version 1.4 here as well. If it makes any difference I am running it under Win7 x64 on a Sandy Bridge Xeon 3440 (2.4Ghz Quad Core) processor.

One other thing you might try is re-rip the song from the original CD (or maybe re-download it from somewhere). Also if you want to upload your copy to somewhere (mediafire is good) I can try analyzing it here to see if it gets hung up on my system too.

Unfortunately there doesn't seem to be too many people left using genpuid anymore so I don't know if you're the outlier or I am. I know the linux and MacOS versions were both hanging on me but the Windows version has been working just fine (aside from having to perform local analysis on just about everything).

[slamhound]

I tried to find a freely available song that didn't work for me. I just downloaded the first song from this album: http://www.amazon.com/gp/product/B00...rd_i=678551011 and it didn't work.

So you could try that one (if you can download songs from Amazon).