PDA

View Full Version : Plugin to import a CD into the library



iwp
2008-01-21, 14:51
Hi all,

As I alluded to in the 3rd Party Plugins forum, I've been playing with a plugin which greatly simplifies the task of importing a CD into your SqueezeCenter music library. The idea is that, with as little user intervention as possible, it will rip, flac and tag the CD, fetch the correct coverart, add it to your music directory and trigger a rescan of just the new music.

I've attached my current attempt. It works well enough but - be warned - this is very much early and experimental code. They say the best way to write code is to write it once, then throw it away and write it again - well this is the version that will probably get thrown away! So I'm not really interested in bug reports/beta testing at the moment (there's plenty wrong with the attached code) - I'm publishing it looking really for any comments or suggestions on the overall approach before spending time coding it properly :)

It only works with SqueezeCenter v7 on Linux (I've developed it on Ubuntu Gutsy), and you will need to make sure the "cdparanoia" and "flac" commands are installed and available. The squeezecenter user needs permission to read from the CD drive (on Ubuntu, you need to add the squeezecenter user to the cdrom group) and to write to your audio directory. Obviously make sure you've got your music backed up, or use a test system! Install the plugin as usual; it appears under "Extras" on the SC web app. It also only works with /dev/cdrom at present, if you have multiple CD drives make sure you put the disc into the right one!

Reflecting this plugin's prototype status, it has debug trace enabled by default - error handling is not entirely complete, so if it's not working, keep an eye on the server log. The ripping process does most of its work inside a directory in /tmp, so you may have to clear up in there if things go wrong. The plugin will move the ripped music files into a "cdimport" directory in your audio directory; this lets you clear out anything that's added incorrectly, and also means you don't have to mark your whole audio directory as being writable by squeezecenter. Underneath "cdimport" it uses a artist/album/trackname directory structure.

The process of ripping seems to take 10-15 minutes for most CDs, I feel it should be quicker than that but I'm not sure why it's taking so long. If you try this, please let me know how long it takes so I can see whether it's just my system or a genuine problem.

Tag data is obtained from Musicbrainz - if MB doesn't recognise the disc, it will currently refuse to import it. Album art comes from Amazon, using the Amazon ASID which is provided by Musicbrainz. If MB doesn't know the ASID, or Amazon doesn't have an image, then it should let you import the CD but you won't get any coverart. There's a couple of problems here: MB doesn't give us "genre" information so the tracks are tagged without any genre. Also Amazon's coverart seems to be very variable in quality. I've been looking for solutions to these problems, but haven't found anything really good yet... Finally, I've had to reverse engineer the MusicBrainz tag names by looking at some files tagged by the MB applications, and there's a couple I haven't worked out yet (see comments in CDInfo.pm).

I'd like to be able to include MusicIP and ReplayGain tags as well, but I've not looked at these yet. Are there any other tags that could usefully be added when importing music?

The web front end to this is a simple AJAX thing; this is the first time I've tried AJAX techniques so I'd particularly appreciate any expert reviews of this :) It's only tested with Firefox so far; it's pretty ugly but it's a reasonable proof-of-concept. The plugin tries to be non-blocking, so you can navigate away from the page while it's ripping and come back later.

All time-consuming work is shipped off to other processes using Perl forking, using:

open "$command |"and IO::Select to handle non-blocking reading of the responses. This seems to work ok, but I'm concerned that forking SqueezeCenter might not be very wise. Could some kindly perl/SC expert put me at ease or suggest a better way of handling this? The foking is encapsulated in Fork.pm.

The scan is triggered using:

Slim::Utils::Scanner->scanDirectory({'url' => $albumdir,
'recursive' => 0,
'types' => 'audio',
'scanName' => 'directory',
});Again, this mostly works ok, but could someone please comment on whether this is the right way to do this? There's one bug - this doesn't pick up the artwork until you navigate to the album page of a newly-ripped album. Is there some other call I should be making to do this?

Thanks for reading to the end of this long post.... any and all comments most welcome!

Ian

mflint
2008-01-21, 15:04
Ian,

That is a funky piece of work, nice one!

(A great pity it's no use to me as my SqueezeCenter box doesn't have a CD drive, but...)

Matthew

dean
2008-01-21, 16:10
Very nice! I'll give it a try with my Ubuntu server here...

mrfantasy
2008-01-21, 17:27
Cool idea!

I've got my linux box running abcde (http://www.hispalinux.es/~data/abcde.php, although the SVN server seems to be down) and it handles the autorip, and tagging, but it doesn't do cover art or the adding to the library. So I'll give this a shot!

kdf
2008-01-21, 20:04
On 21-Jan-08, at 1:51 PM, iwp wrote:

>
> --------------------
> Again, this mostly works ok, but could someone please comment on
> whether
> this is the right way to do this? There's one bug - this doesn't
> pick up
> the artwork until you navigate to the album page of a newly-ripped
> album. Is there some other call I should be making to do this?

Have you considered Slim::Utils::Misc::findAndScanDirectoryTree?

This is what is called while using BMF and it handles adding artwork
where present.
it takes a hashref as a param, and you can use the key 'url' for the
top level.
-kdf

iwp
2008-01-22, 14:02
Have you considered Slim::Utils::Misc::findAndScanDirectoryTree?


Thanks, kdf! I've got it working with that, but I've had to work around what I think is a problem in findAndScanDirectoryTree. If I invoke it after building the output directory in the library, nothing appears to happen. The following gets logged:


[08-01-22 20:19:03.3408] Plugins::CDImport::Ripper::finalize (252) Rescanning directory using URL file:///home/iwp/Music/cdimport/Muse/Showbiz
[08-01-22 20:19:03.4572] Slim::Utils::Misc::findAndScanDirectoryTree (1013) findAndScanDirectoryTree( /home/iwp/Music/cdimport/Muse/Showbiz ): fsMTime: 1201033143, dbMTime: 1201033143
[08-01-22 20:19:03.4617] Plugins::CDImport::Ripper::finalize (256) Done

Looking at the findAndScanDirectoryTree code, the problem is that fsMTime==dbMTime, so it thinks there's nothing to do. I guess that, because this is the first time it's seen this directory, it's created a new 'track object' with an up-to-date timestamp; and then compared this to the timestamp of the directory on the disk.

I've worked around it by driving findAndScanDirectoryTree before constructing the output directory, to initialize the track object; and then driving it again at the end. This works a treat, including album art.

I'm invoking it with:

my $pathAsURL = Slim::Utils::Misc::fileURLFromPath($albumdir);
Slim::Utils::Misc::findAndScanDirectoryTree({
'url' => $pathAsURL
});

...am I doing something wrong or is this a problem in findAndScanDirectoryTree?

Thanks again,
Ian

kdf
2008-01-22, 14:31
> ...am I doing something wrong or is this a problem in
> findAndScanDirectoryTree?

The problem is that this is the bit of code used by Browse Music Folder
when collecting the folder list. This includes the check against the
timestamps in order to trigger a scan, or not. Since this is a brand new
folder, the timestamps do not differ and you end up not scanning, as you
have discovered.

This is the expected behaviour, but your use case clearly shows the
limitation of how this is done. There isn't really any way for this
routine to know whether or not a new folder has created a new track item
or not. In the designed case, you'd always trigger a scan at the parent
folder.

Your workaround seems perfectly valid as long as it's not triggering a
full file scan prematurely. It might be simpler to tighten one more level
and use the objectForUrl call:
$topLevelObj = Slim::Schema->rs('Track')->objectForUrl({
'url' => $url,
'create' => 1,
'readTags' => 1,
'commit' => 1,
});

If you do this when you've defined the albumdir, then the path will
already exist in the db and should allow findAndScan to work. Another
option would be to do this at the default top level for your album
folders, then run the scan on that, but then you may end up scanning more
files that you want to if they aren't always being moved away.

-kdf

iwp
2008-01-23, 04:40
Thanks again, kdf. I'll stick to my present workaround for the time being I think - bringing knowledge of the DB schema stuff into the plugin feels like bad encapsulation; and going to the top level folder feels like overkill.

Ian

SuperQ
2008-02-06, 19:59
Awesome, I'm glad someone finally started working on a plugin like this. Two ideas:

I use a lastfm plugin for MusicBrainz Picard to gather genere data which seems to work really well. I don't know how much work it would be to add that data.

You could also grab cover art from Apple, although I don't know about the legality of that. There is code out there:
http://avantlumiere.com/projects/i7awf/

iwp
2008-02-07, 04:23
Awesome, I'm glad someone finally started working on a plugin like this. Two ideas:

I use a lastfm plugin for MusicBrainz Picard to gather genere data which seems to work really well. I don't know how much work it would be to add that data.

You could also grab cover art from Apple, although I don't know about the legality of that. There is code out there:
http://avantlumiere.com/projects/i7awf/

Hi SuperQ,

Many thanks for the input! I spotted the last.fm/Picard plugin a couple of days ago but haven't tried it yet. You're the second user I've seen saying that it works nicely, so I will take a closer look at how it works.

I haven't seen the Apple code before - again this sounds like a good idea. Will take a look.

BTW, I'm slowly going through the code making the plugin more robust. If anyone wants to play with more up to date code, see http://sourceforge.net/projects/squimport/ . (And if anyone would like to contribute some code... well there's plenty to do. I'd particularly like to find someone who understands SC skins and javascript to improve the web interface; and perhaps somebody would like to take a crack at making it work on Windows?)

Cheers,

Ian

pippin
2008-02-07, 05:10
Awesome, I'm glad someone finally started working on a plugin like this. Two ideas:

I use a lastfm plugin for MusicBrainz Picard to gather genere data which seems to work really well. I don't know how much work it would be to add that data.

You could also grab cover art from Apple, although I don't know about the legality of that. There is code out there:
http://avantlumiere.com/projects/i7awf/

Does the Apple code still work? Thought they encode the artwork now...

funkstar
2008-02-07, 12:26
Fantastic idea for a plugin iwp. Major cudos to you :)

I wonder if you could write a module for the N5200 that would add a USB CD drive and then bundle this plugin to turn the NAS box into a full ripping and SqueezeCetner box.

Could be a killer combination :)

iwp
2008-02-07, 12:53
Does the Apple code still work? Thought they encode the artwork now...

Just tried it, albeit very quickly - I get a URL back but my browser won't render the image; so I guess you're right. Also, that source code looks like it tries to masquerade as the iTunes application. Oh well, a nice thought.

funkstar - thanks. A full-featured dedicated Squeezecenter server is exactly the inspiration for this.

Ian

kolding
2008-02-07, 17:35
Just tried it, albeit very quickly - I get a URL back but my browser won't render the image; so I guess you're right. Also, that source code looks like it tries to masquerade as the iTunes application. Oh well, a nice thought.
Ian

Apple actually sends a JPEG or GIF file with some sort of data chunk in front of it. In general, if you strip off the first 500 bytes, it seems to become a valid image file. There's a description (see below) out on the web, and as I recall, the chunk could be a different size, as specified by a couple of fields in the data, but always appears to be 500 bytes.

Here's the description I saw a while back...

http://www.falsecognate.org/2007/01/deciphering_the_itunes_itc_fil.php

Eric

reverber
2008-02-15, 15:54
Very nice idea!
I will be trying it out. Does it rip to a single flac per album? If not, please consider this a feature request ;).

TIA,

Cody

bpa
2008-02-28, 19:20
Attached is a version of the CDImport plugin which supports Windows. The plugin is still a prototype but feedback from Windows users would be useful to focus development effort.

On Windows the plugin uses CDDA2WAV instead of CDparanoia. I have bundled the necessary file (cdda2waw.exe, cygwin1.dll annd ,metaflac.exe) into a single zip file to simplify installation.

To install
1. Download the attached zip file and unzip and put CDImport folder into SqueezeCenter server/Plugins directory.
If you have a standard windows installation - then just right click the ZIP file and "Extract All" to
C:\Program Files\SqueezeCenter\server\Plugins
2. Download the zipped binary files from http://homepage.eircom.net/~altondsl/slim/CDImportBin.zip . Then unzip the file
into the Plugins\CDImport directory such that there is a Bin directory.
If you have a standard windows installation - then just right click the ZIP file and "Extract All" to
C:\Program Files\SqueezeCenter\server\Plugins
3. If you already have Cygwin installed - delete the cygwin1.dll from the Bin airectory.
4. Restart SqueezeCenter - there should be a "CD Import" menu option under Extras. Check the "CD Importer" plugin settings to see that the correct CD drive has been identified. This is necessary for user with more than one CD-ROM drive.

Also in settings there are two tick boxes
(i) to choose whether to copy the ripped and compressed files into a directory in your SqueezeCenter music directory.
(ii) if (i) is enabled then the 2nd check box will trigger a SqueezeCenter scan of the new directory.

In normal operation both these boxes will be checked but as shipped they are unchecked as user may initially want to test rips without copying files into SqueezeCenter.

During testing I found problems in the following area.
1. IE7 caching and some other IE7 DOM quirks. No problems when using Firefox.
2. A DVD-RW drive needed a firmware update before it would work properly.
3. A Laptop with only 512Mb RAM could not rip a CD when it was running Squeezecenter and a Firefox browser. Not enough memory.

So if you find problems besides normal details please Browser type, drive info incl firmware rev and system details esp. memory.


Caution: The "Abort" while scanning is bit unreliable at present and the scanning process CDDA2WAV may keep running. After pressing abort wait until all "Rip" activity stops before click another button.

CDImport has not been tested on Vista but cdda2wav.exe is supposed to work on Vista.

iwp
2008-02-29, 06:21
Brilliant stuff, thanks for doing all this! (And I know I'm yet to reply to your last email)

Ian

pippin
2008-02-29, 06:53
Great tool.
May I file a feature request?
What I would like to see is the possibility to play a CD while ripping. This is how it could look like: You enter a CDD, ripping starts, and after track 1 is finished, you do a "loadtracks" with the URL of the first track and all subsequent tracks are being added as soon as they are ready through "addtracks". AFAIK this would also add those tracks to the database so a seperate scan would no longer be necessary.

Possible?
If I can be of any help regarding the UI (can't do Perl ;-), let me know.

bpa
2008-02-29, 07:40
My feeling is that the RIP process needs to be improved before adding other features. The improvement I think include

1. Allow user to override or provide some tag info in addition to the MusicBrainz information. For example, man albums have no genre assigned or a user may have their own generes.

2. Provide alternative album art choices such as from Yahoo, Walmart or Google images.

3. When MB fails to provide info - look up other databases.

4. Just RIP selected tracks not necessarily whole albums.

5. Finer control over compress format and options.

6. Support flexible naming and positioning of ripped files so that they are copied into a directory and file structure using Album, Artist names etc. details according to user requirements.

One of the target user group of the Plugin will be new users of SC and the need to make it easy to rip their CD collections. Having a UI to support the above features would make it complicated so there is probably a needs for an "Advanced" mode.

pippin
2008-02-29, 08:06
My feeling is that the RIP process needs to be improved before adding other features. The improvement I think include

1. Allow user to override or provide some tag info in addition to the MusicBrainz information. For example, man albums have no genre assigned or a user may have their own generes.

2. Provide alternative album art choices such as from Yahoo, Walmart or Google images.

3. When MB fails to provide info - look up other databases.

4. Just RIP selected tracks not necessarily whole albums.

5. Finer control over compress format and options.

6. Support flexible naming and positioning of ripped files so that they are copied into a directory and file structure using Album, Artist names etc. details according to user requirements.

One of the target user group of the Plugin will be new users of SC and the need to make it easy to rip their CD collections. Having a UI to support the above features would make it complicated so there is probably a needs for an "Advanced" mode.

I understand that. I have a different target group in mind ;-)
To avoid complicated, drop flexibility and have defaults that make sense. And have a good UI, of course ;-)

I disagree on two of your points:

1. No. Not necessary. Allow users to change what is in SC. That's a different task but solves quite a few issues.
6. That is really advanced and only needs to be provided in an "options" section.

bpa
2008-02-29, 08:18
1. No. Not necessary. Allow users to change what is in SC. That's a different task but solves quite a few issues.

One of the fundamental principles of SC is that it doesn't change the user's files or directories nor does it move files. In keeping with that principle - I felt it would be better to get the tags as right as possible when creating the file rather than have a plugin change the tags and force a rescan

From my perspective - Point 6 is the easiest one to implement just using the "normal" approach of extending a printf format (something like %A = Artist %B = Album name %N = Track number %T = Track name) as used by many file management apps.

iwp
2008-02-29, 12:19
Just to cast my votes.... I'm absolutely with bpa on getting the tags right upfront and avoid changing them later. Pragmatically this may not entirely be possible but it's a reasonable goal to aim for. In any case, there's a billion and one retagging applications out there already.

But I'd be with Pippin on putting directory structure configuration in the options.

I also agree that we should focus on the rip process for now. "Playing while ripping" is a great idea for the future. Something that would be easier in the shorter term: I'd like to be able to play a recently ripped CD from the CDImport plugin's web page - not quite as immediate as "playing while ripping" but avoids all sorts of nasty issues about skipping ahead of the rip etc. :)

Ian

dean
2008-02-29, 12:44
I agree, ideally the tag information would be in editable fields on
the page so that they could be tweaked before hitting the Import button.

Extra credit for an upload button to replace the artwork.

-dean

john_lenfr
2017-03-05, 03:24
Attached is a version of the CDImport plugin which supports Windows. The plugin is still a prototype but feedback from Windows users would be useful to focus development effort.

On Windows the plugin uses CDDA2WAV instead of CDparanoia. I have bundled the necessary file (cdda2waw.exe, cygwin1.dll annd ,metaflac.exe) into a single zip file to simplify installation.

To install
1. Download the attached zip file and unzip and put CDImport folder into SqueezeCenter server/Plugins directory.
If you have a standard windows installation - then just right click the ZIP file and "Extract All" to
C:\Program Files\SqueezeCenter\server\Plugins
2. Download the zipped binary files from . Then unzip the file
into the Plugins\CDImport directory such that there is a Bin directory.
If you have a standard windows installation - then just right click the ZIP file and "Extract All" to
C:\Program Files\SqueezeCenter\server\Plugins
3. If you already have Cygwin installed - delete the cygwin1.dll from the Bin airectory.
4. Restart SqueezeCenter - there should be a "CD Import" menu option under Extras. Check the "CD Importer" plugin settings to see that the correct CD drive has been identified. This is necessary for user with more than one CD-ROM drive.

Also in settings there are two tick boxes
(i) to choose whether to copy the ripped and compressed files into a directory in your SqueezeCenter music directory.
(ii) if (i) is enabled then the 2nd check box will trigger a SqueezeCenter scan of the new directory.

In normal operation both these boxes will be checked but as shipped they are unchecked as user may initially want to test rips without copying files into SqueezeCenter.

During testing I found problems in the following area.
1. IE7 caching and some other IE7 DOM quirks. No problems when using Firefox.
2. A DVD-RW drive needed a firmware update before it would work properly.
3. A Laptop with only 512Mb RAM could not rip a CD when it was running Squeezecenter and a Firefox browser. Not enough memory.

So if you find problems besides normal details please Browser type, drive info incl firmware rev and system details esp. memory.


Caution: The "Abort" while scanning is bit unreliable at present and the scanning process CDDA2WAV may keep running. After pressing abort wait until all "Rip" activity stops before click another button.

CDImport has not been tested on Vista but cdda2wav.exe is supposed to work on Vista.


Hi bpa,

I downloaded the zip file and copy/paste dll and exe files in plugin directory.

But I think I missed something because I don't have access to the pluggin parameters.

Do you have any idea why?

Also, I can't find the "Download the attached zip file " on your older post.

Thanks,

john_lenfr
2017-03-05, 03:29
Attached is a version of the CDImport plugin which supports Windows. The plugin is still a prototype but feedback from Windows users would be useful to focus development effort.

On Windows the plugin uses CDDA2WAV instead of CDparanoia. I have bundled the necessary file (cdda2waw.exe, cygwin1.dll annd ,metaflac.exe) into a single zip file to simplify installation.

To install
1. Download the attached zip file and unzip and put CDImport folder into SqueezeCenter server/Plugins directory.
If you have a standard windows installation - then just right click the ZIP file and "Extract All" to
C:\Program Files\SqueezeCenter\server\Plugins
2. Download the zipped binary files from . Then unzip the file
into the Plugins\CDImport directory such that there is a Bin directory.
If you have a standard windows installation - then just right click the ZIP file and "Extract All" to
C:\Program Files\SqueezeCenter\server\Plugins
3. If you already have Cygwin installed - delete the cygwin1.dll from the Bin airectory.
4. Restart SqueezeCenter - there should be a "CD Import" menu option under Extras. Check the "CD Importer" plugin settings to see that the correct CD drive has been identified. This is necessary for user with more than one CD-ROM drive.

Also in settings there are two tick boxes
(i) to choose whether to copy the ripped and compressed files into a directory in your SqueezeCenter music directory.
(ii) if (i) is enabled then the 2nd check box will trigger a SqueezeCenter scan of the new directory.

In normal operation both these boxes will be checked but as shipped they are unchecked as user may initially want to test rips without copying files into SqueezeCenter.

During testing I found problems in the following area.
1. IE7 caching and some other IE7 DOM quirks. No problems when using Firefox.
2. A DVD-RW drive needed a firmware update before it would work properly.
3. A Laptop with only 512Mb RAM could not rip a CD when it was running Squeezecenter and a Firefox browser. Not enough memory.

So if you find problems besides normal details please Browser type, drive info incl firmware rev and system details esp. memory.


Caution: The "Abort" while scanning is bit unreliable at present and the scanning process CDDA2WAV may keep running. After pressing abort wait until all "Rip" activity stops before click another button.

CDImport has not been tested on Vista but cdda2wav.exe is supposed to work on Vista.

Is it abandonned pluggin, I can't install it?