SBS 7.6: please test automatic artwork download

[mherger]

I've added some (experimental) code to 7.6 which allows you to have the music scanner automatically download missing album artwork for your music. It's working good enough to give it some broader testing. Scanning my test library of about 670 albums it was able to find roughly 90% of all artwork automatically.

The artwork download is an additional step during the scanning process. It's disabled by default, but I'd love to hear some feedback how it's working for you.

In order to enable the automatic artwork download, please follow these instructions:
- install SBS 7.6/r31180 or later
- make sure it's using test.mysqueezebox.com: you should have use_sn_test:1 in your server.prefs file. If you don't, add it, while SBS is shut down, then restart the server
- enable the download in Settings/Advanced/Formatting
- re-run a scan

If you're an iTunes user (using our iTunes integration) I would be interested to know how this compares in performance and results with our iTunes artwork importer (which can be enabled/disabled on the iTunes settings page).

A few caveats:
- automatic artwork download will take some time, as it's communicating with external resources
- mysb.com is using last.fm as the source for the covers. Thus the quality of the artwork can vary a lot (I've seen >1600x1600px PNG as well as 100x100px JPG)
- mysb.com will currently limit the artwork size to max. 500x500px. Larger files are being scaled down. This value is a deliberate decision based on the experience during testing, but it's not set in stone.

Have fun!

--

Michael

[pippin]

Any chance for 768x768? That would be cool for iPad...

[erland]

I've added some (experimental) code to 7.6 which allows you to have the music scanner automatically download missing album artwork for your music. It's working good enough to give it some broader testing. Scanning my test library of about 670 albums it was able to find roughly 90% of all artwork automatically.

The artwork download is an additional step during the scanning process. It's disabled by default, but I'd love to hear some feedback how it's working for you.

Where will the artwork be stored ? In cache, music folder or somewhere else ?
Can I somehow run it on my library without removing all cover.jpg file I currently have in the folder of all albums ? I want to make it retrieve artwork for all albums just to see how good it works.

[mherger]

> Any chance for 768x768? That would be cool for iPad...

Try the following patch to override the default value:

Index: Artwork.pm
===================================================================
--- Artwork.pm	(revision 31181)
+++ Artwork.pm	(working copy)
@@ -638,6 +638,7 @@
  				my $url = $snURL
  					. '?album=' . URI::Escape::uri_escape_utf8( $albumname )
  					. '&artist=' . URI::Escape::uri_escape_utf8( $contributor )
+					. '&width=768'
  					. '&mbid=' . $album_mbid;

Unfortunately test.mysb.com is encountering some issues right now. The service isn't working :-(. I hope we'll manage to fix it before the weekend...

--

Michael

[mherger]

> Where will the artwork be stored ? In cache, music folder or somewhere
> else ?

[sb cachefolder]/DownloadedArtwork

> Can I somehow run it on my library without removing all cover.jpg file
> I currently have in the folder of all albums ? I want to make it
> retrieve artwork for all albums just to see how good it works.

Good plan - what I did too. Simplest solution: disable the filter on the query.

Index: Artwork.pm
===================================================================
--- Artwork.pm	(revision 31181)
+++ Artwork.pm	(working copy)
@@ -558,7 +558,7 @@
  	# Find distinct albums to check for artwork.
  	my $tracks = Slim::Schema->search('Track', {
  		'me.audio'   => 1,
-		'me.coverid' => { '='  => undef },
+#		'me.coverid' => { '='  => undef },
  	}, {
  		'join'     => 'album',
  	});

You'd then be able to inspect the downloaded files in the aforementioned folder. Unfortunately SBS would still show whatever artwork it did find, not only the downloaded. This might require a larger patch (which I haven't tested before posting, but I did something similar earlier today):

Index: Artwork.pm
===================================================================
--- Artwork.pm	(revision 31181)
+++ Artwork.pm	(working copy)
@@ -558,7 +558,7 @@
  	# Find distinct albums to check for artwork.
  	my $tracks = Slim::Schema->search('Track', {
  		'me.audio'   => 1,
-		'me.coverid' => { '='  => undef },
+#		'me.coverid' => { '='  => undef },
  	}, {
  		'join'     => 'album',
  	});
@@ -688,6 +688,15 @@
  			
  			else {
  				main::DEBUGLOG && $importlog->is_debug && $importlog->debug( "Failed to download artwork for $albumname" );
+
+				# remove any artwork if nothing has been found
+				$track->cover( undef );
+				$track->update;
+
+				$track->coverid( undef );
+				$track->update;
+
+				$track->album->artwork( undef );
  				
  				$cache{"artwork_download_failed_$albumid"} = 1;
  			}

This will basically remove existing artwork data from tracks/albums which were not found online.

test.mysb.com seems to be back online...

--

Michael

[copperstate]

Hi Michael,

here are my results. Each test was done using a clean rescan of the whole library:

Version: 7.6.0 - r31181 @ Sun Aug 8 02:00:52 PDT 2010
Operating system: Mac OS X 10.6.4 - EN - utf8
Database Version: MySQL 5.0.22-standard


1. Without any deliberate artwork scan:

Music Scan Details
Discovering files (3912 of 3912) Complete 00:00:06
Scan new files (2756 of 2756) Complete 00:02:35
Discovering playlists (1 of 1) Complete 00:00:00
iTunes Import (2947 of 2947) Complete 00:00:28
iTunes Playlist Import (217 of 217) Complete 00:00:35
Pre-caching Artwork (19 of 19) Complete 00:00:31

The server has finished scanning your music collection.
Total Time: 00:04:15 (Sunday, 8 August 2010 / 13.23)


2. Artwork scan using iTunes (done yesterday thus still 7.6.0 - r31176):

Music Scan Details
Discovering files (3912 of 3912) Complete 00:00:09
Scan new files (2756 of 2756) Complete 00:02:47
Discovering playlists (1 of 1) Complete 00:00:01
iTunes Import (2947 of 2947) Complete 00:00:38
iTunes Playlist Import (217 of 217) Complete 00:00:54

iTunes Downloaded Artwork Import (2955 of 2955) Complete 00:01:51
iTunes User-Added Artwork Import (2674 of 2674) Complete 00:18:38
Pre-caching Artwork (306 of 306) Complete 00:00:34

The server has finished scanning your music collection.
Total Time: 00:25:32 (Saturday, 7 August 2010 / 15.46)


3. Artwork scan using test.mysb.com / last.fm:

Music Scan Details
Discovering files (3912 of 3912) Complete 00:00:07
Scan new files (2756 of 2756) Complete 00:01:52
Discovering playlists (1 of 1) Complete 00:00:00
iTunes Import (2947 of 2947) Complete 00:00:46
iTunes Playlist Import (217 of 217) Complete 00:00:42

Downloading Artwork (2674 of 2674) Complete 00:14:40
Pre-caching Artwork (119 of 119) Complete 00:00:06

The server has finished scanning your music collection.
Total Time: 00:18:13 (Sunday, 8 August 2010 / 13.45)

I mostly listen to classical music so that might an explanation why iTunes seems to find more covers than last.fm.

[verypsb]

It's working good enough to give it some broader testing. Scanning my test library of about 670 albums it was able to find roughly 90% of all artwork automatically.

If I could get 7.6 running on my system, I would gladly test it. However, I haven't been able to run 7.6 (or 7.5e) for a very long time. (I never had a working 7.6 (or 7.5e) squeezeboxserver) All builds that I've tried are unusable (scanner crashes, scanner stalls, squeezeboxserver unresponsive, squeezeboxserver crashes, etc)

All I have to do is installing a squeezeboxserver version that doesn't use sqlite, and all is up & running again.

OS: Windows 7 x64

[copperstate]

If I could get 7.6 running on my system, I would gladly test it. However, I haven't been able to run 7.6 (or 7.5e) for a very long time. (I never had a working 7.6 (or 7.5e) squeezeboxserver) All builds that I've tried are unusable (scanner crashes, scanner stalls, squeezeboxserver unresponsive, squeezeboxserver crashes, etc)

All I have to do is installing a squeezeboxserver version that doesn't use sqlite, and all is up & running again.

OS: Windows 7 x64

If sqlite seems to be the problem, that may be easily remedied by selecting mysql in the 7.6 web ui settings (advanced/performance). Never tried it with Windows though.

[verypsb]

If sqlite seems to be the problem, that may be easily remedied by selecting mysql in the 7.6 web ui settings (advanced/performance). Never tried it with Windows though.

My guess is that SQLite isn't the problem, but the new scanner engine. Squeezeboxserver <7.6 uses MySQL and the old scanner engine, Squeezeboxserver >7.5e/7.6 uses SQLite and the new scanner engine.

[erland]

I've added some (experimental) code to 7.6 which allows you to have the music scanner automatically download missing album artwork for your music. It's working good enough to give it some broader testing. Scanning my test library of about 670 albums it was able to find roughly 90% of all artwork automatically.

Scanning the music files took 90 seconds, downloading artwork took about 10 minutes. This is my rather small 262 album 3437 track library. Very acceptable times considering how much it seems to have got the right album art for.

Some problems:
- Most compilation albums failed, I'm not sure if this is because most of my compilation albums are only released in Sweden or if it's some generic problem with compilation albums.
- There are a few albums that got an alternative album cover, probably because the international version of the album might have a different cover than the version released in Sweden.
- There are a few albums without a cover that seems to have special image instead, you should probably not use these as album covers if possible, for example:
http://www.last.fm/music/Tommy+K%C3%B6rberg/Stilla+Natt
- I had one album where it got the wrong cover, it got this one:
http://www.last.fm/music/Mainstreet+.../Love+And+Play
But should have got this one:
http://www.last.fm/music/Mainstreet+...den/Mainstreet

Total statistics for my library:
- Albums with correct cover: 228
- Albums with the special image instead of a real album cover: 3
- Albums where it failed to retrieve cover: 20 (mostly compilations)
- Albums with cover from wrong album: 1
- Albums with alternative cover that doesn't match Swedish release: 10

Overall I have to say that it worked pretty good. Assuming your retrieval code works correctly I don't think there is any point of going to a higher resolution than 500x500. Most of my albums besides big releases from famous artists were retrived in 300x300 or 160x160 resolution, presumably because there isn't any higher resolution image on LastFM. So even though I'd love to get 768x768 images for the iPad I have a feeling there would be very few images with that resolution available from LastFM.