PDA

View Full Version : Last.fm & Trackstat



lrossouw
2009-08-23, 21:43
Hi

I'm writing a perl script which would import the last.fm counts to Erland's Trackstats plugin's tables. It would compare weekly playcounts and increase the Trackstats counts based on this. This should allow for a way to get plays form any Last.fm compatible player into the trackstats. I.e. iTunes, WMP, ipod etc.

I'm a novice (perl) writer and decided it would be too difficult (for me) to integrate as a plugin. It's freestanding script which accesses the mysql database directly, adds some tables and update them.

It also fetches global / country wide playcounts for the tracks in your library. This allows you to create dynamic playlists like "play me music from my library that is popular on last.fm in South Africa". Or "play me music from my library that was popular in the last week on last.fm" etc.

I'm already listening to dynamic playlists like the above using the sqlplaylist/dynamicplaylist plugin. However I have some utf8 issues (which I have no idea how to resolve). And I still need to write the bit that compares what is in trackstat versus what is in last.fm and updates.

This script could be useful to someone who wants to
a) run it once off and get all their last.fm playcounts into their library when they are starting out using trackstat (sort of backfill their data)
b) run it regularly to get updated last.fm playcounts into their library
c) run it regularly to get total playcount stats from last.fm (weekly and overall stats).
d) run it regularly to get country wise playcount stats from last.fm (weekly stats).
e) run it regularly to get group wise playcount stats from last.fm (weekly stats).

(b)-(d) is sort of working except that tracks\artists with special characters in their names don't get the counts imported.
I'm doing (a) next.
(e) should be relatively easy to add once a and d are working although I won't use it.

I need some pointers:
1) The tables I create seem to survive rescans. Do they survive squeezecentre upgrades? Only needed to keep the weekly history of (c)-(e).
2) I get the artist/track name (blob value) from the contributors/tracks table then query last.fm using their api using this to get the data. How do I ensure that I don't get into utf8 problems? At the moment I pass it unadjusted to the the "get" that fetches the last.fm data via xml (other than url escaping it).
3) This would seem to be a useful add-on to custom scan, but want to get it working standalone first.

Regards
Louis

erland
2009-08-24, 12:00
1) The tables I create seem to survive rescans. Do they survive squeezecentre upgrades? Only needed to keep the weekly history of (c)-(e).

There are no guarantees.

Mine has survived all upgrades beside the upgrades where the database engine changed (6.3(SQLite) -> 6.5(MySQL) and 7.3(MySQL) -> early beta of 7.4(SQLite) and early beta of 7.4(SQLite) -> late beta of 7.4(MySQL)).

You can probably expect a database change between 7.4 and 8.0 as 7.4 will be based on MySQL while 8.0 is planned to be based on SQLite.

If it's important that the data survives a rescan, remember the situation where the url changes during a rescan due to a moved or renamed file. In TrackStat I solve this by also storing the musicbrainz_id and after the rescan I synchronize my tables with the tracks table based on url and musicbrainz_id values.

However, I suppose you could just retrieve new data from LastFM if a counter is lost, so lost data might not be a big problem in your case.



2) I get the artist/track name (blob value) from the contributors/tracks table then query last.fm using their api using this to get the data. How do I ensure that I don't get into utf8 problems? At the moment I pass it unadjusted to the the "get" that fetches the last.fm data via xml (other than url escaping it).

From the LastFM API it sounds like they want utf8 encoding.

In the LastFM scanning module bundled with Custom Scan I use escape($artist->name) where $artist is a Slim::Schema::Contributor object retrieved from the database. Although, I'm not completely sure if this works correctly or not.

Character encoding could be a mess, there is a Slim::Utils::Unicode module which contains a number of conversion methods but I suppose you've already seen that ?



3) This would seem to be a useful add-on to custom scan, but want to get it working standalone first.

Get back to me when you want to integrate it.

I suppose you are aware of the license limitation that you can only send 5 requests per second to LastFM ?

Earlier it was 1 request per second but it seems like they've raised the limit a bit since I did the LastFM module bundled with Custom Scan.

lrossouw
2009-08-24, 19:52
Thanks for your comprehensive reply.


You can probably expect a database change between 7.4 and 8.0 as 7.4 will be based on MySQL while 8.0 is planned to be based on SQLite.

I read about this and not sure how I will handle this as yet.


If it's important that the data survives a rescan, remember the situation where the url changes during a rescan due to a moved or renamed file. In TrackStat I solve this by also storing the musicbrainz_id and after the rescan I synchronize my tables with the tracks table based on url and musicbrainz_id values.

I get your point. It is not that important. To keep some last.fm history it would be nice to keep the tables but not important. As effectively Last.fm operates on an artist\track title "key" I keep things at this level. So only if you change the artist or track title would you loose some last.fm data.


In the LastFM scanning module bundled with Custom Scan I use escape($artist->name) where $artist is a Slim::Schema::Contributor object retrieved from the database. Although, I'm not completely sure if this works correctly or not.

Character encoding could be a mess, there is a Slim::Utils::Unicode module which contains a number of conversion methods but I suppose you've already seen that ?


I am already doing what you suggest above. And I agree it is a mess. Will have a look to see what the scrobbler does when it submits tracks. If I can find it...



Get back to me when you want to integrate it.


Will do, if there is interest. No point otherwise.


I suppose you are aware of the license limitation that you can only send 5 requests per second to LastFM ?

I have limited this (in most places) to 1 sec. I looked for the actual limit in the documentation but could not find it.

Cheers
Louis

erland
2009-08-24, 20:45
Will have a look to see what the scrobbler does when it submits tracks. If I can find it...

It's in Slim/Plugin/AudioScrobbler/Plugin.pm and it seems to do the same thing as far as I can see.

I'm guessing you read the artist name from a Slim::Schema::* object and not directly using SQL ?

I think it can get more complicated if you use SQL directly to retrieve things from the standard SqueezeCenter tables. I think I had to do some extra manipulation in Custom Scan, Custom Browse and SQL Playlist regarding this. I use SQL directly in some places in these plugins also when accessing the standard SqueezeCenter tables.



I have limited this (in most places) to 1 sec. I looked for the actual limit in the documentation but could not find it.

It's in the license agreement you approve to when getting the API key.

lrossouw
2009-08-25, 00:07
I'm guessing you read the artist name from a Slim::Schema::* object and not directly using SQL ?
Oops no :) I'm using SQL as my script is not written as a plugin. Everything is done with SQL statements and get statements to fetch xml from the Last.fm website.

I think it can get more complicated if you use SQL directly to retrieve things from the standard SqueezeCenter tables. I think I had to do some extra manipulation in Custom Scan, Custom Browse and SQL Playlist regarding this. I use SQL directly in some places in these plugins also when accessing the standard SqueezeCenter tables.
Is there a place I can look at the source for your plugins to learn a few lessons on this? I suppose I should be switching to the "Schema" but maybe I'd just fix what I have now first.

It's in the license agreement you approve to when getting the API key.
Must go an read it again...

erland
2009-08-25, 09:12
Oops no :) I'm using SQL as my script is not written as a plugin. Everything is done with SQL statements and get statements to fetch xml from the Last.fm website.

Is there a place I can look at the source for your plugins to learn a few lessons on this? I suppose I should be switching to the "Schema" but maybe I'd just fix what I have now first.

I'm not sure if Slim::Schema::* will work for a standalone application.

Please, note that I'm not sure I've done this the correct way. I think I asked in the Developers section but I didn't get an answer so I just tried different variants until it worked.

Have a look at the _exectue method in this file:
http://erlandplugins.googlecode.com/svn/CustomBrowse/trunk/src/MenuHandler/SQLHandler.pm

As you can see I'm in SqueezeCenter 7.3 and earlier I'm using:


Slim::Utils::Unicode::utf8on(Slim::Utils::Unicode: :utf8decode($name,'utf8'));


Or in Squeezebox Server 7.4 and later:


Slim::Utils::Unicode::utf8decode($name,'utf8');


I think the different behaviors is because some change in the Slim::Utils::Unicode module between 7.3 and 7.4.

Also please note that you can get really strange behavior if you use the strings in log messages. The reason is that perl sometime converts strings when you concatenate two strings with different encoding, the result is that the code could suddenly start to work when you add a log message.

craigim
2009-08-25, 14:15
Will do, if there is interest. No point otherwise.

I actually came to the forum today to see if there was a plugin to do this very thing. I am not a programmer, so I can offer no advice whatsoever, but if it ever becomes a working plugin, I would use it every day and would be happy to test out any beta versions you might have. Right now, I kind of do this manually by going to last.fm, expanding my list of artists, scrolling down to the bottom, and seeing if there is anything that grabs me, and then putting it into my playlist. Since I listen to Pandora a lot at work (which I scrobble using pandorafm.real-ity.com), my TrackStat does not match my last.fm profile.

I would use a-d, especially if you could break it down by timescale as well (for instance, 'view the artists least played in the last 6 months')

lrossouw
2009-08-25, 19:10
I am not a programmer

Neither am I :D

Will get something done over the weekend. But it won't be a plugin. It will be a separate script that you need to run outside of SqueezeCentre. Something like the flac2mp3 script if you've used that.

lrossouw
2009-08-27, 07:14
Managed the utf8. I hadn't set the connection with MySQL to utf8 mode. This simplifies things a lot. Only needed decode statements in one or two places. And seems to be working.

So now did a full scan of last.fm global play count data for my library. Also imported my last.fm history. Now just need to ad a bit of code to merge my last.fm playcount data into the trackstat data then I'll have something workable.

hajnes
2010-10-08, 03:55
Well, quite a time since the last post in this thread...

Have you made any progress with your script??

Would realy be nice to update Trackstat with the information from Last.fm

lrossouw
2010-10-21, 05:14
Well, quite a time since the last post in this thread...

Have you made any progress with your script??

Would realy be nice to update Trackstat with the information from Last.fm

My script is running more or less but something has recently changed resulting in problems in getting data. It fetches the data, but I haven't gotten it to update the trackstat tables yet, though that should be simple. At the moment it is stored separately.

I've also played arround with measuring the similarity of two tracks using tag data, though not part of the script this could be very useful for a musicip replacement.

Also I have actually been quite busy and have just moved back to South Africa from Singapore. In fact my squeezies and the server are somewhere on ship off the coast of Africa right now, though they made it past the pirates.

I also need to find a job :P

Once that is all done hopefully I'll only start in the new year, which would give me some time to work on things like this :)

erland
2010-10-21, 11:04
I've also played arround with measuring the similarity of two tracks using tag data, though not part of the script this could be very useful for a musicip replacement.

Can you give us some more details about this, sounds interesting.