PDA

View Full Version : TrackStat and iTunes Statistics Import



ElMuto
2009-08-08, 07:49
I am wondering, why statistics of some songs are imported and some are not. I havent' found a pattern yet, to which kind of tracks this problem applies. About 70% of my iTunes-rated tracks don't have a TrackStat in Squeezecenter rating after running an import.

I haven't found any suspicios messages in the server.log, even when I turned debugging mode on for CustomScann and TrackStat logging messages.

Any ideas?
Any help is appreciated!

btw: The problem I have, ist not related to special characters in the file path (like '[', '&', etc.). I fixed it by applying a sed-script to the library-xml.

erland
2009-08-08, 08:17
Could you find a specific song that doesn't work ?

In that case, I would like you to enable debug level logging on plugin.trackstat and then post or mail me the section of the log that contains rows related to the server.log file. You can post/send the whole log file if it isn't too large or you can cut out the section related to the file plus 20 extra rows before and after in a text editor.

If you run TrackStat with debug logging enabled, you should see rows that indicates that ratings or other statistics is imported for each specific track.

ElMuto
2009-08-09, 23:28
Hi Erland, let me mention, how much I like your plugin. You are doing a great job!

Thanks for the fast reply. It lead me to where really the problem was and I have to correct myself: the problem I described *is* related to special characters in the file path:

My own sed-based workaround for the special-char-problem generated a non-well-formed xml file:

transformation.sed:
s/&/&/g
s/%5B/\[/g
s/%5D/\]/g

Log file entry:
[09-08-08 19:52:45.7814] Plugins::CustomScan::Scanner::exitScanTrack (1707) CustomScan: Failed to call exitScanTrack on module Plugins::TrackStat::Plugin->itunesimport:
not well-formed (invalid token) at line 18, column 41, byte 817:
<key>Track ID</key><integer>3040</integer>
<key>Name</key><string>Dream Machine (Down Tempo Mix)</string>
<key>Artist</key><string>Mark Farina & Sean Hayes</string>
========================================^

I got rid of all the 'No matching alternative path found' messages, but ignored the error message above. Bugger! The songs with imported rating have been imported *before* I implemented this workaround.

Deactivating my sed script leads me back to the problem described in Thread "CustomScan - iTunes statistics import notworking". In this thread I haven't found a hint how to excactly deal with those characters like '[]&' in my filenames.
Nevertheless, focussing (as advised) on the plugin.trackstat-messages, I found out, that for some reason SC does NOT url-Encode the above Characters. Eg. '&' in 'Girls & Boys':

mysql> select url, rating from tracks_persistent where url like '%Blur%Parklife%Girls%Boys%';
+------------------------------------------------------------------------+--------+
| url | rating |
+------------------------------------------------------------------------+--------+
| file:///mnt/sda1/mp3/A-Z/B/Blur%20-%20Parklife/01_Girls%20&%20Boys.mp3 | NULL |
+------------------------------------------------------------------------+--------+
1 row in set (2.89 sec)

I would expect this url to be stored as TrackStat does: 'Girls%20%26%20Boys' (thanks again for the hint to the logfile), but SC stores it as 'Girls%20&%20Boys'

Log Messages:
[09-08-09 22:12:53.6025] Plugins::TrackStat::Storage::findTrack (495) findTrack(): URL: file:///mnt/sda1/mp3/A-Z/B/Blur%20-%20Parklife/10_London%20Loves.mp3

[09-08-09 22:12:53.6195] Plugins::TrackStat::iTunes::Import::normalize_loca tion (689) normalized file:///mnt/sda1/mp3/A-Z/B/Blur%20-%20Parklife/01_Girls%20%26%20Boys.mp3 to file:///mnt/sda1/mp3/A-Z/B/Blur%20-%20Parklife/01_Girls%20%26%20Boys.mp3

[09-08-09 22:12:53.6238] Plugins::TrackStat::iTunes::Import::handleTrack (448) got a track named Girls %26 Boys location: file:///mnt/sda1/mp3/A-Z/B/Blur%20-%20Parklife/01_Girls%20%26%20Boys.mp3

[09-08-09 22:12:53.6557] Plugins::TrackStat::Storage::mergeTrack (1065) File file:///mnt/sda1/mp3/A-Z/B/Blur%20-%20Parklife/01_Girls%20%26%20Boys.mp3 doesn't exist - will try to find on alternative path.
[09-08-09 22:12:53.6584] Plugins::TrackStat::Storage::findObjectForMovedUrl (899) Find track urls ending with /B/Blur%20-%20Parklife/01_Girls%20%26%20Boys.
[09-08-09 22:12:59.9177] Plugins::TrackStat::Storage::findObjectForMovedUrl (904) Found 0 matching tracks
[09-08-09 22:12:59.9196] Plugins::TrackStat::Storage::findObjectForMovedUrl (924) Find track urls ending with /Blur%20-%20Parklife/01_Girls%20%26%20Boys.
[09-08-09 22:13:06.2219] Plugins::TrackStat::Storage::findObjectForMovedUrl (929) Found 0 matching tracks
[09-08-09 22:13:06.2237] Plugins::TrackStat::Storage::mergeTrack (1072) No matching alternative path found for file:///mnt/sda1/mp3/A-Z/B/Blur%20-%20Parklife/01_Girls%20%26%20Boys.mp3.
[09-08-09 22:13:06.2315] Plugins::TrackStat::iTunes::Import::normalize_loca tion (689) normalized file:///mnt/sda1/mp3/A-Z/B/Blur%20-%20Parklife/12_Clover%20Over%20Dover.mp3 to file:///mnt/sda1/mp3/A-Z/B/Blur%20-%20Parklife/12_Clover%20Over%20Dover.mp3

Maybe a bug in SC?

Cheers
ElMuto

Chunkywizard
2009-08-10, 01:54
Hi ElMuto,

I am the other guy seeing this (I started the other thread). I remember erland saying that this bug isn't present in the Linux build, it only seems to affect Windows builds. I guess that wouldn't point to a bug in SC...

CW

ElMuto
2009-08-10, 02:28
Hi ChunkyWizard,

I was hoping, we could find some synergies :-)

Think, we're getting closer to the problem: I experience those symptoms on a

- Linux-Installation: Ubuntu Server 8.04.1 LTS (Hardy)
- SC 7.3.3 - 27044

System. It would be interesting to know if on other (than our two) systems, characters like '&' in filepaths are *not* URL-encoded, when stored in SC's tracks.url and tracks_persistent.url columns. If so, I would really like to know the reason for it. Will search other threads about URL encoding in SC tonight.

ElMuto

Chunkywizard
2009-08-10, 02:31
Aha, you are not who I thought you were! We are three, see: http://code.google.com/p/erlandplugins/issues/detail?id=72&sort=reportedby&colspec=ID%20Plugin%20Type%20Status%20Priority%20M ilestone%20ReportedBy%20Summary

They are running a WHS system as well. Can you link your findings to the bug report?

CW

ElMuto
2009-08-10, 03:35
good idea! done.

Seems as if I am the missing SC-Linux-User, doesn't it?

erland
2009-08-10, 10:00
It would be interesting to know if on other (than our two) systems, characters like '&' in filepaths are *not* URL-encoded, when stored in SC's tracks.url and tracks_persistent.url columns. If so, I would really like to know the reason for it. Will search other threads about URL encoding in SC tonight.

Could you check which locale setting SqueezeCenter uses ?
It's shown in the "Information" tab in the Server Settings as something like:
"Operating system: Debian - EN - utf8"

How does the paths look like in the iTunes xml file ? Do they contain & or is it encoded with %26 ?

With Chunkywizard's test files my development machine seems to only encode # and ' characters, I'm not sure if this was the case earlier or if it behaves differently now.

ElMuto
2009-08-10, 11:35
Could you check which locale setting SqueezeCenter uses ?
It's shown in the "Information" tab in the Server Settings as something like:
"Operating system: Debian - EN - utf8"
Debian - DE - utf8


How does the paths look like in the iTunes xml file ? Do they contain & or is it encoded with %26 ?
Interesting point:
- in the iTunes (WinXP) xml file it's encoded with: & #38; (please ignore the space between the ampersand and the pound symbol)
- TrackStat (log) reads it as: %26
- SC stores it as: &


With Chunkywizard's test files my development machine seems to only encode # and ' characters, I'm not sure if this was the case earlier or if it behaves differently now.
Another interesting observation on my system:
- iTunes xml: '
- Trackstat log: '
- SC: %27

Let me know, if you need any more infos/things I can do

ElMuto
2009-08-12, 13:01
In order deepen my findings, I modified the method "normalize_location" in /usr/share/perl5/Slim/Plugin/iTunes/Common.pm and wanted to see, what happens. So I inserted following code-snippet at Line 358 (just before returning $url):

# ElMuto's extension
# replace %26 by ampersand to get
# matching file urls in SCs DB
$log->warn("original url was $url");
$url =~ s/%26/\&/;
$log->warn("..changed url is $url");
# end of ElMuto's extension

Interestingly enough, even after restart of the squeezecenter-service, this code-modification seems to have no effect in the execution. Not even the messages are written to the log.

Does anybody know, what I can do to test my code modifications on a standard Ubuntu-SC-installation? I would really like to avoid setting up an SC-dev-environment for testing one line of effective code.

erland
2009-08-12, 22:00
In order deepen my findings, I modified the method "normalize_location" in /usr/share/perl5/Slim/Plugin/iTunes/Common.pm and wanted to see, what happens. So I inserted following code-snippet at Line 358 (just before returning $url):

# ElMuto's extension
# replace %26 by ampersand to get
# matching file urls in SCs DB
$log->warn("original url was $url");
$url =~ s/%26/\&/;
$log->warn("..changed url is $url");
# end of ElMuto's extension

Interestingly enough, even after restart of the squeezecenter-service, this code-modification seems to have no effect in the execution. Not even the messages are written to the log.

Does anybody know, what I can do to test my code modifications on a standard Ubuntu-SC-installation? I would really like to avoid setting up an SC-dev-environment for testing one line of effective code.

If you like to change the behavior in the iTunes Statistics Import scanning module, you will need to change the files within the TrackStat plugin. You probably use the latest official version of TrackStat where the code is packaged in a *.par file which makes it a bit hard to modify. Anyway, to make it a bit easier, a source code version of TrackStat 2.8 is attached to this post.

The correct way to install it is:
1. Uninstall TrackStat 2.8 through Extension Downloader (if you have installed it that way) or alternatively remove it manually if you have downloaded the zip and unzipped it yourself.
2. Shutdown SqueezeCenter
3. Unzip the attached TrackStat-2.8 zip in the /var/lib/squeezecenter/Plugins directory
4. Start SqueezeCenter and make sure TrackStat still works

After this you are ready to do any modifications to the files, after you have modified something you will need to restart SqueezeCenter for the changes to take effect.

The file you should make modifications in are:
/var/lib/squeezecenter/Plugins/TrackStat/lib/Plugins/TrackStat/iTunes/Import.pm

erland
2009-08-12, 22:05
And the source version which I of course forgot to attach comes here.

It was a bit large to attach, so you get a link to my home page instead:
http://erland.isaksson.info/squeezebox/TrackStat-2.8.zip

Chunkywizard
2009-08-13, 09:17
Hi ElMuto,

if you get anywhere with this and want me to test out a possible fix just shout!

CW

ElMuto
2009-08-14, 00:42
@erland: thanks a lot for your support! Following your instructions and thanks to the great documentation / clear code of TrackStat, it was really easy to get some first results. And they look promising: most of the files with filenames containing []&' are properly imported. I will do some additional checks over the weekend and post the result.

@cw: thanks for your offer. I will definitely come back to it, as soon as I reached a first 'milestone'.

ElMuto
2009-08-15, 07:59
Got it working on my system now. All files are now imported into TrackStat

The modified Import.pm is attached to this post.

@cw: let me know, if it works on your system as well. I'm really curious ...

Chunkywizard
2009-08-15, 08:16
Got it working on my system now. All files are now imported into TrackStat

The modified Import.pm is attached to this post.

@cw: let me know, if it works on your system as well. I'm really curious ...

Hi ElMuto,

so let me just check I have this right. I need to:
1. Uninstall TrackStat 2.8 through Extension Downloader
2. Shutdown SqueezeCenter
3. Unzip the attached TrackStat-2.8 zip in the C:/Program Files/squeezebox/server/Plugins directory
4. Replace C:\Program Files\Squeezebox\server\Plugins\TrackStat\lib\Plug ins\TrackStat\iTunes\Import.pm with your copy.
5. Start Squeezecenter
6. Check it works.

I'll try now. Let me know if I have got anything wrong!

CW

Chunkywizard
2009-08-15, 08:33
Yee-ha! At first glance, that's fixed it good and proper. Many thanks for your help. I'll give my library a complete look over when I have the chance but I think it's worked.

erland, can you incorporate the fix back into the main code?

CW

ElMuto
2009-08-15, 09:39
Man, you're really fast!

Great to hear, that it worked for you as well on a first test.

I cross my fingers for your complete library scan ;-)

Chunkywizard
2009-08-16, 01:28
Hi ElMuto,

my library was completely rescanned last night and everything still looks AOK, job done! Thanks for fixing this, it was my number 1 annoyance with this fantastic plugin. Here's hoping this fix gets incorporated into the main code stream.

Cheers

CW

erland
2009-08-16, 21:32
Got it working on my system now. All files are now imported into TrackStat

The modified Import.pm is attached to this post.

Thanks, I'll include it in next beta version of TrackStat for Squeezebox Server 7.4.

Dondo
2009-08-29, 07:22
I am having a problem importing stats from itunes to trackstat. Everything in my system is working as usual, except the CustomScan itunes import is not working. (export is working perfectly)
I have chosen a sample from my logs to show what is happening in the scan:




[09-08-23 13:35:47.4016] Plugins::TrackStat::iTunes::Import::normalize_loca tion (689) normalized file://localhost/X:/zz-music/Music1/A/Dan%20Auerbach/Keep%20It%20Hid/04-Because%20I%20Should.mp4 to file:///X:/zz-music/Music1/A/Dan%20Auerbach/Keep%20It%20Hid/04-Because%20I%20Should.flac

[09-08-23 13:35:47.4023] Plugins::TrackStat::iTunes::Import::handleTrack (424) file not found: /X:/zz-music/Music1/A/Dan Auerbach/Keep It Hid/04-Because I Should.flac

[09-08-23 13:35:47.4038] Plugins::TrackStat::iTunes::Import::normalize_loca tion (689) normalized file://localhost/X:/zz-music/Music1/A/Dan%20Auerbach/Keep%20It%20Hid/06-Real%20Desire.mp4 to file:///X:/zz-music/Music1/A/Dan%20Auerbach/Keep%20It%20Hid/06-Real%20Desire.flac

[09-08-23 13:35:47.4044] Plugins::TrackStat::iTunes::Import::handleTrack (424) file not found: /X:/zz-music/Music1/A/Dan Auerbach/Keep It Hid/06-Real Desire.flac


my configuration:

Computer 1 (virtual machine)
Windows XP as operating system (on Sun VirtualBox)
Running iTunes
Music files available as aac format with paths:
X:\zz-music\Music**\Artist1\Album1\Track1.mp4 (X:\zz-music is a shortcut folder located on the shared drives)
X:\zz-music\Music**\Artist2\Album1\Track1.mp4

Computer 2
Ubuntu 9.04 as operating system
Running SqueezeCenter 7.3.2
SqueezeCenter Music Folder points to: /media/Music5/zz-music (this is the same shortcut folder that itunes points at)
Music files available in FLAC format with paths:
/media/Music5/zz-music/Music**/Artist1/Album1/Track1.flac
/media/Music5/zz-music/Music**/Artist2/Album1/Track1.flac
since the drives are the same, the paths to the files Computer 1 sees in MP4 is available as:
/media/Music5/zz-music/Music**/Artist1/Album1/Track1.mp4


settings in "Module Settings iTunes Statistics Export"
Output directory - /media/Music5/zz-itunes/Squeezecenter
Continously write history file - enabled
Music path in iTunes setting - X:\zz-music
Music path in SqueezeCenter/SlimServer setting /media/Music5/zz-music
File extension in iTunes - .mp4

settings in "Module Settings iTunes Statistics Import"
iTunes Library XML file - /media/Music5/zz-itunes/iTunes Library.xml
File extension in SqueezeCenter/SlimServer - .flac
Music path in SqueezeCenter/SlimServer - /media/Music5/zz-music (this is the same path listed in the Squeezecenter Basic Settings as my Music folder)

Is there any obvious problems that i am overlooking?
Is it possible that the updated plugin and import.pm mentioned above will fix this, or is it a different situation?

erland
2009-08-30, 02:25
Is there any obvious problems that i am overlooking?
Is it possible that the updated plugin and import.pm mentioned above will fix this, or is it a different situation?

I had a quick look at the code and I suspect it's the shortcut folder that causes the problem.

Just to make sure, could you open the iTunes Library.xml file in a text editor and look for a Music Folder key in the first 10 rows. What does it say ?

Dondo
2009-08-30, 07:26
Here is the first bit of my library xml
Thanks Erland


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Major Version</key><integer>1</integer>
<key>Minor Version</key><integer>1</integer>
<key>Application Version</key><string>8.2.1</string>
<key>Features</key><integer>5</integer>
<key>Show Content Ratings</key><true/>
<key>Music Folder</key><string>file://localhost/X:/zz-itunes/iTunes%20Music/</string>
<key>Library Persistent ID</key><string>E6073E2DF83FA4E6</string>
<key>Tracks</key>
<dict>

erland
2009-08-30, 10:07
Are the music files stored in:
X:/zz-itunes/iTunes%20Music/Music*

Or in:
X:/zz-itunes/Music*

I think the current iTunes Import logic will try to replace "X:/zz-itunes/iTunes%20Music/" with the "Music path in SqueezeCenter" setting you have entered. So if the music is stored in "X:/zz-itunes/Music*" this explains why the replace doesn't happen.

Dondo
2009-08-30, 10:50
the shortcut folder is
X:/zz-music/
with shortcuts pointing to the drives Music1, Music2, Music3, Music4.
So the "Itunes Music" is very wrong!
I have no idea how itunes assumed that, but I will try to fix it.

Thanks for the help Erland..I had a feeling that I would be missing something obvious!



It was the iTunes music folder that threw me off. I've never used that folder to "organize" my music, so it was turned off soon after I installed it. The only thing that goes in that folder is the singles I buy from the Itunes store. So that never even entered my mind as being a problem. All of the scanning logs I looked at didn't even seem to have any errors.
The scanning is progressing now, and appears to be successful!

Christophenonstop
2010-01-01, 09:44
Hi- I run a similar install to that of Dondo, with iTunes setup on a virtual WinXP with VirtualBox.

Both iTunes and Squeezebox Server point to the same directory which is /media/Media/MUSIC-AUDIO in Ubuntu and M:\MUSIC-AUDIO in virtual XP

Now when I scan the iTunes xml file (which I first moved and renamed), the log returns 424 errors for every track :


[10-01-01 16:37:13.5855] Plugins::TrackStat::iTunes::Import::handleTrack (424) file not found: /M:/MUSIC-AUDIO/Diplo - Favela Strikes Back/01 - Diplo - FSB - Track 17.mp3


Do you have a clue of what I am missing ?


My iTunes Statistics Import Settings are as follow :
iTunes Library XML file = /media/Media/DATABASE/winampdb.xml
File extension in Squeezebox Server = empty
Music path in Squeezebox Server = empty
Ignore disabled songs = No
Use album rating = No


Also, below is a snippet of the xml file :

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Major Version</key><integer>1</integer>
<key>Minor Version</key><integer>1</integer>
<key>Application Version</key><string>9.0.2</string>
<key>Features</key><integer>5</integer>
<key>Music Folder</key><string>file://localhost/C:/Documents%20and%20Settings/Christophenonstop/My%20Documents/My%20Music/iTunes/iTunes%20Media/</string>
<key>Library Persistent ID</key><string>2F1BCA4CAF9C1016</string>
<key>Tracks</key>
<dict>
<key>119</key>
<dict>
<key>Track ID</key><integer>119</integer>
<key>Name</key><string>FSB - Track 17</string>
<key>Artist</key><string>Diplo</string>
<key>Album Artist</key><string>Diplo</string>
<key>Album</key><string>Favela Strikes Back</string>
<key>Genre</key><string>Hip-Hop Latin DJset</string>
<key>Kind</key><string>MPEG audio file</string>
<key>Size</key><integer>5796793</integer>
<key>Total Time</key><integer>241475</integer>
<key>Track Number</key><integer>17</integer>
<key>Track Count</key><integer>20</integer>
<key>Year</key><integer>2005</integer>
<key>Date Modified</key><date>2009-04-14T18:39:42Z</date>
<key>Date Added</key><date>2010-01-01T15:33:34Z</date>
<key>Bit Rate</key><integer>192</integer>
<key>Sample Rate</key><integer>44100</integer>
<key>Rating</key><integer>80</integer>
<key>Album Rating</key><integer>80</integer>
<key>Album Rating Computed</key><true/>
<key>Persistent ID</key><string>975050377F9130F4</string>
<key>Track Type</key><string>File</string>
<key>Location</key><string>file://localhost/M:/MUSIC-AUDIO/Diplo%20-%20Favela%20Strikes%20Back/17%20-%20Diplo%20-%20FSB%20-%20Track%2017.mp3</string>
<key>File Folder Count</key><integer>-1</integer>
<key>Library Folder Count</key><integer>-1</integer>
</dict>

erland
2010-01-01, 11:13
You definitely need to set the "Music path in Squeezebox Server" parameter in the Custom Scan iTunes Statistics Import settings, should be set to "/media/Media/MUSIC-AUDIO" or whatever you have set the Squeezebox Server Music Folder setting to.

However, I think this line in your iTunes XML file might cause problems


<key>Music Folder</key><string>file://localhost/C:/Documents%20and%20Settings/Christophenonstop/My%20Documents/My%20Music/iTunes/iTunes%20Media/</string>


I would have expected this to point to "file://localhost/M:/MUSIC-AUDIO" instead. I think the import replaces this path with the "Music path in Squeezebox Server", so at the moment it will probably not replace anything since all the music files are in "M:/MUSIC-AUDIO" and not in "C:/Documents and Settings/Christophenonstop/My Documents/My Music/iTunes/iTUnes Media"

I think you probably need to reconfigure the path setting in iTunes to change this.

Christophenonstop
2010-01-03, 04:22
Thanks Erland, that was it!

I edited my xml file as a text file in gedit to turn the "Music Folder" path into the new path "/media/Media/MUSIC-AUDIO" and it did the trick. Now I still get a bunch of 424 errors that I figure are due to special characters.

However, as you suggested I also tried the import after setting the "Music path in Squeezebox Server" parameter in the Custom Scan settings and without altering the original xml file. And no, the import did not replace the old path with the new one.

Chunkywizard
2010-01-03, 06:17
Hi Erland,

I have been using CustomScan import for a while now and have found one thing I would like to know if it's possible to change. I have found over time my listening habits have changed and I listen to more music on my Squeezeboxes with Dynamic Playlists than via iTunes and Smart Playlists. This means that rather than replace my play counts and last played data every night I would rather now run the Squeezeboxes independently of iTunes so that my Dynamic Playlists work better (I have found that my playlists play least played tunes first. This isn't an issue, but once they play on the Squeezeboxes and then the last played data is over written then next time the play list is invoked the same tunes will be played again). However I would still like rating syncing. So my question is this. Is there anyway to only sync ratings using CustomScan, and not last played data and playcounts?

Regards

CW

erland
2010-01-03, 06:21
However, as you suggested I also tried the import after setting the "Music path in Squeezebox Server" parameter in the Custom Scan settings and without altering the original xml file. And no, the import did not replace the old path with the new one.

You need to correct the XML file, with the XML you posted earlier it will never do any path conversion.

erland
2010-01-03, 06:30
Hi Erland,

I have been using CustomScan import for a while now and have found one thing I would like to know if it's possible to change. I have found over time my listening habits have changed and I listen to more music on my Squeezeboxes with Dynamic Playlists than via iTunes and Smart Playlists. This means that rather than replace my play counts and last played data every night I would rather now run the Squeezeboxes independently of iTunes so that my Dynamic Playlists work better (I have found that my playlists play least played tunes first. This isn't an issue, but once they play on the Squeezeboxes and then the last played data is over written then next time the play list is invoked the same tunes will be played again). However I would still like rating syncing. So my question is this. Is there anyway to only sync ratings using CustomScan, and not last played data and playcounts?

Regards

CW

Are you saying that an older last played time in iTunes overwrites a newer last played time in SBS during import ?
That's not supposed to happen, the import should ignore any play times that's older in iTunes than in SBS. It should also ignore any play counts that's lower in iTunes than in SBS.

Or are you talking about exporting statistics from SBS to iTunes ?

Besides the above logic that should work there isn't any other way to just synchronize ratings.

Chunkywizard
2010-01-03, 09:17
Are you saying that an older last played time in iTunes overwrites a newer last played time in SBS during import ?
That's not supposed to happen, the import should ignore any play times that's older in iTunes than in SBS. It should also ignore any play counts that's lower in iTunes than in SBS.

Or are you talking about exporting statistics from SBS to iTunes ?

Besides the above logic that should work there isn't any other way to just synchronize ratings.

Hi Erland

the way you have explaIned should be fine. I didn't understand there was no overwriting logic. I guess I wasn't seeing tracks replayed, it just felt like that! It might have been down to duplicates in my library. I'll keep an eye on it. I have now amended my dynamic playlist logic to cut out duplicates (I have a keyword in comments I now exclude against)

thanks for the help

CW

Christophenonstop
2010-01-03, 10:57
Hi ElMuto,
so let me just check I have this right. I need to:
1. Uninstall TrackStat 2.8 through Extension Downloader
2. Shutdown SqueezeCenter
3. Unzip the attached TrackStat-2.8 zip in the C:/Program Files/squeezebox/server/Plugins directory
4. Replace C:\Program Files\Squeezebox\server\Plugins\TrackStat\lib\Plug ins\TrackStat\iTunes\Import.pm with your copy.
5. Start Squeezecenter
6. Check it works.


Guys, I tried this but I still can't import my stats from files with special characters.

What I get instead is two types of errors, (424) and (1072):



[10-01-03 18:42:24.0699] Plugins::TrackStat::Storage::mergeTrack (1072) No matching alternative path found for file:///media/Media/MUSIC-AUDIO/Lee Perry/Arkology%2C Reel I/01 - Lee Perry & The Upsetters - Dub Revolution %28Part 1%29.mp3.
[10-01-03 18:42:48.0983] Plugins::TrackStat::iTunes::Import::handleTrack (424) file not found: /media/Media/MUSIC-AUDIO/Marousse/Skanka Fé/07 - Je m'éclate au Sénégal.mp3


Also after updating to TrackStat 2.8 and restarting the server I read multiple chunks like this :


[10-01-03 18:38:21.0891] Slim::Web::HTTP::addPageFunction (2373) Backtrace:

frame 0: Slim::Utils::Log::logBacktrace (/usr/share/perl5/Slim/Web/HTTP.pm line 2373)
frame 1: Slim::Web::HTTP::addPageFunction (/usr/share/squeezeboxserver/Plugins/TrackStat/lib/Plugins/TrackStat/Plugin.pm line 1144)
frame 2: Plugins::TrackStat::Plugin::webPages (/usr/share/perl5/Slim/Plugin/Base.pm line 68)
frame 3: Slim::Plugin::Base::initPlugin (/usr/share/squeezeboxserver/Plugins/TrackStat/lib/Plugins/TrackStat/Plugin.pm line 2255)
frame 4: Plugins::TrackStat::Plugin::initPlugin (/usr/share/perl5/Slim/Utils/PluginManager.pm line 323)
frame 5: (eval) (/usr/share/perl5/Slim/Utils/PluginManager.pm line 323)
frame 6: Slim::Utils::PluginManager::load (/usr/sbin/squeezeboxserver line 501)
frame 7: main::init (/usr/sbin/squeezeboxserver line 572)
frame 8: main::main (/usr/sbin/squeezeboxserver line 1065)

[10-01-03 18:38:21.0905] Slim::Web::HTTP::addPageFunction (2373) Error: Slim::Web::HTTP::addPageFunction() is deprecated - please use Slim::Web::Pages->addPageFunction() instead

erland
2010-01-03, 22:41
Guys, I tried this but I still can't import my stats from files with special characters.

What I get instead is two types of errors, (424) and (1072):



[10-01-03 18:42:24.0699] Plugins::TrackStat::Storage::mergeTrack (1072) No matching alternative path found for file:///media/Media/MUSIC-AUDIO/Lee Perry/Arkology%2C Reel I/01 - Lee Perry & The Upsetters - Dub Revolution %28Part 1%29.mp3.
[10-01-03 18:42:48.0983] Plugins::TrackStat::iTunes::Import::handleTrack (424) file not found: /media/Media/MUSIC-AUDIO/Marousse/Skanka Fé/07 - Je m'éclate au Sénégal.mp3


You don't have to use TrackStat 2.8, the changes mentioned earlier in the thread is already part of the latest release.

Try enable debug logging of plugin.trackstat in SBS Settings/Advanced/Logging, perform and import and post a section with 10-20 lines before and after a problematic track.

The if you are able to, it would be great if you could see how the same track looks in the database. If you don't have a gigantic database, you should be able to install the Database Query plugin and create a free form query with the statement "select url from tracks", if you have a large database you can instead use "select url from tracks where title like '%some part of title%'" where the "some part of title" is a part of the track title.

The database url and the url from the debug log needs to match exactly, if they doesn't that's the reason for your problem. There is replacement function in TrackStat/iTunes/Import.pm called "replace_problematic_url_chars" that replaces some characters, you might be able to add more characters that needs to be replaced there if you find out which ones that causes the problem.

I suspect your problem might be different file name encodings on Windows and Linux but I'm not completely sure. If this is the case, replacing single characters like above might not be the easiest solution because then it might be a lot of problematic characters.



Also after updating to TrackStat 2.8 and restarting the server I read multiple chunks like this :


[10-01-03 18:38:21.0891] Slim::Web::HTTP::addPageFunction (2373) Backtrace:

frame 0: Slim::Utils::Log::logBacktrace (/usr/share/perl5/Slim/Web/HTTP.pm line 2373)
frame 1: Slim::Web::HTTP::addPageFunction (/usr/share/squeezeboxserver/Plugins/TrackStat/lib/Plugins/TrackStat/Plugin.pm line 1144)
frame 2: Plugins::TrackStat::Plugin::webPages (/usr/share/perl5/Slim/Plugin/Base.pm line 68)
frame 3: Slim::Plugin::Base::initPlugin (/usr/share/squeezeboxserver/Plugins/TrackStat/lib/Plugins/TrackStat/Plugin.pm line 2255)
frame 4: Plugins::TrackStat::Plugin::initPlugin (/usr/share/perl5/Slim/Utils/PluginManager.pm line 323)
frame 5: (eval) (/usr/share/perl5/Slim/Utils/PluginManager.pm line 323)
frame 6: Slim::Utils::PluginManager::load (/usr/sbin/squeezeboxserver line 501)
frame 7: main::init (/usr/sbin/squeezeboxserver line 572)
frame 8: main::main (/usr/sbin/squeezeboxserver line 1065)

[10-01-03 18:38:21.0905] Slim::Web::HTTP::addPageFunction (2373) Error: Slim::Web::HTTP::addPageFunction() is deprecated - please use Slim::Web::Pages->addPageFunction() instead


It has been corrected in latest TrackStat version, it's just a warning so it doesn't cause any harm.

Christophenonstop
2010-01-05, 14:16
Hi Erland, thanks for support.

OK- back to TrackStat 2.10, I have enabled debug logging and here is the outcome for the import of one single problematic track :


[10-01-05 21:28:00.6149] Plugins::TrackStat::iTunes::Import::handleEndEleme nt (590) starting track parsing

[10-01-05 21:28:00.6191] Plugins::TrackStat::iTunes::Import::replace_proble matic_url_chars (698) original url was file:///media/Media/MUSIC-AUDIO/Kana/Entre Fr%E8res/09 - L%27h%E9ritage.mp3
[10-01-05 21:28:00.6193] Plugins::TrackStat::iTunes::Import::normalize_loca tion (690) normalized file://localhost/M:/MUSIC-AUDIO/Kana/Entre Fr%E8res/09 - L%27h%E9ritage.mp3 to file:///media/Media/MUSIC-AUDIO/Kana/Entre Fr%E8res/09 - L%27h%E9ritage.mp3

[10-01-05 21:28:00.6198] Plugins::TrackStat::iTunes::Import::handleTrack (424) file not found: /media/Media/MUSIC-AUDIO/Kana/Entre Frères/09 - L'héritage.mp3

[10-01-05 21:28:00.6201] Plugins::TrackStat::iTunes::Import::handleEndEleme nt (628) Finished scanning iTunes XML



I can tell from the log that the very same track is stored in the database under a different url


[10-01-05 21:29:58.3048] Plugins::TrackStat::Storage::findTrack (503) findTrack(): URL: file:///media/Media/MUSIC-AUDIO/Kana/Entre%20Fr%C3%A8res/09%20-%20L%27h%C3%A9ritage.mp3


So you were right, character encoding seems different. One lame option is seek and replace text strings within the xml, but I'd rather put up a permanent fix for later batches. Do you have a clue ?

---
Also, although I wish to, I could not set the Database Query plugin up, simply because I don't understand how to. I have set the following directories both owned by user (not root) with drwxr-xr-x rights :
- Data queries directory = /home/christophe/Backups/Configs/Squeezebox_Server/Database_Query/Queries
- Data queries templates directory = /home/christophe/Backups/Configs/Squeezebox_Server/Database_Query/Queries_Templates
But then I don't know what to do.

erland
2010-01-05, 15:16
So you were right, character encoding seems different. One lame option is seek and replace text strings within the xml, but I'd rather put up a permanent fix for later batches. Do you have a clue ?

How does your setup look like ?
Do you have the music files on the Linux box and publish them on the network with samba ?
I think you can set the character encoding on the samba share but it will probably result in that your current iTunes library will point to non existing files.

It would probably be possible to add a character encoding option in TrackStat and let the user select which conversion he wants. Alternatively it might be possible to change TrackStat so it tries multiple encoding but that might affect the performance of the import.



Also, although I wish to, I could not set the Database Query plugin up, simply because I don't understand how to. I have set the following directories both owned by user (not root) with drwxr-xr-x rights :
- Data queries directory = /home/christophe/Backups/Configs/Squeezebox_Server/Database_Query/Queries
- Data queries templates directory = /home/christophe/Backups/Configs/Squeezebox_Server/Database_Query/Queries_Templates
But then I don't know what to do.

The user running Squeezebox Server needs to have write access to these directories.
After that, you can goto Extras/Database Query in the web interface and create a new data query of the type "Free form query" and enter the SQL statement you like to execute.

Christophenonstop
2010-01-05, 23:50
How does your setup look like ?
Ubuntu 9.10 64-bit is my main OS with locale=en_US.UTF-8. It hosts SBS, the music files and also iTunes/XP thanks to VirtualBox.


Do you have the music files on the Linux box and publish them on the network with samba ?
Samba is next on the list after we solve this!


It would probably be possible to add a character encoding option in TrackStat
Sounds like the best option. I would test it if you need.

erland
2010-01-06, 00:13
Changed the path to /usr/share/squeezeboxserver/Queries but still nothing changed.
If you have a standard Ubuntu installation, the directory have to be owned by the user "squeezeboxserver".

To change the ownership you can run:
chown squeezeboxserver /home/christophe/Backups/Configs/Squeezebox_Server/Database_Query/Queries

If you don't use samba to share you music files between the Windows and Linux box how do both Windows and Linux get access to them ?

Do you use a shared folder in Virtualbox to accomplish it ?

Ubuntu typically use UTF8 encoded filenames while at least my Swedish Windows installation use ISO8859-1

Christophenonstop
2010-01-06, 12:15
Yes, I use shared folders in VirtualBox and 'map a network drive' in XP.

Christophenonstop
2010-01-06, 14:34
Nice tool for character transcoding :


iconv -f iso8859-1 -t utf8 input_file.xml -o output_file.xml


Unfortunately it does not work. It seems TrackStat requires original characters as an input, e.g. "" instead of "%E9".

%E9 %20 etc. is URL encoding which is absent from
iconv -l

erland
2010-01-06, 17:57
What's in the iTunes XML file ? or %E9 ?
It would also be interesting to know what's in the SBS database if you could get Database Query plugin to work.

Christophenonstop
2010-01-07, 00:04
Extract from xml as in gedit :


<dict>
<key>Track ID</key><integer>1877</integer>
<key>Name</key><string>L&apos;hritage</string>
<key>Artist</key><string>Kana</string>
<key>Album Artist</key><string>Kana</string>
<key>Album</key><string>Entre frres...</string>
<key>Genre</key><string>Reggae Franais</string>
<key>Kind</key><string>MPEG audio file</string>
<key>Size</key><integer>6120</integer>
<key>Total Time</key><integer>261000</integer>
<key>Track Number</key><integer>9</integer>
<key>Year</key><integer>2003</integer>
<key>Date Modified</key><date>Sun Sep 16 14:39:22 2007</date>
<key>Date Added</key><date>Sun May 18 17:10:40 2008</date>
<key>Play Date</key><integer>1172928668</integer>
<key>Play Date UTC</key><date>Sat Mar 03 14:31:08 2007</date>
<key>Bitrate</key><integer>192</integer>
<key>Play Count</key><integer>1</integer>
<key>Location</key><string>file://localhost/M:/MUSIC-AUDIO/Kana/Entre Fr%E8res/09 - L%27h%E9ritage.mp3</string>
<key>File Folder Count</key><integer>-1</integer>
<key>Library Folder Count</key><integer>-1</integer>
</dict>

You can see there is both and %E9.

Sorry but I can't get Database Query to work! I must be doing something wrong...
The path are still the same, I changed the rights to :


ls -l /home/christophe/Backups/Configs/
drwxr-xr-x 5 christophe christophe 4096 2010-01-06 20:03 Squeezebox_Server
ls -l /home/christophe/Backups/Configs/Squeezebox_Server/
drwxr-xr-x 4 squeezeboxserver christophe 4096 2010-01-06 20:03 Database_Queries
ls -l /home/christophe/Backups/Configs/Squeezebox_Server/Database_Queries/
drwxr-xr-x 2 squeezeboxserver christophe 4096 2010-01-06 20:03 Queries
drwxr-xr-x 2 squeezeboxserver christophe 4096 2010-01-06 20:03 Queries_Templates

Debug logging of the plugin does not report anything.

erland
2010-01-07, 01:37
You can see there is both and %E9.

iTunes just encodes the path (Location) and in your case it seems to be iso-8859-1.

The problem is probably that the paths on the Linux host is encoded as utf-8. I've no idea why, I think it should work because I've had a similar setup myself where it worked.

One solution might be to setup samba instead and see if it solves the problem.

Another alternative is to make sure the filenames and directories only contain us ascii characters.

If you do "ls" on the Linux box, do you see or some other corrupted character on the same file ?

If you see and the locale is en_US.UTF-8 Linux and Windows definitely seems to use different encoding for some reason.

In the same way on the Windows machine in Explorer(file manager) do you see or some corrupted character ?

I'm guessing that you have tried to play one of these tracks in iTunes and know that works ?



Sorry but I can't get Database Query to work! I must be doing something wrong...

Could you post your databasequery.prefs file, I'm guessing it's in /var/lib/squeezeboxserver/prefs/plugin/

Also check the owner of databasequery.prefs file, that should also tell us which user that runs SBS.

Christophenonstop
2010-01-11, 13:30
Ok it seems the character encoding problem is not because of the OS. I figured the XML database I am trying to import is actually part of a series of backups I made with Winamp 5 and not iTunes.

The problem seems to be because of the builtin ml_impex 2.04 plugin shipped with Winamp 5.56. This translator generates XML files featuring path with URL encoding (says Version 7.6.1 in the header). TrackStat can't read URL encoded path.

Instead, if I try with even older XML back ups, I can see it works fine with all files encoded with the original separate ml_impex 1.3 plugin. It means TrackStat and ml_impex together are a way to migrate a music database from Winamp to SBS. Oh by the way, XML files generated with iTunes 9.0 worked fine ;)

Erland, thanks a lot for your help. :)

Now I'm still stuck with my URL encoded XML file. I am looking for a tool to easily translate a mixed URL/plain text file to UTF8. Anyone ?



Could you post your databasequery.prefs file, I'm guessing it's in /var/lib/squeezeboxserver/prefs/plugin/
Also check the owner of databasequery.prefs file, that should also tell us which user that runs SBS.

Owner is "squeezeboxserver nogroup" like other files in /plugin.