Need help to verify duplicate detection

[erland]

Thanks, that did it.

I scanned my +115k library and 9,548 duplicates were found with the default setting. When I clicked show duplicates SBS stalled (music playing stopped, mysql at 50% CPU) and I left it that way overnight. No change the morning after so I had to force SBS to quit. The same happened when I after a restart of SBS tried showing the duplicates again.

When I checked the show duplicates .txt file in the beginning of the scan it worked and reported duplicates that were not duplicates (it was perhaps only half a page of data then).

Log file is attached.

It's not optimized for long lists of duplicates yet. Could you try to run it again but increasing the setting to 100 000 or even 250 000 ?

[vagskal]

It's not optimized for long lists of duplicates yet. Could you try to run it again but increasing the setting to 100 000 or even 250 000 ?

The scan with a 100,000 setting took almost 3 hrs and here is the result:

Detecting using (number of bytes): 100000
Detected: 115777
Duplicates: 1017 Show duplicates

Trying to show the duplicates stalled SBS again so I cannot tell for sure if those are true duplicates, but the number seems high.

Let me know if you have use for the server.log.

[copperstate]

I tried your plugin, but am not able to get it to display the list of duplicates.

[vagskal]

The result for a 250,000 setting is below:

Detecting using (number of bytes): 250000
Detected: 115777
Duplicates: 1001 Show duplicates

[erland]

I've now released a new 0.2 version which includes three duplicate lists:
- Checksum duplicates: Tracks with the same checksum
- Incorrect duplicates: Tracks with the same checksum but different number of audio bytes
- Duplicates: Real duplicates which both have the same checksum and the same number of audio bytes

I also think I've fixed the performance problem which made it impossible to show identified duplicates in a large library when using MySQL as database.

Anyone that runs this, please:
- Report back the result for your library (the above mentioned numbers)
- Try it with a number of different size settings to see how low you can go without getting any incorrect duplicates

As Andy mentioned, we know that incorrect duplicates will be generated with lower settings and LAME encoded MP3 files. We also know that duplicates will be generated with lower settings and songs that begins with silence.

[vagskal]

Below are the results on 7.5.2 latest build and the first part of the server.log where the plugin seemed to have an issue:

Detecting using (number of bytes): 1000000
Detected: 115777
Checksum duplicates: 975 Show checksum duplicates  (Incorrect duplicates: 52) 
Duplicates: 923 Show duplicates 

Detecting using (number of bytes): 500000
Detected: 115777
Checksum duplicates: 975 Show checksum duplicates  (Incorrect duplicates: 52) 
Duplicates: 923 Show duplicates 

Detecting using (number of bytes): 250000
Detected: 115777
Checksum duplicates: 1001 Show checksum duplicates  (Incorrect duplicates: 78) 
Duplicates: 923 Show duplicates 




[10-09-05 09:54:27.4051] main::init (323) Starting Squeezebox Server (v7.5.2, r31264, Sat Aug 28 02:06:44 PDT 2010) perl 5.010000
[10-09-05 09:54:36.6707] Slim::Utils::Strings::parseStrings (351) Error: Parsing line 1: # Max Spicer, May 2007
[10-09-05 09:54:47.9365] Slim::Utils::Misc::msg (1165) Warning: [09:54:47.9362] "my" variable $dbh masks earlier declaration in same scope at C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 319.
[10-09-05 09:54:47.9368] Slim::Utils::Misc::msg (1165) Warning: [09:54:47.9366] "my" variable $sth masks earlier declaration in same scope at C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 320.
[10-09-05 09:54:53.5301] Slim::Schema::Storage::throw_exception (82) Error: DBI Exception: DBD::mysql::db do failed: Unknown column 'audiosize' in 'field list'
[10-09-05 09:54:53.5305] Slim::Schema::Storage::throw_exception (82) Backtrace:

   frame 0: Slim::Utils::Log::logBacktrace (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>Slim/Schema/Storage.pm line 82)
   frame 1: Slim::Schema::Storage::throw_exception (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>DBIx/Class/Storage/DBI.pm line 957)
   frame 2: DBIx::Class::Storage::DBI::__ANON__ (C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 104)
   frame 3: (eval) (C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 104)
   frame 4: Plugins::DuplicateDetector::Plugin::initDatabase (C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 81)
   frame 5: Plugins::DuplicateDetector::Plugin::initPlugin (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>Slim/Utils/PluginManager.pm line 328)
   frame 6: (eval) (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>Slim/Utils/PluginManager.pm line 328)
   frame 7: Slim::Utils::PluginManager::load (slimserver.pl line 507)
   frame 8: main::init (slimserver.pl line 578)
   frame 9: main::main (slimserver.pl line 99)
   frame 10: PerlSvc::Interactive (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>PerlSvc.pm line 99)
   frame 11: PerlSvc::_interactive (slimserver.pl line 0)
   frame 12: (eval) (slimserver.pl line 0)
[10-09-05 09:54:53.8575] Plugins::DuplicateDetector::Plugin::initDatabase (111) Duplicate Detector: Creating database tables
[10-09-05 09:54:53.9546] Plugins::DuplicateDetector::Plugin::createIndex (137) No smdidIndex index found in duplicatedetector_tracks, creating index...

Let me know if you want me to try with an even higher setting or if you would like to see the duplicates lists or the entire server.log.

[erland]

Below are the results on 7.5.2 latest build and the first part of the server.log where the plugin seemed to have an issue:

You can ignore the SQL exception in the server.log, I didn't find any way to hide it. It's harmless and it's only output once for users which have had the previous version of the plugin installed.

Let me know if you want me to try with an even higher setting or if you would like to see the duplicates lists or the entire server.log.

It's the "Incorrect duplicates" lists that are mostly interested, could you please:
- Post the incorrectduplicates.txt file for one of the executions, doesn't matter which one.
- Try and see if there is anything special with those tracks that could cause incorrect duplicates, for example a lot of silence in the beginning or something similar.

If one or several of the rows in the incorrectduplicates.txt file starts with "NOCHECKSUM-", that indicates that no checksum calculation could be performed for these files. In that case it's very interesting to know the file format of these files and to verify that they are possible to play through SBS. I've seen some issues like this for m4a files from another user.

[MrSinatra]

hey Erland,

as always, you impress. questions for you on this, although i can see you're still beta testing it:

what is defined as a duplicate? same title, artist, etc? some kind of audio fingerprint? what if bitrates or formats are different of the same song? what if the source is different, like a remastered cd?

will you be able to checkmark dupes you want to delete?

[erland]

what is defined as a duplicate? same title, artist, etc? some kind of audio fingerprint? what if bitrates or formats are different of the same song? what if the source is different, like a remastered cd?

It needs to be the exact same rip as it checks that the compressed audio bytes section of the file is the same. So remastered cd, different formats, bitrates will not be detected as duplicates.

will you be able to checkmark dupes you want to delete?

No, for three reasons:
1. SBS might not have write access to the file system where the music files are.
2. I don't want to make it easy for users to accidentally delete their music files.
3. The intention of this plugin is to verify the algorithm that's used to identify a specific music file even if it has been re-tagged or renamed/moved. The algorithm will later be used to connect metadata and statistics to a specific music file and make sure that relation survives a rename/move or retagging of the file.

It is possible to export all duplicates to a text file.

[andyg]

On Sep 5, 2010, at 11:36 PM, erland wrote:

>
> MrSinatra;574572 Wrote:
>>
>> what is defined as a duplicate? same title, artist, etc? some kind of
>> audio fingerprint? what if bitrates or formats are different of the
>> same song? what if the source is different, like a remastered cd?
>>
> It needs to be the exact same rip as it checks that the compressed
> audio bytes section of the file is the same. So remastered cd,
> different formats, bitrates will not be detected as duplicates.

Right, this is not doing audio fingerprinting, just a checksum. Fingerprinting is hard and slow, as you have to decode every type of audio format to PCM and then process the raw audio data.