Home of the Squeezebox™ & Transporter® network music players.
Page 2 of 8 FirstFirst 1234 ... LastLast
Results 11 to 20 of 72
  1. #11
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    10,689
    Quote Originally Posted by vagskal View Post
    Thanks, that did it.

    I scanned my +115k library and 9,548 duplicates were found with the default setting. When I clicked show duplicates SBS stalled (music playing stopped, mysql at 50% CPU) and I left it that way overnight. No change the morning after so I had to force SBS to quit. The same happened when I after a restart of SBS tried showing the duplicates again.

    When I checked the show duplicates .txt file in the beginning of the scan it worked and reported duplicates that were not duplicates (it was perhaps only half a page of data then).

    Log file is attached.
    It's not optimized for long lists of duplicates yet. Could you try to run it again but increasing the setting to 100 000 or even 250 000 ?
    Erland Isaksson (My homepage)
    (Developer of many plugins/applets (both free and commercial).
    If you like to encourage future presence on this forum and/or third party plugin/applet development, consider purchasing some plugins)

    Interested in the future of music streaming ? ickStream - A world of music at your fingertips.

  2. #12
    Senior Member vagskal's Avatar
    Join Date
    Oct 2008
    Location
    Sweden
    Posts
    643
    Quote Originally Posted by erland View Post
    It's not optimized for long lists of duplicates yet. Could you try to run it again but increasing the setting to 100 000 or even 250 000 ?
    The scan with a 100,000 setting took almost 3 hrs and here is the result:

    Detecting using (number of bytes): 100000
    Detected: 115777
    Duplicates: 1017 Show duplicates

    Trying to show the duplicates stalled SBS again so I cannot tell for sure if those are true duplicates, but the number seems high.

    Let me know if you have use for the server.log.
    2 x SB3 (wired), Receiver (wired), Boom (wireless), Controller, iPeng on iPhone 4 & iPad, muso on remote computer running Win 7 64-bit | 7.7.3 on Win XP

  3. #13
    Senior Member
    Join Date
    Dec 2009
    Location
    Germany
    Posts
    727
    I tried your plugin, but am not able to get it to display the list of duplicates.

  4. #14
    Senior Member vagskal's Avatar
    Join Date
    Oct 2008
    Location
    Sweden
    Posts
    643
    The result for a 250,000 setting is below:

    Detecting using (number of bytes): 250000
    Detected: 115777
    Duplicates: 1001 Show duplicates
    2 x SB3 (wired), Receiver (wired), Boom (wireless), Controller, iPeng on iPhone 4 & iPad, muso on remote computer running Win 7 64-bit | 7.7.3 on Win XP

  5. #15
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    10,689
    I've now released a new 0.2 version which includes three duplicate lists:
    - Checksum duplicates: Tracks with the same checksum
    - Incorrect duplicates: Tracks with the same checksum but different number of audio bytes
    - Duplicates: Real duplicates which both have the same checksum and the same number of audio bytes

    I also think I've fixed the performance problem which made it impossible to show identified duplicates in a large library when using MySQL as database.

    Anyone that runs this, please:
    - Report back the result for your library (the above mentioned numbers)
    - Try it with a number of different size settings to see how low you can go without getting any incorrect duplicates

    As Andy mentioned, we know that incorrect duplicates will be generated with lower settings and LAME encoded MP3 files. We also know that duplicates will be generated with lower settings and songs that begins with silence.
    Erland Isaksson (My homepage)
    (Developer of many plugins/applets (both free and commercial).
    If you like to encourage future presence on this forum and/or third party plugin/applet development, consider purchasing some plugins)

    Interested in the future of music streaming ? ickStream - A world of music at your fingertips.

  6. #16
    Senior Member vagskal's Avatar
    Join Date
    Oct 2008
    Location
    Sweden
    Posts
    643
    Below are the results on 7.5.2 latest build and the first part of the server.log where the plugin seemed to have an issue:
    Code:
    Detecting using (number of bytes): 1000000
    Detected: 115777
    Checksum duplicates: 975 Show checksum duplicates  (Incorrect duplicates: 52) 
    Duplicates: 923 Show duplicates 
    
    Detecting using (number of bytes): 500000
    Detected: 115777
    Checksum duplicates: 975 Show checksum duplicates  (Incorrect duplicates: 52) 
    Duplicates: 923 Show duplicates 
    
    Detecting using (number of bytes): 250000
    Detected: 115777
    Checksum duplicates: 1001 Show checksum duplicates  (Incorrect duplicates: 78) 
    Duplicates: 923 Show duplicates 
    
    
    
    
    [10-09-05 09:54:27.4051] main::init (323) Starting Squeezebox Server (v7.5.2, r31264, Sat Aug 28 02:06:44 PDT 2010) perl 5.010000
    [10-09-05 09:54:36.6707] Slim::Utils::Strings::parseStrings (351) Error: Parsing line 1: ´╗┐# Max Spicer, May 2007
    [10-09-05 09:54:47.9365] Slim::Utils::Misc::msg (1165) Warning: [09:54:47.9362] "my" variable $dbh masks earlier declaration in same scope at C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 319.
    [10-09-05 09:54:47.9368] Slim::Utils::Misc::msg (1165) Warning: [09:54:47.9366] "my" variable $sth masks earlier declaration in same scope at C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 320.
    [10-09-05 09:54:53.5301] Slim::Schema::Storage::throw_exception (82) Error: DBI Exception: DBD::mysql::db do failed: Unknown column 'audiosize' in 'field list'
    [10-09-05 09:54:53.5305] Slim::Schema::Storage::throw_exception (82) Backtrace:
    
       frame 0: Slim::Utils::Log::logBacktrace (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>Slim/Schema/Storage.pm line 82)
       frame 1: Slim::Schema::Storage::throw_exception (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>DBIx/Class/Storage/DBI.pm line 957)
       frame 2: DBIx::Class::Storage::DBI::__ANON__ (C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 104)
       frame 3: (eval) (C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 104)
       frame 4: Plugins::DuplicateDetector::Plugin::initDatabase (C:\Documents and Settings\All Users\Application Data\Squeezebox\Cache\InstalledPlugins/Plugins/DuplicateDetector/Plugin.pm line 81)
       frame 5: Plugins::DuplicateDetector::Plugin::initPlugin (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>Slim/Utils/PluginManager.pm line 328)
       frame 6: (eval) (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>Slim/Utils/PluginManager.pm line 328)
       frame 7: Slim::Utils::PluginManager::load (slimserver.pl line 507)
       frame 8: main::init (slimserver.pl line 578)
       frame 9: main::main (slimserver.pl line 99)
       frame 10: PerlSvc::Interactive (/<C:\Program\SQUEEZ~1\server\SQUEEZ~3.EXE>PerlSvc.pm line 99)
       frame 11: PerlSvc::_interactive (slimserver.pl line 0)
       frame 12: (eval) (slimserver.pl line 0)
    [10-09-05 09:54:53.8575] Plugins::DuplicateDetector::Plugin::initDatabase (111) Duplicate Detector: Creating database tables
    [10-09-05 09:54:53.9546] Plugins::DuplicateDetector::Plugin::createIndex (137) No smdidIndex index found in duplicatedetector_tracks, creating index...
    Let me know if you want me to try with an even higher setting or if you would like to see the duplicates lists or the entire server.log.
    2 x SB3 (wired), Receiver (wired), Boom (wireless), Controller, iPeng on iPhone 4 & iPad, muso on remote computer running Win 7 64-bit | 7.7.3 on Win XP

  7. #17
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    10,689
    Quote Originally Posted by vagskal View Post
    Below are the results on 7.5.2 latest build and the first part of the server.log where the plugin seemed to have an issue:
    You can ignore the SQL exception in the server.log, I didn't find any way to hide it. It's harmless and it's only output once for users which have had the previous version of the plugin installed.

    Quote Originally Posted by vagskal View Post
    Let me know if you want me to try with an even higher setting or if you would like to see the duplicates lists or the entire server.log.
    It's the "Incorrect duplicates" lists that are mostly interested, could you please:
    - Post the incorrectduplicates.txt file for one of the executions, doesn't matter which one.
    - Try and see if there is anything special with those tracks that could cause incorrect duplicates, for example a lot of silence in the beginning or something similar.

    If one or several of the rows in the incorrectduplicates.txt file starts with "NOCHECKSUM-", that indicates that no checksum calculation could be performed for these files. In that case it's very interesting to know the file format of these files and to verify that they are possible to play through SBS. I've seen some issues like this for m4a files from another user.
    Erland Isaksson (My homepage)
    (Developer of many plugins/applets (both free and commercial).
    If you like to encourage future presence on this forum and/or third party plugin/applet development, consider purchasing some plugins)

    Interested in the future of music streaming ? ickStream - A world of music at your fingertips.

  8. #18
    MrSinatra
    Guest
    hey Erland,

    as always, you impress. questions for you on this, although i can see you're still beta testing it:

    what is defined as a duplicate? same title, artist, etc? some kind of audio fingerprint? what if bitrates or formats are different of the same song? what if the source is different, like a remastered cd?

    will you be able to checkmark dupes you want to delete?

  9. #19
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    10,689
    Quote Originally Posted by MrSinatra View Post
    what is defined as a duplicate? same title, artist, etc? some kind of audio fingerprint? what if bitrates or formats are different of the same song? what if the source is different, like a remastered cd?
    It needs to be the exact same rip as it checks that the compressed audio bytes section of the file is the same. So remastered cd, different formats, bitrates will not be detected as duplicates.

    Quote Originally Posted by MrSinatra View Post
    will you be able to checkmark dupes you want to delete?
    No, for three reasons:
    1. SBS might not have write access to the file system where the music files are.
    2. I don't want to make it easy for users to accidentally delete their music files.
    3. The intention of this plugin is to verify the algorithm that's used to identify a specific music file even if it has been re-tagged or renamed/moved. The algorithm will later be used to connect metadata and statistics to a specific music file and make sure that relation survives a rename/move or retagging of the file.

    It is possible to export all duplicates to a text file.
    Erland Isaksson (My homepage)
    (Developer of many plugins/applets (both free and commercial).
    If you like to encourage future presence on this forum and/or third party plugin/applet development, consider purchasing some plugins)

    Interested in the future of music streaming ? ickStream - A world of music at your fingertips.

  10. #20
    Administrator andyg's Avatar
    Join Date
    Jan 2006
    Location
    Pittsburgh, PA
    Posts
    7,395

    Need help to verify duplicate detection

    On Sep 5, 2010, at 11:36 PM, erland wrote:

    >
    > MrSinatra;574572 Wrote:
    >>
    >> what is defined as a duplicate? same title, artist, etc? some kind of
    >> audio fingerprint? what if bitrates or formats are different of the
    >> same song? what if the source is different, like a remastered cd?
    >>

    > It needs to be the exact same rip as it checks that the compressed
    > audio bytes section of the file is the same. So remastered cd,
    > different formats, bitrates will not be detected as duplicates.


    Right, this is not doing audio fingerprinting, just a checksum. Fingerprinting is hard and slow, as you have to decode every type of audio format to PCM and then process the raw audio data.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •