Home of the Squeezebox™ & Transporter® network music players.
Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 22
  1. #11
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    10,888
    Thanks both of you.

    Just for reference, this is what I ended up with:
    Code:
    dd if=/dev/urandom bs=2 count=100 2>/dev/null|flac -s --no-padding --endian=big --sign=signed --channels=2 --bps=16 --sample-rate=44100 - -f -o somefile.flac --tag="ALBUM=Some Album" --tag="ARTIST=Some Artist" --tag="TITLE=Some Track"
    If anyone have similar needs to generate a large library, let me know and I'll package it up a bit so it's usable by someone else. The scripts will generate a library of specified number of flac files with tag data and directory structure based on data from a discogs database dump. A 50 000 track library takes up 212 MB storage space on the disk. Took about 8 minutes to rescan in my setup without any third party plugins activated.

    If someone just wants a 50 000 tracks library to to some performance testing with, I've uploaded the ones I generated here:
    http://erland.isaksson.info/arkiv/flac_test_libraries/
    (I'll upload a 100 000 tracks library a bit later if there is an interest for this)
    Erland Isaksson (My homepage)
    (Developer of many plugins/applets (both free and commercial).
    If you like to encourage future presence on this forum and/or third party plugin/applet development, consider purchasing some plugins)

    Interested in the future of music streaming ? ickStream - A world of music at your fingertips.

  2. #12
    Senior Member
    Join Date
    Mar 2008
    Location
    Paris
    Posts
    153
    Quote Originally Posted by erland View Post
    (I'll upload a 100 000 tracks library a bit later if there is an interest for this)
    Erland, i'm interested in the 100 000 "big one" for test purpose.
    Regards
    STREAM => SqueezeBoxServer 7.7.3 / ReadyNas Pro (x86) | SB3 - Duet - Boom - Ipeng (Iphone) - Touch
    AUDIO => Rega DAC | Rega Brio amp | Triangle Celius 202 speakers

  3. #13
    Senior Member
    Join Date
    Apr 2005
    Location
    Colorado
    Posts
    10,969
    Is 8 minutes a normal amount of time to scan 50k tracks on your server? Just wondering how realistic scanning times are with no music data in the files. I would think it shouldn't make a difference, but maybe there is. The only reason I can see for having a very large test library like this would be performance testing.

    How do you pull down the tagging data? Just by requesting data for random albums?

  4. #14
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    10,888
    Quote Originally Posted by JJZolx View Post
    Is 8 minutes a normal amount of time to scan 50k tracks on your server? Just wondering how realistic scanning times are with no music data in the files. I would think it shouldn't make a difference, but maybe there is.
    I can't compare it exactly because normally the files are on a NAS but in this case it's on a local disk. Also the SBS is running in a virtual machine so it can be affected by other stuff running on the physical machine.

    But I did a quick check and generated a random library in with the same size as my main FLAC library and also one with the same metadata with the flactruncate.pl script linked in a previous post in this thread and the result with all files on the NAS was as follows (all libraries contains 3495 tracks)
    - Real library (91 GB): full rescan 92 seconds
    - flactruncate.pl library (31 MB): full rescan 32 seconds
    - Random generated flac library (15 MB): full rescan 25 seconds

    Just for reference I also tried to put the random generated flac library on the local drive and then it takes 18 seconds to do a full rescan.

    So based on this, I'd say that it can't be used to realistically test the performance of the scanner as it's mostly restricted by I/O and larger files result in more I/O.

    Quote Originally Posted by JJZolx View Post
    The only reason I can see for having a very large test library like this would be performance testing.
    Yes, that's the plan I have, I've had trouble testing performance of my plugins previously since my own library is fairly small compared to the library sizes some other users have.

    Based on the above analysis I think it's mostly usable for performance testing of browsing functionality. However, it could possibly be used to find database performance issues with scanner in larger libraries, but not not to measure the scanner performance as that's heavily affected by I/O which will be very different with these small FLAC files.

    Quote Originally Posted by JJZolx View Post
    How do you pull down the tagging data? Just by requesting data for random albums?
    From a local discogs dump, they have XML dump files you can download from here:
    http://www.discogs.com/data/

    It won't work to call their web service, that would take too much time and it would violate their web service license restrictions which doesn't allow you to make more than a certain number of calls per minute.

    I've based it on the "releases" dump and just picks albums from the beginning until I have as many as I like, skipping albums with very few tracks to make it a bit more realistic.
    Erland Isaksson (My homepage)
    (Developer of many plugins/applets (both free and commercial).
    If you like to encourage future presence on this forum and/or third party plugin/applet development, consider purchasing some plugins)

    Interested in the future of music streaming ? ickStream - A world of music at your fingertips.

  5. #15
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    10,888
    Quote Originally Posted by volpone View Post
    Erland, i'm interested in the 100 000 "big one" for test purpose.
    Regards
    It's now on my web site, you will find it here:
    http://erland.isaksson.info/arkiv/flac_test_libraries/

    I just noticed that it doesn't contain any YEAR information so I'll see if I can generate new dumps some time later this week which also contains YEAR(or DATE) tags.

    Please take a look and see if there is any other tags you are also missing and I'll see if I can add those too.
    Erland Isaksson (My homepage)
    (Developer of many plugins/applets (both free and commercial).
    If you like to encourage future presence on this forum and/or third party plugin/applet development, consider purchasing some plugins)

    Interested in the future of music streaming ? ickStream - A world of music at your fingertips.

  6. #16
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    10,888
    Quote Originally Posted by erland View Post
    It's now on my web site, you will find it here:
    http://erland.isaksson.info/arkiv/flac_test_libraries/

    I just noticed that it doesn't contain any YEAR information so I'll see if I can generate new dumps some time later this week which also contains YEAR(or DATE) tags.

    Please take a look and see if there is any other tags you are also missing and I'll see if I can add those too.
    I've uploaded new versions of these test libraries, now with YEAR tags to make them a bit more similar to a real setup.
    Erland Isaksson (My homepage)
    (Developer of many plugins/applets (both free and commercial).
    If you like to encourage future presence on this forum and/or third party plugin/applet development, consider purchasing some plugins)

    Interested in the future of music streaming ? ickStream - A world of music at your fingertips.

  7. #17
    Member
    Join Date
    Mar 2008
    Location
    Belgium
    Posts
    73
    I think there might be some incompatibilities with Windows utilities and file/directory names - for the 100.000 archive WinRAR displays "Unexpected end of archive". The 50.000 archive tests OK, but during extraction it fails to create some 550 files - most likely those which don't comply with Windows naming standards...
    Anyways, what I have is enough for doing my tests.

  8. #18
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    10,888
    Quote Originally Posted by amatala View Post
    I think there might be some incompatibilities with Windows utilities and file/directory names - for the 100.000 archive WinRAR displays "Unexpected end of archive". The 50.000 archive tests OK, but during extraction it fails to create some 550 files - most likely those which don't comply with Windows naming standards...
    Anyways, what I have is enough for doing my tests.
    Just out of interest.

    Are you sure the zip was downloaded successfully ?
    Do you know which file inside the zip the error occurred on or which file inside the zip that was the last one successfully extracted ?

    If you have the ability to run a MD5 checksum tool on the files their MD5 checksums should be:
    Code:
    9ee47150f20a9766686496fed35b3e88  flac100000.zip
    6911cbc4fa4827eda69a5f2288d530b8  flac50000.zip
    Erland Isaksson (My homepage)
    (Developer of many plugins/applets (both free and commercial).
    If you like to encourage future presence on this forum and/or third party plugin/applet development, consider purchasing some plugins)

    Interested in the future of music streaming ? ickStream - A world of music at your fingertips.

  9. #19
    Member
    Join Date
    Mar 2008
    Location
    Belgium
    Posts
    73
    I downloaded the files twice without any error.
    I think for the 100K archive it's a WinRAR issue - I know that RAR archives which contain very large number of files should be created with a larger dictionary size - maybe a similar limitation applies to the way it handles ZIP files.
    For the 50K it all tests fine and it extracts all but 550 files - it does not stop on error, it just skips the files and continues. I will retrieve the list of files from the logs once I get home and post it for your info.

  10. #20
    Member
    Join Date
    Mar 2008
    Location
    Belgium
    Posts
    73
    Here are some of the files which could not get created on my Windows system - it seems to be mostly names including a "\" or folders starting with a space:

    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Suzuki In Dub_028800a0-751e-4ede-a1fc-de6ba0d867fa\1.Suzuki (Burnt Friedman \ Nonplace Dub).flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Gemini_c74d7c21-17f3-444b-b417-08ed8ad17252\10.Serrano Anthem (Amor \ Amor).flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Surrender (Australia \ NZ Tour Collection)_5a33e34a-9f04-48c0-8015-cd0b334b6a43
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\The Jackal_c03295c1-fe20-4d6f-9272-252773ad7f70\5.Swallowed (Goldie \ Toasted On Both Sides Mix).flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Second Toughest In The Infants_7fb49dc8-2d8d-47d9-82ee-aa81b0af670c\1-1.Juanita \ Kiteless
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Second Toughest In The Infants_7fb49dc8-2d8d-47d9-82ee-aa81b0af670c\1-2.Banstyle \ Sappy's Curry.flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Centripetal_aa5a1d9f-615a-4569-be93-e21a3e03f17f\17.Ebb \ Esho Funi.flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Renaissance_ The Masters Series Part Two_ Ibiza_a54ab817-79ab-4d01-b3d8-807792750836\1-13.Rise (Deep Dish Hi-Rise Dub \ Remix).flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Renaissance_ The Masters Series Part Two_ Ibiza_a54ab817-79ab-4d01-b3d8-807792750836\2-04.Everytime (Timo Maas Dub \ Vocal Remix).flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Renaissance_ The Masters Series Part Two_ Ibiza_a54ab817-79ab-4d01-b3d8-807792750836\1-04.Temperamental (Hex Hector \ Mac Quale Reverse Drum Dub
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Preaching Through The Converted_6baf7b27-2a26-4e59-a7c6-7189bfeabbab\5.Part1_ In The Company Of Myself \ Interlude
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Seminal Live_adf8b9f9-c668-45c3-9be7-33da16ff3453\13.Introduction \ Cruisers Creek.flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Seminal Live_adf8b9f9-c668-45c3-9be7-33da16ff3453\10.Elf Prefix \ L.A.flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create folder flac50000\Colongib \ Pacman Split_6045d38e-ec74-4096-a36f-aa00555977a9
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Back And Forth Series Two_324cdafd-8ff2-4d29-9beb-7f688fd8ff63\10.A.M. \ Meat Flavour.flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Back And Forth Series Two_324cdafd-8ff2-4d29-9beb-7f688fd8ff63\7.Sore In A Masterpiece \ Dead Of Winter.flac
    The system cannot find the path specified.
    ! D:\TEMP\flac50000.zip: Cannot create flac50000\Hold Up_aa3398c0-44c7-4f33-abb4-647fcb239b1c\1.To Marseille From Paris (Dim's Megamix \ Interlude By Dimitri From Paris).flac
    The system cannot find the path specified.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •