PDA

View Full Version : 18 hrs for wipe & rescan - Normal?



vagskal
2009-07-03, 07:38
A wipe and rescan in SC took over 18 hours with no significant messages in the scanner log.

Mappsökning (98053 av 98053) Fullständig 08:08:54

MusicIP-import (98052 av 98052) Fullständig 03:14:08

Sökning i spellista (6 av 6) Fullständig 00:00:36

Slå ihop Blandade artister (6674 av 6674) Fullständig 00:06:17

Sökning efter albumomslag (7614 av 7614) Fullständig 06:38:00

Optimering av databas Fullständig 00:09:05

Sökningen av ditt musikbibliotek har slutförts.
Total tid: 18:17:00 (fredag 3 juli 2009 / 09.27)


Could that be normal? Is there anything I could do to speed things up?

The scanner log says I have got version 7.3.3, but I am on 7.3.4.

Below are my details and the scanner log:

Version: 7.3.4 - 27205 @ Sat Jun 27 03:01:49 PDT 2009
Värdnamn: Musikmaskinen
Serverns IP-adress: 192.168.1.4
HTTP-serverns portnummer: 9000
Operativsystem: Windows XP - SV - cp1252
Plattformsarkitektur: 586
Perl-version: 5.8.8 - MSWin32-x86-multi-thread
MySQL-version: 5.0.22-community-nt
Totalt antal anslutna spelare: 4

Computer:
Operating System Microsoft Windows XP Professional
OS Service Pack Service Pack 3
Motherboard:
CPU Type Intel Pentium 4E, 2800 MHz (3.5 x 800)
Motherboard Name Abit IC7-MAX3 (5 PCI, 1 AGP Pro, 4 DIMM, Audio, Gigabit LAN, IEEE-1394)
Motherboard Chipset Intel Canterwood i875P
System Memory 1024 MB (DDR SDRAM)
BIOS Type Award (10/30/03)

Scanner log:
[09-07-01 19:56:03.8125] main::main (122) Starting SqueezeCenter scanner (v7.3.3, r27205, Sat Jun 27 03:01:49 PDT 2009)
[09-07-02 04:00:35.0156] main::main (122) Starting SqueezeCenter scanner (v7.3.3, r27205, Sat Jun 27 03:01:49 PDT 2009)
[09-07-02 12:56:11.6562] main::main (122) Starting SqueezeCenter scanner (v7.3.3, r27205, Sat Jun 27 03:01:49 PDT 2009)
[09-07-02 19:00:35.6879] Slim::Formats::Playlists::CUE::parse (167) Error: Couldn't find referenced FILE: [\\Musikmaskinen\Elements (I)\MP3\Min musik\Artister\Marvin Gaye\What's Going On [Deluxe Edition] (1971)\Marvin Gaye - What's Going On Deluxe Edition (Disc-1).flac] on disk! Skipping!
[09-07-02 19:00:35.7042] Slim::Formats::Playlists::CUE::parse (167) Error: Couldn't find referenced FILE: [\\Musikmaskinen\Elements (I)\MP3\Min musik\Artister\Marvin Gaye\What's Going On [Deluxe Edition] (1971)\Marvin Gaye - What's Going On Deluxe Edition (Disc-1).flac] on disk! Skipping!

[etc.]

[09-07-02 19:00:35.7857] Slim::Formats::Playlists::CUE::parse (167) Error: Couldn't find referenced FILE: [\\Musikmaskinen\Elements (I)\MP3\Min musik\Artister\Marvin Gaye\What's Going On [Deluxe Edition] (1971)\Marvin Gaye - What's Going On Deluxe Edition (Disc-1).flac] on disk! Skipping!
[09-07-02 19:00:36.2243] Slim::Formats::Playlists::CUE::parse (167) Error: Couldn't find referenced FILE: [\\Musikmaskinen\Elements (I)\MP3\Min musik\Artister\Marvin Gaye\What's Going On [Deluxe Edition] (1971)\Marvin Gaye - What's Going On Deluxe Edition (Disc-2).flac] on disk! Skipping!
[09-07-02 19:00:36.2288] Slim::Formats::Playlists::CUE::parse (167) Error: Couldn't find referenced FILE: [\\Musikmaskinen\Elements (I)\MP3\Min musik\Artister\Marvin Gaye\What's Going On [Deluxe Edition] (1971)\Marvin Gaye - What's Going On Deluxe Edition (Disc-2).flac] on disk! Skipping!

[etc.]

[09-07-02 19:00:36.2891] Slim::Formats::Playlists::CUE::parse (167) Error: Couldn't find referenced FILE: [\\Musikmaskinen\Elements (I)\MP3\Min musik\Artister\Marvin Gaye\What's Going On [Deluxe Edition] (1971)\Marvin Gaye - What's Going On Deluxe Edition (Disc-2).flac] on disk! Skipping!

amcluesent
2009-07-03, 07:51
18 hours is a loooong time.

>\\Musikmaskinen\Elements....<

How are you mounting the music files on the server running Squeezcenter? Is that a network share, USB disk or a local disk?

Could just be a dog-slow NAS...

vagskal
2009-07-03, 08:02
How are you mounting the music files on the server running Squeezcenter? Is that a network share, USB disk or a local disk?

It is a USB disc connected directly to the PC running SC.

When I investigated things I noticed that the USB disc with the music files is FAT32, while all internal discs on that PC is NTFS.

The PC running SC is only used for that and for backup during the night.

SuperQ
2009-07-03, 09:02
I wonder if the USB disk is connected at high speed or "full" speed (usb 1.1)

vagskal
2009-07-03, 09:07
I wonder if the USB disk is connected at high speed or "full" speed (usb 1.1)

I suppose it is a high speed usb connection. How can I verify this (where do I look or how do I test it)?

Playing music from the disc works well, as does copying files to the disc over the network.

vagskal
2009-07-03, 09:22
I suppose it is a high speed usb connection. How can I verify this (where do I look or how do I test it)?

Playing music from the disc works well, as does copying files to the disc over the network.

I think I have found the information:

Port BC00-BC1F Exclusive Intel(R) 82801EB USB Universal Host Controller - 24D2
Port B800-B81F Exclusive Intel(R) 82801EB USB Universal Host Controller - 24DE
Port B400-B41F Exclusive Intel(R) 82801EB USB Universal Host Controller - 24D7
Port B000-B01F Exclusive Intel(R) 82801EB USB Universal Host Controller - 24D4
Memory FC100000-FC1003FF Exclusive Intel(R) 82801EB USB2 Enhanced Host Controller - 24DD
IRQ 23 Shared Intel(R) 82801EB USB2 Enhanced Host Controller - 24DD
IRQ 19 Shared Intel(R) 82801EB USB Universal Host Controller - 24D4
IRQ 18 Shared Intel(R) 82801EB USB Universal Host Controller - 24D7
IRQ 16 Shared Intel(R) 82801EB USB Universal Host Controller - 24D2
IRQ 16 Shared Intel(R) 82801EB USB Universal Host Controller - 24DE

I have an "Enhanced" USB2 host controller, which (according to this page http://www.usbman.com/Guides/checking_for_usb_2.htm) means that I have High Speed USB (USB 2.0).

sebp
2009-07-03, 15:46
That's a whole lot of files on a USB disk, driven by a rather old machine.
Are you sure your USB disk enclosure is USB2 compatible?

vagskal
2009-07-04, 00:41
Are you sure your USB disk enclosure is USB2 compatible?

Yes, it is a fairly new disc from Western Digital 1 TB. I checked the specifications again:

Western Digital Extern HDD Elements 1TB - Svart (USB 2.0)

Fast, simple, reliable add-on storage.

This external USB 2.0 drive is preformatted for immediate use. Simply plug it in to a USB port and you've got instant storage for your PC or Macintosh® computer.

Features:
• High-speed USB 2.0 - For maximum performance; backward compatible with USB 1.1 ports.
• Plug and Play Storage - Just plug it in and connect the cable to add high-speed storage for both desktop and notebook computers.
• Rugged Metal Case - Compact, durable metal case for extra data protection.

Specifikationer:
• Data transfer rate 480 Mb/s (maximum)
• Read seek time 8.9 ms (average)
• Rotational speed 7200 RPM
• RoHS compliant Yes
• Warranty 2-year limited

vagskal
2009-07-04, 01:35
BTW, just the MusicIP-import took over 3 hours, and the MusicIP library file is in its default location, i.e. on the internal (C:) disc where SC also is; I think the internal discs are all IDE, not SATA.

MusicIP-import (98052 av 98052) Fullständig 03:14:08

Phil Leigh
2009-07-04, 05:46
How many tracks do you have. My system is very similar to yours and takes 40 mins for 30K tracks (of which 32 mins is the MusicIP import). This is on 7.4 but 7.3.x was about 5 mins slower.

You don't have both a music folder AND musicIP import set do you?

vagskal
2009-07-04, 05:53
How many tracks do you have. My system is very similar to yours and takes 40 mins for 30K tracks (of which 32 mins is the MusicIP import). This is on 7.4 but 7.3.x was about 5 mins slower.

Some 98K tracks.


You don't have both a music folder AND musicIP import set do you?

No, I have the music folder set in SC and have chosen to import from Music-IP only the mix status.

Phil Leigh
2009-07-04, 05:56
Some 98K tracks.



No, I have the music folder set in SC and have chosen to import from Music-IP only the mix status.

Ah - OK. 98K tracks will take a while. On my system I'd expect that to take maybe 3-4 hours?

You might like to try and do what I do, and set no music folder and then use MusicIP as the source for all data (not just mix status). At least that way you will only do 1 read of the files...

vagskal
2009-07-04, 06:20
Ah - OK. 98K tracks will take a while. On my system I'd expect that to take maybe 3-4 hours?

Because of the duration I seldom do a wipe and rescan. The last time (before this 18 hours session) I think it took 6-8 hours. A scan for new and changed music will take about two hours, or a little less.


You might like to try and do what I do, and set no music folder and then use MusicIP as the source for all data (not just mix status). At least that way you will only do 1 read of the files...

Thanks, but I have deliberately chosen not to do that, because all my tags would not be imported into SC that way (I cannot now remember exactly what the problem was). I would rather skip the MusicIP-import altogether, i.e. not use MusicIP. Still, if it takes 18 hours with MusicIP-import and three hours less without it, I think that is also too long.

I am hoping to get some more suggestions in the forum and then apply the suggested measures (including disabling MusicIP), cross my fingers and take a deep breath and do another wipe and rescan and hope for the best. For me, the wipe and rescan takes too long to resort to testing the measures one by one.

Another thing I will do before the next wipe and rescan is to disable completely CustomScan. Now I scan automatically for two custom tags, REVIEW and ORIGYEAR, where REVIEW can contain a lot of text (the review from AMG). All other scans in CustomScan are set to Manual, i.e. they should not be performed during a wipe and rescan. Could this CustomScan have anything to do with the long scanning time?

Phil Leigh
2009-07-04, 06:47
Because of the duration I seldom do a wipe and rescan. The last time (before this 18 hours session) I think it took 6-8 hours. A scan for new and changed music will take about two hours, or a little less.



Thanks, but I have deliberately chosen not to do that, because all my tags would not be imported into SC that way (I cannot now remember exactly what the problem was). I would rather skip the MusicIP-import altogether, i.e. not use MusicIP. Still, if it takes 18 hours with MusicIP-import and three hours less without it, I think that is also too long.

I am hoping to get some more suggestions in the forum and then apply the suggested measures (including disabling MusicIP), cross my fingers and take a deep breath and do another wipe and rescan and hope for the best. For me, the wipe and rescan takes too long to resort to testing the measures one by one.

Another thing I will do before the next wipe and rescan is to disable completely CustomScan. Now I scan automatically for two custom tags, REVIEW and ORIGYEAR, where REVIEW can contain a lot of text (the review from AMG). All other scans in CustomScan are set to Manual, i.e. they should not be performed during a wipe and rescan. Could this CustomScan have anything to do with the long scanning time?

Yes it could be CustomScan (which I have disabled at the moment because of issues with 7.4).
But I don't understand why you wouldn't just import from MusicIp - It doesn't lose any Tag data (believe me I'd notice if it did!).

vagskal
2009-07-04, 07:07
But I don't understand why you wouldn't just import from MusicIp - It doesn't lose any Tag data (believe me I'd notice if it did!).

If I remember correctly it had something to do with Vorbis comments in flac files. I think that the ALBUMARTIST tag was not read at all and that only one of several tags for ARTIST and COMPOSER was read. I seem to recall that if I tried to save the track id to the file, all existing tags were replaced with what was in the MusicIP database. That was a while ago, but I thought that MusicIP was no longer actively developed.

mherger
2009-07-04, 22:53
> [09-07-02 19:00:35.6879] Slim::Formats::Playlists::CUE::parse (167)
> Error: Couldn't find referenced FILE: [\\Musikmaskinen\Elements
> (I)\MP3\Min musik\Artister\Marvin Gaye\What's Going On [Deluxe Edition]
> (1971)\Marvin Gaye - What's Going On Deluxe Edition (Disc-1).flac] on
> disk! Skipping!

Is this just a bad CUE sheet or are you indeed accessing your local files
using an UNC path? What's your audiodir set to?

--

Michael

vagskal
2009-07-05, 01:28
> [09-07-02 19:00:35.6879] Slim::Formats::Playlists::CUE::parse (167)
> Error: Couldn't find referenced FILE: [\\Musikmaskinen\Elements
> (I)\MP3\Min musik\Artister\Marvin Gaye\What's Going On [Deluxe Edition]
> (1971)\Marvin Gaye - What's Going On Deluxe Edition (Disc-1).flac] on
> disk! Skipping!

Is this just a bad CUE sheet or are you indeed accessing your local files
using an UNC path? What's your audiodir set to?

--

Michael

Yes, the cue sheet was bad. I had changed the name of the files without editing the cue sheets. I have corrected that now and a scan for new or changed music picked the album up correctly.

And yes, I use unc paths for local discs. My music directory in SC is set to \\Musikmaskinen\Elements (I)\MP3\Min musik. In the referenced folder (on an usb 2.0 disc) there are folders containing the bulk of my music files AND a short cut with the unc path "\\Musikmaskinen\Musik (H)\Mina CD (Flac)" pointing to an internal disc (IDE, I think) where some additional music files (flac) are stored.

The paths were that way also when I did the penultimate wipe and rescan which took 6-8 hours, instead of 18 hours for the last one.


The reason for using unc paths is that I had trouble with muso, http://klarita.net/forum/index.php?topic=175.0. I think the issue has been addressed now.

Could using unc paths slow things down in SC?

mherger
2009-07-05, 06:34
> And yes, I use unc paths for local discs. My music directory in SC is
> set to \\Musikmaskinen\Elements (I)\MP3\Min musik. In the referenced

I'd not recommend using UNC paths for local file. It will always be slower.

--

Michael

vagskal
2009-07-08, 02:47
Thanks all for the hints.

This is what I did:

1. Installed the latest 7.3.4. beta (r27392).

2. Switched usb port for the usb disc containing the bulk of the music files and removed on usb thing that I do not use. - I also checked that the usb disc did not need defragmentation.

3. Disabled MusicIP.

4. In CustomScan I set all custom scans to Manual, i.e. disabled them.

5. I considered to not cache the album artwork, but decided to keep the caching since I enjoy my album artwork. (My album artwork is often very large - 1400x1400 or so - and perhaps the resampling is taken a long time?)

6. Restarted the PC.

7. I changed my music folder from an unc path to a local path (but first I did the same for the shortcut in the music folder). That triggered a wipe and rescan.

The rescan is under way now and I will report the result when it has finished...

vagskal
2009-07-08, 14:16
This time the wipe and rescan took some 8 hours according to the progress report from SC below. A definite improvement over the previous 18 hours, but still a long time.

Mappsökning (99463 av 99463) Fullständig 04:02:32

Sökning i spellista (6 av 6) Fullständig 00:00:18

Slå ihop Blandade artister (6784 av 6784) Fullständig 00:03:29

Sökning efter albumomslag (7725 av 7725) Fullständig 03:46:37

Optimering av databas Fullständig 00:07:54

Sökningen av ditt musikbibliotek har slutförts.
Total tid: 08:00:50 (onsdag 8 juli 2009 / 21.45)

The scan really started at 11.41 (see the scanner log below), so the actual time to completion at 21.45 would be some 10 hours.

From the scanner log:
[09-07-08 11:41:51.3906] main::main (122) Starting SqueezeCenter scanner (v7.3.3, r27392, Sat Jul 4 03:01:50 PDT 2009)

vagskal
2009-07-09, 06:14
After changing the music folder in SC from an unc path to a local path something strange happened.

My set up now:
Music folder set in SC: I:\MP3\Min musik
In that folder there is a shortcut pointing to: H:\Mina CD (Flac)

I have checked those two folders and they do not contain any other shortcut.

The files in the two folders show up OK in SC, but in addition to that one album on \\MUSIKMASKINEN\Media (D)\Flac\Artister\Muriel Zoe\Neon Blue (2005)\ (notice the unc path) shows up in SC. The disc in question (D:\) is also a local disc.

How can this happen?

I have not listened to the songs on that album after the wipe and rescan, i.e. they could not have been pushed into SC by another application, like muso. And I have not browsed the Music folder in SC at all after the wipe and rescan, i.e. the files could not have been added through browsing. I know of no other way to enter files into the SC database.