PDA

View Full Version : Cross referencing 2 large music collections ?



swayzak
2016-09-11, 00:14
Hello

My main music collection died last night (on a WD USB hard drive - one accidental drop too many !).

I have a double back up of this (another USB HD & an external, AC-powered HD), but this has been done manually over the years so am uncertain as to how accurate the backups are.

Is there an easy way to cross reference these collections, to ensure the maximum number of folders/files are restored to the new drive (hopefully without any missing) ? e.g. to find which files (if any) are absent from either drive that are present on the other...

DJanGo
2016-09-11, 00:26
Is there an easy way to cross reference these collections, to ensure the maximum number of folders/files are restored to the new drive (hopefully without any missing) ? e.g. to find which files (if any) are absent from either drive that are present on the other...

Depends on your os your folderstructure and your skills.

If you rename/move files in one disk - you could expect doublette files.

So whats your os?

swayzak
2016-09-11, 00:32
Depends on your os your folderstructure and your skills.

If you rename/move files in one disk - you could expect doublette files.

So whats your os?

Win 10

kidstypike
2016-09-11, 01:02
Win 10

FreeFileSync http://www.freefilesync.org/ can copy any missing files from one folder to another, and vice versa, so you would finish up with both folders containing the same files.
Or you could just press the "Compare" button and sift through the list manually.

slate
2016-09-11, 01:03
When I update my backups i use "Total Commander"; menu "Commands" -> "Synchronize dirs"

I also use it at work when I have to compare source code trees ( files including content).

dasmueller
2016-09-11, 06:38
FreeFileSync http://www.freefilesync.org/ can copy any missing files from one folder to another, and vice versa, so you would finish up with both folders containing the same files.
Or you could just press the "Compare" button and sift through the list manually.


When I update my backups i use "Total Commander"; menu "Commands" -> "Synchronize dirs"

I also use it at work when I have to compare source code trees ( files including content).

Do these solutions take into account that both drives have the same files but tags may be different ?

garym
2016-09-11, 06:59
FreeFileSync http://www.freefilesync.org/ can copy any missing files from one folder to another, and vice versa,

I also use freefilesync to keep my backup drives synched.

regarding whether same file, different tags will be synched???? I use the comparison option of "compare file size and time". See link:
http://www.freefilesync.org/manual.php?topic=comparison-settings

If I've changed the tags in a file, it will change the file modification date (I use mp3tag for tagging; it has an option to either change or not change the file modification date and I allow it to change the modification date because I do want my backups to pickup any new version of a file even if all I did was edit the tags).

edit: In my own use case, I'm not doing two way synching as suggested to the OP. This would be needed initially to make sure files are same on BOTH backup drives. Once this is done however, I'd do one way mirroring. That is, I have the base HDD that I've made changes to, then I'd make sure that the backup drives contain the new/changed files. in freefilesync this is "synchronize MIRROR >>>" so that the HDD on the right side of screen is a mirror of the HDD on the left side of the screen (my base HDD).

kidstypike
2016-09-11, 08:04
edit: In my own use case, I'm not doing two way synching as suggested to the OP. This would be needed initially to make sure files are same on BOTH backup drives. Once this is done however, I'd do one way mirroring. That is, I have the base HDD that I've made changes to, then I'd make sure that the backup drives contain the new/changed files. in freefilesync this is "synchronize MIRROR >>>" so that the HDD on the right side of screen is a mirror of the HDD on the left side of the screen (my base HDD).

Yes, that's exactly how I make backups, from a master copy of all my music on my "L" drive, and only make changes to tags/files on this drive.

I suggested two way synching only as a means of getting all music files onto both drives.

garym
2016-09-11, 08:05
Yes, that's exactly how I make backups, from a master copy of all my music on my "L" drive, and only make changes to tags/files on this drive.

I suggested two way synching only as a means of getting all music files onto both drives.

agree! the two way synching is the answer to get both drives consistent with each other since neither is the "master".

d6jg
2016-09-12, 08:36
I'd suggest that the mirroring approach only fulfills half of the requirements for a good backup.
What happens if you accidentally delete some files on your main drive but don't notice for a period of time. A period of time during which the "deletion" has been written to your backup.

I use rsync from my main server to 2 others. One is local (i.e. in my house), the other is remote but I don't sync deletions. The upside is even if I accidentally delete a file its still on the backups. The downside is that when I replace an MP3 rip of a particular CD with a FLAC one I can, and do, end up with 2 copies of the CD on the backups.

Or does the backup software allow you to mirror without deletions ?

DJanGo
2016-09-12, 08:49
Do these solutions take into account that both drives have the same files but tags may be different ?

why not? unless you didnt change filestamps.

btw: xcopy /D /Y source:\ destination:\ or rsync is always a good solution

garym
2016-09-12, 08:56
I'd suggest that the mirroring approach only fulfills half of the requirements for a good backup.
What happens if you accidentally delete some files on your main drive but don't notice for a period of time. A period of time during which the "deletion" has been written to your backup.

I use rsync from my main server to 2 others. One is local (i.e. in my house), the other is remote but I don't sync deletions. The upside is even if I accidentally delete a file its still on the backups. The downside is that when I replace an MP3 rip of a particular CD with a FLAC one I can, and do, end up with 2 copies of the CD on the backups.

Or does the backup software allow you to mirror without deletions ?

excellent point and I agree. This is why all my backups are not created from a single "master". I didn't go into all that, but more specifically, I have 3 working systems (home, office, and weekend cottage). Each of these has a "master". Additions/deletions from these 3 masters are NOT done automatically. I manually add new files, remove duplicates, etc. The only comparisons I do are that when all 3 are manually updated with changes, I can check to see that the (1) total number of files/directories, (2) total number of tracks, and (3) total number of albums are all identical. Then at each of these 3 locations I have backup HDDs that I use freefilesync to keep in sync with the master from that location. At office and weekend cottage I only have one external backup HDD. At home (which I consider the "main" system) I have two USB HDDs containing synched copy of "home" master. The backups are stored not connected to power or a computer.

The downside of this is that I have to keep a temporary drive containing files I need to add to all 3 locations and keep a log reminding me what I need to do at each location to keep all up to date. This sounds way more complicated than it is. Just a few lines in a file, with some dates that things happened so I know the status. I suspect that I am not as good about keeping track of minor tagging tweaks across the 3 locations, but I do try.

The upside is that I have 3 parallel systems, all with backups, and the files are not cross-contaminating across the 3 sets. Accidental deletion at one location doesn't taint another location. And after adding new files, deleting files, etc. at any location, I always confirm that the totals mentioned above are consistent across the 3 locations. Again, all sounds complex, but really simple and something I just do on a regular basis (I add new rips, etc. (or re-rips in FLAC of old mp3 rips) about every week or two; I don't add files each time I rip a new CD and instead wait until I get a week or two's worth and then do all the updating.

DJanGo
2016-09-12, 09:12
I don't add files each time I rip a new CD and instead wait until I get a week or two's worth and then do all the updating.

I do listen to my new stuff asap. (Hard lesson learned)
But i rip them into a special folder, tag these and do a copy job into my library once i am finished.
eg. "sucking samplers" some spell artists different than others and i want to be sure that "my" artist matches the new files.

garym
2016-09-12, 09:44
Or does the backup software allow you to mirror without deletions ?

Yes, freefilesync has 3 options for files that are being deleted from the mirror drive: put them in recycle bin, permanently delete, or move files to a user defined folder.

dasmueller
2016-09-24, 09:25
Was playing w freefilesync earlier and ran into the following issue. It appears that it looks at and compares files within a folder but not the folders themselves. What I did was load a folder into the left side and then selected a destination location on the right. The folder did not previously exist in the location folder on the right. The files then appeared in the location on the right, but not the folder itself. I would guess I am missing something settings wise. If I want to update drive F to match drive K if drive K has folders that do not currently exist on drive F what settings should I choose ?

garym
2016-09-24, 09:29
It should create the missing (new) folder. Maybe you need to be choosing the parent directory just above the folder. For example if I have
flac/artist/album/tracks

i set c:/flac to be the directory synched.

I dont recall call any user settings related to folders, but I'm away from computer and can't check

kidstypike
2016-09-24, 09:54
Was playing w freefilesync earlier and ran into the following issue. It appears that it looks at and compares files within a folder but not the folders themselves. What I did was load a folder into the left side and then selected a destination location on the right. The folder did not previously exist in the location folder on the right. The files then appeared in the location on the right, but not the folder itself. I would guess I am missing something settings wise. If I want to update drive F to match drive K if drive K has folders that do not currently exist on drive F what settings should I choose ?

The contents (subfolders and files) of the folder on the left, will be copied to the folder you select on the right. Note the word "contents".

In your test example there were probably just files, no subfolders?

In the example below, the contents of the folder "Candy Dulfer" (files and sufolders) will be copied to the destination on the right.

dasmueller
2016-09-24, 11:44
You are correct, there were no subfolders. I think I understand better now what I need to do next time. Thanks !

dasmueller
2017-11-28, 09:58
Played w FreeFileSync again this AM. What I need to do is copy my music to a brand new HDD. I played w some newly created folders and used two way sync. It appeared to work Am I missing anything ?

I have 860 G to process any thoughts on time frame ? Should I do in batches ?

garym
2017-11-28, 11:44
Played w FreeFileSync again this AM. What I need to do is copy my music to a brand new HDD. I played w some newly created folders and used two way sync. It appeared to work Am I missing anything ?

I have 860 G to process any thoughts on time frame ? Should I do in batches ?

In my settings, I use the "synchronize Mirror __>" setting on the upper right side. That is, I want the target drive (right side) to be an exact copy of the source drive (left side). I prefer this over two way sync. With two way sync, something odd that I did with the target drive would then end up polluting the source drive. See pic. (EDIT, and I just noticed a more detailed discussion of two way vs one way mirror a few posts above that are useful to review.)

Also, I've done over 1TB in one batch. Shouldn't be an issue, just takes a while.
24092

dasmueller
2017-11-29, 07:04
Thanks gary

I'll probably run it overnight

garym
2017-11-29, 07:08
Thanks gary

I'll probably run it overnight

that's what I did. Good news is that once you have a backup drive, the updating with changes is very fast and automatic.

dasmueller
2017-11-29, 09:17
I have a backup already. The reason I am doing this is that my primary music folder is on a 1 TB HDD which has only about 30 G free space. backup is a 1.5 TB HDD so it has plenty of space. I bought a 2 TB Toshiba external from Amazon for $64 which I thought was a great price. The plan is to use the 2 TB as the primary and then take the 1 TB and convert everything to mp3 and give it to my son who only uses mp3s.

garym
2017-11-29, 10:23
I have a backup already. The reason I am doing this is that my primary music folder is on a 1 TB HDD which has only about 30 G free space. backup is a 1.5 TB HDD so it has plenty of space. I bought a 2 TB Toshiba external from Amazon for $64 which I thought was a great price. The plan is to use the 2 TB as the primary and then take the 1 TB and convert everything to mp3 and give it to my son who only uses mp3s.

good plan. amazing how cheap USB storage is getting. I recently bought some 4TB USB drives the size of a pack of cigarettes and they were pretty cheap as I recall.

dasmueller
2017-12-07, 07:47
I finally ran FreeFileSync yesterday. Took a little over 19 hrs. Odd thing is I went to each of the 2 folders, right clicked and selected properties. The total folders and files were different. I then went to FreeFileSync and hit the compare button. The screen flashed and if one looked quickly there was a temporary msg that all files were in sync.

Any thoughts ?

garym
2017-12-07, 07:59
I finally ran FreeFileSync yesterday. Took a little over 19 hrs. Odd thing is I went to each of the 2 folders, right clicked and selected properties. The total folders and files were different. I then went to FreeFileSync and hit the compare button. The screen flashed and if one looked quickly there was a temporary msg that all files were in sync.

Any thoughts ?

hmmm, something is wrong I'd say. When you compare the two drives after synching the two, you should have exactly the same number of files and folders (and even exact total bytes). One thing to check (and this happens to me sometimes). Make sure when you do properties that you are only selecting the top folder of the music files you synched and then select properties, and not the entire drive (which could have other files on it *other than* the music folders you synched.

for example, if you compare:

d:\musicfiles\

to

e:\

you may get different count that includes files outside of \musicfiles\ folder. So you want:

d:\musicfiles
e:\musicfiles

dasmueller
2017-12-07, 08:10
What you speak of Gary is what I did-just compared the 2 music folders
Here is a screenshot of the properties of the 2 folders-bytes are different as are file quantities

garym
2017-12-07, 08:39
Hmmm, I see you are comparing same folders, so that's good. Number of subfolders is exactly the same, but files different. That makes me nervous. When I do this, the files, folders, and bytes (not bytes on disc, but the first listing of total bytes) are all identical. I assume the drive with the greater number of files is your source disk and the smaller number of files is your new backup (target)? Not sure what's causing this. Is one of these a NAS? Are there hidden files somewhere not being transferred.

I recall you use foobar2000 as well. If so, open up foobar2000 and change your music library to the "tagged" folder of the original drive. Then in foobar2000 what does it tell you about the total number of music tracks? Then change your music library to the tagged folder of the new drive. What does foobar2000 tell you about total tracks? Are both the same or different?

dasmueller
2017-12-07, 08:49
You are correct, the smaller number is the target folder. I do not use a NAS and know of no hidden files. Foobar is loading now-will take awhile.

I did post the question in the FreeFileSync forum to see what thoughts folks there might have.

kidstypike
2017-12-07, 09:19
You are correct, the smaller number is the target folder. I do not use a NAS and know of no hidden files. Foobar is loading now-will take awhile.

I did post the question in the FreeFileSync forum to see what thoughts folks there might have.

I assume you are Mirroring the left side to the right side?

24159

If yes, try hitting the button (circled below) which reverses the sync direction, and then hit the compare button, this may show if there are files that are going to be deleted.
Do not hit the Synchronize button, and remember to switch back to how it was.

24160

DJanGo
2017-12-07, 09:20
no hidden files

a simple dir /AH /B /S k:\ could proove if you're right or wrong...

dasmueller
2017-12-07, 09:44
DJanGo;

Tried that nothing showed up. I did see at the bottom 0 of 43657 rows in both directions

dasmueller
2017-12-07, 09:54
https://www.freefilesync.org/faq.php#explorer-numbers-mismatch

here is what I got at FreeFileSync forum-not much help

slartibartfast
2017-12-07, 10:37
https://www.freefilesync.org/faq.php#explorer-numbers-mismatch

here is what I got at FreeFileSync forum-not much helpIt could be thumbs.db files filtered out by freefilesync. What do you see if you untick "Hide excluded items" ?

Sent from my SM-G900F using Tapatalk

garym
2017-12-07, 11:47
It could be thumbs.db files filtered out by freefilesync. What do you see if you untick "Hide excluded items" ?

Sent from my SM-G900F using Tapatalk

and search each drive's folder for *.db and see if files show up on one and not the other.

dasmueller
2017-12-07, 12:39
It could be thumbs.db files filtered out by freefilesync. What do you see if you untick "Hide excluded items" ?

Sent from my SM-G900F using Tapatalk

When I removed thumbs from the filters thumbs showed on the source drive


and search each drive's folder for *.db and see if files show up on one and not the other.

entering*.db showed nothing

both folders are set to show hidden files

garym
2017-12-07, 12:44
did you ever get the foobar2000 counts?

dasmueller
2017-12-07, 13:00
did not see anywhere that had a number of files so I highlighted everything and right clicked, selected properties both folders showed 934915307406 bytes

garym
2017-12-07, 13:02
did not see anywhere that had a number of files so I highlighted everything and right clicked, selected properties both folders showed 934915307406 bytes

that's a good sign. Can you add all your files from a drive to a single playlist (I think just right click and add). And then the playlist shows you how many tracks? I use facets, so it looks a bit different than basic foobar.

slartibartfast
2017-12-07, 13:19
When I removed thumbs from the filters thumbs showed on the source drive



entering*.db showed nothing

both folders are set to show hidden filesIf I search for *.db in my Music folder I find hundreds of files. Are you sure you are searching correctly as turning off the filter reveals them to be there.

Sent from my SM-G900F using Tapatalk

dasmueller
2017-12-07, 13:31
gary, that's what I was describing above-it does not show # of files

dasmueller
2017-12-07, 13:39
If I search for *.db in my Music folder I find hundreds of files. Are you sure you are searching correctly as turning off the filter reveals them to be there.

Sent from my SM-G900F using Tapatalk

I typed *.db in the search filter area top right

slartibartfast
2017-12-07, 13:43
I typed *.db in the search filter area top rightOK but you did say that thumbs showed up in freefilesync so they must be there.

Sent from my SM-G900F using Tapatalk

dasmueller
2017-12-07, 14:15
OK but you did say that thumbs showed up in freefilesync so they must be there.

Sent from my SM-G900F using Tapatalk

The why can't I see them in the folder itself ?

and there are still 451 files unaccounted for

slartibartfast
2017-12-07, 14:21
The why can't I see them in the folder itself ?

and there are still 451 files unaccounted forAre you showing extensions for files? I don't know but maybe if extensions are hidden then a search will not find them. If you just open an album folder is there a thumbs file in there?

Sent from my SM-G900F using Tapatalk
Or search for "thumbs" instead of *.db

dasmueller
2017-12-07, 14:29
OK, if I go into folder options and unclick-Hide protected operating system files the Thumbs are then visible.

That leaves 2 questions

1. why do some folders have them and others not ?
2. how to find what and where are the other 451 extra files in the source folder

slartibartfast
2017-12-07, 14:32
OK, if I ggo into folder options and unclick-Hide protected operating system files the Thumbs are then visible.

That leaves 2 questions

1. why do some folders have them and others not ?
2. how to find what and where are the other 451 extra files in the source folder

When you untick "Hide excluded files" in freefilesync at the bottom of the page you should see all the extra files in the source folder.

Sent from my SM-G900F using Tapatalk

dasmueller
2017-12-07, 14:49
When you untick "Hide excluded files" in freefilesync at the bottom of the page you should see all the extra files in the source folder.

Sent from my SM-G900F using Tapatalk

I'm running version 8.4 and do not see the box you refer to

garym
2017-12-07, 15:01
I'm running version 8.4 and do not see the box you refer to

At the top of page, between the "compare" box and the "synchronize" box, there is a graphic of a filter. Click the filter and you'll see options (3 different tabs of options)

slartibartfast
2017-12-07, 15:05
I'm running version 8.4 and do not see the box you refer toI'm on 5.15. Just updated to 9.6 and I cannot see it either. Seems to have been removed. You could clear the filter using the drop down menu next to the red funnel at the top of the page.

Sent from my SM-G900F using Tapatalk

dasmueller
2017-12-07, 15:19
I'm on 5.15. Just updated to 9.6 and I cannot see it either. Seems to have been removed. You could clear the filter using the drop down menu next to the red funnel at the top of the page.

Sent from my SM-G900F using Tapatalk

I did that for the Thumbs

the others listed are
\System Volume Information\
\$Recycle.Bin\
\RECYCLER\
\RECYCLED\
*\desktop.ini

are you suggesting I delete those also ?

garym
2017-12-07, 15:25
I did that for the Thumbs

the others listed are
\System Volume Information\
\$Recycle.Bin\
\RECYCLER\
\RECYCLED\
*\desktop.ini

are you suggesting I delete those also ?

I have nothing in my exclusions settings. then again, I only use this for my music files folder, so none of those directories should even be there for me.

slartibartfast
2017-12-07, 16:07
I did that for the Thumbs

the others listed are
\System Volume Information\
\$Recycle.Bin\
\RECYCLER\
\RECYCLED\
*\desktop.ini

are you suggesting I delete those also ?

No maybe just desktop.ini. I have some of those in my music folder but only 19.

Sent from my SM-G900F using Tapatalk

slartibartfast
2017-12-07, 16:09
I have nothing in my exclusions settings. then again, I only use this for my music files folder, so none of those directories should even be there for me.Like garym said the other exclusions are not relevant if you only scan the Music folder

Sent from my SM-G900F using Tapatalk