PDA

View Full Version : Monitoring Music Database



norderney
2007-01-27, 18:26
I currently have 23832 songs by 4631 artists stored on my 750GB hard drive. I have invested in 2 additional 750Gb hard drives which I use for backuping up my music database. I use an application called SyncBackSE V4.1.1.0 to backup changes to my database to both backup hard drives.

Just wondering if there are any utilities I should run on a regular basis to check all my music files are ok on both the live drive and the 2 backup drives?

As with any computer system, files can become corrupted at any time. Just worried that if any of my files on the backup drive became corrupted I may not find out about it until I need to restore them to the live drive! Or am I being too paranoid?

It took me many months to get all my music ripped to the hard drive, and I do not want to have to do it again - ever!!!

Andrew Smith

mherger
2007-01-28, 00:22
> Just wondering if there are any utilities I should run on a regular
> basis to check all my music files are ok on both the live drive and the
> 2 backup drives?

I don't know about a utility. Probably DOS' fc or something like diff?
More important seems to be not to keep one single copy, but more than one.
What happens if you accidentally delete/alter a file and sync it before
you notice? The copy will be the same - deleted or altered.

--

Michael

-----------------------------------------------------------------
http://www.herger.net/SlimCD - your SlimServer on a CD
http://www.herger.net/slim - AlbumReview, Biography, MusicInfoSCR

norderney
2007-01-28, 01:31
[QUOTE=Michael Herger;174929]> More important seems to be not to keep one single copy, but more than one.
What happens if you accidentally delete/alter a file and sync it before
you notice? The copy will be the same - deleted or altered.


That's why I have 2 backup hard drives - so that I have 2 separate copies of my database. It also means I can keep one of them off site.

erland
2007-01-28, 02:03
Just worried that if any of my files on the backup drive became corrupted I may not find out about it until I need to restore them to the live drive! Or am I being too paranoid?No you are not being paranoid, I started to think that this could actually happen in my own setup. I have automatic backups setup that is written to 2 different harddrives. There is one full backup and incremental backups runs every week. And one copy of the backup on both harddrives.

As Michael says, the first thing to make sure is that you have more than one backup and these are stored on different hard drives. But since you have ordered two drives it sounds like you have already thought about this. The best solution is also if you are able to store these drives at different places. Two hard drives within the same computer could easily both crash if you get some heat problems. Two drives within the same building could easily both be destroyed in case of a fire in the house, but handling that might be starting to be a bit paranoid. Its also a good idea to keep several copies of a changed file, so you can go back to a previous version.

What file format are you using for your music ?

FLAC and MP3 are compressed formats so they probably have some internal checksum which you should be able to check against. The flac tool on Linux has a -t flag that does some test, I'm not sure its the correct type of test though. The mp3-decoder tool on Linux also have a -t flag that does some testing. I guess both these probably fails if run on a corrupt file.

Another solution is to just store MD5 checksum of all music files and compare these during the backup process. It would report errors for files with changed checksums, this could be intentionally changed files but also modified files.

I don't know if there are any solutions ready to use around these tools of if you would have to write the scripts around them yourself.


It took me many months to get all my music ripped to the hard drive, and I do not want to have to do it again - ever!!!It took me a about a week for my pretty small collection of about 3000 tracks, but I still don't want to do it again so I can see your point.

Hard drives do crash, its just a matter of time. I have already had two crashes, for one I didn't have any backup but fortunately the drive didn't contain any important stuff. The other one did have a lot of important stuff and I hadn't any backup on all digital photos because I thought it took to much space. Fortunately the drive didn't totally crash so I was able to get back everything from either the drive or the backups I had. After this I started to do backups on everything and I also started to do backups to two different drives.

Wonko
2007-01-28, 02:55
I'm not aware of any tools that can 100% check for corruption inside the mp3/flac/whatever files - if the tags got deleted somehow on the 'live' disk then you could have a perfectly valid and readable but damaged file.

erland's suggestion of using md5s should work well. The problem for me comes when 'updating' the backup as I tend to make quite a few changes such as tidying capitalisation of tags, adding album art, applying replaygain, etc. - the difficulty is knowing whether I'm about to overwrite my uncorrupted backup file with an updated corrupted one.

I happen to use syncback (freeware version) too - one of the things I like about it is the preview of changes it is about to make after doing the comparison of the master to the backup drive. I use the 'slower but more reliable' file comparison mode (under 'compare options' in the earlier version I use) and then carefully check the preview list. This generates and compares hashes for the source and backup files, and so should (a) check all files are readable and (b) detect any differences. If you review the differences carefully for any unexpected changes it could be a fairly safe scheme.

As well as having good backups, it's obviously better to try to avoid the HDD failures in the first place. If you haven't already, I suggest you make sure all the HDDs are cooled as much as possible as heat can lead to premature failure. This can be a bigger problem for external drives.

peter
2007-01-28, 03:18
Wonko wrote:
> I'm not aware of any tools that can 100% check for corruption inside the
> mp3/flac/whatever files - if the tags got deleted somehow on the 'live'
> disk then you could have a perfectly valid and readable but damaged
> file.
>
> erland's suggestion of using md5s should work well. The problem for me
> comes when 'updating' the backup as I tend to make quite a few changes
> such as tidying capitalisation of tags, adding album art, applying
> replaygain, etc. - the difficulty is knowing whether I'm about to
> overwrite my uncorrupted backup file with an updated corrupted one.
>
> I happen to use syncback (freeware version) too - one of the things I
> like about it is the preview of changes it is about to make after doing
> the comparison of the master to the backup drive. I use the 'slower but
> more reliable' file comparison mode (under 'compare options' in the
> earlier version I use) and then carefully check the preview list. This
> generates and compares hashes for the source and backup files, and so
> should (a) check all files are readable and (b) detect any differences.
> If you review the differences carefully for any unexpected changes it
> could be a fairly safe scheme.
>

Even that scheme is somewhat risky. What if your tag editing tool
corrupts your music while writing the tags? Or the OS/Hard Disk is
unstable and messes up anythng (re)written. You'd end up with thrash
anyway. And you have to check your backup change lists every time... :(

I guess backup up all new files to DVD as well would be a wise
precaution in addition to any automatic backup/snapshot system.
Especially for real important stuff like pictures and movies. Problem
is, I'm too lazy... :(

Regards,
Peter

Wonko
2007-01-28, 03:51
Hi Peter - I'd completely agree, and would suspect that no-one has a completely 'risk free' approach. Risk is a relative thing, and in engineering terms is normally defined as the combination of the likelihood of a 'loss' with the severity of the consequences. It's all about choosing a level of risk that you can live with given the importance of the data and the 'cost' (in time and money) of doing the backups.

If my tag editor messed up and corrupted some tracks, which then got written to a backup before I had noticed the corruption, then I've decided I could live with that (I might end up re-ripping a couple of CDs). Like most people, I suspect the thought of losing an entire collection is less palatable.

This is what I was trying to suggest (perhaps badly) in my last post - that checking the integrity of an existing backup is only part of the problem, and that checking that you're not overwriting a valid backup with corrupted data is probably harder to deal with. This is one of the reasons for doing multiple / incremental backups, but then you need to decide how many copies you need to keep over what period of time...

For irreplaceable personal files and photos, etc. I do periodic backups to DVD (kept offsite) as well as more frequent copying between PCs and to external HDD. Putting all my music and other 'stuff' onto DVD would just take too long to do, and even then I'm not sure that a single DVD would be reliable enough!
BRs

joek
2007-01-28, 06:05
I've written a checksum .bat program (5-10 lines with comments) that uses fsum and can write checksums recursively for each file in my directories of choice.

I usually use it when moving or copying important files. I then verify the files after the move or copy with their checksum.

I also use it when I write DVD's, so that I could easily check the integrity of the bits over time.

It may take some time with 100's of GB's in your case, but so what. You don't have to sit there and watch it.

BTW, a good sync application should take data integrity into consideration when transferring data. I use DeltaCopy and have never had an issue. DeltaCopy is a free one-direction sync based on rsync. There is also other free apps like Unison (bi-direction) sync and jfilesync.

Joe