PDA

View Full Version : Backing up music library



gutted
2007-10-22, 07:39
I currently use MS Backup, and this weekend ran a manual backup after upgrading a few drives for larger capacity. The backup time is crazy (it took well over 24 hours to backup and verify around 240GB of data) so I was wondering if anyone can offer some advice...

I currently use "normal" backup option. There are several different types (differential, incremental etc). I chose "normal" because I wanted to make sure that every change gets backed up - even if I modify FLAC tags, but maintain the last update timestamp (which is my preferred way to go).

I wasn't sure if MS backup is clever enough to know that the file has changed even if the last update timestamp remains unchanged. So to be safe, I just chose "normal" backup.

So basically - is MS backup clever enough to detect differences in files (e.g. tag updates) even if the last update timestamp is unchanged? If so, what is the best backup option to use - differential, maybe?
If MS backup won't be able to detect that change, is there any other backup utility I can use instead, and that will detect updates to tags?

Cheers!
Dan.

4mula1
2007-10-22, 07:46
Check out SyncBack. I'm using it and it can detect changed files. Works like a champ for me.

There is a freeware version and a $$$ version. The paid version obviously is a bit more powerful but the free version will do the trick.

gutted
2007-10-22, 08:37
Cool - thanks dude.

And this can detect changes to tags, even if I've configured MP3tag to maintain the last modified timestamp? If so, then it's a winner :)

4mula1
2007-10-22, 08:40
Cool - thanks dude.

And this can detect changes to tags, even if I've configured MP3tag to maintain the last modified timestamp? If so, then it's a winner :)

That I can't answer. I use it to back up my pictures and wife's itunes collection, so I've pretty much relied on it using timestamps. I run SlimServer on Solaris, so I use rsync to do the backups of my music.

autopilot
2007-10-22, 08:51
I have tried so many different options for backing up my files, including writing complicated batch scripts for Robocopy, but in the end i found Synctoy to be the best. It's free, simple and from microsoft - http://www.microsoft.com/windowsxp/using/digitalphotography/prophoto/synctoy.mspx

I use 'echo' mode most often.

gutted
2007-10-22, 08:52
Hmmm. Yeah, that's the problem. I configured MP3tag to maintain the last modified timestamp... The reason for this is so that I can easily see which albums were recently ripped to FLAC and which ones are still MP3 just by looking through Windows Explorer.

If the timestamp changes when I modify tags, then it makes it tricky to tell which albums have recently been re-ripped to FLAC (I might have changed the tags in some of my MP3s, so then it would look like these albums were recently added).

There are ways round this, and I can just adopt a different system - but if I can find a backup tool that is clever enough to detect changes to tags, then that would be the ideal solution...

autopilot
2007-10-22, 09:01
Hmmm. Yeah, that's the problem. I configured MP3tag to maintain the last modified timestamp... The reason for this is so that I can easily see which albums were recently ripped to FLAC and which ones are still MP3 just by looking through Windows Explorer.

If the timestamp changes when I modify tags, then it makes it tricky to tell which albums have recently been re-ripped to FLAC (I might have changed the tags in some of my MP3s, so then it would look like these albums were recently added).

There are ways round this, and I can just adopt a different system - but if I can find a backup tool that is clever enough to detect changes to tags, then that would be the ideal solution...

Synctoy will detect changes to the tags, even if the time stamp has not changed. You just need to select the 'check file contents' option. Takes a lot longer but works perfectly.

gutted
2007-10-22, 09:32
Nice one - thanks Auto Pilot: I'll give that a try. Even if it takes a lot longer, I guess that it surely must be quicker to run a differential backup using Synctoy than running a complete backup each time... I'll post back with an update :)

EDIT: Bummer. Looks like it is not designed to run on Win2k. My fileserver is still on Win2k OS :( Will google to see if it still actually works on Win2k even if it's supposed to run on XP or above...

Skittler
2007-10-22, 09:45
Be careful with "Echo". If you delete a load of files accidentally then Synctoy will happily delete them from your backup too!

At least I think so.

You may be better off with "Contribute" and then if you really do mean to delete stuff, do it manually from your backup too.

iPhone
2007-10-22, 12:21
I currently use MS Backup, and this weekend ran a manual backup after upgrading a few drives for larger capacity. Cheers! Dan.

Hello Dan,

I have two possible suggestions and one comment. This is for anybody backing up a single drive with compression on the same drive or to a secondary drive: beware of bad media, bad sectors, and bad clusters. Any data written to them or any area that becomes bad is lost forever. And if the backup area or drive gets them it is a useless backup.

Suggestion one is to get a NAS and run Raid 5 or X-Raid. No more backups to schedule or sit through, as itís all automatic. If cost is an issue (suggestion two), I recommend doing uncompressed direct copying of FLAC files to DL DVD. Doing it this way you never have to waste time doing another backup again. If a drive or section of drive goes bad you just replace the songs you are missing from the archive DVDs. First one does a complete copying of all their songs to DL DVD. Then when one rips more songs to FLAC (or whatever format ones song files are in) and has 8 Gigs worth copy them to a DL DVD. Now one has a permanent archive and no longer needs to perform backups. It is best to keep track of the new files on the DL DVDs in a simple database or at least in a spreadsheet. I have mine in a database with each DL DVD having a unique number assigned to it. My neighbor is using Excel to track his. Depending on ones system, it can be hard to listen to tunes when a backup is taking place. Once one converts to a system similar to this and all the data and DL DVDs are current it is simple just to rip new CDs to ones preferred format then spend the rest of the time enjoying tunes until it is time to archive 8 gigs of new tunes then back to the music. Most people will have great piece of mind with either one of these systems.

Personally, I am a secure data fool now when it comes to my FLAC Files so I am doing both. I am running a 3TB ReadyNAS NV+ in X-Raid format (auto backup all data, spans data, allows for swapping out drives to larger as they become available, data safe from any one full drive failure) plus I have all my FLAC Files archived to DL DVD.

Just some thoughts from somebody that has had to rip everything three time. I had everything in MP3 to save space (was only listening to them in the car and MP3 player). Found Squeezebox and ripped them all again to FLAC to play on the main system and through out the house. Was backing up to a second drive and had a main drive complete failure, bought new drive and went to put the backup on it only to find out the backup drive had bad sector and clusters! Bought a ReadyNAS NV+ and ripped them again as well as archived on DL DVD, and knock on wood, have never had any issue since switching to this system two years ago. My only hope is that my bad luck can help some else avoid all the ripping I have done by changing the way they approach protecting all the time they have invested in ripping a large CD library.

radish
2007-10-22, 13:08
RAID is not a backup solution, nor is it a valid replacement for a backup solution. RAID serves two main purposes - improved performance (striping) and reduced downtime (fault tolerance/hotswap). Unless you're running a radio station from slimserver (how cool would that be?!) I don't think either of those criteria are particularly important for typical slim usage.

Backing up to DVD (or something similar) is a good idea, but don't forget to (a) test the backups every so often - discs don't last forever and (b) keep them somewhere else.

egd
2007-10-22, 14:16
First off, I don't think there are necessarily any easy solutions to this unless you have a relatively small library. I'll deal with small and larger libraries separately because I think they warrant different approaches.

Small libraries
If your library fits on a single drive I'd simply make a 1:1 copy to a second drive that is hosted externally from your PC and only accessed for the purposes of making/ restoring a backup. There are various tools that can automate this and use compare files or rely on date/timestamps/archive attribute etc. If you're really paranoid, make two copies on two external drives.

Larger libraries
This is where things begin to get a little more complex and/ or expensive depending on your approach and appetite for spending. In general, I would suggest the easiest way of dealing with a large library that spans say a Terabyte, is to have an external RAID enclosure that serves the same purpose I discussed above for small libraries. For the really paranoid this approach can be further augmented by the addition of a tape-based backup solution, ideally LTO2 or LTO3 - but this is where things take an expensive turn.

My FLAC library is housed on a NAS and duplicated to an external RAID5 enclosure that is only powered up for the purposes of backup - yep, it's not automated, but the fact is my library is not that dynamic either. The RAID enclosure is in turn backed up to LTO2 tape using an incremental backup strategy. If I was to do this again I might consider backing up to another external RAID enclosure rather than to tape - it would be a lot faster. The advantage of tape over DASD is that it can grow with your needs whereas you could conceivably outgrow a RAID configuration and expanding a RAID array is not a risk free exercise in and of itself.

An alternative approach - separate metadata from audio
Whilst I can't point to a tool that can do this I remember reading somewhere (in this forum I believe) of an approach that involved separating backup of metadata from the actual audio, with the underlying thinking being that the audio stream seldom changes whereas the associated metadata is often times updated, augmented etc. This would in my view be an ideal approach insofar as my incremental backups would shrink tremendously in size.

Conclusion
There are as many backup strategies and tools out there as you can poke a stick at - all of which will generally culminate in disk or tape backup, or perhaps even blu-ray in time (for those who like watching paint dry). The important thing, irrespective of selected strategy, is to ensure you incorporate redundancy, that you are organised to keep backups current, and design it such that you eliminate single points of failure. Offsite storage of your backups is generally a good thing.

Wirrunna
2007-10-22, 16:12
Autopilot, thank you for the link to SyncToy.
My slimserver PC runs 4 x 320Gb disks in Raid5 and I backup onto a clunky old PC with a single WD 750Gb drive. The Backup PC is only run once every few weeks for backup.
I just downloaded SyncToy and ran it rather than the filecopy I have done in the past. Results - SyncToy ran in about 15 minutes compared to 6 hours, it reported 33,220 files that did not require action, and 67 new files and 5 new folders - the CDs added to my library since the last backup.
Thanks again for the tip.

ob_kook
2007-10-22, 18:06
RAID is not a backup solution, nor is it a valid replacement for a backup solution. RAID serves two main purposes - improved performance (striping) and reduced downtime (fault tolerance/hotswap). Unless you're running a radio station from slimserver (how cool would that be?!) I don't think either of those criteria are particularly important for typical slim usage.

Backing up to DVD (or something similar) is a good idea, but don't forget to (a) test the backups every so often - discs don't last forever and (b) keep them somewhere else.

Radish is correct that you also need a backup solution. Allow me to expand on the point about RAID. There are many kinds of RAID: 0,1,5, 10 are the most popular. RAID itself stands for Redundant Array if Independent Disks.

RAID0, or striping, as he mentioned, is used to improve performance, but offers no high availability - if you lose a single disk, you've lost your entire LUN.

RAID1 is a mirror done at the disk level. This will halve your available disk, but give you a continuous synchronous copy. There is a bit of a performance hit on writes since you have to write everything twice.

RAID5 creates a parity disk. Basically, you can string together 4 disks, lose about 20% capacity, and be able to withstand the loss of 1 disk without losing data. There is a nominal read performance gain, and a definite write performance hit (the overhead of writing to multiple places).

RAID10 first mirrors the disks, then stripes them for speed. downside is losing 1/2 your capacity.

Just remember 2 things:
1. it is not a question of IF your disk fails, it is a matter of WHEN. These are spinning physical media. They will absolutely fail over time.

2. HDD do not like heat. If you are packing a bunch of drives in a tight enclosure, make sure to provide enough space between them and adequate air flow across them. Hot disks = shorter life. It is not difficult to cook your disk.

st2lemans
2007-10-22, 23:32
I keep a copy on a 500MB USB drive, a copy at work, and a copy at my
girlfriend's (using the USB drive to transport changes).

Tom

kjg
2007-10-23, 00:03
4mula1 wrote:
> gutted;237084 Wrote:
>> Cool - thanks dude.
>>
>> And this can detect changes to tags, even if I've configured MP3tag to
>> maintain the last modified timestamp? If so, then it's a winner :)
>
> That I can't answer. I use it to back up my pictures and wife's itunes
> collection, so I've pretty much relied on it using timestamps. I run
> SlimServer on Solaris, so I use rsync to do the backups of my music.
>
>

According to the documentation, SyncBack uses (can use) the following to determine if the source and destination files are different.

1. Last file modification date & time: All files record the date & time of when they were last changed.

2. File size: All files record the number of bytes they contain.

3. Hash value: A unique value can be computed based on the contents of a file. These values can be used to check if a files contents is the same as another's.

4. File attributes: Files have special attributes, e.g. read-only, hidden, etc., and SyncBack can optionally check for changes in these attributes

Using the hash value (#3) will obviously take longer, but it should be completely reliable. I use SyncBack and it's an excellent program all around.

Schindler
2007-10-23, 14:19
I use Allway Sync to backup to an external HD. Its great, you can even have several profiles... If you use it too often you have to buy it. It also detects changes in tags...

http://allwaysync.com/

christian

norderney
2008-03-22, 10:27
I store my music on a 750GB hard drive. I have about 21,000 FLAC tracks and 9,000 MP3s 320kbps VBR.

I bought my SB1 in November 2004 and recently replaced it with a Transporter. Having spent hours and hours ripping CDs and tagging, I wanted a backup system that would be as fool poof as possible. So over the last 3.5 years I now have 3 extra 750GB HDDs, which I use for backup. 1 of these disks I keep off site.

Is this overkill paranoia or sensible backp planning?


I use SyncBack to manage the backup process. When I run SyncBack it first gives me a list of all new/modified files it is going to copy and a list of files it is going to delete. If I agree with this list I press the continue button.

I also run FLACTEST regularly on my main music HDD and my 3 backup HDDs as well.

Bizarroterl
2008-03-22, 10:48
I've used SyncToy and AllwaySync. Both are comparable, though everywhere I've seen them compared AllwaySync came out on top. The free version of AllwaySync has a limit on the number of files it will sync each month (it tells you when you hit this limit). The paid version doesn't have this limitation.

One thing you should be aware of with SyncToy - It has a limit to the number of files it will sync reliably, IIRC about 50,000. Go over that amount and it doesn't sync reliably and it will not indicate this is happening. You could use it for years and then, when you needed to recover file, find out not everything was saved.

To determine if this is happening:
Run a full synchronization.
Run a Preview and see if any files are out of sync. If they are then you can split your job into smaller batches or pony up the $20 for AllwaySync.

JJZolx
2008-03-22, 11:10
I store my music on a 750GB hard drive. I have about 21,000 FLAC tracks and 9,000 MP3s 320kbps VBR.

I bought my SB1 in November 2004 and recently replaced it with a Transporter. Having spent hours and hours ripping CDs and tagging, I wanted a backup system that would be as fool poof as possible. So over the last 3.5 years I now have 3 extra 750GB HDDs, which I use for backup. 1 of these disks I keep off site.

Is this overkill paranoia or sensible backp planning?
I know a guy that washes his hands about 50 times a day. Overkill or sensible?


I use SyncBack to manage the backup process. When I run SyncBack it first gives me a list of all new/modified files it is going to copy and a list of files it is going to delete. If I agree with this list I press the continue button.

I also run FLACTEST regularly on my main music HDD and my 3 backup HDDs as well.

The question remains whether the backup software can detect changes to files where only some tag data has been modified. If you don't maintain file mod times when you change tags then it's not something you need to worry about. But if you do, since Flac tag blocks are written with some padding to allow tags to be modified in place, the file size as well as the date last modified usually remain the same.

norderney
2008-03-22, 11:38
I've used SyncToy and AllwaySync. Both are comparable, though everywhere I've seen them compared AllwaySync came out on top. The free version of AllwaySync has a limit on the number of files it will sync each month (it tells you when you hit this limit). The paid version doesn't have this limitation.

One thing you should be aware of with SyncToy - It has a limit to the number of files it will sync reliably, IIRC about 50,000. Go over that amount and it doesn't sync reliably and it will not indicate this is happening. You could use it for years and then, when you needed to recover file, find out not everything was saved.

To determine if this is happening:
Run a full synchronization.
Run a Preview and see if any files are out of sync. If they are then you can split your job into smaller batches or pony up the $20 for AllwaySync.

I use SyncBack not SyncToy. SyncBack does seem to work ok for me. Also SyncBack does detect changes to files where only some tag data has been modified

808
2008-03-23, 04:22
I've got a cron job running rysnc daily from my ClarkConnect drive to a external USB, there is also a pearl script checking a couple times a day if my laptop is ping-able, if yes it then runs a My Documents rsync back to the box. cwRsync Server runs on the laptop.

I'm not sure what would happen if the laptop turned off mid-sync. Any ideas ?

Jeff Flowerday
2008-03-23, 05:47
A second HD and robocopy. Vista now included robocopy.

robopy m:\music b:\music * /E /COPYALL /PURGE

Using 2 eSata drives I can backup a full 600+ GB in approx 2 hrs. After that it only gets quicker by applying what's changed to the second directory not another full backup.