PDA

View Full Version : How do YOU backup your music?



autopilot
2006-03-16, 10:09
I currently have 2 HDD's in my PC. I have 2 large important folder trees, one with my music and on with my photo's. Currently i am manually copying everything between drives, but this is becoming a bit of a chore and often i forget.

So what software people here use and would recommend for me?

I don't want anything fancy, i.e. compression is needed as obviously its already compressed. Just a simple solution with folder syncing and preset times (over night).

Oh, and free would be nice too :D

Steven Moore
2006-03-16, 10:25
As I scan a cd into itunes I keep a copy of it in a folder, when the folder reaches 4.4gb I burn it to dvd.

fuzzyT
2006-03-16, 10:27
dangerous_dom wrote:
> So what software people here use and would recommend for me?

What OS are you running?

For Linux, look into rsync.

For Windows look into robocopy.

Either can be scripted and scripts scheduled.

--rt

jonheal
2006-03-16, 10:29
I've got my music on a drive connected to a Linksys NSLU2. Once a week, the NSLU2 copied the entire drive to the other identical drive that is connected to it.

kjg
2006-03-16, 10:31
Folks had been touting a tool from Microsoft called SyncToy which I
tried, but for me was very unreliable. I found a simple, free program on
the net called ICE Mirror that does the trick for me. Just point it to a
source and destination directory on any drive and it'll take care of
mirroring the files exactly. You have to start it up manually, but it
takes care of all of the sync'ing chores for you, including removing
files on the destination that have been renamed on the source, etc. You
can also run more than one instance if you want to sync multiple
directories simultaneously. Nice tool for a freebie.

http://www.ice-graphics.com/ICEMirror/IndexE.html

dangerous_dom wrote:
> I currently have 2 HDD's in my PC. I have 2 large important folder
> trees, one with my music and on with my photo's. Currently i am
> manually copying everything between drives, but this is becoming a bit
> of a chore and often i forget.
>
> So what software people here use and would recommend for me?
>
> I don't want anything fancy, i.e. compression is needed as obviously
> its already compressed. Just a simple solution with folder syncing and
> preset times (over night).
>
> Oh, and free would be nice too :D
>
>
>

autopilot
2006-03-16, 10:31
Sorry, running XP. Don't need CD/DVD buring either.

bklaas
2006-03-16, 10:34
All new music goes into a folder called new_additions. Periodically I burn this to CD or DVD, at which point I parse the new_additions content to appropriate folders.

I also have a 2HD system that I backup daily from one to the other. I use rsnapshot (http://rsnapshot.org), which is a great backup utility built on rsync for *nix operating systems. Given you didn't mention your OS, I'm assuming you are on Windows, so this won't be available to you. :(

cheers,
#!/ben

twylie
2006-03-16, 10:51
I, too got tired of backing up to external drives and wasn't happy about having to keep burning plastic. It wasn't the music so much as a video archiving project that drove me over the edge - 12GB per hour of video in raw format. About 3 years ago, I sucked it up and built a dedicated server with hardware RAID. It has been running 6 * 250GB drives in RAID5 with hotspare for 1TB of storage. A year later, I built a second server with 1.75TB in RAID5 using 8 * 250GB drives. Both servers have been runnign 24x7 in my basement since they were built. I use RDP to control and admin each of them. Although this gave me some improvement over losing a single hard drive, all my digital financial records, pictures, and source videos were loaded on a 300GB USB drive I sent to my parents house in another state. I update that drive about once a quarter with them.

jmpage2
2006-03-16, 10:52
All of my music and other data is backed up from a server with mirrored hard disks to an external hard drive in another part of the house.

I run M$FT robocopy utility to synch the data nightly.

If I was really anal retentive I would buy a 2nd external enclosure and keep a rotated backup in my fire safe or periodically send it to Aunt Glenda in the Ozarks for safe keeping.

mherger
2006-03-16, 10:59
> I also have a 2HD system that I backup daily from one to the other. I
> use rsnapshot (http://rsnapshot.org), which is a great backup utility
> built on rsync for *nix operating systems. Given you didn't mention
> your OS, I'm assuming you are on Windows, so this won't be available to
> you. :(

Add my vote for rsnapshot - I even use it to backup Windows machines to my
Linux server (with cwRsync on the Windows side). Not as effective as with
Linux only systems, but still great to have a history of your files
available.

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
StringEditor Plugin (http://www.herger.net/slim/)

Heuer
2006-03-16, 11:20
Norton Ghost to a second HD. Does scheduled incremental backups so saves on disk space. You can also use a USB drive.

sfraser
2006-03-16, 11:45
I Set up a file server in the basement with 1.6 TB of disk space, of which 900GB is on a RAID5 system. Music and other important docs goes on the RAID file system. Also in the process of setting up a 100 mbps ethernet network with my neighbor (squeezebox owner as well) he wants to use my slimserver, instead of duplicating my setup. When our houses are networked, I think I will setup a NAS at his house for some offsite storage. It cost a bit more than I liked, but it has taken me way to much time to burn my entire CD catalog, just to lose it "WHEN" a hard drive fails.

Cheers,

sfraser
2006-03-16, 11:47
I, too got tired of backing up to external drives and wasn't happy about having to keep burning plastic. It wasn't the music so much as a video archiving project that drove me over the edge - 12GB per hour of video in raw format. About 3 years ago, I sucked it up and built a dedicated server with hardware RAID. It has been running 6 * 250GB drives in RAID5 with hotspare for 1TB of storage. A year later, I built a second server with 1.75TB in RAID5 using 8 * 250GB drives. Both servers have been runnign 24x7 in my basement since they were built. I use RDP to control and admin each of them. Although this gave me some improvement over losing a single hard drive, all my digital financial records, pictures, and source videos were loaded on a 300GB USB drive I sent to my parents house in another state. I update that drive about once a quarter with them.

I have been doing something similar, have you ran into RDP issues with certain microsoft security patchs? I ended up un-installing a few patchs because they kept on causing RDp encryption errors.

Scott

kefa
2006-03-16, 12:01
dangerous_dom wrote:
> So what software people here use and would recommend for me?

What OS are you running?

For Linux, look into rsync.

For Windows look into robocopy.

Either can be scripted and scripts scheduled.

--rt

rsync is the bomb with 2 x HDD! I don't schedule anything automatically, I just kick off an rysnc script manually every few weeks as I'm sure if I ran this automagically I would end up deleting some files before the deletions were automatigcally rsynced over.

As an aside, I also use rsync on my windows laptop (under cygwin) to backup to my linux desktop.

Pale Blue Ego
2006-03-16, 12:33
I have my music files on 2 drives in my Linux fileserver. To backup, I connect a 400GB external USB drive to my Windows computer and use a tool called xxcopy

(www.xxcopy.com)

xxcopy is a Windows command-line program that is very flexible, but basically I only use the /clone switch, which will mirror the source drive(s) to the destination drive.

The reason I use a (slower) backup over the network is the Linux box is tucked away and hard to get at, plus it doesn't have USB 2.0.

I also back up to DVD-R as I add new stuff.

radish
2006-03-16, 12:38
I have a seperate server which gets robocopy'd to by a simple script every night.

autopilot
2006-03-16, 12:42
Can robocopy do multiple root folders at once, or can it only do one folder and it sub folders?

Also, say i deleted a file(s) or folder(s) from the source, will it remove the one(s) in the backup. If it does, i will give it a go.

fuzzyT
2006-03-16, 12:51
dangerous_dom wrote:
> Can robocopy do multiple root folders at once, or can it only do one
> folder and it sub folders?

i've only used it for one root and it's subs per command line
invocation. a simple batch file will let you do multi.

> Also, say i deleted a file(s) or folder(s) from the source, will it
> remove the one(s) in the backup. If it does, i will give it a go.

there are command line switches that control this behavior. i usually
make my automated passes w/o deletes, and then occasionally do a manual
run with deletes (i keep two batch files). this protects against
accidental deletions of the sources unintentionally carrying over to the
destination.

--rt

cliveb
2006-03-16, 13:02
About 3 years ago, I sucked it up and built a dedicated server with hardware RAID.
I have a rather unfortunate tale to tell which may make you rethink whether total reliance on RAID is a good idea.

In my day job, I support a database for a large national company involved in the transport business (I better not identify them). That database was stored on a hardware RAID array. Back in 2002, we suffered a hardware or firmware fault (we never found out which) in the RAID controller, which decided to scribble over the file system allocation tables on all five disks. Moral of the story: valuable data should be backed up, even if it's on a nominally fault-tolerant RAID array. (I suppose in the case of CD rips, you could rationalise that a fault like this is so unlikely that you're prepared to re-rip everything if the unthinkable happens).

Browny
2006-03-16, 13:03
Also, say i deleted a file(s) or folder(s) from the source, will it remove the one(s) in the backup. If it does, i will give it a go.

Yes - this is the /MIR switch - it will ensure the destination always has the same contents as the source.

Kyle
2006-03-16, 13:14
Is robocopy part of Windows XP or does it require a separate install? It seems that robocopy and xxcopy are similar. Let's say I've got an updated music folder on drive H and an outdated music folder (some new files not there) on drive I. What command would I use to update the drive I folder to the drive H one?

fuzzyT
2006-03-16, 13:19
Kyle wrote:
> Is robocopy part of Windows XP or does it require a separate install?

robocopy is part of a windows resource kit. google it and pull the
installer and docs from the microsoft site.

> It seems that robocopy and xxcopy are similar. Let's say I've got an
> updated music folder on drive H and an outdated music folder (some new
> files not there) on drive I. What command would I use to update the
> drive I folder to the drive H one?

ms is the source for docs. there are quite a few CL switches for
various options.

for most cases it's something like this:

robocopy /MIR H:\dir I:\dir

--rt

Steve Baumgarten
2006-03-16, 13:23
Kyle wrote:

> Is robocopy part of Windows XP or does it require a separate install?
> It seems that robocopy and xxcopy are similar. Let's say I've got an
> updated music folder on drive H and an outdated music folder (some new
> files not there) on drive I. What command would I use to update the
> drive I folder to the drive H one?

You'd do:

robocopy /mir h:\source i:\dest

The "/mir" switch tells it to delete files in the destination folder
that have been deleted from the source folder. There are other ways to
run robocopy such that it only copies/updates, but doesn't delete.
Depends on what you want. I always run it as above so I keep a perfect
mirror copy.

It's freely available as part of the Windows Server 2003 Resource Kit Tools:

http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&DisplayLang=en

Don't let the name fool you -- it works fine with XP Home as well.

SBB











Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses. The sender therefore
does not accept liability for any errors or omissions in the contents
of this message which arise as a result of e-mail transmission. If
verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities or
related financial instruments.

Richie
2006-03-16, 13:24
> Is robocopy part of Windows XP or does it require a separate install?
> It seems that robocopy and xxcopy are similar. Let's say I've got an
> updated music folder on drive H and an outdated music folder (some new
> files not there) on drive I. What command would I use to update the
> drive I folder to the drive H one?

It's available as part of the Windows Server 2003 Resource Kit Tools
available here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en

The help is quite good. I can't remember what commands you'd use
offhand, but I recall it's syntax being quite easy to pick up. Just
have a practice on some test files first.

Richard

autopilot
2006-03-16, 13:37
Looks like robocopy + batch file might be just the ticket then.

I notice a few GUI's are around for it, but the ones i have seen only do single folders. Any worth bothering with?

One other thing, i take it that robocopy only copies new/changed files, right? I won't copy the whoel lot, regardless of if they have been changed will it?

Kyle
2006-03-16, 13:41
You'd do:

robocopy /mir h:\source i:\dest


This sounds like what I need. If my music files are in a folder called "FLAC Files" on both drives, and I only want to mirror that folder, how would I write the command? Would it be:

/mir h:FLAC Files\source i:FLAC Files\dest

fuzzyT
2006-03-16, 13:51
dangerous_dom wrote:

> I notice a few GUI's are around for it, but the ones i have seen only
> do single folders. Any worth bothering with?

not if you can read simple docs and save text files with a .bat extension.

> One other thing, i take it that robocopy only copies new/changed files,
> right?

correct. it only does the copy if the file has changed.

this may be an overrideable default. and you may be able to define
what constitutes 'changed'. don't recall precisely. it's all in the docs.

autopilot
2006-03-16, 13:58
Could some kind person please email me just the robocopy.exe to domrevans--at--yahoo.co.uk

Thanks!

boerner
2006-03-16, 14:00
I have been using an application called SyncBack on Windows which proves very flexible. Check it out at:

http://www.2brightsparks.com

jeffmeh
2006-03-16, 14:44
You can also use the Windows "Briefcase" functionality to keep a synchronized set of folders.

Steve Baumgarten
2006-03-16, 14:51
Kyle wrote:

> This sounds like what I need. If my music files are in a folder called
> "FLAC Files" on both drives, and I only want to mirror that folder, how
> would I write the command? Would it be:
>
> /mir h:FLAC Files\source i:FLAC Files\dest

That's correct, assuming you want to mirror that folder *and all
sub-folders underneath it*. (You'll probably have to put the names in
quotes, though, since they have a space character in them.)

SBB










Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses. The sender therefore
does not accept liability for any errors or omissions in the contents
of this message which arise as a result of e-mail transmission. If
verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities or
related financial instruments.

Steve Baumgarten
2006-03-16, 14:53
The other nice thing about robocopy is that it does write verification;
the "robo" prefix stands for "robust", according to the docs.

SBB










Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses. The sender therefore
does not accept liability for any errors or omissions in the contents
of this message which arise as a result of e-mail transmission. If
verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities or
related financial instruments.

Dondi
2006-03-16, 15:04
My $.02...

I have my SlimServer on a dedicated WinXP workstaion
with an external 640GB FireWire800 RAID-5 array
attached to the PC in my back room. In the living room
I also have a M$ Media Center edition with a dedicated
500GB LaCie FireWire800 HD for a duplicate music
library. In having a very very very large library
(>67,000) that has been painstakenly tagged to
accommodate both SlimServer & Windows Media Player to
my liking, there are times when retagging and/or
tweaking is necessary. As I only like to do this once
and forget about it, I have incorporated a free M$
PowerToy applet named SyncToy. It doesn't matter which
library I amend, add-to or delete from. If I make a
change on one, it can update the other as well. It
also ensures that files don't get totally deleted. I
manually run SyncToy, but you can have this as an
automated process. I only run it when I have changed
anything in either of my libraries. It's free, it
works on attached drives on seperate PCs, it has a GUI
and hasn't messed-up anything in my libraries.

-- D


--- Steve Baumgarten <sbb (AT) panix (DOT) com> wrote:

> The other nice thing about robocopy is that it does
> write verification;
> the "robo" prefix stands for "robust", according to
> the docs.
>
> SBB
>
>
>
>
>
>
>
>
>
>
> Visit our website at http://www.ubs.com
>
> This message contains confidential information and
> is intended only
> for the individual named. If you are not the named
> addressee you
> should not disseminate, distribute or copy this
> e-mail. Please
> notify the sender immediately by e-mail if you have
> received this
> e-mail by mistake and delete this e-mail from your
> system.
>
> E-mail transmission cannot be guaranteed to be
> secure or error-free
> as information could be intercepted, corrupted,
> lost, destroyed,
> arrive late or incomplete, or contain viruses. The
> sender therefore
> does not accept liability for any errors or
> omissions in the contents
> of this message which arise as a result of e-mail
> transmission. If
> verification is required please request a hard-copy
> version. This
> message is provided for informational purposes and
> should not be
> construed as a solicitation or offer to buy or sell
> any securities or
> related financial instruments.
>
>

ceejay
2006-03-16, 15:19
Put me down as another happy SyncToy user. My main slimserver library is backed up when I feel like it, using SyncToy, to an external disc on another computer.

I also use SyncToy to copy just the AAC branch of the library (I have FLAC, MP3 and AAC) to my laptop to be played with iTunes while travelling.

The external disc is only powered and plugged in when I'm doing a backup (eliminates the lightning risk) and I've deliberately gone for a manually triggered backup to avoid a case where my main system went bananas, damaged its own data and then, before I had a chance to stop it, wrecked the backup as well.

Even if I leave it a month, thats at worst a few CDs worth of data. If I ever do a big retagging effort, I know its time to backup.

So... just at the moment I'm feeling snug and smug. My problem now is that the "huge" disc I thought I'd bought a year ago - with plenty of room for growth - has nearly been filled up by the extraordinary amount of music I've bought in that time! So now I've got to consider buying another drive for the server, and another external drive for backup, and all of a sudden the backup scenario looks a lot harder. Hmmmm...

Ceejay

marty
2006-03-16, 15:52
I've not seen anybody use ntbackup (the backup utility built-in to Windows
XP and 2k).

I have my music in a single RAID 1 array on a dedicated music server, and I
schedule an ntbackup to run an incremental backup twice a week and a full
backup once a week onto a separate disk on the same server. All backups are
verified as well to ensure the success of the backup. Works well for me.

twylie
2006-03-16, 15:56
I have been doing something similar, have you ran into RDP issues with certain microsoft security patchs? I ended up un-installing a few patchs because they kept on causing RDp encryption errors.

Scott

No issues on my side. My first "server" runs XP Pro and my secodn one runs Server2k3. Sorry, but no help on that one.

twylie

Mr_Smiley
2006-03-16, 17:24
I use mp3tunes.com It's a nominal fee ($30/year) for unlimited music storage. It supports mp3, windows and apple drm formats. I think that's it for formats. I stick with mp3's since my ear isn't good enough to notice much difference. It'll even backup iTunes purcahsed music. It has a sync program witch can up/down load files to sync up. It also has an iTunes sync ability. You have the added benefit of accessing your library and playlists through a web interface. It's relieved my wories of hardware failure. Your also alowed 2 simultaneous log ins, in case you and your spouse both listen to the web interface at work.

Thomas J. Angstman
2006-03-16, 17:31
I use rsync over the 'net from my server at work to my server at home.

Mine is a *nix system (http://samba.org/rsync/), but it works on
windows, too:

http://www.gaztronics.net/rsync.php


TJ


-----Original Message-----
From: discuss-bounces (AT) lists (DOT) slimdevices.com
[mailto:discuss-bounces (AT) lists (DOT) slimdevices.com] On Behalf Of
dangerous_dom
Sent: Thursday, March 16, 2006 10:46 AM
Subject: [slim] Re: How do YOU backup your music?


Sorry, running XP. Don't need CD/DVD buring either.


--
dangerous_dom
------------------------------------------------------------------------
dangerous_dom's Profile:
http://forums.slimdevices.com/member.php?userid=1763
View this thread: http://forums.slimdevices.com/showthread.php?t=22161

dancarne
2006-03-16, 18:07
I use Acronis True Image for backups. And, my music drive is RAID1.

rupped
2006-03-16, 19:18
I have been using an application called SyncBack on Windows which proves very flexible. Check it out at:

http://www.2brightsparks.com

I have been using SyncBack for 6 months with multiple external drives. It is simple to set up, and previews what it is going to do (and allows you to make changes). Plus it's free - what's not to like??

David

Kyle
2006-03-16, 19:27
I cannot get the syntax right for robocopy. I have tried to copy a test folder named "folder1" on my desktop to another folder named "folder2", but I keep getting syntax errors in WindowsXP. Can someone help before I try my actual music files?

JJZolx
2006-03-16, 21:56
I cannot get the syntax right for robocopy. I have tried to copy a test folder named "folder1" on my desktop to another folder named "folder2", but I keep getting syntax errors in WindowsXP. Can someone help before I try my actual music files?
Post the code you're using. And _definitely_ use test folders until you're 110% sure you have both the syntax and the logic correct. Otherwise, if you're using robocopy's mirroring commands you risk wiping out your music collection if you were to accidentally mirror in the wrong direction.

JJZolx
2006-03-16, 22:13
I have my music library on an Infrant NAS running RAID 5 with 878 GB of storage on my home network. I bought the NAS primarily for the large, contiguous file space that it offers and I also use it for storing video and computer backups. The RAID part is secondary. I did have a bad disk when I first set it up last summer, but that meant I had to run the array in a degraded state (cross your fingers - lose a secondnd drive and you're toast) for about 10 days, since the only alternative was to shut it down and wait, while the disk was RMA'd.

My library still only takes up about 180 MB, so I back it up by mirroring it using robocopy to a spare drive in one of my Windows XP PCs on the network. I could just as easily do it the other way around - storing the library on a local disk in the music server and doing backups to the NAS. That would certainly be faster for SlimServer scanning, but I'm anticipating eventually needing the large amount of storage that the NAS gives. Backups at that point may have to be to another similarly configured NAS, or else I'll build a dedicated storage server.

One of the downside to living in an age where you have enormous amounts of cheap disk storage available is that if you actually start filling it up with _valuable_ data, then you have no good, inexpensive way to back it up. Disk-to-disk backups end up being about the cheapest way, when you factor in any kind of convenience, but it means that you need to at least double your storage space to accomplish it.

mherger
2006-03-17, 00:11
> I use rsync over the 'net from my server at work to my server at home.
> http://www.gaztronics.net/rsync.php

If you don't want to install cygwin (which is a great tool not only for
Linux users bound to a Windows PC :-)), but still would like to use rsync,
there's a stripped down rsync package with only the few files needed to
run it on Windows (plus documentation):

http://itefix.no/cwrsync/

Much easier to configure than cygwin - at a fraction of the download size.

--

Michael

-----------------------------------------------------------
Help translate SlimServer by using the
SlimString Translation Helper (http://www.herger.net/slim/)

autopilot
2006-03-17, 05:18
You can also use the Windows "Briefcase" functionality to keep a synchronized set of folders.

NO! Correct me if i am wrong, but if you lost some of your master files and the briefcase gets updated it will delete the copies in the briefcase! And i think briefcase updates on start up. So, you lose your files somehow and then your PC reboots. Boom, back up gone :(

Briefcase is a synchronization solution for working documents, not a backup solution.

Right, this is starting to do my head in now. I just can't find the solution that is _just_ right for me. But i think with a bit of playing around, Robocopy will do what i want.

What i want is this;
Automated copying of new/modified files to my second HDD. If i delete my master files, i dont want my backup files deleted. However, i do want my _modified_ files copying over. Can robocopy do this?

Browny
2006-03-17, 05:38
What i want is this;
Automated copying of new/modified files to my second HDD. If i delete my master files, i dont want my backup files deleted. However, i do want my _modified_ files copying over. Can robocopy do this?

Yes - should be something like:

Robocopy /E /IT

so long as you don't use the /MIR or /PURGE options Robocopy won't delete files from the backup.

Robocopy is your friend!! Probably the best freeby from M$ - don't know why they have'nt put it in the base OS yet.

autopilot
2006-03-17, 05:45
Could some please email me the lastet robocopy.exe to domrevans-AT-yahoo-DOT-co-DOT-uk please! :)

jeffmeh
2006-03-17, 06:56
No, briefcase does not update upon startup, or even automatically, unless you set it up to do so.

You are correct that if I lose a source file and then update the briefcase, then the update will delete the associated file in the briefcase. However, I initiate the update myself, and the update process also tells me everything it is going to do and asks me whether I wish to continue. If I see that it intends to delete a file, I can decide whether that is consistent with my expectations and either continue or bail out. In my view, that gives me enough of a margin of error.

I also find the synchronization very helpful, particularly when I am renaming and reorganizing files, as those changes are reflected in the briefcase upon update. But you seem to be looking for a simple automated copy function. Just pick one of the suggestions and try it out.

By the way, many of the tools people are citing would work the same way as briefcase if you set them up to "mirror" or "synchronize." Some people have said they do this with automated scripts, which I think is pretty risky for just the reason you cited.

My music sits on an Infrant NAS in RAID 0, so I already have a "mirroring" solution which will protect me from a disk failure. I set up the briefcase on another disk on the network, and I am very careful when I update it. This provides some additional protection from bugs, viruses, and my own stupidity, unless that stupidity includes updating the briefcase without being careful, as you have pointed out.

Every backup scheme has to deal with this in one way or another. Even if you keep many generations of full backups, you can still get into a situation where a problem was introduced 5 days ago, you did not catch it until today, but you did a significant amount of work yesterday. If you restore the backup from before the problem you no longer have the problem, but you lose all the work you did yesterday. If you stay where you are and do not restore a backup, you do not have to repeat yesterday's work, but you do have to figure out how to fix the problem some other way. Either approach can be the better one, depending upon the specifics of the circumstances.

I am in no way saying that briefcase is a great tool for backups, merely that if you are on Windows you already have it, and that what it provides it provides pretty well. YMMV


NO! Correct me if i am wrong, but if you lost some of your master files and the briefcase gets updated it will delete the copies in the briefcase! And i think briefcase updates on start up. So, you lose your files somehow and then your PC reboots. Boom, back up gone :(

Briefcase is a synchronization solution for working documents, not a backup solution.

Right, this is starting to do my head in now. I just can't find the solution that is _just_ right for me. But i think with a bit of playing around, Robocopy will do what i want.

What i want is this;
Automated copying of new/modified files to my second HDD. If i delete my master files, i dont want my backup files deleted. However, i do want my _modified_ files copying over. Can robocopy do this?

Kyle
2006-03-17, 07:25
Post the code you're using. And _definitely_ use test folders until you're 110% sure you have both the syntax and the logic correct. Otherwise, if you're using robocopy's mirroring commands you risk wiping out your music collection if you were to accidentally mirror in the wrong direction.
I downloaded SyncToy and played around with it last night. It seems to work fine with a GUI, so it's much easier for me. It also moves deleted files to the trashcan, so you can retreive them if you lose something you didn't mean to.

Kevin O. Lepard
2006-03-17, 08:20
Multiple layers of protection:

1) Original CDs in storage
2) Copy of ripped music on 2 different hard drives
3) Copy burned to 2 sets of DVD-Rs, one off-site
4) Copy on a RAID 5 NAS
--
Kevin O. Lepard

Happiness is being 100% Microsoft free.

Robin Bowes
2006-03-17, 10:15
Kevin O. Lepard wrote:
> Multiple layers of protection:
>
> 1) Original CDs in storage
> 2) Copy of ripped music on 2 different hard drives
> 3) Copy burned to 2 sets of DVD-Rs, one off-site
> 4) Copy on a RAID 5 NAS

That's nice and thorough.

I'm not quite there yet. I've currently got:

1. Original CDs in storage. Well, for *most* of my collection ;)
2. Ripped music on RAID5 drive (linux software RAID). Server on UPS.
3. Copy of music on 500GB drive in Xbox offsite.

I really must get round to either burning to DVD-R or 15GB DLTs before
the inevitable happens.

R.

JJZolx
2006-03-17, 10:35
> 1) Original CDs in storage
> 2) Copy of ripped music on 2 different hard drives
> 3) Copy burned to 2 sets of DVD-Rs, one off-site
> 4) Copy on a RAID 5 NAS[/color]

That's nice and thorough.

I'm not quite there yet. I've currently got:

1. Original CDs in storage. Well, for *most* of my collection ;)
2. Ripped music on RAID5 drive (linux software RAID). Server on UPS.
3. Copy of music on 500GB drive in Xbox offsite.

I really must get round to either burning to DVD-R or 15GB DLTs before
the inevitable happens.
What's the "inevitable"? A freak tornado hits both your house and your offsite backup location and blows away your original CDs, your RAID 5 array and the XBox backup? You must live under a very dark cloud if you think that's inevitable.

m1abrams
2006-03-17, 10:41
What's the "inevitable"? A freak tornado hits both your house and your offsite backup location and blows away your original CDs, your RAID 5 array and the XBox backup? You must live under a very dark cloud if you think that's inevitable.

When it comes to backups you can NEVER have too much.

pfarrell
2006-03-17, 10:56
m1abrams wrote:
> JJZolx Wrote:
>>What's the "inevitable"? A freak tornado hits both your house and your
>>offsite backup location and blows away your original CDs, your RAID 5
>>array and the XBox backup? You must live under a very dark cloud if
>>you think that's inevitable.
>
> When it comes to backups you can NEVER have too much.

It really is both an emotional and an engineering question.
Economics play a big part of the decision process.
This is true of any backup situation.

If you have the need, you have to have two physical locations
preferably a thousand or more miles apart.

I have a bit over 700 CDs. at $10 a piece, that is $7,000
worth of music. So if it all was destroyed, I have to look at
$7000 plus time to re-rip. There is no way I'm going to spend
$25,000 on a backup strategy for $7000 worth of music.

I live "inside the Beltway" in the Washington DC area. If the
Russians drop a nuke on the Pentagon, my house will be vaporized.
If that happens, all my music is gone. But I don't worry about
that, because if I'm home at the time, I've got other problems.
Or maybe no more problems ever.

If I really cared, I bet that there is some SqueezeBox
owner in Europe that would be willing to hold a 400GB disk for me
while I did the same for them. Access could be instant over the
'net, or via DHL in a couple of days. You can get about
anything you want to pay for.


--
Pat
http://www.pfarrell.com/music/slimserver/slimsoftware.html

jimdibb
2006-03-17, 10:59
If the cost of the backups is more than the data is worth, you have too
much.

On 3/17/06, m1abrams <m1abrams.24trp01142617502 (AT) no-mx (DOT) forums.slimdevices.com>
wrote:
>
>
>
>
> When it comes to backups you can NEVER have too much.
>
>
> --
> m1abrams
>

JJZolx
2006-03-17, 11:05
When it comes to backups you can NEVER have too much.
Yes, you can. At some point it becomes either prohibitively expensive or overly time consuming to manage, just to gain some very small extra degree of security.

stevieweevie
2006-03-17, 12:36
I have a 250GB external drive ... using XXCopy I either write a full backup, or an incremental either based on new files, or changed files.

Prior to making any backup I run MP3Test against the source data. My MP3's are mostly encoded with CRC's, so this helps with regard to integrity.

Robin Bowes
2006-03-17, 12:58
JJZolx wrote:
> Robin Bowes Wrote:
>
>>>1) Original CDs in storage
>>>2) Copy of ripped music on 2 different hard drives
>>>3) Copy burned to 2 sets of DVD-Rs, one off-site
>>>4) Copy on a RAID 5 NAS
>>
>>That's nice and thorough.
>>
>>I'm not quite there yet. I've currently got:
>>
>>1. Original CDs in storage. Well, for *most* of my collection ;)
>>2. Ripped music on RAID5 drive (linux software RAID). Server on UPS.
>>3. Copy of music on 500GB drive in Xbox offsite.
>>
>>I really must get round to either burning to DVD-R or 15GB DLTs before
>>the inevitable happens.
>
> What's the "inevitable"? A freak tornado hits both your house and your
> offsite backup location and blows away your original CDs, your RAID 5
> array and the XBox backup? You must live under a very dark cloud if
> you think that's inevitable.[/color]

Sorry, I wasn't clear.

Whilst I have a (nearly) full back up of my lossless collection (18GB of
Mozart wouldn't fit on the 500GB drive) I need to back up my lossy
collection, plus DR backups for the rest of my systems.

As for the inevitable, it's exceedingly common for >1 drives to fail in
a RAID5 system because the stress of rebuilding the array following a
failure often causes one or more of the the other drives to fail too.

I don't worry about whether my discs will fail or not - they will, it's
just a matter of time.

R.

m1abrams
2006-03-17, 13:16
Yes, you can. At some point it becomes either prohibitively expensive or overly time consuming to manage, just to gain some very small extra degree of security.

Sorry I am coming from a industry background not a music background on this topic. In industry the price of the data sometimes is priceless. Yes with music you can usually get a replacement at some cost. (I do have CDs I have lost that are virtually unreplacable). However photos of your childs first step, what is the price of that?

And the example of nuke inside DC, well yea that is silly to protect against. However fire damage of ones home is not so silly to protect against.

autopilot
2006-03-18, 16:00
Well thanks for al the info guys.

I now have robocopy and it's just what i needed. I had not writen a complex batch file for some years, so i was a bit rusty, but now i have a nice batch with a menu that can select mirror or backup modes. If anyone wants to see it, i will post the code.

autopilot
2006-03-19, 06:10
If anyone is interested, this is the batch script i made for robocopy;



@ECHO OFF
:startbatch
cls
echo.
echo Dom's Backup Script
echo -------------------
echo.
set /p choice= Backup, Mirror or Quit (B/M/Q)? :

if '%choice%'=='b' goto startbackup
if '%choice%'=='m' goto startmirror
if '%choice%'=='q' goto end
goto startbatch


:startbackup
robocopy /S /IT "F:\Audio" "D:\BACKUP\Audio"
pause
robocopy /S /IT "F:\Drivers & Manuals" "D:\BACKUP\Drivers & Manuals"
pause
robocopy /S /IT "F:\Downloads" "D:\BACKUP\Downloads"
pause
robocopy /S /IT "F:\Photos" "D:\BACKUP\Photos"
pause
goto finish

:startmirror
robocopy /MIR "F:\Audio" "D:\BACKUP\Audio"
pause
robocopy /MIR "F:\Drivers & Manuals" "D:\BACKUP\Drivers & Manuals"
pause
robocopy /MIR "F:\Downloads" "D:\BACKUP\Downloads"
pause
robocopy /MIR "F:\Photos" "D:\BACKUP\Photos"
pause
goto finish


:end
echo.
echo Aborted

:finish

echo.
pause


Thanks again for pointing me in the right direction. If anyone can suggest any ways i could improve this, please do.

Also, if anyone does not want to use robocopy and write a batch, i also came acrross this usful backup utility - http://www.simplysafebackup.com/

hickinbottoms
2006-03-20, 01:35
I use boxbackup to continuously sync my music (and anything I want to back up), to a disc I've installed remotely at a friends house over ADSL.

http://boxbackup.hostworks.ca

This is a secure solution in that the connection is encrypted with SSL and authenticated with certificates, is totally automatic (it continually syncs, keeps old versions of files, as well as deleted files), and is stored encyrpted at the server end and so is most useful when the server is in an untrusted network (ie not your own).

Thoroughly recommended - the only drawback is that there isn't a Windows server at the moment (although there are Windows clients that work well).

It's saved my bacon in the past.

JJZolx
2006-03-20, 12:47
I use boxbackup to continuously sync my music (and anything I want to back up), to a disc I've installed remotely at a friends house over ADSL.

...

It's saved my bacon in the past.
Although I know a lot of people are doing automated syncs using any number of software solutions, I have a real problem with considering this a safe backup solution. It's not so much the sync part - it's the _automated_ part. For example: You accidentally delete a few folders of important data without noticing, the sync then deletes them on the "backup" copy. You're SOL.

Syncing is a nice, easy way to make backup copies, but I'd reserve it for manual operations only. It's best to have a very high level of confidence in the integrity of the original before you push it to your only sync'd copy.

Now, if you can keep multiple generations of copies, automating the process isn't so unsafe, so long as you think you'll catch a deletion or corruption before all of the copies are overwritten. That's usually not an option, though, when we're talking about multi-hundred-gigabyte media libraries.

stinkingpig
2006-03-20, 13:01
JJZolx wrote:
> hickinbottoms Wrote:
>
>> I use boxbackup to continuously sync my music (and anything I want to
>> back up), to a disc I've installed remotely at a friends house over
>> ADSL.
>>
>> ...
>>
>> It's saved my bacon in the past.
>>
> Although I know a lot of people are doing automated syncs using any
> number of software solutions, I have a real problem with considering
> this a safe backup solution. It's not so much the sync part - it's the
> _automated_ part. For example: You accidentally delete a few folders of
> important data without noticing, the sync then deletes them on the
> "backup" copy. You're SOL.
>
all solutions are vulnerable to user error. At some point, you just
gotta give up on trying to make the computer be smarter than the user.
Still,

....
>
> Now, if you can keep multiple generations of copies, automating the
> process isn't so unsafe, so long as you think you'll catch a deletion
> or corruption before all of the copies are overwritten. That's usually
> not an option, though, when we're talking about multi-hundred-gigabyte
> media libraries.
>
>

Once again, this problem was solved years ago on *nix. It's called
hard-linking, and it allows the unchanged copies to exist without making
multiples. http://www.monkeynoodle.org/comp/tools/backups

--
Jack at Monkeynoodle dot Org: It's a Scientific Venture...
Riding the Emergency Third Rail Power Trip Since 1996

bklaas
2006-03-20, 13:23
rsnapshot, previously mentioned by me and a few others in this thread, is a nice packaged implementation of just what Jack described in the previous post. It uses rsync together with hardlinking to be able to store multiple snapshots of whatever it is you want to backup with only a "delta" more HD space used for each snapshot. It is highly configurable and works like a charm.

For my personal data, I take and save a snapshot of my HD each day (saving the last 5), each week (saving the last 2), each month (saving the last 3). I've never, ever lost a file with this system, and I've deleted several files that I didn't want to (grabbed it from the snapshot backup without incident), as well as survived a HD crash (rsnapshot restored my data easily after HD was replaced).

And these are the sizes of the data drive and the BACKUP drive.
/dev/sda4 208G 101G 108G 49% /data
/dev/sdb1 230G 122G 97G 56% /BACKUP
10 full HD backups for only a small amount more space than the current. Pretty sweet.

I also find this a MUCH better solution than RAID, which I find a troubling amount of people using it as 100% of their backup system. RAID's a really cool technology, but that was never what it was intended for...

cheers,
#!/ben

stevieweevie
2006-03-20, 14:06
Using XXCopy ... (WIN32)

set source=\\htpc\d\media\music\
set dest=g:\backups\music\
xxcopy %source% %dest% /bb /s /oNxxcopylog.txt

Only copies new files to the backup. Any changed files get ignored. If I know a file has changed, I delete it from the destination beforehand.

Kyle
2006-03-23, 14:37
If anyone is interested, this is the batch script i made for robocopy;

dom, how does your script work? I copied it to test and thought it would ask me whether I wanted to backup, mirror or quit, but it didn't.

autopilot
2006-03-24, 03:12
dom, how does your script work? I copied it to test and thought it would ask me whether I wanted to backup, mirror or quit, but it didn't.

Assuming you have robocopy installed and your paths are right (go to the command prompt and type 'robocopy' to test it), you will then have to edit the drive/folders to match yours. Works great for me but you will have different folder names. Also, what OS are you using?

For more help with batch file commands, try here http://www.computerhope.com/batch.htm

Kyle
2006-03-24, 07:24
Assuming you have robocopy installed and your paths are right (go to the command prompt and type 'robocopy' to test it), you will then have to edit the drive/folders to match yours. Works great for me but you will have different folder names. Also, what OS are you using?

For more help with batch file commands, try here http://www.computerhope.com/batch.htm
I adjusted the paths to match mine but it appeared to actually perform the functions rather than offer me a choice of mirror or backup. Here's what I get:

echo.

echo Dom's Backup Script
Dom's Backup Script
echo -------------------
-------------------
echo.

set /p choice= Backup, Mirror or Quit (B/M/Q)? :
Backup, Mirror or Quit (B/M/Q)? :
if '%choice%'=='b' goto startbackup
if '%choice%'=='m' goto startmirror
if '%choice%'=='q' goto end
goto startbatch


:startbackup
robocopy /S /IT "C:\documents and settings\jim\desktop\folder1" "C:\documents a
nd settings\jim\desktop\folder3"

-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows :: Version XP010
-------------------------------------------------------------------------------

Started : Fri Mar 24 09:22:19 2006

Source : C:\documents and settings\jim\desktop\folder1\
Dest : C:\documents and settings\jim\desktop\folder3\

Files : *.*

Options : *.* /S /COPY:DAT /IT /R:1000000 /W:30

------------------------------------------------------------------------------

0 C:\documents and settings\jim\desktop\folder1\
3 C:\documents and settings\jim\desktop\folder1\Bi
ll\
3 C:\documents and settings\jim\desktop\folder1\Bo
b\

------------------------------------------------------------------------------

Total Copied Skipped Mismatch FAILED Extras
Dirs : 3 0 3 0 0 0
Files : 6 0 6 0 0 0
Bytes : 705.0 k 0 705.0 k 0 0 0
Times : 0:00:00 0:00:00 0:00:00 0:00:00

Ended : Fri Mar 24 09:22:19 2006
goto finish

:startmirror
robocopy /MIR "C:\documents and settings\jim\desktop\folder1" "C:\documents and
settings\jim\desktop\folder3"

-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows :: Version XP010
-------------------------------------------------------------------------------

Started : Fri Mar 24 09:22:20 2006

Source : C:\documents and settings\jim\desktop\folder1\
Dest : C:\documents and settings\jim\desktop\folder3\

Files : *.*

Options : *.* /S /E /COPY:DAT /PURGE /MIR /R:1000000 /W:30

Shouldn't it ask for input from me?
Also, is there a way to keep this script somewhere and run it automatically, or do I need to open a command window and copy it there each time? I use WindowsXP.

autopilot
2006-03-24, 07:42
Shouldn't it ask for input from me?
Also, is there a way to keep this script somewhere and run it automatically, or do I need to open a command window and copy it there each time? I use WindowsXP.

Oh i see, yes it's ment to be used in a batch file, not cut and pasted to the command prompt.

Create a new file called, say, backup.BAT. It's the .BAT thats the standard extention thats been used since the old DOS days. Then just double click it and off you go. You can edit as and when you like. I should ask you to input an option then.

Take a look at the URL i gave you above for more info on making batch files.

Kyle
2006-03-24, 08:06
Create a new file called, say, backup.BAT. It's the .BAT thats the standard extention thats been used since the old DOS days. Then just double click it and off you go. You can edit as and when you like. I should ask you to input an option then.

Thanks! Works perfectly.

autopilot
2006-03-24, 10:52
No problem, play around and make it your own. Just make sure you always put it in the order of 'robocopy [source] [destination]' so you don't overwrite the orginals.

Also, you can take the pauses out to make run all the way through without asking for a key press if you want. It's just that i like to see the summory to check for any errors etc.

Mark Lanctot
2006-03-24, 12:54
Since I just got a USB drive enclosure and a 250 GB backup drive, I'm playing around with this today.

I wanted to use XXCOPY, and so far I have it doing a nice backup with the \BU switch. This will ignore any files that haven't changed, add any that were added in the source drive and modify any that were modified in the source drive.

The only thing I can't get it to do is to delete files that were deleted in the source directory. I tried the /RD and /RX switches but both wanted to remove the whole destination directory! :-O Thank goodness it asked first.

Did I miss something with XXCOPY? The other delete switches all deal with deleting files in the source directory, which is not what you want to do with a backup...

So I'll play with robocopy now and see if it does what I want.

Edit: Yup, robocopy /MIR does exactly what I want except that it tries to copy the System Volume Information over and errors out. I guess I can live with that, but it would be nice to have a report saying what it did and reporting speed like XXCOPY does.

autopilot
2006-03-24, 13:11
Yup, robocopy /MIR does exactly what I want except that it tries to copy the System Volume Information over and errors out.

I'm sure there is a switch to stop that.


it would be nice to have a report saying what it did and reporting speed like XXCOPY does.

Are you sure you are not missing something? I always get a full report on exactly what it has done, any errors and time etc. Is the
cmd window closing before you have had cance to read it? If so, use a batch file and put a pause at the end.

http://www.ss64.com/nt/robocopy.html

Kyle
2006-03-24, 13:13
... but it would be nice to have a report saying what it did and reporting speed like XXCOPY does.
Mark, are you not seeing the data that robocopy gives at the end of its process, or are you used to getting more data than that?

Mark Lanctot
2006-03-24, 13:26
Are you sure you are not missing something? I always get a full report on exactly what it has done, any errors and time etc. Is the
cmd window closing before you have had cance to read it? If so, use a batch file and put a pause at the end.

http://www.ss64.com/nt/robocopy.html

It's because it's erroring out at the end. It tries, errors, waits 30 seconds, tries again, errors, waits 30 seconds etc. I have to kill the process.

I'm pretty sure if it completed successfully I'd get a report.

JJZolx
2006-03-24, 13:26
Use robocopy's ability to create log files and check the logs afterwards. A typical robocopy operation that operates on thousands of files isn't necessarily something you can read onscreen.



Logging options
/L : List only - don't copy, timestamp or delete any files.
/NP : No Progress - don't display % copied.
/LOG:file : output status to LOG file (overwrite existing log).
/LOG+:file : output status to LOG file (append to existing log).

/TS : include source file Time Stamps in the output.
/FP : include Full Pathname of files in the output.
/NS : No Size - don't log file sizes.
/NC : No Class - don't log file classes.
/NFL : No File List - don't log file names.
/NDL : No Directory List - don't log directory names.
/TEE : output to console window, as well as the log file.
/NJH : No Job Header.
/NJS : No Job Summary.

Steve Baumgarten
2006-03-24, 13:53
Mark Lanctot wrote:

> It's because it's erroring out at the end. It tries, errors, waits 30
> seconds, tries again, errors, waits 30 seconds etc. I have to kill the
> process.
>
> I'm pretty sure if it completed successfully I'd get a report.

You need to tell it to not retry open files a million times (really,
that's the default retry number: 1000000). Or you can tell it skip
particular files/folders that you know you don't want backed up or that
will always be open/locked. I have a robocopy script that backs up my
Firefox application data directory and I have it skip the cache folder
as well as the ".lock" file so all runs smoothly.

Add these options:

/R:n : number of Retries on failed copies - default is 1 million.
/W:n : Wait time between retries - default is 30 seconds.

So you could do:

/R:0

That will cause robocopy to report on files it can't copy but then just
skip them.

To always skip particular files or directories (like that System Volume
Information folder you mentioned) that you know ahead of time you aren't
interested in:

/XF file [file]... : eXclude Files matching given names/paths/wildcards.
/XD dirs [dirs]... : eXclude Directories matching given names/paths.

XF and XD can be used in combination e.g.
ROBOCOPY c:\source d:\dest /XF *.doc *.xls /XD c:\unwanted /S

It's a powerful utility, but yes, there are an awful lot of options.

http://www.ss64.com/nt/robocopy.html

SBB








Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses. The sender therefore
does not accept liability for any errors or omissions in the contents
of this message which arise as a result of e-mail transmission. If
verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities or
related financial instruments.

Mark Lanctot
2006-03-24, 14:03
Awesome guys, thanks for the help. I got robocopy to skip System Volume Information using the /XD switch.

Haven't quite got /LOG working yet. What's strange is that it says it's bad, yet writes a log anyway! :-)

Edit: got /LOG working, it just needed a full path. Mirroring my music directory now. I can see why you'd want to supress progress bars in the logfile. :-P

ceejay
2006-03-25, 02:51
FYI, I've just added a link to this thread from the BeginnersGuideToServers (backup section)

So if there are any more bright ideas of tackling this problem, lets have them!

Ceejay.

Mark Lanctot
2006-03-25, 08:06
BTW the final options I ended up with:


robocopy k:\ m:\ /mir /xd "System Volume Information" /log:"C:\Documents and Settings\Mark\Desktop\Music Backup.log" /np /tee

So that's:

- copying from k:\, my internal 250 GB music drive, to m:\, my external backup drive
- /mir: mirroring the drives, which entails copying over all new files, any file that has changed and deleting files in the destination that have been deleted in the source
- /xd: ignoring the "System Volume Information" directory that Windows XP puts in each drive yet is locked out
- /log: logging to a file on my desktop
- /np: don't show progress bars in the log (kind of useless after the job is finished and you look at the file, no? :-) )
- /tee: display the progress on the screen. Useful to see what it's currently doing. I suppose that might make the progress bars useful again, but now that the bulk of my music collection is backed up there will only be incremental changes from now on.

darrenyeats
2007-03-21, 12:26
All,
I recommend "unison" file synchronizer. It works on all platforms - and even between platforms as I understand it.

I am on Linux. This should be available as a package for your distro - it certainly is with Debian. Works a charm:

1. Read The (unison) Manual

2. Set up your stuff in ~/.unison/default.prf and you can invoke it as 'unison -batch' to sync your main dir with the backup dir.

unison deals with any changed, new or deleted items - it just syncs the dirs exactly.

Regards, Darren

peter
2007-03-21, 13:41
darrenyeats wrote:
> All,
> I recommend "unison" file synchronizer. It works on all platforms - and
> even between platforms as I understand it.
>
> I am on Linux. This should be available as a package for your distro -
> it certainly is with Debian. Works a charm:
>
> 1. Read The (unison) Manual
>
> 2. Set up your stuff in ~/.unison/default.prf and you can invoke it as
> 'unison -batch' to sync your main dir with the backup dir.
>
> unison deals with any changed, new or deleted items - it just syncs the
> dirs exactly.
>

Wonderful, as long as you didn't inadvertently wipe out you main storage
en didn't notice until after unison did its destructive work...

Snapshots are what is needed: rsnapshot to be precise.

Regards,
Peter

Mark Miksis
2007-03-21, 14:22
Wonderful, as long as you didn't inadvertently wipe out you main storage
en didn't notice until after unison did its destructive work...

And with unison, it's twice as scary. Unison is basically a two-way rsync so if the backup somehow gets destroyed those changes will be propagated back to the main storage.

JJZolx
2007-03-21, 14:43
> unison deals with any changed, new or deleted items - it just syncs the
> dirs exactly.

Wonderful, as long as you didn't inadvertently wipe out you main storage
en didn't notice until after unison did its destructive work...

Snapshots are what is needed: rsnapshot to be precise.

How does rsnapshot work? From what I read it's similar to a traditional data backup strategy of doing a full backup, then incremental backups.

studley
2007-03-21, 15:16
How does rsnapshot work? From what I read it's similar to a traditional data backup strategy of doing a full backup, then incremental backups.

I'm interested in this too. Could someone point us to a good site/FAQ?

Thanks, David

bklaas
2007-03-21, 15:32
more than adequate documentation on the website for rsnapshot:
http://rsnapshot.org/

from the top of the website:
"rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.

Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals."

I use rsnapshot for both home and work use, and it's fantastic. It's mostly for the Unix/Linux/BSD crowd, but I've heard of people using it for Win backups as well.

cheers,
#!/ben

snarlydwarf
2007-03-21, 15:35
I'm interested in this too. Could someone point us to a good site/FAQ?


http://www.rsnapshot.org/faq.html

Make sure you follow the link to Wikipedia about Hard Links if you're not familiar with them. An ancient Unix filesystem trick, but very useful in this case.

(Not at all the same as Symlinks which are more like "Shortcuts" in Windows... a hardlink is exactly the same as whatever it copies. It uses the same disk space, and you can have thousands of filenames pointing to exactly the same spot on the disk. You can rename or delete each without affecting the others... overwriting is another matter, though.)

JJZolx
2007-03-21, 15:50
The disk space required is just a little more than the space of one full backup, plus incrementals."

Doesn't sound like what I understand to be a snapshot. Usually only pointers are stored at the time of the snapshot, then writes to the filesystem are actively monitored and only the changes stored.

For most people, any backup strategy for music or video that takes up more space than the original probably isn't feasible. If you have the storage space, it would certainly be _nice_, since it offers better protection than doing a sync/mirror. Imagine running a script on your library to add a COMPILATION tag. Unless you start all over, the incremental will be as big as the full backup and the entire thing will require 2x the storage of the library itself.

With a simple mirror you only need as much space as the original. You could buy two 400 GB drives and when the first is nearly full you won't be running out of space on the other used for backup. Yes, you have to be very careful to not mirror a corrupted library, or one where files have been deleted.

hickinbottoms
2007-03-21, 17:32
If you have a server that doesn't run Windows, and want automatic off-site backups, then I recommend Box Backup:

http://bbdev.fluffy.co.uk/trac/wiki

Secure, encrypted, incremental backups to a remote location. Keeps old and deleted versions of files (until the store is full, in which case it starts removing the oldest 'old versions'), can run continuously and is very bandwidth efficient.

The big advantage is that the server can be untrusted - it cannot decrypt the backups it holds as the clients keep the keys and encrypt the traffic before uploading. Perfect for backing up to a friends house or another server you can't trust.

It's saved my bacon a few times over the last couple of years.

And it's free and open source. Note that although the server can't be Windows-based, clients can be. One server can also keep separate backup stores for any number of clients.

Stuart


I currently have 2 HDD's in my PC. I have 2 large important folder trees, one with my music and on with my photo's. Currently i am manually copying everything between drives, but this is becoming a bit of a chore and often i forget.

So what software people here use and would recommend for me?

I don't want anything fancy, i.e. compression is needed as obviously its already compressed. Just a simple solution with folder syncing and preset times (over night).

Oh, and free would be nice too :D

Pale Blue Ego
2007-03-21, 20:14
Since I just got a USB drive enclosure and a 250 GB backup drive, I'm playing around with this today.

I wanted to use XXCOPY, and so far I have it doing a nice backup with the \BU switch. This will ignore any files that haven't changed, add any that were added in the source drive and modify any that were modified in the source drive.

The only thing I can't get it to do is to delete files that were deleted in the source directory. I tried the /RD and /RX switches but both wanted to remove the whole destination directory! :-O Thank goodness it asked first.

Did I miss something with XXCOPY? The other delete switches all deal with deleting files in the source directory, which is not what you want to do with a backup...

So I'll play with robocopy now and see if it does what I want.

Edit: Yup, robocopy /MIR does exactly what I want except that it tries to copy the System Volume Information over and errors out. I guess I can live with that, but it would be nice to have a report saying what it did and reporting speed like XXCOPY does.

With xxcopy, I always use the /clone switch:

xxcopy x:\ n:\ /clone /yy

Where x:\ is the source, n:\ the destination. /yy suppresses all prompts so it can run unattended. The /clone switch will make an exact copy, including any changes or deletes.

mherger
2007-03-21, 21:13
>> The disk space required is just a little more than the space of one full
>> backup, plus incrementals."
>
> Doesn't sound like what I understand to be a snapshot. Usually only
> pointers are stored at the time of the snapshot, then writes to the
> filesystem are actively monitored and only the changes stored.

That "little more than the space of one backup" is the space needed for
pointers (hardlinks). I'm backing up about 15GB of data (mainly office
documents, mail, photos, without the music). Daily incrementals are around
40MB. If I edit a 100MB image file, the daily backup will of course
increase by these 100MB.

> be running out of space on the other used for backup. Yes, you have to
> be very careful to not mirror a corrupted library, or one where files
> have been deleted.

That's exactly the point: with snapshots/incrementals you need more space,
but you often can go back days, weeks or even months to restore a file. I
have 7 dailies, 4 weeklies and 8 monthlies of those 15GB of files. That
backup is using a total of 23GB - 50% addition for a max of 19 versions
per file.

--

Michael

-----------------------------------------------------------------
http://www.herger.net/SlimCD - your SlimServer on a CD
http://www.herger.net/slim - AlbumReview, Biography, MusicInfoSCR

peter
2007-03-21, 23:52
JJZolx wrote:
> bklaas;189372 Wrote:
>
>> The disk space required is just a little more than the space of one full
>> backup, plus incrementals."
>>
>
> Doesn't sound like what I understand to be a snapshot. Usually only
> pointers are stored at the time of the snapshot, then writes to the
> filesystem are actively monitored and only the changes stored.
>
> For most people, any backup strategy for music or video that takes up
> more space than the original probably isn't feasible. If you have the
> storage space, it would certainly be _nice_, since it offers better
> protection than doing a sync/mirror. Imagine running a script on your
> library to add a COMPILATION tag. Unless you start all over, the
> incremental will be as big as the full backup and the entire thing will
> require 2x the storage of the library itself.
>

Yes, that's the disadvantage. You have to be careful with operations
like that or reorganizations of your directory structure. In my case
stuff is backed up to a colo server, but my upload bandwidth isn't
unlimited (1 Mbit) so I have to remind myself of the physical limits.

Of course, the bandwidth limitation would be the same in case of a
mirror setup, so in fact because I use a colo server, the snapshot
solution has no real disadvantages to a mirror. Extra disk usage is
minimal and the bandwidth limitation makes sure I don't change too much
at once. I'm not much of a tag control freak, thankfully.

I keep 8 daily and 8 weekly snapshots (I can go back 8 weeks).

The latest snapshot was 126G and disk space used by all snapshots is
131G, so in practice it's working pretty well for me.

> With a simple mirror you only need as much space as the original. You
> could buy two 400 GB drives and when the first is nearly full you won't
> be running out of space on the other used for backup. Yes, you have to
> be very careful to not mirror a corrupted library, or one where files
> have been deleted.
>

For me backups have to be automatic, and I'd be in grave danger with
such a solution. Keep in mind that I don't backup just the music, but
important documents, digital pictures and home movies as well.

Regards,
Peter

peter
2007-03-22, 02:51
hickinbottoms wrote:
> If you have a server that doesn't run Windows, and want automatic
> off-site backups, then I recommend Box Backup:
>
> http://bbdev.fluffy.co.uk/trac/wiki
>
> Secure, encrypted, incremental backups to a remote location. Keeps old
> and deleted versions of files (until the store is full, in which case
> it starts removing the oldest 'old versions'), can run continuously and
> is very bandwidth efficient.
>
> The big advantage is that the server can be untrusted - it cannot
> decrypt the backups it holds as the clients keep the keys and encrypt
> the traffic before uploading. Perfect for backing up to a friends house
> or another server you can't trust.
>
> It's saved my bacon a few times over the last couple of years.
>
> And it's free and open source. Note that although the server can't be
> Windows-based, clients can be. One server can also keep separate backup
> stores for any number of clients.
>

That looks very interesting. In fact I've tried to talk a colleague into
writing something like this quite some time ago. The encryption part
that allows backups to be stored on a friends server without privacy
issues is very cool and was part of my idea as well.

Thanks for the link.

Regards,
Peter

darrenyeats
2007-07-13, 03:35
And with unison, it's twice as scary. Unison is basically a two-way rsync so if the backup somehow gets destroyed those changes will be propagated back to the main storage.

Yes I should have mentioned that originally! However, I am happy to carry on using unison until I find something better.

I see unison as a replacement for disk mirroring, and in that context it works well. I never write to the backup disk (other than indirectly via unison) nor do I have any reason to, so changes being propagated back isn't a worry.
Darren

PS: This post was edited because, initially, I made some comments about rsnapshot in error. I will look into rsnapshot more carefully and if it suits me better than unison I will change to rsnapshot. Regards, D.

Robin Bowes
2007-07-13, 08:12
darrenyeats wrote:

> From what I've read from your link, rsnapshot isn't what I want. It
> seems rsnapshot is great for, erm, snapshots. But snapshots aren't
> backups.
>
> I want backups for when a disk fails, and it seems rsnapshot, by
> relying on hard links, is not intended to propagate data outside of the
> original filesystem. If my disk fails I lose the original data and all
> the snapshots, which is kind of disappointing! If I have got this wrong
> then, firstly, I apologise, and secondly, I am sure you will put me
> right :-)

You have got it wrong!

The "r" in rsnapshot stands for remote.

The basic idea is that the first run of the tool makes a copy of your
data, then subsequent runs use hard links to files that haven't changed
and only copy over files that have changed meaning that the total backup
size is much smaller.

HTH,

R.

kolding
2007-07-13, 14:57
I, personally, have a rather complicated strategy.
I rip all my CD's to FLAC for archival purposes, but do to disk space costs, I keep my online storage in MP3.

I store my MP3's on a RAID array, and backup weekly to a second hard drive in an external moveable enclosure.

My FLAC's I currently keep on an external hard drive, which I backup to DVD. I then keep this collection of DVD's offsite (ok, at my desk at work).

I threw together a couple scripts to manage backing stuff up to optical discs. These are aimed at systems which are write-mostly, ie, you keep adding new content without really updating what's already there.

My scripts are at:
http://koldware.com/SlimStuff/MusicUtils/makeBackupSet
and
http://koldware.com/SlimStuff/MusicUtils/makeISOs

The former script maintains a database of files, and allocates the files to various DVD's. The latter script takes that database and creates the individual ISO images that you can use to burn the physical DVD's.

Using the makeBackupSet script, you can grow the database, and it will add to the end of the backup set.

Sample usage would be:
makeBackupSet -db /etc/MusicBackup.db -k -p -f -v /music/FLAC
This will create (or extend) the database in /etc/MusicBackup.db.
It will attempt to keep all the files in a directory together (-k), pack directories into the disks it's creating (-p), fill the last previously created disk (-f) and tell you about it all (-v).

You can then run makeISOs on this DB:
makeISOs -d /tmp/FlacBackup -i ~/tmp/Backup -m -s -c -p 79-80
which will generate ISO images of discs 79 and 80, call tmp ~/tmp/Backup.79 and ~/tmp/Backup.80.

They're kindof poorly documented, your best bet is to read the code for help. Maybe I'll update the -h screens.

darrenyeats
2007-07-16, 02:02
You have got it wrong!


Curse that send button! Er, thank you.

I think I'm going to stick with unison for the time being because what I'm after is a kind of mirror function (without using RAID) rather than a backup tool per se. Space is (going to be) tight on both the disks.

Perhaps I can configure rsnapshot to act like a mirror? In which case it will be slightly better than unison - as someone pointed out, it removes even the theoretical chance of propagating changes in the wrong direction. So, if I configured rsnapshot to keep only one generation of backup, would it remove / rename files for me like unison? And never use more storage than the original files?

Thanks, Darren

Robin Bowes
2007-07-16, 04:46
darrenyeats wrote:
> Robin Bowes;214209 Wrote:
>> You have got it wrong!
>>
>
> Curse that send button! Er, thank you.
>
> I think I'm going to stick with unison for the time being because what
> I'm after is a kind of mirror function (without using RAID) rather than
> a backup tool per se. Space is (going to be) tight on both the disks.
>
> Perhaps I can configure rsnapshot to act like a mirror? In which case
> it will be slightly better than unison - as someone pointed out, it
> removes even the theoretical chance of propagating changes in the wrong
> direction. So, if I configured rsnapshot to keep only one generation of
> backup, would it remove / rename files for me like unison? And never
> use more storage than the original files?

If you don't want the snapshots, you just need rsync, not rsnapshot. Or
unison will work fine too.

R.

dalar
2007-07-20, 04:55
I Set up a file server in the basement with 1.6 TB of disk space, of which 900GB is on a RAID5 system. Music and other important docs goes on the RAID file system. Also in the process of setting up a 100 mbps ethernet network with my neighbor (squeezebox owner as well) he wants to use my slimserver, instead of duplicating my setup. When our houses are networked, I think I will setup a NAS at his house for some offsite storage. It cost a bit more than I liked, but it has taken me way to much time to burn my entire CD catalog, just to lose it "WHEN" a hard drive fails.

Cheers,

Excellent Idea... I tell my family and friends the same thing... your hardrive *WILL* fail, but you will hardly ever know when - so backup at least once a month!

Ashy72
2007-07-20, 06:07
I use a separate disk in my main pc just for music/photo's etc then back up to an external usb drive.
I built a hard drive into an external usb enclosure, it came from maplins about 20 as i recall. I comes with sync quick software which allows you to mirror two different drives or even just individual folders it can be set up to synch automatically at any time you want. It does not delete files though and if you delete a file on the source and forget to delete on the backup the file re-appears on the source disk. This is ok for me though as i'd rather make sure its me that does any deleting. I never leave the external disk powered up very long so as to minimise the chance of virus attacks.

zano65
2007-07-20, 10:14
hello, i've been reading this interesting thread from the first post, here's my small contribution. Some of you guys know much more than i do and i think that if i should follow some advice given here ( scripts for mirroring or else) i would loose all my collection before i clearly understand what's going on. So here's my way of doing:
when i have ripped or downloaded about 4.4 GB of data, i burn it to a DVD-R (100 dvds= 440 GB for 50 Euros) and put it in the HDD used for playing the music. When this HDD is full, i make a copy on a usb drive to save me the pain of using the dvds. I don't need incremental backup because i don't change anything in the files. When i begun, 8 years ago, all i used was MP3, and saved on CD-Rs. A HDD died later and i had to reload a hundred CD-Rs to a new disk, it was a little boring but i did not lose any music, and it reinforced my feeling that HDDs are not to be trusted, even if nothing bad has happened since.
Jean

Audiotic
2007-07-20, 11:15
I currently have 2 HDD's in my PC. I have 2 large important folder trees, one with my music and on with my photo's. Currently i am manually copying everything between drives, but this is becoming a bit of a chore and often i forget.

So what software people here use and would recommend for me?

I don't want anything fancy, i.e. compression is needed as obviously its already compressed. Just a simple solution with folder syncing and preset times (over night).

Oh, and free would be nice too :D

Entirely free won't work... I have my music every night back-upped (actually, synced/incremental, only changes are written) with a free utility* to a second USB external HD. And best of all - I have a 3rd HD safe at a friend (he somehow didn't mind getting it) far away from home, that we sync every other month or so. External HD's are very cheap these days - a 500GB is < 120 euro, 320GB < 90. That is a good investment for your valuable music you bought, and spent countless hours ripping!!

*: http://www.lacie.com/silverkeeper/ - Mac only, but they have a PC version too, if it is as good as the Mac version it's great: http://www.lacie.com/download/drivers/LaCie1Click_1_5_2378_setup.exe