PDA

View Full Version : lesson Learned with RAID 5 Implementation



sfraser
2007-01-22, 10:06
Hi, I thought I would post this, it might save others some grief.
Last year (XMAS 2005) I built a new PC platform which became the new household server (Slimserver as well). Santa Clause brought me a thermaltake Tower case and I purchased 3200+ AMD chip, new motherboard etc. I was seriously considering purchasing a RAID 5 Controller to connect the 4x300G SATA drives I picked up during the boxing day sales, however the Hardware controllers were quite expensive, so I decided to implement it the "cheap way" and purchased a ASUS motherboard with a built in Silicon Image controller with RAID capabilities. Installation went fine, Gigabit LAN between the server and the HTPC (EAC ripping platform) Fast Ethernet and wifi everywhere else throughout the house connecting the families pc's, laptops printers and 4 squeezebox's. 900 Gig RAID5 file volume for WAV files, FLAC files and important family pictures, documents etc. I encourged the family to store important doc's on the RAID5 volume. Everything was fine until early Nov 2006, when a power cable (splitter) which had a faulty ground pin similtanously powered down 2 of the 4 hardrives within the RAID volume. The drives were instantly "orphaned" and the RAID volume was no more. This effectively deleted 5-6 years of family vacation pictures, and 450 Gig of ripped CD's (WAV format). I shuddered at the thought of the time involved to re-rip the 600 CD's. Also the "minister of domestic affairs" stated that I shall recover the familly pic's regardless of the cost. I checked data recovery services, but all started at several thousand dollars, I called ASUS, they were no help, I called Silicon Image, their technical support basically told me to "piss up a rope" (thanks guys, I will remember that when making future purchases). I was reasonably sure the data was still on the disks, but how to access it? Neither ASUS or SI bundled any RAID recovery utilities with the motherboard software. I eventually came across a company called Runtime Software http://www.runtime.org/ who sell utilities to recover files off corrupted drives etc. They also had a utility for retrieving files off of damaged RAID5 volumes. In order to use the software they need to create a duplicate image of the RAID vol. that they would work from. Therefore I needed approx. 850G of disk space. So I purchased a Buffalo Terrabit station (NAS). The SI controller built into the motherboard would not represent the 4 SATA drives independantly, therefore i had to purchase a Promise 4 port SATA controller. Once the duplicate image is created from the original damaged RAID vol., Runtime need enough disk space to copy the recovered data to from the copied Image, therefore I required another 500G drive. Eventually I recovered all my data. I now have the original 4x300g SATA drives configured as JBOD on the promise controller. The JBOD is backed up to the NAS nightly using a filesync application. I have run 300' of direct burial CAT5 cable to my nieghbours house, I will be placing the NAS at his place, and in turn he can connect his squeezebox to my Slimserver over the network. I am also considering a power conditioner for the server. So that is my little story on how a 30 cent power lead cost me $1300 and a month with no Slimserver!

Lessons learned, don't go cheap with RAID5, especially if you are putting all your eggs in that one basket. Thumbs up to Runtime software, and there support. Thumbs down to SI and ther elack of support.

P.S. for the Network Engineers out there, my neighbour and I are thinking of also connecting our WIFI routers together and Running OSPF ECMP over the wifi and physical ethernet. Nothing like some more diversity, in most cases you are worried about the "random back hoe" taking down your primary link....in our case it's the "random garden hoe" :)

Cheers,

Scott

cliveb
2007-01-22, 10:34
Two comments:

1. RAID is no substitute for backups. In my day job, I support a database we maintain for a large national organisation. Back in 2002, a RAID controller (Adaptec's top of the line model, not a cheapie) went faulty and scribbled over ALL of the disks. So no matter how advanced your RAID setup, you can't rely on it never to lose your precious data.

2. Some friendly advice: try to format your posts so they don't look like a "stream of consciousness". It makes them much easier to read.

mherger
2007-01-22, 10:49
> 1. RAID is no substitute for backups.

Your right. IMHO raid is overrated for most private uses. It's sold as if
it were backup, but it's only about _availability_, reduced downtime etc.
- which is quite unimportant to most home users.

--

Michael

-----------------------------------------------------------------
http://www.herger.net/SlimCD - your SlimServer on a CD
http://www.herger.net/slim - AlbumReview, Biography, MusicInfoSCR

peter
2007-01-22, 10:51
sfraser wrote:
> Lessons learned, don't go cheap with RAID5, especially if you are
> putting all your eggs in that one basket. Thumbs up to Runtime
> software, and there support. Thumbs down to SI and ther elack of
> support.
>

Thanks for sharing that, I'm sre it will save someone. I'll bet
someone's making a backup to DVD right now ;)

But it sounds like your data is still in danger. What happens if you or
one of your family make a mistake and delete the five years of family photos
and *fail to notice*. Wouldn't the NAS backup faithfully overwrite the
previous backup and delete the photos?

Or do you have some kind of snapshot/incremental system in place? I
decided I was going to suffer something catastrophic sooner or later so
I rented a colo location, put a Linux server there and run rsnapshot
against it every night. If I screw something up, I'll have 7 weeks to
notice it. No RAID at home for me. Up to the minute file integrity is
not too important here.

Regards,
Peter

Mitch Harding
2007-01-22, 11:00
I use RAID 5 at home, primarily to avoid problems from failing hard drives.
In the 6 years I've had a RAID 5 setup, I've had one drive fail. Just being
able to swap in a new drive and have it running again was worth it for me.

I agree backups are also needed, preferably incremental backups with
versioning.

On 1/22/07, Michael Herger <slim (AT) herger (DOT) net> wrote:
>
> > 1. RAID is no substitute for backups.
>
> Your right. IMHO raid is overrated for most private uses. It's sold as if
> it were backup, but it's only about _availability_, reduced downtime etc.
> - which is quite unimportant to most home users.
>
> --
>
> Michael
>
> -----------------------------------------------------------------
> http://www.herger.net/SlimCD - your SlimServer on a CD
> http://www.herger.net/slim - AlbumReview, Biography, MusicInfoSCR
>

Mark Miksis
2007-01-22, 11:26
RAID is no substitute for backups.

I agree too.

RAID will save you from a drive failure. Backups are needed to protect from user error (rm -rf /) or some other hardware failure (RAM, drive controller, etc) that causes bad data to be written to the arrray.

Offsite backups are best as they will also protect from fire or other catastrophic failure. If you have to keep them on site, use a different machine. If you really want to backup to the same machine, use a different drive on a different controller.

JJZolx
2007-01-22, 11:35
But it sounds like your data is still in danger. What happens if you or
one of your family make a mistake and delete the five years of family photos
and *fail to notice*. Wouldn't the NAS backup faithfully overwrite the
previous backup and delete the photos?

My thoughts as well. Automated disk/folder syncs are a kind of poor-man's backup plan and are subject to failures exactly like this. The problem is that this type of backup only guards against hardware failure and fails to guard against file corruption and human error.

If you're going to have two backups done via sync, don't have more than one of them done automatically on a daily basis. Do the other manually only after you're certain the data is complete and non-corrupted. That one probably only needs to be done once or twice a month and be a safeguard against something happening to both the original and the backup. It should probably also be kept offsite to guard against fire or theft.

MrD
2007-01-22, 12:03
You may also want to electrically isolate yourself from your neighbor's power. Ground loops or power spikes could come across that CAT-5 and zap your equipment and visa-versa.

Not sure though, how this would be done with wired ethernet, perhaps there are some transformer based solutions.

Maybe you could run fiber between houses :)

-MrD

sfraser
2007-01-22, 12:36
My thoughts as well. Automated disk/folder syncs are a kind of poor-man's backup plan and are subject to failures exactly like this. The problem is that this type of backup only guards against hardware failure and fails to guard against file corruption and human error.

If you're going to have two backups done via sync, don't have more than one of them done automatically on a daily basis. Do the other manually only after you're certain the data is complete and non-corrupted. That one probably only needs to be done once or twice a month and be a safeguard against something happening to both the original and the backup. It should probably also be kept offsite to guard against fire or theft.



The software backup software has the option not to "match" deletions. From time to time I will run it manually list the delta's and then match deletions.

Thanks for the input and suggestions..

Scott

mustbemad
2007-01-22, 13:34
Hey Guys

Interesting thread... I was in the process of splitting my single Linux box into two linux boxes, so I could at least back up one to another nightly... When the primary disk in it went North.

Good News: My Slimserver Library is on the good disk & I just backed up 20 Gigs of digital photos to DVD...
Bad News: The other disk had another 60Meg of everything else I had ever wanted to keep on it: Videos, Kids' GCSE course work & other life-limiting-if-you-lose-it data...

Anyone know any decent software which might be able to analyse & extract data from the bad disk? It seems mechanically OK, but has lost ALL the Superblock backups & possibly more. I've tried the usual CLI utils to no avail and a free prog called TestDisk which wasn't able to do anything for me.

Any ideas anyone?

Cheers

ftlight
2007-01-22, 13:44
mustbemad wrote:

> Anyone know any decent software which might be able to analyse &
> extract data from the bad disk? It seems mechanically OK, but has lost
> ALL the Superblock backups & possibly more. I've tried the usual CLI
> utils to no avail and a free prog called TestDisk which wasn't able to
> do anything for me.
>
> Any ideas anyone?

I can't help with the recovery, but before you mess with a single bit of
the content, best to clone the drive to a backup. I've been using an
excellent bootable command line bit-level disk copier which works with
any OS and drive type:

http://fbim.fh-regensburg.de/~feyrer/g4u/

--
Bill Burns
Long Island NY USA
http://ftldesign.com

peter
2007-01-22, 13:49
On Mon, 22 Jan 2007 15:44:22 -0500, "Bill Burns" <billb (AT) ftldesign (DOT) com>
said:
>
> I can't help with the recovery, but before you mess with a single bit of
> the content, best to clone the drive to a backup. I've been using an
> excellent bootable command line bit-level disk copier which works with
> any OS and drive type:
>
> http://fbim.fh-regensburg.de/~feyrer/g4u/

On Linux (which I assume he's running) that's just:

cat /dev/hda | gzip > /tmp/image.gz ;)

On the down side I find recovering deleted files under Linux a lot more
problematic than under Windows.

Regards,
Peter

bklaas
2007-01-22, 13:50
Hey Guys

Interesting thread... I was in the process of splitting my single Linux box into two linux boxes, so I could at least back up one to another nightly... When the primary disk in it went North.

Good News: My Slimserver Library is on the good disk & I just backed up 20 Gigs of digital photos to DVD...
Bad News: The other disk had another 60Meg of everything else I had ever wanted to keep on it: Videos, Kids' GCSE course work & other life-limiting-if-you-lose-it data...

Anyone know any decent software which might be able to analyse & extract data from the bad disk? It seems mechanically OK, but has lost ALL the Superblock backups & possibly more. I've tried the usual CLI utils to no avail and a free prog called TestDisk which wasn't able to do anything for me.

Any ideas anyone?

Cheers

There are commercial software packages that do this, even with Linux FS's, and I believe they tend to run in the $50-$100 range. I don't know how well they work. If that fails, you might try a professional data recovery service. Two excellent ones are Ontrack (ontrack.com) and Drive Savers (drivesavers.com). Downside is that these are very expensive. I'd expext $500-$2K for data recovery.

One other comment-- why would you need to setup two linux boxes for drive backup purposes? I use rsnapshot (basically an elegant wrapper over rsync) to backup one drive to another drive in the same machine daily. It works beautifully. Incidentally, you can use rsnapshot to back up via ssh to remote machines as well.

http://rsnapshot.org

#!/ben

note: I'm surprised people keep thinking RAID is for backup, when pretty much any thread or doc you read on RAID states fervently in the first paragraph "RAID IS NOT A BACKUP SYSTEM".

mherger
2007-01-22, 14:05
> http://rsnapshot.org

I love it myself :-). And I'm also using the
second-disk-mounted-read-only-during-the-day approach. Additionally I
started rsyncing the daily.0 to a remote machine...

> note: I'm surprised people keep thinking RAID is for backup, when
> pretty much any thread or doc you read on RAID states fervently in the
> first paragraph "RAID IS NOT A BACKUP SYSTEM".

That's not written on that colorful, fancy Asus/Gigabyte/Younameit
motherboard box.

--

Michael

-----------------------------------------------------------------
http://www.herger.net/SlimCD - your SlimServer on a CD
http://www.herger.net/slim - AlbumReview, Biography, MusicInfoSCR

bklaas
2007-01-22, 14:17
>
That's not written on that colorful, fancy Asus/Gigabyte/Younameit
motherboard box.


Glossy Marketing Text = Gospel ;)

this is a very nice quick read on what RAID can't do for you:
http://en.wikipedia.org/wiki/RAID#What_RAID_Cannot_Do

mustbemad
2007-01-22, 14:34
why would you need to setup two linux boxes for drive backup purposes?

Cos if I'm using one machine & the power supply catches fire - or something similar - I'd potentially lose both disks. With two - the house my burn down, but maybe I'd still have data in one place!

Thanks for the replys guys! I am running Linux - Debian on one machine, and Xebian on the other - a hacked Xbox - fun but a tad slow for snappy page loads on Slimserver.

I think I'll source another big disk & clone to that before I mess about with it in any other ways...

bklaas
2007-01-22, 14:45
Cos if I'm using one machine & the power supply catches fire - or something similar - I'd potentially lose both disks. With two - the house my burn down, but maybe I'd still have data in one place!

Thanks for the replys guys! I am running Linux - Debian on one machine, and Xebian on the other - a hacked Xbox - fun but a tad slow for snappy page loads on Slimserver.

I think I'll source another big disk & clone to that before I mess about with it in any other ways...

Far be it from me to discourage someone from hacking an Xbox...I'm running XBMC on a hacked xbox myself and couldn't be happier with the results.

If you're happy with your power bills from running a couple servers 24/7, what you are doing should work just fine (my recommendation of rsnapshot would still apply well here). You're going to draw less power if you keep your drive backups to one machine though.

I don't have real data for this, but I'm willing to bet that power supply fires happen a few orders of magnitude less frequently than HD failures. In the context of drive backups, it's just not something I worry about.

Also FWIW, truly important data needs to periodically get written to something less error-prone than HDs. For me, "truly important data" = digital pics and my music library. I do a less frequent backup to DVD-Rs (~ quarterly) that I physically store at a friend's house. It's not that I'm worried that my house is going to burn down, but if it were to catch ablaze, I'd like to avoid the inevitable moronic thought "Honey, get the kids! I have to run downstairs and grab my server!" ;)

#!/ben

Pale Blue Ego
2007-01-22, 20:35
Here's a couple things I do that have saved me a lot of grief.

First, anything that goes on the "file server" (music, photos, documents, family videos) gets backed up to DVD-R first. I also keep extra DVD-R copies of irreplaceable stuff like family photos in a safe deposit box.

The file server gets mirrored to an external USB drive after additions or changes. I don't keep the external drive connected except during mirror operations. I store it in a closet away from the computer room.

Also, the stuff on the file server is kept "read-only" except during specific times, mainly when I am updating tags.

Since the server is a Linux system, I take the extra precaution of making the files "immutable":

chattr -R +i /mnt/hdc1/music

This makes it so even root can't screw up and delete them.

When I want to add files or update file tags, I reverse the immutable attribute:

chattr -R -i /mnt/hdc1/music

Actually, I've created aliases, "lock_music", "unlock_music", "lock_photos", etc. so I can lock/unlock different parts of the server as needed.

mustbemad
2007-01-23, 03:18
Interesting point - I had never quantified the actual cost of leaving a server on - so here are some live measurements & costs per month given my electricity supplier's rate of GBP 0.11 per KWh:

Xbox with one HD ~ 260mA ~ GBP 5
Libretto 70CT (weather station) ~ 38mA ~ GBP 0.72!
Vaio Laptop ~ 100mA ~ GBP 1.90

Take your point about frequency of server fires... I may pension off the Xbox...

Cheers.

Mustbemad

BTW, [OT] the weather & stuff from the Libretto can be found at www.mustbemad.com

mustbemad
2007-01-23, 14:50
Found a piece of software from Disk Internals Research called Linux Reader - free - seemed too good to be true!

http://www.diskinternals.com/

It mounted my disk on a Windows XP machine via USB, and reads every file on it without any trouble. Just extracting all the data now.

This is a great bit of software - highly reccomended!


Mustbemad

exile
2007-01-23, 17:03
an unfortunate way to learn about the vulnerability of RAID.

my two cents: I'm a television editor and the use of raids for storing and quickly accessing massive amounts of video/audio media is commonplace. what's also commonplace is Raids failing. Every company I've ever worked for has gone through failed raid arrays.

In television, the media on the raid drives is always expendable and therefore we don't back it up. the source tapes can always be re-digitized if something goes wrong with the computer. but with personal media it's very different.

It sounds dull but being paranoid and contantly backing up personal media is a must for peace of mind and to minimize the headaches when those drives eventually fail you.

peter
2007-01-24, 01:51
mustbemad wrote:
> Found a piece of software from Disk Internals Research called Linux
> Reader - free - seemed too good to be true!
>
> http://www.diskinternals.com/
>
> It mounted my disk on a Windows XP machine via USB, and reads every
> file on it without any trouble. Just extracting all the data now.
>

AFAICS it's just a ext2/ext3 filesystem reader, right? No obvious
recovery tools. I believe there are two files ystem drivers that just
allow you to read & write ext2/ext3 partitions under Linux. I've used
one in the past (forgot which) and it seemed to work OK.

Regards,
Peter