PDA

View Full Version : 8 TB Storage and more...



jimzak
2013-12-17, 04:32
I'm currently using a laptop with 4 external 2 TB drives for storage of 215,000 Flac and MP3 files.

Things are a bit unwieldy and backup is time consuming. Each drive takes 12-24 hours to backup over USB 2.0.

I need to add another drive soon as I am running over about 7 TB of actual files.

Any ideas as to what would be a more efficient system for storage and backup (and possibly the server)???

I've thought about getting a custom-built server computer with at least 10-15 TB of internal storage and backup via a USB 3.0 external hard dock.

The hard drives would be of the slower energy efficient variety and I would specify an energy efficient processor.

I'm not crazy about a NAS because of the vagaries of running LMS on a nonstandard OS and a very low-power processor.

Let me know your thoughts.

Happy Holidaze.

DJanGo
2013-12-17, 06:08
Hi,

you must decide which way you wanna die...

~138 MB/s is the fastest SATA Harddisk thats aviable for Money - if you wanna go higher you have to chosse SSD.
But 9 TB made of a bunch of SSD is a bit expensive?

Small Backuptimes and real Big Disks are not possible.
For this Case a good Networker always uses identical Storage Systems that rsync permanetly to solve that "Problem".

Permantent Rsyncing and energy efficient processor is boring cause of a energy efficient processor is capable to minimize the Power if its not needed - but rsyncing needs Power.

SO which way you wanna die and what size of Money you wanna spend?

JohnSwenson
2013-12-17, 12:04
What I did was get an aluminum case (the one I bought is not made any more, but there are others) with a "wind tunnel" of drive mounts with a fan at both ends of the tunnel to keep drives cool. A medium type motherboard (no need for super graphics etc). I put vortexbox software on this and it became the "server" for the house. I left it on 24/7, the wind tunnel in the case kept the drives cool for very long life. It sat in an out of the way spot in the spare bedroom. It does make some noise, but really not very much.

For backups you have some choices. The easiest is just build another system the same as the first and rsync over ethernet between them. This gives you a good local backup. Then do a rotating backup off the secondary on external drives stored offsite. Since you are essentially building your own NAS, you can put whatever hardware interfaces you want on it such as multiple USB3 ports, whatever. The backup server does not have to run 24/7, just enough to handle the rsync, which is probably not much once a full copy has been done.

John S.

w3wilkes
2013-12-17, 12:37
I use Windows Home Server 2011 with a product called DriveBender and specify duplication for my music folder. DriveBender maintains duplicate files on separate drives on the same server. This way I only have offsite backup to address.

DriveBender takes a group of drives and makes it appear as a single drive to the OS and applications. If the server itself crashes I can just read the drives on another PC as native drives without additional software. With the duplication specified for the music folder I'm protected from a individual drive failure.

Just my 2˘

Wirrunna
2013-12-17, 14:21
As DJanGo asks, "SO which way you wanna die and what size of Money you wanna spend? "

What works for me is a master / slave two server combination. Both systems run Win 7/64
The master is an old Antec P182 case with a quad core Intel chip and a bunch of 2GB drives, 2 for music, one for video and one for TV series. All new music is ripped, tagged, processed by MusicIP etc and stored on the master. New versions of LMS 7.8 are also run there. This system is powered down when not in use.

The slave is a mini-ITX system with a low power AMD motherboard and 2 2TB drives. Power draw during a rescan is about 35 watts, normal LMS playing power draw is well below that. This system starts with Wake on Lan and is shut down by the Server Power Control plugin. It lives under the 40" TV and plays Video and DVDs when needed.

The music library is kept in sync using Syncback - http://www.2brightsparks.com/syncback/ . Backup is also done by Syncback and Seagate USB 2TB Desktop Expansion Drives.

I suggest that a dedicated server will be better for you than utilising a laptop that is surplus to requirements. You could keep using the laptop as the Master like I do for processing new music and use the dedicated server as the slave. I would replace the 2TB USB drives with Seagate 4TB drives.

Now the reading list - This case looks good - http://www.silentpcreview.com/Fractal_Design_Node_304, pick a motherboard from - http://www.tomshardware.com/reviews/z87-motherboard-roundup,3656.html, disks from - http://www.storagereview.com/wd_black_4tb_desktop_hard_drive_review_wd4003fzex .

All the decisions - Single system or Master / Slave, then Operating system, CPU, motherboard (my next one will be a mini-ITX with an onboard mSATA 120GB SSD for drive C), HDD size, HDD speed (and therefore power draw) await you.

Happy reading.

matka
2013-12-17, 14:47
I'm currently using a laptop with 4 external 2 TB drives for storage of 215,000 Flac and MP3 files.

Things are a bit unwieldy and backup is time consuming. Each drive takes 12-24 hours to backup over USB 2.0.

I need to add another drive soon as I am running over about 7 TB of actual files.

Any ideas as to what would be a more efficient system for storage and backup (and possibly the server)???

I've thought about getting a custom-built server computer with at least 10-15 TB of internal storage and backup via a USB 3.0 external hard dock.

The hard drives would be of the slower energy efficient variety and I would specify an energy efficient processor.

I'm not crazy about a NAS because of the vagaries of running LMS on a nonstandard OS and a very low-power processor.

Let me know your thoughts.

Happy Holidaze.
I have 6TB but not all is taken as I prune video rather quickly, however my setup might be of interest to you. I've got Mediasonic Raid box that can take up to 4 drives, I know they have enclosures that can take up to 8. In a common raid 5 configuration with 3TB drives that will give you potentially 20TB give or take. This enclosures have their own raid controller and connect to a PC either via usb3 or eSata. I have my 6TB connected to a simple Acer nettop which serves it to the network as a NFS storage via gigabyte (it could be served via Samba for Windows). You do not need a lot of power to run network storage. My slim server runs on another server (with more horse power) so scans and directory browsing is snappy. Acer nettop runs linux.

Mediasonic Raid enclosure that I have does not provide great write speeds after a while when data is fragmented among the drives, but after initial backup, rsync should (or the other canned programs that do the same) relatively quickly sync your backup and main storage as it only goes after the differences and not a mirror backup.

You could use Mediasonic as your backup device, I'm using it as a main storage server and have rsync backups to docked drives via usb2. Would like to purchase another Mediasonic just for the backups but I'm trying to economize.

Good luck and how many hours do you need to listen to all that flac that you have ? :-)

Edit - I just checked newegg user comments, "Cannot use RAID5 using all 8 disks, only option is to use RAID 50 which is 2 sets RAID 5 in RAID 0.". Still an option though.

Pascal Hibon
2013-12-17, 15:49
I would recommend a good NAS for such storage needs. USB drives are probably the worst storage means, especially for large storage needs; and 8 TB is huge.
A decent NAS has plenty of power these days, technology has improved a lot.
There are different ways to backup a NAS and the great part is that they can be automated and running during the night. The first time you backup the entire 8 TB will take a long time but consecutive backups will be very fast. Rsync will be your friend.
I always recommend ReadyNAS; they are packed with great features (such as fast and easy storage growth) and they provide great value for the money.
If I were you I would have a look at the ReadyNAS 314.

DJanGo
2013-12-17, 17:24
Edit - I just checked newegg user comments, "Cannot use RAID5 using all 8 disks, only option is to use RAID 50 which is 2 sets RAID 5 in RAID 0.". Still an option though.

Raid is just another thing...
All my Raid Adapters (did i wrote that my job is managing Servers and Networks already?) have more Power than any lightweight or called here more efficient system Processor just for the XOR doing things thats needed for a raid.
Another issue why "energy efficient" is just a buzz word.

He should get a NAS with Red WD Disks.

And just again...
You cant backup a bunch of disks on one disk - if they are all the same size.
So Backup and and "Production" System / Disks have to match each other (usualy the Backupdisk should be bigger)

JJZolx
2013-12-17, 19:38
A headless Linux or Windows server. For Windows, it doesn't need to be a server OS - Windows 7 Pro would be my recommendation if you go that route, with native Remote Desktop Server rather than using VNC. If you'd like some disk pooling and perhaps disk redundancy, I would use flexRAID (http://www.flexraid.com/) (on Windows) instead of hardware RAID. More flexible, easier to manage, and MUCH less prone to user f***ups when you lose a drive.

For backup, a mirror computer system. It could have identical specs as the server above, then could be used in its place if the first system fails. Or, it could have lesser specs to save a few dollars and be used solely for backup. With that much data, a significant percentage of the cost will be in the hard drives themselves, so building a second system for backup should be reasonable. Make sure you have a gigabit network, as you'll be doing backups across the network. The time to do backups shouldn't be a concern any more, as you won't be swapping drives.

erland
2013-12-17, 21:45
I'm currently using a laptop with 4 external 2 TB drives for storage of 215,000 Flac and MP3 files.

Things are a bit unwieldy and backup is time consuming. Each drive takes 12-24 hours to backup over USB 2.0.

I need to add another drive soon as I am running over about 7 TB of actual files.

Any ideas as to what would be a more efficient system for storage and backup (and possibly the server)???

Sorry if I'm stating the obvious, but my general thoughts are that you should separate:
1. LMS server
2. Storage
3. Backup

These three things have very different needs.

For LMS server, you want to use a computer for best performance. It can be a small, silent computer or a old used cheap computer, both these alternatives are generally significantly better than running LMS on a NAS. Even if the NAS is powerful today, you will eventually end up in a situation where you need to purchase a complete new NAS just because you need more CPU or memory and this gets expensive.

For storage, a NAS is perfect, it's extensible and it's easy to add more storage when you need it. Just make sure you select one with enough drive slots so it's easy to extend it with more storage as you need. I would personally warn you a bit regarding using RAID, with RAID you typically need exactly the same hard drives in all slots and while this is easy when you initially set it up, it can be hard to purchase new drives later since the ones you purchased initially might no longer be sold at that time. Look for JBOD support, which most NAS boxes support, that makes it possible to expose the drives as individual drives and you can add drives of different types to the NAS box when you need more storage. RAID sounds good, but it does NOT replace backup as some people seems to think. Also, pretty much all times I've been involved in a setup with RAID where a disk has crashed it has result in that the system has become more or less useless during the time it takes to re-build the RAID when you replace a crashed disk. It might be that the setups I've seen has been setup the wrong way or have incapable RAID support, but I just want to warn you a bit about it so you don't think RAID solves all the problems. If you are willing to purchase all the storage initially, using RAID can make a bit more sense because then you can always make sure to purchase exactly the same drives, just be a aware that you have to be a bit selective when you select the hard drives because some have issues in RAID setups.

For backup you want to use something different than the original NAS. Main reason for this is that you don't want the backup to be in the same box as you main storage in case of fire or power malfunctions as it can easily destroy both your main storage and your backup at the same time. It can be a good idea to use a NAS also for backup if you need a lot of storage, but separate USB/e-SATA drives is also a solution. It's important that you have an incremental backup scheme so you don't need to backup everything every time, backing up just the changes will be a lot faster and then it's also feasible to do the backup over the network or a slower USB connection.

My personal setup consists of:
- A Linux based server computer running LMS in a VirtualBox virtual machine
- A QNAP 409 Pro NAS storing the music files, drives configured as single drives (JBOD), mounted via NFS
- An external USB drive where the music files are backed up using an rsync based backup solution(dirvish) automatically every night

Pascal Hibon
2013-12-17, 23:33
For storage, a NAS is perfect, it's extensible and it's easy to add more storage when you need it. Just make sure you select one with enough drive slots so it's easy to extend it with more storage as you need. I would personally warn you a bit regarding using RAID, with RAID you typically need exactly the same hard drives in all slots and while this is easy when you initially set it up, it can be hard to purchase new drives later since the ones you purchased initially might no longer be sold at that time. Look for JBOD support, which most NAS boxes support, that makes it possible to expose the drives as individual drives and you can add drives of different types to the NAS box when you need more storage. RAID sounds good, but it does NOT replace backup as some people seems to think. Also, pretty much all times I've been involved in a setup with RAID where a disk has crashed it has result in that the system has become more or less useless during the time it takes to re-build the RAID when you replace a crashed disk. It might be that the setups I've seen has been setup the wrong way or have incapable RAID support, but I just want to warn you a bit about it so you don't think RAID solves all the problems. If you are willing to purchase all the storage initially, using RAID can make a bit more sense because then you can always make sure to purchase exactly the same drives, just be a aware that you have to be a bit selective when you select the hard drives because some have issues in RAID setups.


With a decent NAS there is no need to use all the same hard drives. That's also a thing of the past. Of course, you do need to look at the NAS manufacturers hardware compatibility list when selecting a suitable hard drive. But in general WD Red hard drives are a good choice (and supported by most decent NAS's).
I personally would recommend using RAID for your primary NAS because that gives you drive redundancy; if one drive fails you don't loose data. Where you do loose data with JBOD. I would only use JBOD in a second NAS that you use to backup your primary NAS.

JohnB
2013-12-18, 03:17
The way I handle things is that I use a HP Microserver running Ubuntu for storage. I don't use RAID or any volume spanning. I then periodically mirror each HDD in the HP Microserver to a separate external backup HDD (formatted as NTFS) of the same size (2TB) using an eSata connection. As only the changes are copied over this doesn't take that much time.

If anything goes wrong I can always access the data on the backup HDDs from Windows.

Not perhaps the most elegant solution but it suits me.

aspendl828
2013-12-18, 09:23
Another HP Microserver user. This time using a Windows 8 Storage Space to manage the data disks giving me parity redundancy - effectively RAID 5. I know I don't really need it but it was fun setting it up!
Backup each month or following major changes / additions to usb drives using synctoy. I have flac, flac24 and mp3 tracks in separate directory trees so I have 3 synctoy pairs.
Backup drive kept offsite. Currently offsite means the boot of my car!

JohnB
2013-12-18, 10:00
Backup each month or following major changes / additions to usb drives using synctoy.

I tried using synctoy but found it to be one of the most frustrating pieces of software I have encountered - it just didn't do what I wanted, and what it did do it did to its own "rules".

In the end I got BeyondCompare which is great for mirroring, etc, etc, etc. Lots of options and it works flawlessly.

aspendl828
2013-12-18, 10:21
I think my only issue with Synctoy was initially caused by me not fully understanding differences between the echo, syncronise and the other option (at work and can't remember what the other one is). After that it works fine. Although, obviously there are plenty of other solutions.
I now find Echo works best for me in the Backup situation.

Otto-Wilhelm
2013-12-18, 13:37
My music collection, currently nearly 80000 files, mainly flac files, is quite static. Only new files or files which have been changed, are copied for being backed up, which indeed is only a sync between different hard discs in my server. I use a tool called AllSync (http://www.allsync.de) which works fine. Such a backup run doesn´t take very long.

Additional I back up to internet (Livedrive.com, only EUR 13/month for unlimited backup for 5 PCs, runs in the background). As soon as available again, I might additionally or alternatively back up to internet via Online Cloud Backup www.audiosafe.com (http://forum.dbpoweramp.com/forumdisplay.php?59-AudioSafe, runs in the background, had been free of charge, only downloading for restoring a lost music collection was charged).

The music collection is also synced to a NAS in my local network, as a second local backup and for being the music collection of a second LMS on this Synology NAS for having also a "backup LMS server".

According to my sync and backup schedules I could deal without any problems with a substantially greater music collection e.g. of 300000 flac files.

pgnyc
2013-12-18, 14:09
HI,
you can also take a look at UNRAID. <http://lime-technology.com/>

for short , unraid , is a raid but with one parity disk and your data disk .
in case of problem , you can take the data disk and read them in any linux.
the data disk are equal of less than the size of the parity disk.
unraid is free for 3 disks ( so you can test)

i use it with a HP microserver N54L , for my music, photo &video.

Pascal

banned for life
2013-12-19, 09:37
We had a Black Armor NAS go tits up while rebuilding the array. Most of it was backed-up. Some of the folders had not been backed-up for a couple of months but it was better than a total loss.

The replacement is two Synology DS1513+'s that are separated by about half a mile and connected with 1Gbps fiber. They are configured for Raid-6 and using High Availability, they constantly mirror each other. They also support hot spares if you are made of money but we just went with 5 each 3gb WD Red drives.

The only issues have occurred when a switch got fried. This results in "split brain" which required about a day and a half to rebuild the passive server.

BFL

Two phrases can guide YOUR life:
Don't fart around.
Sign me up.

JJZolx
2013-12-19, 10:07
We had a Black Armor NAS go tits up while rebuilding the array. Most of it was backed-up. Some of the folders had not been backed-up for a couple of months but it was better than a total loss.

I've tried to convince some technically semi-literate friends that if they buy a NAS and use RAID there is a very good chance that when (not if) they lose a hard drive, they or the hardware or the software are going to f**k thingS up before or during the rebuilding process to make that array unrecoverable. In most cases it will be user error. I also try to convince them of the need for backups, but some see that RAID array as the solution to their problems. Two of them lost arrays completely, and one had no backup and lost everything.


The replacement is two Synology DS1513+'s that are separated by about half a mile and connected with 1Gbps fiber. They are configured for Raid-6 and using High Availability, they constantly mirror each other. They also support hot spares if you are made of money but we just went with 5 each 3gb WD Red drives.

Constantly mirroring is both senseless and dangerous. If you consider this to be a 'backup', then it completely defeats the purpose of having a backup. If you accidentally delete something, or the files are inadvertently changed or corrupted or, then the deletions, unintended changes and corruptions are then mirrored to the second system. If you're using mirroring for backup, you should only do backups manually and you should do something to assure yourself that the original library is healthy before performing a backup. For a music library, you can get a file count to guard against deletions, and if the files are encoded in FLAC, you can run a test across the library to be sure files are not corrupted. You can also often run mirroring software with a test switch to first see how many files will be copied.

erland
2013-12-19, 15:03
I've tried to convince some technically semi-literate friends that if they buy a NAS and use RAID there is a very good chance that when (not if) they lose a hard drive, they or the hardware or the software are going to f**k thingS up before or during the rebuilding process to make that array unrecoverable. In most cases it will be user error. I also try to convince them of the need for backups, but some see that RAID array as the solution to their problems. Two of them lost arrays completely, and one had no backup and lost everything.

It's nice to hear that I'm not alone with this kind of experience, I started think that my experience with people recovering a RAID system and ending up in a scenario where the recovery takes too long or fails was a rare exception.

As you say, it's probably often user errors of people doing something wrong during the recovery process or a system not configured properly, but still.



Constantly mirroring is both senseless and dangerous. If you consider this to be a 'backup', then it completely defeats the purpose of having a backup. If you accidentally delete something, or the files are inadvertently changed or corrupted or, then the deletions, unintended changes and corruptions are then mirrored to the second system. If you're using mirroring for backup, you should only do backups manually and you should do something to assure yourself that the original library is healthy before performing a backup. For a music library, you can get a file count to guard against deletions, and if the files are encoded in FLAC, you can run a test across the library to be sure files are not corrupted. You can also often run mirroring software with a test switch to first see how many files will be copied.

Totally agree, a good backup should either be based on a consistenty check before the backup or doing backups where historical versions are kept. Without this it's very easy to accidentally end up with a a non working backup that's useless when something happens with your primary storage.

JohnSwenson
2013-12-20, 12:18
It's nice to hear that I'm not alone with this kind of experience, I started think that my experience with people recovering a RAID system and ending up in a scenario where the recovery takes too long or fails was a rare exception.



I had a RAID-5 array for many years, using linux software RAID. That array lasted through 3 computers, 4 operating systems and two bad disks, but one day it went bad and would not recover no matter what I did. But I DID keep full backups (the last of which had just happened two days before) so I was able to completely recover everything. Even with the array messed up I could still get the data off the array, but it was not functioning like it should. (but that was unnecessary because of the backup).

But this backing up to hard drives is still so much better than "in the old days" using tapes to do backup. I had two different tape systems and was diligent in making backups. But the two times I had a total disk meltdown and needed to do a restore the tapes would not read on the same drive that wrote them. A complete waste of time and money.

John S.

Mark Miksis
2013-12-20, 12:34
For anyone considering cloud backup, I'm very happy with crashplan.com. It's about $60 per year for unlimited storage. The initial backup can take quite a while depending on how much data you have, but from then on it just updates itself incrementally. Upload speed seems to be limited only by my own cable upload speeds. I chose them over other cloud based backup services because of unlimited storage and Linux support.

jimzak
2013-12-22, 08:05
I've been watching all the great replies to my question.

This is what I have come up with:

storage: Mediasonic 8-bay hard drive enclosure (USB-3/eSATA) with four to eight 3-4 TB hard drives installed

http://www.amazon.com/Mediasonic-H82-SU3S2-ProBox-External-Enclosure/dp/B005GYDMYG/ref=sr_1_2?ie=UTF8&qid=1387724400&sr=8-2&keywords=mediasonic

backup: either a unit identical to above or use the USB-3 or eSATA port to backup incrementally to a single external USB-3/eSATA hard drive dock

server: same laptop as I am currently using

I like using the single hard drive dock as backup because I can keep the backups off-site more easily. The Mediasonic unit with a bunch of hard drives looks to be one of the cheaper options for my situation.

If you have further comments or suggestions, let me know.

Happy Holidazed.

Jim

adhawkins
2013-12-22, 13:27
In article <Mark.Miksis.66nwtz (AT) no-mx (DOT) forums.slimdevices.com>,
Mark Miksis<Mark.Miksis.66nwtz (AT) no-mx (DOT) forums.slimdevices.com> wrote:
>
> For anyone considering cloud backup, I'm very happy with crashplan.com.
> It's about $60 per year for unlimited storage. The initial backup can
> take quite a while depending on how much data you have, but from then on
> it just updates itself incrementally. Upload speed seems to be limited
> only by my own cable upload speeds. I chose them over other cloud based
> backup services because of unlimited storage and Linux support.

Seconded. I have the 'family' subscription, that means I can update multiple
computers.

Occasionally have issues with the (java based) software, but their support
are generally pretty good at sorting things out.

Andy

jimzak
2013-12-22, 14:04
For anyone considering cloud backup, I'm very happy with crashplan.com. It's about $60 per year for unlimited storage. The initial backup can take quite a while depending on how much data you have, but from then on it just updates itself incrementally. Upload speed seems to be limited only by my own cable upload speeds. I chose them over other cloud based backup services because of unlimited storage and Linux support.

I wonder if backing up 7 TB of files is even feasible? Seeing that even over USB 2, a 2TB drive takes 12-24 hours, I doubt it.

Mark Miksis
2013-12-22, 14:09
I wonder if backing up 7 TB of files is even feasible? Seeing that even over USB 2, a 2TB drive takes 12-24 hours, I doubt it.

crashplan.com offers an option where you can ship them a drive for the initial backup at an extra cost. Otherwise, yeah it can take quite a while. Either way, once the initial backup is done, it's kept up to date incrementally which seems to work fine for my ~2.5 TB.

JJZolx
2013-12-22, 15:08
I wonder if backing up 7 TB of files is even feasible? Seeing that even over USB 2, a 2TB drive takes 12-24 hours, I doubt it.

Backup to the cloud? Not likely.

But backup to disk, why not? It's not like you're breaking rocks while that backup is happening. You're off doing something else. The data is (or should be) nearly static, so the initial full backup might take 24 hours, but subsequent backups will only be backing up a few gigabytes and take minutes. The majority of the time taken isn't even going to be in transferring files across the USB/eSATA/ethernet interface ... it's going to be spent comparing the hundreds of thousands of files' timestamps and sizes.

matka
2013-12-23, 07:35
storage: Mediasonic 8-bay hard drive enclosure (USB-3/eSATA) with four to eight 3-4 TB hard drives installed

If you have further comments or suggestions, let me know.

Happy Holidazed.

Jim
You might want to check mediasonic user forums, I believe you can upgrade firmware on these boxes and this should improve performance. I have also seen posts about people reverting the firmware to older version if they did not want drives to spin down. Ideally you should do that before you have actual data on the devices.

adhawkins
2013-12-23, 13:37
Hi,

In article <Mark.Miksis.66rqin (AT) no-mx (DOT) forums.slimdevices.com>,
Mark Miksis<Mark.Miksis.66rqin (AT) no-mx (DOT) forums.slimdevices.com> wrote:
> crashplan.com offers an option where you can ship them a drive for the
> initial backup at an extra cost.

Not sure of your location, but I believe this option is only available in
the US.

Andy

audiomuze
2013-12-23, 14:50
HP Microserver running Ubuntu server performs brilliantly as LMS host and audio library store for libraries the size you've referred to. Backup to another Microserver using rsync and keep a 3rd offsite for safekeeping. Quickly becomes pricy, but rather spend the money than lose the files and have to rerip and tag.

banned for life
2013-12-24, 09:55
I've tried to convince some technically semi-literate friends that if they buy a NAS and use RAID there is a very good chance that when (not if) they lose a hard drive, they or the hardware or the software are going to f**k thingS up before or during the rebuilding process to make that array unrecoverable. In most cases it will be user error. I also try to convince them of the need for backups, but some see that RAID array as the solution to their problems. Two of them lost arrays completely, and one had no backup and lost everything.



Constantly mirroring is both senseless and dangerous. If you consider this to be a 'backup', then it completely defeats the purpose of having a backup. If you accidentally delete something, or the files are inadvertently changed or corrupted or, then the deletions, unintended changes and corruptions are then mirrored to the second system. If you're using mirroring for backup, you should only do backups manually and you should do something to assure yourself that the original library is healthy before performing a backup. For a music library, you can get a file count to guard against deletions, and if the files are encoded in FLAC, you can run a test across the library to be sure files are not corrupted. You can also often run mirroring software with a test switch to first see how many files will be copied.

In this case constant mirroring provides instant fallback operation should 2 drives in one enclosure fail. I couldn't possibly consider that either senseless or dangerous. Perhaps I did not make it clear that this is TWO NAS enclosures and that the mirroring provides a CONSTANT backup.

Unlike most Linux-based home-office NAS the Synology also support Recycle Bin on a folder by folder basis and there is also a setting that can restrict Recycle Bin access to administrators only.

I agree that it is not a matter of if a drive will fail but when. The Black Armor was killed by software problems. Even before the first disk failure, FTP had the flakes-- instead of "GOODBYE" it returned "DIRECTORY SUCCESSFULLY CHANGED"

JJZolx
2013-12-24, 11:22
In this case constant mirroring provides instant fallback operation should 2 drives in one enclosure fail. I couldn't possibly consider that either senseless or dangerous. Perhaps I did not make it clear that this is TWO NAS enclosures and that the mirroring provides a CONSTANT backup.

Yes, that's clear and exactly why it's so dangerous. Yes, a constant backup is the only way to ensure that you don't lose a day or a week or a month's updates, but I consider that to be a very small amount of work to duplicate considering the potential danger. You can always backup more often if you consider it too risky. You're essentially implementing another level of RAID, protecting against an increasingly unlikely catastrophic event, rather than a backup solution.

jimzak
2013-12-26, 06:13
Thanks again to all who offered me help.

On a related note, what type of hard drive should I get for my new enclosure situation?

Should I get a "Red" NAS specific drive or go with perhaps cheaper "Green" drives that offer energy savings with lower RPM (the "Red" drives do too)?

Here's a sample "Red" drive on sale at Tiger.com:

http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=3580092&CatId=139

Here's a sample "Green" drive:

http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=940885&CatId=4357

Any thoughts on this? I realize a slower drive may also backup slower but for the main purpose of these drives, serving music files, a fast HD is probably not needed.

Thanks. Hope you're having a decent holiday season.

Jim

Pascal Hibon
2013-12-26, 06:26
Thanks again to all who offered me help.

On a related note, what type of hard drive should I get for my new enclosure situation?

Should I get a "Red" NAS specific drive or go with perhaps cheaper "Green" drives that offer energy savings with lower RPM (the "Red" drives do too)?

Here's a sample "Red" drive on sale at Tiger.com:

http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=3580092&CatId=139

Here's a sample "Green" drive:

http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=940885&CatId=4357

Any thoughts on this? I realize a slower drive may also backup slower but for the main purpose of these drives, serving music files, a fast HD is probably not needed.

Thanks. Hope you're having a decent holiday season.

Jim

That would depend on your personal preferences but I would go for WD Red. That drive can spin at higher speeds which makes the drive more robust.

jimzak
2013-12-29, 09:33
I'm currently copying all the files over to the new enclosure. I have about 2 TB more to go. I've been copying files for about 48 hours now.

I got two 3 TB Red drives on sale, and I already had 4 other drives lying around. That will give me plenty of room to grow.

Now I need an incremental backup software application. I have used Macrium in the past for disk images.

Does Windows 7 do incremental backup?

Suggestions?

I'll be backing up to an external hard drive dock over USB 3 or eSATA.

Thanks again.

Pascal Hibon
2013-12-29, 09:56
Now I need an incremental backup software application. I have used Macrium in the past for disk images.

Does Windows 7 do incremental backup?

Suggestions?


On Windows I use the free edition of SynckBack (called SynckBackFree (http://www.2brightsparks.com/download-syncbackfree.html))
Even the free edition has great features and it does incremental backup. It is possible to create a profile for your backups and create a schedule so that they run automatically.

matka
2013-12-30, 11:46
I'm currently copying all the files over to the new enclosure. I have about 2 TB more to go. I've been copying files for about 48 hours now.

I got two 3 TB Red drives on sale, and I already had 4 other drives lying around. That will give me plenty of room to grow.

Now I need an incremental backup software application. I have used Macrium in the past for disk images.

Does Windows 7 do incremental backup?

Suggestions?

I'll be backing up to an external hard drive dock over USB 3 or eSATA.

Thanks again.
I'm on linux and use rsync. It is a great utility that can do incremental backups among other things and it is for free. I believe rsync was ported to Windows but I do not have a personal experience using it on this environment.

jimzak
2013-12-31, 03:31
Mission accomplished (except for backup)!

It's not hugely different from my previous arrangement in some ways:

USB connection: 3.0 vs 2.0
4 hard drives (so far): single enclosure vs separate external drives

But the difference is huge in terms of simplicity. My media closet now has 4 less power blocks plugged in and 3 less USB cables.

Backup will be simpler with incremental backups as suggested in this thread and faster via USB 3.0.

The enclosure still has 4 empty slots for mega-future-expansion.

Total Tracks: 218,888
Total Albums: 21,887
Total Artists: 17,047
Total Genres: 1,122
Total Playing Time: 27085:46:22

Media Scan Details
Discovering files/directories: L: (48342 of 48342) Complete 00:02:31
Scanning new music files: L: (43549 of 43549) Complete 00:16:56
Discovering files/directories: M: (67636 of 67636) Complete 00:04:59
Scanning new music files: M: (60866 of 60866) Complete 00:34:20
Discovering files/directories: N: (48752 of 48752) Complete 00:02:28
Scanning new music files: N: (43738 of 43738) Complete 00:33:09
Discovering files/directories: O: (79242 of 79242) Complete 00:05:34
Scanning new music files: O: (72106 of 72106) Complete 01:03:12
Discovering playlists: C:\Users\Music\Playlists (1 of 1) Complete 00:00:00
Pre-caching Artwork (21659 of 21659) Complete 00:18:39

The server has finished scanning your media library.
Total Time: 03:01:48 (Monday, December 30, 2013 / 10:49 PM)

Thanks to all who helped and I hope everyone has a happy, musical New Year.

jimzak
2014-01-18, 13:03
USB 3.0 is a bit tricky for this enclosure/notebook combo.

I enlarged the total storage to four 3TB drives, two of which are NAS type.

Apparently the enclosure had not been connecting at USB 3.0 speeds which slowed down transfers internally between drives as I rearranged the drives, replaced some 2 TB drives, and tried out backup software.

To get USB 3.0 speeds, I had to unplug my USB 2.0 mouse and start the enclosure up already plugged into the notebook computer. By doing so I quadrupled transfer speeds to the backup hard drive enclosure and internally.

It also seems to have accelerated the scanning process by 17%.

Total Tracks: 225,657
Total Albums: 22,559
Total Artists: 17,502
Total Genres: 1,169
Total Playing Time: 28487:39:31

Media Scan Details
Discovering files/directories: L: (49350 of 49350) Complete 00:00:55
Scanning new music files: L: (44439 of 44439) Complete 00:12:27
Discovering files/directories: M: (69478 of 69478) Complete 00:01:57
Scanning new music files: M: (62477 of 62477) Complete 00:27:16
Discovering files/directories: N: (50384 of 50384) Complete 00:01:36
Scanning new music files: N: (45185 of 45185) Complete 00:29:06
Discovering files/directories: O: (82356 of 82356) Complete 00:02:09
Scanning new music files: O: (75000 of 75000) Complete 00:56:21
Discovering playlists: C:\Users\Music\Playlists (1 of 1) Complete 00:00:00
Pre-caching Artwork (22392 of 22392) Complete 00:18:44

The server has finished scanning your media library.

get.amped
2014-01-25, 06:54
I'm currently copying all the files over to the new enclosure. I have about 2 TB more to go. I've been copying files for about 48 hours now.

I got two 3 TB Red drives on sale, and I already had 4 other drives lying around. That will give me plenty of room to grow.

Now I need an incremental backup software application. I have used Macrium in the past for disk images.

Does Windows 7 do incremental backup?

Suggestions?

I'll be backing up to an external hard drive dock over USB 3 or eSATA.

Thanks again.

If you are just looking to replicate your directory structures, robocopy works well. Use the /MIR flag to mirror entire source directories to the destination. I have 3 x 3TB internal drives in my main system and another 3 x 3TB external drives connected to a laptop I use for mobile/remote use. As I rip new CDs to the main system, I use robocopy .CMD files to update the drives connected to the laptop across my network. Since it only copies changes (including deletions of lossy tracks being replaced by lossless ones), it doesn't take long to keep everything in sync.

menno
2014-01-25, 15:53
I'm on linux and use rsync. It is a great utility that can do incremental backups among other things and it is for free. I believe rsync was ported to Windows but I do not have a personal experience using it on this environment.

I recommend looking into rdiff-backup, which combines the best features of a mirror and incremental backups (uses librsync). Also runs on windows.

I run it on my Dockstar, with two USB disks plugged in. Means I have a full mirror and incremental backup history. Also means I can plugin a 3rd drive occasionally to backup another mirror for offsite storage. And I can just plugin bigger drives when I need/as they get more affordable.

My Dockstar also runs LMS and my Calibre ebook library via COPS. I prefer this setup as it means I can have one tiny low powered device on 24/7 for all these tasks.

jimzak
2014-03-02, 03:23
The Mediasonic hard drive enclosure has really helped decrease the complexity of my server. However, it has not been without a few glitches. The main one being that it's firmware is finicky about USB 3. Initially the enclosure only connected to the server laptop at USB 2 speeds which really makes transfers and backup exponentially longer.

I updated the laptop BIOS and found one setting for USB 3 that could be varied (I cannot remember what it was, and I don't want to reboot the laptop to find out what that setting was because it's a shoutcast server also). I changed that setting and since that time the enclosure has connected consistently at USB 3. The setting change was not intuitive. I just go lucky.

I have been using a USB 3 single hard drive docking station as a backup device, and I adopted "Backup Maker" (https://www.ascomp.de/en/) as as my backup tool for the following reasons:

1. Wizard setup of saved backup profiles for each drive
2. Incremental backup
3. Freeware (with a nag at the end of each backup.)

So once the complete backup of each drive was completed, subsequent incremental backups take only a few minutes once a week.

I'm also implementing a new server computer as per another thread which has led to whole new series of technical challenges (lesson so far: don't buy HP laptops).

Jim

matka
2014-03-04, 08:53
The Mediasonic hard drive enclosure has really helped decrease the complexity of my server. However, it has not been without a few glitches. The main one being that it's firmware is finicky about USB 3. Initially the enclosure only connected to the server laptop at USB 2 speeds which really makes transfers and backup exponentially longer.

I updated the laptop BIOS and found one setting for USB 3 that could be varied (I cannot remember what it was, and I don't want to reboot the laptop to find out what that setting was because it's a shoutcast server also). I changed that setting and since that time the enclosure has connected consistently at USB 3. The setting change was not intuitive. I just go lucky.

I have been using a USB 3 single hard drive docking station as a backup device, and I adopted "Backup Maker" (https://www.ascomp.de/en/) as as my backup tool for the following reasons:

1. Wizard setup of saved backup profiles for each drive
2. Incremental backup
3. Freeware (with a nag at the end of each backup.)

So once the complete backup of each drive was completed, subsequent incremental backups take only a few minutes once a week.

I'm also implementing a new server computer as per another thread which has led to whole new series of technical challenges (lesson so far: don't buy HP laptops).

Jim
Thanks for an update.
Did you consider eSata ? Speeds are compatible and it is an older technology, probably more stable. I'm using eSata on mine without any driver issues (all linux).

jimzak
2014-03-08, 15:24
Thanks for an update.
Did you consider eSata ? Speeds are compatible and it is an older technology, probably more stable. I'm using eSata on mine without any driver issues (all linux).

The USB 3.0 connection gives me almost identical speeds to the eSATA. After I changed the BIOS on the i5 server, USB 3.0 worked fine.

I just connected a new Haswell i7 laptop as the music server. I left my shoutcast for now on the i5 computer.

The i7 server scanned the entire music library 33% faster.

I gave up on the SSD upgrade. HP makes it too difficult. I don't need it anyway.

Reconnecting all the players to the new server has been a bit problematic but having been with SB for 6 years now, I know the tricks.:cool:

jimzak
2014-03-30, 14:24
...and now for some uncool news.

The Mediasonic 8-bay enclosure has died a premature death today during a server reboot. I've contacted the mfr for a RMA.

I'm back to the 4 separate external drives for now, minus the 3 months of new music that I have put on the drives in the enclosure, which I am assuming are fine.

Such is life in the First World.

J

castalla
2014-03-30, 17:35
...and now for some uncool news.

Such is life in the First World.

J

What a bummer compared with having to search daily for water, fuel, food - such is life in the 3rd World.

jimzak
2014-03-30, 18:02
What a bummer compared with having to search daily for water, fuel, food - such is life in the 3rd World.

Exactly what I was referring to. I didn't have to have tree bark for dinner.

Otto-Wilhelm
2014-04-05, 14:44
...and now for some uncool news.

The Mediasonic 8-bay enclosure has died a premature death today during a server reboot. I've contacted the mfr for a RMA.

I'm back to the 4 separate external drives for now, minus the 3 months of new music that I have put on the drives in the enclosure, which I am assuming are fine.

Such is life in the First World.

J

If you want switch to different hardware you might try Sharkoon´s 8-Bay RAID-Station
see http://www.sharkoon.com/?q=de/content/8-bay-raid-station
or Sharkoon´s 5-Bay RAID-Station
see http://www.sharkoon.com/?q=de/content/5-bay-raid-station

jimzak
2014-04-27, 11:21
If you want switch to different hardware you might try Sharkoon´s 8-Bay RAID-Station
see http://www.sharkoon.com/?q=de/content/8-bay-raid-station
or Sharkoon´s 5-Bay RAID-Station
see http://www.sharkoon.com/?q=de/content/5-bay-raid-station

Thanks for the suggestion. I'm definitely looking at alternatives if this new unit from Mediasonic fails like the fist one did.

Another side note: when the enclosure was offline, I fiddled with my home ethernet network. Usually fiddling leads to problems; however, something I did boosted my network transfers by a factor of 8-10x. For years I had "put up with" transfers over my network at about 13 mbps. The network now allows transfer speeds of 80-100 mbps. About the only thing I can think of is that I rebooted an ethernet switch.

Here's to fiddling!!!