PDA

View Full Version : 6 SATA Hard Drives full of FLAC files - need suggestions



jimzak
2018-12-27, 11:41
I started a home music system in 2009 based on Squeezebox. I am still VERY happy with it.

Gradually my library has expanded to approx 12 TB of mainly FLAC files.

Currently the music files are spread over six 3 TB hard drives that are housed in an Medisonic USB 3.0 hard drive enclosure:

Mediasonic H82-SU3S2 ProBox 8 Bay 3.5" Hard Drive External Hard Drive Enclosure - USB 3.0 & eSATA

https://www.amazon.com/dp/B005GYDMYG/ref=pe_385040_3033...

It has been in service since 2013 and seems to have been a pretty good solution. I had to replace one early due to complete failure but it was under warranty.

The hard drive enclosure is connected to a dedicated NUC computer which runs the Squeezebox server software and keeps the music indexed.

HOWEVER, recently I notice hashes mismatches when transferring files to the discs.

When I transfer files to the Mediasonic enclosure, I get the mismatches; when I transfer files to any other attached USB drive, there are no mismatches (I use Teracopy to verify the file transfers)

I read in a couple of reviews that the problem with these units lies in its jmicron controller.

So I have a few questions:

1. Should I just replace the obviously problematic unit? or,
2. Should I consider other options?

Other options might be a NAS but I really want something faster such as my NUC because I use that computer as a Shoutcast server and as a Madsonic server which allows me to access my music outside of the house via phone apps or other computers. I doubt a NAS would be able to run all this software and give reasonable performance.

I suppose I could build a computer with 6 drives but that would obviously be pretty expensive and not sure how many cases will hold 6-8 drives.

Interesting to note that I have not had a drive failure in 5 years of continuous use. I use NAS drives and I back them up regularly to an external hard drive dock, one at a time.

I'm not interested in other music streaming options than Squeezebox as I have a number of still functional Squeezebox players including a Touch, a couple of Booms, and a Radio. Additionally I have built several picoreplayers.

Thanks for reading this and helping if you can.

Jim

DJanGo
2018-12-27, 13:00
12 TB thats pretty bad...

..because i run a wd red 10TB Disk for small cash.
and a single drive fits proper in a single disk case. but 12 TB doesnt fit in the 8.8TB drive labeld as 10TB.

Since your able to backup one drive after another i dont think you're using any raid just a bunch of disks?

teracopy looks like you running Windows. Rsync can handle a bandwith limit but the rsync windows version cant handle UTF8.
So rsync isnt a good idea...

drmatt
2018-12-27, 13:09
I would definitely look to stay ahead of your storage requirements by buying fewer larger drives. You can get four 8tb without having to ride the bleeding edge of price or performance and have two copies and room to spare; then you're looking at a simple 4 bay pc like the hp microserver or similar.

With a suitable Linux distro you can run two stripe pairs with a hourly rsync for replication. Add encryption and you're covered.

Remember every spinning HDD costs about £20 a year in power alone so put that money into fewer larger drives to offset the cost. Not to mention the higher likelihood of failure from having many drives.

Transcoded from Matt's brain by Tapatalk

Roland0
2018-12-27, 17:27
As someone who also experienced data corruption issues with a jmicron-based enclosure, I'd advise against using any type of consumer-grade HW based solution of that kind.

If you want a (moderately) high-performance NAS, building your own is imo the best solution (I've been using a similar setup for the past ~6 years).
Cases are not that expensive: overview (https://geizhals.at/?cat=gehatx&xf=536_4%7E550_2+-+ITX%2FDTX&sort=p#productlist)
Intel's Gemini Lake SoC mainboards (same as in the current NUCs) are fairly cheap and only need 6-20W: overview (https://geizhals.at/?cat=mbson&xf=11832_Gemini+Lake%7E317_(SoC)%7E3760_Intel%7E44 00_Mini-ITX&sort=p#productlist)
Add a power supply and RAM and you're done (if you really need >4 drives, add a SATA card)

There are a number of open source NAS OSes (FreeNAS, Open Media Vault, ...) available as well.

bakker_be
2018-12-28, 03:59
As someone who also experienced data corruption issues with a jmicron-based enclosure, I'd advise against using any type of consumer-grade HW based solution of that kind.

If you want a (moderately) high-performance NAS, building your own is imo the best solution (I've been using a similar setup for the past ~6 years).
Cases are not that expensive: overview (https://geizhals.at/?cat=gehatx&xf=536_4%7E550_2+-+ITX%2FDTX&sort=p#productlist)
Intel's Gemini Lake SoC mainboards (same as in the current NUCs) are fairly cheap and only need 6-20W: overview (https://geizhals.at/?cat=mbson&xf=11832_Gemini+Lake%7E317_(SoC)%7E3760_Intel%7E44 00_Mini-ITX&sort=p#productlist)
Add a power supply and RAM and you're done (if you really need >4 drives, add a SATA card)

There are a number of open source NAS OSes (FreeNAS, Open Media Vault, ...) available as well.

Given the needs of the OP, I'd advise NOT to go towards one of the NAS OS'es, good as they are. In a similar setup I've been using Ubuntu Server with ZFS (https://wiki.ubuntu.com/ZFS) to my complete satisfaction: one "small" OS disk, 8x 4TB disks in Raid-z1. Using Ubuntu (or another generic Linux flavour) gives the advantage of more documented solutions for just about everything you'd like to do, which is especially useful when it comes to multimedia ...

d6jg
2018-12-28, 05:54
I'd opt for a high grade NAS (raided) to store files but keep your NUC as the server. Its easy to mount said NAS via NFS and you won't see any performance loss at all.

jimzak
2018-12-28, 09:26
I'd opt for a high grade NAS (raided) to store files but keep your NUC as the server. Its easy to mount said NAS via NFS and you won't see any performance loss at all.

Coincidentally, I have an unused DS1815+ that I just put into service for the first time.

I have mounted the "test" drive as NFS.

I am assuming that I will be able to "see" this folder/drive in Squeezebox server and via Madsonic's add folders?

Thanks for a great suggestion.

Additionally, I will implement 6 or 8 TB drives (thanks for the suggestion above) in some sort of RAID to ensure that drive failure will not be catastrophic.

Unfortunately buying 4-6 high capacity NAS drives will NOT come cheaply.

drmatt
2018-12-28, 11:37
Unfortunately buying 4-6 high capacity NAS drives will NOT come cheaply.

No but you will be surprised to find how much you can get back when you sell your old drives. This will be enough to take some of the pain away... In truth I'd be surprised if you don't get at least half the cost of a 6tb back by selling two 3tb drives. Add in a lifetime of five years at twenty quid a year electricity saving and there you go, cost neutral..

(What can I say, I'm adept at justifying upgrades to myself..)


Transcoded from Matt's brain by Tapatalk

d6jg
2018-12-28, 15:25
Coincidentally, I have an unused DS1815+ that I just put into service for the first time.

I have mounted the "test" drive as NFS.

I am assuming that I will be able to "see" this folder/drive in Squeezebox server and via Madsonic's add folders?

Thanks for a great suggestion.

Additionally, I will implement 6 or 8 TB drives (thanks for the suggestion above) in some sort of RAID to ensure that drive failure will not be catastrophic.

Unfortunately buying 4-6 high capacity NAS drives will NOT come cheaply.

To answer your specific question - yes LMS should be able to see your music via nfs Mount.
Remember the playlist folder needs to be writable.
Permissions are the only real issue operating in this mode but once you understand whatís going on with users etc and why then itís a piece of ....
I have a Vortexbox 2.4 running on an HP54 with a single SSD but which reads and writes to my 4 bay QNAP.
I donít run subsonic or mad sonic but I do run both LMS and Plex on the HP with all files stored on NAS.
A Pi running PiCorePlayer with files on NAS is also a realistic way of operating but I found the Piís networking capability to be a bit borked from time to time when operating a number of players in sync and therefore prefer the processing power of a proper PC

jimzak
2018-12-28, 16:19
To answer your specific question - yes LMS should be able to see your music via nfs Mount.
Remember the playlist folder needs to be writable.
Permissions are the only real issue operating in this mode but once you understand whatís going on with users etc and why then itís a piece of ....
I have a Vortexbox 2.4 running on an HP54 with a single SSD but which reads and writes to my 4 bay QNAP.
I donít run subsonic or mad sonic but I do run both LMS and Plex on the HP with all files stored on NAS.
A Pi running PiCorePlayer with files on NAS is also a realistic way of operating but I found the Piís networking capability to be a bit borked from time to time when operating a number of players in sync and therefore prefer the processing power of a proper PC

The web interface doesn't see the NAS but the Control Panel in the Windows tray allows one to put in the login credentials for the NAS and then the NAS is visible.

I just scored a couple of 8 TB NAS drives. I'll wait for a sale, and get two more.

I checked the Madsonic forum and the process is similar, via logging into the NAS via the Madsonic service.

My only concern is that transferring all this music to the NAS will take days but I'm not in a hurry, and I can leave the status quo in place during the transfer process.

d6jg
2018-12-28, 17:10
I didnít realise your NUC was running Windows! It would be a lot faster / more stable if it was Linux.

Make sure your AV doesnít scan the NAS Mount

Roland0
2018-12-29, 01:21
If OP doesn't need Windows, he could simply run everything (including LMS) on the DS1815+ (which has a performance comparable to a mid-range NUC)

bakker_be
2018-12-30, 03:02
My only concern is that transferring all this music to the NAS will take days but I'm not in a hurry, and I can leave the status quo in place during the transfer process.

Your NAS support USB file transfers, just plug the drives into it ...

jimzak
2019-01-06, 06:42
I thought long and hard about all the suggestions above.

I ended up with an inertial solution.

I got the 4 bay version of the problematic hard drive enclosure for less than 100 bucks. I tested it for hash mismatches and there were none.

I got two 8 TB drives and transferred the 12 TB or so music to these drives and had a few empty TB left over.

I'm going to get two more 8 GB drives to keep backups but I will not put them in the unused bays in the enclosure because I don't want them spinning 24/7 just for backups.

I'll likely purpose the NAS as a household backup system or at some point move the music over.

It's good to have options.

Thanks for everyone's help.

sgmlaw
2019-01-17, 11:04
This may be controversial to the anti-RAID contingent who views audio and video files as Ďstatic dataí, but another option is a quality outboard HW RAID enclosure to the server machine of your choice.

We have a similar library to the OP: rapidly approaching 15TB. We have not just a lot of media, but a lot of hours invested in organizing and tagging it. It is a dynamic data volume, as new media is constantly being added, and old media occasionally updated, retagged, re-coded, or archived as new media and clients come and go.

Weíve kept it (all backed up locally and offsite of course), on a Areca 4-bay TB enclusure for a number of years. This type of enclosure and connection will easily saturate the bus of any host machine, and any ethernet connection that machine is attached to. It is compact, quiet, scalable and expandable, and is fault tolerant. Host machine migration is a snap, you simply move the enclosure TB cable over to the new machine.

The newest version of it is TB3 and USB3 capable, so it will accomodate just about any host machine.

I am not a fan of NAS-based data servers for larger libraries and or with many clients. They require so much processing power to handle large libraries with mixed multiple clients that you are ultimately better off with a freestanding server machine. Smaller libraries or those with fewer clients might pull it off, however. But not in our environment.

d6jg
2019-01-17, 11:21
This may be controversial to the anti-RAID contingent who views audio and video files as Ďstatic dataí, but another option is a quality outboard HW RAID enclosure to the server machine of your choice.

We have a similar library to the OP: rapidly approaching 15TB. We have not just a lot of media, but a lot of hours invested in organizing and tagging it. It is a dynamic data volume, as new media is constantly being added, and old media occasionally updated, retagged, re-coded, or archived as new media and clients come and go.

Weíve kept it (all backed up locally and offsite of course), on a Areca 4-bay TB enclusure for a number of years. This type of enclosure and connection will easily saturate the bus of any host machine, and any ethernet connection that machine is attached to. It is compact, quiet, scalable and expandable, and is fault tolerant. Host machine migration is a snap, you simply move the enclosure TB cable over to the new machine.

The newest version of it is TB3 and USB3 capable, so it will accomodate just about any host machine.

I am not a fan of NAS-based data servers for larger libraries and or with many clients. They require so much processing power to handle large libraries with mixed multiple clients that you are ultimately better off with a freestanding server machine. Smaller libraries or those with fewer clients might pull it off, however. But not in our environment.

I agree but also disagree.

It is best to run the server software on a machine that has the processing power to do it.

As to where the files reside. A NAS is made for file storage and usually has the backup tools built in to make backup painless.

I run LMS on a small but powerful machine with a small SSD that holds only the Linux operating system and the LMS installation. The NAS is mounted by NFS.

I have changed LMS servers many times over the years but the storage has persisted.

sgmlaw
2019-01-18, 15:15
I agree but also disagree.

It is best to run the server software on a machine that has the processing power to do it.

As to where the files reside. A NAS is made for file storage and usually has the backup tools built in to make backup painless.

I run LMS on a small but powerful machine with a small SSD that holds only the Linux operating system and the LMS installation. The NAS is mounted by NFS.

I have changed LMS servers many times over the years but the storage has persisted.

I think we are ultimately saying the same thing. A NAS device is particularly proficient when used primarily as a network data target for other machines. But a NAS is not a server. All but the most powerful models can be quickly overwhelmed when running more resource-intensive server machine applications and services, or forced to host numerous concurrent services serving many clients and targets.

We also have a NAS, but it used strictly as a backup volume target on the network for the other machines and the working data volume. Most data is pushed to it in a non-priority context where QOS or streaming interruption considerations do not arise.

Placing working data for a server machine elsewhere on the network does add additional packet traffic, and creates an additional fault layer should network buffers fail. A local working volume also allows a working drive separate from the machine's boot drive to host high-bandwidth tasks such as file transcoding, saving a write-limited OS SSD from unnecessary wear. A gigabit ethernet connection to a networked target drive is simply incapable of that luxury.

But I can certainly understand those who do use a NAS as a working data volume for a server machine elsewhere on the network, as setting up and maintaining multiple independent arrays and HDDs in a home environment is rather expensive.

d6jg
2019-01-19, 06:13
I think we are ultimately saying the same thing. A NAS device is particularly proficient when used primarily as a network data target for other machines. But a NAS is not a server. All but the most powerful models can be quickly overwhelmed when running more resource-intensive server machine applications and services, or forced to host numerous concurrent services serving many clients and targets.

We also have a NAS, but it used strictly as a backup volume target on the network for the other machines and the working data volume. Most data is pushed to it in a non-priority context where QOS or streaming interruption considerations do not arise.

Placing working data for a server machine elsewhere on the network does add additional packet traffic, and creates an additional fault layer should network buffers fail. A local working volume also allows a working drive separate from the machine's boot drive to host high-bandwidth tasks such as file transcoding, saving a write-limited OS SSD from unnecessary wear. A gigabit ethernet connection to a networked target drive is simply incapable of that luxury.

But I can certainly understand those who do use a NAS as a working data volume for a server machine elsewhere on the network, as setting up and maintaining multiple independent arrays and HDDs in a home environment is rather expensive.

In a LMS on one machine data on another via NFS/CIFS scenario the transcoding (if required) is always going to be done on the LMS machine. If the OS and software are on one drive and the data is on another in the same machine then any transcoding will be done on the OS/LMS drive so I really donít understand your point.
Surely transcoding is precisely the sort of task where an SSD is a good idea.

Streaming is a low level network activity and you would be very hard pressed to choke a gigabit Ethernet connection by reading a music file from one machine to another and then streaming it out again.

People successfully run Pi based LMS with data on NAS. Personally I donít because my experience of running this way was that the Piís network card was its weakest point so I moved to an HP54 with single SSD and all files on NAS but I regularly build new Piís and connect them to the NAS. If every time I wanted to build a new LMS server I had to build new Raid arrays on attached USB enclosure (also a Pi weak spot as the LAN port shares the same bus) then it would be a nightmare.

sgmlaw
2019-01-27, 07:54
In a LMS on one machine data on another via NFS/CIFS scenario the transcoding (if required) is always going to be done on the LMS machine. If the OS and software are on one drive and the data is on another in the same machine then any transcoding will be done on the OS/LMS drive so I really donít understand your point.
Surely transcoding is precisely the sort of task where an SSD is a good idea.

Streaming is a low level network activity and you would be very hard pressed to choke a gigabit Ethernet connection by reading a music file from one machine to another and then streaming it out again.

People successfully run Pi based LMS with data on NAS. Personally I donít because my experience of running this way was that the Piís network card was its weakest point so I moved to an HP54 with single SSD and all files on NAS but I regularly build new Piís and connect them to the NAS. If every time I wanted to build a new LMS server I had to build new Raid arrays on attached USB enclosure (also a Pi weak spot as the LAN port shares the same bus) then it would be a nightmare.

You misunderstood my point about transcoding over a network connection. I was not talking about stream transcoding on the fly through LMS. I was speaking of file transcoding through the server machine using DB Poweramp or other application, with the NAS over ethernet as the host volume. Processing will often stall and plummet in speed in the latter environment. Once you move up to video files, processing can slow to a crawl or fail outright. File transcoding, container conversions, and metadata revisions are much more common in today's larger home A/V libraries than ten or fifteen years ago. These tasks are much more efficiently performed over a wide local bus. And a local TB or USB3 target volume provides that luxury, and avoids write-cycling wear through an SSD OS drive (which are much more perishable in that environment).

I stand by my opinion that a remotely networked working data target can effectively double the network traffic going through the server machine interface, and network health and traffic buffering then becomes more critical to server performance than with a local data volume. With increasing numbers of clients, the added network traffic only increases proportionally. For most smaller client groups, it should not pose too much of an issue if the aggregate traffic does not saturate the effective network bus bandwidth. Otherwise, in a very busy environment, QOS rules and router configuration best practices can become critical.

But again, maintaining multiple networked data arrays is too expensive for most home environments, so I completely understand the widespread practice of using a NAS as a working data volume target.

The OP using a local 6 drive data stack is somewhat excessive even for me, however. And juggling multiple bare drives in an ad hoc rotation for backups would drive me crazy. Which is why I think a more robust prosumer or enterprise grade 4 drive H/W RAID solution a better option for a local data target in that scenario. A well-designed stack will perform automatic file checking and parity checks and corrections as part of regular RAID scrubbing (and perhaps mitigate some of the hash errors the OP is describing with a less sophisticated stack machine). If someone does not want to constantly juggle bare drives in rotation, a further networked and/or cloud back up solution are some other more automated backup alternatives.

Backup strategies are in the eye of the beholder, and how they prioritize and value the particular data. Some treat this type of data as static, and off-line archive drive backups to be sufficient. Others, like me, treat it as dynamic, and deserving of continuous active backup systems with some fault-tolerance. Each has to make their own decisions based on how they manage and value their data.

jonmyatt
2019-02-18, 15:02
Out of interest - how many albums/songs in the 12TB collection?