PDA

View Full Version : Large Backups on a Cloud



mandryka
2017-05-01, 02:10
Hello everyone

Amazon are now offering unlimited storage for £55 p.a in the UK, and they accept lossless music files. So the idea of having a cloud backup is more attractive.

I have 3TB of music, it's going to take forever to upload it via my phone line -- I doubt it's feasible.

Is there a one off fast upload service from anyone? Someone who I could send a hard drive to and they do the upload for me for a fee?

drmatt
2017-05-01, 02:34
Took me about a week to upload 2TB to exactly that service using my BT fibre line. Tedious but doable. And next time the incremental backups are trivia to send up to the cloud.

Amazon can do media-based ingest to S3, but a) you pay for it and b) it's not Drive. Otherwise find a friend with a good upload speed...

Sent from my ONEPLUS A3003 using Tapatalk

mandryka
2017-05-01, 02:53
Oh good, thout sounds promising.

Is there a backup utility? Something that will check for changes and upload them, things like changes to metadata.

drmatt
2017-05-01, 03:20
No, there are no utilities for accessing Drive, I scripted backups to large files using dar on my Linux box then just copied the files to a Windows box to drag and drop to Drive. AFAIK there is no API to access Drive directly yet, so nothing going for automatic updates etc.

bpa
2017-05-01, 03:26
AFAIK there is no API to access Drive directly yet, so nothing going for automatic updates etc.

Does this not work ? https://developers.google.com/drive/v3/web/simple-upload

drmatt
2017-05-01, 04:24
Awesome, that's new. They didn't publish anything like that when I I started using it.

https://developers.google.com/drive/v3/web/resumable-upload

This looks like what I want.. (2GB blobs are impractical to restart from scratch each time.. :) )

bpa
2017-05-01, 04:29
Also some examples of using the API.
https://github.com/googledrive

bakker_be
2017-05-01, 06:45
Ehm, aren't we confusing GOOGLE drive and AMAZON drive here?

Back to Amazon Drive:
rclone (https://rclone.org/) seems to support it, and it has a "check" function that compares file hashes allowing you to skip files that calculate as identical.

mandryka
2017-05-01, 07:25
Amazon have a desktop app which will allow you to synch to a folder on your C drive. It will NOT allow you to synch to a folder on an external hard drive (so no good for me)

I tried to upload via the web interface several times but it crashed for my big "downloads" folder. It worked for one or two small files.

I just started an upload of the whole folder via the windows 10 app and so far so good, will report back.

JohnB
2017-05-01, 08:17
Is there any reason why you prefer to backup to the cloud rather than backing up to external HDDs which are under your control?

mandryka
2017-05-01, 08:40
Is there any reason why you prefer to backup to the cloud rather than backing up to external HDDs which are under your control?

Just fire and theft, that's all. Offsite backups don't seem to work for me.

It's done 100 files (out of 105,000) in the past two hours, and it may not be keeping the structure of the folders, so so far it's a bit worrying really.

drmatt
2017-05-01, 08:51
That is why I went with a virtual tape type thing. Stage the backups to large encrypted files and move the encrypted virtual tapes to the cloud.

Sent from my ONEPLUS A3003 using Tapatalk

drmatt
2017-05-01, 08:51
Ehm, aren't we confusing GOOGLE drive and AMAZON drive here?

Bah, yes you are right. Doh

bakker_be
2017-05-01, 10:22
Just fire and theft, that's all. Offsite backups don't seem to work for me.

It's done 100 files (out of 105,000) in the past two hours, and it may not be keeping the structure of the folders, so so far it's a bit worrying really.

Seems a bit slow to me. I don't know what kind of connection you have of course, but I've started the 3-month free trial, just to get a look at it and with rclone from my Ubuntu Server LMS I have around 3 GB (195 files) of data in around half an hour of toying with it.
Given that traffic between midnight and noon only counts for half with my provider I've scheduled a CRON job (timeout 12h rclone sync -c -v /mnt/music AMAZON_DRIVE:/music), so it doesn't get me "punished" for backing up my data :p rclone works on Windows as well, both 32 and 64 bit versions are available, and using task scheduler you could probably accomplish the same thing. The -c option tells it to do a checksum calculation. To me this has 2 advantages:

It allows me to interrupt the upload process, picking up where it left off
If you set your tagging software to not change file modification times, it still detects that a file has been changed.

mandryka
2017-05-01, 11:18
It's not going to work for me., 400 files in 6 hours, to upload the lot will take about two months.

SlimChances
2017-05-01, 11:59
An external HD in my safety deposit box works for me. It is far cheaper and much larger storage options

drmatt
2017-05-01, 12:26
It's not going to work for me., 400 files in 6 hours, to upload the lot will take about two months.
Depends on your broadband upload speed. Unlimited fibre is the only way this makes sense, that's the only way to get 18-20 Mbit upload. With that, I put 2TB up in a week and two days. If your upload is 2Mbit it will take ten times longer...

Squeezemenicely
2017-05-01, 12:27
I am using Amazon Drive and have about 2 TB music on there, it is great I can use it to stream music on the go using CloudBeats, plus I have an extra Backup of the entire music as an incremental Backup done via my NAS.
Yes it takes long to upload at first, but afterwards it is worth it.

drmatt
2017-05-01, 12:29
An external HD in my safety deposit box works for me. It is far cheaper and much larger storage options
Larger than infinite? And the price? Fifty five quid a year for infinite? Personally I have found that a new HDD every three years or so actually costs more than this. There is a tipping point where if your storage is small enough then fixed price £55 a year is too much; there is another tipping point where you have so much data that getting it out there is impractical. In between those points Amazon's deal is excellent value for money.

Sent from my ONEPLUS A3003 using Tapatalk

SlimChances
2017-05-01, 14:23
Larger than infinite? And the price? Fifty five quid a year for infinite? Personally I have found that a new HDD every three years or so actually costs more than this. There is a tipping point where if your storage is small enough then fixed price £55 a year is too much; there is another tipping point where you have so much data that getting it out there is impractical. In between those points Amazon's deal is excellent value for money.

Sent from my ONEPLUS A3003 using Tapatalk

Infinite is a bit of an exaggeration. But I can fit 4 -3 1/2" drives in my box. Right now it is easy to obtain a 10 TB Hard Drive. How much is Amazon going to charge for 40 TB of data? Also how long does it take to download that information if your house catches fire and you lose your data? For me I take my HD out of my Safety Deposit Box and connect it to a laptop and I am in business. Personally I am not too comfortable with Amazon or Google or another company having control of my data, whether it's 1 TB or 40TB
The Hard Drive in the SD box should not wear out for a long time as it is only ocasionally used

mandryka
2017-05-01, 14:33
Depends on your broadband upload speed. Unlimited fibre is the only way this makes sense, that's the only way to get 18-20 Mbit upload. With that, I put 2TB up in a week and two days. If your upload is 2Mbit it will take ten times longer...

Upload speed is at the moment between 10 and 20 mbps. This calculator suggests it will take between 2 and 3 weeks for 2Tb. I think it is fibre, from BT infinity.

http://www.thecloudcalculator.com/calculators/file-transfer.html

drmatt
2017-05-01, 14:55
Infinite is a bit of an exaggeration. But I can fit 4 -3 1/2" drives in my box. Right now it is easy to obtain a 10 TB Hard Drive. How much is Amazon going to charge for 40 TB of data? Also how long does it take to download that information if your house catches fire and you lose your data? For me I take out of my SD Box and connect it to a laptop and I am in business. Personally I am not too comfortable with Amazon or Google or another company having control of my data, whether it's 1 TB or 40TB
The Hard Drive in the SD box should not wear out for a long time as it is only ocasionally used
Amazon is going to charge £55 a year, that's the contract terms. "Unlimited". Of course they invoke a "fair use" clause too but who knows when they would actually bother to use that. So there are use cases where it makes sense. And for me it makes sense. Not that that means I don't *also* have HDD stored locally, it's just another tier of disaster recovery options and quite a reasonably priced one too.

Download is obviously many times quicker, but it's infinitely quicker than getting my camera and attempting to persuade the kids to pose for a hundred thousand pictures again..

As for control of the data, sure that's always a concern with the cloud, so I decided encrypt before upload. Even pathnames and filenames are encrypted, they literally have nothing without breaking my encryption.

drmatt
2017-05-01, 14:59
Ehm, aren't we confusing GOOGLE drive and AMAZON drive here?

Back to Amazon Drive:
rclone (https://rclone.org/) seems to support it, and it has a "check" function that compares file hashes allowing you to skip files that calculate as identical.
No sign of an open API (yet) but some developers have had access by the look of it. https://developer.amazon.com/amazon-drive

Sent from my ONEPLUS A3003 using Tapatalk

bpa
2017-05-01, 15:59
No sign of an open API (yet) but some developers have had access by the look of it. https://developer.amazon.com/amazon-drive

Sent from my ONEPLUS A3003 using Tapatalk

This link lets you download sdk for IO and Android. https://developer.amazon.com/public/apis/experience/cloud-drive/sdk-download/sdks-download-release-notes

There seems to be no SDK for Windows or Linux but I assume rclone (and simialr) just replicate data exchange.

CloudAtlas
2017-05-02, 02:56
Infinite is a bit of an exaggeration. But I can fit 4 -3 1/2" drives in my box. Right now it is easy to obtain a 10 TB Hard Drive. How much is Amazon going to charge for 40 TB of data? Also how long does it take to download that information if your house catches fire and you lose your data? For me I take out of my SD Box and connect it to a laptop and I am in business. Personally I am not too comfortable with Amazon or Google or another company having control of my data, whether it's 1 TB or 40TB
The Hard Drive in the SD box should not wear out for a long time as it is only ocasionally used

I personally believe that using the cloud as a sole backup is very unwise. As it is not directly under you control, it should be viewed as ephemeral (or even open source if not encrypted). A multi-layered approach is far safer e.g. local backup, off-site backup and cloud.

Just my thoughts.

RonM
2017-05-02, 05:20
I personally believe that using the cloud as a sole backup is very unwise. As it is not directly under you control, it should be viewed as ephemeral (or even open source if not encrypted). A multi-layered approach is far safer e.g. local backup, off-site backup and cloud.

Just my thoughts.

I'm not sure when cloud backup for large datasets is actually a best solution, unless you have really fast connections and also alternative back-ups. Purely from a security point of view, it's not clear to me why local/offsite/cloud is better than local and multiple offsites. For instance, one or more local copies, perhaps on a main data hard drive plus on the server, plus two hard drive back-ups stored in safety deposit boxes in separate banks. The incremental cost of a second safety deposit box won't be more than the cost of cloud storage, and won't have the inconvenience.

Important to bear in mind that a safety deposit box does not guarantee against loss. If the bank burns down so will your drive in the SD. It's offsite back-up, but that doesn't mean immune from loss.

I personally have music data stored on three different locations at home (main data archive, server, and the lossy version, all on separate hard drives), and a periodic backup to the safety deposit box. I use two or three different HDs for the bank storage -- one at the bank, and at least one (maybe two) that used to be in the bank and were retrieved when I added the most recent back-up. This is reminding me to do another backup, and I'll use the drive I took out of the bank last time I did this. I'll then take it to the bank and retrieve the one that's there for next time. That retrieved drive is another (incomplete) backup itself.

I haven't gone so far as to get a second location for an SD.

R.

lostboy
2017-05-02, 10:45
I use Crashplan as part of tiered backup strategy, and have done for past five years or so. It's $60 p.a. for unlimited encrypted storage for one PC, so a little cheaper than Amazon with current exchange rates, but backup specific. They will backup anything 'local' to a licensed machine, and provide a sensible background backup application (backup sets, file version retention including deleted files, etc.). Their app will also allow backup to other PCs including across the net, so if you have friend with a 24/7 server.... (their free version has this capability - you only pay for central cloud services).

I've found it reliable and it's saved my bacon on more than one occasion. I find it quicker to find/restore the odd file from the cloud, than go to local backups. However, full disk restores are much quicker from physical drives. In my server I use a disk pooling app with file duplication as well and have found that this has taken care of wholesale disk failures - Crashplan helps me when I've done something stupid and haven't noticed at the time (deleted the wrong file and so on).

I wouldn't contemplate remote backup without a fast internet connection, and at the moment my unlimited fibre connection isn't so fast that I'd be prepared to do without local backups either. I'm not sure that the cloud backup would work well from a machine that is only on for short periods.

Chris

d6jg
2017-05-03, 06:31
I have my data stored on a Qnap with LMS running on either a Pi (piCorePlayer) or an old PC (Vortexbox). The QNAP is mounted either via SMB or NFS.

One of the officially available Qnap apps is "Connect to Cloud Drive". I haven't tried it but it supports Amazon Cloud Drive. If it works (and I have no reason to doubt that it does) then this would do the job nicely.
"You can remotely connect to a cloud service (such as Dropbox, Microsoft OneDrive, Google Drive, Amazon Cloud Drive, Box or Yandex Disk) in File Station after installing this add-on. You can easily carry out file management tasks such as copying and moving from cloud services to the NAS, or from NAS to cloud services by using remote connection at File Station. Note: Limitations of your cloud service account may affect what files can be transferred. Please check the account details with your cloud service for more information regarding what files can and cannot be transferred."

Another Qnap developed app is Hybrid Backup Sync - in Beta
"Hybrid Backup Sync is a comprehensive data backup and disaster recovery solution for the files stored on your QNAP NAS. It integrates backup, restoration and synchronization functions to provide you with a variety of options for data backup and synchronization including one-touch USB backup; Time Machine backup; RTRR backup and synchronization through RTRR, Rsync, FTP, CIFS/SMB and cloud services such as Google Drive™, Microsoft® OneDrive®, Dropbox®, Box®, Yandex® Disk, Amazon® Cloud Drive, Amazon® S3, Amazon® Glacier, Azure™ Storage, Google Cloud Storage™, S3/OpenStack Swift/WebDAV compatible services. Hybrid Backup Sync allows you to manage multiple storage, remote and cloud accounts to help save time in creating backup jobs. Note: You must install Cloud Backup Sync from the App Center to remotely connect to cloud services."

I don't need to use them though as I do a RTRR sync to another Qnap at a remote location.

atrocity
2017-05-17, 10:32
Ehm, aren't we confusing GOOGLE drive and AMAZON drive here?

Back to Amazon Drive:
rclone (https://rclone.org/) seems to support it, and it has a "check" function that compares file hashes allowing you to skip files that calculate as identical.

THANK YOU for this!!! I just set it up yesterday and it seems to be working fine. Greatly preferable to trying to sync using Amazon's horrific W*nd*ws client.

bakker_be
2017-05-18, 09:05
THANK YOU for this!!! I just set it up yesterday and it seems to be working fine. Greatly preferable to trying to sync using Amazon's horrific W*nd*ws client.

Since posting this, I've uploaded ±1TB of music to Amazon :p

atrocity
2017-05-19, 08:19
Since posting this, I've uploaded ±1TB of music to Amazon :p

And, just like that, Amascum has banned rclone.

https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314/10

drmatt
2017-05-19, 10:50
They don't like the word "unlimited" as much as their marketing would have you believe, apparently...

Sent from my ONEPLUS A3003 using Tapatalk

atrocity
2017-05-19, 12:31
They don't like the word "unlimited" as much as their marketing would have you believe, apparently...

If I understand the situation correctly (and I'm not sure that I do) rclone's developer had to work with Amazon to get the app approved, but Amazon banned it without warning or consulting him and has yet to respond to his inquiries. For now, I've told Amazon not to renew my plan when it expires in August. If they allow rclone again, I'll probably keep the service, but for people with large music libraries the inability to automate a sync makes even "unlimited" useless. I'd be OK (not thrilled, but OK) if they'd just allow it with --dry-run so I could see what needs to be done and do it manually, but absent that or a similar tool, forget it.

I'm currently testing a 10GB backup to https://hubic.com - they have a 25GB free version that works with rclone. Unfortunately, they don't have an unlimited plan, though their 10TB should work (for now!) for me. The test isn't finished yet, but it's been running for a few hours and seems to be working well:


Transferred: 7.864 GBytes (726.991 kBytes/s)
Errors: 0
Checks: 1
Transferred: 4942
Elapsed time: 3h9m1.9s

drmatt
2017-05-19, 17:12
So this is why I used "dar". Keep a local metadata copy and I can run differential backups against the metadata to upload as a tarball (um, darball) without having to keep the data local too.

Sent from my ONEPLUS A3003 using Tapatalk

atrocity
2017-05-21, 10:01
Testing hubiC at the moment. 25GB free and it works with rclone. Their largest plan is 10TB, which works for me for now, so I just might go for it.

rclone developer is hopeful that he can work things out with Amazon.

norderney
2017-05-21, 15:15
I use Dropbox for backing up my music database which is about 1Tb / 60,000 tracks. As a bonus I have installed an app called FLACBox on my iPhone which allows my to play music stored directly on Dropbox. So I have access to my complete music library when I'm away from home using either 3G or WiFi.
I already had a 2Tb Dropbox account but I had only a few documents stored on it. This way I can make use of the storage capacity.

mandryka
2017-05-30, 07:19
I use Dropbox for backing up my music database which is about 1Tb / 60,000 tracks. As a bonus I have installed an app called FLACBox on my iPhone which allows my to play music stored directly on Dropbox. So I have access to my complete music library when I'm away from home using either 3G or WiFi.
I already had a 2Tb Dropbox account but I had only a few documents stored on it. This way I can make use of the storage capacity.

But how did you get the data uploaded?

norderney
2017-05-30, 08:57
Using the Dropbox web site, I logged into my Dropbox account and then upload all my FLAC files to my Music folder within Dropbox. Quite easy actually. Takes a long time but only has to be done once. Works best if you have a fast fibre broadband connection.

mandryka
2017-05-30, 11:00
Using the Dropbox web site, I logged into my Dropbox account and then upload all my FLAC files to my Music folder within Dropbox. Quite easy actually. Takes a long time but only has to be done once. Works best if you have a fast fibre broadband connection.

What sort of upload speeds did you get? How long did it take to upload 1TB?

drmatt
2017-05-30, 11:06
Maths to the rescue! 1TB is 8000000 Mbits. This will take 5.1 days on an 18Mbit/s uplink (typical unlimited fibre package).

mandryka
2017-05-30, 11:32
The reason I was pushing the point is that my internet upload speed is about 17mbps, but using Backblaze, I was getting an upload speed of 1.5mbps.

Anyway I just spoke to their support people and they told me make some changes in the settings file and now I have 16mbps. So problem solved.

I have 2TB to upload so I don't think dropbox will work for me. So far, Backblaze looks fine though. Fingers crossed.

drmatt
2017-05-30, 15:11
Ok, so you had a specific problem. What did you change? Are they artificially limiting upload speed in the client package?

I had no issues with Amazon drive on upload speed.

Sent from my ONEPLUS A3003 using Tapatalk

mandryka
2017-05-31, 23:37
Ok, so you had a specific problem. What did you change? Are they artificially limiting upload speed in the client package?

I had no issues with Amazon drive on upload speed.

Sent from my ONEPLUS A3003 using Tapatalk

You have to go to performance in settings, turn the throttle to maximum upload speed, and choose the maximum number of threads.

drmatt
2017-05-31, 23:42
So it seems they restricted everyone by default to prevent impact to browsing experience and general connectivity whilst uploading. They seem to have somewhat overcooked the throttling if it went to less than 10% of your link speed, however..

Sent from my ONEPLUS A3003 using Tapatalk

pablolie
2017-06-07, 17:40
I see the appeal for backup, but unless it's automatically mirrored -which I haven't seen in cheaper storage- I don't see the appeal for everyday operations. Local NAS still seems to rule the cost-benefit analysis for me... (unfortunately, I'd love a cloud based cost effective solution, but local storage is MUCH cheaper over the mid term for me).

Blisteringblue
2017-06-08, 04:21
Wow, talk about long winded. I wanted a set and forget style backup for this, been looking for reasonable cloud for a while. I've 1TB OneDrive from MS through work, 500GB BT cloud with my broadband but £55 for unlimited Amazon Cloud gets the nod.

Just spent 2 days playing with various applications to get it there, boy do most blow chunks.

My music is only about 900GB but it sits on my Proliant Microserver running WHS 2011. Therein lies the issue, I tried Odrive but it was dreadful, was about to download cloudberry backup but again looked a messy solution.

I've a QNAP NAS which I use for my media server and although I don't use it for Squeeze (need Intel for MusicIP and Sugarcube) I came up with an ingenuous plan.

1. Download the Hybrid Cloud Backup app for QNAS which does 1 or 2-way sync direct into the Cloud Drive (using for home photo/video backup to Amazon)
2. Plugged my current Backup Music Drive (stand alone USB 2TB WD Drive) directly into the QNAP and it automatically shares as an External USB Drive
3. Setup a second cloud one-way sync job from the QNAP to Amazon for all the music files on the WD USB Drive which is now running happily away through the 900GB
4. I can then amend my SyncToy job that was my backup for my music files to send to the new Shared QNAS drive over the network, so pretty much a double backup, local and cloud.

Not bad for £55 a year, plus the added bonus that if I am working away I can get to my music files on a adhoc basis, although spotify sort of negates this these days.

Happy Days and thanks to the OP for the heads up.

d6jg
2017-06-08, 06:35
Wow, talk about long winded. I wanted a set and forget style backup for this, been looking for reasonable cloud for a while. I've 1TB OneDrive from MS through work, 500GB BT cloud with my broadband but £55 for unlimited Amazon Cloud gets the nod.

Just spent 2 days playing with various applications to get it there, boy do most blow chunks.

My music is only about 900GB but it sits on my Proliant Microserver running WHS 2011. Therein lies the issue, I tried Odrive but it was dreadful, was about to download cloudberry backup but again looked a messy solution.

I've a QNAP NAS which I use for my media server and although I don't use it for Squeeze (need Intel for MusicIP and Sugarcube) I came up with an ingenuous plan.

1. Download the Hybrid Cloud Backup app for QNAS which does 1 or 2-way sync direct into the Cloud Drive (using for home photo/video backup to Amazon)
2. Plugged my current Backup Music Drive (stand alone USB 2TB WD Drive) directly into the QNAP and it automatically shares as an External USB Drive
3. Setup a second cloud one-way sync job from the QNAP to Amazon for all the music files on the WD USB Drive which is now running happily away through the 900GB
4. I can then amend my SyncToy job that was my backup for my music files to send to the new Shared QNAS drive over the network, so pretty much a double backup, local and cloud.

Not bad for £55 a year, plus the added bonus that if I am working away I can get to my music files on a adhoc basis, although spotify sort of negates this these days.

Happy Days and thanks to the OP for the heads up.

Why don't you store your music on the QNAP and have LMS / WHS read it from there? I agree that a QNAP won't do MusicIP etc but if you had your music files there then the Cloud Sync is going to be easy. There is some performance hit by going through the USB port.

I have all my files on a QNAP but LMS runs on a server class PC running Vortexbox with NFS mount to the QNAP (but it could just as easily be SMB). I am lucky enough to have a second QNAP sitting in a remote location and I sync them using the QNAP built in Remote Replication. If I didn't have that then I'd use the Hybrid Cloud Backup App like you are.

Blisteringblue
2017-06-08, 08:07
Why don't you store your music on the QNAP and have LMS / WHS read it from there? I agree that a QNAP won't do MusicIP etc but if you had your music files there then the Cloud Sync is going to be easy. There is some performance hit by going through the USB port.

I have all my files on a QNAP but LMS runs on a server class PC running Vortexbox with NFS mount to the QNAP (but it could just as easily be SMB). I am lucky enough to have a second QNAP sitting in a remote location and I sync them using the QNAP built in Remote Replication. If I didn't have that then I'd use the Hybrid Cloud Backup App like you are.

Not really worth it, I've not exactly changed much this way, my local backup is still to the WD Drive, I will just sync via shares to the QNAP USB rather than direct to my PC. WHS has plenty of space too and I would be scratching around on the QNAP (single drive version) as the 3TB there is pretty full of DVDs etc. I just get an automatic copy of the WD drive up to Amazon without thinking about it now.

And just as I embrace unlimited cloud I see Amazon have pulled the plug, hopefully I come under 1TB

https://betanews.com/2017/06/08/amazon-drive-unlimited-cloud-storage-is-no-more/

Squeezemenicely
2017-06-09, 02:36
Oh no! Amazon have killed "Music in the cloud" for me. This is really terrible. I had setup my NAS to sync my music files to the cloud - all was great.

Now they changed the game, no more unlimited and I would have to pay 180 Dollars for my storage, so obviously I will not renew my plan.
What a shame, to think of all the time it took to upload to the cloud, now just having to use an oldfashioned backup on drive again.

Two things I loved about the cloud - being able to stream my music on the go and second immediate backup of my files. With a disc that is stored at my workplace, I will probably only do a backup every month or so - knowing myself probably even less than that.
That could mean loosing many files in a worstcase scenario.

Ridiculous going from unlimited to 1TB, in our day and age. They got us to use and upload files and know they are trying to get us to pay more and more.

bakker_be
2017-06-09, 07:01
Oh no! Amazon have killed "Music in the cloud" for me. This is really terrible. I had setup my NAS to sync my music files to the cloud - all was great.

Now they changed the game, no more unlimited and I would have to pay 180 Dollars for my storage, so obviously I will not renew my plan.
What a shame, to think of all the time it took to upload to the cloud, now just having to use an oldfashioned backup on drive again.

Two things I loved about the cloud - being able to stream my music on the go and second immediate backup of my files. With a disc that is stored at my workplace, I will probably only do a backup every month or so - knowing myself probably even less than that.
That could mean loosing many files in a worstcase scenario.

Ridiculous going from unlimited to 1TB, in our day and age. They got us to use and upload files and know they are trying to get us to pay more and more.
Ever since they stopped allowing the use of rclone, I've been investigating alternatives. This is just another factor prompting a switch. Since I only need a backup solution I intend to try Crashplan ...

atrocity
2017-06-11, 07:10
Ever since they stopped allowing the use of rclone, I've been investigating alternatives. This is just another factor prompting a switch. Since I only need a backup solution I intend to try Crashplan ...

I love that I learned about rclone mere days before it was banned. But that ban got me to immediately request non-renewal of my ACD account. I signed up with hubiC, which works just fine with rclone, but my multi-terabyte library is currently on day five of the backup process. I don't think it's painfully slow, it's just not blazingly fast.

I also use CrashPlan, but "painfully slow" definitely applies. The price and feature set are fantastic, but it's definitely something you use AFTER you have your local backup figured out.

atrocity
2017-06-11, 07:11
I see the appeal for backup, but unless it's automatically mirrored -which I haven't seen in cheaper storage- I don't see the appeal for everyday operations.

I run rclone to hubiC wrapped in a shell script. I haven't scheduled it yet because the first full backup hasn't completed, but I intend to.

pgnyc
2017-06-11, 08:19
hi,
like some of you i have 2 HP microservers , one with unraid (nas server) , one under Slackware ( lms, mail server, imap server, calendar server , etc..) .
the lms is connected thru SMB to the unraid.

everyday a backup is done to an attached external drive. (total on monday, incremental other day, 4 weeks kept)

i have also a raspberry pi 3 (in a different room) as a rcp server (using raspbian) with also an attached external drive formatted exfat.
so i do a rcp (remote copy) from the unraid to the the pi.

Pascal

matka
2017-06-12, 10:06
I use Crashplan as part of tiered backup strategy, and have done for past five years or so. It's $60 p.a. for unlimited encrypted storage for one PC, so a little cheaper than Amazon with current exchange rates, but backup specific. They will backup anything 'local' to a licensed machine, and provide a sensible background backup application (backup sets, file version retention including deleted files, etc.). Their app will also allow backup to other PCs including across the net, so if you have friend with a 24/7 server.... (their free version has this capability - you only pay for central cloud services).

I've found it reliable and it's saved my bacon on more than one occasion. I find it quicker to find/restore the odd file from the cloud, than go to local backups. However, full disk restores are much quicker from physical drives. In my server I use a disk pooling app with file duplication as well and have found that this has taken care of wholesale disk failures - Crashplan helps me when I've done something stupid and haven't noticed at the time (deleted the wrong file and so on).

I wouldn't contemplate remote backup without a fast internet connection, and at the moment my unlimited fibre connection isn't so fast that I'd be prepared to do without local backups either. I'm not sure that the cloud backup would work well from a machine that is only on for short periods.

Chris
Another Crashplan user here, I have 2.5TB backed up. As mentioned already, the software runs in the background, does incremental backups, its throttled. The original backup took about 3 months, 24/7, as I throttled the uplink to 500KB, but it was eventually done. Incremental backups, especially for flac (I do not backup movies) do not require much time, I'm throttling at 250KB 24/7 - this is far from fibre but gets the job done and it does not affect my web usage experience. CrashPlan desktop app provides a control on the uplink speed. I'm also with an ISP provider where upload does not count against your bandwidth.

If you are in US, you can mail a harddrive for an initial, seed backup, there are restrictions for mailing hard drives across borders so I did not bother.

I had to recover a small portion of my files once and it was very easy, painless experience.

I'm on linux and there were some changes to be made to config files to allow more than 2TB of backup data but there was enough support from CrashPlan to do so.

Overall I'm quite happy with this solution, I have reduced much of the backup clutter that I had after moving to a smaller place. And unless CrashPlan crashes, I don't have to worry about hard drive crashes :-)

mandryka
2017-06-23, 22:52
I have very positive feelings about Backblaze. Once the initial backup is made you set the controls to automatic and subsequent backups occur automatically and slowly in the background, and with no noticeable effects on the computer or network. It took 3 weeks to upload 2TB.

Upload speeds slow down in the initial upload the more you upload. That's because there's a "deduplication" programme which has more to check. The more RAM the better of course. Mine took no time at all for the first third, the last third went up at about 100GB or less in 24 hours. It's not a real problem, apart from my impatience. My upload speeds seemed to average at about 15mbps. The computer was still usable but the UI became arthritic. But streaming music from it was fine as was a virus check.

Music files are particularly slow to upload because Backblaze makes a copy of any file over 30MB before uploading.

Customer service is excellent and the UI is simple.

There are some setting adjustments worth making for the initial upload - throttle to max and no. of threads to max.

ftlight
2017-06-24, 16:26
I have very positive feelings about Backblaze. Once the initial backup is made you set the controls to automatic and subsequent backups occur automatically and slowly in the background, and with no noticeable effects on the computer or network. It took 3 weeks to upload 2TB.
Thanks for the update! How much are you paying for 2TB?

Bill

mandryka
2017-06-30, 09:36
Thanks for the update! How much are you paying for 2TB?

Bill

£50 pa