Cloud Backup to Amazon S3 unsuccessful

Hi folks,

I’ve been attempting to make a backup to Amazon S3 using the Cloud Backup feature. My aim is to do a one off initial backup of everything (currently around 500GB) and have this maintained incrementally thereafter. 

Obviously backing up 500GB of data is a fairly time consuming process … I have a reasonable amount of bandwidth but I’d expect this to take about a week to complete. Unfortunately every time I’ve tried this so far the process has stopped with an indeterminate error after a couple of days (the partial transfer to S3 seems to be just fine).

This shouldn’t be more than a minor irritation, because I have the backup mode set to “Incremental”, so I should just be able to restart the backup process and have it pick up from where it left off, right? Sadly not … it appears to restart from the beginning overwriting files which have already been transferred (there have been no local writes in the interim), which makes it seem unlikley that the process will ever complete. Even if it did, I don’t have a lot of confidence that subsequent incrementals wouldn’t just attempt to transfer the whole lot again.

Has anyone else experienced these issues or similar? Does anyone have any success stories, and if so do they have any advice for me?

Cheers,

Miles

Very few Users are known to run Amazon S3 backups from a WD NAS, but hopefully they will be able to offer you some guidance. On the meanwhile you could try contacting WD’s Technical Support about this. You can do so either by phone or email.

To Contact WD for Technical Support
http://support.wdc.com/contact/index.asp?lang=en

Regards,

@Trancer I’ve tried contacting WD support, but so far not had any useful feedback. Are you able to put me in touch with someone at WD who would be able to help me troubleshoot this problem?

Please check your private messages.

Regards,

I have been in touch again, but I’m afraid still no progress … has anyone had any success with this?

I have a feeling the Amazon S3 feature is buggy - I haven’t tried it myself but my hunch is based on feedback by couple other users (one of EX2 and one of EX4 - and both EX2 and EX4 use the same codebase as Mirror’s, so whatever problem is plaguing the Amazon S3 support on your Mirror is almost certainly an issue on those models as well). There are certain issues that WD takes forever to fix in their firmware, if at all…and given how few people might be using the S3 backup feature, I doubt it is even on their radar. The best way to deal with these bugs is to definitively prove to their tech support that this is really a bug (like I did with an FTP bug that was present on my EX2 and on Mirror and they fixed it after five months from when I reported it in their firmware release last week).

This is where you can find more about those other users - my advice - band together with others in your shoes and be in touch outside the forum and build your case if you want the issue to be addressed by WD at all. Ad oh, you do need to open a trouble ticket (which WD has a bad policy of closing even if an issue is unresolved if there is no activity on the case for one week) ->  http://community.wd.com/t5/WD-My-Cloud-EX2/Problems-with-Amazon-S3-Jobs/m-p/759676

I’m still not making any headway on this … my support ticket is still open, but noone on the other end of it appears to be very responsive.

Should we conclude that the Amazon S3 cloud backup feature (one of the main reasons I purchased this device) is basically unfinished and practically useless?

Bump. Same issue here, S3 backup fails immediately.  Contacted support and they recommended restarting to factory defaults, no improvement. Hoping they push out a new firmware update fast or I’m returning for a Synology.  Cloud backup is a must for me to have protection in case the My Cloud is damaged or stolen.

Also, why S3 and not Glacier option? Just more expensive.

I know I’m reviving an old post but I’m having the same problem with an EX4 I recently purchased. I’ve updated to the latest firmware (1.05.30) and hunted all around the ssh console looking for logs but can’t find much.

the /etc/s3.conf file does have an error_code of 47 but I can’t work out what that is. I do see what I think is a problem though with the cron setup and the S3 binary. All my problems started when I scheduled the backup, if I run it manually it works. The conf file sets out a schedule (assuming you’ve set one in the interface, mines a daily incremental) but cron has one setup as well that matches the s3.config file. So even though the S3 binary runs and stays running because of it’s own schedule cron still launches a new one every day anyway. It would explain why there are multiple S3 processes running using the same schedule. I’ll test the theory tonight when I can restart the device.

  • m0tter 

Hi,

I also have a the same problem. Please let us know if your solution works.

I feel let down by this product.

Thanks.

I just tried setting up an incremental Cloud Backup to S3 on my My Cloud Mirror WD NAS, and hit exactly the same problem that milessabin thoroughly described about a year and a half ago, and then shaleens seemed to indicate was experiencing the same behavior a year ago.

My scenario was an attempt to backup about 700GB of data (10s of thousands of files) to S3. I am seeing exactly what milessabin reported:

  • The Backup job reports “Failed” after several hours of run time, without any apparent error details. It crossed my mind that the job might be exceeding S3 throttling, and I just hoped running it multiple times would eventually get to completion and a happy steady state, due to having chosen the incremental option.
  • However, despite using the incremental option, I see all the files getting uploaded each time and the Last Modified timestamps in S3 being updated across the board, and the Storage Class changed back to “Standard,” after I had used the S3 web interface to change them to “Standard - Infrequent Access” after a previous partially completed backup run.

Given this thread, it sounds like the assumption is that there aren’t many users of this feature, and it is just not functional, which I’m very disappointed to learn.

Assuming there aren’t any updates on this from the community, I’ll attempt to test an isolated scenario focused on just the incremental nature (since that at least is hopefully easily reproducible), and then submit a support ticket to WD. Very frustrating…

Pete

@Pete I gave up and now do periodic manual backups with Duplicity. The feature hasn’t ever worked properly, and clearly hasn’t been tested … it shouldn’t be advertised.

Thanks for your quick reply milessabin. So I guess there is some hope that if I poke tech support, that I tip the scale that motivates WD digging into this, and maybe they would have a firmware fix out in a few months, but I am not comfortable living without an automatic offsite backup for that long…

I’m only vaguely familiar with Duplicity, and was really hoping the out-of-the-box Cloud Backup feature would “just work.” Apparently, no such luck… I’m assuming you did something along the lines of https://theholyjava.wordpress.com/2015/04/03/backup-wd-mycloud-to-s3glacier-with-duplicity-build-instructions-included/? I now need to decide whether I bite the bullet and do that… Have you had any problems with WD upgrades messing with your configuration of Duplicity on the WD Mirror?

I thought about trying the built-in Cloud backup to Elephant Drive, on the off chance that works where as the S3 target doesn’t. But it seems Elephant Drive is cost prohibitive for backing up close to 1TB. Also, the fact that I had never heard of Elephant Drive before doesn’t make me want to jump on it (but that could just be my small world…).

I reported an issue and chased it for months with no useful response.

I run Linux on my laptop so run Duplicity directly on that machine rather than on the NAS. If you go down the other route I’d be keen to know how you get on.

I tried a simple test case, and the incremental backup process worked in a reasonable way. I then tried to progressively build up my backup set, with successful incremental backups along the way. However, not only was this tedious, but I seemed to possible hit some limit, at which point the backups seemed to consistently fail… I didn’t spend a lot of time isolating whether the problem was throttling on the S3 side, something to do with my ISP, or something on the WD My Cloud Mirror. I’m crossing my fingers, and upgrading to a QNAP 4-bay NAS that seems to have much richer S3 backup options, supposedly with out-of-the-box support for S3 Glacier (but the unit is also about 3 times the price). When that arrives and I’ve determined whether I have any more luck on the AWS S3 backup front, I’ll try to remember to post back here.

Sad that this is not working I just signed up for Amazon S3 to attempt to get this setup.

Looks like WD needs to address some issues on their end to get this working.

I was getting this.
I had created a client bucket and a folder bucket within it.
I was trying to direct the folder to client/folder
When I took out the folder and just used the main bucket the job worked.
It created its own folder called Folder_Backup
and the job started.

S3 backup issues appear persistent through the EX4100 model with firmware v2.21.119.

We have an open ticket with tech support for a couple of weeks now and no solution.

Our backup jobs will report “Backup Failed” after a couple of seconds of running the jobs. One of our backup jobs is a simple folder with a 0 byte test file and it would not backup. We checked to make sure that the S3 bucket has correct permissions and that the account used in the backup job has the ability to copy files in the destination bucket.

Very disappointing…

If you’re willing, enable ssh, login and post the error code line in the /etc/smb.conf file

        cat /etc/smb.conf

Or you could enable S3 Bucket Logging and parse the logs for error messages
http://docs.aws.amazon.com/AmazonS3/latest/UG/ManagingBucketLogging.html

I’m having the same problem and have been for months. I have a 300mb down speed and 20mb up speed. I’ve contacted WD support with no solution. They say keep trying and also check with Amazon and your ISP. I’ve worked with Amazon web service to trouble shoot. I’ve had Spectrum out to my house twice to double check connection and speed. I’ve tried backing up dozens of times and the backup fails every time. My conclusion is that the WD My Mirror does not function as advertised. Lesson learned. I’m researching a replacement that offers a cloud backup solution that works. Obviously not a WD product!