Happy accident - INCLUDES AN ALTERNATIVE TO SAFEPOINT

My 13 month old 4TB WDMC has been giving me sporadic problems for some time now, losing network connection, media stuttering, all sorts. I’ve ‘tinkered at the edges’ to keep things going but all to no avail, things have kept going wrong. I do have a certain amount of tech knowledge, so I knew what I was doing.

Then one day last week I managed to accidentally hit the ‘Quick restore’ button on the dashboard, don’t ask me how, I even managed to hit the confirmation button as well, so within seconds all data on the MC was gone, oops.

I have a backup regime in place, (not Safepoint, it just does not inspire confidence and is sooo slow) using rsync in a script which I’ve developed which runs at 0100 every day, backing up to an attached USB drive. It’s faster than Safepoint, not so hungry of resources, and I’ve even got it to send me an email when it’s finished. Having configured the MC after it restarted, I wrote a script to restore my shares (amounting to to 2.1TB) - the next day or so was a little anxious as I awaited the outcome, but all my data was restored, intact, within 48 hours - phew!

Since then the MC is more stable, it feels faster and so far none of the little niggles of the last few months are making themselves evident. It shouldn’t really take a wipe of all one’s data to get things running properly again, but it has worked, and I feel more than a little satisfaction that my backup solution has worked so well for me!

What I do have to say is that I’ve developed my scripts from information that I’ve gleaned and learned from in this forum, so a big thank you to everyone in here for your help. If I have some time I’ll post the scripts for my backup and restore solution in here, but in the meantime, thanks again.

3 Likes

Good Job nicktee55!!

Now please post up your rsync + email script minus your email addy and secret backup paths. I’ve been meaning to do this for ages but it has been a year and a half.

Now a couple of questions

  1. what firmware are you on and

  2. does the attached USB drive stay connected?  

Thanx

Then one day last week I managed to accidentally hit the ‘Quick restore’ button on the dashboard, don’t ask me how, I even managed to hit the confirmation button as well, so within seconds all data on the MC was gone, oops.

It’s very easy to not read confirmation boxes (they pop up so often they’ve lost their impact), and just click “yeah, of course I know what I’m doing”.  Only to then go “oh *bleep*, what have I just done…?”  I think we’ve all been there, and felt that cold sweat, heart-sinking feeling…

But that’s what a good backup regime is for…

Which brings me to echo Ralphael’s comment: yes please to posting your rsync script here; it obviously works, and would be better than the clunky old xcopy-based script I use when I remember to plug drives into my PC. Even if only to serve to spark our own scripts.

Thanks.

OK here goes, but I’ll add a couple of caveats -

  1. Use at your own risk, I do not know whether this invalidates your warranty

  2. There may be more elegant ways of doing the actual scripts, I’m just doing what worked for me.

  3. Others may recognise some of their work in here, thank you for your help in getting me this far!

I also assume that anyone doing this has basic knowledge of WinSCP and PuTTY and has SSH access. I’m using the latest version of firmware. This doesn’t give an incremental backup, just an ongoing mirror of your shares.

Using WinSCP (easiest) or PuTTY create a folder called “Backup” on your USB drive, inside that folder create a separate folder for each of your shares. It may also be worth renaming the share name of your USB drive to something simple such as “USBbackup” which is the name I’ll use here.

To prevent your scripts being overwritten by future firmware updates, create a share called ‘system’ (for example) and place a folder in there called, say 'Tasks". I shall use these and the above names in what’s to follow.

With acknowledgements to Arty2, we’ll create the Python email script. Go into PuTTY then:

nano /shares /system/Tasks/mail.py

Paste (right click) the following into the editor

#!/usr/bin/env python

-*- coding: utf-8 -*-from email.header import Header

from email.mime.text import MIMEText
from getpass import getpass
from smtplib import SMTP_SSL
import sys
#edit the areas between the quotes in the line line below with your own settings
login, password, server, recipients = “the e.mail.account.you’re.sending.from@gmail.com”, “password”, “smtp.gmail.com”, “recipient@domain.com
#send email
subject = sys.argv[1]
body = sys.argv[2]
msg = MIMEText(body, ‘plain’, ‘utf-8’)
msg[‘Subject’] = Header(subject, ‘utf-8’)
msg[‘From’] = login
msg[‘To’] = recipients
s = SMTP_SSL(server, 465, timeout=10)
s.set_debuglevel(1)
try:
s.login(login, password)
s.sendmail(msg[‘From’], recipients, msg.as_string())
except Exception, error:
print “Unable to send e-mail: ‘%s’.” % str(error)
finally:
s.quit()

Come out of the editor. Now we’ll create the backup script

nano /shares/system/Tasks/backup.sh

add your share names where share1 etc appear below

rsync -a /shares/share1/ /var/media/USBbackup/Backup/share1/
rsync -a /shares/share2/ /var/media/USBbackup/Backup/share2/

Continue adding as many shares as you like using the above syntax

Finally, the email to confirm the backup completed

python /shares/system/Tasks/mail.py “Message from WD MyCloud” “System backup has been completed successfully”

Having come out of the editor, make the script executable:

chmod +x /shares/system/Tasks/backup.sh

Now add a task with crontab, the following will run the backup at 0100 each day

crontab -e

add the line

00 01 * * * /shares/system/Tasks/backup.sh

Your backup will run at the required time and will send you an email when the task is completed.

That’s it for the backup. By the sheer speed of what happened, my USB drive stayed connected throughout the ‘wipe’ and reboot without problems.

For restoring, have a script ready, once again using nano, as my shares are quite large I wanted to keep an eye on progress so asked it to send me an email after each share had been restored:

nano /shares/system/Tasks/restore.sh

rsync -a /var/media/USBbackup/share1/ /shares/share1/

python /shares/system/Tasks/mail.py “Message from WD MyCloud” “Share1 restore has been completed successfully”

rsync -a /var/media/USBbackup/share2/ /shares/share2/

python /shares/system/Tasks/mail.py “Message from WD MyCloud” “Share2 restore has been completed successfully”

add further shares as you wish, exit and save

make this script executable

chmod +x /shares/system/Tasks/restore.sh

now a final script, this will enable you to send an instruction to the MC to execute the restore script unattended

nano /shares/system/Tasks/startrestore.sh

nohup /shares/system/Tasks/restore.sh > /dev/null 2>&1 &

exit and make this script executable

chmod +x /shares/system/Tasks/startrestore.sh

That’s it!!!

To recover after a wipe of your MC, once you’ve manually set up your shares again, configured the MC and given yourself SSH access (not forgetting changing the root password), use WinSCP to manually copy the backup of your ‘system’ share to the MC (should only take a min), then open PuTTY.

Give it the command:

. /shares/system/Tasks/startrestore.sh

The restore will start running, PuTTY will return almost immediately to the command prompt, the restore will run in the background without any further intervention, and you’ll get emails to tell you how it’s going.

Of course if you just want to restore the odd file or folder that you’ve accidentally lost you can use WinSCP to copy the files/folders across from the USB drive.

Hope this is of help. There may be typos or unintended line-breaks in the above for which I apologise.

Nick :slight_smile:

3 Likes

Thank you very much…

I have given you two kudos for this; please don’t spend it all at WD :stuck_out_tongue:

I’ll be using the email script to tell me when my cloud wakes up :stuck_out_tongue: I’m obsessed with cloud sleeping…

and yes the cron backup will alleviate my weekly “Beyond compare” software that compares the two drives and copies the difference. Makes me fall in love with my system so much more…

Also from now on anybody that wants to do backup will be refered to your post.

Lastly, this also answers the second question of Firmware 4.x that it does not disconnect the USB due to firmware issues. 

Appreciate that my formatting has been messed about by moving to the new forum system. I will work at getting it right again in due course

Nick

@Ralphael
Do you have a script that sends an E-Mail after wake up?
I want to check if and how oftern my drive goes to sleep and this seems a good way.

unfortunately I never got around to it… :stuck_out_tongue:

everyday I still do it the old fashion way by checking the logs :smile:

Do you know if it is even possible?
I fear that i wake up the WDMyCloud with the script to check if it is awake :confused:

of course it is do-able… you put it in monitorio.sh where it checks if the standby flag is on and the drive is awake (meaning that the drive was suppose to be sleeping (standby flag) but the drive is awake, then as it logs the time of awake to the logs, you send an email with the time. You will need the email script from above and the bash line to invoke the script with the wake time just after logging.

Totally simple for someone that knows, but that isn’t me :stuck_out_tongue: (well it could be me, but I would have to test the email script from a bash script, figure how to format a nice date string… then finally insert it into monitorio.sh.

I sort of got this to work. I didn’t get to the point where I set it up as an automatic job but I did get the rsync and the email to work. Where I’m having the problem is the speed. When running rsync from MyCloud to another NAS, MyCloud Mirror, I am getting speed around 6.5MB/s. When I do a copy paste from windows my speed is around 50MB/s.

Here is the rsync command I am using:
rsync -a --progress --delete /shares/media1/ sshd@192.168.1.2:/shares/media2/Test/

The ‘–delete’ switch deletes any files in the destination that don’t exist in the source. I want the destination to be a mirror.

The ‘–progress’ switch gives you the speed and will show you what is happening.

I tried the rsync two different ways with no noticeable change in speed; running on the MC pulling from the MCM and running on MC and pushing to MCM. It seems odd to me that a windows copy is so much faster since it is the man in the middle. The directory I was using as a test contains .mkv files that average about 300KB in size.

A couple of questions:
Is 6.5MB/s all I can expect running rsync on MC?
Are there any switch changes I can make so it will run faster?
Should I try running rsync from MCM?
(does it have a faster CPU or more RAM)
Can I do initial copy in Windows then let MC handle changes going forward?
(most days will be less than 50GB of changes)

I’ve now posted my script as a new topic as his was getting a bit long winded. Have a look at

Alternative to Safepoint

Appreciate that my formatting has been messed about by moving to the new forum system.

Here’s my attempt to sort out the formatting mess. I’ve tried to format

command line

And script file

line1
line2
line3
etc

With acknowledgements to Arty2, we’ll create the Python email script. Log in via SSH using PuTTY then:

nano /shares/system/Tasks/mail.py

Paste (right click) the following into the editor

#!/usr/bin/env python

#-*- coding: utf-8 -*-
from email.header import Header
from email.mime.text import MIMEText
from getpass import getpass
from smtplib import SMTP_SSL
import sys  

# edit the areas between the quotes in the line line below with your own settings

login, password, server, recipients = "the e.mail.account.you're.sending.from@gmail.com", "password", "smtp.gmail.com", "recipient@domain.com"  

# send email

subject = sys.argv[1]
body = sys.argv[2]
msg = MIMEText(body, 'plain', 'utf-8')
msg['Subject'] = Header(subject, 'utf-8')
msg['From'] = login
msg['To'] = recipients
s = SMTP_SSL(server, 465, timeout=10)
s.set_debuglevel(1)
try:s.login(login, password)
s.sendmail(msg['From'], recipients, msg.as_string())
except Exception, error:
print "Unable to send e-mail: '%s'." % str(error)
finally:
s.quit()

Come out of the editor. Now we’ll create the backup script

nano /shares/system/Tasks/backup.sh

# add your share names where share1 etc appear below

rsync -a /shares/share1/ /var/media/USBbackup/Backup/share1/rsync -a /shares/share2/ /var/media/USBbackup/Backup/share2/`  

# Continue adding as many shares as you like using the above syntax

# Finally, the email to confirm the backup completed

python /shares/system/Tasks/mail.py "Message from WD MyCloud" "System backup has been completed successfully"`

Having come out of the editor, make the script executable:

chmod +x /shares/system/Tasks/backup.sh

Now add a task with crontab, the following will run the backup at 0100 each day

crontab -e

add the line

00 01 * * * /shares/system/Tasks/backup.sh

Your backup will run at the required time and will send you an email when the task is completed.

That’s it for the backup. By the sheer speed of what happened, my USB drive stayed connected throughout the ‘wipe’ and reboot without problems.

For restoring, have a script ready, once again using nano, as my shares are quite large I wanted to keep an eye on progress so asked it to send me an email after each share had been restored:

nano /shares/system/Tasks/restore.sh

rsync -a /var/media/USBbackup/share1/ /shares/share1/

python /shares/system/Tasks/mail.py "Message from WD MyCloud" "Share1 restore has been completed successfully"

rsync -a /var/media/USBbackup/share2/ /shares/share2/

python /shares/system/Tasks/mail.py "Message from WD MyCloud" "Share2 restore has been completed successfully"

add further shares as you wish, exit and save

make this script executable

chmod +x /shares/system/Tasks/restore.sh

now a final script, this will enable you to send an instruction to the MC to execute the restore script unattended

nano /shares/system/Tasks/startrestore.sh

nohup /shares/system/Tasks/restore.sh > /dev/null 2>&1 &

exit and make this script executable

chmod +x /shares/system/Tasks/startrestore.sh

That’s it!!!

To recover after a wipe of your MC, once you’ve manually set up your shares again, configured the MC and given yourself SSH access (not forgetting changing the root password), use WinSCP to manually copy the backup of your ‘system’ share to the MC (should only take a min), then open PuTTY.

Give it the command:

./shares/system/Tasks/startrestore.sh

I would think an easier way would be to create a cron job to run at 03:05 every morning. The system is always awake at 03:05AM. In the cron job run the scripts that I posted in previous posts and email the output of those scripts once a day.
sleep.awk
awk ’
NR==1 { a=substr($1,6,2); b=substr($1,9,2); c=substr($1,12,8); y=substr($1,1,4)}
{if ($7 ~ /after/){
hh1=int($8/3600)
mm1=int(($8%3600)/60)
ss1=($8%3600)%60
TotalSleep = TotalSleep + $8
printf “%2s %2s %8s %8s %5d %2d:%02d:%02d\n”,substr($1,6,2),substr($1,9,2) ,substr($11,0,9),substr($1,12,8),$8 ,hh1,mm1,ss1}}
END {
hh1=int(TotalSleep/3600)
mm1=mm1=int(($8%3600)/60)
ss1=($8%3600)%60
printf “Total Sleep Time: %2d:%02d:%02d\n” ,hh1,mm1,ss1
start= a “/” b “/” y " " c
“date +%Y”|getline year
end= substr($1,6,2) “/” substr($1,9,2) “/” year " " substr($1,12,8)
$1=start
cmd=“date --date="”$1"" +%s"
cmd|getline st
printf “Start %s\n” ,st
$1=end
cmd=“date --date="”$1"" +%s"
cmd|getline end
printf “End %s\n” ,end
Total=end-st
hh1=int(Total/3600)
mm1=mm1=int((Total%3600)/60)
ss1=(Total%3600)%60
Percent=int((TotalSleep/Total)*100)
WakeTime= int(100 - Percent)
printf “Total Up Time: %2d:%02d:%02d %2d%% Sleep %2d%% Wake\n” ,hh1,mm1,ss1,Percent,WakeTime
}’

Save this script in /nfs/George/sleep.awk
To use the script run the following command.
cat /var/log/user.log1 /var/log/user.log|/nfs/George/sleep.awk > sleeptimes.txt

This should work without waking the system.

RAC

D’oh! I missed this! And I’d even replied to the thread… Sometimes bookmarks aren’t helpful…

Once the python script was fixed I was able to make it work. So here’s my script that runs in crontab every Saturday at 1AM - the backup is on a network connected MyBookLive. Its located and run from a share named System. the exclude.txt file can be found at /etc/nas/NSPT and the reason I copy it to my local share is I added the USB share when its plugged in so I don’t backup what’s on it.

mount //IP-ADDR-MyBookLive/NAS_BACKUP /media/WDSAFE -o password=“ROOT-PASSWORD”
sleep 6
cp /dev/null /shares/System/backup.log
sleep 3
rsync -rtDvH --delete --log-file /shares/System/backup.log --exclude-from /shares/System/exclude.txt /shares/ /media/WDSAFE >/dev/null
sleep 6
umount /media/WDSAFE
sleep 3
/usr/bin/python /shares/System/backup-email.py “Message from WD MyCloud” “System backup has been completed” 2>/dev/null

I did this too. And it works for me although I chose to backup only the folders I care about which come to about 2.9 Terrabytes. Photos and music mostly but also backups of some of the other systems in the house, Windows, Linux, SunOS, and apple.

One day I realized that although I had other drives which had most of the photos on them, as well as the original SD’s, it would take forever to reassemble the archive, so fool that I turned out to be, I bought a WD Elements 4tb USB drive, plugged it into the back of the 4tb MyCloud, and set up a safe point for sunday morning. When I got going that day, the backup had quit after several hundred gigs and locked up exports.

I had to reboot the system to get it exporting again. (you know, exportfs -a) Following Sunday same thing happened.

I can’t imagine how I could have been confused. I thought that if I bought WD’s USB drive and plugged it into the back of their system and used their software it would work. Silly me.

So now I have the rsync system up and running and it emails me reports of its success. Thanks so much for such an intelligent backup scheme. Too bad WD couldn’t muster the smarts or discipline (probably the latter) to provide something that works. I wonder how the folks who can’t do what we did get on?

Maybe they don’t backup.

john