Alternative to Safepoint - reposted

My Cloud OS 3 Personal & Network Attached Storage My Cloud
1

I posted an alternative to Safepoint under a previous posting commencing “Happy accident…”. The move to the new forum style corrupted the formatting. Here, for anyone who wants it is my original post, correctly formatted. It’s worked well for me for five months now, including a disaster recovery that went smoothly, I appreciate that some of the scripts could possibly be more elegant to those more experienced than myself, but this does have one advantage - it works!

This will back up any shares that you want, and is far more reliable, to my mind at least than Safepoint

Using WinSCP (easiest) or PuTTY create a folder called “Backup” on your USB drive, inside that folder create a separate folder for each of your shares. It may also be worth renaming the share name of your USB drive to something simple such as “USBbackup” which is the name I’ll use here.

To prevent your scripts being overwritten by future firmware updates, create a share called ‘system’ (for example) and place a folder in there called, say 'Tasks". I shall use these and the above names in what’s to follow.

With acknowledgements to Arty2, we’ll create the Python email script. Go into PuTTY then:

nano /shares /system/Tasks/mail.py

Paste (right click) the following into the editor

!/usr/bin/env python
-- coding: utf-8 --
from email.header import Header
from email.mime.text import MIMEText
from getpass import getpass
from smtplib import SMTP_SSL
import sys
# edit the areas between the quotes in the line line below with your own settings
login, password, server, recipients = “the e.mail.account.you’re.sending.from@gmail.com”, “password”, “smtp.gmail.com”, “recipient@domain.com”
send email
subject = sys.argv[1]
body = sys.argv[2]
msg = MIMEText(body, ‘plain’, ‘utf-8’)
msg[‘Subject’] = Header(subject, ‘utf-8’)
msg[‘From’] = login
msg[‘To’] = recipients
s = SMTP_SSL(server, 465, timeout=10)
s.set_debuglevel(1)
try:
s.login(login, password)
s.sendmail(msg[‘From’], recipients, msg.as_string())
except Exception, error:
print “Unable to send e-mail: ‘%s’.” % str(error)
finally:
s.quit()

Come out of the editor. Now we’ll create the backup script

nano /shares/system/Tasks/backup.sh

add your share names where share1 etc appear below

rsync -a /shares/share1/ /var/media/USBbackup/Backup/share1/
rsync -a /shares/share2/ /var/media/USBbackup/Backup/share2/

Continue adding as many shares as you like using the above syntax.

Finally, the email to confirm the backup completed:

python /shares/system/Tasks/mail.py “Message from WD MyCloud” “System backup has been completed successfully”

Having come out of the editor, make the script executable:

chmod +x /shares/system/Tasks/backup.sh

Now add a task with crontab, the following will run the backup at 0100 each day

crontab -e

add the line

00 01 * * * /shares/system/Tasks/backup.sh

Your backup will run at the required time and will send you an email when the task is completed.
That’s it for the backup.

For restoring, have a script ready, once again using nano, as my shares are quite large I wanted to keep an eye on progress so asked it to send me an email after each share had been restored:

nano /shares/system/Tasks/restore.sh
rsync -a /var/media/USBbackup/share1/ /shares/share1/
python /shares/system/Tasks/mail.py “Message from WD MyCloud” “Share1 restore has been completed successfully”
rsync -a /var/media/USBbackup/share2/ /shares/share2/
python /shares/system/Tasks/mail.py “Message from WD MyCloud” “Share2 restore has been completed successfully”

add further shares as you wish, exit and save

make this script executable

chmod +x /shares/system/Tasks/restore.sh

now a final script, this will enable you to send an instruction to the MC to execute the restore script unattended

nano /shares/system/Tasks/startrestore.sh

nohup /shares/system/Tasks/restore.sh > /dev/null 2>&1 &

exit and make this script executable

chmod +x /shares/system/Tasks/startrestore.sh

That’s it!!!

To recover after a wipe of your MC, once you’ve manually set up your shares again, configured the MC and given yourself SSH access (not forgetting changing the root password), use WinSCP to manually copy the backup of your ‘system’ share to the MC (should only take a min), then open PuTTY.

Give it the command:

. /shares/system/Tasks/startrestore.sh

The restore will start running, PuTTY will return almost immediately to the command prompt, the restore will run in the background without any further intervention, and you’ll get emails to tell you how it’s going.

Of course if you just want to restore the odd file or folder that you’ve accidentally lost you can use WinSCP to copy the files/folders across from the USB drive.
Hope this is of help. There may be typos or unintended line-breaks in the above for which I apologise.

6 Likes
2

Thanks for sharing.

1 Like
3

Hi

I’ve been a Mycloud user for 3-4 years already with 4 different Mycloud/Mypassport products that I have been using at my home and office. I have tried many many times to make safepoint work with the different advice given throughout the years to no avail. I actually gave up on it and have just been using Ubuntu Deja Dup on a dedicated device which basically defeats the purpose of Mycloud. Until I saw your post.

I just want to thank you for sharing this to the community even if I have yet to try it myself (just about to- just clearing up one mypassport drive to use as my backup). I just registered here on this forum to be able to thank you.

Keep it up.

1 Like
4

@ nicktee55

I’m happy to report back that I was able to make this one work except for the email.
For the scheduling I wanted the Mycloud to do rsync twice day a since there are a numbers of pc’s sync"ed to this device whose files (mostly MS office related) are constantly updated throughout the day. So I used this particular cron command instead for rsync’ng at 12 am and 12 pm:

00 01,12 * * * /shares/system/Tasks/backup.sh

However, on the email part I’m not yet able to make it work and have yet to figure out the problem.
Will post back for any development.

Thanks, man.

5

Might be your email server. With Gmail I had to change a gmail setting that allowed ‘less secure apps’

gmail security settings

6

Email is just a nice touch.

Alternatively, get it to simply write a message to a log file. Put the log file somewhere in admin user space, so you can see it from the network without having to SSH in.

7

When I run the python script I get syntax errors. I even typed the code by hand instead of copy/paste

8

Looks like the forum formatter has mangled the code, interpreting some characters as format commands. There’s certainly a missing # from the first line, which I’m sure should read

#!/usr/bin/env python

Oh; I’ve already tried to correct that in the original thread:

9

More than just that:

File “./kkk.py”, line 2
– coding: utf-8 –

10

I can confirm that the forum formatter has mangled the first two lines for reasons best known to itself. I’ve managed to now post the lines correctly by adding a space at the start of lines 1 and 2.

First two lines should be (if you copy and paste please delete the space I have had to insert at the beginning of each line)

#!/usr/bin/env python

-- coding: utf-8 --

There should also be a leading # on the line “send email”

I think it should be OK now, but any difficulties have a look at arty2’s post at Sending email with My Cloud

11

Use the ‘code formatting’ to stop it interfering with code; the </> icon in the format bar.

I guessed right about the utf-8 line, then…

12

See my full corrections at the link i posted. Was the link that hard to follow?

13

Great post. Set everything up, checked everything working, and then kicked off the cron only to discover … half an hour later … that it was part way through the third share and had stopped copying the first two and these were incomplete.

Stopped everything and dry ran the first rsync to be greeted with the error.

Stack Smashing Detected … Terminated.

So I’m currently using cp -R to make a full copy of all the shares before re-enabling the crontab entry in the hopes that it will cope.

I’m presuming that it’s the volume of files that it is trying to sync up. If I make a few changes then I can imagine it will work, but if something touches a large number of files, this is going to hit again.

Any idea how to avoid it?

14

Hi DarkRayven

Sometimes I have discovered that rsync can have problems with corrupted files, specially jpg and mp3 for some reason. Rsync tries, and tries again several times, and sometime quit or give up.

Have you tried rsync with log-file output, or double or triple verbose (-vvv)? May be you can find a clue there.

Also, you can try to look at what exit values rsync is returning to investigate further.

15

Tracked it down to an apparent lack of memory issue.The initial copy was going to be over 100,000 files (and that was just in one directory structure). If I ran the same rsync command again, it skipped the files that were copied and picked up from where it left off, only to fail again later. I think it had too much to cope with (2.4Tb of files in the first run)

I ended up aborting this for the initial copy and using

cp -Rp

instead to make an initial copy.

Once that had been done (took about 12 hours), I scheduled a backup script using

rsync --archive --delete

to keep the mirror copy in sync.

Touch wood it has worked so far, but I’ve had it send me an initial email before the backup starts warning me that if I don’t get a completed one afterwards, to check for fails.

16

I run this script once per week - backup between MyCloud and older MyBookLive:

mount //192.168.10.3/NAS_BACKUP /media/RSBACKUP -o password=“welc0me”

rc=$?
if [ $rc != 0 ]
then
/usr/bin/python /shares/System/backup-email.py “Message from WD MyCloud” “Failed to mount remote share to RSBACKUP”
exit
fi

sleep 6
cp /dev/null /shares/System/backup.log
sleep 3

rsync -rtDvH --delete --log-file /shares/System/backup.log --exclude-from /shares/System/exclude.txt /shares/ /media/RSBACKUP >/dev/null
rc=$?
sleep 6
umount /media/RSBACKUP
sleep 3

if [ $rc != 0 ]
then
/usr/bin/python /shares/System/backup-email.py “Message from WD MyCloud” “System backup has failed” 2>/dev/null
else
/usr/bin/python /shares/System/backup-email.py “Message from WD MyCloud” “System backup has been completed” 2>/dev/null
fi

17

Hi whsbuss

Nice mirror script…

I see that you are using -H option (preserve hard links), so this must be a mirror to another Linux file system.

I mirror my files to a My Passport Ultra connected to My Cloud. My Passport uses a Windows file system (maybe exFAT).

I‘m considering formatting My Passport to ext4 so I can have a more complete backup like yours. Is rsync working ok every week, or is the exit value sometimes different from 0?

Also I wonder why you stop execution of batch-file for 3 and 6 seconds. Is it necessary?

18

So far, rsync has worked flawlessly for a year now. Be advised that when you update the firmware the mount point on /media/RSBACKUP will be removed and will have to be manually reconnected.

I’m an old Unix guy and I like giving the operating system a few seconds to allow the shell to not conflict executing commands.

19

Bumping this old thread. Used a variation of the code whsbuss posted above to backup select folders to an attached USB hard drive. While the following code isn’t elegant and there are probably better ways to code it, it does appear to work for me.

The following mail.py code will use Yahoo.com email and will attach the backup.log file to the email.

#!/usr/bin/env python

# -*- coding: utf-8 -*-
import smtplib
from email.header import Header
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.MIMEBase import MIMEBase
from email import Encoders
from getpass import getpass
from smtplib import SMTP_SSL
import sys  

# edit the areas between the quotes in the line line below with your own settings. 
# Change the backup.log file location to match file to be attached to email.

login, password, server, recipients, logfile = "full.Yahoo.Email.Address.Goes.Here", "Yahoo.Password", "smtp.mail.yahoo.com", "Recipient.Email.Address", "/DataVolume/shares/system/backup.log"  

# send email

subject = sys.argv[1]
body = sys.argv[2]
msg = MIMEMultipart()
msg.attach(MIMEText(body, 'plain', 'utf-8'))
msg['Subject'] = Header(subject, 'utf-8')
msg['From'] = login
msg['To'] = recipients
part = MIMEBase('application', "octet-stream")
part.set_payload(open(logfile, "rb").read())
Encoders.encode_base64(part)
# Change file name in following line to match filename of attached file
part.add_header('Content-Disposition', 'attachment; filename="backup.log"')
msg.attach(part)
s = SMTP_SSL(server, 465, timeout=10)
s.set_debuglevel(1)
try:
    s.login(login, password)
    s.sendmail(msg['From'], recipients, msg.as_string()) 
except Exception, error:
    print "Unable to send e-mail: '%s'." % str(error)
finally:
    s.quit()

The reason for using this code was due to Safepoint being an all or nothing affair and wanting a backup process that would backup a select number of Shares and then email (with log file attached) me. The log file will show what files (if any) have been mirrored to the USB hard drive.

Can post full information/code for anyone interested.

1 Like
20

The email by python won’t work for firmware v 2 xxx.
Internally, it is a busybox. Its python does not have SMTP_SSL