Hi to all and Happy new year.
Since I upgraded my EX2 ultra to OS5, I’m no more able to run my weekly backup from QNAP server. I set the rsync process following the instructions as the attached file and all was always perfect till the upgrade.
I have realized that something has changed (new rsync version updated to 3.1.3-6 and the root command now is sshd instead root), but I haven’t find the solution yet. QNAP find the connection through rsync but it gives me an error in the transmission speed.
Is anybody experienced my same issue? Have you find a work around to solve it?
@alibabau can you provide the rsync command from the QNAP device?
You should be able to see the full command string via SSH on QNAP by running is ‘ps |grep sync’ command after the QNAP backup job is started.
One change to rsync in OS 5 is that OS 5 doesn’t support the blowfish encryption cipher so if QNAP is using blowfish, the rsync process that transfers the data will fail.
I have now realized that the OS5 doesn’t support the blowfish encryption. That’s the reason. On my point of view this is a very big issue so that I have switched off the my cloud forever since it has no more reason to live.
No words to describe this downgrading.
This is the latest posting on rsync - I have a few comments
I am testing just using the USB backup app ( that just got updates since the original OS5 update )
The commands are not working like the older version
I am hoping someone that has looked at the source can tell what changed.
I am looking into why the copy is replacing large 100GB+ Acronis backup files when NOTHING is changed on the NAS ( you can see it moving over a extra file with a random file extension then replacing the old file
I am using copy as it is the closest to the old Mirror backup command.
this link may help on other rsync items
here is what I thing may be different
USB disk format
–modify-window
When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situations. In particular, when transferring to or from an MS Windows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second).
So using any of the FAT/ EXFAT formatting on a external backup disk may not work well
( most suggest Unix or basic windows only )
rsync finds files that need to be transferred using a “quick check” algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file’s data does not need to be updated.
– one option is checksum —
-c, --checksum
This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a “quick check” that (by default) checks if each file’s size and time of last modification match between the sender and receiver. This option changes this to compare a 128-bit checksum for each file that has a matching size. Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer (and this is prior to any reading that will be done to transfer changed files), so this can slow things down significantly.
Just suggesting we ask a few of our regular poster if they know from the backup source code how the command options have changes.
I use rsync from a remote server to pull data from my EX2 ultra. I was getting rsync errors which prevented data transfers to complete, for example:
receiving incremental file list
unexpected tag 88 [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(1169) [receiver=3.0.3]
rsync error: sibling process terminated abnormally (code 16) at main.c(1524) [generator=3.0.3]
Maybe this is related to your problem or just an additional one.
I discovered that the WD is supplying a bizzare version of rsync with OS5:
mc$ /usr/sbin/rsync --version
rsync version 3.1.3 protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
no socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, no ACLs, xattrs, iconv, symtimes, prealloc
Panic Action: “xterm -display :0 -T Panic -n Panic -e gdb /proc/%d/exe %d”
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
mc$
Note the “Panic Action:” line. This is obviously a debug build of rsync. Bizzare because OS5 does not provide xterm or gdb.
After I override the WD version with one from debian (raspberian) the errors go away and my remote rsync now works properly:
mc$ /usr/bin/rsync --version
rsync version 3.1.3 protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, prealloc
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
mc$
Notice the placement of the rsync override link is in /usr/bin and not /usr/sbin.
I copied rsync binary from my Raspberry Pi. Then I created a persistant softlink /usr/bin/rsync to the version I downloaded from the raspbery. To make it persistant there needs to be a script to re-create the link that runs at boot. My script does other things but here is the part that creates the link: (change GOOD_RSYNC to path you placed the copied rsync binary, I suggest somewhere on /mnt/HD/HD_a2/ )
GOOD_RSYNC=/cust/bin/rsync
RSYNC_LINK=/usr/bin/rsync
if [ -x $GOOD_RSYNC ]
then
ln -sf $GOOD_RSYNC $RSYNC_LINK
echo $RSYNC_LINK set to $GOOD_RSYNC >> $LOG
else
echo $GOOD_RSYNC is missing or not executable >> $LOG
fi
unset GOOD_RSYNC RSYNC_LINK
To get my script to run a boot I installed the Internal Backups app and added a line to call it in
/mnt/HD/HD_a2/Nas_Prog/InternalBackups/start.sh
If you don’t have a raspberry Pi you should be able to download the .deb file from the raspberry repository and use archive tools to extract the rsync binary. The following to works for me:
10/22/21 Note: WD uses rsync for remote backups. It is likely they had a reason to supply a custom build of rsync. If remote backups are not working properly it would be better to ask WD for a solution.
I was able to get a Synology NAS to run HyperBackup to my MyCloud that was unfortunately upgraded to OS5. This is using the latest OS5 firmware available.
Observations:
This only works if ‘Encrypted Transfer’ is selected in the HyperBackup app with port 22 and the ‘sshd’ username.
HyperBackup sends the rsync password as plaintext, not base64 as MyCloud uses in /etc/rsyncd.secrets (many checks of /var/log/daemon.log on MyCloud and peeking at rsync source code to figure this out… “no rule” then “password mismatch”).
HyperBackup doesn’t let you specify a separate SSH username and password and rsync username and password, creating the problem.
MyCloud expects ‘sshd’ as the username when connecting via SSH, but the rsync ‘auth’ entries for shares in /etc/rsyncd.conf only include the ‘root’ user, which requires the backup username to be ‘root’, but you can only specify 1 username!
You don’t have to restart rsync on the MyCloud after editing rsyncd.conf and rsyncd.secrets.
You DO have to restart sshd if you edit /etc/ssh/sshd_config to add ‘root’ as an allowed SSH user (use ‘ps aux | grep sshd’ and re-enter that command line exactly as shown after using ‘kill’ on the sshd PID). I haven’t found the proper WD script to elegantly restart sshd.
With those experiment observations out of the way, these are basic notes to get Synology’s HyperBackup to work with a WD MyCloud running OS5 (temporarily until power cycle).
Allow Synology HyperBackup to work with MyCloud OS5: (only for those who love MyCloud ssh command lines and editing files with ‘nano’)
First in the MyCloud OS5 dashboard enable Settings->Network->SSH and set the sshd password.
Next in Settings->Network->Remote Backup Server with same password as sshd.
Option 1: Via ssh to MyCloud: Use “nano /etc/rsyncd.conf” to comment out or rename ‘auth = root’ line of share being used for the backup (i.e. change ‘auth’ to ‘xauth’).
Option 2: Via ssh to MyCloud (I used PuTTY): Use “nano /etc/rsyncd.conf” to add ‘sshd’ to the auth line (or replace ‘root’), then use “nano /etc/rsyncd.secrets” to append a new line with “sshd:PutPlainTextPasswordEnteredAboveHere” (no quotes and password is NOT base64 encoded here).
Any reboot or reconfiguration will require editing the rsyncd configuration files again, everything gets restored on reboot.
I was able to edit /etc/ssh/ssh_config to add ‘root’ as an allowed user and restart sshd, but still had to change the rsyncd.secrets password to be the plaintext root password instead of base64 and have HyperBackup work that way also.
I’m not sure why using a Linux or Cygwin command line rsync doesn’t have these problems doing the same thing to the MyCloud OS5, it could use sshd@1.2.3.4:/shares/MyBackup without issue.
I’m hoping these clues could steer someone else more experienced with Synology and MyCloud products to offer a better workaround. It seems like HyperBackup would need to be enhanced to separate the SSH and rsync username&passwords to have a real solution. The root issue is the MyCloud’s rsync ‘auth’ authentication check and what username and password is provided by HyperBackup.
I’m looking the reverse solution: Backup WDMyCloud to Synology, so Hyperbackup is of no use. I implemeted RSYNC in Daemon mode in WD and execute a cron job on the Synology within my LAN. Not the best, but Synology is dreadful…I’d rather prefer WD My Cloud.
