12.2MB/s write speed seems really low.
So, I was wondering if it’s just the max I can get from this device or if something with its configuration might be wrong.
Thank you for your help!
BTW: My firmware version is 2.21.126.
CPU utilization is around 20-70% during the write process.
I’m surprised that command worked. ON the V4.xxx firmware it gets the following error.AbbottandCostello:/nfs/George# dd if=/dev/zero of=./testfile bs=1G count=1
dd: memory exhausted by input buffer of size 1073741824 bytes (1.0 GiB)
But this command works
AbbottandCostello:/nfs/George# dd if=/dev/zero of=./testfile bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 10.8101 s, 99.3 MB/s
So the reason why I see high iperf network speeds of up to 930MBit/sec but only 15-16 MBytes/sec file transfer speed via scp is not the hard disk write speed. I could have also checked that via running the following command on my laptop that is connected to the MyCloud:
which gives also only 15-16 MByte/sec transfer speed without any hard drives invoked.
It turned out that using a different ssh cipher increases file transfer speed. With security in my network not being an issue, I now use
ssh does not allow for unencrypted file transfers. So using netcat might be an option. @rac8006
Since you seem to know the MyCloud device and Linux well, do you know whether I can get netcat for the busybox OS on the MyCloud?
I have no knowledge of netcat. It may be possible to compile netcat from scratch. But you need to compile it with pagesize=64k and link with -static. The first thing you need to do is create a build environment for the gen2 My Cloud.
You can search the forum to instructions to setup a build environment.
Thanks for your help. Setting up a build environment sounds like a lot of work and probably involves installing some 3rd party software on the device (haven’t yet researched which steps are necessary though). I will first try to directly transmit via tcp sockets. I have seen that the busybox OS on my MyCloud has python installed. Should be possible to code up a tcp-socket client there.
Ok, I have now finished TCP-socket client and server in Python. The server is running on the MyCloud device and receives unencrypted data (split into packets of size 1MB by the client). With this I get file transfer speeds of 78MB/sec which is close to the max disk write speed f 85MB/sec. I have checked the md5 checksums and the data is the same on both client and server side.
So, the conclusion is that all data transfer tools that depend on ssh (like scp) cannot get to the max Gigabit Ethernet speed (or the max write speed of the MyCloud disk – whatever is lower). The reason is that encryption takes too much overhead. Using the arcfour cipher will get you some improvement, but not much. Newer versions of openSSH seem to have even faster ciphers but these are not supported by the WD MyCloud SSH version.
It is unclear to me what you are trying to do. Why use ssh to transfer files? Why not use the mapped drive method?
You should be aware that data being transferred over the network is broken up into blocks of 1500 bytes.
I was trying to find out why scp cannot achieve the max Gigabit Ethernet speed. Don’t know what “mapped drive method” means. I have had mounted the MyCloud disk via NFS to my Linux Laptop before (done via fstab). That gave me some weird behavior when shutting down Mint (I think I would have had to add a script that explicitly unmounts the device at shutdown).
When I wrote that the file is split into 1MB chunks, I meant that this happens before the data is passed to the TCP socket (smaller files one could possibly completely load into RAM and then send to the socket).
In Windows we map the My Cloud Public folder to a drive letter. Then access the My Cloud using the drive letter the same way that we access any other drive. In Linux we can mount the drive using samba and access the drive like any local drive on the Linux system. I have an alias on my Linux system. I use the following alias to mount my network disk.
mount -t cifs //192.168.xx.xxx/Volume_2 /Volume_2
After this I just cp files to /Volume_2
Not sure why you were using scp?
Yes, like I said, I have been using something similar before - but using nfs instead of cifs. Plus, I did the mount automatically in fstab. I had abandoned that solution because it made problems at shutdown. Had forgotten that the performance with
sudo mount -t nfs MYCLOUDIP:/nfs/user /media/nas
is not bad. I get around 50MByte/sec. In order to get around any issues at shutdown I could write a little copy script that mounts, copies and un-mounts. BTW: How much do you get when you copy a 1GByte file via cifs mount?
The reason why I used scp in the first place is that I work a lot on remote machines via ssh. And then it’s kind of natural to use scp. However, I am always connected to these machines via the internet with much lower bandwidth available. So I had never to care about scp encryption overhead. Actually, I was never aware that the overhead could be an issue. My first, incorrect tests with dd also set me on the wrong track believing that the disk speed was the culprit. Thanks again for pointing out that I should use smaller blocksizes when using dd on the device.
Also: Coding up the TCP socket connection was really fun. Plus, I get very close to the max disk write speed. It even does public/private key authentication. And now I also know how to use python on the MyCloud device + how to install new Python packages there.
