Import files into WD My Cloud directly from another server - alternative to the lack of ftp client

My Cloud OS 3 Personal & Network Attached Storage My Cloud
1

I have bought the WDMC lately and I have an old DLINK DNS-321 as a previous NAS.  There is more than 1TB of data in the DNS321 and I wanted to import all that into the WDMC, wipe it out of the DNS-321 and make that device the backup storage place for WDCM Safepoints.

 

But how to transfer 1TB from a NAS to the WDMC ?

 

Both NAS are set as FTP server and both are connected to the same router.  But if I use my Pc (file manager or other tools) to copy/paste folder from the DNS-321 to the WDMC it is awfully slow because the data transits trough the PC ; upload from first NAS and download into the other simultaneously…

 

The better way is to SSH into the WDMC and import the data directly from there.  But there is no ftp client in the WDMC hence no way to ftp to the DNS-321 and get the data.  Perhaps the WDMC could be an SFTP client but my DNS-321 does not handle the secure shell…

 

Then I found this older post about the same question on a My Book Live : http://community.wd.com/t5/My-Book-Live/How-to-use-ftp-client-on-Mybook-live/td-p/250834.  It suggests the command “wget” to do the job.  That command is great for this, indeed.  The only thing missing in that post is how to list the content of the remote server, so you know the path to the data. I found how to do that somewhere else.

 

I now have a very simple and effective way of copying tons of data into the WDMC from another server: 

 

Prerequisites: 

 

-       Enable SSH on the WDMC  (WDMC interface -> Parameters, Network -> Services SSH (it will show the password as you turn it on))

 

 

-       activate FTP server on the other NAS

 

 

-       Have a user with read right to all folders in the other NAS

 

 

Procedure:

(let’s say that the WDMC IP address is 192.168.1.101 and the DNS-321 is 192.168.1.102)

 

-       SSH connect to the WDMC (easy with MAC, requires Cygwin with a Pc … does anybody have an alternative ?)

ssh root@192.168.1.101

 

 

-       List the content of the other server : 

curl -L ftp://<remote_user>:<remote_password>@192.168.1.102

 

Example :

WDMyCloud:~# curl -L ftp://..:..@192.168.1.102/

drwxrwxrwx    5 501      501          4096 Dec 31 17:03 Aimie

drwxrwxrwx    7 501      501          4096 Jul 12  2012 Invite

drwxrwxrwx    3 501      501          4096 Aug 30  2013 Nathalie

drwxrwxrwx    5 501      501          4096 Feb 25 22:45 Réjean

drwxrwxrwx    3 501      501          4096 Jun 19  2013 Réjean_20120306

drwxrwxrwx    3 501      501          4096 Sep 16  2012 WindowsImageBackup

-rwxrwxrwx    1 501      501            68 Jun 20  2010 desktop.ini

WDMyCloud:~#

WDMyCloud:~#

-       Let’s say I want to import all files under other server’s /Nathalie into her My Cloud  folder :

cd /shares/Nathalie 

wget -m -nv ftp://<remote_user>:<remote_password>@192.168.1.102/Nathalie

Note : -m is for mirror, -nv is for non-verbose

 

Example :

WDMyCloud:/shares/Nathalie# wget -m -nv ftp://<remote_user>:<remote_password>@192.168.1.102/Nathalie

2014-02-28 23:24:04 URL: ftp://rejean:*password*@192.168.1.102/Nathalie [663] -> “192.168.1.102/.listing” [1]

2014-02-28 23:24:04 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Nathalie [185] -> “192.168.1.102/Nathalie/.listing” [1]

2014-02-28 23:24:04 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Pictures/Nathalie [5353] -> “192.168.1.102/Nathalie/Pictures/.listing” [1]

2014-02-28 23:24:04 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Pictures/2010-06-07/Nathalie [282] -> “192.168.1.102/Nathalie/Pictures/2010-06-07/.listing” [1]

2014-02-28 23:24:04 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Pictures/2011-04-01/Nathalie [27699] -> “192.168.1.102/Nathalie/Pictures/2011-04-01/.listing” [1]

2014-02-28 23:24:04 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0032.JPG [2945677] -> “192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0032.JPG” [1]

2014-02-28 23:24:05 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0033.JPG [4243952] -> “192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0033.JPG” [1]

2014-02-28 23:24:05 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0034.JPG [3645003] -> “192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0034.JPG” [1]

2014-02-28 23:24:05 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0035.JPG [3668067] -> “192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0035.JPG” [1]

2014-02-28 23:24:06 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0036.JPG [3671609] -> “192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0036.JPG” [1]

2014-02-28 23:24:06 URL: ftp://rejean:*password*@192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0037.JPG [3549615] -> “192.168.1.102/Nathalie/Pictures/2011-04-01/DSC_0037.JPG”

…

-       The complete path of the imported data is replicated in the WDMC local folder hence the full path of the data is now “/shares/Nathalie/192.168.1.102/Nathalie/…”.  To fix that, do something like the following :

cd /shares/Nathalie             # be sure to be in the correct folder

mv 192.168.1.102/Nathalie/* .   # move all data to current folder
rm -rf 192.168.1.102            # remove old path and everything under 

. 

Voilà !

I get 10MB/s transfer rate like this which is good considering that the DNS-321 only has a 10/100 network port.

2 Likes
2

Hi rejdrouin,

Thanks for your instructional post. Hopefully other users will chime in and report if they experience similar speeds on their setups.

Edit:

Forgot to mention that users should be aware that SSH is not supported and this could void the warranty and render the drive unusable if performed incorrectly.

3

Hi  John012,

What is the recommended method for impoort large amount of data into the WDMC from another NAS or server ?   Something in the GUI would be usefull.

4

Using dd shouldn’t be a big deal over a network as all you should need to do is mount your share and run dd using that directory as the destination.

for example first mount the folder where you want to copy new data

mkdir /mnt/windows
mount -t cifs //server/sharename -o username=login,password=password /mnt/windows

then using dd command you can copy whatever you want

2 Likes
5

Thanks rejdrouin and andreaspf8585 … I think this will help a lot of users, I know for me it will. :slight_smile:

6

Hi andreaspf,

Thanks for your comment, I learned something new.

mount is really usefull, indeed ; it makes the remote NAS look like an internal folder of the WDMC :

WDMyCloud:~# mkdir /mnt/dns321

WDMyCloud:~# mount -t cifs //<PI address>/Volume_1 -o username=<remote user>,password=<remote passwd> /mnt/dns321

WDMyCloud:~#  ls -l /mnt/dns321
total 4
drwxrwxrwx 2 daapd 501 0 Mar 1 20:36 Aimie
drwxrwxrwx 5 daapd 501 0 Mar 1 21:36 Invite
drwxrwxrwx 3 daapd 501 0 Aug 30 2013 Nathalie
drwxrwxrwx 5 daapd 501 0 Feb 25 22:45 Rejean
-rwxrwxrwx 1 daapd 501 68 Jun 20 2010 desktop.ini
WDMyCloud:~# 

Now I can have  ls, rsync, mv cp, etc. commands executed on the remote NAS, great !  And I guess I can do the same from my pc (cygwin) to mount the NAS and the WDMC.

I am not familiar with dd but from what I see it does copy disk or partition to disk or partition.  I will not need that now that my NAS data is imported into the WDMC (using wget).  Also, it does require sepcial care when the input and output sides disk are not the same size.  

Question ; does dd overwrites the complete destination disk ?  Should I create a destination folder (under /mnt ?), dd to there and then use mv to have the data under the desired path ?

Do you know if, because dd does not handle files but disk sectors, it is significantly faster that other data transfer methods ??