Firmware 1.05.21 - File copy fails due to long .wdmc paths

Yes this is a problem. It’s not a bug per se, but as John Madden would say, '‘Ill conceived and poorly executed!’ The cloud service is creating hidden thumbnails in the .wdmc directories in order to supply low-res images to mobile apps without consumming a lot of bandwidth (and data cost) on mobile networks. However, because the .wdmc files are created by the Unix\root account you do not own them and cannot change them in any way. Any folder you attempt to delete or copy that contains a hidden .wdmc folder will fail.

If the .wdmc directories had been created using open permissions, then it’s likely we’d never have even noticed them and only blissfully enjoyed the data conservation and quick load times from mobile access. Instead it’s a major pain.

The implememtation has two major flaws.

  1. the permissions prevent deleting or copying our own directories, because that would in turn also modify the hidden .wdmc subdirectory. This is the major flaw.

  2. The thumbnail images are created immediately upon saving a JPG to the drive. When copying large numbers of JPG files, the data transfer slows to a crawl because the drive is trying to thumbnail everything as it arrives. Much better to wait unitl idle times to generate thumbnails.

I’m sure WD is aware of the issue and is fixing it, even if customer support doesn’t have any information. It’s as obvious a problem as they come. However, I can’t wait months for the next firmware update to correct it. I’ll try to find a workaround that corrects the issue without fundimentaly breaking the intention of the feature and I’ll post what I find.

Ben

Okay, here is the solution.

First, you’ll need to enable SSH access for the device. Open the device webpage and go to Settings | Network and turn on SSH. It will ask you to create a password. Do so.

Ok, now you’ll need an SSH client since Windows doesn’t come with one (don’t know about Mac).

Download PuTTY from here. I recommend just putting the putty.exe file into C:\Windows. Then you can access it anytime by running “putty” command.

Assuming you’ve put it where I said, go to Start | Run…

Enter putty in the run command, then enter.

Now you have the PuTTY Configuration dialog open. In the Host Name (or IP Address) field enter the name of your device. For instance WDMyCloudEX2. Or you can just put the IP address if you know it. Then press the Open button.

Now you’ll have an SSH shell open. It will ask you to logon. Use sshd for the username. Use the password you setup earlier.

Now you have a command prompt that looks like:

~ #

Type the following command and press enter:

find /mnt/HD -type d -name .wdmc -printf “"%p"\n” | xargs -n 1 chmod o+w -R

The drive will churn away for awhile and then you’ll have the command prompt again.

What has happened is all .wdmc directories, and their subdirectories, now have write permission added for all users. By default, these directories only have write permission for the owner and the owner group.

To double check the command was executed properly, type the following command and press enter:

find /mnt/HD -type d -name transcoded_files -printf “"%p"\n” | xargs -n 1 ls -ld

You should see a listing of all the transcoded_files directories. This is the directory just under .wdmc. We are listing these directories to ensure the permission wasn’t only changed on the .wdmc directory, but also it’s subdirectories.

The list should have entries like this:

drwxrwxrwx    3 root     share         4096 Nov 25 10:18 /mnt/HD/HD_a2/Public/Shared Music/Mercyful Fate/Melissa/.wdmc/transcoded_files

What we are looking for is the drwxrwxrwx portion. If it looks like that then success. If it looks like drwxrwxr-x then failure. Check the original command was type exactly as shown and try again.

Once successful close putty. Now you should be able to manipulate your folders without the root permission issue getting in the way. Unfortunately, if you add a new folder with JPG files to the drive, it will again create .wdmc subdirectories with limited permissions. So in that case run the command again if the folder needs to be manipulated.

Regarding the second issue of the thumbnail generation slowing down transfers, I suspect turning off the cloud access service before moving a large number of JPGs to the drive will solve it. However, I haven’t tested the theory yet. (Settings | General | Cloud Access = off)


A few additional notes for those who are interested and may want to take this further. I had hoped the user could simply be added to the owner group and that would fix it for all current folders and future adds.

The .wdmc directories are owned by root and the owner group is share. The directories are created with rwx permissions for both the owner and owner group, so it should be a matter of ensuring the user is a member of share and it should work fine.

However, I see the nobody user and the created user (ben in my case) are both members of share. So if windows explorer was logging into the network share using either of those users, then the .wdmc directory should be freely modified. I think that is the intention, but it isn’t working. Likely explorer is connecting to the share using some other credentials, a guest account or something. Anyway, if that was working properly then the whole issue goes away. But I don’t know enough about Unix to figure it out quickly and I need to get to work. So the above workaround will work for anyone who needs it. If someone finds a solution that works permanently please post. Otherwise, I’m sure a future firmware update will get this sorted out permanently.

I have almost done what you describe.

I turned Cloud Access OFF.

I opened a SSH shell.

I deleted all -wdmc folders and their contents.

like this: find /mnt/HD/HD_a2 -name .wdmc -exec rm -rf {} ;

Unfortunately something seems to turn Cloud Access back ON, now and then.

I have needed to redo the above a couple of times in the last few weeks.

I still think this as a bug.

A NAS is a Networked Attached Storage.

It should store files where I tell it to store them, nothing else.

It should NOT create anything by itself, without being specifically asked to do so.

Oh I don’t know, this device is marketed as cloud-like storage, not a simple barebones NAS. To support all the features it has it will need to do a bunch of things under-the-hood to make those features work. So, I think it should do whatever it’s designed to do, but do it well. That’s just how I feel about it, probably because I’m a software engineer myself. Design a feature well and users love it. Design a feature poorly and its worse than not having it at all. Maybe a better method would have been to put the thumbnails in a database or mirrored directory tree, instead of as subfolders, so that there was no chance the user would bump into them. 

Anyway, back to the point at hand, since you’d like to actually stop the files from being created at all, you’ll need to permanently stop the services that make them. Give this a shot:

From ssh execute these commands

/etc/init.d/wdmcserverd stop

/etc/init.d/wdphotodbmergerd stop

update-rc.d wdphotodbmergerd disable

update-rc.d wdmcserverd disable

The first two will stop the service that makes the files. And the second two will disable them from restarting after a reboot.

Let us know how you make out.

Does “cloud like storage” automatically imply that it it is allowed

to contaminate my file structure with unwanted entries?

I believe I read in another post that the update command is not available.

Can’t recall if I tried it myself, but I know I gave up on it.

Perhaps I’m wrong about that,

and just settled for disabling Cloud Access, and removing the unwanted folders.

You’re right about update-rc.d not being available. Apparently it was removed in more recent firmware versions.

I copied that suggestion from this post but didn’t test it because I’m not interested in killing the cloud service.

However, later in the thread is another suggestion that should work just as well. Simply remove execute permission from those services. Like this:

chmod -x /etc/init.d/wdmcserverd

chmod -x /etc/init.d/wdphotodbmergerd

That should prevent those services from being able to start. I don’t plan to test it, but if you do please report what you find.

thanks for the tips bdpratt

used for ex4 lets see if gets better performance

bdpratt wrote:

So if windows explorer was logging into the network share using either of those users, then the .wdmc directory should be freely modified. I think that is the intention, but it isn’t working. 

The .wdmc directory is just not visible when you pull up a share in Windows because directory and filenames beginning with . are hidden by Windows but you can see the .wdmc directories from a Mac (since a Mac OS has a linux kernel).

Thanks for contributing Cybernut. The issue isn’t about visibility, per se. The issue is that the .wdmc directory is a subdirectory of the users directory, but does not have the same permissions as its parent. So when the user attempts to modify his own visible directory, say for instance deleting it, the file system naturally also attempts to delete the child .wdmc subdirectory. That operation fails because the user doesn’t have permission to delete .wdmc. The user isn’t explicitly trying to delete .wdmc…in most cases the user doesn’t even know it exists, because as you say, it’s hidden. But modification to the users own directory, which should be freely allowed, fails because the hidden subdirectories reject the whole operation.

To see .wdmc and its contents i Windows,

you need to turn on visibility of hidden files and folders.

But,

as pointed out by bdpratt, visibility is NOT the issue here,

but visibility helps in understanding the issue.

Agree!  I have been waiting for a fix to this for MONTHA.  Support level 2 acted like it was normal since they were ‘system’ files

AGREE!!!   Same annoying issue.  I have all kind of files I created that can not delete!!!  

Because, detailed instructions have been posted previously for getting rid of the pesky autocreated .wdmc files.

 /etc/init.d/wdmcserverd stop
/etc/init.d/wdphotodbmergerd stop

chmod -x /etc/init.d/wdmcserverd
chmod -x /etc/init.d/wdphotodbmergerd

Then what happens if I hard power cycle it?  I bet these changes dont persist.

WD: Provide a PERMENANT solution to this nonsense.  You should _NEVER_ alter my file structure unless I tell you to.

I can confirm that these changes do not persist.  Again I had to setup a snippet in JuiceSSH on my phone so I can quickly disable these useless ‘services’ every time I reboot the device.  

Discovered that the creation of all the wdmc files was one of the main reasons I was having issues with webDAV shares and btsync.  Both the processes that create the .wdmc folders were fighting for CPU and RAM with everything else - vast majority of the 4TBs on my EX2 are photography backups, then all my music and video files have albums and box covers in the folders so combined with the general slowness of the device it was taking forever to work through them all.  Additionally, btsync uses 500 bytes of RAM per file being shared so the EX2 proliferating my photography folders with literally thousands of unnecessary files was what was causing btsync to intermittantly work one minute and stop working the next - as the EX2 no longer had enough RAM for it to run properly.  After deleting all the unnecessary files btsync now works completely fine with no issues.  

On seperate note - is there a list anywhere of all the processes that are safe to kill?  On reboot I have roughly 58 processes running and I’m sure like wdmcserverd and wdphotodmergerd a few of them are unneeded and just hogging up resources.

bdpratt wrote:

Okay, here is the solution.

 

First, you’ll need to enable SSH access for the device. Open the device webpage and go to Settings | Network and turn on SSH. It will ask you to create a password. Do so.

 

Ok, now you’ll need an SSH client since Windows doesn’t come with one (don’t know about Mac).

Download PuTTY from here. I recommend just putting the putty.exe file into C:\Windows. Then you can access it anytime by running “putty” command.

 

Assuming you’ve put it where I said, go to Start | Run…

Enter putty in the run command, then enter.

 

Now you have the PuTTY Configuration dialog open. In the Host Name (or IP Address) field enter the name of your device. For instance WDMyCloudEX2. Or you can just put the IP address if you know it. Then press the Open button.

 

Now you’ll have an SSH shell open. It will ask you to logon. Use sshd for the username. Use the password you setup earlier.

 

Now you have a command prompt that looks like:

~ #

 

Type the following command and press enter:

 

find /mnt/HD -type d -name .wdmc -printf “"%p"\n” | xargs -n 1 chmod o+w -R

 

The drive will churn away for awhile and then you’ll have the command prompt again.

 

What has happened is all .wdmc directories, and their subdirectories, now have write permission added for all users. By default, these directories only have write permission for the owner and the owner group.

 

To double check the command was executed properly, type the following command and press enter:

 

find /mnt/HD -type d -name transcoded_files -printf “"%p"\n” | xargs -n 1 ls -ld

 

You should see a listing of all the transcoded_files directories. This is the directory just under .wdmc. We are listing these directories to ensure the permission wasn’t only changed on the .wdmc directory, but also it’s subdirectories.

 

The list should have entries like this:

 

drwxrwxrwx    3 root     share         4096 Nov 25 10:18 /mnt/HD/HD_a2/Public/Shared Music/Mercyful Fate/Melissa/.wdmc/transcoded_files

 

What we are looking for is the drwxrwxrwx portion. If it looks like that then success. If it looks like drwxrwxr-x then failure. Check the original command was type exactly as shown and try again.

 

Once successful close putty. Now you should be able to manipulate your folders without the root permission issue getting in the way. Unfortunately, if you add a new folder with JPG files to the drive, it will again create .wdmc subdirectories with limited permissions. So in that case run the command again if the folder needs to be manipulated.

 

Regarding the second issue of the thumbnail generation slowing down transfers, I suspect turning off the cloud access service before moving a large number of JPGs to the drive will solve it. However, I haven’t tested the theory yet. (Settings | General | Cloud Access = off)

 


 

A few additional notes for those who are interested and may want to take this further. I had hoped the user could simply be added to the owner group and that would fix it for all current folders and future adds.

 

The .wdmc directories are owned by root and the owner group is share. The directories are created with rwx permissions for both the owner and owner group, so it should be a matter of ensuring the user is a member of share and it should work fine.

 

However, I see the nobody user and the created user (ben in my case) are both members of share. So if windows explorer was logging into the network share using either of those users, then the .wdmc directory should be freely modified. I think that is the intention, but it isn’t working. Likely explorer is connecting to the share using some other credentials, a guest account or something. Anyway, if that was working properly then the whole issue goes away. But I don’t know enough about Unix to figure it out quickly and I need to get to work. So the above workaround will work for anyone who needs it. If someone finds a solution that works permanently please post. Otherwise, I’m sure a future firmware update will get this sorted out permanently.

this post is GOLD.

I have a ffew questions for WD, since they state that .wdmc folder is normal as it is part of the system:

  • If i have to backup all my 30k pictures from over 150 folders to another location, let’s say a windows server,  why do i have to also copy the .wdmc and triple the amount of pictures?

  • if i have to give access to someone to take all the picture folders via FTP or VPN, why does he have to copy also the .wdmc folder since he can’t otherwise ?

  • if i have to copy the picture folders on another NAS, why do i have to copy the .wdmc since the other NAS doesn’t know what that folder is ?

Wow that last post was a lot of stuff.

Lets try to simplifiy.  Anyone doing this stuff already knows about SSH.

Remove all WDMC .dirs:

find /mnt/HD/HD_a2 -name .wdmc -exec rm -rf {} ;

Stop the service:

killall wdmcserverd ; chmod -x /etc/init.d/wdmcserverd
killall wdphotodbmergerd ; chmod -x /etc/init.d/wdphotodbmergerd

Must be done again after hard power cycle.

After many iterations with WD support
they claimed that this is how the EX2 is designed to work.

That is when I ran out of patience,
and replaced the EX2 with a Synology DS713+.

All issues are gone now.

//AndersJ

I wanted to verify; I must do all these steps because the EX2 has taken all my files and placed them in a .wdmc hidden folder for which I can see, because I have hidden folders visible and within these .wdmc hidden folders is another folder called transcoded and within that folder are folders with as mentioned in a previous post a long file name consisting of letters and numbers, roughly over 35 characters long ?

[Original Folder Name] > [.wdmc] > [transcoded folder] > [35+ character folder] 

Your description is almost correct.
You put your files where you want them.
After that,
the EX2 creates the unwanted folders and files in the folders where you put your files.

Anders J