"No space left on device" but df shows 1.22 TB available!

[taamberg]

Hi all-

I’ve tried searching here on the forum for an answer to this, but haven’t hit one that matches my scenario, yet.

I have a 4 TB MyCloud EX2 - set up in a RAID1 config, and have been moving files from an older Mac (acting as a server) over to it.  Media and cloud serving is turned OFF.  The only thing unusual I’m doing here is that, due to wretched performance issues for folders full of many small files (like the “Library” folder in an OS X user account), I’ll tar the directory on the Mac side, transfer the tarball, and then extract the tarball on the EX2 by using its ssh access feature.  

However, I can no longer do that, as the command reports “No space left on device”.  Indeed, so does Mac OS X when trying to write anything to it.  However, the HTML gui and df tell me I have 1.22 TB available!

Here is the report from df

------------------------------:

root@myWDEX2 / # df

Filesystem           1K-blocks      Used Available Use% Mounted on

%root%                   55529     29493     23169  56% /

/dev/ram0                55529     29493     23169  56% /

mdev                    254368        32    254336   0% /dev

ubi0:config              12500       272     11556   2% /usr/local/config

/dev/loop0               94976     94976         0 100% /usr/local/modules

tmpfs                     1024         0      1024   0% /mnt

tmpfs                    20480      1216     19264   6% /var/log

tmpfs                   102400      1568    100832   2% /tmp

/dev/sda4               967576     27856    929400   3% /mnt/HD_a4

/dev/sdb4               967576     17692    939564   2% /mnt/HD_b4

/dev/md1             3901593320 2608955584 1253609688  68% /mnt/HD/HD_a2

/dev/sdc1            3906983932 2987141176 919842756  76% /mnt/USB/USB4_c1

/dev/sdd1            3906983932 3332476464 574507468  85% /mnt/USB/USB1_d1

The internal RAID1 is /dev/md1, and that shows that it is only 68% full.

When I go to do a du  report on /, this is what it shows:

---

root@myWDEX2 / # du -s -h -x /*

24K /CacheVolume

567K /bin

32K /dev

4.2M /etc

5.0K /home

3.6M /lib

0 /linuxrc

12K /lost+found

16K /mnt

0 /nfs

1.0K /opt

du: cannot access `/proc/3273/task/3273/fd/4’: No such file or directory

du: cannot access `/proc/3273/task/3273/fdinfo/4’: No such file or directory

du: cannot access `/proc/3273/fd/4’: No such file or directory

du: cannot access `/proc/3273/fdinfo/4’: No such file or directory

0 /proc

1.0K /root

1.0K /sbin

1.0K /shares

0 /sys

1.6M /tmp

3.1M /usr

17M /var

---

Can anyone guide me to a solution here?  Thanks!

[Hamlet]

Hello,

Make sure you didn’t set any quota rules

check page # 40 of the user’s manual for more info.

http://www.wdc.com/wdproducts/library/UM/ENG/4779-705119.pdf#page=45

[taamberg]

No, no quota rules are set.  

Indeed, I only have an “admin” account for purposes of changing settings, and I only transfer files to it as “guest”.  (Using the SSH login, I can see that this gets set in the filesystem as user “nobody”.

Does the “nobody” account have a hidden quota limit?  There is no “quota” command on the device, at least not in any of the set paths.

[taamberg]

I called tech support and they had me reset the device (without reformatting the drives).  I thought at first this fixed the problem, but the reality is that it only helped out a tiny amount.

I suspect that the reset cleared out cache files, opening up some space, but I soon ran into the same problem.

Does anyone have any advice here?  It seems ridiculous that I can’t use a full third of the capacity I’m paying for!!

[Hamlet]

Hello,

We have passed this along to support. Someone will be contacting you soon. Please check your private messages.

[cyriller]

Hello,

thanks to share your feedback on this topic.

I got the same issue of “no more space available”. I re-initialze it and it’s OK bur problem gets back.

[klaasbaaaas]

Dear Hamlet, isn’t the purpose of this forum to help each other with problems?

By sending the answer to his queries by PM is a total waste in my opinion.

Can you please share the solution with the rest of the world or do I really have to create another topic with exactly the same questions? Because like many many others, I experience the same as the TS. but only with a WD MyCloud.

Thank you.

[benramadan]

Hamlet, please could you share the solution to this problem as I too have hit 100% capacity on %root%.  I’m on an EX4.

df -h

Filesystem                Size      Used Available Use% Mounted on
%root%                   54.2M     54.2M         0 100% /
/dev/ram0                54.2M     54.2M         0 100% /
mdev                    251.6M     64.0K    251.5M   0% /dev
cgroup_root             251.6M         0    251.6M   0% /cgroup
ubi0:config              12.2M    696.0K     10.9M   6% /usr/local/config
/usr/local/tmp/image.cfs
                         86.4M     86.4M         0 100% /usr/local/modules
tmpfs                     1.0M         0      1.0M   0% /mnt
tmpfs                    20.0M      1.6M     18.4M   8% /var/log
tmpfs                   100.0M      1.8M     98.3M   2% /tmp
/dev/sda4               944.9M     47.2M    897.7M   5% /mnt/HD_a4
/dev/sdb4               944.9M     17.3M    927.6M   2% /mnt/HD_b4
/dev/sdc4               944.9M     17.3M    927.6M   2% /mnt/HD_c4
/dev/sdd4               944.9M     17.3M    927.6M   2% /mnt/HD_d4
/dev/md1                 10.8T    692.3G     10.0T   6% /mnt/HD/HD_a2

[Hamlet]

Hello,

The WD Community was created for Users -such as yourself- to help one another and to share your experiences and ideas with one another.  As Moderators, we are simply here to facilitate and oversee the Community. If you have a question or issue related to the WD Community itself, us Moderators will be happy to respond and assist you. However, if you are looking for technical support from WD, you will need to contact them directly.

To Contact WD for Technical Support:

http://support.wdc.com/contact/index.asp?lang=en

[scalax]

I have exactly the same issue, when i try to create a folder or a simple text file on the nas, in any of the shares available i get the no space left on device message.

This happens if i use windows network resources, the wdmycloudex2 dashboard (either to create a folder or a new share) and even if i try to create the folder with ssh.

However i have plenty of free space on the NAS

root@WDMyCloudEX2 Private # df -h
Filesystem Size Used Available Use% Mounted on
%root% 54.2M 30.2M 21.2M 59% /
/dev/ram0 54.2M 30.2M 21.2M 59% /
mdev 248.4M 32.0K 248.4M 0% /dev
ubi0:config 12.2M 404.0K 11.2M 3% /usr/local/config
/dev/loop0 94.1M 94.1M 0 100% /usr/local/modules
tmpfs 1.0M 0 1.0M 0% /mnt
tmpfs 40.0M 1.2M 38.8M 3% /var/log
tmpfs 100.0M 2.1M 97.9M 2% /tmp
/dev/sda4 944.9M 387.3M 547.6M 41% /mnt/HD_a4
/dev/sdb4 944.9M 17.3M 917.5M 2% /mnt/HD_b4
/dev/md1 1.8T 180.7G 1.6T 10% /mnt/HD/HD_a2

[SBrown]

Hello scalax,

1. What is returned by the df -i command?
      Does it say 100% on /mnt/HDD/_XYZ  ? (XYZ being the name of your HDD device)

2. What’s the firmware version in the UI or the output to the command below?

    cat /etc/version

If the firmware version is 2.10.302, then you’ve upgraded to My Cloud OS 3 already
and you can run the following command as root to correct your out of inode issue if ineed you’re out of inodes.

    /usr/sbin/inode_growth.sh

If the returned version is not 2.10.302, then please contact support for assistance.

Regards,

Samuel Brown

[webdigit]

Hello,

Same issue.
EX2 with 4T and 3,6T free
Impossible to copy files anymore.
Here is a screenShoot : Dropbox - Capture d'écran 2015-12-11 08.44.00.png - Simplify your life
I need some help
Thank you
JC

[scotty03921]

I haven’t yet run out of space, but I seem to be bleeding storage space like crazy on my EX2 (2x3TB Red, JBOD, v. 2.10.310), which I received Dec 5 this year. I manually mirror files from a subdir of Public on Vol 1 to Vol 2, big GB mp4s. Aside from a 65 GB .dmg, everything of any size on the EX2 is just those mp4s.

tl;dr: Recycle bin can contain a lot of used storage, and seems to do so on Macs, even though you think you’re using AFP rather then SMB (Time Machine can’t use SMB). Waiting for Support to respond wi the best way to deal with this on a Mac.

Details included for Support and WDs dev team:

Vol 2 looks fine, showing 675G usage, about as expected, in the Share profile of the dashboard. Other shares except Public show usage in KB except Vol 1, which also shows 675GB usage, but total available space is << 5TB at 3.96 GB.

OS X 10.11.1 shows 2.15GB available on Vol 2, but only 1.77GB available on Vol 1, though they have the exact same content wrt large mp4 files.

From the shell, things are anomalous-looking, but bottom line is:

EX2: df -h
…
/dev/sda2 2.7T 1.0T 1.6T 39% /mnt/HD/HD_a2
/dev/sdb2 2.7T 712.8G 2.0T 26% /mnt/HD/HD_b2

Groveling down in Vol 1 with du -sh, I get this anomaly:

EX2: du -sh /mnt/HD/HD_a2/
1.1T /mnt/HD/HD_a2/

EX2: du -sh /mnt/HD/HD_a2/*/
295M /mnt/HD/HD_a2/Nas_Prog/
20K /mnt/HD/HD_a2/P2P/
675G /mnt/HD/HD_a2/Public/
8.0K /mnt/HD/HD_a2/SmartWare/
8.0K /mnt/HD/HD_a2/TimeMachineBackup/
16K /mnt/HD/HD_a2/lost+found/

HD_a2 is using 1.1TB, but summing its subdirs shows only 675G. A chunk is missing…

There are no symlinked subdirs messing up the total, as seen here:

EX2: ls -l /mnt/HD/HD_a2/
drwxrwxrwx 5 root root 4096 Dec 6 10:57 Nas_Prog
drwxrwxrwx 5 root root 4096 Dec 10 21:44 P2P
drwxrwxrwx 11 root root 4096 Dec 16 23:12 Public
drwxrwxrwx 2 root root 4096 Dec 12 23:30 SmartWare
drwxrwxrwx 2 root root 4096 Dec 12 23:30 TimeMachineBackup
drwx------ 2 root root 16384 Sep 18 10:35 lost+found
lrwxrwxrwx 1 root root 32 Dec 6 10:32 plex_conf → /mnt/HD/HD_a2/Nas_Prog/plex_conf

EX2: du -sh /mnt/HD/HD_a2/plex_conf/Plex\ Media\ Server/*/
51M /mnt/HD/HD_a2/plex_conf/Plex Media Server/Cache/
4.0K /mnt/HD/HD_a2/plex_conf/Plex Media Server/Crash Reports/
1.1M /mnt/HD/HD_a2/plex_conf/Plex Media Server/Logs/
4.0K /mnt/HD/HD_a2/plex_conf/Plex Media Server/Metadata/
3.4M /mnt/HD/HD_a2/plex_conf/Plex Media Server/Plug-in Support/
60K /mnt/HD/HD_a2/plex_conf/Plex Media Server/Plug-ins/
8.0K /mnt/HD/HD_a2/plex_conf/Plex Media Server/Thumbnails/

Where is the 1.1TB - 675GB missing space?

Looks like it’s mostly here:

EX2: du -sh /mnt/HD/HD_a2/.!*
392G /mnt/HD/HD_a2/.!@#$recycle

(What a dir name!) I did turn on Recycle bin, but noted the following in the Manual:

Note:
The Recycle Bin is only available for Windows SMB files.

and

Note: Recycle Bin is only available for SMB protocols.

(The first statement needs a bit of work, since SMB is a protocol, not a file type and no longer a creature of Win exclusively)

The Finder does see the Recycle bins and they do have GBs in them, many GBs on Vol 1.

Since it’s a Mac, and top on the EX2 shows /usr/sbin/afpd going crazy at 50-70% CPU and smbd and nmbd at 0% CPU when I copy an mp4 to it from a different NAS, I believe(d) I am not using SMB (or else the note above is wrong, and Recycle is enabled for AFP). To be clear, there are no Win PCs in the place, only Macs, iThings, and a couple WD MyBook Lives, and I have no backups scheduled on any computer except Time Machine to USB drives.

I had turned on the Recycle on both Vols in case friends or family came a-visiting with Win PCs in hand. I haven’t set up any Active Directory on the NAS, and don’t want to, at least not until it’s needed.

So because no AD setup, I can’t find the Clear button referenced on p 118 (I assume it shows up somewhere in the AD setup, the Manual doesn’t really say where it’s to be found):

The Recycle Bin contains any files deleted from the WD My Cloud EX2 device. To clear the recycle bin for your device, click Clear.

In any case the Manual seems out-of-date wrt latest firmware, with many screens appearing different.

I tried disabling Recycle bins on both Vols but the space is not reclaimed,

A thought about Dashboard: when user disables Recycle and there are files in the Bin, ask whether s/he wants to delete the files; could save some grief.

I guess I could use the shell to delete the files, but the dir name /mnt/HD/HD_a2/.!@#$recycle is a beast to work with in bash.

I did try deleting Recycle bin contents from Finder before I disabled it in the Dashboard, but there was no apparent progress, so I hit Stop in the Finder progress window. I do see that some space was freed up during the apparently inoperative Finder delete:

EX2: du -sh /mnt/HD/HD_a2/.!*
282G /mnt/HD/HD_a2/.!@#$recycle

a gain of 110 GB, so maybe I’ll re-enable Recycle and retry the Finder delete.

Anyway, I’ve got a ton of space still in use on the Recycle bins, so this may be the problem for other folks too, especially if they’ve turned on Recycle, deleted lots of stuff, then turned it off again, rendering the storage usage effectively invisible without Ninja shell tricks.

[scotty03921]

Correction (I hit Esc in the browser window because I’d been using bash and vi so much while writing this that my fingers forgot I shouldn’t do so in the Submission window:-) :

but total available space is << 6TB at 3.96 GB.

and the request:

Please advise the best/easiest way to clear files that have already entered the Recycle Bin, without needing to set up AD.

[scotty03921]

Last tedious details: I started with default RAID1, copied almost all the mp4s to it in RAID 1 mode, then decided protection from accidental deletion was more important than drive failure protection, so I split the RAID to JBOD, leaving Vol 1 as the active drive, formatted Vol 2, and copied all files over from Vol 1, using Finder in order to retain Tags. Don’t know if this matters.

Also, while working on a .py to drive the fan speed per user taste (rev 0.1 is almost tested and ready, I’ll post in the other thread when I’ve got it done), I did 3-4 Finder Duplicates of a dir containing tons of my mp4s, in order to work the NAS hard and raise the temps inside. When done, I of course deleted the files, with Recycle Bin enabled on both Vols…

Hence the big Recycle Bin usage.

[scotty03921]

Re-enabled the Recycle Bins on both Vols, used Finder to delete the stuff there. The progress report doesn’t update (very often) but it does actually delete them, returning the space for use.

It’s all good.

The final question: does Recycle Bin work with AFP connections, or only with SMB as documented?

I’m still pretty convinced it’s using AFP on my system, even though I’m not using Time Machine to the NAS.

[aplocher]

I’m having same issue. EX2, 2 x 2TB drivers (RAID1 - mirrored for a total of 2TB usable space). I was down to less than 100GB, started receiving this error, and cleared up about 580GB free now (says df -h and the web UI), but still says it’s full.

I can’t even create a folder in Windows Explorer, it says “Please free up 0 bytes, not enough free space”

[SBrown]

@aplocher what does the df -i command return?
If you Mirror, EX2 or EX4 are out of inodes, then check this KBA

http://support.wdc.com/knowledgebase/answer.aspx?ID=14954

[aplocher]

Thanks for the response, I got this addressed back when I originally asked the questions. As you’ve alluded to, I ran out of I-Nodes - df -i was returning “0”.

I kept Windows File History snapshots on this drive for about 1.5 years, including my entire source code dir (with a ridiculous amount of small files). There were a LOT of small files and I just ended up needing to truncate my backup down to about ~1yr of history.

[Budbo]

Has anyone figured this out?