Solid red LED light - cannot mount file system

My Cloud OS 3 Personal & Network Attached Storage My Cloud
6

Hi All, as mentioned I managed to fix my red solid led issue and get access to all my files again. Here is how I did it… Apologies in advance, this is a long guide with a lot of info. I wanted to be as descriptive as possible.

DISCLAIMER

I’m not a Linux expert - just a guy who had important files on my 2TB WD My Cloud, so was willing to do anything to get them back. I’m also not claiming that my solution works for everyone, it did for me, so that’s why I’m sharing.

Please only proceed with caution, as most of what I’ve written below requires that you have super user (root) privileges to the operating system running your WD My Cloud, which can be very dangerous, because it’s very easy to destroy the whole thing by running the wrong commands. I won’t take any responsibility if you break your WD My Cloud - you’ve been warned.

Also, I’m not sure if this will void your warranty or not, please check with WD if you’re not sure before proceeding.

Having said all of that, I will help you where I can though… :slight_smile:

THE PROBLEM

First off, the issue I had is as follows:

  • The LED on the front of the WD My Cloud was solid red
  • On the WD My Cloud Dashboard (accessed by opening your browser and going to http://wdmycloud/UI/ - NOTE: For me Chrome didn’t work, but Firefox did), under the “Diagnosis” box it said “Critical”. Clicking on the arrow to view the status, it said that the “Content Status” was “Not Mounted”

If the above describes your problem, then read on, otherwise, sorry, you’ll need to keep searching the forums.

THE CAUSE AND RESOLUTION

The cause of the problem was that the superblock of my primary partition, /dev/sda4 (i.e. the place where the WD My Cloud keeps all my files) was corrupted. There are backups for this superblock, so it’s just really a matter of restoring the corrupted superblock from one of the backups, rebooting and then everything should be working fine again. Or in my case, it was a matter of running the file system check utilty to find and fix any errors, then rebooting.

I’m not sure how this happened in the first place, best guess is that the power outage I had the previous night possibly had something to do with it… I guess I’ll never know for sure, but the main thing is to get access back to my files.

THE FIX - ATTEMPT 1

First I tried going through the steps outlined by the user Blueoriontiger in this post - https://community.wd.com/t/data-volume-failed-to-mount/91074/7. The same user put a tutorial together here: http://janeilh.com/2015/03/wd-my-cloud-data-recovery/. This user goes on to say that after running the fix, that you’ll need to restore the drive to factory settings (after you copy your data off) - I can’t verify this, as I never got that far, so do that at your own discretion.

In a nutshell, you’ll need to do the following:

  1. Access your WD My Cloud Dashboard via the browser, i.e. go to http://wdmycloud/UI/ and login

  2. Enable SSH by going to “Settings” → “Network” → Click the switch next to “SSH”

  3. Download the terminal emulator Putty from here: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

  4. Get the IP address of the WD My Cloud. I have a Windows 10 machine, so all I had to do was open file explorer (WINDOWS Key + E), click on “Network” on the left, then on the right, right-click on WDMyCloud under “Storage”, click properties and the IP address is under the section “Troubleshooting Information”

  5. Open Putty, then for ‘Host Name’, enter in the IP address of the WD My Cloud, and click the “Open” button. The default login details are:

    • username: root
    • password: welc0me

WARNING: You’re now logged into the WD My Cloud operating system as the super user “root”. You can do some serious damage to the operating system by running the wrong commands, so just be careful.

  1. Find your main partition with the command: parted -l. For me, I found it by looking for the line with the biggest size. From the below output, I knew that:

  • the partition is on the disk /dev/sda

  • partition number 4 is the biggest, so the partition with my files on it is going to be /dev/sda4

    WDMyCloud:/media/myusb# parted -l
    Model: ATA WDC WD30EZRX-00D (scsi)
    Disk /dev/sda: 3001GB <— This is the disk device
    Sector size (logical/physical): 512B/4096B
    Partition Table: gpt

    Number Start End Size File system Name Flags
    3 15.7MB 528MB 513MB linux-swap(v1) primary
    1 528MB 2576MB 2048MB ext3 primary raid
    2 2576MB 4624MB 2048MB ext3 primary raid
    5 4624MB 4724MB 99.6MB primary
    6 4724MB 4824MB 101MB primary
    7 4824MB 4826MB 1049kB primary
    8 4826MB 4828MB 2097kB primary
    4 4828MB 3001GB 2996GB ext4 primary <— This is the partition

  1. Next, find out which blocks are storing the superblock backups by running mke2fs -n [partition], i.e. mke2fs -n /dev/sda4. The blocks will be listed right at the bottom

    WDMyCloud:~# mke2fs -n /dev/sda4
    mke2fs 1.42.5 (29-Jul-2012)
    Filesystem label=
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    Stride=0 blocks, Stripe width=0 blocks
    182853632 inodes, 731387904 blocks
    36569395 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=0
    22321 block groups
    32768 blocks per group, 32768 fragments per group
    8192 inodes per group
    Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
    102400000, 214990848, 512000000, 550731776, 644972544 <---- they are kept on these blocks

  2. To restore the superblock from backup, run the e2fsck command, providing the block where the backup is kept. So trying the backup located at block 32768, the command will be e2fsck -b 32768 /dev/sda4. Note this will take a long time and seem like it’s actually doing nothing. You just need to be patient - Pass 1 will take a while, Pass 2, 3 and 4 won’t take as long, but Pass 5 will take forever again… it did for me at least. If you want to see the progress, you can run the command with -C0, i.e. e2fsck -C0 -b 32768 /dev/sda4, this will give you a progress bar. I only figured that out later on, so I didn’t include it in my screencap:

    WDMyCloud:~# e2fsck -b 32768 /dev/sda4
    e2fsck 1.42.5 (29-Jul-2012)
    /dev/sda4 contains a file system with errors, check forced.
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information

After Pass 5, one of two things will happen, either you’ll see the word “Killed” or you’ll see a summary similar to this:

/dev/sda4: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda4: 635740/182853632 files (0.2% non-contiguous), 329909675/731387904 blocks

If you see the summary, good for you (I didn’t!), in that case, all you need to do is run the command: reboot. This will reboot the WD My Cloud, and you’re putty session will disconnect. Wait for it to reboot and let it do it’s thing - it’ll be rescanning and running all sorts of other things. Eventually the LED on the WD My Cloud should turn from white to blue and you should be able to access your WD My Cloud as per normal. Please also refer to the “Optional Step 7” in the link to the tutorial above to see if you’ll need to do that. I can’t comment on this, because my process was killed off at this point, so I didn’t do the reboot etc here.

If you see the word “Killed”, this means that the e2fsck process was killed off by the ‘Out Of Memory Killer’, which is an automated process that stops any processes from draining the operating system from all of it’s memory. You can verify this by looking at the system logs with the command: dmesg | tail. Apparently there is a bug with the version of e2fsck causing it to consume too much memory, or maybe I have too many files for it to process, I don’t know for sure, either way I needed to supplement the memory. Since I can’t simply just add more RAM to the WD My Cloud, or pull out the disk and shove it into another Linux machine with more resources, the next best thing is to increase the swap space of the system (read here if you’re interested in what swap space is: https://www.linux.com/news/all-about-linux-swap-space), so this process can complete. If this is you too, then move on to “THE FIX - ATTEMPT 2”.

THE FIX - ATTEMPT 2

For this fix attempt you’ll need at least a 8GB USB drive - we’ll be formatting it, so make sure any files you need are removed beforehand.

At a high level, this attempt to fix the corrupted superblock will be as follows:

  • Format the USB drive
  • Configure e2fsck to use the USB drive for it’s temporary files
  • Create a 6GB swap file on the USB for the operating system to use to supplement its memory
  • Run the file system check utility (fsck) which will find and fix any errors
  • Remove the 6GB swap file
  • Reboot

  1. Ensure that you still have a Putty session open to the WD My Cloud, if not go back up to “THE FIX - ATTEMPT 1” to get the instructions

  2. Plug the USB stick into the back of the WD My Cloud and find it using the command: fdisk -l. Since it’ll be the second disk, it’ll most likely be /dev/sdb and since there is only likely one partition on the USB, it’ll most likely be the device /dev/sdb1. But please take a look at YOUR output and find yours - ensure that it is the USB before you proceed to step 3!

My output was:

WDMyCloud:~# fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

Disk /dev/sdb: 8178 MB, 8178892800 bytes		   	      <--- this is my USB disk, note the size
2 heads, 63 sectors/track, 126780 cylinders, total 15974400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x011909fb

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *          64    15974399     7987168    b  W95 FAT32     <--- this is my USB device, note the system

Disk /dev/md1: 2047 MB, 2047803392 bytes
2 heads, 4 sectors/track, 499952 cylinders, total 3999616 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table
WDMyCloud:~#
  1. Format the USB drive using Linux ext4 file system.

EXTRA SUPER HIGH INTENSE WARNING:
Make sure you’re referencing the USB device before you run this command, I cannot stress this HIGH enough
The last thing you want to do is format the wrong file system! Because then all bets are over… and you might as well throw your WD My Cloud in the bin and buy a new one. I don’t want to see comments saying “Alex, I accidentally formatted my main/the wrong drive, what do I do now??”, because I’ll be like, “Dunno man … Google is your friend now, I warned you, I’m out!”.

In fact, if you don’t feel comfortable running this command, then please don’t. Just make sure you’ve removed everything off the USB so there is the maximum amount of free space… the next steps should, potentially, still work with the Windows FAT32 file system on the USB drive (I just can’t verify this, because I formatted my USB drive as part of one of the things I did when trying to fix my Cloud drive).

The command to format the USB drive is: mkfs.ext4 [USB Device found in step 2], e.g. mkfs.ext4 /dev/sdb1. This is my output

WDMyCloud:~# mkfs.ext4 /dev/sdb1
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
499712 inodes, 1996784 blocks
99839 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2046820352
61 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
  1. Create a directory called /media/myusb and mount the USB stick on that directory:

  • Create Create the directory

    WDMyCloud:~# cd /media/ WDMyCloud:/media# ls -l total 0 WDMyCloud:/media# mkdir myusb

  • Mount the USB drive with: mount [USB device] [Target folder], e.g. mount /dev/sdb1 /media/myusb

    WDMyCloud:/media# mount /dev/sdb1 /media/myusb WDMyCloud:/media# cd /media/myusb WDMyCloud:/media/myusb# df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb1 7.5G 146M 7.0G 2% /media/myusb

USB is now mounted on /media/myusb … onwards.

  1. Create the directory /media/myusb/scratch on the USB, then create /etc/e2fsck.conf and point scratch_files directive to /media/myusb/scratch - this will tell e2fsck to use that directory for any temporary files it needs to create, which should help with memory issues.

  • Create the directory /media/myusb/scratch

    WDMyCloud:/media/myusb# mkdir scratch WDMyCloud:/media/myusb# ls -l total 20 drwx------ 2 root root 16384 Sep 20 20:45 lost+found drwxrwxrwx 2 root root 4096 Sep 20 20:50 scratch

  • use the command vi to create the file /etc/e2fsck.conf and add this:

    [scratch_files] directory = /media/myusb/scratch

So, run the command: vi /etc/e2fsck.conf. Then press “i”, then start typing the above lines exactly as is. When done, type colon, wq, then enter, i.e. :wq[ENTER]. This will write the contents to the file and quit the “vi” editor

WDMyCloud:/media/myusb# vi /etc/e2fsck.conf
WDMyCloud:/media/myusb# cat /etc/e2fsck.conf
[scratch_files]
directory = /media/myusb/scratch
  1. Create an additional 6GB of swap space on the USB drive; call the swap file swap2. Why 6GB? Because I tried with 2GB and 4GB and it still ran out of memory… The swap file is created in several steps:

a) Create a 6GB (6 * 1024 * 1024^2 / 4096 = 1572864) file on the USB to be dedicated to the swap space. Give this command a long while to run:

dd if=/dev/zero of=/media/myusb/swap2 bs=4096 count=1572864

WDMyCloud:/media/myusb# dd if=/dev/zero of=/media/myusb/swap2 bs=4096 count=1572864
1572864+0 records in
1572864+0 records out
6442450944 bytes (6.4 GB) copied, 3077.3 s, 2.1 MB/s

b) Set the swap area with mkswap: mkswap /media/myusb/swap2

WDMyCloud:/media/myusb# mkswap /media/myusb/swap2
Setting up swapspace version 1, size = 6291392 KiB
no label, UUID=eb214df5-abd2-49ac-b00b-b6e77579e2e4

c) Enable the swap space: swapon -p 1000 /media/myusb/swap2

WDMyCloud:/media/myusb# swapon -p 1000 /media/myusb/swap2
WDMyCloud:/media/myusb#

The new additional 6GB swap file is created and switched on… now to finally scan and fix the partition

  1. Run the command: fsck /dev/sda4 to scan the partition and answer yes when prompted to fix any issues that are found. Below is my output when I ran it on my WD My Cloud - I think you could have run this with the -y flag to autofix, but I didn’t know that at the time, hence I manually answered “y” for yes, when prompted:

    WDMyCloud:/media/myusb# fsck /dev/sda4
    fsck from util-linux 2.20.1
    e2fsck 1.42.5 (29-Jul-2012)
    One or more block group descriptor checksums are invalid. Fix? yes
    Group descriptor 15872 checksum is 0x29c7, should be 0x86b6. FIXED.
    /dev/sda4 contains a file system with errors, check forced.
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    Free blocks count wrong (401732028, counted=401478229).
    Fix? yes
    Free inodes count wrong (182238332, counted=182217892).
    Fix? yes

    /dev/sda4: ***** FILE SYSTEM WAS MODIFIED *****
    /dev/sda4: 635740/182853632 files (0.2% non-contiguous), 329909675/731387904 blocks

At this stage, with any luck it should have found issues, and as mentioned you’ll need to manually answer “yes” to fix them. Once the fsck command has finished running, double-check the exit code, by running the command: echo $?. You should get the number 1, as per this output:

WDMyCloud:/media/myusb# echo $?
1
WDMyCloud:/media/myusb#

You can check the manual for fsck for the meaning of the exit codes here: https://linux.die.net/man/8/fsck. Effectively, an exit code of “1” means that “File system errors corrected”… Sweet, that’s what we want to hear! Ok, now to clean up …

  1. Turn off the additional swap space, remove the swap file and then unmount the USB drive:
  • Still under the directory: /media/myusb, run
    `# swapoff swap2

    rm swap2`

s

WDMyCloud:/media/myusb# swapoff swap2
WDMyCloud:/media/myusb# rm swap2
rm: remove regular file `swap2'? y

  • Go to the home directory (with the command: cd), and unmount the USB drive with the command: umount /media/myusb

    WDMyCloud:/media/myusb# cd WDMyCloud:~# umount /media/myusb

  1. Lastly, remove the /etc/e2fsck.conf file, take the USB out of the My Cloud and then reboot the WD My Cloud with the command: reboot

    WDMyCloud:# rm /etc/e2fsck.conf
    rm: remove regular file `/etc/e2fsck.conf’? y

    WDMyCloud:~# reboot

    Broadcast message from root@WDMyCloud (pts/1) (Wed Sep 21 03:35:46 2016):

    The system is going down for reboot NOW!

At this point, give the WD My Cloud a little while to sort itself out. The LED should eventually turn from white to blue. It’ll be a little slow to begin with, but I think it’s running through a scan in the background. But you should now be in a position to access it and copy off all your important files. After you have salvaged your important files, you can, if you want to, do a factory restore, but I’ll leave that up to you.

Hope this guide works for people that have the same issue.

Thanks,
Alex

4 Likes