Disks OK but deteriorated RAID and data lost

Cerberus · November 12, 2023, 4:01pm

Didn’t expect to see /dev/md2 so a bit more info is needed. Run the following commands, one at a time, then post the results.

`mdadm --detail /dev/md0;
`mdadm --detail /dev/md2;

I also need to see the output of the third command, requested previously, but you may be in the process of posting it, so ignore this part if you are.

dmesg -t -l warn,emerg,alert,crit,err;

laucha131 · November 12, 2023, 4:09pm

Still me, laucha13, as new member I have a lot of limitation, no more than seven posts per day was the last shown … here is the last of the 3 commands, I will do next 2

usyBox v1.30.1 (2023-01-16 16:39:18 UTC) built-in shell (ash)
Enter ‘help’ for a list of built-in commands.

root@WDMyCloudEX4100 ~ # cat /proc/mdstat;
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md2 : active raid1 sdd2[1]
3902822264 blocks super 1.0 [2/1] [_U]
bitmap: 1/1 pages [32KB], 262144KB chunk

md1 : active raid1 sdb2[0] sda2[1]
9762240320 blocks super 1.0 [2/2] [UU]
bitmap: 0/10 pages [0KB], 65536KB chunk

md0 : active raid1 sda1[3] sdb1[2] sdd1[1] sdc1[0]
2094080 blocks super 1.2 [4/4] [UUUU]
bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices:
root@WDMyCloudEX4100 ~ # mdadm --detail /dev/md1;
/dev/md1:
Version : 1.0
Creation Time : Wed Nov 8 18:41:32 2023
Raid Level : raid1
Array Size : 9762240320 (9310.00 GiB 9996.53 GB)
Used Dev Size : 9762240320 (9310.00 GiB 9996.53 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

 Intent Bitmap : Internal

   Update Time : Sun Nov 12 10:55:54 2023
         State : clean
Active Devices : 2

Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Consistency Policy : bitmap

          Name : WDMyCloudEX4100:1  (local to host WDMyCloudEX4100)
          UUID : 264015ac:0e40e9ec:0684827a:bc239a93
        Events : 2

Number   Major   Minor   RaidDevice State
   0       8       18        0      active sync   /dev/sdb2
   1       8        2        1      active sync   /dev/sda2

root@WDMyCloudEX4100 ~ # ^C
root@WDMyCloudEX4100 ~ # dmesg -t -l warn,emerg,alert,crit,err;
mvebu-pmsu: CPU hotplug support is currently broken on Armada 38x: disabling
mvebu-pmsu: CPU idle is currently broken on Armada 38x: disabling
ahci-mvebu f10a8000.sata: masking port_map 0x3 → 0x3
ahci-mvebu f10e0000.sata: masking port_map 0x3 → 0x3
marvell-nfc f10d0000.flash: Timeout on CMDD (NDSR: 0x00000080)
marvell-nfc f10d0000.flash: Timeout on CMDD (NDSR: 0x00000280)
(NULL device *): hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info().
EXT4-fs (ram0): couldn’t mount as ext3 due to feature incompatibilities
jnl: loading out-of-tree module taints kernel.
ufsd: module license ‘Commercial product’ taints kernel.
Disabling lock debugging due to kernel taint
restart sysinfod…
md: kicking non-fresh sdc2 from array!
root@WDMyCloudEX4100 ~ #

Cerberus · November 12, 2023, 4:19pm

Indeed, those limitations sometimes make it difficult to help new users. I’m working on an app to automate much of this stuff, but it’s not quite done yet.

Just as I expected. The /dev/sdc drive (the one with issues) was kicked from the /dev/md2 RAID 1 array. Ignore all previous command requests, and run the following command, then post the results.

mdadm --detail /dev/md2;

Afterwards, I will post additional instructions for running commands which may take a long time to finish.

laucha131 · November 12, 2023, 4:45pm

I’m having trouble writing the last command and I can’t use ctrl - v to paste your commands in putty.

First character is Alt 96 in ASCII code right ?
What is the character after mdadm ?
I’ve tried “–” but does not work, cursor go to next line and stay there …

root@WDMyCloudEX4100 ~ # `mdadm --detail /dev/md2;

root@WDMyCloudEX4100 ~ # `mdadm --detail /dev/md2;

I have a French keyboard what doesn’t help … sorry

Cerberus · November 12, 2023, 4:56pm

It’s easy. Just right-click to paste in Putty. Also, the command had a typo, the leading tick mark should not be there. Here is the correct command.

mdadm --detail /dev/md2;

laucha131 · November 12, 2023, 5:00pm

root@WDMyCloudEX4100 ~ # mdadm --detail /dev/md2;
/dev/md2:
Version : 1.0
Creation Time : Thu Apr 9 12:24:08 2015
Raid Level : raid1
Array Size : 3902822264 (3722.02 GiB 3996.49 GB)
Used Dev Size : 3902822264 (3722.02 GiB 3996.49 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent

 Intent Bitmap : Internal

   Update Time : Sun Nov 12 10:07:01 2023
         State : clean, degraded
Active Devices : 1

Working Devices : 1
Failed Devices : 0
Spare Devices : 0

Consistency Policy : bitmap

          Name : 1
          UUID : 38833c04:ff5275a5:98bf3f32:2d62e8e6
        Events : 21241

Number   Major   Minor   RaidDevice State
   -       0        0        0      removed
   1       8       50        1      active sync   /dev/sdd2

root@WDMyCloudEX4100 ~ #

Cerberus · November 12, 2023, 5:07pm

Ok, this corresponds with /dev/sdc2 getting kicked out of the RAID 1 array, so /dev/md2 is the correct device. I had to be sure, because the next commands are potentially dangerous if they’re ran on the wrong device.

Execute the following commands to kill running processes and unmount the /dev/md2 device.

/usr/sbin/kill_process.sh /dev/md2;
umount /dev/md2;

Next, as quickly as possible after running the first two commands, run the following command to start the e2fsck filesystem repair process. It will probably take a long time (hours to days) to run, and it may generate error messages, etc.

Type the command EXACTLY as shown below, and DO NOT close the Putty window until it’s finished.

e2fsck -p -f -C 0 /dev/md2;

When it is finished, copy the output and save it to a text file for later reference, if needed. Finally, reboot and hope for the best.

Let me know how it goes.

laucha131 · November 12, 2023, 5:19pm

root@WDMyCloudEX4100 ~ # /usr/sbin/kill_process.sh /dev/md2;
kill_process.sh /dev/md2: 1 process(es) need to be stopped
kill_process.sh /dev/md2: stop smbd(pid=8371)
USER PID ACCESS COMMAND
/dev/md2: root kernel mount /mnt/HD/HD_b2
root@WDMyCloudEX4100 ~ # umount /dev/md2;
umount: /mnt/HD/HD_b2: target is busy.
root@WDMyCloudEX4100 ~ #

Is it normal to have the busy message ?

Cerberus · November 12, 2023, 5:21pm

Yes, if the umount command is ran too quickly. Give it a minute and try again. DO NOT run the final e2fsck command until the umount command is successful.

laucha131 · November 12, 2023, 5:34pm

root@WDMyCloudEX4100 ~ # umount /dev/md2;
umount: /mnt/HD/HD_b2: target is busy.
root@WDMyCloudEX4100 ~ # umount /dev/md2;
umount: /mnt/HD/HD_b2: target is busy.
root@WDMyCloudEX4100 ~ # /usr/sbin/kill_process.sh /dev/md2;

How do I know if unmount command is succesfull ? Only if cursor goes to next line as in the text below?
I imagine I need to execute both commands one more time since I didn’t do the last one immediately ?
Maybe they are stupid questions but …

kill_process.sh /dev/md2: 1 process(es) need to be stopped
kill_process.sh /dev/md2: stop smbd(pid=27472)
USER PID ACCESS COMMAND
/dev/md2: root kernel mount /mnt/HD/HD_b2
root@WDMyCloudEX4100 ~ # umount /dev/md2;
root@WDMyCloudEX4100 ~ #

Cerberus · November 12, 2023, 5:41pm

Yes, it’s best to run them all again, just to be sure.

I should have mentioned it, but if the umount command has no output and skips back to a blank command line afterwards, you should be good to go.

For example:

# mkdir /tmp/test
# mount /tmp/test /dev/mmcblk0p5
# mount /dev/mmcblk0p5 /tmp/test
# umount /dev/mmcblk0p5
#

laucha131 · November 12, 2023, 6:18pm

As the putty interface was no moving and disk was quiet I thought process was finished,
and I wrote “ctrl -c” what I think stopped the process, I know now how to copy from putty but it’s maybe a little late.

Send you the last information shown in putty.
Can I restart the process ?
Shared folders are not available anymore from Win file explorer.

/dev/md2: Inode 130024809 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024837 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024838 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024876 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024878 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024880 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130025551 extent tree (at level 2) could be narrower. IGNORED.
Inode 134086662 extent tree (at level 1) could be narrower. IGNORED.
Inode 151519235 extent tree (at level 1) could be narrower. IGNORED.
Inode 153616389 extent tree (at level 2) could be narrower. IGNORED.
/dev/md2: Inode 154140674 extent tree (at level 2) could be narrower. IGNORED.
Inode 154533890 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 155189254 extent tree (at level 2) could be narrower. IGNORED.
Inode 157420229 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: 500526/243933184 files (0.9% non-contiguous), 951836752/975701248 blocks
root@WDMyCloudEX4100 ~ # ^C
root@WDMyCloudEX4100 ~ # ev/md2: Inode 87306112 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 87306632 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 87818413 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 87949325 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 87949326 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 87949926 extent tree (at level 2) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 116785200 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 116785235 extent tree (at level 2) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 118358522 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 118358555 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 122159648 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 122159676 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 122159677 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 122159715 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 122159717 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 122159719 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 129238215 extent tree (at level 2) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 129238216 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 130023470 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 130024247 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 130024809 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 130024837 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 130024838 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 130024876 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 130024878 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 130024880 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 130025551 extent tree (at level 2) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 134086662 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 151519235 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 153616389 extent tree (at level 2) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 154140674 extent tree (at level 2) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 154533890 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: Inode 155189254 extent tree (at level 2) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # Inode 157420229 extent tree (at level 1) could be narrower. IGNORED.
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # /dev/md2: 500526/243933184 files (0.9% non-contiguous), 951836752/975701248 blocks
-sh: syntax error: unexpected “(”
root@WDMyCloudEX4100 ~ # root@WDMyCloudEX4100 ~ # ^C
-sh: root@WDMyCloudEX4100: not found
root@WDMyCloudEX4100 ~ # root@WDMyCloudEX4100 ~ #
-sh: root@WDMyCloudEX4100: not found

Cerberus · November 12, 2023, 6:27pm

Yes, but the e2fsck process should NEVER be terminated until it’s finished because it can have unpredictable consequences. As I said, it will take a long time, from hours to days.

Simply repeat the steps outlined previously. You may need to reboot first.

laucha131 · November 12, 2023, 6:35pm

I restarted it … we will see.
Mi intention was not to stop it, just to copy the messages I had on putty but I don’t know at all putty and I did’t think it will stop all.
How do I know the process is finished ?

Anyways … thanks a lot for your help!

Cerberus · November 12, 2023, 6:42pm

It’s been a while, but it should drop back to the command line, similar to the umount command.

laucha131 · November 12, 2023, 7:48pm

As previously, the system is in stand by mode, disks are not turning, CPU doesn’t seem to do anything, no noise, and cursor came to the next line on putty …

Inode 153616389 extent tree (at level 2) could be narrower. IGNORED.
/dev/md2: Inode 154140674 extent tree (at level 2) could be narrower. IGNORED.
Inode 154533890 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 155189254 extent tree (at level 2) could be narrower. IGNORED.
Inode 157420229 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: 500526/243933184 files (0.9% non-contiguous), 951836752/975701248 blocks
root@WDMyCloudEX4100 ~ #

Cerberus · November 12, 2023, 7:52pm

The e2fsck command appears to be finished. Now, try rebooting and see what happens. Files still missing, etc…?

laucha131 · November 12, 2023, 8:21pm

After reboot, no changes … the files are still missing looking from my file explorer.
Each time I execute the command it takes 2 minutes to finish, you said it should take hours or days, not sure it’s working.

v/md2: Inode 67373298 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373299 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373300 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373301 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373302 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373305 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373318 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373330 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373331 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373332 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373364 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373365 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373366 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373367 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373368 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 67373371 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 70385942 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 70909954 extent tree (at level 1) could be narrower. IGNORED.
Inode 71565414 extent tree (at level 1) could be narrower. IGNORED.
Inode 82444291 extent tree (at level 1) could be narrower. IGNORED.
Inode 84410371 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 86245399 extent tree (at level 1) could be narrower. IGNORED.
Inode 87306112 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 87306632 extent tree (at level 1) could be narrower. IGNORED.
Inode 87818413 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 87949325 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 87949326 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 87949926 extent tree (at level 2) could be narrower. IGNORED.
Inode 116785200 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 116785235 extent tree (at level 2) could be narrower. IGNORED.
Inode 118358522 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 118358555 extent tree (at level 1) could be narrower. IGNORED.
Inode 122159648 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 122159676 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 122159677 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 122159715 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 122159717 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 122159719 extent tree (at level 1) could be narrower. IGNORED.
Inode 129238215 extent tree (at level 2) could be narrower. IGNORED.
/dev/md2: Inode 129238216 extent tree (at level 1) could be narrower. IGNORED.
Inode 130023470 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024247 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024809 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024837 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024838 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024876 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024878 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130024880 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 130025551 extent tree (at level 2) could be narrower. IGNORED.
Inode 134086662 extent tree (at level 1) could be narrower. IGNORED.
Inode 151519235 extent tree (at level 1) could be narrower. IGNORED.
Inode 153616389 extent tree (at level 2) could be narrower. IGNORED.
Inode 154140674 extent tree (at level 2) could be narrower. IGNORED.
/dev/md2: Inode 154533890 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: Inode 155189254 extent tree (at level 2) could be narrower. IGNORED.
Inode 157420229 extent tree (at level 1) could be narrower. IGNORED.
/dev/md2: 500526/243933184 files (0.9% non-contiguous), 951836752/975701248 blocks
root@WDMyCloudEX4100 ~ #

Cerberus · November 12, 2023, 8:42pm

No, I said it MAY take hours or days, it all depends on how many problems the e2fsck program encounters, etc. It’s the same command that’s ran by the “Scan Disk” option via the dashboard, except this way the results can be seen.

Scan Disk File System Check and Repair

Manufacturers sell RAID as if it were some sort of magic replacement for proper backups, but it’s quite the opposite. In fact, RAID1 is even more dangerous than RAID0 because it creates a false sense of security, while files are often silently corrupted or lost in the background. As in this case, the primary drive developed problems, files became corrupted or were lost, and those changes were immediately “mirrored” to the secondary drive.

Unfortunately, I’m not sure that much more can be done at this point.

laucha131 · November 12, 2023, 8:50pm

I agree, I thought I was doing the right thing but I understand better now.

Thank you for your help !