After firmware 5.27.157 update, RAID configuration is gone and consequently no access to data

birthi · November 11, 2023, 7:06pm

After update completed, I was notified via email “No access to volume 1”.
Drive tests (full) and system test passed without errors. I have no physical access to the device (it’s 6000 mi away), but SSH works.
I pulled some information:
root@HB-Cloud2 ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid1 sdb1[1] sda1[0]
2094080 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices:

root@HB-Cloud2 ~ # mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Nov 9 21:37:57 2023
Raid Level : raid1
Array Size : 2094080 (2045.00 MiB 2144.34 MB)
Used Dev Size : 2094080 (2045.00 MiB 2144.34 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

 Intent Bitmap : Internal

   Update Time : Thu Nov  9 21:38:00 2023
         State : clean
Active Devices : 2

Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Consistency Policy : bitmap

          Name : HB-Cloud2:0  (local to host HB-Cloud2)
          UUID : c6bc57b1:0b090abb:00cd92b3:d600a8d1
        Events : 2

Number   Major   Minor   RaidDevice State
   0       8        1        0      active sync   /dev/sda1
   1       8       17        1      active sync   /dev/sdb1

Any advice what to look for would be highly appreciated! I’m not a Linux expert, so right now I’m poking around in the dark.

Cerberus · November 11, 2023, 7:48pm

The first thing you should do is check the drives for errors. Run the following commands, one at a time, then post the results.

smartctl -s on -a /dev/sda;
smartctl -s on -a /dev/sdb;

birthi · November 11, 2023, 10:29pm

root@HB-Cloud2 ~ # smartctl -s on -a /dev/sda
smartctl 7.2 2020-12-30 r5155 [armv7l-linux-4.14.22-armada-18.09.3] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (SMR)
Device Model: WDC WD40EFAX-68JH4N1
Serial Number: WD-WXC2D90A78ZK
LU WWN Device Id: 5 0014ee 2be207503
Firmware Version: 83.00A83
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Nov 11 23:29:12 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 7740) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 485) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3039) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 209 206 021 Pre-fail Always - 2525
4 Start_Stop_Count 0x0032 091 091 000 Old_age Always - 9359
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 071 071 000 Old_age Always - 21422
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 26
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 18
193 Load_Cycle_Count 0x0032 197 197 000 Old_age Always - 9341
194 Temperature_Celsius 0x0022 116 094 000 Old_age Always - 31
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 21372 -

2 Extended offline Completed without error 00% 21329 -

3 Short offline Completed without error 00% 21320 -

4 Extended offline Completed without error 00% 21081 -

5 Short offline Completed without error 00% 0 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@HB-Cloud2 ~ # smartctl -s on -a /dev/sdb
smartctl 7.2 2020-12-30 r5155 [armv7l-linux-4.14.22-armada-18.09.3] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red (SMR)
Device Model: WDC WD40EFAX-68JH4N1
Serial Number: WD-WXC2D90KF2ZV
LU WWN Device Id: 5 0014ee 268caa869
Firmware Version: 83.00A83
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Nov 11 23:29:25 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (21524) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 506) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3039) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 208 202 021 Pre-fail Always - 2575
4 Start_Stop_Count 0x0032 091 091 000 Old_age Always - 9498
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 071 071 000 Old_age Always - 21424
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 25
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 18
193 Load_Cycle_Count 0x0032 197 197 000 Old_age Always - 9480
194 Temperature_Celsius 0x0022 116 094 000 Old_age Always - 31
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 21374 -

2 Extended offline Completed without error 00% 21332 -

3 Short offline Completed without error 00% 21322 -

4 Short offline Completed without error 00% 21100 -

5 Short offline Completed without error 00% 21100 -

6 Extended offline Completed without error 00% 21084 -

7 Short offline Completed without error 00% 0 -

8 Short offline Aborted by host 20% 0 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Cerberus · November 11, 2023, 10:42pm

Neither hard drive has any errors, but both of them are WD Red WD40EFAX models that use SMR technology, which is the likely source of your problem.

The performance of SMR hard drives is abysmal, which makes them totally unsuitable for use in a RAID environment.

Post the results of the following command.

dmesg -t -l warn,emerg,alert,crit,err;

NAS_user · November 11, 2023, 10:44pm

I have my suspicions:

But - - - ok; you can access the drives via SSH. Good.
Can you access via VPN?
Or the OS/5 dashboard?
Or the files via the OS/5 app (in other words. . . .is the device working, and you are investigating an error message; or are you not able to access the system other than via SSH?)

birthi · November 11, 2023, 10:52pm

root@HB-Cloud2 ~ # dmesg -t -l warn,emerg,alert,crit,err;
mvebu-pmsu: CPU hotplug support is currently broken on Armada 38x: disabling
mvebu-pmsu: CPU idle is currently broken on Armada 38x: disabling
marvell-nfc f10d0000.flash: Timeout on CMDD (NDSR: 0x00000080)
marvell-nfc f10d0000.flash: Timeout on CMDD (NDSR: 0x00000280)
(NULL device *): hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info().
ahci-mvebu f10a8000.sata: masking port_map 0x3 → 0x3
EXT4-fs (ram0): couldn’t mount as ext3 due to feature incompatibilities
jnl: loading out-of-tree module taints kernel.
ufsd: module license ‘Commercial product’ taints kernel.
Disabling lock debugging due to kernel taint
restart sysinfod…
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
EXT4-fs (md1): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Inode table for group 64 overlaps superblock
EXT4-fs (md1): ext4_check_descriptors: Checksum for group 64 failed (15047!=0)
EXT4-fs (md1): group descriptors corrupted!
root@HB-Cloud2 ~ #

I have VPN access, ssh access, I see the dashboard and can execute all tabs, but capacity is 0k and file access is aborting

Cerberus · November 11, 2023, 10:55pm

Next, run the following commands, one at a time. The second command may take a while to run, and may appear to fail. Before doing anything else, post the results of both commands and we’ll take it from there.

mknod /dev/md1 b 9 1;
mdadm --assemble --run /dev/md1 /dev/sda2 /dev/sdb2;

birthi · November 11, 2023, 10:57pm

root@HB-Cloud2 ~ # mknod /dev/md1 b 9 1
root@HB-Cloud2 ~ # mdadm --assemble --run /dev/md1 /dev/sda2 /dev/sdb2
mdadm: Fail create md1 when using /sys/module/md_mod/parameters/new_array
mdadm: /dev/md1 has been started with 2 drives.
root@HB-Cloud2 ~ #

Cerberus · November 11, 2023, 11:01pm

Ok, it was able to start the RAID 1 array, which is a very good sign. Now, reboot, then log into the dashboard and report the results.

birthi · November 11, 2023, 11:18pm

Rebooted, still showing no RAID volume, 0 kB free

Cerberus · November 11, 2023, 11:19pm

I suspected it might do that, but wanted to confirm before taking the next step. It’s a process. Run the following commands, one at a time, then post the results.

cat /proc/mdstat;
mdadm --detail /dev/md1;

birthi · November 11, 2023, 11:28pm

root@HB-Cloud2 ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid1 sdb1[1] sda1[0]
2094080 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices:
root@HB-Cloud2 ~ # madm --detail /dev/md1
-sh: madm: not found
root@HB-Cloud2 ~ # mdadm --detail /dev/md1
mdadm: cannot open /dev/md1: No such file or directory
root@HB-Cloud2 ~ #

Cerberus · November 11, 2023, 11:30pm

Ok, manually creating the /dev/md1 device didn’t survive a reboot. Run the following commands, one at a time, then post the results. Don’t reboot.

mknod /dev/md1 b 9 1;
mdadm --assemble --run /dev/md1 /dev/sda2 /dev/sdb2;
cat /proc/mdstat;
mdadm --detail /dev/md1;

birthi · November 11, 2023, 11:34pm

root@HB-Cloud2 ~ # mknod /dev/md1 b 9 1
root@HB-Cloud2 ~ # mdadm --assemble --run /dev/md1 /dev/sda2 /dev/sdb2
mdadm: Fail create md1 when using /sys/module/md_mod/parameters/new_array
mdadm: /dev/md1 has been started with 2 drives.
root@HB-Cloud2 ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid1 sda2[0] sdb2[2]
3902822264 blocks super 1.0 [2/2] [UU]
bitmap: 0/1 pages [0KB], 262144KB chunk

md0 : active raid1 sdb1[1] sda1[0]
2094080 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices:
root@HB-Cloud2 ~ # mdadm --detail /dev/md1
/dev/md1:
Version : 1.0
Creation Time : Fri Feb 26 14:18:36 2021
Raid Level : raid1
Array Size : 3902822264 (3722.02 GiB 3996.49 GB)
Used Dev Size : 3902822264 (3722.02 GiB 3996.49 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

 Intent Bitmap : Internal

   Update Time : Sun Nov 12 00:08:00 2023
         State : clean
Active Devices : 2

Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Consistency Policy : bitmap

          Name : 1
          UUID : d7807461:9d6f590d:c98d1f5c:3a3bef51
        Events : 33146

Number   Major   Minor   RaidDevice State
   0       8        2        0      active sync   /dev/sda2
   2       8       18        1      active sync   /dev/sdb2

Cerberus · November 11, 2023, 11:40pm

Ok, the /dev/md1 device is operational again. Next, log into the dashboard and run the following tests, then report the results. They’re located in the Settings / Utilities / System Diagnostics section.

Quick Disk Test
System Test

This should force OS5 to recreate any missing configuration files, and allow the /dev/md1 device to survive reboots. If not, we can try something else.

birthi · November 11, 2023, 11:47pm

OK, thanks. Will take a minute!

birthi · November 11, 2023, 11:51pm

All tests passed. Will reboot now…

birthi · November 11, 2023, 11:57pm

Unfortunately, same result. I remembered when I had an unfounded disk error few weeks ago, it persisted until I did a full disk test, which eventually cleared it. Would it it make sense to try that?

Cerberus · November 12, 2023, 12:08am

It might come to that, but we’re not quite there yet. This bug is a bit of a mystery, so it will take a bit of trial and error to find the cause and solution. The good news is that your data appears to be fine, it’s just not accessible via traditional means just yet.

First, recreate the /dev/md1 device again and start the RAID 1 array by running the following commands.

mknod /dev/md1 b 9 1;
mdadm --assemble --run /dev/md1 /dev/sda2 /dev/sdb2;

Next, run the following commands, one at a time, then post the results. You’ll need to post the results as </> preformatted text this time, to prevent the forum from munging the XML.

cat /mnt/HD_a4/.systemfile/hd_volume_info.xml
cat /mnt/HD_b4/.systemfile/hd_volume_info.xml

Afterwards, try running a “Scan Disk” test via the dashboard. It’s in the same section as before, just a bit further down. Don’t reboot yet.

birthi · November 12, 2023, 12:14am

I’m not familiar with the preformatted text, how/where to apply the </>?