PR4100 with group descriptors corrupted!

brightskyapps · September 9, 2019, 10:36am

Ok, so my guess is that the USB drive is /dev/sde as the others were part of the Raid.

I created a Table and partition, then when I check I can see that it should create the swap on /dev/sde1 but I get this error when creating a swap:

# blkid

/dev/sdd2: UUID="7b532c3f-a908-9a56-57b1-77b6a8f67641" UUID_SUB="373be9a8-0fcb-5be2-5cba-d71edb8f89fb" LABEL="1" TYPE="linux_raid_member" PARTLABEL="Microsoft basic data" PARTUUID="cbee0f08-f474-419f-859f-71b7f193ff46"

/dev/loop0: TYPE="squashfs"

/dev/mmcblk0p1: LABEL="wdnas_efi" UUID="A8D3-03D2" TYPE="vfat" PARTLABEL="EFI System" PARTUUID="fcddf37a-68bc-45f6-84fe-a5b2f6ee3c50"

/dev/mmcblk0p2: LABEL="wdnas_kernel" UUID="3db962ce-8b1b-4712-a438-6ae8e20f68b0" TYPE="ext4" PARTLABEL="kernel" PARTUUID="6e493cbc-164b-4872-88c9-ada9df3028b8"

/dev/mmcblk0p3: LABEL="wdnas_initramfs" UUID="9f89a790-3d7c-46b0-a391-47a4081c9c4b" TYPE="ext4"

/dev/mmcblk0p4: LABEL="wdnas_image.cfs" UUID="99eedd29-ee30-499a-ba8e-05c05fb443fa" TYPE="ext4"

/dev/mmcblk0p5: LABEL="wdnas_rescue_fw" UUID="375858b6-2fcd-4784-860a-7e57669a327d" TYPE="ext4"

/dev/mmcblk0p6: LABEL="wdnas_config" UUID="395d6777-7073-4e52-a831-13526990a093" TYPE="ext4"

/dev/mmcblk0p9: LABEL="wdnas_backup" UUID="e288be5b-8210-487d-a8d3-99b57181e7b7" TYPE="ext4"

/dev/sda1: UUID="bf698bed-acf9-1fae-6d3d-418575ee90b2" TYPE="linux_raid_member" PARTLABEL="Linux swap" PARTUUID="45a0bc36-2b44-44ef-a6d1-a0839d69595b"

/dev/sda2: UUID="7b532c3f-a908-9a56-57b1-77b6a8f67641" UUID_SUB="3f70ce6e-6f78-c325-224f-2464c643d3de" LABEL="1" TYPE="linux_raid_member" PARTLABEL="Microsoft basic data" PARTUUID="4ebcf81b-029a-4232-b11e-b1c48256fc61"

/dev/sda4: UUID="0524ecfa-5d9b-4ca9-a9d7-9ba243ae0d78" TYPE="ext4"

/dev/sdc1: UUID="bf698bed-acf9-1fae-6d3d-418575ee90b2" TYPE="linux_raid_member" PARTLABEL="Linux swap" PARTUUID="7872f91d-fb15-4327-873f-68471864a4c3"

/dev/sdc2: UUID="7b532c3f-a908-9a56-57b1-77b6a8f67641" UUID_SUB="d8fc0980-ed27-b7a1-4b39-4661efdfe26a" LABEL="1" TYPE="linux_raid_member" PARTLABEL="Microsoft basic data" PARTUUID="ab3f13de-8b41-4494-ad57-2568229e6f18"

/dev/sdc4: UUID="a7ddaba5-7d13-42bc-aa7b-07692bc90fdf" TYPE="ext4"

/dev/sdb1: UUID="bf698bed-acf9-1fae-6d3d-418575ee90b2" TYPE="linux_raid_member" PARTLABEL="Linux swap" PARTUUID="fb718fd3-8518-402c-803a-59ba443225f1"

/dev/sdb2: UUID="7b532c3f-a908-9a56-57b1-77b6a8f67641" UUID_SUB="3ab23e03-3ebc-cdb2-f28c-db5cf82d96bf" LABEL="1" TYPE="linux_raid_member" PARTLABEL="Microsoft basic data" PARTUUID="7d4d6c77-3f74-4d96-bd66-5570d103d4bd"

/dev/sdb4: UUID="0b10701b-4aa2-4a04-9c95-45ab677937ae" TYPE="ext4"

/dev/sdd1: UUID="bf698bed-acf9-1fae-6d3d-418575ee90b2" TYPE="linux_raid_member" PARTLABEL="Linux swap" PARTUUID="32da8593-8045-4de6-a7d7-44d776cf1eb6"

/dev/sdd4: UUID="11b5a998-a33f-4b2c-8ff5-a5f36b54f21b" TYPE="ext4"

/dev/md0: UUID="047f2844-554c-492c-b115-a01ca14b46bb" TYPE="swap"

/dev/md1: UUID="4bd33a2e-3a58-4f99-8cc0-870781e0e56a" TYPE="ext4"

/dev/sde2: LABEL="WDElements" UUID="5B42-1DDC" TYPE="exfat" PARTUUID="3fa6ecb0-904d-43db-aba1-9a09d13bb8c3"

/dev/sde1: LABEL="EFI" UUID="67E3-17ED" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="7011c988-e3ab-4f6e-9e75-2a258313107c"

/dev/mmcblk0: PTTYPE="gpt"

/dev/sda3: PARTLABEL="Microsoft basic data" PARTUUID="8f1086e0-97eb-412c-a7b6-5c28fbdd0726"

/dev/sdc3: PARTLABEL="Microsoft basic data" PARTUUID="fb46d30d-1722-4354-a3c8-5e3e10a764ba"

/dev/sdb3: PARTLABEL="Microsoft basic data" PARTUUID="397c9a29-ea8b-45ff-a981-b71dbadadc83"

/dev/sdd3: PARTLABEL="Microsoft basic data" PARTUUID="f4027bcc-1abd-4022-bf62-9686119faeb0"

# swapon /dev/sde1

swapon: /dev/sde1: Invalid argument

brightskyapps · September 9, 2019, 10:50am

I can use mkswap:

mkswap /dev/sde1

Setting up swapspace version 1, size = 209711104 bytes

UUID=1e0e7e41-21c4-41eb-b1e1-9cb37eb985d2

brightskyapps · September 9, 2019, 11:02am

running again with swap and nohup
Just to confirm:
nohup e2fsck -y -b 214990848 /dev/md1 > /tmp/fsck.log 2>&1 &

Tfl · September 9, 2019, 12:38pm

Myeah 200MB, that’s not going to cut it.
On my EX2Ultra, the OS prepares a RAID1 swap on the disks of 2G.

You can use the free command to get memory usage in kilobytes.

# free
              total        used        free      shared  buff/cache   available
Mem:        1037312      435520      104896           0      496896           0
-/+ buffers/cache:       435520      601792
Swap:       2097056      102560     1994496

Now for your huge operation, I’d try to get at least 8 or 16G of swap memory… up to 64G.
Fsck may have to move large blocks/files around, which requires a lot of memory.

brightskyapps · September 9, 2019, 12:53pm

That must be what the default size is for mkswap /dev/sde1, I’ll have to go look how to make one for 64GB

Tfl · September 9, 2019, 12:55pm

/dev/sde2: LABEL="WDElements" UUID="5B42-1DDC" TYPE="exfat" PARTUUID="3fa6ecb0-904d-43db-aba1-9a09d13bb8c3"
/dev/sde1: LABEL="EFI" UUID="67E3-17ED" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="7011c988-e3ab-4f6e-9e75-2a258313107c"

Maybe use /dev/sde2 instead
Make sure you do this first

umount /dev/sde2

brightskyapps · September 9, 2019, 12:56pm

mkswap /dev/sde2

Setting up swapspace version 1, size = 999990226944 bytes

UUID=e1a66956-9c8e-456f-9123-d2987735346f

Tfl · September 9, 2019, 12:57pm

I’m not going to count those nines, but it looks like enough

brightskyapps · September 9, 2019, 12:57pm

yeah its a terabyte

brightskyapps · September 9, 2019, 12:59pm

I think it might be too big, as if I cat /proc/swaps. then it doesn’t show

cat /proc/swaps
Filename				Type		Size	Used	Priority
/dev/md0                                partition	2097084	91912	-1

brightskyapps · September 9, 2019, 1:27pm

Doesn’t want to add a swap anymore since I removed the last one

blkid
/dev/sdd2: UUID="7b532c3f-a908-9a56-57b1-77b6a8f67641" UUID_SUB="373be9a8-0fcb-5be2-5cba-d71edb8f89fb" LABEL="1" TYPE="linux_raid_member" PARTLABEL="Microsoft basic data" PARTUUID="cbee0f08-f474-419f-859f-71b7f193ff46" 
/dev/loop0: TYPE="squashfs" 
/dev/mmcblk0p1: LABEL="wdnas_efi" UUID="A8D3-03D2" TYPE="vfat" PARTLABEL="EFI System" PARTUUID="fcddf37a-68bc-45f6-84fe-a5b2f6ee3c50" 
/dev/mmcblk0p2: LABEL="wdnas_kernel" UUID="3db962ce-8b1b-4712-a438-6ae8e20f68b0" TYPE="ext4" PARTLABEL="kernel" PARTUUID="6e493cbc-164b-4872-88c9-ada9df3028b8" 
/dev/mmcblk0p3: LABEL="wdnas_initramfs" UUID="9f89a790-3d7c-46b0-a391-47a4081c9c4b" TYPE="ext4" 
/dev/mmcblk0p4: LABEL="wdnas_image.cfs" UUID="99eedd29-ee30-499a-ba8e-05c05fb443fa" TYPE="ext4" 
/dev/mmcblk0p5: LABEL="wdnas_rescue_fw" UUID="375858b6-2fcd-4784-860a-7e57669a327d" TYPE="ext4" 
/dev/mmcblk0p6: LABEL="wdnas_config" UUID="395d6777-7073-4e52-a831-13526990a093" TYPE="ext4" 
/dev/mmcblk0p9: LABEL="wdnas_backup" UUID="e288be5b-8210-487d-a8d3-99b57181e7b7" TYPE="ext4" 
/dev/sda1: UUID="bf698bed-acf9-1fae-6d3d-418575ee90b2" TYPE="linux_raid_member" PARTLABEL="Linux swap" PARTUUID="45a0bc36-2b44-44ef-a6d1-a0839d69595b" 
/dev/sda2: UUID="7b532c3f-a908-9a56-57b1-77b6a8f67641" UUID_SUB="3f70ce6e-6f78-c325-224f-2464c643d3de" LABEL="1" TYPE="linux_raid_member" PARTLABEL="Microsoft basic data" PARTUUID="4ebcf81b-029a-4232-b11e-b1c48256fc61" 
/dev/sda4: UUID="0524ecfa-5d9b-4ca9-a9d7-9ba243ae0d78" TYPE="ext4" 
/dev/sdc1: UUID="bf698bed-acf9-1fae-6d3d-418575ee90b2" TYPE="linux_raid_member" PARTLABEL="Linux swap" PARTUUID="7872f91d-fb15-4327-873f-68471864a4c3" 
/dev/sdc2: UUID="7b532c3f-a908-9a56-57b1-77b6a8f67641" UUID_SUB="d8fc0980-ed27-b7a1-4b39-4661efdfe26a" LABEL="1" TYPE="linux_raid_member" PARTLABEL="Microsoft basic data" PARTUUID="ab3f13de-8b41-4494-ad57-2568229e6f18" 
/dev/sdc4: UUID="a7ddaba5-7d13-42bc-aa7b-07692bc90fdf" TYPE="ext4" 
/dev/sdb1: UUID="bf698bed-acf9-1fae-6d3d-418575ee90b2" TYPE="linux_raid_member" PARTLABEL="Linux swap" PARTUUID="fb718fd3-8518-402c-803a-59ba443225f1" 
/dev/sdb2: UUID="7b532c3f-a908-9a56-57b1-77b6a8f67641" UUID_SUB="3ab23e03-3ebc-cdb2-f28c-db5cf82d96bf" LABEL="1" TYPE="linux_raid_member" PARTLABEL="Microsoft basic data" PARTUUID="7d4d6c77-3f74-4d96-bd66-5570d103d4bd" 
/dev/sdb4: UUID="0b10701b-4aa2-4a04-9c95-45ab677937ae" TYPE="ext4" 
/dev/sdd1: UUID="bf698bed-acf9-1fae-6d3d-418575ee90b2" TYPE="linux_raid_member" PARTLABEL="Linux swap" PARTUUID="32da8593-8045-4de6-a7d7-44d776cf1eb6" 
/dev/sdd4: UUID="11b5a998-a33f-4b2c-8ff5-a5f36b54f21b" TYPE="ext4" 
/dev/md0: UUID="047f2844-554c-492c-b115-a01ca14b46bb" TYPE="swap" 
/dev/md1: UUID="4bd33a2e-3a58-4f99-8cc0-870781e0e56a" TYPE="ext4" 
/dev/sde1: LABEL="EFI" UUID="67E3-17ED" TYPE="vfat" PARTLABEL="Linux swap" PARTUUID="6a842dc1-8ef8-4cea-8999-a9275c9c1095" 
/dev/mmcblk0: PTTYPE="gpt" 
/dev/sda3: PARTLABEL="Microsoft basic data" PARTUUID="8f1086e0-97eb-412c-a7b6-5c28fbdd0726" 
/dev/sdc3: PARTLABEL="Microsoft basic data" PARTUUID="fb46d30d-1722-4354-a3c8-5e3e10a764ba" 
/dev/sdb3: PARTLABEL="Microsoft basic data" PARTUUID="397c9a29-ea8b-45ff-a981-b71dbadadc83" 
/dev/sdd3: PARTLABEL="Microsoft basic data" PARTUUID="f4027bcc-1abd-4022-bf62-9686119faeb0" 
root@McCloud / # umount /dev/sde1
umount: /dev/sde1: not mounted
root@McCloud / # mkswap /dev/sde1
Setting up swapspace version 1, size = 68718424576 bytes
UUID=049181d7-cf8f-4f8c-a0e4-3c0c54125c07
root@McCloud / # cat /proc/swaps
Filename				Type		Size	Used	Priority
/dev/md0                                partition	2097084	90732	-1

brightskyapps · September 9, 2019, 2:56pm

Do I need to add the swap to the /etc/fstab/ ?

The partition UUID is 8c1893a7-f00c-4b6d-a33f-fbd05467975b, so I think I have to add:

UUID=8c1893a7-f00c-4b6d-a33f-fbd05467975b none swap sw 0 0

I have a Swap partition created on /dev/sde1 but whenever I add it by using mkswap /dev/sde1 it never shows up in the cat /proc/swaps

Ok, it’s now added, and running in nohup, back in 5 days…or more

Tfl · September 10, 2019, 11:07am

I’ll update my prior instructions. You need to create swap with mkswap (similar to mkfs to setup a file system on a partition) and then swapon to enable it (similar to mount for a filesystem).

brightskyapps · September 27, 2019, 2:48pm

So it has been running for over a week now, it mysteriously stopped and when I checked there wasn’t even a log file in the tmp directory so I have no idea why it stopped, I started it again and it is currently still running but the output to the log file has not changed in over 5 days.

The last line in the log file is:
Inode 44232998, i_blocks is 49794562579095, should be 15. Fix? yes

brightskyapps · September 27, 2019, 11:37pm

The drives are fine, they have no issues at all, the problem was caused because of a power failure and again this last one looks like it was also caused by the same issue as I have now seen through the alert in the admin console that a power failure occurred last weekend whilst I was away.

brightskyapps · October 25, 2019, 11:10am

Still waiting, its crunching away and I’ve checked if the process is still running and it is, so my guess is that it is still doing something, have heard some people take a month to do fsck sometimes when doing a superblock replacement. It is 32TB of HD’s after all.

brightskyapps · November 11, 2019, 7:45pm

Just a quick update, it is actually still going, still writing to the log file and still churning away. This could be the longest e2fsck in history…

Tfl · November 12, 2019, 11:17am

Wow, I’m impressed.
Scrubbing my healthy zfs pool with 26TB of data took only 23h
Every now and then I make an offline backup of the critical data ofc.

brightskyapps · February 16, 2020, 2:33pm

This may be the longest fsck in history, it’s still going but is still writing to the log file, shall I just leave it or do you think this is just going to go on forever? Attached log file, was written to again a few days ago since its been on the last fix. The box is still crunching away. Thats almost 4 and a half months now.
fsck.txt (425.1 KB)

brightskyapps · March 29, 2020, 3:46pm

So, after Months if e2fsk going on, it has finally finished.

this is the last part of the log file:
Deleted inode 69941889 has zero dtime. Fix? yes

Deleted inode 69941890 has zero dtime. Fix? yes

Deleted inode 69941891 has zero dtime. Fix? yes

Deleted inode 69941892 has zero dtime. Fix? yes

Deleted inode 69941894 has zero dtime. Fix? yes

Deleted inode 69941895 has zero dtime. Fix? yes

Deleted inode 69941896 has zero dtime. Fix? yes

Deleted inode 69941898 has zero dtime. Fix? yes

Deleted inode 69941899 has zero dtime. Fix? yes

Deleted inode 69941900 has zero dtime. Fix? yes

Deleted inode 69941902 has zero dtime. Fix? yes

Deleted inode 69941903 has zero dtime. Fix? yes

Deleted inode 69941904 has zero dtime. Fix? yes

Inode 69941918 has an invalid extent node (blk 5377662985, lblk 0)
Clear? yes

e2fsck: e2fsck_read_bitmaps: illegal bitmap block(s) for /dev/md1

/dev/md1: ***** FILE SYSTEM WAS MODIFIED *****
e2fsck: aborted

/dev/md1: ***** FILE SYSTEM WAS MODIFIED *****

What do I have to do now to get files that were recovered etc?