SMART errors blocking boot

Maturity · September 8, 2023, 2:07pm

I have a Linux Centos 6 machine with 2SSDs for the OS and 4 WD spinning drives for files. 2 drives are older 3TB drives (WDC WD30EFRX-68AX9N0) and have been running fine for years and 2 are recently added 6TB drives (WDC WD60EFAX-68JH4N1) added less than a year ago. The spinning drives are in mirrored pairs. Now the machine spins failing some SMART error on boot until I unplug one of the 6TB drives (either one it seems). Then after boot the drive can be plugged back in and connected to the RAID pair and all works as expected. We thought it was maybe the motherboard failing so I ordered and installed a separate SATA card but the symptoms remain the same.

Running smartctl against the drive shows the following but removing a drive from the system and running the WD Windows tools against the drive for a quick test seems to indicate that it is fine.

Any ideas about this?

Thanks, M

SMART Error Log Version: 1
ATA Error Count: 8494 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 8494 occurred at disk power-on lifetime: 33 hours (1 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 08 00 00 00 e0  Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0

Maturity · September 8, 2023, 2:13pm

I received another SMART error email and checking the reported drive it now shows the following. Keep in mind that the 6TB drives have been running fine for months (since install Nov 2022).

# smartctl -a /dev/sdb
smartctl 5.43 2016-09-28 r4347 [x86_64-linux-2.6.32-754.35.1.el6.centos.plus.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               /1:0:0:0
Product:
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

Maturity · September 8, 2023, 3:37pm

Using a lowercase a gives more results it seems, here is both:

[root@swifter ~]# smartctl -a /dev/sdb
smartctl 5.43 2016-09-28 r4347 [x86_64-linux-2.6.32-754.35.1.el6.centos.plus.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               /1:0:0:0
Product:
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

[root@swifter ~]# smartctl -A /dev/sdb
smartctl 5.43 2016-09-28 r4347 [x86_64-linux-2.6.32-754.35.1.el6.centos.plus.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

Maturity · September 8, 2023, 4:06pm

I mostly just administer the machine and the PBX that runs on it and didn’t have any say in the purchase of these drives. My guess would be that they were selected because the they are the same as the 3TB drives that have been running fine for years. Not a lot of traffic on them from what I understand. What is the expected lifetime of these drives?

The other drive of the pair gives a lot of info:

I am thinking I’ll pull both drives and have the other fellow run the WD tool against them on a Windows machine and see what it says.


[root@swifter ~]# smartctl -a /dev/sde
smartctl 5.43 2016-09-28 r4347 [x86_64-linux-2.6.32-754.35.1.el6.centos.plus.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD60EFAX-68JH4N1
Serial Number:    WD-WX92D91E14H7
LU WWN Device Id: 5 0014ee 214c46e3f
Firmware Version: 83.00A83
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ACS-3 (unknown minor revision code: 0x006d)
Local Time is:    Fri Sep  8 12:03:22 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (14384) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 748) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3039) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   227   222   021    Pre-fail  Always       -       3608
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       20
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       6059
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       20
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       12
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2448
194 Temperature_Celsius     0x0022   124   104   000    Old_age   Always       -       26
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
ATA Error Count: 8494 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 8494 occurred at disk power-on lifetime: 33 hours (1 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 08 00 00 00 e0  Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 00 00 00 e0 08   1d+08:59:20.245  READ DMA
  ef 10 02 00 00 00 a0 08   1d+08:59:20.245  SET FEATURES [Reserved for Serial ATA]
  ec 00 00 00 00 00 a0 08   1d+08:59:20.244  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 08   1d+08:59:20.244  SET FEATURES [Set transfer mode]

Error 8493 occurred at disk power-on lifetime: 33 hours (1 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 08 00 00 00 e0  Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 00 00 00 e0 08   1d+08:59:20.244  READ DMA
  ef 10 02 00 00 00 a0 08   1d+08:59:20.244  SET FEATURES [Reserved for Serial ATA]
  ec 00 00 00 00 00 a0 08   1d+08:59:20.243  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 08   1d+08:59:20.243  SET FEATURES [Set transfer mode]

Error 8492 occurred at disk power-on lifetime: 33 hours (1 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 08 00 00 00 e0  Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 00 00 00 e0 08   1d+08:59:20.243  READ DMA
  ef 10 02 00 00 00 a0 08   1d+08:59:20.243  SET FEATURES [Reserved for Serial ATA]
  ec 00 00 00 00 00 a0 08   1d+08:59:20.242  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 08   1d+08:59:20.242  SET FEATURES [Set transfer mode]

Error 8491 occurred at disk power-on lifetime: 33 hours (1 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 08 00 00 00 e0  Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 00 00 00 e0 08   1d+08:59:20.242  READ DMA
  ef 10 02 00 00 00 a0 08   1d+08:59:20.242  SET FEATURES [Reserved for Serial ATA]
  ec 00 00 00 00 00 a0 08   1d+08:59:20.242  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 08   1d+08:59:20.241  SET FEATURES [Set transfer mode]

Error 8490 occurred at disk power-on lifetime: 33 hours (1 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 08 00 00 00 e0  Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 00 00 00 e0 08   1d+08:59:20.241  READ DMA
  ef 10 02 00 00 00 a0 08   1d+08:59:20.241  SET FEATURES [Reserved for Serial ATA]
  ec 00 00 00 00 00 a0 08   1d+08:59:20.241  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 08   1d+08:59:20.241  SET FEATURES [Set transfer mode]

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      5972         -
# 2  Short offline       Completed without error       00%        33         -
# 3  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

fzabkar · September 9, 2023, 1:43am

I don’t understand.

You begin by showing us /dev/sdb which has an absurd block size and capacity, but at the end you show us /dev/sde which appears normal apart from the error log which suggests that it wasn’t able to read physical sector 0.

Is /dev/sdb your SSD? :-?

Maturity · September 9, 2023, 2:23am

Ah yes, I had just determined that as well. I’d moved things around on the machine to see if it is a problem with a port but apparently not.

Looking at the SMART output for the 3TB drives that have been running for years and they show the same kind of errors so I suspect that the SMART messages are a red-herring.

I was able to capture some of the lines that fly past at startup and it looks like the slow write speed could very well be the problem, now to figure out a way around it. Are they still OK as individual drives? I’m thinking I could split the pair and just run a cron job to rsync from the main drive to the backup “mirror”

__ratelimit: 574 callbacks suppressed
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 0f b8 b8 f5 00 00 02 00
__ratelimit: 574 callbacks suppressed
end_request: I/O error, dev sdg, sector 2110113704
Buffer I/O error on device sdg1, logical block 263763957
lost page write due to I/O error on sdg1
Buffer I/O error on device sdg1, logical block 263763958
lost page write due to I/O error on sdg1
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 16 e3 ac a9 00 00 01 00
end_request: I/O error, dev sdg, sector 3072157000
Buffer I/O error on device sdg1, logical block 384019369
lost page write due to I/O error on sdg1
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 1d 18 9b b8 00 00 1e 00
end_request: I/O error, dev sdg, sector 3905215936
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 1d 18 9b d6 00 00 03 00
end_request: I/O error, dev sdg, sector 3905216176
Aborting journal on device sdg1-8.
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 1d 18 81 00 00 00 01 00
end_request: I/O error, dev sdg, sector 3905161216
Buffer I/O error on device sdg1, logical block 488144896
lost page write due to I/O error on sdg1
JBD2: I/O error detected when updating journal superblock for sdg1-8.
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 16 e4 62 27 00 00 02 00
end_request: I/O error, dev sdg, sector 3072528696
Buffer I/O error on device sdg1, logical block 384065831
lost page write due to I/O error on sdg1
Buffer I/O error on device sdg1, logical block 384065832
lost page write due to I/O error on sdg1
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 16 e4 63 4b 00 00 02 00
end_request: I/O error, dev sdg, sector 3072531032
Buffer I/O error on device sdg1, logical block 384066123
lost page write due to I/O error on sdg1
Buffer I/O error on device sdg1, logical block 384066124
lost page write due to I/O error on sdg1
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 16 e4 66 83 00 00 02 00
end_request: I/O error, dev sdg, sector 3072537624
Buffer I/O error on device sdg1, logical block 384066947
lost page write due to I/O error on sdg1
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 16 e4 67 bb 00 00 02 00
end_request: I/O error, dev sdg, sector 3072540120
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 16 e4 69 35 00 00 02 00
end_request: I/O error, dev sdg, sector 3072543144
EXT4-fs error (device sdg1) in ext4_da_writepages: IO failure
EXT4-fs error (device sdg1): ext4_journal_start_sb: Detected aborted journal
EXT4-fs (sdg1): Remounting filesystem read-only
EXT4-fs (sdg1): ext4_da_writepages: jbd2_start: 880 pages, ino 66331825; err -30

__ratelimit: 257 callbacks suppressed
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 0f b8 01 2a 00 00 01 00
__ratelimit: 257 callbacks suppressed
end_request: I/O error, dev sdg, sector 2109737296
__ratelimit: 463 callbacks suppressed
Buffer I/O error on device sdg1, logical block 263716906
lost page write due to I/O error on sdg1
sd 8:0:0:0: [sdg]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 8:0:0:0: [sdg]  Sense Key : Not Ready [current]
sd 8:0:0:0: [sdg]  Add. Sense: Medium not present
sd 8:0:0:0: [sdg] CDB: Write(10): 2a 00 0f b8 0f ee 00 00 01 00
end_request: I/O error, dev sdg, sector 2109767536
Buffer I/O error on device sdg1, logical block 263720686
lost page write due to I/O error on sdg1

fzabkar · September 9, 2023, 2:37am

It sounds like the drive is going offline, or not spinning up fast enough after waking up from sleep???

fzabkar · September 9, 2023, 3:46am

The following error occurs because a SCSI command is being sent to a SATA drive:

Terminate command early due to bad response to IEC mode page

To avoid this error, you need to append “-d ata” to your command line.

https://linux.die.net/man/8/smartctl

Use the forms “/dev/hd[a-t]” for IDE/ATA devices, and “/dev/sd[a-z]” for SCSI devices. For SATA disks accessed with libata, use “/dev/sd[a-z]” and append “-d ata” .

-d TYPE, --device=TYPE
Specifies the type of the device. The valid arguments to this option are:

auto - attempt to guess the device type from the device name or from controller type info provided by the operating system or from a matching USB ID entry in the drive database. This is the default.

test - prints the guessed type, then opens the device and prints the (possibly changed) TYPE name and then exists without performing any further commands.

ata - the device type is ATA. This prevents smartctl from issuing SCSI commands to an ATA device.