Sentinel DX4000 Recovery - Stuck on Recreating My Storage

I am looking for some information (that I can’t find in the WD documentation) regarding a rebuild of a Sentinel DX4000 unit.
I am rebuilding a Sentinel DX4000 from parts - purchased the diskless unit (WDBLGT0160KBK) and using two new 2TB WD2002FYPS drives (same drives as in my other two WD Sentinel units). I downloaded the PANAM .ISO image PANAM_SvrRecovery_1_7_7_21_.iso and mounted the image using Gizmo’s Virtual Disk utility. I created two thumb drives, one for the RECOVERY and one for RECREATE STORAGE. I attached the two new drives, one at a time, to a worker PC and used diskpart to clean each drives, with diskpart reporting success. I then installed the drives in the NAS unit and booted with the RECREATE MY STORAGE thumb drive. The boot process went according to the WD Documentation. According to the documentation, when the unit’s display reaches “INITIALIZING OK LOADING RECOVERY”, the instruction says to “Allow the recovery process to complete.” That process has now been running for 22 hours for a pair of 2 TB drives. As I recall, a full setup of four 2 TB drives took 24 hours, so this seems excessively long. Since all the other documentation has been consistent with the documentation, I am reluctant to stop the process just yet to troubleshoot more. However there is no “progress” information such as “percent complete” and the documentation does not indicate that any other display information should appear.
My question is: is this RECREATE MY STORAGE process stuck and I need to restart or change some parameters, or do I just need to wait for completion? If I need to wait, how long does it usually take to RECREATE STORAGE on a pair of 2 TB drives?
Patiently awaiting a friendly reply.
Skip

Hi,
First of all you need to verify that you have a good ISO by comparing the hash.
It has been a long time, but I think it says recreate success, not loading recovery
Recreate goes very fast, less than 5 minutes. There will be a log file on the thumb with details
C:\Users\Budro\Downloads\PANAM_SvrRecovery_1_7_6_21_.iso:
Verified: Unsigned
File date: 1:53 AM 1/14/2017
MD5: 32031F2454D3510D7CE2BF5891F7D25D
SHA1: EF0E132813182B2202A5FF54180C840BEE44F59E
SHA256: 6D511FC43FEFBC088E6A069397539B575AA6E81067D0A5BC81E1FEF12CDCD1B1

Thanks, Gramps!. For PANAM_SvrRecovery_1_7_6_21_.iso which I downloaded from WDC.com, I do get that hash code exactly. So, I am convinced that I have a good image. I also received exactly the same “success” message as shown in the documentation both when I created the RECOVERY thumb drive and the RECREATE MY STORAGE thumb drive.

… The WDC Server Administrator documentation does not show any message after “INITIALIZING OK LOADING RECOVERY” so I am guessing that your “success” message would be correct. I also appreciate that the storage creation completion should have been within a few minutes, unless the RECREATE MY STORAGE process also performs a low level format of both 2 TB drives. But more than 24 hours now is a long time! I guess I am turning back here to ask, is the process hung? And, if it is, what can I do next. I am presuming that reading the log file is the next step. I will do that, but I’m not sure what to do with that info afterwards and how to prevent a recurrence when I retry the RECREATE process. And, yes, I will remake the thumb drive - I understand about the XML file deletion.

Do I need to re-clean both 2 TB drives using diskpart since I never received a completion message?

look at the log file on the thumb

could you have used the wrong thumb drive to boot from?

It is quite doubtful that I mixed up the thumb drives. I was pretty careful to keep them separate. I also compared them side by side just now and I did use the correct drive. So, I recreated the RECREATE MY STORAGE thumb drive and retried according to the instructions shown on the “success” page. I am wondering if my Norton antivirus has anything to do with what gets written to the thumb drive. Since I don’t have a list of the files to be contained on the RECREATE thumb, I have no idea if all the files are there or not. All I have is the application saying “success”.

I have now tried the process again. Same thing: The process still seems to hang at “INITIALIZING OK STARTING RECOVERY” - the display does not move beyond that point.

As for looking at the recovery.log file on the thumb, I was surprised to see that there was no log file, either at the root directory or in the WDRECOVERY folder. I don’t think that the process is ever getting to the actual loading phase.

Here’s a question about using RECREATE: When I originally CLEANed both drives using diskpart, should I have created a boot partition on one and done a format as well, or is the RECREATE MY STORAGE process supposed to do that for me?

No just diskpart >Clean
Try a diff “brand” thumbdrive
make sure you are using the recreate thumb, the small one

No clue if it matters but put both drives in the left side of the box

sorry, you say recreate completes with success? There should be a log file on the recreate thumb
Your PC must remain on during the recovery process you should have an indication on your PC screen for next etc
You may need another PC without a 3rd party AV

Let me clarify. The process to create the RECREATE MY STORAGE thumbdrive reports a successful creation (that is from the virtually mounted .ISO image) while the thumbdrive is still inserted in the worker PC. When I insert that thumbdrive into the DX4000 and boot the unit from the newly-created thumbdrive, the process hangs at “INITIALIZING OK LOADING RECOVERY” and never completes, i.e., the message never changes and never says that it was successful. Moreover, if I perform a hard shutdown (holding the on/off butt for 4 seconds) on the DX4000, there is no log file on the thumbdrive. That is what makes me believe that the recreate/recovery process never loaded and never began to execute.

The two drives are located in Slots 0 and 1 (i.e. the left side) of the array. I have assumed all along that Slot 0 will eventually have the drive with the boot partition and the bootable image.

“Your PC must remain on during the recovery process you should have an indication on your PC screen for next etc
You may need another PC without a 3rd party AV”
The (worker) PC must remain on? It is, in fact on. Bot h the worker PC and the DX4000 are connected to the network. The router reports that it gave an IP address to the unit from the DHCP pool. But what application would be reporting to the PC or showing status from the DX4000? At the end of the Recreate Storage wizard, it says the next steps are 1. Shutdown WD Sentinel, 2. Restart WD Sentinal with USB plugged in and pressing button. It says the server reboots in Recovery mode. There is a “Finish” button. Clicking that closes the Recreate Storage wizard. What app talks to the DX4000 over the network?

The recovery process talks to the pc. Both drives will end up with the same stuff on both.

Guess we need to start over, do your drives have the same letters after what you posted? WD2002FYPS-02W3B0 OR WD2002FYPS-02W3B1

Is this diskless unit good?

Yes you run setup.exe from the virtual drive and make a thumb using recreate
insert it in the DX and hold in the reset button on the back while you power on. Hold the button about 20 seconds or until the screen says something. It should say recreate success quickly

Both drives are WD2002FYPS-01U1B1, new, out of the box and they are identical to the eight other drives I have in two other DX4000 boxes (so I am pretty sure they should work on this box).
I am pretty sure the box is good, or should I say, I have no reason to believe the box is bad. it sounds rights, it powers up and the lights blink as I would expect and (they say) it was pulled from an operational system to update to larger servers. I will say that I THINK it works because it shows no sign of not working.
I have been running the recreate scenario as you describe: Run setup.exe from the virtual drive and make a thumb using recreate. Insert it in the DX and hold in the reset button on the back while you power on. Hold the button about 20 seconds or until the screen says “INITIALIZING OK LOADING RECOVERY”. From that point on the LCD display on the DX does not change, nor does the process complete.
But there is no other application running on the worker PC once the Recreate drive is inserted into the DX, correct? It’s all on the DX at this point, right?

If the box(s) came with those drives perhaps
See if there is a whitelist.xml on the thumb and edit it just replace one of the others with your values
I do not see that on the list
https://support.wdc.com/knowledgebase/answer.aspx?ID=9443#panam

Okay, I will try editing the whitelist.
Meanwhile, I also found this factoid on the WDC/support regarding recovery:
Create a bootable USB flash drive for server recovery’ appears during Server Recovery on a WD Sentinel RX4100 or DX4000
Answer ID 9889
Issue:
During a server recovery on a WD Sentinel RX4100 or a WD Sentinel DX4000, a Create a bootable USB flash drive for server recovery message appears. It requests the use of an 8 GB or larger USB flash drive. This occurs even if a drive larger than 8 GB is connected.
Cause:
Due to certain limitations with the Server Recovery image size, there are some requirements that should be considered before running the recovery process on a WD Sentinel RX4100 or a DX4000:
Solution:
Based on our USB compatibility tests, the following limitations have been found:
1. The USB Flash Drive must be a USB 2.0 flash drive (not USB 3.0).
2. Despite what the message says, The USB Flash Drive must have a capacity of 16 GB or 32 GB in size.
A USB Flash Drive with a storage capacity greater than 32 GB will not work.

Based on this info, I replaced the 4GB thumb drive I was using with a 16GB drive and regenerated the “Recreate My Storage” thumbdrive. I also re-cleaned the drives using diskpart on the worker PC.
Because I saw the red lgiths on Port 1, 3, and 4 flash red initially (I later learned this had nothing to do with the drives present in the slots), I booted the DX with the drives in slots 2 and 4. This time, the process proceeded beyond where it was stuck and started loading the recovery OS. The processed died, however, after cleaning the drives with a message “INVALID DRIVE COUNT”. A search of the recovery.log file showed an “invalid numberof drives”:

2009.01.01 05:15:14 - INFO - RaidCfg64 detected 0 drives
2009.01.01 05:15:14 - WARN - No RAID volume detected!
2009.01.01 05:15:14 - INFO - Checking RAID status (2): Volume Missing
2009.01.01 05:15:15 - WARN - RAID volume status is bad or volume is missing, need to re-create the RAID volume: Volume Missing
2009.01.01 05:15:16 - INFO - Deleting all RAID volumes
2009.01.01 05:15:21 - INFO - Re-creating RAID volume
2009.01.01 05:15:21 - ERROR - Invalid number of drives for RAID volume create: 0
2009.01.01 05:15:21 - INFO - =========== Recovery app END (4) ===========

So, it appears that the DX found the two drives and accepted them (or I think I would have received a “bad storage” response. I believe I now need to go looking for the reason for an “INVALID DRIVE COUNT” or how to force the DX to reconstruct the system as RAID 1 rather than RAID 5. However, I’ll try adding the WD2002FYPS-01U1B1 drives to the whitelist first.

Oops! Just opened lcd0.log, whose contents state “INVALID DRIVE”. Now I am definitely headed to edit the whitelist.

Well, my bad! I have two confessions to make:

  1. In looking at the compatibility lists, I realize that my other two DX4000s are 4TB boxes (WDBLGT0040KBK) and the box I am trying to rehab is a 16TB box (WDGLBT0160KBK), so they are NOT identical.
  2. I was wrong about the exact model of the drives. While the two drives I am trying to use in the DX in question are WD2002FYPS-01U1B1, the drives in my original boxes are the canonical WD2002FYPS-02W3B0 (I pulled them out and looked).
    So, I probably do have an incompatible drive problem. The question then becomes whether I can make them compatible using any of the other WDC utilities. While WDC released a drive compatibility utility some years ago, it does not appear that the RECREATE MY STORAGE or RECOVERY drives (OSes) were updated to accept these drives - probably because it would mean updating the DX firmware too. Other than going out and buying two new drives that are on the list, is there a way to make these drives compatible?