[GUIDE] Computing New CRC and Size Values for Kernel Config Partitions

usury · September 20, 2021, 2:57pm

[GUIDE] Computing New CRC and Size Values for Kernel Config Partitions

WD MyCloud Gen1 (Comcerto CPU with Barebox bootloader)

In This Guide

use factory-supplied GPL Source Code
use files available in factory firmware (or firmware sq-*.deb archives)
analyze leading binary data within k1m0.env (partition 7) or k1m1.env (partition 8)
determine fields to edit
edit and null-pad kernel config, possibly changing its length
- store new length values to binary area of kernel config files
compute new crc32 values
- store new crc values to binary area of kernel config files

This guide assumes an advanced use case for your WD MyCloud Gen1.
This is not a beginner guide.

Assumptions

you have some insight about barebox bootloader
you are probably using UART to examine the device as it boots
you are able to examine WD-Supplied GPL Source Code to confirm the accuracy of the information here
you know what a kernel configs is
you know how to write kernel config to the appropriate partition(s)
you are confident with commandline tools and their usage
- particularly hexedit, hexdump, and dd

You may have landed here due to the following errors observed over UART during boot:

sataenv: partition 7 wrong crc on env data
sataenv: partition 8 wrong crc on env data

## editorial comments
## - notice how mac_addr is a bogus value and other values are blank
## - barebox appears to load a placeholder config if no config partitions pass crc
## - you can find the placeholder mac address by searching barebox source code
commandline:$console=ttyS0,115200n8, init=/sbin/init rootfstype=ext4 rw noinitrd mac_addr=00:11:22:33:44:55 model= serial= board_test= btn_status=0

FOREWORD

The source code files referenced in this guide are found inside WD GPL Source Code available on the support downloads page.

A binary representation of a struct named envfs_super and another struct named envfs_inode appear at the start of the kernel config partitions (required by bootloader) - thought process follows.

Using the instructions in this guide, I have successfully

edited the four “size” and “crc” fields described below for an unmodified factory k1m0.env
- the results are indistinguishable from the original, confirmed using hexdump (usage shown below)
edited only the two “crc” fields based on a modified k1m0.env (one byte of the content changed, size did not)
- the resulting k1m0.env passes crc validation during boot on actual hardware

WHERE DOES `"wrong crc on env data"` OCCUR?

Using an IDE (or grep), search within barebox-2011.06.0/ subdir inside WD GPL Source Code for the string wrong crc on env data.

That string appears only in the file barebox-2011.06.0/commands/sataenv.c. That file shows how a struct of type envfs_super is populated from the first few bytes of partition 7 (or 8). It also shows how the bootloader validates the magic number and two different crc values found inside envfs_super, as well as an additional magic number found inside a struct of type envfs_inode (which itself is loaded from the bytes following envfs_super).

DECLARATIONS

From the file barebox-2011.06.0/include/envfs.h

all comments here are original to factory-supplied code (though re-aligned for readability below)

#define ENVFS_MAGIC		    0x798fba79	/* some random number */
#define ENVFS_INODE_MAGIC	0x67a8c78d

/*
 * Superblock information at the beginning of the FS.
 */
struct envfs_super {
    uint32_t magic;			/* ENVFS_MAGIC */
    uint32_t priority;
    uint32_t crc;			/* crc for the data */
    uint32_t size;			/* size of data */
    uint32_t flags;			/* feature flags */
    uint32_t future;		/* reserved for future use */
    uint32_t sb_crc;		/* crc for the superblock */
};

struct envfs_inode {
    uint32_t magic;	    /* ENVFS_INODE_MAGIC */
    uint32_t size;	    /* data size in bytes  */
    uint32_t namelen;   /* The length of the filename _including_ a trailing 0 */
    char data[0];	    /* The filename (zero terminated) + padding to 4 byte boundary
                         * followed by the data for this inode.
                         * The next inode follows after the data + padding to 4 byte
                         * boundary.
                         */
};

EXAMINATION

From examining /usr/share/k1m0.env (which becomes the contents of partition 7) from factory firmware, it seems clear that k1m0.env begins with envfs_super followed immediately by envsfs_inode (much more follows).

There are exactly 44 bytes of binary data at the beginning of each *.env file. Assuming sizeof uint32_t is 32bits (4-bytes), that makes sizeof( envfs_super ) 28 bytes (seven fields times four bytes each).

That still leaves 16 bytes .

Using hexdump with a more complex formatting string

first, seven 4-byte chunks (28 bytes total), formatted as unsigned int, on their own line
then, four 4-byte chunks (16 bytes total), formatted as unsigned int, also on their own line
finally, 524 bytes (522 rounded up to a multiple of 4) as a big character block
- why 522?
  - That value appears on the second output line
  - It corresponds to the expected position of the field envfs_inode.size
  - It corresponds to the observed size of the actual data (without including the null padding) that follows envfs_inode
- why 524?
  - Source code comments mention null-padding actual data portion to the next 4-byte boundary
  - It corresponds to the observed size of the actual data, now including the null padding
  - This consumes all the bytes so hexdump does not begin repeating the formatting pattern
- also 524 + 16 (16 is the observed size of the likely envfs_inode struct) exactly matches 540, which corresponds to observed value in envfs_super.size.
- the sizes of the actual structs envfs_super and envfs_inode are both accounted for in the file barebox-2011.06.0/commands/sataenv.c
- The size field of each struct is also utilized in a manner similar to how we are using it here as far as knowing how much data to read

$> hexdump -e '7/4 "%u\t""\n" 4/4 "%u\t""\n" 524/1 "%c""\n"' k1m0.env 

2039462521	0	546255751	540	0	0	4161781478
1739114381	522	1	0
#!/bin/sh
## Button initial state
btn_status=0
get_button_status
sata 
satapart 0x3008000 5 0x5000
sata stop
# This is customized for each environment variable script
bootargs="console=ttyS0,115200n8, init=/sbin/init"
bootargs="$bootargs root=/dev/md0 raid=autodetect"
bootargs="$bootargs rootfstype=ext3 rw noinitrd debug initcall_debug swapaccount=1 panic=3"
bootargs="$bootargs mac_addr=$eth0.ethaddr"
bootargs="$bootargs model=$model serial=$serial board_test=$board_test btn_status=$btn_status"
bootm /dev/mem.uImage

hexdump again, this time only the leading binary chunk (44 bytes) representing what I believe are the structs envfs_super and envfs_inode, appearing here (mostly) as unsigned int

first, one 4-byte chunk as hex, followed by six 4-byte chunks as unsigned int
- still seven chunks total
then, one 4-byte chunk as hex, followed by three 4-byte chunks as unsigned int
- still four chunks total
fields are all separated by tab character

$> hexdump -n 44 -e '1/4 "x%08x\t" 6/4 "%u\t""\n" 1/4 "x%08x\t" 3/4 "%u\t""\n"' k1m0.env 

x798fba79	0	546255751	540	0	0	4161781478
x67a8c78d	522	1	0

You can see the first hex chunk on each line exactly matches the magic number referenced in the comments for the first field of each of the two structs defined the file barebox-2011.06.0/include/envfs.h.

The file barebox-2011.06.0/commands/sataenv.c does confirm the presence of those magic numbers before proceeding to other tests.

That, and the rest of how barebox-2011.06.0/commands/sataenv.c appears to work, gives me a good amount of confidence in the following:

the first 28 bytes are envfs_super
- envfs_super.magic is first field of struct
  - observed hex value matches hard-coded hex value ENVFS_MAGIC
- envfs_super.size is fourth field of struct
  - observed value 540 corresponds to observed actual data (524 bytes when rounded up to a multiple of 4-bytes) plus the size of envfs_inode (16 bytes)
the next 16 bytes are envfs_inode
- envfs_inode.magic is first field of struct
  - observed hex value matches hard-coded hex value for ENVFS_INODE_MAGIC
- envfs_inode.size is second field of struct
  - Observed value 522 corresponds to observed actual length of data in *.env file (when rounded up to a multiple of 4-bytes and null-padded)
it seems envfs_inode could hold a filename? However…
- envfs_inode.namelen == 1 (a length which includes the null terminator according to the code comments)
- envfs_inode.data == 0 (which is the null terminator)
- the fact envfs_inode is 16 bytes on disk is a consequence of it having a data field (filename) containing an empty string (null terminator only)

TWO CHECKSUMS

There are two checksum fields in envfs_super

envfs_super.crc (third field)
envfs_super.sb_crc (seventh field)
those fields have very similar names
- I have taken care to use them correctly, below
- If you notice an error, please post a comment and I will edit to fix it

In barebox-2011.06.0/commands/sataenv.c, it…

reads sizeof( envfs_super ) (28 bytes)
- well, it reads a whole sector but ultimately looks at only the first 28 bytes
- bailout with error if "crc32( first 24 bytes of that data )" fails to match envfs_super.sb_crc
  - first 24 bytes is the first six fields of envfs_super
  - envfs_super.sb_crc is the seventh field and final four bytes of the 28 bytes read
- this means: changing anything in envfs_super (like the other crc field) requires calculating/writing a new envfs_super.sb_crc from envfs_super itself
reads from disk again, this time based on envfs_super.size (540 observed using hexdump, above)
- bailout with error if "crc32( all the data including envfs_inode but not envfs_super )" fails to match envfs_super.crc
  - envfs_super.crs is the third field of envfs_super
- this means: changing even a single byte in the actual human-readable kernel args requires calculating/writing a new envfs_super.crc
- this also means: changing the length of the actual human-readable kernel args requires storing the correct length value in envfs_inode.size and "( that_length + 16 + possible_padding )" in envfs_super.size (prior to calculating/writing new CRC values)

MY EDUCATED GUESS

Starting at byte zero of your k1m0.env config file (or config partition)

the first field of envfs_super occupies byte positions 0-3 (zero-based indexing)
envfs_super.crc (third field) occupies byte positions 8-11
envfs_super.size (fourth field) occupies byte positions 12-15
envfs_super.sb_crc (seventh field) occupies byte positions 24-27
envfs_inode.size (second field) occupies byte positions 32-34

IF YOU ARE NOT CHANGING THE UNPADDED LENGTH

Are you changing things in a way that the unpadded length within k1m0.env does not change?

only a single byte, like ext3 → ext4
a single byte in satapart directive to load from some other partition number
the satapart directive for loading a kernel occupying more than 0x5000 512-byte sectors (10MiB)

As long as the unpadded length does not change by even one byte, envfs_inode.size and envfs_super.size stay the same. Only the crc values need re-calculating.

Skip the section immediately below called “MODIFY k1m0.env WITH NEW LENGTH VALUES”
Go directly to “MODIFY k1m0.env WITH NEW CRC VALUES”

MODIFY `k1m0.env` WITH NEW LENGTH VALUES

Make a backup copy of your k1m0.env before proceeding
Read the actual code and double check my math before proceeding.

!! WARNING !! I have not personally tested changing the length of kernel config files though I used these steps to store factory values (the ones shown below) to the appropriate fields. The resulting file is identical to the original when inspected with hexdump (as shown earlier in this guide). I believe the procedure described here is accurate.

edit your config file with a text editor that will not alter/strip the 44 bytes of binary data at the beginning of the document, then save and exit.
then, edit your config file with hexedit (or use hexedit for all your editing)
- make certain to null-pad the end of the data to the next four-byte boundary
  - ie, insert 0 (zero, null)
- if your last actual char just happens to land at a four-byte boundary
  - I am not certain any null bytes are actually required
    - the bootloader reads based entirely upon the size it is told to read
  - I do not think there is any harm in adding four null bytes, though
hexedit reports your position in hex at the bottom of the screen
- remember, zero-based indexing here
- when your cursor is on position 0, it is the first byte
- when your cursor is on position 0x235 (hex), it is byte 0x236th byte
- envfs_super.size and envfs_inode.size are expecting the appropriate number of bytes, not the byte position

Determine and Store envfs_inode.size

determine your unpadded_end_position
- using hexedit, arrow-key to your last actual non-null character
- for instance, byte 0x235 (hex) is byte 565 (decimal)
  - use the hex-to-decimal functionality of a calculator
determine new value for envfs_inode.size
- ( unpadded_end_position_in_decimal) + 1 - 44 )
- ie ( 565 + 1 - 44 ) = 522
  - 44 is total length of the binary section at beginning of file
  - 44 is sizeof(envfs_super) + sizeof(envfs_inode)

Write New envfs_inode.size (using one-liner, below)

this is the second field of envfs_inode, beginning at byte position 32 in k1m0.env
REPLACE 522 IN printf WITH YOUR NEW VALUE FOR envfs_inode.size (determined immediately above)

## use `printf` to produce ascii hex string from the value `522`, padded to eight hex digits
## pipe that ascii  to `xxd` to "revert" ascii hex representation to binary (`-plain` with `-revert` means the *input* lacks hexdump-stype line numbers)
## pipe that binary to `xxd` again to swap the byte order (only works on binary input and only produces **ASCII output with hexdump-style line numbers**)
## pipe that ascii  to `xxd` one final time to "revert" to binary again (byte-swapped this time) (xxd knows to ignore hexdump-style line numbers)
## pipe that byte-swapped binary value to `dd` (`if=<stdin>` when not specified)
##   using four-byte blocks because `bs=4`
##   seek past *eight* four-byte blocks (`seek=8`)
##   write *one* four-byte block starting **byte position 32** (`count=1`)
##   DO NOT TRUNCATE THE THE FILE

$> printf "%08x" 522 | xxd -revert -plain | xxd -e -g4 | xxd -revert | dd of=k1m0.env bs=4 count=1 seek=8 conv=notrunc

Determine and Store envfs_super.size

determine your padded_end_position
- using hexedit, arrow-key to your final null byte
- for instance, byte 0x237 (hex) is byte 567 (decimal)
determine the new value for envfs_super.size
- ( padded_end_position_in_decimal) + 1 - 28 )
- ie ( 567 + 1 - 28 ) = 540
  - 28 is sizeof(envfs_super)

Write New envfs_super.size (using one-liner, below)

this is the fourth field of envfs_super, beginning at byte position 12 in k1m0.env
REPLACE 540 IN printf WITH YOUR NEW VALUE FOR envfs_super.size (determined immediately above)

## use `printf` to produce ascii hex string from the value `540`, padded to eight hex digits
## pipe that ascii  to `xxd` to "revert" ascii hex representation to binary (`-plain` with `-revert` means the *input* lacks hexdump-stype line numbers)
## pipe that binary to `xxd` again to swap the byte order (only works on binary input and only produces **ASCII output with hexdump-style line numbers**)
## pipe that ascii  to `xxd` one final time to "revert" to binary again (byte-swapped this time) (xxd knows to ignore hexdump-style line numbers)
## pipe that byte-swapped binary value to `dd` (`if=<stdin>` when not specified)
##   using four-byte blocks because `bs=4`
##   seek past *three* four-byte blocks (`seek=3`)
##   write *one* four-byte block starting at **byte position 12** (`count=1`)
##   DO NOT TRUNCATE THE THE FILE

$> printf "%08x" 540 | xxd -revert -plain | xxd -e -g4 | xxd -revert | dd of=k1m0.env bs=4 count=1 seek=3 conv=notrunc

YOU STILL MUST COMPUTE CRC VALUES

length values are part of the data used in computing crc values
you must set both of the new length values prior to performing the steps immediately below in “Modify k1m0.env with New CRC Values”

MODIFY `k1m0.env` WITH NEW CRC VALUES

Make a backup copy of your k1m0.env before proceeding

This section uses (obviously named) temporary files to compute CRC
crc32 reads only from a file (at least mine does)
Delete temporary files when you are finished

Calculate and Store envfs_super.crc

this is the third field of envfs_super, beginning at byte position 8 in k1m0.env

Extract fragment from k1m0.env

read envfs_super.size bytes starting at byte 28
byte 28 is where envfs_inode begins

## replace `count=540` with your `envfs_super.size` value
## read `540` one-byte blocks, skipping the first `28` one-byte blocks

$> dd if=k1m0.env bs=1 skip=28 count=540 of=temp-envfs_super.crc

temp file now contains fragment of k1m0.env needed to compute new value for envfs_super.crc

Calculate CRC and Store CRC (using one-liner, below)

## calculate checksum of the fragment within temp file (produces ascii hex string)
## pipe that ascii  to `xxd` to "revert" ascii hex representation to binary (`-plain` with `-revert` means the *input* lacks hexdump-stype line numbers)
## pipe that binary to `xxd` again to swap the byte order (only works on binary input and only produces **ASCII output with hexdump-style line numbers**)
## pipe that ascii  to `xxd` one final time to "revert" to binary again (byte-swapped this time) (xxd knows to ignore hexdump-style line numbers)
## pipe that byte-swapped binary value to `dd` (`if=<stdin>` when not specified)
##   using four-byte blocks because `bs=4`
##   seek past *two* four-byte blocks (`seek=2`)
##   write *one* four-byte block starting at **byte position 8** (`count=1`)
##   DO NOT TRUNCATE THE THE FILE

$> crc32 temp-envfs_super.crc | xxd -revert -plain | xxd -e -g4 | xxd -revert | dd of=k1m0.env bs=4 count=1 seek=2 conv=notrunc

** OR **
You could take the hex value provided by $> crc32 temp-envfs_super.crc and
use hexedit to manually swap bytes and enter it in the proper place

Calculate and Store envfs_super.sb_crc

this is the seventh field of envfs_super, beginning at byte position 24 in k1m0.env
Extract fragment from k1m0.env
- read 24 bytes starting at byte 0
- this is the first six fields of envfs_super
- the seventh field of envfs_super is not used when bootloader confirms CRC
- it should not be used when generating CRC (plus, you are about to change field seven anyway)
```
## read 6 four-byte blocks starting at the beginning of `k1m0.env`
## this temp file does **NOT** depend on any size value you previously computed

$> dd if=k1m0.env bs=4 count=6 of=temp-envfs_super.sb_crc
```
- temp file now contains fragment of k1m0.env needed to compute new value for envfs_super.sb_crc

Calculate SB_CRC and Store SB_CRC (using one-liner, below)

## calculate checksum of the fragment within temp file (produces ascii hex string)
## pipe that ascii  to `xxd` to "revert" ascii hex representation to binary (`-plain` with `-revert` means the *input* lacks hexdump-stype line numbers)
## pipe that binary to `xxd` again to swap the byte order (only works on binary input and only produces **ASCII output with hexdump-style line numbers**)
## pipe that ascii  to `xxd` one final time to "revert" to binary again (byte-swapped this time) (xxd knows to ignore hexdump-style line numbers)
## pipe that byte-swapped binary value to `dd` (`if=<stdin>` when not specified)
##   using four-byte blocks because `bs=4`
##   seek past *six* four-byte blocks (`seek=6`)
##   write *one* four-byte block starting at **byte position 24** (`count=1)
##   DO NOT TRUNCATE THE THE FILE

$> crc32 temp-envfs_super.sb_crc | xxd -revert -plain | xxd -e -g4 | xxd -revert | dd of=k1m0.env bs=4 count=1 seek=6 conv=notrunc

!! DONE !!

Confirm your k1m0.env looks correct by using hexdump as shown much earlier

edits: typos, formatting, clarification

Fox_exe · September 21, 2021, 8:06am

Or just use u-boot-tools:
mkimage -A arm -O linux -T script -C none -n boot.scr -d boot.scr boot.img
dd if=boot.img of=/dev/sda7

[GUIDE] Computing New CRC and Size Values for Kernel Config Partitions