Any interests in Kernel 4.0 on My book live?

WD Legacy Products My Book Live
81

Hi. Many thanks for all the howtos and kernel patches. I successfully debootsrapped debian jessie and compiled a 4.1 kernel with patches from GitHub - KL-Yang/mbl-linux-4.0: WD My Book Live Kernel 4.1.y patches . On power on, jessie boots. But on a reboot, I only get the uboot prompt on the serial console:

U-Boot 2009.0I2C:   ready
DRAM:  256 MB
FLASH: 512 kB
DTT:   1 FAILED INIT
Net:   PHY EC1 Register: 0x2c8c
ppc_4xx_eth0

Type run flash_nfs to mount root filesystem over NFS

Hit any key to stop autoboot:  0
=>
(... boot stops here ...)

The boot.scr does not seem to be processed. When I cut power and restore it within under ~5 seconds, same result. Only after I cut power for longer (~10 s seems to be safe), the boot proceeds as exoected:

U-Boot 2009.08 (Sep 02 2010 - 21:48:57), Build: 0.2.3

CPU:   AMCC PowerPC APM82181 Rev. C at 800 MHz (PLB=200, OPB=100, EBC=100 MHz)
       Security support
       Bootstrap Option E - Boot ROM Location NOR/SRAM (8 bits)
       32 kB I-Cache 32 kB D-Cache
Board: Apollo-3G - APM82181 Board, 2*SATA, 1*USB
I2C:   ready
DRAM:  Auto calibration 256 MB
FLASH: 512 kB
DTT:   1 FAILED INIT
Net:   PHY EC1 Register: 0x2c8c
ppc_4xx_eth0

Type run flash_nfs to mount root filesystem over NFS

Hit any key to stop autoboot:  0
SATA DWC initialization 1
(... boot continues ...)

I suppose this is some kind of hard reset whereas the first case is a kind of softer reset? Is there any way to either force the hard reset from linux of make the soft reset perform a nornal boot?

Thanks!

82

hello IOdev,

  1. the NETCONSOLE entries are the same.
  2. the grep gives expected results.
  3. /sys/kernel/config/netconsole does not exist.

did your 4.9.24 kernel pass the torture tests?!

83

Sorry for the delay, I have more than a full time job and was traveling for several weeks, so no access to my MBL. 4.9.24 did not pass the tests due to two bugs which were fixed in subsequent kernels. 4.9.31 passed 96 hours of torture test, at least with Jumbo packets disabled. I also managed to fix the SATA driver to support NCQ, but in contrast to 2.6.32.70, I am seeing very little performance increase :frowning: I’ll try a few more Synopsys controller options, but I am nearly out of ideas. Still with 166 MB/s read and 103 MB/s write, that is just 4 to 5% away from the 2.6.32.70 kernel performance, and better than the WD specs for the drive. Maybe that is the limit a 4.9 kernel can achieve as some of the changes in libata and co improve portability and overall functionality but at the expensive of performance. Sometimes as we see 3 to 4 levels of functions being called, none of which are inlined, and each of them doing barely more than a test followed by another function call. I can post my work on 4.9.31 as-is if anyone is interested, otherwise I’ll see if I can fix the jumbo packets issue in the network driver and post afterwards.
Don’t have a DUO to test on, so the new SATA driver may fail, but I also post the 4.9.24 (more conservative) version.
Ewald

84

@jbohac,

Did you modify the board to attach a serial console ? If so, the port settings or your serial-to-USB converter may cause uboot to pause. The slightest port activity makes uboot think a key has been hit. I had a similar problem with one of my USB-to-serial converters on my “production” MBL, but it’s back in it’s enclosure now, so I can’t attach a console to replicate.
Try removing the serial data out pin (make it a read only console) and see if that let’s it it “auto boot”.
I assume you reboot with “systemctl reboot”.

I managed to install a double netconsole on uboot and MBL kernel boot, so no more need for a UART type console (my development MBL is a very early version with no pre-soldered UART pin island)

Ewald

85

the performance improvement has given the box a new lease of life - it’s more than enough. thanks for your amazing efforts!

how about the apollo3g.dtb ! :blush:
cheers

86

@Ewald:
thanks so much, that was it; I feel very stupid not to have tried booting with the serial cable detached.
My MBL had unpopulated solder pads on the board, I soldered a 4 pin header to them.

87

Here are my dts files, with a script to compile them. There are pre-compiled versions for both the MBL and MBL DUO. You may also try the LEDE project 4.9 kernel DTS files, but I was unable to make these work, so I have not included them. As far as I remember from the old days, the APM82181 does not have a GPIO1 or 2 and my early version board definitely does not have these. Maybe that is why those dtb’s are failing. There is a version for the DUO that enables the second sata. USB is enabled too, but I yet have to build a USB enabled kernel. At first look the USB_DWC2 driver should work, but I was not able to make it work. Maybe I can get the old DWC_OTG driver to work, but that could be a stretch…
Ewald

88

@IOdev:
Here is a 4.9.31 compiled kernel with usb support. Porting of the old usb_otg driver was easier than I thought. However I have no USB ports to test. Adding a USB port to my development MBL might be doable as the APM82181 has a build-in USB clock, but I would have to make modifications to the driver and the soldering might turn out to be challenge. The driver might use an upgrade anyway as it leaves out some major capabilities of the HW, like the FIFO buffers etc. Note that this kernel does not have WIFI support, so a WIFI dongle will not work. Next I will upgrade to 4.9.33 as it fixes a problem I encountered with the kernel watchdog but was unable to pinpoint.
Forget to mention: this kernel requires the dtb/dts files from the previous post and has not been validated as stable (stable = 96 hour NAS torture test with no fault or performance degradation). The goal is just to check whether the USB driver works. Will post a stable 4.9.33 kernel shortly (without USB).

PS. When running Debian Jessie, the command to list USB devices is “usb-devices” as lib-usb, lsusb and USBFS kernel driver are all obsolete. Another place to look is /sys/kernel/debug/usb.

89

Here is version 4.9.33. The main update for this version is a re-write of the IBM EMAC driver. The standard upstream driver was not performing very well, had a few bugs that caused hangs/data corruption under stress and was missing key HW functionalities supported by the APM81281 CPU and the Broadcom 54610 NIC. Consequently, at the standard MTU of 1500 bytes, netcat (nc) was only able to transfer 43MB/s over TCP. Jumbo packet support was broken as well.
In the new driver there is support for TAH (only TSO support still missing), Jumbo packets up to 9000 bytes, full MAL support, full MDIO and OCM support, Interrupt Coalescing, SYSFS (meaning you can modify EMAC parameters via /sys/class/net/eth0/, Mask Carrier Extension signals, Powerdown mode, WOL (wake on LAN) support, Broadcom NIC 54610 tuning etc. This kernel has sustained a 48 hour torture test with Jumbo packets disabled. One of the challenges is to validate all the combinations of options and MTU’s, I simply don’t have the time…

The most optimal MTU size is 4080 bytes, due to the MAL hardware, but performance for 9000 bytes is within 5%.
At MTU 4080, netcat transmits now at 122MB/s for TCP, the theoretical limit for 1GE. AT MTU=1500 this is around 95MB/s or more than twice the performance of the original driver. That is with all of the above features enabled, some of which offer power saving and reduced CPU usage at the expense of extra code in critical paths (=performance loss)…
With standard Debian Jessie SAMBA, a 1GB file read from Windows 10 Anniversary release is around 96MB/s, while writes are around 84GB/s. With some changes to the SAMBA code, I was able to get 101MB/s writes (clearly now limited by the SATA driver) and 118MB/s reads. However, a full torture test of this SAMBA build will takes weeks on our simple MBL’s and without this test, the risk for data corruption, memory leaks and/or loss is simply too high. Hence I am not planning to make this Samba modification available.
Next on the plan, if I get a few free hours late in the evening :sleeping:, is some more performance tuning of the SATA driver (110MB/s write and 175MB/s read is the goal, or I need to buy a faster WD hard drive…) and fixing the encryption HW to accelerate rsync and ssh.

Installation: extract the compressed tar archive to any directory (paths are relative). Copy boot/apollo3g.dtb_4.9.33 to /boot/apollo3g.dtb (make a copy of your original dtb file first), copy boot/uImage_4.9.33.nc.usb to /boot/uImage (again: backup first) and copy the whole lib/modules/4.9.33-mbl+ directory to /lib/modules.

Performance verification: on your Linux box (or Windows) make sure you configure the MTU at 4080 bytes (ifconfig eth0 mtu 4080 up).
On the WBL: nc -l -n -q 0 -T Maximize-Throughput IP-of-Linux-box -p 2222 >/dev/null
On my Linux Mint PC: dd if=/dev/zero bs=1024K count=1024 | nc -n -q 0 -T throughput IP-of-WBL 2222
To validate Samba performance on Windows, don’t forget to modify the MTU of your NIC, restart the SMB daemon (/etc/init.d/smbd restart) first and ideally unmap/remap the share.

Disclaimer: This voids your warranty. Despite all due diligence and testing done, data loss or corruption is at your own risk.

90

New version of the 4.9.33 kernel, now with hardware large segment offloading (TSO) implemented, as well as a few minor optimizations of the kernel sk_buff structure. With standard Debian Jessie Samba on Windows 10 x64, I now get 122MB/s read (the theoretical maximum on 1GE) and 84MB/s writes with both MTU’s of 4080 and 9000. No improvements on write speed, since TSO is only relevant on packet transmit (aka read speed from NAS clients). With modified Samba, I currently hit a ceiling of 120MB/s write speeds. Please note that I have a more recent hybrid WD drive, so on regular WD green or red drives, the drive may limit the speeds you will see. That said 110MB/s reads on SAMBA should be possible.

I am back-porting these drivers to kernel version 2.6.32.71, which should make it possible to run just standard WD software with just the kernel and dtb files replaced.
Ewald

1 Like
91

Does this kernel works on top of std Debian by kl-yang?

Just extract, rename and reboot?

I reply to myself, NO. Extracted the file to the correct locations, reboot and it never came back. This was extracted over a kl-yang std debian install. I’ll troubleshoot it when I get a chance to hook up a serial console.

92

Is there a way we can get the patches so we can compile it for other Linux distros like Gentoo? I have Gentoo running on my 3 MBL’s (1T, 2T, and 3T) and have been running 4.4.2 for quite awhile.

93

I am revisiting this thread after several years and quite astonished to see that everyone is running a more recent OS and a newer kernel.

Thank you, developers!

My WD MBL (with 2TB) is mainly used to download torrents and acts as a NAS drive for my Kodi.

Since I have 1.4 TB of data already on the drive, I assume before I begin I need to move that over to some other storage.

I have reasonably OK knowledge of Linux commands and have a Windows desktop to connect to the WD MBL.
Do I need to open up the WDMBL? Or, can I do the upgrade of the OS just by commands?
What is the risk of permanent loss of hardware?
Is there any option to upgrade to Debian -- Debian “stretch” Release Information?

94

@asdfe,
Sorry for the delay. Yes, you should be able to install it on the image of kl-yang.
In fact, that is what I did to get started. That said, kl-yang created a minimal image, meaning there were many packages missing. In addition, I had to fix a number of issues, a few Debian defects, but mainly configuration issues as the MBL is aging HW compared to modern systems. For example, I added or updated 491 packages as reported by “dpkg --get-selections | sed -n ‘s/\t+install$//p’|wc -l”. Also configured samba (added several patches), nfs, rtc, ntp etc.

One way to get started is to use the netconsole installation documented here on top of KL Yang’s image. If you follow that it will give you a 4.1.39 kernel and allow you to follow boot messages on a remote windows/linux system.
After that you can install the 4.9.33 kernel as per my message above.

Alternatively, you can just take my Debian Jessie tar image posted below. It has the everything included, in less than an hour you should be done.

Are you using a DUO version? If so, your luck may very as I have no duo hardware to test with.
Ewald

95

@vakharia,
While it should be possible to create a MBL update package that updates an official system with Jessie and a new kernel, many of us got into this because our WD drive died. We were left with no support, all data lost (or to be restored) and forced to open up the cover to replace the drive. It’s also not a full functional replacement. Yes, you get the latest Debian release, a very recent kernel, almost 2x performance increase, but you don’t get the nice WD web GUI and some of the included 3rd party packages.
In addition, the MBL uses a user data partition with 64K block sizes, which makes it impossible to mount on a standard Intel/AMD Linux box.
But, if you have made a full backup of all the MBL partitions (dd), it should be possible to just copy a new boot OS and keep all your data as-is, either using KL-Yang’s Jessie image, or the image I will make available shortly.

To your final questions

  1. totally possible to update to Debian Stretch! I did it as a test but reverted from it as a few packages were not available for PPC at the time and I did not have the time to compile them. So your luck may vary…
  2. I have “bricked” my system at least 100 times, but the de-brick process which you can find in this forum has always worked. And I don’t have a serial console, since my HW is a proto/early version with no pads to solder a console.
    If you take a dd copy of all your partitions, you can even restore your drive as-is with the current software.

Ewald

96

@Simba7,
I have posted here the patches for 4.9.33.
The only things that are not included are:

  • latest version of the SATA driver (but a very recent one that will give you 90%+ of the performance). The latest rewrite is using advanced capabilities of the DWC SATA core (like continuous FIFO DMA) which on one of the 12 types of WD’s drives I tested created a single corrupted data block in over a million writes. Still, until 100% sure, it makes no sense to release.
  • the rewrite of the network driver, for the simple reason I have not had the time to package this in a patch.
  • Changes to samba source
  • Debian Jessie fixes (provided upstream)
    Ewald
97

@asdfe,
First of all, thanks for giving it a try and reporting back. I think I found what could have caused the issue. If you happened to use DHCP, the system would probably have come up fine, but you would not have been able to connect via the network. In Debian Jessie the system startup process is highly parallelized. This already caused many issues on our fast Intel/AMD servers (e.g. see here), but on a much slower MBL system this resulted in many issues that no one had found before. So, I’ve had to create quite a number of fixes and workarounds. In this case, it’s a racing issue between the mounting of /run as tmpfs and the start of dhclient. My regular test drive is a ultra fast sshd which never has the issue. But on a first gen WD green drive, it’s a 50% hit with 4.9.33, but always fine with 4.1.39. Anyhow, I added a few lines of kernel code to one of the drivers that dhclient needs to avoid this problem and it’s also integrated into the Debian Jessie image I will post shortly.
Sorry for the inconvenience,

Ewald

99

Tried compiling your sata_dwc_pmp driver, but it is giving me an incompatible pointer type and stops.

CC drivers/ata/sata_dwc_pmp.o
drivers/ata/sata_dwc_pmp.c: In function ‘clear_chan_interrupts’:
drivers/ata/sata_dwc_pmp.c:538:36: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
struct dmaIntrRegs* __iomem reg = &(sata_dma_regs->interrupt_clear);
^
drivers/ata/sata_dwc_pmp.c:539:15: error: dereferencing pointer to incomplete type ‘struct dmaIntrRegs’
out_le32(&reg->tfr.low, dma_chan);
^~
drivers/ata/sata_dwc_pmp.c: In function ‘sata_dwc_hardreset’:
drivers/ata/sata_dwc_pmp.c:1172:26: warning: unused variable ‘hsdev’ [-Wunused-variable]
struct sata_dwc_device *hsdev = HSDEV_FROM_HSDEVP(hsdevp);
^~~~~
drivers/ata/sata_dwc_pmp.c: In function ‘sata_dwc_qc_issue’:
drivers/ata/sata_dwc_pmp.c:1882:6: warning: unused variable ‘status’ [-Wunused-variable]
u32 status, sactive;
^~~~~~
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:294: drivers/ata/sata_dwc_pmp.o] Error 1
make[1]: *** [scripts/Makefile.build:544: drivers/ata] Error 2
make: *** [Makefile:991: drivers] Error 2

100

@Simba7,

Sorry for late reply and the challenges in compilation. It compiles without errors (but there might be warnings of unused variables) on both the MBL itself and on Linux OS (Ubuntu/Linux Mint with cross compiler).

Let’s check a few things:

  1. What kernel version are you trying to compile? It should be 4.9.x with x between 31 and 52.
  2. Did all patches install without errors? For the ATA driver, I posted the latest files [here] (https://drive.google.com/open?id=0B8Nj6R0Fc58pLXJFODRpaHJDUjg) so you can be 100% to have the same or working versions and exclude potential errors in the patch file. They maybe slightly newer, but definitely working for kernel 4.9.33. For kernels like 4.11+ or 4.14, I rewrote the drivers to take advantage of newer kernel features and EXT4 changes.
  3. What system + version are you compiling it on ? I have not tried to compile this on the official firmware for quite a while, so there might be issues with gcc/as versions etc. as I use advanced PowerPC options for both assembler and compiler
  4. try to build with “make V=1 uImage” for more verbose info

I am aware there is some cleanup to be done. More precisely there is one unused variable (statusIntr) and one function declared which is not defined/used (sata_reg2txt). Sorry about that, I posted a snapshot of development at the time and moved on to 4.14 for a few reasons.

To verify that the patches I posted compile OK, I took a fresh copy of kernel sources 4.9.52, applied the patches and ran “make uImage”. Everything builds with no problems. I will now validate the new 4.9.52 kernel first and then post a pre-build kernel for those not wanting to compile it themselves.

Hope this helps,
Ewald

101

Hi Ewald,

First of all, thanks for that all you doing.
I try to build kernel 4.9.33 (using cross compiler), but have an error related to building wndr4700. Could you please to look it?

List of command I perform:

export PATH=$PATH:~/x-tools/powerpc-405-linux-gnu/bin
export ARCH=powerpc
export CROSS_COMPILE=powerpc-405-linux-gnu-
wget -O ‘patches_4.9.33.7z’ ‘https://drive.google.com/uc?export=download&id=0B8Nj6R0Fc58pb3hwSWxCcEZVZzA’
mkdir patches_4.9.33
cd patches_4.9.33
p7zip -d …/patches_4.9.33.7z
cd …
wget https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.9.33.tar.xz
tar -xJf linux-4.9.33.tar.xz
cd linux-4.9.33
for f in …/patches_4.9.33/*.patch ; do patch -p1 < $f ; done
cp …/patches_4.9.33/config-4.9.txt .config
make menuconfig
make -j 7
make uImage

Output for last command:

…
make -f ./scripts/Makefile.build obj=arch/powerpc/kernel/vdso32
make -f ./scripts/Makefile.build obj=arch/powerpc/mm
make -f ./scripts/Makefile.build obj=arch/powerpc/lib
make -f ./scripts/Makefile.build obj=arch/powerpc/sysdev
make -f ./scripts/Makefile.build obj=arch/powerpc/platforms
make -f ./scripts/Makefile.build obj=arch/powerpc/platforms/44x
make[2]: *** No rule to make target ‘arch/powerpc/platforms/44x/wndr4700.o’, needed by ‘arch/powerpc/platforms/44x/built-in.o’. Stop.
scripts/Makefile.build:544: recipe for target ‘arch/powerpc/platforms/44x’ failed
make[1]: *** [arch/powerpc/platforms/44x] Error 2
Makefile:988: recipe for target ‘arch/powerpc/platforms’ failed
make: *** [arch/powerpc/platforms] Error 2

It seems there is missing wndr4700.c (there is no file arch/powerpc/platforms/44x/wndr4700.c).
UPDATE: It seems compiled when i found necessary file:

wget -O arch/powerpc/platforms/44x/wndr4700.c https://raw.githubusercontent.com/lede-project/source/master/target/linux/apm821xx/files/arch/powerpc/platforms/44x/wndr4700.c

Hope this will works :slight_smile:

One more question: your config has CONFIG_PPC_4K_PAGES=y, but KL-Yang noted rootfs will NOT work on stock filesystem! The stock kernel use 64K page size (filesystem is also 64K page), so maybe I need to set CONFIG_PPC_64K_PAGES=y instead.

Any suggestion?

Thanks.