Any interests in Kernel 4.0 on My book live?

Ewald · December 13, 2018, 8:23am

@goldstar611,
The work from the OpenWRT team has been a blessing and is fully leveraged. They fixed a lot of the upstream issues in the core kernel modules and provide a working base. Plus they have a phenomenal build system, that includes cross-compiling. Lately they have also been fixing some of the crypto modules. It has saved me at least 100 hours of work…

So now I can focus pretty much on Debian Jessie and NAS performance and all the stuff that the OpenWRT is not so interested in such as getting 16K/64K pages to work, sata I/O performance with NCQ, ext4 performance (versus squashfs), NFS performance, netconsole support, kernel MD5 acceleration support for Rsync, samba performance etc. The generic Broadcom network driver does not have all HW acceleration that the APM821xx chip brings, such as interrupt coalescence etc. Lately I rewrote the 3 DMA drivers (slave, sync, async) into a single driver, and can now get 122MB/s sustained read and write over Samba using optimized splice and peripheral DMA. Will take some more time for release though (I am on assignment abroad + a few memory leaks + need to figure out why module unload does not work in 4.14). Their DWC sata drivers is not too bad though and if time permits, I may be integrated my optimizations with it.

I have not verified the performance difference with their latest kernels, but on 4.9 (OpenWRT 17.x) it was significant enough (50%+ on certain functions) to spend the extra effort. After all OpenWRT focuses on routers/networking and not on storage/NAS.

Ewald.

trogper · January 5, 2019, 9:53pm

Hi
I admire work you have done on the kernel, keep it up.
Have you considered putting it on Github? I think it would be more readable and they even have release hosting. I’d like to help you with that if you need.

Simba7 · January 11, 2019, 5:12am

Thanks for the kernel! I was able to get all 3 of the MyBook Live drives (1T, 2T, and 3T) up and running with Gentoo. No problems here!

Linux wdmbl1t 4.9.135-gentoo-mbl+ #2 Tue Oct 23 05:55:24 UTC 2018 ppc APM821XX GNU/Linux

Ewald · January 18, 2019, 6:15pm

@trogper,
I have a github account, that’s not the issue. I have no plans to fork the Linux kernel or replicate OpenWRT, but I can post the patches for each release when I get a little time.

trogper · January 18, 2019, 8:33pm

that’s what I meant, just the patches, a readme.md and maybe some releases/builds

Ewald · January 20, 2019, 10:56am

Version 1.0 of Github repository released
It’s long overdue, not because of a lack of Github skills but simply because of lack of time.
From now on, I will post patches, how-to’s etc. on Github. About 70% of documentation has been migrated,
Feel free to suggest improvements. Happy browsing.

trogper · February 16, 2019, 1:05pm

Since my hdd doesn’t spin down itself and after using hdparm -y, it spins back up after a few seconds.
lsof doesn’t show anything significant, I have tried log2ram (ramdisk for /var/log).
next thing I wanted to do was iotop, but it requires kernel with io accounting, however I can’t find it in menuconfig. Adding it to .config does nothing and is removed upon build.
Does ppc44x not support io accounting?

Ewald · February 17, 2019, 10:30am

@trogper,

Does ppc44x not support io accounting?

Yes, it does and I have tested/validated that for 4.9.x
With respect to disk standby, it depends a bit on the type of disk you have. Some disks honor hdparam -S, others don’t. For WD drives, there is an idle3ctl. For a Seagate drive that is not “well behaved”, I use a script in /etc/cron.hourly:

#!/bin/bash
# Simple cron script - set disk standby timer
hdparm -S 60 /dev/sda
out=$(hdparm -C /dev/sda)
if [[ $out = *“standby”* ]] ; then
hdparm -Y /dev/sda
fi

trogper · February 17, 2019, 5:07pm

Are you able to run iotop?
my disk is wd blue WD3200AAJS. When connected to uart, there was an error/timeout just before spinning the drive back up, but I’m unable to replicate it right now.
This seems to me like the OS is constantly reading from the disk or something

Ewald · February 18, 2019, 1:44pm

Yes, I have tested iotop. It requires CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING to be compiled in. If I recall well, I fixed a few things to have it pass my NAS torture test, but for development work it’s OK as-is. I have created a pre-compiled 4.9.149 kernel that supports it out of the box here.

Ewald · February 18, 2019, 1:55pm

Kernel 4.9.149 released: on GitHub

Available precompiled and in patch form
Two configurations: optimized for size and performance (~400K size difference, little performance difference)
Netconsole enabled by default (remove netconsole from /boot/scr and/or remove CONFIG_NETCONSOLE from kernel config file
removed IPv6, netfilter and other modules (simplifies testing and reduces amount of patchwork needed).
patch updates and backports in preparation of 4.19 (e.g. networking & sata driver)

Next planned release: 4.19.19 or later

Ewald · February 18, 2019, 5:54pm

I have seen IO errors when spinning an active disk down, so one should no use hdparm -C on a disk that is not in standby state (see my script). Reversely, the torture test I run includes 200 cycles of IO workloads (50% read and 50% write, 50% local and 50% over SMB/NFS) on a drive in spin down mode and I have not seen the error. That said, all drives I have feature 64MB of cache and that could explain why I am not seeing it. I noticed your drive has only 8MB, maybe that could explain it. Are you sure write cache is enabled?
Any error logged in dmesg/kern.log ?

pprobi · February 23, 2019, 7:01pm

Hi, I’m new to this, any tutorials on how to patch the kernel for v.2.6.32.70/71 ? I I currently would like to keep the OEM software for now and patching it with the patch available on github.

So I guess I have to compile the linux kernel matching the version 2.6.32.70/71 ?

I’ll appreciate you guys pointing me in the right directions ; )

Thanks,

Phil

From github (and what I want to do for now);

“I want to keep the original OEM software because of it’s web user interface and vendor support”
In that case the only custom kernel available is 2.6.32.70/71. You will find the patch in source code here. It will provide a considerable performance improvement over standard software, up to 2x on certain tests.

trogper · February 23, 2019, 10:16pm

Hi
if you don’t need any custom modifications, just download the precompiled version, which has the patches applied.

you should only do system modifications if you have a way to recover, if you can connect the disk to another linux system and revert changes.
before installing the new kernel, backup your old one ( /boot/uImage)

pprobi · February 23, 2019, 11:51pm

So I guess it goes like this;

I put the newer apollo3g.dtb with the script kdtc.sh to work with precompiled kernel.
I copy the archive file kernel-2.6.32.70-ncq.tgz to tmp folder.
I run this to copy the file and reboot on the precompiled kernel.

cd /
tar -xzf /tmp/kernel-2.6.32.70-ncq.tgz
cp /boot/uImage.2.6.32.70.64K_NCQ /boot/uImage
systemctl reboot

I just don’t want to screw this up, and don’t feel to install the full netconsole setup for now…

pprobi · February 24, 2019, 12:13am

I just read the infos on the appolo3g.dtb on github, so I guess I could skip step one from what I understand ? It will use the one already installed ?

pprobi · February 24, 2019, 1:48am

I got it finally (Skipped step 1)… But I’ll probably have to change the internal drive toe. I tought something was going on with the firmware, but as I reset everything, put this precompiled firmware, I still have really weird thicking noise from the drive. I tought it might have been related to whatever cron job or a dead loop scanning upnp files, but I think the drive is about to die.

Ewald · March 2, 2019, 10:25am

@pprobi,
Apologies, I should have been more clear on how to install 2.6.32.70 pre-compiled kernels, but you found it out yourself. I have updated the readme’s…
Sorry about your disk. Most likely it has failed, but just to be sure you could try a different power supply, ideally a 3A one. There is a capacitor on the PCB to help with startup peak power needs but on my development WBL this cap was not (no longer?) working properly. If the head got stuck, you might try your luck and look at this.

Ewald · March 11, 2019, 6:44pm

Kernel 4.19.24 released.
After skipping 4.14 due to stability issues, we finally have a 4.19 kernel that passed 96 hours of NAS torture testing and delivers performance that is very close to 4.9. As always, the work from the OpenWRT team has been leveraged. The performance of the DWC SATA driver was so close to the custom driver, that I decided to include it. On the plus side, the standard DWC Sata drivers has room for much more improvement, so we should be able to even outperform the custom driver when NCQ is added. On the minus side: there is a defect with 64K page sizes, but 4K and 16K work, and performance is good enough for release. Please note that stability has not been proven beyond 96 hours of testing and there are a few small performance issues I noticed for specific tests.

With a 4.19.x customer kernel, standard 4K block size ext4 file system, Debian 8.11, page size of 16K, network MTU size of 4080, one can expect:

Sequential disk reads of 121MB/s (dd if=tst.dd of=/dev/null bs=4k count=256K)
Sequential disk writes of 120MB/s to 160MB/s (dd if=/dev/zero of=tst.dd bs=1M or 4K)
Samba read speed of 110 MB/s (1GB file read, Windows 10 64-bit)
Samba write speed of 75 MB/s (1GB file write, Windows 10 64-bit)
NFS read/write of 112 MB/s and 85MB/s respectively

Pre-compiled kernel and patches here

Rasteirinho · March 14, 2019, 6:47pm

Hello,

With this kernel, my device not mount swap partition (sda2).
Can it be my mistake?

The led don’t work too, but I think the path is not ready yet.

Many thanks for your work!! I follow your github long time ago.