I’m using your 4.1.5-mbl+ kernel for a while now and it does very well for me.
(two MBL with noWD HDDs< 1TB)
Is there an update on LED support for WD MBL or MBLD.
Can someone give me a hint wich part of Linux is responisble for this part to work properly?
I was hoping that apollo3g is a way to get it working, But with a line:
setenv load_part1 'sata init; ext2load sata 1:1 ${kernel_addr_r} /boot/uImage; ext2load sata 1:1 ${fdt_addr_r} /boot/apollo3g.dtb
as the hardware initialization on boot does not do it on my system, I don’t have a proper ttys0 on boot, so I sadly can’t see if there is a error message happening.
I looked trough
wd-lib*.deb
wd-nas*.deb
als well but no mentions of LED funktioniallity except the scripts
ledConfig.sh and ledCtr.sh but they are using a preexisting functionality of something else, kernel? module?
Any hints, where to start looking to get that working?
I gave the 4.4 Kernel a try by using the port of the LEDE/OpenWRT project.
Unfortunately the resulting performance was not anywhere as good as the original 2.6 or 4.1 kernels.
Find my writeup here:
Regarding the LED device; mine works fine by writeing red/green/blue/yellow/white to /sys/class/leds/a3g_led/color.
If I use green - the ATA-driver interacts with the LED on disk access.
There is a third part to my post which outlines some useful scripts, managing disk sleep, logging to ram, etc.
I also tried out OpenMediaVault (last post), but cannot recommend it at all on the MBL as the memory is rather limited and its single core CPU is just not up to the tasks the nifty GUI lays upon it.
I have tried windows ext2fs and it wasn’t stable; rigged an old machine with linux and mounted using fuseext2 for 2TB, at the rate it looks it will take a month to copy. I did not have direct SATA, going via USB2.0. using rsync to ensure everythiing gets copied even if its breaks off intermittently.
wanted to check if this would really go faster if I bothered with better I/O or CPU. my machine shows 80% CPU.
I am done with backup of my WD harddisk now after 4 weeks! I want to install debian jessie after formatting drive to something regular (4K blocks etc) so that copying in future is not such a hassle. of course, I am in position to reformat the drive.
with so many doc links I am not clear on how to go about it:
format the old WD disk, in what way, what are the commands
hook the reformatted old WD disk onto some linux system as mountpoint
install debian jessie kernel, from where? exact link for clean install
move hdd back into WD enclosure & wait for magic
I will use it as slow but steady owncloud server, and can install on debian. will LED work? will wakeup on LAN work?
I finally moved my Debian Wheezy and self-compiled Kernel 2.6.32.71 (I rewrote major parts of the SATA driver, DMA, crypto, splice memory DMA/page remapping) to my production NAS, which freed up my second NAS for some testing/development on Debbian Jessie and Kernel 4.x. While performance and stability in Wheezy/2.6.32.x is outstanding, it’s time to move to to modern versions…
So I installed KL Yang’s Debian image and kernel flawlessly (thanks!). The 4.1.20 kernel performance is quite OK, but the supplied kernel was unfortunately very unstable:
time dd if=/dev/zero of=/shares/software/tst.dd bs=1M count=1K: HANG ! (123MB/s on Wheezy/2.6.32.71)
time dd if=/shares/software/tst.dd of=/dev/null bs=1M count=1K: 170MB/s (176MB/s on Wheezy/2.6.32.71) → very promising!
copy 1GB file from Windows 10 to NAS via Samba: HANG! (around 500MB seems to be where it starts to go wrong)
copy 1GB file from NAS to Windows 10: 85MB/s (108MB/s on Wheezy/2.6.32.71 with Jumbo packets)
config eth0 mtu 4088: HANG! → looks like no Jumbo packet support
Then I checked out the latest 4.1.39 kernel, using KL Yang’s 4.1 patches plus a few extra ones from the Lede project, cross-compiled on a Linux Mint 18.1 server, activated kernel NFS etc:
time dd if=/dev/zero of=/shares/software/tst.dd bs=1M count=1K: 97MB/s (versus 123MB/s on Wheezy/my kernel 2.6.32.71)
time dd if=/shares/software/tst.dd of=/dev/null bs=1M count=1K: 173MB/s (versus 176MB/s on Wheezy/2.6.32.71) → very promising!
copy 1GB file from Windows 10 to NAS via Samba: 55MB/s (versus 86MB/s on Wheezy/my kernel 2.6.32.71 with Jumbo packets)
copy 1GB file from NAS to Windows 10: 85MB/s (versus 108MB/s on Wheezy/my kernel 2.6.32.71 with Jumbo)
config eth0 mtu 4088: HANG! → looks like no Jumbo packet support
48 hour torture test → stable
First tests on HW Crypto are promising too, some things are working others are broken.
Lots of trouble though to get Debian Jessie extended (NFS/Samba,ntpdate,…) and upgraded to the latest version 8.7 with security patches. E.g. renaming the server was a challenge and led many times to no boot, disk refused to go in standby at night, samba would not come up (bug in Jessie when /run is tmpfs), wrong package dependencies, se-linux troubles etc…). Once I get everything stable, I will post my Debian image, my SAMBA config files and all instructions to get Debian Jessie configured.
For those that want to rename KL Yang’s Debian image:
export hostname=newname
hostname newname
echo “127.0.0.1 newname localhost” > /etc/hosts
hostnamectl set-hostname newname (this sometimes fails, but no worries)
echo “newname” > /etc/hostname # uneeded if previous command succeeds
rm /etc/ssh/ssh_host_*
dpkg-reconfigure openssh-server
systemctl restart ssh
invoke-rc.d hostname.sh start
invoke-rc.d networking force-reload
invoke-rc.d network-manager force-reload
systemctl reboot (execute once everything succeeds)
if you don’t have a fully qualified hostname (with domain name) you might want to remove the exim4 (email server) package, since this might also give renaming issues (there are workarounds though).
There is great work done by KL Yang and Alex (see earlier in this thread), but there no real easy way to get both a stable and performing system that is running a fully configured (Samba, NFS server, MiniDLNA, twonky etc…), updated version Debian Jessie (8.7) like you have with the standard WD image.
The easiest way to get started is:
read this thread and specifically: KL Yangs notes on github, Alex’s blogs
mount the mybook drive on a Linux system (mine is Linux Mint 18.1)
if your drive is not bricked (you have the 4 partitions) format the first 2GB partition (root OS) to ext3 (I know KL has it in ext2, but I prefer ext3)
mount the partition and cd to root directory
make sure root (/) is owned by root (chown root:root .) or you get into lots of trouble with SE linux
restore KL Yangs image using “tar xvf /jessie_sda1_rootfs_password_4.1.21_64k.tar --strip 1”
unmount the drive from your Linux system and boot on MBL HW
you now are running Debian Jessie, but this kernel is not stable. KL Yang has other kernels which I assume don;t have issues, but he is using a 16K page size, whereas I prefer a 64K one. As an alternative you can compile your own
for your data drive: you can format this with 4K, 16K or 64K (blocksize has to be <= to kernel page size), but keep in mind that performance with 4K is known to be weak. So you will likely end up using 16K or 64K both of which have the same problem: you can’t mount the partition easily on a standard Linux Intel box which has a 4K pagesize. One work around is to have a 16/64K kernel on e.g an ARM64 system using CONFIG_ARM64_64K_PAGES=y. See link
however, it’s not such a big issue once you went through teh trouble of getting your WD drive out of the NAS box. Using NFS I copy data out of the MBL at 100MB/s and once you make a good backup using tar of your MBL Debian OS you can restore your system rapidly.
Thanks Ewald for the tips. I am not too well versed but these are my takeaways
a) move kernel to something modern debian jessie so that new stuff can be installed. if OS partition need to be formatted to ext3/64K for stability, will do as guided by experts on this forum all I need is corresponding image that is known to work!
b) not too particular about samba/twonky/nfs to be preinstalled if apt-get from standard repositories would work? is this not the case? can I not install plex/emby/owncloud once regular debian jessie is setup? to me that is the charm of this upgrade to be able to deploy new stuff, not constrained by WD offering
c) data blocks: 64k vs 16k vs 4k: my goal isto ensure that data copying off the WD disk is easier if something goes wrong. with the native format, I had to wait for 4 weeks using fusext2 etc. the same stuff was copied in 6hours once I got onto standard ext4 disk. if I format as 16k (for performance as suggested), will an old amd64 linux machine with debian be able to read it off somehow? installing some special kernel on old lenvo thinkpad in 6 hours is acceptable but 4 weeks of nail biting data transfer is not. If I have to compromise 20% runtime performance for this safety by using 4k blocks, I am ok.
Hi Jains,
on your questions:
a) No stability impact for ext2 vs. ext3. Ext3 has journalling and ensures better/faster recovery when your MBL loses power. Ext2 has slightly better performance. I now have a $20 mini 12V UPS (great stuff) and considering to go back to Ext2. Wrt the file system block size, again no stability impact. Always use 4K for root filesystem and your choice of 4K, 16K or 64K for data volume.Just keep in mind that kernel page size has to >= to file system block size and that 4K kernel page size will most likely have 40% less performance.
As I had not done any tests on 4.1.x kernels, I decided to break my soft raid setup and format /dev/sda2 to test the difference in speed using kernel 4.1.39 with 64K page size.
Read speed: 160/160/159 MB/s for block sizes on Ext4 of 4K/16K/64K → no difference
Write speed: 96/102/103 MB/s for block sizes on Ext4 of 4K/16K/64K → very minor ~5% difference
Samba read from Windows 10: 85/85/85 MB/s for block sizes on Ext4 of 4K/16K/64K → no difference
Samba write to Windows10: 55/55/55 MB/s for block sizes on Ext4 of 4K/16K/64K → no difference Conclusion: a 4K (default) ext4 file system is absolutely fine for a NAS/samba data volume and will allow direct mounts on a Linux Intel system. I had not had the chance to compile kernels with 4K/16K/64K page sizes, but i am quite convinced that 16K or 64K page size will be significantly superior.
b) in that case you can use KLY’s image. Just know you will spend a few days adding packages, upgrading, struggling with systemd (unless you know that stuff) and a whole series of Debian Jessie problems (e.g. journald is loosing kernel messages, samba package fails because /run is a tmpfs etc.). As soon as I get everything fixed I will post my Jessie image. If urgent, I am happy to post it as-is (just need to remove users/smbpasswd etc).
c) given the test results above, this is now easy: just use standard 4K.
There are better solutions than fusext2 though. Dump a simple root file system on sda1 and then use NFS over TCP to copy the files over at over 100MB/s.
I have been using the 16K page size kernel for maybe more than two years, so far so good.
The stable issue encountered is there are sometimes SATA reset error relate with DMA if the uptime goes beyond 30 days.
I have changed the default config “CONFIG_CONSISTENT_SIZE=0x02000000” and haven’s observe those issue with uptime around 60 days.
There are minidlna server in debian, and nfs/samba are easy to install with apt-get.
The real issue is disk/Ethernet performance is worse than factory firmware but acceptable. Especially the hardware encryption engine driver, without that the SCP speed is quite slow at around 8MB/s.
The MyBook Live has overheat issue if you live in a hot country, so I just remove the case and put it in a closet and run 24x7, and didn’t bother with the LED driver . That’s roughly how I use it.
KL, when I installed your kernel 4.1.20 with 64K pages with the corresponding Debian Jessie image and started my stability test scripts, it failed within 3 to 5 minutes albeit at different places in the test. One command though fails quite reliably with a hang:
time dd if=/dev/zero of=/shares/software/tst.dd bs=1M count=1K
Not clear what the issue is. I compiled 4.1.39, changed kernel config a bit and added a few extra patches, but nothing really major enough to explain why one kernel is “torture test” stable and the other not. It’s not related to the Debian version as I can reproduce it by just switching the kernel. I will do some tests with “CONFIG_CONSISTENT_SIZE=0x02000000”. Performance-wise, I am getting quite close to the factory version even without jumbo packets, but not yet as good as on my highly tuned 2.6.32.71 kernel as I rewrote major parts of the SATA DMA to include NCQ and other improvements (e.g. IBM newemac ethernet changes). I also noted Jessie with systemd decreases performance a bit an I still need to fix the fact that I am losing the first 25 kernel messages in journal-d. I found a few bugs in the kernel ring buffer code that were fixed in Android kernel but somehow were never implemented in mainstream kernel.
I tried with 16K page kernel, seems fine with dd, however those write numbers are low. When I got time, I would like to try your 2.6.32.x kernel. I am switching to use MBL as storage only, and use an ARM box with mainline kernel as front end server, a higher performance kernel is more suitable there.
Regards
root@mbl:/mybook# uname -a
Linux mbl 4.1.39-mbl+ #7 Sat Apr 29 07:44:01 UTC 2017 ppc GNU/Linux
root@mbl:/mybook# getconf PAGE_SIZE
16384
root@mbl:/mybook# time dd if=/dev/zero of=tst.dd bs=1M count=16K
16384+0 records in
16384+0 records out
17179869184 bytes (17 GB) copied, 646.947 s, 26.6 MB/s
Hi KL,
Version 2.6.32.70 kernel posted here. It has SATA NCQ support (rewrote parts of SATA driver) and I fixed some of the Crypto libraries. I can make it available as a patch on the sources. Will take a few days as I need to reinstall Wheezy on my development MBL or maybe I try to cross compile it. The kernel has 64K pages.
Right now I am struggling to get 4.9.24 to run properly as somehow 64K pages and other things are broken and the Lede project has apparently not hit/discovered these. 4.9.x fixes some issues I hit in 4.1.x and back-porting the fixes would be quite some work.
Ewald
My development MBL was an early version I received as a gift. One of the challenges is that it does not have UART solder patches and that makes it difficult to debug kernels if one can’t even see the kernel boot logs. Specifically developing SATA drivers is painful as nothing will log to disk if your driver has a bug. The good news is that the Linux kernel does have a remote console logging capability called NETCONSOLE. The bad news is that the IBM PPC EMAC driver does not support it as it does not support NETPOLL. NETCONSOLE requires the network card to support NETPOLL but for my use it only needs to send messages out. Hence I managed to patch the driver in a simple way to enable NETPOLL on eth0 and make NETCONSOLE work.
I posted a full HOW-TO here. It provides instructions for:
trying out a prebuild 4.1.39 kernel with tuned SATA PMP driver (but without NCQ). This driver achieves 160MB/s dd read and 101MB/s dd write. Adding NCQ won’t add a lot of additional performance on sequential read/write, but first limited tests show ~170MB/s read and 108MB/s write, so very minor improvement. These results seem quite independent of the file system block size (4K, 16K and 64K tested).
testing NETCONSOLE both after kernel boot and with uboot
building your own 4.1.39 kernel with NETCONSOLE but without tuned SATA driver. Once I get NCQ working using NETCONSOLE debugging I will update the patches. Also fixed are the missing kernel boot messages on Debian 8.7 by increasing CONFIG_LOG_BUF_SHIFT to 15 (from 14).
Happy testing, let me know if there are issues.
Ewald
I’ve been giving these kernels a try as I’ve wanted improved performance for years now.
but also ability to use the usb port for a wifi dongle or usb cable it to a RPi.
I believe a 3. series kernel is needed for that.
I got 102/115 write/read on the 2.6.32-70 as opposed to the incredibly low 9/32 for stock! (greenpower disks)
but then I went for the 4.1.39 for the usb and it would not boot.
I’m using the duo - could it be the md module is not in the kernel?
have now built and installed a 4.1.39 kernel.
I found that the eth0 driver was not configured in so I put smsc95xx in plus the RAID driver and I have booting system - wonderful!
the numbers are 96/116 for a dd test, which makes it quite a bit short of your 160 read result, Kwald ! any idea?
I tried the linuxConsole cmds above but the /sys/…/netconfig directory didn’t exist.
but no matter, with the right boot.scr I got output to another machine anyway. fantastic!
so just distributing a kernel uImage with eth0 and md drivers in-built is enough to get MBLD owners going with at least triple the read performance !
@IOdev: apologies for the troubles, I don’t have a duo to test, nor do I have USB. Happy to see you figured it all out . With respect to performance, I fixed/rewrote the SATA-DWC driver for 2.6.32-70 and enabled NCQ, so you will see maximum performance on that kernel, even though I think another 5% improvement is possible. The 102 write/115 read is pretty much the maximum you can get on a WD Green! On my development MBL, I have a WD40E31X which is a SSHD drive. That explains my ~170MB/s read and ~110MB/s write performance, which in fact is better than the 150MB/s sustained that the drive’s datasheet mentions . I figured a faster, more modern drive is better when you are working on the SATA driver itself, specifically to measure progress. Your 96/116 numbers are totally in line with expectations as NCQ is not yet working on the SATA driver. I have given up making that work on 4.1.39 because of too many issues with that kernel with respect to the PowerPC architecture. Things looks better on 4.9.24. Different problems (e.g. 64K pages does not even compile!) but much less! We have to thank the Lede team for many of the fixes.
With respect to USB and dwc_otg: I have this working on 4.9.24, so I assume it’s not difficult to make work on 4.1.39. My guess is that issue is in your /boot/apollo3g.dtb file. I will make sure I test USB OTG on 4.9.24 and will post an updated apollo3g.dtb file. Don’t give up hope on the WIFI! I am sure we will get that to work
all I need is corresponding image that is known to work!
. With respect to performance, I fixed/rewrote the SATA-DWC driver for 2.6.32-70 and enabled NCQ, so you will see maximum performance on that kernel, even though I think another 5% improvement is possible. The 102 write/115 read is pretty much the maximum you can get on a WD Green! On my development MBL, I have a WD40E31X which is a SSHD drive. That explains my ~170MB/s read and ~110MB/s write performance, which in fact is better than the 150MB/s sustained that the drive’s datasheet mentions 
. I figured a faster, more modern drive is better when you are working on the SATA driver itself, specifically to measure progress. Your 96/116 numbers are totally in line with expectations as NCQ is not yet working on the SATA driver. I have given up making that work on 4.1.39 because of too many issues with that kernel with respect to the PowerPC architecture. Things looks better on 4.9.24. Different problems (e.g. 64K pages does not even compile!) but much less! We have to thank the Lede team for many of the fixes.