I figured that the new kernel package was just a matter of unpacking the .tar image at root.
Thanks for the rest of the instructions.
Using uname -r verifies the new kernel 4.9.44
I have a good gigabit network and with my EX2 ultra I can get 118MB/s read and write (disk spanning not RAID) with a standard MTU, so with all of my other devices it does not make sense changing my network to support jumbo frames.
With kernel 4.9.44 and MTU 1500 on the MBL, the large sequential write test was 96.7MB/s
SAMBA read 76 MB/s
SAMBA write 41 MB/s
which is a good performance jump from stock firmware
and I would hope that there are more security patches with the new kernel than with the stock firmware
it just took me a bit to get the SAMBA shares, users configured (I’m still somewhat new to linux)
Thanks for all your work - great update for some aging hardware.
One last question, should one be able to perform
apt-get update
apt-get upgrade
without any problems?
edit: the network shares are not showing up in Windows; what needs to be done to have the MBL broadcast the shares?
@thegooddoctor,
It looks like your hard drive might be the bottleneck to get much higher performance, but an MTU of 4080 on the MBL might still give you 50MB/s+ of SAMBA writes and >100MB/s reads, given the good quality of your net work.
Security updates is the main reason of this initiative, as it’s not possible to improve on work I did on kernel 2.6.32.70 on the original WBL image (121MB/s SAMBA read, 117MB/s SAMBA write is very close to the theoretical maximum).
It’s possible to perform apt-get-update/upgrade (I have done a few but not in the last months), but definitely take a full backup before, so you can always roll back.
The easiest way to do this is the use the second root partition that is normally used as soft RAID in the original MBL image (your choice of using dd/tar or soft raid methods).
I filed all Debian defects and fixes upstream so most likely they have made it into new releases, but left and right there might be a little glitch. Nothing to worry about though, but be prepared to invest some time in learning and dealing with the changes to Linux. For example, between KL Yangs version on Debian Jessie (8.1 I believe) and the one I posted (8.7), there is huge difference e.g. Debian moved to systemd for system and service management.
With “the network shares are not showing up in Windows”, I assume you mean using the Windows Explorer “Network” shortcut, because it’s definitely possible to connect to shares. Yes, I am aware of that but never bothered. Let me look into this…
Ewald
@oddstan,
In order to build a kernels 4.9 and above, you need an up to date development system with the correct gcc and development libraries.
My development system is a MBL (very early version, maybe proto, bought in the bay area for 15$, no duo).
From there you can compile your kernel of choice.
Patches and kernel config file for kernel 4.9.x here.
The patches can be applied to any 4.9.x kernel but be aware that some versions have stability and/or performance issues. 4.9.31 to 44 are OK as well as 4.9.77 and 78 as I managed to get patches approved upstream that fixed a series of defects/performance penalties after 4.9.44.
Please note that 4.9.77 and above still have some issues for which I need to create separate patches (e.g. kernel perf tools crash with segment violation) but as a NAS run-time this is OK.
With this combination you can compile your kernel with pnfs support on the WBL itself. Crross compile is another option, but you need anyhow something different than official FW as I have not been able to run kernels 4.x on Debian Squeeze.
I installed Debian “Jessie 8.7 with performance-tuned kernel 4.9.33” on a blank disk, following instruction given in August, installed the HD back in the WD enclosure and…
I can ping the WD, but I can’t log in
I got “connection refused” for both ssh and telnet.
for some reason while building an uImage it returns an error:
In file included from drivers/leds/led-core.c:20:0:
drivers/leds/leds.h: In function ‘led_set_color’:
drivers/leds/leds.h:24:11: error: ‘struct led_classdev’ has no member named ‘color_set’
led_cdev->color_set(led_cdev, value);
^
drivers/leds/leds.h: In function ‘led_set_blink’:
drivers/leds/leds.h:35:17: error: ‘struct led_classdev’ has no member named ‘blink_set_3g’
led_cdev->blink_set_3g(led_cdev, value);
^
LD drivers/misc/built-in.o
LD firmware/built-in.o
scripts/Makefile.build:293: recipe for target ‘drivers/leds/led-core.o’ failed
make[2]: *** [drivers/leds/led-core.o] Error 1
scripts/Makefile.build:544: recipe for target ‘drivers/leds’ failed
make[1]: *** [drivers/leds] Error 2
make[1]: *** Waiting for unfinished jobs…
I’m crosscomiling from virtual machine on Jessie 8.1.
Do you know how to solve problem? Do not apply patch for leds?
@oddstan,
I do recall a similar issue with one of the patch bundles I created, but it was resolved a while ago. Did you use the latest version of the patches posted here ?
The fact that you are cross compiling from a VM seems irrelevant for this issue IMHO.
For your convenience I posted the patched files in drivers/leds here. They are applied to kernel version 4.9.77. Maybe this helps to figure out where the patch is going wrong.
Please note that sometimes small differences exist between versions that cause a patch hunk to (unintentionally) fail (e.g. an added comment).
What kernel version are you compiling? I will try to replicate…
@Olivier,
It’s probably an issue with my image which has many security patches and lock-downs. I will take a fresh disk and try it out. Maybe I need to enable telnet on the initial image as ssh has many gotchas and provide instructions to disable it afterwards…
Also a good time to update the image with the latest Debian packages and patches. A number of my patches got accepted by the Debian team so it will be much cleaner…
Thanks for letting me know. Sorry for the time lost
Ewald
I’ve used patches from your post in which you replied me:
The kernel I’ve used was 4.9.33. as i understood from the thread it’s most stable for MBL.
I just added some services to the config you’ve provided (ext4 encryption, nfs4 +pnfs).
At the moment on NAS I have:
root@NAS:~# cat /proc/version
Linux version 4.1.35-25102016 (symo@symo-ubuntu) (gcc version 5.3.0 (crosstool-N G crosstool-ng-1.22.0-134-ge1d494a) ) #1 Tue Oct 25 16:48:41 +03 2016
root@NAS:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 8.6 (jessie)
Release: 8.6
Codename: jessie
@oddstan,
Looks excellent, great work.
4.9.33, 4.9.40, 4.9.44 are all very stable with the exception of Jumbo frames. I would stay away from anything above that up until 4.9.77, for which i have posted updated patches. I fixed a few things but most of the credit really goes to LEDE/openWRT team who (back)ported some of the source code from 4.15.x.
The on thing you may try is to use of of my pre-compiled kernels e.g. 4.9.44 and see if you experience a speed increase in SMB/Samba and networking in general. I rewrote most of the network driver, borrowing code from more recent broadcom drivers, but also implementing more advanced capabilities of the APM chipset.
This kernel is also able to run 9000 and 4080 byte Jumbo frames.
@Olivier,
Not many folks working on old (powerPC) hardware these days. Lots of this maintenance has moved to embedded Linux OS’s, like OpenWRT or OpenEmbedded. Most issues were system.d and timing/race condition related as in recent Debian Jessie most startup code is being run in parallel which with older/resource limited HW can create some unique issues.
Let us know how things go.
Ewald
OpenWrt\LEDE now using kernel 4.14 for My Book Live. Why don’t you conribute your awesome patches there? @Ewald
How to apply fix_64k_pages.patch and sata_mbl.patch to LEDE?
@Ewald Keep up the good work. I’m glad this trio is still supported with the latest kernel instead of being stuck on a crappy 2.6.32 kernel forever. I don’t plan on retiring them.
After suffering a devastating house fire, the 1TB, 2TB, and 3TB MyBook Lives were at the other end of the house. So, after I clean the soot off of them, I can start recompiling Gentoo back onto them.
I assume you are referring to Debian Stretch. The problem is that PowerPC is not longer supported in Debian 9.x aka Stretch. In fact, it’s no longer actively supported in Jessie either (no or very limited bug fixes, no security patches). That means I’ve had my hands full to keep Debian 8 running properly e.g. back-porting some security patches. I make my current Debian 8.11 version available, which has a whole series of extra packages like rsync 3.1.3 modified for cyropto acceleration, newer versions of Samba, minidlna, libkcapi (user space API for kernel crypto acceleration, modified to support extra kernel PowerPC crypto functions).
In addition, since we already have upgraded gcc, kernel, Samba, rsync (and fixed udev rules), I am not expecting lots of benefits from Stretch. Right now, I am working on:
kernels beyond 4.9.77: somehow the more recent 4.9 kernels like 4.9.109 have stability problems (freezes) and performance issues with disk writes and I can not find what patch since 4.9.77 is causing it. Trying to get 4.9.99 to work.
kernel 4.14.x: starting to debug 4.14.48. The Lede team has done a tremendous job fixing many of the issues, but unfortunately it does not pass NAS torture tests. And beside unstable it’s also 30% slower, so there is still a lot of work…
Linux Kernel 4.9.99 (and 4.9.119)
Kernel 4.9.99 pre-compiled here and as patch set to compile your own here
Stress tested for 96 hours (NAS functions, jumbo packets, rsync, disk copies, NFS, kernel/user space crypto, SAMBA/CIFS)
For best stability and performance, remove custom sysctl settings. It’s difficult to beat the kernel defaults.
This will be the last 4.9 kernel (unless some unexpected issue shows up). Focus has moved to 4.14.x.
Open issues: MSI not working, occasionally netdev watchdog will reset the network when the network hangs (problem identified, but fix is complex to code)
CIFS (Samba) reads @ 115MB/s, writes @ 82MB/s (1GB file, MTU 4080, Windows 10 target, protocol version 3), disk sequential reads @175MB/s, writes @ 106 MB/s
Kernels above 4.9.99 might be unstable and perform less good. At least 4.9.109 has significant issues, but the root cause has not been identified.
UPDATE: kernel 4.9.119 has all the issues identified and fixed. Feel free to use this version with the same patches,
@Ewald: xcellent performance of your Debian 8.11 system. I wanted to retire the old system because of the security flaws, but this makes it worthwhile to look for a second one on eBay.
How did you optimize for this samba performance, only patches? I noticed a lot of fine-tuning in your smb.conf, but mainly commented out. I would like to adapt this to other devices, which run linux on ARM, for example, so any hint is welcome.
Regarding testing your system: I got rid of some last traces of your config with grep -rnw '/etc' -e 'ETNas', and editing them out. On the other hand, I rebuilt the samba shares from your overall_share file because they make a lot of sense on a one-disk NAS, also important for the minidlna configuration.
Overall, a real good starting point. A couple of questions:
When doing a apt install, the system complains with /sbin/ldconfig.real: /usr/lib/libkcapi.so.hmac is not an elf file - it has the wrong magic bytes at the start. - I assume this is safe to ignore?
Did you switch off nmbd for performance/memory reasons?
Same question for the users using dash instead of bash; only root uses bash in your configuration? Dash is hard to work with outside of scripts, it doesn’t even have tab completion.
DNS seems to be not working. I can neither ping the device by name from another computer on the internal network, nor can I ping out by name (again, internal only). IP ping works, as expected. Any insights on this?
EDIT: After some more peeking inside, I found out that my DNS setup relies on mDNS/Avahi, which your image doesn’t have. I kind of assumed it should be there because the stock firmware also has it. So, either install apt-get install avahi-daemon avahi-discover libnss-mdns or live without it. I chose the latter.
