blog: WIP: draft the nixos-kernel-hacking article
This commit is contained in:
245
content/blog/DRAFT-2024-11-15-nixos-kernel-hacking.md
Normal file
245
content/blog/DRAFT-2024-11-15-nixos-kernel-hacking.md
Normal file
@@ -0,0 +1,245 @@
|
||||
+++
|
||||
title = "Painless kernel hacking + device bringup on NixOS"
|
||||
description = "patching kernels and tweaking device trees *without* lengthy rebuilds"
|
||||
# date = 2024-11-15
|
||||
# to publish: uncomment the `date` field and remove the `DRAFT` prefix
|
||||
+++
|
||||
|
||||
one of my NixOS devices is a [PinePhone Pro].
|
||||
at time of writing, it does boot to a working display/touchscreen and even WiFi on stock NixOS,
|
||||
however other basic features like audio and battery readouts aren't available.
|
||||
actually, these features *are* available on distros like [postmarketOS] which ship kernel forks
|
||||
specifically for this. but maintaining a fork is at least a bit of work, and NixOS gives you lots
|
||||
_better_ ways to tweak your kernel than simply building the whole thing from a different source tree.
|
||||
|
||||
## Kernel Pain Points
|
||||
|
||||
skip ahead if you're already familiar with the NixOS kernel options.
|
||||
the straightforward way to ship custom kernels on NixOS is something like:
|
||||
|
||||
```nix
|
||||
{ pkgs, ... }:
|
||||
{
|
||||
boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.buildLinux {
|
||||
src = fetchgit {
|
||||
# ...
|
||||
};
|
||||
# ...
|
||||
});
|
||||
}
|
||||
```
|
||||
or if your kernel patches are simple enough, then track the NixOS kernel and patch it instead:
|
||||
```nix
|
||||
{ ... }:
|
||||
{
|
||||
boot.kernelPatches = [
|
||||
{
|
||||
name = "my-kernel-patch";
|
||||
patch = ./my-kernel-patch.patch;
|
||||
}
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
well the huge downside to this is that tweaking just a single line in `my-kernel-patch.patch` forces
|
||||
a complete rebuild of your kernel: your iteration cycle time might well be 30 minutes with this approach.
|
||||
|
||||
|
||||
## Faster Kernel Iterations
|
||||
|
||||
until we get something like [dynamic derivations], the path out of costly iteration is to ship whichever tweaks you're pursuing via some other mechanism than as part of the top-level kernel derivation.
|
||||
the kernel is fairly pluggable; consider:
|
||||
|
||||
1. device tree files can be loaded at runtime.
|
||||
2. kernel modules can be loaded at runtime.
|
||||
|
||||
with the right config, either of these things can be freely modified without forcing a kernel build.
|
||||
|
||||
## Shipping Device Tree Patches
|
||||
|
||||
Linux uses [device tree] files to determine which peripherals are available on your
|
||||
device and how to use them. it's more relevant for embedded devices than for traditional x86 PCs.
|
||||
|
||||
the typical boot flow for an embedded device is that the bootloader (U-Boot) knows the name of
|
||||
the device it's running on (perhaps because it was built _specifically_ for that device) and communicates
|
||||
this to the kernel when it hands over control of the device. the name of this device tree might look something like "pine64,pinephone-pro".
|
||||
|
||||
device trees are composable by design; the spec already defines the concept of an "overlay",
|
||||
such that the actual device tree to apply to a device should be the union of whatever's
|
||||
defined in the kernel source tree plus any overlays that apply to the same device.
|
||||
NixOS makes this feature available via the [`hardware.deviceTree.overlays`] option.
|
||||
|
||||
one issue with the PinePhone Pro on stock Linux is that the volume-down button doesn't work.
|
||||
but it works in the pine64 kernel, because they patched the device tree file like so:
|
||||
```diff
|
||||
--- arch/arm64/boot/dts/rockchip/rk3399-pinephone-pro.dts
|
||||
+++ arch/arm64/boot/dts/rockchip/rk3399-pinephone-pro.dts
|
||||
@@ -48,11 +48,11 @@ / {
|
||||
adc-keys {
|
||||
compatible = "adc-keys";
|
||||
io-channels = <&saradc 1>;
|
||||
io-channel-names = "buttons";
|
||||
keyup-threshold-microvolt = <1600000>;
|
||||
poll-interval = <100>;
|
||||
button-down {
|
||||
label = "Volume Down";
|
||||
linux,code = <KEY_VOLUMEDOWN>;
|
||||
- press-threshold-microvolt = <600000>;
|
||||
+ press-threshold-microvolt = <400000>;
|
||||
};
|
||||
```
|
||||
|
||||
this patch can alternately be represented as a [Device Tree Overlay] (DTO):
|
||||
|
||||
```dts
|
||||
// file: rk3399-pinephone-pro-lradc-fix.dtso
|
||||
|
||||
// metadata for the device tree compiler
|
||||
/dts-v1/;
|
||||
/plugin/;
|
||||
|
||||
// instruct the system to only apply this overlay
|
||||
// to PinePhone Pro, and not any other devices.
|
||||
/ {
|
||||
compatible = "pine64,pinephone-pro";
|
||||
};
|
||||
|
||||
// address the parent node we want to patch,
|
||||
// and then inject a new property value.
|
||||
// this overrides `press-threshold-microvolt`
|
||||
// while leaving all other properties unchanged.
|
||||
&{/adc-keys/button-down} {
|
||||
press-threshold-microvolt = <400000>;
|
||||
};
|
||||
```
|
||||
|
||||
and then we can ship it in NixOS like so:
|
||||
|
||||
```nix
|
||||
{ ... }:
|
||||
{
|
||||
hardware.deviceTree.overlays = [
|
||||
{
|
||||
name = "rk3399-pinephone-pro-lradc-fix";
|
||||
dtsFile = ./rk3399-pinephone-pro-lradc-fix.dtso;
|
||||
}
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
after adding the above to a stock NixOS config, and deploying
|
||||
to a PinePhone Pro, the volume down button should now be fixed!
|
||||
|
||||
## Shipping Custom Kernel Modules
|
||||
|
||||
one of the less trivial patches yet to be mainlined is support for
|
||||
battery monitoring on the PinePhone Pro. this feature is supported
|
||||
in downstream kernels by shipping two new kernel modules:
|
||||
`rk818_battery` and `rk818_charger`.
|
||||
|
||||
these modules are device drivers: they detect a `rk818-battery`
|
||||
device somewhere on the system, and then run code in response
|
||||
to that (namely, enable some voltage regulators, and create sysfs
|
||||
nodes to let userspace interact with the device).
|
||||
|
||||
code wise, the typical driver is a single .c file with few if any
|
||||
direct dependencies on other drivers. most driver modules are
|
||||
standalone in the same sense that each nix package is standalone.
|
||||
|
||||
then, we can ship new kernel modules by:
|
||||
- defining a nix derivation that compiles the kernel module.
|
||||
- telling nix to deploy that on the kernel's module search path.
|
||||
- instructing the kernel to load our module.
|
||||
- shipping a device tree overlay to associate the relevant devices
|
||||
with the driver our module provides.
|
||||
|
||||
### Building a Kernel Module
|
||||
|
||||
TODO
|
||||
|
||||
### Deploying a Kernel Module
|
||||
|
||||
TODO
|
||||
|
||||
### Device Tree + Module Integration
|
||||
|
||||
TODO
|
||||
|
||||
## Patching an Upstream Kernel Module
|
||||
|
||||
this all works, however the PinePhone Pro battery integration
|
||||
requires not just a new kernel module, but also to patch an existing
|
||||
kernel module (`rk8xx-i2c`). since the kernel module is defined in-tree, the natural
|
||||
way to do that would force a kernel rebuild every time we adjust the patches.
|
||||
|
||||
we can build our patched module out-of-tree using the same method
|
||||
as we build a wholly new module out-of-tree (writing the derivation
|
||||
as some `patches` atop `src = linux-latest`, or just copying the entire
|
||||
module source into our repo instead of tracking patches). then we just
|
||||
configure the kernel to prefer our module over its in-tree module.
|
||||
|
||||
a first approach might be to configure the kernel with `CONFIG_MFD_RK8XX_I2C=n`,
|
||||
then ship our module as above. this works:
|
||||
|
||||
```nix
|
||||
{ lib, ... }:
|
||||
{
|
||||
boot.kernelPatches = [
|
||||
{
|
||||
name = "rk8xx-i2c-out-of-tree";
|
||||
patch = null;
|
||||
extraStructuredConfig = with lib.kernel; {
|
||||
MFD_RK8XX_I2C = no;
|
||||
};
|
||||
}
|
||||
];
|
||||
|
||||
}
|
||||
```
|
||||
|
||||
this will result in _one_ kernel rebuild, and then you can freely
|
||||
edit your out-of-tree `rk8xx-i2c` module without costly rebuilds.
|
||||
if you find yourself patching a lot of modules, then it may be
|
||||
preferable to simply build _all_ in-tree modules as dynamic modules,
|
||||
and configure out-of-tree modules to take precedence over the in-tree ones.
|
||||
remove the `MFD_RK8XX_I2C = no` patch and replace it with this:
|
||||
|
||||
```nix
|
||||
{ ... }:
|
||||
{
|
||||
nixpkgs.config.hostPlatform.linux-kernel = {
|
||||
# build every module known to the mainline kernel
|
||||
autoModules = true;
|
||||
# and build those as dynamically loaded modules (`=m`), instead of builtins (`=y`).
|
||||
preferBuiltin = false;
|
||||
};
|
||||
|
||||
# default nixos behavior is to error if a kernel module is provided by more
|
||||
# than one package. but we're doing that intentionally, so patch the logic
|
||||
# from <nixos/modules/system/boot/kernel.nix> (AKA pkgs.aggregateModules)
|
||||
# to not complain.
|
||||
system.modulesTree = lib.mkForce [(
|
||||
(pkgs.aggregateModules
|
||||
( config.boot.extraModulePackages ++ [ config.boot.kernelPackages.kernel ])
|
||||
).overrideAttrs {
|
||||
# when collisions are ignored, earlier items override the contents of later items
|
||||
ignoreCollisions = true;
|
||||
}
|
||||
)];
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## Conclusion
|
||||
|
||||
and there you have it! if you found anything here useful, then my request to you
|
||||
is to upstream your kernel work so that the next reader of this blog doesn't have
|
||||
to go through the same pain 😛
|
||||
|
||||
|
||||
[PinePhone Pro]: https://en.wikipedia.org/wiki/PinePhone_Pro
|
||||
[postmarketOS]: https://postmarketos.org/
|
||||
[dynamic-derivations]: https://github.com/NixOS/rfcs/pull/92
|
||||
[device tree]: https://docs.kernel.org/devicetree/usage-model.html
|
||||
[`hardware.deviceTree.overlays`]: https://search.nixos.org/options?channel=unstable&show=hardware.deviceTree.overlays&from=0&size=50&sort=relevance&type=packages&query=hardware.deviceTree.overlays
|
||||
[Device Tree Overlay]: TODO
|
||||
Reference in New Issue
Block a user