uninsane/content/blog/DRAFT-2024-11-15-nixos-kernel-hacking.md

+++
title = "Painless kernel hacking + device bringup on NixOS"
description = "patching kernels and tweaking device trees *without* lengthy rebuilds"
# date = 2024-11-15
# to publish: uncomment the `date` field and remove the `DRAFT` prefix
+++

one of my NixOS devices is a [PinePhone Pro].
at time of writing, it does boot to a working display/touchscreen and even WiFi on stock NixOS,
however other basic features like audio and battery readouts aren't available.
actually, these features *are* available on distros like [postmarketOS] which ship kernel forks
specifically for this. but maintaining a fork is at least a bit of work, and NixOS gives you lots
_better_ ways to tweak your kernel than simply building the whole thing from a different source tree.

## Kernel Pain Points

skip ahead if you're already familiar with the NixOS kernel options.
the straightforward way to ship custom kernels on NixOS is something like:

```nix
{ pkgs, ... }:
{
  boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.buildLinux {
    src = fetchgit {
      # ...
    };
    # ...
  });
}
```
or if your kernel patches are simple enough, then track the NixOS kernel and patch it instead:
```nix
{ ... }:
{
  boot.kernelPatches = [
    {
      name = "my-kernel-patch";
      patch = ./my-kernel-patch.patch;
    }
  ];
}
```

well the huge downside to this is that tweaking just a single line in `my-kernel-patch.patch` forces
a complete rebuild of your kernel: your iteration cycle time might well be 30 minutes with this approach.


## Faster Kernel Iterations

until we get something like [dynamic derivations], the path out of costly iteration is to ship whichever tweaks you're pursuing via some other mechanism than as part of the top-level kernel derivation.
the kernel is fairly pluggable; consider:

1. device tree files can be loaded at runtime.
2. kernel modules can be loaded at runtime.

with the right config, either of these things can be freely modified without forcing a kernel build.

## Shipping Device Tree Patches

Linux uses [device tree] files to determine which peripherals are available on your
device and how to use them. it's more relevant for embedded devices than for traditional x86 PCs.

the typical boot flow for an embedded device is that the bootloader (U-Boot) knows the name of
the device it's running on (perhaps because it was built _specifically_ for that device) and communicates
this to the kernel when it hands over control of the device. the name of this device tree might look something like "pine64,pinephone-pro".

device trees are composable by design; the spec already defines the concept of an "overlay",
such that the actual device tree to apply to a device should be the union of whatever's
defined in the kernel source tree plus any overlays that apply to the same device.
NixOS makes this feature available via the [`hardware.deviceTree.overlays`] option.

one issue with the PinePhone Pro on stock Linux is that the volume-down button doesn't work.
but it works in the pine64 kernel, because they patched the device tree file like so:
```diff
--- arch/arm64/boot/dts/rockchip/rk3399-pinephone-pro.dts
+++ arch/arm64/boot/dts/rockchip/rk3399-pinephone-pro.dts
@@ -48,11 +48,11 @@ / {
 	adc-keys {
 		compatible = "adc-keys";
 		io-channels = <&saradc 1>;
 		io-channel-names = "buttons";
 		keyup-threshold-microvolt = <1600000>;
 		poll-interval = <100>;
 		button-down {
 			label = "Volume Down";
 			linux,code = <KEY_VOLUMEDOWN>;
-			press-threshold-microvolt = <600000>;
+			press-threshold-microvolt = <400000>;
 		};
```

this patch can alternately be represented as a [Device Tree Overlay] (DTO):

```dts
// file: rk3399-pinephone-pro-lradc-fix.dtso

// metadata for the device tree compiler
/dts-v1/;
/plugin/;

// instruct the system to only apply this overlay
// to PinePhone Pro, and not any other devices.
/ {
        compatible = "pine64,pinephone-pro";
};

// address the parent node we want to patch,
// and then inject a new property value.
// this overrides `press-threshold-microvolt`
// while leaving all other properties unchanged.
&{/adc-keys/button-down} {
        press-threshold-microvolt = <400000>;
};
```

and then we can ship it in NixOS like so:

```nix
{ ... }:
{
  hardware.deviceTree.overlays = [
    {
      name = "rk3399-pinephone-pro-lradc-fix";
      dtsFile = ./rk3399-pinephone-pro-lradc-fix.dtso;
    }
  ];
}
```

after adding the above to a stock NixOS config, and deploying
to a PinePhone Pro, the volume down button should now be fixed!

## Shipping Custom Kernel Modules

one of the less trivial patches yet to be mainlined is support for
battery monitoring on the PinePhone Pro. this feature is supported
in downstream kernels by shipping two new kernel modules:
`rk818_battery` and `rk818_charger`.

these modules are device drivers: they detect a `rk818-battery`
device somewhere on the system, and then run code in response
to that (namely, enable some voltage regulators, and create sysfs
nodes to let userspace interact with the device).

code wise, the typical driver is a single .c file with few if any
direct dependencies on other drivers. most driver modules are
standalone in the same sense that each nix package is standalone.

then, we can ship new kernel modules by:
- defining a nix derivation that compiles the kernel module.
- telling nix to deploy that on the kernel's module search path.
- instructing the kernel to load our module.
- shipping a device tree overlay to associate the relevant devices
  with the driver our module provides.

### Building a Kernel Module

TODO

### Deploying a Kernel Module

TODO

### Device Tree + Module Integration

TODO

## Patching an Upstream Kernel Module

this all works, however the PinePhone Pro battery integration
requires not just a new kernel module, but also to patch an existing
kernel module (`rk8xx-i2c`). since the kernel module is defined in-tree, the natural
way to do that would force a kernel rebuild every time we adjust the patches.

we can build our patched module out-of-tree using the same method
as we build a wholly new module out-of-tree (writing the derivation
as some `patches` atop `src = linux-latest`, or just copying the entire
module source into our repo instead of tracking patches). then we just
configure the kernel to prefer our module over its in-tree module.

a first approach might be to configure the kernel with `CONFIG_MFD_RK8XX_I2C=n`,
then ship our module as above. this works:

```nix
{ lib, ... }:
{
  boot.kernelPatches = [
    {
      name = "rk8xx-i2c-out-of-tree";
      patch = null;
      extraStructuredConfig = with lib.kernel; {
        MFD_RK8XX_I2C = no;
      };
    }
  ];

}
```

this will result in _one_ kernel rebuild, and then you can freely
edit your out-of-tree `rk8xx-i2c` module without costly rebuilds.
if you find yourself patching a lot of modules, then it may be
preferable to simply build _all_ in-tree modules as dynamic modules,
and configure out-of-tree modules to take precedence over the in-tree ones.
remove the `MFD_RK8XX_I2C = no` patch and replace it with this:

```nix
{ ... }:
{
  nixpkgs.config.hostPlatform.linux-kernel = {
    # build every module known to the mainline kernel
    autoModules = true;
    # and build those as dynamically loaded modules (`=m`), instead of builtins (`=y`).
    preferBuiltin = false;
  };

  # default nixos behavior is to error if a kernel module is provided by more
  # than one package. but we're doing that intentionally, so patch the logic
  # from <nixos/modules/system/boot/kernel.nix> (AKA pkgs.aggregateModules)
  # to not complain.
  system.modulesTree = lib.mkForce [(
    (pkgs.aggregateModules
      ( config.boot.extraModulePackages ++ [ config.boot.kernelPackages.kernel ])
    ).overrideAttrs {
      # when collisions are ignored, earlier items override the contents of later items
      ignoreCollisions = true;
    }
  )];
}
```


## Conclusion

and there you have it! if you found anything here useful, then my request to you
is to upstream your kernel work so that the next reader of this blog doesn't have
to go through the same pain 😛


[PinePhone Pro]: https://en.wikipedia.org/wiki/PinePhone_Pro
[postmarketOS]: https://postmarketos.org/
[dynamic-derivations]: https://github.com/NixOS/rfcs/pull/92
[device tree]: https://docs.kernel.org/devicetree/usage-model.html
[`hardware.deviceTree.overlays`]: https://search.nixos.org/options?channel=unstable&show=hardware.deviceTree.overlays&from=0&size=50&sort=relevance&type=packages&query=hardware.deviceTree.overlays
[Device Tree Overlay]: TODO