mpv: sane-sysvol: monitor pipewire for changes and relay that to uosc

it's a one-way volume control, but that's a start

.sops.yaml

@@ -3,7 +3,6 @@ keys:
   - &user_lappy_colin age1j2pqnl8j0krdzk6npe93s4nnqrzwx978qrc0u570gzlamqpnje9sc8le2g
   - &user_servo_colin age1z8fauff34cdecr6sjkre260luzxcca05kpcwvhx988d306tpcejsp63znu
   - &user_moby_colin age1zsrsvd7j6l62fjxpfd2qnhqlk8wk4p8r0dtxpe4sdgnh2474095qdu7xj9
-  - &host_crappy age1hl50ufuxnqy0jnk8fqeu4tclh4vte2xn2d59pxff0gun20vsmv5sp78chj
   - &host_desko age1vnw7lnfpdpjn62l3u5nyv5xt2c965k96p98kc43mcnyzpetrts9q54mc9v
   - &host_lappy age1w7mectcjku6x3sd8plm8wkn2qfrhv9n6zhzlf329e2r2uycgke8qkf9dyn
   - &host_servo age1tzlyex2z6t88tg9h82943e39shxhmqeyr7ywhlwpdjmyqsndv3qq27x0rf
@@ -16,7 +15,6 @@ creation_rules:
       - *user_lappy_colin
       - *user_servo_colin
       - *user_moby_colin
-      - *host_crappy
       - *host_desko
       - *host_lappy
       - *host_servo

README.md

@@ -2,8 +2,6 @@
 
 # .❄️≡We|_c0m3 7o m`/ f14k≡❄️.
 
-(er, it's not a flake anymore. welcome to my nix files.)
-
 ## What's Here
 
 this is the top-level repo from which i configure/deploy all my NixOS machines:
@@ -18,12 +16,13 @@ building [hosts/](./hosts/) will require [sops][sops].
 
 you might specifically be interested in these files (elaborated further in #key-points-of-interest):
 - ~~[`sxmo-utils`](./pkgs/additional/sxmo-utils/default.nix)~~
+  - ~~[example SXMO deployment](./hosts/modules/gui/sxmo/default.nix)~~
   - these files will remain until my config settles down, but i no longer use or maintain SXMO.
 - [my implementation of impermanence](./modules/persist/default.nix)
 - my way of deploying dotfiles/configuring programs per-user:
   - [modules/fs/](./modules/fs/default.nix)
   - [modules/programs/](./modules/programs/default.nix)
-  - [modules/users/](./modules/users/default.nix)
+  - [modules/users.nix](./modules/users.nix)
 
 [nixpkgs]: https://github.com/NixOS/nixpkgs
 [sops]: https://github.com/Mic92/sops-nix
@@ -31,7 +30,11 @@ you might specifically be interested in these files (elaborated further in #key-
 
 ## Using This Repo In Your Own Config
 
-follow the instructions [here][NUR] to access my packages through the Nix User Repositories.
+this should be a pretty "standard" flake. just reference it, and import either
+- `nixosModules.sane` (for the modules)
+- `overlays.pkgs` (for the packages)
+
+or follow the instructions [here][NUR] to use it via the Nix User Repositories.
 
 [NUR]: https://nur.nix-community.org/
 
@@ -39,15 +42,19 @@ follow the instructions [here][NUR] to access my packages through the Nix User R
 - `doc/`
   - instructions for tasks i find myself doing semi-occasionally in this repo.
 - `hosts/`
-  - configs which aren't factored with external use in mind.
+  - the bulk of config which isn't factored with external use in mind.
   - that is, if you were to add this repo to a flake.nix for your own use,
     you won't likely be depending on anything in this directory.
 - `integrations/`
-  - code intended for consumption by external tools (e.g. the Nix User Repos).
+  - code intended for consumption by external tools (e.g. the Nix User Repos)
 - `modules/`
-  - config which is gated behind `enable` flags, in similar style to nixpkgs' `nixos/` directory.
-  - if you depend on this repo for anything besides packages, it's most likely for something in this directory.
+  - config which is gated behind `enable` flags, in similar style to nixpkgs'
+    `nixos/` directory.
+  - if you depend on this repo, it's most likely for something in this directory.
+- `nixpatches/`
+  - literally, diffs i apply atop upstream nixpkgs before performing further eval.
 - `overlays/`
+  - exposed via the `overlays` output in `flake.nix`.
   - predominantly a list of `callPackage` directives.
 - `pkgs/`
   - derivations for things not yet packaged in nixpkgs.
@@ -55,12 +62,13 @@ follow the instructions [here][NUR] to access my packages through the Nix User R
   - inline code for wholly custom packages (e.g. `pkgs/additional/sane-scripts/` for CLI tools
     that are highly specific to my setup).
 - `scripts/`
-  - scripts which aren't reachable on a deployed system, but may aid manual deployments.
+  - scripts which aren't reachable on a deployed system, but may aid manual deployments
 - `secrets/`
   - encrypted keys, API tokens, anything which one or more of my machines needs
     read access to but shouldn't be world-readable.
-  - not much to see here.
+  - not much to see here
 - `templates/`
+  - exposed via the `templates` output in `flake.nix`.
   - used to instantiate short-lived environments.
   - used to auto-fill the boiler-plate portions of new packages.
 
@@ -101,10 +109,9 @@ i.e. you might find value in using these in your own config:
     - `sane.programs.firefox.sandbox.whitelistWayland = true;  # allow it to render a wayland window`
     - `sane.programs.firefox.sandbox.extraHomePaths = [ "Downloads" ];  # allow it read/write access to ~/Downloads`
     - integrated with `fs` and `persist` modules so that programs' config files and persisted data stores are linked into the sandbox w/o any extra involvement.
-- `modules/users/`
+- `modules/users.nix`
   - convenience layer atop the above modules so that you can just write
     `fs.".config/git"` instead of `fs."/home/colin/.config/git"`
-  - per-user services managed by [s6-rc](https://www.skarnet.org/software/s6-rc/)
 
 some things in here could easily find broader use. if you would find benefit in
 them being factored out of my config, message me and we could work to make that happen.

TODO.md

@@ -1,36 +1,17 @@
 ## BUGS
-- `rmDbusServices` may break sandboxing
-  - e.g. if the package ships a systemd unit which references $out, then make-sandboxed won't properly update that unit.
-  - `rmDbusServicesInPlace` is not affected
-- when moby wlan is explicitly set down (via ip link set wlan0 down), /var/lib/trust-dns/dhcp-configs doesn't get reset
-  - `ip monitor` can detect those manual link state changes (NM-dispatcher it seems cannot)
-  - or try dnsmasq?
-- trust-dns can't resolve `abs.twimg.com`
-- trust-dns can't resolve `social.kernel.org`
-- sandbox: link cache means that if i update ~/.config/... files inline, sandboxed programs still see the old version
-- mpv: continues to play past the end of some audio files
-- mpv: audiocast has mpv sending its output to the builtin speakers unless manually changed
-- mpv: no way to exit fullscreen video on moby
-  - uosc hides controls on FS, and touch doesn't support unhiding
-- `ssh` access doesn't grant same linux capabilities as login
-- syshud (volume overlay): when casting with `blast`, syshud doesn't react to volume changes
-- moby: kaslr is effectively disabled
-  - `dmesg | grep "KASLR disabled due to lack of seed"`
-  - fix by adding `kaslrseed` to uboot script before `booti`
-    - <https://github.com/armbian/build/pull/4352>
-    - not sure how that's supposed to work with tow-boot; maybe i should just update tow-boot
-- moby: bpf is effectively disabled?
-  - `dmesg | grep 'systemd[1]: bpf-lsm: Failed to load BPF object: No such process'`
-  - `dmesg | grep 'hid_bpf: error while preloading HID BPF dispatcher: -22'`
-- `s6` is not re-entrant
-  - so if the desktop crashes, the login process from `unl0kr` fails to re-launch the GUI
+- Signal restart loop drains battery
+  - decrease s6 restart time?
+- mpv `player-mode=pseudo-gui` swallows all loggin
+- ringer (i.e. dino incoming call) doesn't prevent moby from sleeping
+- sway mouse/kb hotplug doesn't work
+- `nix` operations from lappy hang when `desko` is unreachable
+  - could at least direct the cache to `http://desko-hn:5001`
 
 ## REFACTORING:
-- add import checks to my Python nix-shell scripts
+- REMOVE DEPRECATED `crypt` from sftpgo_auth_hook
 - consolidate ~/dev and ~/ref
   - ~/dev becomes a link to ~/ref/cat/mine
 - fold hosts/common/home/ssh.nix -> hosts/common/users/colin.nix
-- don't hardcode IP addresses so much in servo
 
 ### sops/secrets
 - rework secrets to leverage `sane.fs`
@@ -44,23 +25,13 @@
 
 ### upstreaming
 - add updateScripts to all my packages in nixpkgs
+- REVIEW/integrate jellyfin dataDir config: <https://github.com/NixOS/nixpkgs/pull/233617>
 
 #### upstreaming to non-nixpkgs repos
 - gtk: build schemas even on cross compilation: <https://github.com/NixOS/nixpkgs/pull/247844>
 
 
 ## IMPROVEMENTS:
-- systemd/journalctl: use a less shit pager
-  - there's an env var for it: SYSTEMD_PAGER? and a flag for journalctl
-- kernels: ship the same kernel on every machine
-  - then i can tune the kernels for hardening, without duplicating that work 4 times
-- zfs: replace this with something which doesn't require a custom kernel build
-- mpv: add media looping controls (e.g. loop song, loop playlist)
-- curlftpfs: replace with something better
-  - safer (rust? actively maintained? sandboxable?)
-  - handles spaces/symbols in filenames
-  - has better multi-stream perf (e.g. `sane-sync-music` should be able to copy N items in parallel)
-
 ### security/resilience
 - validate duplicity backups!
 - encrypt more ~ dirs (~/archives, ~/records, ..?)
@@ -68,9 +39,6 @@
 - /mnt/desko/home, etc, shouldn't include secrets (~/private)
   - 95% of its use is for remote media access and stuff which isn't in VCS (~/records)
 - port all sane.programs to be sandboxed
-  - sandbox `curlftpfs`
-  - sandbox `nix`
-  - sandbox `sshfs-fuse`
   - enforce that all `environment.packages` has a sandbox profile (or explicitly opts out)
   - revisit "non-sandboxable" apps and check that i'm not actually just missing mountpoints
     - LL_FS_RW=/ isn't enough -- need all mount points like `=/:/proc:/sys:...`.
@@ -79,18 +47,24 @@
   - lock down dbus calls within the sandbox
     - otherwise anyone can `systemd-run --user ...` to potentially escape a sandbox
     - <https://github.com/flatpak/xdg-dbus-proxy>
-  - port sanebox to a compiled language (hare?)
+  - remove `.ssh` access from Firefox!
+    - limit access to `~/knowledge/secrets` through an agent that requires GUI approval, so a firefox exploit can't steal all my logins
+  - port sane-sandboxed to a compiled language (hare?)
     - it adds like 50-70ms launch time _on my laptop_. i'd hate to know how much that is on the pinephone.
+  - remove /run/wrappers from the sandbox path
+    - they're mostly useless when using no-new-privs, just an opportunity to forget to specify deps
 - make dconf stuff less monolithic
   - i.e. per-app dconf profiles for those which need it. possible static config.
-  - flatpak/spectrum has some stuff to proxy dconf per-app
+- canaries for important services
+  - e.g. daily email checks; daily backup checks
+  - integrate `nix check` into Gitea actions?
 
 ### user experience
-- rofi: sort items case-insensitively
+- xdg-desktop-portal shouldn't kill children on exit
+  - *maybe* a job for `setsid -f`?
 - replace starship prompt with something more efficient
   - watch `forkstat`: it does way too much
-- cleanup waybar/nwg-panel so that it's not invoking playerctl every 2 seconds
-  - nwg-panel: doesn't know that virtual-desktop 10/TV exists
+- cleanup waybar so that it's not invoking playerctl every 2 seconds
 - install apps:
   - display QR codes for WiFi endpoints: <https://linuxphoneapps.org/apps/noappid.wisperwind.wifi2qr/>
   - shopping list (not in nixpkgs): <https://linuxphoneapps.org/apps/ro.hume.cosmin.shoppinglist/>
@@ -98,9 +72,8 @@
   - offline docs viewer (gtk): <https://github.com/workbenchdev/Biblioteca>
   - some type of games manager/launcher
     - Gnome Highscore (retro games)?: <https://gitlab.gnome.org/World/highscore>
-  - better maps for mobile (Osmin (QtQuick)? Pure Maps (Qt/Kirigami)?)
+  - better maps for mobile (Osmin (QtQuick)? Pure Maps (Qt/Kirigami)? Gnome Maps is improved in 45)
   - note-taking app: <https://linuxphoneapps.org/categories/note-taking/>
-    - Folio is nice, uses standard markdown, though it only supports flat repos
   - OSK overlay specifically for mobile gaming
     - i.e. mock joysticks, for use with SuperTux and SuperTuxKart
 - install mobile-friendly games:
@@ -111,8 +84,6 @@
   - numberlink (generic name for Flow Free). not packaged in Nix
   - Neverball (https://neverball.org/screenshots.php). nix: as `neverball`
   - blurble (https://linuxphoneapps.org/games/app.drey.blurble/). nix: not as of 2024-02-05
-  - Trivia Quiz (https://linuxphoneapps.org/games/io.github.nokse22.trivia-quiz/)
-- sane-sync-music: remove empty dirs
 
 #### moby
 - fix cpuidle (gets better power consumption): <https://xnux.eu/log/077.html>
@@ -121,12 +92,13 @@
   - don't show MPRIS if no players detected
     - this is a problem of playerctld, i guess
   - add option to change audio output
+  - fix colors (red alert) to match overall theme
 - moby: tune GPS
-  - fix iio-sensor-proxy magnetometer scaling
+  - run only geoclue, and not gpsd, to save power?
   - tune QGPS setting in eg25-control, for less jitter?
+  - direct mepo to prefer gpsd, with fallback to geoclue, for better accuracy?
   - configure geoclue to do some smoothing?
-  - manually do smoothing, as some layer between mepo and geoclue?
-- moby: port `freshen-agps` timer service to s6 (maybe i want some `s6-cron` or something)
+  - manually do smoothing, as some layer between mepo and geoclue/gpsd?
 - moby: show battery state on ssh login
 - moby: improve gPodder launch time
 - moby: theme GTK apps (i.e. non-adwaita styles)
@@ -148,15 +120,13 @@
 - have xdg-open parse `<repo:...> URIs (or adjust them so that it _can_ parse)
 - sane-bt-search: show details like 5.1 vs stereo, h264 vs h265
   - maybe just color these "keywords" in all search results?
-- transmission: apply `sane-tag-media` path fix in `torrent-done` script
-  - many .mkv files do appear to be tagged: i'd just need to add support in my own tooling
 - uninsane.org: make URLs relative to allow local use (and as offline homepage)
 - email: fix so that local mail doesn't go to junk
   - git sendmail flow adds the DKIM signatures, but gets delivered locally w/o having the sig checked, so goes into Junk
   - could change junk filter from "no DKIM success" to explicit "DKIM failed"
-  - add an auto-reply address (e.g. `reply-test@uninsane.org`) which reflects all incoming mail; use this (or a friend running this) for liveness checks
 
 ### perf
+- debug nixos-rebuild times
 - add `pkgs.impure-cached.<foo>` package set to build things with ccache enabled
   - every package here can be auto-generated, and marked with some env var so that it doesn't pollute the pure package set
   - would be super handy for package prototyping!

default.nix

@@ -1,5 +1,9 @@
-{ ... }@args:
-let
-  sane-nix-files = import ./pkgs/additional/sane-nix-files { };
-in
-  import "${sane-nix-files}/impure.nix" args
+# limited, non-flake interface to this repo.
+# this file exposes the same view into `pkgs` which the flake would see when evaluated.
+#
+# the primary purpose of this file is so i can run `updateScript`s which expect
+# the root to be `default.nix`
+{ pkgs ? import <nixpkgs> {} }:
+pkgs.appendOverlays [
+  (import ./overlays/all.nix)
+]

doc/adding-a-host.md

@@ -1,25 +0,0 @@
-to add a host:
-- create the new nix targets
-  - hosts/by-name/HOST
-  - let the toplevel (flake.nix) know about HOST
-- build and flash an image
-- optionally expand the rootfs
-  - `cfdisk /dev/sda2` -> resize partition
-  - `mount /dev/sda2 boot`
-  - `btrfs filesystem resize max root`
-- setup required persistent directories
-  - `mkdir -p root/persist/private`
-  - `gocryptfs -init root/persist/private`
-  - then boot the device, and for every dangling symlink in ~/.local/share, ~/.cache, do `mkdir -p` on it
-- setup host ssh
-  - `mkdir -p root/persist/plaintext/etc/ssh/host_keys`
-  - boot the machine and let it create its own ssh keys
-  - add the pubkey to `hosts/common/hosts.nix`
-- setup user ssh
-  - `ssh-keygen`. don't enter any password; it's stored in a password-encrypted fs.
-  - add the pubkey to `hosts/common/hosts.nix`
-- allow the new host to view secrets
-  - instructions in hosts/common/secrets.nix
-  - run `ssh-to-age` on user/host pubkeys
-  - add age key to .sops.yaml
-  - update encrypted secrets: `sops updatekeys path/to/secret.yaml`

doc/recovery.md

@@ -1,12 +0,0 @@
-## deploying to SD card
-- build a toplevel config: `nix build '.#hostSystems.moby'`
-- mount a system:
-  - `mkdir -p root/{nix,boot}`
-  - `mount /dev/sdX1 root/boot`
-  - `mount /dev/sdX2 root/nix`
-- copy the config:
-  - `sudo nix copy --no-check-sigs --to root/ $(readlink result)`
-    - nix will copy stuff to `root/nix/store`
-- install the boot files:
-  - `sudo /nix/store/sbwpwngjlgw4f736ay9hgi69pj3fdwk5-extlinux-conf-builder.sh -d ./root/boot -t 5 -c $(readlink ./result)`
-    - extlinux-conf-builder can be found in `/run/current-system/bin/switch-to-configuration`

flake.lock

@@ -0,0 +1,330 @@
+{
+  "nodes": {
+    "flake-compat": {
+      "locked": {
+        "lastModified": 1688025799,
+        "narHash": "sha256-ktpB4dRtnksm9F5WawoIkEneh1nrEvuxb5lJFt1iOyw=",
+        "owner": "nix-community",
+        "repo": "flake-compat",
+        "rev": "8bf105319d44f6b9f0d764efa4fdef9f1cc9ba1c",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "flake-compat",
+        "type": "github"
+      }
+    },
+    "flake-parts": {
+      "inputs": {
+        "nixpkgs-lib": [
+          "nixpkgs-wayland",
+          "nix-eval-jobs",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1701473968,
+        "narHash": "sha256-YcVE5emp1qQ8ieHUnxt1wCZCC3ZfAS+SRRWZ2TMda7E=",
+        "owner": "hercules-ci",
+        "repo": "flake-parts",
+        "rev": "34fed993f1674c8d06d58b37ce1e0fe5eebcb9f5",
+        "type": "github"
+      },
+      "original": {
+        "owner": "hercules-ci",
+        "repo": "flake-parts",
+        "type": "github"
+      }
+    },
+    "flake-utils": {
+      "inputs": {
+        "systems": "systems"
+      },
+      "locked": {
+        "lastModified": 1710146030,
+        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
+        "type": "github"
+      },
+      "original": {
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "type": "github"
+      }
+    },
+    "lib-aggregate": {
+      "inputs": {
+        "flake-utils": "flake-utils",
+        "nixpkgs-lib": "nixpkgs-lib"
+      },
+      "locked": {
+        "lastModified": 1711886936,
+        "narHash": "sha256-D2WENp9GuaCostvNcQ7vElekk0V5cuMdnFZ7NfRhVrQ=",
+        "owner": "nix-community",
+        "repo": "lib-aggregate",
+        "rev": "9c06929b83e57c18d125f1105ba6a423f24083d2",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "lib-aggregate",
+        "type": "github"
+      }
+    },
+    "mobile-nixos": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1694749521,
+        "narHash": "sha256-MiVokKlpcJmfoGuWAMeW1En7gZ5hk0rCQArYm6P9XCc=",
+        "owner": "nixos",
+        "repo": "mobile-nixos",
+        "rev": "d25d3b87e7f300d8066e31d792337d9cd7ecd23b",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nixos",
+        "ref": "d25d3b87e7f300d8066e31d792337d9cd7ecd23b",
+        "repo": "mobile-nixos",
+        "type": "github"
+      }
+    },
+    "nix-eval-jobs": {
+      "inputs": {
+        "flake-parts": "flake-parts",
+        "nix-github-actions": "nix-github-actions",
+        "nixpkgs": "nixpkgs",
+        "treefmt-nix": "treefmt-nix"
+      },
+      "locked": {
+        "lastModified": 1705242886,
+        "narHash": "sha256-TLj334vRwFtSym3m+NnKcNCnKKPNoTC/TDZL40vmOso=",
+        "owner": "nix-community",
+        "repo": "nix-eval-jobs",
+        "rev": "6b03a93296faf174b97546fd573c8b379f523a8d",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "nix-eval-jobs",
+        "type": "github"
+      }
+    },
+    "nix-github-actions": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-wayland",
+          "nix-eval-jobs",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1701208414,
+        "narHash": "sha256-xrQ0FyhwTZK6BwKhahIkUVZhMNk21IEI1nUcWSONtpo=",
+        "owner": "nix-community",
+        "repo": "nix-github-actions",
+        "rev": "93e39cc1a087d65bcf7a132e75a650c44dd2b734",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "nix-github-actions",
+        "type": "github"
+      }
+    },
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1703134684,
+        "narHash": "sha256-SQmng1EnBFLzS7WSRyPM9HgmZP2kLJcPAz+Ug/nug6o=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "d6863cbcbbb80e71cecfc03356db1cda38919523",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "nixpkgs-unstable",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-lib": {
+      "locked": {
+        "lastModified": 1711846064,
+        "narHash": "sha256-cqfX0QJNEnge3a77VnytM0Q6QZZ0DziFXt6tSCV8ZSc=",
+        "owner": "nix-community",
+        "repo": "nixpkgs.lib",
+        "rev": "90b1a963ff84dc532db92f678296ff2499a60a87",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "nixpkgs.lib",
+        "type": "github"
+      }
+    },
+    "nixpkgs-next-unpatched": {
+      "locked": {
+        "lastModified": 1712383280,
+        "narHash": "sha256-YL8miM11o/jMqOwt5DsdyhPgh/JgCl1kOIzvX7ukniY=",
+        "owner": "nixos",
+        "repo": "nixpkgs",
+        "rev": "7c74352f2f7eca1925729f5c9c80cb89df8e74a2",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nixos",
+        "ref": "staging-next",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-stable": {
+      "locked": {
+        "lastModified": 1711819797,
+        "narHash": "sha256-tNeB6emxj74Y6ctwmsjtMlzUMn458sBmwnD35U5KIM4=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "2b4e3ca0091049c6fbb4908c66b05b77eaef9f0c",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "release-23.11",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-unpatched": {
+      "locked": {
+        "lastModified": 1712398506,
+        "narHash": "sha256-oopwPeBKBXQEw2BlyK2jEs2farZ5uMjAZU7H4FpGuGE=",
+        "owner": "nixos",
+        "repo": "nixpkgs",
+        "rev": "c58702222e0a29fd01cc42d70737d699995f6389",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nixos",
+        "ref": "master",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-wayland": {
+      "inputs": {
+        "flake-compat": "flake-compat",
+        "lib-aggregate": "lib-aggregate",
+        "nix-eval-jobs": "nix-eval-jobs",
+        "nixpkgs": [
+          "nixpkgs-unpatched"
+        ]
+      },
+      "locked": {
+        "lastModified": 1712237761,
+        "narHash": "sha256-NoMBBCADTms3yx5BL+sbc7vfDivNiYULO6t9GBAsPt0=",
+        "owner": "nix-community",
+        "repo": "nixpkgs-wayland",
+        "rev": "9b77653338f52da4b498abdf4835efb6ff6e453e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "nixpkgs-wayland",
+        "type": "github"
+      }
+    },
+    "root": {
+      "inputs": {
+        "mobile-nixos": "mobile-nixos",
+        "nixpkgs-next-unpatched": "nixpkgs-next-unpatched",
+        "nixpkgs-unpatched": "nixpkgs-unpatched",
+        "nixpkgs-wayland": "nixpkgs-wayland",
+        "sops-nix": "sops-nix",
+        "uninsane-dot-org": "uninsane-dot-org"
+      }
+    },
+    "sops-nix": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-unpatched"
+        ],
+        "nixpkgs-stable": "nixpkgs-stable"
+      },
+      "locked": {
+        "lastModified": 1711855048,
+        "narHash": "sha256-HxegAPnQJSC4cbEbF4Iq3YTlFHZKLiNTk8147EbLdGg=",
+        "owner": "Mic92",
+        "repo": "sops-nix",
+        "rev": "99b1e37f9fc0960d064a7862eb7adfb92e64fa10",
+        "type": "github"
+      },
+      "original": {
+        "owner": "Mic92",
+        "repo": "sops-nix",
+        "type": "github"
+      }
+    },
+    "systems": {
+      "locked": {
+        "lastModified": 1681028828,
+        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "owner": "nix-systems",
+        "repo": "default",
+        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-systems",
+        "repo": "default",
+        "type": "github"
+      }
+    },
+    "treefmt-nix": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-wayland",
+          "nix-eval-jobs",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1702979157,
+        "narHash": "sha256-RnFBbLbpqtn4AoJGXKevQMCGhra4h6G2MPcuTSZZQ+g=",
+        "owner": "numtide",
+        "repo": "treefmt-nix",
+        "rev": "2961375283668d867e64129c22af532de8e77734",
+        "type": "github"
+      },
+      "original": {
+        "owner": "numtide",
+        "repo": "treefmt-nix",
+        "type": "github"
+      }
+    },
+    "uninsane-dot-org": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-unpatched"
+        ]
+      },
+      "locked": {
+        "lastModified": 1711371733,
+        "narHash": "sha256-+brjlMyLVnVADY31sN82Ap0IsPE2WZEwHUd94sY6BXI=",
+        "ref": "refs/heads/master",
+        "rev": "b9502e6f190752d327f8cee7fa4b139094bd7c16",
+        "revCount": 237,
+        "type": "git",
+        "url": "https://git.uninsane.org/colin/uninsane"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://git.uninsane.org/colin/uninsane"
+      }
+    }
+  },
+  "root": "root",
+  "version": 7
+}

flake.nix

@@ -0,0 +1,644 @@
+# FLAKE FEEDBACK:
+# - if flake inputs are meant to be human-readable, a human should be able to easily track them down given the URL.
+#   - this is not the case with registry URLs, like `nixpkgs/nixos-22.11`.
+#   - this is marginally the case with schemes like `github:nixos/nixpkgs`.
+#   - given the *existing* `git+https://` scheme, i propose expressing github URLs similarly:
+#     - `github+https://github.com/nixos/nixpkgs/tree/nixos-22.11`
+#     - this would allow for the same optimizations as today's `github:nixos/nixpkgs`, but without obscuring the source.
+#       a code reader could view the source being referenced simply by clicking the https:// portion of that URI.
+# - need some way to apply local patches to inputs.
+#
+#
+# DEVELOPMENT DOCS:
+# - Flake docs: <https://nixos.wiki/wiki/Flakes>
+# - Flake RFC: <https://github.com/tweag/rfcs/blob/flakes/rfcs/0049-flakes.md>
+#   - Discussion: <https://github.com/NixOS/rfcs/pull/49>
+# - <https://serokell.io/blog/practical-nix-flakes>
+#
+#
+# COMMON OPERATIONS:
+# - update a specific flake input:
+#   - `nix flake lock --update-input nixpkgs`
+
+{
+  # XXX: use the `github:` scheme instead of the more readable git+https: because it's *way* more efficient
+  # preferably, i would rewrite the human-readable https URLs to nix-specific github: URLs with a helper,
+  # but `inputs` is required to be a strict attrset: not an expression.
+  inputs = {
+    # branch workflow:
+    # - daily:
+    #   - nixos-unstable cut from master after enough packages have been built in caches.
+    # - every 6 hours:
+    #   - master auto-merged into staging and staging-next
+    #   - staging-next auto-merged into staging.
+    # - manually, approximately once per month:
+    #   - staging-next is cut from staging.
+    #   - staging-next merged into master.
+    #
+    # which branch to source from?
+    # - nixos-unstable: for everyday development; it provides good caching
+    # - master: temporarily if i'm otherwise cherry-picking lots of already-applied patches
+    # - staging-next: if testing stuff that's been PR'd into staging, i.e. base library updates.
+    # - staging: maybe if no staging-next -> master PR has been cut yet?
+    #
+    # <https://github.com/nixos/nixpkgs/tree/nixos-unstable>
+    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-unstable";
+    nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=master";
+    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-staging";
+    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-staging-next";
+    nixpkgs-next-unpatched.url = "github:nixos/nixpkgs?ref=staging-next";
+
+    nixpkgs-wayland = {
+      url = "github:nix-community/nixpkgs-wayland";
+      inputs.nixpkgs.follows = "nixpkgs-unpatched";
+    };
+
+    mobile-nixos = {
+      # <https://github.com/nixos/mobile-nixos>
+      # only used for building disk images, not relevant after deployment
+      # TODO: replace with something else. commit `0f3ac0bef1aea70254a3bae35e3cc2561623f4c1`
+      # replaces the imageBuilder with a "new implementation from celun" and wildly breaks my use.
+      # pinning to d25d3b... is equivalent to holding at 2023-09-15
+      url = "github:nixos/mobile-nixos?ref=d25d3b87e7f300d8066e31d792337d9cd7ecd23b";
+      flake = false;
+    };
+    sops-nix = {
+      # <https://github.com/Mic92/sops-nix>
+      # used to distribute secrets to my hosts
+      url = "github:Mic92/sops-nix";
+      # inputs.nixpkgs.follows = "nixpkgs";
+      inputs.nixpkgs.follows = "nixpkgs-unpatched";
+    };
+    uninsane-dot-org = {
+      # provides the package to deploy <https://uninsane.org>, used only when building the servo host
+      url = "git+https://git.uninsane.org/colin/uninsane";
+      # inputs.nixpkgs.follows = "nixpkgs";
+      inputs.nixpkgs.follows = "nixpkgs-unpatched";
+    };
+  };
+
+  outputs = {
+    self,
+    nixpkgs-unpatched,
+    nixpkgs-next-unpatched ? nixpkgs-unpatched,
+    nixpkgs-wayland,
+    mobile-nixos,
+    sops-nix,
+    uninsane-dot-org,
+    ...
+  }@inputs:
+    let
+      inherit (builtins) attrNames elem listToAttrs map mapAttrs;
+      # redefine some nixpkgs `lib` functions to avoid the infinite recursion
+      # of if we tried to use patched `nixpkgs.lib` as part of the patching process.
+      mapAttrs' = f: set:
+        listToAttrs (map (attr: f attr set.${attr}) (attrNames set));
+      optionalAttrs = cond: attrs: if cond then attrs else {};
+      # mapAttrs but without the `name` argument
+      mapAttrValues = f: mapAttrs (_: f);
+
+      # rather than apply our nixpkgs patches as a flake input, do that here instead.
+      # this (temporarily?) resolves the bad UX wherein a subflake residing in the same git
+      # repo as the main flake causes the main flake to have an unstable hash.
+      patchNixpkgs = variant: nixpkgs: (import ./nixpatches/flake.nix).outputs {
+        inherit variant nixpkgs;
+        self = patchNixpkgs variant nixpkgs;
+      };
+
+      nixpkgs' = patchNixpkgs "master" nixpkgs-unpatched;
+      nixpkgsCompiledBy = system: nixpkgs'.legacyPackages."${system}";
+
+      evalHost = { name, local, target, light ? false, nixpkgs ? nixpkgs' }: nixpkgs.lib.nixosSystem {
+        system = target;
+        modules = [
+          {
+            nixpkgs.buildPlatform.system = local;
+            # nixpkgs.config.replaceStdenv = { pkgs }: pkgs.ccacheStdenv;
+          }
+          (optionalAttrs (local != target) {
+            # XXX(2023/12/11): cache.nixos.org uses `system = ...` instead of `hostPlatform.system`, and that choice impacts the closure of every package.
+            # so avoid specifying hostPlatform.system on non-cross builds, so i can use upstream caches.
+            nixpkgs.hostPlatform.system = target;
+          })
+          (optionalAttrs light {
+            sane.enableSlowPrograms = false;
+          })
+          (import ./hosts/instantiate.nix { hostName = name; })
+          self.nixosModules.default
+          self.nixosModules.passthru
+          {
+            nixpkgs.overlays = [
+              self.overlays.passthru
+              self.overlays.sane-all
+            ];
+          }
+        ];
+      };
+    in {
+      nixosConfigurations = let
+        hosts = {
+          servo       = { name = "servo";  local = "x86_64-linux"; target = "x86_64-linux";  };
+          desko       = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  };
+          desko-light = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  light = true; };
+          lappy       = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  };
+          lappy-light = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  light = true; };
+          moby        = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; };
+          moby-light  = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; light = true; };
+          rescue      = { name = "rescue"; local = "x86_64-linux"; target = "x86_64-linux";  };
+        };
+        hostsNext = mapAttrs' (h: v: {
+          name = "${h}-next";
+          value = v // { nixpkgs = patchNixpkgs "staging-next" nixpkgs-next-unpatched; };
+        }) hosts;
+      in mapAttrValues evalHost (
+        hosts // hostsNext
+      );
+
+      # unofficial output
+      # this produces a EFI-bootable .img file (GPT with a /boot partition and a system (/ or /nix) partition).
+      # after building this:
+      #   - flash it to a bootable medium (SD card, flash drive, HDD)
+      #   - resize the root partition (use cfdisk)
+      #   - mount the part
+      #      - chown root:nixbld <part>/nix/store
+      #      - chown root:root -R <part>/nix/store/*
+      #      - chown root:root -R <part>/persist  # if using impermanence
+      #      - populate any important things (persist/, home/colin/.ssh, etc)
+      #   - boot
+      #   - if fs wasn't resized automatically, then `sudo btrfs filesystem resize max /`
+      #   - checkout this flake into /etc/nixos AND UPDATE THE FS UUIDS.
+      #   - `nixos-rebuild --flake './#<host>' switch`
+      imgs = mapAttrValues (host: host.config.system.build.img) self.nixosConfigurations;
+
+      # unofficial output
+      hostConfigs = mapAttrValues (host: host.config) self.nixosConfigurations;
+      hostSystems = mapAttrValues (host: host.config.system.build.toplevel) self.nixosConfigurations;
+      hostPkgs = mapAttrValues (host: host.config.system.build.pkgs) self.nixosConfigurations;
+      hostPrograms = mapAttrValues (host: mapAttrValues (p: p.package) host.config.sane.programs) self.nixosConfigurations;
+
+      patched.nixpkgs = nixpkgs';
+
+      overlays = {
+        # N.B.: `nix flake check` requires every overlay to take `final: prev:` at defn site,
+        #   hence the weird redundancy.
+        default = final: prev: self.overlays.pkgs final prev;
+        sane-all = final: prev: import ./overlays/all.nix final prev;
+        pkgs = final: prev: import ./overlays/pkgs.nix final prev;
+        pins = final: prev: import ./overlays/pins.nix final prev;
+        preferences = final: prev: import ./overlays/preferences.nix final prev;
+        passthru = final: prev:
+          let
+            mobile = (import "${mobile-nixos}/overlay/overlay.nix");
+            uninsane = uninsane-dot-org.overlays.default;
+            wayland = final: prev: {
+              # default is to dump the packages into `waylandPkgs` *and* the toplevel.
+              # but i just want the `waylandPkgs` set
+              inherit (nixpkgs-wayland.overlays.default final prev)
+                waylandPkgs
+                new-wayland-protocols  #< 2024/03/10: nixpkgs-wayland assumes this will be in the toplevel
+              ;
+            };
+          in
+            (mobile final prev)
+            // (uninsane final prev)
+            // (wayland final prev)
+          ;
+      };
+
+      nixosModules = rec {
+        default = sane;
+        sane = import ./modules;
+        passthru = { ... }: {
+          imports = [
+            sops-nix.nixosModules.sops
+          ];
+        };
+      };
+
+      # this includes both our native packages and all the nixpkgs packages.
+      legacyPackages =
+        let
+          allPkgsFor = sys: (nixpkgsCompiledBy sys).appendOverlays [
+            self.overlays.passthru self.overlays.pkgs
+          ];
+        in {
+          x86_64-linux = allPkgsFor "x86_64-linux";
+          aarch64-linux = allPkgsFor "aarch64-linux";
+        };
+
+      # extract only our own packages from the full set.
+      # because of `nix flake check`, we flatten the package set and only surface x86_64-linux packages.
+      packages = mapAttrs
+        (system: passthruPkgs: passthruPkgs.lib.filterAttrs
+          (name: pkg:
+            # keep only packages which will pass `nix flake check`, i.e. keep only:
+            # - derivations (not package sets)
+            # - packages that build for the given platform
+            (! elem name [ "feeds" "pythonPackagesExtensions" ])
+            && (passthruPkgs.lib.meta.availableOn passthruPkgs.stdenv.hostPlatform pkg)
+          )
+          (
+            # expose sane packages and chosen inputs (uninsane.org)
+            (import ./pkgs { pkgs = passthruPkgs; }) // {
+              inherit (passthruPkgs) uninsane-dot-org;
+            }
+          )
+        )
+        # self.legacyPackages;
+        {
+          x86_64-linux = (nixpkgsCompiledBy "x86_64-linux").appendOverlays [
+            self.overlays.passthru
+          ];
+        }
+      ;
+
+      apps."x86_64-linux" =
+        let
+          pkgs = self.legacyPackages."x86_64-linux";
+          sanePkgs = import ./pkgs { inherit pkgs; };
+          deployScript = host: addr: action: pkgs.writeShellScript "deploy-${host}" ''
+            set -e
+
+            host="${host}"
+            addr="${addr}"
+            action="${if action != null then action else ""}"
+            runOnTarget() {
+              # run the command ($@) on the machine we're deploying to.
+              # if that's a remote machine, then do it via ssh, else local shell.
+              if [ -n "$addr" ]; then
+                ssh "$addr" "$@"
+              else
+                "$@"
+              fi
+            }
+
+            nix build ".#nixosConfigurations.$host.config.system.build.toplevel" --out-link "./build/result-$host" "$@"
+            storePath="$(readlink ./build/result-$host)"
+
+            # mimic `nixos-rebuild --target-host`, in effect:
+            # - nix-copy-closure ...
+            # - nix-env --set ...
+            # - switch-to-configuration <boot|dry-activate|switch|test|>
+            # avoid the actual `nixos-rebuild` for a few reasons:
+            # - fewer nix evals
+            # - more introspectability and debuggability
+            # - sandbox friendliness (especially: `git` doesn't have to be run as root)
+
+            if [ -n "$addr" ]; then
+              sudo nix store sign -r -k /run/secrets/nix_serve_privkey "$storePath"
+              # add more `-v` for more verbosity (up to 5).
+              # builders-use-substitutes false: optimizes so that the remote machine doesn't try to get paths from its substituters.
+              #   we already have all paths here, and the remote substitution is slow to check and SERIOUSLY flaky on moby in particular.
+              nix copy -vv --option builders-use-substitutes false --to "ssh-ng://$addr" "$storePath"
+            fi
+
+            if [ -n "$action" ]; then
+              runOnTarget sudo nix-env -p /nix/var/nix/profiles/system --set "$storePath"
+              runOnTarget sudo "$storePath/bin/switch-to-configuration" "$action"
+            fi
+          '';
+          deployApp = host: addr: action: {
+            type = "app";
+            program = ''${deployScript host addr action}'';
+          };
+
+          # pkg updating.
+          # a cleaner alternative lives here: <https://discourse.nixos.org/t/how-can-i-run-the-updatescript-of-personal-packages/25274/2>
+          # mkUpdater :: [ String ] -> { type = "app"; program = path; }
+          mkUpdater = attrPath: {
+            type = "app";
+            program = let
+              pkg = pkgs.lib.getAttrFromPath attrPath sanePkgs;
+              strAttrPath = pkgs.lib.concatStringsSep "." attrPath;
+              commandArgv = pkg.updateScript.command or pkg.updateScript;
+              command = pkgs.lib.escapeShellArgs commandArgv;
+            in builtins.toString (pkgs.writeShellScript "update-${strAttrPath}" ''
+              export UPDATE_NIX_NAME=${pkg.name}
+              export UPDATE_NIX_PNAME=${pkg.pname}
+              export UPDATE_NIX_OLD_VERSION=${pkg.version}
+              export UPDATE_NIX_ATTR_PATH=${strAttrPath}
+              ${command}
+            '');
+          };
+          mkUpdatersNoAliases = opts: basePath: pkgs.lib.concatMapAttrs
+            (name: pkg:
+              if pkg.recurseForDerivations or false then {
+                "${name}" = mkUpdaters opts (basePath ++ [ name ]);
+              } else if pkg.updateScript or null != null then {
+                "${name}" = mkUpdater (basePath ++ [ name ]);
+              } else {}
+            )
+            (pkgs.lib.getAttrFromPath basePath sanePkgs);
+          mkUpdaters = { ignore ? [], flakePrefix ? [] }@opts: basePath:
+            let
+              updaters = mkUpdatersNoAliases opts basePath;
+              invokeUpdater = name: pkg:
+                let
+                  fullPath = basePath ++ [ name ];
+                  doUpdateByDefault = !builtins.elem fullPath ignore;
+
+                  # in case `name` has a `.` in it, we have to quote it
+                  escapedPath = builtins.map (p: ''"${p}"'') fullPath;
+                  updatePath = builtins.concatStringsSep "." (flakePrefix ++ escapedPath);
+                in pkgs.lib.optionalString doUpdateByDefault (
+                  pkgs.lib.escapeShellArgs [
+                    "nix" "run" ".#${updatePath}"
+                  ]
+                );
+            in {
+              type = "app";
+              # top-level app just invokes the updater of everything one layer below it
+              program = builtins.toString (pkgs.writeShellScript
+                (builtins.concatStringsSep "-" (flakePrefix ++ basePath))
+                (builtins.concatStringsSep
+                  "\n"
+                  (pkgs.lib.mapAttrsToList invokeUpdater updaters)
+                )
+              );
+            } // updaters;
+        in {
+          help = {
+            type = "app";
+            program = let
+              helpMsg = builtins.toFile "nixos-config-help-message" ''
+                commands:
+                - `nix run '.#help'`
+                  - show this message
+                - `nix run '.#update.pkgs'`
+                  - updates every package
+                - `nix run '.#update.feeds'`
+                  - updates metadata for all feeds
+                - `nix run '.#init-feed' <url>`
+                - `nix run '.#deploy.{desko,lappy,moby,servo}[-light|-test]' [nix args ...]`
+                  - build and deploy the host
+                - `nix run '.#preDeploy.{desko,lappy,moby,servo}[-light]' [nix args ...]`
+                  - copy closures to a host, but don't activate it
+                  - or `nix run '.#preDeploy'` to target all hosts
+                - `nix run '.#check'`
+                  - make sure all systems build; NUR evaluates
+                - `nix run '.#bench'`
+                  - benchmark the eval time of common targets this flake provides
+
+                specific build targets of interest:
+                - `nix build '.#imgs.rescue'`
+              '';
+            in builtins.toString (pkgs.writeShellScript "nixos-config-help" ''
+              cat ${helpMsg}
+              echo ""
+              echo "complete flake structure:"
+              nix flake show --option allow-import-from-derivation true
+            '');
+          };
+          # wrangle some names to get package updaters which refer back into the flake, but also conditionally ignore certain paths (e.g. sane.feeds).
+          # TODO: better design
+          update = rec {
+            _impl.pkgs.sane = mkUpdaters { flakePrefix = [ "update" "_impl" "pkgs" ]; ignore = [ [ "sane" "feeds" ] ]; } [ "sane" ];
+            pkgs = _impl.pkgs.sane;
+            _impl.feeds.sane.feeds = mkUpdaters { flakePrefix = [ "update" "_impl" "feeds" ]; } [ "sane" "feeds" ];
+            feeds = _impl.feeds.sane.feeds;
+          };
+
+          init-feed = {
+            type = "app";
+            program = "${pkgs.feeds.init-feed}";
+          };
+
+          deploy = {
+            desko       = deployApp "desko"       "desko" "switch";
+            desko-light = deployApp "desko-light" "desko" "switch";
+            lappy       = deployApp "lappy"       "lappy" "switch";
+            lappy-light = deployApp "lappy-light" "lappy" "switch";
+            moby        = deployApp "moby"        "moby"  "switch";
+            moby-light  = deployApp "moby-light"  "moby"  "switch";
+            moby-test   = deployApp "moby"        "moby"  "test";
+            servo       = deployApp "servo"       "servo" "switch";
+
+            # like `nixos-rebuild --flake . switch`
+            self        = deployApp "$(hostname)"       ""      "switch";
+            self-light  = deployApp "$(hostname)-light" ""      "switch";
+
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "deploy-all" ''
+              nix run '.#deploy.lappy'
+              nix run '.#deploy.moby'
+              nix run '.#deploy.desko'
+              nix run '.#deploy.servo'
+            '');
+          };
+          preDeploy = {
+            # build the host and copy the runtime closure to that host, but don't activate it.
+            desko       = deployApp "desko"       "desko" null;
+            desko-light = deployApp "desko-light" "desko" null;
+            lappy       = deployApp "lappy"       "lappy" null;
+            lappy-light = deployApp "lappy-light" "lappy" null;
+            moby        = deployApp "moby"        "moby"  null;
+            moby-light  = deployApp "moby-light"  "moby"  null;
+            servo       = deployApp "servo"       "servo" null;
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "predeploy-all" ''
+              # copy the -light variants first; this might be run while waiting on a full build. or the full build failed.
+              nix run '.#preDeploy.moby-light' -- "$@"
+              nix run '.#preDeploy.lappy-light' -- "$@"
+              nix run '.#preDeploy.desko-light' -- "$@"
+              nix run '.#preDeploy.lappy' -- "$@"
+              nix run '.#preDeploy.servo' -- "$@"
+              nix run '.#preDeploy.moby' -- "$@"
+              nix run '.#preDeploy.desko' -- "$@"
+            '');
+          };
+
+          sync = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-all" ''
+              RC_lappy=$(nix run '.#sync.lappy' -- "$@")
+              RC_moby=$(nix run '.#sync.moby' -- "$@")
+              RC_desko=$(nix run '.#sync.desko' -- "$@")
+
+              echo "lappy: $RC_lappy"
+              echo "moby: $RC_moby"
+              echo "desko: $RC_desko"
+            '');
+          };
+
+          sync.desko = {
+            # copy music from servo to desko
+            # can run this from any device that has ssh access to desko and servo
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-to-desko" ''
+              sudo mount /mnt/desko/home
+              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music --compat /mnt/servo/media/Music /mnt/desko/home/Music "$@"
+            '');
+          };
+
+          sync.lappy = {
+            # copy music from servo to lappy
+            # can run this from any device that has ssh access to lappy and servo
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-to-lappy" ''
+              sudo mount /mnt/lappy/home
+              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music --compress --compat /mnt/servo/media/Music /mnt/lappy/home/Music "$@"
+            '');
+          };
+
+          sync.moby = {
+            # copy music from servo to moby
+            # can run this from any device that has ssh access to moby and servo
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-to-moby" ''
+              sudo mount /mnt/moby/home
+              sudo mount /mnt/desko/home
+              ${pkgs.rsync}/bin/rsync -arv --exclude servo-macros /mnt/moby/home/Pictures/ /mnt/desko/home/Pictures/moby/
+              # N.B.: limited by network/disk -> reduce job count to improve pause/resume behavior
+              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music --compress --compat --jobs 4 /mnt/servo/media/Music /mnt/moby/home/Music "$@"
+            '');
+          };
+
+          check = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "check-all" ''
+              nix run '.#check.nur'
+              RC0=$?
+              nix run '.#check.hostConfigs'
+              RC1=$?
+              nix run '.#check.rescue'
+              RC2=$?
+              echo "nur: $RC0"
+              echo "hostConfigs: $RC1"
+              echo "rescue: $RC2"
+              exit $(($RC0 | $RC1 | $RC2))
+            '');
+          };
+
+          check.nur = {
+            # `nix run '.#check-nur'`
+            # validates that my repo can be included in the Nix User Repository
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "check-nur" ''
+              cd ${./.}/integrations/nur
+              NIX_PATH= NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 nix-env -f . -qa \* --meta --xml \
+                --allowed-uris https://static.rust-lang.org \
+                --option restrict-eval true \
+                --option allow-import-from-derivation true \
+                --drv-path --show-trace \
+                -I nixpkgs=${nixpkgs-unpatched} \
+                -I nixpkgs-overlays=${./.}/hosts/common/nix/overlay \
+                -I ../../ \
+                | tee  # tee to prevent interactive mode
+            '');
+          };
+
+          check.hostConfigs = {
+            type = "app";
+            program = let
+              checkHost = host: let
+                shellHost = pkgs.lib.replaceStrings [ "-" ] [ "_" ] host;
+              in ''
+                nix build -v '.#nixosConfigurations.${host}.config.system.build.toplevel' --out-link ./build/result-${host} -j2 "$@"
+                RC_${shellHost}=$?
+              '';
+            in builtins.toString (pkgs.writeShellScript
+              "check-host-configs"
+              ''
+                # build minimally-usable hosts first, then their full image.
+                # this gives me a minimal image i can deploy or copy over, early.
+                ${checkHost "desko-light"}
+                ${checkHost "moby-light"}
+                ${checkHost "lappy-light"}
+
+                ${checkHost "desko"}
+                ${checkHost "lappy"}
+                ${checkHost "servo"}
+                ${checkHost "moby"}
+                ${checkHost "rescue"}
+
+                # still want to build the -light variants first so as to avoid multiple simultaneous webkitgtk builds
+                ${checkHost "desko-light-next"}
+                ${checkHost "moby-light-next"}
+
+                ${checkHost "desko-next"}
+                ${checkHost "lappy-next"}
+                ${checkHost "servo-next"}
+                ${checkHost "moby-next"}
+                ${checkHost "rescue-next"}
+
+                echo "desko: $RC_desko"
+                echo "lappy: $RC_lappy"
+                echo "servo: $RC_servo"
+                echo "moby: $RC_moby"
+                echo "rescue: $RC_rescue"
+
+                echo "desko-next: $RC_desko_next"
+                echo "lappy-next: $RC_lappy_next"
+                echo "servo-next: $RC_servo_next"
+                echo "moby-next: $RC_moby_next"
+                echo "rescue-next: $RC_rescue_next"
+
+                # i don't really care if the -next hosts fail. i build them mostly to keep the cache fresh/ready
+                exit $(($RC_desko | $RC_lappy | $RC_servo | $RC_moby | $RC_rescue))
+              ''
+            );
+          };
+
+          check.rescue = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "check-rescue" ''
+              nix build -v '.#imgs.rescue' --out-link ./build/result-rescue-img -j2
+            '');
+          };
+
+          bench = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "bench" ''
+              doBench() {
+                attrPath="$1"
+                shift
+                echo -n "benchmarking eval of '$attrPath'... "
+                /run/current-system/sw/bin/time -f "%e sec" -o /dev/stdout \
+                  nix eval --no-eval-cache --quiet --raw ".#$attrPath" --apply 'result: if result != null then "" else "unexpected null"' $@ 2> /dev/null
+              }
+
+              if [ -n "$1" ]; then
+                doBench "$@"
+              else
+                doBench hostConfigs
+                doBench hostConfigs.lappy
+                doBench hostConfigs.lappy.sane.programs
+                doBench hostConfigs.lappy.sane.users.colin
+                doBench hostConfigs.lappy.sane.fs
+                doBench hostConfigs.lappy.environment.systemPackages
+              fi
+            '');
+          };
+        };
+
+      templates = {
+        env.python-data = {
+          # initialize with:
+          # - `nix flake init -t '/home/colin/dev/nixos/#env.python-data'`
+          # then enter with:
+          # - `nix develop`
+          path = ./templates/env/python-data;
+          description = "python environment for data processing";
+        };
+        pkgs.rust-inline = {
+          # initialize with:
+          # - `nix flake init -t '/home/colin/dev/nixos/#pkgs.rust-inline'`
+          path = ./templates/pkgs/rust-inline;
+          description = "rust package and development environment (inline rust sources)";
+        };
+        pkgs.rust = {
+          # initialize with:
+          # - `nix flake init -t '/home/colin/dev/nixos/#pkgs.rust'`
+          path = ./templates/pkgs/rust;
+          description = "rust package fit to ship in nixpkgs";
+        };
+        pkgs.make = {
+          # initialize with:
+          # - `nix flake init -t '/home/colin/dev/nixos/#pkgs.make'`
+          path = ./templates/pkgs/make;
+          description = "default Makefile-based derivation";
+        };
+      };
+    };
+}
+

hosts/by-name/crappy/default.nix

@@ -1,45 +0,0 @@
-# Samsung chromebook XE303C12
-# - <https://wiki.postmarketos.org/wiki/Samsung_Chromebook_(google-snow)>
-{ ... }:
-{
-  imports = [
-    ./fs.nix
-  ];
-
-  sane.hal.samsung.enable = true;
-  sane.roles.client = true;
-  # sane.roles.pc = true;
-
-  users.users.colin.initialPassword = "147147";
-  sane.programs.sway.enableFor.user.colin = true;
-
-  sane.programs.calls.enableFor.user.colin = false;
-  sane.programs.consoleMediaUtils.enableFor.user.colin = true;
-  sane.programs.epiphany.enableFor.user.colin = true;
-  sane.programs.geary.enableFor.user.colin = false;
-  # sane.programs.firefox.enableFor.user.colin = true;
-  sane.programs.portfolio-filemanager.enableFor.user.colin = true;
-  sane.programs.signal-desktop.enableFor.user.colin = false;
-  sane.programs.wike.enableFor.user.colin = true;
-
-  sane.programs.dino.config.autostart = false;
-  sane.programs.dissent.config.autostart = false;
-  sane.programs.fractal.config.autostart = false;
-  sane.programs.sway.config.mod = "Mod1";  #< alt key instead of Super
-
-  # sane.programs.guiApps.enableFor.user.colin = false;
-
-  # sane.programs.pcGuiApps.enableFor.user.colin = false;  #< errors!
-
-  sane.programs.blueberry.enableFor.user.colin = false;  # bluetooth manager: doesn't cross compile!
-  # sane.programs.brave.enableFor.user.colin = false;  # 2024/06/03: fails eval if enabled on cross
-  # sane.programs.firefox.enableFor.user.colin = false;  # 2024/06/03: this triggers an eval error in yarn stuff -- i'm doing IFD somewhere!!?
-  sane.programs.mepo.enableFor.user.colin = false;  # 2024/06/04: doesn't cross compile (nodejs)
-  sane.programs.mercurial.enableFor.user.colin = false;  # 2024/06/03: does not cross compile
-  sane.programs.nixpkgs-review.enableFor.user.colin = false;  # 2024/06/03: OOMs when cross compiling
-  sane.programs.ntfy-sh.enableFor.user.colin = false;  # 2024/06/04: doesn't cross compile (nodejs)
-  sane.programs.pwvucontrol.enableFor.user.colin = false;  # 2024/06/03: doesn't cross compile (libspa-sys)
-  sane.programs."sane-scripts.bt-search".enableFor.user.colin = false;  # 2024/06/03: does not cross compile
-  sane.programs.sequoia.enableFor.user.colin = false;  # 2024/06/03: does not cross compile
-  sane.programs.zathura.enableFor.user.colin = false;  # 2024/06/03: does not cross compile
-}

hosts/by-name/crappy/fs.nix

@@ -1,16 +0,0 @@
-{ ... }:
-{
-  fileSystems."/nix" = {
-    device = "/dev/disk/by-uuid/55555555-0303-0c12-86df-eda9e9311526";
-    fsType = "btrfs";
-    options = [
-      "compress=zstd"
-      "defaults"
-    ];
-  };
-
-  fileSystems."/boot" = {
-    device = "/dev/disk/by-uuid/303C-5A37";
-    fsType = "vfat";
-  };
-}

hosts/by-name/desko/default.nix

@@ -1,58 +1,52 @@
-{ config, lib, pkgs, ... }:
+{ config, pkgs, ... }:
 {
   imports = [
     ./fs.nix
   ];
 
-  sane.services.trust-dns.asSystemResolver = false;  # TEMPORARY: TODO: re-enable trust-dns
-  # sane.programs.devPkgs.enableFor.user.colin = true;
   # sane.guest.enable = true;
 
-  # don't enable wifi by default: it messes with connectivity.
-  # systemd.services.iwd.enable = false;
-  # networking.wireless.enable = false;
-  # systemd.services.wpa_supplicant.enable = false;
-  # sane.programs.wpa_supplicant.enableFor.user.colin = lib.mkForce false;
-  # sane.programs.wpa_supplicant.enableFor.system = lib.mkForce false;
-  # don't auto-connect to wifi networks
-  # see: <https://networkmanager.dev/docs/api/latest/NetworkManager.conf.html#device-spec>
-  networking.networkmanager.unmanaged = [ "type:wifi" ];
+  # services.distccd.enable = true;
+  # sane.programs.distcc.enableFor.user.guest = true;
+
+  # TODO: remove emulation, but need to fix nixos-rebuild to moby for that.
+  # sane.roles.build-machine.emulation = true;
 
   sops.secrets.colin-passwd.neededForUsers = true;
 
+  sane.ports.openFirewall = true;  # for e.g. nix-serve
+
   sane.roles.build-machine.enable = true;
   sane.roles.client = true;
   sane.roles.dev-machine = true;
   sane.roles.pc = true;
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."desko".wg-home.ip;
-  sane.ovpn.addrV4 = "172.26.55.21";
-  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:20c1:a73c";
   sane.services.duplicity.enable = true;
-  sane.services.rsync-net.enable = true;
+  sane.services.nixserve.secretKeyFile = config.sops.secrets.nix_serve_privkey.path;
 
+  sane.nixcache.substituters.desko = false;
   sane.nixcache.remote-builders.desko = false;
 
-  sane.programs.sane-private-unlock-remote.enableFor.user.colin = true;
-  sane.programs.sane-private-unlock-remote.config.hosts = [ "servo" ];
-
   sane.programs.sway.enableFor.user.colin = true;
   sane.programs.iphoneUtils.enableFor.user.colin = true;
   sane.programs.steam.enableFor.user.colin = true;
 
-  sane.programs.geary.config.autostart = true;
+  # sane.programs.devPkgs.enableFor.user.colin = true;
+
+  sane.programs."gnome.geary".config.autostart = true;
   sane.programs.signal-desktop.config.autostart = true;
 
-  sane.programs.nwg-panel.config = {
-    battery = false;
-    brightness = false;
-  };
-
+  boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
 
   # needed to use libimobiledevice/ifuse, for iphone sync
   services.usbmuxd.enable = true;
 
+  # don't enable wifi by default: it messes with connectivity.
+  systemd.services.iwd.enable = false;
+  systemd.services.wpa_supplicant.enable = false;
+
   # default config: https://man.archlinux.org/man/snapper-configs.5
   # defaults to something like:
   #   - hourly snapshots
@@ -64,4 +58,7 @@
     # TODO: ALLOW_USERS doesn't seem to work. still need `sudo snapper -c nix list`
     ALLOW_USERS = [ "colin" ];
   };
+
+  # docs: https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion
+  system.stateVersion = "21.05";
 }

hosts/by-name/lappy/default.nix

@@ -9,25 +9,18 @@
   sane.roles.pc = true;
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."lappy".wg-home.ip;
-  sane.ovpn.addrV4 = "172.23.119.72";
-  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:0332:aa96/128";
 
   # sane.guest.enable = true;
+  boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
 
-  sane.programs.sane-private-unlock-remote.enableFor.user.colin = true;
-  sane.programs.sane-private-unlock-remote.config.hosts = [ "servo" ];
-
-  sane.programs.stepmania.enableFor.user.colin = true;
   sane.programs.sway.enableFor.user.colin = true;
-
-  sane.programs.geary.config.autostart = true;
+  sane.programs."gnome.geary".config.autostart = true;
   sane.programs.signal-desktop.config.autostart = true;
+  sane.programs.stepmania.enableFor.user.colin = true;
 
   sops.secrets.colin-passwd.neededForUsers = true;
 
-  sane.services.rsync-net.enable = true;
-
   # default config: https://man.archlinux.org/man/snapper-configs.5
   # defaults to something like:
   #   - hourly snapshots
@@ -38,4 +31,7 @@
     SUBVOLUME = "/nix";
     ALLOW_USERS = [ "colin" ];
   };
+
+  # docs: https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion
+  system.stateVersion = "21.05";
 }

hosts/by-name/lappy/xkb_mobile_normal_buttons

@@ -0,0 +1,7 @@
+xkb_keymap {
+	xkb_keycodes  { include "evdev+aliases(qwerty)"	};
+	xkb_types     { include "complete"	};
+	xkb_compat    { include "complete"	};
+	xkb_symbols   { include "pc+us+inet(evdev)"	};
+	xkb_geometry  { include "pc(pc105)"	};
+};

hosts/by-name/moby/bootloader.nix

@@ -0,0 +1,22 @@
+# tow-boot: <https://tow-boot.org>
+# docs (pinephone specific): <https://github.com/Tow-Boot/Tow-Boot/tree/development/boards/pine64-pinephoneA64>
+# LED and button behavior is defined here: <https://github.com/Tow-Boot/Tow-Boot/blob/development/modules/tow-boot/phone-ux.nix>
+# - hold VOLDOWN: enter recovery mode
+#   - LED will turn aqua instead of yellow
+#   - recovery mode would ordinarily allow a selection of entries, but for pinephone i guess it doesn't do anything?
+# - hold VOLUP: force it to load the OS from eMMC?
+#   - LED will turn blue instead of yellow
+# boot LEDs:
+# - yellow = entered tow-boot
+# - 10 red flashes => poweroff means tow-boot couldn't boot into the next stage (i.e. distroboot)
+#   - distroboot: <https://source.denx.de/u-boot/u-boot/-/blob/v2022.04/doc/develop/distro.rst>)
+{ config, pkgs, ... }:
+{
+  # we need space in the GPT header to place tow-boot.
+  # only actually need 1 MB, but better to over-allocate than under-allocate
+  sane.image.extraGPTPadding = 16 * 1024 * 1024;
+  sane.image.firstPartGap = 0;
+  sane.image.installBootloader = ''
+    dd if=${pkgs.tow-boot-pinephone}/Tow-Boot.noenv.bin of=$out/nixos.img bs=1024 seek=8 conv=notrunc
+  '';
+}

hosts/by-name/moby/default.nix

@@ -1,4 +1,7 @@
 # Pinephone
+# other setups to reference:
+# - <https://hamblingreen.gitlab.io/2022/03/02/my-pinephone-setup.html>
+#   - sxmo Arch user. lots of app recommendations
 #
 # wikis, resources, ...:
 # - Linux Phone Apps: <https://linuxphoneapps.org/>
@@ -9,27 +12,37 @@
 { config, pkgs, lib, ... }:
 {
   imports = [
+    ./bootloader.nix
     ./fs.nix
+    ./gps.nix
+    ./kernel.nix
+    ./polyfill.nix
   ];
 
-  sane.hal.pine64.enable = true;
   sane.roles.client = true;
   sane.roles.handheld = true;
+  sane.zsh.showDeadlines = false;  # unlikely to act on them when in shell
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."moby".wg-home.ip;
-  sane.ovpn.addrV4 = "172.24.87.255";
-  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:18cd:a72b";
+
+  # for some reason desko -> moby deploys are super flaky when desko is also a nixcache (not true of desko -> lappy deploys, though!)
+  # > unable to download 'http://desko:5001/<hash>.narinfo': Server returned nothing (no headers, no data) (52)
+  sane.nixcache.substituters.desko = false;
 
   # XXX colin: phosh doesn't work well with passwordless login,
   # so set this more reliable default password should anything go wrong
   users.users.colin.initialPassword = "147147";
+  # services.getty.autologinUser = "root";  # allows for emergency maintenance?
 
   sops.secrets.colin-passwd.neededForUsers = true;
 
-  sane.services.rsync-net.enable = true;
-
+  # sane.gui.sxmo.enable = true;
   sane.programs.sway.enableFor.user.colin = true;
-  sane.programs.sway.config.mod = "Mod1";  #< alt key instead of Super
+  sane.programs.swaylock.enableFor.user.colin = false;  #< not usable on touch
+  sane.programs.schlock.enableFor.user.colin = true;
+  sane.programs.swayidle.config.actions.screenoff.delay = 300;
+  sane.programs.swayidle.config.actions.screenoff.enable = true;
+  sane.programs.sane-input-handler.enableFor.user.colin = true;
   sane.programs.blueberry.enableFor.user.colin = false;  # bluetooth manager: doesn't cross compile!
   sane.programs.fcitx5.enableFor.user.colin = false;  # does not cross compile
   sane.programs.mercurial.enableFor.user.colin = false;  # does not cross compile
@@ -37,16 +50,30 @@
 
   # enabled for easier debugging
   sane.programs.eg25-control.enableFor.user.colin = true;
-  # sane.programs.rtl8723cs-wowlan.enableFor.user.colin = true;
+  sane.programs.rtl8723cs-wowlan.enableFor.user.colin = true;
 
   # sane.programs.ntfy-sh.config.autostart = true;
   sane.programs.dino.config.autostart = true;
-  sane.programs.signal-desktop.config.autostart = true;
-  # sane.programs.geary.config.autostart = true;
+  # sane.programs.signal-desktop.config.autostart = true;  # TODO: enable once electron stops derping.
+  # sane.programs."gnome.geary".config.autostart = true;
   # sane.programs.calls.config.autostart = true;
 
-  sane.programs.pipewire.config = {
-    # tune so Dino doesn't drop audio
+  sane.programs.firefox.mime.priority = 300;  # prefer other browsers when possible
+  # HACK/TODO: make `programs.P.env.VAR` behave according to `mime.priority`
+  sane.programs.firefox.env = lib.mkForce {};
+  sane.programs.epiphany.env.BROWSER = "epiphany";
+
+  # note the .conf.d approach: using ~/.config/pipewire/pipewire.conf directly breaks all audio,
+  # presumably because that deletes the defaults entirely whereas the .conf.d approach selectively overrides defaults
+  sane.user.fs.".config/pipewire/pipewire.conf.d/10-fix-dino-mic-cutout.conf".symlink.text = ''
+    # config docs: <https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-PipeWire#properties>
+    # useful to run `pw-top` to see that these settings are actually having effect,
+    # and `pw-metadata` to see if any settings conflict (e.g. max-quantum < min-quantum)
+    #
+    # restart pipewire after editing these files:
+    # - `systemctl --user restart pipewire`
+    # - pipewire users will likely stop outputting audio until they are also restarted
+    #
     # there's seemingly two buffers for the mic (see: <https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/FAQ#pipewire-buffering-explained>)
     # 1. Pipewire buffering out of the driver and into its own member.
     # 2. Pipewire buffering into Dino.
@@ -57,11 +84,67 @@
     # `pw-metadata -n settings 0 clock.force-quantum 1024` reduces to about 1 error per second.
     # `pw-metadata -n settings 0 clock.force-quantum 2048` reduces to 1 error every < 10s.
     # pipewire default config includes `clock.power-of-two-quantum = true`
-    min-quantum = 2048;
-    max-quantum = 8192;
-  };
+    context.properties = {
+      default.clock.min-quantum = 2048
+      default.clock.max-quantum = 8192
+    }
+  '';
 
+  boot.loader.efi.canTouchEfiVariables = false;
   # /boot space is at a premium. default was 20.
   # even 10 can be too much
   boot.loader.generic-extlinux-compatible.configurationLimit = 8;
+  # mobile.bootloader.enable = false;
+  # mobile.boot.stage-1.enable = false;
+  # boot.initrd.systemd.enable = false;
+  # boot.initrd.services.swraid.enable = false;  # attempt to fix dm_mod stuff
+
+  # hardware.firmware makes the referenced files visible to the kernel, for whenever a driver explicitly asks for them.
+  # these files are visible from userspace by following `/sys/module/firmware_class/parameters/path`
+  #
+  # mobile-nixos' /lib/firmware includes:
+  #   rtl_bt          (bluetooth)
+  #   anx7688-fw.bin  (USB-C chip: power negotiation, HDMI/dock)
+  #   ov5640_af.bin   (camera module)
+  # hardware.firmware = [ config.mobile.device.firmware ];
+  # hardware.firmware = [ pkgs.rtl8723cs-firmware ];
+  hardware.firmware = [
+    (pkgs.linux-firmware-megous.override {
+      # rtl_bt = false probably means no bluetooth connectivity.
+      # N.B.: DON'T RE-ENABLE without first confirming that wake-on-lan works during suspend (rtcwake).
+      # it seems the rtl_bt stuff ("bluetooth coexist") might make wake-on-LAN radically more flaky.
+      rtl_bt = false;
+    })
+  ];
+
+  system.stateVersion = "21.11";
+
+  # defined: https://www.freedesktop.org/software/systemd/man/machine-info.html
+  # XXX colin: not sure which, if any, software makes use of this
+  environment.etc."machine-info".text = ''
+    CHASSIS="handset"
+  '';
+
+  # enable rotation sensor
+  hardware.sensor.iio.enable = true;
+
+  # TODO: move elsewhere...
+  systemd.services.ModemManager.serviceConfig = {
+    # N.B.: the extra "" in ExecStart serves to force upstream ExecStart to be ignored
+    ExecStart = [ "" "${pkgs.modemmanager}/bin/ModemManager --debug" ];
+    # --debug sets DEBUG level logging: so reset
+    ExecStartPost = [ "${pkgs.modemmanager}/bin/mmcli --set-logging=INFO" ];
+  };
+
+  services.udev.extraRules = let
+    chmod = "${pkgs.coreutils}/bin/chmod";
+    chown = "${pkgs.coreutils}/bin/chown";
+  in ''
+    # make Pinephone flashlight writable by user.
+    # taken from postmarketOS: <repo:postmarketOS/pmaports:device/main/device-pine64-pinephone/60-flashlight.rules>
+    SUBSYSTEM=="leds", DEVPATH=="*/*:flash", RUN+="${chmod} g+w /sys%p/brightness /sys%p/flash_strobe", RUN+="${chown} :video /sys%p/brightness /sys%p/flash_strobe"
+
+    # make Pinephone front LEDs writable by user.
+    SUBSYSTEM=="leds", DEVPATH=="*/*:indicator", RUN+="${chmod} g+w /sys%p/brightness", RUN+="${chown} :video /sys%p/brightness"
+  '';
 }

hosts/by-name/moby/gps.nix

@@ -0,0 +1,69 @@
+# pinephone GPS happens in EG25 modem
+# serial control interface to modem is /dev/ttyUSB2
+# after enabling GPS, readout is /dev/ttyUSB1
+#
+# minimal process to enable modem and GPS:
+# - `echo 1 > /sys/class/modem-power/modem-power/device/powered`
+# - `screen /dev/ttyUSB2 115200`
+#   - `AT+QGPSCFG="nmeasrc",1`
+#   - `AT+QGPS=1`
+# this process is automated by my `eg25-control` program and services (`eg25-control-powered`, `eg25-control-gps`)
+# - see the `modules/` directory further up this repository.
+#
+# now, something like `gpsd` can directly read from /dev/ttyUSB1,
+# or geoclue can query the GPS directly through modem-manager
+#
+# initial GPS fix can take 15+ minutes.
+# meanwhile, services like eg25-manager or eg25-control-freshen-agps can speed this up by uploading assisted GPS data to the modem.
+#
+# support/help:
+# - geoclue, gnome-maps
+#   - irc: #gnome-maps on irc.gimp.org
+#   - Matrix: #gnome-maps:gnome.org  (unclear if bridged to IRC)
+#
+# programs to pair this with:
+# - `satellite-gtk`: <https://codeberg.org/tpikonen/satellite>
+#   - shows/tracks which satellites the GPS is connected to; useful to understand fix characteristics
+# - `gnome-maps`: uses geoclue, has route planning
+# - `mepo`: uses gpsd, minimalist, flaky, and buttons are kinda hard to activate on mobile
+# - puremaps?
+# - osmin?
+#
+# known/outstanding bugs:
+# - `systemctl start eg25-control-gps` can the hang the whole system (2023/10/06)
+#   - i think it's actually `eg25-control-powered` which does this (started by the gps)
+#   - best guess is modem draws so much power at launch that other parts of the system see undervoltage
+#   - workaround is to hard power-cycle the system. the modem may not bring up after reboot: leave unpowered for 60s and boot again.
+#
+# future work:
+# - integrate with [wigle](https://www.wigle.net/) for offline equivalent to Mozilla Location Services
+
+{ config, lib, ... }:
+{
+  # test gpsd with `gpspipe -w -n 10 2> /dev/null | grep -m 1 TPV | jq '.lat, .lon' | tr '\n' ' '`
+  # ^ should return <lat> <long>
+  services.gpsd.enable = true;
+  services.gpsd.devices = [ "/dev/ttyUSB1" ];
+
+  # test geoclue2 by building `geoclue2-with-demo-agent`
+  # and running "${geoclue2-with-demo-agent}/libexec/geoclue-2.0/demos/where-am-i"
+  # note that geoclue is dbus-activated, and auto-stops after 60s with no caller
+  services.geoclue2.enable = true;
+  services.geoclue2.appConfig.where-am-i = {
+    # this is the default "agent", shipped by geoclue package: allow it to use location
+    isAllowed = true;
+    isSystem = false;
+    # XXX: setting users != [] might be causing `where-am-i` to time out
+    users = [
+      # restrict to only one set of users. empty array (default) means "allow any user to access geolocation".
+      (builtins.toString config.users.users.colin.uid)
+    ];
+  };
+  systemd.services.geoclue.after = lib.mkForce [];  #< defaults to network-online, but not all my sources require network
+  users.users.geoclue.extraGroups = [
+    "dialout"  # TODO: figure out if dialout is required. that's for /dev/ttyUSB1, but geoclue probably doesn't read that?
+  ];
+
+  sane.services.eg25-control.enable = true;
+  sane.programs.where-am-i.enableFor.user.colin = true;
+}

hosts/by-name/moby/kernel.nix

@@ -0,0 +1,91 @@
+{ pkgs, ... }:
+let
+  dmesg = "${pkgs.util-linux}/bin/dmesg";
+  grep = "${pkgs.gnugrep}/bin/grep";
+  modprobe = "${pkgs.kmod}/bin/modprobe";
+  ensureHWReady = ''
+    # common boot failure:
+    # blank screen (no backlight even), with the following log:
+    # ```syslog
+    # sun8i-dw-hdmi 1ee0000.hdmi: Couldn't get the HDMI PHY
+    # ...
+    # sun4i-drm display-engine: Couldn't bind all pipelines components
+    # ...
+    # sun8i-dw-hdmi: probe of 1ee0000.hdmi failed with error -17
+    # ```
+    #
+    # in particular, that `probe ... failed` occurs *only* on failed boots
+    # (the other messages might sometimes occur even on successful runs?)
+    #
+    # reloading the sun8i hdmi driver usually gets the screen on, showing boot text.
+    # then restarting display-manager.service gets us to the login.
+    #
+    # NB: the above log is default level. though less specific, there's a `err` level message that also signals this:
+    # sun4i-drm display-engine: failed to bind 1ee0000.hdmi (ops sun8i_dw_hdmi_ops [sun8i_drm_hdmi]): -17
+    # NB: this is the most common, but not the only, failure mode for `display-manager`.
+    # another error seems characterized by these dmesg logs, in which reprobing sun8i_drm_hdmi does not fix:
+    # ```syslog
+    # sun6i-mipi-dsi 1ca0000.dsi: Couldn't get the MIPI D-PHY
+    # sun4i-drm display-engine: Couldn't bind all pipelines components
+    # sun6i-mipi-dsi 1ca0000.dsi: Couldn't register our component
+    # ```
+
+    if (${dmesg} --kernel --level err --color=never --notime | ${grep} -q 'sun4i-drm display-engine: failed to bind 1ee0000.hdmi')
+    then
+      echo "reprobing sun8i_drm_hdmi"
+      # if a command here fails it errors the whole service, so prefer to log instead
+      ${modprobe} -r sun8i_drm_hdmi || echo "failed to unload sun8i_drm_hdmi"
+      ${modprobe} sun8i_drm_hdmi || echo "failed to load sub8i_drm_hdmi"
+    fi
+  '';
+in
+{
+  boot.kernelPackages = pkgs.linuxPackagesFor pkgs.linux-megous;
+  # boot.kernelPackages = pkgs.linuxPackagesFor pkgs.linux-manjaro;
+  # boot.kernelPackages = pkgs.linuxPackagesFor pkgs.linux_latest;
+
+  # alternatively, apply patches directly to stock nixos kernel:
+  # boot.kernelPatches = manjaroPatches ++ [
+  #   (patchDefconfig kernelConfig)
+  # ];
+
+  # configure nixos to build a compressed kernel image, since it doesn't usually do that for aarch64 target.
+  # without this i run out of /boot space in < 10 generations
+  nixpkgs.hostPlatform.linux-kernel = {
+    # defaults:
+    name = "aarch64-multiplatform";
+    baseConfig = "defconfig";
+    DTB = true;
+    autoModules = true;
+    preferBuiltin = true;
+    # extraConfig = ...
+    # ^-- raspberry pi stuff: we don't need it.
+
+    # target = "Image";  # <-- default
+    target = "Image.gz";  # <-- compress the kernel image
+    # target = "zImage";  # <-- confuses other parts of nixos :-(
+  };
+
+  # disable proximity sensor.
+  # the filtering/calibration is bad that it causes the screen to go fully dark at times.
+  boot.blacklistedKernelModules = [ "stk3310" ];
+
+  boot.kernelParams = [
+    # without this some GUI apps fail: `DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory`
+    # this is because they can't allocate enough video ram.
+    # see related nixpkgs issue: <https://github.com/NixOS/nixpkgs/issues/260222>
+    # TODO(2023/12/03): remove once mesa 23.3.1 lands: <https://github.com/NixOS/nixpkgs/pull/265740>
+    #
+    # the default CMA seems to be 32M.
+    # i was running fine with 256MB from 2022/07-ish through 2022/12-ish, but then the phone quit reliably coming back from sleep (phosh): maybe a memory leak?
+    # `cat /proc/meminfo` to see CmaTotal/CmaFree if interested in tuning this.
+    "cma=512M"
+    # 2023/10/20: potential fix for the lima (GPU) timeout bugs:
+    # - <https://gitlab.com/postmarketOS/pmaports/-/issues/805#note_890467824>
+    "lima.sched_timeout_ms=2000"
+  ];
+
+  # services.xserver.displayManager.job.preStart = ensureHWReady;
+  # systemd.services.greetd.preStart = ensureHWReady;
+  systemd.services.unl0kr.preStart = ensureHWReady;
+}

hosts/by-name/moby/polyfill.nix

@@ -0,0 +1,45 @@
+# this file configures preferences per program, without actually enabling any programs.
+# the goal is to separate the place where we decide *what* to use (i.e. `sane.programs.firefox.enable = true` -- at the toplevel)
+# from where we specific how that thing should behave *if* it's in use.
+#
+# NixOS backgrounds:
+# - <https://github.com/NixOS/nixos-artwork>
+#   - <https://github.com/NixOS/nixos-artwork/issues/50>  (colorful; unmerged)
+#   - <https://github.com/NixOS/nixos-artwork/pull/60/files>  (desktop-oriented; clean; unmerged)
+# - <https://itsfoss.com/content/images/2023/04/nixos-tutorials.png>
+
+{ lib, pkgs, sane-lib, ... }:
+{
+  sane.programs.firefox.config = {
+    # compromise impermanence for the sake of usability
+    persistCache = "private";
+    persistData = "private";
+
+    # i don't do crypto stuff on moby
+    addons.ether-metamask.enable = false;
+    # sidebery UX doesn't make sense on small screen
+    addons.sidebery.enable = false;
+  };
+  sane.programs.swaynotificationcenter.config = {
+    backlight = "backlight";  # /sys/class/backlight/*backlight*/brightness
+  };
+
+  sane.programs.alacritty.config.fontSize = 9;
+
+  sane.programs.sway.config = {
+    font = "pango:monospace 10";
+    mod = "Mod1";  # prefer Alt
+    workspace_layout = "tabbed";
+  };
+
+  sane.programs.waybar.config = {
+    fontSize = 14;
+    height = 26;
+    persistWorkspaces = [ "1" "2" "3" "4" "5" ];
+    modules.media = false;
+    modules.network = false;
+    modules.perf = false;
+    modules.windowTitle = false;
+    # TODO: show modem state
+  };
+}

hosts/by-name/rescue/default.nix

@@ -4,6 +4,7 @@
     ./fs.nix
   ];
 
+  boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
   sane.persist.enable = false;  # what we mean here is that the image is immutable; `/` is still tmpfs.
   sane.nixcache.enable = false;  # don't want to be calling out to dead machines that we're *trying* to rescue
@@ -11,4 +12,7 @@
   # auto-login at shell
   services.getty.autologinUser = "colin";
   # users.users.colin.initialPassword = "colin";
+
+  # docs: https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion
+  system.stateVersion = "21.05";
 }

hosts/by-name/servo/default.nix

@@ -7,38 +7,37 @@
     ./services
   ];
 
-  # for administering services
-  sane.programs.clightning-sane.enableFor.user.colin = true;
-  # sane.programs.freshrss.enableFor.user.colin = true;
-  # sane.programs.signaldctl.enableFor.user.colin = true;
-  # sane.programs.matrix-synapse.enableFor.user.colin = true;
+  sane.programs = {
+    # for administering services
+    freshrss.enableFor.user.colin = true;
+    matrix-synapse.enableFor.user.colin = true;
+    signaldctl.enableFor.user.colin = true;
+  };
 
   sane.roles.build-machine.enable = true;
-  sane.programs.sane-deadlines.config.showOnLogin = false;  # ~/knowledge doesn't always exist
+  sane.zsh.showDeadlines = false;  # ~/knowledge doesn't always exist
   sane.programs.consoleUtils.suggestedPrograms = [
     "consoleMediaUtils"  # notably, for go2tv / casting
     "pcConsoleUtils"
     "sane-scripts.stop-all-servo"
   ];
   sane.services.dyn-dns.enable = true;
-  sane.services.trust-dns.asSystemResolver = false;  # TODO: enable once it's all working well
   sane.services.wg-home.enable = true;
   sane.services.wg-home.visibleToWan = true;
   sane.services.wg-home.forwardToWan = true;
   sane.services.wg-home.routeThroughServo = false;
   sane.services.wg-home.ip = config.sane.hosts.by-name."servo".wg-home.ip;
-  sane.ovpn.addrV4 = "172.23.174.114";
-  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:8df3:14b0";
+  sane.nixcache.substituters.servo = false;
+  sane.nixcache.substituters.desko = false;
   sane.nixcache.remote-builders.desko = false;
   sane.nixcache.remote-builders.servo = false;
   # sane.services.duplicity.enable = true;  # TODO: re-enable after HW upgrade
-  sane.services.rsync-net.enable = true;
 
   # automatically log in at the virtual consoles.
-  # using root here makes sure we always have an escape hatch.
-  # XXX(2024-07-27): this is incompatible with my s6-rc stuff, which needs to auto-login as `colin` to start its user services.
-  # services.getty.autologinUser = "root";
+  # using root here makes sure we always have an escape hatch
+  services.getty.autologinUser = "root";
 
+  boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
 
   # both transmission and ipfs try to set different net defaults.
@@ -46,5 +45,13 @@
   boot.kernel.sysctl = {
     "net.core.rmem_max" = 4194304;  # 4MB
   };
+
+  # This value determines the NixOS release from which the default
+  # settings for stateful data, like file locations and database versions
+  # on your system were taken. It‘s perfectly fine and recommended to leave
+  # this value at the release version of the first install of this system.
+  # Before changing this value read the documentation for this option
+  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
+  system.stateVersion = "21.11";
 }
 

hosts/by-name/servo/fs.nix

@@ -14,7 +14,7 @@
 # show zfs datasets: `zfs list` (will be empty if haven't imported)
 # show zfs properties (e.g. compression): `zfs get all pool`
 # set zfs properties: `zfs set compression=on pool`
-{ lib, pkgs, ... }:
+{ ... }:
 
 {
   # hostId: not used for anything except zfs guardrail?
@@ -54,7 +54,7 @@
     options = [ "acl" ];  #< not sure if this `acl` flag is actually necessary. it mounts without it.
   };
   # services.zfs.zed = ... # TODO: zfs can send me emails when disks fail
-  sane.programs.sysadminUtils.suggestedPrograms = [ "zfs-tools" ];
+  sane.programs.sysadminUtils.suggestedPrograms = [ "zfs" ];
 
   sane.persist.stores."ext" = {
     origin = "/mnt/pool/persist";
@@ -131,20 +131,6 @@
     the contents should be a subset of what's in ../media/datasets.
   '';
 
-  systemd.services.dedupe-media = {
-    description = "transparently de-duplicate /var/media entries by using block-level hardlinks";
-    script = ''
-      ${lib.getExe' pkgs.util-linux "hardlink"} /var/media --reflink=always --ignore-time --verbose
-    '';
-  };
-  systemd.timers.dedupe-media = {
-    wantedBy = [ "multi-user.target" ];
-    timerConfig = {
-      OnStartupSec = "23min";
-      OnUnitActiveSec = "720min";
-    };
-  };
-
   # btrfs doesn't easily support swapfiles
   # swapDevices = [
   #   { device = "/nix/persist/swapfile"; size = 4096; }

hosts/by-name/servo/net.nix

@@ -3,19 +3,9 @@
 let
   portOpts = with lib; types.submodule {
     options = {
-      visibleTo.ovpns = mkOption {
+      visibleTo.ovpn = mkOption {
         type = types.bool;
         default = false;
-        description = ''
-          whether to forward inbound traffic on the OVPN vpn port to the corresponding localhost port.
-        '';
-      };
-      visibleTo.doof = mkOption {
-        type = types.bool;
-        default = false;
-        description = ''
-          whether to forward inbound traffic on the doofnet vpn port to the corresponding localhost port.
-        '';
       };
     };
   };
@@ -23,21 +13,13 @@ in
 {
   options = with lib; {
     sane.ports.ports = mkOption {
-      # add the `visibleTo.{doof,ovpns}` options
+      # add the `visibleTo.ovpn` option
       type = types.attrsOf portOpts;
     };
   };
 
   config = {
     networking.domain = "uninsane.org";
-    systemd.network.networks."50-eth0" = {
-      matchConfig.Name = "eth0";
-      networkConfig.Address = [
-        "205.201.63.12/32"
-        "10.78.79.51/22"
-      ];
-      networkConfig.DNS = [ "10.78.79.1" ];
-    };
 
     sane.ports.openFirewall = true;
     sane.ports.openUpnp = true;
@@ -56,47 +38,37 @@ in
     #   FallbackDNS=1.1.1.1 9.9.9.9
     # '';
 
-    # tun-sea config
-    sane.dns.zones."uninsane.org".inet.A."doof.tunnel" = "205.201.63.12";
-    # sane.dns.zones."uninsane.org".inet.AAAA."doof.tunnel" = "2602:fce8:106::51";  #< TODO: enable IPv6
-    networking.wireguard.interfaces.wg-doof = {
-      privateKeyFile = config.sops.secrets.wg_doof_privkey.path;
-      # wg is active only in this namespace.
-      # run e.g. ip netns exec doof <some command like ping/curl/etc, it'll go through wg>
-      #   sudo ip netns exec doof ping www.google.com
-      interfaceNamespace = "doof";
-      ips = [
-        "205.201.63.12"
-        # "2602:fce8:106::51/128"  #< TODO: enable IPv6
-      ];
-      peers = [
-        {
-          publicKey = "nuESyYEJ3YU0hTZZgAd7iHBz1ytWBVM5PjEL1VEoTkU=";
-          # TODO: configure DNS within the doof ns and use tun-sea.doof.net endpoint
-          # endpoint = "tun-sea.doof.net:53263";
-          endpoint = "205.201.63.44:53263";
-          allowedIPs = [ "0.0.0.0/0" "::/0" ];
-          persistentKeepalive = 25; #< keep the NAT alive
-        }
-      ];
-    };
-    sane.netns.doof.hostVethIpv4 = "10.0.2.5";
-    sane.netns.doof.netnsVethIpv4 = "10.0.2.6";
-    sane.netns.doof.netnsPubIpv4 = "205.201.63.12";
-    sane.netns.doof.routeTable = 12;
-
     # OVPN CONFIG (https://www.ovpn.com):
     # DOCS: https://nixos.wiki/wiki/WireGuard
     # if you `systemctl restart wireguard-wg-ovpns`, make sure to also restart any other services in `NetworkNamespacePath = .../ovpns`.
     # TODO: why not create the namespace as a seperate operation (nix config for that?)
     networking.wireguard.enable = true;
-    networking.wireguard.interfaces.wg-ovpns = {
+    networking.wireguard.interfaces.wg-ovpns = let
+      ip = "${pkgs.iproute2}/bin/ip";
+      in-ns = "${ip} netns exec ovpns";
+      iptables = "${pkgs.iptables}/bin/iptables";
+      veth-host-ip = "10.0.1.5";
+      veth-local-ip = "10.0.1.6";
+      vpn-ip = "185.157.162.178";
+      # DNS = 46.227.67.134, 192.165.9.158, 2a07:a880:4601:10f0:cd45::1, 2001:67c:750:1:cafe:cd45::1
+      vpn-dns = "46.227.67.134";
+      bridgePort = port: proto: ''
+        ${in-ns} ${iptables} -A PREROUTING -t nat -p ${proto} --dport ${port} -m iprange --dst-range ${vpn-ip} \
+          -j DNAT --to-destination ${veth-host-ip}
+      '';
+      bridgeStatements = lib.foldlAttrs
+        (acc: port: portCfg: acc ++ (builtins.map (bridgePort port) portCfg.protocol))
+        []
+        config.sane.ports.ports;
+    in {
       privateKeyFile = config.sops.secrets.wg_ovpns_privkey.path;
       # wg is active only in this namespace.
       # run e.g. ip netns exec ovpns <some command like ping/curl/etc, it'll go through wg>
       #   sudo ip netns exec ovpns ping www.google.com
       interfaceNamespace = "ovpns";
-      ips = [ "185.157.162.178" ];
+      ips = [
+        "185.157.162.178/32"
+      ];
       peers = [
         {
           publicKey = "SkkEZDCBde22KTs/Hc7FWvDBfdOCQA4YtBEuC3n5KGs=";
@@ -114,11 +86,99 @@ in
           # dynamicEndpointRefreshRestartSeconds = 5;
         }
       ];
+      preSetup = ''
+        ${ip} netns add ovpns || (test -e /run/netns/ovpns && echo "ovpns already exists")
+      '';
+      postShutdown = ''
+        ${in-ns} ip link del ovpns-veth-b || echo "couldn't delete ovpns-veth-b"
+        ${ip} link del ovpns-veth-a || echo "couldn't delete ovpns-veth-a"
+        ${ip} netns delete ovpns || echo "couldn't delete ovpns"
+        # restore rules/routes
+        ${ip} rule del from ${veth-host-ip} lookup ovpns pref 50 || echo "couldn't delete init -> ovpns rule"
+        ${ip} route del default via ${veth-local-ip} dev ovpns-veth-a proto kernel src ${veth-host-ip} metric 1002 table ovpns || echo "couldn't delete init -> ovpns route"
+        ${ip} rule add from all lookup local pref 0
+        ${ip} rule del from all lookup local pref 100
+      '';
+      postSetup = ''
+        # DOCS:
+        # - some of this approach is described here: <https://josephmuia.ca/2018-05-16-net-namespaces-veth-nat/>
+        # - iptables primer: <https://danielmiessler.com/study/iptables/>
+        # create veth pair
+        ${ip} link add ovpns-veth-a type veth peer name ovpns-veth-b
+        ${ip} addr add ${veth-host-ip}/24 dev ovpns-veth-a
+        ${ip} link set ovpns-veth-a up
+
+        # mv veth-b into the ovpns namespace
+        ${ip} link set ovpns-veth-b netns ovpns
+        ${in-ns} ip addr add ${veth-local-ip}/24 dev ovpns-veth-b
+        ${in-ns} ip link set ovpns-veth-b up
+
+        # make it so traffic originating from the host side of the veth
+        # is sent over the veth no matter its destination.
+        ${ip} rule add from ${veth-host-ip} lookup ovpns pref 50
+        # for traffic originating at the host veth to the WAN, use the veth as our gateway
+        # not sure if the metric 1002 matters.
+        ${ip} route add default via ${veth-local-ip} dev ovpns-veth-a proto kernel src ${veth-host-ip} metric 1002 table ovpns
+        # give the default route lower priority
+        ${ip} rule add from all lookup local pref 100
+        ${ip} rule del from all lookup local pref 0
+
+        # in order to access DNS in this netns, we need to route it to the VPN's nameservers
+        # - alternatively, we could fix DNS servers like 1.1.1.1.
+        ${in-ns} ${iptables} -A OUTPUT -t nat -p udp --dport 53 -m iprange --dst-range 127.0.0.53 \
+          -j DNAT --to-destination ${vpn-dns}:53
+      '' + (lib.concatStringsSep "\n" bridgeStatements);
     };
-    sane.netns.ovpns.hostVethIpv4 = "10.0.1.5";
-    sane.netns.ovpns.netnsVethIpv4 = "10.0.1.6";
-    sane.netns.ovpns.netnsPubIpv4 = "185.157.162.178";
-    sane.netns.ovpns.routeTable = 11;
-    sane.netns.ovpns.dns = "46.227.67.134";  #< DNS requests inside the namespace are forwarded here
+
+    # create a new routing table that we can use to proxy traffic out of the root namespace
+    # through the ovpns namespace, and to the WAN via VPN.
+    networking.iproute2.rttablesExtraConfig = ''
+      5 ovpns
+    '';
+    networking.iproute2.enable = true;
+
+
+    # HURRICANE ELECTRIC CONFIG:
+    # networking.sits = {
+    #   hurricane = {
+    #     remote = "216.218.226.238";
+    #     local = "192.168.0.5";
+    #     # local = "10.0.0.5";
+    #     # remote = "10.0.0.1";
+    #     # local = "10.0.0.22";
+    #     dev = "eth0";
+    #     ttl = 255;
+    #   };
+    # };
+    # networking.interfaces."hurricane".ipv6 = {
+    #   addresses = [
+    #     # mx.uninsane.org (publically routed /64)
+    #     {
+    #       address = "2001:470:b:465::1";
+    #       prefixLength = 128;
+    #     }
+    #     # client addr
+    #     # {
+    #     #   address = "2001:470:a:466::2";
+    #     #   prefixLength = 64;
+    #     # }
+    #   ];
+    #   routes = [
+    #     {
+    #       address = "::";
+    #       prefixLength = 0;
+    #       # via = "2001:470:a:466::1";
+    #     }
+    #   ];
+    # };
+
+    # # after configuration, we want the hurricane device to look like this:
+    # # hurricane: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1480
+    # #            inet6 2001:470:a:450::2  prefixlen 64  scopeid 0x0<global>
+    # #            inet6 fe80::c0a8:16  prefixlen 64  scopeid 0x20<link>
+    # #            sit  txqueuelen 1000  (IPv6-in-IPv4)
+    # # test with:
+    # #   curl --interface hurricane http://[2607:f8b0:400a:80b::2004]
+    # #   ping 2607:f8b0:400a:80b::2004
   };
 }

hosts/by-name/servo/services/calibre.nix

@@ -0,0 +1,34 @@
+{ config, lib, ... }:
+
+let
+  cweb-cfg = config.services.calibre-web;
+  inherit (cweb-cfg) user group;
+  inherit (cweb-cfg.listen) ip port;
+  svc-dir = "/var/lib/${cweb-cfg.dataDir}";
+in
+# XXX: disabled because of runtime errors like:
+# > File "/nix/store/c7jqvx980nlg9xhxi065cba61r2ain9y-calibre-web-0.6.19/lib/python3.10/site-packages/calibreweb/cps/db.py", line 926, in speaking_language
+# > languages = self.session.query(Languages) \
+# > AttributeError: 'NoneType' object has no attribute 'query'
+lib.mkIf false
+{
+  sane.persist.sys.byStore.plaintext = [
+    { inherit user group; mode = "0700"; path = svc-dir; method = "bind"; }
+  ];
+
+  services.calibre-web.enable = true;
+  services.calibre-web.listen.ip = "127.0.0.1";
+  # XXX: externally populate `${svc-dir}/metadata.db` (once) from
+  #   <https://github.com/janeczku/calibre-web/blob/master/library/metadata.db>
+  # i don't know why you have to do this??
+  # services.calibre-web.options.calibreLibrary = svc-dir;
+
+  services.nginx.virtualHosts."calibre.uninsane.org" = {
+    addSSL = true;
+    enableACME = true;
+    locations."/" = {
+      proxyPass = "http://${ip}:${builtins.toString port}";
+    };
+  };
+  sane.dns.zones."uninsane.org".inet.CNAME."calibre" = "native";
+}

hosts/by-name/servo/services/coturn.nix

@@ -24,65 +24,50 @@
 #   that is NOT the case when the STUN server and client A are on the same LAN
 #   even if client A contacts the STUN server via its WAN address with port reflection enabled.
 #   hence, there's no obvious way to put the STUN server on the same LAN as either client and expect the rest to work.
-# - there an old version which *half worked*, which is:
-#   - run the turn server in the root namespace.
-#   - bind the turn server to the veth connecting it to the VPN namespace (so it sends outgoing traffic to the right place).
-#   - NAT the turn port range from VPN into root namespace (so it receives incomming traffic).
-#   - this approach would fail the prosody conversations.im check, but i didn't notice *obvious* call routing errors.
-#
-# debugging:
-# - log messages like 'usage: realm=<turn.uninsane.org>, username=<1715915193>, rp=14, rb=1516, sp=8, sb=684'
-#   - rp = received packets
-#   - rb = received bytes
-#   - sp = sent packets
-#   - sb = sent bytes
-
-{ config, lib, ... }:
+{ lib, ... }:
 let
-  # TURN port range (inclusive).
-  # default coturn behavior is to use the upper quarter of all ports. i.e. 49152 - 65535.
-  # i believe TURN allocations expire after either 5 or 10 minutes of inactivity.
-  turnPortLow = 49152; # 49152 = 0xc000
-  turnPortHigh = turnPortLow + 256;
+  # TODO: this range could be larger, but right now that's costly because each element is its own UPnP forward
+  # TURN port range (inclusive)
+  turnPortLow = 49152;
+  turnPortHigh = 49167;
   turnPortRange = lib.range turnPortLow turnPortHigh;
 in
 {
-  # the port definitions are only needed if running in the root net namespace
-  # sane.ports.ports = lib.mkMerge ([
-  #   {
-  #     "3478" = {
-  #       # this is the "control" port.
-  #       # i.e. no client data is forwarded through it, but it's where clients request tunnels.
-  #       protocol = [ "tcp" "udp" ];
-  #       # visibleTo.lan = true;
-  #       # visibleTo.wan = true;
-  #       visibleTo.ovpns = true;  # forward traffic from the VPN to the root NS
-  #       description = "colin-stun-turn";
-  #     };
-  #     "5349" = {
-  #       # the other port 3478 also supports TLS/DTLS, but presumably clients wanting TLS will default 5349
-  #       protocol = [ "tcp" ];
-  #       # visibleTo.lan = true;
-  #       # visibleTo.wan = true;
-  #       visibleTo.ovpns = true;
-  #       description = "colin-stun-turn-over-tls";
-  #     };
-  #   }
-  # ] ++ (builtins.map
-  #   (port: {
-  #     "${builtins.toString port}" = let
-  #       count = port - turnPortLow + 1;
-  #       numPorts = turnPortHigh - turnPortLow + 1;
-  #     in {
-  #       protocol = [ "tcp" "udp" ];
-  #       # visibleTo.lan = true;
-  #       # visibleTo.wan = true;
-  #       visibleTo.ovpns = true;
-  #       description = "colin-turn-${builtins.toString count}-of-${builtins.toString numPorts}";
-  #     };
-  #   })
-  #   turnPortRange
-  # ));
+  sane.ports.ports = lib.mkMerge ([
+    {
+      "3478" = {
+        # this is the "control" port.
+        # i.e. no client data is forwarded through it, but it's where clients request tunnels.
+        protocol = [ "tcp" "udp" ];
+        # visibleTo.lan = true;
+        # visibleTo.wan = true;
+        visibleTo.ovpn = true;
+        description = "colin-stun-turn";
+      };
+      "5349" = {
+        # the other port 3478 also supports TLS/DTLS, but presumably clients wanting TLS will default 5349
+        protocol = [ "tcp" ];
+        # visibleTo.lan = true;
+        # visibleTo.wan = true;
+        visibleTo.ovpn = true;
+        description = "colin-stun-turn-over-tls";
+      };
+    }
+  ] ++ (builtins.map
+    (port: {
+      "${builtins.toString port}" = let
+        count = port - turnPortLow + 1;
+        numPorts = turnPortHigh - turnPortLow + 1;
+      in {
+        protocol = [ "tcp" "udp" ];
+        # visibleTo.lan = true;
+        # visibleTo.wan = true;
+        visibleTo.ovpn = true;
+        description = "colin-turn-${builtins.toString count}-of-${builtins.toString numPorts}";
+      };
+    })
+    turnPortRange
+  ));
 
   services.nginx.virtualHosts."turn.uninsane.org" = {
     # allow ACME to procure a cert via nginx for this domain
@@ -118,28 +103,22 @@ in
   services.coturn.realm = "turn.uninsane.org";
   services.coturn.cert = "/var/lib/acme/turn.uninsane.org/fullchain.pem";
   services.coturn.pkey = "/var/lib/acme/turn.uninsane.org/key.pem";
-
-  #v disable to allow unauthenticated access (or set `services.coturn.no-auth = true`)
   services.coturn.use-auth-secret = true;
   services.coturn.static-auth-secret-file = "/var/lib/coturn/shared_secret.bin";
-  services.coturn.lt-cred-mech = true; #< XXX: use-auth-secret overrides lt-cred-mech
-
+  services.coturn.lt-cred-mech = true;
   services.coturn.min-port = turnPortLow;
   services.coturn.max-port = turnPortHigh;
   # services.coturn.secure-stun = true;
   services.coturn.extraConfig = lib.concatStringsSep "\n" [
     "verbose"
-    # "Verbose"  #< even MORE verbosity than "verbose"  (it's TOO MUCH verbosity really)
-    "no-multicast-peers"  # disables sending to IPv4 broadcast addresses (e.g. 224.0.0.0/3)
-    # "listening-ip=${config.sane.netns.ovpns.hostVethIpv4}" "external-ip=${config.sane.netns.ovpns.netnsPubIpv4}"  #< 2024/04/25: works, if running in root namespace
-    "listening-ip=${config.sane.netns.ovpns.netnsPubIpv4}" "external-ip=${config.sane.netns.ovpns.netnsPubIpv4}"
-
-    # old attempts:
-    # "external-ip=${config.sane.netns.ovpns.netnsPubIpv4}/${config.sane.netns.ovpns.hostVethIpv4}"
+    # "Verbose"  #< even MORE verbosity than "verbose"
+    # "no-multicast-peers"  # disables sending to IPv4 broadcast addresses (e.g. 224.0.0.0/3)
+    "listening-ip=10.0.1.5"
+    # "external-ip=185.157.162.178/10.0.1.5"
+    "external-ip=185.157.162.178"
     # "listening-ip=10.78.79.51"  # can be specified multiple times; omit for *
     # "external-ip=97.113.128.229/10.78.79.51"
     # "external-ip=97.113.128.229"
     # "mobility"  # "mobility with ICE (MICE) specs support" (?)
   ];
-  systemd.services.coturn.serviceConfig.NetworkNamespacePath = "/run/netns/ovpns";
 }

hosts/by-name/servo/services/cryptocurrencies/bitcoin.nix

@@ -16,17 +16,14 @@
 # - validate with `bitcoin-cli -netinfo`
 { config, lib, pkgs, sane-lib, ... }:
 let
-  # bitcoind = config.sane.programs.bitcoind.packageUnwrapped;
-  bitcoind = pkgs.bitcoind;
   # wrapper to run bitcoind with the tor onion address as externalip (computed at runtime)
-  _bitcoindWithExternalIp = pkgs.writeShellScriptBin "bitcoind" ''
-    set -xeu
+  _bitcoindWithExternalIp = with pkgs; writeShellScriptBin "bitcoind" ''
     externalip="$(cat /var/lib/tor/onion/bitcoind/hostname)"
     exec ${bitcoind}/bin/bitcoind "-externalip=$externalip" "$@"
   '';
   # the package i provide to services.bitcoind ends up on system PATH, and used by other tools like clightning.
   # therefore, even though services.bitcoind only needs `bitcoind` binary, provide all the other bitcoin-related binaries (notably `bitcoin-cli`) as well:
-  bitcoindWithExternalIp = pkgs.symlinkJoin {
+  bitcoindWithExternalIp = with pkgs; symlinkJoin {
     name = "bitcoind-with-external-ip";
     paths = [ _bitcoindWithExternalIp bitcoind ];
   };
@@ -64,62 +61,23 @@ in
       passwordHMAC = "30002c05d82daa210550e17a182db3f3$6071444151281e1aa8a2729f75e3e2d224e9d7cac3974810dab60e7c28ffaae4";
     };
     extraConfig = ''
-      # checkblocks: default 6: how many blocks to verify on start
-      checkblocks=3
       # don't load the wallet, and disable wallet RPC calls
       disablewallet=1
       # proxy all outbound traffic through Tor
       proxy=127.0.0.1:9050
     '';
-    extraCmdlineOptions = [
-      # "-debug"
-      # "-debug=estimatefee"
-      # "-debug=http"
-      # "-debug=net"
-      "-debug=proxy"
-      "-debug=rpc"
-      # "-debug=validation"
-    ];
   };
 
   users.users.bitcoind-mainnet.extraGroups = [ "tor" ];
 
-  systemd.services.bitcoind-mainnet = {
-    after = [ "tor.service" ];
-    requires = [ "tor.service" ];
-    serviceConfig.RestartSec = "30s";  #< default is 0
-
-    # hardening (systemd-analyze security bitcoind-mainnet)
-    serviceConfig.StateDirectory = "bitcoind-mainnet";
-    serviceConfig.LockPersonality = true;
-    serviceConfig.MemoryDenyWriteExecute = "true";
-    serviceConfig.NoNewPrivileges = "true";
-    serviceConfig.PrivateDevices = "true";
-    serviceConfig.PrivateMounts = true;
-    serviceConfig.PrivateTmp = "true";
-    serviceConfig.PrivateUsers = true;
-    serviceConfig.ProcSubset = "pid";
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;
-    serviceConfig.ProtectHostname = true;
-    serviceConfig.ProtectKernelLogs = true;
-    serviceConfig.ProtectKernelModules = true;
-    serviceConfig.ProtectKernelTunables = true;
-    serviceConfig.ProtectProc = "invisible";
-    serviceConfig.ProtectSystem = lib.mkForce "strict";
-    serviceConfig.RemoveIPC = true;
-    serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
-    serviceConfig.RestrictNamespaces = true;
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";
-    serviceConfig.SystemCallFilter = [ "@system-service" ];
-  };
+  systemd.services.bitcoind-mainnet.serviceConfig.RestartSec = "30s";  #< default is 0
 
+  sane.users.colin.fs.".bitcoin/bitcoin.conf" = sane-lib.fs.wantedSymlinkTo config.sops.secrets."bitcoin.conf".path;
   sops.secrets."bitcoin.conf" = {
     mode = "0600";
     owner = "colin";
     group = "users";
   };
 
-  sane.programs.bitcoin-cli.enableFor.user.colin = true;  # for debugging/administration: `bitcoin-cli`
+  sane.programs.bitcoind.enableFor.user.colin = true;  # for debugging/administration: `bitcoin-cli`
 }

hosts/by-name/servo/services/cryptocurrencies/clightning-sane/clightning-sane

@@ -1,21 +1,5 @@
 #!/usr/bin/env nix-shell
-#!nix-shell -i python3 -p pyln-client -p python3
-
-"""
-clightning-sane: helper to perform common Lightning node admin operations:
-- view channel balances
-- rebalance channels
-
-COMMON OPERATIONS:
-- view channel balances: `clightning-sane status`
-- rebalance channels to improve routability (without paying any fees): `clightning-sane autobalance`
-
-FULL OPERATION:
-- `clightning-sane status --full`
-  - `P$`: represents how many msats i've captured in fees from this channel.
-  - `COST`: rough measure of how much it's "costing" me to let my channel partner hold funds on his side of the channel.
-            this is based on the notion that i only capture fees from outbound transactions, and so the channel partner holding all liquidity means i can't capture fees on that liquidity.
-"""
+#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.pyln-client ])"
 
 # pyln-client docs: <https://github.com/ElementsProject/lightning/tree/master/contrib/pyln-client>
 # terminology:
@@ -742,7 +726,7 @@ def main():
     logging.basicConfig()
     logger.setLevel(logging.INFO)
 
-    parser = argparse.ArgumentParser(usage=__doc__)
+    parser = argparse.ArgumentParser(description="rebalance lightning channel balances")
     parser.add_argument("--verbose", action="store_true", help="more logging")
     parser.add_argument("--min-msat", default="999", help="min transaction size")
     parser.add_argument("--max-msat", default="1000000", help="max transaction size")

hosts/by-name/servo/services/cryptocurrencies/clightning.nix

@@ -72,11 +72,13 @@
 
 { config, pkgs, ... }:
 {
-  sane.persist.sys.byStore.private = [
-    # clightning takes up only a few MB. but then several hundred MB of crash logs that i should probably GC.
+  sane.persist.sys.byStore.ext = [
     { user = "clightning"; group = "clightning"; mode = "0710"; path = "/var/lib/clightning"; method = "bind"; }
   ];
 
+  # `lightning-cli` finds its RPC file via `~/.lightning/bitcoin/lightning-rpc`, to message the daemon
+  sane.user.fs.".lightning".symlink.target = "/var/lib/clightning";
+
   # see bitcoin.nix for how to generate this
   services.bitcoind.mainnet.rpc.users.clightning.passwordHMAC =
     "befcb82d9821049164db5217beb85439$2c31ac7db3124612e43893ae13b9527dbe464ab2d992e814602e7cb07dc28985";
@@ -103,7 +105,6 @@
   users.users.clightning.extraGroups = [ "tor" ];
 
   systemd.services.clightning.after = [ "tor.service" ];
-  systemd.services.clightning.requires = [ "tor.service" ];
 
   # lightning-config contains fields from here:
   # - <https://docs.corelightning.org/docs/configuration>
@@ -115,16 +116,11 @@
   # - fee-per-satoshi=<ppm>
   # - feature configs (i.e. experimental-xyz options)
   sane.services.clightning.extraConfig = ''
-    # log levels: "io", "debug", "info", "unusual", "broken"
-    log-level=info
-    # log-level=info:lightningd
-    # log-level=debug:lightningd
-    # log-level=debug
-
+    log-level=debug:lightningd
     # peerswap:
     # - config example: <https://github.com/fort-nix/nix-bitcoin/pull/462/files#diff-b357d832705b8ce8df1f41934d613f79adb77c4cd5cd9e9eb12a163fca3e16c6>
     # XXX: peerswap crashes clightning on launch. stacktrace is useless.
-    # plugin={pkgs.peerswap}/bin/peerswap
+    # plugin=${pkgs.peerswap}/bin/peerswap
     # peerswap-db-path=/var/lib/clightning/peerswap/swaps
     # peerswap-policy-path=...
   '';
@@ -135,5 +131,5 @@
     group = "clightning";
   };
 
-  sane.programs.lightning-cli.enableFor.user.colin = true;  # for debugging/admin:
+  sane.programs.clightning.enableFor.user.colin = true;  # for debugging/admin: `lightning-cli`
 }

hosts/by-name/servo/services/cryptocurrencies/i2p.nix

@@ -1,5 +1,4 @@
-{ lib, ... }:
-lib.mkIf false  #< 2024/07/27: i don't use it, too much surface-area for me to run it pro-bono (`systemd-analyze security monero`)
+{ ... }:
 {
   services.i2p.enable = true;
 }

hosts/by-name/servo/services/cryptocurrencies/monero.nix

@@ -1,6 +1,5 @@
 # as of 2023/11/26: complete downloaded blockchain should be 200GiB on disk, give or take.
-{ lib, ... }:
-lib.mkIf false  #< 2024/07/27: i don't use it, too much surface-area for me to run it pro-bono (`systemd-analyze security monero`)
+{ ... }:
 {
   sane.persist.sys.byStore.ext = [
     # /var/lib/monero/lmdb is what consumes most of the space

hosts/by-name/servo/services/cryptocurrencies/tor.nix

@@ -1,10 +1,10 @@
 # tor settings: <https://2019.www.torproject.org/docs/tor-manual.html.en>
 { lib, ... }:
 {
-  sane.persist.sys.byStore.ephemeral = [
-    # N.B.: tor hidden service hostnames aren't deterministic, so if you need them
-    # to be preserved across reboots then persist /var/lib/tor/onion in "private" store.
-   { user = "tor"; group = "tor"; mode = "0710"; path = "/var/lib/tor"; method = "bind"; }
+  # tor hidden service hostnames aren't deterministic, so persist.
+  # might be able to get away with just persisting /var/lib/tor/onion, not sure.
+  sane.persist.sys.byStore.plaintext = [
+    { user = "tor"; group = "tor"; mode = "0710"; path = "/var/lib/tor"; method = "bind"; }
   ];
 
   # tor: `tor.enable` doesn't start a relay, exit node, proxy, etc. it's minimal.

hosts/by-name/servo/services/default.nix

@@ -1,6 +1,7 @@
 { ... }:
 {
   imports = [
+    ./calibre.nix
     ./coturn.nix
     ./cryptocurrencies
     ./email
@@ -10,7 +11,7 @@
     ./gitea.nix
     ./goaccess.nix
     ./ipfs.nix
-    ./jackett
+    ./jackett.nix
     ./jellyfin.nix
     ./kiwix-serve.nix
     ./komga.nix
@@ -19,13 +20,14 @@
     ./navidrome.nix
     ./nginx.nix
     ./nixos-prebuild.nix
+    ./nixserve.nix
     ./ntfy
     ./pict-rs.nix
     ./pleroma.nix
     ./postgres.nix
     ./prosody
     ./slskd.nix
-    ./transmission
+    ./transmission.nix
     ./trust-dns.nix
     ./wikipedia.nix
   ];

hosts/by-name/servo/services/ejabberd.nix

@@ -44,61 +44,61 @@ in
 # everything configured below was fine: used ejabberd for several months.
 lib.mkIf false
 {
-  sane.persist.sys.byStore.private = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "ejabberd"; group = "ejabberd"; path = "/var/lib/ejabberd"; method = "bind"; }
   ];
   sane.ports.ports = lib.mkMerge ([
     {
       "3478" = {
         protocol = [ "tcp" "udp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-xmpp-stun-turn";
       };
       "5222" = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-xmpp-client-to-server";
       };
       "5223" = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-xmpps-client-to-server";  # XMPP over TLS
       };
       "5269" = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
+        visibleTo.wan = true;
         description = "colin-xmpp-server-to-server";
       };
       "5270" = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
+        visibleTo.wan = true;
         description = "colin-xmpps-server-to-server";  # XMPP over TLS
       };
       "5280" = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-xmpp-bosh";
       };
       "5281" = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-xmpp-bosh-https";
       };
       "5349" = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-xmpp-stun-turn-over-tls";
       };
       "5443" = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-xmpp-web-services";  # file uploads, websockets, admin
       };
     }
@@ -109,8 +109,8 @@ lib.mkIf false
         numPorts = turnPortHigh - turnPortLow + 1;
       in {
         protocol = [ "tcp" "udp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-xmpp-turn-${builtins.toString count}-of-${builtins.toString numPorts}";
       };
     })

hosts/by-name/servo/services/email/dovecot.nix

@@ -8,14 +8,14 @@
 {
   sane.ports.ports."143" = {
     protocol = [ "tcp" ];
-    visibleTo.doof = true;
     visibleTo.lan = true;
+    visibleTo.wan = true;
     description = "colin-imap-imap.uninsane.org";
   };
   sane.ports.ports."993" = {
     protocol = [ "tcp" ];
-    visibleTo.doof = true;
     visibleTo.lan = true;
+    visibleTo.wan = true;
     description = "colin-imaps-imap.uninsane.org";
   };
 
@@ -83,8 +83,8 @@
     #   sieve_plugins = sieve_imapsieve
     # }
 
-    # mail_debug = yes
-    # auth_debug = yes
+    mail_debug = yes
+    auth_debug = yes
     # verbose_ssl = yes
   '';
 

hosts/by-name/servo/services/email/postfix.nix

@@ -1,6 +1,6 @@
 # postfix config options: <https://www.postfix.org/postconf.5.html>
 
-{ config, lib, pkgs, ... }:
+{ lib, pkgs, ... }:
 
 let
   submissionOptions = {
@@ -18,10 +18,10 @@ let
   };
 in
 {
-  sane.persist.sys.byStore.private = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
     { user = "opendkim"; group = "opendkim"; path = "/var/lib/opendkim"; method = "bind"; }
-    { user = "root"; group = "root"; path = "/var/lib/postfix"; method = "bind"; }  #< probably not *all* of postfix needs to actually be persisted (e.g. not the conf dir)
+    { user = "root"; group = "root"; path = "/var/lib/postfix"; method = "bind"; }
     { user = "root"; group = "root"; path = "/var/spool/mail"; method = "bind"; }
     # *probably* don't need these dirs:
     # "/var/lib/dhparams"          # https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/security/dhparams.nix
@@ -56,7 +56,8 @@ in
 
   sane.dns.zones."uninsane.org".inet = {
     MX."@" = "10 mx.uninsane.org.";
-    A."mx" = "%AOVPNS%"; #< XXX: RFC's specify that the MX record CANNOT BE A CNAME. TODO: use "%AOVPNS%?
+    # XXX: RFC's specify that the MX record CANNOT BE A CNAME
+    A."mx" = "185.157.162.178";
 
     # Sender Policy Framework:
     #   +mx     => mail passes if it originated from the MX

hosts/by-name/servo/services/export/default.nix

@@ -2,7 +2,7 @@
 {
   imports = [
     ./nfs.nix
-    ./sftpgo
+    ./sftpgo.nix
   ];
 
   users.groups.export = {};
@@ -37,8 +37,7 @@
     wantedBy = [ "nfs.service" "sftpgo.service" ];
     file.text = ''
       - media/         read-only:  Videos, Music, Books, etc
-      - playground/    read-write: use it to share files with other users of this server, inaccessible from the www
-      - pub/           read-only:  content made to be shared with the www
+      - playground/    read-write: use it to share files with other users of this server
     '';
   };
 
@@ -51,11 +50,4 @@
       - be a friendly troll
     '';
   };
-
-  sane.fs."/var/export/.public_for_test/test" = {
-    wantedBy = [ "nfs.service" "sftpgo.service" ];
-    file.text = ''
-      automated tests read this file to probe connectivity
-    '';
-  };
 }

hosts/by-name/servo/services/export/nfs.nix

@@ -15,7 +15,6 @@
 # - could maybe be done with some mount option?
 
 { config, lib, ... }:
-lib.mkIf false  #< TODO: remove nfs altogether! it's not exactly the most secure
 {
   services.nfs.server.enable = true;
 

hosts/by-name/servo/services/export/sftpgo.nix

@@ -9,68 +9,40 @@
 
 { config, lib, pkgs, sane-lib, ... }:
 let
-  external_auth_hook = pkgs.static-nix-shell.mkPython3 {
-    pname = "external_auth_hook";
+  sftpgo_external_auth_hook = pkgs.static-nix-shell.mkPython3Bin {
+    pname = "sftpgo_external_auth_hook";
     srcRoot = ./.;
-    pkgs = [ "python3.pkgs.passlib" ];
   };
+in
+{
   # Client initiates a FTP "control connection" on port 21.
   # - this handles the client -> server commands, and the server -> client status, but not the actual data
   # - file data, directory listings, etc need to be transferred on an ephemeral "data port".
   # - 50000-50100 is a common port range for this.
   #   50000 is used by soulseek.
-  passiveStart = 50050;
-  passiveEnd   = 50070;
-in
-{
   sane.ports.ports = {
     "21" = {
       protocol = [ "tcp" ];
       visibleTo.lan = true;
+      visibleTo.wan = true;
       description = "colin-FTP server";
     };
-    "990" = {
-      protocol = [ "tcp" ];
-      visibleTo.doof = true;
-      visibleTo.lan = true;
-      description = "colin-FTPS server";
-    };
   } // (sane-lib.mapToAttrs
     (port: {
       name = builtins.toString port;
       value = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-FTP server data port range";
       };
     })
-    (lib.range passiveStart passiveEnd)
+    (lib.range 50050 50100)
   );
 
-  # use nginx/acme to produce a cert for FTPS
-  services.nginx.virtualHosts."ftp.uninsane.org" = {
-    addSSL = true;
-    enableACME = true;
-  };
-  sane.dns.zones."uninsane.org".inet.CNAME."ftp" = "native";
-
   services.sftpgo = {
     enable = true;
     group = "export";
-
-    package = pkgs.sftpgo.overrideAttrs (upstream: {
-      patches = (upstream.patches or []) ++ [
-        # fix for compatibility with kodi:
-        # ftp LIST operation returns entries over-the-wire like:
-        # - dgrwxrwxr-x 1 ftp ftp            9 Apr  9 15:05 Videos
-        # however not all clients understand all mode bits (like that `g`, indicating SGID / group sticky bit).
-        # instead, only send mode bits which are well-understood.
-        # the full set of bits, from which i filter, is found here: <https://pkg.go.dev/io/fs#FileMode>
-        ./safe_fileinfo.patch
-      ];
-    });
-
     settings = {
       ftpd = {
         bindings = [
@@ -80,68 +52,37 @@ in
             port = 21;
             debug = true;
           }
-          {
-            # binding this means any wireguard client can connect
-            address = "10.0.10.5";
-            port = 990;
-            debug = true;
-            tls_mode = 2;  # 2 = "implicit FTPS": client negotiates TLS before any FTP command.
-          }
           {
             # binding this means any LAN client can connect (also WAN traffic forwarded from the gateway)
             address = "10.78.79.51";
             port = 21;
             debug = true;
           }
-          {
-            # binding this means any LAN client can connect (also WAN traffic forwarded from the gateway)
-            address = "10.78.79.51";
-            port = 990;
-            debug = true;
-            tls_mode = 2;  # 2 = "implicit FTPS": client negotiates TLS before any FTP command.
-          }
-          {
-            # binding this means any doof client can connect (TLS only)
-            address = config.sane.netns.doof.hostVethIpv4;
-            port = 990;
-            debug = true;
-            tls_mode = 2;  # 2 = "implicit FTPS": client negotiates TLS before any FTP command.
-          }
-          {
-            # binding this means any LAN client can connect via `ftp.uninsane.org` (TLS only)
-            address = config.sane.netns.doof.netnsPubIpv4;
-            port = 990;
-            debug = true;
-            tls_mode = 2;  # 2 = "implicit FTPS": client negotiates TLS before any FTP command.
-          }
         ];
 
         # active mode is susceptible to "bounce attacks", without much benefit over passive mode
         disable_active_mode = true;
         hash_support = true;
         passive_port_range = {
-          start = passiveStart;
-          end = passiveEnd;
+          start = 50050;
+          end = 50100;
         };
 
-        certificate_file = "/var/lib/acme/ftp.uninsane.org/full.pem";
-        certificate_key_file = "/var/lib/acme/ftp.uninsane.org/key.pem";
-
         banner = ''
           Welcome, friends, to Colin's FTP server! Also available via NFS on the same host, but LAN-only.
 
-          Read-only access (LAN clients see everything; WAN clients can only see /pub):
+          Read-only access (LAN-restricted):
           Username: "anonymous"
           Password: "anonymous"
 
-          CONFIGURE YOUR CLIENT FOR "PASSIVE" MODE, e.g. `ftp --passive ftp.uninsane.org`.
+          CONFIGURE YOUR CLIENT FOR "PASSIVE" mode, e.g. `ftp --passive uninsane.org`.
           Please let me know if anything's broken or not as it should be. Otherwise, browse and transfer freely :)
         '';
 
       };
       data_provider = {
         driver = "memory";
-        external_auth_hook = "${external_auth_hook}/bin/external_auth_hook";
+        external_auth_hook = "${sftpgo_external_auth_hook}/bin/sftpgo_external_auth_hook";
         # track_quota:
         # - 0: disable quota tracking
         # - 1: quota is updated on every upload/delete, even if user has no quota restriction
@@ -154,7 +95,6 @@ in
   users.users.sftpgo.extraGroups = [
     "export"
     "media"
-    "nginx"  # to access certs
   ];
 
   systemd.services.sftpgo = {

hosts/by-name/servo/services/export/sftpgo/safe_fileinfo.patch

@@ -1,32 +0,0 @@
-diff --git a/internal/ftpd/handler.go b/internal/ftpd/handler.go
-index 036c3977..33211261 100644
---- a/internal/ftpd/handler.go
-+++ b/internal/ftpd/handler.go
-@@ -169,7 +169,7 @@ func (c *Connection) Stat(name string) (os.FileInfo, error) {
- 		}
- 		return nil, err
- 	}
--	return fi, nil
-+	return vfs.NewFileInfo(name, fi.IsDir(), fi.Size(), fi.ModTime(), false), nil
- }
-
- // Name returns the name of this connection
-@@ -315,7 +315,17 @@ func (c *Connection) ReadDir(name string) (ftpserver.DirLister, error) {
- 		}, nil
- 	}
-
--	return c.ListDir(name)
-+	lister, err := c.ListDir(name)
-+	if err != nil {
-+		return nil, err
-+	}
-+	return &patternDirLister{
-+		DirLister:      lister,
-+		pattern:        "*",
-+		lastCommand:    c.clientContext.GetLastCommand(),
-+		dirName:        name,
-+		connectionPath: c.clientContext.Path(),
-+	}, nil
- }
-
- // GetHandle implements ClientDriverExtentionFileTransfer

hosts/by-name/servo/services/export/sftpgo_external_auth_hook

@@ -1,5 +1,5 @@
 #!/usr/bin/env nix-shell
-#!nix-shell -i python3 -p python3 -p python3.pkgs.passlib
+#!nix-shell -i python3 -p "python3.withPackages (ps: [  ])"
 # vim: set filetype=python :
 #
 # available environment variables:
@@ -37,16 +37,14 @@
 # - it seems (empirically) that a user can't cd above their home directory.
 #   though i don't have a reference for that in the docs.
 
+import crypt
 import json
 import os
-import passlib.hosts
 
 from hmac import compare_digest
 
 authFail = dict(username="")
 
-PERM_DENY = []
-PERM_LIST = [ "list" ]
 PERM_RO = [ "list", "download" ]
 PERM_RW = [
   # read-only:
@@ -114,8 +112,10 @@ def isWireguard(ip: str) -> bool:
 
 def isTrustedCred(password: str) -> bool:
   for cred in TRUSTED_CREDS:
-    if passlib.hosts.linux_context.verify(password, cred):
-      return True
+    _, method, salt, hash_ = cred.split("$")
+    # assert method == "6", f"unrecognized crypt entry: {cred}"
+    if crypt.crypt(password, f"${method}${salt}") == cred:
+        return True
 
   return False
 
@@ -129,14 +129,12 @@ def getAuthResponse(ip: str, username: str, password: str) -> dict:
     return mkAuthOk(username, permissions = {
       "/": PERM_RW,
       "/playground": PERM_RW,
-      "/.public_for_test": PERM_RO,
     })
   if isWireguard(ip):
     # allow any user from wireguard
     return mkAuthOk(username, permissions = {
       "/": PERM_RW,
       "/playground": PERM_RW,
-      "/.public_for_test": PERM_RO,
     })
   if isLan(ip):
     if username == "anonymous":
@@ -144,19 +142,7 @@ def getAuthResponse(ip: str, username: str, password: str) -> dict:
       return mkAuthOk("anonymous", permissions = {
         "/": PERM_RO,
         "/playground": PERM_RW,
-        "/.public_for_test": PERM_RO,
       })
-  if username == "anonymous":
-    # anonymous users from the www can have even more limited access.
-    # mostly because i need an easy way to test WAN connectivity :-)
-    return mkAuthOk("anonymous", permissions = {
-      # "/": PERM_DENY,
-      "/": PERM_LIST,  #< REQUIRED, even for lftp to list a subdir
-      "/media": PERM_DENY,
-      "/playground": PERM_DENY,
-      "/.public_for_test": PERM_RO,
-      # "/README.md": PERM_RO,  #< does not work
-    })
 
   return authFail
 

hosts/by-name/servo/services/freshrss.nix

@@ -10,7 +10,6 @@
 # ```
 
 { config, lib, pkgs, sane-lib, ... }:
-lib.mkIf false  #< 2024/07/04: i haven't actively used this for months
 {
   sops.secrets."freshrss_passwd" = {
     owner = config.users.users.freshrss.name;

hosts/by-name/servo/services/gitea.nix

@@ -2,8 +2,9 @@
 { config, pkgs, lib, ... }:
 
 {
-  sane.persist.sys.byStore.private = [
-    { user = "git"; group = "gitea"; mode = "0750"; path = "/var/lib/gitea"; method = "bind"; }
+  sane.persist.sys.byStore.plaintext = [
+    # TODO: mode? could be more granular
+    { user = "git"; group = "gitea"; path = "/var/lib/gitea"; method = "bind"; }
   ];
   services.gitea.enable = true;
   services.gitea.user = "git";  # default is 'gitea'
@@ -37,34 +38,28 @@
       ROOT_URL = "https://git.uninsane.org/";
     };
     service = {
-      # timeout for email approval. 5760 = 4 days. 10080 = 7 days
-      ACTIVE_CODE_LIVE_MINUTES = 10080;
+      # timeout for email approval. 5760 = 4 days
+      ACTIVE_CODE_LIVE_MINUTES = 5760;
       # REGISTER_EMAIL_CONFIRM = false;
       # REGISTER_MANUAL_CONFIRM = true;
       REGISTER_EMAIL_CONFIRM = true;
-      # not sure what this notifies *on*...
+      # not sure what this notified on?
       ENABLE_NOTIFY_MAIL = true;
       # defaults to image-based captcha.
       # also supports recaptcha (with custom URLs) or hCaptcha.
       ENABLE_CAPTCHA = true;
       NOREPLY_ADDRESS = "noreply.anonymous.git@uninsane.org";
     };
-    session = {
-      COOKIE_SECURE = true;
-      # keep me logged in for 30 days
-      SESSION_LIFE_TIME = 60 * 60 * 24 * 30;
-    };
+    session.COOKIE_SECURE = true;
     repository = {
       DEFAULT_BRANCH = "master";
-      ENABLE_PUSH_CREATE_USER = true;
-      ENABLE_PUSH_CREATE_ORG = true;
     };
     other = {
       SHOW_FOOTER_TEMPLATE_LOAD_TIME = false;
     };
     ui = {
-      # options: "gitea-auto" (adapt to system theme), "gitea-dark", "gitea-light"
-      # DEFAULT_THEME = "gitea-auto";
+      # options: "auto", "gitea", "arc-green"
+      DEFAULT_THEME = "arc-green";
       # cache frontend assets if true
       # USE_SERVICE_WORKER = true;
     };
@@ -73,10 +68,9 @@
       # alternative is to use nixos-level config:
       # services.gitea.mailerPasswordFile = ...
       ENABLED = true;
+      MAILER_TYPE = "sendmail";
       FROM = "notify.git@uninsane.org";
-      PROTOCOL = "sendmail";
       SENDMAIL_PATH = "${pkgs.postfix}/bin/sendmail";
-      SENDMAIL_ARGS = "--";  # most "sendmail" programs take options, "--" will prevent an email address being interpreted as an option.
     };
     time = {
       # options: ANSIC, UnixDate, RubyDate, RFC822, RFC822Z, RFC850, RFC1123, RFC1123Z, RFC3339, RFC3339Nano, Kitchen, Stamp, StampMilli, StampMicro, StampNano
@@ -96,8 +90,6 @@
     ];
   };
 
-  services.openssh.settings.UsePAM = true;  #< required for `git` user to authenticate
-
   # hosted git (web view and for `git <cmd>` use
   # TODO: enable publog?
   services.nginx.virtualHosts."git.uninsane.org" = {
@@ -108,10 +100,6 @@
     locations."/" = {
       proxyPass = "http://127.0.0.1:3000";
     };
-    # fuck you @anthropic
-    locations."= /robots.txt".extraConfig = ''
-      return 200 "User-agent: *\nDisallow: /\n";
-    '';
     # gitea serves all `raw` files as content-type: plain, but i'd like to serve them as their actual content type.
     # or at least, enough to make specific pages viewable (serving unoriginal content as arbitrary content type is dangerous).
     locations."~ ^/colin/phone-case-cq/raw/.*.html" = {
@@ -137,7 +125,7 @@
   sane.ports.ports."22" = {
     protocol = [ "tcp" ];
     visibleTo.lan = true;
-    visibleTo.doof = true;
+    visibleTo.wan = true;
     description = "colin-git@git.uninsane.org";
   };
 }

hosts/by-name/servo/services/goaccess.nix

@@ -20,7 +20,7 @@
           --ignore-panel=HOSTS \
           --ws-url=wss://sink.uninsane.org:443/ws \
           --port=7890 \
-          -o /var/lib/goaccess/index.html
+          -o /var/lib/uninsane/sink/index.html
       '';
       ExecReload = "${pkgs.coreutils}/bin/kill -HUP $MAINPID";
       Type = "simple";
@@ -28,19 +28,17 @@
       RestartSec = "10s";
 
       # hardening
-      # TODO: run as `goaccess` user and add `goaccess` user to group `nginx`.
+      WorkingDirectory = "/tmp";
       NoNewPrivileges = true;
-      PrivateDevices = "yes";
       PrivateTmp = true;
       ProtectHome = "read-only";
+      ProtectSystem = "strict";
+      SystemCallFilter = "~@clock @cpu-emulation @debug @keyring @memlock @module @mount @obsolete @privileged @reboot @resources @setuid @swap @raw-io";
+      ReadOnlyPaths = "/";
+      ReadWritePaths = [ "/proc/self" "/var/lib/uninsane/sink" ];
+      PrivateDevices = "yes";
       ProtectKernelModules = "yes";
       ProtectKernelTunables = "yes";
-      ProtectSystem = "strict";
-      ReadOnlyPaths = [ "/var/log/nginx" ];
-      ReadWritePaths = [ "/proc/self" "/var/lib/goaccess" ];
-      StateDirectory = "goaccess";
-      SystemCallFilter = "~@clock @cpu-emulation @debug @keyring @memlock @module @mount @obsolete @privileged @reboot @resources @setuid @swap @raw-io";
-      WorkingDirectory = "/var/lib/goaccess";
     };
     after = [ "network.target" ];
     wantedBy = [ "multi-user.target" ];
@@ -51,7 +49,7 @@
     addSSL = true;
     enableACME = true;
     # inherit kTLS;
-    root = "/var/lib/goaccess";
+    root = "/var/lib/uninsane/sink";
 
     locations."/ws" = {
       proxyPass = "http://127.0.0.1:7890";

hosts/by-name/servo/services/ipfs.nix

@@ -10,7 +10,7 @@
 
 lib.mkIf false # i don't actively use ipfs anymore
 {
-  sane.persist.sys.byStore.private = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
     { user = "261"; group = "261"; path = "/var/lib/ipfs"; method = "bind"; }
   ];

hosts/by-name/servo/services/jackett.nix

@@ -0,0 +1,33 @@
+{ ... }:
+
+{
+  sane.persist.sys.byStore.plaintext = [
+    # TODO: mode? we only need this to save Indexer creds ==> migrate to config?
+    { user = "root"; group = "root"; path = "/var/lib/jackett"; method = "bind"; }
+  ];
+  services.jackett.enable = true;
+
+  systemd.services.jackett.after = [ "wireguard-wg-ovpns.service" ];
+  systemd.services.jackett.partOf = [ "wireguard-wg-ovpns.service" ];
+  systemd.services.jackett.serviceConfig = {
+    # run this behind the OVPN static VPN
+    NetworkNamespacePath = "/run/netns/ovpns";
+    # patch jackett to listen on the public interfaces
+    # ExecStart = lib.mkForce "${pkgs.jackett}/bin/Jackett --NoUpdates --DataFolder /var/lib/jackett/.config/Jackett --ListenPublic";
+  };
+
+  # jackett torrent search
+  services.nginx.virtualHosts."jackett.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+    # inherit kTLS;
+    locations."/" = {
+      # proxyPass = "http://ovpns.uninsane.org:9117";
+      proxyPass = "http://10.0.1.6:9117";
+      recommendedProxySettings = true;
+    };
+  };
+
+  sane.dns.zones."uninsane.org".inet.CNAME."jackett" = "native";
+}
+

hosts/by-name/servo/services/jackett/default.nix

@@ -1,68 +0,0 @@
-{ config, lib, pkgs, ... }:
-
-let
-  cfg = config.services.jackett;
-in
-{
-  sane.persist.sys.byStore.private = [
-    # TODO: mode? we only need this to save Indexer creds ==> migrate to config?
-    { user = "jackett"; group = "jackett"; path = "/var/lib/jackett"; method = "bind"; }
-  ];
-  services.jackett.enable = true;
-
-  systemd.services.jackett.after = [ "wireguard-wg-ovpns.service" ];
-  systemd.services.jackett.partOf = [ "wireguard-wg-ovpns.service" ];
-  systemd.services.jackett = {
-    # run this behind the OVPN static VPN
-    serviceConfig.NetworkNamespacePath = "/run/netns/ovpns";
-    serviceConfig.ExecStartPre = [ "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect ${config.sane.netns.ovpns.netnsPubIpv4}" ];  # abort if public IP is not as expected
-    # patch in `--ListenPublic` so that it's reachable from the netns veth.
-    # this also makes it reachable from the VPN pub address. oh well.
-    serviceConfig.ExecStart = lib.mkForce "${cfg.package}/bin/Jackett --ListenPublic --NoUpdates --DataFolder '${cfg.dataDir}'";
-    serviceConfig.RestartSec = "30s";
-
-    # hardening (systemd-analyze security jackett)
-    # TODO: upstream into nixpkgs
-    serviceConfig.StateDirectory = "jackett";
-    serviceConfig.LockPersonality = true;
-    serviceConfig.NoNewPrivileges = true;
-    # serviceConfig.MemoryDenyWriteExecute = true;  #< Failed to create CoreCLR, HRESULT: 0x80004005
-    serviceConfig.PrivateDevices = true;
-    serviceConfig.PrivateMounts = true;
-    serviceConfig.PrivateTmp = true;
-    serviceConfig.PrivateUsers = true;
-    serviceConfig.ProcSubset = "pid";
-    serviceConfig.ProtectClock = true;
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;
-    serviceConfig.ProtectHostname = true;
-    serviceConfig.ProtectKernelLogs = true;
-    serviceConfig.ProtectKernelModules = true;
-    serviceConfig.ProtectKernelTunables = true;
-    serviceConfig.ProtectProc = "invisible";
-    serviceConfig.ProtectSystem = "strict";
-    serviceConfig.RemoveIPC = true;
-    serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
-    serviceConfig.RestrictNamespaces = true;
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";
-    serviceConfig.SystemCallFilter = [ "@system-service" "~@privileged" ];
-  };
-
-  # jackett torrent search
-  services.nginx.virtualHosts."jackett.uninsane.org" = {
-    forceSSL = true;
-    enableACME = true;
-    # inherit kTLS;
-    locations."/" = {
-      proxyPass = "http://${config.sane.netns.ovpns.netnsVethIpv4}:9117";
-      recommendedProxySettings = true;
-    };
-    locations."= /robots.txt".extraConfig = ''
-      return 200 "User-agent: *\nDisallow: /\n";
-    '';
-  };
-
-  sane.dns.zones."uninsane.org".inet.CNAME."jackett" = "native";
-}
-

hosts/by-name/servo/services/jellyfin.nix

@@ -75,7 +75,7 @@
   # Jellyfin multimedia server
   # this is mostly taken from the official jellfin.org docs
   services.nginx.virtualHosts."jelly.uninsane.org" = {
-    forceSSL = true;
+    addSSL = true;
     enableACME = true;
     # inherit kTLS;
 

hosts/by-name/servo/services/kiwix-serve.nix

@@ -21,9 +21,6 @@
     enableACME = true;
     # inherit kTLS;
     locations."/".proxyPass = "http://127.0.0.1:8013";
-    locations."= /robots.txt".extraConfig = ''
-      return 200 "User-agent: *\nDisallow: /\n";
-    '';
   };
 
   sane.dns.zones."uninsane.org".inet.CNAME."w" = "native";

hosts/by-name/servo/services/komga.nix

@@ -12,14 +12,11 @@ in
   services.komga.port = 11319;  # chosen at random
 
   services.nginx.virtualHosts."komga.uninsane.org" = {
-    forceSSL = true;
+    addSSL = true;
     enableACME = true;
     locations."/" = {
       proxyPass = "http://127.0.0.1:${builtins.toString port}";
     };
-    locations."= /robots.txt".extraConfig = ''
-      return 200 "User-agent: *\nDisallow: /\n";
-    '';
   };
   sane.dns.zones."uninsane.org".inet.CNAME."komga" = "native";
 }

hosts/by-name/servo/services/lemmy.nix

@@ -22,7 +22,7 @@ let
   #   #      "Change commandline flag to allow disabling video, since it is enabled by default"
   #   postPatch = (upstream.postPatch or "") + ''
   #     substituteInPlace src/validate.rs \
-  #       --replace-fail 'if transcode_options.needs_reencode() {' 'if false {'
+  #       --replace 'if transcode_options.needs_reencode() {' 'if false {'
   #   '';
   # });
 in {
@@ -38,10 +38,14 @@ in {
     nginx.enable = true;
   };
 
+  systemd.services.lemmy.serviceConfig = {
+    # fix to use a normal user so we can configure perms correctly
+    DynamicUser = mkForce false;
+    User = "lemmy";
+    Group = "lemmy";
+  };
   systemd.services.lemmy.environment = {
     RUST_BACKTRACE = "full";
-    RUST_LOG = "error";
-    # RUST_LOG = "warn";
     # RUST_LOG = "debug";
     # RUST_LOG = "trace";
     # upstream defaults LEMMY_DATABASE_URL = "postgres:///lemmy?host=/run/postgresql";
@@ -68,73 +72,6 @@ in {
 
   sane.dns.zones."uninsane.org".inet.CNAME."lemmy" = "native";
 
-  systemd.services.lemmy = {
-    # fix to use a normal user so we can configure perms correctly
-    # XXX(2024-07-28): this hasn't been rigorously tested:
-    # possible that i've set something too strict and won't notice right away
-    serviceConfig.DynamicUser = mkForce false;
-    serviceConfig.User = "lemmy";
-    serviceConfig.Group = "lemmy";
-
-    # hardening (systemd-analyze security lemmy)
-    # a handful of these are specified in upstream nixpkgs, but mostly not
-    serviceConfig.LockPersonality = true;
-    serviceConfig.NoNewPrivileges = true;
-    serviceConfig.MemoryDenyWriteExecute = true;
-    serviceConfig.PrivateDevices = true;
-    serviceConfig.PrivateMounts = true;
-    serviceConfig.PrivateTmp = true;
-    serviceConfig.PrivateUsers = true;
-    serviceConfig.ProcSubset = "pid";
-
-    serviceConfig.ProtectClock = true;
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;
-    serviceConfig.ProtectHostname = true;
-    serviceConfig.ProtectKernelLogs = true;
-    serviceConfig.ProtectKernelModules = true;
-    serviceConfig.ProtectKernelTunables = true;
-    serviceConfig.ProtectProc = "invisible";
-    serviceConfig.ProtectSystem = "strict";
-    serviceConfig.RemoveIPC = true;
-    serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
-
-    serviceConfig.RestrictNamespaces = true;
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";
-    serviceConfig.SystemCallFilter = [ "@system-service" ];
-  };
-
-  systemd.services.lemmy-ui = {
-    # hardening (systemd-analyze security lemmy-ui)
-    # TODO: upstream into nixpkgs
-    serviceConfig.LockPersonality = true;
-    serviceConfig.NoNewPrivileges = true;
-    # serviceConfig.MemoryDenyWriteExecute = true;  #< it uses v8, JIT
-    serviceConfig.PrivateDevices = true;
-    serviceConfig.PrivateMounts = true;
-    serviceConfig.PrivateTmp = true;
-    serviceConfig.PrivateUsers = true;
-    serviceConfig.ProcSubset = "pid";
-
-    serviceConfig.ProtectClock = true;
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;
-    serviceConfig.ProtectHostname = true;
-    serviceConfig.ProtectKernelLogs = true;
-    serviceConfig.ProtectKernelModules = true;
-    serviceConfig.ProtectKernelTunables = true;
-    serviceConfig.ProtectProc = "invisible";
-    serviceConfig.ProtectSystem = "strict";
-    serviceConfig.RemoveIPC = true;
-    serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
-
-    serviceConfig.RestrictNamespaces = true;
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";
-    serviceConfig.SystemCallFilter = [ "@system-service" "@pkey" "@sandbox" ];
-  };
-
   #v DO NOT REMOVE: defaults to 0.3, instead of latest, so always need to explicitly set this.
   services.pict-rs.package = pict-rs;
 
@@ -144,38 +81,10 @@ in {
   # - via CLI flags (overrides everything above)
   # some of the CLI flags have defaults, making it the only actual way to configure certain things even when docs claim otherwise.
   # CLI args: <https://git.asonix.dog/asonix/pict-rs#user-content-running>
-  systemd.services.pict-rs = {
-    serviceConfig.ExecStart = lib.mkForce (lib.concatStringsSep " " [
-      "${lib.getBin pict-rs}/bin/pict-rs run"
-      "--media-video-max-frame-count" (builtins.toString (30*60*60))
-      "--media-process-timeout 120"
-      "--media-video-allow-audio"  # allow audio
-    ]);
-
-    # hardening (systemd-analyze security pict-rs)
-    # TODO: upstream into nixpkgs
-    serviceConfig.LockPersonality = true;
-    serviceConfig.NoNewPrivileges = true;
-    serviceConfig.MemoryDenyWriteExecute = true;
-    serviceConfig.PrivateDevices = true;
-    serviceConfig.PrivateMounts = true;
-    serviceConfig.PrivateTmp = true;
-    serviceConfig.PrivateUsers = true;
-    serviceConfig.ProcSubset = "pid";
-    serviceConfig.ProtectClock = true;
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;
-    serviceConfig.ProtectHostname = true;
-    serviceConfig.ProtectKernelLogs = true;
-    serviceConfig.ProtectKernelModules = true;
-    serviceConfig.ProtectKernelTunables = true;
-    serviceConfig.ProtectProc = "invisible";
-    serviceConfig.ProtectSystem = "strict";
-    serviceConfig.RemoveIPC = true;
-    serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
-    serviceConfig.RestrictNamespaces = true;
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";
-    serviceConfig.SystemCallFilter = [ "@system-service" ];
-  };
+  systemd.services.pict-rs.serviceConfig.ExecStart = lib.mkForce (lib.concatStringsSep " " [
+    "${lib.getBin pict-rs}/bin/pict-rs run"
+    "--media-video-max-frame-count" (builtins.toString (30*60*60))
+    "--media-process-timeout 120"
+    "--media-video-allow-audio"  # allow audio
+  ]);
 }

hosts/by-name/servo/services/matrix/default.nix

@@ -1,6 +1,6 @@
 # docs: <https://nixos.wiki/wiki/Matrix>
 # docs: <https://nixos.org/manual/nixos/stable/index.html#module-services-matrix-synapse>
-# example config: <https://github.com/element-hq/synapse/blob/develop/docs/sample_config.yaml>
+# example config: <https://github.com/matrix-org/synapse/blob/develop/docs/sample_config.yaml>
 #
 # ENABLING PUSH NOTIFICATIONS (with UnifiedPush/ntfy):
 # - Matrix "pushers" API spec: <https://spec.matrix.org/latest/client-server-api/#post_matrixclientv3pushersset>
@@ -20,12 +20,14 @@
     ./signal.nix
   ];
 
-  sane.persist.sys.byStore.private = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/matrix-synapse"; method = "bind"; }
   ];
   services.matrix-synapse.enable = true;
-  services.matrix-synapse.log.root.level = "ERROR";  # accepts "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL" (?)
   services.matrix-synapse.settings = {
+    # this changes the default log level from INFO to WARN.
+    # maybe there's an easier way?
+    log_config = ./synapse-log_level.yaml;
     server_name = "uninsane.org";
 
     # services.matrix-synapse.enable_registration_captcha = true;

hosts/by-name/servo/services/matrix/discord-puppet.nix

@@ -5,7 +5,7 @@
 # - recommended to use mautrix-discord: <https://github.com/NixOS/nixpkgs/pull/200462>
 lib.mkIf false
 {
-  sane.persist.sys.byStore.private = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/mx-puppet-discord"; method = "bind"; }
   ];
 

hosts/by-name/servo/services/matrix/irc.nix

@@ -1,13 +1,15 @@
 # config docs:
 # - <https://github.com/matrix-org/matrix-appservice-irc/blob/develop/config.sample.yaml>
+#       probably want to remove that.
 { config, lib, ... }:
 
 let
-  ircServer = { name, additionalAddresses ? [], ssl ? true, sasl ? true, port ? if ssl then 6697 else 6667 }: let
+  ircServer = { name, additionalAddresses ? [], sasl ? true, port ? 6697 }: let
     lowerName = lib.toLower name;
   in {
     # XXX sasl: appservice doesn't support NickServ identification (only SASL, or PASS if sasl = false)
-    inherit additionalAddresses name port sasl ssl;
+    inherit name additionalAddresses sasl port;
+    ssl = true;
     botConfig = {
       # bot has no presence in IRC channel; only real Matrix users
       enabled = false;
@@ -99,7 +101,7 @@ in
     })
   ];
 
-  sane.persist.sys.byStore.private = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode?
     { user = "matrix-appservice-irc"; group = "matrix-appservice-irc"; path = "/var/lib/matrix-appservice-irc"; method = "bind"; }
   ];
@@ -127,7 +129,6 @@ in
     };
 
     ircService = {
-      logging.level = "warn";  # "error", "warn", "info", "debug"
       servers = {
         "irc.esper.net" = ircServer {
           name = "esper";
@@ -155,10 +156,6 @@ in
           # - #sxmo-offtopic
         };
         "irc.rizon.net" = ircServer { name = "Rizon"; };
-        "wigle.net" = ircServer {
-          name = "WiGLE";
-          ssl = false;
-        };
       };
     };
   };

hosts/by-name/servo/services/matrix/signal.nix

@@ -4,7 +4,7 @@
 
 lib.mkIf false  # disabled 2024/01/11: i don't use it, and pkgs.mautrix-signal had some API changes
 {
-  sane.persist.sys.byStore.private = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "mautrix-signal"; group = "mautrix-signal"; path = "/var/lib/mautrix-signal"; method = "bind"; }
     { user = "signald"; group = "signald"; path = "/var/lib/signald"; method = "bind"; }
   ];

hosts/by-name/servo/services/matrix/synapse-log_level.yaml

@@ -0,0 +1,27 @@
+version: 1
+
+# In systemd's journal, loglevel is implicitly stored, so let's omit it
+# from the message text.
+formatters:
+    journal_fmt:
+        format: '%(name)s: [%(request)s] %(message)s'
+
+filters:
+    context:
+        (): synapse.util.logcontext.LoggingContextFilter
+        request: ""
+
+handlers:
+    journal:
+        class: systemd.journal.JournalHandler
+        formatter: journal_fmt
+        filters: [context]
+        SYSLOG_IDENTIFIER: synapse
+
+# default log level: INFO
+root:
+    level: WARN
+    handlers: [journal]
+
+disable_existing_loggers: False
+

hosts/by-name/servo/services/nginx.nix

@@ -17,24 +17,18 @@ in
   sane.ports.ports."80" = {
     protocol = [ "tcp" ];
     visibleTo.lan = true;
-    visibleTo.ovpns = true;  # so that letsencrypt can procure a cert for the mx record
-    visibleTo.doof = true;
+    visibleTo.wan = true;
+    visibleTo.ovpn = true;  # so that letsencrypt can procure a cert for the mx record
     description = "colin-http-uninsane.org";
   };
   sane.ports.ports."443" = {
     protocol = [ "tcp" ];
     visibleTo.lan = true;
-    visibleTo.doof = true;
+    visibleTo.wan = true;
     description = "colin-https-uninsane.org";
   };
 
   services.nginx.enable = true;
-  # nginxStable is one release behind nginxMainline.
-  # nginx itself recommends running mainline; nixos defaults to stable.
-  # services.nginx.package = pkgs.nginxMainline;
-  # XXX(2024-07-31): nixos defaults to zlib-ng -- supposedly more performant, but spams log with
-  # "gzip filter failed to use preallocated memory: ..."
-  services.nginx.package = pkgs.nginxMainline.override { zlib = pkgs.zlib; };
   services.nginx.appendConfig = ''
     # use 1 process per core.
     # may want to increase worker_connections too, but `ulimit -n` must be increased first.
@@ -50,10 +44,8 @@ in
     log_format vcombined '$host:$server_port $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referrer" "$http_user_agent"';
     access_log /var/log/nginx/private.log vcombined;
   '';
-  # enables gzip and sets gzip_comp_level = 5
+  # sets gzip_comp_level = 5
   services.nginx.recommendedGzipSettings = true;
-  # enables zstd and sets zstd_comp_level = 9
-  services.nginx.recommendedZstdSettings = true;
   # enables OCSP stapling (so clients don't need contact the OCSP server -- i do instead)
   # - doesn't seem to, actually: <https://www.ssllabs.com/ssltest/analyze.html?d=uninsane.org>
   # caches TLS sessions for 10m
@@ -97,26 +89,6 @@ in
         disable_symlinks on;
       '';
     };
-    locations."/share/Milkbags/" = {
-      alias = "/var/media/Videos/Milkbags/";
-      extraConfig = ''
-        # autoindex => render directory listings
-        autoindex on;
-        # don't follow any symlinks when serving files
-        # otherwise it allows a directory escape
-        disable_symlinks on;
-      '';
-    };
-    locations."/share/Ubunchu/" = {
-      alias = "/var/media/Books/Visual/HiroshiSeo/Ubunchu/";
-      extraConfig = ''
-        # autoindex => render directory listings
-        autoindex on;
-        # don't follow any symlinks when serving files
-        # otherwise it allows a directory escape
-        disable_symlinks on;
-      '';
-    };
 
     # allow matrix users to discover that @user:uninsane.org is reachable via matrix.uninsane.org
     locations."= /.well-known/matrix/server".extraConfig =
@@ -198,15 +170,8 @@ in
 
   sane.persist.sys.byStore.plaintext = [
     { user = "acme"; group = "acme"; path = "/var/lib/acme"; method = "bind"; }
-  ];
-  sane.persist.sys.byStore.private = [
     { user = "colin"; group = "users"; path = "/var/www/sites"; method = "bind"; }
   ];
-  sane.persist.sys.byStore.ephemeral = [
-    # logs *could* be persisted to private storage, but then there's the issue of
-    # "what if servo boots, isn't unlocked, and the whole / tmpfs is consumed by logs"
-    { user = "nginx"; group = "nginx"; path = "/var/log/nginx"; method = "bind"; }
-  ];
 
   # let's encrypt default chain looks like:
   # - End-entity certificate ← R3 ← ISRG Root X1 ← DST Root CA X3

hosts/by-name/servo/services/nixserve.nix

@@ -0,0 +1,21 @@
+{ config, ... }:
+
+{
+  services.nginx.virtualHosts."nixcache.uninsane.org" = {
+    addSSL = true;
+    enableACME = true;
+    # inherit kTLS;
+    # serverAliases = [ "nixcache" ];
+    locations."/".extraConfig = ''
+      proxy_pass http://localhost:${toString config.services.nix-serve.port};
+      proxy_set_header Host $host;
+      proxy_set_header X-Real-IP $remote_addr;
+      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    '';
+  };
+
+  sane.dns.zones."uninsane.org".inet.CNAME."nixcache" = "native";
+
+  sane.services.nixserve.enable = true;
+  sane.services.nixserve.secretKeyFile = config.sops.secrets.nix_serve_privkey.path;
+}

hosts/by-name/servo/services/ntfy/ntfy-sh.nix

@@ -30,7 +30,7 @@ let
   altPort = 2587;
 in
 {
-  sane.persist.sys.byStore.private = [
+  sane.persist.sys.byStore.plaintext = [
     # not 100% necessary to persist this, but ntfy does keep a 12hr (by default) cache
     # for pushing notifications to users who become offline.
     # ACLs also live here.
@@ -46,7 +46,7 @@ in
     # defaults to 45s.
     # note that the client may still do its own TCP-level keepalives, typically every 30s
     keepalive-interval = "15m";
-    log-level = "info";  # trace, debug, info (default), warn, error
+    log-level = "trace";  # trace, debug, info (default), warn, error
     auth-default-access = "deny-all";
   };
   systemd.services.ntfy-sh.serviceConfig.DynamicUser = lib.mkForce false;
@@ -86,7 +86,7 @@ in
   sane.ports.ports."${builtins.toString altPort}" = {
     protocol = [ "tcp" ];
     visibleTo.lan = true;
-    visibleTo.doof = true;
+    visibleTo.wan = true;
     description = "colin-ntfy.uninsane.org";
   };
 }

hosts/by-name/servo/services/ntfy/ntfy-waiter

@@ -1,5 +1,5 @@
 #!/usr/bin/env nix-shell
-#!nix-shell -i python3 -p ntfy-sh -p python3
+#!nix-shell -i python3 -p "python3.withPackages (ps: [  ])" -p ntfy-sh
 
 import argparse
 import logging

hosts/by-name/servo/services/ntfy/ntfy-waiter.nix

@@ -47,7 +47,7 @@ in
     };
     sane.ntfy-waiter.package = mkOption {
       type = types.package;
-      default = pkgs.static-nix-shell.mkPython3 {
+      default = pkgs.static-nix-shell.mkPython3Bin {
         pname = "ntfy-waiter";
         srcRoot = ./.;
         pkgs = [ "ntfy-sh" ];
@@ -62,8 +62,8 @@ in
     sane.ports.ports = lib.mkMerge (lib.forEach portRange (port: {
       "${builtins.toString port}" = {
         protocol = [ "tcp" ];
-        visibleTo.doof = true;
         visibleTo.lan = true;
+        visibleTo.wan = true;
         description = "colin-notification-waiter-${builtins.toString (port - portLow + 1)}-of-${builtins.toString numPorts}";
       };
     }));

hosts/by-name/servo/services/pleroma.nix

@@ -7,15 +7,14 @@
 # to run it in a oci-container: <https://github.com/barrucadu/nixfiles/blob/master/services/pleroma.nix>
 #
 # admin frontend: <https://fed.uninsane.org/pleroma/admin>
-{ config, lib, pkgs, ... }:
+{ config, pkgs, ... }:
 
 let
   logLevel = "warn";
   # logLevel = "debug";
 in
 {
-  sane.persist.sys.byStore.private = [
-    # contains media i've uploaded to the server
+  sane.persist.sys.byStore.plaintext = [
     { user = "pleroma"; group = "pleroma"; path = "/var/lib/pleroma"; method = "bind"; }
   ];
   services.pleroma.enable = true;
@@ -26,7 +25,7 @@ in
 
     config :pleroma, Pleroma.Web.Endpoint,
       url: [host: "fed.uninsane.org", scheme: "https", port: 443],
-      http: [ip: {127, 0, 0, 1}, port: 4040]
+      http: [ip: {127, 0, 0, 1}, port: 4000]
     #   secret_key_base: "{secrets.pleroma.secret_key_base}",
     #   signing_salt: "{secrets.pleroma.signing_salt}"
 
@@ -136,53 +135,26 @@ in
     # something inside pleroma invokes `sh` w/o specifying it by path, so this is needed to allow pleroma to start
     pkgs.bash
     # used by Pleroma to strip geo tags from uploads
-    config.sane.programs.exiftool.package
+    pkgs.exiftool
     # i saw some errors when pleroma was shutting down about it not being able to find `awk`. probably not critical
-    config.sane.programs.gawk.package
+    pkgs.gawk
     # needed for email operations like password reset
     pkgs.postfix
   ];
 
-  systemd.services.pleroma = {
+  systemd.services.pleroma.serviceConfig = {
     # postgres can be slow to service early requests, preventing pleroma from starting on the first try
-    serviceConfig.Restart = "on-failure";
-    serviceConfig.RestartSec = "10s";
-
-    # hardening (systemd-analyze security pleroma)
-    # XXX(2024-07-28): this hasn't been rigorously tested:
-    # possible that i've set something too strict and won't notice right away
-    # make sure to test:
-    # - image/media uploading
-    serviceConfig.CapabilityBoundingSet = "~CAP_SYS_ADMIN";  #< TODO: reduce this. try: CAP_SYS_NICE CAP_DAC_READ_SEARCH CAP_SYS_CHROOT CAP_SETGID CAP_SETUID
-    serviceConfig.LockPersonality = true;
-    serviceConfig.NoNewPrivileges = true;
-    serviceConfig.MemoryDenyWriteExecute = true;
-    serviceConfig.PrivateDevices = lib.mkForce true;  #< dunno why nixpkgs has this set false; it seems to work as true
-    serviceConfig.PrivateMounts = true;
-    serviceConfig.PrivateTmp = true;
-    serviceConfig.PrivateUsers = true;
-
-    serviceConfig.ProtectProc = "invisible";
-    serviceConfig.ProcSubset = "all";  #< needs /proc/sys/kernel/overflowuid for bwrap
-
-    serviceConfig.ProtectClock = true;
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;
-    serviceConfig.ProtectKernelModules = true;
-    serviceConfig.ProtectSystem = lib.mkForce "strict";
-    serviceConfig.RemoveIPC = true;
-    serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6 AF_NETLINK";
-
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";
-    serviceConfig.SystemCallFilter = [ "@system-service" "@mount" "@sandbox" ];  #< "sandbox" might not actually be necessary
-
-    serviceConfig.ProtectHostname = false;  #< else brap can't mount /proc
-    serviceConfig.ProtectKernelLogs = false;  #< else breaks exiftool  ("bwrap: Can't mount proc on /newroot/proc: Operation not permitted")
-    serviceConfig.ProtectKernelTunables = false;  #< else breaks exiftool
-    serviceConfig.RestrictNamespaces = false;  # media uploads require bwrap
+    Restart = "on-failure";
+    RestartSec = "10s";
   };
 
+  # systemd.services.pleroma.serviceConfig = {
+  #   # required for sendmail. see https://git.pleroma.social/pleroma/pleroma/-/issues/2259
+  #   NoNewPrivileges = lib.mkForce false;
+  #   PrivateTmp = lib.mkForce false;
+  #   CapabilityBoundingSet = lib.mkForce "~";
+  # };
+
   # this is required to allow pleroma to send email.
   # raw `sendmail` works, but i think pleroma's passing it some funny flags or something, idk.
   # hack to fix that.
@@ -195,7 +167,7 @@ in
     enableACME = true;
     # inherit kTLS;
     locations."/" = {
-      proxyPass = "http://127.0.0.1:4040";
+      proxyPass = "http://127.0.0.1:4000";
       recommendedProxySettings = true;
       # documented: https://git.pleroma.social/pleroma/pleroma/-/blob/develop/installation/pleroma.nginx
       extraConfig = ''

hosts/by-name/servo/services/postgres.nix

@@ -6,9 +6,9 @@ let
   KiB = n: 1024*n;
 in
 {
-  sane.persist.sys.byStore.private = [
-    { user = "postgres"; group = "postgres"; mode = "0750"; path = "/var/lib/postgresql"; method = "bind"; }
-    { user = "postgres"; group = "postgres"; mode = "0750"; path = "/var/backup/postgresql"; method = "bind"; }
+  sane.persist.sys.byStore.plaintext = [
+    # TODO: mode?
+    { user = "postgres"; group = "postgres"; path = "/var/lib/postgresql"; method = "bind"; }
   ];
   services.postgresql.enable = true;
 

hosts/by-name/servo/services/prosody/default.nix

@@ -56,48 +56,47 @@ let
   enableDebug = false;
 in
 {
-  sane.persist.sys.byStore.private = [
-    # TODO: mode?
+  sane.persist.sys.byStore.plaintext = [
     { user = "prosody"; group = "prosody"; path = "/var/lib/prosody"; method = "bind"; }
   ];
   sane.ports.ports."5000" = {
     protocol = [ "tcp" ];
-    visibleTo.doof = true;
     visibleTo.lan = true;
+    visibleTo.wan = true;
     description = "colin-xmpp-prosody-fileshare-proxy65";
   };
   sane.ports.ports."5222" = {
     protocol = [ "tcp" ];
-    visibleTo.doof = true;
     visibleTo.lan = true;
+    visibleTo.wan = true;
     description = "colin-xmpp-client-to-server";
   };
   sane.ports.ports."5223" = {
     protocol = [ "tcp" ];
-    visibleTo.doof = true;
     visibleTo.lan = true;
+    visibleTo.wan = true;
     description = "colin-xmpps-client-to-server";  # XMPP over TLS
   };
   sane.ports.ports."5269" = {
     protocol = [ "tcp" ];
-    visibleTo.doof = true;
+    visibleTo.wan = true;
     description = "colin-xmpp-server-to-server";
   };
   sane.ports.ports."5270" = {
     protocol = [ "tcp" ];
-    visibleTo.doof = true;
+    visibleTo.wan = true;
     description = "colin-xmpps-server-to-server";  # XMPP over TLS
   };
   sane.ports.ports."5280" = {
     protocol = [ "tcp" ];
-    visibleTo.doof = true;
     visibleTo.lan = true;
+    visibleTo.wan = true;
     description = "colin-xmpp-bosh";
   };
   sane.ports.ports."5281" = {
     protocol = [ "tcp" ];
-    visibleTo.doof = true;
     visibleTo.lan = true;
+    visibleTo.wan = true;
     description = "colin-xmpp-prosody-https";  # necessary?
   };
 

hosts/by-name/servo/services/slskd.nix

@@ -7,12 +7,9 @@
 # debugging:
 # - soulseek is just *flaky*. if you see e.g. DNS errors, even though you can't replicate them via `dig` or `getent ahostsv4`, just give it 10 minutes to work out:
 #   - "Soulseek.AddressException: Failed to resolve address 'vps.slsknet.org': Resource temporarily unavailable"
-{ config, lib, pkgs, ... }:
-
+{ config, lib, ... }:
 {
-  sane.persist.sys.byStore.ephemeral = [
-    # {data,downloads,incomplete,logs}: contains logs, search history, and downloads
-    # so, move the downloaded data to persistent storage regularly, or configure the downloads/incomplete dirs to point to persisted storage (in nixpkgs slskd config)
+  sane.persist.sys.byStore.plaintext = [
     { user = "slskd"; group = "media"; path = "/var/lib/slskd"; method = "bind"; }
   ];
   sops.secrets."slskd_env" = {
@@ -24,7 +21,8 @@
 
   sane.ports.ports."50300" = {
     protocol = [ "tcp" ];
-    # visibleTo.ovpns = true;  #< not needed: it runs in the ovpns namespace
+    # not visible to WAN: i run this in a separate netns
+    visibleTo.ovpn = true;
     description = "colin-soulseek";
   };
 
@@ -34,7 +32,7 @@
     forceSSL = true;
     enableACME = true;
     locations."/" = {
-      proxyPass = "http://${config.sane.netns.ovpns.netnsVethIpv4}:5030";
+      proxyPass = "http://10.0.1.6:5030";
       proxyWebsockets = true;
     };
   };
@@ -71,19 +69,11 @@
   };
 
   systemd.services.slskd = {
-    # run this behind the OVPN static VPN
-    serviceConfig.NetworkNamespacePath = "/run/netns/ovpns";
-    serviceConfig.ExecStartPre = [ "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect ${config.sane.netns.ovpns.netnsPubIpv4}" ];  # abort if public IP is not as expected
-
-    serviceConfig.Restart = lib.mkForce "always";  # exits "success" when it fails to connect to soulseek server
-    serviceConfig.RestartSec = "60s";
-
-    # hardening (systemd-analyze security slskd)
-    # upstream nixpkgs specifies moderate defaults; these are supplementary
-    # serviceConfig.MemoryDenyWriteExecute = true;
-    # serviceConfig.ProcSubset = "pid";
-    # serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
-    # serviceConfig.SystemCallArchitectures = "native";
-    # serviceConfig.SystemCallFilter = [ "@system-service" ];
+    serviceConfig = {
+      # run this behind the OVPN static VPN
+      NetworkNamespacePath = "/run/netns/ovpns";
+      Restart = lib.mkForce "always";  # exits "success" when it fails to connect to soulseek server
+      RestartSec = "60s";
+    };
   };
 }

hosts/by-name/servo/services/transmission.nix

@@ -22,23 +22,46 @@ let
         --replace-fail 'set(TR_USER_AGENT_PREFIX "''${TR_SEMVER}")' 'set(TR_USER_AGENT_PREFIX "3.00")'
     '';
   });
-  download-dir = "/var/media/torrents";  #< keep in sync with consts embedded in `torrent-done`
-  torrent-done = pkgs.static-nix-shell.mkBash {
-    pname = "torrent-done";
-    srcRoot = ./.;
-    pkgs = [
-      "acl"
-      "coreutils"
-      "findutils"
-      "rsync"
+  download-dir = "/var/media/torrents";
+  torrent-done = pkgs.writeShellApplication {
+    name = "torrent-done";
+    runtimeInputs = with pkgs; [
+      rsync
+      util-linux
     ];
+    text = ''
+      destructive() {
+        if [ -n "''${TR_DRY_RUN-}" ]; then
+          echo "$*"
+        else
+          "$@"
+        fi
+      }
+      if [[ "$TR_TORRENT_DIR" =~ ^.*freeleech.*$ ]]; then
+        # freeleech torrents have no place in my permanent library
+        echo "freeleech: nothing to do"
+        exit 0
+      fi
+      if ! [[ "$TR_TORRENT_DIR" =~ ^${download-dir}/.*$ ]]; then
+        echo "unexpected torrent dir, aborting: $TR_TORRENT_DIR"
+        exit 0
+      fi
+
+      REL_DIR="''${TR_TORRENT_DIR#${download-dir}/}"
+      MEDIA_DIR="/var/media/$REL_DIR"
+
+      destructive mkdir -p "$(dirname "$MEDIA_DIR")"
+      destructive rsync -arv "$TR_TORRENT_DIR/" "$MEDIA_DIR/"
+      # dedupe the whole media library.
+      # yeah, a bit excessive: move this to a cron job if that's problematic.
+      destructive hardlink /var/media --reflink=always --ignore-time --verbose
+    '';
   };
 in
 {
-  sane.persist.sys.byStore.private = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? we need this specifically for the stats tracking in .config/
     { user = "transmission"; group = config.users.users.transmission.group; path = "/var/lib/transmission"; method = "bind"; }
-    { user = "transmission"; group = config.users.users.transmission.group; path = "/var/backup/torrents"; method = "bind"; }
   ];
   users.users.transmission.extraGroups = [ "media" ];
 
@@ -58,8 +81,8 @@ in
     # DOCUMENTATION/options list: <https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md#options>
 
     # message-level = 3;  #< enable for debug logging. 0-3, default is 2.
-    # ovpns.netnsVethIpv4 => allow rpc only from the root servo ns. it'll tunnel things to the net, if need be.
-    rpc-bind-address = config.sane.netns.ovpns.netnsVethIpv4;
+    # 0.0.0.0 => allow rpc from any host: we gate it via firewall and auth requirement
+    rpc-bind-address = "0.0.0.0";
     #rpc-host-whitelist = "bt.uninsane.org";
     #rpc-whitelist = "*.*.*.*";
     rpc-authentication-required = true;
@@ -69,10 +92,6 @@ in
     rpc-password = "{503fc8928344f495efb8e1f955111ca5c862ce0656SzQnQ5";
     rpc-whitelist-enabled = false;
 
-    # force behind ovpns in case the NetworkNamespace fails somehow
-    bind-address-ipv4 = config.sane.netns.ovpns.netnsPubIpv4;
-    port-forwarding-enabled = false;
-
     # hopefully, make the downloads world-readable
     # umask = 0;  #< default is 2: i.e. deny writes from world
 
@@ -107,31 +126,14 @@ in
     script-torrent-done-filename = "${torrent-done}/bin/torrent-done";
   };
 
-  systemd.services.transmission = {
-    after = [ "wireguard-wg-ovpns.service" ];
-    partOf = [ "wireguard-wg-ovpns.service" ];
-    environment.TR_DEBUG = "1";
+  systemd.services.transmission.after = [ "wireguard-wg-ovpns.service" ];
+  systemd.services.transmission.partOf = [ "wireguard-wg-ovpns.service" ];
+  systemd.services.transmission.serviceConfig = {
     # run this behind the OVPN static VPN
-    serviceConfig.NetworkNamespacePath = "/run/netns/ovpns";
-    serviceConfig.ExecStartPre = [ "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect ${config.sane.netns.ovpns.netnsPubIpv4}" ];  # abort if public IP is not as expected
-
-    serviceConfig.Restart = "on-failure";
-    serviceConfig.RestartSec = "30s";
-    serviceConfig.BindPaths = [ "/var/media" ];  #< so it can move completed torrents into the media library
-    serviceConfig.SystemCallFilter = lib.mkForce [
-      # the torrent-done script does stuff which fails the nixos default syscall filter.
-      # allow a bunch of stuff, speculatively, to hopefully fix that:
-      "@aio"
-      "@basic-io"
-      "@chown"
-      "@file-system"
-      "@io-event"
-      "@process"
-      "@sandbox"
-      "@sync"
-      "@system-service"
-      "quotactl"
-    ];
+    NetworkNamespacePath = "/run/netns/ovpns";
+    Restart = "on-failure";
+    RestartSec = "30s";
+    BindPaths = [ "/var/media" ];  #< so it can move completed torrents into the media library
   };
 
   # service to automatically backup torrents i add to transmission
@@ -157,14 +159,14 @@ in
     # inherit kTLS;
     locations."/" = {
       # proxyPass = "http://ovpns.uninsane.org:9091";
-      proxyPass = "http://${config.sane.netns.ovpns.netnsVethIpv4}:9091";
+      proxyPass = "http://10.0.1.6:9091";
     };
   };
 
   sane.dns.zones."uninsane.org".inet.CNAME."bt" = "native";
   sane.ports.ports."51413" = {
     protocol = [ "tcp" "udp" ];
-    # visibleTo.ovpns = true;  #< not needed: it runs in the ovpns namespace
+    visibleTo.ovpn = true;
     description = "colin-bittorrent";
   };
 }

hosts/by-name/servo/services/transmission/torrent-done

@@ -1,69 +0,0 @@
-#!/usr/bin/env nix-shell
-#!nix-shell -i bash -p acl -p bash -p coreutils -p findutils -p rsync
-
-# transmission invokes this with no args, and the following env vars:
-# - TR_TORRENT_DIR: full path to the folder i told transmission to download it to.
-#     e.g. /var/media/torrents/Videos/Film/Jason.Bourne-2016
-# optionally:
-# - TR_DRY_RUN=1
-# - TR_DEBUG=1
-
-DOWNLOAD_DIR=/var/media/torrents
-
-destructive() {
-  if [ -n "${TR_DRY_RUN-}" ]; then
-    echo "[dry-run] $*"
-  else
-    debug "$@"
-    "$@"
-  fi
-}
-debug() {
-  if [ -n "${TR_DEBUG-}" ]; then
-    echo "$@"
-  fi
-}
-
-echo "TR_TORRENT_DIR=$TR_TORRENT_DIR torrent-done $*"
-
-if [[ "$TR_TORRENT_DIR" =~ ^.*freeleech.*$ ]]; then
-  # freeleech torrents have no place in my permanent library
-  echo "freeleech: nothing to do"
-  exit 0
-fi
-if ! [[ "$TR_TORRENT_DIR" =~ ^$DOWNLOAD_DIR/.*$ ]]; then
-  echo "unexpected torrent dir, aborting: $TR_TORRENT_DIR"
-  exit 0
-fi
-
-REL_DIR="${TR_TORRENT_DIR#$DOWNLOAD_DIR/}"
-MEDIA_DIR="/var/media/$REL_DIR"
-
-destructive mkdir -p "$(dirname "$MEDIA_DIR")"
-destructive rsync -arv "$TR_TORRENT_DIR/" "$MEDIA_DIR/"
-# make the media rwx by anyone in the group
-destructive find "$MEDIA_DIR" -type d -exec setfacl --recursive --modify d:g::rwx,o::rx {} \;
-destructive find "$MEDIA_DIR" -type d -exec chmod g+rw,a+rx {} \;
-
-# if there's a single directory inside the media dir, then inline that
-subdirs=("$MEDIA_DIR"/*)
-debug "top-level items in torrent dir:" "${subdirs[@]}"
-if [ ${#subdirs[@]} -eq 1 ]; then
-  dirname="${subdirs[0]}"
-  debug "exactly one top-level item, checking if directory: $dirname"
-  if [ -d "$dirname" ]; then
-    destructive mv "$dirname"/* "$MEDIA_DIR/" && destructive rmdir "$dirname"
-  fi
-fi
-
-# remove noisy files:
-# -iname means "insensitive", but the syntax is NOT regex -- more similar to shell matching
-destructive find "$MEDIA_DIR/" -type f \(\
-     -iname '*downloaded?from*' \
-  -o -iname 'source.txt' \
-  -o -iname '*upcoming?releases*' \
-  -o -iname 'www.YTS*.jpg' \
-  -o -iname 'WWW.YIFY*.COM.jpg' \
-  -o -iname 'YIFY*.com.txt' \
-  -o -iname 'YTS*.com.txt' \
-  \) -exec rm {} \;

hosts/by-name/servo/services/trust-dns.nix

@@ -2,16 +2,24 @@
 { config, lib, pkgs, ... }:
 
 let
-  dyn-dns = config.sane.services.dyn-dns;
   nativeAddrs = lib.mapAttrs (_name: builtins.head) config.sane.dns.zones."uninsane.org".inet.A;
-in
+  bindOvpn = "10.0.1.5";
+in lib.mkMerge [
 {
+  services.trust-dns.enable = true;
+
+  # don't bind to IPv6 until i explicitly test that stack
+  services.trust-dns.settings.listen_addrs_ipv6 = [];
+  services.trust-dns.quiet = true;
+  # FIXME(2023/11/26): services.trust-dns.debug doesn't log requests: use RUST_LOG=debug env for that.
+  # - see: <https://github.com/hickory-dns/hickory-dns/issues/2082>
+  # services.trust-dns.debug = true;
+
   sane.ports.ports."53" = {
     protocol = [ "udp" "tcp" ];
     visibleTo.lan = true;
-    # visibleTo.wan = true;
-    visibleTo.ovpns = true;
-    visibleTo.doof = true;
+    visibleTo.wan = true;
+    visibleTo.ovpn = true;
     description = "colin-dns-hosting";
   };
 
@@ -39,7 +47,6 @@ in
     CNAME."native" = "%CNAMENATIVE%";
     A."@" =      "%ANATIVE%";
     A."servo.wan" = "%AWAN%";
-    A."servo.doof" = "%ADOOF%";
     A."servo.lan" = config.sane.hosts.by-name."servo".lan-ip;
     A."servo.hn" = config.sane.hosts.by-name."servo".wg-home.ip;
 
@@ -47,99 +54,157 @@ in
     # it's best that we keep this identical, or a superset of, what org. lists as our NS.
     # so, org. can specify ns2/ns3 as being to the VPN, with no mention of ns1. we provide ns1 here.
     A."ns1" =    "%ANATIVE%";
-    A."ns2" =    "%ADOOF%";
-    A."ovpns" =  "%AOVPNS%";
+    A."ns2" =    "185.157.162.178";
+    A."ns3" =    "185.157.162.178";
+    A."ovpns" =  "185.157.162.178";
     NS."@" = [
       "ns1.uninsane.org."
       "ns2.uninsane.org."
+      "ns3.uninsane.org."
     ];
   };
 
   services.trust-dns.settings.zones = [ "uninsane.org" ];
 
+  # TODO: can i transform this into some sort of service group?
+  # have `systemctl restart trust-dns.service` restart all the individual services?
+  systemd.services.trust-dns.serviceConfig = {
+    DynamicUser = lib.mkForce false;
+    User = "trust-dns";
+    Group = "trust-dns";
+    wantedBy = lib.mkForce [];
+  };
+  systemd.services.trust-dns.enable = false;
 
-  networking.nat.enable = true;  #< TODO: try removing this?
-  # networking.nat.extraCommands = ''
-  #   # redirect incoming DNS requests from LAN addresses
-  #   #   to the LAN-specialized DNS service
-  #   # N.B.: use the `nixos-*` chains instead of e.g. PREROUTING
-  #   #   because they get cleanly reset across activations or `systemctl restart firewall`
-  #   #   instead of accumulating cruft
-  #   iptables -t nat -A nixos-nat-pre -p udp --dport 53 \
-  #     -m iprange --src-range 10.78.76.0-10.78.79.255 \
-  #     -j DNAT --to-destination :1053
-  #   iptables -t nat -A nixos-nat-pre -p tcp --dport 53 \
-  #     -m iprange --src-range 10.78.76.0-10.78.79.255 \
-  #     -j DNAT --to-destination :1053
-  # '';
-  # sane.ports.ports."1053" = {
-  #   # because the NAT above redirects in nixos-nat-pre, LAN requests behave as though they arrived on the external interface at the redirected port.
-  #   # TODO: try nixos-nat-post instead?
-  #   # TODO: or, don't NAT from port 53 -> port 1053, but rather nat from LAN addr to a loopback addr.
-  #   # - this is complicated in that loopback is a different interface than eth0, so rewriting the destination address would cause the packets to just be dropped by the interface
-  #   protocol = [ "udp" "tcp" ];
-  #   visibleTo.lan = true;
-  #   description = "colin-redirected-dns-for-lan-namespace";
-  # };
-
-
-  sane.services.trust-dns.enable = true;
-  sane.services.trust-dns.instances = let
-    mkSubstitutions = flavor: {
-      "%ADOOF%" = config.sane.netns.doof.netnsPubIpv4;
-      "%ANATIVE%" = nativeAddrs."servo.${flavor}";
-      "%AOVPNS%" = config.sane.netns.ovpns.netnsPubIpv4;
-      "%AWAN%" = "$(cat '${dyn-dns.ipPath}')";
-      "%CNAMENATIVE%" = "servo.${flavor}";
-    };
-  in
-  {
-    doof = {
-      substitutions = mkSubstitutions "doof";
-      listenAddrsIpv4 = [
-        config.sane.netns.doof.hostVethIpv4
-        config.sane.netns.doof.netnsPubIpv4
-        nativeAddrs."servo.lan"
-        # config.sane.netns.ovpns.hostVethIpv4
-      ];
-    };
-    hn = {
-      substitutions = mkSubstitutions "hn";
-      listenAddrsIpv4 = [ nativeAddrs."servo.hn" ];
-      enableRecursiveResolver = true;  #< allow wireguard clients to use this as their DNS resolver
-      # extraConfig = {
-      #   zones = [
-      #     {
-      #       # forward the root zone to the local DNS resolver
-      #       # to allow wireguard clients to use this as their DNS resolver
-      #       zone = ".";
-      #       zone_type = "Forward";
-      #       stores = {
-      #         type = "forward";
-      #         name_servers = [
-      #           {
-      #             socket_addr = "127.0.0.53:53";
-      #             protocol = "udp";
-      #             trust_nx_responses = true;
-      #           }
-      #         ];
-      #       };
-      #     }
-      #   ];
-      # };
-    };
-    # lan = {
-    #   substitutions = mkSubstitutions "lan";
-    #   listenAddrsIpv4 = [ nativeAddrs."servo.lan" ];
-    #   # port = 1053;
-    # };
-    # wan = {
-    #   substitutions = mkSubstitutions "wan";
-    #   listenAddrsIpv4 = [
-    #     nativeAddrs."servo.lan"
-    #   ];
-    # };
+  users.groups.trust-dns = {};
+  users.users.trust-dns = {
+    group = "trust-dns";
+    isSystemUser = true;
   };
 
-  sane.services.dyn-dns.restartOnChange = lib.map (c: "${c.service}.service") (builtins.attrValues config.sane.services.trust-dns.instances);
+  # sane.services.dyn-dns.restartOnChange = [ "trust-dns.service" ];
+
+  networking.nat.enable = true;
+  networking.nat.extraCommands = ''
+    # redirect incoming DNS requests from LAN addresses
+    #   to the LAN-specialized DNS service
+    # N.B.: use the `nixos-*` chains instead of e.g. PREROUTING
+    #   because they get cleanly reset across activations or `systemctl restart firewall`
+    #   instead of accumulating cruft
+    iptables -t nat -A nixos-nat-pre -p udp --dport 53 \
+      -m iprange --src-range 10.78.76.0-10.78.79.255 \
+      -j DNAT --to-destination :1053
+    iptables -t nat -A nixos-nat-pre -p tcp --dport 53 \
+      -m iprange --src-range 10.78.76.0-10.78.79.255 \
+      -j DNAT --to-destination :1053
+  '';
+  sane.ports.ports."1053" = {
+    # because the NAT above redirects in nixos-nat-pre, LAN requests behave as though they arrived on the external interface at the redirected port.
+    # TODO: try nixos-nat-post instead?
+    # TODO: or, don't NAT from port 53 -> port 1053, but rather nat from LAN addr to a loopback addr.
+    # - this is complicated in that loopback is a different interface than eth0, so rewriting the destination address would cause the packets to just be dropped by the interface
+    protocol = [ "udp" "tcp" ];
+    visibleTo.lan = true;
+    description = "colin-redirected-dns-for-lan-namespace";
+  };
 }
+{
+  systemd.services =
+    let
+      sed = "${pkgs.gnused}/bin/sed";
+      stateDir = "/var/lib/trust-dns";
+      zoneTemplate = pkgs.writeText "uninsane.org.zone.in" config.sane.dns.zones."uninsane.org".rendered;
+
+      zoneDirFor = flavor: "${stateDir}/${flavor}";
+      zoneFor = flavor: "${zoneDirFor flavor}/uninsane.org.zone";
+      mkTrustDnsService = opts: flavor: let
+        flags = let baseCfg = config.services.trust-dns; in
+          (lib.optional baseCfg.debug "--debug") ++ (lib.optional baseCfg.quiet "--quiet");
+        flagsStr = builtins.concatStringsSep " " flags;
+
+        anative = nativeAddrs."servo.${flavor}";
+
+        toml = pkgs.formats.toml { };
+        configTemplate = opts.config or (toml.generate "trust-dns-${flavor}.toml" (
+          (
+            lib.filterAttrsRecursive (_: v: v != null) config.services.trust-dns.settings
+          ) // {
+            listen_addrs_ipv4 = opts.listen or [ anative ];
+          }
+        ));
+        configFile = "${stateDir}/${flavor}-config.toml";
+
+        port = opts.port or 53;
+      in {
+        description = "trust-dns Domain Name Server (serving ${flavor})";
+        unitConfig.Documentation = "https://trust-dns.org/";
+
+        preStart = ''
+          wan=$(cat '${config.sane.services.dyn-dns.ipPath}')
+          ${sed} s/%AWAN%/$wan/ ${configTemplate} > ${configFile}
+        '' + lib.optionalString (!opts ? config) ''
+          mkdir -p ${zoneDirFor flavor}
+          ${sed} \
+            -e s/%CNAMENATIVE%/servo.${flavor}/ \
+            -e s/%ANATIVE%/${anative}/ \
+            -e s/%AWAN%/$wan/ \
+            -e s/%AOVPNS%/185.157.162.178/ \
+            ${zoneTemplate} > ${zoneFor flavor}
+        '';
+        serviceConfig = config.systemd.services.trust-dns.serviceConfig // {
+          ExecStart = ''
+            ${pkgs.trust-dns}/bin/${pkgs.trust-dns.meta.mainProgram} \
+            --port ${builtins.toString port} \
+            --zonedir ${zoneDirFor flavor}/ \
+            --config ${configFile} ${flagsStr}
+          '';
+        };
+
+        after = [ "network.target" ];
+        wantedBy = [ "multi-user.target" ];
+      };
+    in {
+      trust-dns-wan = mkTrustDnsService { listen = [ nativeAddrs."servo.lan" bindOvpn ]; } "wan";
+      trust-dns-lan = mkTrustDnsService { port = 1053; } "lan";
+      trust-dns-hn = mkTrustDnsService { port = 1053; } "hn";
+      trust-dns-hn-resolver = mkTrustDnsService {
+        config = pkgs.writeText "hn-resolver-config.toml" ''
+          # i host a resolver in the wireguard VPN so that clients can resolve DNS through the VPN.
+          # (that's what this file achieves).
+          #
+          # one would expect this resolver could host the authoritative zone for `uninsane.org`, and then forward everything else to the system resolver...
+          # and while that works for `dig`, it breaks for `nslookup` (and so `ssh`, etc).
+          #
+          # DNS responses include a flag for if the responding server is the authority of the zone queried.
+          # it seems that default Linux stub resolvers either:
+          # - expect DNSSEC when the response includes that bit, or
+          # - expect A records to be in the `answer` section instead of `additional` section.
+          # or perhaps something more nuanced. but for `nslookup` to be reliable, it has to talk to an
+          # instance of trust-dns which is strictly a resolver, with no authority.
+          # hence, this config: a resolver which forwards to the actual authority.
+
+          listen_addrs_ipv4 = ["${nativeAddrs."servo.hn"}"]
+          listen_addrs_ipv6 = []
+
+          [[zones]]
+          zone = "uninsane.org"
+          zone_type = "Forward"
+          stores = { type = "forward", name_servers = [{ socket_addr = "${nativeAddrs."servo.hn"}:1053", protocol = "udp", trust_nx_responses = true }] }
+
+          [[zones]]
+          # forward the root zone to the local DNS resolver
+          zone = "."
+          zone_type = "Forward"
+          stores = { type = "forward", name_servers = [{ socket_addr = "127.0.0.53:53", protocol = "udp", trust_nx_responses = true }] }
+        '';
+      } "hn-resolver";
+    };
+
+  sane.services.dyn-dns.restartOnChange = [
+    "trust-dns-wan.service"
+    "trust-dns-lan.service"
+    "trust-dns-hn.service"
+    # "trust-dns-hn-resolver.service"  # doesn't need restart because it doesn't know about WAN IP
+  ];
+}
+]

hosts/common/boot.nix

@@ -1,51 +0,0 @@
-{ lib, pkgs, ... }:
-{
-  boot.initrd.supportedFilesystems = [ "ext4" "btrfs" "ext2" "ext3" "vfat" ];
-  # useful emergency utils
-  boot.initrd.extraUtilsCommands = ''
-    copy_bin_and_libs ${pkgs.btrfs-progs}/bin/btrfstune
-    copy_bin_and_libs ${pkgs.util-linux}/bin/{cfdisk,lsblk,lscpu}
-    copy_bin_and_libs ${pkgs.gptfdisk}/bin/{cgdisk,gdisk}
-    copy_bin_and_libs ${pkgs.smartmontools}/bin/smartctl
-    copy_bin_and_libs ${pkgs.e2fsprogs}/bin/resize2fs
-  '' + lib.optionalString pkgs.stdenv.hostPlatform.isx86_64 ''
-    copy_bin_and_libs ${pkgs.nvme-cli}/bin/nvme  # doesn't cross compile
-  '';
-  boot.kernelParams = [
-    "boot.shell_on_fail"
-    #v experimental full pre-emption for hopefully better call/audio latency on moby.
-    # also toggleable at runtime via /sys/kernel/debug/sched/preempt
-    # defaults to preempt=voluntary
-    # "preempt=full"
-  ];
-  # other kernelParams:
-  #   "boot.trace"
-  #   "systemd.log_level=debug"
-  #   "systemd.log_target=console"
-
-  # moby has to run recent kernels (defined elsewhere).
-  # meanwhile, kernel variation plays some minor role in things like sandboxing (landlock) and capabilities.
-  # simpler to keep near the latest kernel on all devices,
-  # and also makes certain that any weird system-level bugs i see aren't likely to be stale kernel bugs.
-  # servo needs zfs though, which doesn't support every kernel.
-  boot.kernelPackages = lib.mkDefault pkgs.zfs.latestCompatibleLinuxPackages;
-
-  # hack in the `boot.shell_on_fail` arg since that doesn't always seem to work.
-  boot.initrd.preFailCommands = "allowShell=1";
-
-  # default: 4 (warn). 7 is debug
-  boot.consoleLogLevel = 7;
-
-  boot.loader.grub.enable = lib.mkDefault false;
-  boot.loader.generic-extlinux-compatible.enable = lib.mkDefault true;
-
-  hardware.enableAllFirmware = true;  # firmware with licenses that don't allow for redistribution. fuck lawyers, fuck IP, give me the goddamn firmware.
-  # hardware.enableRedistributableFirmware = true;  # proprietary but free-to-distribute firmware (extraneous to `enableAllFirmware` option)
-
-  # default is 252274, which is too low particularly for servo.
-  # manifests as spurious "No space left on device" when trying to install watches,
-  # e.g. in dyn-dns by `systemctl start dyn-dns-watcher.path`.
-  # see: <https://askubuntu.com/questions/828779/failed-to-add-run-systemd-ask-password-to-directory-watch-no-space-left-on-dev>
-  boot.kernel.sysctl."fs.inotify.max_user_watches" = 4194304;
-  boot.kernel.sysctl."fs.inotify.max_user_instances" = 4194304;
-}

hosts/common/default.nix

@@ -1,29 +1,24 @@
 { config, lib, pkgs, ... }:
 {
   imports = [
-    ./boot.nix
     ./feeds.nix
     ./fs.nix
+    ./hardware
     ./home
     ./hosts.nix
     ./ids.nix
     ./machine-id.nix
     ./net
-    ./nix.nix
+    ./nix
+    ./persist.nix
     ./polyunfill.nix
     ./programs
-    ./quirks.nix
     ./secrets.nix
     ./ssh.nix
     ./systemd.nix
     ./users
   ];
 
-
-  # docs: https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion
-  # this affects where nixos modules look for stateful data which might have been migrated across releases.
-  system.stateVersion = "21.11";
-
   sane.nixcache.enable-trusted-keys = true;
   sane.nixcache.enable = lib.mkDefault true;
   sane.persist.enable = lib.mkDefault true;
@@ -31,6 +26,9 @@
   sane.programs.sysadminUtils.enableFor.system = lib.mkDefault true;
   sane.programs.consoleUtils.enableFor.user.colin = lib.mkDefault true;
 
+  nixpkgs.config.allowUnfree = true;  # NIXPKGS_ALLOW_UNFREE=1
+  nixpkgs.config.allowBroken = true;  # NIXPKGS_ALLOW_BROKEN=1
+
   # time.timeZone = "America/Los_Angeles";
   time.timeZone = "Etc/UTC";  # DST is too confusing for me => use a stable timezone
 

hosts/common/feeds.nix

@@ -1,11 +1,14 @@
 # where to find good stuff?
 # - universal search/directory: <https://podcastindex.org>
-# - list of lists: <https://en.wikipedia.org/wiki/Category:Lists_of_podcasts>
 # - podcasts w/ a community: <https://lemmyverse.net/communities?query=podcast>
-# - podcast recs:
-#   - active lemmy: <https://slrpnk.net/c/podcasts>
-#   - old thread: <https://lemmy.ml/post/1565858>
+# - podcast rec thread: <https://lemmy.ml/post/1565858>
 #
+# candidates:
+# - The Nonlinear Library (podcast): <https://forum.effectivealtruism.org/posts/JTZTBienqWEAjGDRv/listen-to-more-ea-content-with-the-nonlinear-library>
+#   - has ~10 posts per day, text-to-speech; i would need better tagging before adding this
+# - <https://www.metaculus.com/questions/11102/introducing-the-metaculus-journal-podcast/>
+#   - dead since 2022/10 - 2023/03
+
 { lib, sane-data, ... }:
 let
   hourly = { freq = "hourly"; };
@@ -72,25 +75,20 @@ let
     (fromDb "feeds.feedburner.com/radiolab" // pol)  # Radiolab -- also available here, but ONLY OVER HTTP: <http://feeds.wnyc.org/radiolab>
     (fromDb "feeds.megaphone.fm/behindthebastards" // pol)  # also Maggie Killjoy
     (fromDb "feeds.megaphone.fm/recodedecode" // tech)  # The Verge - Decoder
+    (fromDb "feeds.simplecast.com/54nAGcIl" // pol)  # The Daily
     (fromDb "feeds.simplecast.com/82FI35Px" // pol)  # Ezra Klein Show
     (fromDb "feeds.simplecast.com/wgl4xEgL" // rat)  # Econ Talk
     (fromDb "feeds.simplecast.com/xKJ93w_w" // uncat)  # Atlas Obscura
     (fromDb "feeds.transistor.fm/acquired" // tech)
-    (fromDb "feeds.transistor.fm/complex-systems-with-patrick-mckenzie-patio11" // tech)  # Patrick Mackenzie (from Bits About Money)
-    (fromDb "feeds.twit.tv/floss.xml" // tech)
     (fromDb "fulltimenix.com" // tech)
-    (fromDb "futureofcoding.org/episodes" // tech)
     (fromDb "hackerpublicradio.org" // tech)
     (fromDb "lexfridman.com/podcast" // rat)
-    (fromDb "linktr.ee/betteroffline" // pol)
     (fromDb "mapspodcast.libsyn.com" // uncat)  # Multidisciplinary Association for Psychedelic Studies
-    (fromDb "microarch.club" // tech)
-    (fromDb "mintcast.org" // tech)
     (fromDb "omegataupodcast.net" // tech)  # 3/4 German; 1/4 eps are English
     (fromDb "omny.fm/shows/cool-people-who-did-cool-stuff" // pol)  # Maggie Killjoy -- referenced by Cory Doctorow
-    (fromDb "omny.fm/shows/money-stuff-the-podcast")  # Matt Levine
     (fromDb "omny.fm/shows/the-dollop-with-dave-anthony-and-gareth-reynolds")  # The Dollop history/comedy
     (fromDb "originstories.libsyn.com" // uncat)
+    (fromDb "podcast.posttv.com/itunes/post-reports.xml" // pol)
     (fromDb "politicalorphanage.libsyn.com" // pol)
     (fromDb "reverseengineering.libsyn.com/rss" // tech)  # UnNamed Reverse Engineering Podcast
     (fromDb "rss.acast.com/deconstructed")  # The Intercept - Deconstructed
@@ -101,11 +99,8 @@ let
     (fromDb "seattlenice.buzzsprout.com" // pol)
     (fromDb "srslywrong.com" // pol)
     (fromDb "sharkbytes.transistor.fm" // tech)  # Wireshark Podcast o_0
-    (fromDb "sharptech.fm/feed/podcast" // tech)
     (fromDb "sscpodcast.libsyn.com" // rat)  # Astral Codex Ten
     (fromDb "talesfromthebridge.buzzsprout.com" // tech)  # Sci-Fi? has Peter Watts; author of No Moods, Ads or Cutesy Fucking Icons (rifters.com)
-    (fromDb "theamphour.com" // tech)
-    (fromDb "techtalesshow.com" // tech)  # Corbin Davenport
     (fromDb "techwontsave.us" // pol)  # rec by Cory Doctorow
     (fromDb "wakingup.libsyn.com" // pol)  # Sam Harris
     (fromDb "werenotwrong.fireside.fm" // pol)
@@ -113,9 +108,7 @@ let
 
     # (fromDb "feeds.libsyn.com/421877" // rat)  # Less Wrong Curated
     # (fromDb "feeds.megaphone.fm/hubermanlab" // uncat)  # Daniel Huberman on sleep
-    # (fromDb "feeds.simplecast.com/54nAGcIl" // pol)  # The Daily
     # (fromDb "feeds.simplecast.com/l2i9YnTd" // tech // pol)  # Hard Fork (NYtimes tech)
-    # (fromDb "podcast.posttv.com/itunes/post-reports.xml" // pol)
     # (fromDb "podcast.thelinuxexp.com" // tech)  # low-brow linux/foss PR announcements
     # (fromDb "rss.art19.com/your-welcome" // pol)  # Michael Malice - Your Welcome -- also available here: <https://origin.podcastone.com/podcast?categoryID2=2232>
     # (fromDb "rss.prod.firstlook.media/deconstructed/podcast.rss" // pol)  #< possible URL rot
@@ -131,12 +124,11 @@ let
     (fromDb "acoup.blog/feed")  # history, states. author: <https://historians.social/@bretdevereaux/following>
     (fromDb "amosbbatto.wordpress.com" // tech)
     (fromDb "anish.lakhwara.com" // tech)
-    (fromDb "apenwarr.ca/log/rss.php" // tech)  # CEO of tailscale
     (fromDb "applieddivinitystudies.com" // rat)
     (fromDb "artemis.sh" // tech)
     (fromDb "ascii.textfiles.com" // tech)  # Jason Scott
     (fromDb "austinvernon.site" // tech)
-    (fromDb "buttondown.email" // tech)
+    # (fromDb "balajis.com" // pol)  # Balaji
     (fromDb "ben-evans.com/benedictevans" // pol)
     (fromDb "bitbashing.io" // tech)
     (fromDb "bitsaboutmoney.com" // uncat)
@@ -147,6 +139,8 @@ let
     (fromDb "blog.thalheim.io" // tech)  # Mic92
     (fromDb "bunniestudios.com" // tech)  # Bunnie Juang
     (fromDb "capitolhillseattle.com" // pol)
+    # (fromDb "drewdevault.com" // tech)
+    # (fromDb "econlib.org" // pol)
     (fromDb "edwardsnowden.substack.com" // pol // text)
     (fromDb "fasterthanli.me" // tech)
     (fromDb "gwern.net" // rat)
@@ -157,9 +151,9 @@ let
     (fromDb "interconnected.org/home/feed" // rat)  # Matt Webb -- engineering-ish, but dreamy
     (fromDb "jeffgeerling.com" // tech)
     (fromDb "jefftk.com" // tech)
-    (fromDb "jwz.org/blog" // tech // pol)  # DNA lounge guy, loooong-time blogger
     (fromDb "kill-the-newsletter.com/feeds/joh91bv7am2pnznv.xml" // pol)  # Matt Levine - Money Stuff
     (fromDb "kosmosghost.github.io/index.xml" // tech)
+    # (fromDb "lesswrong.com" // rat)
     (fromDb "linmob.net" // tech)
     (fromDb "lwn.net" // tech)
     (fromDb "lynalden.com" // pol)
@@ -174,13 +168,13 @@ let
     (fromDb "pomeroyb.com" // tech)
     (fromDb "postmarketos.org/blog" // tech)
     (fromDb "preposterousuniverse.com" // rat)  # Sean Carroll
+    (fromDb "profectusmag.com" // uncat)
     (fromDb "project-insanity.org" // tech)  # shared blog by a few NixOS devs, notably onny
     (fromDb "putanumonit.com" // rat)  # mostly dating topics. not advice, or humor, but looking through a social lens
     (fromDb "richardcarrier.info" // rat)
     (fromDb "rifters.com/crawl" // uncat)  # No Moods, Ads or Cutesy Fucking Icons
     (fromDb "righto.com" // tech)  # Ken Shirriff
     (fromDb "rootsofprogress.org" // rat)  # Jason Crawford
-    (fromDb "samuel.dionne-riel.com" // tech)  # SamuelDR
     (fromDb "sagacioussuricata.com" // tech)  # ian (Sanctuary)
     (fromDb "semiaccurate.com" // tech)
     (fromDb "sideways-view.com" // rat)  # Paul Christiano
@@ -189,43 +183,33 @@ let
     (fromDb "spectrum.ieee.org" // tech)
     (fromDb "stpeter.im/atom.xml" // pol)
     (fromDb "thediff.co" // pol)  # Byrne Hobart
+    # (fromDb "theregister.com" // tech)
     (fromDb "thisweek.gnome.org" // tech)
     (fromDb "tuxphones.com" // tech)
     (fromDb "uninsane.org" // tech)
     (fromDb "unintendedconsequenc.es" // rat)
+    # (fromDb "vitalik.ca" // tech)  # moved to vitalik.eth.limo
     (fromDb "vitalik.eth.limo" // tech)  # Vitalik Buterin
-    (fromDb "weekinethereumnews.com" // tech)
+    # (fromDb "webcurious.co.uk" // uncat)  # link aggregator; defunct?
     (fromDb "willow.phantoma.online")  # wizard@xyzzy.link
     (fromDb "xn--gckvb8fzb.com" // tech)
-    (fromDb "xorvoid.com" // tech)
-    (fromDb "www.thebignewsletter.com" // pol)
     (mkSubstack "astralcodexten" // rat // daily)  # Scott Alexander
+    # (mkSubstack "doomberg" // tech // weekly)  # articles are all pay-walled
     (mkSubstack "eliqian" // rat // weekly)
     (mkSubstack "oversharing" // pol // daily)
     (mkSubstack "samkriss" // humor // infrequent)
     (mkText "http://benjaminrosshoffman.com/feed" // pol // weekly)
     (mkText "http://boginjr.com/feed" // tech // infrequent)
     (mkText "https://forum.merveilles.town/rss.xml" // pol // infrequent)  #quality RSS list here: <https://forum.merveilles.town/thread/57/share-your-rss-feeds%21-6/>
+    # (mkText "https://github.com/Kaiteki-Fedi/Kaiteki/commits/master.atom" // tech // infrequent)
     (mkText "https://jvns.ca/atom.xml" // tech // weekly)  # Julia Evans
     (mkText "https://linuxphoneapps.org/blog/atom.xml" // tech // infrequent)
     (mkText "https://nixos.org/blog/announcements-rss.xml" // tech // infrequent)  # more nixos stuff here, but unclear how to subscribe: <https://nixos.org/blog/categories.html>
     (mkText "https://nixos.org/blog/stories-rss.xml" // tech // weekly)
     (mkText "https://solar.lowtechmagazine.com/posts/index.xml" // tech // weekly)
-    (mkText "https://www.stratechery.com/rss" // pol // weekly)  # Ben Thompson
-
-    # (fromDb "balajis.com" // pol)  # Balaji
-    # (fromDb "drewdevault.com" // tech)
-    # (fromDb "econlib.org" // pol)
-    # (fromDb "lesswrong.com" // rat)
-    # (fromDb "profectusmag.com" // pol)  # some conservative/libertarian think tank
-    # (fromDb "thesideview.co" // uncat)  # spiritual journal; RSS items are stubs
-    # (fromDb "theregister.com" // tech)
-    # (fromDb "vitalik.ca" // tech)  # moved to vitalik.eth.limo
-    # (fromDb "webcurious.co.uk" // uncat)  # link aggregator; defunct?
-    # (mkSubstack "doomberg" // tech // weekly)  # articles are all pay-walled
-    # (mkText "https://github.com/Kaiteki-Fedi/Kaiteki/commits/master.atom" // tech // infrequent)
     # (mkText "https://til.simonwillison.net/tils/feed.atom" // tech // weekly)
     # (mkText "https://www.bloomberg.com/opinion/authors/ARbTQlRLRjE/matthew-s-levine.rss" // pol // weekly)  # Matt Levine (preview/paywalled)
+    (mkText "https://www.stratechery.com/rss" // pol // weekly)  # Ben Thompson
   ];
 
   videos = [
@@ -237,20 +221,18 @@ let
     (fromDb "youtube.com/@JackStauber")
     (fromDb "youtube.com/@NativLang")
     (fromDb "youtube.com/@PolyMatter")
+    # (fromDb "youtube.com/@rossmanngroup" // pol // tech)  # Louis Rossmann
     (fromDb "youtube.com/@TechnologyConnections" // tech)
     (fromDb "youtube.com/@TheB1M")
     (fromDb "youtube.com/@TomScottGo")
     (fromDb "youtube.com/@Vihart")
-
-    # (fromDb "youtube.com/@Vox")
-    # (fromDb "youtube.com/@Vsauce")  # they're all like 1-minute long videos now? what happened @Vsauce?
-    # (fromDb "youtube.com/@rossmanngroup" // pol // tech)  # Louis Rossmann
+    (fromDb "youtube.com/@Vox")
+    (fromDb "youtube.com/@Vsauce")
   ];
 
   images = [
     (fromDb "catandgirl.com" // img // humor)
     (fromDb "davidrevoy.com" // img // art)
-    (fromDb "grumpy.website" // img // humor)
     (fromDb "miniature-calendar.com" // img // art // daily)
     (fromDb "pbfcomics.com" // img // humor)
     (fromDb "poorlydrawnlines.com/feed" // img // humor)

hosts/common/fs.nix

@@ -26,6 +26,10 @@ let
     # lazyMount: defer mounting until first access from userspace.
     # see: `man systemd.automount`, `man automount`, `man autofs`
     lazyMount = noauto ++ automount;
+    wg = [
+      "x-systemd.requires=wireguard-wg-home.service"
+      "x-systemd.after=wireguard-wg-home.service"
+    ];
 
     fuse = [
       "allow_other"  # allow users other than the one who mounts it to access it. needed, if systemd is the one mounting this fs (as root)
@@ -103,8 +107,7 @@ let
     ftp = common ++ fuseColin ++ [
       # "ftpfs_debug=2"
       "user=colin:ipauth"
-      # connect_timeout=10: casting shows to T.V. fails partway through about half the time
-      "connect_timeout=20"
+      "connect_timeout=10"
     ];
   };
   remoteHome = host: {
@@ -121,61 +124,52 @@ let
       dir.acl.mode = "0700";
     };
   };
-  remoteServo = subdir: let
-    localPath = "/mnt/servo/${subdir}";
-    systemdName = utils.escapeSystemdPath localPath;
-  in {
+  remoteServo = subdir: {
     sane.programs.curlftpfs.enableFor.system = true;
-    sane.fs."${localPath}" = sane-lib.fs.wanted {
+    sane.fs."/mnt/servo/${subdir}" = sane-lib.fs.wanted {
       dir.acl.user = "colin";
       dir.acl.group = "users";
       dir.acl.mode = "0750";
     };
-    fileSystems."${localPath}" = {
+    fileSystems."/mnt/servo/${subdir}" = {
       device = "ftp://servo-hn:/${subdir}";
       noCheck = true;
       fsType = "fuse.curlftpfs";
-      options = fsOpts.ftp ++ fsOpts.noauto;
+      options = fsOpts.ftp ++ fsOpts.noauto ++ fsOpts.wg;
       # fsType = "nfs";
-      # options = fsOpts.nfs ++ fsOpts.lazyMount;
+      # options = fsOpts.nfs ++ fsOpts.lazyMount ++ fsOpts.wg;
     };
-
-    systemd.mounts = let
-      fsEntry = config.fileSystems."${localPath}";
-    in [{
-      #VVV repeat what systemd would ordinarily scrape from /etc/fstab
-      where = localPath;
-      what = fsEntry.device;
-      type = fsEntry.fsType;
-      options = lib.concatStringsSep "," fsEntry.options;
+    systemd.services."automount-servo-${utils.escapeSystemdPath subdir}" = let
+      fs = config.fileSystems."/mnt/servo/${subdir}";
+    in {
+      # this is a *flaky* network mount, especially on moby.
+      # if done as a normal autofs mount, access will eternally block when network is dropped.
+      # notably, this would block *any* sandboxed app which allows media access, whether they actually try to use that media or not.
+      # a practical solution is this: mount as a service -- instead of autofs -- and unmount on timeout error, in a restart loop.
+      # until the ftp handshake succeeds, nothing is actually mounted to the vfs, so this doesn't slow down any I/O when network is down.
+      description = "automount /mnt/servo/${subdir} in a fault-tolerant and non-blocking manner";
       after = [ "network-online.target" ];
       requires = [ "network-online.target" ];
-      wantedBy = [ "default.target" ];  #< TODO: move this into nixos fileSystems
-      #VVV patch so that when the mount fails, we start a timer to remount it.
-      #    and for a disconnection after a good mount (onSuccess), restart the timer to be more aggressive
-      onFailure = [ "${systemdName}.timer" ];
-      onSuccess = [ "${systemdName}-restart-timer.target" ];
-    }];
-    systemd.targets."${systemdName}-restart-timer" = {
-      # hack unit which, when started, stops the timer (if running), and then starts it again.
-      after = [ "${systemdName}.timer" ];
-      conflicts = [ "${systemdName}.timer" ];
-      upholds = [ "${systemdName}.timer" ];
-      unitConfig.StopWhenUnneeded = true;
-    };
-    systemd.timers."${systemdName}" = {
-      timerConfig.Unit = "${systemdName}.mount";
-      timerConfig.AccuracySec = "2s";
-      timerConfig.OnActiveSec = [
-        # try to remount at these timestamps, backing off gradually
-        # there seems to be an implicit mount attempt at t=0.
-        "10s"
-        "30s"
-        "60s"
-        "120s"
+      wantedBy = [ "default.target" ];
+
+      serviceConfig.Type = "simple";
+      serviceConfig.ExecStart = lib.escapeShellArgs [
+        "/usr/bin/env"
+        "PATH=/run/current-system/sw/bin"
+        "mount.${fs.fsType}"
+        "-f"  # foreground (i.e. don't daemonize)
+        "-s"  # single-threaded (TODO: it's probably ok to disable this?)
+        "-o"
+        (lib.concatStringsSep "," (lib.filter (o: !lib.hasPrefix "x-systemd." o) fs.options))
+        fs.device
+        "/mnt/servo/${subdir}"
       ];
-      # cap the backoff to a fixed interval.
-      timerConfig.OnUnitActiveSec = [ "120s" ];
+      # not sure if this configures a linear, or exponential backoff.
+      # but the first restart will be after `RestartSec`, and the n'th restart (n = RestartSteps) will be RestartMaxDelaySec after the n-1'th exit.
+      serviceConfig.Restart = "always";
+      serviceConfig.RestartSec = "10s";
+      serviceConfig.RestartMaxDelaySec = "120s";
+      serviceConfig.RestartSteps = "5";
     };
   };
 in
@@ -221,7 +215,6 @@ lib.mkMerge [
     programs.fuse.userAllowOther = true;  #< necessary for `allow_other` or `allow_root` options.
   }
 
-  (remoteHome "crappy")
   (remoteHome "desko")
   (remoteHome "lappy")
   (remoteHome "moby")
@@ -232,10 +225,10 @@ lib.mkMerge [
   (remoteServo "media/Books")
   (remoteServo "media/collections")
   # (remoteServo "media/datasets")
+  (remoteServo "media/freeleech")
   (remoteServo "media/games")
   (remoteServo "media/Music")
   (remoteServo "media/Pictures/macros")
-  (remoteServo "media/torrents")
   (remoteServo "media/Videos")
   (remoteServo "playground")
 ]

hosts/common/hardware/default.nix

@@ -0,0 +1,101 @@
+{ config, lib, pkgs, ... }:
+
+{
+  imports = [
+    ./x86_64.nix
+  ];
+
+  boot.initrd.supportedFilesystems = [ "ext4" "btrfs" "ext2" "ext3" "vfat" ];
+  # useful emergency utils
+  boot.initrd.extraUtilsCommands = ''
+    copy_bin_and_libs ${pkgs.btrfs-progs}/bin/btrfstune
+    copy_bin_and_libs ${pkgs.util-linux}/bin/{cfdisk,lsblk,lscpu}
+    copy_bin_and_libs ${pkgs.gptfdisk}/bin/{cgdisk,gdisk}
+    copy_bin_and_libs ${pkgs.smartmontools}/bin/smartctl
+    copy_bin_and_libs ${pkgs.e2fsprogs}/bin/resize2fs
+  '' + lib.optionalString pkgs.stdenv.hostPlatform.isx86_64 ''
+    copy_bin_and_libs ${pkgs.nvme-cli}/bin/nvme  # doesn't cross compile
+  '';
+  boot.kernelParams = [
+    "boot.shell_on_fail"
+    #v experimental full pre-emption for hopefully better call/audio latency on moby.
+    # also toggleable at runtime via /sys/kernel/debug/sched/preempt
+    # defaults to preempt=voluntary
+    # "preempt=full"
+  ];
+  # other kernelParams:
+  #   "boot.trace"
+  #   "systemd.log_level=debug"
+  #   "systemd.log_target=console"
+
+  # moby has to run recent kernels (defined elsewhere).
+  # meanwhile, kernel variation plays some minor role in things like sandboxing (landlock) and capabilities.
+  # simpler to keep near the latest kernel on all devices,
+  # and also makes certain that any weird system-level bugs i see aren't likely to be stale kernel bugs.
+  # servo needs zfs though, which doesn't support every kernel.
+  boot.kernelPackages = lib.mkDefault pkgs.zfs.latestCompatibleLinuxPackages;
+
+  # TODO: remove after linux 6.9. see: <https://github.com/axboe/liburing/issues/1113>
+  # - <https://github.com/neovim/neovim/issues/28149>
+  # - <https://git.kernel.dk/cgit/linux/commit/?h=io_uring-6.9&id=e5444baa42e545bb929ba56c497e7f3c73634099>
+  # when removing, try starting and suspending (ctrl+z) two instances of neovim simultaneously.
+  # if the system doesn't freeze, then this is safe to remove.
+  # added 2024-04-04
+  sane.user.fs.".profile".symlink.text = lib.mkBefore ''
+    export UV_USE_IO_URING=0
+  '';
+
+  # hack in the `boot.shell_on_fail` arg since that doesn't always seem to work.
+  boot.initrd.preFailCommands = "allowShell=1";
+
+  # default: 4 (warn). 7 is debug
+  boot.consoleLogLevel = 7;
+
+  boot.loader.grub.enable = lib.mkDefault false;
+  boot.loader.generic-extlinux-compatible.enable = lib.mkDefault true;
+
+  # non-free firmware
+  hardware.enableRedistributableFirmware = true;
+
+  # default is 252274, which is too low particularly for servo.
+  # manifests as spurious "No space left on device" when trying to install watches,
+  # e.g. in dyn-dns by `systemctl start dyn-dns-watcher.path`.
+  # see: <https://askubuntu.com/questions/828779/failed-to-add-run-systemd-ask-password-to-directory-watch-no-space-left-on-dev>
+  boot.kernel.sysctl."fs.inotify.max_user_watches" = 1048576;
+
+  # powertop will default to putting USB devices -- including HID -- to sleep after TWO SECONDS
+  powerManagement.powertop.enable = false;
+  # linux CPU governor: <https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt>
+  # - options:
+  #   - "powersave" => force CPU to always run at lowest supported frequency
+  #   - "performance" => force CPU to always run at highest frequency
+  #   - "ondemand" => adjust frequency based on load
+  #   - "conservative"  (ondemand but slower to adjust)
+  #   - "schedutil"
+  #   - "userspace"
+  # - not all options are available for all platforms
+  #   - intel (intel_pstate) appears to manage scaling w/o intervention/control from the OS.
+  #   - AMD (acpi-cpufreq) appears to manage scaling via the OS *or* HW. but the ondemand defaults never put it to max hardware frequency.
+  #   - qualcomm (cpufreq-dt) appears to manage scaling *only* via the OS. ondemand governor exercises the full range.
+  # - query details with `sudo cpupower frequency-info`
+  powerManagement.cpuFreqGovernor = "ondemand";
+
+  services.logind.extraConfig = ''
+    # see: `man logind.conf`
+    # don’t shutdown when power button is short-pressed (commonly done an accident, or by cats).
+    #   but do on long-press: useful to gracefully power-off server.
+    HandlePowerKey=lock
+    HandlePowerKeyLongPress=poweroff
+    HandleLidSwitch=lock
+  '';
+
+  # services.snapper.configs = {
+  #   root = {
+  #     subvolume = "/";
+  #     extraConfig = {
+  #       ALLOW_USERS = "colin";
+  #     };
+  #   };
+  # };
+  # services.snapper.snapshotInterval = "daily";
+}

hosts/common/hardware/x86_64.nix

@@ -0,0 +1,15 @@
+{ lib, pkgs, ... }:
+
+{
+  config = lib.mkIf (pkgs.system == "x86_64-linux") {
+    boot.initrd.availableKernelModules = [
+      "xhci_pci" "ahci" "sd_mod" "sdhci_pci"  # nixos-generate-config defaults
+      "usb_storage"   # rpi needed this to boot from usb storage, i think.
+      "nvme"  # to boot from nvme devices
+      # efi_pstore evivars
+    ];
+
+    hardware.cpu.amd.updateMicrocode = true;    # desktop
+    hardware.cpu.intel.updateMicrocode = true;  # laptop
+  };
+}

hosts/common/home/fs.nix

@@ -1,13 +1,21 @@
 { config, lib, ... }:
 {
   sane.user.persist.byStore.plaintext = [
-    # TODO: some of ~/dev should be private too, but maybe not all 800+ GB of it
-    # perhaps i ought to rethink how it's organized
+    "archive"
     "dev"
+    # TODO: records should be private
+    "records"
     "ref"
+    "tmp"
     "use"
     "Books/local"
     "Music"
+    "Pictures/albums"
+    "Pictures/cat"
+    "Pictures/from"
+    "Pictures/Screenshots"  #< XXX: something is case-sensitive about this?
+    "Pictures/Photos"
+    "Videos/local"
 
     # these are persisted simply to save on RAM.
     # ~/.cache/nix can become several GB.
@@ -17,17 +25,7 @@
     ".cache/nix"
   ];
   sane.user.persist.byStore.private = [
-    "archive"
-    "Pictures/albums"
-    "Pictures/cat"
-    "Pictures/from"
-    "Pictures/Screenshots"  #< XXX: something is case-sensitive about this?
-    "Pictures/Photos"
-    "records"
-    "tmp"
-
     "knowledge"
-    "Videos/local"
   ];
 
   # convenience
@@ -36,7 +34,7 @@
   in {
     ".persist/private" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.private.origin; };
     ".persist/plaintext" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.plaintext.origin; };
-    ".persist/ephemeral" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.ephemeral.origin; };
+    ".persist/ephemeral" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.cryptClearOnBoot.origin; };
 
     "nixos".symlink.target = "dev/nixos";
 

hosts/common/home/mime.nix

@@ -50,7 +50,7 @@ let
   localShareApplicationsPkg = (pkgs.symlinkJoin {
     name = "user-local-share-applications";
     paths = builtins.map
-      (p: builtins.toString p.package)
+      (p: "${p.package}")
       (enabledProgramsWithPackage ++ [ { package=mimeappsListPkg; } ]);
   }).overrideAttrs (orig: {
     # like normal symlinkJoin, but don't error if the path doesn't exist

hosts/common/home/xdg-dirs.nix

@@ -3,15 +3,11 @@
 {
   # XDG defines things like ~/Desktop, ~/Downloads, etc.
   # these clutter the home, so i mostly don't use them.
-  # note that several of these are not actually standardized anywhere.
-  # some are even non-conventional, like:
-  # - XDG_PHOTOS_DIR: only works because i patch e.g. megapixels
   sane.user.fs.".config/user-dirs.dirs".symlink.text = ''
     XDG_DESKTOP_DIR="$HOME/.xdg/Desktop"
     XDG_DOCUMENTS_DIR="$HOME/dev"
     XDG_DOWNLOAD_DIR="$HOME/tmp"
     XDG_MUSIC_DIR="$HOME/Music"
-    XDG_PHOTOS_DIR="$HOME/Pictures/Photos"
     XDG_PICTURES_DIR="$HOME/Pictures"
     XDG_PUBLICSHARE_DIR="$HOME/.xdg/Public"
     XDG_SCREENSHOTS_DIR="$HOME/Pictures/Screenshots"
@@ -23,5 +19,11 @@
   # see <https://manpages.ubuntu.com/manpages/bionic/man5/user-dirs.conf.5.html>
   sane.user.fs.".config/user-dirs.conf".symlink.text = "enabled=False";
 
-  sane.user.fs.".config/environment.d/30-user-dirs.conf".symlink.target = "../user-dirs.dirs";
+  sane.user.fs.".profile".symlink.text = ''
+    # configure XDG_<type>_DIR preferences (e.g. for downloads, screenshots, etc)
+    # surround with `set -o allexport` since user-dirs.dirs doesn't `export` its vars
+    set -a
+    source $HOME/.config/user-dirs.dirs
+    set +a
+  '';
 }

hosts/common/hosts.nix

@@ -2,14 +2,6 @@
 
 {
   # TODO: this should be populated per-host
-  sane.hosts.by-name."crappy" = {
-    ssh.user_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMIvSQAGKqmymXIL4La9B00LPxBIqWAr5AsJxk3UQeY5";
-    ssh.host_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMN0cpRAloCBOE5/2wuzgik35iNDv5KLceWMCVaa7DIQ";
-    # wg-home.pubkey = "TODO";
-    # wg-home.ip = "10.0.10.55";
-    lan-ip = "10.78.79.55";
-  };
-
   sane.hosts.by-name."desko" = {
     ssh.user_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPU5GlsSfbaarMvDA20bxpSZGWviEzXGD8gtrIowc1pX";
     ssh.host_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFw9NoRaYrM6LbDd3aFBc4yyBlxGQn8HjeHd/dZ3CfHk";

hosts/common/ids.nix

@@ -4,9 +4,6 @@
 { ... }:
 
 {
-  # partially supported in nixpkgs  <repo:nixos/nixpkgs:nixos/modules/misc/ids.nix>
-  sane.ids.networkmanager.uid = 57;  #< nixpkgs unofficially reserves this, to match networkmanager's gid
-
   # legacy servo users, some are inconvenient to migrate
   sane.ids.dhcpcd.gid = 991;
   sane.ids.dhcpcd.uid = 992;
@@ -21,7 +18,7 @@
   sane.ids.matrix-appservice-irc.uid = 993;
   sane.ids.matrix-appservice-irc.gid = 992;
 
-  # greetd (legacy)
+  # greetd (used by sway)
   sane.ids.greeter.uid = 999;
   sane.ids.greeter.gid = 999;
 
@@ -62,7 +59,6 @@
   sane.ids.clightning.gid = 2419;
   sane.ids.nix-serve.uid = 2420;
   sane.ids.nix-serve.gid = 2420;
-  sane.ids.plugdev.gid = 2421;
 
   sane.ids.colin.uid = 1000;
   sane.ids.guest.uid = 1100;
@@ -82,7 +78,6 @@
 
   # found on graphical hosts
   sane.ids.nm-iodine.uid = 2101;  # desko/moby/lappy
-  sane.ids.seat.gid = 2102;
 
   # found on desko host
   # from services.usbmuxd

hosts/common/net/default.nix

@@ -4,15 +4,12 @@
   imports = [
     ./dns.nix
     ./hostnames.nix
-    ./modemmanager.nix
-    ./networkmanager.nix
     ./upnp.nix
     ./vpn.nix
   ];
 
   systemd.network.enable = true;
   networking.useNetworkd = true;
-  networking.usePredictableInterfaceNames = false;  #< set false to get `eth0`, `wlan0`, etc instead of `enp3s0`/etc
 
   # view refused/dropped packets with: `sudo journalctl -k`
   # networking.firewall.logRefusedPackets = true;
@@ -27,4 +24,44 @@
   # this is required separately by servo and by any `sane-vpn` users,
   # however Nix requires this be set centrally, in only one location (i.e. here)
   boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
+
+  # the default backend is "wpa_supplicant".
+  # wpa_supplicant reliably picks weak APs to connect to.
+  # see: <https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/474>
+  # iwd is an alternative that shouldn't have this problem
+  # docs:
+  # - <https://nixos.wiki/wiki/Iwd>
+  # - <https://iwd.wiki.kernel.org/networkmanager>
+  # - `man iwd.config`  for global config
+  # - `man iwd.network` for per-SSID config
+  # use `iwctl` to control
+  # networking.networkmanager.wifi.backend = "iwd";
+  # networking.wireless.iwd.enable = true;
+  # networking.wireless.iwd.settings = {
+  #   # auto-connect to a stronger network if signal drops below this value
+  #   # bedroom -> bedroom connection is -35 to -40 dBm
+  #   # bedroom -> living room connection is -60 dBm
+  #   General.RoamThreshold = "-52";  # default -70
+  #   General.RoamThreshold5G = "-52";  # default -76
+  # };
+
+  # plugins mostly add support for establishing different VPN connections.
+  # the default plugin set includes mostly proprietary VPNs:
+  # - fortisslvpn (Fortinet)
+  # - iodine (DNS tunnels)
+  # - l2tp
+  # - openconnect (Cisco Anyconnect / Juniper / ocserv)
+  # - openvpn
+  # - vpnc (Cisco VPN)
+  # - sstp
+  #
+  # i don't use these, and notably they drag in huge dependency sets and don't cross compile well.
+  # e.g. openconnect drags in webkitgtk (for SSO)!
+  networking.networkmanager.plugins = lib.mkForce [];
+
+  # keyfile.path = where networkmanager should look for connection credentials
+  networking.networkmanager.extraConfig = ''
+    [keyfile]
+    path=/var/lib/NetworkManager/system-connections
+  '';
 }

hosts/common/net/dns.nix

@@ -1,6 +1,7 @@
 # things to consider when changing these parameters:
 # - temporary VPN access (`sane-vpn up ...`)
 # - servo `ovpns` namespace (it *relies* on /etc/resolv.conf mentioning 127.0.0.53)
+# - jails: `firejail --net=br-ovpnd-us --noprofile --dns=46.227.67.134 ping 1.1.1.1`
 #
 # components:
 # - /etc/nsswitch.conf:
@@ -17,22 +18,17 @@
 #   - modern implementations hardcodes `127.0.0.53` and then systemd-resolved proxies everything (and caches).
 #
 # namespacing:
-# - each namespace may use a different /etc/resolv.conf to specify different DNS servers
-# - nscd breaks namespacing: the host nscd is unaware of the guest's /etc/resolv.conf, and so directs the guest's DNS requests to the host's servers.
-#   - this is fixed by either removing `/var/run/nscd/socket` from the namespace, or disabling nscd altogether.
-{ config, lib, pkgs, ... }:
-lib.mkMerge [
+# - each namespace can use a different /etc/resolv.conf to specify different DNS servers (see `firejail --dns=...`)
+# - nscd breaks namespacing: the host nscd is unaware of the guest's /etc/resolv.conf, and so direct's the guest's DNS requests to the host's servers.
+#   - this is fixed by either `firejail --blacklist=/var/run/nscd/socket`, or disabling nscd altogether.
+{ lib, ... }:
 {
-  sane.services.trust-dns.enable = lib.mkDefault config.sane.services.trust-dns.asSystemResolver;
-  sane.services.trust-dns.asSystemResolver = lib.mkDefault true;
-}
-(lib.mkIf (!config.sane.services.trust-dns.asSystemResolver) {
   # use systemd's stub resolver.
   # /etc/resolv.conf isn't sophisticated enough to use different servers per net namespace (or link).
   # instead, running the stub resolver on a known address in the root ns lets us rewrite packets
   # in servo's ovnps namespace to use the provider's DNS resolvers.
   # a weakness is we can only query 1 NS at a time (unless we were to clone the packets?)
-  # TODO: improve trust-dns recursive resolver and then remove this
+  # TODO: rework servo's netns to use `firejail`, which is capable of spoofing /etc/resolv.conf.
   services.resolved.enable = true;  #< to disable, set ` = lib.mkForce false`, as other systemd features default to enabling `resolved`.
   # without DNSSEC:
   # - dig matrix.org => works
@@ -48,8 +44,7 @@ lib.mkMerge [
     # stub resolver (just forwards upstream) lives on 127.0.0.54
     "127.0.0.53"
   ];
-})
-{
+
   # nscd -- the Name Service Caching Daemon -- caches DNS query responses
   # in a way that's unaware of my VPN routing, so routes are frequently poor against
   # services which advertise different IPs based on geolocation.
@@ -59,35 +54,14 @@ lib.mkMerge [
   # in the netns and we query upstream DNS more often than needed. hm.
   # services.nscd.enableNsncd = true;
 
-  # disabling nscd LOSES US SOME FUNCTIONALITY. in particular, only the glibc-builtin modules are accessible via /etc/resolv.conf (er, did i mean /etc/nsswitch.conf?).
+  # disabling nscd LOSES US SOME FUNCTIONALITY. in particular, only the glibc-builtin modules are accessible via /etc/resolv.conf.
   # - dns: glibc-bultin
   # - files: glibc-builtin
   # - myhostname: systemd
   # - mymachines: systemd
   # - resolve: systemd
   # in practice, i see no difference with nscd disabled.
-  # - the exception is when the system dns resolver doesn't do everything.
-  #   for example, systemd-resolved does mDNS. hickory-dns does not. a hickory-dns system won't be mDNS-capable.
   # disabling nscd VASTLY simplifies netns and process isolation. see explainer at top of file.
   services.nscd.enable = false;
-  # system.nssModules = lib.mkForce [];
-  sane.silencedAssertions = [''.*Loading NSS modules from system.nssModules.*requires services.nscd.enable being set to true.*''];
-  # add NSS modules into their own subdirectory.
-  # then i can add just the NSS modules library path to the global LD_LIBRARY_PATH, rather than ALL of /run/current-system/sw/lib.
-  # TODO: i'm doing this so as to achieve mdns DNS resolution (avahi). it would be better to just have trust-dns delegate .local to avahi
-  #   (except avahi doesn't act as a local resolver over DNS protocol -- only dbus).
-  environment.systemPackages = [(pkgs.symlinkJoin {
-    name = "nss-modules";
-    paths = config.system.nssModules.list;
-    postBuild = ''
-      mkdir nss
-      mv $out/lib/libnss_* nss
-      rm -rf $out
-      mkdir -p $out/lib
-      mv nss $out/lib
-    '';
-  })];
-  environment.variables.LD_LIBRARY_PATH = [ "/run/current-system/sw/lib/nss" ];
-  systemd.globalEnvironment.LD_LIBRARY_PATH = "/run/current-system/sw/lib/nss";  #< specifically for `geoclue.service`
+  system.nssModules = lib.mkForce [];
 }
-]

hosts/common/net/modemmanager.nix

@@ -1,77 +0,0 @@
-{ config, lib, pkgs, ... }:
-{
-  networking.modemmanager.package = pkgs.modemmanager-split.daemon.overrideAttrs (upstream: {
-    # patch to allow the dbus endpoints to be owned by networkmanager user
-    postInstall = (upstream.postInstall or "") + ''
-      substitute $out/share/dbus-1/system.d/org.freedesktop.ModemManager1.conf \
-        $out/share/dbus-1/system.d/networkmanager-org.freedesktop.ModemManager1.conf \
-        --replace-fail 'user="root"' 'group="networkmanager"'
-    '';
-  });
-
-  systemd.services.ModemManager = {
-    # aliases = [ "dbus-org.freedesktop.ModemManager1.service" ];
-    # after = [ "polkit.service" ];
-    # requires = [ "polkit.service" ];
-    wantedBy = [ "network.target" ];  #< default is `multi-user.target`, somehow it doesn't auto-start with that...
-
-    # serviceConfig.Type = "dbus";
-    # serviceConfig.BusName = "org.freedesktop.ModemManager1";
-
-    # only if started with `--debug` does mmcli let us issue AT commands like
-    # `mmcli --modem any --command=<AT_CMD>`
-    serviceConfig.ExecStart = [
-      ""  # first blank line is to clear the upstream `ExecStart` field.
-      "${lib.getExe' config.networking.modemmanager.package "ModemManager"} --debug"
-    ];
-    # --debug sets DEBUG level logging: so reset
-    serviceConfig.ExecStartPost = "${lib.getExe config.sane.programs.mmcli.package} --set-logging=INFO";
-
-    # v this is what upstream ships
-    # serviceConfig.Restart = "on-abort";
-    # serviceConfig.StandardError = "null";
-    # serviceConfig.CapabilityBoundingSet = "CAP_SYS_ADMIN CAP_NET_ADMIN";
-    # serviceConfig.ProtectSystem = true;  # makes empty: /boot, /usr
-    # serviceConfig.ProtectHome = true;  # makes empty: /home, /root, /run/user
-    # serviceConfig.PrivateTmp = true;
-    # serviceConfig.RestrictAddressFamilies = "AF_NETLINK AF_UNIX AF_QIPCRTR";
-    # serviceConfig.NoNewPrivileges = true;
-
-    serviceConfig.CapabilityBoundingSet = [ "CAP_NET_ADMIN" ];  #< TODO: make sure this is *really* taking effect, and isn't supplemental to upstream's `CAP_SYS_ADMIN` setting
-    serviceConfig.LockPersonality = true;
-    # serviceConfig.PrivateUsers = true;  #< untried, not likely to work since it needs capabilities
-    serviceConfig.PrivateTmp = true;
-    serviceConfig.ProtectClock = true;  # syscall filter to prevent changing the RTC
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;  # makes empty: /home, /root, /run/user
-    serviceConfig.ProtectHostname = true;  # prevents changing hostname
-    serviceConfig.ProtectKernelLogs = true;  # disable /proc/kmsg, /dev/kmsg
-    serviceConfig.ProtectKernelModules = true;  # syscall filter to prevent module calls
-    serviceConfig.ProtectKernelTunables = true;
-    serviceConfig.ProtectSystem = "strict";  # makes read-only all but /dev, /proc, /sys
-    serviceConfig.RestrictAddressFamilies = [
-      "AF_NETLINK"
-      "AF_QIPCRTR"
-      "AF_UNIX"
-    ];
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";  # prevents e.g. aarch64 syscalls in the event that the kernel is multi-architecture.
-
-    # from earlier `landlock` sandboxing, i know it needs these directories:
-    # - # "/"
-    # - "/dev" #v modem-power + net are not enough
-    # - # "/dev/modem-power"
-    # - # "/dev/net"
-    # - "/proc"
-    # - # /run  #v can likely be reduced more
-    # - "/run/dbus"
-    # - "/run/NetworkManager"
-    # - "/run/resolvconf"
-    # - "/run/systemd"
-    # - "/run/udev"
-    # - "/sys"
-  };
-
-  # so that ModemManager can discover when the modem appears
-  # services.udev.packages = lib.mkIf cfg.enabled [ cfg.package ];
-}

hosts/common/net/networkmanager.nix

@@ -1,281 +0,0 @@
-{ config, pkgs, ... }:
-let
-  # networkmanager = pkgs.networkmanager;
-  networkmanager = pkgs.networkmanager.overrideAttrs (upstream: {
-    # src = pkgs.fetchFromGitea {
-    #   domain = "git.uninsane.org";
-    #   owner = "colin";
-    #   repo = "NetworkManager";
-    #   # patched to fix polkit permissions (with `nmcli`) when NetworkManager runs as user networkmanager
-    #   rev = "dev-sane-1.48.0";
-    #   hash = "sha256-vGmOKtwVItxjYioZJlb1og3K6u9s4rcmDnjAPLBC3ao=";
-    # };
-    patches = (upstream.patches or []) ++ [
-      (pkgs.fetchpatch {
-        name = "polkit: add owner annotations to all actions";
-        url = "https://git.uninsane.org/colin/NetworkManager/commit/a01293861fa24201ffaeb84c07f1c71136c49759.patch";
-        hash = "sha256-th1/M2slo7rjkVBwETZII53Lmhyw8OMS0aT9QYI5Uvk=";
-      })
-    ];
-  });
-  # split the package into `daemon` and `nmcli` outputs, because the networkmanager *service*
-  # doesn't need `nmcli`/`nmtui` tooling
-  networkmanager-split = pkgs.networkmanager-split.override { inherit networkmanager; };
-in {
-  networking.networkmanager.enable = true;
-  systemd.network.wait-online.enable = false;  # systemd-networkd-wait-online.service reliably fails on lappy. docs don't match behavior. shit software.
-  # plugins mostly add support for establishing different VPN connections.
-  # the default plugin set includes mostly proprietary VPNs:
-  # - fortisslvpn (Fortinet)
-  # - iodine (DNS tunnels)
-  # - l2tp
-  # - openconnect (Cisco Anyconnect / Juniper / ocserv)
-  # - openvpn
-  # - vpnc (Cisco VPN)
-  # - sstp
-  #
-  # i don't use these, and notably they drag in huge dependency sets and don't cross compile well.
-  # e.g. openconnect drags in webkitgtk (for SSO)!
-  # networking.networkmanager.plugins = lib.mkForce [];
-  networking.networkmanager.enableDefaultPlugins = false;
-
-  networking.networkmanager.package = networkmanager-split.daemon.overrideAttrs (upstream: {
-    # postPatch = (upstream.postPatch or "") + ''
-    #   substituteInPlace src/{core/org.freedesktop.NetworkManager,nm-dispatcher/nm-dispatcher}.conf --replace-fail \
-    #     'user="root"' 'user="networkmanager"'
-    # '';
-    postInstall = (upstream.postInstall or "") + ''
-      # allow the bus to owned by either root or networkmanager users
-      # use the group here, that way ordinary users can be elevated to control networkmanager
-      # (via e.g. `nmcli`)
-      for f in org.freedesktop.NetworkManager.conf nm-dispatcher.conf ; do
-        substitute $out/share/dbus-1/system.d/$f \
-          $out/share/dbus-1/system.d/networkmanager-$f \
-          --replace-fail 'user="root"' 'group="networkmanager"'
-      done
-
-      # remove unused services to prevent any unexpected interactions
-      rm $out/etc/systemd/system/{nm-cloud-setup.service,nm-cloud-setup.timer,nm-priv-helper.service}
-    '';
-  });
-
-  # fixup the services to run as `networkmanager` and with less permissions
-  systemd.services.NetworkManager = {
-    serviceConfig.RuntimeDirectory = "NetworkManager";  #< tells systemd to create /run/NetworkManager
-    # serviceConfig.StateDirectory = "NetworkManager";  #< tells systemd to create /var/lib/NetworkManager
-    serviceConfig.User = "networkmanager";
-    serviceConfig.Group = "networkmanager";
-    serviceConfig.AmbientCapabilities = [
-      # "CAP_DAC_OVERRIDE"
-      "CAP_NET_ADMIN"
-      "CAP_NET_RAW"  #< required, else `libndp: ndp_sock_open: Failed to create ICMP6 socket.`
-      "CAP_NET_BIND_SERVICE"  #< this *does* seem to be necessary, though i don't understand why. DHCP?
-      # "CAP_SYS_MODULE"
-      # "CAP_AUDIT_WRITE"  #< allow writing to the audit log (optional)
-      # "CAP_KILL"
-    ];
-    serviceConfig.LockPersonality = true;
-    serviceConfig.NoNewPrivileges = true;
-    serviceConfig.PrivateDevices = true;  # remount /dev with just the basics, syscall filter to block @raw-io
-    serviceConfig.PrivateIPC = true;
-    serviceConfig.PrivateTmp = true;
-    # serviceConfig.PrivateUsers = true;  #< BREAKS NetworkManager (presumably, it causes a new user namespace, breaking CAP_NET_ADMIN & others).  "platform-linux: do-change-link[3]: failure 1 (Operation not permitted)"
-    serviceConfig.ProtectClock = true;  # syscall filter to prevent changing the RTC
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;  # makes empty: /home, /root, /run/user
-    serviceConfig.ProtectHostname = true;  # probably not upstreamable: prevents changing hostname
-    serviceConfig.ProtectKernelLogs = true;  # disable /proc/kmsg, /dev/kmsg
-    serviceConfig.ProtectKernelModules = true;  # syscall filter to prevent module calls (probably not upstreamable: NM will want to load modules like `ppp`)
-    serviceConfig.ProtectKernelTunables = true;  # but NM might need to write /proc/sys/net/...
-    serviceConfig.ProtectSystem = "strict";  # makes read-only: all but /dev, /proc, /sys.
-    serviceConfig.RestrictAddressFamilies = [
-      "AF_INET"
-      "AF_INET6"
-      "AF_NETLINK"  # breaks near DHCP without this
-      "AF_PACKET"  # for DHCP
-      "AF_UNIX"
-      # AF_ALG ?
-      # AF_BLUETOOTH ?
-      # AF_BRIDGE ?
-    ];
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";  # prevents e.g. aarch64 syscalls in the event that the kernel is multi-architecture.
-    # from earlier `landlock` sandboxing, i know it needs these directories:
-    # - "/proc/net"
-    # - "/proc/sys/net"
-    # - "/run/NetworkManager"
-    # - "/run/systemd"  # for trust-dns-nmhook
-    # - "/run/udev"
-    # - # "/run/wg-home.priv"
-    # - "/sys/class"
-    # - "/sys/devices"
-    # - "/var/lib/NetworkManager"
-    # - "/var/lib/trust-dns"  #< for trust-dns-nmhook
-    # - "/run/systemd"
-  };
-
-  systemd.services.NetworkManager-wait-online = {
-    serviceConfig.User = "networkmanager";
-    serviceConfig.Group = "networkmanager";
-  };
-
-  # fix NetworkManager-dispatcher to actually run as a daemon,
-  # and sandbox it a bit
-  systemd.services.NetworkManager-dispatcher = {
-    after = [ "trust-dns-localhost.service" ];  #< so that /var/lib/trust-dns will exist
-    # serviceConfig.ExecStart = [
-    #   ""  # first blank line is to clear the upstream `ExecStart` field.
-    #   "${cfg.package}/libexec/nm-dispatcher --persist"  # --persist is needed for it to actually run as a daemon
-    # ];
-    # serviceConfig.Restart = "always";
-    # serviceConfig.RestartSec = "1s";
-
-    # serviceConfig.DynamicUser = true;  #< not possible, else we lose group perms (so can't write to `trust-dns`'s files in the nm hook)
-    serviceConfig.User = "networkmanager";  # TODO: should arguably use `DynamicUser`
-    serviceConfig.Group = "networkmanager";
-    serviceConfig.LockPersonality = true;
-    serviceConfig.NoNewPrivileges = true;
-    serviceConfig.PrivateDevices = true;  # remount /dev with just the basics, syscall filter to block @raw-io
-    serviceConfig.PrivateIPC = true;
-    serviceConfig.PrivateTmp = true;
-    serviceConfig.PrivateUsers = true;
-    serviceConfig.ProtectClock = true;  # syscall filter to prevent changing the RTC
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;  # makes empty: /home, /root, /run/user
-    serviceConfig.ProtectHostname = true;  # probably not upstreamable: prevents changing hostname
-    serviceConfig.ProtectKernelLogs = true;  # disable /proc/kmsg, /dev/kmsg
-    serviceConfig.ProtectKernelModules = true;  # syscall filter to prevent module calls
-    serviceConfig.ProtectKernelTunables = true;
-    serviceConfig.ProtectSystem = "full";  # makes read-only: /boot, /etc/, /usr. `strict` isn't possible due to trust-dns hook
-    serviceConfig.RestrictAddressFamilies = [
-      "AF_UNIX"  # required, probably for dbus or systemd connectivity
-    ];
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";  # prevents e.g. aarch64 syscalls in the event that the kernel is multi-architecture.
-  };
-
-  # harden wpa_supplicant (used by NetworkManager)
-  systemd.services.wpa_supplicant = {
-    serviceConfig.User = "networkmanager";
-    serviceConfig.Group = "networkmanager";
-    serviceConfig.AmbientCapabilities = [
-      "CAP_NET_ADMIN"
-      "CAP_NET_RAW"
-    ];
-    serviceConfig.LockPersonality = true;
-    serviceConfig.NoNewPrivileges = true;
-    # serviceConfig.PrivateDevices = true;  # untried, not likely to work. remount /dev with just the basics, syscall filter to block @raw-io
-    serviceConfig.PrivateIPC = true;
-    serviceConfig.PrivateTmp = true;
-    # serviceConfig.PrivateUsers = true;  #< untried, not likely to work
-    serviceConfig.ProtectClock = true;  # syscall filter to prevent changing the RTC
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;  # makes empty: /home, /root, /run/user
-    serviceConfig.ProtectHostname = true;  # prevents changing hostname
-    serviceConfig.ProtectKernelLogs = true;  # disable /proc/kmsg, /dev/kmsg
-    serviceConfig.ProtectKernelModules = true;  # syscall filter to prevent module calls
-    serviceConfig.ProtectKernelTunables = true;  #< N.B.: i think this makes certain /proc writes fail
-    serviceConfig.ProtectSystem = "strict";  # makes read-only: all but /dev, /proc, /sys.
-    serviceConfig.RestrictAddressFamilies = [
-      "AF_INET"  #< required
-      "AF_INET6"
-      "AF_NETLINK"  #< required
-      "AF_PACKET"  #< required
-      "AF_UNIX"  #< required (wpa_supplicant wants to use dbus)
-    ];
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";  # prevents e.g. aarch64 syscalls in the event that the kernel is multi-architecture.
-
-    # from earlier `landlock` sandboxing, i know it needs only these paths:
-    # - "/dev/net"
-    # - "/dev/rfkill"
-    # - "/proc/sys/net"
-    # - "/sys/class/net"
-    # - "/sys/devices"
-    # - "/run/systemd"
-  };
-
-  networking.networkmanager.settings = {
-    # keyfile.path = where networkmanager should look for connection credentials
-    keyfile.path = "/var/lib/NetworkManager/system-connections";
-
-    # wifi.backend = "wpa_supplicant";  #< default
-    # wifi.scan-rand-mac-address = true;  #< default
-
-    # logging.audit = false;  #< default
-    logging.level = "INFO";
-
-    # main.dhcp = "internal";  #< default
-    # main.dns controls what to do when NM gets a DNS server via DHCP
-    # - "none"  (populate /run/NetworkManager/resolv.conf with DHCP settings)
-    # - "internal" (?)
-    # - "systemd-resolved" (tell systemd-resolved about it, and point /run/NetworkManager/resolv.conf -> systemd)
-    #   without this, systemd-resolved won't be able to resolve anything (because it has no upstream servers)
-    # note that NM's resolv.conf isn't (necessarily) /etc/resolv.conf -- that is managed by nixos (via symlinking)
-    main.dns = if config.services.resolved.enable then
-      "systemd-resolved"
-    else if config.sane.services.trust-dns.enable && config.sane.services.trust-dns.asSystemResolver then
-      "none"
-    else
-      "internal"
-    ;
-    main.systemd-resolved = false;
-  };
-  environment.etc."NetworkManager/system-connections".source = "/var/lib/NetworkManager/system-connections";
-
-  # the default backend is "wpa_supplicant".
-  # wpa_supplicant reliably picks weak APs to connect to.
-  # see: <https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/474>
-  # iwd is an alternative that shouldn't have this problem
-  # docs:
-  # - <https://nixos.wiki/wiki/Iwd>
-  # - <https://iwd.wiki.kernel.org/networkmanager>
-  # - `man iwd.config`  for global config
-  # - `man iwd.network` for per-SSID config
-  # use `iwctl` to control
-  # networking.networkmanager.wifi.backend = "iwd";
-  # networking.wireless.iwd.enable = true;
-  # networking.wireless.iwd.settings = {
-  #   # auto-connect to a stronger network if signal drops below this value
-  #   # bedroom -> bedroom connection is -35 to -40 dBm
-  #   # bedroom -> living room connection is -60 dBm
-  #   General.RoamThreshold = "-52";  # default -70
-  #   General.RoamThreshold5G = "-52";  # default -76
-  # };
-
-  # allow networkmanager to control systemd-resolved,
-  # which it needs to do to apply new DNS settings when using systemd-resolved.
-  security.polkit.extraConfig = ''
-    polkit.addRule(function(action, subject) {
-      if (subject.isInGroup("networkmanager") && action.id.indexOf("org.freedesktop.resolve1.") == 0) {
-        return polkit.Result.YES;
-      }
-    });
-  '';
-
-  users.users.networkmanager = {
-    isSystemUser = true;
-    group = "networkmanager";
-    extraGroups = [ "trust-dns" ];
-  };
-
-  # there is, unfortunately, no proper interface by which to plumb wpa_supplicant into the NixOS service, except by overlay.
-  nixpkgs.overlays = [(self: super: {
-    wpa_supplicant = super.wpa_supplicant.overrideAttrs (upstream: {
-      # postPatch = (upstream.postPatch or "") + ''
-      #   substituteInPlace wpa_supplicant/dbus/dbus-wpa_supplicant.conf --replace-fail \
-      #     'user="root"' 'user="networkmanager"'
-      # '';
-      postInstall = (upstream.postInstall or "") + ''
-        substitute $out/share/dbus-1/system.d/dbus-wpa_supplicant.conf \
-          $out/share/dbus-1/system.d/networkmanager-wpa_supplicant.conf \
-          --replace-fail 'user="root"' 'group="networkmanager"'
-      '';
-
-      postFixup = (upstream.postFixup or "") + ''
-        # remove unused services to avoid unexpected interactions
-        rm $out/etc/systemd/system/{wpa_supplicant-nl80211@,wpa_supplicant-wired@,wpa_supplicant@}.service
-      '';
-    });
-  })];
-}

hosts/common/net/vpn.nix

@@ -7,62 +7,50 @@
 
 { config, lib, pkgs, ... }:
 let
-  # N.B.: OVPN issues each key (i.e. device) a different IP (addrV4), and requires you use it.
-  # the IP it issues can be used to connect to any of their VPNs.
-  # effectively the IP and key map 1-to-1.
-  # it seems to still be possible to keep two active tunnels on one device, using the same key/IP address, though.
-  def-ovpn = name: { endpoint, publicKey, id }: let
-    inherit (config.sane.ovpn) addrV4;
-  in {
-    sane.vpn."ovpnd-${name}" = lib.mkIf (addrV4 != null) {
-      inherit addrV4 endpoint publicKey id;
-      privateKeyFile = config.sops.secrets."ovpn_privkey".path;
+  def-ovpn = name: { endpoint, publicKey, addrV4, id }: {
+    sane.vpn."ovpnd-${name}" = {
+      inherit endpoint publicKey addrV4 id;
+      privateKeyFile = config.sops.secrets."wg/ovpnd_${name}_privkey".path;
       dns = [
         "46.227.67.134"
         "192.165.9.158"
-        # "2a07:a880:4601:10f0:cd45::1"
-        # "2001:67c:750:1:cafe:cd45::1"
       ];
     };
 
-    sops.secrets."ovpn_privkey" = lib.mkIf (addrV4 != null) {
+    sops.secrets."wg/ovpnd_${name}_privkey" = {
       # needs to be readable by systemd-network or else it says "Ignoring network device" and doesn't expose it to networkctl.
       owner = "systemd-network";
     };
   };
-in {
-  options = with lib; {
-    sane.ovpn.addrV4 = mkOption {
-      type = types.nullOr types.str;
-      default = null;
-      description = ''
-        ovpn issues one IP address per device.
-        set `null` to disable OVPN for this host.
-      '';
-    };
-  };
-
-  config = lib.mkMerge [
-    (def-ovpn "us" {
-      endpoint = "vpn31.prd.losangeles.ovpn.com:9929";
-      publicKey = "VW6bEWMOlOneta1bf6YFE25N/oMGh1E1UFBCfyggd0k=";
-      id = 1;
-    })
-    (def-ovpn "us-mi" {
-      endpoint = "vpn34.prd.miami.ovpn.com:9929";
-      publicKey = "VtJz2irbu8mdkIQvzlsYhU+k9d55or9mx4A2a14t0V0=";
-      id = 2;
-    })
-    (def-ovpn "ukr" {
-      endpoint = "vpn96.prd.kyiv.ovpn.com:9929";
-      publicKey = "CjZcXDxaaKpW8b5As1EcNbI6+42A6BjWahwXDCwfVFg=";
-      id = 3;
-    })
-    # TODO: us-atl disabled until i need it again, i guess.
-    # (def-ovpn "us-atl" {
-    #   endpoint = "vpn18.prd.atlanta.ovpn.com:9929";
-    #   publicKey = "Dpg/4v5s9u0YbrXukfrMpkA+XQqKIFpf8ZFgyw0IkE0=";
-    #   id = 4;
-    # })
-  ];
-}
+in lib.mkMerge [
+  (def-ovpn "us" {
+    endpoint = "vpn31.prd.losangeles.ovpn.com:9929";
+    publicKey = "VW6bEWMOlOneta1bf6YFE25N/oMGh1E1UFBCfyggd0k=";
+    id = 1;
+    addrV4 = "172.27.237.218";
+    # addrV6 = "fd00:0000:1337:cafe:1111:1111:ab00:4c8f";
+  })
+  # TODO: us-atl disabled until i can give it a different link-local address and wireguard key than us-mi
+  # (def-ovpn "us-atl" {
+  #   endpoint = "vpn18.prd.atlanta.ovpn.com:9929";
+  #   publicKey = "Dpg/4v5s9u0YbrXukfrMpkA+XQqKIFpf8ZFgyw0IkE0=";
+  #   address = [
+  #     "172.21.182.178/32"
+  #     "fd00:0000:1337:cafe:1111:1111:cfcb:27e3/128"
+  #   ];
+  # })
+  (def-ovpn "us-mi" {
+    endpoint = "vpn34.prd.miami.ovpn.com:9929";
+    publicKey = "VtJz2irbu8mdkIQvzlsYhU+k9d55or9mx4A2a14t0V0=";
+    id = 2;
+    addrV4 = "172.21.182.178";
+    # addrV6 = "fd00:0000:1337:cafe:1111:1111:cfcb:27e3";
+  })
+  (def-ovpn "ukr" {
+    endpoint = "vpn96.prd.kyiv.ovpn.com:9929";
+    publicKey = "CjZcXDxaaKpW8b5As1EcNbI6+42A6BjWahwXDCwfVFg=";
+    id = 3;
+    addrV4 = "172.18.180.159";
+    # addrV6 = "fd00:0000:1337:cafe:1111:1111:ec5c:add3";
+  })
+]

hosts/common/nix/default.nix

@@ -53,26 +53,22 @@
 
   # allow `nix-shell` (and probably nix-index?) to locate our patched and custom packages.
   # this is actually a no-op, and the real action happens in assigning `nix.settings.nix-path`.
-  nix.nixPath = (lib.optionals (config.sane.maxBuildCost >= 2) [
+  nix.nixPath = (lib.optionals config.sane.enableSlowPrograms [
     "nixpkgs=${pkgs.path}"
   ]) ++ [
     # note the import starts at repo root: this allows `./overlay/default.nix` to access the stuff at the root
     # "nixpkgs-overlays=${../../..}/hosts/common/nix-path/overlay"
     # as long as my system itself doesn't rely on NIXPKGS at runtime, we can point the overlays to git
-    # to avoid `switch`ing so much during development.
-    # TODO: it would be nice to remove this someday!
-    # it's an impurity that touches way more than i need and tends to cause hard-to-debug eval issues
-    # when it goes wrong. should i port my `nix-shell` scripts to something more tailored to my uses
-    # and then delete `nixpkgs-overlays`?
-    "nixpkgs-overlays=/home/colin/dev/nixos/integrations/nixpkgs/nixpkgs-overlays.nix"
+    # to avoid switching so much during development
+    "nixpkgs-overlays=/home/colin/dev/nixos/hosts/common/nix/overlay"
   ];
 
   # ensure new deployments have a source of this repo with which they can bootstrap.
   # this however changes on every commit and can be slow to copy for e.g. `moby`.
-  environment.etc."nixos" = lib.mkIf (config.sane.maxBuildCost >= 3) {
-    source = pkgs.sane-nix-files;
+  environment.etc."nixos" = lib.mkIf config.sane.enableSlowPrograms {
+    source = ../../..;
   };
-  environment.etc."nix/registry.json" = lib.mkIf (config.sane.maxBuildCost < 3) {
+  environment.etc."nix/registry.json" = lib.mkIf (!config.sane.enableSlowPrograms) {
     enable = false;
   };
 

hosts/common/nix/overlay/default.nix

@@ -0,0 +1,4 @@
+# XXX: NIX_PATH=...:nixpkgs-overlays=... will import every overlay in the directory
+# so we prefer to give it a directory with just this *one* overlay, otherwise it imports conflicting overlays
+# and gets stuck in a loop until it OOMs
+import ../../../../overlays/all.nix

hosts/common/persist.nix

@@ -0,0 +1,17 @@
+{ ... }:
+
+{
+  # store /home/colin/a/b in /mnt/persist/private/a/b instead of /mnt/persist/private/home/colin/a/b
+  sane.persist.stores.private.prefix = "/home/colin";
+
+  sane.persist.sys.byStore.initrd = [
+    "/var/log"
+  ];
+  sane.persist.sys.byStore.plaintext = [
+    # TODO: these should be private.. somehow
+    "/var/backup"  # for e.g. postgres dumps
+  ];
+  sane.persist.sys.byStore.cryptClearOnBoot = [
+    "/var/lib/systemd/coredump"
+  ];
+}

hosts/common/polyunfill.nix

@@ -1,229 +1,72 @@
 # strictly *decrease* the scope of the default nixos installation/config
 
-{ config, lib, pkgs, ... }:
-let
-  suidlessPam = pkgs.pam.overrideAttrs (upstream: {
-    # nixpkgs' pam hardcodes unix_chkpwd path to the /run/wrappers one,
-    # but i don't want the wrapper, so undo that.
-    # ideally i would patch this via an overlay, but pam is in the bootstrap so that forces a full rebuild.
-    # TODO: add a `package` option to the nixos' pam module and substitute it that way.
-    postPatch = (if upstream.postPatch != null then upstream.postPatch else "") + ''
-      substituteInPlace modules/pam_unix/Makefile.am --replace-fail \
-        "/run/wrappers/bin/unix_chkpwd" "$out/bin/unix_chkpwd"
-    '';
-  });
-in
+{ lib, ... }:
 {
-  # remove a few items from /run/wrappers we don't need.
-  options.security.wrappers = lib.mkOption {
-    apply = lib.filterAttrs (name: _: !(builtins.elem name [
-      # from <repo:nixos/nixpkgs:nixos/modules/security/polkit.nix>
-      "pkexec"
-      "polkit-agent-helper-1"  #< used by systemd; without this you'll have to `sudo systemctl daemon-reload` instead of unauth'd `systemctl daemon-reload`
-      # from <repo:nixos/nixpkgs:nixos/modules/services/system/dbus.nix>
-      "dbus-daemon-launch-helper"
-      # from <repo:nixos/nixpkgs:nixos/modules/security/wrappers/default.nix>
-      "fusermount"  #< only needed if you want to mount entries declared in /etc/fstab or mtab as unprivileged user
-      "fusermount3"
-      "mount"  #< only needed if you want to mount entries declared in /etc/fstab or mtab as unprivileged user
-      "umount"
-      # from <repo:nixos/nixpkgs:nixos/modules/programs/shadow.nix>
-      "newgidmap"
-      "newgrp"
-      "newuidmap"
-      "sg"
-      "su"
-      # from: <repo:nixos/nixpkgs:nixos/modules/security/pam.nix>
-      # requires associated `pam` patch to not hardcode unix_chkpwd path
-      "unix_chkpwd"
-    ]));
-  };
-  options.security.pam.services = lib.mkOption {
-    apply = services: let
-      filtered = lib.filterAttrs (name: _: !(builtins.elem name [
-        # from <repo:nixos/nixpkgs:nixos/modules/security/pam.nix>
-        "i3lock"
-        "i3lock-color"
-        "vlock"
-        "xlock"
-        "xscreensaver"
-        "runuser"
-        "runuser-l"
-        # from ??
-        "chfn"
-        "chpasswd"
-        "chsh"
-        "groupadd"
-        "groupdel"
-        "groupmems"
-        "groupmod"
-        "useradd"
-        "userdel"
-        "usermod"
-        # from <repo:nixos/nixpkgs:nixos/modules/system/boot/systemd/user.nix>
-        "systemd-user"  #< N.B.: this causes the `systemd --user` service manager to not be started!
-      ])) services;
-    in lib.mapAttrs (_serviceName: service: service // {
-      # replace references with the old pam_unix, which calls into /run/wrappers/bin/unix_chkpwd,
-      # with a pam_unix that calls into unix_chkpwd via the nix store.
-      # TODO: use `security.pam.package` instead once <https://github.com/NixOS/nixpkgs/pull/314791> lands.
-      text = lib.replaceStrings [" pam_unix.so" ] [ " ${suidlessPam}/lib/security/pam_unix.so" ] service.text;
-    }) filtered;
-  };
+  # disable non-required packages like nano, perl, rsync, strace
+  environment.defaultPackages = [];
 
-  options.environment.systemPackages = lib.mkOption {
-    # see: <repo:nixos/nixpkgs:nixos/modules/config/system-path.nix>
-    # it's 31 "requiredPackages", with no explanation of why they're "required"...
-    # most of these can be safely removed without breaking the *boot*,
-    # but some core system services DO implicitly depend on them.
-    # TODO: see which more of these i can remove (or shadow/sandbox)
-    apply = let
-      requiredPackages = builtins.map (pkg: lib.setPrio ((pkg.meta.priority or 5) + 3) pkg) [
-        # pkgs.acl
-        # pkgs.attr
-        # pkgs.bashInteractive
-        # pkgs.bzip2
-        # pkgs.coreutils-full
-        # pkgs.cpio
-        # pkgs.curl
-        # pkgs.diffutils
-        # pkgs.findutils
-        # pkgs.gawk
-        # pkgs.stdenv.cc.libc
-        # pkgs.getent
-        # pkgs.getconf
-        # pkgs.gnugrep
-        # pkgs.gnupatch
-        # pkgs.gnused
-        # pkgs.gnutar
-        # pkgs.gzip
-        # pkgs.xz
-        pkgs.less
-        # pkgs.libcap  #< implicitly required by NetworkManager/wpa_supplicant!
-        # pkgs.ncurses
-        pkgs.netcat
-        # config.programs.ssh.package
-        # pkgs.mkpasswd
-        pkgs.procps
-        # pkgs.su
-        # pkgs.time
-        # pkgs.util-linux
-        # pkgs.which
-        # pkgs.zstd
-      ];
-      conveniencePackages = [
-        config.boot.kernelPackages.cpupower  # <repo:nixos/nixpkgs:nixos/modules/tasks/cpu-freq.nix> places it on PATH for convenience if powerManagement.cpuFreqGovernor is set
-        pkgs.kbd  # <repo:nixos/nixpkgs:nixos/modules/config/console.nix>  places it on PATH as part of console/virtual TTYs, but probably not needed unless you want to set console fonts
-      ];
-    in lib.filter (p: ! builtins.elem p (requiredPackages ++ conveniencePackages));
-  };
+  # remove all the non-existent default directories from XDG_DATA_DIRS, XDG_CONFIG_DIRS to simplify debugging.
+  # this is defaulted in <repo:nixos/nixpkgs:nixos/modules/programs/environment.nix>,
+  # without being gated by any higher config.
+  environment.profiles = lib.mkForce [
+    "/etc/profiles/per-user/$USER"
+    "/run/current-system/sw"
+  ];
 
-  options.system.fsPackages = lib.mkOption {
-    # <repo:nixos/nixpkgs:nixos/modules/tasks/filesystems/vfat.nix>  adds `mtools` and `dosfstools`
-    # dosfstools actually makes its way into the initrd (`fsck.vfat`).
-    # mtools is like "MS-DOS for Linux", ancient functionality i'll never use.
-    apply = lib.filter (p: p != pkgs.mtools);
-  };
+  # NIXPKGS_CONFIG defaults to "/etc/nix/nixpkgs-config.nix" in <nixos/modules/programs/environment.nix>.
+  # that's never existed on my system and everything does fine without it set empty (no nixpkgs API to forcibly *unset* it).
+  environment.variables.NIXPKGS_CONFIG = lib.mkForce "";
+  # XDG_CONFIG_DIRS defaults to "/etc/xdg", which doesn't exist.
+  # in practice, pam appends the values i want to XDG_CONFIG_DIRS, though this approach causes an extra leading `:`
+  environment.sessionVariables.XDG_CONFIG_DIRS = lib.mkForce [];
+  # XCURSOR_PATH: defaults to `[ "$HOME/.icons" "$HOME/.local/share/icons" ]`, neither of which i use, just adding noise.
+  # see: <repo:nixos/nixpkgs:nixos/modules/config/xdg/icons.nix>
+  environment.sessionVariables.XCURSOR_PATH = lib.mkForce [];
 
-  config = {
-    # disable non-required packages like nano, perl, rsync, strace
-    environment.defaultPackages = [];
+  # disable nixos' portal module, otherwise /share/applications gets linked into the system and complicates things (sandboxing).
+  # instead, i manage portals myself via the sane.programs API (e.g. sane.programs.xdg-desktop-portal).
+  xdg.portal.enable = false;
+  xdg.menus.enable = false;  #< links /share/applications, and a bunch of other empty (i.e. unused) dirs
 
-    # remove all the non-existent default directories from XDG_DATA_DIRS, XDG_CONFIG_DIRS to simplify debugging.
-    # this is defaulted in <repo:nixos/nixpkgs:nixos/modules/programs/environment.nix>,
-    # without being gated by any higher config.
-    environment.profiles = lib.mkForce [
-      "/etc/profiles/per-user/$USER"
-      "/run/current-system/sw"
-    ];
+  # xdg.autostart.enable defaults to true, and links /etc/xdg/autostart into the environment, populated with .desktop files.
+  # see: <repo:nixos/nixpkgs:nixos/modules/config/xdg/autostart.nix>
+  # .desktop files are a questionable way to autostart things: i generally prefer a service manager for that.
+  xdg.autostart.enable = false;
 
-    # NIXPKGS_CONFIG defaults to "/etc/nix/nixpkgs-config.nix" in <nixos/modules/programs/environment.nix>.
-    # that's never existed on my system and everything does fine without it set empty (no nixpkgs API to forcibly *unset* it).
-    environment.variables.NIXPKGS_CONFIG = lib.mkForce "";
-    # XDG_CONFIG_DIRS defaults to "/etc/xdg", which doesn't exist.
-    # in practice, pam appends the values i want to XDG_CONFIG_DIRS, though this approach causes an extra leading `:`
-    environment.sessionVariables.XDG_CONFIG_DIRS = lib.mkForce [];
-    # XCURSOR_PATH: defaults to `[ "$HOME/.icons" "$HOME/.local/share/icons" ]`, neither of which i use, just adding noise.
-    # see: <repo:nixos/nixpkgs:nixos/modules/config/xdg/icons.nix>
-    environment.sessionVariables.XCURSOR_PATH = lib.mkForce [];
+  # nix.channel.enable: populates `/nix/var/nix/profiles/per-user/root/channels`, `/root/.nix-channels`, `$HOME/.nix-defexpr/channels`
+  #   <repo:nixos/nixpkgs:nixos/modules/config/nix-channel.nix>
+  # TODO: may want to recreate NIX_PATH, nix.settings.nix-path
+  nix.channel.enable = false;
 
-    # disable nixos' portal module, otherwise /share/applications gets linked into the system and complicates things (sandboxing).
-    # instead, i manage portals myself via the sane.programs API (e.g. sane.programs.xdg-desktop-portal).
-    xdg.portal.enable = false;
-    xdg.menus.enable = false;  #< links /share/applications, and a bunch of other empty (i.e. unused) dirs
+  # environment.stub-ld: populate /lib/ld-linux.so with an object that unconditionally errors on launch,
+  # so as to inform when trying to run a non-nixos binary?
+  # IMO that's confusing: i thought /lib/ld-linux.so was some file actually required by nix.
+  environment.stub-ld.enable = false;
 
-    # xdg.autostart.enable defaults to true, and links /etc/xdg/autostart into the environment, populated with .desktop files.
-    # see: <repo:nixos/nixpkgs:nixos/modules/config/xdg/autostart.nix>
-    # .desktop files are a questionable way to autostart things: i generally prefer a service manager for that.
-    xdg.autostart.enable = false;
+  # `less.enable` sets LESSKEYIN_SYSTEM, LESSOPEN, LESSCLOSE env vars, which does confusing "lesspipe" things, so disable that.
+  # it's enabled by default from `<nixos/modules/programs/environment.nix>`, who also sets `PAGER="less"` and `EDITOR="nano"` (keep).
+  programs.less.enable = lib.mkForce false;
+  environment.variables.PAGER = lib.mkOverride 900 "";  # mkDefault sets 1000. non-override is 100. 900 will beat the nixpkgs `mkDefault` but not anyone else.
+  environment.variables.EDITOR = lib.mkOverride 900 "";
 
-    # nix.channel.enable: populates `/nix/var/nix/profiles/per-user/root/channels`, `/root/.nix-channels`, `$HOME/.nix-defexpr/channels`
-    #   <repo:nixos/nixpkgs:nixos/modules/config/nix-channel.nix>
-    # TODO: may want to recreate NIX_PATH, nix.settings.nix-path
-    nix.channel.enable = false;
+  # several packages (dconf, modemmanager, networkmanager, gvfs, polkit, udisks, bluez/blueman, feedbackd, etc)
+  # will add themselves to the dbus search path.
+  # i prefer dbus to only search XDG paths (/share/dbus-1) for service files, as that's more introspectable.
+  # see: <repo:nixos/nixpkgs:nixos/modules/services/system/dbus.nix>
+  # TODO: sandbox dbus? i pretty explicitly don't want to use it as a launcher.
+  services.dbus.packages = lib.mkForce [
+    "/run/current-system/sw"
+    # config.system.path
+    # pkgs.dbus
+    # pkgs.polkit.out
+    # pkgs.modemmanager
+    # pkgs.networkmanager
+    # pkgs.udisks
+    # pkgs.wpa_supplicant
+  ];
 
-    # environment.stub-ld: populate /lib/ld-linux.so with an object that unconditionally errors on launch,
-    # so as to inform when trying to run a non-nixos binary?
-    # IMO that's confusing: i thought /lib/ld-linux.so was some file actually required by nix.
-    environment.stub-ld.enable = false;
-
-    # `less.enable` sets LESSKEYIN_SYSTEM, LESSOPEN, LESSCLOSE env vars, which does confusing "lesspipe" things, so disable that.
-    # it's enabled by default from `<nixos/modules/programs/environment.nix>`, who also sets `PAGER="less"` and `EDITOR="nano"` (keep).
-    programs.less.enable = lib.mkForce false;
-    environment.variables.PAGER = lib.mkOverride 900 "";  # mkDefault sets 1000. non-override is 100. 900 will beat the nixpkgs `mkDefault` but not anyone else.
-    environment.variables.EDITOR = lib.mkOverride 900 "";
-
-    # several packages (dconf, modemmanager, networkmanager, gvfs, polkit, udisks, bluez/blueman, feedbackd, etc)
-    # will add themselves to the dbus search path.
-    # i prefer dbus to only search XDG paths (/share/dbus-1) for service files, as that's more introspectable.
-    # see: <repo:nixos/nixpkgs:nixos/modules/services/system/dbus.nix>
-    # TODO: sandbox dbus? i pretty explicitly don't want to use it as a launcher.
-    services.dbus.packages = lib.mkForce [
-      "/run/current-system/sw"
-      # config.system.path
-      # pkgs.dbus
-      # pkgs.polkit.out
-      # pkgs.modemmanager
-      # pkgs.networkmanager
-      # pkgs.udisks
-      # pkgs.wpa_supplicant
-    ];
-
-    # systemd by default forces shitty defaults for e.g. /tmp/.X11-unix.
-    # nixos propagates those in: <nixos/modules/system/boot/systemd/tmpfiles.nix>
-    # by overwriting this with an empty file, we can effectively remove it.
-    environment.etc."tmpfiles.d/x11.conf".text = "# (removed by Colin)";
-
-    # see: <nixos/modules/tasks/swraid.nix>
-    # it was enabled by default before 23.11
-    boot.swraid.enable = lib.mkDefault false;
-
-    # see: <nixos/modules/tasks/bcache.nix>
-    # these allow you to use the Linux block cache (cool! doesn't need to be a default though)
-    boot.bcache.enable = lib.mkDefault false;
-
-    # see: <nixos/modules/system/boot/kernel.nix>
-    # by default, it adds to boot.initrd.availableKernelModules:
-    # - SATA: "ahci" "sata_nv" "sata_via" "sata_sis" "sata_uli" "ata_piix" "pata_marvell"
-    # - "nvme"
-    # - scsi: "sd_mod" "sr_mod"
-    # - SD/eMMC: "mmc_block"
-    # - USB keyboards: "uhci_hcd" "ehci_hcd" "ehci_pci" "ohci_hcd" "ohci_pci" "xhci_hcd" "xhci_pci" "usbhid" "hid_generic" "hid_lenovo" "hid_apple" "hid_roccat" "hid_logitech_hidpp" "hid_logitech_dj" "hid_microsoft" "hid_cherry" "hid_corsair"
-    # - LVM: "dm_mod"
-    # - on x86 only: more keyboard stuff: "pcips2" "atkbd" "i8042"
-
-    boot.initrd.includeDefaultModules = lib.mkDefault false;
-
-    # see: <repo:nixos/nixpkgs:nixos/modules/virtualisation/nixos-containers.nix>
-    boot.enableContainers = lib.mkDefault false;
-
-    # see: <repo:nixos/nixpkgs:nixos/modules/tasks/lvm.nix>
-    # lvm places `pkgs.lvm2` onto PATH, which has like 100 binaries.
-    # it is, actually, needed for some userspace tools (cryptsetup). probably just the udev rules. try to reduce this set?
-    services.lvm.enable = lib.mkDefault false;
-    services.udev.packages = [ pkgs.lvm2.out ];  #< N.B. `lvm2.out` != `lvm2`
-    # systemd.packages = [ pkgs.lvm2 ];
-    # systemd.tmpfiles.packages = [ pkgs.lvm2.out ];
-    # environment.systemPackages = [ pkgs.lvm2 ];
-  };
+  # systemd by default forces shitty defaults for e.g. /tmp/.X11-unix.
+  # nixos propagates those in: <nixos/modules/system/boot/systemd/tmpfiles.nix>
+  # by overwriting this with an empty file, we can effectively remove it.
+  environment.etc."tmpfiles.d/x11.conf".text = "# (removed by Colin)";
 }

hosts/common/programs/alsa-ucm-conf/default.nix

@@ -15,33 +15,39 @@ in
     };
 
     # upstream alsa ships with PinePhone audio configs, but they don't actually produce sound.
-    # - still true as of 2024-05-26
-    # - see: <https://github.com/alsa-project/alsa-ucm-conf/pull/134>
+    # see: <https://github.com/alsa-project/alsa-ucm-conf/pull/134>
+    # these audio files come from some revision of:
+    # - <https://gitlab.manjaro.org/manjaro-arm/packages/community/phosh/alsa-ucm-pinephone>
     #
-    # we can substitute working UCM conf in two ways:
-    # 1. nixpkgs' override for the `alsa-ucm-conf` package
-    #   - that forces a rebuild of ~500 packages (including webkitgtk).
-    # 2. set ALSA_CONFIG_UCM2 = /path/to/ucm2 in the relevant places
-    #   - e.g. pulsewire service.
-    #   - easy to miss places, though.
+    # alternative to patching is to plumb `ALSA_CONFIG_UCM2 = "${./ucm2}"` environment variable into the relevant places
+    # e.g. `systemd.services.pulseaudio.environment`.
+    # that leaves more opportunity for gaps (i.e. missing a service),
+    # on the other hand this method causes about 500 packages to be rebuilt (including qt5 and webkitgtk).
     #
-    # alsa-ucm-pinephone-manjaro (2024-05-26):
-    # - headphones work
-    # - "internal earpiece" works
-    # - "internal speaker" is silent (maybe hardware issue)
-    # - 3.5mm connection is flapping when playing to my car, which eventually breaks audio and requires restarting wireplumber
-    # packageUnwrapped = pkgs.alsa-ucm-pinephone-manjaro.override {
-    #   inherit (cfg.config) preferEarpiece;
-    # };
-    # alsa-ucm-pinephone-pmos (2024-05-26):
-    # - headphones work
-    # - "internal earpiece" works
-    # - "internal speaker" is silent (maybe hardware issue)
-    packageUnwrapped = pkgs.alsa-ucm-pinephone-pmos.override {
-      inherit (cfg.config) preferEarpiece;
-    };
+    # note that with these files, the following audio device support:
+    # - headphones work.
+    # - "internal earpiece" works.
+    # - "internal speaker" doesn't work (but that's probably because i broke the ribbon cable)
+    # - "analog output" doesn't work.
+    packageUnwrapped = pkgs.alsa-ucm-conf.overrideAttrs (upstream: {
+      postPatch = (upstream.postPatch or "") + ''
+        cp ${./ucm2/PinePhone}/* ucm2/Allwinner/A64/PinePhone/
 
-    sandbox.enable = false;  #< only provides $out/share/alsa
+        # fix the self-contained ucm files i source from to have correct path within the alsa-ucm-conf source tree
+        substituteInPlace ucm2/Allwinner/A64/PinePhone/PinePhone.conf \
+          --replace-fail 'HiFi.conf' '/Allwinner/A64/PinePhone/HiFi.conf'
+        substituteInPlace ucm2/Allwinner/A64/PinePhone/PinePhone.conf \
+          --replace-fail 'VoiceCall.conf' '/Allwinner/A64/PinePhone/VoiceCall.conf'
+      '' + lib.optionalString cfg.config.preferEarpiece ''
+        # decrease the priority of the internal speaker so that sounds are routed
+        # to the earpiece by default.
+        # this is just personal preference.
+        substituteInPlace ucm2/Allwinner/A64/PinePhone/{HiFi.conf,VoiceCall.conf} \
+          --replace-fail 'PlaybackPriority 300' 'PlaybackPriority 100'
+      '';
+    });
+
+    sandbox.enable = false;  #< only provides #out/share/alsa
 
     # alsa-lib package only looks in its $out/share/alsa to find runtime config data, by default.
     # but ALSA_CONFIG_UCM2 is an env var that can override that.

hosts/common/programs/animatch.nix

@@ -30,8 +30,6 @@
       });
     };
 
-    buildCost = 1;
-
     sandbox.method = "bwrap";
     sandbox.whitelistWayland = true;
 

hosts/common/programs/assorted.nix

@@ -34,31 +34,26 @@ in
     ];
 
     sysadminUtils = declPackageSet [
-      "ausyscall"
       "bridge-utils"  # for brctl; debug linux "bridge" inet devices
       "btrfs-progs"
       "cacert.unbundled"  # some services require unbundled /etc/ssl/certs
       "cryptsetup"
-      "curl"
       "ddrescue"
       "dig"
       "dtc"  # device tree [de]compiler
       "e2fsprogs"  # resize2fs
       "efibootmgr"
-      "errno"
       "ethtool"
       "fatresize"
       "fd"
       "file"
       "forkstat"  # monitor every spawned/forked process
-      "free"
       # "fwupd"
       "gawk"
       "gdb"  # to debug segfaults
       "git"
       "gptfdisk"  # gdisk
       "hdparm"
-      "hping"
       "htop"
       "iftop"
       "inetutils"  # for telnet
@@ -68,27 +63,20 @@ in
       "jq"
       "killall"
       "less"
-      "lftp"
       # "libcap_ng"  # for `netcap`
       "lsof"
-      "man-pages"
-      "man-pages-posix"
       # "miniupnpc"
-      "mmcli"
       "nano"
       #  "ncdu"  # ncurses disk usage. doesn't cross compile (zig)
       "neovim"
       "netcat"
       "nethogs"
-      "nix"
       "nmap"
-      "nmcli"
       "nvme-cli"  # nvme
       # "openssl"
       "parted"
       "pciutils"
       "powertop"
-      "ps"
       "pstree"
       "ripgrep"
       "s6-rc"  # service manager
@@ -101,8 +89,6 @@ in
       "tree"
       "usbutils"  # lsusb
       "util-linux"  # lsblk, lscpu, etc
-      "valgrind"
-      "watch"
       "wget"
       "wirelesstools"  # iwlist
       # "xq"  # jq for XML
@@ -122,12 +108,12 @@ in
     #   - debugging?
     consoleUtils = declPackageSet [
       "alsaUtils"  # for aplay, speaker-test
+      "strings"
       # "cdrtools"
       # "clinfo"
       # "dmidecode"
       "dtrx"  # `unar` alternative, "Do The Right eXtraction"
       # "efivar"
-      "exiftool"
       "eza"  # a better 'ls'
       # "flashrom"
       "git"  # needed as a user package, for config.
@@ -151,25 +137,18 @@ in
       "nmon"
       # "node2nix"
       # "oathToolkit"  # for oathtool
-      "objdump"
       # "ponymix"
       "pulsemixer"
       "python3-repl"
-      # "python3.pkgs.eyeD3"  # music tagging
+      # "python3Packages.eyeD3"  # music tagging
       "ripgrep"  # needed as a user package so that its user-level config file can be installed
       "rsync"
-      "rsyslog"  # KEEP THIS HERE if you want persistent logging
-      "sane-deadlines"
       "sane-scripts.bittorrent"
       "sane-scripts.cli"
-      "sane-secrets-unlock"
-      "sane-sysload"
-      "sc-im"
       # "snapper"
       "sops"  # for manually viewing secrets; outside `sane-secrets` (TODO: improve sane-secrets!)
       "speedtest-cli"
       # "ssh-to-age"
-      "strings"
       "sudo"
       # "tageditor"  # music tagging
       # "unar"
@@ -184,12 +163,8 @@ in
       # "gh"  # MS GitHub cli
       "nix-index"
       "nixpkgs-review"
-      "qmk-udev-rules"
       "sane-scripts.dev"
       "sequoia"
-      # "via"
-      "wally-cli"
-      # "zsa-udev-rules"
     ];
 
     consoleMediaUtils = declPackageSet [
@@ -198,7 +173,6 @@ in
       "ffmpeg"
       "go2tv"  # cast videos to UPNP/DLNA device (i.e. tv).
       "imagemagick"
-      "sane-cast"  # cast videos to UPNP/DLNA, with compatibility
       "sox"
       "yt-dlp"
     ];
@@ -229,179 +203,6 @@ in
       # "tree-sitter"
     ];
 
-    gameApps = declPackageSet [
-      "animatch"
-      "gnome-2048"
-      "gnome.hitori"  # like sudoku
-    ];
-
-    pcGameApps = declPackageSet [
-      # "andyetitmoves" # TODO: fix build!
-      # "armagetronad"  # tron/lightcycles; WAN and LAN multiplayer
-      "celeste64"
-      # "cutemaze"      # meh: trivial maze game; qt6 and keyboard-only
-      # "cuyo"          # trivial puyo-puyo clone
-      "endless-sky"     # space merchantilism/exploration
-      # "factorio"
-      "frozen-bubble"   # WAN + LAN + 1P/2P bubble bobble
-      "hase"            # WAN worms game
-      # "hedgewars"     # WAN + LAN worms game (5~10 people online at any moment; <https://hedgewars.org>)
-      # "libremines"    # meh: trivial minesweeper; qt6
-      # "mario0"        # SMB + portal
-      # "mindustry"
-      # "minesweep-rs"  # CLI minesweeper
-      # "nethack"
-      # "osu-lazer"
-      # "pinball"       # 3d pinball; kb/mouse. old sourceforge project
-      # "powermanga"    # STYLISH space invaders derivative (keyboard-only)
-      "shattered-pixel-dungeon"  # doesn't cross compile
-      "space-cadet-pinball"  # LMB/RMB controls (bindable though. volume buttons?)
-      "steam"
-      "superTux"  # keyboard-only controls
-      "superTuxKart"  # poor FPS on pinephone
-      "tumiki-fighters" # keyboard-only
-      "vvvvvv"  # keyboard-only controls
-      # "wine"
-    ];
-
-    guiApps = declPackageSet [
-      # package sets
-      "gameApps"
-      "guiBaseApps"
-    ];
-
-    guiBaseApps = declPackageSet [
-      # "abaddon"  # discord client
-      "alacritty"  # terminal emulator
-      "calls"  # gnome calls (dialer/handler)
-      "dbus"
-      "dconf"  # required by many packages, but not well-documented :(
-      # "delfin"  # Jellyfin client
-      "dialect"  # language translation
-      "dino"  # XMPP client
-      "dissent"  # Discord client (formerly known as: gtkcord4)
-      # "emote"
-      # "evince"  # PDF viewer
-      # "flare-signal"  # gtk4 signal client
-      # "foliate"  # e-book reader
-      "fractal"  # matrix client
-      "g4music"  # local music player
-      # "gnome.cheese"
-      # "gnome-feeds"  # RSS reader (with claimed mobile support)
-      # "gnome.file-roller"
-      "geary"  # adaptive e-mail client; uses webkitgtk 4.1
-      "gnome-calculator"
-      "gnome-calendar"
-      "gnome.gnome-clocks"
-      "gnome.gnome-maps"
-      # "gnome-podcasts"
-      # "gnome.gnome-system-monitor"
-      # "gnome.gnome-terminal"  # works on phosh
-      "gnome.gnome-weather"
-      # "seahorse"  # keyring/secret manager
-      "gnome-frog"  # OCR/QR decoder
-      "gpodder"
-      # "gst-device-monitor"  # for debugging audio/video
-      # "gthumb"
-      # "lemoa"  # lemmy app
-      # "libcamera"  # for `cam` binary (useful for debugging cameras)
-      "libnotify"  # for notify-send; debugging
-      # "lollypop"
-      "loupe"  # image viewer
-      "mate.engrampa"  # archive manager
-      "mepo"  # maps viewer
-      # "mesa-demos"  # for eglinfo, glxinfo & other testing tools
-      "mpv"
-      "networkmanagerapplet"  # for nm-connection-editor: it's better than not having any gui!
-      "ntfy-sh"  # notification service
-      # "newsflash"  # RSS viewer
-      "pavucontrol"
-      "pwvucontrol"  # pipewire version of pavu
-      # "picard"  # music tagging
-      # "libsForQt5.plasmatube"  # Youtube player
-      "signal-desktop"
-      # "snapshot"  # camera app
-      "spot"  # Gnome Spotify client
-      # "sublime-music"
-      # "tdesktop"  # broken on phosh
-      # "tokodon"
-      "tuba"  # mastodon/pleroma client (stores pw in keyring)
-      "vulkan-tools"  # vulkaninfo
-      # "whalebird"  # pleroma client (Electron). input is broken on phosh.
-      "xdg-terminal-exec"
-      "youtube-tui"
-      "zathura"  # PDF/CBZ/ePUB viewer
-    ];
-
-    handheldGuiApps = declPackageSet [
-      # "celluloid"  # mpv frontend
-      # "chatty"  # matrix/xmpp/irc client  (2023/12/29: disabled because broken cross build)
-      # "cozy"  # audiobook player
-      "epiphany"  # gnome's web browser
-      # "iotas"  # note taking app
-      "komikku"
-      "koreader"
-      "lgtrombetta-compass"
-      "megapixels"  # camera app
-      "notejot"  # note taking, e.g. shopping list
-      "planify"  # todo-tracker/planner
-      "portfolio-filemanager"
-      "tangram"  # web browser
-      "wike"  # Wikipedia Reader
-      "xarchiver"  # archiver, backup option for when engrampa UI overflows screen and is unusale (xarchiver UI fails in different ways)
-    ];
-
-    pcGuiApps = declPackageSet [
-      # package sets
-      "pcGameApps"
-      "pcTuiApps"
-      ####
-      "audacity"
-      # "blanket"  # ambient noise generator
-      "brave"  # for the integrated wallet -- as a backup
-      # "cantata"  # music player (mpd frontend)
-      # "chromium"  # chromium takes hours to build. brave is chromium-based, distributed in binary form, so prefer it.
-      # "cups"
-      "discord"  # x86-only
-      # "electrum"
-      "element-desktop"
-      "firefox"
-      "font-manager"
-      # "gajim"  # XMPP client. cross build tries to import host gobject-introspection types (2023/09/01)
-      "gimp"  # broken on phosh
-      # "gnome.dconf-editor"
-      # "gnome.file-roller"
-      "gnome-disk-utility"
-      "nautilus"  # file browser
-      # "gnome.totem"  # video player, supposedly supports UPnP
-      # "handbrake"  #< TODO: fix build
-      "inkscape"
-      # "jellyfin-media-player"
-      "kdenlive"
-      # "keymapp"
-      # "kid3"  # audio tagging
-      "krita"
-      "libreoffice"  # TODO: replace with an office suite that uses saner packaging?
-      "losslesscut-bin"  # x86-only
-      # "makemkv"  # x86-only
-      # "monero-gui"  # x86-only
-      # "mumble"
-      # "nheko"  # Matrix chat client
-      "nicotine-plus"  # soulseek client
-      # "obsidian"
-      # "openscad"  # 3d modeling
-      # "rhythmbox"  # local music player
-      # "slic3r"
-      "soundconverter"
-      "spotify"  # x86-only
-      "tor-browser"  # x86-only
-      # "vlc"
-      "wireshark"  # could maybe ship the cli as sysadmin pkg
-      # "xterm"  # requires Xwayland
-      # "zecwallet-lite"  # x86-only
-      # "zulip"
-    ];
-
 
     # INDIVIDUAL PACKAGE DEFINITIONS
 
@@ -410,7 +211,6 @@ in
 
     backblaze-b2 = {};
 
-    blanket.buildCost = 1;
     blanket.sandbox.method = "bwrap";
     blanket.sandbox.whitelistAudio = true;
     # blanket.sandbox.whitelistDbus = [ "user" ];  # TODO: untested
@@ -429,6 +229,14 @@ in
     bridge-utils.sandbox.method = "bwrap";  #< bwrap, landlock: both work
     bridge-utils.sandbox.net = "all";
 
+    brightnessctl.sandbox.method = "landlock";  # also bwrap, but landlock is more responsive
+    brightnessctl.sandbox.extraPaths = [
+      "/sys/class/backlight"
+      "/sys/class/leds"
+      "/sys/devices"
+    ];
+    brightnessctl.sandbox.whitelistDbus = [ "system" ];
+
     btrfs-progs.sandbox.method = "bwrap";  #< bwrap, landlock: both work
     btrfs-progs.sandbox.autodetectCliPaths = "existing";  # e.g. `btrfs filesystem df /my/fs`
 
@@ -438,11 +246,6 @@ in
 
     clang = {};
 
-    clightning-sane.sandbox.method = "bwrap";
-    clightning-sane.sandbox.extraPaths = [
-      "/var/lib/clightning/bitcoin/lightning-rpc"
-    ];
-
     # cryptsetup: typical use is `cryptsetup open /dev/loopxyz mappedName`, and creates `/dev/mapper/mappedName`
     cryptsetup.sandbox.method = "landlock";
     cryptsetup.sandbox.extraPaths = [
@@ -460,14 +263,13 @@ in
     ddrescue.sandbox.method = "landlock";  # TODO:sandbox: untested
     ddrescue.sandbox.autodetectCliPaths = "existingOrParent";
 
-    delfin.buildCost = 1;
+    # auth token, preferences
     delfin.sandbox.method = "bwrap";
     delfin.sandbox.whitelistAudio = true;
     delfin.sandbox.whitelistDbus = [ "user" ];  # else `mpris` plugin crashes the player
     delfin.sandbox.whitelistDri = true;
     delfin.sandbox.whitelistWayland = true;
     delfin.sandbox.net = "clearnet";
-    # auth token, preferences
     delfin.persist.byStore.private = [ ".config/delfin" ];
 
     dig.sandbox.method = "bwrap";
@@ -494,7 +296,11 @@ in
     ];
 
     dtc.sandbox.method = "bwrap";
-    dtc.sandbox.autodetectCliPaths = "existingFile";  # TODO:sandbox: untested
+    dtc.sandbox.autodetectCliPaths = true;  # TODO:sandbox: untested
+
+    dtrx.sandbox.method = "bwrap";
+    dtrx.sandbox.whitelistPwd = true;
+    dtrx.sandbox.autodetectCliPaths = "existing";  #< for the archive
 
     duplicity = {};
 
@@ -508,13 +314,11 @@ in
 
     eg25-control = {};
 
-    electrum.buildCost = 1;
     electrum.sandbox.method = "bwrap";  # TODO:sandbox: untested
     electrum.sandbox.net = "all";  # TODO: probably want to make this run behind a VPN, always
     electrum.sandbox.whitelistWayland = true;
-    electrum.persist.byStore.ephemeral = [ ".electrum" ];  #< TODO: use XDG dirs!
+    electrum.persist.byStore.cryptClearOnBoot = [ ".electrum" ];  #< TODO: use XDG dirs!
 
-    endless-sky.buildCost = 1;
     endless-sky.persist.byStore.plaintext = [ ".local/share/endless-sky" ];
     endless-sky.sandbox.method = "bwrap";
     endless-sky.sandbox.whitelistAudio = true;
@@ -530,11 +334,10 @@ in
     ethtool.sandbox.capabilities = [ "net_admin" ];
 
     # eza `ls` replacement
-    # bwrap causes `/proc` files to be listed differently (e.g. `eza /proc/sys/net/ipv6/conf/`)
-    # bwrap loses group info (so files owned by other users appear as owner "nobody")
-    eza.sandbox.method = "landlock";
-    # eza.sandbox.method = "bwrap";
-    eza.sandbox.autodetectCliPaths = "existing";
+    # landlock is OK, only `whitelistPwd` doesn't make the intermediate symlinks traversable, so it breaks on e.g. ~/Videos/servo/Shows/foo
+    # eza.sandbox.method = "landlock";
+    eza.sandbox.method = "bwrap";
+    eza.sandbox.autodetectCliPaths = true;
     eza.sandbox.whitelistPwd = true;
     eza.sandbox.extraHomePaths = [
       # so that e.g. `eza -l ~` can show which symlink exist
@@ -546,7 +349,7 @@ in
     fatresize.sandbox.autodetectCliPaths = "parent";  # /dev/sda1 -> needs /dev/sda
 
     fd.sandbox.method = "landlock";
-    fd.sandbox.autodetectCliPaths = "existing";
+    fd.sandbox.autodetectCliPaths = true;
     fd.sandbox.whitelistPwd = true;
     fd.sandbox.extraHomePaths = [
       # let it follow symlinks to non-sensitive data
@@ -554,15 +357,14 @@ in
       ".persist/plaintext"
     ];
 
-    ffmpeg.buildCost = 1;
     ffmpeg.sandbox.method = "bwrap";
     ffmpeg.sandbox.autodetectCliPaths = "existingFileOrParent";  # it outputs uncreated files -> parent dir needs mounting
 
     file.sandbox.method = "bwrap";
-    file.sandbox.autodetectCliPaths = "existing";  #< file OR directory, yes
+    file.sandbox.autodetectCliPaths = true;
 
     findutils.sandbox.method = "bwrap";
-    findutils.sandbox.autodetectCliPaths = "existing";
+    findutils.sandbox.autodetectCliPaths = true;
     findutils.sandbox.whitelistPwd = true;
     findutils.sandbox.extraHomePaths = [
       # let it follow symlinks to non-sensitive data
@@ -572,16 +374,16 @@ in
 
     fluffychat-moby.persist.byStore.plaintext = [ ".local/share/chat.fluffy.fluffychat" ];
 
-    font-manager.buildCost = 1;
     font-manager.sandbox.method = "bwrap";
-    font-manager.sandbox.whitelistWayland = true;
     font-manager.packageUnwrapped = pkgs.rmDbusServicesInPlace (pkgs.font-manager.override {
       # build without the "Google Fonts" integration feature, to save closure / avoid webkitgtk_4_0
       withWebkit = false;
     });
 
     forkstat.sandbox.method = "landlock";  #< doesn't seem to support bwrap
-    forkstat.sandbox.isolatePids = false;
+    forkstat.sandbox.extraConfig = [
+      "--sane-sandbox-keep-namespace" "pid"
+    ];
     forkstat.sandbox.extraPaths = [
       "/proc"
     ];
@@ -595,7 +397,11 @@ in
 
     gawk.sandbox.method = "bwrap";  # TODO:sandbox: untested
     gawk.sandbox.wrapperType = "inplace";  # /share/gawk libraries refer to /libexec
-    gawk.sandbox.autodetectCliPaths = "existingFile";
+    gawk.sandbox.autodetectCliPaths = true;
+
+    gdb.sandbox.enable = false;  # gdb doesn't sandbox well. i don't know how you could.
+    # gdb.sandbox.method = "landlock";  # permission denied when trying to attach, even as root
+    gdb.sandbox.autodetectCliPaths = true;
 
     geoclue2-with-demo-agent = {};
 
@@ -603,7 +409,6 @@ in
     # TODO: we can populate gh's stuff statically; it even lets us use the same oauth across machines
     gh.persist.byStore.private = [ ".config/gh" ];
 
-    gimp.buildCost = 1;
     gimp.sandbox.method = "bwrap";
     gimp.sandbox.whitelistX = true;
     gimp.sandbox.whitelistWayland = true;
@@ -623,44 +428,32 @@ in
       "/tmp"  # "Cannot open display:" if it can't mount /tmp 👀
     ];
 
-    gnome-calculator.buildCost = 1;
-    gnome-calculator.sandbox.method = "bwrap";
-    gnome-calculator.sandbox.whitelistWayland = true;
+    "gnome.gnome-calculator".sandbox.method = "bwrap";
+    "gnome.gnome-calculator".sandbox.whitelistWayland = true;
 
-    gnome-calendar.buildCost = 1;
     # gnome-calendar surely has data to persist, but i use it strictly to do date math, not track events.
-    gnome-calendar.sandbox.method = "bwrap";
-    gnome-calendar.sandbox.whitelistWayland = true;
+    "gnome.gnome-calendar".sandbox.method = "bwrap";
+    "gnome.gnome-calendar".sandbox.whitelistWayland = true;
+
+    "gnome.gnome-clocks".sandbox.method = "bwrap";
+    "gnome.gnome-clocks".sandbox.whitelistWayland = true;
+    "gnome.gnome-clocks".suggestedPrograms = [ "dconf" ];
 
     # gnome-disks
-    gnome-disk-utility.buildCost = 1;
-    gnome-disk-utility.sandbox.method = "bwrap";
-    gnome-disk-utility.sandbox.whitelistDbus = [ "system" ];
-    gnome-disk-utility.sandbox.whitelistWayland = true;
-    gnome-disk-utility.sandbox.extraHomePaths = [
-      "tmp"
-      "use/iso"
-      # TODO: probably need /dev and such
-    ];
-
-    hping.sandbox.method = "landlock";
-    hping.sandbox.net = "all";
-    hping.sandbox.capabilities = [ "net_raw" ];
-    hping.sandbox.autodetectCliPaths = "existingFile";  # for sending packet data from file
+    "gnome.gnome-disk-utility".sandbox.method = "bwrap";
+    "gnome.gnome-disk-utility".sandbox.whitelistDbus = [ "system" ];
+    "gnome.gnome-disk-utility".sandbox.whitelistWayland = true;
 
     # seahorse: dump gnome-keyring secrets.
-    seahorse.buildCost = 1;
-    # N.B. it can lso manage ~/.ssh keys, but i explicitly don't add those to the sandbox for now.
-    seahorse.sandbox.method = "bwrap";
-    seahorse.sandbox.whitelistDbus = [ "user" ];
-    seahorse.sandbox.whitelistWayland = true;
+    # N.B.: it can also manage ~/.ssh keys, but i explicitly don't add those to the sandbox for now.
+    "gnome.seahorse".sandbox.method = "bwrap";
+    "gnome.seahorse".sandbox.whitelistDbus = [ "user" ];
+    "gnome.seahorse".sandbox.whitelistWayland = true;
 
-    gnome-2048.buildCost = 1;
     gnome-2048.sandbox.method = "bwrap";
     gnome-2048.sandbox.whitelistWayland = true;
     gnome-2048.persist.byStore.plaintext = [ ".local/share/gnome-2048/scores" ];
 
-    gnome-frog.buildCost = 1;
     gnome-frog.sandbox.method = "bwrap";
     gnome-frog.sandbox.whitelistWayland = true;
     gnome-frog.sandbox.whitelistDbus = [ "user" ];
@@ -678,7 +471,7 @@ in
       "Pictures/Screenshots"
       "Pictures/servo-macros"
     ];
-    gnome-frog.persist.byStore.ephemeral = [
+    gnome-frog.persist.byStore.cryptClearOnBoot = [
       ".local/share/tessdata"  # 15M; dunno what all it is.
     ];
 
@@ -687,12 +480,11 @@ in
     # 1. no number may appear unshaded more than once in the same row/column
     # 2. no two shaded tiles can be direct N/S/E/W neighbors
     # - win once (1) and (2) are satisfied
-    "gnome.hitori".buildCost = 1;
     "gnome.hitori".sandbox.method = "bwrap";
     "gnome.hitori".sandbox.whitelistWayland = true;
 
     gnugrep.sandbox.method = "bwrap";
-    gnugrep.sandbox.autodetectCliPaths = "existing";
+    gnugrep.sandbox.autodetectCliPaths = true;
     gnugrep.sandbox.whitelistPwd = true;
     gnugrep.sandbox.extraHomePaths = [
       # let it follow symlinks to non-sensitive data
@@ -700,6 +492,7 @@ in
       ".persist/plaintext"
     ];
 
+    # sed: there is an edgecase of `--file=<foo>`, wherein `foo` won't be whitelisted.
     gnused.sandbox.method = "bwrap";
     gnused.sandbox.autodetectCliPaths = "existingFile";
     gnused.sandbox.whitelistPwd = true;  #< `-i` flag creates a temporary file in pwd (?) and then moves it.
@@ -716,7 +509,6 @@ in
     grim.sandbox.autodetectCliPaths = "existingOrParent";
     grim.sandbox.whitelistWayland = true;
 
-    hase.buildCost = 1;
     hase.sandbox.method = "bwrap";
     hase.sandbox.net = "clearnet";
     hase.sandbox.whitelistAudio = true;
@@ -725,7 +517,7 @@ in
 
     # hdparm: has to be run as sudo. e.g. `sudo hdparm -i /dev/sda`
     hdparm.sandbox.method = "bwrap";
-    hdparm.sandbox.autodetectCliPaths = "existingFile";
+    hdparm.sandbox.autodetectCliPaths = true;
 
     host.sandbox.method = "landlock";
     host.sandbox.net = "all";  #< technically, only needs to contact localhost's DNS server
@@ -737,7 +529,6 @@ in
     # N.B.: inetutils' `ping` is shadowed by iputils' ping (by nixos, intentionally).
     inetutils.sandbox.method = "landlock";  # want to keep the same netns, at least.
 
-    inkscape.buildCost = 1;
     inkscape.sandbox.method = "bwrap";
     inkscape.sandbox.whitelistWayland = true;
     inkscape.sandbox.extraHomePaths = [
@@ -759,17 +550,14 @@ in
     ];
     iotop.sandbox.capabilities = [ "net_admin" ];
 
-    # provides `ip`, `routel`, `bridge`, others.
-    # landlock works fine for most of these, but `ip netns exec` wants to attach to an existing namespace
-    # and that means we can't use ANY sandboxer for it.
-    iproute2.sandbox.enable = false;
-    # iproute2.sandbox.net = "all";
-    # iproute2.sandbox.capabilities = [ "net_admin" ];
-    # iproute2.sandbox.extraPaths = [
-    #   "/run/netns"  # for `ip netns ...` to work, but maybe not needed anymore?
-    #   "/sys/class/net"  # for `ip netns ...` to work
-    #   "/var/run/netns"
-    # ];
+    # provides `ip`, `routel`, others
+    iproute2.sandbox.method = "landlock";
+    iproute2.sandbox.net = "all";
+    iproute2.sandbox.capabilities = [ "net_admin" ];
+    iproute2.sandbox.extraPaths = [
+      "/run/netns"  # for `ip netns ...` to work
+      "/var/run/netns"
+    ];
 
     iptables.sandbox.method = "landlock";
     iptables.sandbox.net = "all";
@@ -792,7 +580,6 @@ in
       "/proc"
     ];
 
-    krita.buildCost = 1;
     krita.sandbox.method = "bwrap";
     krita.sandbox.whitelistWayland = true;
     krita.sandbox.autodetectCliPaths = "existing";
@@ -808,23 +595,11 @@ in
       "tmp"
     ];
 
-    libcamera = {};
-
-    libcap.sandbox.enable = false;  #< for `capsh`, which i use as a sandboxer
     libcap_ng.sandbox.enable = false;  # there's something about /proc/$pid/fd which breaks `readlink`/stat with every sandbox technique (except capsh-only)
 
     libnotify.sandbox.method = "bwrap";
     libnotify.sandbox.whitelistDbus = [ "user" ];  # notify-send
 
-    lightning-cli.packageUnwrapped = pkgs.linkBinIntoOwnPackage pkgs.clightning "lightning-cli";
-    lightning-cli.sandbox.method = "bwrap";
-    lightning-cli.sandbox.extraHomePaths = [
-      ".lightning/bitcoin/lightning-rpc"
-    ];
-    # `lightning-cli` finds its RPC file via `~/.lightning/bitcoin/lightning-rpc`, to message the daemon
-    lightning-cli.fs.".lightning".symlink.target = "/var/lib/clightning";
-
-    losslesscut-bin.buildCost = 1;
     losslesscut-bin.sandbox.method = "bwrap";
     losslesscut-bin.sandbox.extraHomePaths = [
       "Music"
@@ -840,24 +615,14 @@ in
     losslesscut-bin.sandbox.whitelistX = true;
 
     lsof.sandbox.method = "capshonly";  # lsof doesn't sandbox under bwrap or even landlock w/ full access to /
-    lsof.sandbox.capabilities = [ "dac_override" "sys_ptrace" ];
 
     lua = {};
 
-    man-pages.sandbox.enable = false;
-    man-pages-posix.sandbox.enable = false;
-
     mercurial.sandbox.method = "bwrap";  # TODO:sandbox: untested
     mercurial.sandbox.net = "clearnet";
     mercurial.sandbox.whitelistPwd = true;
 
-    mesa-demos.sandbox.method = "bwrap";
-    mesa-demos.sandbox.whitelistDri = true;
-    mesa-demos.sandbox.whitelistWayland = true;
-    mesa-demos.sandbox.whitelistX = true;
-
     # actual monero blockchain (not wallet/etc; safe to delete, just slow to regenerate)
-    monero-gui.buildCost = 1;
     # XXX: is it really safe to persist this? it doesn't have info that could de-anonymize if captured?
     monero-gui.persist.byStore.plaintext = [ ".bitmonero" ];
     monero-gui.sandbox.method = "bwrap";
@@ -866,7 +631,6 @@ in
       "records/finance/cryptocurrencies/monero"
     ];
 
-    mumble.buildCost = 1;
     mumble.persist.byStore.private = [ ".local/share/Mumble" ];
 
     nano.sandbox.method = "bwrap";
@@ -894,15 +658,9 @@ in
     nixpkgs-review.sandbox.wrapperType = "inplace";  #< shell completions use full paths
     nixpkgs-review.sandbox.net = "clearnet";
     nixpkgs-review.sandbox.whitelistPwd = true;
-    nixpkgs-review.sandbox.extraHomePaths = [
-      ".config/git"  #< it needs to know commiter name/email, even if not posting
-    ];
     nixpkgs-review.sandbox.extraPaths = [
       "/nix"
     ];
-    nixpkgs-review.persist.byStore.ephemeral = [
-      ".cache/nixpkgs-review"  #< help it not exhaust / tmpfs
-    ];
 
     nmap.sandbox.method = "bwrap";
     nmap.sandbox.net = "all";  # clearnet and lan
@@ -931,8 +689,6 @@ in
     # settings (electron app)
     obsidian.persist.byStore.plaintext = [ ".config/obsidian" ];
 
-    passt.sandbox.enable = false;  #< sandbox helper (netns specifically)
-
     parted.sandbox.method = "landlock";
     parted.sandbox.extraPaths = [
       "/dev"
@@ -951,7 +707,7 @@ in
       "/sys/devices"
     ];
 
-    "perlPackages.FileMimeInfo" = {};
+    "perlPackages.FileMimeInfo".sandbox.enable = false;  #< TODO: sandbox `mimetype` but not `mimeopen`.
 
     powertop.sandbox.method = "landlock";
     powertop.sandbox.capabilities = [ "ipc_lock" "sys_admin" ];
@@ -964,7 +720,9 @@ in
 
     # procps: free, pgrep, pidof, pkill, ps, pwait, top, uptime, couple others
     procps.sandbox.method = "bwrap";
-    procps.sandbox.isolatePids = false;
+    procps.sandbox.extraConfig = [
+      "--sane-sandbox-keep-namespace" "pid"
+    ];
 
     pstree.sandbox.method = "landlock";
     pstree.sandbox.extraPaths = [
@@ -976,17 +734,12 @@ in
     pulsemixer.sandbox.method = "landlock";
     pulsemixer.sandbox.whitelistAudio = true;
 
-    pwvucontrol.buildCost = 1;
     pwvucontrol.sandbox.method = "bwrap";
     pwvucontrol.sandbox.whitelistAudio = true;
-    pwvucontrol.sandbox.whitelistDri = true;  # else perf on moby is unusable
     pwvucontrol.sandbox.whitelistWayland = true;
 
     python3-repl.packageUnwrapped = pkgs.python3.withPackages (ps: with ps; [
-      psutil
-      pykakasi
       requests
-      unidecode
     ]);
     python3-repl.sandbox.method = "bwrap";
     python3-repl.sandbox.net = "clearnet";
@@ -996,7 +749,7 @@ in
     ];
 
     qemu.sandbox.enable = false;  #< it's a launcher
-    qemu.buildCost = 2;
+    qemu.slowToBuild = true;
 
     rsync.sandbox.method = "bwrap";
     rsync.sandbox.net = "clearnet";
@@ -1004,31 +757,17 @@ in
 
     rustc = {};
 
-    sane-cast.sandbox.method = "bwrap";
-    sane-cast.sandbox.net = "clearnet";
-    sane-cast.sandbox.autodetectCliPaths = "existingFile";
-    sane-cast.suggestedPrograms = [ "go2tv" ];
-
-    sane-die-with-parent.sandbox.enable = false;  #< it's a launcher; can't sandbox
-
-    sane-weather.sandbox.method = "bwrap";
-    sane-weather.sandbox.net = "clearnet";
-
-    sc-im.sandbox.method = "bwrap";
-    sc-im.sandbox.autodetectCliPaths = "existingFile";
+    sane-open-desktop.sandbox.enable = false;  #< trivial script, and all our deps are sandboxed
+    sane-open-desktop.suggestedPrograms = [
+      "gdbus"
+    ];
 
     screen.sandbox.enable = false;  #< tty; needs to run anything
 
-    sequoia.packageUnwrapped = pkgs.sequoia.overrideAttrs (_: {
-      # XXX(2024-07-30): sq_autocrypt_import test failure: "Warning: 9B7DD433F254904A is expired."
-      doCheck = false;
-    });
-    sequoia.buildCost = 1;
-    sequoia.sandbox.method = "bwrap";
+    sequoia.sandbox.method = "bwrap";  # TODO:sandbox: untested
     sequoia.sandbox.whitelistPwd = true;
-    sequoia.sandbox.autodetectCliPaths = "existingFileOrParent";  # supports `-o <file-to-create>`
+    sequoia.sandbox.autodetectCliPaths = true;
 
-    shattered-pixel-dungeon.buildCost = 1;
     shattered-pixel-dungeon.persist.byStore.plaintext = [ ".local/share/.shatteredpixel/shattered-pixel-dungeon" ];
     shattered-pixel-dungeon.sandbox.method = "bwrap";
     shattered-pixel-dungeon.sandbox.whitelistAudio = true;
@@ -1036,7 +775,6 @@ in
     shattered-pixel-dungeon.sandbox.whitelistWayland = true;
 
     # printer/filament settings
-    slic3r.buildCost = 1;
     slic3r.persist.byStore.plaintext = [ ".Slic3r" ];
 
     slurp.sandbox.method = "bwrap";
@@ -1048,29 +786,22 @@ in
     smartmontools.sandbox.autodetectCliPaths = "existing";
     smartmontools.sandbox.capabilities = [ "sys_rawio" ];
 
-    # snapshot camera, based on libcamera
-    # TODO: enable dma heaps for more efficient buffer sharing: <https://gitlab.com/postmarketOS/pmaports/-/issues/2789>
-    snapshot = {};
-
     sops.sandbox.method = "bwrap";  # TODO:sandbox: untested
     sops.sandbox.extraHomePaths = [
       ".config/sops"
-      "nixos"
+      "dev/nixos"
       # TODO: sops should only need access to knowledge/secrets,
       # except that i currently put its .sops.yaml config in the root of ~/knowledge
       "knowledge"
     ];
 
-    soundconverter.buildCost = 1;
     soundconverter.sandbox.method = "bwrap";
     soundconverter.sandbox.whitelistWayland = true;
     soundconverter.sandbox.extraHomePaths = [
       "Music"
       "tmp"
       "use"
-      ".config/dconf"
     ];
-    soundconverter.sandbox.whitelistDbus = [ "user" ];  # for dconf
     soundconverter.sandbox.extraPaths = [
       "/mnt/servo/media/Music"
       "/mnt/servo/media/games"
@@ -1081,7 +812,6 @@ in
     sox.sandbox.autodetectCliPaths = "existingFileOrParent";
     sox.sandbox.whitelistAudio = true;
 
-    space-cadet-pinball.buildCost = 1;
     space-cadet-pinball.persist.byStore.plaintext = [ ".local/share/SpaceCadetPinball" ];
     space-cadet-pinball.sandbox.method = "bwrap";
     space-cadet-pinball.sandbox.whitelistAudio = true;
@@ -1102,7 +832,6 @@ in
     subversion.sandbox.whitelistPwd = true;
     sudo.sandbox.enable = false;
 
-    superTux.buildCost = 1;
     superTux.sandbox.method = "bwrap";
     superTux.sandbox.wrapperType = "inplace";  # package Makefile incorrectly installs to $out/games/superTux instead of $out/share/games
     superTux.sandbox.whitelistAudio = true;
@@ -1121,14 +850,12 @@ in
 
     tdesktop.persist.byStore.private = [ ".local/share/TelegramDesktop" ];
 
-    tokodon.buildCost = 1;
     tokodon.persist.byStore.private = [ ".cache/KDE/tokodon" ];
 
     tree.sandbox.method = "landlock";
-    tree.sandbox.autodetectCliPaths = "existing";
+    tree.sandbox.autodetectCliPaths = true;
     tree.sandbox.whitelistPwd = true;
 
-    tumiki-fighters.buildCost = 1;
     tumiki-fighters.sandbox.method = "bwrap";
     tumiki-fighters.sandbox.whitelistAudio = true;
     tumiki-fighters.sandbox.whitelistDri = true;  #< not strictly necessary, but triples CPU perf
@@ -1147,13 +874,12 @@ in
       "/sys/bus/usb"
     ];
 
-    valgrind.buildCost = 1;
-    valgrind.sandbox.enable = false;  #< it's a launcher: can't sandbox
+    visidata.sandbox.method = "bwrap";  # TODO:sandbox: untested
+    visidata.sandbox.autodetectCliPaths = true;
 
     # `vulkaninfo`, `vkcube`
     vulkan-tools.sandbox.method = "landlock";
 
-    vvvvvv.buildCost = 1;
     vvvvvv.sandbox.method = "bwrap";
     vvvvvv.sandbox.whitelistAudio = true;
     vvvvvv.sandbox.whitelistDri = true;  #< playable without, but burns noticably more CPU
@@ -1167,8 +893,6 @@ in
       "tmp"
     ];
 
-    watch.sandbox.enable = false;  #< it executes the command it's given
-
     wdisplays.sandbox.method = "bwrap";
     wdisplays.sandbox.whitelistWayland = true;
 
@@ -1176,25 +900,20 @@ in
     wget.sandbox.net = "all";
     wget.sandbox.whitelistPwd = true;  # saves to pwd by default
 
-    whalebird.buildCost = 1;
     whalebird.persist.byStore.private = [ ".config/Whalebird" ];
 
     # `wg`, `wg-quick`
     wireguard-tools.sandbox.method = "landlock";
-    wireguard-tools.sandbox.net = "all";
     wireguard-tools.sandbox.capabilities = [ "net_admin" ];
 
     # provides `iwconfig`, `iwlist`, `iwpriv`, ...
     wirelesstools.sandbox.method = "landlock";
-    wirelesstools.sandbox.net = "all";
     wirelesstools.sandbox.capabilities = [ "net_admin" ];
 
     wl-clipboard.sandbox.method = "bwrap";
     wl-clipboard.sandbox.whitelistWayland = true;
 
     wtype = {};
-    wtype.sandbox.method = "bwrap";
-    wtype.sandbox.whitelistWayland = true;
 
     xwayland.sandbox.method = "bwrap";
     xwayland.sandbox.wrapperType = "inplace";  #< consumers use it as a library (e.g. wlroots)
@@ -1209,45 +928,11 @@ in
     yt-dlp.sandbox.method = "bwrap";  # TODO:sandbox: untested
     yt-dlp.sandbox.net = "all";
     yt-dlp.sandbox.whitelistPwd = true;  # saves to pwd by default
+
+    zfs = {};
   };
 
-  sane.persist.sys.byStore.plaintext = lib.mkIf config.sane.programs.guiApps.enabled [
-    # "/var/lib/alsa"                # preserve output levels, default devices
-    { path = "/var/lib/systemd/backlight"; method = "bind"; }   # backlight brightness; bind because systemd T_T
-  ];
-
-  systemd.services."systemd-backlight@" = lib.mkIf config.sane.programs.guiApps.enabled {
-    after = [
-      "ensure-var-lib-systemd-backlight.service"
-    ];
-    wants = [
-      "ensure-var-lib-systemd-backlight.service"
-    ];
-  };
-
-  hardware.graphics = lib.mkIf config.sane.programs.guiApps.enabled ({
+  programs.feedbackd = lib.mkIf config.sane.programs.feedbackd.enabled {
     enable = true;
-  } // (lib.optionalAttrs pkgs.stdenv.isx86_64 {
-    # for 32 bit applications
-    # upstream nixpkgs forbids setting enable32Bit unless specifically x86_64 (so aarch64 isn't allowed)
-    enable32Bit = lib.mkDefault true;
-  }));
-
-  system.activationScripts.notifyActive = lib.mkIf config.sane.programs.guiApps.enabled {
-    text = lib.concatStringsSep "\n" ([
-        ''
-          tryNotifyUser() {
-            local user="$1"
-            local new_path="$PATH:${pkgs.sudo}/bin:${pkgs.libnotify}/bin"
-            local version="$(cat $systemConfig/nixos-version)"
-            PATH="$new_path" sudo -u "$user" \
-              env PATH="$new_path" NIXOS_VERSION="$version" /bin/sh -c \
-              '. $HOME/.profile; dbus_file="$XDG_RUNTIME_DIR/bus"; if [ -z "$DBUS_SESSION_BUS_ADDRESS" ] && [ -e "$dbus_file" ]; then export DBUS_SESSION_BUS_ADDRESS="unix:path=$dbus_file"; fi ; if [ -n "$DBUS_SESSION_BUS_ADDRESS" ]; then notify-send "nixos activated" "version: $NIXOS_VERSION" ; fi'
-          }
-        ''
-      ] ++ lib.mapAttrsToList
-        (user: en: lib.optionalString en "tryNotifyUser ${user}")
-        config.sane.programs.guiApps.enableFor.user
-    );
   };
 }

hosts/common/programs/audacity.nix

@@ -14,12 +14,10 @@
       };
     };
 
-    buildCost = 1;
-
     sandbox.method = "bwrap";
     sandbox.whitelistAudio = true;
     sandbox.whitelistWayland = true;
-    sandbox.autodetectCliPaths = "existingFile";
+    sandbox.autodetectCliPaths = true;
     sandbox.extraHomePaths = [
       # support media imports via file->open dir to some common media directories
       "tmp"

hosts/common/programs/ausyscall.nix

@@ -1,10 +0,0 @@
-# `ausyscall --dump`: lists all syscalls by number and name
-{ pkgs, ... }:
-{
-  sane.programs.ausyscall = {
-    packageUnwrapped = pkgs.linkBinIntoOwnPackage pkgs.audit "ausyscall";
-
-    sandbox.method = "landlock";
-  };
-}
-

hosts/common/programs/avahi.nix

@@ -1,43 +0,0 @@
-# Avahi zeroconf (mDNS) implementation.
-# runs as systemd `avahi-daemon.service`
-#
-# - <https://avahi.org/>
-# - code: <https://github.com/avahi/avahi>
-# - IRC: #avahi on irc.libera.chat
-#
-# - `avahi-browse --help` for usage
-# - `man avahi-daemon.conf`
-# - `LD_LIBRARY_PATH=/nix/store/ngwj3jqmxh8k4qji2z0lj7y1f8vzqrn2-nss-mdns-0.15.1/lib getent hosts desko.local`
-#   nss-mdns goes through avahi-daemon, so there IS caching here
-#
-{ config, lib, ... }:
-{
-  sane.programs.avahi = {
-    sandbox.method = "bwrap";
-    sandbox.whitelistDbus = [ "system" ];
-    sandbox.net = "all";  #< otherwise it will show 'null' in place of each interface name.
-    sandbox.extraPaths = [
-      "/"  #< else the daemon exits immediately. TODO: decrease this scope.
-    ];
-  };
-  services.avahi = lib.mkIf config.sane.programs.avahi.enabled {
-    enable = true;
-    package = config.sane.programs.avahi.package;
-    publish.enable = true;
-    publish.userServices = true;
-    nssmdns4 = true;
-    nssmdns6 = true;
-    # reflector = true;
-    allowInterfaces = [
-      # particularly, the default config disallows loopback, which is kinda fucking retarded, right?
-      "ens1"  #< servo
-      "enp5s0"  #< desko
-      "eth0"
-      "lo"
-      "wg-home"
-      "wlan0"  #< moby
-      "wlp3s0"  #< lappy
-      "wlp4s0"  #< desko
-    ];
-  };
-}

hosts/common/programs/bemenu.nix

@@ -87,7 +87,7 @@ let
 in
 {
   sane.programs.bemenu = {
-    sandbox.method = "bwrap";  # landlock works, but requires *all* of $XDG_RUNTIME_DIR to be granted.
+    sandbox.method = "bwrap";  # landlock works, but requires *all* of /run/user/$ID to be granted.
     sandbox.whitelistWayland = true;
     sandbox.extraHomePaths = [
       ".cache/fontconfig"  #< else it complains, and is *way* slower
@@ -95,7 +95,7 @@ in
 
     packageUnwrapped = pkgs.bemenu.overrideAttrs (upstream: {
       nativeBuildInputs = (upstream.nativeBuildInputs or []) ++ [
-        pkgs.makeBinaryWrapper
+        pkgs.makeWrapper
       ];
       # can alternatively be specified as CLI flags
       postInstall = (upstream.postInstall or "") + ''

hosts/common/programs/bitcoin-cli.nix

@@ -1,13 +0,0 @@
-{ pkgs, ... }:
-{
-  sane.programs.bitcoin-cli = {
-    packageUnwrapped = pkgs.linkBinIntoOwnPackage pkgs.bitcoind "bitcoin-cli";
-    sandbox.method = "bwrap";
-    sandbox.autodetectCliPaths = "existing";  #< for `bitcoin-cli -datadir=/var/lib/...`
-    sandbox.extraHomePaths = [
-      ".bitcoin/bitcoin.conf"
-    ];
-    sandbox.net = "all";  # actually needs only localhost
-    secrets.".bitcoin/bitcoin.conf" = ../../../secrets/servo/bitcoin.conf.bin;
-  };
-}

hosts/common/programs/blast-ugjka/blast-to-default

@@ -1,8 +1,11 @@
 #!/usr/bin/env nix-shell
-#!nix-shell -i python3 -p blast-ugjka -p python3
+#!nix-shell -i python3 -p "python3.withPackages (ps: [  ])" -p blast-ugjka
 # vim: set filetype=python :
 
+import ctypes
 import logging
+import os
+import signal
 import socket
 import subprocess
 
@@ -13,9 +16,53 @@ logger = logging.getLogger(__name__)
 # map from known devices -> required flags
 DEVICE_MAP = {
   "Theater TV": [],
-  "Cuddlevision": [ "-usewav" ],
+  "[LG] webOS TV OLED55C9PUA": [ "-usewav" ],
 }
 
+def set_pdeathsig(sig=signal.SIGTERM):
+    """
+    helper function to ensure once parent process exits, its children processes will automatically die.
+    see: <https://stackoverflow.com/a/43152455>
+    see: <https://www.man7.org/linux/man-pages/man2/prctl.2.html>
+    """
+    libc = ctypes.CDLL("libc.so.6")
+    return libc.prctl(1, sig)
+
+MY_PID = None
+
+def reap_children(sig=None, frame=None):
+    global MY_PID
+    # reset SIGTERM handler to avoid recursing
+    signal.signal(signal.SIGTERM, signal.Handlers.SIG_DFL)
+    logger.info("killing all children (of pid %d)", MY_PID)
+    os.killpg(MY_PID, signal.SIGTERM)
+
+def reap_on_exit():
+    """
+    catch when the parent exits, and map that to SIGTERM for this process.
+    when this process receives SIGTERM, also terminate all descendent processes.
+
+    this is done because:
+    1. mpv invokes this, but (potentially) via the sandbox wrapper.
+    2. when mpv exits, it `SIGKILL`s that sandbox wrapper.
+    3. bwrap does not pass SIGKILL or SIGTERM to its child.
+    4. hence, we neither receive that signal NOR can we pass it on simply by killing our immediate children
+       (since any bwrap'd children wouldn't pass that signal on...)
+    really, the proper fix would be on mpv's side:
+    - mpv should create a new process group when it launches a command, and kill that process group on exit.
+    or fix this in the sandbox wrapper:
+    - why *doesn't* bwrap forward the signals?
+    - there's --die-with-parent, but i can't apply that *system wide* and expect reasonably behavior
+      <https://github.com/containers/bubblewrap/issues/529>
+    """
+    global MY_PID
+    MY_PID = os.getpid()
+    # create a new process group, pgid = gid
+    os.setpgid(MY_PID, MY_PID)
+
+    set_pdeathsig(signal.SIGTERM)
+    signal.signal(signal.SIGTERM, reap_children)
+
 def get_ranked_ip_addrs():
     """
     return the IP addresses most likely to be LAN addresses
@@ -49,6 +96,8 @@ class BlastDriver:
             stdin=subprocess.PIPE,
             stdout=subprocess.PIPE,
             stderr=subprocess.PIPE,
+            # this pdeathsig isn't necessary; seems it might result in leaked pulse outputs
+            # preexec_fn=set_pdeathsig
         )
         self.blast_flags = list(blast_flags)
         self.receiver_names = []
@@ -153,11 +202,15 @@ def main():
     logging.basicConfig()
     logging.getLogger().setLevel(logging.DEBUG)
 
+    reap_on_exit()
+
     blast = try_blast()
 
     if blast is not None:
         logger.info("waiting until blast exits")
         blast.blast.wait()
 
+    reap_children()
+
 if __name__ == "__main__":
     main()

hosts/common/programs/blast-ugjka/default.nix

@@ -31,7 +31,7 @@ in
 
   sane.programs.blast-to-default = {
     # helper to deal with blast's interactive CLI
-    packageUnwrapped = pkgs.static-nix-shell.mkPython3 {
+    packageUnwrapped = pkgs.static-nix-shell.mkPython3Bin {
       pname = "blast-to-default";
       pkgs = [ "blast-ugjka" ];
       srcRoot = ./.;
@@ -39,10 +39,12 @@ in
     sandbox.method = "bwrap";
     sandbox.whitelistAudio = true;
     sandbox.net = "clearnet";
-    #v  else it fails to reap its children (or, maybe, it fails to hook its parent's death signal?)
-    #v  might be possible to remove this, but kinda hard to see a clean way.
-    sandbox.isolatePids = false;
-    suggestedPrograms = [ "blast-ugjka" "sane-die-with-parent" ];
+    sandbox.extraConfig = [
+      # else it fails to reap its children (or, maybe, it fails to hook its parent's death signal?)
+      # might be possible to remove this, but kinda hard to see a clean way.
+      "--sane-sandbox-keep-namespace" "pid"
+    ];
+    suggestedPrograms = [ "blast-ugjka" ];
   };
 
   networking.firewall.allowedTCPPorts = lib.mkIf cfg.enabled [ 9000 ];