nixpkgs: 0-unstable-2025-02-08 -> 0-unstable-2025-02-17

servo is really running out of space

README.md

@@ -17,24 +17,27 @@ the only hard dependency for my exported pkgs/modules should be [nixpkgs][nixpkg
 building [hosts/](./hosts/) will require [sops][sops].
 
 you might specifically be interested in these files (elaborated further in #key-points-of-interest):
-- ~~[`sxmo-utils`](./pkgs/additional/sxmo-utils/default.nix)~~
-  - these files will remain until my config settles down, but i no longer use or maintain SXMO.
+- [my packages](./pkgs/by-name)
 - [my implementation of impermanence](./modules/persist/default.nix)
 - my way of deploying dotfiles/configuring programs per-user:
   - [modules/fs/](./modules/fs/default.nix)
   - [modules/programs/](./modules/programs/default.nix)
   - [modules/users/](./modules/users/default.nix)
 
+if you find anything here genuinely useful, message me so that i can work to upstream it!
+
 [nixpkgs]: https://github.com/NixOS/nixpkgs
 [sops]: https://github.com/Mic92/sops-nix
 [uninsane-org]: https://uninsane.org
 
+
 ## Using This Repo In Your Own Config
 
 follow the instructions [here][NUR] to access my packages through the Nix User Repositories.
 
 [NUR]: https://nur.nix-community.org/
 
+
 ## Layout
 - `doc/`
   - instructions for tasks i find myself doing semi-occasionally in this repo.
@@ -52,7 +55,7 @@ follow the instructions [here][NUR] to access my packages through the Nix User R
 - `pkgs/`
   - derivations for things not yet packaged in nixpkgs.
   - derivations for things from nixpkgs which i need to `override` for some reason.
-  - inline code for wholly custom packages (e.g. `pkgs/additional/sane-scripts/` for CLI tools
+  - inline code for wholly custom packages (e.g. `pkgs/by-name/sane-scripts/` for CLI tools
     that are highly specific to my setup).
 - `scripts/`
   - scripts which aren't reachable on a deployed system, but may aid manual deployments.
@@ -79,44 +82,40 @@ i.e. you might find value in using these in your own config:
     - populated with some statically-defined data
     - populated according to some script
     - created as a dependency of some service (e.g. `nginx`)
-  - values defined here are applied neither at evaluation time _nor_ at activation time.
-    - rather, they become systemd services.
-    - systemd manages dependencies
-    - e.g. link `/var/www -> /mnt/my-drive/www` only _after_ `/mnt/my-drive/www` appears)
   - this is akin to using [Home Manager's][home-manager] file API -- the part which lets you
     statically define `~/.config` files -- just with a different philosophy.
+    namely, it avoids any custom activation scripts by leveraging `systemd-tmpfiles`.
 - `modules/persist/`
-  - my alternative to the Impermanence module.
-  - this builds atop `modules/fs/` to achieve things stock impermanence can't:
-    - persist things to encrypted storage which is unlocked at login time (pam_mount).
+  - my implementation of impermanence, built atop the above `fs` module, with a few notable features:
+    - no custom activation scripts or services (uses `systemd-tmpfiles` and `.mount` units)
     - "persist" cache directories -- to free up RAM -- but auto-wipe them on mount
       and encrypt them to ephemeral keys so they're unreadable post shutdown/unmount.
+    - persist to encrypted storage which is unlocked at login time.
 - `modules/programs/`
   - like nixpkgs' `programs` options, but allows both system-wide or per-user deployment.
   - allows `fs` and `persist` config values to be gated behind program deployment:
     - e.g. `/home/<user>/.mozilla/firefox` is persisted only for users who
       `sane.programs.firefox.enableFor.user."<user>" = true;`
   - allows aggressive sandboxing any program:
-    - `sane.programs.firefox.sandbox.method = "bwrap";  # sandbox with bubblewrap`
+    - `sane.programs.firefox.sandbox.enable = true;  # wraps the program so that it isolates itself into a new namespace when invoked`
     - `sane.programs.firefox.sandbox.whitelistWayland = true;  # allow it to render a wayland window`
     - `sane.programs.firefox.sandbox.extraHomePaths = [ "Downloads" ];  # allow it read/write access to ~/Downloads`
     - integrated with `fs` and `persist` modules so that programs' config files and persisted data stores are linked into the sandbox w/o any extra involvement.
 - `modules/users/`
   - convenience layer atop the above modules so that you can just write
     `fs.".config/git"` instead of `fs."/home/colin/.config/git"`
-  - per-user services managed by [s6-rc](https://www.skarnet.org/software/s6-rc/)
-
-some things in here could easily find broader use. if you would find benefit in
-them being factored out of my config, message me and we could work to make that happen.
+  - simplified `systemd.services` API
 
 [home-manager]: https://github.com/nix-community/home-manager
 
+
 ## Mirrors
 
 this repo exists in a few known locations:
 - primary: <https://git.uninsane.org/colin/nix-files>
 - mirror: <https://github.com/nix-community/nur-combined/tree/master/repos/colinsane>
 
+
 ## Contact
 
 if you want to contact me for questions, or collaborate to split something useful into a shared repo, etc,

TODO.md

@@ -1,34 +1,15 @@
 ## BUGS
+- alacritty Ctrl+N frequently fails to `cd` to the previous directory
+- bunpen dbus sandboxing can't be *nested* (likely a problem in xdg-dbus-proxy)
+- dissent has a memory leak (3G+ after 24hr)
+  - set a max memory use in the systemd service, to force it to restart as it leaks?
 - `rmDbusServices` may break sandboxing
   - e.g. if the package ships a systemd unit which references $out, then make-sandboxed won't properly update that unit.
   - `rmDbusServicesInPlace` is not affected
-- when moby wlan is explicitly set down (via ip link set wlan0 down), /var/lib/hickory-dns/dhcp-configs doesn't get reset
-  - `ip monitor` can detect those manual link state changes (NM-dispatcher it seems cannot)
-  - or try dnsmasq?
-- hickory-dns can't resolve `abs.twimg.com`
-- hickory-dns can't resolve `social.kernel.org`
-- hickory-dns can't resolve `pe.usps.com`
-- hickory-dns can't resolve `social.seattle.wa.us`
-- hickory-dns can't resolve `support.mozilla.org`
-- sandbox: link cache means that if i update ~/.config/... files inline, sandboxed programs still see the old version
-- mpv: continues to play past the end of some audio files
 - mpv: audiocast has mpv sending its output to the builtin speakers unless manually changed
-- `ssh` access doesn't grant same linux capabilities as login
 - syshud (volume overlay): when casting with `blast`, syshud doesn't react to volume changes
-- moby: after bringing the modem up, powering it down loses *complete* net connectivity (i.e. wlan is gone as well)
 - dissent: if i launch it without net connectivity, it gets stuck at the login, and never tries again
-- calls: seems that it starts before net access, and then is forever disconnected (until i manually restart it)
-- moby: kaslr is effectively disabled
-  - `dmesg | grep "KASLR disabled due to lack of seed"`
-  - fix by adding `kaslrseed` to uboot script before `booti`
-    - <https://github.com/armbian/build/pull/4352>
-    - not sure how that's supposed to work with tow-boot; maybe i should just update tow-boot
-- moby: bpf is effectively disabled?
-  - `dmesg | grep 'systemd[1]: bpf-lsm: Failed to load BPF object: No such process'`
-  - `dmesg | grep 'hid_bpf: error while preloading HID BPF dispatcher: -22'`
-- `s6` is not re-entrant
-  - so if the desktop crashes, the login process from `unl0kr` fails to re-launch the GUI
-- newflash on moby can't play videos
+- newsflash on moby can't play videos
   - "open in browser" works though -- in mpv
 - gnome-maps can't use geoclue *and* openstreetmap at the same time
   - get gnome-maps to speak xdg-desktop-portal, and this will be fixed
@@ -36,8 +17,12 @@
   - see under "preferences", cookies are disabled
   - prevents logging into websites (OpenStreetMap)
   - works when sandbox is disabled
+- rsync to ssh target fails because of restrictive sandboxing
+- `/mnt/.servo_ftp` retries every 10s, endlessly, rather than doing a linear backoff
+  - repro by `systemctl stop sftpgo` on servo, then watching `mnt-.servo_ftp.{mount,timer}` on desko
 
 ## REFACTORING:
+- fold hosts/modules/ into toplevel modules/
 - add import checks to my Python nix-shell scripts
 - consolidate ~/dev and ~/ref
   - ~/dev becomes a link to ~/ref/cat/mine
@@ -45,27 +30,24 @@
 - don't hardcode IP addresses so much in servo
 
 ### sops/secrets
-- rework secrets to leverage `sane.fs`
-- remove sops activation script as it's covered by my systemd sane.fs impl
 - user secrets could just use `gocryptfs`, like with ~/private?
   - can gocryptfs support nested filesystems, each with different perms (for desko, moby, etc)?
 
-### roles
-- allow any host to take the role of `uninsane.org`
-  - will make it easier to test new services?
-
 ### upstreaming
-- add updateScripts to all my packages in nixpkgs
+- upstream blueprint-compiler cross fixes -> nixpkgs
+- upstream cargo cross fixes -> nixpkgs
+- upstream `gps-share` package -> nixpkgs
 
 #### upstreaming to non-nixpkgs repos
+- gnome-calls: retry net connection when DNS is down
 - gtk: build schemas even on cross compilation: <https://github.com/NixOS/nixpkgs/pull/247844>
+- linux: upstream PinePhonePro device trees
+- nwg-panel: configurable media controls
+- nwg-panel / playerctl hang fix (i think nwg-panel is what should be patched here)
 
 
 ## IMPROVEMENTS:
-- kernels: ship the same kernel on every machine
-  - then i can tune the kernels for hardening, without duplicating that work 4 times
-- zfs: replace this with something which doesn't require a custom kernel build
-- mpv: add media looping controls (e.g. loop song, loop playlist)
+- sane-deadlines: show day of the week for upcoming items
 - curlftpfs: replace with something better
   - safer (rust? actively maintained? sandboxable?)
   - handles spaces/symbols in filenames
@@ -75,45 +57,60 @@
   - matrix room links *just work*.
   - `network.protocol-handler.external.https = true` in about:config *seems* to do this,
     but breaks some webpages (e.g. Pleroma)
+- associate http(s)://*.pdf with my pdf handler
+  - can't do that because lots of applications don't handle URIs
+  - could workaround using a wrapper that downloads the file and then passes it to the program
+- geary: replace with envelope
+  - likely requires updating envelope to a more recent version (for multi-accounting), and therefore updating libadwaita...
 
 ### security/resilience
 - enable `snapper` btrfs snapshots (`services.snapper`)
 - /mnt/desko/home, etc, shouldn't include secrets (~/private)
   - 95% of its use is for remote media access and stuff which isn't in VCS (~/records)
+- harden systemd services:
+  - servo: `coturn.service`
+  - servo: `postgresql.service`
+  - servo: `postfix.service`
+  - servo: `prosody.service`
+  - servo: `slskd.service`
+  - desko: `usbmuxd.service`
+  - servo: `backup-torrents.service`
+  - servo: `dedupe-media.service`
+  - remove SGID /run/wrappers/bin/sendmail, and just add senders to `postdrop` group
 - port all sane.programs to be sandboxed
   - sandbox `nix`
   - enforce that all `environment.packages` has a sandbox profile (or explicitly opts out)
-  - revisit "non-sandboxable" apps and check that i'm not actually just missing mountpoints
-    - LL_FS_RW=/ isn't enough -- need all mount points like `=/:/proc:/sys:...`.
-  - ensure non-bin package outputs are linked for sandboxed apps
-    - i.e. `outputs.man`, `outputs.debug`, `outputs.doc`, ...
-  - lock down dbus calls within the sandbox
-    - otherwise anyone can `systemd-run --user ...` to potentially escape a sandbox
-    - <https://github.com/flatpak/xdg-dbus-proxy>
-  - port sanebox to a compiled language (hare?)
-    - it adds like 50-70ms launch time _on my laptop_. i'd hate to know how much that is on the pinephone.
+  - enforce granular dbus sandboxing (bunpen-dbus-*)
+- make gnome-keyring-daemon less monolithic
+  - no reason every application with _a_ secret needs to see _all_ secrets
+  - check out oo7-daemon?
+  - also unix-pass based provider: <https://github.com/mdellweg/pass_secret_service>
 - make dconf stuff less monolithic
   - i.e. per-app dconf profiles for those which need it. possible static config.
   - flatpak/spectrum has some stuff to proxy dconf per-app
 
 ### user experience
+- setup a real calendar system, for recurring events
 - rofi: sort items case-insensitively
+- rofi: enable mouse mode?
+- mpv: add media looping controls (e.g. loop song, loop playlist)
+- mpv: add/implement an extension to search youtube
+  - apparently `yt-dlp` does searching!
 - replace starship prompt with something more efficient
   - watch `forkstat`: it does way too much
-- cleanup waybar/nwg-panel so that it's not invoking playerctl every 2 seconds
+- cleanup nwg-panel so that it's not invoking swaync every second
   - nwg-panel: doesn't know that virtual-desktop 10/TV exists
 - install apps:
   - display QR codes for WiFi endpoints: <https://linuxphoneapps.org/apps/noappid.wisperwind.wifi2qr/>
   - shopping list (not in nixpkgs): <https://linuxphoneapps.org/apps/ro.hume.cosmin.shoppinglist/>
   - offline Wikipedia (or, add to `wike`)
-  - offline docs viewer (gtk): <https://github.com/workbenchdev/Biblioteca>
   - some type of games manager/launcher
     - Gnome Highscore (retro games)?: <https://gitlab.gnome.org/World/highscore>
-  - better maps for mobile (Osmin (QtQuick)? Pure Maps (Qt/Kirigami)?)
   - note-taking app: <https://linuxphoneapps.org/categories/note-taking/>
     - Folio is nice, uses standard markdown, though it only supports flat repos
   - OSK overlay specifically for mobile gaming
     - i.e. mock joysticks, for use with SuperTux and SuperTuxKart
+  - game: Hedgewars
 - install mobile-friendly games:
   - Shattered Pixel Dungeon (nixpkgs `shattered-pixel-dungeon`; doesn't cross-compile b/c openjdk/libIDL) <https://github.com/ebolalex/shattered-pixel-dungeon>
   - UnCiv (Civ V clone; nixpkgs `unciv`; doesn't cross-compile):  <https://github.com/yairm210/UnCiv>
@@ -124,56 +121,48 @@
   - blurble (https://linuxphoneapps.org/games/app.drey.blurble/). nix: not as of 2024-02-05
   - Trivia Quiz (https://linuxphoneapps.org/games/io.github.nokse22.trivia-quiz/)
 - sane-sync-music: remove empty dirs
+- soulseek: install a CLI app usable over ssh
 
 #### moby
+- moby: port battery support to something upstreamable
+- moby: install transito/mobroute public transit app: <https://sr.ht/~mil/mobroute/> <https://git.sr.ht/~mil/transito>
+  - see: <https://github.com/NixOS/nixpkgs/pull/335613>
+- moby: consider honeybee instead of gnome-calls for calling? <https://git.sr.ht/~anjan/honeybee>
+  - uses XMPP, so more NAT/WoWLAN-friendly
 - fix cpuidle (gets better power consumption): <https://xnux.eu/log/077.html>
 - fix cpupower for better power/perf
   - `journalctl -u cpupower --boot` (problem is present on lappy, at least)
+- use dynamic DRAM clocking to reduce power by 0.5W: <https://xnux.eu/log/083.html>
+  - coreboot implements DRAM training for rk3399: <https://gitlab.com/vicencb/kevinboot/-/blob/master/cb/sdram.c>
 - moby: tune keyboard layout
-- SwayNC: add option to change audio output
-- moby: tune GPS
-  - fix iio-sensor-proxy magnetometer scaling
-  - tune QGPS setting in eg25-control, for less jitter?
-  - configure geoclue to do some smoothing?
-  - manually do smoothing, as some layer between mepo and geoclue?
-  - email wigle.net people to unlock API access
-- moby: port `freshen-agps` timer service to s6 (maybe i want some `s6-cron` or something)
-- moby: improve gPodder launch time
-- moby: theme GTK apps (i.e. non-adwaita styles)
-  - especially, make the menubar collapsible
-  - try Gradience tool specifically for theming adwaita? <https://linuxphoneapps.org/apps/com.github.gradienceteam.gradience/>
+- SwayNC/nwg-panel: add option to change audio output
+- Newsflash: sync OPML on start, same way i do with gpodder
+- better podcasting client?
+- hardware upgrade (OnePlus)?
 
 #### non-moby
 - RSS: integrate a paywall bypass
   - e.g. self-hosted [ladder](https://github.com/everywall/ladder) (like 12ft.io)
 - RSS: have podcasts get downloaded straight into ~/Videos/...
   - and strip the ads out using Whisper transcription + asking a LLM where the ad breaks are
-- neovim: set up language server (lsp; rnix-lsp; nvim-lspconfig)
-- neovim: integrate LLMs
-- Helix: make copy-to-system clipboard be the default
-- firefox/librewolf: persist history
+- neovim: integrate ollama
+- neovim: better docsets (e.g. c++, glib)
+- firefox: persist history
   - just not cookies or tabs
-- package Nix/NixOS docs for Zeal
-  - install [doc-browser](https://github.com/qwfy/doc-browser)
-  - this supports both dash (zeal) *and* the datasets from <https://devdocs.io> (which includes nix!)
-  - install [devhelp](https://wiki.gnome.org/Apps/Devhelp)  (gnome)
 - have xdg-open parse `<repo:...> URIs (or adjust them so that it _can_ parse)
 - sane-bt-search: show details like 5.1 vs stereo, h264 vs h265
   - maybe just color these "keywords" in all search results?
 - transmission: apply `sane-tag-media` path fix in `torrent-done` script
   - many .mkv files do appear to be tagged: i'd just need to add support in my own tooling
+  - more aggressively cleanup non-media files after DL (ripper logos, info txts)
 - uninsane.org: make URLs relative to allow local use (and as offline homepage)
 - email: fix so that local mail doesn't go to junk
   - git sendmail flow adds the DKIM signatures, but gets delivered locally w/o having the sig checked, so goes into Junk
   - could change junk filter from "no DKIM success" to explicit "DKIM failed"
   - add an auto-reply address (e.g. `reply-test@uninsane.org`) which reflects all incoming mail; use this (or a friend running this) for liveness checks
 
-### perf
-- add `pkgs.impure-cached.<foo>` package set to build things with ccache enabled
-  - every package here can be auto-generated, and marked with some env var so that it doesn't pollute the pure package set
-  - would be super handy for package prototyping!
-
 ## NEW FEATURES:
+- migrate Kodi box to nix
 - migrate MAME cabinet to nix
   - boot it from PXE from servo?
 - enable IPv6

default.nix

@@ -1,5 +1,5 @@
 { ... }@args:
 let
-  sane-nix-files = import ./pkgs/additional/sane-nix-files { };
+  sane-nix-files = import ./pkgs/by-name/sane-nix-files/package.nix { };
 in
   import "${sane-nix-files}/impure.nix" args

hosts/by-name/crappy/default.nix

@@ -30,16 +30,4 @@
   # sane.programs.guiApps.enableFor.user.colin = false;
 
   # sane.programs.pcGuiApps.enableFor.user.colin = false;  #< errors!
-
-  sane.programs.blueberry.enableFor.user.colin = false;  # bluetooth manager: doesn't cross compile!
-  # sane.programs.brave.enableFor.user.colin = false;  # 2024/06/03: fails eval if enabled on cross
-  # sane.programs.firefox.enableFor.user.colin = false;  # 2024/06/03: this triggers an eval error in yarn stuff -- i'm doing IFD somewhere!!?
-  sane.programs.mepo.enableFor.user.colin = false;  # 2024/06/04: doesn't cross compile (nodejs)
-  sane.programs.mercurial.enableFor.user.colin = false;  # 2024/06/03: does not cross compile
-  sane.programs.nixpkgs-review.enableFor.user.colin = false;  # 2024/06/03: OOMs when cross compiling
-  sane.programs.ntfy-sh.enableFor.user.colin = false;  # 2024/06/04: doesn't cross compile (nodejs)
-  sane.programs.pwvucontrol.enableFor.user.colin = false;  # 2024/06/03: doesn't cross compile (libspa-sys)
-  sane.programs."sane-scripts.bt-search".enableFor.user.colin = false;  # 2024/06/03: does not cross compile
-  sane.programs.sequoia.enableFor.user.colin = false;  # 2024/06/03: does not cross compile
-  sane.programs.zathura.enableFor.user.colin = false;  # 2024/06/03: does not cross compile
 }

hosts/by-name/desko/default.nix

@@ -1,10 +1,9 @@
-{ config, lib, pkgs, ... }:
+{ config, pkgs, ... }:
 {
   imports = [
     ./fs.nix
   ];
 
-  sane.services.hickory-dns.asSystemResolver = false;  # TEMPORARY: TODO: re-enable hickory-dns
   # sane.programs.devPkgs.enableFor.user.colin = true;
   # sane.guest.enable = true;
 
@@ -22,8 +21,8 @@
 
   sane.roles.build-machine.enable = true;
   sane.roles.client = true;
-  sane.roles.dev-machine = true;
   sane.roles.pc = true;
+  sane.services.ollama.enable = true;
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."desko".wg-home.ip;
   sane.ovpn.addrV4 = "172.26.55.21";
@@ -32,38 +31,26 @@
 
   sane.nixcache.remote-builders.desko = false;
 
+  sane.programs.firefox.config.formFactor = "desktop";
+
   sane.programs.sane-private-unlock-remote.enableFor.user.colin = true;
   sane.programs.sane-private-unlock-remote.config.hosts = [ "servo" ];
 
   sane.programs.sway.enableFor.user.colin = true;
-  sane.programs.iphoneUtils.enableFor.user.colin = true;
   sane.programs.steam.enableFor.user.colin = true;
 
-  sane.programs.geary.config.autostart = true;
-  sane.programs.signal-desktop.config.autostart = true;
-
   sane.programs.nwg-panel.config = {
     battery = false;
     brightness = false;
   };
 
+  sane.programs.mpv.config.defaultProfile = "high-quality";
+
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
 
   # needed to use libimobiledevice/ifuse, for iphone sync
   services.usbmuxd.enable = true;
 
-  # TODO: enable snapper (need to make `/nix` or `/nix/persist` a subvolume, somehow).
-  # default config: https://man.archlinux.org/man/snapper-configs.5
-  # defaults to something like:
-  #   - hourly snapshots
-  #   - auto cleanup; keep the last 10 hourlies, last 10 daylies, last 10 monthlys.
-  # to list snapshots: `sudo snapper --config nix list`
-  # to take a snapshot: `sudo snapper --config nix create`
-  # services.snapper.configs.nix = {
-  #   # TODO: for the impermanent setup, we'd prefer to just do /nix/persist,
-  #   # but that also requires setting up the persist dir as a subvol
-  #   SUBVOLUME = "/nix";
-  #   # TODO: ALLOW_USERS doesn't seem to work. still need `sudo snapper -c nix list`
-  #   ALLOW_USERS = [ "colin" ];
-  # };
+  # TODO(2025-01-01): re-enable once rocm build is fixed: <https://github.com/NixOS/nixpkgs/pull/367695>
+  # hardware.amdgpu.opencl.enable = true;  # desktop (AMD's opencl implementation AKA "ROCM"); probably required for ollama
 }

hosts/by-name/desko/fs.nix

@@ -3,7 +3,7 @@
 {
   # increase /tmp space (defaults to 50% of RAM) for building large nix things.
   # a cross-compiled kernel, particularly, will easily use 30+GB of tmp
-  fileSystems."/tmp".options = [ "size=64G" ];
+  fileSystems."/tmp".options = [ "size=128G" ];
 
   fileSystems."/nix" = {
     device = "/dev/disk/by-uuid/845d85bf-761d-431b-a406-e6f20909154f";

hosts/by-name/lappy/default.nix

@@ -5,7 +5,6 @@
   ];
 
   sane.roles.client = true;
-  sane.roles.dev-machine = true;
   sane.roles.pc = true;
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."lappy".wg-home.ip;
@@ -18,16 +17,19 @@
   sane.programs.sane-private-unlock-remote.enableFor.user.colin = true;
   sane.programs.sane-private-unlock-remote.config.hosts = [ "servo" ];
 
-  sane.programs.stepmania.enableFor.user.colin = true;
+  sane.programs.firefox.config.formFactor = "laptop";
+  sane.programs.itgmania.enableFor.user.colin = true;
+  # sane.programs.stepmania.enableFor.user.colin = true;  #< TODO: fix build
   sane.programs.sway.enableFor.user.colin = true;
 
-  sane.programs.geary.config.autostart = true;
-  sane.programs.signal-desktop.config.autostart = true;
-
   sops.secrets.colin-passwd.neededForUsers = true;
 
   sane.services.rsync-net.enable = true;
 
+  # starting 2024/09, under default settings (apparently 256 quantum), audio would crackle under load.
+  # 1024 solves *most* crackles, but still noticable under heavier loads.
+  sane.programs.pipewire.config.min-quantum = 2048;
+
   # TODO: enable snapper (need to make `/nix` or `/nix/persist` a subvolume, somehow).
   # default config: https://man.archlinux.org/man/snapper-configs.5
   # defaults to something like:

hosts/by-name/moby/default.nix

@@ -6,13 +6,13 @@
 # - Mobian wiki: <https://wiki.mobian-project.org/doku.php?id=start>
 #   - recommended apps, chatrooms
 
-{ config, pkgs, lib, ... }:
+{ config, ... }:
 {
   imports = [
     ./fs.nix
   ];
 
-  sane.hal.pine64.enable = true;
+  sane.hal.pine64-pinephone-pro.enable = true;
   sane.roles.client = true;
   sane.roles.handheld = true;
   sane.services.wg-home.enable = true;
@@ -30,20 +30,16 @@
 
   sane.programs.sway.enableFor.user.colin = true;
   sane.programs.sway.config.mod = "Mod1";  #< alt key instead of Super
-  sane.programs.blueberry.enableFor.user.colin = false;  # bluetooth manager: doesn't cross compile!
-  sane.programs.fcitx5.enableFor.user.colin = false;  # does not cross compile
-  sane.programs.mercurial.enableFor.user.colin = false;  # does not cross compile
-  sane.programs.nvme-cli.enableFor.system = false;  # does not cross compile (libhugetlbfs)
 
   # enabled for easier debugging
   sane.programs.eg25-control.enableFor.user.colin = true;
   # sane.programs.rtl8723cs-wowlan.enableFor.user.colin = true;
+  # sane.programs.eg25-manager.enableFor.user.colin = true;
 
   # sane.programs.ntfy-sh.config.autostart = true;
   sane.programs.dino.config.autostart = true;
-  sane.programs.signal-desktop.config.autostart = true;
-  # sane.programs.geary.config.autostart = true;
-  # sane.programs.calls.config.autostart = true;
+  sane.programs.signal-desktop.config.autostart = false;
+  sane.programs.geary.config.autostart = false;
 
   sane.programs.pipewire.config = {
     # tune so Dino doesn't drop audio
@@ -61,7 +57,8 @@
     max-quantum = 8192;
   };
 
-  # /boot space is at a premium. default was 20.
-  # even 10 can be too much
-  boot.loader.generic-extlinux-compatible.configurationLimit = 8;
+  sane.programs.mpv.config.defaultProfile = "fast";
+
+  # /boot space is at a premium, especially with uncompressed kernels. default was 20.
+  boot.loader.generic-extlinux-compatible.configurationLimit = 10;
 }

hosts/by-name/servo/default.nix

@@ -3,8 +3,9 @@
 {
   imports = [
     ./fs.nix
-    ./net.nix
+    ./net
     ./services
+    ./users
   ];
 
   # for administering services
@@ -21,22 +22,14 @@
     "sane-scripts.stop-all-servo"
   ];
   sane.services.dyn-dns.enable = true;
-  sane.services.hickory-dns.asSystemResolver = false;  # TODO: enable once it's all working well
-  sane.services.wg-home.enable = true;
-  sane.services.wg-home.visibleToWan = true;
-  sane.services.wg-home.forwardToWan = true;
-  sane.services.wg-home.routeThroughServo = false;
-  sane.services.wg-home.ip = config.sane.hosts.by-name."servo".wg-home.ip;
-  sane.ovpn.addrV4 = "172.23.174.114";
-  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:8df3:14b0";
   sane.nixcache.remote-builders.desko = false;
   sane.nixcache.remote-builders.servo = false;
   sane.services.rsync-net.enable = true;
 
   # automatically log in at the virtual consoles.
   # using root here makes sure we always have an escape hatch.
-  # XXX(2024-07-27): this is incompatible with my s6-rc stuff, which needs to auto-login as `colin` to start its user services.
-  # services.getty.autologinUser = "root";
+  # XXX(2024-07-27): this is incompatible if using s6, which needs to auto-login as `colin` to start its user services.
+  services.getty.autologinUser = "root";
 
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
 

hosts/by-name/servo/fs.nix

@@ -1,60 +1,9 @@
-# zfs docs:
-# - <https://nixos.wiki/wiki/ZFS>
-# - <repo:nixos/nixpkgs:nixos/modules/tasks/filesystems/zfs.nix>
-#
-# zfs check health: `zpool status`
-#
-# zfs pool creation (requires `boot.supportedFilesystems = [ "zfs" ];`
-# - 1. identify disk IDs: `ls -l /dev/disk/by-id`
-# - 2. pool these disks: `zpool create -f -m legacy pool raidz ata-ST4000VN008-2DR166_WDH0VB45 ata-ST4000VN008-2DR166_WDH17616 ata-ST4000VN008-2DR166_WDH0VC8Q ata-ST4000VN008-2DR166_WDH17680`
-#   - legacy documented: <https://superuser.com/questions/790036/what-is-a-zfs-legacy-mount-point>
-# - 3. enable acl support: `zfs set acltype=posixacl pool`
-#
-# import pools: `zpool import pool`
-# show zfs datasets: `zfs list` (will be empty if haven't imported)
-# show zfs properties (e.g. compression): `zfs get all pool`
-# set zfs properties: `zfs set compression=on pool`
 { lib, pkgs, ... }:
 
 {
   # hostId: not used for anything except zfs guardrail?
   #   [hex(ord(x)) for x in 'serv']
-  networking.hostId = "73657276";
-  boot.supportedFilesystems = [ "zfs" ];
-  # boot.zfs.enabled = true;
-  boot.zfs.forceImportRoot = false;
-  # scrub all zfs pools weekly:
-  services.zfs.autoScrub.enable = true;
-  boot.extraModprobeConfig = ''
-    ### zfs_arc_max tunable:
-    # ZFS likes to use half the ram for its own cache and let the kernel push everything else to swap.
-    # so, reduce its cache size
-    # see: <https://askubuntu.com/a/1290387>
-    # see: <https://serverfault.com/a/1119083>
-    # see: <https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-arc-max>
-    # for all tunables, see: `man 4 zfs`
-    # to update these parameters without rebooting:
-    # - `echo '4294967296' | sane-sudo-redirect /sys/module/zfs/parameters/zfs_arc_max`
-    ### zfs_bclone_enabled tunable
-    # this allows `cp --reflink=always FOO BAR` to work. i.e. shallow copies.
-    # it's unstable as of 2.2.3. led to *actual* corruption in 2.2.1, but hopefully better by now.
-    # - <https://github.com/openzfs/zfs/issues/405>
-    # note that `du -h` won't *always* show the reduced size for reflink'd files (?).
-    # `zpool get all | grep clone` seems to be the way to *actually* see how much data is being deduped
-    options zfs zfs_arc_max=4294967296 zfs_bclone_enabled=1
-  '';
-  # to be able to mount the pool like this, make sure to tell zfs to NOT manage it itself.
-  # otherwise local-fs.target will FAIL and you will be dropped into a rescue shell.
-  # - `zfs set mountpoint=legacy pool`
-  # if done correctly, the pool can be mounted before this `fileSystems` entry is created:
-  # - `sudo mount -t zfs pool /mnt/persist/pool`
-  fileSystems."/mnt/pool" = {
-    device = "pool";
-    fsType = "zfs";
-    options = [ "acl" ];  #< not sure if this `acl` flag is actually necessary. it mounts without it.
-  };
-  # services.zfs.zed = ... # TODO: zfs can send me emails when disks fail
-  sane.programs.sysadminUtils.suggestedPrograms = [ "zfs-tools" ];
+  # networking.hostId = "73657276";
 
   sane.persist.stores."ext" = {
     origin = "/mnt/pool/persist";
@@ -80,19 +29,33 @@
     fsType = "vfat";
   };
 
-  # slow, external storage (for archiving, etc)
-  fileSystems."/mnt/usb-hdd" = {
-    device = "/dev/disk/by-uuid/aa272cff-0fcc-498e-a4cb-0d95fb60631b";
+  fileSystems."/mnt/pool" = {
+    # all btrfs devices of the same RAID volume use the same UUID.
+    device = "UUID=40fc6e1d-ba41-44de-bbf3-1aa02c3441df";
     fsType = "btrfs";
     options = [
-      "compress=zstd"
+      # "compress=zstd"  #< not much point in compressing... mostly videos and music; media.
       "defaults"
+      # `device=...` only needed if `btrfs scan` hasn't yet been run
+      # see: <https://askubuntu.com/a/484374>
+      # i don't know what guarantees NixOS/systemd make about that, so specifying all devices for now
+      # "device=/dev/disk/by-partuuid/14a7d00a-be53-2b4e-96f9-7e2c964674ec"  #< removed 2024-11-24 (for capacity upgrade)
+      "device=/dev/disk/by-partuuid/409a147e-2282-49eb-87a7-c968032ede88"  #< added 2024-11-24
+      "device=/dev/disk/by-partuuid/6b86cc10-c3cc-ec4d-b20d-b6688f0959a6"
+      # "device=/dev/disk/by-partuuid/7fd85cac-b6f3-8248-af4e-68e703d11020"  #< removed 2024-11-13 (early drive failure)
+      "device=/dev/disk/by-partuuid/d9ad5ebc-0fc4-4d89-9fd0-619ce5210f1b"  #< added 2024-11-13
+      "device=/dev/disk/by-partuuid/ef0e5c7b-fccf-f444-bac4-534424326159"
+      "nofail"
+      # "x-systemd.before=local-fs.target"
+      "x-systemd.device-bound=false"  #< don't unmount when `device` disappears (i thought this was necessary, for drive replacement, but it might not be)
+      "x-systemd.device-timeout=60s"
+      "x-systemd.mount-timeout=60s"
     ];
   };
-  sane.fs."/mnt/usb-hdd".mount = {};
 
+  # TODO: move this elsewhere and automate the ACLs!
   # FIRST TIME SETUP FOR MEDIA DIRECTORY:
-  # - set the group stick bit: `sudo find /var/media -type d -exec chmod g+s {} +`
+  # - set the group sticky bit: `sudo find /var/media -type d -exec chmod g+s {} +`
   #   - this ensures new files/dirs inherit the group of their parent dir (instead of the user who creates them)
   # - ensure everything under /var/media is mounted with `-o acl`, to support acls
   # - ensure all files are rwx by group: `setfacl --recursive --modify d:g::rwx /var/media`
@@ -115,7 +78,6 @@
   sane.fs."/var/media/Books/Books".dir = {};
   sane.fs."/var/media/Books/Visual".dir = {};
   sane.fs."/var/media/collections".dir = {};
-  # sane.fs."/var/media/datasets".dir = {};
   sane.fs."/var/media/freeleech".dir = {};
   sane.fs."/var/media/Music".dir = {};
   sane.fs."/var/media/Pictures".dir = {};
@@ -124,13 +86,6 @@
   sane.fs."/var/media/Videos/Shows".dir = {};
   sane.fs."/var/media/Videos/Talks".dir = {};
 
-  # this is file.text instead of symlink.text so that it may be read over a remote mount (where consumers might not have any /nix/store/.../README.md path)
-  sane.fs."/var/lib/uninsane/datasets/README.md".file.text = ''
-    this directory may seem redundant with ../media/datasets. it isn't.
-    this directory exists on SSD, allowing for speedy access to specific datasets when necessary.
-    the contents should be a subset of what's in ../media/datasets.
-  '';
-
   systemd.services.dedupe-media = {
     description = "transparently de-duplicate /var/media entries by using block-level hardlinks";
     script = ''
@@ -144,28 +99,5 @@
       OnUnitActiveSec = "720min";
     };
   };
-
-  # btrfs doesn't easily support swapfiles
-  # swapDevices = [
-  #   { device = "/nix/persist/swapfile"; size = 4096; }
-  # ];
-
-  # this can be a partition. create with:
-  #   fdisk <dev>
-  #     n
-  #     <default partno>
-  #     <start>
-  #     <end>
-  #     t
-  #     <partno>
-  #     19  # set part type to Linux swap
-  #     w   # write changes
-  #   mkswap -L swap <part>
-  # swapDevices = [
-  #   {
-  #     label = "swap";
-  #     # TODO: randomEncryption.enable = true;
-  #   }
-  # ];
 }
 

hosts/by-name/servo/net.nix

@@ -1,124 +0,0 @@
-{ config, lib, pkgs, ... }:
-
-let
-  portOpts = with lib; types.submodule {
-    options = {
-      visibleTo.ovpns = mkOption {
-        type = types.bool;
-        default = false;
-        description = ''
-          whether to forward inbound traffic on the OVPN vpn port to the corresponding localhost port.
-        '';
-      };
-      visibleTo.doof = mkOption {
-        type = types.bool;
-        default = false;
-        description = ''
-          whether to forward inbound traffic on the doofnet vpn port to the corresponding localhost port.
-        '';
-      };
-    };
-  };
-in
-{
-  options = with lib; {
-    sane.ports.ports = mkOption {
-      # add the `visibleTo.{doof,ovpns}` options
-      type = types.attrsOf portOpts;
-    };
-  };
-
-  config = {
-    networking.domain = "uninsane.org";
-    systemd.network.networks."50-eth0" = {
-      matchConfig.Name = "eth0";
-      networkConfig.Address = [
-        "205.201.63.12/32"
-        "10.78.79.51/22"
-      ];
-      networkConfig.DNS = [ "10.78.79.1" ];
-    };
-
-    sane.ports.openFirewall = true;
-    sane.ports.openUpnp = true;
-
-    # unless we add interface-specific settings for each VPN, we have to define nameservers globally.
-    # networking.nameservers = [
-    #   "1.1.1.1"
-    #   "9.9.9.9"
-    # ];
-
-    # services.resolved.extraConfig = ''
-    #   # docs: `man resolved.conf`
-    #   # DNS servers to use via the `wg-ovpns` interface.
-    #   #   i hope that from the root ns, these aren't visible.
-    #   DNS=46.227.67.134%wg-ovpns 192.165.9.158%wg-ovpns
-    #   FallbackDNS=1.1.1.1 9.9.9.9
-    # '';
-
-    # tun-sea config
-    sane.dns.zones."uninsane.org".inet.A."doof.tunnel" = "205.201.63.12";
-    # sane.dns.zones."uninsane.org".inet.AAAA."doof.tunnel" = "2602:fce8:106::51";  #< TODO: enable IPv6
-    networking.wireguard.interfaces.wg-doof = {
-      privateKeyFile = config.sops.secrets.wg_doof_privkey.path;
-      # wg is active only in this namespace.
-      # run e.g. ip netns exec doof <some command like ping/curl/etc, it'll go through wg>
-      #   sudo ip netns exec doof ping www.google.com
-      interfaceNamespace = "doof";
-      ips = [
-        "205.201.63.12"
-        # "2602:fce8:106::51/128"  #< TODO: enable IPv6
-      ];
-      peers = [
-        {
-          publicKey = "nuESyYEJ3YU0hTZZgAd7iHBz1ytWBVM5PjEL1VEoTkU=";
-          # TODO: configure DNS within the doof ns and use tun-sea.doof.net endpoint
-          # endpoint = "tun-sea.doof.net:53263";
-          endpoint = "205.201.63.44:53263";
-          allowedIPs = [ "0.0.0.0/0" "::/0" ];
-          persistentKeepalive = 25; #< keep the NAT alive
-        }
-      ];
-    };
-    sane.netns.doof.hostVethIpv4 = "10.0.2.5";
-    sane.netns.doof.netnsVethIpv4 = "10.0.2.6";
-    sane.netns.doof.netnsPubIpv4 = "205.201.63.12";
-    sane.netns.doof.routeTable = 12;
-
-    # OVPN CONFIG (https://www.ovpn.com):
-    # DOCS: https://nixos.wiki/wiki/WireGuard
-    # if you `systemctl restart wireguard-wg-ovpns`, make sure to also restart any other services in `NetworkNamespacePath = .../ovpns`.
-    # TODO: why not create the namespace as a seperate operation (nix config for that?)
-    networking.wireguard.enable = true;
-    networking.wireguard.interfaces.wg-ovpns = {
-      privateKeyFile = config.sops.secrets.wg_ovpns_privkey.path;
-      # wg is active only in this namespace.
-      # run e.g. ip netns exec ovpns <some command like ping/curl/etc, it'll go through wg>
-      #   sudo ip netns exec ovpns ping www.google.com
-      interfaceNamespace = "ovpns";
-      ips = [ "185.157.162.178" ];
-      peers = [
-        {
-          publicKey = "SkkEZDCBde22KTs/Hc7FWvDBfdOCQA4YtBEuC3n5KGs=";
-          endpoint = "185.157.162.10:9930";
-          # alternatively: use hostname, but that presents bootstrapping issues (e.g. if host net flakes)
-          # endpoint = "vpn36.prd.amsterdam.ovpn.com:9930";
-          allowedIPs = [ "0.0.0.0/0" ];
-          # nixOS says this is important for keeping NATs active
-          persistentKeepalive = 25;
-          # re-executes wg this often. docs hint that this might help wg notice DNS/hostname changes.
-          # so, maybe that helps if we specify endpoint as a domain name
-          # dynamicEndpointRefreshSeconds = 30;
-          # when refresh fails, try it again after this period instead.
-          # TODO: not avail until nixpkgs upgrade
-          # dynamicEndpointRefreshRestartSeconds = 5;
-        }
-      ];
-    };
-    sane.netns.ovpns.hostVethIpv4 = "10.0.1.5";
-    sane.netns.ovpns.netnsVethIpv4 = "10.0.1.6";
-    sane.netns.ovpns.netnsPubIpv4 = "185.157.162.178";
-    sane.netns.ovpns.routeTable = 11;
-    sane.netns.ovpns.dns = "46.227.67.134";  #< DNS requests inside the namespace are forwarded here
-  };
-}

hosts/by-name/servo/net/default.nix

@@ -0,0 +1,60 @@
+# debugging:
+# - enable logs (shows handshake attempts)
+#   - `echo module wireguard +p | sane-sudo-redirect /sys/kernel/debug/dynamic_debug/control`
+#   - `sudo dmesg --follow`
+#     patterns: "Sending keepalive packet to peer NN (N.N.N.N:NNNNN)"
+#     patterns: "Sending handshake initiation to peer NN (N.N.N.N:NNNNN)"
+# - when wg-doof and wg-ovpns stop routing traffic, restart with:
+#   - `systemctl restart netns-doof-wg`
+# - handshaking:
+#   - `wg show` should *always* show "latest handshake: N", with N < 2 minutes ago.
+{ lib, ... }:
+let
+  portOpts = with lib; types.submodule {
+    options = {
+      visibleTo.ovpns = mkOption {
+        type = types.bool;
+        default = false;
+        description = ''
+          whether to forward inbound traffic on the OVPN vpn port to the corresponding localhost port.
+        '';
+      };
+      visibleTo.doof = mkOption {
+        type = types.bool;
+        default = false;
+        description = ''
+          whether to forward inbound traffic on the doofnet vpn port to the corresponding localhost port.
+        '';
+      };
+    };
+  };
+in
+{
+  options = with lib; {
+    sane.ports.ports = mkOption {
+      # add the `visibleTo.{doof,ovpns}` options
+      type = types.attrsOf portOpts;
+    };
+  };
+
+  imports = [
+    ./doof.nix
+    ./ovpn.nix
+    ./wg-home.nix
+  ];
+
+  config = {
+    networking.domain = "uninsane.org";
+    systemd.network.networks."50-eth0" = {
+      matchConfig.Name = "eth0";
+      networkConfig.Address = [
+        "205.201.63.12/32"
+        "10.78.79.51/22"
+      ];
+      networkConfig.DNS = [ "10.78.79.1" ];
+    };
+
+    sane.ports.openFirewall = true;
+    sane.ports.openUpnp = true;
+  };
+}

hosts/by-name/servo/net/doof.nix

@@ -0,0 +1,27 @@
+{ config, ... }:
+{
+  # tun-sea config
+  sane.dns.zones."uninsane.org".inet.A."doof.tunnel" = "205.201.63.12";
+  # sane.dns.zones."uninsane.org".inet.AAAA."doof.tunnel" = "2602:fce8:106::51";  #< TODO: enable IPv6 (i have /128)
+
+  # if the tunnel breaks, restart it manually:
+  # - `systemctl restart netns-doof.service`
+  sane.netns.doof = {
+    veth.initns.ipv4 = "10.0.2.5";
+    veth.netns.ipv4 = "10.0.2.6";
+    routeTable = 12;
+    # wg.port = 51821;
+    wg.privateKeyFile = config.sops.secrets.wg_doof_privkey.path;
+    wg.address.ipv4 = "205.201.63.12";
+    wg.peer.publicKey = "nuESyYEJ3YU0hTZZgAd7iHBz1ytWBVM5PjEL1VEoTkU=";
+    wg.peer.endpoint = "tun-sea.doof.net:53263";
+    # wg.peer.endpoint = "205.201.63.44:53263";
+  };
+
+  # inside doof, forward DNS requests back to the root machine
+  # this is fine: nothing inside the ns performs DNS except for wireguard,
+  # and we're not forwarding external DNS requests here
+  # XXX: ACTUALLY, CAN'T EASILY DO THAT BECAUSE HICKORY-DNS IS ALREADY USING PORT 53
+  # but that's ok, we don't really need DNS *inside* this namespace.
+  # sane.netns.doof.dns.ipv4 = config.sane.netns.doof.veth.netns.ipv4;
+}

hosts/by-name/servo/net/ovpn.nix

@@ -0,0 +1,20 @@
+{ config, ... }:
+{
+  sane.ovpn.addrV4 = "172.23.174.114";
+  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:8df3:14b0";
+
+  # OVPN CONFIG (https://www.ovpn.com):
+  # DOCS: https://nixos.wiki/wiki/WireGuard
+  sane.netns.ovpns = {
+    veth.initns.ipv4 = "10.0.1.5";
+    veth.netns.ipv4 = "10.0.1.6";
+    routeTable = 11;
+    dns.ipv4 = "46.227.67.134";  #< DNS requests inside the namespace are forwarded here
+    # wg.port = 51822;
+    wg.privateKeyFile = config.sops.secrets.wg_ovpns_privkey.path;
+    wg.address.ipv4 = "185.157.162.178";
+    wg.peer.publicKey = "SkkEZDCBde22KTs/Hc7FWvDBfdOCQA4YtBEuC3n5KGs=";
+    wg.peer.endpoint = "vpn36.prd.amsterdam.ovpn.com:9930";
+    # wg.peer.endpoint = "185.157.162.10:9930";
+  };
+}

hosts/by-name/servo/net/wg-home.nix

@@ -0,0 +1,15 @@
+{ config, ... }:
+{
+  sane.services.wg-home.enable = true;
+  sane.services.wg-home.visibleToWan = true;
+  sane.services.wg-home.forwardToWan = true;
+  sane.services.wg-home.routeThroughServo = false;
+  sane.services.wg-home.ip = config.sane.hosts.by-name."servo".wg-home.ip;
+  services.unbound.settings.server.interface = [
+    # provide DNS to my wireguard clients
+    config.sane.hosts.by-name."servo".wg-home.ip
+  ];
+  services.unbound.settings.server.access-control = [
+    "${config.sane.hosts.by-name."servo".wg-home.ip}/24 allow"
+  ];
+}

hosts/by-name/servo/services/coturn.nix

@@ -104,13 +104,6 @@ in
     SRV."_turns._tcp" =                       "5 50 5349 turn";
   };
 
-  sane.derived-secrets."/var/lib/coturn/shared_secret.bin" = {
-    encoding = "base64";
-    # TODO: make this not globally readable
-    acl.mode = "0644";
-  };
-  sane.fs."/var/lib/coturn/shared_secret.bin".wantedBeforeBy = [ "coturn.service" ];
-
   # provide access to certs
   users.users.turnserver.extraGroups = [ "nginx" ];
 
@@ -119,9 +112,14 @@ in
   services.coturn.cert = "/var/lib/acme/turn.uninsane.org/fullchain.pem";
   services.coturn.pkey = "/var/lib/acme/turn.uninsane.org/key.pem";
 
+  # N.B.: prosody needs to read this shared secret
+  sops.secrets."coturn_shared_secret".owner = "turnserver";
+  sops.secrets."coturn_shared_secret".group = "turnserver";
+  sops.secrets."coturn_shared_secret".mode = "0440";
+
   #v disable to allow unauthenticated access (or set `services.coturn.no-auth = true`)
   services.coturn.use-auth-secret = true;
-  services.coturn.static-auth-secret-file = "/var/lib/coturn/shared_secret.bin";
+  services.coturn.static-auth-secret-file = "/run/secrets/coturn_shared_secret";
   services.coturn.lt-cred-mech = true; #< XXX: use-auth-secret overrides lt-cred-mech
 
   services.coturn.min-port = turnPortLow;
@@ -131,11 +129,11 @@ in
     "verbose"
     # "Verbose"  #< even MORE verbosity than "verbose"  (it's TOO MUCH verbosity really)
     "no-multicast-peers"  # disables sending to IPv4 broadcast addresses (e.g. 224.0.0.0/3)
-    # "listening-ip=${config.sane.netns.ovpns.hostVethIpv4}" "external-ip=${config.sane.netns.ovpns.netnsPubIpv4}"  #< 2024/04/25: works, if running in root namespace
-    "listening-ip=${config.sane.netns.ovpns.netnsPubIpv4}" "external-ip=${config.sane.netns.ovpns.netnsPubIpv4}"
+    # "listening-ip=${config.sane.netns.ovpns.veth.initns.ipv4}" "external-ip=${config.sane.netns.ovpns.wg.address.ipv4}"  #< 2024/04/25: works, if running in root namespace
+    "listening-ip=${config.sane.netns.ovpns.wg.address.ipv4}" "external-ip=${config.sane.netns.ovpns.wg.address.ipv4}"
 
     # old attempts:
-    # "external-ip=${config.sane.netns.ovpns.netnsPubIpv4}/${config.sane.netns.ovpns.hostVethIpv4}"
+    # "external-ip=${config.sane.netns.ovpns.wg.address.ipv4}/${config.sane.netns.ovpns.veth.initns.ipv4}"
     # "listening-ip=10.78.79.51"  # can be specified multiple times; omit for *
     # "external-ip=97.113.128.229/10.78.79.51"
     # "external-ip=97.113.128.229"

hosts/by-name/servo/services/cryptocurrencies/bitcoin.nix

@@ -22,7 +22,7 @@ let
   _bitcoindWithExternalIp = pkgs.writeShellScriptBin "bitcoind" ''
     set -xeu
     externalip="$(cat /var/lib/tor/onion/bitcoind/hostname)"
-    exec ${bitcoind}/bin/bitcoind "-externalip=$externalip" "$@"
+    exec ${lib.getExe' bitcoind "bitcoind"} "-externalip=$externalip" "$@"
   '';
   # the package i provide to services.bitcoind ends up on system PATH, and used by other tools like clightning.
   # therefore, even though services.bitcoind only needs `bitcoind` binary, provide all the other bitcoin-related binaries (notably `bitcoin-cli`) as well:

hosts/by-name/servo/services/cryptocurrencies/clightning.nix

@@ -124,7 +124,7 @@
     # peerswap:
     # - config example: <https://github.com/fort-nix/nix-bitcoin/pull/462/files#diff-b357d832705b8ce8df1f41934d613f79adb77c4cd5cd9e9eb12a163fca3e16c6>
     # XXX: peerswap crashes clightning on launch. stacktrace is useless.
-    # plugin={pkgs.peerswap}/bin/peerswap
+    # plugin={lib.getExe' pkgs.peerswap "peerswap"}
     # peerswap-db-path=/var/lib/clightning/peerswap/swaps
     # peerswap-policy-path=...
   '';

hosts/by-name/servo/services/default.nix

@@ -8,23 +8,25 @@
     ./freshrss.nix
     ./export
     ./hickory-dns.nix
+    ./gerbera.nix
     ./gitea.nix
     ./goaccess.nix
     ./ipfs.nix
     ./jackett
-    ./jellyfin.nix
+    ./jellyfin
     ./kiwix-serve.nix
     ./komga.nix
     ./lemmy.nix
     ./matrix
+    ./minidlna.nix
+    ./mumble.nix
     ./navidrome.nix
     ./nginx.nix
     ./nixos-prebuild.nix
     ./ntfy
-    ./ollama.nix
     ./pict-rs.nix
     ./pleroma.nix
-    ./postgres.nix
+    ./postgresql
     ./prosody
     ./slskd.nix
     ./transmission

hosts/by-name/servo/services/ejabberd.nix

@@ -457,13 +457,12 @@ lib.mkIf false
         mod_version = {};
       };
     });
-    sed = "${pkgs.gnused}/bin/sed";
   in ''
     ip=$(cat '${config.sane.services.dyn-dns.ipPath}')
     # config is 444 (not 644), so we want to write out-of-place and then atomically move
     # TODO: factor this out into `sane-woop` helper?
     rm -f /var/lib/ejabberd/ejabberd.yaml.new
-    ${sed} "s/%ANATIVE%/$ip/g" ${config-in} > /var/lib/ejabberd/ejabberd.yaml.new
+    ${lib.getExe pkgs.gnused} "s/%ANATIVE%/$ip/g" ${config-in} > /var/lib/ejabberd/ejabberd.yaml.new
     mv /var/lib/ejabberd/ejabberd.yaml{.new,}
   '';
 

hosts/by-name/servo/services/email/postfix.nix

@@ -162,16 +162,13 @@ in
   services.postfix.enableSubmissions = true;
   services.postfix.submissionsOptions = submissionOptions;
 
-  systemd.services.postfix.after = [ "wireguard-wg-ovpns.service" ];
-  systemd.services.postfix.partOf = [ "wireguard-wg-ovpns.service" ];
   systemd.services.postfix.unitConfig.RequiresMountsFor = [
     "/var/spool/mail"  # spooky errors when postfix is run w/o this: `warning: connect #1 to subsystem private/proxymap: Connection refused`
     "/var/lib/opendkim"
   ];
-  systemd.services.postfix.serviceConfig = {
-    # run this behind the OVPN static VPN
-    NetworkNamespacePath = "/run/netns/ovpns";
-  };
+
+  # run these behind the OVPN static VPN
+  sane.netns.ovpns.services = [ "opendkim" "postfix" ];
 
 
   #### OPENDKIM
@@ -190,11 +187,7 @@ in
   # keeping this the same as the hostname seems simplest
   services.opendkim.selector = "mx";
 
-  systemd.services.opendkim.after = [ "wireguard-wg-ovpns.service" ];
-  systemd.services.opendkim.partOf = [ "wireguard-wg-ovpns.service" ];
   systemd.services.opendkim.serviceConfig = {
-    # run this behind the OVPN static VPN
-    NetworkNamespacePath = "/run/netns/ovpns";
     # /run/opendkim/opendkim.sock needs to be rw by postfix
     UMask = lib.mkForce "0011";
   };

hosts/by-name/servo/services/export/default.nix

@@ -10,7 +10,7 @@
   fileSystems."/var/export/media" = {
     # everything in here could be considered publicly readable (based on the viewer's legal jurisdiction)
     device = "/var/media";
-    options = [ "rbind" ];
+    options = [ "rbind" "nofail" ];
   };
   # fileSystems."/var/export/playground" = {
   #   device = config.fileSystems."/mnt/persist/ext".device;
@@ -34,18 +34,16 @@
   ];
 
   sane.fs."/var/export/README.md" = {
-    wantedBy = [ "nfs.service" "sftpgo.service" ];
     file.text = ''
       - media/         read-only:  Videos, Music, Books, etc
-      - playground/    read-write: use it to share files with other users of this server, inaccessible from the www
-      - pub/           read-only:  content made to be shared with the www
+      - playground/    read-write*: use it to share files with other users of this server, inaccessible from the www
+                                    *if you can't write to it, make sure you're connected to the WiFi and not mobile.
     '';
   };
 
   sane.fs."/var/export/playground/README.md" = {
-    wantedBy = [ "nfs.service" "sftpgo.service" ];
     file.text = ''
-      this directory is intentionally read+write by anyone with access (i.e. on the LAN).
+      this directory is intentionally read+write by anyone with access.
       - share files
       - write poetry
       - be a friendly troll
@@ -53,7 +51,6 @@
   };
 
   sane.fs."/var/export/.public_for_test/test" = {
-    wantedBy = [ "nfs.service" "sftpgo.service" ];
     file.text = ''
       automated tests read this file to probe connectivity
     '';

hosts/by-name/servo/services/export/sftpgo/default.nix

@@ -102,14 +102,14 @@ in
           }
           {
             # binding this means any doof client can connect (TLS only)
-            address = config.sane.netns.doof.hostVethIpv4;
+            address = config.sane.netns.doof.veth.initns.ipv4;
             port = 990;
             debug = true;
             tls_mode = 2;  # 2 = "implicit FTPS": client negotiates TLS before any FTP command.
           }
           {
             # binding this means any LAN client can connect via `ftp.uninsane.org` (TLS only)
-            address = config.sane.netns.doof.netnsPubIpv4;
+            address = config.sane.netns.doof.wg.address.ipv4;
             port = 990;
             debug = true;
             tls_mode = 2;  # 2 = "implicit FTPS": client negotiates TLS before any FTP command.
@@ -141,7 +141,7 @@ in
       };
       data_provider = {
         driver = "memory";
-        external_auth_hook = "${external_auth_hook}/bin/external_auth_hook";
+        external_auth_hook = lib.getExe external_auth_hook;
         # track_quota:
         # - 0: disable quota tracking
         # - 1: quota is updated on every upload/delete, even if user has no quota restriction
@@ -158,14 +158,15 @@ in
   ];
 
   systemd.services.sftpgo = {
-    after = [ "network-online.target" ];
+    after = [ "network-online.target" ];  #< so that it reliably binds to all interfaces/netns's?
     wants = [ "network-online.target" ];
-    serviceConfig = {
-      ReadWritePaths = [ "/var/export" ];
-
-      Restart = "always";
-      RestartSec = "20s";
-      UMask = lib.mkForce "0002";
-    };
+    unitConfig.RequiresMountsFor = [
+      "/var/export/media"
+      "/var/export/playground"
+    ];
+    serviceConfig.ReadWritePaths = [ "/var/export" ];
+    serviceConfig.Restart = "always";
+    serviceConfig.RestartSec = "20s";
+    serviceConfig.UMask = lib.mkForce "0002";
   };
 }

hosts/by-name/servo/services/export/sftpgo/external_auth_hook

@@ -69,8 +69,12 @@ TRUSTED_CREDS = [
   # /etc/shadow style creds.
   # mkpasswd -m sha-512
   # $<method>$<salt>$<hash>
-  "$6$Zq3c2u4ghUH4S6EP$pOuRt13sEKfX31OqPbbd1LuhS21C9MICMc94iRdTAgdAcJ9h95gQH/6Jf6Ie4Obb0oxQtojRJ1Pd/9QHOlFMW."  #< m. rocket boy
+  "$6$Zq3c2u4ghUH4S6EP$pOuRt13sEKfX31OqPbbd1LuhS21C9MICMc94iRdTAgdAcJ9h95gQH/6Jf6Ie4Obb0oxQtojRJ1Pd/9QHOlFMW.",  #< m. rocket boy
+  "$6$B0NLGNdCL51PNse1$46G.aA1ATWIv5v.jUsKf4F3NS7emV2jB2gkZ3MytZtMvw2pjniHmRl0fywRjKW9TuXTeK9T50v.H0f2BaQ4PT1",  #< v. telephony
 ]
+TRUSTED_VIEWING_OR_PLAYGROUND_CREDS = [
+  # "$6$iikDajz5b.YH1.on$tfSzzBEtX8IeDiJJXCasOTxRTd7cFDKXU6dhlWYVhK6xDeJhV2fh6bmm1WIHItjIth9Eh9zNgUB8xibMIWCm/."  # fedi (2024-08-27); music appreciation
+];
 
 def mkAuthOk(username: str, permissions: dict[str, list[str]]) -> dict:
   return dict(
@@ -112,8 +116,8 @@ def isLan(ip: str) -> bool:
 def isWireguard(ip: str) -> bool:
   return ip.startswith("10.0.10.")
 
-def isTrustedCred(password: str) -> bool:
-  for cred in TRUSTED_CREDS:
+def isTrustedCred(password: str, credlist: list[str] = TRUSTED_CREDS) -> bool:
+  for cred in credlist:
     if passlib.hosts.linux_context.verify(password, cred):
       return True
 
@@ -130,6 +134,22 @@ def getAuthResponse(ip: str, username: str, password: str) -> dict:
       "/": PERM_RW,
       "/playground": PERM_RW,
       "/.public_for_test": PERM_RO,
+      "/media/Music": PERM_RO,  #< i am too picky about Music organization
+    })
+  if isTrustedCred(password, TRUSTED_VIEWING_OR_PLAYGROUND_CREDS) and username != "colin":
+    return mkAuthOk(username, permissions = {
+      # error prone, but... not the worst if i miss something
+      "/": PERM_LIST,
+      "/media/archive": PERM_DENY,
+      "/media/Books": PERM_RO,
+      "/media/collections": PERM_DENY,
+      "/media/games": PERM_RO,
+      "/media/Music": PERM_RO,
+      "/media/Pictures": PERM_RO,
+      "/media/torrents": PERM_DENY,
+      "/media/Videos": PERM_RO,
+      "/playground": PERM_RW,
+      "/.public_for_test": PERM_RO,
     })
   if isWireguard(ip):
     # allow any user from wireguard

hosts/by-name/servo/services/gerbera.nix

@@ -0,0 +1,38 @@
+# gerbera UPNP/media server
+# accessible from TVs on the LAN
+# unauthenticated admin and playback UI at http://servo:49152/
+#
+# supposedly does transcoding, but i poked at it for 10 minutes and couldn't get that working
+#
+# compatibility:
+# - LG TV: music: all working
+# - LG TV: videos: mixed
+{ lib, ... }:
+lib.mkIf false  #< XXX(2024-11-17): WORKS, but no better than any other service; slow to index and transcoding doesn't work
+{
+  sane.ports.ports."1900" = {
+    protocol = [ "udp" ];
+    visibleTo.lan = true;
+    description = "colin-upnp-for-gerbera";
+  };
+  sane.ports.ports."49152" = {
+    protocol = [ "tcp" "udp" ];  # TODO: is udp required?
+    visibleTo.lan = true;
+    description = "colin-gerbera-http";
+  };
+
+  sane.persist.sys.byStore.plaintext = [
+    # persist the index database, since it takes a good 30 minutes to scan the media collection
+    { user = "mediatomb"; group = "mediatomb"; mode = "0700"; path = "/var/lib/gerbera"; method = "bind"; }
+  ];
+
+  services.mediatomb.enable = true;
+  services.mediatomb.serverName = "servo";
+  services.mediatomb.transcoding = true;
+  services.mediatomb.mediaDirectories = [
+    { path = "/var/media/Music"; recursive = true; hidden-files = false; }
+    { path = "/var/media/Videos/Film"; recursive = true; hidden-files = false; }
+    { path = "/var/media/Videos/Shows"; recursive = true; hidden-files = false; }
+  ];
+  users.users.mediatomb.extraGroups = [ "media" ];
+}

hosts/by-name/servo/services/gitea.nix

@@ -1,6 +1,6 @@
 # config options: <https://docs.gitea.io/en-us/administration/config-cheat-sheet/>
 # TODO: service shouldn't run as `git` user, but as `gitea`
-{ config, pkgs, lib, ... }:
+{ pkgs, lib, ... }:
 
 {
   sane.persist.sys.byStore.private = [
@@ -11,15 +11,23 @@
 
   services.gitea.enable = true;
   services.gitea.user = "git";  # default is 'gitea'
-  services.gitea.database.type = "postgres";
-  services.gitea.database.user = "git";
   services.gitea.appName = "Perfectly Sane Git";
   # services.gitea.disableRegistration = true;
 
-  services.gitea.database.createDatabase = false;  #< silence warning which wants db user and name to be equal
-  # TODO: remove this after merge: <https://github.com/NixOS/nixpkgs/pull/268849>
+  services.gitea.database.createDatabase = false;  # can only createDatabase if user ("git") == dbname ("gitea")
+  services.gitea.database.type = "postgres";
+  services.gitea.database.user = "git";
+  # createDatabase=false means manually specify the connection; see: <https://github.com/NixOS/nixpkgs/pull/268849>
+  services.gitea.database.name = "gitea";
   services.gitea.database.socket = "/run/postgresql";  #< would have been set if createDatabase = true
 
+  services.postgresql.enable = true;
+  services.postgresql.ensureDatabases = [ "gitea" ];
+  services.postgresql.ensureUsers = [{
+    name = "git";
+    # ensureDBOwnership = true;  # not possible if db name ("gitea") != db username ("git"); one-time manual setup required to grant user ownership of the relevant db
+  }];
+
   # gitea doesn't create the git user
   users.users.git = {
     description = "Gitea Service";
@@ -86,7 +94,7 @@
       ENABLED = true;
       FROM = "notify.git@uninsane.org";
       PROTOCOL = "sendmail";
-      SENDMAIL_PATH = "${pkgs.postfix}/bin/sendmail";
+      SENDMAIL_PATH = lib.getExe' pkgs.postfix "sendmail";
       SENDMAIL_ARGS = "--";  # most "sendmail" programs take options, "--" will prevent an email address being interpreted as an option.
     };
     time = {
@@ -96,18 +104,23 @@
     };
   };
 
+  systemd.services.gitea.wants = [ "postgresql.service" ];
   systemd.services.gitea.serviceConfig = {
     # nix default is AF_UNIX AF_INET AF_INET6.
     # we need more protos for sendmail to work. i thought it only needed +AF_LOCAL, but that didn't work.
     RestrictAddressFamilies = lib.mkForce "~";
     # add maildrop to allow sendmail to work
-    ReadWritePaths = lib.mkForce [
+    ReadWritePaths = [
       "/var/lib/postfix/queue/maildrop"
-      "/var/lib/gitea"
     ];
+    # rate limit the restarts to prevent systemd from disabling it
+    RestartSec = 5;
+    RestartMaxDelaySec = 30;
+    StartLimitBurst = 120;
+    RestartSteps = 5;
   };
 
-  services.openssh.settings.UsePAM = true;  #< required for `git` user to authenticate
+  # services.openssh.settings.UsePAM = true;  #< required for `git` user to authenticate
 
   # hosted git (web view and for `git <cmd>` use
   # TODO: enable publog?

hosts/by-name/servo/services/goaccess.nix

@@ -1,4 +1,5 @@
-{ pkgs, ... }:
+{ lib, pkgs, ... }:
+lib.mkIf false  #< 2024/09/30: disabled because i haven't used it in several months
 {
   # based on <https://bytes.fyi/real-time-goaccess-reports-with-nginx/>
   # log-format setting can be derived with this tool if custom:
@@ -10,7 +11,7 @@
     description = "GoAccess server monitoring";
     serviceConfig = {
       ExecStart = ''
-        ${pkgs.goaccess}/bin/goaccess \
+        ${lib.getExe pkgs.goaccess} \
           -f /var/log/nginx/public.log \
           --log-format=VCOMBINED \
           --real-time-html \
@@ -22,7 +23,7 @@
           --port=7890 \
           -o /var/lib/goaccess/index.html
       '';
-      ExecReload = "${pkgs.coreutils}/bin/kill -HUP $MAINPID";
+      ExecReload = "${lib.getExe' pkgs.coreutils "kill"} -HUP $MAINPID";
       Type = "simple";
       Restart = "on-failure";
       RestartSec = "10s";

hosts/by-name/servo/services/hickory-dns.nix

@@ -86,9 +86,9 @@ in
   sane.services.hickory-dns.enable = true;
   sane.services.hickory-dns.instances = let
     mkSubstitutions = flavor: {
-      "%ADOOF%" = config.sane.netns.doof.netnsPubIpv4;
+      "%ADOOF%" = config.sane.netns.doof.wg.address.ipv4;
       "%ANATIVE%" = nativeAddrs."servo.${flavor}";
-      "%AOVPNS%" = config.sane.netns.ovpns.netnsPubIpv4;
+      "%AOVPNS%" = config.sane.netns.ovpns.wg.address.ipv4;
       "%AWAN%" = "$(cat '${dyn-dns.ipPath}')";
       "%CNAMENATIVE%" = "servo.${flavor}";
     };
@@ -97,37 +97,37 @@ in
     doof = {
       substitutions = mkSubstitutions "doof";
       listenAddrsIpv4 = [
-        config.sane.netns.doof.hostVethIpv4
-        config.sane.netns.doof.netnsPubIpv4
+        config.sane.netns.doof.veth.initns.ipv4
+        config.sane.netns.doof.wg.address.ipv4
         nativeAddrs."servo.lan"
-        # config.sane.netns.ovpns.hostVethIpv4
+        # config.sane.netns.ovpns.veth.initns.ipv4
       ];
     };
-    hn = {
-      substitutions = mkSubstitutions "hn";
-      listenAddrsIpv4 = [ nativeAddrs."servo.hn" ];
-      enableRecursiveResolver = true;  #< allow wireguard clients to use this as their DNS resolver
-      # extraConfig = {
-      #   zones = [
-      #     {
-      #       # forward the root zone to the local DNS resolver
-      #       # to allow wireguard clients to use this as their DNS resolver
-      #       zone = ".";
-      #       zone_type = "Forward";
-      #       stores = {
-      #         type = "forward";
-      #         name_servers = [
-      #           {
-      #             socket_addr = "127.0.0.53:53";
-      #             protocol = "udp";
-      #             trust_nx_responses = true;
-      #           }
-      #         ];
-      #       };
-      #     }
-      #   ];
-      # };
-    };
+    # hn = {
+    #   substitutions = mkSubstitutions "hn";
+    #   listenAddrsIpv4 = [ nativeAddrs."servo.hn" ];
+    #   enableRecursiveResolver = true;  #< allow wireguard clients to use this as their DNS resolver
+    #   # extraConfig = {
+    #   #   zones = [
+    #   #     {
+    #   #       # forward the root zone to the local DNS resolver
+    #   #       # to allow wireguard clients to use this as their DNS resolver
+    #   #       zone = ".";
+    #   #       zone_type = "Forward";
+    #   #       stores = {
+    #   #         type = "forward";
+    #   #         name_servers = [
+    #   #           {
+    #   #             socket_addr = "127.0.0.53:53";
+    #   #             protocol = "udp";
+    #   #             trust_nx_responses = true;
+    #   #           }
+    #   #         ];
+    #   #       };
+    #   #     }
+    #   #   ];
+    #   # };
+    # };
     # lan = {
     #   substitutions = mkSubstitutions "lan";
     #   listenAddrsIpv4 = [ nativeAddrs."servo.lan" ];
@@ -141,5 +141,10 @@ in
     # };
   };
 
+  systemd.services.hickory-dns-doof.after = [
+    # service will fail to bind the veth, otherwise
+    "netns-doof-veth.service"
+  ];
+
   sane.services.dyn-dns.restartOnChange = lib.map (c: "${c.service}.service") (builtins.attrValues config.sane.services.hickory-dns.instances);
 }

hosts/by-name/servo/services/jackett/default.nix

@@ -10,15 +10,16 @@ in
   ];
   services.jackett.enable = true;
 
-  systemd.services.jackett.after = [ "wireguard-wg-ovpns.service" ];
-  systemd.services.jackett.partOf = [ "wireguard-wg-ovpns.service" ];
+  # run this behind the OVPN static VPN
+  sane.netns.ovpns.services = [ "jackett" ];
   systemd.services.jackett = {
-    # run this behind the OVPN static VPN
-    serviceConfig.NetworkNamespacePath = "/run/netns/ovpns";
-    serviceConfig.ExecStartPre = [ "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect ${config.sane.netns.ovpns.netnsPubIpv4}" ];  # abort if public IP is not as expected
+    serviceConfig.ExecStartPre = [
+      # abort if public IP is not as expected
+      "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect ${config.sane.netns.ovpns.wg.address.ipv4}"
+    ];
     # patch in `--ListenPublic` so that it's reachable from the netns veth.
     # this also makes it reachable from the VPN pub address. oh well.
-    serviceConfig.ExecStart = lib.mkForce "${cfg.package}/bin/Jackett --ListenPublic --NoUpdates --DataFolder '${cfg.dataDir}'";
+    serviceConfig.ExecStart = lib.mkForce "${lib.getExe' cfg.package "Jackett"} --ListenPublic --NoUpdates --DataFolder '${cfg.dataDir}'";
     serviceConfig.RestartSec = "30s";
 
     # hardening (systemd-analyze security jackett)
@@ -55,7 +56,7 @@ in
     enableACME = true;
     # inherit kTLS;
     locations."/" = {
-      proxyPass = "http://${config.sane.netns.ovpns.netnsVethIpv4}:9117";
+      proxyPass = "http://${config.sane.netns.ovpns.veth.netns.ipv4}:9117";
       recommendedProxySettings = true;
     };
     locations."= /robots.txt".extraConfig = ''

hosts/by-name/servo/services/jellyfin.nix

@@ -1,127 +0,0 @@
-# configuration options (today i don't store my config in nix):
-#
-# - jellyfin-web can be statically configured (result/share/jellyfin-web/config.json)
-#   - <https://jellyfin.org/docs/general/clients/web-config>
-#   - configure server list, plugins, "menuLinks", colors
-#
-# - jellfyin server is configured in /var/lib/jellfin/
-#   - root/default/<LibraryType>/
-#     - <LibraryName>.mblink: contains the directory name where this library lives
-#     - options.xml: contains preferences which were defined in the web UI during import
-#       - e.g. `EnablePhotos`, `EnableChapterImageExtraction`, etc.
-#   - config/encoding.xml: transcoder settings
-#   - config/system.xml: misc preferences like log file duration, audiobook resume settings, etc.
-#   - data/jellyfin.db: maybe account definitions? internal state?
-
-{ config, lib, ... }:
-
-{
-  # https://jellyfin.org/docs/general/networking/index.html
-  sane.ports.ports."1900" = {
-    protocol = [ "udp" ];
-    visibleTo.lan = true;
-    description = "colin-upnp-for-jellyfin";
-  };
-  sane.ports.ports."7359" = {
-    protocol = [ "udp" ];
-    visibleTo.lan = true;
-    description = "colin-jellyfin-specific-client-discovery";
-    # ^ not sure if this is necessary: copied this port from nixos jellyfin.openFirewall
-  };
-  # not sure if 8096/8920 get used either:
-  sane.ports.ports."8096" = {
-    protocol = [ "tcp" ];
-    visibleTo.lan = true;
-    description = "colin-jellyfin-http-lan";
-  };
-  sane.ports.ports."8920" = {
-    protocol = [ "tcp" ];
-    visibleTo.lan = true;
-    description = "colin-jellyfin-https-lan";
-  };
-
-  sane.persist.sys.byStore.plaintext = [
-    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin"; method = "bind"; }
-  ];
-  sane.fs."/var/lib/jellyfin/config/logging.json" = {
-    # "Emby.Dlna" logging: <https://jellyfin.org/docs/general/networking/dlna>
-    symlink.text = ''
-      {
-        "Serilog": {
-          "MinimumLevel": {
-            "Default": "Information",
-            "Override": {
-              "Microsoft": "Warning",
-              "System": "Warning",
-              "Emby.Dlna": "Debug",
-              "Emby.Dlna.Eventing": "Debug"
-            }
-          },
-          "WriteTo": [
-            {
-              "Name": "Console",
-              "Args": {
-                "outputTemplate": "[{Timestamp:HH:mm:ss}] [{Level:u3}] [{ThreadId}] {SourceContext}: {Message:lj}{NewLine}{Exception}"
-              }
-            }
-          ],
-          "Enrich": [ "FromLogContext", "WithThreadId" ]
-        }
-      }
-    '';
-    wantedBeforeBy = [ "jellyfin.service" ];
-  };
-
-  # Jellyfin multimedia server
-  # this is mostly taken from the official jellfin.org docs
-  services.nginx.virtualHosts."jelly.uninsane.org" = {
-    forceSSL = true;
-    enableACME = true;
-    # inherit kTLS;
-
-    locations."/" = {
-      proxyPass = "http://127.0.0.1:8096";
-      extraConfig = ''
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header X-Forwarded-Protocol $scheme;
-        proxy_set_header X-Forwarded-Host $http_host;
-
-        # Disable buffering when the nginx proxy gets very resource heavy upon streaming
-        proxy_buffering off;
-      '';
-    };
-    # locations."/web/" = {
-    #   proxyPass = "http://127.0.0.1:8096/web/index.html";
-    #   extraConfig = ''
-    #     proxy_set_header Host $host;
-    #     proxy_set_header X-Real-IP $remote_addr;
-    #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    #     proxy_set_header X-Forwarded-Proto $scheme;
-    #     proxy_set_header X-Forwarded-Protocol $scheme;
-    #     proxy_set_header X-Forwarded-Host $http_host;
-    #   '';
-    # };
-    locations."/socket" = {
-      proxyPass = "http://127.0.0.1:8096";
-      extraConfig = ''
-        proxy_http_version 1.1;
-        proxy_set_header Upgrade $http_upgrade;
-        proxy_set_header Connection "upgrade";
-
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header X-Forwarded-Protocol $scheme;
-        proxy_set_header X-Forwarded-Host $http_host;
-      '';
-    };
-  };
-
-  sane.dns.zones."uninsane.org".inet.CNAME."jelly" = "native";
-
-  services.jellyfin.enable = true;
-}

hosts/by-name/servo/services/jellyfin/default.nix

@@ -0,0 +1,171 @@
+# configuration options (today only a *subset* of the config is done in nix)
+# - jellyfin-web can be statically configured (result/share/jellyfin-web/config.json)
+#   - <https://jellyfin.org/docs/general/clients/web-config>
+#   - configure server list, plugins, "menuLinks", colors
+#
+# - jellfyin server is configured in /var/lib/jellfin/
+#   - root/default/<LibraryType>/
+#     - <LibraryName>.mblink: contains the directory name where this library lives
+#     - options.xml: contains preferences which were defined in the web UI during import
+#       - e.g. `EnablePhotos`, `EnableChapterImageExtraction`, etc.
+#   - config/encoding.xml: transcoder settings
+#   - config/system.xml: misc preferences like log file duration, audiobook resume settings, etc.
+#   - data/jellyfin.db: maybe account definitions? internal state?
+#
+# N.B.: default install DOES NOT SUPPORT DLNA out of the box.
+#       one must install it as a "plugin", which can be done through the UI.
+{ lib, ... }:
+
+# lib.mkIf false  #< XXX(2024-11-17): disabled because it hasn't been working for months; web UI hangs on load, TVs see no files
+{
+  # https://jellyfin.org/docs/general/networking/index.html
+  sane.ports.ports."1900" = {
+    protocol = [ "udp" ];
+    visibleTo.lan = true;
+    description = "colin-upnp-for-jellyfin";
+  };
+  sane.ports.ports."7359" = {
+    protocol = [ "udp" ];
+    visibleTo.lan = true;
+    description = "colin-jellyfin-specific-client-discovery";
+    # ^ not sure if this is necessary: copied this port from nixos jellyfin.openFirewall
+  };
+  # not sure if 8096/8920 get used either:
+  sane.ports.ports."8096" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    description = "colin-jellyfin-http-lan";
+  };
+  sane.ports.ports."8920" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    description = "colin-jellyfin-https-lan";
+  };
+
+  sane.persist.sys.byStore.plaintext = [
+    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin/data"; method = "bind"; }
+    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin/metadata"; method = "bind"; }
+    # TODO: ship plugins statically, via nix. that'll be less fragile
+    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin/plugins/DLNA_5.0.0.0"; method = "bind"; }
+    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin/root"; method = "bind"; }
+  ];
+  sane.persist.sys.byStore.ephemeral = [
+    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin/log"; method = "bind"; }
+    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin/transcodes"; method = "bind"; }
+  ];
+
+  services.jellyfin.enable = true;
+  users.users.jellyfin.extraGroups = [ "media" ];
+
+  sane.fs."/var/lib/jellyfin".dir.acl = {
+    user = "jellyfin";
+    group = "jellyfin";
+    mode = "0700";
+  };
+
+  # `"Jellyfin.Plugin.Dlna": "Debug"` logging: <https://jellyfin.org/docs/general/networking/dlna>
+  # TODO: switch Dlna back to 'Information' once satisfied with stability
+  sane.fs."/var/lib/jellyfin/config/logging.json".symlink.text = ''
+    {
+      "Serilog": {
+        "MinimumLevel": {
+          "Default": "Information",
+          "Override": {
+            "Microsoft": "Warning",
+            "System": "Warning",
+            "Jellyfin.Plugin.Dlna": "Debug"
+          }
+        },
+        "WriteTo": [
+          {
+            "Name": "Console",
+            "Args": {
+              "outputTemplate": "[{Timestamp:HH:mm:ss}] [{Level:u3}] [{ThreadId}] {SourceContext}: {Message:lj}{NewLine}{Exception}"
+            }
+          }
+        ],
+        "Enrich": [ "FromLogContext", "WithThreadId" ]
+      }
+    }
+  '';
+
+  sane.fs."/var/lib/jellyfin/config/network.xml".file.text = ''
+    <?xml version="1.0" encoding="utf-8"?>
+    <NetworkConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
+      <BaseUrl />
+      <EnableHttps>false</EnableHttps>
+      <RequireHttps>false</RequireHttps>
+      <InternalHttpPort>8096</InternalHttpPort>
+      <InternalHttpsPort>8920</InternalHttpsPort>
+      <PublicHttpPort>8096</PublicHttpPort>
+      <PublicHttpsPort>8920</PublicHttpsPort>
+      <AutoDiscovery>true</AutoDiscovery>
+      <EnableUPnP>false</EnableUPnP>
+      <EnableIPv4>true</EnableIPv4>
+      <EnableIPv6>false</EnableIPv6>
+      <EnableRemoteAccess>true</EnableRemoteAccess>
+      <LocalNetworkSubnets>
+        <string>10.78.76.0/22</string>
+      </LocalNetworkSubnets>
+      <KnownProxies>
+        <string>127.0.0.1</string>
+        <string>localhost</string>
+        <string>10.78.79.1</string>
+      </KnownProxies>
+      <IgnoreVirtualInterfaces>false</IgnoreVirtualInterfaces>
+      <VirtualInterfaceNames />
+      <EnablePublishedServerUriByRequest>false</EnablePublishedServerUriByRequest>
+      <PublishedServerUriBySubnet />
+      <RemoteIPFilter />
+      <IsRemoteIPFilterBlacklist>false</IsRemoteIPFilterBlacklist>
+    </NetworkConfiguration>
+  '';
+
+  # guest user id is `5ad194d60dca41de84b332950ffc4308`
+  sane.fs."/var/lib/jellyfin/plugins/configurations/Jellyfin.Plugin.Dlna.xml".file.text = ''
+    <?xml version="1.0" encoding="utf-8"?>
+    <DlnaPluginConfiguration xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
+      <EnablePlayTo>true</EnablePlayTo>
+      <ClientDiscoveryIntervalSeconds>60</ClientDiscoveryIntervalSeconds>
+      <BlastAliveMessages>true</BlastAliveMessages>
+      <AliveMessageIntervalSeconds>180</AliveMessageIntervalSeconds>
+      <SendOnlyMatchedHost>true</SendOnlyMatchedHost>
+      <DefaultUserId>5ad194d6-0dca-41de-84b3-32950ffc4308</DefaultUserId>
+    </DlnaPluginConfiguration>
+  '';
+
+  # fix LG TV to play more files.
+  # there are certain files for which it only supports Direct Play (not even "Direct Stream" -- but "Direct Play").
+  # this isn't a 100% fix: patching the profile allows e.g. Azumanga Daioh to play,
+  # but A Place Further Than the Universe still fails as before.
+  #
+  # profile is based on upstream: <https://github.com/jellyfin/jellyfin-plugin-dlna>
+  sane.fs."/var/lib/jellyfin/plugins/DLNA_5.0.0.0/profiles/LG Smart TV.xml".symlink.target = ./dlna/user/LG_Smart_TV.xml;
+  # XXX(2024-11-17): old method, but the file referenced seems not to be used and setting just it causes failures:
+  # > [DBG] Jellyfin.Plugin.Dlna.ContentDirectory.ContentDirectoryService: Not eligible for DirectPlay due to unsupported subtitles
+  # sane.fs."/var/lib/jellyfin/plugins/configurations/dlna/user/LG Smart TV.xml".symlink.target = ./dlna/user/LG_Smart_TV.xml;
+
+  systemd.services.jellyfin.unitConfig.RequiresMountsFor = [
+    "/var/media"
+  ];
+
+  # Jellyfin multimedia server
+  # this is mostly taken from the official jellfin.org docs
+  services.nginx.virtualHosts."jelly.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+    # inherit kTLS;
+
+    locations."/" = {
+      proxyPass = "http://127.0.0.1:8096";
+      proxyWebsockets = true;
+      recommendedProxySettings = true;
+      # extraConfig = ''
+      #   # Disable buffering when the nginx proxy gets very resource heavy upon streaming
+      #   proxy_buffering off;
+      # '';
+    };
+  };
+
+  sane.dns.zones."uninsane.org".inet.CNAME."jelly" = "native";
+}

hosts/by-name/servo/services/jellyfin/dlna/user/LG_Smart_TV.xml

@@ -0,0 +1,91 @@
+<?xml version="1.0"?>
+<Profile xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+  xmlns:xsd="http://www.w3.org/2001/XMLSchema">
+  <Name>LG Smart TV</Name>
+  <Identification>
+    <ModelName>LG TV</ModelName>
+    <Headers />
+  </Identification>
+  <Manufacturer>Jellyfin</Manufacturer>
+  <ManufacturerUrl>https://github.com/jellyfin/jellyfin</ManufacturerUrl>
+  <ModelName>Jellyfin Server</ModelName>
+  <ModelDescription>UPnP/AV 1.0 Compliant Media Server</ModelDescription>
+  <ModelNumber>01</ModelNumber>
+  <ModelUrl>https://github.com/jellyfin/jellyfin</ModelUrl>
+  <EnableAlbumArtInDidl>false</EnableAlbumArtInDidl>
+  <EnableSingleAlbumArtLimit>false</EnableSingleAlbumArtLimit>
+  <EnableSingleSubtitleLimit>false</EnableSingleSubtitleLimit>
+  <SupportedMediaTypes>Audio,Photo,Video</SupportedMediaTypes>
+  <AlbumArtPn>JPEG_SM</AlbumArtPn>
+  <MaxAlbumArtWidth>480</MaxAlbumArtWidth>
+  <MaxAlbumArtHeight>480</MaxAlbumArtHeight>
+  <MaxIconWidth>48</MaxIconWidth>
+  <MaxIconHeight>48</MaxIconHeight>
+  <MaxStreamingBitrate>140000000</MaxStreamingBitrate>
+  <MaxStaticBitrate>140000000</MaxStaticBitrate>
+  <MusicStreamingTranscodingBitrate>192000</MusicStreamingTranscodingBitrate>
+  <MaxStaticMusicBitrate xsi:nil="true" />
+  <ProtocolInfo>http-get:*:video/mpeg:*,http-get:*:video/mp4:*,http-get:*:video/vnd.dlna.mpeg-tts:*,http-get:*:video/avi:*,http-get:*:video/x-matroska:*,http-get:*:video/x-ms-wmv:*,http-get:*:video/wtv:*,http-get:*:audio/mpeg:*,http-get:*:audio/mp3:*,http-get:*:audio/mp4:*,http-get:*:audio/x-ms-wma:*,http-get:*:audio/wav:*,http-get:*:audio/L16:*,http-get:*:image/jpeg:*,http-get:*:image/png:*,http-get:*:image/gif:*,http-get:*:image/tiff:*</ProtocolInfo>
+  <TimelineOffsetSeconds>10</TimelineOffsetSeconds>
+  <RequiresPlainVideoItems>false</RequiresPlainVideoItems>
+  <RequiresPlainFolders>false</RequiresPlainFolders>
+  <EnableMSMediaReceiverRegistrar>false</EnableMSMediaReceiverRegistrar>
+  <IgnoreTranscodeByteRangeRequests>false</IgnoreTranscodeByteRangeRequests>
+  <XmlRootAttributes />
+  <DirectPlayProfiles>
+    <DirectPlayProfile container="ts,mpegts,avi,mkv,m2ts" audioCodec="aac,ac3,eac3,mp3,dca,dts" videoCodec="h264,hevc" type="Video" />
+    <DirectPlayProfile container="mp4,m4v" audioCodec="aac,ac3,eac3,mp3,dca,dts" videoCodec="h264,mpeg4,hevc" type="Video" />
+    <DirectPlayProfile container="mp3" type="Audio" />
+    <DirectPlayProfile container="jpeg" type="Photo" />
+    <DirectPlayProfile container="" audioCodec="" videoCodec="" type="Video" />
+  </DirectPlayProfiles>
+  <TranscodingProfiles>
+    <TranscodingProfile container="mp3" type="Audio" audioCodec="mp3" estimateContentLength="false" enableMpegtsM2TsMode="false" transcodeSeekInfo="Auto" copyTimestamps="false" context="Streaming" enableSubtitlesInManifest="false" minSegments="0" segmentLength="0" breakOnNonKeyFrames="false" />
+    <TranscodingProfile container="ts" type="Video" videoCodec="h264" audioCodec="ac3,aac,mp3" estimateContentLength="false" enableMpegtsM2TsMode="false" transcodeSeekInfo="Auto" copyTimestamps="false" context="Streaming" enableSubtitlesInManifest="false" minSegments="0" segmentLength="0" breakOnNonKeyFrames="false" />
+    <TranscodingProfile container="jpeg" type="Photo" estimateContentLength="false" enableMpegtsM2TsMode="false" transcodeSeekInfo="Auto" copyTimestamps="false" context="Streaming" enableSubtitlesInManifest="false" minSegments="0" segmentLength="0" breakOnNonKeyFrames="false" />
+  </TranscodingProfiles>
+  <ContainerProfiles>
+    <ContainerProfile type="Photo">
+      <Conditions>
+        <ProfileCondition condition="LessThanEqual" property="Width" value="1920" isRequired="true" />
+        <ProfileCondition condition="LessThanEqual" property="Height" value="1080" isRequired="true" />
+      </Conditions>
+    </ContainerProfile>
+  </ContainerProfiles>
+  <CodecProfiles>
+    <CodecProfile type="Video" codec="mpeg4">
+      <Conditions>
+        <ProfileCondition condition="LessThanEqual" property="Width" value="1920" isRequired="true" />
+        <ProfileCondition condition="LessThanEqual" property="Height" value="1080" isRequired="true" />
+        <ProfileCondition condition="LessThanEqual" property="VideoFramerate" value="30" isRequired="true" />
+      </Conditions>
+      <ApplyConditions />
+    </CodecProfile>
+    <CodecProfile type="Video" codec="h264">
+      <Conditions>
+        <ProfileCondition condition="LessThanEqual" property="Width" value="1920" isRequired="true" />
+        <ProfileCondition condition="LessThanEqual" property="Height" value="1080" isRequired="true" />
+        <ProfileCondition condition="LessThanEqual" property="VideoLevel" value="41" isRequired="true" />
+      </Conditions>
+      <ApplyConditions />
+    </CodecProfile>
+    <CodecProfile type="VideoAudio" codec="ac3,eac3,aac,mp3">
+      <Conditions>
+        <ProfileCondition condition="LessThanEqual" property="AudioChannels" value="6" isRequired="true" />
+      </Conditions>
+      <ApplyConditions />
+    </CodecProfile>
+  </CodecProfiles>
+  <ResponseProfiles>
+    <ResponseProfile container="m4v" type="Video" mimeType="video/mp4">
+      <Conditions />
+    </ResponseProfile>
+    <ResponseProfile container="ts,mpegts" type="Video" mimeType="video/mpeg">
+      <Conditions />
+    </ResponseProfile>
+  </ResponseProfiles>
+  <SubtitleProfiles>
+    <SubtitleProfile format="srt" method="Embed" />
+    <SubtitleProfile format="srt" method="External" />
+  </SubtitleProfiles>
+</Profile>

hosts/by-name/servo/services/kiwix-serve.nix

@@ -1,19 +1,26 @@
-# how to update wikipedia snapshot:
-# - browse for later snapshots:
-#   - <https://mirror.accum.se/mirror/wikimedia.org/other/kiwix/zim/wikipedia>
-# - DL directly, or via rsync (resumable):
-#   - `rsync --progress --append-verify rsync://mirror.accum.se/mirror/wikimedia.org/other/kiwix/zim/wikipedia/wikipedia_en_all_maxi_2022-05.zim .`
-
-{ ... }:
+{ pkgs, ... }:
 {
-  sane.persist.sys.byStore.ext = [
-    { user = "colin"; group = "users"; path = "/var/lib/kiwix"; method = "bind"; }
-  ];
-
   sane.services.kiwix-serve = {
     enable = true;
     port = 8013;
-    zimPaths = [ "/var/lib/kiwix/wikipedia_en_all_maxi_2023-11.zim" ];
+    zimPaths = with pkgs.zimPackages; [
+      alpinelinux_en_all_maxi.zimPath
+      archlinux_en_all_maxi.zimPath
+      bitcoin_en_all_maxi.zimPath
+      devdocs_en_nix.zimPath
+      gentoo_en_all_maxi.zimPath
+      # khanacademy_en_all.zimPath  #< TODO: enable
+      openstreetmap-wiki_en_all_maxi.zimPath
+      psychonautwiki_en_all_maxi.zimPath
+      rationalwiki_en_all_maxi.zimPath
+      # wikipedia_en_100.zimPath
+      wikipedia_en_all_maxi.zimPath
+      # wikipedia_en_all_mini.zimPath
+      zimgit-food-preparation_en.zimPath
+      zimgit-medicine_en.zimPath
+      zimgit-post-disaster_en.zimPath
+      zimgit-water_en.zimPath
+    ];
   };
 
   services.nginx.virtualHosts."w.uninsane.org" = {

hosts/by-name/servo/services/komga.nix

@@ -1,8 +1,9 @@
-{ config, ... }:
+{ config, lib, ... }:
 let
   svc-cfg = config.services.komga;
   inherit (svc-cfg) user group port stateDir;
 in
+lib.mkIf false  #< 2024/09/30: disabled because i haven't used this for several months
 {
   sane.persist.sys.byStore.plaintext = [
     { inherit user group; mode = "0700"; path = stateDir; method = "bind"; }

hosts/by-name/servo/services/lemmy.nix

@@ -3,28 +3,26 @@
 # - <repo:LemmyNet/lemmy:docker/nginx.conf>
 # - <repo:LemmyNet/lemmy-ansible:templates/nginx.conf>
 
-{ config, lib, pkgs, ... }:
+{ lib, pkgs, ... }:
 let
-  inherit (builtins) toString;
-  inherit (lib) mkForce;
   uiPort = 1234;  # default ui port is 1234
   backendPort = 8536; # default backend port is 8536
   #^ i guess the "backend" port is used for federation?
   pict-rs = pkgs.pict-rs;
-  # pict-rs = pkgs.pict-rs.overrideAttrs (upstream: {
-  #   # as of v0.4.2, all non-GIF video is forcibly transcoded.
-  #   # that breaks lemmy, because of the request latency.
-  #   # and it eats up hella CPU.
-  #   # pict-rs is iffy around video altogether: mp4 seems the best supported.
-  #   # XXX: this patch no longer applies after 0.5.10 -> 0.5.11 update.
-  #   #      git log is hard to parse, but *suggests* that video is natively supported
-  #   #      better than in the 0.4.2 days, e.g. 5fd59fc5b42d31559120dc28bfef4e5002fb509e
-  #   #      "Change commandline flag to allow disabling video, since it is enabled by default"
-  #   postPatch = (upstream.postPatch or "") + ''
-  #     substituteInPlace src/validate.rs \
-  #       --replace-fail 'if transcode_options.needs_reencode() {' 'if false {'
-  #   '';
-  # });
+  # pict-rs configuration is applied in this order:
+  # - via toml
+  # - via env vars (overrides everything above)
+  # - via CLI flags (overrides everything above)
+  # some of the CLI flags have defaults, making it the only actual way to configure certain things even when docs claim otherwise.
+  # CLI args: <https://git.asonix.dog/asonix/pict-rs#user-content-running>
+  # TOML args: <https://git.asonix.dog/asonix/pict-rs/src/branch/main/pict-rs.toml>
+  toml = pkgs.formats.toml { };
+  tomlConfig = toml.generate "pict-rs.toml" pictrsConfig;
+  pictrsConfig = {
+    media.process_timeout = 120;
+    media.video.allow_audio = true;
+    media.video.max_frame_count = 30 * 60 * 60;
+  };
 in {
   services.lemmy = {
     enable = true;
@@ -52,8 +50,8 @@ in {
     # - postgresql://[user[:password]@][netloc][:port][/dbname][?param1=value1&...]
     # LEMMY_DATABASE_URL = "postgres://lemmy@/run/postgresql";  # connection to server on socket "/run/postgresql/.s.PGSQL.5432" failed: FATAL:  database "run/postgresql" does not exist
     # LEMMY_DATABASE_URL = "postgres://lemmy?host=/run/postgresql";  # no PostgreSQL user name specified in startup packet
-    # LEMMY_DATABASE_URL = mkForce "postgres://lemmy@?host=/run/postgresql";  # WORKS
-    LEMMY_DATABASE_URL = mkForce "postgres://lemmy@/lemmy?host=/run/postgresql";
+    # LEMMY_DATABASE_URL = lib.mkForce "postgres://lemmy@?host=/run/postgresql";  # WORKS
+    LEMMY_DATABASE_URL = lib.mkForce "postgres://lemmy@/lemmy?host=/run/postgresql";
   };
   users.groups.lemmy = {};
   users.users.lemmy = {
@@ -72,10 +70,14 @@ in {
     # fix to use a normal user so we can configure perms correctly
     # XXX(2024-07-28): this hasn't been rigorously tested:
     # possible that i've set something too strict and won't notice right away
-    serviceConfig.DynamicUser = mkForce false;
+    serviceConfig.DynamicUser = lib.mkForce false;
     serviceConfig.User = "lemmy";
     serviceConfig.Group = "lemmy";
 
+    # switch postgres from Requires -> Wants, so that postgres may restart without taking lemmy down with it.
+    requires = lib.mkForce [];
+    wants = [ "postgresql.service" ];
+
     # hardening (systemd-analyze security lemmy)
     # a handful of these are specified in upstream nixpkgs, but mostly not
     serviceConfig.LockPersonality = true;
@@ -138,18 +140,12 @@ in {
   #v DO NOT REMOVE: defaults to 0.3, instead of latest, so always need to explicitly set this.
   services.pict-rs.package = pict-rs;
 
-  # pict-rs configuration is applied in this order:
-  # - via toml
-  # - via env vars (overrides everything above)
-  # - via CLI flags (overrides everything above)
-  # some of the CLI flags have defaults, making it the only actual way to configure certain things even when docs claim otherwise.
-  # CLI args: <https://git.asonix.dog/asonix/pict-rs#user-content-running>
   systemd.services.pict-rs = {
     serviceConfig.ExecStart = lib.mkForce (lib.concatStringsSep " " [
-      "${lib.getBin pict-rs}/bin/pict-rs run"
-      "--media-video-max-frame-count" (builtins.toString (30*60*60))
-      "--media-process-timeout 120"
-      "--media-video-allow-audio"  # allow audio
+      (lib.getExe pict-rs)
+      "--config-file"
+      tomlConfig
+      "run"
     ]);
 
     # hardening (systemd-analyze security pict-rs)

hosts/by-name/servo/services/matrix/default.nix

@@ -12,7 +12,9 @@
 # - delete a notification destination by setting `kind` to `null` (otherwise, request is identical to above)
 #
 { config, lib, pkgs, ... }:
-
+let
+  ntfy = config.services.ntfy-sh.enable;
+in
 {
   imports = [
     ./discord-puppet.nix
@@ -68,21 +70,30 @@
     config.sops.secrets."matrix_synapse_secrets.yaml".path
   ];
 
-  systemd.services.matrix-synapse.postStart = ''
-    ACCESS_TOKEN=$(${pkgs.coreutils}/bin/cat ${config.sops.secrets.matrix_access_token.path})
-    TOPIC=$(${pkgs.coreutils}/bin/cat ${config.sops.secrets.ntfy-sh-topic.path})
+  # tune restart settings to ensure systemd doesn't disable it, and we don't overwhelm postgres
+  systemd.services.matrix-synapse.serviceConfig.RestartSec = 5;
+  systemd.services.matrix-synapse.serviceConfig.RestartMaxDelaySec = 20;
+  systemd.services.matrix-synapse.serviceConfig.StartLimitBurst = 120;
+  systemd.services.matrix-synapse.serviceConfig.RestartSteps = 3;
+  # switch postgres from Requires -> Wants, so that postgres may restart without taking matrix down with it.
+  systemd.services.matrix-synapse.requires = lib.mkForce [];
+  systemd.services.matrix-synapse.wants = [ "postgresql.service" ];
+
+  systemd.services.matrix-synapse.postStart = lib.optionalString ntfy ''
+    ACCESS_TOKEN=$(${lib.getExe' pkgs.coreutils "cat"} ${config.sops.secrets.matrix_access_token.path})
+    TOPIC=$(${lib.getExe' pkgs.coreutils "cat"} ${config.sops.secrets.ntfy-sh-topic.path})
 
     echo "ensuring ntfy push gateway"
-    ${pkgs.curl}/bin/curl \
+    ${lib.getExe pkgs.curl} \
       --header "Authorization: Bearer $ACCESS_TOKEN" \
       --data "{ \"app_display_name\": \"ntfy-adapter\", \"app_id\": \"ntfy.uninsane.org\", \"data\": { \"url\": \"https://ntfy.uninsane.org/_matrix/push/v1/notify\", \"format\": \"event_id_only\" }, \"device_display_name\": \"ntfy-adapter\", \"kind\": \"http\", \"lang\": \"en-US\", \"profile_tag\": \"\", \"pushkey\": \"$TOPIC\" }" \
       localhost:8008/_matrix/client/v3/pushers/set
 
     echo "registered push gateways:"
-    ${pkgs.curl}/bin/curl \
+    ${lib.getExe pkgs.curl} \
       --header "Authorization: Bearer $ACCESS_TOKEN" \
       localhost:8008/_matrix/client/v3/pushers \
-      | ${pkgs.jq}/bin/jq .
+      | ${lib.getExe pkgs.jq} .
   '';
 
 
@@ -159,5 +170,5 @@
     owner = config.users.users.matrix-synapse.name;
   };
   # provide access to ntfy-sh-topic secret
-  users.users.matrix-synapse.extraGroups = [ "ntfy-sh" ];
+  users.users.matrix-synapse.extraGroups = lib.optionals ntfy [ "ntfy-sh" ];
 }

hosts/by-name/servo/services/matrix/irc.nix

@@ -1,6 +1,6 @@
 # config docs:
 # - <https://github.com/matrix-org/matrix-appservice-irc/blob/develop/config.sample.yaml>
-{ config, lib, ... }:
+{ lib, ... }:
 
 let
   ircServer = { name, additionalAddresses ? [], ssl ? true, sasl ? true, port ? if ssl then 6697 else 6667 }: let
@@ -128,6 +128,7 @@ in
 
     ircService = {
       logging.level = "warn";  # "error", "warn", "info", "debug"
+      mediaProxy.publicUrl = "https://irc.matrix.uninsane.org/media";
       servers = {
         "irc.esper.net" = ircServer {
           name = "esper";
@@ -153,8 +154,17 @@ in
           # notable channels:
           # - #sxmo
           # - #sxmo-offtopic
+          # supposedly also available at <irc://37lnq2veifl4kar7.onion:6667/> (unofficial)
         };
         "irc.rizon.net" = ircServer { name = "Rizon"; };
+        # "irc.sdf.org" = ircServer {
+        #   # XXX(2024-11-06): seems it can't connect. "matrix-appservice-irc: WARN:Provisioner Provisioner only handles text 'yes'/'y' (from BASHy2-EU on irc.sdf.org)"
+        #   # use instead? <https://lemmy.sdf.org/c/sdfpubnix>
+        #   name = "sdf";
+        #   # sasl = false;
+        #   # notable channels (see: <https://sdf.org/?tutorials/irc-channels>)
+        #   # - #sdf
+        # };
         "wigle.net" = ircServer {
           name = "WiGLE";
           ssl = false;
@@ -168,4 +178,16 @@ in
     # the service actively uses at least one of these, and both of them are fairly innocuous
     SystemCallFilter = lib.mkForce "~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io @setuid @swap";
   };
+
+  services.nginx.virtualHosts."irc.matrix.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+    locations."/media" = {
+      proxyPass = "http://127.0.0.1:11111";
+    };
+  };
+
+  sane.dns.zones."uninsane.org".inet = {
+    CNAME."irc.matrix" = "native";
+  };
 }

hosts/by-name/servo/services/minidlna.nix

@@ -0,0 +1,39 @@
+# - `man 5 minidlna.conf`
+# - `man 8 minidlnad`
+#
+# this is an extremely simple (but limited) DLNA server:
+# - no web UI
+# - no runtime configuration -- just statically configure media directories instead
+# - no transcoding
+# compatibility:
+# - LG TV: music: all working
+# - LG TV: videos: mixed. i can't see the pattern; HEVC works; H.264 sometimes works.
+{ lib, ... }:
+lib.mkIf false  #< XXX(2024-11-17): WORKS, but i'm trying gerbera instead for hopefully better transcoding
+{
+  sane.ports.ports."1900" = {
+    protocol = [ "udp" ];
+    visibleTo.lan = true;
+    description = "colin-upnp-for-minidlna";
+  };
+  sane.ports.ports."8200" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    description = "colin-minidlna-http";
+  };
+
+  services.minidlna.enable = true;
+
+  services.minidlna.settings = {
+    media_dir = [
+      # A/V/P to restrict a directory to audio/video/pictures
+      "A,/var/media/Music"
+      "V,/var/media/Videos/Film"
+      # "V,/var/media/Videos/Milkbags"
+      "V,/var/media/Videos/Shows"
+    ];
+    notify_interval = 60;
+  };
+
+  users.users.minidlna.extraGroups = [ "media" ];
+}

hosts/by-name/servo/services/mumble.nix

@@ -0,0 +1,66 @@
+# murmur is the server component of mumble.
+# - docs: <https://www.mumble.info/documentation/>
+# - config docs: <https://www.mumble.info/documentation/administration/config-file/>
+#
+# default port is 64738 (UDP and TCP)
+#
+# FIRST-RUN:
+# - login from mumble client as `SuperUser`, password taken from `journalctl -u murmur`.
+# - login from another machine and right click on self -> 'Register'
+# - as SuperUser, right click on server root -> edit
+#   - Groups tab: select "admin", then add the other registered user to the group.
+# - log out as SuperUser and manage the server using that other user now.
+#
+# USAGE:
+# - 'auth' group = any user who has registered a cert with the server.
+{ ... }:
+{
+  sane.persist.sys.byStore.private = [
+    { user = "murmur"; group = "murmur"; mode = "0700"; path = "/var/lib/murmur"; method = "bind"; }
+  ];
+
+  services.murmur.enable = true;
+  services.murmur.welcometext = "welcome to Colin's mumble voice chat server";
+  # max bandwidth (bps) **per user**. i believe this affects both voice and uploads?
+  # mumble defaults to 558000, but nixos service defaults to 72000.
+  services.murmur.bandwidth = 558000;
+  services.murmur.imgMsgLength = 8 * 1024 * 1024;
+
+  services.murmur.sslCert = "/var/lib/acme/mumble.uninsane.org/fullchain.pem";
+  services.murmur.sslKey = "/var/lib/acme/mumble.uninsane.org/key.pem";
+  services.murmur.sslCa = "/etc/ssl/certs/ca-bundle.crt";
+
+  # allow clients on the LAN to discover this server
+  services.murmur.bonjour = true;
+
+  # mumble has a public server listing.
+  # my server doesn't associate with that registry (unless i specify registerPassword).
+  # however these settings appear to affect how the server presents itself to clients, regardless of registration.
+  services.murmur.registerName = "mumble.uninsane.org";
+  services.murmur.registerUrl = "https://mumble.uninsane.org";
+  services.murmur.registerHostname = "mumble.uninsane.org";
+
+  # defaultchannel=ID makes it so that unauthenticated users are placed in some specific channel when they join
+  services.murmur.extraConfig = ''
+    defaultchannel=2
+  '';
+
+  users.users.murmur.extraGroups = [
+    "nginx"  # provide access to certs
+  ];
+  services.nginx.virtualHosts."mumble.uninsane.org" = {
+    # allow ACME to procure a cert via nginx for this domain
+    enableACME = true;
+  };
+
+  sane.dns.zones."uninsane.org".inet = {
+    CNAME."mumble" = "native";
+  };
+
+  sane.ports.ports."64738" = {
+    protocol = [ "tcp" "udp" ];
+    visibleTo.lan = true;
+    visibleTo.doof = true;
+    description = "colin-mumble";
+  };
+}

hosts/by-name/servo/services/nginx.nix

@@ -83,6 +83,28 @@ in
     # unversioned files
     locations."@fallback" = {
       root = "/var/www/sites/uninsane.org";
+      extraConfig = ''
+        # instruct Google to not index these pages.
+        # see: <https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#xrobotstag>
+        add_header X-Robots-Tag 'none, noindex, nofollow';
+
+        # best-effort attempt to block archive.org from archiving these pages.
+        # reply with 403: Forbidden
+        # User Agent is *probably* "archive.org_bot"; maybe used to be "ia_archiver"
+        # source: <https://archive.org/details/archive.org_bot>
+        # additional UAs: <https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker>
+        #
+        # validate with: `curl -H 'User-Agent: "bot;archive.org_bot;like: something else"' -v https://uninsane.org/dne`
+        if ($http_user_agent ~* "(?:\b)archive.org_bot(?:\b)") {
+          return 403;
+        }
+        if ($http_user_agent ~* "(?:\b)archive.org(?:\b)") {
+          return 403;
+        }
+        if ($http_user_agent ~* "(?:\b)ia_archiver(?:\b)") {
+          return 403;
+        }
+      '';
     };
 
     # uninsane.org/share/foo => /var/www/sites/uninsane.org/share/foo.
@@ -235,7 +257,7 @@ in
   # to accept it.
   system.activationScripts.generate-x509-self-signed.text = ''
     mkdir -p /var/www/certs/wildcard
-    test -f /var/www/certs/wildcard/key.pem || ${pkgs.openssl}/bin/openssl \
+    test -f /var/www/certs/wildcard/key.pem || ${lib.getExe pkgs.openssl} \
       req -x509 -newkey rsa:4096 \
       -keyout /var/www/certs/wildcard/key.pem \
       -out /var/www/certs/wildcard/cert.pem \

hosts/by-name/servo/services/ntfy/default.nix

@@ -1,12 +1,12 @@
 # ntfy: UnifiedPush notification delivery system
 # - used to get push notifications out of Matrix and onto a Phone (iOS, Android, or a custom client)
-{ config, ... }:
+{ config, lib, ... }:
 {
   imports = [
     ./ntfy-waiter.nix
     ./ntfy-sh.nix
   ];
-  sops.secrets."ntfy-sh-topic" = {
+  sops.secrets."ntfy-sh-topic" = lib.mkIf config.services.ntfy-sh.enable {
     mode = "0440";
     owner = config.users.users.ntfy-sh.name;
     group = config.users.users.ntfy-sh.name;

hosts/by-name/servo/services/ntfy/ntfy-sh.nix

@@ -29,6 +29,7 @@ let
   # at the IP layer, to enable e.g. wake-on-lan.
   altPort = 2587;
 in
+lib.mkIf false  #< 2024/09/30: disabled because i haven't used it in several months
 {
   sane.persist.sys.byStore.private = [
     # not 100% necessary to persist this, but ntfy does keep a 12hr (by default) cache
@@ -58,7 +59,7 @@ in
     # note that this will fail upon first run, i.e. before ntfy has created its db.
     # just restart the service.
     topic=$(cat ${config.sops.secrets.ntfy-sh-topic.path})
-    ${pkgs.ntfy-sh}/bin/ntfy access everyone "$topic" read-write
+    ${lib.getExe' pkgs.ntfy-sh "ntfy"} access everyone "$topic" read-write
   '';
 
 

hosts/by-name/servo/services/ntfy/ntfy-waiter.nix

@@ -14,7 +14,7 @@ let
     silence = port - portLow;
     flags = lib.optional cfg.verbose "--verbose";
     cli = [
-      "${cfg.package}/bin/ntfy-waiter"
+      (lib.getExe cfg.package)
       "--port"
       "${builtins.toString port}"
       "--silence"
@@ -31,7 +31,7 @@ let
         ExecStart = lib.concatStringsSep " " cli;
       };
       after = [ "network.target" ];
-      wantedBy = [ "default.target" ];
+      wantedBy = [ "ntfy-sh.service" ];
     };
   };
 in
@@ -39,7 +39,7 @@ in
   options = with lib; {
     sane.ntfy-waiter.enable = mkOption {
       type = types.bool;
-      default = true;
+      default = config.services.ntfy-sh.enable;
     };
     sane.ntfy-waiter.verbose = mkOption {
       type = types.bool;

hosts/by-name/servo/services/ollama.nix

@@ -1,23 +0,0 @@
-# ollama: <https://github.com/ollama/ollama>
-# use: `ollama run llama3.1`
-# or: `ollama run llama3.1:70b`
-# or use a remote session: <https://github.com/ggozad/oterm>
-{ lib, ... }:
-lib.mkIf false  #< WIP
-{
-  sane.persist.sys.byStore.plaintext = [
-    { user = "ollama"; group = "ollama"; path = "/var/lib/ollama"; method = "bind"; }
-  ];
-  services.ollama.enable = true;
-  services.ollama.user = "ollama";
-  services.ollama.group = "ollama";
-
-  users.groups.ollama = {};
-
-  users.users.ollama = {
-    group = "ollama";
-    isSystemUser = true;
-  };
-
-  systemd.services.ollama.serviceConfig.DynamicUser = lib.mkForce false;
-}

hosts/by-name/servo/services/pleroma.nix

@@ -10,7 +10,7 @@
 { config, lib, pkgs, ... }:
 
 let
-  logLevel = "warn";
+  logLevel = "warning";
   # logLevel = "debug";
 in
 {
@@ -46,7 +46,7 @@ in
     config :pleroma, Pleroma.Emails.Mailer,
       enabled: true,
       adapter: Swoosh.Adapters.Sendmail,
-      cmd_path: "${pkgs.postfix}/bin/sendmail"
+      cmd_path: "${lib.getExe' pkgs.postfix "sendmail"}"
 
     config :pleroma, Pleroma.User,
       restricted_nicknames: [ "admin", "uninsane", "root" ]
@@ -88,6 +88,12 @@ in
     # strip metadata from uploaded images
     config :pleroma, Pleroma.Upload, filters: [Pleroma.Upload.Filter.Exiftool.StripLocation]
 
+    # fix log spam: <https://git.pleroma.social/pleroma/pleroma/-/issues/1659>
+    # specifically, remove LAN addresses from `reserved`
+    config :pleroma, Pleroma.Web.Plugs.RemoteIp,
+      enabled: true,
+      reserved: ["127.0.0.0/8", "::1/128", "fc00::/7", "172.16.0.0/12"]
+
     # TODO: GET /api/pleroma/captcha is broken
     # there was a nixpkgs PR to fix this around 2022/10 though.
     config :pleroma, Pleroma.Captcha,
@@ -136,9 +142,10 @@ in
     # something inside pleroma invokes `sh` w/o specifying it by path, so this is needed to allow pleroma to start
     pkgs.bash
     # used by Pleroma to strip geo tags from uploads
-    config.sane.programs.exiftool.package
+    pkgs.exiftool
+    # config.sane.programs.exiftool.package  #< XXX(2024-10-20): breaks image uploading
     # i saw some errors when pleroma was shutting down about it not being able to find `awk`. probably not critical
-    config.sane.programs.gawk.package
+    # config.sane.programs.gawk.package
     # needed for email operations like password reset
     pkgs.postfix
   ];
@@ -153,7 +160,7 @@ in
     # possible that i've set something too strict and won't notice right away
     # make sure to test:
     # - image/media uploading
-    serviceConfig.CapabilityBoundingSet = "~CAP_SYS_ADMIN";  #< TODO: reduce this. try: CAP_SYS_NICE CAP_DAC_READ_SEARCH CAP_SYS_CHROOT CAP_SETGID CAP_SETUID
+    serviceConfig.CapabilityBoundingSet = lib.mkForce [ "" "" ];  # nixos default is `~CAP_SYS_ADMIN`
     serviceConfig.LockPersonality = true;
     serviceConfig.NoNewPrivileges = true;
     serviceConfig.MemoryDenyWriteExecute = true;
@@ -199,34 +206,7 @@ in
       recommendedProxySettings = true;
       # documented: https://git.pleroma.social/pleroma/pleroma/-/blob/develop/installation/pleroma.nginx
       extraConfig = ''
-        # XXX colin: this block is in the nixos examples: i don't understand all of it
-        add_header 'Access-Control-Allow-Origin' '*' always;
-        add_header 'Access-Control-Allow-Methods' 'POST, PUT, DELETE, GET, PATCH, OPTIONS' always;
-        add_header 'Access-Control-Allow-Headers' 'Authorization, Content-Type, Idempotency-Key' always;
-        add_header 'Access-Control-Expose-Headers' 'Link, X-RateLimit-Reset, X-RateLimit-Limit, X-RateLimit-Remaining, X-Request-Id' always;
-        if ($request_method = OPTIONS) {
-            return 204;
-        }
-
-        add_header X-XSS-Protection "1; mode=block";
-        add_header X-Permitted-Cross-Domain-Policies none;
-        add_header X-Frame-Options DENY;
-        add_header X-Content-Type-Options nosniff;
-        add_header Referrer-Policy same-origin;
-        add_header X-Download-Options noopen;
-
-        # proxy_http_version 1.1;
-        # proxy_set_header Upgrade $http_upgrade;
-        # proxy_set_header Connection "upgrade";
-        # # proxy_set_header Host $http_host;
-        # proxy_set_header Host $host;
-        # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-
-        # colin: added this due to Pleroma complaining in its logs
-        # proxy_set_header X-Real-IP $remote_addr;
-        # proxy_set_header X-Forwarded-Proto $scheme;
-
-        # NB: this defines the maximum upload size
+        # client_max_body_size defines the maximum upload size
         client_max_body_size 16m;
       '';
     };

hosts/by-name/servo/services/postgresql/default.nix

@@ -29,9 +29,10 @@ in
   # - as `sudo su postgres`:
   #   - `cd /var/lib/postgreql`
   #   - `psql -f state.sql`
+  #     (for a compressed dump: `gunzip --stdout state.sql.gz | psql`)
   # - restart dependent services (maybe test one at a time)
 
-  services.postgresql.package = pkgs.postgresql_15;
+  services.postgresql.package = pkgs.postgresql_16;
 
 
   # XXX colin: for a proper deploy, we'd want to include something for Pleroma here too.
@@ -44,34 +45,46 @@ in
   #     LC_CTYPE = "C";
   # '';
 
-  # perf tuning
-  # - for recommended values see: <https://pgtune.leopard.in.ua/>
-  # - for official docs (sparse), see: <https://www.postgresql.org/docs/11/config-setting.html#CONFIG-SETTING-CONFIGURATION-FILE>
   services.postgresql.settings = {
-    # DB Version: 15
+    # perf tuning
+    # - for recommended values see: <https://pgtune.leopard.in.ua/>
+    # - for official docs (sparse), see: <https://www.postgresql.org/docs/11/config-setting.html#CONFIG-SETTING-CONFIGURATION-FILE>
+    # DB Version: 16
     # OS Type: linux
     # DB Type: web
-    # Total Memory (RAM): 32 GB
+    # vvv artificially constrained because the server's resources are shared across maaany services
+    # Total Memory (RAM): 12 GB
     # CPUs num: 12
     # Data Storage: ssd
     max_connections = 200;
-    shared_buffers = "8GB";
-    effective_cache_size = "24GB";
-    maintenance_work_mem = "2GB";
+    shared_buffers = "3GB";
+    effective_cache_size = "9GB";
+    maintenance_work_mem = "768MB";
     checkpoint_completion_target = 0.9;
     wal_buffers = "16MB";
     default_statistics_target = 100;
     random_page_cost = 1.1;
     effective_io_concurrency = 200;
-    work_mem = "10485kB";
+    work_mem = "3932kB";
     min_wal_size = "1GB";
     max_wal_size = "4GB";
     max_worker_processes = 12;
     max_parallel_workers_per_gather = 4;
     max_parallel_workers = 12;
     max_parallel_maintenance_workers = 4;
+
+    # DEBUG OPTIONS:
+    log_min_messages = "DEBUG1";
   };
 
+  # regulate the restarts, so that systemd never disables it
+  systemd.services.postgresql.serviceConfig.Restart = "on-failure";
+  systemd.services.postgresql.serviceConfig.RestartSec = 2;
+  systemd.services.postgresql.serviceConfig.RestartMaxDelaySec = 10;
+  systemd.services.postgresql.serviceConfig.RestartSteps = 4;
+  systemd.services.postgresql.serviceConfig.StartLimitBurst = 120;
+  # systemd.services.postgresql.serviceConfig.TimeoutStartSec = "14400s";  #< 14400 = 4 hours; recoveries are long
+
   # daily backups to /var/backup
   services.postgresqlBackup.enable = true;
 

hosts/by-name/servo/services/postgresql/recollate.sh

@@ -0,0 +1,81 @@
+#!/bin/sh
+# source: <https://gist.githubusercontent.com/troykelly/616df024050dd50744dde4a9579e152e/raw/fe84e53cedf0caa6903604894454629a15867439/reindex_and_refresh_collation.sh>
+#
+# run this whenever postgres complains like:
+# > WARNING:  database "gitea" has a collation version mismatch
+# > DETAIL:  The database was created using collation version 2.39, but the operating system provides version 2.40.
+# > HINT:  Rebuild all objects in this database that use the default collation and run ALTER DATABASE gitea REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
+#
+# this script checks which databases are in need of a collation update,
+# and re-collates them as appropriate.
+# invoking this script should have low perf impact in the non-upgrade case,
+# so safe to do this as a cron job.
+#
+# invoke as postgres user
+
+log_info() {
+  >&2 echo "$@"
+}
+
+list_databases() {
+  log_info "Retrieving list of databases from the PostgreSQL server..."
+  psql --dbname="postgres" -Atc \
+    "SELECT datname FROM pg_database WHERE datistemplate = false"
+}
+
+refresh_collation_version() {
+  local db=$1
+  log_info "Refreshing collation version for database: $db..."
+  psql --dbname="$db" -c \
+    "ALTER DATABASE \"$db\" REFRESH COLLATION VERSION;"
+}
+
+check_collation_mismatches() {
+  local error=
+  log_info "Checking for collation mismatches in all databases..."
+  # Loop through each database and check for mismatching collations in table columns.
+  while IFS= read -r db; do
+    if [ -n "$db" ]; then
+      log_info "Checking database: $db for collation mismatches..."
+      local mismatches=$(psql --dbname="$db" -Atc \
+        "SELECT 'Mismatch in table ' || table_name || ' column ' || column_name || ' with collation ' || collation_name
+         FROM information_schema.columns
+         WHERE collation_name IS NOT NULL AND collation_name <> 'default' AND table_schema = 'public'
+         EXCEPT
+         SELECT 'No mismatch - default collation of ' || datcollate || ' used.'
+         FROM pg_database WHERE datname = '$db';"
+      )
+      if [ -z "$mismatches" ]; then
+        log_info "No collation mismatches found in database: $db"
+      else
+        # Print an informational message to stderr.
+        log_info "Collation mismatches found in database: $db:"
+        log_info "$mismatches"
+        error=1
+      fi
+    fi
+  done
+
+  if [ -n "$error" ]; then
+    exit 1
+  fi
+}
+
+log_info "Starting the reindexing and collation refresh process for all databases..."
+
+databases=$(list_databases)
+
+if [ -z "$databases" ]; then
+  log_info "No databases found for reindexing or collation refresh. Please check connection details to PostgreSQL server."
+  exit 1
+fi
+
+for db in $databases; do
+  refresh_collation_version "$db"
+done
+
+# Checking for collation mismatches after reindexing and collation refresh.
+# Pass the list of databases to the check_collation_mismatches function through stdin.
+echo "$databases" | check_collation_mismatches
+
+log_info "Reindexing and collation refresh process completed."

hosts/by-name/servo/services/prosody/default.nix

@@ -49,7 +49,7 @@
 # - disable or fix bosh (jabber over http):
 #   - "certmanager: No certificate/key found for client_https port 0"
 
-{ lib, pkgs, ... }:
+{ config, lib, pkgs, ... }:
 
 let
   # enables very verbose logging
@@ -104,6 +104,7 @@ in
   users.users.prosody.extraGroups = [
     "nginx"  # provide access to certs
     "ntfy-sh"  # access to secret ntfy topic
+    "turnserver"  # to access the coturn shared secret
   ];
 
   security.acme.certs."uninsane.org".extraDomainNames = [
@@ -149,14 +150,8 @@ in
   # pointing it to /var/lib/acme doesn't quite work because it expects the private key
   # to be named `privkey.pem` instead of acme's `key.pem`
   # <https://prosody.im/doc/certificates#automatic_location>
-  sane.fs."/etc/prosody/certs/uninsane.org/fullchain.pem" = {
-    symlink.target = "/var/lib/acme/uninsane.org/fullchain.pem";
-    wantedBeforeBy = [ "prosody.service" ];
-  };
-  sane.fs."/etc/prosody/certs/uninsane.org/privkey.pem" = {
-    symlink.target = "/var/lib/acme/uninsane.org/key.pem";
-    wantedBeforeBy = [ "prosody.service" ];
-  };
+  environment.etc."prosody/certs/uninsane.org/fullchain.pem".source = "/var/lib/acme/uninsane.org/fullchain.pem";
+  environment.etc."prosody/certs/uninsane.org/privkey.pem".source = "/var/lib/acme/uninsane.org/key.pem";
 
   services.prosody = {
     enable = true;
@@ -242,6 +237,7 @@ in
       # legacy coturn integration
       # see: <https://modules.prosody.im/mod_turncredentials.html>
       # "turncredentials"
+    ] ++ lib.optionals config.services.ntfy-sh.enable [
       "sane_ntfy"
     ] ++ lib.optionals enableDebug [
       "stanza_debug"  #< logs EVERY stanza as debug: <https://prosody.im/doc/modules/mod_stanza_debug>
@@ -273,18 +269,34 @@ in
       s2s_direct_tls_ports = { 5270 }
 
       turn_external_host = "turn.uninsane.org"
-      turn_external_secret = readAll("/var/lib/coturn/shared_secret.bin")
+      turn_external_secret = readAll("/run/secrets/coturn_shared_secret")
       -- turn_external_user = "prosody"
 
       -- legacy mod_turncredentials integration
       -- turncredentials_host = "turn.uninsane.org"
-      -- turncredentials_secret = readAll("/var/lib/coturn/shared_secret.bin")
-
-      ntfy_binary = "${pkgs.ntfy-sh}/bin/ntfy"
-      ntfy_topic = readAll("/run/secrets/ntfy-sh-topic")
+      -- turncredentials_secret = readAll("/run/secrets/coturn_shared_secret")
 
       -- s2s_require_encryption = true
       -- c2s_require_encryption = true
+    '' + lib.optionalString config.services.ntfy-sh.enable ''
+      ntfy_binary = "${lib.getExe' pkgs.ntfy-sh "ntfy"}"
+      ntfy_topic = readAll("/run/secrets/ntfy-sh-topic")
     '';
   };
+
+  systemd.services.prosody = {
+    # hardening (systemd-analyze security prosody)
+    serviceConfig.LockPersonality = true;
+    serviceConfig.NoNewPrivileges = true;
+    serviceConfig.PrivateUsers = true;
+    serviceConfig.ProcSubset = "pid";
+    serviceConfig.ProtectClock = true;
+    serviceConfig.ProtectKernelLogs = true;
+    serviceConfig.ProtectProc = "invisible";
+    serviceConfig.ProtectSystem = "strict";
+    serviceConfig.RemoveIPC = true;
+    serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
+    serviceConfig.SystemCallArchitectures = "native";
+    serviceConfig.SystemCallFilter = [ "@system-service" "~@privileged" "~@resources" ];
+  };
 }

hosts/by-name/servo/services/slskd.nix

@@ -34,7 +34,7 @@
     forceSSL = true;
     enableACME = true;
     locations."/" = {
-      proxyPass = "http://${config.sane.netns.ovpns.netnsVethIpv4}:5030";
+      proxyPass = "http://${config.sane.netns.ovpns.veth.netns.ipv4}:5030";
       proxyWebsockets = true;
     };
   };
@@ -73,7 +73,10 @@
   systemd.services.slskd = {
     # run this behind the OVPN static VPN
     serviceConfig.NetworkNamespacePath = "/run/netns/ovpns";
-    serviceConfig.ExecStartPre = [ "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect ${config.sane.netns.ovpns.netnsPubIpv4}" ];  # abort if public IP is not as expected
+    serviceConfig.ExecStartPre = [
+      # abort if public IP is not as expected
+      "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect ${config.sane.netns.ovpns.wg.address.ipv4}"
+    ];
 
     serviceConfig.Restart = lib.mkForce "always";  # exits "success" when it fails to connect to soulseek server
     serviceConfig.RestartSec = "60s";

hosts/by-name/servo/services/transmission/default.nix

@@ -58,8 +58,8 @@ in
     # DOCUMENTATION/options list: <https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md#options>
 
     # message-level = 3;  #< enable for debug logging. 0-3, default is 2.
-    # ovpns.netnsVethIpv4 => allow rpc only from the root servo ns. it'll tunnel things to the net, if need be.
-    rpc-bind-address = config.sane.netns.ovpns.netnsVethIpv4;
+    # ovpns.veth.netns.ipv4 => allow rpc only from the root servo ns. it'll tunnel things to the net, if need be.
+    rpc-bind-address = config.sane.netns.ovpns.veth.netns.ipv4;
     #rpc-host-whitelist = "bt.uninsane.org";
     #rpc-whitelist = "*.*.*.*";
     rpc-authentication-required = true;
@@ -70,7 +70,7 @@ in
     rpc-whitelist-enabled = false;
 
     # force behind ovpns in case the NetworkNamespace fails somehow
-    bind-address-ipv4 = config.sane.netns.ovpns.netnsPubIpv4;
+    bind-address-ipv4 = config.sane.netns.ovpns.wg.address.ipv4;
     port-forwarding-enabled = false;
 
     # hopefully, make the downloads world-readable
@@ -104,16 +104,17 @@ in
     # - TR_TORRENT_NAME - Name of torrent (not filename)
     # - TR_TORRENT_TRACKERS - A comma-delimited list of the torrent's trackers' announce URLs
     script-torrent-done-enabled = true;
-    script-torrent-done-filename = "${torrent-done}/bin/torrent-done";
+    script-torrent-done-filename = lib.getExe torrent-done;
   };
 
+  # run this behind the OVPN static VPN
+  sane.netns.ovpns.services = [ "transmission" ];
   systemd.services.transmission = {
-    after = [ "wireguard-wg-ovpns.service" ];
-    partOf = [ "wireguard-wg-ovpns.service" ];
     environment.TR_DEBUG = "1";
-    # run this behind the OVPN static VPN
-    serviceConfig.NetworkNamespacePath = "/run/netns/ovpns";
-    serviceConfig.ExecStartPre = [ "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect ${config.sane.netns.ovpns.netnsPubIpv4}" ];  # abort if public IP is not as expected
+    serviceConfig.ExecStartPre = [
+      # abort if public IP is not as expected
+      "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect ${config.sane.netns.ovpns.wg.address.ipv4}"
+    ];
 
     serviceConfig.Restart = "on-failure";
     serviceConfig.RestartSec = "30s";
@@ -138,7 +139,7 @@ in
   systemd.services.backup-torrents = {
     description = "archive torrents to storage not owned by transmission";
     script = ''
-      ${pkgs.rsync}/bin/rsync -arv /var/lib/transmission/.config/transmission-daemon/torrents/ /var/backup/torrents/
+      ${lib.getExe pkgs.rsync} -arv /var/lib/transmission/.config/transmission-daemon/torrents/ /var/backup/torrents/
     '';
   };
   systemd.timers.backup-torrents = {
@@ -157,7 +158,7 @@ in
     # inherit kTLS;
     locations."/" = {
       # proxyPass = "http://ovpns.uninsane.org:9091";
-      proxyPass = "http://${config.sane.netns.ovpns.netnsVethIpv4}:9091";
+      proxyPass = "http://${config.sane.netns.ovpns.veth.netns.ipv4}:9091";
     };
   };
 

hosts/by-name/servo/users/default.nix

@@ -0,0 +1,6 @@
+{ ... }:
+{
+  imports = [
+    ./shelvacu.nix
+  ];
+}

hosts/by-name/servo/users/shelvacu.nix

@@ -0,0 +1,65 @@
+{ lib, pkgs, ... }:
+{
+  users.users.shelvacu = {
+    isNormalUser = true;
+    home = "/home/shelvacu";
+    subUidRanges = [
+      { startUid=300000; count=1; }
+    ];
+    group = "users";
+    initialPassword = lib.mkDefault "";
+    shell = pkgs.bash;
+
+    openssh.authorizedKeys.keys = [
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKoy1TrmfhBGWtVedgOM1FB1oD2UdodN3LkBnnLx6Tug compute-deck"
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAxAFFxQMXAgi+0cmGaNE/eAkVfEl91wafUqFIuAkI5I compute-deck-root"
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINQ2c0GzlVMjV06CS7bWbCaAbzG2+7g5FCg/vClJPe0C fw"
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGHLPOxRd68+DJ/bYmqn0wsgwwIcMSMyuU1Ya16hCb/m fw-root"
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOre0FnYDm3arsFj9c/l5H2Q8mdmv7kmvq683pL4heru legtop"
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINznGot+L8kYoVQqdLV/R17XCd1ILMoDCILOg+I3s5wC pixel9pro-nod"
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDcRDekd8ZOYfQS5X95/yNof3wFYIbHqWeq4jY0+ywQX pro1x-nod"
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJNFbzt0NHVTaptBI38YtwLG+AsmeNYy0Nr5yX2zZEPE root@vacuInstaller toptop-root"
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICVeSzDkGTueZijB0xUa08e06ovAEwwZK/D+Cc7bo91g triple-dezert"
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOtwtao/TXbiuQOYJbousRPVesVcb/2nP0PCFUec0Nv8 triple-dezert-root"
+    ];
+  };
+
+  security.sudo.extraRules = [
+    {
+      users = [ "shelvacu" ];
+      runAs = "postgres";
+      commands = [
+        {
+          command = "ALL";
+          options = [ "NOPASSWD" ];
+        }
+      ];
+    }
+  ];
+
+  security.polkit.extraConfig = ''
+    // allow:
+    // - systemctl restart|start|stop SERVICE
+    polkit.addRule(function(action, subject) {
+      if (subject.user == "shelvacu" && action.id == "org.freedesktop.systemd1.manage-units") {
+        switch (action.lookup("verb")) {
+          // case "cancel":
+          // case "reenable":
+          case "restart":
+          // case "reload":
+          // case "reload-or-restart":
+          case "start":
+          case "stop":
+          // case "try-reload-or-restart":
+          // case "try-restart":
+            return polkit.Result.YES;
+          default:
+        }
+      }
+    })
+  '';
+
+  sane.persist.sys.byStore.private = [
+    { path = "/home/shelvacu/persist"; user = "shelvacu"; group = "users"; mode = "0700"; }
+  ];
+}

hosts/common/boot.nix

@@ -3,13 +3,12 @@
   boot.initrd.supportedFilesystems = [ "ext4" "btrfs" "ext2" "ext3" "vfat" ];
   # useful emergency utils
   boot.initrd.extraUtilsCommands = ''
-    copy_bin_and_libs ${pkgs.btrfs-progs}/bin/btrfstune
-    copy_bin_and_libs ${pkgs.util-linux}/bin/{cfdisk,lsblk,lscpu}
-    copy_bin_and_libs ${pkgs.gptfdisk}/bin/{cgdisk,gdisk}
-    copy_bin_and_libs ${pkgs.smartmontools}/bin/smartctl
-    copy_bin_and_libs ${pkgs.e2fsprogs}/bin/resize2fs
-  '' + lib.optionalString pkgs.stdenv.hostPlatform.isx86_64 ''
-    copy_bin_and_libs ${pkgs.nvme-cli}/bin/nvme  # doesn't cross compile
+    copy_bin_and_libs ${lib.getExe' pkgs.btrfs-progs "btrfstune"}
+    copy_bin_and_libs ${lib.getExe' pkgs.util-linux "{cfdisk,lsblk,lscpu}"}
+    copy_bin_and_libs ${lib.getExe' pkgs.gptfdisk "{cgdisk,gdisk}"}
+    copy_bin_and_libs ${lib.getExe' pkgs.smartmontools "smartctl"}
+    copy_bin_and_libs ${lib.getExe' pkgs.e2fsprogs "resize2fs"}
+    copy_bin_and_libs ${lib.getExe pkgs.nvme-cli}
   '';
   boot.kernelParams = [
     "boot.shell_on_fail"
@@ -25,10 +24,10 @@
 
   # moby has to run recent kernels (defined elsewhere).
   # meanwhile, kernel variation plays some minor role in things like sandboxing (landlock) and capabilities.
+  # - as of 2024/08/xx, my boot fails on 6.6, but works on 6.9 and (probably; recently) 6.8.
   # simpler to keep near the latest kernel on all devices,
   # and also makes certain that any weird system-level bugs i see aren't likely to be stale kernel bugs.
-  # servo needs zfs though, which doesn't support every kernel.
-  boot.kernelPackages = lib.mkDefault pkgs.zfs.latestCompatibleLinuxPackages;
+  boot.kernelPackages = lib.mkDefault pkgs.linuxPackages_latest;
 
   # hack in the `boot.shell_on_fail` arg since that doesn't always seem to work.
   boot.initrd.preFailCommands = "allowShell=1";

hosts/common/default.nix

@@ -1,9 +1,9 @@
-{ config, lib, pkgs, ... }:
+{ lib, pkgs, ... }:
 {
   imports = [
     ./boot.nix
     ./feeds.nix
-    ./fs.nix
+    ./fs
     ./home
     ./hosts.nix
     ./ids.nix
@@ -14,6 +14,7 @@
     ./programs
     ./quirks.nix
     ./secrets.nix
+    ./snapper.nix
     ./ssh.nix
     ./systemd.nix
     ./users
@@ -31,6 +32,15 @@
   sane.programs.sysadminUtils.enableFor.system = lib.mkDefault true;
   sane.programs.consoleUtils.enableFor.user.colin = lib.mkDefault true;
 
+  services.buffyboard.enable = true;
+  services.buffyboard.settings.theme.default = "pmos-light";
+  # services.buffyboard.settings.quirks.fbdev_force_refresh = true;
+  services.buffyboard.extraFlags = [ "--verbose" ];
+
+  # irqbalance monitors interrupt count (as a daemon) and assigns high-frequency interrupts to different CPUs.
+  # that reduces contention between simultaneously-fired interrupts.
+  services.irqbalance.enable = true;
+
   # time.timeZone = "America/Los_Angeles";
   time.timeZone = "Etc/UTC";  # DST is too confusing for me => use a stable timezone
 
@@ -41,7 +51,7 @@
       # source: <https://github.com/luishfonseca/dotfiles/blob/32c10e775d9ec7cc55e44592a060c1c9aadf113e/modules/upgrade-diff.nix>
       # modified to not error on boot (when /run/current-system doesn't exist)
       if [ -d /run/current-system ]; then
-        ${pkgs.nvd}/bin/nvd --nix-bin-dir=${pkgs.nix}/bin diff /run/current-system "$systemConfig"
+        ${lib.getExe pkgs.nvd} --nix-bin-dir=${pkgs.nix}/bin diff /run/current-system "$systemConfig"
       fi
     '';
   };

hosts/common/feeds.nix

@@ -1,5 +1,8 @@
 # where to find good stuff?
 # - universal search/directory: <https://podcastindex.org>
+#   - the full database is downloadable
+# - find adjacent podcasts: <https://rephonic.com/graph>
+# - charts: <https://rephonic.com/charts/apple/united-states/technology>
 # - list of lists: <https://en.wikipedia.org/wiki/Category:Lists_of_podcasts>
 # - podcasts w/ a community: <https://lemmyverse.net/communities?query=podcast>
 # - podcast recs:
@@ -57,8 +60,9 @@ let
     };
 
   podcasts = [
+    (fromDb "404media.co/the-404-media-podcast" // tech)
     (fromDb "acquiredlpbonussecretsecret.libsyn.com" // tech)  # ACQ2 - more "Acquired" episodes
-    (fromDb "allinchamathjason.libsyn.com" // pol)
+    (fromDb "adventofcomputing.com" // tech)  # computing history
     (fromDb "api.oyez.org/podcasts/oral-arguments/2015" // pol)  # Supreme Court Oral Arguments ("2015" in URL means nothing -- it's still updated)
     (fromDb "anchor.fm/s/34c7232c/podcast/rss" // tech)  # Civboot -- https://anchor.fm/civboot
     (fromDb "anchor.fm/s/2da69154/podcast/rss" // tech)  # POD OF JAKE -- https://podofjake.com/
@@ -66,16 +70,25 @@ let
     (fromDb "congressionaldish.libsyn.com" // pol)  # Jennifer Briney
     (fromDb "craphound.com" // pol)  # Cory Doctorow -- both podcast & text entries
     (fromDb "darknetdiaries.com" // tech)
-    (fromDb "feed.podbean.com/matrixlive/feed.xml" // tech)  # Matrix (chat) Live
+    (fromDb "dwarkeshpatel.com" // tech)
     (fromDb "feeds.99percentinvisible.org/99percentinvisible" // pol)  # 99% Invisible -- also available here: <https://feeds.simplecast.com/BqbsxVfO>
+    (fromDb "feeds.acast.com/public/shows/lawfare" // pol)  # <https://www.lawfaremedia.org/podcasts-multimedia/podcast/the-lawfare-podcast>
+    (fromDb "feeds.buzzsprout.com/2412334.rss")  # Matt Stoller's _Organized Money_ <https://www.organizedmoney.fm/>
+    (fromDb "feeds.eff.org/howtofixtheinternet" // pol)
     (fromDb "feeds.feedburner.com/80000HoursPodcast" // rat)
     (fromDb "feeds.feedburner.com/dancarlin/history" // rat)
     (fromDb "feeds.feedburner.com/radiolab" // pol)  # Radiolab -- also available here, but ONLY OVER HTTP: <http://feeds.wnyc.org/radiolab>
+    (fromDb "feeds.megaphone.fm/CHTAL4990341033" // pol)  # ChinaTalk: https://www.chinatalk.media/podcast
+    (fromDb "feeds.megaphone.fm/GLT1412515089" // pol)  # JRE: Joe Rogan Experience
     (fromDb "feeds.megaphone.fm/behindthebastards" // pol)  # also Maggie Killjoy
+    (fromDb "feeds.megaphone.fm/cspantheweekly" // pol)
+    (fromDb "feeds.megaphone.fm/econ102")  # Noah Smith + Erik Torenberg <https://www.podpage.com/econ102/>
+    (fromDb "feeds.megaphone.fm/history102")  # <https://www.podpage.com/history-102-with-whatifalthist/>
     (fromDb "feeds.megaphone.fm/recodedecode" // tech)  # The Verge - Decoder
-    (fromDb "feeds.simplecast.com/82FI35Px" // pol)  # Ezra Klein Show
+    (fromDb "feeds.megaphone.fm/thiswontlast" // tech)  # <https://www.podpage.com/thiswontlast/>
+    (fromDb "feeds.megaphone.fm/unexplainable")
     (fromDb "feeds.simplecast.com/wgl4xEgL" // rat)  # Econ Talk
-    (fromDb "feeds.simplecast.com/xKJ93w_w" // uncat)  # Atlas Obscura
+    (fromDb "feeds.simplecast.com/whlwDbyc" // tech)  # Tech Lounge: <https://chrischinchilla.com/podcast/techlounge/>
     (fromDb "feeds.transistor.fm/acquired" // tech)
     (fromDb "feeds.transistor.fm/complex-systems-with-patrick-mckenzie-patio11" // tech)  # Patrick Mackenzie (from Bits About Money)
     (fromDb "feeds.twit.tv/floss.xml" // tech)
@@ -84,45 +97,61 @@ let
     (fromDb "hackerpublicradio.org" // tech)
     (fromDb "lexfridman.com/podcast" // rat)
     (fromDb "linktr.ee/betteroffline" // pol)
+    (fromDb "linuxdevtime.com" // tech)
+    (fromDb "malicious.life" // tech)
     (fromDb "mapspodcast.libsyn.com" // uncat)  # Multidisciplinary Association for Psychedelic Studies
     (fromDb "microarch.club" // tech)
-    (fromDb "mintcast.org" // tech)
     (fromDb "omegataupodcast.net" // tech)  # 3/4 German; 1/4 eps are English
     (fromDb "omny.fm/shows/cool-people-who-did-cool-stuff" // pol)  # Maggie Killjoy -- referenced by Cory Doctorow
     (fromDb "omny.fm/shows/money-stuff-the-podcast")  # Matt Levine
+    (fromDb "omny.fm/shows/stuff-you-should-know-1")
     (fromDb "omny.fm/shows/the-dollop-with-dave-anthony-and-gareth-reynolds")  # The Dollop history/comedy
     (fromDb "omny.fm/shows/weird-little-guys")  # Cool Zone Media
     (fromDb "originstories.libsyn.com" // uncat)
+    (fromDb "podcast.ergaster.org/@flintandsilicon" // tech)  # Thib's podcast: public interest tech, gnome, etc: <https://fed.uninsane.org/users/$ALLO9MZ5g5CsQTCBH6>
     (fromDb "politicalorphanage.libsyn.com" // pol)
     (fromDb "reverseengineering.libsyn.com/rss" // tech)  # UnNamed Reverse Engineering Podcast
-    (fromDb "rss.acast.com/deconstructed")  # The Intercept - Deconstructed
-    (fromDb "rss.acast.com/ft-tech-tonic" // tech)
-    (fromDb "rss.acast.com/intercepted-with-jeremy-scahill")  # The Intercept - Intercepted
-    (fromDb "rss.art19.com/60-minutes" // pol)
+    (fromDb "rss.acast.com/ft-tech-tonic" // tech)  # Financial Time's: Tech Tonic
     (fromDb "rss.art19.com/the-portal" // rat)  # Eric Weinstein
-    (fromDb "seattlenice.buzzsprout.com" // pol)
+    (fromDb "seattlenice.buzzsprout.com" // pol)  # Seattle Nice
+    (fromDb "sites.libsyn.com/438684" // humor)  # Quorators - digging up *weird* Quota questions
+    (fromDb "speedboatdope.com" // pol)  # Chapo Trap House (premium feed)
     (fromDb "srslywrong.com" // pol)
     (fromDb "sharkbytes.transistor.fm" // tech)  # Wireshark Podcast o_0
-    (fromDb "sharptech.fm/feed/podcast" // tech)
-    (fromDb "sscpodcast.libsyn.com" // rat)  # Astral Codex Ten
+    (fromDb "sharptech.fm/feed/podcast" // tech)  # Ben Thompson
+    (fromDb "sscpodcast.libsyn.com" // rat)  # Astral Codex Ten; Scott Alexander
     (fromDb "talesfromthebridge.buzzsprout.com" // tech)  # Sci-Fi? has Peter Watts; author of No Moods, Ads or Cutesy Fucking Icons (rifters.com)
-    (fromDb "theamphour.com" // tech)
     (fromDb "techtalesshow.com" // tech)  # Corbin Davenport
-    (fromDb "techwontsave.us" // pol)  # rec by Cory Doctorow
-    (fromDb "wakingup.libsyn.com" // pol)  # Sam Harris
-    (fromDb "werenotwrong.fireside.fm" // pol)
+    (fromDb "theamphour.com" // tech)  # The Amp Hour
+    (fromDb "the-ben-marc-show.simplecast.com" // tech // pol)  # Ben Horowitz + Marc Andreessen; love to hate em
+    (fromDb "timclicks.dev/compose-podcast" // tech)  # Rust-heavy dev interviews
+    (fromDb "werenotwrong.fireside.fm" // pol)  # We're Not Wrong
+    (fromDb "whycast.podcast.audio/@whycast" // tech)  # What Hackers Yearn [for]: <https://why2025.org/>
     (mkPod "https://sfconservancy.org/casts/the-corresponding-source/feeds/ogg/" // tech)
 
+    # (fromDb "allinchamathjason.libsyn.com" // pol)
+    # (fromDb "feed.podbean.com/matrixlive/feed.xml" // tech)  # Matrix (chat) Live
     # (fromDb "feeds.libsyn.com/421877" // rat)  # Less Wrong Curated
     # (fromDb "feeds.megaphone.fm/hubermanlab" // uncat)  # Daniel Huberman on sleep
     # (fromDb "feeds.simplecast.com/54nAGcIl" // pol)  # The Daily
+    # (fromDb "feeds.simplecast.com/82FI35Px" // pol)  # Ezra Klein Show
     # (fromDb "feeds.simplecast.com/l2i9YnTd" // tech // pol)  # Hard Fork (NYtimes tech)
+    # (fromDb "feeds.simplecast.com/xKJ93w_w" // uncat)  # Atlas Obscura
+    # (fromDb "lastweekinai.com" // tech)  # Last Week in AI
+    # (fromDb "mintcast.org" // tech)
+    # (fromDb "politicspoliticspolitics.com" // pol)  # don't judge me. Justin Robert Young.
     # (fromDb "podcast.posttv.com/itunes/post-reports.xml" // pol)
+    # (fromDb "podcast.sustainoss.org" // tech)  # "Sustainable tech", only... it somehow manages to avoid any tech which is actually sustainable, and most of the time doesn't even talk about Open Source Software (!). normie/surface-level/"feel good"
     # (fromDb "podcast.thelinuxexp.com" // tech)  # low-brow linux/foss PR announcements
+    # (fromDb "rss.acast.com/deconstructed")  # The Intercept - Deconstructed
+    # (fromDb "rss.acast.com/intercepted-with-jeremy-scahill")  # The Intercept - Intercepted
+    # (fromDb "rss.art19.com/60-minutes" // pol)
     # (fromDb "rss.art19.com/your-welcome" // pol)  # Michael Malice - Your Welcome -- also available here: <https://origin.podcastone.com/podcast?categoryID2=2232>
     # (fromDb "rss.prod.firstlook.media/deconstructed/podcast.rss" // pol)  #< possible URL rot
     # (fromDb "rss.prod.firstlook.media/intercepted/podcast.rss" // pol)  #< possible URL rot
+    # (fromDb "techwontsave.us" // pol)  # rec by Cory Doctorow, but way too info-sparse
     # (fromDb "trashfuturepodcast.podbean.com" // pol)  # rec by Cory Doctorow, but way rambly
+    # (fromDb "wakingup.libsyn.com" // pol)  # Sam Harris, but he just repeats himself now
     # (mkPod "https://anchor.fm/s/21bc734/podcast/rss" // pol // infrequent)  # Emerge: making sense of what's next -- <https://www.whatisemerging.com/emergepodcast>
     # (mkPod "https://audioboom.com/channels/5097784.rss" // tech)  # Lateral with Tom Scott
     # (mkPod "https://feeds.megaphone.fm/RUNMED9919162779" // pol // infrequent)  # The Witch Trials of J.K. Rowling: <https://www.thefp.com/witchtrials>
@@ -130,9 +159,11 @@ let
   ];
 
   texts = [
+    (fromDb "ergaster.org/blog" // tech)  # Thib's blog: public interest tech, gnome, etc: <https://fed.uninsane.org/users/$ALLO9MZ5g5CsQTCBH6>
     (fromDb "acoup.blog/feed")  # history, states. author: <https://historians.social/@bretdevereaux/following>
     (fromDb "amosbbatto.wordpress.com" // tech)
     (fromDb "anish.lakhwara.com" // tech)
+    (fromDb "antipope.org")  # Charles Stross
     (fromDb "apenwarr.ca/log/rss.php" // tech)  # CEO of tailscale
     (fromDb "applieddivinitystudies.com" // rat)
     (fromDb "artemis.sh" // tech)
@@ -147,6 +178,7 @@ let
     (fromDb "blog.jmp.chat" // tech)
     (fromDb "blog.rust-lang.org" // tech)
     (fromDb "blog.thalheim.io" // tech)  # Mic92
+    (fromDb "blog.brixit.nl" // tech)  # Martijn Braam
     (fromDb "bunniestudios.com" // tech)  # Bunnie Juang
     (fromDb "capitolhillseattle.com" // pol)
     (fromDb "edwardsnowden.substack.com" // pol // text)
@@ -159,6 +191,7 @@ let
     (fromDb "interconnected.org/home/feed" // rat)  # Matt Webb -- engineering-ish, but dreamy
     (fromDb "jeffgeerling.com" // tech)
     (fromDb "jefftk.com" // tech)
+    (fromDb "justine.lol" // tech)
     (fromDb "jwz.org/blog" // tech // pol)  # DNA lounge guy, loooong-time blogger
     (fromDb "kill-the-newsletter.com/feeds/joh91bv7am2pnznv.xml" // pol)  # Matt Levine - Money Stuff
     (fromDb "kosmosghost.github.io/index.xml" // tech)
@@ -168,6 +201,7 @@ let
     (fromDb "mako.cc/copyrighteous" // tech // pol)  # rec by Cory Doctorow
     (fromDb "mg.lol" // tech)
     (fromDb "mindingourway.com" // rat)
+    (fromDb "momi.ca" // tech)  # Anjan, pmOS
     (fromDb "morningbrew.com/feed" // pol)
     (fromDb "nixpkgs.news" // tech)
     (fromDb "overcomingbias.com" // rat)  # Robin Hanson
@@ -190,7 +224,6 @@ let
     (fromDb "slimemoldtimemold.com" // rat)
     (fromDb "spectrum.ieee.org" // tech)
     (fromDb "stpeter.im/atom.xml" // pol)
-    (fromDb "thediff.co" // pol)  # Byrne Hobart
     (fromDb "thisweek.gnome.org" // tech)
     (fromDb "tuxphones.com" // tech)
     (fromDb "uninsane.org" // tech)
@@ -202,12 +235,14 @@ let
     (fromDb "xorvoid.com" // tech)
     (fromDb "www.thebignewsletter.com" // pol)
     (mkSubstack "astralcodexten" // rat // daily)  # Scott Alexander
+    (mkSubstack "chlamchowder" // tech)  # details CPU advancements
     (mkSubstack "eliqian" // rat // weekly)
     (mkSubstack "oversharing" // pol // daily)
     (mkSubstack "samkriss" // humor // infrequent)
     (mkText "http://benjaminrosshoffman.com/feed" // pol // weekly)
     (mkText "http://boginjr.com/feed" // tech // infrequent)
     (mkText "https://forum.merveilles.town/rss.xml" // pol // infrequent)  #quality RSS list here: <https://forum.merveilles.town/thread/57/share-your-rss-feeds%21-6/>
+    (mkText "https://icm.museum/rss20.xml" // tech // infrequent)  # Interim Computer Museum
     (mkText "https://jvns.ca/atom.xml" // tech // weekly)  # Julia Evans
     (mkText "https://linuxphoneapps.org/blog/atom.xml" // tech // infrequent)
     (mkText "https://nixos.org/blog/announcements-rss.xml" // tech // infrequent)  # more nixos stuff here, but unclear how to subscribe: <https://nixos.org/blog/categories.html>
@@ -220,6 +255,7 @@ let
     # (fromDb "econlib.org" // pol)
     # (fromDb "lesswrong.com" // rat)
     # (fromDb "profectusmag.com" // pol)  # some conservative/libertarian think tank
+    # (fromDb "thediff.co" // pol)  # Byrne Hobart; 80% is subscriber-only
     # (fromDb "thesideview.co" // uncat)  # spiritual journal; RSS items are stubs
     # (fromDb "theregister.com" // tech)
     # (fromDb "vitalik.ca" // tech)  # moved to vitalik.eth.limo
@@ -232,21 +268,26 @@ let
 
   videos = [
     (fromDb "youtube.com/@Channel5YouTube" // pol)
-    (fromDb "youtube.com/@ColdFusion")
     (fromDb "youtube.com/@ContraPoints" // pol)
     (fromDb "youtube.com/@Exurb1a")
     (fromDb "youtube.com/@hbomberguy")
     (fromDb "youtube.com/@JackStauber")
+    (fromDb "youtube.com/@mii_beta" // tech)  # Baby Wogue / gnome reviewer
+    (fromDb "youtube.com/@Matrixdotorg" // tech)  # Matrix Live
     (fromDb "youtube.com/@NativLang")
     (fromDb "youtube.com/@PolyMatter")
     (fromDb "youtube.com/@TechnologyConnections" // tech)
-    (fromDb "youtube.com/@TheB1M")
     (fromDb "youtube.com/@TomScottGo")
+    (fromDb "youtube.com/@TVW_Washington" // pol)  # interviews with WA public officials
     (fromDb "youtube.com/@Vihart")
+    (fromDb "youtube.com/@InnuendoStudios" // pol)  # breaks down the nastier political strategies, from a "politics is power" angle
 
+    # (fromDb "youtube.com/@ColdFusion")
+    # (fromDb "youtube.com/@rossmanngroup" // pol // tech)  # Louis Rossmann
+    # (fromDb "youtube.com/@TheB1M")
+    # (fromDb "youtube.com/@tested" // tech)  # Adam Savage  (uploads too frequently)
     # (fromDb "youtube.com/@Vox")
     # (fromDb "youtube.com/@Vsauce")  # they're all like 1-minute long videos now? what happened @Vsauce?
-    # (fromDb "youtube.com/@rossmanngroup" // pol // tech)  # Louis Rossmann
   ];
 
   images = [

hosts/common/fs.nix

@@ -1,375 +0,0 @@
-# docs
-# - x-systemd options: <https://www.freedesktop.org/software/systemd/man/systemd.mount.html>
-# - fuse options: `man mount.fuse`
-
-{ config, lib, pkgs, sane-lib, utils, ... }:
-
-let
-  fsOpts = rec {
-    common = [
-      "_netdev"
-      "noatime"
-      # user: allow any user with access to the device to mount the fs.
-      #       note that this requires a suid `mount` binary; see: <https://zameermanji.com/blog/2022/8/5/using-fuse-without-root-on-linux/>
-      "user"
-      "x-systemd.requires=network-online.target"
-      "x-systemd.after=network-online.target"
-      "x-systemd.mount-timeout=10s"  # how long to wait for mount **and** how long to wait for unmount
-    ];
-    # x-systemd.automount: mount the fs automatically *on first access*.
-    # creates a `path-to-mount.automount` systemd unit.
-    automount = [ "x-systemd.automount" ];
-    # noauto: don't mount as part of remote-fs.target.
-    # N.B.: `remote-fs.target` is a dependency of multi-user.target, itself of graphical.target.
-    # hence, omitting `noauto` can slow down boots.
-    noauto = [ "noauto" ];
-    # lazyMount: defer mounting until first access from userspace.
-    # see: `man systemd.automount`, `man automount`, `man autofs`
-    lazyMount = noauto ++ automount;
-
-    fuse = [
-      "allow_other"  # allow users other than the one who mounts it to access it. needed, if systemd is the one mounting this fs (as root)
-      # allow_root: allow root to access files on this fs (if mounted by non-root, else it can always access them).
-      #             N.B.: if both allow_root and allow_other are specified, then only allow_root takes effect.
-      # "allow_root"
-      # default_permissions: enforce local permissions check. CRUCIAL if using `allow_other`.
-      # w/o this, permissions mode of sshfs is like:
-      # - sshfs runs all remote commands as the remote user.
-      # - if a local user has local permissions to the sshfs mount, then their file ops are sent blindly across the tunnel.
-      # - `allow_other` allows *any* local user to access the mount, and hence any local user can now freely become the remote mapped user.
-      # with default_permissions, sshfs doesn't tunnel file ops from users until checking that said user could perform said op on an equivalent local fs.
-      "default_permissions"
-    ];
-    fuseColin = fuse ++ [
-      "uid=1000"
-      "gid=100"
-    ];
-
-    ssh = common ++ fuseColin ++ [
-      "identityfile=/home/colin/.ssh/id_ed25519"
-      # i *think* idmap=user means that `colin` on `localhost` and `colin` on the remote are actually treated as the same user, even if their uid/gid differs?
-      # i.e., local colin's id is translated to/from remote colin's id on every operation?
-      "idmap=user"
-    ];
-    sshColin = ssh ++ fuseColin ++ [
-      # follow_symlinks: remote files which are symlinks are presented to the local system as ordinary files (as the target of the symlink).
-      #                  if the symlink target does not exist, the presentation is unspecified.
-      #                  symlinks which point outside the mount ARE followed. so this is more capable than `transform_symlinks`
-      "follow_symlinks"
-      # symlinks on the remote fs which are absolute paths are presented to the local system as relative symlinks pointing to the expected data on the remote fs.
-      # only symlinks which would point inside the mountpoint are translated.
-      "transform_symlinks"
-    ];
-    # sshRoot = ssh ++ [
-    #   # we don't transform_symlinks because that breaks the validity of remote /nix stores
-    #   "sftp_server=/run/wrappers/bin/sudo\\040/run/current-system/sw/libexec/sftp-server"
-    # ];
-    # in the event of hunt NFS mounts, consider:
-    # - <https://unix.stackexchange.com/questions/31979/stop-broken-nfs-mounts-from-locking-a-directory>
-
-    # NFS options: <https://linux.die.net/man/5/nfs>
-    # actimeo=n  = how long (in seconds) to cache file/dir attributes (default: 3-60s)
-    # bg         = retry failed mounts in the background
-    # retry=n    = for how many minutes `mount` will retry NFS mount operation
-    # intr       = allow Ctrl+C to abort I/O (it will error with `EINTR`)
-    # soft       = on "major timeout", report I/O error to userspace
-    # softreval  = on "major timeout", service the request using known-stale cache results instead of erroring -- if such cache data exists
-    # retrans=n  = how many times to retry a NFS request before giving userspace a "server not responding" error (default: 3)
-    # timeo=n    = number of *deciseconds* to wait for a response before retrying it (default: 600)
-    #              note: client uses a linear backup, so the second request will have double this timeout, then triple, etc.
-    # proto=udp  = encapsulate protocol ops inside UDP packets instead of a TCP session.
-    #              requires `nfsvers=3` and a kernel compiled with `NFS_DISABLE_UDP_SUPPORT=n`.
-    #              UDP might be preferable to TCP because the latter is liable to hang for ~100s (kernel TCP timeout) after a link drop.
-    #              however, even UDP has issues with `umount` hanging.
-    #
-    # N.B.: don't change these without first testing the behavior of sandboxed apps on a flaky network.
-    nfs = common ++ [
-      # "actimeo=5"
-      # "bg"
-      "retrans=1"
-      "retry=0"
-      # "intr"
-      "soft"
-      "softreval"
-      "timeo=30"
-      "nofail"  # don't fail remote-fs.target when this mount fails  (not an option for sshfs else would be common)
-      # "proto=udp"  # default kernel config doesn't support NFS over UDP: <https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1964093> (see comment 11).
-      # "nfsvers=3"  # NFSv4+ doesn't support UDP at *all*. it's ok to omit nfsvers -- server + client will negotiate v3 based on udp requirement. but omitting causes confusing mount errors when the server is *offline*, because the client defaults to v4 and thinks the udp option is a config error.
-      # "x-systemd.idle-timeout=10"  # auto-unmount after this much inactivity
-    ];
-
-    # manually perform a ftp mount via e.g.
-    #   curlftpfs -o ftpfs_debug=2,user=anonymous:anonymous,connect_timeout=10 -f -s ftp://servo-hn /mnt/my-ftp
-    ftp = common ++ fuseColin ++ [
-      # "ftpfs_debug=2"
-      "user=colin:ipauth"
-      # connect_timeout=10: casting shows to T.V. fails partway through about half the time
-      "connect_timeout=20"
-    ];
-  };
-
-  ifSshAuthorized = lib.mkIf config.sane.hosts.by-name."${config.networking.hostName}".ssh.authorized;
-
-  remoteHome = host: {
-    sane.programs.sshfs-fuse.enableFor.system = true;
-    system.fsPackages = [
-      config.sane.programs.sshfs-fuse.package
-    ];
-    fileSystems."/mnt/${host}/home" = {
-      device = "sshfs#colin@${host}:/home/colin";
-      fsType = "fuse3";
-      options = fsOpts.sshColin ++ fsOpts.lazyMount ++ [
-        # drop_privileges: after `mount.fuse3` opens /dev/fuse, it will drop all capabilities before invoking sshfs
-        "drop_privileges"
-        "auto_unmount"  #< ensures that when the fs exits, it releases its mountpoint. then systemd can recognize it as failed.
-      ];
-      noCheck = true;
-    };
-    sane.fs."/mnt/${host}/home" = {
-      dir.acl.user = "colin";
-      dir.acl.group = "users";
-      dir.acl.mode = "0700";
-      wantedBy = [ "default.target" ];
-      mount.depends = [ "network-online.target" ];
-      mount.mountConfig.ExecSearchPath = [ "/run/current-system/sw/bin" ];
-      mount.mountConfig.User = "colin";
-      mount.mountConfig.AmbientCapabilities = "CAP_SETPCAP CAP_SYS_ADMIN";
-      # hardening (systemd-analyze security mnt-desko-home.mount):
-      # TODO: i can't use ProtectSystem=full here, because i can't create a new mount space; but...
-      # with drop_privileges, i *could* sandbox the actual `sshfs` program using e.g. bwrap
-      mount.mountConfig.CapabilityBoundingSet = "CAP_SETPCAP CAP_SYS_ADMIN";
-      mount.mountConfig.LockPersonality = true;
-      mount.mountConfig.MemoryDenyWriteExecute = true;
-      mount.mountConfig.NoNewPrivileges = true;
-      mount.mountConfig.ProtectClock = true;
-      mount.mountConfig.ProtectHostname = true;
-      mount.mountConfig.RemoveIPC = true;
-      mount.mountConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
-      #VVV this includes anything it reads from, e.g. /bin/sh; /nix/store/...
-      # see `systemd-analyze filesystems` for a full list
-      mount.mountConfig.RestrictFileSystems = "@common-block @basic-api fuse";
-      mount.mountConfig.RestrictRealtime = true;
-      mount.mountConfig.RestrictSUIDSGID = true;
-      mount.mountConfig.SystemCallArchitectures = "native";
-      mount.mountConfig.SystemCallFilter = [
-        "@system-service"
-        "@mount"
-        "~@chown"
-        "~@cpu-emulation"
-        "~@keyring"
-        # could remove almost all io calls, however one has to keep `open`, and `write`, to communicate with the fuse device.
-        # so that's pretty useless as a way to prevent write access
-      ];
-      mount.mountConfig.IPAddressDeny = "any";
-      mount.mountConfig.IPAddressAllow = "10.0.0.0/8";
-      mount.mountConfig.DevicePolicy = "closed";  # only allow /dev/{null,zero,full,random,urandom}
-      mount.mountConfig.DeviceAllow = "/dev/fuse";
-      # mount.mountConfig.RestrictNamespaces = true;  #< my sshfs sandboxing uses bwrap
-    };
-  };
-  remoteServo = subdir: let
-    localPath = "/mnt/servo/${subdir}";
-    systemdName = utils.escapeSystemdPath localPath;
-  in {
-    sane.programs.curlftpfs.enableFor.system = true;
-    system.fsPackages = [
-      config.sane.programs.curlftpfs.package
-    ];
-    fileSystems."${localPath}" = {
-      device = "curlftpfs#ftp://servo-hn:/${subdir}";
-      noCheck = true;
-      fsType = "fuse3";
-      options = fsOpts.ftp ++ fsOpts.noauto ++ [
-        # drop_privileges: after `mount.fuse3` opens /dev/fuse, it will drop all capabilities before invoking sshfs
-        "drop_privileges"
-        "auto_unmount"  #< ensures that when the fs exits, it releases its mountpoint. then systemd can recognize it as failed.
-      ];
-      # fsType = "nfs";
-      # options = fsOpts.nfs ++ fsOpts.lazyMount;
-    };
-    sane.fs."${localPath}" = {
-      dir.acl.user = "colin";
-      dir.acl.group = "users";
-      dir.acl.mode = "0750";
-      wantedBy = [ "default.target" ];
-      mount.depends = [ "network-online.target" "${systemdName}-reachable.service" ];
-      #VVV patch so that when the mount fails, we start a timer to remount it.
-      #    and for a disconnection after a good mount (onSuccess), restart the timer to be more aggressive
-      mount.unitConfig.OnFailure = [ "${systemdName}.timer" ];
-      mount.unitConfig.OnSuccess = [ "${systemdName}-restart-timer.target" ];
-
-      mount.mountConfig.TimeoutSec = "10s";
-      mount.mountConfig.ExecSearchPath = [ "/run/current-system/sw/bin" ];
-      mount.mountConfig.User = "colin";
-      mount.mountConfig.AmbientCapabilities = "CAP_SETPCAP CAP_SYS_ADMIN";
-      # hardening (systemd-analyze security mnt-servo-playground.mount)
-      mount.mountConfig.CapabilityBoundingSet = "CAP_SETPCAP CAP_SYS_ADMIN";
-      mount.mountConfig.LockPersonality = true;
-      mount.mountConfig.MemoryDenyWriteExecute = true;
-      mount.mountConfig.NoNewPrivileges = true;
-      mount.mountConfig.ProtectClock = true;
-      mount.mountConfig.ProtectHostname = true;
-      mount.mountConfig.RemoveIPC = true;
-      mount.mountConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
-      #VVV this includes anything it reads from, e.g. /bin/sh; /nix/store/...
-      # see `systemd-analyze filesystems` for a full list
-      mount.mountConfig.RestrictFileSystems = "@common-block @basic-api fuse";
-      mount.mountConfig.RestrictRealtime = true;
-      mount.mountConfig.RestrictSUIDSGID = true;
-      mount.mountConfig.SystemCallArchitectures = "native";
-      mount.mountConfig.SystemCallFilter = [
-        "@system-service"
-        "@mount"
-        "~@chown"
-        "~@cpu-emulation"
-        "~@keyring"
-        # could remove almost all io calls, however one has to keep `open`, and `write`, to communicate with the fuse device.
-        # so that's pretty useless as a way to prevent write access
-      ];
-      mount.mountConfig.IPAddressDeny = "any";
-      mount.mountConfig.IPAddressAllow = "10.0.10.5";
-      mount.mountConfig.DevicePolicy = "closed";  # only allow /dev/{null,zero,full,random,urandom}
-      mount.mountConfig.DeviceAllow = "/dev/fuse";
-      # mount.mountConfig.RestrictNamespaces = true;
-    };
-
-    systemd.services."${systemdName}-reachable" = {
-      serviceConfig.ExecSearchPath = [ "/run/current-system/sw/bin" ];
-      serviceConfig.ExecStart = lib.escapeShellArgs [
-        "curlftpfs"
-        "ftp://servo-hn:/${subdir}"
-        "/dev/null"
-        "-o"
-        (lib.concatStringsSep "," ([ "exit_after_connect" ] ++ config.fileSystems."${localPath}".options))
-      ];
-      serviceConfig.RemainAfterExit = true;
-      serviceConfig.Type = "oneshot";
-      unitConfig.BindsTo = [ "${systemdName}.mount" ];
-      # hardening (systemd-analyze security mnt-servo-playground-reachable.service)
-      serviceConfig.AmbientCapabilities = "";
-      serviceConfig.CapabilityBoundingSet = "";
-      serviceConfig.DynamicUser = true;
-      serviceConfig.LockPersonality = true;
-      serviceConfig.MemoryDenyWriteExecute = true;
-      serviceConfig.NoNewPrivileges = true;
-      serviceConfig.PrivateDevices = true;
-      serviceConfig.PrivateMounts = true;
-      serviceConfig.PrivateTmp = true;
-      serviceConfig.PrivateUsers = true;
-      serviceConfig.ProcSubset = "all";
-      serviceConfig.ProtectClock = true;
-      serviceConfig.ProtectControlGroups = true;
-      serviceConfig.ProtectHome = true;
-      serviceConfig.ProtectKernelModules = true;
-      serviceConfig.ProtectProc = "invisible";
-      serviceConfig.ProtectSystem = "strict";
-      serviceConfig.RemoveIPC = true;
-      serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
-      # serviceConfig.RestrictFileSystems = "@common-block @basic-api";  #< NOPE
-      serviceConfig.RestrictRealtime = true;
-      serviceConfig.RestrictSUIDSGID = true;
-      serviceConfig.SystemCallArchitectures = "native";
-      serviceConfig.SystemCallFilter = [
-        "@system-service"
-        "@mount"
-        "~@chown"
-        "~@cpu-emulation"
-        "~@keyring"
-        # "~@privileged"  #< NOPE
-        "~@resources"
-        # could remove some more probably
-      ];
-      serviceConfig.IPAddressDeny = "any";
-      serviceConfig.IPAddressAllow = "10.0.10.5";
-      serviceConfig.DevicePolicy = "closed";
-      # exceptions
-      serviceConfig.ProtectHostname = false;
-      serviceConfig.ProtectKernelLogs = false;
-      serviceConfig.ProtectKernelTunables = false;
-    };
-
-    systemd.targets."${systemdName}-restart-timer" = {
-      # hack unit which, when started, stops the timer (if running), and then starts it again.
-      after = [ "${systemdName}.timer" ];
-      conflicts = [ "${systemdName}.timer" ];
-      upholds = [ "${systemdName}.timer" ];
-      unitConfig.StopWhenUnneeded = true;
-    };
-    systemd.timers."${systemdName}" = {
-      timerConfig.Unit = "${systemdName}.mount";
-      timerConfig.AccuracySec = "2s";
-      timerConfig.OnActiveSec = [
-        # try to remount at these timestamps, backing off gradually
-        # there seems to be an implicit mount attempt at t=0.
-        "10s"
-        "30s"
-        "60s"
-        "120s"
-      ];
-      # cap the backoff to a fixed interval.
-      timerConfig.OnUnitActiveSec = [ "120s" ];
-    };
-  };
-in
-lib.mkMerge [
-  {
-    # some services which use private directories error if the parent (/var/lib/private) isn't 700.
-    sane.fs."/var/lib/private".dir.acl.mode = "0700";
-
-    # in-memory compressed RAM
-    # defaults to compressing at most 50% size of RAM
-    # claimed compression ratio is about 2:1
-    # - but on moby w/ zstd default i see 4-7:1  (ratio lowers as it fills)
-    # note that idle overhead is about 0.05% of capacity (e.g. 2B per 4kB page)
-    # docs: <https://www.kernel.org/doc/Documentation/blockdev/zram.txt>
-    #
-    # to query effectiveness:
-    # `cat /sys/block/zram0/mm_stat`. whitespace separated fields:
-    # - *orig_data_size*  (bytes)
-    # - *compr_data_size* (bytes)
-    # - mem_used_total    (bytes)
-    # - mem_limit         (bytes)
-    # - mem_used_max      (bytes)
-    # - *same_pages*  (pages which are e.g. all zeros (consumes no additional mem))
-    # - *pages_compacted*  (pages which have been freed thanks to compression)
-    # - huge_pages  (incompressible)
-    #
-    # see also:
-    # - `man zramctl`
-    zramSwap.enable = true;
-    # how much ram can be swapped into the zram device.
-    # this shouldn't be higher than the observed compression ratio.
-    # the default is 50% (why?)
-    # 100% should be "guaranteed" safe so long as the data is even *slightly* compressible.
-    # but it decreases working memory under the heaviest of loads by however much space the compressed memory occupies (e.g. 50% if 2:1; 25% if 4:1)
-    zramSwap.memoryPercent = 100;
-
-    # environment.pathsToLink = [
-    #   # needed to achieve superuser access for user-mounted filesystems (see sshRoot above)
-    #   # we can only link whole directories here, even though we're only interested in pkgs.openssh
-    #   "/libexec"
-    # ];
-
-    programs.fuse.userAllowOther = true;  #< necessary for `allow_other` or `allow_root` options.
-  }
-
-  (ifSshAuthorized (remoteHome "crappy"))
-  (ifSshAuthorized (remoteHome "desko"))
-  (ifSshAuthorized (remoteHome "lappy"))
-  (ifSshAuthorized (remoteHome "moby"))
-  (ifSshAuthorized (remoteHome "servo"))
-  # this granularity of servo media mounts is necessary to support sandboxing:
-  # for flaky mounts, we can only bind the mountpoint itself into the sandbox,
-  # so it's either this or unconditionally bind all of media/.
-  (remoteServo "media/archive")
-  (remoteServo "media/Books")
-  (remoteServo "media/collections")
-  # (remoteServo "media/datasets")
-  (remoteServo "media/games")
-  (remoteServo "media/Music")
-  (remoteServo "media/Pictures/macros")
-  (remoteServo "media/torrents")
-  (remoteServo "media/Videos")
-  (remoteServo "playground")
-]
-

hosts/common/fs/default.nix

@@ -0,0 +1,51 @@
+{ ... }:
+{
+  imports = [
+    ./remote-home.nix
+    ./remote-servo.nix
+  ];
+  # some services which use private directories error if the parent (/var/lib/private) isn't 700.
+  sane.fs."/var/lib/private".dir.acl.mode = "0700";
+
+
+  # allocate a proper /tmp fs, else its capacity will be limited as per impermanence defaults (i.e. 1 GB).
+  fileSystems."/tmp" = {
+    device = "none";
+    fsType = "tmpfs";
+    options = [
+      "mode=777"
+      "defaults"
+    ];
+  };
+
+  # in-memory compressed RAM
+  # defaults to compressing at most 50% size of RAM
+  # claimed compression ratio is about 2:1
+  # - but on moby w/ zstd default i see 4-7:1  (ratio lowers as it fills)
+  # note that idle overhead is about 0.05% of capacity (e.g. 2B per 4kB page)
+  # docs: <https://www.kernel.org/doc/Documentation/blockdev/zram.txt>
+  #
+  # to query effectiveness:
+  # `cat /sys/block/zram0/mm_stat`. whitespace separated fields:
+  # - *orig_data_size*  (bytes)
+  # - *compr_data_size* (bytes)
+  # - mem_used_total    (bytes)
+  # - mem_limit         (bytes)
+  # - mem_used_max      (bytes)
+  # - *same_pages*  (pages which are e.g. all zeros (consumes no additional mem))
+  # - *pages_compacted*  (pages which have been freed thanks to compression)
+  # - huge_pages  (incompressible)
+  #
+  # see also:
+  # - `man zramctl`
+  zramSwap.enable = true;
+  # how much ram can be swapped into the zram device.
+  # this shouldn't be higher than the observed compression ratio.
+  # the default is 50% (why?)
+  # 100% should be "guaranteed" safe so long as the data is even *slightly* compressible.
+  # but it decreases working memory under the heaviest of loads by however much space the compressed memory occupies (e.g. 50% if 2:1; 25% if 4:1)
+  zramSwap.memoryPercent = 100;
+
+  programs.fuse.userAllowOther = true;  #< necessary for `allow_other` or `allow_root` options.
+}
+

hosts/common/fs/fs-opts.nix

@@ -0,0 +1,76 @@
+# docs
+# - x-systemd options: <https://www.freedesktop.org/software/systemd/man/systemd.mount.html>
+# - fuse options: `man mount.fuse`
+rec {
+  common = [
+    "_netdev"
+    "noatime"
+    # user: allow any user with access to the device to mount the fs.
+    #       note that this requires a suid `mount` binary; see: <https://zameermanji.com/blog/2022/8/5/using-fuse-without-root-on-linux/>
+    "user"
+    "x-systemd.requires=network-online.target"
+    "x-systemd.after=network-online.target"
+    "x-systemd.mount-timeout=10s"  # how long to wait for mount **and** how long to wait for unmount
+    # disable defaults: don't fail local-fs.target if this mount fails
+    "nofail"
+  ];
+  # x-systemd.automount: mount the fs automatically *on first access*.
+  # creates a `path-to-mount.automount` systemd unit.
+  automount = [ "x-systemd.automount" ];
+  # noauto: don't mount as part of remote-fs.target.
+  # N.B.: `remote-fs.target` is a dependency of multi-user.target, itself of graphical.target.
+  # hence, omitting `noauto` can slow down boots.
+  noauto = [ "noauto" ];
+  # lazyMount: defer mounting until first access from userspace.
+  # see: `man systemd.automount`, `man automount`, `man autofs`
+  lazyMount = noauto ++ automount;
+
+  fuse = [
+    "allow_other"  # allow users other than the one who mounts it to access it. needed, if systemd is the one mounting this fs (as root)
+    # allow_root: allow root to access files on this fs (if mounted by non-root, else it can always access them).
+    #             N.B.: if both allow_root and allow_other are specified, then only allow_root takes effect.
+    # "allow_root"
+    # default_permissions: enforce local permissions check. CRUCIAL if using `allow_other`.
+    # w/o this, permissions mode of sshfs is like:
+    # - sshfs runs all remote commands as the remote user.
+    # - if a local user has local permissions to the sshfs mount, then their file ops are sent blindly across the tunnel.
+    # - `allow_other` allows *any* local user to access the mount, and hence any local user can now freely become the remote mapped user.
+    # with default_permissions, sshfs doesn't tunnel file ops from users until checking that said user could perform said op on an equivalent local fs.
+    "default_permissions"
+    "drop_privileges"
+    "auto_unmount"  #< ensures that when the fs exits, it releases its mountpoint. then systemd can recognize it as failed.
+  ];
+  fuseColin = fuse ++ [
+    "uid=1000"
+    "gid=100"
+  ];
+
+  ssh = common ++ fuseColin ++ [
+    "identityfile=/home/colin/.ssh/id_ed25519"
+    # i *think* idmap=user means that `colin` on `localhost` and `colin` on the remote are actually treated as the same user, even if their uid/gid differs?
+    # i.e., local colin's id is translated to/from remote colin's id on every operation?
+    "idmap=user"
+  ];
+  sshColin = ssh ++ fuseColin ++ [
+    # follow_symlinks: remote files which are symlinks are presented to the local system as ordinary files (as the target of the symlink).
+    #                  if the symlink target does not exist, the presentation is unspecified.
+    #                  symlinks which point outside the mount ARE followed. so this is more capable than `transform_symlinks`
+    "follow_symlinks"
+    # symlinks on the remote fs which are absolute paths are presented to the local system as relative symlinks pointing to the expected data on the remote fs.
+    # only symlinks which would point inside the mountpoint are translated.
+    "transform_symlinks"
+  ];
+  # sshRoot = ssh ++ [
+  #   # we don't transform_symlinks because that breaks the validity of remote /nix stores
+  #   "sftp_server=/run/wrappers/bin/sudo\\040/run/current-system/sw/libexec/sftp-server"
+  # ];
+
+  # manually perform a ftp mount via e.g.
+  #   curlftpfs -o ftpfs_debug=2,user=anonymous:anonymous,connect_timeout=10 -f -s ftp://servo-hn /mnt/my-ftp
+  ftp = common ++ fuseColin ++ [
+    # "ftpfs_debug=2"
+    "user=colin:ipauth"
+    # connect_timeout=10: casting shows to T.V. fails partway through about half the time
+    "connect_timeout=20"
+  ];
+}

hosts/common/fs/remote-home.nix

@@ -0,0 +1,73 @@
+{ config, lib, ... }:
+let
+  fsOpts = import ./fs-opts.nix;
+  ifSshAuthorized = lib.mkIf (((config.sane.hosts.by-name."${config.networking.hostName}" or {}).ssh or {}).authorized or false);
+
+  remoteHome = name: { host ? name }: let
+    mountpoint = "/mnt/${name}/home";
+    device = "sshfs#colin@${host}:/home/colin";
+    fsType = "fuse3";
+    options = fsOpts.sshColin ++ fsOpts.lazyMount;
+  in {
+    sane.programs.sshfs-fuse.enableFor.system = true;
+    system.fsPackages = [
+      config.sane.programs.sshfs-fuse.package
+    ];
+    fileSystems."${mountpoint}" = {
+      inherit device fsType options;
+      noCheck = true;
+    };
+    # tell systemd about the mount so that i can sandbox it
+    systemd.mounts = [{
+      where = mountpoint;
+      what = device;
+      type = fsType;
+      options = lib.concatStringsSep "," options;
+      wantedBy = [ "default.target" ];
+      after = [ "network-online.target" ];
+      requires = [ "network-online.target" ];
+
+      mountConfig.ExecSearchPath = [ "/run/current-system/sw/bin" ];
+      mountConfig.User = "colin";
+      mountConfig.AmbientCapabilities = "CAP_SETPCAP CAP_SYS_ADMIN";
+      # hardening (systemd-analyze security mnt-desko-home.mount):
+      # TODO: i can't use ProtectSystem=full here, because i can't create a new mount space; but...
+      # with drop_privileges, i *could* sandbox the actual `sshfs` program using e.g. bwrap
+      mountConfig.CapabilityBoundingSet = "CAP_SETPCAP CAP_SYS_ADMIN";
+      mountConfig.LockPersonality = true;
+      mountConfig.MemoryDenyWriteExecute = true;
+      mountConfig.NoNewPrivileges = true;
+      mountConfig.ProtectClock = true;
+      mountConfig.ProtectHostname = true;
+      mountConfig.RemoveIPC = true;
+      mountConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
+      #VVV this includes anything it reads from, e.g. /bin/sh; /nix/store/...
+      # see `systemd-analyze filesystems` for a full list
+      mountConfig.RestrictFileSystems = "@common-block @basic-api fuse";
+      mountConfig.RestrictRealtime = true;
+      mountConfig.RestrictSUIDSGID = true;
+      mountConfig.SystemCallArchitectures = "native";
+      mountConfig.SystemCallFilter = [
+        "@system-service"
+        "@mount"
+        "~@chown"
+        "~@cpu-emulation"
+        "~@keyring"
+        # could remove almost all io calls, however one has to keep `open`, and `write`, to communicate with the fuse device.
+        # so that's pretty useless as a way to prevent write access
+      ];
+      mountConfig.IPAddressDeny = "any";
+      mountConfig.IPAddressAllow = "10.0.0.0/8";
+      mountConfig.DevicePolicy = "closed";  # only allow /dev/{null,zero,full,random,urandom}
+      mountConfig.DeviceAllow = "/dev/fuse";
+      # mount.mountConfig.RestrictNamespaces = true;  #< my sshfs sandboxing uses bwrap
+    }];
+  };
+in
+lib.mkMerge [
+  (ifSshAuthorized (remoteHome "crappy" {}))
+  (ifSshAuthorized (remoteHome "desko" {}))
+  (ifSshAuthorized (remoteHome "lappy" {}))
+  (ifSshAuthorized (remoteHome "moby" { host = "moby-hn"; }))
+  (ifSshAuthorized (remoteHome "servo" {}))
+]

hosts/common/fs/remote-servo.nix

@@ -0,0 +1,134 @@
+{ config, lib, utils, ... }:
+let
+  fsOpts = import ./fs-opts.nix;
+  commonOptions = fsOpts.ftp ++ fsOpts.noauto;
+  mountpoint = "/mnt/.servo_ftp";
+  systemdName = utils.escapeSystemdPath mountpoint;
+  device = "curlftpfs#ftp://servo-hn:/";
+  fsType = "fuse3";
+  options = commonOptions ++ [
+    # systemd (or maybe fuse?) swallows stderr of mount units with no obvious fix.
+    # instead, use this flag to log the mount output to disk
+    "stderr_path=/var/log/curlftpfs/servo-hn.stderr"
+  ];
+
+  remoteServo = subdir: let
+    systemdBindName = utils.escapeSystemdPath "/mnt/servo/${subdir}";
+  in {
+    # sane.fs."/mnt/servo/${subdir}".mount.bind = "/mnt/.servo_ftp/${subdir}";
+    systemd.mounts = [{
+      where = "/mnt/servo/${subdir}";
+      what = "/mnt/.servo_ftp/${subdir}";
+      options = "bind,nofail";
+      type = "auto";
+
+      after = [ "${systemdName}.mount" ];
+      upheldBy = [ "${systemdName}.mount" ];  #< start this mount whenever the underlying becomes available
+      bindsTo = [ "${systemdName}.mount" ];  #< stop this mount whenever the underlying disappears
+    }];
+  };
+in
+lib.mkMerge [
+  {
+    sane.programs.curlftpfs.enableFor.system = true;
+    system.fsPackages = [
+      config.sane.programs.curlftpfs.package
+    ];
+
+    sane.fs."/var/log/curlftpfs".dir.acl.mode = "0777";
+
+    fileSystems."/mnt/.servo_ftp" = {
+      inherit device fsType options;
+      noCheck = true;
+    };
+    systemd.mounts = [{
+      where = mountpoint;
+      what = device;
+      type = fsType;
+      options = lib.concatStringsSep "," options;
+      wantedBy = [ "default.target" ];
+      after = [ "network-online.target" ];
+      requires = [ "network-online.target" ];
+
+      #VVV patch so that when the mount fails, we start a timer to remount it.
+      #    and for a disconnection after a good mount (onSuccess), restart the timer to be more aggressive
+      unitConfig.OnFailure = [ "${systemdName}.timer" ];
+      unitConfig.OnSuccess = [ "${systemdName}-restart-timer.target" ];
+
+      mountConfig.TimeoutSec = "10s";
+      mountConfig.ExecSearchPath = [ "/run/current-system/sw/bin" ];
+      mountConfig.User = "colin";
+      mountConfig.AmbientCapabilities = "CAP_SETPCAP CAP_SYS_ADMIN";
+      # hardening (systemd-analyze security mnt-servo-playground.mount)
+      mountConfig.CapabilityBoundingSet = "CAP_SETPCAP CAP_SYS_ADMIN";
+      mountConfig.LockPersonality = true;
+      mountConfig.MemoryDenyWriteExecute = true;
+      mountConfig.NoNewPrivileges = true;
+      mountConfig.ProtectClock = true;
+      mountConfig.ProtectHostname = true;
+      mountConfig.RemoveIPC = true;
+      mountConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";
+      #VVV this includes anything it reads from, e.g. /bin/sh; /nix/store/...
+      # see `systemd-analyze filesystems` for a full list
+      mountConfig.RestrictFileSystems = "@common-block @basic-api fuse";
+      mountConfig.RestrictRealtime = true;
+      mountConfig.RestrictSUIDSGID = true;
+      mountConfig.SystemCallArchitectures = "native";
+      mountConfig.SystemCallFilter = [
+        "@system-service"
+        "@mount"
+        "~@chown"
+        "~@cpu-emulation"
+        "~@keyring"
+        # could remove almost all io calls, however one has to keep `open`, and `write`, to communicate with the fuse device.
+        # so that's pretty useless as a way to prevent write access
+      ];
+      mountConfig.IPAddressDeny = "any";
+      mountConfig.IPAddressAllow = "10.0.10.5";
+      mountConfig.DevicePolicy = "closed";  # only allow /dev/{null,zero,full,random,urandom}
+      mountConfig.DeviceAllow = "/dev/fuse";
+      # mountConfig.RestrictNamespaces = true;
+    }];
+
+    systemd.targets."${systemdName}-restart-timer" = {
+      # hack unit which, when started, stops the timer (if running), and then starts it again.
+      after = [ "${systemdName}.timer" ];
+      conflicts = [ "${systemdName}.timer" ];
+      upholds = [ "${systemdName}.timer" ];
+      unitConfig.StopWhenUnneeded = true;
+    };
+    systemd.timers."${systemdName}" = {
+      timerConfig.Unit = "${systemdName}.mount";
+      timerConfig.AccuracySec = "2s";
+      timerConfig.OnActiveSec = [
+        # try to remount at these timestamps, backing off gradually
+        # there seems to be an implicit mount attempt at t=0.
+        "10s"
+        "30s"
+        "60s"
+        "120s"
+      ];
+      # cap the backoff to a fixed interval.
+      timerConfig.OnUnitActiveSec = [ "120s" ];
+    };
+  }
+
+  # this granularity of servo media mounts is necessary to support sandboxing. consider:
+  #   1. servo offline
+  #   2. launch a long-running app
+  #   3. servo comes online
+  # in order for the servo mount to be propagated into the app's namespace, we need to bind
+  # the root mountpoint into the app namespace. if we wish to only grant the app selective access
+  # to servo, we must create *multiple* mountpoints: /mnt/servo/FOO directories which always exist,
+  # and are individually bound to /mnt/.servo_ftp/FOO as the latter becomes available.
+  (remoteServo "media/archive")
+  (remoteServo "media/Books")
+  (remoteServo "media/collections")
+  # (remoteServo "media/datasets")
+  (remoteServo "media/games")
+  (remoteServo "media/Music")
+  (remoteServo "media/Pictures/macros")
+  (remoteServo "media/torrents")
+  (remoteServo "media/Videos")
+  (remoteServo "playground")
+]

hosts/common/home/fs.nix

@@ -6,13 +6,17 @@
     "dev"
     "ref"
     "use"
+    "Books/Audiobooks"
+    "Books/Books"
+    "Books/Visual"
     "Books/local"
     "Music"
-
-    # this is persisted simply to save on RAM. mesa_shader_cache is < 10 MB.
-    # TODO: integrate with sane.programs.sandbox?
-    ".cache/mesa_shader_cache"
   ];
+
+  sane.user.persist.byStore.ephemeral = [
+    # this is persisted simply to save on RAM. mesa_shader_cache is < 10 MB per boot.
+  ];
+
   sane.user.persist.byStore.private = [
     "archive"
     "Pictures/albums"
@@ -25,15 +29,43 @@
 
     "knowledge"
     "Videos/local"
+
+    # TODO: pre-compile mesa shaders, and then run in read-only mode?
+    # mesa shader cache can be configured with e.g.:
+    # - MESA_SHADER_CACHE_DISABLE=true
+    # - MESA_SHADER_CACHE_DIR=/path/to/cache_db
+    # - MESA_DISK_CACHE_SINGLE_FILE=1  (in which case default cache file is ~/.cache/mesa_shader_cache_sf)
+    # - MESA_DISK_CACHE_MULTI_FILE=1  (in which case default cache dir is ~/.cache/mesa_shader_cache)
+    # - MESA_DISK_CACHE_READ_ONLY_FOZ_DBS=foo,bar
+    #   - to use read-only mesa caches, one from foo.db the other bar.db
+    # - MESA_DISK_CACHE_READ_ONLY_FOZ_DBS_DYNAMIC_LIST=/path/to/txt
+    #   - where /path/to/txt contains a list of names which represent read-only caches
+    #   - allows to change the cache providers w/o having to update variables
+    #
+    # see also: <https://gitlab.freedesktop.org/mesa/shader-db>
+    # - database of common shaders (gtk4, chromium, etc) & instructions to compile for any arch
+    # see also: <https://github.com/ValveSoftware/Fossilize>
+    #   which may help in generating readonly cache files
+    #
+    # for now, mesa shader cache is persisted because some programs *greatly* benefit from it.
+    # esp gnome-contacts has a first-launch bug where it shows a misleading warning if shaders take too long to compile,
+    # so we persist to private instead of ephemeral.
+    ".cache/mesa_shader_cache_db"
   ];
 
   # convenience
   sane.user.fs = let
     persistEnabled = config.sane.persist.enable;
   in {
-    ".persist/private" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.private.origin; };
-    ".persist/plaintext" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.plaintext.origin; };
-    ".persist/ephemeral" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.ephemeral.origin; };
+    ".persist/private" = lib.mkIf persistEnabled {
+       symlink.target = "${config.sane.persist.stores.private.origin}/home/${config.sane.defaultUser}";
+    };
+    ".persist/plaintext" = lib.mkIf persistEnabled {
+       symlink.target = "${config.sane.persist.stores.plaintext.origin}/home/${config.sane.defaultUser}";
+    };
+    ".persist/ephemeral" = lib.mkIf persistEnabled {
+       symlink.target = "${config.sane.persist.stores.ephemeral.origin}/home/${config.sane.defaultUser}";
+    };
 
     "nixos".symlink.target = "dev/nixos";
 

hosts/common/home/mime.nix

@@ -53,19 +53,24 @@ let
       (p: builtins.toString p.package)
       (enabledProgramsWithPackage ++ [ { package=mimeappsListPkg; } ]);
   }).overrideAttrs (orig: {
-    # like normal symlinkJoin, but don't error if the path doesn't exist
+    # like normal symlinkJoin, but don't error if the path doesn't exist.
+    # additionally, remove `DBusActivatable=true` from any .desktop files encountered;
+    # my dbus session is sandboxed such that it can't activate services even if i thought that was a good idea.
     buildCommand = ''
       mkdir -p $out/share/applications
       for i in $(cat $pathsPath); do
         if [ -e "$i/share/applications" ]; then
-          ${pkgs.buildPackages.xorg.lndir}/bin/lndir -silent $i/share/applications $out/share/applications
+          local files=($(cd "$i/share/applications"; ls .))
+          for f in "''${files[@]}"; do
+            sed '/DBusActivatable=true/d' $i/share/applications/$f > $out/share/applications/$f
+          done
         fi
       done
       runHook postBuild
     '';
     postBuild = ''
       # rebuild `mimeinfo.cache`, used by file openers to show the list of *all* apps, not just the user's defaults.
-      ${pkgs.buildPackages.desktop-file-utils}/bin/update-desktop-database $out/share/applications
+      ${lib.getExe' pkgs.buildPackages.desktop-file-utils "update-desktop-database"} $out/share/applications
     '';
   });
 

hosts/common/ids.nix

@@ -6,6 +6,7 @@
 {
   # partially supported in nixpkgs  <repo:nixos/nixpkgs:nixos/modules/misc/ids.nix>
   sane.ids.networkmanager.uid = 57;  #< nixpkgs unofficially reserves this, to match networkmanager's gid
+  sane.ids.mediatomb.uid = 187;  # <repo:nixos/nixpkgs:nixos/modules/misc/ids.nix>
 
   # legacy servo users, some are inconvenient to migrate
   sane.ids.dhcpcd.gid = 991;
@@ -65,6 +66,7 @@
   sane.ids.plugdev.gid = 2421;
   sane.ids.ollama.uid = 2422;
   sane.ids.ollama.gid = 2422;
+  sane.ids.shelvacu.uid = 5431;
 
   sane.ids.colin.uid = 1000;
   sane.ids.guest.uid = 1100;
@@ -81,6 +83,15 @@
   sane.ids.wireshark.gid = 2006;
   sane.ids.nixremote.uid = 2007;
   sane.ids.nixremote.gid = 2007;
+  sane.ids.unbound.uid = 2008;
+  sane.ids.unbound.gid = 2008;
+  sane.ids.resolvconf.gid = 2009;
+  sane.ids.smartd.uid = 2010;
+  sane.ids.smartd.gid = 2010;
+  sane.ids.radicale.uid = 2011;
+  sane.ids.radicale.gid = 2011;
+  sane.ids.named.uid = 2012;
+  sane.ids.named.gid = 2012;
 
   # found on graphical hosts
   sane.ids.nm-iodine.uid = 2101;  # desko/moby/lappy

hosts/common/net/default.nix

@@ -1,11 +1,12 @@
-{ lib, ... }:
+{ ... }:
 
 {
   imports = [
-    ./dns.nix
+    ./dns
     ./hostnames.nix
     ./modemmanager.nix
     ./networkmanager.nix
+    ./ntp.nix
     ./upnp.nix
     ./vpn.nix
   ];

hosts/common/net/dns/bind.nix

@@ -0,0 +1,108 @@
+# debugging:
+# - `man named`
+# - `man named.conf`
+# - `systemctl stop bind`
+# - `sudo /nix/store/0zpdy93sd3fgbxgvf8dsxhn8fbbya8d2-bind-9.18.28/sbin/named -g -u named -4 -c /nix/store/f1mp0myzmfms71h9vinwxpn2i9362a9a-named.conf`
+#   - `-g` = don't fork
+#   - `-u named` = start as superuser (to claim port 53), then drop to user `named`
+{ config, lib, pkgs, ... }:
+let
+  hostCfg = config.sane.hosts.by-name."${config.networking.hostName}";
+  bindCfg = config.services.bind;
+in
+{
+  config = lib.mkIf (!config.sane.services.hickory-dns.asSystemResolver) {
+    services.resolved.enable = lib.mkForce false;
+
+    networking.nameservers = [
+      # be compatible with systemd-resolved
+      # "127.0.0.53"
+      # or don't be compatible with systemd-resolved, but with libc and pasta instead
+      #   see <pkgs/by-name/sane-scripts/src/sane-vpn>
+      "127.0.0.1"
+      # enable IPv6, or don't; unbound is spammy when IPv6 is enabled but unroutable
+      # "::1"
+    ];
+
+    networking.resolvconf.extraConfig = ''
+      # DNS serviced by `BIND` recursive resolver
+      name_servers='127.0.0.1'
+    '';
+
+    services.bind.enable = lib.mkDefault true;
+    services.bind.forwarders = [];  #< don't forward queries to upstream resolvers
+    services.bind.cacheNetworks = [
+      "127.0.0.0/24"
+      "::1/128"
+      "10.0.10.0/24"  #< wireguard clients (servo)
+    ];
+    services.bind.listenOn = [
+      "127.0.0.1"
+    ] ++ lib.optionals (hostCfg.wg-home.ip != null) [
+      # allow wireguard clients to use us as a recursive resolver (only needed for servo)
+      hostCfg.wg-home.ip
+    ];
+    services.bind.listenOnIpv6 = [
+      # "::1"
+    ];
+
+    services.bind.ipv4Only = true;  # unbound is spammy when it tries IPv6 without a routable address
+
+    # when testing, deploy on a port other than 53
+    # services.bind.extraOptions = ''
+    #   listen-on port 953 { any; };
+    # '';
+
+    networking.resolvconf.useLocalResolver = false;  #< we manage resolvconf explicitly, above
+
+    # TODO: how to exempt `pool.ntp.org` from DNSSEC checks, as i did when using unbound?
+
+    # allow runtime insertion of zones or other config changes:
+    # add your supplemental config as a toplevel file in /run/named/dhcp-configs/, then `systemctl restart bind`
+    services.bind.extraConfig = ''
+      include "/run/named/dhcp-configs.conf";
+    '';
+    services.bind.extraOptions = ''
+      // we can't guarantee that all forwarders support DNSSEC,
+      // and as of 2025-01-30 BIND9 gives no way to disable DNSSEC per-forwarder/zone,
+      // so just disable it globally
+      dnssec-validation no;
+    '';
+    # re-implement the nixos default bind config, but without `options { forwarders { }; };`,
+    # as having an empty `forwarders` at the top-level prevents me from forwarding the `.` zone in a separate statement
+    # (which i want to do to allow sane-vpn to forward all DNS).
+    services.bind.configFile = pkgs.writeText "named.conf" ''
+      include "/etc/bind/rndc.key";
+      controls {
+        inet 127.0.0.1 allow {localhost;} keys {"rndc-key";};
+      };
+
+      acl cachenetworks { ${lib.concatMapStrings (entry: " ${entry}; ") bindCfg.cacheNetworks} };
+      acl badnetworks { ${lib.concatMapStrings (entry: " ${entry}; ") bindCfg.blockedNetworks} };
+
+      options {
+        listen-on { ${lib.concatMapStrings (entry: " ${entry}; ") bindCfg.listenOn} };
+        listen-on-v6 { ${lib.concatMapStrings (entry: " ${entry}; ") bindCfg.listenOnIpv6} };
+        allow-query-cache { cachenetworks; };
+        blackhole { badnetworks; };
+        //v disable top-level forwards, so that i can do forwarding more generically in `zone FOO { ... }` directives.
+        // forward ${bindCfg.forward};
+        // forwarders { ${lib.concatMapStrings (entry: " ${entry}; ") bindCfg.forwarders} };
+        directory "${bindCfg.directory}";
+        pid-file "/run/named/named.pid";
+        ${bindCfg.extraOptions}
+      };
+
+      ${bindCfg.extraConfig}
+    '';
+
+    systemd.services.bind.serviceConfig.ExecStartPre = pkgs.writeShellScript "named-generate-config" ''
+      mkdir -p /run/named/dhcp-configs
+      chmod g+w /run/named/dhcp-configs
+      echo "// FILE GENERATED BY bind.service's ExecStartPre: CHANGES TO THIS FILE WILL BE OVERWRITTEN" > /run/named/dhcp-configs.conf
+      for c in $(ls /run/named/dhcp-configs/); do
+        cat "/run/named/dhcp-configs/$c" >> /run/named/dhcp-configs.conf
+      done
+    '';
+  };
+}

hosts/common/net/dns/default.nix

@@ -20,36 +20,14 @@
 # - each namespace may use a different /etc/resolv.conf to specify different DNS servers
 # - nscd breaks namespacing: the host nscd is unaware of the guest's /etc/resolv.conf, and so directs the guest's DNS requests to the host's servers.
 #   - this is fixed by either removing `/var/run/nscd/socket` from the namespace, or disabling nscd altogether.
-{ config, lib, pkgs, ... }:
-lib.mkMerge [
+{ config, pkgs, ... }:
 {
-  sane.services.hickory-dns.enable = lib.mkDefault config.sane.services.hickory-dns.asSystemResolver;
-  sane.services.hickory-dns.asSystemResolver = lib.mkDefault true;
-}
-(lib.mkIf (!config.sane.services.hickory-dns.asSystemResolver) {
-  # use systemd's stub resolver.
-  # /etc/resolv.conf isn't sophisticated enough to use different servers per net namespace (or link).
-  # instead, running the stub resolver on a known address in the root ns lets us rewrite packets
-  # in servo's ovnps namespace to use the provider's DNS resolvers.
-  # a weakness is we can only query 1 NS at a time (unless we were to clone the packets?)
-  # TODO: improve hickory-dns recursive resolver and then remove this
-  services.resolved.enable = true;  #< to disable, set ` = lib.mkForce false`, as other systemd features default to enabling `resolved`.
-  # without DNSSEC:
-  # - dig matrix.org => works
-  # - curl https://matrix.org => works
-  # with default DNSSEC:
-  # - dig matrix.org => works
-  # - curl https://matrix.org => fails
-  # i don't know why. this might somehow be interfering with the DNS run on this device (hickory-dns)
-  services.resolved.dnssec = "false";
-  networking.nameservers = [
-    # use systemd-resolved resolver
-    # full resolver (which understands /etc/hosts) lives on 127.0.0.53
-    # stub resolver (just forwards upstream) lives on 127.0.0.54
-    "127.0.0.53"
+  imports = [
+    ./bind.nix
+    ./hickory-dns.nix
+    ./unbound.nix
   ];
-})
-{
+
   # nscd -- the Name Service Caching Daemon -- caches DNS query responses
   # in a way that's unaware of my VPN routing, so routes are frequently poor against
   # services which advertise different IPs based on geolocation.
@@ -72,6 +50,7 @@ lib.mkMerge [
   services.nscd.enable = false;
   # system.nssModules = lib.mkForce [];
   sane.silencedAssertions = [''.*Loading NSS modules from system.nssModules.*requires services.nscd.enable being set to true.*''];
+
   # add NSS modules into their own subdirectory.
   # then i can add just the NSS modules library path to the global LD_LIBRARY_PATH, rather than ALL of /run/current-system/sw/lib.
   # TODO: i'm doing this so as to achieve mdns DNS resolution (avahi). it would be better to just have hickory-dns delegate .local to avahi
@@ -90,4 +69,3 @@ lib.mkMerge [
   environment.variables.LD_LIBRARY_PATH = [ "/run/current-system/sw/lib/nss" ];
   systemd.globalEnvironment.LD_LIBRARY_PATH = "/run/current-system/sw/lib/nss";  #< specifically for `geoclue.service`
 }
-]

hosts/common/net/dns/hickory-dns.nix

@@ -0,0 +1,31 @@
+{ config, lib, ... }:
+lib.mkIf false  #< XXX(2024-10-xx): hickory-dns recursive resolution is too immature; switched to `unbound`
+(lib.mkMerge [
+ {
+   sane.services.hickory-dns.enable = lib.mkDefault config.sane.services.hickory-dns.asSystemResolver;
+   # sane.services.hickory-dns.asSystemResolver = lib.mkDefault true;
+ }
+ (lib.mkIf (!config.sane.services.hickory-dns.asSystemResolver && config.sane.services.hickory-dns.enable) {
+   # use systemd's stub resolver.
+   # /etc/resolv.conf isn't sophisticated enough to use different servers per net namespace (or link).
+   # instead, running the stub resolver on a known address in the root ns lets us rewrite packets
+   # in servo's ovnps namespace to use the provider's DNS resolvers.
+   # a weakness is we can only query 1 NS at a time (unless we were to clone the packets?)
+   # TODO: improve hickory-dns recursive resolver and then remove this
+   services.resolved.enable = true;  #< to disable, set ` = lib.mkForce false`, as other systemd features default to enabling `resolved`.
+   # without DNSSEC:
+   # - dig matrix.org => works
+   # - curl https://matrix.org => works
+   # with default DNSSEC:
+   # - dig matrix.org => works
+   # - curl https://matrix.org => fails
+   # i don't know why. this might somehow be interfering with the DNS run on this device (hickory-dns)
+   services.resolved.dnssec = "false";
+   networking.nameservers = [
+     # use systemd-resolved resolver
+     # full resolver (which understands /etc/hosts) lives on 127.0.0.53
+     # stub resolver (just forwards upstream) lives on 127.0.0.54
+     "127.0.0.53"
+   ];
+ })
+])

hosts/common/net/dns/unbound.nix

@@ -0,0 +1,93 @@
+# `man unbound.conf` for info on settings
+# it's REALLY EASY to combine settings in a way that produce bad effects.
+# generally, prefer to stay close to defaults unless there's a compelling reason to differ.
+{ config, lib, ... }:
+lib.optionalAttrs false  #< XXX(2024-12-29): unbound caches failed DNS resolutions, just randomly breaks connectivity daily
+{
+  config = lib.mkIf (!config.sane.services.hickory-dns.asSystemResolver) {
+    services.resolved.enable = lib.mkForce false;
+
+    networking.nameservers = [
+      # be compatible with systemd-resolved
+      # "127.0.0.53"
+      # or don't be compatible with systemd-resolved, but with libc and pasta instead
+      #   see <pkgs/by-name/sane-scripts/src/sane-vpn>
+      "127.0.0.1"
+      # enable IPv6, or don't, because having just a single name server makes monkey-patching it easier
+      # "::1"
+    ];
+    networking.resolvconf.extraConfig = ''
+      # DNS serviced by `unbound` recursive resolver
+      name_servers='127.0.0.1'
+    '';
+
+    # resolve DNS recursively with Unbound.
+    services.unbound.enable = lib.mkDefault true;
+    services.unbound.resolveLocalQueries = false;  #< disable, so that i can manage networking.nameservers manually
+    services.unbound.settings.server.interface = [ "127.0.0.1" ];
+    services.unbound.settings.server.access-control = [ "127.0.0.0/8 allow" ];
+
+    # allow control via `unbound-control`. user must be a member of the `unbound` Unix group.
+    services.unbound.localControlSocketPath = "/run/unbound/unbound.ctl";
+
+    # exempt `pool.ntp.org` from DNSSEC checks to avoid a circular dependency between DNS resolution and NTP.
+    # without this, if the RTC fails, then both time and DNS are unrecoverable.
+    services.unbound.settings.server.domain-insecure = config.networking.timeServers;
+
+    # XXX(2024-12-03): BUG: during boot (before network is up), or during network blips, Unbound will
+    # receive a query, fail to evaluate it, and then resolve future identical queries with a no-answers response for the next ~15m.
+    # this *appears* to be some bug in Unbound's "infra-cache", as evidenced by `unbound-control flush_infra all`.
+    #
+    # the infra cache is a per-nameserver liveness and latency cache which Unbound uses to decide which of N applicable nameservers to route a given query to.
+    #
+    # there is apparently NO simple solution.
+    # the closest fix is to reduce the TTL of the infra-cache (`infra-host-ttl`) so as to limit the duration of this error.
+    # tried, but failed fixes:
+    # - server.harden-dnssec-stripped = false
+    # - services.unbound.enableRootTrustAnchor = false;  #< disable DNSSEC
+    #   - server.trust-anchor-file = "${pkgs.dns-root-data}/root.key";  #< hardcode root keys instead of dynamically probing them
+    # - server.disable-dnssec-lame-check = true;
+    # - server.infra-keep-probing = true;  #< if unbound fails to reach a host (NS), it by default *does not try again* for 900s. keep-probing tells it to keep trying, with a backoff.
+    # - server.infra-cache-min-rtt = 1000;
+    # - server.infra-cache-max-rtt = 1000;
+    #
+    # see also:
+    # - <https://forum.opnsense.org/index.php?topic=32852.0>
+    # - <https://unbound.docs.nlnetlabs.nl/en/latest/reference/history/info-timeout-server-selection.html>
+    #
+    services.unbound.settings.server.infra-host-ttl = 30;  #< cache each NS's liveness for a max of 30s
+
+    # perf tuning; see: <https://unbound.docs.nlnetlabs.nl/en/latest/topics/core/performance.html>
+    # resource usage:
+    # - defaults (num-threads = 1; so-{rcvbuf,sndbuf} = 0, prefetch = false): 12.7M memory usage
+    # - num-threads = 2: 17.2M memory usage
+    # - num-threads = 4: 26.2M memory usage
+    # - num-threads = 4; so-{rcvbuf,sndbuf}=4m: 26.7M memory usage
+    # - prefetch = true: no increased memory; supposed 10% increase in traffic
+    #
+    # # i suspect most operations are async; the only serialized bits are either CPU or possibly local IO (i.e. syscalls to write sockets).
+    # # threading is probably only rarely helpful
+    # services.unbound.settings.server.num-threads = 4;
+    #
+    # higher so-rcvbuf means less likely to drop client queries...
+    # default is `cat /proc/sys/net/core/wmem_default`, i.e. 208k
+    # services.unbound.settings.server.so-rcvbuf = "1m";
+    # services.unbound.settings.server.so-sndbuf = "1m";
+    #
+    # `prefetch`: prefetch RRs which are about to expire from the cache, to keep them primed.
+    # services.unbound.settings.server.prefetch = true;
+
+    # if a resolution fails, or takes excessively long, reply with expired cache entries
+    # see: <https://unbound.docs.nlnetlabs.nl/en/latest/topics/core/serve-stale.html#rfc-8767>
+    services.unbound.settings.server.serve-expired = true;
+    services.unbound.settings.server.serve-expired-ttl = 86400;  #< don't serve any records more outdated than this
+    services.unbound.settings.server.serve-expired-client-timeout = 2800;  #< only serve expired records if the client has been waiting this long, ms
+
+    # `cache-max-negative-ttl`: intended to limit damage during networking flakes, but instead seems to cause unbound to cache error responses it *wouldn't* otherwise cache
+    # services.unbound.settings.server.cache-max-negative-ttl = 60;
+
+    # `user-caps-for-id`: randomizes casing to avoid spoofing, but causes unbound to reply with no results to queries after boot (likely a infra-cache issue)
+    # services.unbound.settings.server.use-caps-for-id = true;
+  };
+}
+

hosts/common/net/networkmanager.nix

@@ -1,27 +1,5 @@
-{ config, pkgs, ... }:
-let
-  # networkmanager = pkgs.networkmanager;
-  networkmanager = pkgs.networkmanager.overrideAttrs (upstream: {
-    # src = pkgs.fetchFromGitea {
-    #   domain = "git.uninsane.org";
-    #   owner = "colin";
-    #   repo = "NetworkManager";
-    #   # patched to fix polkit permissions (with `nmcli`) when NetworkManager runs as user networkmanager
-    #   rev = "dev-sane-1.48.0";
-    #   hash = "sha256-vGmOKtwVItxjYioZJlb1og3K6u9s4rcmDnjAPLBC3ao=";
-    # };
-    patches = (upstream.patches or []) ++ [
-      (pkgs.fetchpatch {
-        name = "polkit: add owner annotations to all actions";
-        url = "https://git.uninsane.org/colin/NetworkManager/commit/a01293861fa24201ffaeb84c07f1c71136c49759.patch";
-        hash = "sha256-th1/M2slo7rjkVBwETZII53Lmhyw8OMS0aT9QYI5Uvk=";
-      })
-    ];
-  });
-  # split the package into `daemon` and `nmcli` outputs, because the networkmanager *service*
-  # doesn't need `nmcli`/`nmtui` tooling
-  networkmanager-split = pkgs.networkmanager-split.override { inherit networkmanager; };
-in {
+{ config, lib, pkgs, ... }:
+{
   networking.networkmanager.enable = true;
   systemd.network.wait-online.enable = false;  # systemd-networkd-wait-online.service reliably fails on lappy. docs don't match behavior. shit software.
   # plugins mostly add support for establishing different VPN connections.
@@ -36,10 +14,9 @@ in {
   #
   # i don't use these, and notably they drag in huge dependency sets and don't cross compile well.
   # e.g. openconnect drags in webkitgtk (for SSO)!
-  # networking.networkmanager.plugins = lib.mkForce [];
-  networking.networkmanager.enableDefaultPlugins = false;
+  networking.networkmanager.plugins = lib.mkForce [];
 
-  networking.networkmanager.package = networkmanager-split.daemon.overrideAttrs (upstream: {
+  networking.networkmanager.package = pkgs.networkmanager-split.daemon.overrideAttrs (upstream: {
     # postPatch = (upstream.postPatch or "") + ''
     #   substituteInPlace src/{core/org.freedesktop.NetworkManager,nm-dispatcher/nm-dispatcher}.conf --replace-fail \
     #     'user="root"' 'user="networkmanager"'
@@ -48,7 +25,9 @@ in {
       # allow the bus to owned by either root or networkmanager users
       # use the group here, that way ordinary users can be elevated to control networkmanager
       # (via e.g. `nmcli`)
-      for f in org.freedesktop.NetworkManager.conf nm-dispatcher.conf ; do
+      confs=(nm-dispatcher.conf)
+      confs+=(org.freedesktop.NetworkManager.conf)
+      for f in "''${confs[@]}" ; do
         substitute $out/share/dbus-1/system.d/$f \
           $out/share/dbus-1/system.d/networkmanager-$f \
           --replace-fail 'user="root"' 'group="networkmanager"'
@@ -66,15 +45,23 @@ in {
     serviceConfig.User = "networkmanager";
     serviceConfig.Group = "networkmanager";
     serviceConfig.AmbientCapabilities = [
+      "CAP_KILL"  #< required, else `nmcli d disconnect blah` says "Unable to determine UID of the request"
       "CAP_NET_ADMIN"
       "CAP_NET_RAW"
       "CAP_NET_BIND_SERVICE"
+
+      # "CAP_DAC_OVERRIDE"
+      # "CAP_SYS_MODULE"
+      # "CAP_AUDIT_WRITE"  #< allow writing to the audit log (optional)
+      # "CAP_KILL"
     ];
     serviceConfig.CapabilityBoundingSet = [
-      # "CAP_DAC_OVERRIDE"
+      "CAP_KILL"  #< required, else `nmcli d disconnect blah` says "Unable to determine UID of the request"
       "CAP_NET_ADMIN"
       "CAP_NET_RAW"  #< required, else `libndp: ndp_sock_open: Failed to create ICMP6 socket.`
       "CAP_NET_BIND_SERVICE"  #< this *does* seem to be necessary, though i don't understand why. DHCP?
+
+      # "CAP_DAC_OVERRIDE"
       # "CAP_SYS_MODULE"
       # "CAP_AUDIT_WRITE"  #< allow writing to the audit log (optional)
       # "CAP_KILL"
@@ -92,9 +79,9 @@ in {
     serviceConfig.ProtectHostname = true;  # probably not upstreamable: prevents changing hostname
     serviceConfig.ProtectKernelLogs = true;  # disable /proc/kmsg, /dev/kmsg
     serviceConfig.ProtectKernelModules = true;  # syscall filter to prevent module calls (probably not upstreamable: NM will want to load modules like `ppp`)
-    serviceConfig.ProtectKernelTunables = true;  # but NM might need to write /proc/sys/net/...
+    # serviceConfig.ProtectKernelTunables = true;  # causes errors/warnings when opening files in /proc/sys/net/...; also breaks IPv6 SLAAC / link-local address creation!
     serviceConfig.ProtectProc = "invisible";
-    serviceConfig.ProcSubset = "pid";
+    serviceConfig.ProcSubset = "all";
     serviceConfig.ProtectSystem = "strict";  # makes read-only: all but /dev, /proc, /sys.
     serviceConfig.RemoveIPC = true;
     serviceConfig.RestrictAddressFamilies = [
@@ -140,7 +127,7 @@ in {
     #VVV so that /var/lib/hickory-dns will exist (the hook needs to write here).
     # but this creates a cycle: hickory-dns-localhost > network.target > NetworkManager-dispatcher > hickory-dns-localhost.
     # (seemingly) impossible to remove the network.target dep on NetworkManager-dispatcher.
-    # beffore would be to have the dispatcher not write hickory-dns files
+    # before would be to have the dispatcher not write hickory-dns files
     # but rather just its own, and create a .path unit which restarts hickory-dns appropriately.
     # after = [ "hickory-dns-localhost.service" ];
     # serviceConfig.ExecStart = [
@@ -234,7 +221,9 @@ in {
     # note that NM's resolv.conf isn't (necessarily) /etc/resolv.conf -- that is managed by nixos (via symlinking)
     main.dns = if config.services.resolved.enable then
       "systemd-resolved"
-    else if config.sane.services.hickory-dns.enable && config.sane.services.hickory-dns.asSystemResolver then
+    else if
+      (config.sane.services.hickory-dns.enable && config.sane.services.hickory-dns.asSystemResolver)
+      || (config.services.unbound.enable && config.services.unbound.resolveLocalQueries) then
       "none"
     else
       "internal"

hosts/common/net/ntp.nix

@@ -0,0 +1,32 @@
+# NTP and DNS/DNSSEC have a chicken-and-egg issue:
+# - NTP needs to resolve DNS to know how to query the servers (`0.nixos.pool.ntp.org`, etc)
+# - DNS needs to have a semi-accurate clock to validate DNSSEC for resolutions
+#
+# nixos and systemd-timesyncd overcome this in the default installation by:
+# - setting `SYSTEMD_NSS_RESOLVE_VALIDATE=0` in the systemd-timesyncd.service unit file
+# - systemd nss module which plumbs that to systemd-resolved
+# that ONLY WORKS if using systemd-resolved.
+#
+# my alternative fix here is to hardcode a list of fallback NTP IP addresses, to use when DNS resolution of the primaries fails.
+#
+# lastly, the clock can be manually set:
+# - `systemctl stop systemd-timesyncd`
+# - `sudo timedatectl --adjust-system-clock set-time '2024-01-01 00:00:01 UTC'`
+# - `systemctl start systemd-timesyncd`
+#
+# XXX(2024-12-03): i fixed the NTP-DNS circularity by exempting `pool.ntp.org` from DNSSEC validation in unbound conf
+{ config, ... }:
+{
+  # services.timesyncd.servers = config.networking.timeServers;
+  # services.timesyncd.fallbackServers = [
+  #   "129.6.15.28"  # time-a-g.nist.gov
+  #   "132.163.97.1"  # time-a-wwv.nist.gov
+  #   "132.163.96.1"  # time-a-b.nist.gov
+  #   "128.138.140.44"  # utcnist.colorado.edu
+  #   "162.159.200.1"  # time.cloudflare.com
+  # ];
+
+  # more feature-complete NTP implementations exist, like `chrony`, should i ever wish to also be a NTP **server**:
+  # services.chrony.enable = true;
+  # services.chrony.enableNTS = true;
+}

hosts/common/net/upnp.nix

@@ -16,5 +16,9 @@
     ${ipset}/bin/ipset create -! upnp hash:ip,port timeout 10
     ${iptables}/bin/iptables -A OUTPUT -d 239.255.255.250/32 -p udp -m udp --dport 1900 -j SET --add-set upnp src,src --exist
     ${iptables}/bin/iptables -A INPUT -p udp -m set --match-set upnp dst,dst -j ACCEPT
+    # IPv6 ruleset. ff02::/16 means *any* link-local multicast group (so this is probably more broad than it needs to be)
+    ${ipset}/bin/ipset create -! upnp6 hash:ip,port timeout 10 family inet6
+    ${iptables}/bin/ip6tables -A OUTPUT -d ff02::/16 -p udp -m udp --dport 1900 -j SET --add-set upnp6 src,src --exist
+    ${iptables}/bin/ip6tables -A INPUT -p udp -m set --match-set upnp6 dst,dst -j ACCEPT
   '';
 }

hosts/common/net/vpn.nix

@@ -5,7 +5,7 @@
 # - generate config @ OVPN.com
 # - copy the Address, PublicKey, Endpoint from OVPN's config
 
-{ config, lib, pkgs, ... }:
+{ config, lib, ... }:
 let
   # N.B.: OVPN issues each key (i.e. device) a different IP (addrV4), and requires you use it.
   # the IP it issues can be used to connect to any of their VPNs.

hosts/common/nix.nix

@@ -64,7 +64,12 @@
     # it's an impurity that touches way more than i need and tends to cause hard-to-debug eval issues
     # when it goes wrong. should i port my `nix-shell` scripts to something more tailored to my uses
     # and then delete `nixpkgs-overlays`?
-    "nixpkgs-overlays=/home/colin/dev/nixos/integrations/nixpkgs/nixpkgs-overlays.nix"
+    # "nixpkgs-overlays=/home/colin/dev/nixos/integrations/nixpkgs/nixpkgs-overlays.nix"
+    # XXX(2024-09-02): nix 2.24.4 errors when nixpkgs-overlays includes a symlink component:
+    # "error: path '/home/colin/dev' is a symlink"
+    # apparently nix has to explicitly handle symlinks in every place it might encounter them,
+    # so the fixes inside nix for this are manual and fragile. dereference it ourselves:
+    "nixpkgs-overlays=${config.sane.fs."/home/colin/dev".symlink.target}/nixos/integrations/nixpkgs/nixpkgs-overlays.nix"
   ];
 
   # ensure new deployments have a source of this repo with which they can bootstrap.

hosts/common/polyunfill.nix

@@ -6,7 +6,7 @@ let
     # nixpkgs' pam hardcodes unix_chkpwd path to the /run/wrappers one,
     # but i don't want the wrapper, so undo that.
     # ideally i would patch this via an overlay, but pam is in the bootstrap so that forces a full rebuild.
-    postPatch = (if upstream.postPatch != null then upstream.postPatch else "") + ''
+    postPatch = (upstream.postPatch or "") + ''
       substituteInPlace modules/pam_unix/Makefile.am --replace-fail \
         "/run/wrappers/bin/unix_chkpwd" "$out/bin/unix_chkpwd"
     '';
@@ -59,7 +59,7 @@ in
       "userdel"
       "usermod"
       # from <repo:nixos/nixpkgs:nixos/modules/system/boot/systemd/user.nix>
-      "systemd-user"  #< N.B.: this causes the `systemd --user` service manager to not be started!
+      # "systemd-user"  #< N.B.: this causes the `systemd --user` service manager to fail 224/PAM!
     ]));
   };
 
@@ -106,6 +106,7 @@ in
       conveniencePackages = [
         config.boot.kernelPackages.cpupower  # <repo:nixos/nixpkgs:nixos/modules/tasks/cpu-freq.nix> places it on PATH for convenience if powerManagement.cpuFreqGovernor is set
         pkgs.kbd  # <repo:nixos/nixpkgs:nixos/modules/config/console.nix>  places it on PATH as part of console/virtual TTYs, but probably not needed unless you want to set console fonts
+        pkgs.nixos-firewall-tool  # <repo:nixos/nixpkgs:nixos/modules/services/networking/firewall.nix>  for end-user management of the firewall? cool but doesn't cross-compile
       ];
     in lib.filter (p: ! builtins.elem p (requiredPackages ++ conveniencePackages));
   };
@@ -151,7 +152,6 @@ in
 
     # nix.channel.enable: populates `/nix/var/nix/profiles/per-user/root/channels`, `/root/.nix-channels`, `$HOME/.nix-defexpr/channels`
     #   <repo:nixos/nixpkgs:nixos/modules/config/nix-channel.nix>
-    # TODO: may want to recreate NIX_PATH, nix.settings.nix-path
     nix.channel.enable = false;
 
     # environment.stub-ld: populate /lib/ld-linux.so with an object that unconditionally errors on launch,

hosts/common/programs/aerc.nix

@@ -3,8 +3,7 @@
 
 {
   sane.programs.aerc = {
-    sandbox.method = "bwrap";
-    sandbox.wrapperType = "inplace";  #< /share/aerc/aerc.conf refers to other /share files by absolute path
+    sandbox.wrapperType = "inplace";  #< /share/aerc/aerc.conf mentions (in comments) other (non-sandboxed) /share files by absolute path
     sandbox.net = "clearnet";
     secrets.".config/aerc/accounts.conf" = ../../../secrets/common/aerc_accounts.conf.bin;
     mime.associations."x-scheme-handler/mailto" = "aerc.desktop";

hosts/common/programs/alacritty.nix

@@ -26,6 +26,9 @@ in
       [font]
       size = ${builtins.toString cfg.config.fontSize}
 
+      [cursor.style]
+      blinking = "Always"
+
       [[keyboard.bindings]]
       mods = "Control"
       key = "N"

hosts/common/programs/alsa-ucm-conf/default.nix

@@ -4,43 +4,12 @@ let
 in
 {
   sane.programs.alsa-ucm-conf = {
-    configOption = with lib; mkOption {
-      default = {};
-      type = types.submodule {
-        options.preferEarpiece = mkOption {
-          type = types.bool;
-          default = true;
-        };
+    packageUnwrapped = pkgs.alsa-ucm-conf.overrideAttrs (base: {
+      meta = (base.meta or {}) // {
+        # let the other alsa ucm packages override configs from this one
+        priority = ((base.meta or {}).priority or 10) + 20;
       };
-    };
-
-    # upstream alsa ships with PinePhone audio configs, but they don't actually produce sound.
-    # - still true as of 2024-05-26
-    # - see: <https://github.com/alsa-project/alsa-ucm-conf/pull/134>
-    #
-    # we can substitute working UCM conf in two ways:
-    # 1. nixpkgs' override for the `alsa-ucm-conf` package
-    #   - that forces a rebuild of ~500 packages (including webkitgtk).
-    # 2. set ALSA_CONFIG_UCM2 = /path/to/ucm2 in the relevant places
-    #   - e.g. pulsewire service.
-    #   - easy to miss places, though.
-    #
-    # alsa-ucm-pinephone-manjaro (2024-05-26):
-    # - headphones work
-    # - "internal earpiece" works
-    # - "internal speaker" is silent (maybe hardware issue)
-    # - 3.5mm connection is flapping when playing to my car, which eventually breaks audio and requires restarting wireplumber
-    # packageUnwrapped = pkgs.alsa-ucm-pinephone-manjaro.override {
-    #   inherit (cfg.config) preferEarpiece;
-    # };
-    # alsa-ucm-pinephone-pmos (2024-05-26):
-    # - headphones work
-    # - "internal earpiece" works
-    # - "internal speaker" is silent (maybe hardware issue)
-    packageUnwrapped = pkgs.alsa-ucm-pinephone-pmos.override {
-      inherit (cfg.config) preferEarpiece;
-    };
-
+    });
     sandbox.enable = false;  #< only provides $out/share/alsa
 
     # alsa-lib package only looks in its $out/share/alsa to find runtime config data, by default.
@@ -48,6 +17,8 @@ in
     # this is particularly needed by wireplumber;
     #   also *maybe* pipewire and pipewire-pulse.
     # taken from <repo:nixos/mobile-nixos:modules/quirks/audio.nix>
+    # the other option is to `override` pkgs.alsa-ucm-conf,
+    # but that triggers 500+ rebuilds
     env.ALSA_CONFIG_UCM2 = "/run/current-system/sw/share/alsa/ucm2";
 
     enableFor.system = lib.mkIf (builtins.any (en: en) (builtins.attrValues cfg.enableFor.user)) true;

hosts/common/programs/animatch.nix

@@ -32,7 +32,6 @@
 
     buildCost = 1;
 
-    sandbox.method = "bwrap";
     sandbox.whitelistWayland = true;
 
     persist.byStore.plaintext = [

hosts/common/programs/audacity.nix

@@ -6,17 +6,22 @@
 { pkgs, ... }:
 {
   sane.programs.audacity = {
-    packageUnwrapped = pkgs.audacity.override {
+    packageUnwrapped = (pkgs.audacity.override {
       # wxGTK32 uses webkitgtk-4.0.
       # audacity doesn't actually need webkit though, so diable to reduce closure
       wxGTK32 = pkgs.wxGTK32.override {
         withWebKit = false;
       };
-    };
+    }).overrideAttrs (base: {
+      # upstream audacity.desktop specifies GDK_BACKEND=x11, with which it doesn't actually launch :|
+      postInstall = (base.postInstall or "") + ''
+        substituteInPlace $out/share/applications/audacity.desktop \
+          --replace-fail 'GDK_BACKEND=x11 ' ""
+      '';
+    });
 
     buildCost = 1;
 
-    sandbox.method = "bwrap";
     sandbox.whitelistAudio = true;
     sandbox.whitelistWayland = true;
     sandbox.autodetectCliPaths = "existingFile";

hosts/common/programs/ausyscall.nix

@@ -4,7 +4,6 @@
   sane.programs.ausyscall = {
     packageUnwrapped = pkgs.linkBinIntoOwnPackage pkgs.audit "ausyscall";
 
-    sandbox.method = "landlock";
   };
 }
 

hosts/common/programs/avahi.nix

@@ -11,12 +11,15 @@
 #   nss-mdns goes through avahi-daemon, so there IS caching here
 #
 { config, lib, pkgs, ... }:
+let
+  cfg = config.sane.programs.avahi;
+in
 {
   sane.programs.avahi = {
     packageUnwrapped = pkgs.avahi.overrideAttrs (upstream: {
       # avahi wants to do its own sandboxing opaque to systemd & maybe in conflict with my bwrap.
       # --no-drop-root disables that, so that i can e.g. run it as User=avahi, etc.
-      # do this here, because the service isn't so easily patched.
+      # do this here, because the nixos service isn't so easily patched.
       postInstall = (upstream.postInstall or "") + ''
         wrapProgram "$out/sbin/avahi-daemon" \
           --add-flags --no-drop-root
@@ -25,16 +28,14 @@
         pkgs.makeBinaryWrapper
       ];
     });
-    sandbox.method = "bwrap";
-    sandbox.whitelistDbus = [ "system" ];
+    sandbox.whitelistDbus.system = true;
     sandbox.net = "all";  #< otherwise it will show 'null' in place of each interface name.
-    sandbox.extraPaths = [
-      "/"  #< TODO: decrease this, but be weary that the daemon might exit immediately
-    ];
+    # sandbox.extraPaths = [ ];  #< may be missing some paths; only tried service discovery, not service advertisement.
   };
-  services.avahi = lib.mkIf config.sane.programs.avahi.enabled {
+
+  services.avahi = lib.mkIf cfg.enabled {
     enable = true;
-    package = config.sane.programs.avahi.package;
+    package = cfg.packageUnwrapped;  #< use systemd sandboxing... not my own
     publish.enable = true;
     publish.userServices = true;
     nssmdns4 = true;
@@ -53,39 +54,26 @@
     ];
   };
 
-  systemd.services.avahi-daemon = lib.mkIf config.sane.programs.avahi.enabled {
+  # fix "rpfilter drop ..." dmesg logspam.
+  # this might not be necessary?
+  networking.firewall.extraCommands = lib.mkIf cfg.enabled (with pkgs; ''
+    # after an outgoing mDNS query to the multicast address, open FW for incoming responses.
+    # ipset -! means "don't fail if set already exists"
+    ${lib.getExe' ipset "ipset"} create -! mdns hash:ip,port timeout 10
+    ${lib.getExe' iptables "iptables"} -A OUTPUT -d 239.255.255.250/32 -p udp -m udp --dport 5353 -j SET --add-set mdns src,src --exist
+    ${lib.getExe' iptables "iptables"} -A INPUT -p udp -m set --match-set mdns dst,dst -j ACCEPT
+    # IPv6 ruleset. ff02::/16 means *any* link-local multicast group (so this is probably more broad than it needs to be)
+    ${lib.getExe' ipset "ipset"} create -! mdns6 hash:ip,port timeout 10 family inet6
+    ${lib.getExe' iptables "ip6tables"} -A OUTPUT -d ff02::/16 -p udp -m udp --dport 5353 -j SET --add-set mdns6 src,src --exist
+    ${lib.getExe' iptables "ip6tables"} -A INPUT -p udp -m set --match-set mdns6 dst,dst -j ACCEPT
+  '');
+
+  systemd.services.avahi-daemon = lib.mkIf cfg.enabled {
     # hardening: see `systemd-analyze security avahi-daemon`
     serviceConfig.User = "avahi";
     serviceConfig.Group = "avahi";
     serviceConfig.AmbientCapabilities = "";
-    serviceConfig.CapabilityBoundingSet = "";
-    serviceConfig.LockPersonality = true;
-    serviceConfig.MemoryDenyWriteExecute = true;
-    serviceConfig.NoNewPrivileges = true;
-    serviceConfig.PrivateDevices = true;
-    serviceConfig.PrivateMounts = true;
-    serviceConfig.PrivateTmp = true;
-    serviceConfig.PrivateUsers = true;
-    serviceConfig.ProcSubset = "all";
-    serviceConfig.ProtectClock = true;
-    serviceConfig.ProtectControlGroups = true;
-    serviceConfig.ProtectHome = true;
-    serviceConfig.ProtectHostname = true;
-    serviceConfig.ProtectKernelLogs = true;
-    serviceConfig.ProtectKernelModules = true;
-    serviceConfig.ProtectKernelTunables = true;
-    serviceConfig.ProtectProc = "noaccess";
-    serviceConfig.ProtectSystem = "strict";
-    serviceConfig.RemoveIPC = true;  #< this *might* slow down the initial connection?
-    serviceConfig.RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6 AF_NETLINK";
-    serviceConfig.RestrictRealtime = true;
-    serviceConfig.RestrictSUIDSGID = true;
-    serviceConfig.SystemCallArchitectures = "native";
-    serviceConfig.SystemCallFilter = [
-      "@system-service"
-      "@mount"
-      "~@resources"
-      # "~@privileged"
-    ];
+    serviceConfig.CapabilityBoundingSet = lib.mkForce "";
+    serviceConfig.PrivateUsers = lib.mkForce true;
   };
 }

hosts/common/programs/bemenu.nix

@@ -87,11 +87,7 @@ let
 in
 {
   sane.programs.bemenu = {
-    sandbox.method = "bwrap";  # landlock works, but requires *all* of $XDG_RUNTIME_DIR to be granted.
     sandbox.whitelistWayland = true;
-    sandbox.extraHomePaths = [
-      ".cache/fontconfig"  #< else it complains, and is *way* slower
-    ];
 
     packageUnwrapped = pkgs.bemenu.overrideAttrs (upstream: {
       nativeBuildInputs = (upstream.nativeBuildInputs or []) ++ [

hosts/common/programs/bitcoin-cli.nix

@@ -2,7 +2,6 @@
 {
   sane.programs.bitcoin-cli = {
     packageUnwrapped = pkgs.linkBinIntoOwnPackage pkgs.bitcoind "bitcoin-cli";
-    sandbox.method = "bwrap";
     sandbox.autodetectCliPaths = "existing";  #< for `bitcoin-cli -datadir=/var/lib/...`
     sandbox.extraHomePaths = [
       ".bitcoin/bitcoin.conf"

hosts/common/programs/blanket.nix

@@ -0,0 +1,13 @@
+{ ... }:
+{
+  sane.programs.blanket = {
+    # com.rafaelmardojai.Blanket
+    buildCost = 1;
+    sandbox.whitelistAudio = true;
+    sandbox.whitelistDbus.user.own = [
+      "com.rafaelmardojai.Blanket"
+      "org.mpris.MediaPlayer2.Blanket"
+    ];
+    sandbox.whitelistWayland = true;
+  };
+}

hosts/common/programs/blueberry.nix

@@ -0,0 +1,24 @@
+{ config, lib, ... }:
+let
+  cfg = config.sane.programs.blueberry;
+in
+{
+  sane.programs.blueberry = {
+    sandbox.wrapperType = "inplace";  #< it places binaries in /lib and then /etc/xdg/autostart files refer to the /lib paths, and fail to be patched
+    sandbox.whitelistWayland = true;
+    sandbox.extraPaths = [
+      "/dev/rfkill"
+      "/run/dbus"
+      "/sys/class/rfkill"
+      "/sys/devices"
+    ];
+    sandbox.keepPids = true;  #< not sure why, but it fails to launch GUI without this
+  };
+
+  # TODO: hardware.bluetooth puts like 100 binaries from `bluez` onto PATH;
+  # i can probably patch this so it's just `bluetoothd`.
+  # see: <repo:nixos/nixpkgs:nixos/modules/services/hardware/bluetooth.nix>
+  hardware.bluetooth = lib.mkIf cfg.enabled {
+    enable = true;
+  };
+}

hosts/common/programs/blueman.nix

@@ -0,0 +1,6 @@
+{ ... }:
+{
+  sane.programs.blueman = {
+    sandbox.method = null;  #< TODO: sandbox
+  };
+}

hosts/common/programs/bonsai.nix

@@ -1,93 +1,7 @@
 # bonsai docs: <https://sr.ht/~stacyharper/bonsai/>
-{ config, lib, pkgs, ... }:
+{ config, lib, options, pkgs, ... }:
 let
   cfg = config.sane.programs.bonsai;
-
-  delayType = with lib; types.submodule {
-    options = {
-      type = mkOption {
-        type = types.enum [ "delay" ];
-        # default = "delay";
-      };
-      delay_duration = mkOption {
-        type = types.int;
-        description = ''
-          used for "delay" types only.
-          nanoseconds until the event is finalized.
-        '';
-      };
-      transitions = mkOption {
-        type = types.listOf transitionType;
-        default = [];
-        description = ''
-          list of transitions out of this state (i.e. after completing the delay).
-        '';
-      };
-    };
-  };
-  eventType = with lib; types.submodule {
-    options = {
-      type = mkOption {
-        type = types.enum [ "event" ];
-        # default = "event";
-      };
-      event_name = mkOption {
-        type = types.str;
-        description = ''
-          name of event which this transition applies to.
-        '';
-      };
-      transitions = mkOption {
-        type = types.listOf transitionType;
-        default = [];
-        description = ''
-          list of transitions out of this state.
-        '';
-      };
-    };
-  };
-  execType = with lib; types.submodule {
-    options = {
-      type = mkOption {
-        type = types.enum [ "exec" ];
-        # default = "exec";
-      };
-      command = mkOption {
-        type = types.listOf types.str;
-        description = ''
-          command to run when the event is triggered.
-        '';
-      };
-      transitions = mkOption {
-        type = types.listOf transitionType;
-        default = [];
-        description = ''
-          list of transitions out of this state (i.e. after successfully executing the command)
-        '';
-      };
-    };
-  };
-  isDelay = x: delayType.check x && x.type == "delay";
-  isEvent = x: eventType.check x && x.type == "event";
-  isExec = x: execType.check x && x.type == "exec";
-  # unfortunately, `types.oneOf` is naive about submodules, so we need our own type.
-  #   transitionType = lib.types.oneOf [ delayType eventType execType ];
-  transitionType = with lib.types; mkOptionType {
-    name = "transition";
-    check = x: isDelay x || isEvent x || isExec x;
-    merge = loc: defs: let
-      defList = builtins.map (d: d.value) defs;
-    in
-      if builtins.all isDelay defList then
-        delayType.merge loc defs
-      else if builtins.all isEvent defList then
-        eventType.merge loc defs
-      else if builtins.all isExec defList then
-        execType.merge loc defs
-      else
-        mergeOneOption loc defs
-    ;
-  };
 in
 {
   sane.programs.bonsai = {
@@ -96,7 +10,7 @@ in
       type = types.submodule {
         options = {
           transitions = mkOption {
-            type = types.listOf transitionType;
+            type = options.services.bonsaid.settings.type;
             default = [];
           };
         };
@@ -111,9 +25,8 @@ in
       '';
     });
 
-    fs.".config/bonsai/bonsai_tree.json".symlink.target = pkgs.writers.writeJSON "bonsai_tree.json" cfg.config.transitions;
+    fs.".config/bonsai/bonsai_tree.json".symlink.target = config.services.bonsaid.configFile;
 
-    sandbox.method = "bwrap";
     sandbox.extraRuntimePaths = [
       "bonsai"
     ];
@@ -127,13 +40,35 @@ in
         # TODO: don't create the sway directory here!
         # i do it for now because sway and bonsai call into eachother; circular dependency:
         # - sway -> bonsai -> sane-input-handler -> swaymsg
-        mkdir -p $XDG_RUNTIME_DIR/{bonsai,sway}
-        exec nice -n -11 bonsaid -t $HOME/.config/bonsai/bonsai_tree.json
+        mkdir -p ''${XDG_RUNTIME_DIR}/{bonsai,sway}
+        exec nice -n -11 bonsaid -t ''${HOME}/.config/bonsai/bonsai_tree.json
       '';
-      cleanupCommand = "rm -f $XDG_RUNTIME_DIR/bonsai/bonsai";
+      cleanupCommand = ''rm -f ''${XDG_RUNTIME_DIR}/bonsai/bonsai'';
       readiness.waitExists = [
-        "$XDG_RUNTIME_DIR/bonsai/bonsai"
+        ''''${XDG_RUNTIME_DIR}/bonsai/bonsai''
       ];
     };
   };
+
+  # plug into the nixpkgs bonsaid service.
+  # it's a user service, and since i don't use the service manager it doesn't actually activate:
+  # i just steal the config file generation from it :)
+  services.bonsaid.package = config.sane.programs.bonsai.package;
+  services.bonsaid.settings = lib.mkIf cfg.enabled (lib.mkMerge [
+    cfg.config.transitions
+    [{
+      type = "delay";
+      transitions = [];
+      # speculative: i've observed a hang inside bonsai (rather, hare-ev) where it
+      # attempts to read from a timer, assuming it to have expired, and the read *never* returns.
+      # i think this can happen when an `exec` and a `delay` trigger simultaneously?
+      # particularly, hare-ev does the exec action callback, during which bonsaid enters a node w/o delay and *disables* the timer, and then reading the timer hangs.
+      # if true, then adding a delay to the root node alleviates that (so long as all other nodes also have delays).
+      #
+      # long term, it may be best to move away from bonsai. aside from the above, it's really easy to get it to segfault.
+      delay_duration = 30000 * 1000000;
+    }]
+  ]);
+  # vvv not actually necessary. TODO: delete this line once the service is upstreamed?
+  services.bonsaid.enable = lib.mkIf cfg.enabled true;
 }

hosts/common/programs/brave.nix

@@ -3,12 +3,17 @@
   sane.programs.brave = {
     # convert eval error to build failure
     packageUnwrapped = if (builtins.tryEval pkgs.brave).success then
-      pkgs.brave
+      pkgs.brave.overrideAttrs (upstream: {
+        # brave does crimes with `$0` which break under transparent wrapping
+        preFixup = (upstream.preFixup or "") + ''
+          substituteInPlace $out/opt/brave.com/brave/brave-browser \
+            --replace '$0' "$out/opt/brave.com/brave/brave-browser"
+        '';
+      })
     else
       pkgs.runCommandLocal "brave-not-supported" {} "false"
     ;
-    sandbox.method = "bwrap";
-    sandbox.wrapperType = "inplace";  # /opt/share/brave.com vendor-style packaging
+    sandbox.wrapperType = "inplace";  #< package contains dangling symlinks which my wrapper doesn't understand
     sandbox.net = "all";
     sandbox.extraHomePaths = [
       "dev"  # for developing anything web-related
@@ -17,6 +22,7 @@
     sandbox.extraPaths = [
       "/tmp"  # needed particularly if run from `sane-vpn do`
     ];
+    sandbox.mesaCacheDir = ".cache/BraveSoftware/mesa";
     sandbox.whitelistAudio = true;
     sandbox.whitelistDri = true;
     sandbox.whitelistWayland = true;

hosts/common/programs/brightnessctl.nix

@@ -4,18 +4,17 @@ let
 in
 {
   sane.programs.brightnessctl = {
-    sandbox.method = "landlock";  # also bwrap, but landlock is more responsive
     sandbox.extraPaths = [
       "/sys/class/backlight"
       "/sys/class/leds"
       "/sys/devices"
     ];
-    # sandbox.whitelistDbus = [ "system" ];  #< only necessary if not granting udev perms
+    # sandbox.whitelistDbus.system = true;  #< only necessary if not granting udev perms
   };
 
   services.udev.extraRules = let
-    chmod = "${pkgs.coreutils}/bin/chmod";
-    chown = "${pkgs.coreutils}/bin/chown";
+    chmod = lib.getExe' pkgs.coreutils "chmod";
+    chown = lib.getExe' pkgs.coreutils "chown";
   in lib.mkIf cfg.enabled ''
     # make backlight controllable by members of `video`
     SUBSYSTEM=="backlight", RUN+="${chown} :video $sys$devpath/brightness", RUN+="${chmod} g+w $sys$devpath/brightness"

hosts/common/programs/btrfs-progs.nix

@@ -0,0 +1,25 @@
+{ config, lib, ... }:
+let
+  cfg = config.sane.programs.btrfs-progs;
+in
+{
+  sane.programs.btrfs-progs = {
+    # sandbox.autodetectCliPaths = "existing";  # e.g. `btrfs filesystem df /my/fs`
+    sandbox.autodetectCliPaths = "parent";  # e.g. `btrfs subvolume create ./my_subvol`
+    sandbox.extraPaths = [
+      "/dev/btrfs-control"
+      #vvv required for `sudo btrfs filesystem show` with no args
+      "/dev"
+      "/sys/block"
+      "/sys/dev/block"
+      "/sys/devices"
+    ];
+    sandbox.tryKeepUsers = true;
+    sandbox.capabilities = [ "sys_admin" ];  # for `btrfs scrub`
+  };
+
+  # TODO: service sandboxing
+  services.btrfs.autoScrub.enable = lib.mkIf cfg.enabled true;
+  services.btrfs.autoScrub.interval = "weekly";
+}
+

hosts/common/programs/bunpen.nix

@@ -0,0 +1,20 @@
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.sane.programs.bunpen;
+in
+{
+  sane.programs.bunpen = {
+    packageUnwrapped = pkgs.bunpen.overrideAttrs (base: {
+      # create a directory which holds just the `bunpen` so that we
+      # can add bunpen as a dependency to binaries via `PATH=/run/current-system/libexec/bunpen` without forcing rebuild every time bunpen changes
+      postInstall = (base.postInstall or "") + ''
+        mkdir -p $out/libexec/bunpen
+        ln -s $out/bin/bunpen $out/libexec/bunpen/bunpen
+      '';
+    });
+    sandbox.enable = false;
+    sandbox.method = null;  #< TODO: avoids infinite recursion in the sane.programs system
+  };
+
+  environment.pathsToLink = lib.mkIf cfg.enabled [ "/libexec/bunpen" ];
+}

hosts/common/programs/callaudiod.nix

@@ -13,9 +13,8 @@
   sane.programs.callaudiod = {
     packageUnwrapped = pkgs.rmDbusServices pkgs.callaudiod;
 
-    sandbox.method = "bwrap";
     sandbox.whitelistAudio = true;
-    sandbox.whitelistDbus = [ "user" ];
+    sandbox.whitelistDbus.user = true;  #< TODO: reduce
 
     services.callaudiod = {
       description = "callaudiod: dbus service to switch audio profiles and mute microphone";

hosts/common/programs/calls.nix

@@ -12,6 +12,18 @@
 # user guide:
 # - "Use for Calls" means, "when i click a tel: URI, use this account": <https://gitlab.gnome.org/GNOME/calls/-/issues/513>
 # - `calls -vvv` for verbosity
+# - `SOFIA_DEBUG=9 NEA_DEBUG=9 NUA_DEBUG=9 NTA_DEBUG=9 SU_DEBUG=8 gnome-calls` to debug SIP related stuff
+#
+# LIMITATIONS, COMPATIBILITY
+# - 2024-08-20: when switching from wifi -> wwan (4g), may experience about a minute of audio loss.
+#   the call stays alive, but no sound in either direction.
+#   this appears to be ~40s of general net loss to servo-hn (NetworkManager being slow to switch the default device? wireguard being slow to refresh?),
+#   unknown how much time is lost in the upper layers (e.g. dns being refreshed)
+# - 2024-08-20: wwan -> wifi switching is (near) flawless. prefer to keep modem powered until end of call, because of audio routing, but OK to power it off.
+# - 2024-08-20: audio is not always routed to a good device when the modem is powered.
+#   solve by opening `pavucontrol`, go to "configuration" tab, change "Built-in audio" to anything and then back to "Make a phone call (Earpiece, Mic)".
+#   i expect my eg25-control-powered script messes with the audio routing.
+# - 2024-12-12: contacts are visible when evolution-data-server is enabled, however attempting to call triggers "Can't submit call with no origin"
 { config, lib, pkgs, ... }:
 let
   cfg = config.sane.programs.calls;
@@ -29,56 +41,88 @@ in
     };
 
     packageUnwrapped = pkgs.rmDbusServicesInPlace ((pkgs.calls.override {
-      gtk3 = pkgs.gtk4;
-      libpeas = pkgs.libpeas2;
-      wrapGAppsHook3 = pkgs.wrapGAppsHook4;
-    }).overrideAttrs (upstream: {
-      # XXX(2024-08-08): v46.3 has a bug where if it has no network connection on launch, it forever stays disconnected & never retries
-      version = "47_beta.0-unstable-2024-08-08";
-      src = lib.warnIf (lib.versionOlder "47.0" upstream.version) "gnome-calls outdated; remove src override? (keep UI patches though!)" pkgs.fetchFromGitLab {
-        domain = "gitlab.gnome.org";
-        owner = "GNOME";
-        repo = "calls";
-        fetchSubmodules = true;
-        # rev = "main";
-        rev = "ff213579a52222e7c95e585843d97b5b817b2a8b";
-        hash = "sha256-0QYC8FJpfg/X2lIjBDooba2idUfpJNQhcpv8Z5I/B4k=";
+      evolution-data-server-gtk4 = pkgs.evolution-data-server-gtk4.override {
+        # drop webkitgtk_6_0 dependency.
+        # it's normally cached, but if modifying low-level deps (e.g. pipewire) it's nice to not have to rebuild it,
+        # especially since `calls` is part of `moby-min`.
+        withGtk4 = false;
       };
+      folks = pkgs.folks.override {
+        evolution-data-server-gtk4 = pkgs.evolution-data-server-gtk4.override {
+          # drop webkitgtk_6_0 dependency.
+          withGtk4 = false;
+        };
+      };
+      sofia_sip = pkgs.sofia_sip.overrideAttrs (upstream: {
+        # use linphone's sofia_sip.
+        # Freeswitch sofia_sip has a bug where a failed DNS query will never return to the caller.
+        # see `outgoing_answer_a`: in linphone's this already calls the user's callback; in Freeswitch there's a branch which leaves the caller hanging.
+        version = "1.13.45bc-unstable-2024-08-05";
+        src = pkgs.fetchFromGitLab {
+          domain = "gitlab.linphone.org";
+          owner = "BC/public/external";
+          repo = "sofia-sip";
+          rev = "b924a57e8eeb24e8b9afc5fd0fb9b51d5993fe5d";
+          hash = "sha256-1VbKV+eAJ80IMlubNl7774B7QvLv4hE8SXANDSD9sRU=";
+        };
+        patches = [];
+      });
+    }).overrideAttrs (upstream: {
+      # src = lib.warnIf (lib.versionOlder "47.0" upstream.version) "gnome-calls outdated; remove src override? (keep UI patches though!)" pkgs.fetchFromGitLab {
+      #   domain = "gitlab.gnome.org";
+      #   owner = "GNOME";
+      #   repo = "calls";
+      #   fetchSubmodules = true;
+      #   # rev = "main";
+      #   # rev = "ff213579a52222e7c95e585843d97b5b817b2a8b";
+      #   # hash = "sha256-0QYC8FJpfg/X2lIjBDooba2idUfpJNQhcpv8Z5I/B4k=";
+      #   rev = "75c4072c4e2ba8619c8067703fb65fe622af8b42";
+      #   hash = "sha256-99B1GS2IXt3per8XnbBRCTChlcwT3zWnhwgG1ift0QQ=";
+      # };
 
       patches = (upstream.patches or []) ++ [
         (pkgs.fetchpatch {
-          # usability improvement... if the UI is visible, then i can receive calls. otherwise, i can't!
-          url = "https://git.uninsane.org/colin/gnome-calls/commit/a19166d85927e59662fae189a780eed18bf876ce.patch";
+          # usability improvement... ties the UI visibility to the connection state, so if the UI is gone, then i can't receive calls (and will hopefully notice that more easily!)
+          # TODO: see about a more maintainable solution:
+          # 1. create gobject-introspection bindings, then a python wrapper which binds the MainWindow and CallWindow notify::visible signals?
+          # 2. move this functionality into a gnome calls `plugin`?
+          # 3. upstream this; use the Nautilus approach of controlling behavior here with an env var?
+          # also TODO: write a nix test for this functionality so that it doesn't break during an upgrade!
+          url = "https://git.uninsane.org/colin/gnome-calls/commit/88dbe108a8cf82f9c0766c310218902a8a2a7cd5.patch";
           name = "exit on close (i.e. never daemonize)";
-          hash = "sha256-NoVQV2TlkCcsBt0uwSyK82hBKySUW4pADrJVfLFvWgU=";
+          hash = "sha256-QggVM28X9A2f9SbHMMM38M4zKhjYZrTvsZoitxyczdo=";
+        })
+        (pkgs.fetchpatch {
+          # solves the issue where flakey DNS (especially at boot) could take down call connectivity indefinitely.
+          # see: <https://gitlab.gnome.org/GNOME/calls/-/issues/659>
+          url = "https://git.uninsane.org/colin/gnome-calls/commit/db9192a69cff2b20b5e8870e34a9b1e694a81c7f.patch";
+          name = "sip: attempt reconnection anytime network is routable, not just when routability changes";
+          hash = "sha256-agPM3XKXiP5Rxrl26DNA+pnhEPTBEBQBxZe3CoptgII=";
         })
-      ];
-
-      nativeBuildInputs = upstream.nativeBuildInputs ++ [
-        pkgs.dbus  #< for dbus-run-session (should be test only, but it's not)
-      ];
-
-      buildInputs = upstream.buildInputs ++ [
-        pkgs.libadwaita
       ];
     }));
 
-    sandbox.method = "bwrap";
+    sandbox.mesaCacheDir = ".cache/calls/mesa";
     sandbox.net = "vpn.wg-home";  #< XXX(2024/07/05): my cell carrier seems to block RTP, so tunnel it.
     sandbox.whitelistAudio = true;
-    sandbox.whitelistDbus = [ "user" ];  # necessary for secrets, at the minimum
+    sandbox.whitelistDbus.user.call."org.freedesktop.secrets" = "*";  #< TODO: restrict to a subset of secrets
+    sandbox.whitelistDbus.user.call."org.mobian_project.CallAudio" = "*";
+    sandbox.whitelistDbus.user.call."org.sigxcpu.Feedback" = "*";
+    sandbox.whitelistDbus.user.call."org.gnome.evolution.dataserver.*" = "*";  #< TODO: reduce; only needs address book and maybe sources
+    sandbox.whitelistDbus.user.own = [ "org.gnome.Calls" ];
+    sandbox.whitelistSendNotifications = true;  # for missed calls
     sandbox.whitelistWayland = true;
 
     persist.byStore.private = [
       # ".cache/folks"      # contact avatars?
       # ".config/calls"
       ".local/share/calls"  # call "records"
-      # .local/share/folks  # contacts?
+      # .local/share/folks  # contacts  (e.g. `.local/share/folks/relationships.ini` with gsetting org/freedesktop/folks/primary-store='key-file'
     ];
     # this is only the username/endpoint: the actual password appears to be stored in gnome-keyring
     secrets.".config/calls/sip-account.cfg" = ../../../secrets/common/gnome_calls_sip-account.cfg.bin;
     suggestedPrograms = [
-      "callaudiod"  # runtime dependency (optional, but probably needed for mic muting?)
+      "callaudiod"  # runtime dependency (optional; without this the mute and speaker buttons do not work (ordinarily they function by changing the GLOBAL audio config))
       "feedbackd"  # needs `phone-incoming-call`, in particular
       "gnome-keyring"  # to remember the password
     ];

hosts/common/programs/captree.nix

@@ -1,8 +1,7 @@
 { pkgs, ... }:
 {
   sane.programs.captree = {
-    packageUnwrapped = pkgs.linkBinIntoOwnPackage pkgs.libcap-with-captree "captree";
-    sandbox.method = "bwrap";
-    sandbox.isolatePids = false;
+    packageUnwrapped = pkgs.linkBinIntoOwnPackage pkgs.libcap "captree";
+    sandbox.keepPidsAndProc = true;
   };
 }

hosts/common/programs/celeste64.nix

@@ -3,7 +3,6 @@
   sane.programs.celeste64 = {
     buildCost = 1;
 
-    sandbox.method = "bwrap";
     sandbox.whitelistAudio = true;
     sandbox.whitelistDri = true;
     sandbox.whitelistWayland = true;
@@ -15,5 +14,6 @@
       # save data, controls map
       ".local/share/Celeste64"
     ];
+    sandbox.mesaCacheDir = ".cache/Celeste64/mesa";
   };
 }

hosts/common/programs/confy.nix

@@ -0,0 +1,19 @@
+{ ... }:
+{
+  sane.programs.confy = {
+    sandbox.net = "all";
+    sandbox.whitelistDri = true;
+    sandbox.whitelistWayland = true;
+    sandbox.mesaCacheDir = ".cache/net.kirgroup.confy/mesa";
+    sandbox.whitelistDbus.user.own = [ "net.kirgroup.confy" ];
+    sandbox.whitelistPortal = [
+      "NetworkMonitor"
+      "OpenURI"
+    ];
+
+    persist.byStore.private = [
+      ".cache/net.kirgroup.confy"
+      # ".local/share/net.kirgroup.confy"  #< empty
+    ];
+  };
+}

hosts/common/programs/conky/default.nix

@@ -1,7 +1,6 @@
-{ pkgs, ... }:
+{ ... }:
 {
   sane.programs.conky = {
-    sandbox.method = "bwrap";
     sandbox.net = "clearnet";  #< for the scripts it calls (weather)
     sandbox.extraPaths = [
       "/sys/class/power_supply"

hosts/common/programs/cozy.nix

@@ -15,11 +15,11 @@
 
     buildCost = 1;
 
-    sandbox.method = "bwrap";  # landlock gives: _multiprocessing.SemLock: Permission Denied
     sandbox.whitelistAudio = true;
-    sandbox.whitelistDbus = [ "user" ];  # mpris
+    sandbox.whitelistDbus.user = true;  #< TODO: reduce  # mpris
     sandbox.whitelistWayland = true;
     sandbox.extraHomePaths = [
+      "Books/Audiobooks"
       "Books/local"
       "Books/servo"
     ];

hosts/common/programs/cups.nix

@@ -11,11 +11,12 @@ let
 in
 {
   sane.programs.cups = {
+    sandbox.method = null;  #< TODO: sandbox
     suggestedPrograms = [
       "system-config-printer"
     ];
   };
-  sane.programs.system-config-printer = {};
+  sane.programs.system-config-printer.sandbox.method = null;  #< TODO: sandbox
 
   services.printing = lib.mkIf cfg.enabled {
     enable = true;