make-sandboxed: preserve meta.mainProgram

how did this work earlier? does lappy have binfmt enabled??

.gitignore

@@ -1,2 +1,4 @@
+.working
 result
+result-*
 /secrets/local.nix

.sops.yaml

@@ -8,7 +8,7 @@ keys:
   - &host_servo age1tzlyex2z6t88tg9h82943e39shxhmqeyr7ywhlwpdjmyqsndv3qq27x0rf
   - &host_moby age18vq5ktwgeaysucvw9t67drqmg5zd5c5k3le34yqxckkfj7wqdqgsd4ejmt
 creation_rules:
-  - path_regex: secrets/universal*
+  - path_regex: secrets/common*
     key_groups:
     - age:
       - *user_desko_colin
@@ -26,19 +26,19 @@ creation_rules:
       - *user_lappy_colin
       - *user_servo_colin
       - *host_servo
-  - path_regex: secrets/desko.yaml$
+  - path_regex: secrets/desko*
     key_groups:
     - age:
       - *user_desko_colin
       - *user_lappy_colin
       - *host_desko
-  - path_regex: secrets/lappy.yaml$
+  - path_regex: secrets/lappy*
     key_groups:
     - age:
       - *user_lappy_colin
       - *user_desko_colin
       - *host_lappy
-  - path_regex: secrets/moby.yaml$
+  - path_regex: secrets/moby*
     key_groups:
     - age:
       - *user_desko_colin

README.md

@@ -0,0 +1,133 @@
+![hello](doc/hello.gif)
+
+# .❄️≡We|_c0m3 7o m`/ f14k≡❄️.
+
+## What's Here
+
+this is the top-level repo from which i configure/deploy all my NixOS machines:
+- desktop
+- laptop
+- server
+- mobile phone (Pinephone)
+
+everything outside of [hosts/](./hosts/) and [secrets/](./secrets/) is intended for export, to be importable for use by 3rd parties.
+the only hard dependency for my exported pkgs/modules should be [nixpkgs][nixpkgs].
+building [hosts/](./hosts/) will require [sops][sops].
+
+you might specifically be interested in these files (elaborated further in #key-points-of-interest):
+- ~~[`sxmo-utils`](./pkgs/additional/sxmo-utils/default.nix)~~
+  - these files will remain until my config settles down, but i no longer use or maintain SXMO.
+- [my implementation of impermanence](./modules/persist/default.nix)
+- my way of deploying dotfiles/configuring programs per-user:
+  - [modules/fs/](./modules/fs/default.nix)
+  - [modules/programs/](./modules/programs/default.nix)
+  - [modules/users/](./modules/users/default.nix)
+
+[nixpkgs]: https://github.com/NixOS/nixpkgs
+[sops]: https://github.com/Mic92/sops-nix
+[uninsane-org]: https://uninsane.org
+
+## Using This Repo In Your Own Config
+
+this should be a pretty "standard" flake. just reference it, and import either
+- `nixosModules.sane` (for the modules)
+- `overlays.pkgs` (for the packages)
+
+or follow the instructions [here][NUR] to use it via the Nix User Repositories.
+
+[NUR]: https://nur.nix-community.org/
+
+## Layout
+- `doc/`
+  - instructions for tasks i find myself doing semi-occasionally in this repo.
+- `hosts/`
+  - the bulk of config which isn't factored with external use in mind.
+  - that is, if you were to add this repo to a flake.nix for your own use,
+    you won't likely be depending on anything in this directory.
+- `integrations/`
+  - code intended for consumption by external tools (e.g. the Nix User Repos)
+- `modules/`
+  - config which is gated behind `enable` flags, in similar style to nixpkgs'
+    `nixos/` directory.
+  - if you depend on this repo, it's most likely for something in this directory.
+- `nixpatches/`
+  - literally, diffs i apply atop upstream nixpkgs before performing further eval.
+- `overlays/`
+  - exposed via the `overlays` output in `flake.nix`.
+  - predominantly a list of `callPackage` directives.
+- `pkgs/`
+  - derivations for things not yet packaged in nixpkgs.
+  - derivations for things from nixpkgs which i need to `override` for some reason.
+  - inline code for wholly custom packages (e.g. `pkgs/additional/sane-scripts/` for CLI tools
+    that are highly specific to my setup).
+- `scripts/`
+  - scripts which aren't reachable on a deployed system, but may aid manual deployments
+- `secrets/`
+  - encrypted keys, API tokens, anything which one or more of my machines needs
+    read access to but shouldn't be world-readable.
+  - not much to see here
+- `templates/`
+  - exposed via the `templates` output in `flake.nix`.
+  - used to instantiate short-lived environments.
+  - used to auto-fill the boiler-plate portions of new packages.
+
+
+## Key Points of Interest
+
+i.e. you might find value in using these in your own config:
+
+- `modules/fs/`
+  - use this to statically define leafs and nodes anywhere in the filesystem,
+    not just inside `/nix/store`.
+  - e.g. specify that `/var/www` should be:
+    - owned by a specific user/group
+    - set to a specific mode
+    - symlinked to some other path
+    - populated with some statically-defined data
+    - populated according to some script
+    - created as a dependency of some service (e.g. `nginx`)
+  - values defined here are applied neither at evaluation time _nor_ at activation time.
+    - rather, they become systemd services.
+    - systemd manages dependencies
+    - e.g. link `/var/www -> /mnt/my-drive/www` only _after_ `/mnt/my-drive/www` appears)
+  - this is akin to using [Home Manager's][home-manager] file API -- the part which lets you
+    statically define `~/.config` files -- just with a different philosophy.
+- `modules/persist/`
+  - my alternative to the Impermanence module.
+  - this builds atop `modules/fs/` to achieve things stock impermanence can't:
+    - persist things to encrypted storage which is unlocked at login time (pam_mount).
+    - "persist" cache directories -- to free up RAM -- but auto-wipe them on mount
+      and encrypt them to ephemeral keys so they're unreadable post shutdown/unmount.
+- `modules/programs/`
+  - like nixpkgs' `programs` options, but allows both system-wide or per-user deployment.
+  - allows `fs` and `persist` config values to be gated behind program deployment:
+    - e.g. `/home/<user>/.mozilla/firefox` is persisted only for users who
+      `sane.programs.firefox.enableFor.user."<user>" = true;`
+  - allows aggressive sandboxing any program:
+    - `sane.programs.firefox.sandbox.method = "bwrap";  # sandbox with bubblewrap`
+    - `sane.programs.firefox.sandbox.whitelistWayland = true;  # allow it to render a wayland window`
+    - `sane.programs.firefox.sandbox.extraHomePaths = [ "Downloads" ];  # allow it read/write access to ~/Downloads`
+    - integrated with `fs` and `persist` modules so that programs' config files and persisted data stores are linked into the sandbox w/o any extra involvement.
+- `modules/users/`
+  - convenience layer atop the above modules so that you can just write
+    `fs.".config/git"` instead of `fs."/home/colin/.config/git"`
+  - per-user services managed by [s6-rc](https://www.skarnet.org/software/s6-rc/)
+
+some things in here could easily find broader use. if you would find benefit in
+them being factored out of my config, message me and we could work to make that happen.
+
+[home-manager]: https://github.com/nix-community/home-manager
+
+## Mirrors
+
+this repo exists in a few known locations:
+- primary: <https://git.uninsane.org/colin/nix-files>
+- mirror: <https://github.com/nix-community/nur-combined/tree/master/repos/colinsane>
+
+## Contact
+
+if you want to contact me for questions, or collaborate to split something useful into a shared repo, etc,
+you can reach me via any method listed [here](https://uninsane.org/about).
+patches, for this repo or any other i host, will be warmly welcomed in any manner you see fit:
+`git send-email`, DM'ing the patch over Matrix/Lemmy/ActivityPub/etc, even a literal PR where you
+link me to your own clone.

TODO.md

@@ -0,0 +1,171 @@
+## BUGS
+- `rmDbusServices` may break sandboxing
+  - e.g. if the package ships a systemd unit which references $out, then make-sandboxed won't properly update that unit.
+  - `rmDbusServicesInPlace` is not affected
+- moby: touchscreen input is still enabled when screen is off
+- when moby wlan is explicitly set down (via ip link set wlan0 down), /var/lib/trust-dns/dhcp-configs doesn't get reset
+  - `ip monitor` can detect those manual link state changes (NM-dispatcher it seems cannot)
+  - or try dnsmasq?
+- trust-dns: can't recursively resolve api.mangadex.org
+  - and *sometimes* apple.com fails
+- sandbox: link cache means that if i update ~/.config/... files inline, sandboxed programs still see the old version
+- mpv: audiocast has mpv sending its output to the builtin speakers unless manually changed
+- mpv: no way to exit fullscreen video on moby
+  - uosc hides controls on FS, and touch doesn't support unhiding
+- Signal restart loop drains battery
+  - decrease s6 restart time?
+- `ssh` access doesn't grant same linux capabilities as login
+- ringer (i.e. dino incoming call) doesn't prevent moby from sleeping
+- sysvol (volume overlay): when casting with `blast`, sysvol doesn't react to volume changes
+- moby: kaslr is effectively disabled
+  - `dmesg | grep "KASLR disabled due to lack of seed"`
+  - fix by adding `kaslrseed` to uboot script before `booti`
+    - <https://github.com/armbian/build/pull/4352>
+    - not sure how that's supposed to work with tow-boot; maybe i should just update tow-boot
+- moby: bpf is effectively disabled?
+  - `dmesg | grep 'systemd[1]: bpf-lsm: Failed to load BPF object: No such process'`
+  - `dmesg | grep 'hid_bpf: error while preloading HID BPF dispatcher: -22'`
+
+## REFACTORING:
+- add import checks to my Python nix-shell scripts
+- consolidate ~/dev and ~/ref
+  - ~/dev becomes a link to ~/ref/cat/mine
+- fold hosts/common/home/ssh.nix -> hosts/common/users/colin.nix
+
+### sops/secrets
+- rework secrets to leverage `sane.fs`
+- remove sops activation script as it's covered by my systemd sane.fs impl
+- user secrets could just use `gocryptfs`, like with ~/private?
+  - can gocryptfs support nested filesystems, each with different perms (for desko, moby, etc)?
+
+### roles
+- allow any host to take the role of `uninsane.org`
+  - will make it easier to test new services?
+
+### upstreaming
+- add updateScripts to all my packages in nixpkgs
+
+#### upstreaming to non-nixpkgs repos
+- gtk: build schemas even on cross compilation: <https://github.com/NixOS/nixpkgs/pull/247844>
+
+
+## IMPROVEMENTS:
+- systemd/journalctl: use a less shit pager
+  - there's an env var for it: SYSTEMD_PAGER? and a flag for journalctl
+- kernels: ship the same kernel on every machine
+  - then i can tune the kernels for hardening, without duplicating that work 4 times
+- zfs: replace this with something which doesn't require a custom kernel build
+- mpv: add media looping controls (e.g. loop song, loop playlist)
+
+### security/resilience
+- validate duplicity backups!
+- encrypt more ~ dirs (~/archives, ~/records, ..?)
+  - best to do this after i know for sure i have good backups
+- /mnt/desko/home, etc, shouldn't include secrets (~/private)
+  - 95% of its use is for remote media access and stuff which isn't in VCS (~/records)
+- port all sane.programs to be sandboxed
+  - enforce that all `environment.packages` has a sandbox profile (or explicitly opts out)
+  - revisit "non-sandboxable" apps and check that i'm not actually just missing mountpoints
+    - LL_FS_RW=/ isn't enough -- need all mount points like `=/:/proc:/sys:...`.
+  - ensure non-bin package outputs are linked for sandboxed apps
+    - i.e. `outputs.man`, `outputs.debug`, `outputs.doc`, ...
+  - lock down dbus calls within the sandbox
+    - otherwise anyone can `systemd-run --user ...` to potentially escape a sandbox
+    - <https://github.com/flatpak/xdg-dbus-proxy>
+  - remove `.ssh` access from Firefox!
+    - limit access to `~/knowledge/secrets` through an agent that requires GUI approval, so a firefox exploit can't steal all my logins
+  - port sanebox to a compiled language (hare?)
+    - it adds like 50-70ms launch time _on my laptop_. i'd hate to know how much that is on the pinephone.
+- make dconf stuff less monolithic
+  - i.e. per-app dconf profiles for those which need it. possible static config.
+- canaries for important services
+  - e.g. daily email checks; daily backup checks
+  - integrate `nix check` into Gitea actions?
+
+#### sudo-free world
+- `systemctl restart FOO`: needs `sudo`
+
+### user experience
+- rofi: sort items case-insensitively
+- xdg-desktop-portal shouldn't kill children on exit
+  - *maybe* a job for `setsid -f`?
+- replace starship prompt with something more efficient
+  - watch `forkstat`: it does way too much
+- cleanup waybar so that it's not invoking playerctl every 2 seconds
+- install apps:
+  - display QR codes for WiFi endpoints: <https://linuxphoneapps.org/apps/noappid.wisperwind.wifi2qr/>
+  - shopping list (not in nixpkgs): <https://linuxphoneapps.org/apps/ro.hume.cosmin.shoppinglist/>
+  - offline Wikipedia (or, add to `wike`)
+  - offline docs viewer (gtk): <https://github.com/workbenchdev/Biblioteca>
+  - some type of games manager/launcher
+    - Gnome Highscore (retro games)?: <https://gitlab.gnome.org/World/highscore>
+  - better maps for mobile (Osmin (QtQuick)? Pure Maps (Qt/Kirigami)?
+  - note-taking app: <https://linuxphoneapps.org/categories/note-taking/>
+  - OSK overlay specifically for mobile gaming
+    - i.e. mock joysticks, for use with SuperTux and SuperTuxKart
+- install mobile-friendly games:
+  - Shattered Pixel Dungeon (nixpkgs `shattered-pixel-dungeon`; doesn't cross-compile b/c openjdk/libIDL) <https://github.com/ebolalex/shattered-pixel-dungeon>
+  - UnCiv (Civ V clone; nixpkgs `unciv`; doesn't cross-compile):  <https://github.com/yairm210/UnCiv>
+  - Simon Tatham's Puzzle Collection (not in nixpkgs) <https://git.tartarus.org/?p=simon/puzzles.git>
+  - Shootin Stars  (Godot; not in nixpkgs) <https://gitlab.com/greenbeast/shootin-stars>
+  - numberlink (generic name for Flow Free). not packaged in Nix
+  - Neverball (https://neverball.org/screenshots.php). nix: as `neverball`
+  - blurble (https://linuxphoneapps.org/games/app.drey.blurble/). nix: not as of 2024-02-05
+  - Trivia Quiz (https://linuxphoneapps.org/games/io.github.nokse22.trivia-quiz/)
+- sane-sync-music: remove empty dirs
+
+#### moby
+- fix cpuidle (gets better power consumption): <https://xnux.eu/log/077.html>
+- moby: tune keyboard layout
+- SwayNC:
+  - don't show MPRIS if no players detected
+    - this is a problem of playerctld, i guess
+  - add option to change audio output
+  - fix colors (red alert) to match overall theme
+- moby: tune GPS
+  - run only geoclue, and not gpsd, to save power?
+  - tune QGPS setting in eg25-control, for less jitter?
+  - direct mepo to prefer gpsd, with fallback to geoclue, for better accuracy?
+  - configure geoclue to do some smoothing?
+  - manually do smoothing, as some layer between mepo and geoclue/gpsd?
+- moby: port `freshen-agps` timer service to s6 (maybe i want some `s6-cron` or something)
+- moby: show battery state on ssh login
+- moby: improve gPodder launch time
+- moby: theme GTK apps (i.e. non-adwaita styles)
+  - especially, make the menubar collapsible
+  - try Gradience tool specifically for theming adwaita? <https://linuxphoneapps.org/apps/com.github.gradienceteam.gradience/>
+
+#### non-moby
+- RSS: integrate a paywall bypass
+  - e.g. self-hosted [ladder](https://github.com/everywall/ladder) (like 12ft.io)
+- neovim: set up language server (lsp; rnix-lsp; nvim-lspconfig)
+- neovim: integrate LLMs
+- Helix: make copy-to-system clipboard be the default
+- firefox/librewolf: persist history
+  - just not cookies or tabs
+- package Nix/NixOS docs for Zeal
+  - install [doc-browser](https://github.com/qwfy/doc-browser)
+  - this supports both dash (zeal) *and* the datasets from <https://devdocs.io> (which includes nix!)
+  - install [devhelp](https://wiki.gnome.org/Apps/Devhelp)  (gnome)
+- have xdg-open parse `<repo:...> URIs (or adjust them so that it _can_ parse)
+- sane-bt-search: show details like 5.1 vs stereo, h264 vs h265
+  - maybe just color these "keywords" in all search results?
+- uninsane.org: make URLs relative to allow local use (and as offline homepage)
+- email: fix so that local mail doesn't go to junk
+  - git sendmail flow adds the DKIM signatures, but gets delivered locally w/o having the sig checked, so goes into Junk
+  - could change junk filter from "no DKIM success" to explicit "DKIM failed"
+
+### perf
+- debug nixos-rebuild times
+  - use `systemctl list-jobs` to show what's being waited on
+  - i think it's `systemd-networkd-wait-online.service` that's blocking this?
+    - i wonder what interface it's waiting for. i should use `--ignore=...` to ignore interfaces i don't care about.
+  - also `wireguard-wg-home.target` when net is offline
+- add `pkgs.impure-cached.<foo>` package set to build things with ccache enabled
+  - every package here can be auto-generated, and marked with some env var so that it doesn't pollute the pure package set
+  - would be super handy for package prototyping!
+
+## NEW FEATURES:
+- migrate MAME cabinet to nix
+  - boot it from PXE from servo?
+- enable IPv6

default.nix

@@ -0,0 +1,9 @@
+# limited, non-flake interface to this repo.
+# this file exposes the same view into `pkgs` which the flake would see when evaluated.
+#
+# the primary purpose of this file is so i can run `updateScript`s which expect
+# the root to be `default.nix`
+{ pkgs ? import <nixpkgs> {} }:
+pkgs.appendOverlays [
+  (import ./overlays/all.nix)
+]

doc/adding-a-program.md

@@ -0,0 +1,13 @@
+to ship `pkgs.foo` on some host, either:
+- add it as an entry in `suggestedPrograms` to the appropriate category in `hosts/common/programs/assorted.nix`, or
+- `sane.programs.foo.enableFor.user.colin = true` in `hosts/by-name/myhost/default.nix`
+
+if the program needs customization (persistence, configs, secrets):
+- add a file for it at `hosts/common/programs/<foo>.nix`
+- set the options, `sane.programs.foo.{fs,persist}`
+
+if it's unclear what fs paths a program uses:
+- run one of these commands, launch the program, run it again, and `diff`:
+  - `du -x --apparent-size ~`
+  - `find ~ -xdev`
+- or, inspect the whole tmpfs root with `ncdu -x /`

flake.lock

@@ -1,12 +1,52 @@
 {
   "nodes": {
-    "flake-utils": {
+    "flake-compat": {
       "locked": {
-        "lastModified": 1659877975,
-        "narHash": "sha256-zllb8aq3YO3h8B/U0/J1WBgAL8EX5yWf5pMj3G0NAmc=",
+        "lastModified": 1688025799,
+        "narHash": "sha256-ktpB4dRtnksm9F5WawoIkEneh1nrEvuxb5lJFt1iOyw=",
+        "owner": "nix-community",
+        "repo": "flake-compat",
+        "rev": "8bf105319d44f6b9f0d764efa4fdef9f1cc9ba1c",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "flake-compat",
+        "type": "github"
+      }
+    },
+    "flake-parts": {
+      "inputs": {
+        "nixpkgs-lib": [
+          "nixpkgs-wayland",
+          "nix-eval-jobs",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1712014858,
+        "narHash": "sha256-sB4SWl2lX95bExY2gMFG5HIzvva5AVMJd4Igm+GpZNw=",
+        "owner": "hercules-ci",
+        "repo": "flake-parts",
+        "rev": "9126214d0a59633752a136528f5f3b9aa8565b7d",
+        "type": "github"
+      },
+      "original": {
+        "owner": "hercules-ci",
+        "repo": "flake-parts",
+        "type": "github"
+      }
+    },
+    "flake-utils": {
+      "inputs": {
+        "systems": "systems"
+      },
+      "locked": {
+        "lastModified": 1710146030,
+        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
         "owner": "numtide",
         "repo": "flake-utils",
-        "rev": "c0e246b9b83f637f4681389ecabcb2681b4f3af0",
+        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
         "type": "github"
       },
       "original": {
@@ -15,76 +55,193 @@
         "type": "github"
       }
     },
+    "lib-aggregate": {
+      "inputs": {
+        "flake-utils": "flake-utils",
+        "nixpkgs-lib": "nixpkgs-lib"
+      },
+      "locked": {
+        "lastModified": 1716725378,
+        "narHash": "sha256-bNTVDAVBLFSSTU+q54cJnntmFKBi+F/D8sSqlZwBGiM=",
+        "owner": "nix-community",
+        "repo": "lib-aggregate",
+        "rev": "dbc9130fe1455e0f6ee4d8f5f799f9be551f866b",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "lib-aggregate",
+        "type": "github"
+      }
+    },
     "mobile-nixos": {
       "flake": false,
       "locked": {
-        "lastModified": 1674880620,
-        "narHash": "sha256-JMALuC7xcoH/T66sKTVLuItHfOJBCWsNKpE49Qrvs80=",
+        "lastModified": 1694749521,
+        "narHash": "sha256-MiVokKlpcJmfoGuWAMeW1En7gZ5hk0rCQArYm6P9XCc=",
         "owner": "nixos",
         "repo": "mobile-nixos",
-        "rev": "7478a9ffad737486951186b66f6c5535dc5802e2",
+        "rev": "d25d3b87e7f300d8066e31d792337d9cd7ecd23b",
         "type": "github"
       },
       "original": {
         "owner": "nixos",
+        "ref": "d25d3b87e7f300d8066e31d792337d9cd7ecd23b",
         "repo": "mobile-nixos",
         "type": "github"
       }
     },
+    "nix-eval-jobs": {
+      "inputs": {
+        "flake-parts": "flake-parts",
+        "nix-github-actions": "nix-github-actions",
+        "nixpkgs": "nixpkgs",
+        "treefmt-nix": "treefmt-nix"
+      },
+      "locked": {
+        "lastModified": 1715804156,
+        "narHash": "sha256-GtIHP86Cz1kD9xZO/cKbNQACHKdoT9WFbLJAq6W2EDY=",
+        "owner": "nix-community",
+        "repo": "nix-eval-jobs",
+        "rev": "bb95091f6c6f38f6cfc215a1797a2dd466312c8b",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "nix-eval-jobs",
+        "type": "github"
+      }
+    },
+    "nix-github-actions": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-wayland",
+          "nix-eval-jobs",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1703863825,
+        "narHash": "sha256-rXwqjtwiGKJheXB43ybM8NwWB8rO2dSRrEqes0S7F5Y=",
+        "owner": "nix-community",
+        "repo": "nix-github-actions",
+        "rev": "5163432afc817cf8bd1f031418d1869e4c9d5547",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "nix-github-actions",
+        "type": "github"
+      }
+    },
     "nixpkgs": {
-      "inputs": {
-        "nixpkgs": [
-          "nixpkgs-unpatched"
-        ]
-      },
       "locked": {
-        "lastModified": 1,
-        "narHash": "sha256-arp7Uy7ct5ryTcmSY032eN7hr33i7D2XvjTRLliCFDc=",
-        "path": "/nix/store/jblp2g67p3wid2qarcyd8bzrbs9wg5lb-source/nixpatches",
-        "type": "path"
-      },
-      "original": {
-        "path": "/nix/store/jblp2g67p3wid2qarcyd8bzrbs9wg5lb-source/nixpatches",
-        "type": "path"
-      }
-    },
-    "nixpkgs-stable": {
-      "locked": {
-        "lastModified": 1674352297,
-        "narHash": "sha256-OkAnJPrauEcUCrst4/3DKoQfUn2gXKuU6CFvhtMrLgg=",
+        "lastModified": 1715037484,
+        "narHash": "sha256-OUt8xQFmBU96Hmm4T9tOWTu4oCswCzoVl+pxSq/kiFc=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "918b760070bb8f48cb511300fcd7e02e13058a2e",
+        "rev": "ad7efee13e0d216bf29992311536fce1d3eefbef",
         "type": "github"
       },
       "original": {
         "owner": "NixOS",
-        "ref": "release-22.11",
+        "ref": "nixpkgs-unstable",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-lib": {
+      "locked": {
+        "lastModified": 1716684580,
+        "narHash": "sha256-sIbMJWJr4hl2PWd9/iWlh89QfVzBn1NJ3u5RjeZADuM=",
+        "owner": "nix-community",
+        "repo": "nixpkgs.lib",
+        "rev": "d0d27192931680482081aa1c38389da2af84a651",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "nixpkgs.lib",
+        "type": "github"
+      }
+    },
+    "nixpkgs-next-unpatched": {
+      "locked": {
+        "lastModified": 1717243271,
+        "narHash": "sha256-M3VxP6DtREz5Lq6MKg1gQ2EeVdIbq6AEp/N1tDSrvoc=",
+        "owner": "nixos",
+        "repo": "nixpkgs",
+        "rev": "f7de25c01e4c073c06e0525226a0c2311d530cee",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nixos",
+        "ref": "staging-next",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-stable": {
+      "locked": {
+        "lastModified": 1716655032,
+        "narHash": "sha256-kQ25DAiCGigsNR/Quxm3v+JGXAEXZ8I7RAF4U94bGzE=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "59a450646ec8ee0397f5fa54a08573e8240eb91f",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "release-23.11",
         "repo": "nixpkgs",
         "type": "github"
       }
     },
     "nixpkgs-unpatched": {
       "locked": {
-        "lastModified": 1674641431,
-        "narHash": "sha256-qfo19qVZBP4qn5M5gXc/h1MDgAtPA5VxJm9s8RUAkVk=",
+        "lastModified": 1717242134,
+        "narHash": "sha256-2X835ZESUaQ/KZEuG9HkoEB7h0USG5uvkSUmLzFkxAE=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "9b97ad7b4330aacda9b2343396eb3df8a853b4fc",
+        "rev": "61c1d282153dbfcb5fe413c228d172d0fe7c2a7e",
         "type": "github"
       },
       "original": {
         "owner": "nixos",
-        "ref": "nixos-unstable",
+        "ref": "master",
         "repo": "nixpkgs",
         "type": "github"
       }
     },
+    "nixpkgs-wayland": {
+      "inputs": {
+        "flake-compat": "flake-compat",
+        "lib-aggregate": "lib-aggregate",
+        "nix-eval-jobs": "nix-eval-jobs",
+        "nixpkgs": [
+          "nixpkgs-unpatched"
+        ]
+      },
+      "locked": {
+        "lastModified": 1717175759,
+        "narHash": "sha256-KiM5ue/UNQt8ktoqCV4yFqhHxM31U94Mf/piKW9dZ4c=",
+        "owner": "nix-community",
+        "repo": "nixpkgs-wayland",
+        "rev": "93b225ddba91179248b378913a91defbc6aeb899",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "repo": "nixpkgs-wayland",
+        "type": "github"
+      }
+    },
     "root": {
       "inputs": {
         "mobile-nixos": "mobile-nixos",
-        "nixpkgs": "nixpkgs",
+        "nixpkgs-next-unpatched": "nixpkgs-next-unpatched",
         "nixpkgs-unpatched": "nixpkgs-unpatched",
+        "nixpkgs-wayland": "nixpkgs-wayland",
         "sops-nix": "sops-nix",
         "uninsane-dot-org": "uninsane-dot-org"
       }
@@ -92,16 +249,16 @@
     "sops-nix": {
       "inputs": {
         "nixpkgs": [
-          "nixpkgs"
+          "nixpkgs-unpatched"
         ],
         "nixpkgs-stable": "nixpkgs-stable"
       },
       "locked": {
-        "lastModified": 1674546403,
-        "narHash": "sha256-vkyNv0xzXuEnu9v52TUtRugNmQWIti8c2RhYnbLG71w=",
+        "lastModified": 1716692524,
+        "narHash": "sha256-sALodaA7Zkp/JD6ehgwc0UCBrSBfB4cX66uFGTsqeFU=",
         "owner": "Mic92",
         "repo": "sops-nix",
-        "rev": "b6ab3c61e2ca5e07d1f4eb1b67304e2670ea230c",
+        "rev": "962797a8d7f15ed7033031731d0bb77244839960",
         "type": "github"
       },
       "original": {
@@ -110,19 +267,55 @@
         "type": "github"
       }
     },
-    "uninsane-dot-org": {
+    "systems": {
+      "locked": {
+        "lastModified": 1681028828,
+        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "owner": "nix-systems",
+        "repo": "default",
+        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-systems",
+        "repo": "default",
+        "type": "github"
+      }
+    },
+    "treefmt-nix": {
       "inputs": {
-        "flake-utils": "flake-utils",
         "nixpkgs": [
+          "nixpkgs-wayland",
+          "nix-eval-jobs",
           "nixpkgs"
         ]
       },
       "locked": {
-        "lastModified": 1675131883,
-        "narHash": "sha256-yBgJDG72YqIr1bltasqHD1E/kHc9uRFgDjxDmy6kI8M=",
+        "lastModified": 1711963903,
+        "narHash": "sha256-N3QDhoaX+paWXHbEXZapqd1r95mdshxToGowtjtYkGI=",
+        "owner": "numtide",
+        "repo": "treefmt-nix",
+        "rev": "49dc4a92b02b8e68798abd99184f228243b6e3ac",
+        "type": "github"
+      },
+      "original": {
+        "owner": "numtide",
+        "repo": "treefmt-nix",
+        "type": "github"
+      }
+    },
+    "uninsane-dot-org": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-unpatched"
+        ]
+      },
+      "locked": {
+        "lastModified": 1715894399,
+        "narHash": "sha256-h1EdA/h74zgNPNEYbH+0mgOMlJgLVcxuZ8/ewsZlgEc=",
         "ref": "refs/heads/master",
-        "rev": "b099c24091cc192abf3997b94342d4b31cc5757b",
-        "revCount": 170,
+        "rev": "e6f88f563bdd1700c04018951de4f69862646dd1",
+        "revCount": 240,
         "type": "git",
         "url": "https://git.uninsane.org/colin/uninsane"
       },

flake.nix

@@ -4,6 +4,8 @@
 #   - this is marginally the case with schemes like `github:nixos/nixpkgs`.
 #   - given the *existing* `git+https://` scheme, i propose expressing github URLs similarly:
 #     - `github+https://github.com/nixos/nixpkgs/tree/nixos-22.11`
+#     - this would allow for the same optimizations as today's `github:nixos/nixpkgs`, but without obscuring the source.
+#       a code reader could view the source being referenced simply by clicking the https:// portion of that URI.
 # - need some way to apply local patches to inputs.
 #
 #
@@ -12,89 +14,150 @@
 # - Flake RFC: <https://github.com/tweag/rfcs/blob/flakes/rfcs/0049-flakes.md>
 #   - Discussion: <https://github.com/NixOS/rfcs/pull/49>
 # - <https://serokell.io/blog/practical-nix-flakes>
+#
+#
+# COMMON OPERATIONS:
+# - update a specific flake input:
+#   - `nix flake lock --update-input nixpkgs`
 
 {
   # XXX: use the `github:` scheme instead of the more readable git+https: because it's *way* more efficient
   # preferably, i would rewrite the human-readable https URLs to nix-specific github: URLs with a helper,
   # but `inputs` is required to be a strict attrset: not an expression.
   inputs = {
-    # <https://github.com/nixos/nixpkgs/tree/nixos-22.11>
-    # nixpkgs-stable.url = "github:nixos/nixpkgs?ref=nixos-22.11";
-
+    # branch workflow:
+    # - daily:
+    #   - nixos-unstable cut from master after enough packages have been built in caches.
+    # - every 6 hours:
+    #   - master auto-merged into staging and staging-next
+    #   - staging-next auto-merged into staging.
+    # - manually, approximately once per month:
+    #   - staging-next is cut from staging.
+    #   - staging-next merged into master.
+    #
+    # which branch to source from?
+    # - nixos-unstable: for everyday development; it provides good caching
+    # - master: temporarily if i'm otherwise cherry-picking lots of already-applied patches
+    # - staging-next: if testing stuff that's been PR'd into staging, i.e. base library updates.
+    # - staging: maybe if no staging-next -> master PR has been cut yet?
+    #
     # <https://github.com/nixos/nixpkgs/tree/nixos-unstable>
-    nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-unstable";
-    nixpkgs = {
-      url = "./nixpatches";
+    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-unstable";
+    nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=master";
+    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-staging";
+    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-staging-next";
+    nixpkgs-next-unpatched.url = "github:nixos/nixpkgs?ref=staging-next";
+
+    nixpkgs-wayland = {
+      url = "github:nix-community/nixpkgs-wayland";
       inputs.nixpkgs.follows = "nixpkgs-unpatched";
     };
+
     mobile-nixos = {
       # <https://github.com/nixos/mobile-nixos>
-      url = "github:nixos/mobile-nixos";
+      # only used for building disk images, not relevant after deployment
+      # TODO: replace with something else. commit `0f3ac0bef1aea70254a3bae35e3cc2561623f4c1`
+      # replaces the imageBuilder with a "new implementation from celun" and wildly breaks my use.
+      # pinning to d25d3b... is equivalent to holding at 2023-09-15
+      url = "github:nixos/mobile-nixos?ref=d25d3b87e7f300d8066e31d792337d9cd7ecd23b";
       flake = false;
     };
     sops-nix = {
       # <https://github.com/Mic92/sops-nix>
+      # used to distribute secrets to my hosts
       url = "github:Mic92/sops-nix";
-      inputs.nixpkgs.follows = "nixpkgs";
+      # inputs.nixpkgs.follows = "nixpkgs";
+      inputs.nixpkgs.follows = "nixpkgs-unpatched";
     };
     uninsane-dot-org = {
+      # provides the package to deploy <https://uninsane.org>, used only when building the servo host
       url = "git+https://git.uninsane.org/colin/uninsane";
-      inputs.nixpkgs.follows = "nixpkgs";
+      # inputs.nixpkgs.follows = "nixpkgs";
+      inputs.nixpkgs.follows = "nixpkgs-unpatched";
     };
   };
 
   outputs = {
     self,
-    nixpkgs,
     nixpkgs-unpatched,
+    nixpkgs-next-unpatched ? nixpkgs-unpatched,
+    nixpkgs-wayland,
     mobile-nixos,
     sops-nix,
     uninsane-dot-org,
     ...
   }@inputs:
     let
-      nixpkgsCompiledBy = local: nixpkgs.legacyPackages."${local}";
+      inherit (builtins) attrNames elem listToAttrs map mapAttrs;
+      # redefine some nixpkgs `lib` functions to avoid the infinite recursion
+      # of if we tried to use patched `nixpkgs.lib` as part of the patching process.
+      mapAttrs' = f: set:
+        listToAttrs (map (attr: f attr set.${attr}) (attrNames set));
+      optionalAttrs = cond: attrs: if cond then attrs else {};
+      # mapAttrs but without the `name` argument
+      mapAttrValues = f: mapAttrs (_: f);
 
-      evalHost = { name, local, target }:
-        let
-          # XXX: we'd prefer to use `nixosSystem = (nixpkgsCompiledBy target).nixos`
-          # but it doesn't propagate config to the underlying pkgs, meaning it doesn't let you use
-          # non-free packages even after setting nixpkgs.allowUnfree.
-          # XXX: patch using the target -- not local -- otherwise the target will
-          # need to emulate the host in order to rebuild!
-          nixosSystem = import ((nixpkgsCompiledBy target).path + "/nixos/lib/eval-config.nix");
-        in
-          (nixosSystem {
-            # we use pkgs built for and *by* the target, i.e. emulation, by default.
-            # cross compilation only happens on explicit access to `pkgs.cross`
-            system = target;
-            modules = [
-              (import ./hosts/instantiate.nix { localSystem = local; hostName = name; })
-              self.nixosModules.default
-              self.nixosModules.passthru
-              {
-                nixpkgs.overlays = [
-                  self.overlays.default
-                  self.overlays.passthru
-                  self.overlays.pins
-                ];
-              }
-            ];
-          });
-    in {
-      nixosConfigurations = {
-        servo = evalHost { name = "servo"; local = "x86_64-linux"; target = "x86_64-linux"; };
-        desko = evalHost { name = "desko"; local = "x86_64-linux"; target = "x86_64-linux"; };
-        lappy = evalHost { name = "lappy"; local = "x86_64-linux"; target = "x86_64-linux"; };
-        moby = evalHost { name = "moby"; local = "aarch64-linux"; target = "aarch64-linux"; };
-        # special cross-compiled variant, to speed up deploys from an x86 box to the arm target
-        # note that these *do* produce different store paths, because the closure for the tools used to cross compile
-        # v.s. emulate differ.
-        # so deploying foo-cross and then foo incurs some rebuilding.
-        moby-cross = evalHost { name = "moby"; local = "x86_64-linux"; target = "aarch64-linux"; };
-        rescue = evalHost { name = "rescue"; local = "x86_64-linux"; target = "x86_64-linux"; };
+      # rather than apply our nixpkgs patches as a flake input, do that here instead.
+      # this (temporarily?) resolves the bad UX wherein a subflake residing in the same git
+      # repo as the main flake causes the main flake to have an unstable hash.
+      patchNixpkgs = variant: nixpkgs: (import ./nixpatches/flake.nix).outputs {
+        inherit variant nixpkgs;
+        self = patchNixpkgs variant nixpkgs;
       };
 
+      nixpkgs' = patchNixpkgs "master" nixpkgs-unpatched;
+      nixpkgsCompiledBy = system: nixpkgs'.legacyPackages."${system}";
+
+      evalHost = { name, local, target, variant ? null, nixpkgs ? nixpkgs' }: nixpkgs.lib.nixosSystem {
+        system = target;
+        modules = [
+          {
+            nixpkgs.buildPlatform.system = local;
+          }
+          (optionalAttrs (local != target) {
+            # XXX(2023/12/11): cache.nixos.org uses `system = ...` instead of `hostPlatform.system`, and that choice impacts the closure of every package.
+            # so avoid specifying hostPlatform.system on non-cross builds, so i can use upstream caches.
+            nixpkgs.hostPlatform.system = target;
+          })
+          (optionalAttrs (variant == "light") {
+            sane.maxBuildCost = 2;
+          })
+          (optionalAttrs (variant == "min") {
+            sane.maxBuildCost = 0;
+          })
+          (import ./hosts/instantiate.nix { hostName = name; })
+          self.nixosModules.default
+          self.nixosModules.passthru
+          {
+            nixpkgs.overlays = [
+              self.overlays.passthru
+              self.overlays.sane-all
+            ];
+          }
+        ];
+      };
+    in {
+      nixosConfigurations = let
+        hosts = {
+          servo       = { name = "servo";  local = "x86_64-linux"; target = "x86_64-linux";  };
+          desko       = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  };
+          desko-light = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  variant = "light"; };
+          lappy       = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  };
+          lappy-light = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  variant = "light"; };
+          lappy-min =   { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  variant = "min"; };
+          moby        = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; };
+          moby-light  = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; variant = "light"; };
+          moby-min  =   { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; variant = "min"; };
+          rescue      = { name = "rescue"; local = "x86_64-linux"; target = "x86_64-linux";  };
+        };
+        hostsNext = mapAttrs' (h: v: {
+          name = "${h}-next";
+          value = v // { nixpkgs = patchNixpkgs "staging-next" nixpkgs-next-unpatched; };
+        }) hosts;
+      in mapAttrValues evalHost (
+        hosts // hostsNext
+      );
+
       # unofficial output
       # this produces a EFI-bootable .img file (GPT with a /boot partition and a system (/ or /nix) partition).
       # after building this:
@@ -109,25 +172,41 @@
       #   - if fs wasn't resized automatically, then `sudo btrfs filesystem resize max /`
       #   - checkout this flake into /etc/nixos AND UPDATE THE FS UUIDS.
       #   - `nixos-rebuild --flake './#<host>' switch`
-      imgs = builtins.mapAttrs (_: host-dfn: host-dfn.config.system.build.img) self.nixosConfigurations;
+      imgs = mapAttrValues (host: host.config.system.build.img) self.nixosConfigurations;
 
-      overlays = rec {
-        default = pkgs;
-        pkgs = import ./overlays/pkgs.nix;
-        pins = import ./overlays/pins.nix;  # TODO: move to `nixpatches/` input
-        passthru =
+      # unofficial output
+      hostConfigs = mapAttrValues (host: host.config) self.nixosConfigurations;
+      hostSystems = mapAttrValues (host: host.config.system.build.toplevel) self.nixosConfigurations;
+      hostPkgs = mapAttrValues (host: host.config.system.build.pkgs) self.nixosConfigurations;
+      hostPrograms = mapAttrValues (host: mapAttrValues (p: p.package) host.config.sane.programs) self.nixosConfigurations;
+
+      patched.nixpkgs = nixpkgs';
+
+      overlays = {
+        # N.B.: `nix flake check` requires every overlay to take `final: prev:` at defn site,
+        #   hence the weird redundancy.
+        default = final: prev: self.overlays.pkgs final prev;
+        sane-all = final: prev: import ./overlays/all.nix final prev;
+        pkgs = final: prev: import ./overlays/pkgs.nix final prev;
+        pins = final: prev: import ./overlays/pins.nix final prev;
+        preferences = final: prev: import ./overlays/preferences.nix final prev;
+        passthru = final: prev:
           let
-            stable =
-              if inputs ? "nixpkgs-stable" then (
-                next: prev: {
-                  stable = inputs.nixpkgs-stable.legacyPackages."${prev.stdenv.hostPlatform.system}";
-                }
-              ) else (next: prev: {});
             mobile = (import "${mobile-nixos}/overlay/overlay.nix");
-            uninsane = uninsane-dot-org.overlay;
+            uninsane = uninsane-dot-org.overlays.default;
+            wayland = final: prev: {
+              # default is to dump the packages into `waylandPkgs` *and* the toplevel.
+              # but i just want the `waylandPkgs` set
+              inherit (nixpkgs-wayland.overlays.default final prev)
+                waylandPkgs
+                new-wayland-protocols  #< 2024/03/10: nixpkgs-wayland assumes this will be in the toplevel
+              ;
+            };
           in
-            next: prev:
-              (stable next prev) // (mobile next prev) // (uninsane next prev);
+            (mobile final prev)
+            // (uninsane final prev)
+            // (wayland final prev)
+          ;
       };
 
       nixosModules = rec {
@@ -151,36 +230,425 @@
           aarch64-linux = allPkgsFor "aarch64-linux";
         };
 
-      # extract only our own packages from the full set
-      packages = builtins.mapAttrs
-        (_: full: full.sane // { inherit (full) sane uninsane-dot-org; })
-        self.legacyPackages;
+      # extract only our own packages from the full set.
+      # because of `nix flake check`, we flatten the package set and only surface x86_64-linux packages.
+      packages = mapAttrs
+        (system: passthruPkgs: passthruPkgs.lib.filterAttrs
+          (name: pkg:
+            # keep only packages which will pass `nix flake check`, i.e. keep only:
+            # - derivations (not package sets)
+            # - packages that build for the given platform
+            (! elem name [ "feeds" "pythonPackagesExtensions" ])
+            && (passthruPkgs.lib.meta.availableOn passthruPkgs.stdenv.hostPlatform pkg)
+          )
+          (
+            # expose sane packages and chosen inputs (uninsane.org)
+            (import ./pkgs { pkgs = passthruPkgs; }) // {
+              inherit (passthruPkgs) uninsane-dot-org;
+            }
+          )
+        )
+        # self.legacyPackages;
+        {
+          x86_64-linux = (nixpkgsCompiledBy "x86_64-linux").appendOverlays [
+            self.overlays.passthru
+          ];
+        }
+      ;
 
       apps."x86_64-linux" =
         let
           pkgs = self.legacyPackages."x86_64-linux";
-        in {
-          update-feeds = {
+          sanePkgs = import ./pkgs { inherit pkgs; };
+          deployScript = host: addr: action: pkgs.writeShellScript "deploy-${host}" ''
+            set -e
+
+            host="${host}"
+            addr="${addr}"
+            action="${if action != null then action else ""}"
+            runOnTarget() {
+              # run the command ($@) on the machine we're deploying to.
+              # if that's a remote machine, then do it via ssh, else local shell.
+              if [ -n "$addr" ]; then
+                ssh "$addr" "$@"
+              else
+                "$@"
+              fi
+            }
+
+            nix build ".#nixosConfigurations.$host.config.system.build.toplevel" --out-link "./build/result-$host" "$@"
+            storePath="$(readlink ./build/result-$host)"
+
+            # mimic `nixos-rebuild --target-host`, in effect:
+            # - nix-copy-closure ...
+            # - nix-env --set ...
+            # - switch-to-configuration <boot|dry-activate|switch|test|>
+            # avoid the actual `nixos-rebuild` for a few reasons:
+            # - fewer nix evals
+            # - more introspectability and debuggability
+            # - sandbox friendliness (especially: `git` doesn't have to be run as root)
+
+            if [ -n "$addr" ]; then
+              sudo nix store sign -r -k /run/secrets/nix_signing_key "$storePath"
+              # add more `-v` for more verbosity (up to 5).
+              # builders-use-substitutes false: optimizes so that the remote machine doesn't try to get paths from its substituters.
+              #   we already have all paths here, and the remote substitution is slow to check and SERIOUSLY flaky on moby in particular.
+              nix copy -vv --option builders-use-substitutes false --to "ssh-ng://$addr" "$storePath"
+            fi
+
+            if [ -n "$action" ]; then
+              runOnTarget sudo nix-env -p /nix/var/nix/profiles/system --set "$storePath"
+              runOnTarget sudo "$storePath/bin/switch-to-configuration" "$action"
+            fi
+          '';
+          deployApp = host: addr: action: {
             type = "app";
-            program = "${pkgs.feeds.passthru.updateScript}";
+            program = ''${deployScript host addr action}'';
+          };
+
+          # pkg updating.
+          # a cleaner alternative lives here: <https://discourse.nixos.org/t/how-can-i-run-the-updatescript-of-personal-packages/25274/2>
+          # mkUpdater :: [ String ] -> { type = "app"; program = path; }
+          mkUpdater = attrPath: {
+            type = "app";
+            program = let
+              pkg = pkgs.lib.getAttrFromPath attrPath sanePkgs;
+              strAttrPath = pkgs.lib.concatStringsSep "." attrPath;
+              commandArgv = pkg.updateScript.command or pkg.updateScript;
+              command = pkgs.lib.escapeShellArgs commandArgv;
+            in builtins.toString (pkgs.writeShellScript "update-${strAttrPath}" ''
+              set -x
+              env UPDATE_NIX_NAME=${pkg.name} UPDATE_NIX_PNAME=${pkg.pname} UPDATE_NIX_OLD_VERSION=${pkg.version} UPDATE_NIX_ATTR_PATH=${strAttrPath} ${command}
+            '');
+          };
+          mkUpdatersNoAliases = opts: basePath: pkgs.lib.concatMapAttrs
+            (name: pkg:
+              if pkg.recurseForDerivations or false then {
+                "${name}" = mkUpdaters opts (basePath ++ [ name ]);
+              } else if pkg.updateScript or null != null then {
+                "${name}" = mkUpdater (basePath ++ [ name ]);
+              } else {}
+            )
+            (pkgs.lib.getAttrFromPath basePath sanePkgs);
+          mkUpdaters = { ignore ? [], flakePrefix ? [] }@opts: basePath:
+            let
+              updaters = mkUpdatersNoAliases opts basePath;
+              invokeUpdater = name: pkg:
+                let
+                  fullPath = basePath ++ [ name ];
+                  doUpdateByDefault = !builtins.elem fullPath ignore;
+
+                  # in case `name` has a `.` in it, we have to quote it
+                  escapedPath = builtins.map (p: ''"${p}"'') fullPath;
+                  updatePath = builtins.concatStringsSep "." (flakePrefix ++ escapedPath);
+                in pkgs.lib.optionalString doUpdateByDefault (
+                  pkgs.lib.escapeShellArgs [
+                    "nix" "run" ".#${updatePath}"
+                  ]
+                );
+            in {
+              type = "app";
+              # top-level app just invokes the updater of everything one layer below it
+              program = builtins.toString (pkgs.writeShellScript
+                (builtins.concatStringsSep "-" (flakePrefix ++ basePath))
+                (builtins.concatStringsSep
+                  "\n"
+                  (pkgs.lib.mapAttrsToList invokeUpdater updaters)
+                )
+              );
+            } // updaters;
+        in {
+          help = {
+            type = "app";
+            program = let
+              helpMsg = builtins.toFile "nixos-config-help-message" ''
+                commands:
+                - `nix run '.#help'`
+                  - show this message
+                - `nix run '.#update.pkgs'`
+                  - updates every package
+                - `nix run '.#update.feeds'`
+                  - updates metadata for all feeds
+                - `nix run '.#init-feed' <url>`
+                - `nix run '.#deploy.{desko,lappy,moby,servo}[-light|-test]' [nix args ...]`
+                  - build and deploy the host
+                - `nix run '.#preDeploy.{desko,lappy,moby,servo}[-light]' [nix args ...]`
+                  - copy closures to a host, but don't activate it
+                  - or `nix run '.#preDeploy'` to target all hosts
+                - `nix run '.#check'`
+                  - make sure all systems build; NUR evaluates
+                - `nix run '.#bench'`
+                  - benchmark the eval time of common targets this flake provides
+
+                specific build targets of interest:
+                - `nix build '.#imgs.rescue'`
+              '';
+            in builtins.toString (pkgs.writeShellScript "nixos-config-help" ''
+              cat ${helpMsg}
+              echo ""
+              echo "complete flake structure:"
+              nix flake show --option allow-import-from-derivation true
+            '');
+          };
+          # wrangle some names to get package updaters which refer back into the flake, but also conditionally ignore certain paths (e.g. sane.feeds).
+          # TODO: better design
+          update = rec {
+            _impl.pkgs.sane = mkUpdaters { flakePrefix = [ "update" "_impl" "pkgs" ]; ignore = [ [ "sane" "feeds" ] ]; } [ "sane" ];
+            pkgs = _impl.pkgs.sane;
+            _impl.feeds.sane.feeds = mkUpdaters { flakePrefix = [ "update" "_impl" "feeds" ]; } [ "sane" "feeds" ];
+            feeds = _impl.feeds.sane.feeds;
           };
 
           init-feed = {
-            # use like `nix run '.#init-feed' uninsane.org`
             type = "app";
-            program = "${pkgs.feeds.passthru.initFeedScript}";
+            program = "${pkgs.feeds.init-feed}";
+          };
+
+          deploy = {
+            desko       = deployApp "desko"       "desko" "switch";
+            desko-light = deployApp "desko-light" "desko" "switch";
+            lappy       = deployApp "lappy"       "lappy" "switch";
+            lappy-light = deployApp "lappy-light" "lappy" "switch";
+            lappy-min   = deployApp "lappy-min"   "lappy" "switch";
+            moby        = deployApp "moby"        "moby"  "switch";
+            moby-light  = deployApp "moby-light"  "moby"  "switch";
+            moby-min  =   deployApp "moby-min"    "moby"  "switch";
+            moby-test   = deployApp "moby"        "moby"  "test";
+            servo       = deployApp "servo"       "servo" "switch";
+
+            # like `nixos-rebuild --flake . switch`
+            self        = deployApp "$(hostname)"       "" "switch";
+            self-light  = deployApp "$(hostname)-light" "" "switch";
+            self-min    = deployApp "$(hostname)-min"   "" "switch";
+
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "deploy-all" ''
+              nix run '.#deploy.lappy'
+              nix run '.#deploy.moby'
+              nix run '.#deploy.desko'
+              nix run '.#deploy.servo'
+            '');
+          };
+          preDeploy = {
+            # build the host and copy the runtime closure to that host, but don't activate it.
+            desko       = deployApp "desko"       "desko" null;
+            desko-light = deployApp "desko-light" "desko" null;
+            lappy       = deployApp "lappy"       "lappy" null;
+            lappy-light = deployApp "lappy-light" "lappy" null;
+            lappy-min   = deployApp "lappy-min"   "lappy" null;
+            moby        = deployApp "moby"        "moby"  null;
+            moby-light  = deployApp "moby-light"  "moby"  null;
+            moby-min    = deployApp "moby-min"    "moby"  null;
+            servo       = deployApp "servo"       "servo" null;
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "predeploy-all" ''
+              # copy the -min/-light variants first; this might be run while waiting on a full build. or the full build failed.
+              nix run '.#preDeploy.moby-min' -- "$@"
+              nix run '.#preDeploy.lappy-min' -- "$@"
+              nix run '.#preDeploy.moby-light' -- "$@"
+              nix run '.#preDeploy.lappy-light' -- "$@"
+              nix run '.#preDeploy.desko-light' -- "$@"
+              nix run '.#preDeploy.lappy' -- "$@"
+              nix run '.#preDeploy.servo' -- "$@"
+              nix run '.#preDeploy.moby' -- "$@"
+              nix run '.#preDeploy.desko' -- "$@"
+            '');
+          };
+
+          sync = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-all" ''
+              RC_lappy=$(nix run '.#sync.lappy' -- "$@")
+              RC_moby=$(nix run '.#sync.moby' -- "$@")
+              RC_desko=$(nix run '.#sync.desko' -- "$@")
+
+              echo "lappy: $RC_lappy"
+              echo "moby: $RC_moby"
+              echo "desko: $RC_desko"
+            '');
+          };
+
+          sync.desko = {
+            # copy music from servo to desko
+            # can run this from any device that has ssh access to desko and servo
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-to-desko" ''
+              sudo mount /mnt/desko/home
+              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music --compat /mnt/servo/media/Music /mnt/desko/home/Music "$@"
+            '');
+          };
+
+          sync.lappy = {
+            # copy music from servo to lappy
+            # can run this from any device that has ssh access to lappy and servo
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-to-lappy" ''
+              sudo mount /mnt/lappy/home
+              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music --compress --compat /mnt/servo/media/Music /mnt/lappy/home/Music "$@"
+            '');
+          };
+
+          sync.moby = {
+            # copy music from servo to moby
+            # can run this from any device that has ssh access to moby and servo
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-to-moby" ''
+              sudo mount /mnt/moby/home
+              sudo mount /mnt/desko/home
+              ${pkgs.rsync}/bin/rsync -arv --exclude servo-macros /mnt/moby/home/Pictures/ /mnt/desko/home/Pictures/moby/
+              # N.B.: limited by network/disk -> reduce job count to improve pause/resume behavior
+              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music --compress --compat --jobs 4 /mnt/servo/media/Music /mnt/moby/home/Music "$@"
+            '');
+          };
+
+          check = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "check-all" ''
+              nix run '.#check.nur'
+              RC0=$?
+              nix run '.#check.hostConfigs'
+              RC1=$?
+              nix run '.#check.rescue'
+              RC2=$?
+              echo "nur: $RC0"
+              echo "hostConfigs: $RC1"
+              echo "rescue: $RC2"
+              exit $(($RC0 | $RC1 | $RC2))
+            '');
+          };
+
+          check.nur = {
+            # `nix run '.#check-nur'`
+            # validates that my repo can be included in the Nix User Repository
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "check-nur" ''
+              cd ${./.}/integrations/nur
+              NIX_PATH= NIXPKGS_ALLOW_UNSUPPORTED_SYSTEM=1 nix-env -f . -qa \* --meta --xml \
+                --allowed-uris https://static.rust-lang.org \
+                --option restrict-eval true \
+                --option allow-import-from-derivation true \
+                --drv-path --show-trace \
+                -I nixpkgs=${nixpkgs-unpatched} \
+                -I nixpkgs-overlays=${./.}/hosts/common/nix/overlay \
+                -I ../../ \
+                | tee  # tee to prevent interactive mode
+            '');
+          };
+
+          check.hostConfigs = {
+            type = "app";
+            program = let
+              checkHost = host: let
+                shellHost = pkgs.lib.replaceStrings [ "-" ] [ "_" ] host;
+              in ''
+                nix build -v '.#nixosConfigurations.${host}.config.system.build.toplevel' --out-link ./build/result-${host} -j2 "$@"
+                RC_${shellHost}=$?
+              '';
+            in builtins.toString (pkgs.writeShellScript
+              "check-host-configs"
+              ''
+                # build minimally-usable hosts first, then their full image.
+                # this gives me a minimal image i can deploy or copy over, early.
+                ${checkHost "lappy-min"}
+                ${checkHost "moby-min"}
+
+                ${checkHost "desko-light"}
+                ${checkHost "moby-light"}
+                ${checkHost "lappy-light"}
+
+                ${checkHost "desko"}
+                ${checkHost "lappy"}
+                ${checkHost "servo"}
+                ${checkHost "moby"}
+                ${checkHost "rescue"}
+
+                # still want to build the -light variants first so as to avoid multiple simultaneous webkitgtk builds
+                ${checkHost "desko-light-next"}
+                ${checkHost "moby-light-next"}
+
+                ${checkHost "desko-next"}
+                ${checkHost "lappy-next"}
+                ${checkHost "servo-next"}
+                ${checkHost "moby-next"}
+                ${checkHost "rescue-next"}
+
+                echo "desko: $RC_desko"
+                echo "lappy: $RC_lappy"
+                echo "servo: $RC_servo"
+                echo "moby: $RC_moby"
+                echo "rescue: $RC_rescue"
+
+                echo "desko-next: $RC_desko_next"
+                echo "lappy-next: $RC_lappy_next"
+                echo "servo-next: $RC_servo_next"
+                echo "moby-next: $RC_moby_next"
+                echo "rescue-next: $RC_rescue_next"
+
+                # i don't really care if the -next hosts fail. i build them mostly to keep the cache fresh/ready
+                exit $(($RC_desko | $RC_lappy | $RC_servo | $RC_moby | $RC_rescue))
+              ''
+            );
+          };
+
+          check.rescue = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "check-rescue" ''
+              nix build -v '.#imgs.rescue' --out-link ./build/result-rescue-img -j2
+            '');
+          };
+
+          bench = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "bench" ''
+              doBench() {
+                attrPath="$1"
+                shift
+                echo -n "benchmarking eval of '$attrPath'... "
+                /run/current-system/sw/bin/time -f "%e sec" -o /dev/stdout \
+                  nix eval --no-eval-cache --quiet --raw ".#$attrPath" --apply 'result: if result != null then "" else "unexpected null"' $@ 2> /dev/null
+              }
+
+              if [ -n "$1" ]; then
+                doBench "$@"
+              else
+                doBench hostConfigs
+                doBench hostConfigs.lappy
+                doBench hostConfigs.lappy.sane.programs
+                doBench hostConfigs.lappy.sane.users.colin
+                doBench hostConfigs.lappy.sane.fs
+                doBench hostConfigs.lappy.environment.systemPackages
+              fi
+            '');
           };
         };
 
       templates = {
-        python-data = {
+        env.python-data = {
           # initialize with:
-          # - `nix flake init -t '/home/colin/dev/nixos/#python-data'`
+          # - `nix flake init -t '/home/colin/dev/nixos/#env.python-data'`
           # then enter with:
           # - `nix develop`
-          path = ./templates/python-data;
+          path = ./templates/env/python-data;
           description = "python environment for data processing";
         };
+        pkgs.rust-inline = {
+          # initialize with:
+          # - `nix flake init -t '/home/colin/dev/nixos/#pkgs.rust-inline'`
+          path = ./templates/pkgs/rust-inline;
+          description = "rust package and development environment (inline rust sources)";
+        };
+        pkgs.rust = {
+          # initialize with:
+          # - `nix flake init -t '/home/colin/dev/nixos/#pkgs.rust'`
+          path = ./templates/pkgs/rust;
+          description = "rust package fit to ship in nixpkgs";
+        };
+        pkgs.make = {
+          # initialize with:
+          # - `nix flake init -t '/home/colin/dev/nixos/#pkgs.make'`
+          path = ./templates/pkgs/make;
+          description = "default Makefile-based derivation";
+        };
       };
     };
 }

hosts/README.md

@@ -0,0 +1,7 @@
+## directory structure
+- by-name/<hostname>: configuration which is evaluated _only_ for the given hostname
+- common/: configuration which applies to all hosts
+- modules/: nixpkgs-style modules which may be used by multiple hosts, but configured separately per host.
+    - ideally no module here has effect unless `enable`d
+        - however, `enable` may default to true
+        - and in practice some of these modules surely aren't fully "disableable"

hosts/by-name/desko/default.nix

@@ -1,20 +1,39 @@
-{ config, pkgs, ... }:
+{ config, lib, pkgs, ... }:
 {
   imports = [
     ./fs.nix
   ];
 
-  # sane.packages.enableDevPkgs = true;
+  sane.services.trust-dns.asSystemResolver = false;  # TEMPORARY: TODO: re-enable trust-dns
+  # sane.programs.devPkgs.enableFor.user.colin = true;
+  # sane.guest.enable = true;
 
+  # don't enable wifi by default: it messes with connectivity.
+  # systemd.services.iwd.enable = false;
+  # systemd.services.wpa_supplicant.enable = false;
+  sane.programs.wpa_supplicant.enableFor.user.colin = lib.mkForce false;
+  sane.programs.wpa_supplicant.enableFor.system = lib.mkForce false;
+
+  sops.secrets.colin-passwd.neededForUsers = true;
+
+  sane.roles.build-machine.enable = true;
   sane.roles.client = true;
+  sane.roles.dev-machine = true;
+  sane.roles.pc = true;
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."desko".wg-home.ip;
+  sane.ovpn.addrV4 = "172.26.55.21";
+  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:20c1:a73c";
   sane.services.duplicity.enable = true;
-  sane.services.nixserve.enable = true;
-  sane.services.nixserve.sopsFile = ../../../secrets/desko.yaml;
-  sane.persist.enable = true;
 
-  sane.gui.sway.enable = true;
+  sane.nixcache.remote-builders.desko = false;
+
+  sane.programs.sway.enableFor.user.colin = true;
+  sane.programs.iphoneUtils.enableFor.user.colin = true;
+  sane.programs.steam.enableFor.user.colin = true;
+
+  sane.programs."gnome.geary".config.autostart = true;
+  sane.programs.signal-desktop.config.autostart = true;
 
   boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
@@ -22,14 +41,6 @@
   # needed to use libimobiledevice/ifuse, for iphone sync
   services.usbmuxd.enable = true;
 
-  sops.secrets.colin-passwd = {
-    sopsFile = ../../../secrets/desko.yaml;
-    neededForUsers = true;
-  };
-
-  # don't enable wifi by default: it messes with connectivity.
-  systemd.services.iwd.enable = false;
-
   # default config: https://man.archlinux.org/man/snapper-configs.5
   # defaults to something like:
   #   - hourly snapshots
@@ -37,28 +48,11 @@
   services.snapper.configs.nix = {
     # TODO: for the impermanent setup, we'd prefer to just do /nix/persist,
     # but that also requires setting up the persist dir as a subvol
-    subvolume = "/nix";
+    SUBVOLUME = "/nix";
     # TODO: ALLOW_USERS doesn't seem to work. still need `sudo snapper -c nix list`
-    extraConfig = ''
-      ALLOW_USERS = "colin";
-    '';
+    ALLOW_USERS = [ "colin" ];
   };
 
-  sops.secrets.duplicity_passphrase = {
-    sopsFile = ../../../secrets/desko.yaml;
-  };
-
-  programs.steam = {
-    enable = true;
-    # not sure if needed: stole this whole snippet from the wiki
-    remotePlay.openFirewall = true; # Open ports in the firewall for Steam Remote Play
-    dedicatedServer.openFirewall = true; # Open ports in the firewall for Source Dedicated Server
-  };
-  sane.user.persist.plaintext = [
-    ".steam"
-    ".local/share/Steam"
-  ];
-
   # docs: https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion
   system.stateVersion = "21.05";
 }

hosts/by-name/desko/fs.nix

@@ -1,21 +1,12 @@
 { ... }:
 
 {
-  sane.persist.root-on-tmpfs = true;
-  # we need a /tmp for building large nix things.
+  # increase /tmp space (defaults to 50% of RAM) for building large nix things.
   # a cross-compiled kernel, particularly, will easily use 30+GB of tmp
-  fileSystems."/tmp" = {
-    device = "none";
-    fsType = "tmpfs";
-    options = [
-      "mode=777"
-      "size=64G"
-      "defaults"
-    ];
-  };
+  fileSystems."/tmp".options = [ "size=64G" ];
+
   fileSystems."/nix" = {
-    # device = "/dev/disk/by-uuid/985a0a32-da52-4043-9df7-615adec2e4ff";
-    device = "/dev/disk/by-uuid/0ab0770b-7734-4167-88d9-6e4e20bb2a56";
+    device = "/dev/disk/by-uuid/845d85bf-761d-431b-a406-e6f20909154f";
     fsType = "btrfs";
     options = [
       "compress=zstd"
@@ -24,8 +15,7 @@
   };
 
   fileSystems."/boot" = {
-    # device = "/dev/disk/by-uuid/CAA7-E7D2";
-    device = "/dev/disk/by-uuid/41B6-BAEF";
+    device = "/dev/disk/by-uuid/5049-9AFD";
     fsType = "vfat";
   };
 }

hosts/by-name/lappy/default.nix

@@ -5,22 +5,24 @@
   ];
 
   sane.roles.client = true;
+  sane.roles.dev-machine = true;
+  sane.roles.pc = true;
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."lappy".wg-home.ip;
-
-  # sane.packages.enableDevPkgs = true;
+  sane.ovpn.addrV4 = "172.23.119.72";
+  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:0332:aa96/128";
 
   # sane.guest.enable = true;
-  sane.gui.sway.enable = true;
-  sane.persist.enable = true;
-  sane.nixcache.enable = true;
   boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
 
-  sops.secrets.colin-passwd = {
-    sopsFile = ../../../secrets/lappy.yaml;
-    neededForUsers = true;
-  };
+  sane.programs.stepmania.enableFor.user.colin = true;
+  sane.programs.sway.enableFor.user.colin = true;
+
+  sane.programs."gnome.geary".config.autostart = true;
+  sane.programs.signal-desktop.config.autostart = true;
+
+  sops.secrets.colin-passwd.neededForUsers = true;
 
   # default config: https://man.archlinux.org/man/snapper-configs.5
   # defaults to something like:
@@ -29,12 +31,10 @@
   services.snapper.configs.nix = {
     # TODO: for the impermanent setup, we'd prefer to just do /nix/persist,
     # but that also requires setting up the persist dir as a subvol
-    subvolume = "/nix";
+    SUBVOLUME = "/nix";
+    ALLOW_USERS = [ "colin" ];
   };
 
-  # TODO: only here for debugging
-  # services.ipfs.enable = true;
-
   # docs: https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion
   system.stateVersion = "21.05";
 }

hosts/by-name/lappy/fs.nix

@@ -1,17 +1,6 @@
 { ... }:
 
 {
-  sane.persist.root-on-tmpfs = true;
-  # we need a /tmp of default size (half RAM) for building large nix things
-  fileSystems."/tmp" = {
-    device = "none";
-    fsType = "tmpfs";
-    options = [
-      "mode=777"
-      "defaults"
-    ];
-  };
-
   fileSystems."/nix" = {
     device = "/dev/disk/by-uuid/75230e56-2c69-4e41-b03e-68475f119980";
     fsType = "btrfs";
@@ -25,24 +14,4 @@
     device = "/dev/disk/by-uuid/BD79-D6BB";
     fsType = "vfat";
   };
-
-  # fileSystems."/nix" = {
-  #   device = "/dev/disk/by-uuid/5a7fa69c-9394-8144-a74c-6726048b129f";
-  #   fsType = "btrfs";
-  # };
-
-  # fileSystems."/boot" = {
-  #   device = "/dev/disk/by-uuid/4302-1685";
-  #   fsType = "vfat";
-  # };
-
-  # fileSystems."/" = {
-  #   device = "none";
-  #   fsType = "tmpfs";
-  #   options = [
-  #     "mode=755"
-  #     "size=1G"
-  #     "defaults"
-  #   ];
-  # };
 }

hosts/by-name/lappy/xkb_mobile_normal_buttons

@@ -0,0 +1,7 @@
+xkb_keymap {
+	xkb_keycodes  { include "evdev+aliases(qwerty)"	};
+	xkb_types     { include "complete"	};
+	xkb_compat    { include "complete"	};
+	xkb_symbols   { include "pc+us+inet(evdev)"	};
+	xkb_geometry  { include "pc(pc105)"	};
+};

hosts/by-name/moby/bootloader.nix

@@ -0,0 +1,22 @@
+# tow-boot: <https://tow-boot.org>
+# docs (pinephone specific): <https://github.com/Tow-Boot/Tow-Boot/tree/development/boards/pine64-pinephoneA64>
+# LED and button behavior is defined here: <https://github.com/Tow-Boot/Tow-Boot/blob/development/modules/tow-boot/phone-ux.nix>
+# - hold VOLDOWN: enter recovery mode
+#   - LED will turn aqua instead of yellow
+#   - recovery mode would ordinarily allow a selection of entries, but for pinephone i guess it doesn't do anything?
+# - hold VOLUP: force it to load the OS from eMMC?
+#   - LED will turn blue instead of yellow
+# boot LEDs:
+# - yellow = entered tow-boot
+# - 10 red flashes => poweroff means tow-boot couldn't boot into the next stage (i.e. distroboot)
+#   - distroboot: <https://source.denx.de/u-boot/u-boot/-/blob/v2022.04/doc/develop/distro.rst>)
+{ config, pkgs, ... }:
+{
+  # we need space in the GPT header to place tow-boot.
+  # only actually need 1 MB, but better to over-allocate than under-allocate
+  sane.image.extraGPTPadding = 16 * 1024 * 1024;
+  sane.image.firstPartGap = 0;
+  sane.image.installBootloader = ''
+    dd if=${pkgs.tow-boot-pinephone}/Tow-Boot.noenv.bin of=$out/nixos.img bs=1024 seek=8 conv=notrunc
+  '';
+}

hosts/by-name/moby/default.nix

@@ -1,81 +1,103 @@
+# Pinephone
+#
+# wikis, resources, ...:
+# - Linux Phone Apps: <https://linuxphoneapps.org/>
+#   - massive mobile-friendly app database
+# - Mobian wiki: <https://wiki.mobian-project.org/doku.php?id=start>
+#   - recommended apps, chatrooms
+
 { config, pkgs, lib, ... }:
 {
   imports = [
-    ./firmware.nix
+    ./bootloader.nix
     ./fs.nix
+    ./gps.nix
     ./kernel.nix
+    ./polyfill.nix
   ];
 
   sane.roles.client = true;
+  sane.roles.handheld = true;
+  sane.programs.zsh.config.showDeadlines = false;  # unlikely to act on them when in shell
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."moby".wg-home.ip;
-
-  # cross-compiled documentation is *slow*.
-  # no obvious way to natively compile docs (2022/09/29).
-  # entrypoint is nixos/modules/misc/documentation.nix
-  # doc building happens in nixos/doc/manual/default.nix
-  # TODO: we could *maybe* inject pkgs.buildPackages.xyz = cross.buildPackages.xyz?
-  documentation.nixos.enable = false;
+  sane.ovpn.addrV4 = "172.24.87.255";
+  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:18cd:a72b";
 
   # XXX colin: phosh doesn't work well with passwordless login,
   # so set this more reliable default password should anything go wrong
   users.users.colin.initialPassword = "147147";
-  services.getty.autologinUser = "root";  # allows for emergency maintenance?
+  # services.getty.autologinUser = "root";  # allows for emergency maintenance?
 
-  sops.secrets.colin-passwd = {
-    sopsFile = ../../../secrets/moby.yaml;
-    neededForUsers = true;
+  sops.secrets.colin-passwd.neededForUsers = true;
+
+  sane.programs.sway.enableFor.user.colin = true;
+  sane.programs.swaylock.enableFor.user.colin = false;  #< not usable on touch
+  sane.programs.schlock.enableFor.user.colin = true;
+  sane.programs.swayidle.config.actions.screenoff.delay = 300;
+  sane.programs.swayidle.config.actions.screenoff.enable = true;
+  sane.programs.sane-input-handler.enableFor.user.colin = true;
+  sane.programs.blueberry.enableFor.user.colin = false;  # bluetooth manager: doesn't cross compile!
+  sane.programs.fcitx5.enableFor.user.colin = false;  # does not cross compile
+  sane.programs.mercurial.enableFor.user.colin = false;  # does not cross compile
+  sane.programs.nvme-cli.enableFor.system = false;  # does not cross compile (libhugetlbfs)
+
+  # enabled for easier debugging
+  sane.programs.eg25-control.enableFor.user.colin = true;
+  sane.programs.rtl8723cs-wowlan.enableFor.user.colin = true;
+
+  # sane.programs.ntfy-sh.config.autostart = true;
+  sane.programs.dino.config.autostart = true;
+  # sane.programs.signal-desktop.config.autostart = true;  # TODO: enable once electron stops derping.
+  # sane.programs."gnome.geary".config.autostart = true;
+  # sane.programs.calls.config.autostart = true;
+
+  sane.programs.firefox.mime.priority = 300;  # prefer other browsers when possible
+  # HACK/TODO: make `programs.P.env.VAR` behave according to `mime.priority`
+  sane.programs.firefox.env = lib.mkForce {};
+  sane.programs.epiphany.env.BROWSER = "epiphany";
+  sane.programs.pipewire.config = {
+    # tune so Dino doesn't drop audio
+    # there's seemingly two buffers for the mic (see: <https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/FAQ#pipewire-buffering-explained>)
+    # 1. Pipewire buffering out of the driver and into its own member.
+    # 2. Pipewire buffering into Dino.
+    # the latter is fixed at 10ms by Dino, difficult to override via runtime config.
+    # the former defaults low (e.g. 512 samples)
+    # this default configuration causes the mic to regularly drop out entirely for a couple seconds at a time during a call,
+    # presumably because the system can't keep up (pw-top shows incrementing counter in ERR column).
+    # `pw-metadata -n settings 0 clock.force-quantum 1024` reduces to about 1 error per second.
+    # `pw-metadata -n settings 0 clock.force-quantum 2048` reduces to 1 error every < 10s.
+    # pipewire default config includes `clock.power-of-two-quantum = true`
+    min-quantum = 2048;
+    max-quantum = 8192;
   };
 
-  sane.web-browser = {
-    # compromise impermanence for the sake of usability
-    persistCache = "private";
-    persistData = "private";
-
-    # i don't do crypto stuff on moby
-    addons.ether-metamask.enable = false;
-    # addons.sideberry.enable = false;
-  };
-
-  sane.user.persist.plaintext = [
-    ".config/pulse"  # persist pulseaudio volume
-  ];
-
-  # sane.packages.enableGuiPkgs = false;  # XXX faster builds/imaging for debugging
-  sane.packages.extraUserPkgs = [
-    pkgs.plasma5Packages.konsole  # terminal
-  ];
-
-  sane.nixcache.enable = true;
-  sane.persist.enable = true;
-  sane.gui.phosh.enable = true;
-
   boot.loader.efi.canTouchEfiVariables = false;
   # /boot space is at a premium. default was 20.
   # even 10 can be too much
-  # TODO: compress moby kernels!
   boot.loader.generic-extlinux-compatible.configurationLimit = 8;
   # mobile.bootloader.enable = false;
   # mobile.boot.stage-1.enable = false;
   # boot.initrd.systemd.enable = false;
   # boot.initrd.services.swraid.enable = false;  # attempt to fix dm_mod stuff
-  # disable proximity sensor.
-  # the filtering/calibration is bad that it causes the screen to go fully dark at times.
-  boot.blacklistedKernelModules = [ "stk3310" ];
-
-  # without this some GUI apps fail: `DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory`
-  # this is because they can't allocate enough video ram.
-  # the default CMA seems to be 32M.
-  # i was running fine with 256MB from 2022/07-ish through 2022/12-ish, but then the phone quit reliably coming back from sleep: maybe a memory leak?
-  # `cat /proc/meminfo` to see CmaTotal/CmaFree if interested in tuning this.
-  boot.kernelParams = [ "cma=512M" ];
 
+  # hardware.firmware makes the referenced files visible to the kernel, for whenever a driver explicitly asks for them.
+  # these files are visible from userspace by following `/sys/module/firmware_class/parameters/path`
+  #
   # mobile-nixos' /lib/firmware includes:
   #   rtl_bt          (bluetooth)
-  #   anx7688-fw.bin  (USB-C -> HDMI bridge)
+  #   anx7688-fw.bin  (USB-C chip: power negotiation, HDMI/dock)
   #   ov5640_af.bin   (camera module)
   # hardware.firmware = [ config.mobile.device.firmware ];
-  hardware.firmware = [ pkgs.rtl8723cs-firmware ];
+  # hardware.firmware = [ pkgs.rtl8723cs-firmware ];
+  hardware.firmware = [
+    (pkgs.linux-firmware-megous.override {
+      # rtl_bt = false probably means no bluetooth connectivity.
+      # N.B.: DON'T RE-ENABLE without first confirming that wake-on-lan works during suspend (rtcwake).
+      # it seems the rtl_bt stuff ("bluetooth coexist") might make wake-on-LAN radically more flaky.
+      rtl_bt = false;
+    })
+  ];
 
   system.stateVersion = "21.11";
 
@@ -86,16 +108,17 @@
   '';
 
   # enable rotation sensor
-  hardware.sensor.iio.enable = true;
+  # hardware.sensor.iio.enable = true;
 
-  # from https://gitlab.manjaro.org/manjaro-arm/packages/community/phosh/alsa-ucm-pinephone
-  # mobile-nixos does this same thing, with *slightly different settings*.
-  # i trust manjaro more because the guy maintaining that is actively trying to upstream into alsa-ucm-conf.
-  # an alternative may be to build a custom alsa with the PinePhone config patch applied:
-  # - <https://github.com/alsa-project/alsa-ucm-conf/pull/134>
-  # that would make this be not device-specific
-  environment.variables.ALSA_CONFIG_UCM2 = "${./ucm2}";
-  systemd.services.pulseaudio.environment.ALSA_CONFIG_UCM2 = "${./ucm2}";
+  services.udev.extraRules = let
+    chmod = "${pkgs.coreutils}/bin/chmod";
+    chown = "${pkgs.coreutils}/bin/chown";
+  in ''
+    # make Pinephone flashlight writable by user.
+    # taken from postmarketOS: <repo:postmarketOS/pmaports:device/main/device-pine64-pinephone/60-flashlight.rules>
+    SUBSYSTEM=="leds", DEVPATH=="*/*:flash", RUN+="${chmod} g+w /sys%p/brightness /sys%p/flash_strobe", RUN+="${chown} :video /sys%p/brightness /sys%p/flash_strobe"
 
-  hardware.opengl.driSupport = true;
+    # make Pinephone front LEDs writable by user.
+    SUBSYSTEM=="leds", DEVPATH=="*/*:indicator", RUN+="${chmod} g+w /sys%p/brightness", RUN+="${chown} :video /sys%p/brightness"
+  '';
 }

hosts/by-name/moby/firmware.nix

@@ -1,12 +0,0 @@
-{ config, pkgs, ... }:
-{
-  # we need space in the GPT header to place tow-boot.
-  # only actually need 1 MB, but better to over-allocate than under-allocate
-  sane.image.extraGPTPadding = 16 * 1024 * 1024;
-  sane.image.firstPartGap = 0;
-  system.build.img = pkgs.runCommand "nixos_full-disk-image.img" {} ''
-    cp -v ${config.system.build.img-without-firmware}/nixos.img $out
-    chmod +w $out
-    dd if=${pkgs.tow-boot-pinephone}/Tow-Boot.noenv.bin of=$out bs=1024 seek=8 conv=notrunc
-  '';
-}

hosts/by-name/moby/fs.nix

@@ -1,7 +1,6 @@
 { ... }:
 
 {
-  sane.persist.root-on-tmpfs = true;
   fileSystems."/nix" = {
     device = "/dev/disk/by-uuid/1f1271f8-53ce-4081-8a29-60a4a6b5d6f9";
     fsType = "btrfs";

hosts/by-name/moby/gps.nix

@@ -0,0 +1,68 @@
+# pinephone GPS happens in EG25 modem
+# serial control interface to modem is /dev/ttyUSB2
+# after enabling GPS, readout is /dev/ttyUSB1
+#
+# minimal process to enable modem and GPS:
+# - `echo 1 > /sys/class/modem-power/modem-power/device/powered`
+# - `screen /dev/ttyUSB2 115200`
+#   - `AT+QGPSCFG="nmeasrc",1`
+#   - `AT+QGPS=1`
+# this process is automated by my `eg25-control` program and services (`eg25-control-powered`, `eg25-control-gps`)
+# - see the `modules/` directory further up this repository.
+#
+# now, something like `gpsd` can directly read from /dev/ttyUSB1,
+# or geoclue can query the GPS directly through modem-manager
+#
+# initial GPS fix can take 15+ minutes.
+# meanwhile, services like eg25-manager or eg25-control-freshen-agps can speed this up by uploading assisted GPS data to the modem.
+#
+# support/help:
+# - geoclue, gnome-maps
+#   - irc: #gnome-maps on irc.gimp.org
+#   - Matrix: #gnome-maps:gnome.org  (unclear if bridged to IRC)
+#
+# programs to pair this with:
+# - `satellite-gtk`: <https://codeberg.org/tpikonen/satellite>
+#   - shows/tracks which satellites the GPS is connected to; useful to understand fix characteristics
+# - `gnome-maps`: uses geoclue, has route planning
+# - `mepo`: uses gpsd, minimalist, flaky, and buttons are kinda hard to activate on mobile
+# - puremaps?
+# - osmin?
+#
+# known/outstanding bugs:
+# - `systemctl start eg25-control-gps` can the hang the whole system (2023/10/06)
+#   - i think it's actually `eg25-control-powered` which does this (started by the gps)
+#   - best guess is modem draws so much power at launch that other parts of the system see undervoltage
+#   - workaround is to hard power-cycle the system. the modem may not bring up after reboot: leave unpowered for 60s and boot again.
+#
+# future work:
+# - integrate with [wigle](https://www.wigle.net/) for offline equivalent to Mozilla Location Services
+
+{ config, lib, ... }:
+{
+  # test gpsd with `gpspipe -w -n 10 2> /dev/null | grep -m 1 TPV | jq '.lat, .lon' | tr '\n' ' '`
+  # ^ should return <lat> <long>
+  services.gpsd.enable = true;
+  services.gpsd.devices = [ "/dev/ttyUSB1" ];
+
+  # test geoclue2 by building `geoclue2-with-demo-agent`
+  # and running "${geoclue2-with-demo-agent}/libexec/geoclue-2.0/demos/where-am-i"
+  # note that geoclue is dbus-activated, and auto-stops after 60s with no caller
+  services.geoclue2.enable = true;
+  services.geoclue2.appConfig.where-am-i = {
+    # this is the default "agent", shipped by geoclue package: allow it to use location
+    isAllowed = true;
+    isSystem = false;
+    # XXX: setting users != [] might be causing `where-am-i` to time out
+    users = [
+      # restrict to only one set of users. empty array (default) means "allow any user to access geolocation".
+      (builtins.toString config.users.users.colin.uid)
+    ];
+  };
+  systemd.services.geoclue.after = lib.mkForce [];  #< defaults to network-online, but not all my sources require network
+  users.users.geoclue.extraGroups = [
+    "dialout"  # TODO: figure out if dialout is required. that's for /dev/ttyUSB1, but geoclue probably doesn't read that?
+  ];
+
+  sane.programs.where-am-i.enableFor.user.colin = true;
+}

hosts/by-name/moby/kernel.nix

@@ -1,146 +1,271 @@
-{ lib, pkgs, ... }:
+{ pkgs, ... }:
 let
-  # use the last commit on the 5.18 branch (5.18.14)
-  # manjaro's changes between kernel patch versions tend to be minimal if any.
-  manjaroBase = "https://gitlab.manjaro.org/manjaro-arm/packages/core/linux/-/raw/25bd828cd47b1c6e09fcbcf394a649b89d2876dd";
-  manjaroPatch = name: sha256: {
-    inherit name;
-    patch = pkgs.fetchpatch {
-      inherit name;
-      url = "${manjaroBase}/${name}?inline=false";
-      inherit sha256;
-    };
-  };
+  dmesg = "${pkgs.util-linux}/bin/dmesg";
+  grep = "${pkgs.gnugrep}/bin/grep";
+  modprobe = "${pkgs.kmod}/bin/modprobe";
+  ensureHWReady = ''
+    # common boot failure:
+    # blank screen (no backlight even), with the following log:
+    # ```syslog
+    # sun8i-dw-hdmi 1ee0000.hdmi: Couldn't get the HDMI PHY
+    # ...
+    # sun4i-drm display-engine: Couldn't bind all pipelines components
+    # ...
+    # sun8i-dw-hdmi: probe of 1ee0000.hdmi failed with error -17
+    # ```
+    #
+    # in particular, that `probe ... failed` occurs *only* on failed boots
+    # (the other messages might sometimes occur even on successful runs?)
+    #
+    # reloading the sun8i hdmi driver usually gets the screen on, showing boot text.
+    # then restarting display-manager.service gets us to the login.
+    #
+    # NB: the above log is default level. though less specific, there's a `err` level message that also signals this:
+    # sun4i-drm display-engine: failed to bind 1ee0000.hdmi (ops sun8i_dw_hdmi_ops [sun8i_drm_hdmi]): -17
+    # NB: this is the most common, but not the only, failure mode for `display-manager`.
+    # another error seems characterized by these dmesg logs, in which reprobing sun8i_drm_hdmi does not fix:
+    # ```syslog
+    # sun6i-mipi-dsi 1ca0000.dsi: Couldn't get the MIPI D-PHY
+    # sun4i-drm display-engine: Couldn't bind all pipelines components
+    # sun6i-mipi-dsi 1ca0000.dsi: Couldn't register our component
+    # ```
 
-  # the idea for patching off Manjaro's kernel comes from jakewaksbaum:
-  # - https://git.sr.ht/~jakewaksbaum/pi/tree/af20aae5653545d6e67a459b59ee3e1ca8a680b0/item/kernel/default.nix
-  # - he later abandoned this, i think because he's using the Pinephone Pro which received mainline support.
-  manjaroPatches = [
-    (manjaroPatch
-      "1001-arm64-dts-allwinner-add-hdmi-sound-to-pine-devices.patch"
-      "sha256-DApd791A+AxB28Ven/MVAyuyVphdo8KQDx8O7oxVPnc="
-    )
-    # these patches below are critical to enable wifi (RTL8723CS)
-    # - the alternative is a wholly forked kernel by megi/megous:
-    #   - https://xnux.eu/howtos/build-pinephone-kernel.html#toc-how-to-build-megi-s-pinehpone-kernel
-    # - i don't know if these patches are based on megi's or original
-    (manjaroPatch
-      "2001-Bluetooth-Add-new-quirk-for-broken-local-ext-features.patch"
-      "sha256-CExhJuUWivegxPdnzKINEsKrMFx/m/1kOZFmlZ2SEOc="
-    )
-    (manjaroPatch
-      "2002-Bluetooth-btrtl-add-support-for-the-RTL8723CS.patch"
-      "sha256-dDdvOphTcP/Aog93HyH+L9m55laTgtjndPSE4/rnzUA="
-    )
-    (manjaroPatch
-      "2004-arm64-dts-allwinner-enable-bluetooth-pinetab-pinepho.patch"
-      "sha256-o43P3WzXyHK1PF+Kdter4asuyGAEKO6wf5ixcco2kCQ="
-    )
-    # XXX: this one has a Makefile, which hardcodes /sbin/depmod:
-    # - drivers/staging/rtl8723cs/Makefile
-    # - not sure if this is problematic?
-    (manjaroPatch
-      "2005-staging-add-rtl8723cs-driver.patch"
-      "sha256-6ywm3dQQ5JYl60CLKarxlSUukwi4QzqctCj3tVgzFbo="
-    )
-  ];
-
-  # pinephone uses the linux dtb at arch/arm64/boot/dts/allwinner/sun50i-a64-pinephone.dtsi
-  # - this includes sun50i-a64.dtsi
-  # - and sun50i-a64-cpu-opp.dtsi
-  # - no need to touch the allwinner-h6 stuff: that's the SBC pine product
-  # - i think it's safe to ignore sun9i stuff, but i don't know what it is
-  kernelConfig = with lib.kernel; {
-    # NB: nix adds the CONFIG_ prefix to each of these.
-    # if you add the prefix yourself nix will IGNORE YOUR CONFIG.
-    RTL8723CS = module;
-    BT_HCIUART_3WIRE = yes;
-    BT_HCIUART_RTL = yes;
-    RTL8XXXU_UNTESTED = yes;
-    BT_BNEP_MC_FILTER = yes;
-    BT_BNEP_PROTO_FILTER = yes;
-    BT_HS = yes;
-    BT_LE = yes;
-    # relevant configs inherited from nixos defaults (or above additions):
-    # CONFIG_BT=m
-    # CONFIG_BT_BREDR=y
-    # CONFIG_BT_RFCOMM=m
-    # CONFIG_BT_RFCOMM_TTY=y
-    # CONFIG_BT_BNEP=m
-    # CONFIG_BT_HIDP=m
-    # CONFIG_BT_RTL=m
-    # CONFIG_BT_HCIBTUSB=m
-    # CONFIG_BT_HCIBTUSB_BCM=y
-    # CONFIG_BT_HCIBTUSB_RTL=y
-    # CONFIG_BT_HCIUART=m
-    # CONFIG_BT_HCIUART_SERDEV=y
-    # CONFIG_BT_HCIUART_H4=y
-    # CONFIG_BT_HCIUART_LL=y
-    # CONFIG_RTL_CARDS=m
-    # CONFIG_RTLWIFI=m
-    # CONFIG_RTLWIFI_PCI=m
-    # CONFIG_RTLWIFI_USB=m
-    # CONFIG_RTLWIFI_DEBUG=y
-    # CONFIG_RTL8723_COMMON=m
-    # CONFIG_RTLBTCOEXIST=m
-    # CONFIG_RTL8XXXU=m
-    # CONFIG_RTLLIB=m
-    # consider adding (from mobile-nixos):
-    # maybe: CONFIG_BT_HCIUART_3WIRE=y
-    # maybe: CONFIG_BT_HCIUART_RTL=y
-    # maybe: CONFIG_RTL8XXXU_UNTESTED=y
-    # consider adding (from manjaro):
-    # CONFIG_BT_6LOWPAN=m  (not listed as option in nixos kernel)
-    # these are referenced in the rtl8723 source, but not known to config (and not in mobile-nixos config
-    # maybe: CONFIG_RTL_ODM_WLAN_DRIVER
-    # maybe: CONFIG_RTL_TRIBAND_SUPPORT
-    # maybe: CONFIG_SDIO_HCI
-    # maybe: CONFIG_USB_HCI
-  };
-
-  # create a kernelPatch which overrides nixos' defconfig with extra options
-  patchDefconfig = config: {
-    # defconfig options. this method comes from here:
-    # - https://discourse.nixos.org/t/the-correct-way-to-override-the-latest-kernel-config/533/9
-    name = "sane-moby-defconfig";
-    patch = null;
-    extraStructuredConfig = config;
-  };
+    if (${dmesg} --kernel --level err --color=never --notime | ${grep} -q 'sun4i-drm display-engine: failed to bind 1ee0000.hdmi')
+    then
+      echo "reprobing sun8i_drm_hdmi"
+      # if a command here fails it errors the whole service, so prefer to log instead
+      ${modprobe} -r sun8i_drm_hdmi || echo "failed to unload sun8i_drm_hdmi"
+      ${modprobe} sun8i_drm_hdmi || echo "failed to load sub8i_drm_hdmi"
+    fi
+  '';
 in
 {
-  # use Megi's kernel:
-  # even with the Manjaro patches, stock 5.18 has a few issues on Pinephone:
-  # - no battery charging
-  # - phone rotation sensor is off by 90 degrees
-  # - ambient light sensor causes screen brightness to be shakey
-  # - phosh greeter may not appear after wake from sleep
-  boot.kernelPackages = pkgs.cross.linuxPackagesFor pkgs.cross.linux-megous;
+  # kernel compatibility (2024/05/22: 03dab630)
+  # - linux-megous: boots to ssh, desktop
+  #   - camera apps: megapixels (no cameras found), snapshot (no cameras found)
+  # - linux-postmarketos: boots to ssh. desktop ONLY if "anx7688" is in the initrd.availableKernelModules.
+  #   - camera apps: megapixels (both rear and front cameras work), `cam -l` (finds only the rear camera), snapshot (no cameras found)
+  # - linux-megous.override { withMegiPinephoneConfig = true; }: NO SSH, NO SIGNS OF LIFE
+  # - linux-megous.override { withFullConfig = false; }: boots to ssh, no desktop
+  #
+  boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.linux-postmarketos.override {
+    withModemPower = true;
+  });
+  # boot.kernelPackages = pkgs.linuxPackagesFor pkgs.linux-megous;
+  # boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.linux-megous.override {
+  #   withFullConfig = false;
+  # });
+  # boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.linux-megous.override {
+  #   withMegiPinephoneConfig = true;  #< N.B.: does not boot as of 2024/05/22!
+  # });
+  # boot.kernelPackages = pkgs.linuxPackagesFor pkgs.linux-manjaro;
+  # boot.kernelPackages = pkgs.linuxPackagesFor pkgs.linux_latest;
 
-  boot.kernelPatches = [
-    (patchDefconfig (kernelConfig //
-      (with lib.kernel; {
-        # disabling the sun5i_eink driver avoids this compilation error:
-        # CC [M]  drivers/video/fbdev/sun5i-eink-neon.o
-        # aarch64-unknown-linux-gnu-gcc: error: unrecognized command line option '-mfloat-abi=softfp'
-        # aarch64-unknown-linux-gnu-gcc: error: unrecognized command line option '-mfpu=neon'
-        # make[3]: *** [../scripts/Makefile.build:289: drivers/video/fbdev/sun5i-eink-neon.o] Error 1
-        FB_SUN5I_EINK = no;
-        # used by the pinephone pro, but fails to compile with:
-        # ../drivers/media/i2c/ov8858.c:1834:27: error: implicit declaration of function 'compat_ptr'
-        VIDEO_OV8858 = no;
-      })
-    ))
+  # nixpkgs.hostPlatform.linux-kernel becomes stdenv.hostPlatform.linux-kernel
+  nixpkgs.hostPlatform.linux-kernel = {
+    # defaults:
+    name = "aarch64-multiplatform";
+    # baseConfig: defaults to "defconfig";
+    # baseConfig = "pinephone_defconfig";  #< N.B.: ignored by `pkgs.linux-megous`
+    DTB = true;  #< DTB: compile device tree blobs
+    # autoModules (default: true): for config options not manually specified, answer `m` to anything which supports it.
+    # - this effectively builds EVERY MODULE SUPPORTED.
+    autoModules = true;  #< N.B.: ignored by `pkgs.linux-megous`
+    # preferBuiltin (default: false; true for rpi): for config options which default to `Y` upstream, build them as `Y` (overriding `autoModules`)
+    # preferBuiltin = false;
+
+    # build a compressed kernel image: without this i run out of /boot space in < 10 generations
+    # target = "Image";  # <-- default
+    target = "Image.gz";  # <-- compress the kernel image
+    # target = "zImage";  # <-- confuses other parts of nixos :-(
+  };
+
+  # boot.initrd.kernelModules = [
+  #   "drm"  #< force drm to be plugged
+  # ];
+  boot.initrd.availableKernelModules = [
+    # see <repo:postmarketOS/pmaports:device/main/device-pine64-pinephone/modules-initfs>
+    # - they include sun6i_mipi_dsi sun4i_drm pwm_sun4i sun8i_mixer anx7688 gpio_vibra pinephone_keyboard
+    "anx7688" #< required for display initialization and functional cameras
+    # full list of modules active post-boot with the linux-megous kernel + autoModules=true:
+    # - `lsmod | sort | cut -d ' ' -f 1`
+    # "8723cs"
+    # "axp20x_adc"  #< NOT FOUND in megous-no-autoModules
+    # "axp20x_battery"
+    # "axp20x_pek"
+    # "axp20x_usb_power"
+    # "backlight"
+    # "blake2b_generic"
+    # "bluetooth"
+    # "bridge"
+    # "btbcm"
+    # "btqca"
+    # "btrfs"
+    # "btrtl"
+    # "cec"
+    # "cfg80211"
+    # "chacha_neon"
+    # "crc_ccitt"
+    # "crct10dif_ce"
+    # "crypto_engine"
+    # "display_connector"  #< NOT FOUND in pmos
+    # "drm"
+    # "drm_display_helper"
+    # "drm_dma_helper"
+    # "drm_kms_helper"
+    # "drm_shmem_helper"
+    # "dw_hdmi"
+    # "dw_hdmi_cec"  #< NOT FOUND in pmos
+    # "dw_hdmi_i2s_audio"
+    # "ecc"
+    # "ecdh_generic"
+    # "fuse"
+    # "gc2145"  #< NOT FOUND in megous-no-autoModules
+    # "goodix_ts"
+    # "gpio_vibra"  #< NOT FOUND in megous-no-autoModules
+    # "gpu_sched"
+    # "hci_uart"
+    # "i2c_gpio"
+    # "inv_mpu6050"  #< NOT FOUND in megous-no-autoModules
+    # "inv_mpu6050_i2c"  #< NOT FOUND in megous-no-autoModules
+    # "inv_sensors_timestamp"  #< NOT FOUND in megous-no-autoModules
+    # "ip6t_rpfilter"
+    # "ip6_udp_tunnel"
+    # "ip_set"
+    # "ip_set_hash_ipport"
+    # "ip_tables"
+    # "ipt_rpfilter"
+    # "joydev"
+    # "led_class_flash"  #< NOT FOUND in megous-no-autoModules
+    # "leds_sgm3140"  #< NOT FOUND in megous-no-autoModules
+    # "ledtrig_pattern"  #< NOT FOUND in megous-no-autoModules
+    # "libarc4"
+    # "libchacha"
+    # "libchacha20poly1305"
+    # "libcrc32c"
+    # "libcurve25519_generic"
+    # "lima"
+    # "llc"
+    # "mac80211"
+    # "macvlan"
+    # "mc"
+    # "modem_power"
+    # "mousedev"
+    # "nf_conntrack"
+    # "nf_defrag_ipv4"
+    # "nf_defrag_ipv6"
+    # "nf_log_syslog"
+    # "nf_nat"
+    # "nfnetlink"
+    # "nf_tables"
+    # "nft_chain_nat"
+    # "nft_compat"
+    # "nls_cp437"
+    # "nls_iso8859_1"
+    # "nvmem_reboot_mode"
+    # "ov5640"
+    # "panel_sitronix_st7703"
+    # "phy_sun6i_mipi_dphy"
+    # "pinctrl_axp209"  #< NOT FOUND in pmos
+    # "pinephone_keyboard"  #< NOT FOUND in megous-no-autoModules
+    # "poly1305_neon"
+    # "polyval_ce"
+    # "polyval_generic"
+    # "ppkb_manager"  #< NOT FOUND in megous-no-autoModules
+    # "pwm_bl"
+    # "pwm_sun4i"
+    # "qrtr"
+    # "raid6_pq"
+    # "rfkill"
+    # "rtw88_8703b"
+    # "rtw88_8723cs"
+    # "rtw88_8723x"
+    # "rtw88_core"
+    # "rtw88_sdio"
+    # "sch_fq_codel"
+    # "sm4"
+    # "snd_soc_bt_sco"
+    # "snd_soc_ec25"  #< NOT FOUND in megous-no-autoModules
+    # "snd_soc_hdmi_codec"
+    # "snd_soc_simple_amplifier"
+    # "snd_soc_simple_card"
+    # "snd_soc_simple_card_utils"
+    # "stk3310"  #< NOT FOUND in megous-no-autoModules
+    # "st_magn"
+    # "st_magn_i2c"
+    # "st_magn_spi"  #< NOT FOUND in pmos
+    # "stp"
+    # "st_sensors"
+    # "st_sensors_i2c"
+    # "st_sensors_spi"  #< NOT FOUND in pmos
+    # "sun4i_drm"
+    # "sun4i_i2s"
+    # "sun4i_lradc_keys"  #< NOT FOUND in megous-no-autoModules
+    # "sun4i_tcon"
+    # "sun50i_codec_analog"
+    # "sun6i_csi"
+    # "sun6i_dma"
+    # "sun6i_mipi_dsi"
+    # "sun8i_a33_mbus"  #< NOT FOUND in megous-no-autoModules
+    # "sun8i_adda_pr_regmap"
+    # "sun8i_ce"  #< NOT FOUND in pmos
+    # "sun8i_codec"  #< NOT FOUND in megous-no-autoModules
+    # "sun8i_di"  #< NOT FOUND in megous-no-autoModules
+    # "sun8i_drm_hdmi"
+    # "sun8i_mixer"
+    # "sun8i_rotate"  #< NOT FOUND in megous-no-autoModules
+    # "sun8i_tcon_top"
+    # "sun9i_hdmi_audio"  #< NOT FOUND in megous-no-autoModules
+    # "sunxi_wdt"  #< NOT FOUND in pmos
+    # "tap"
+    # "typec"  #< NOT FOUND in pmos
+    # "udp_tunnel"
+    # "uio"  #< NOT FOUND in pmos
+    # "uio_pdrv_genirq"
+    # "v4l2_async"
+    # "v4l2_cci" #< NOT FOUND in pmos
+    # "v4l2_flash_led_class"  #< NOT FOUND in megous-no-autoModules
+    # "v4l2_fwnode"
+    # "v4l2_mem2mem"
+    # "videobuf2_common"
+    # "videobuf2_dma_contig"
+    # "videobuf2_memops"
+    # "videobuf2_v4l2"
+    # "videodev"
+    # "wireguard"
+    # "xor"
+    # "x_tables"
+    # "xt_conntrack"
+    # "xt_LOG"
+    # "xt_nat"
+    # "xt_pkttype"
+    # "xt_set"
+    # "xt_tcpudp"
+    # "zram"
   ];
 
-  # alternatively, use nixos' kernel and add the stuff we want:
-  # # cross-compilation optimization:
-  # boot.kernelPackages =
-  #   let p = (import nixpkgs { localSystem = "x86_64-linux"; });
-  #   in p.pkgsCross.aarch64-multiplatform.linuxPackages_5_18;
-  # # non-cross:
-  # # boot.kernelPackages = pkgs.linuxPackages_5_18;
+  # disable proximity sensor.
+  # the filtering/calibration is bad that it causes the screen to go fully dark at times.
+  # boot.blacklistedKernelModules = [ "stk3310" ];
 
-  # boot.kernelPatches = manjaroPatches ++ [
-  #   (patchDefconfig kernelConfig)
-  # ];
+  boot.kernelParams = [
+    # without this some GUI apps fail: `DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory`
+    # this is because they can't allocate enough video ram.
+    # see related nixpkgs issue: <https://github.com/NixOS/nixpkgs/issues/260222>
+    # TODO(2023/12/03): remove once mesa 23.3.1 lands: <https://github.com/NixOS/nixpkgs/pull/265740>
+    #
+    # the default CMA seems to be 32M.
+    # i was running fine with 256MB from 2022/07-ish through 2022/12-ish, but then the phone quit reliably coming back from sleep (phosh): maybe a memory leak?
+    # bumped to 512M on 2023/01
+    # bumped to 1536M on 2024/05
+    # `cat /proc/meminfo` to see CmaTotal/CmaFree if interested in tuning this.
+    # kernel param mentioned here: <https://cateee.net/lkddb/web-lkddb/CMA_SIZE_PERCENTAGE.html>
+    # i think cma mem isn't exclusive -- it can be used as ordinary `malloc`, still. i heard someone suggest the OS default should just be 50% memory to CMA.
+    "cma=1536M"
+    # 2023/10/20: potential fix for the lima (GPU) timeout bugs:
+    # - <https://gitlab.com/postmarketOS/pmaports/-/issues/805#note_890467824>
+    "lima.sched_timeout_ms=2000"
+  ];
+
+  systemd.services.unl0kr.preStart = ensureHWReady;
 }

hosts/by-name/moby/polyfill.nix

@@ -0,0 +1,45 @@
+# this file configures preferences per program, without actually enabling any programs.
+# the goal is to separate the place where we decide *what* to use (i.e. `sane.programs.firefox.enable = true` -- at the toplevel)
+# from where we specific how that thing should behave *if* it's in use.
+#
+# NixOS backgrounds:
+# - <https://github.com/NixOS/nixos-artwork>
+#   - <https://github.com/NixOS/nixos-artwork/issues/50>  (colorful; unmerged)
+#   - <https://github.com/NixOS/nixos-artwork/pull/60/files>  (desktop-oriented; clean; unmerged)
+# - <https://itsfoss.com/content/images/2023/04/nixos-tutorials.png>
+
+{ lib, pkgs, sane-lib, ... }:
+{
+  sane.programs.firefox.config = {
+    # compromise impermanence for the sake of usability
+    persistCache = "private";
+    persistData = "private";
+
+    # i don't do crypto stuff on moby
+    addons.ether-metamask.enable = false;
+    # sidebery UX doesn't make sense on small screen
+    addons.sidebery.enable = false;
+  };
+  sane.programs.swaynotificationcenter.config = {
+    backlight = "backlight";  # /sys/class/backlight/*backlight*/brightness
+  };
+
+  sane.programs.alacritty.config.fontSize = 9;
+
+  sane.programs.sway.config = {
+    font = "pango:monospace 10";
+    mod = "Mod1";  # prefer Alt
+    workspace_layout = "tabbed";
+  };
+
+  sane.programs.waybar.config = {
+    fontSize = 14;
+    height = 26;
+    persistWorkspaces = [ "1" "2" "3" "4" "5" ];
+    modules.media = false;
+    modules.network = false;
+    modules.perf = false;
+    modules.windowTitle = false;
+    # TODO: show modem state
+  };
+}

hosts/by-name/rescue/default.nix

@@ -1,12 +1,17 @@
-{ config, pkgs, ... }:
+{ pkgs, ... }:
 {
   imports = [
     ./fs.nix
   ];
 
-  boot.loader.generic-extlinux-compatible.enable = true;
   boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
+  sane.persist.enable = false;  # what we mean here is that the image is immutable; `/` is still tmpfs.
+  sane.nixcache.enable = false;  # don't want to be calling out to dead machines that we're *trying* to rescue
+
+  # auto-login at shell
+  services.getty.autologinUser = "colin";
+  # users.users.colin.initialPassword = "colin";
 
   # docs: https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion
   system.stateVersion = "21.05";

hosts/by-name/rescue/fs.nix

@@ -1,7 +1,7 @@
 { ... }:
 
 {
-  fileSystems."/" = {
+  fileSystems."/nix" = {
     device = "/dev/disk/by-uuid/44445555-6666-7777-8888-999900001111";
     fsType = "ext4";
   };

hosts/by-name/servo/default.nix

@@ -4,20 +4,34 @@
   imports = [
     ./fs.nix
     ./net.nix
-    ./secrets.nix
     ./services
   ];
 
-  sane.packages.extraUserPkgs = with pkgs; [
+  sane.programs = {
     # for administering services
-    freshrss
-    matrix-synapse
-    signaldctl
+    freshrss.enableFor.user.colin = true;
+    matrix-synapse.enableFor.user.colin = true;
+    signaldctl.enableFor.user.colin = true;
+  };
+
+  sane.roles.build-machine.enable = true;
+  sane.programs.zsh.config.showDeadlines = false;  # ~/knowledge doesn't always exist
+  sane.programs.consoleUtils.suggestedPrograms = [
+    "consoleMediaUtils"  # notably, for go2tv / casting
+    "pcConsoleUtils"
+    "sane-scripts.stop-all-servo"
   ];
-  sane.persist.enable = true;
   sane.services.dyn-dns.enable = true;
+  sane.services.trust-dns.asSystemResolver = false;  # TODO: enable once it's all working well
   sane.services.wg-home.enable = true;
+  sane.services.wg-home.visibleToWan = true;
+  sane.services.wg-home.forwardToWan = true;
+  sane.services.wg-home.routeThroughServo = false;
   sane.services.wg-home.ip = config.sane.hosts.by-name."servo".wg-home.ip;
+  sane.ovpn.addrV4 = "172.23.174.114";
+  # sane.ovpn.addrV6 = "fd00:0000:1337:cafe:1111:1111:8df3:14b0";
+  sane.nixcache.remote-builders.desko = false;
+  sane.nixcache.remote-builders.servo = false;
   # sane.services.duplicity.enable = true;  # TODO: re-enable after HW upgrade
 
   # automatically log in at the virtual consoles.

hosts/by-name/servo/fs.nix

@@ -1,16 +1,70 @@
+# zfs docs:
+# - <https://nixos.wiki/wiki/ZFS>
+# - <repo:nixos/nixpkgs:nixos/modules/tasks/filesystems/zfs.nix>
+#
+# zfs check health: `zpool status`
+#
+# zfs pool creation (requires `boot.supportedFilesystems = [ "zfs" ];`
+# - 1. identify disk IDs: `ls -l /dev/disk/by-id`
+# - 2. pool these disks: `zpool create -f -m legacy pool raidz ata-ST4000VN008-2DR166_WDH0VB45 ata-ST4000VN008-2DR166_WDH17616 ata-ST4000VN008-2DR166_WDH0VC8Q ata-ST4000VN008-2DR166_WDH17680`
+#   - legacy documented: <https://superuser.com/questions/790036/what-is-a-zfs-legacy-mount-point>
+# - 3. enable acl support: `zfs set acltype=posixacl pool`
+#
+# import pools: `zpool import pool`
+# show zfs datasets: `zfs list` (will be empty if haven't imported)
+# show zfs properties (e.g. compression): `zfs get all pool`
+# set zfs properties: `zfs set compression=on pool`
 { ... }:
 
 {
-  sane.persist.root-on-tmpfs = true;
-  # we need a /tmp for building large nix things
-  fileSystems."/tmp" = {
-    device = "none";
-    fsType = "tmpfs";
-    options = [
-      "mode=777"
-      "defaults"
-    ];
+  # hostId: not used for anything except zfs guardrail?
+  #   [hex(ord(x)) for x in 'serv']
+  networking.hostId = "73657276";
+  boot.supportedFilesystems = [ "zfs" ];
+  # boot.zfs.enabled = true;
+  boot.zfs.forceImportRoot = false;
+  # scrub all zfs pools weekly:
+  services.zfs.autoScrub.enable = true;
+  boot.extraModprobeConfig = ''
+    ### zfs_arc_max tunable:
+    # ZFS likes to use half the ram for its own cache and let the kernel push everything else to swap.
+    # so, reduce its cache size
+    # see: <https://askubuntu.com/a/1290387>
+    # see: <https://serverfault.com/a/1119083>
+    # see: <https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-arc-max>
+    # for all tunables, see: `man 4 zfs`
+    # to update these parameters without rebooting:
+    # - `echo '4294967296' | sane-sudo-redirect /sys/module/zfs/parameters/zfs_arc_max`
+    ### zfs_bclone_enabled tunable
+    # this allows `cp --reflink=always FOO BAR` to work. i.e. shallow copies.
+    # it's unstable as of 2.2.3. led to *actual* corruption in 2.2.1, but hopefully better by now.
+    # - <https://github.com/openzfs/zfs/issues/405>
+    # note that `du -h` won't *always* show the reduced size for reflink'd files (?).
+    # `zpool get all | grep clone` seems to be the way to *actually* see how much data is being deduped
+    options zfs zfs_arc_max=4294967296 zfs_bclone_enabled=1
+  '';
+  # to be able to mount the pool like this, make sure to tell zfs to NOT manage it itself.
+  # otherwise local-fs.target will FAIL and you will be dropped into a rescue shell.
+  # - `zfs set mountpoint=legacy pool`
+  # if done correctly, the pool can be mounted before this `fileSystems` entry is created:
+  # - `sudo mount -t zfs pool /mnt/persist/pool`
+  fileSystems."/mnt/pool" = {
+    device = "pool";
+    fsType = "zfs";
+    options = [ "acl" ];  #< not sure if this `acl` flag is actually necessary. it mounts without it.
   };
+  # services.zfs.zed = ... # TODO: zfs can send me emails when disks fail
+  sane.programs.sysadminUtils.suggestedPrograms = [ "zfs" ];
+
+  sane.persist.stores."ext" = {
+    origin = "/mnt/pool/persist";
+    storeDescription = "external HDD storage";
+    defaultMethod = "bind";  #< TODO: change to "symlink"?
+  };
+
+  # increase /tmp space (defaults to 50% of RAM) for building large nix things.
+  # even the stock `nixpkgs.linux` consumes > 16 GB of tmp
+  fileSystems."/tmp".options = [ "size=32G" ];
 
   fileSystems."/nix" = {
     device = "/dev/disk/by-uuid/cc81cca0-3cc7-4d82-a00c-6243af3e7776";
@@ -27,7 +81,7 @@
   };
 
   # slow, external storage (for archiving, etc)
-  fileSystems."/mnt/persist/ext" = {
+  fileSystems."/mnt/usb-hdd" = {
     device = "/dev/disk/by-uuid/aa272cff-0fcc-498e-a4cb-0d95fb60631b";
     fsType = "btrfs";
     options = [
@@ -35,37 +89,47 @@
       "defaults"
     ];
   };
+  sane.fs."/mnt/usb-hdd".mount = {};
 
-  sane.persist.stores."ext" = {
-    origin = "/mnt/persist/ext/persist";
-    storeDescription = "external HDD storage";
-  };
-  sane.fs."/mnt/persist/ext".mount = {};
+  # FIRST TIME SETUP FOR MEDIA DIRECTORY:
+  # - set the group stick bit: `sudo find /var/media -type d -exec chmod g+s {} +`
+  #   - this ensures new files/dirs inherit the group of their parent dir (instead of the user who creates them)
+  # - ensure everything under /var/media is mounted with `-o acl`, to support acls
+  # - ensure all files are rwx by group: `setfacl --recursive --modify d:g::rwx /var/media`
+  #   - alternatively, `d:g:media:rwx` to grant `media` group even when file has a different owner, but that's a bit complex
+  sane.persist.sys.byStore.ext = [{
+    path = "/var/media";
+    user = "colin";
+    group = "media";
+    mode = "0775";
+  }];
+  sane.fs."/var/media/archive".dir = {};
+  # this is file.text instead of symlink.text so that it may be read over a remote mount (where consumers might not have any /nix/store/.../README.md path)
+  sane.fs."/var/media/archive/README.md".file.text = ''
+    this directory is for media i wish to remove from my library,
+    but keep for a short time in case i reverse my decision.
+    treat it like a system trash can.
+  '';
+  sane.fs."/var/media/Books".dir = {};
+  sane.fs."/var/media/Books/Audiobooks".dir = {};
+  sane.fs."/var/media/Books/Books".dir = {};
+  sane.fs."/var/media/Books/Visual".dir = {};
+  sane.fs."/var/media/collections".dir = {};
+  # sane.fs."/var/media/datasets".dir = {};
+  sane.fs."/var/media/freeleech".dir = {};
+  sane.fs."/var/media/Music".dir = {};
+  sane.fs."/var/media/Pictures".dir = {};
+  sane.fs."/var/media/Videos".dir = {};
+  sane.fs."/var/media/Videos/Film".dir = {};
+  sane.fs."/var/media/Videos/Shows".dir = {};
+  sane.fs."/var/media/Videos/Talks".dir = {};
 
-  sane.persist.sys.plaintext = [
-    # TODO: this is overly broad; only need media and share directories to be persisted
-    { user = "colin"; group = "users"; directory = "/var/lib/uninsane"; }
-  ];
-  # make sure large media is stored to the HDD
-  sane.persist.sys.ext = [
-    {
-      user = "colin";
-      group = "users";
-      mode = "0777";
-      directory = "/var/lib/uninsane/media/Videos";
-    }
-    {
-      user = "colin";
-      group = "users";
-      mode = "0777";
-      directory = "/var/lib/uninsane/media/freeleech";
-    }
-  ];
-
-  # in-memory compressed RAM (seems to be dynamically sized)
-  # zramSwap = {
-  #   enable = true;
-  # };
+  # this is file.text instead of symlink.text so that it may be read over a remote mount (where consumers might not have any /nix/store/.../README.md path)
+  sane.fs."/var/lib/uninsane/datasets/README.md".file.text = ''
+    this directory may seem redundant with ../media/datasets. it isn't.
+    this directory exists on SSD, allowing for speedy access to specific datasets when necessary.
+    the contents should be a subset of what's in ../media/datasets.
+  '';
 
   # btrfs doesn't easily support swapfiles
   # swapDevices = [

hosts/by-name/servo/net.nix

@@ -1,209 +1,217 @@
-{ config, pkgs, ... }:
+{ config, lib, pkgs, ... }:
 
+let
+  portOpts = with lib; types.submodule {
+    options = {
+      visibleTo.ovpn = mkOption {
+        type = types.bool;
+        default = false;
+      };
+    };
+  };
+in
 {
-  networking.domain = "uninsane.org";
-
-  # The global useDHCP flag is deprecated, therefore explicitly set to false here.
-  # Per-interface useDHCP will be mandatory in the future, so this generated config
-  # replicates the default behaviour.
-  networking.useDHCP = false;
-  networking.interfaces.eth0.useDHCP = true;
-  # XXX colin: probably don't need this. wlan0 won't be populated unless i touch a value in networking.interfaces.wlan0
-  networking.wireless.enable = false;
-
-  # networking.firewall.enable = false;
-  networking.firewall.enable = true;
-
-  # this is needed to forward packets from the VPN to the host
-  boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
-
-  # unless we add interface-specific settings for each VPN, we have to define nameservers globally.
-  # networking.nameservers = [
-  #   "1.1.1.1"
-  #   "9.9.9.9"
-  # ];
-
-  # use systemd's stub resolver.
-  # /etc/resolv.conf isn't sophisticated enough to use different servers per net namespace (or link).
-  # instead, running the stub resolver on a known address in the root ns lets us rewrite packets
-  # in the ovnps namespace to use the provider's DNS resolvers.
-  # a weakness is we can only query 1 NS at a time (unless we were to clone the packets?)
-  # there also seems to be some cache somewhere that's shared between the two namespaces.
-  # i think this is a libc thing. might need to leverage proper cgroups to _really_ kill it.
-  # - getent ahostsv4 www.google.com
-  # - try fix: <https://serverfault.com/questions/765989/connect-to-3rd-party-vpn-server-but-dont-use-it-as-the-default-route/766290#766290>
-  services.resolved.enable = true;
-  networking.nameservers = [
-    # use systemd-resolved resolver
-    # full resolver (which understands /etc/hosts) lives on 127.0.0.53
-    # stub resolver (just forwards upstream) lives on 127.0.0.54
-    "127.0.0.53"
-  ];
-
-  # nscd -- the Name Service Caching Daemon -- caches DNS query responses
-  # in a way that's unaware of my VPN routing, so routes are frequently poor against
-  # services which advertise different IPs based on geolocation.
-  # nscd claims to be usable without a cache, but in practice i can't get it to not cache!
-  # nsncd is the Name Service NON-Caching Daemon. it's a drop-in that doesn't cache;
-  # this is OK on the host -- because systemd-resolved caches. it's probably sub-optimal
-  # in the netns and we query upstream DNS more often than needed. hm.
-  # TODO: run a separate recursive resolver in each namespace.
-  services.nscd.enableNsncd = true;
-
-  # services.resolved.extraConfig = ''
-  #   # docs: `man resolved.conf`
-  #   # DNS servers to use via the `wg-ovpns` interface.
-  #   #   i hope that from the root ns, these aren't visible.
-  #   DNS=46.227.67.134%wg-ovpns 192.165.9.158%wg-ovpns
-  #   FallbackDNS=1.1.1.1 9.9.9.9
-  # '';
-
-  # OVPN CONFIG (https://www.ovpn.com):
-  # DOCS: https://nixos.wiki/wiki/WireGuard
-  # if you `systemctl restart wireguard-wg-ovpns`, make sure to also restart any other services in `NetworkNamespacePath = .../ovpns`.
-  # TODO: why not create the namespace as a seperate operation (nix config for that?)
-  networking.wireguard.enable = true;
-  networking.wireguard.interfaces.wg-ovpns = let
-    ip = "${pkgs.iproute2}/bin/ip";
-    in-ns = "${ip} netns exec ovpns";
-    iptables = "${pkgs.iptables}/bin/iptables";
-    veth-host-ip = "10.0.1.5";
-    veth-local-ip = "10.0.1.6";
-    vpn-ip = "185.157.162.178";
-    # DNS = 46.227.67.134, 192.165.9.158, 2a07:a880:4601:10f0:cd45::1, 2001:67c:750:1:cafe:cd45::1
-    vpn-dns = "46.227.67.134";
-  in {
-    privateKeyFile = config.sops.secrets.wg_ovpns_privkey.path;
-    # wg is active only in this namespace.
-    # run e.g. ip netns exec ovpns <some command like ping/curl/etc, it'll go through wg>
-    #   sudo ip netns exec ovpns ping www.google.com
-    interfaceNamespace = "ovpns";
-    ips = [
-      "185.157.162.178/32"
-    ];
-    peers = [
-      {
-        publicKey = "SkkEZDCBde22KTs/Hc7FWvDBfdOCQA4YtBEuC3n5KGs=";
-        endpoint = "185.157.162.10:9930";
-        # alternatively: use hostname, but that presents bootstrapping issues (e.g. if host net flakes)
-        # endpoint = "vpn36.prd.amsterdam.ovpn.com:9930";
-        allowedIPs = [ "0.0.0.0/0" ];
-        # nixOS says this is important for keeping NATs active
-        persistentKeepalive = 25;
-        # re-executes wg this often. docs hint that this might help wg notice DNS/hostname changes.
-        # so, maybe that helps if we specify endpoint as a domain name
-        # dynamicEndpointRefreshSeconds = 30;
-        # when refresh fails, try it again after this period instead.
-        # TODO: not avail until nixpkgs upgrade
-        # dynamicEndpointRefreshRestartSeconds = 5;
-      }
-    ];
-    preSetup = "" + ''
-      ${ip} netns add ovpns || echo "ovpns already exists"
-    '';
-    postShutdown = "" + ''
-      ${in-ns} ip link del ovpns-veth-b || echo "couldn't delete ovpns-veth-b"
-      ${ip} link del ovpns-veth-a || echo "couldn't delete ovpns-veth-a"
-      ${ip} netns delete ovpns || echo "couldn't delete ovpns"
-      # restore rules/routes
-      ${ip} rule del from ${veth-host-ip} lookup ovpns pref 50 || echo "couldn't delete init -> ovpns rule"
-      ${ip} route del default via ${veth-local-ip} dev ovpns-veth-a proto kernel src ${veth-host-ip} metric 1002 table ovpns || echo "couldn't delete init -> ovpns route"
-      ${ip} rule add from all lookup local pref 0
-      ${ip} rule del from all lookup local pref 100
-    '';
-    postSetup = "" + ''
-      # DOCS:
-      # - some of this approach is described here: <https://josephmuia.ca/2018-05-16-net-namespaces-veth-nat/>
-      # - iptables primer: <https://danielmiessler.com/study/iptables/>
-      # create veth pair
-      ${ip} link add ovpns-veth-a type veth peer name ovpns-veth-b
-      ${ip} addr add ${veth-host-ip}/24 dev ovpns-veth-a
-      ${ip} link set ovpns-veth-a up
-
-      # mv veth-b into the ovpns namespace
-      ${ip} link set ovpns-veth-b netns ovpns
-      ${in-ns} ip addr add ${veth-local-ip}/24 dev ovpns-veth-b
-      ${in-ns} ip link set ovpns-veth-b up
-
-      # make it so traffic originating from the host side of the veth
-      # is sent over the veth no matter its destination.
-      ${ip} rule add from ${veth-host-ip} lookup ovpns pref 50
-      # for traffic originating at the host veth to the WAN, use the veth as our gateway
-      # not sure if the metric 1002 matters.
-      ${ip} route add default via ${veth-local-ip} dev ovpns-veth-a proto kernel src ${veth-host-ip} metric 1002 table ovpns
-      # give the default route lower priority
-      ${ip} rule add from all lookup local pref 100
-      ${ip} rule del from all lookup local pref 0
-
-      # bridge HTTP traffic:
-      # any external port-80 request sent to the VPN addr will be forwarded to the rootns.
-      # this exists so LetsEncrypt can procure a cert for the MX over http.
-      # TODO: we could use _acme_challence.mx.uninsane.org CNAME to avoid this forwarding
-      # - <https://community.letsencrypt.org/t/where-does-letsencrypt-resolve-dns-from/37607/8>
-      ${in-ns} ${iptables} -A PREROUTING -t nat -p tcp --dport 80 -m iprange --dst-range ${vpn-ip} \
-        -j DNAT --to-destination ${veth-host-ip}:80
-
-      # we also bridge DNS traffic
-      ${in-ns} ${iptables} -A PREROUTING -t nat -p udp --dport 53 -m iprange --dst-range ${vpn-ip} \
-        -j DNAT --to-destination ${veth-host-ip}:53
-      ${in-ns} ${iptables} -A PREROUTING -t nat -p tcp --dport 53 -m iprange --dst-range ${vpn-ip} \
-        -j DNAT --to-destination ${veth-host-ip}:53
-
-      # in order to access DNS in this netns, we need to route it to the VPN's nameservers
-      # - alternatively, we could fix DNS servers like 1.1.1.1.
-      ${in-ns} ${iptables} -A OUTPUT -t nat -p udp --dport 53 -m iprange --dst-range 127.0.0.53 \
-        -j DNAT --to-destination ${vpn-dns}:53
-    '';
+  options = with lib; {
+    sane.ports.ports = mkOption {
+      # add the `visibleTo.ovpn` option
+      type = types.attrsOf portOpts;
+    };
   };
 
-  # create a new routing table that we can use to proxy traffic out of the root namespace
-  # through the ovpns namespace, and to the WAN via VPN.
-  networking.iproute2.rttablesExtraConfig = ''
-    5 ovpns
-  '';
-  networking.iproute2.enable = true;
+  config = {
+    networking.domain = "uninsane.org";
+
+    sane.ports.openFirewall = true;
+    sane.ports.openUpnp = true;
+
+    # unless we add interface-specific settings for each VPN, we have to define nameservers globally.
+    # networking.nameservers = [
+    #   "1.1.1.1"
+    #   "9.9.9.9"
+    # ];
+
+    # services.resolved.extraConfig = ''
+    #   # docs: `man resolved.conf`
+    #   # DNS servers to use via the `wg-ovpns` interface.
+    #   #   i hope that from the root ns, these aren't visible.
+    #   DNS=46.227.67.134%wg-ovpns 192.165.9.158%wg-ovpns
+    #   FallbackDNS=1.1.1.1 9.9.9.9
+    # '';
+
+    # tun-sea config
+    sane.dns.zones."uninsane.org".inet.A."doof.tunnel" = "205.201.63.12";
+    sane.dns.zones."uninsane.org".inet.AAAA."doof.tunnel" = "2602:fce8:106::51";
+    networking.wireguard.interfaces.wg-doof = let
+      ip = "${pkgs.iproute2}/bin/ip";
+    in {
+      privateKeyFile = config.sops.secrets.wg_doof_privkey.path;
+      # wg is active only in this namespace.
+      # run e.g. ip netns exec doof <some command like ping/curl/etc, it'll go through wg>
+      #   sudo ip netns exec doof ping www.google.com
+      interfaceNamespace = "doof";
+      ips = [
+        "205.201.63.12/32"
+        "2602:fce8:106::51/128"
+      ];
+      peers = [
+        {
+          publicKey = "nuESyYEJ3YU0hTZZgAd7iHBz1ytWBVM5PjEL1VEoTkU=";
+          # TODO: configure DNS within the doof ns and use tun-sea.doof.net endpoint
+          # endpoint = "tun-sea.doof.net:53263";
+          endpoint = "205.201.63.44:53263";
+          allowedIPs = [ "0.0.0.0/0" "::/0" ];
+          persistentKeepalive = 25; #< keep the NAT alive
+        }
+      ];
+      preSetup = ''
+        ${ip} netns add doof || (test -e /run/netns/doof && echo "doof already exists")
+      '';
+      postShutdown = ''
+        ${ip} netns delete doof || echo "couldn't delete doof"
+      '';
+    };
+
+    # OVPN CONFIG (https://www.ovpn.com):
+    # DOCS: https://nixos.wiki/wiki/WireGuard
+    # if you `systemctl restart wireguard-wg-ovpns`, make sure to also restart any other services in `NetworkNamespacePath = .../ovpns`.
+    # TODO: why not create the namespace as a seperate operation (nix config for that?)
+    networking.wireguard.enable = true;
+    networking.wireguard.interfaces.wg-ovpns = let
+      ip = "${pkgs.iproute2}/bin/ip";
+      in-ns = "${ip} netns exec ovpns";
+      iptables = "${pkgs.iptables}/bin/iptables";
+      veth-host-ip = "10.0.1.5";
+      veth-local-ip = "10.0.1.6";
+      vpn-ip = "185.157.162.178";
+      # DNS = 46.227.67.134, 192.165.9.158, 2a07:a880:4601:10f0:cd45::1, 2001:67c:750:1:cafe:cd45::1
+      vpn-dns = "46.227.67.134";
+      bridgePort = port: proto: ''
+        ${in-ns} ${iptables} -A PREROUTING -t nat -p ${proto} --dport ${port} -m iprange --dst-range ${vpn-ip} \
+          -j DNAT --to-destination ${veth-host-ip}
+      '';
+      bridgeStatements = lib.foldlAttrs
+        (acc: port: portCfg: acc ++ (builtins.map (bridgePort port) portCfg.protocol))
+        []
+        config.sane.ports.ports;
+    in {
+      privateKeyFile = config.sops.secrets.wg_ovpns_privkey.path;
+      # wg is active only in this namespace.
+      # run e.g. ip netns exec ovpns <some command like ping/curl/etc, it'll go through wg>
+      #   sudo ip netns exec ovpns ping www.google.com
+      interfaceNamespace = "ovpns";
+      ips = [
+        "185.157.162.178/32"
+      ];
+      peers = [
+        {
+          publicKey = "SkkEZDCBde22KTs/Hc7FWvDBfdOCQA4YtBEuC3n5KGs=";
+          endpoint = "185.157.162.10:9930";
+          # alternatively: use hostname, but that presents bootstrapping issues (e.g. if host net flakes)
+          # endpoint = "vpn36.prd.amsterdam.ovpn.com:9930";
+          allowedIPs = [ "0.0.0.0/0" ];
+          # nixOS says this is important for keeping NATs active
+          persistentKeepalive = 25;
+          # re-executes wg this often. docs hint that this might help wg notice DNS/hostname changes.
+          # so, maybe that helps if we specify endpoint as a domain name
+          # dynamicEndpointRefreshSeconds = 30;
+          # when refresh fails, try it again after this period instead.
+          # TODO: not avail until nixpkgs upgrade
+          # dynamicEndpointRefreshRestartSeconds = 5;
+        }
+      ];
+      preSetup = ''
+        ${ip} netns add ovpns || (test -e /run/netns/ovpns && echo "ovpns already exists")
+      '';
+      postShutdown = ''
+        ${in-ns} ip link del ovpns-veth-b || echo "couldn't delete ovpns-veth-b"
+        ${ip} link del ovpns-veth-a || echo "couldn't delete ovpns-veth-a"
+        ${ip} netns delete ovpns || echo "couldn't delete ovpns"
+        # restore rules/routes
+        ${ip} rule del from ${veth-host-ip} lookup ovpns pref 50 || echo "couldn't delete init -> ovpns rule"
+        ${ip} route del default via ${veth-local-ip} dev ovpns-veth-a proto kernel src ${veth-host-ip} metric 1002 table ovpns || echo "couldn't delete init -> ovpns route"
+        ${ip} rule add from all lookup local pref 0
+        ${ip} rule del from all lookup local pref 100
+      '';
+      postSetup = ''
+        # DOCS:
+        # - some of this approach is described here: <https://josephmuia.ca/2018-05-16-net-namespaces-veth-nat/>
+        # - iptables primer: <https://danielmiessler.com/study/iptables/>
+        # create veth pair
+        ${ip} link add ovpns-veth-a type veth peer name ovpns-veth-b
+        ${ip} addr add ${veth-host-ip}/24 dev ovpns-veth-a
+        ${ip} link set ovpns-veth-a up
+
+        # mv veth-b into the ovpns namespace
+        ${ip} link set ovpns-veth-b netns ovpns
+        ${in-ns} ip addr add ${veth-local-ip}/24 dev ovpns-veth-b
+        ${in-ns} ip link set ovpns-veth-b up
+
+        # make it so traffic originating from the host side of the veth
+        # is sent over the veth no matter its destination.
+        ${ip} rule add from ${veth-host-ip} lookup ovpns pref 50
+        # for traffic originating at the host veth to the WAN, use the veth as our gateway
+        # not sure if the metric 1002 matters.
+        ${ip} route add default via ${veth-local-ip} dev ovpns-veth-a proto kernel src ${veth-host-ip} metric 1002 table ovpns
+        # give the default route lower priority
+        ${ip} rule add from all lookup local pref 100
+        ${ip} rule del from all lookup local pref 0
+
+        # in order to access DNS in this netns, we need to route it to the VPN's nameservers
+        # - alternatively, we could fix DNS servers like 1.1.1.1.
+        ${in-ns} ${iptables} -A OUTPUT -t nat -p udp --dport 53 -m iprange --dst-range 127.0.0.53 \
+          -j DNAT --to-destination ${vpn-dns}:53
+      '' + (lib.concatStringsSep "\n" bridgeStatements);
+    };
+
+    # create a new routing table that we can use to proxy traffic out of the root namespace
+    # through the ovpns namespace, and to the WAN via VPN.
+    networking.iproute2.rttablesExtraConfig = ''
+      5 ovpns
+    '';
+    networking.iproute2.enable = true;
 
 
-  # HURRICANE ELECTRIC CONFIG:
-  # networking.sits = {
-  #   hurricane = {
-  #     remote = "216.218.226.238";
-  #     local = "192.168.0.5";
-  #     # local = "10.0.0.5";
-  #     # remote = "10.0.0.1";
-  #     # local = "10.0.0.22";
-  #     dev = "eth0";
-  #     ttl = 255;
-  #   };
-  # };
-  # networking.interfaces."hurricane".ipv6 = {
-  #   addresses = [
-  #     # mx.uninsane.org (publically routed /64)
-  #     {
-  #       address = "2001:470:b:465::1";
-  #       prefixLength = 128;
-  #     }
-  #     # client addr
-  #     # {
-  #     #   address = "2001:470:a:466::2";
-  #     #   prefixLength = 64;
-  #     # }
-  #   ];
-  #   routes = [
-  #     {
-  #       address = "::";
-  #       prefixLength = 0;
-  #       # via = "2001:470:a:466::1";
-  #     }
-  #   ];
-  # };
+    # HURRICANE ELECTRIC CONFIG:
+    # networking.sits = {
+    #   hurricane = {
+    #     remote = "216.218.226.238";
+    #     local = "192.168.0.5";
+    #     # local = "10.0.0.5";
+    #     # remote = "10.0.0.1";
+    #     # local = "10.0.0.22";
+    #     dev = "eth0";
+    #     ttl = 255;
+    #   };
+    # };
+    # networking.interfaces."hurricane".ipv6 = {
+    #   addresses = [
+    #     # mx.uninsane.org (publically routed /64)
+    #     {
+    #       address = "2001:470:b:465::1";
+    #       prefixLength = 128;
+    #     }
+    #     # client addr
+    #     # {
+    #     #   address = "2001:470:a:466::2";
+    #     #   prefixLength = 64;
+    #     # }
+    #   ];
+    #   routes = [
+    #     {
+    #       address = "::";
+    #       prefixLength = 0;
+    #       # via = "2001:470:a:466::1";
+    #     }
+    #   ];
+    # };
 
-  # # after configuration, we want the hurricane device to look like this:
-  # # hurricane: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1480
-  # #            inet6 2001:470:a:450::2  prefixlen 64  scopeid 0x0<global>
-  # #            inet6 fe80::c0a8:16  prefixlen 64  scopeid 0x20<link>
-  # #            sit  txqueuelen 1000  (IPv6-in-IPv4)
-  # # test with:
-  # #   curl --interface hurricane http://[2607:f8b0:400a:80b::2004]
-  # #   ping 2607:f8b0:400a:80b::2004
+    # # after configuration, we want the hurricane device to look like this:
+    # # hurricane: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1480
+    # #            inet6 2001:470:a:450::2  prefixlen 64  scopeid 0x0<global>
+    # #            inet6 fe80::c0a8:16  prefixlen 64  scopeid 0x20<link>
+    # #            sit  txqueuelen 1000  (IPv6-in-IPv4)
+    # # test with:
+    # #   curl --interface hurricane http://[2607:f8b0:400a:80b::2004]
+    # #   ping 2607:f8b0:400a:80b::2004
+  };
 }

hosts/by-name/servo/secrets.nix

@@ -1,41 +0,0 @@
-{ ... }:
-
-{
-  sops.secrets."ddns_afraid" = {
-    sopsFile = ../../../secrets/servo.yaml;
-  };
-  sops.secrets."ddns_he" = {
-    sopsFile = ../../../secrets/servo.yaml;
-  };
-
-  sops.secrets."dovecot_passwd" = {
-    sopsFile = ../../../secrets/servo.yaml;
-  };
-
-  sops.secrets."duplicity_passphrase" = {
-    sopsFile = ../../../secrets/servo.yaml;
-  };
-
-  sops.secrets."freshrss_passwd" = {
-    sopsFile = ../../../secrets/servo.yaml;
-  };
-
-  sops.secrets."matrix_synapse_secrets" = {
-    sopsFile = ../../../secrets/servo.yaml;
-  };
-  sops.secrets."mautrix_signal_env" = {
-    sopsFile = ../../../secrets/servo/mautrix_signal_env.bin;
-  };
-
-  sops.secrets."mediawiki_pw" = {
-    sopsFile = ../../../secrets/servo.yaml;
-  };
-
-  sops.secrets."pleroma_secrets" = {
-    sopsFile = ../../../secrets/servo.yaml;
-  };
-
-  sops.secrets."wg_ovpns_privkey" = {
-    sopsFile = ../../../secrets/servo.yaml;
-  };
-}

hosts/by-name/servo/services/calibre.nix

@@ -0,0 +1,34 @@
+{ config, lib, ... }:
+
+let
+  cweb-cfg = config.services.calibre-web;
+  inherit (cweb-cfg) user group;
+  inherit (cweb-cfg.listen) ip port;
+  svc-dir = "/var/lib/${cweb-cfg.dataDir}";
+in
+# XXX: disabled because of runtime errors like:
+# > File "/nix/store/c7jqvx980nlg9xhxi065cba61r2ain9y-calibre-web-0.6.19/lib/python3.10/site-packages/calibreweb/cps/db.py", line 926, in speaking_language
+# > languages = self.session.query(Languages) \
+# > AttributeError: 'NoneType' object has no attribute 'query'
+lib.mkIf false
+{
+  sane.persist.sys.byStore.plaintext = [
+    { inherit user group; mode = "0700"; path = svc-dir; method = "bind"; }
+  ];
+
+  services.calibre-web.enable = true;
+  services.calibre-web.listen.ip = "127.0.0.1";
+  # XXX: externally populate `${svc-dir}/metadata.db` (once) from
+  #   <https://github.com/janeczku/calibre-web/blob/master/library/metadata.db>
+  # i don't know why you have to do this??
+  # services.calibre-web.options.calibreLibrary = svc-dir;
+
+  services.nginx.virtualHosts."calibre.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+    locations."/" = {
+      proxyPass = "http://${ip}:${builtins.toString port}";
+    };
+  };
+  sane.dns.zones."uninsane.org".inet.CNAME."calibre" = "native";
+}

hosts/by-name/servo/services/coturn.nix

@@ -0,0 +1,144 @@
+# TURN/STUN NAT traversal service
+# commonly used to establish realtime calls with prosody, or possibly matrix/synapse
+#
+# - <https://github.com/coturn/coturn/>
+# - `man turnserver`
+# - config docs: <https://github.com/coturn/coturn/blob/master/examples/etc/turnserver.conf>
+#
+# N.B. during operation it's NORMAL to see "error 401".
+# during session creation:
+# - client sends Allocate request
+# - server replies error 401, providing a realm and nonce
+# - client uses realm + nonce + shared secret to construct an auth key & call Allocate again
+# - server replies Allocate Success Response
+# - source: <https://stackoverflow.com/a/66643135>
+#
+# N.B. this safest implementation routes all traffic THROUGH A VPN
+# - that adds a lot of latency, but in practice turns out to be inconsequential.
+#   i guess ICE allows clients to prefer the other party's lower-latency server, in practice?
+# - still, this is the "safe" implementation because STUN works with IP addresses instead of domain names:
+#   1. client A queries the STUN server to determine its own IP address/port.
+#   2. client A tells client B which IP address/port client A is visible on.
+#   3. client B contacts that IP address/port
+#   this only works so long as the IP address/port which STUN server sees client A on is publicly routable.
+#   that is NOT the case when the STUN server and client A are on the same LAN
+#   even if client A contacts the STUN server via its WAN address with port reflection enabled.
+#   hence, there's no obvious way to put the STUN server on the same LAN as either client and expect the rest to work.
+# - there an old version which *half worked*, which is:
+#   - run the turn server in the root namespace.
+#   - bind the turn server to the veth connecting it to the VPN namespace (so it sends outgoing traffic to the right place).
+#   - NAT the turn port range from VPN into root namespace (so it receives incomming traffic).
+#   - this approach would fail the prosody conversations.im check, but i didn't notice *obvious* call routing errors.
+#
+# debugging:
+# - log messages like 'usage: realm=<turn.uninsane.org>, username=<1715915193>, rp=14, rb=1516, sp=8, sb=684'
+#   - rp = received packets
+#   - rb = received bytes
+#   - sp = sent packets
+#   - sb = sent bytes
+{ lib, ... }:
+let
+  # TURN port range (inclusive).
+  # default coturn behavior is to use the upper quarter of all ports. i.e. 49152 - 65535.
+  # i believe TURN allocations expire after either 5 or 10 minutes of inactivity.
+  turnPortLow = 49152; # 49152 = 0xc000
+  turnPortHigh = turnPortLow + 256;
+  turnPortRange = lib.range turnPortLow turnPortHigh;
+in
+{
+  # the port definitions are only needed if running in the root net namespace
+  # sane.ports.ports = lib.mkMerge ([
+  #   {
+  #     "3478" = {
+  #       # this is the "control" port.
+  #       # i.e. no client data is forwarded through it, but it's where clients request tunnels.
+  #       protocol = [ "tcp" "udp" ];
+  #       # visibleTo.lan = true;
+  #       # visibleTo.wan = true;
+  #       visibleTo.ovpn = true;  # forward traffic from the VPN to the root NS
+  #       description = "colin-stun-turn";
+  #     };
+  #     "5349" = {
+  #       # the other port 3478 also supports TLS/DTLS, but presumably clients wanting TLS will default 5349
+  #       protocol = [ "tcp" ];
+  #       # visibleTo.lan = true;
+  #       # visibleTo.wan = true;
+  #       visibleTo.ovpn = true;
+  #       description = "colin-stun-turn-over-tls";
+  #     };
+  #   }
+  # ] ++ (builtins.map
+  #   (port: {
+  #     "${builtins.toString port}" = let
+  #       count = port - turnPortLow + 1;
+  #       numPorts = turnPortHigh - turnPortLow + 1;
+  #     in {
+  #       protocol = [ "tcp" "udp" ];
+  #       # visibleTo.lan = true;
+  #       # visibleTo.wan = true;
+  #       visibleTo.ovpn = true;
+  #       description = "colin-turn-${builtins.toString count}-of-${builtins.toString numPorts}";
+  #     };
+  #   })
+  #   turnPortRange
+  # ));
+
+  services.nginx.virtualHosts."turn.uninsane.org" = {
+    # allow ACME to procure a cert via nginx for this domain
+    enableACME = true;
+  };
+  sane.dns.zones."uninsane.org".inet = {
+    # CNAME."turn" = "servo.wan";
+    # CNAME."turn" = "ovpns";
+    # CNAME."turn" = "native";
+    # XXX: SRV records have to point to something with a A/AAAA record; no CNAMEs
+    A."turn" = "%AOVPNS%";
+    # A."turn" = "%AWAN%";
+
+    SRV."_stun._udp" =                        "5 50 3478 turn";
+    SRV."_stun._tcp" =                        "5 50 3478 turn";
+    SRV."_stuns._tcp" =                       "5 50 5349 turn";
+    SRV."_turn._udp" =                        "5 50 3478 turn";
+    SRV."_turn._tcp" =                        "5 50 3478 turn";
+    SRV."_turns._tcp" =                       "5 50 5349 turn";
+  };
+
+  sane.derived-secrets."/var/lib/coturn/shared_secret.bin" = {
+    encoding = "base64";
+    # TODO: make this not globally readable
+    acl.mode = "0644";
+  };
+  sane.fs."/var/lib/coturn/shared_secret.bin".wantedBeforeBy = [ "coturn.service" ];
+
+  # provide access to certs
+  users.users.turnserver.extraGroups = [ "nginx" ];
+
+  services.coturn.enable = true;
+  services.coturn.realm = "turn.uninsane.org";
+  services.coturn.cert = "/var/lib/acme/turn.uninsane.org/fullchain.pem";
+  services.coturn.pkey = "/var/lib/acme/turn.uninsane.org/key.pem";
+
+  #v disable to allow unauthenticated access (or set `services.coturn.no-auth = true`)
+  services.coturn.use-auth-secret = true;
+  services.coturn.static-auth-secret-file = "/var/lib/coturn/shared_secret.bin";
+  services.coturn.lt-cred-mech = true; #< XXX: use-auth-secret overrides lt-cred-mech
+
+  services.coturn.min-port = turnPortLow;
+  services.coturn.max-port = turnPortHigh;
+  # services.coturn.secure-stun = true;
+  services.coturn.extraConfig = lib.concatStringsSep "\n" [
+    "verbose"
+    # "Verbose"  #< even MORE verbosity than "verbose"  (it's TOO MUCH verbosity really)
+    "no-multicast-peers"  # disables sending to IPv4 broadcast addresses (e.g. 224.0.0.0/3)
+    # "listening-ip=10.0.1.5" "external-ip=185.157.162.178"  #< 2024/04/25: works, if running in root namespace
+    "listening-ip=185.157.162.178" "external-ip=185.157.162.178"
+
+    # old attempts:
+    # "external-ip=185.157.162.178/10.0.1.5"
+    # "listening-ip=10.78.79.51"  # can be specified multiple times; omit for *
+    # "external-ip=97.113.128.229/10.78.79.51"
+    # "external-ip=97.113.128.229"
+    # "mobility"  # "mobility with ICE (MICE) specs support" (?)
+  ];
+  systemd.services.coturn.serviceConfig.NetworkNamespacePath = "/run/netns/ovpns";
+}

hosts/by-name/servo/services/cryptocurrencies/bitcoin.nix

@@ -0,0 +1,83 @@
+# as of 2023/12/02: complete blockchain is 530 GiB (on-disk size may be larger)
+#
+# ports:
+# - 8333: for node-to-node communications
+# - 8332: rpc (client-to-node)
+#
+# rpc setup:
+# - generate a password
+#   - use: <https://github.com/bitcoin/bitcoin/blob/master/share/rpcauth/rpcauth.py>
+#     (rpcauth.py is not included in the `'.#bitcoin'` package result)
+#   - `wget https://raw.githubusercontent.com/bitcoin/bitcoin/master/share/rpcauth/rpcauth.py`
+#   - `python ./rpcauth.py colin`
+#   - copy the hash here. it's SHA-256, so safe to be public.
+#   - add "rpcuser=colin" and "rpcpassword=<output>" to secrets/servo/bitcoin.conf  (i.e. ~/.bitcoin/bitcoin.conf)
+#     - bitcoin.conf docs: <https://github.com/bitcoin/bitcoin/blob/master/doc/bitcoin-conf.md>
+# - validate with `bitcoin-cli -netinfo`
+{ config, lib, pkgs, sane-lib, ... }:
+let
+  # wrapper to run bitcoind with the tor onion address as externalip (computed at runtime)
+  _bitcoindWithExternalIp = with pkgs; writeShellScriptBin "bitcoind" ''
+    externalip="$(cat /var/lib/tor/onion/bitcoind/hostname)"
+    exec ${bitcoind}/bin/bitcoind "-externalip=$externalip" "$@"
+  '';
+  # the package i provide to services.bitcoind ends up on system PATH, and used by other tools like clightning.
+  # therefore, even though services.bitcoind only needs `bitcoind` binary, provide all the other bitcoin-related binaries (notably `bitcoin-cli`) as well:
+  bitcoindWithExternalIp = with pkgs; symlinkJoin {
+    name = "bitcoind-with-external-ip";
+    paths = [ _bitcoindWithExternalIp bitcoind ];
+  };
+in
+{
+  sane.persist.sys.byStore.ext = [
+    { user = "bitcoind-mainnet"; group = "bitcoind-mainnet"; path = "/var/lib/bitcoind-mainnet"; method = "bind"; }
+  ];
+
+  # sane.ports.ports."8333" = {
+  #   # this allows other nodes and clients to download blocks from me.
+  #   protocol = [ "tcp" ];
+  #   visibleTo.wan = true;
+  #   description = "colin-bitcoin";
+  # };
+
+  services.tor.relay.onionServices.bitcoind = {
+    version = 3;
+    map = [{
+      # by default tor will route public tor port P to 127.0.0.1:P.
+      # so if this port is the same as clightning would natively use, then no further config is needed here.
+      # see: <https://2019.www.torproject.org/docs/tor-manual.html.en#HiddenServicePort>
+      port = 8333;
+      # target.port; target.addr;  #< set if tor port != clightning port
+    }];
+    # allow "tor" group (i.e. bitcoind-mainnet) to read /var/lib/tor/onion/bitcoind/hostname
+    settings.HiddenServiceDirGroupReadable = true;
+  };
+
+  services.bitcoind.mainnet = {
+    enable = true;
+    package = bitcoindWithExternalIp;
+    rpc.users.colin = {
+      # see docs at top of file for how to generate this
+      passwordHMAC = "30002c05d82daa210550e17a182db3f3$6071444151281e1aa8a2729f75e3e2d224e9d7cac3974810dab60e7c28ffaae4";
+    };
+    extraConfig = ''
+      # don't load the wallet, and disable wallet RPC calls
+      disablewallet=1
+      # proxy all outbound traffic through Tor
+      proxy=127.0.0.1:9050
+    '';
+  };
+
+  users.users.bitcoind-mainnet.extraGroups = [ "tor" ];
+
+  systemd.services.bitcoind-mainnet.serviceConfig.RestartSec = "30s";  #< default is 0
+
+  sane.users.colin.fs.".bitcoin/bitcoin.conf" = sane-lib.fs.wantedSymlinkTo config.sops.secrets."bitcoin.conf".path;
+  sops.secrets."bitcoin.conf" = {
+    mode = "0600";
+    owner = "colin";
+    group = "users";
+  };
+
+  sane.programs.bitcoind.enableFor.user.colin = true;  # for debugging/administration: `bitcoin-cli`
+}

hosts/by-name/servo/services/cryptocurrencies/clightning-sane/clightning-sane

@@ -0,0 +1,782 @@
+#!/usr/bin/env nix-shell
+#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.pyln-client ])"
+
+"""
+clightning-sane: helper to perform common Lightning node admin operations:
+- view channel balances
+- rebalance channels
+
+COMMON OPERATIONS:
+- view channel balances: `clightning-sane status`
+- rebalance channels to improve routability (without paying any fees): `clightning-sane autobalance`
+
+FULL OPERATION:
+- `clightning-sane status --full`
+  - `P$`: represents how many msats i've captured in fees from this channel.
+  - `COST`: rough measure of how much it's "costing" me to let my channel partner hold funds on his side of the channel.
+            this is based on the notion that i only capture fees from outbound transactions, and so the channel partner holding all liquidity means i can't capture fees on that liquidity.
+"""
+
+# pyln-client docs: <https://github.com/ElementsProject/lightning/tree/master/contrib/pyln-client>
+# terminology:
+# - "scid": "Short Channel ID", e.g. 123456x7890x0
+#   from this id, we can locate the actual channel, its peers, and its parameters
+
+import argparse
+import logging
+import math
+import sys
+import time
+
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import dataclass
+from enum import Enum
+
+from pyln.client import LightningRpc, Millisatoshi, RpcError
+
+logger = logging.getLogger(__name__)
+
+RPC_FILE = "/var/lib/clightning/bitcoin/lightning-rpc"
+# CLTV (HLTC delta) of the final hop
+# set this too low and you might get inadvertent channel closures (?)
+CLTV = 18
+
+# for every sequentally failed transaction, delay this much before trying again.
+# note that the initial route building process can involve 10-20 "transient" failures, as it discovers dead channels.
+TX_FAIL_BACKOFF = 0.8
+MAX_SEQUENTIAL_JOB_FAILURES = 200
+
+class LoopError(Enum):
+    """ error when trying to loop sats, or when unable to calculate a route for the loop """
+    TRANSIENT = "TRANSIENT"  # try again, we'll maybe find a different route
+    NO_ROUTE = "NO_ROUTE"
+
+class RouteError(Enum):
+    """ error when calculated a route """
+    HAS_BASE_FEE = "HAS_BASE_FEE"
+    NO_ROUTE = "NO_ROUTE"
+
+class Metrics:
+    looped_msat: int = 0
+    sendpay_fail: int = 0
+    sendpay_succeed: int = 0
+    own_bad_channel: int = 0
+    no_route: int = 0
+    in_ch_unsatisfiable: int = 0
+
+    def __repr__(self) -> str:
+        return f"looped:{self.looped_msat}, tx:{self.sendpay_succeed}, tx_fail:{self.sendpay_fail}, own_bad_ch:{self.own_bad_channel}, no_route:{self.no_route}, in_ch_restricted:{self.in_ch_unsatisfiable}"
+
+@dataclass
+class TxBounds:
+    max_msat: int
+    min_msat: int = 0
+
+    def __repr__(self) -> str:
+        return f"TxBounds({self.min_msat} <= msat <= {self.max_msat})"
+
+    def is_satisfiable(self) -> bool:
+        return self.min_msat <= self.max_msat
+
+    def raise_max_to_be_satisfiable(self) -> "Self":
+        if self.max_msat < self.min_msat:
+            logger.debug(f"raising max_msat to be consistent: {self.max_msat} -> {self.min_msat}")
+            return TxBounds(self.min_msat, self.min_msat)
+        return TxBounds(min_msat=self.min_msat, max_msat=self.max_msat)
+
+    def intersect(self, other: "TxBounds") -> "Self":
+        return TxBounds(
+            min_msat=max(self.min_msat, other.min_msat),
+            max_msat=min(self.max_msat, other.max_msat),
+        )
+
+    def restrict_to_htlc(self, ch: "LocalChannel", why: str = "") -> "Self":
+        """
+        apply min/max HTLC size restrictions of the given channel.
+        """
+        if ch:
+            why = why or ch.directed_scid_to_me
+        if why: why = f"{why}: "
+
+        new_min, new_max = self.min_msat, self.max_msat
+        if ch.htlc_minimum_to_me > self.min_msat:
+            new_min = ch.htlc_minimum_to_me
+            logger.debug(f"{why}raising min_msat due to HTLC requirements: {self.min_msat} -> {new_min}")
+        if ch.htlc_maximum_to_me < self.max_msat:
+            new_max = ch.htlc_maximum_to_me
+            logger.debug(f"{why}lowering max_msat due to HTLC requirements: {self.max_msat} -> {new_max}")
+        return TxBounds(min_msat=new_min, max_msat=new_max)
+
+    def restrict_to_zero_fees(self, ch: "LocalChannel"=None, base: int=0, ppm: int=0, why:str = "") -> "Self":
+        """
+        restrict tx size such that PPM fees are zero.
+        if the channel has a base fee, then `max_msat` is forced to 0.
+        """
+        if ch:
+            why = why or ch.directed_scid_to_me
+            self = self.restrict_to_zero_fees(base=ch.to_me["base_fee_millisatoshi"], ppm=ch.to_me["fee_per_millionth"], why=why)
+
+        if why: why = f"{why}: "
+
+        new_max = self.max_msat
+        ppm_max = math.ceil(1000000 / ppm) - 1 if ppm != 0 else new_max
+        if ppm_max < new_max:
+            logger.debug(f"{why}decreasing max_msat due to fee ppm: {new_max} -> {ppm_max}")
+            new_max = ppm_max
+
+        if base != 0:
+            logger.debug(f"{why}free route impossible: channel has base fees")
+            new_max = 0
+
+        return TxBounds(min_msat=self.min_msat, max_msat=new_max)
+
+
+class LocalChannel:
+    def __init__(self, channels: list, rpc: "RpcHelper"):
+        assert 0 < len(channels) <= 2, f"unexpected: channel count: {channels}"
+        out = None
+        in_ = None
+        for c in channels:
+            if c["source"] == rpc.self_id:
+                assert out is None, f"unexpected: multiple channels from self: {channels}"
+                out = c
+            if c["destination"] == rpc.self_id:
+                assert in_ is None, f"unexpected: multiple channels to self: {channels}"
+                in_ = c
+
+        # assert out is not None, f"no channel from self: {channels}"
+        # assert in_ is not None, f"no channel to self: {channels}"
+
+        if out and in_:
+            assert out["destination"] == in_["source"], f"channel peers are asymmetric?! {channels}"
+            assert out["short_channel_id"] == in_["short_channel_id"], f"channel ids differ?! {channels}"
+
+        self.from_me = out
+        self.to_me = in_
+        self.remote_node = rpc.node(self.remote_peer)
+        self.peer_ch = rpc.peerchannel(self.scid, self.remote_peer)
+        self.forwards_from_me = rpc.rpc.listforwards(out_channel=self.scid, status="settled")["forwards"]
+
+    def __repr__(self) -> str:
+        return self.to_str(with_scid=True, with_bal_ratio=True, with_cost=False, with_ppm_theirs=False)
+
+    def to_str(
+        self,
+        with_peer_id:bool = False,
+        with_scid:bool = False,
+        with_bal_msat:bool = False,
+        with_bal_ratio:bool = False,
+        with_cost:bool = False,
+        with_ppm_theirs:bool = False,
+        with_ppm_mine:bool = False,
+        with_profits:bool = True,
+        with_payments:bool = False,
+    ) -> str:
+        base_flag = "*" if not self.online or self.base_fee_to_me != 0 else ""
+        alias = f"({self.remote_alias}){base_flag}"
+        peerid = f" {self.remote_peer}" if with_peer_id else ""
+        scid = f"  scid:{self.scid:>13}" if with_scid else ""
+        bal = f"  S:{int(self.sendable):11}/R:{int(self.receivable):11}" if with_bal_msat else ""
+        ratio = f"  MINE:{(100*self.send_ratio):>8.4f}%" if with_bal_ratio else ""
+        payments = f"  OUT:{int(self.out_fulfilled_msat):>11}/IN:{int(self.in_fulfilled_msat):>11}" if with_payments else ""
+        profits = f"  P$:{int(self.fees_lifetime_mine):>8}" if with_profits else ""
+        cost = f"  COST:{self.opportunity_cost_lent:>8}" if with_cost else ""
+        ppm_theirs = self.ppm_to_me if self.to_me else "N/A"
+        ppm_theirs = f"  PPM_THEIRS:{ppm_theirs:>6}" if with_ppm_theirs else ""
+        ppm_mine = self.ppm_from_me if self.from_me else "N/A"
+        ppm_mine = f"  PPM_MINE:{ppm_mine:>6}" if with_ppm_mine else ""
+        return f"channel{alias:30}{peerid}{scid}{bal}{ratio}{payments}{profits}{cost}{ppm_theirs}{ppm_mine}"
+
+
+    @property
+    def online(self) -> bool:
+        return self.from_me and self.to_me
+
+    @property
+    def remote_peer(self) -> str:
+        if self.from_me:
+            return self.from_me["destination"]
+        else:
+            return self.to_me["source"]
+
+    @property
+    def remote_alias(self) -> str:
+        return self.remote_node["alias"]
+
+    @property
+    def scid(self) -> str:
+        if self.from_me:
+            return self.from_me["short_channel_id"]
+        else:
+            return self.to_me["short_channel_id"]
+
+    @property
+    def htlc_minimum_to_me(self) -> Millisatoshi:
+        return self.to_me["htlc_minimum_msat"]
+
+    @property
+    def htlc_minimum_from_me(self) -> Millisatoshi:
+        return self.from_me["htlc_minimum_msat"]
+
+    @property
+    def htlc_minimum(self) -> Millisatoshi:
+        return max(self.htlc_minimum_to_me, self.htlc_minimum_from_me)
+
+    @property
+    def htlc_maximum_to_me(self) -> Millisatoshi:
+        return self.to_me["htlc_maximum_msat"]
+
+    @property
+    def htlc_maximum_from_me(self) -> Millisatoshi:
+        return self.from_me["htlc_maximum_msat"]
+
+    @property
+    def htlc_maximum(self) -> Millisatoshi:
+        return min(self.htlc_maximum_to_me, self.htlc_maximum_from_me)
+
+    @property
+    def direction_to_me(self) -> int:
+        return self.to_me["direction"]
+
+    @property
+    def direction_from_me(self) -> int:
+        return self.from_me["direction"]
+
+    @property
+    def directed_scid_to_me(self) -> str:
+        return f"{self.scid}/{self.direction_to_me}"
+
+    @property
+    def directed_scid_from_me(self) -> str:
+        return f"{self.scid}/{self.direction_from_me}"
+
+    @property
+    def delay_them(self) -> str:
+        return self.to_me["delay"]
+
+    @property
+    def delay_me(self) -> str:
+        return self.from_me["delay"]
+
+    @property
+    def ppm_to_me(self) -> int:
+        return self.to_me["fee_per_millionth"]
+
+    @property
+    def ppm_from_me(self) -> int:
+        return self.from_me["fee_per_millionth"]
+        # return self.peer_ch["fee_proportional_millionths"]
+
+    @property
+    def base_fee_to_me(self) -> int:
+        return self.to_me["base_fee_millisatoshi"]
+
+    @property
+    def receivable(self) -> int:
+        return self.peer_ch["receivable_msat"]
+
+    @property
+    def sendable(self) -> int:
+        return self.peer_ch["spendable_msat"]
+
+    @property
+    def in_fulfilled_msat(self) -> Millisatoshi:
+        return self.peer_ch["in_fulfilled_msat"]
+
+    @property
+    def out_fulfilled_msat(self) -> Millisatoshi:
+        return self.peer_ch["out_fulfilled_msat"]
+
+    @property
+    def fees_lifetime_mine(self) -> Millisatoshi:
+        return sum(fwd["fee_msat"] for fwd in self.forwards_from_me)
+
+    @property
+    def send_ratio(self) -> float:
+        cap = self.receivable + self.sendable
+        return self.sendable / cap
+
+    @property
+    def opportunity_cost_lent(self) -> int:
+        """ how much msat did we gain by pushing their channel to its current balance? """
+        return int(self.receivable * self.ppm_from_me / 1000000)
+
+class RpcHelper:
+    def __init__(self, rpc: LightningRpc):
+        self.rpc = rpc
+        self.self_id = rpc.getinfo()["id"]
+
+    def localchannel(self, scid: str) -> LocalChannel:
+        listchan = self.rpc.listchannels(scid)
+        # this assertion would probably indicate a typo in the scid
+        assert listchan and listchan.get("channels", []) != [], f"bad listchannels for {scid}: {listchan}"
+        return LocalChannel(listchan["channels"], self)
+
+    def node(self, id: str) -> dict:
+        nodes = self.rpc.listnodes(id)["nodes"]
+        assert len(nodes) == 1, f"unexpected: multiple nodes for {id}: {nodes}"
+        return nodes[0]
+
+    def peerchannel(self, scid: str, peer_id: str) -> dict:
+        peerchannels = self.rpc.listpeerchannels(peer_id)["channels"]
+        channels = [c for c in peerchannels if c["short_channel_id"] == scid]
+        assert len(channels) == 1, f"expected exactly 1 channel, got: {channels}"
+        return channels[0]
+
+    def try_getroute(self, *args, **kwargs) -> dict | None:
+        """ wrapper for getroute which returns None instead of error if no route exists """
+        try:
+            route = self.rpc.getroute(*args, **kwargs)
+        except RpcError as e:
+            logger.debug(f"rpc failed: {e}")
+            return None
+        else:
+            route = route["route"]
+            if route == []: return None
+            return route
+
+class LoopRouter:
+    def __init__(self, rpc: RpcHelper, metrics: Metrics = None):
+        self.rpc = rpc
+        self.metrics = metrics or Metrics()
+        self.bad_channels = []  # list of directed scid
+        self.nonzero_base_channels = []  # list of directed scid
+
+    def drop_caches(self) -> None:
+        logger.info("LoopRouter.drop_caches()")
+        self.bad_channels = []
+
+    def _get_directed_scid(self, scid: str, direction: int) -> dict:
+        channels = self.rpc.rpc.listchannels(scid)["channels"]
+        channels = [c for c in channels if c["direction"] == direction]
+        assert len(channels) == 1, f"expected exactly 1 channel: {channels}"
+        return channels[0]
+
+    def loop_once(self, out_scid: str, in_scid: str, bounds: TxBounds) -> LoopError|int:
+        out_ch = self.rpc.localchannel(out_scid)
+        in_ch = self.rpc.localchannel(in_scid)
+
+        if out_ch.directed_scid_from_me in self.bad_channels or in_ch.directed_scid_to_me in self.bad_channels:
+            logger.info(f"loop {out_scid} -> {in_scid} failed in our own channel")
+            self.metrics.own_bad_channel += 1
+            return LoopError.TRANSIENT
+
+        # bounds = bounds.restrict_to_htlc(out_ch)  # htlc bounds seem to be enforced only in the outward direction
+        bounds = bounds.restrict_to_htlc(in_ch)
+        bounds = bounds.restrict_to_zero_fees(in_ch)
+        if not bounds.is_satisfiable():
+            self.metrics.in_ch_unsatisfiable += 1
+            return LoopError.NO_ROUTE
+
+        logger.debug(f"route with bounds {bounds}")
+        route = self.route(out_ch, in_ch, bounds)
+        logger.debug(f"route: {route}")
+        if route == RouteError.NO_ROUTE:
+            self.metrics.no_route += 1
+            return LoopError.NO_ROUTE
+        elif route == RouteError.HAS_BASE_FEE:
+            # try again with a different route
+            return LoopError.TRANSIENT
+
+        amount_msat = route[0]["amount_msat"]
+        invoice_id = f"loop-{time.time():.6f}".replace(".", "_")
+        invoice_desc = f"bal {out_scid}:{in_scid}"
+        invoice = self.rpc.rpc.invoice("any", invoice_id, invoice_desc)
+        logger.debug(f"invoice: {invoice}")
+
+        payment = self.rpc.rpc.sendpay(route, invoice["payment_hash"], invoice_id, amount_msat, invoice["bolt11"], invoice["payment_secret"])
+        logger.debug(f"sent: {payment}")
+
+        try:
+            wait = self.rpc.rpc.waitsendpay(invoice["payment_hash"])
+            logger.debug(f"result: {wait}")
+        except RpcError as e:
+            self.metrics.sendpay_fail += 1
+            err_data = e.error["data"]
+            err_scid, err_dir = err_data["erring_channel"], err_data["erring_direction"]
+            err_directed_scid = f"{err_scid}/{err_dir}"
+            logger.debug(f"ch failed, adding to excludes: {err_directed_scid}; {e.error}")
+            self.bad_channels.append(err_directed_scid)
+            return LoopError.TRANSIENT
+        else:
+            self.metrics.sendpay_succeed += 1
+            self.metrics.looped_msat += int(amount_msat)
+            return int(amount_msat)
+
+    def route(self, out_ch: LocalChannel, in_ch: LocalChannel, bounds: TxBounds) -> list[dict] | RouteError:
+        exclude = [
+            # ensure the payment doesn't cross either channel in reverse.
+            # note that this doesn't preclude it from taking additional trips through self, with other peers.
+            # out_ch.directed_scid_to_me,
+            # in_ch.directed_scid_from_me,
+
+            # alternatively, never route through self. this avoids a class of logic error, like what to do with fees i charge "myself".
+            self.rpc.self_id
+        ] + self.bad_channels + self.nonzero_base_channels
+
+        out_peer = out_ch.remote_peer
+        in_peer = in_ch.remote_peer
+
+        route_or_bounds = bounds
+        while isinstance(route_or_bounds, TxBounds):
+            old_bounds = route_or_bounds
+            route_or_bounds = self._find_partial_route(out_peer, in_peer, old_bounds, exclude=exclude)
+            if route_or_bounds == old_bounds:
+                return RouteError.NO_ROUTE
+
+        if isinstance(route_or_bounds, RouteError):
+            return route_or_bounds
+
+        route = self._add_route_endpoints(route_or_bounds, out_ch, in_ch)
+        return route
+
+    def _find_partial_route(self, out_peer: str, in_peer: str, bounds: TxBounds, exclude: list[str]=[]) -> list[dict] | RouteError | TxBounds:
+        route = self.rpc.try_getroute(in_peer, amount_msat=bounds.max_msat, riskfactor=0, fromid=out_peer, exclude=exclude, cltv=CLTV)
+        if route is None:
+            logger.debug(f"no route for {bounds.max_msat}msat {out_peer} -> {in_peer}")
+            return RouteError.NO_ROUTE
+
+        send_msat = route[0]["amount_msat"]
+        if send_msat != Millisatoshi(bounds.max_msat):
+            logger.debug(f"found route with non-zero fee: {send_msat} -> {bounds.max_msat}. {route}")
+
+            error = None
+            for hop in route:
+                hop_scid = hop["channel"]
+                hop_dir = hop["direction"]
+                directed_scid = f"{hop_scid}/{hop_dir}"
+                ch = self._get_directed_scid(hop_scid, hop_dir)
+                if ch["base_fee_millisatoshi"] != 0:
+                    self.nonzero_base_channels.append(directed_scid)
+                    error = RouteError.HAS_BASE_FEE
+                bounds = bounds.restrict_to_zero_fees(ppm=ch["fee_per_millionth"], why=directed_scid)
+
+            return bounds.raise_max_to_be_satisfiable() if error is None else error
+
+        return route
+
+    def _add_route_endpoints(self, route, out_ch: LocalChannel, in_ch: LocalChannel):
+        inbound_hop = dict(
+          id=self.rpc.self_id,
+          channel=in_ch.scid,
+          direction=in_ch.direction_to_me,
+          amount_msat=route[-1]["amount_msat"],
+          delay=route[-1]["delay"],
+          style="tlv",
+        )
+        route = self._add_route_delay(route, in_ch.delay_them) + [ inbound_hop ]
+
+        outbound_hop = dict(
+            id=out_ch.remote_peer,
+            channel=out_ch.scid,
+            direction=out_ch.direction_from_me,
+            amount_msat=route[0]["amount_msat"],
+            delay=route[0]["delay"] + out_ch.delay_them,
+            style="tlv",
+        )
+        route = [ outbound_hop ] + route
+        return route
+
+    def _add_route_delay(self, route: list[dict], delay: int) -> list[dict]:
+        return [ dict(hop, delay=hop["delay"] + delay) for hop in route ]
+
+@dataclass
+class LoopJob:
+    out: str  # scid
+    in_: str  # scid
+    amount: int
+
+@dataclass
+class LoopJobIdle:
+    sec: int = 10
+
+class LoopJobDone(Enum):
+    COMPLETED = "COMPLETED"
+    ABORTED = "ABORTED"
+
+class AbstractLoopRunner:
+    def __init__(self, looper: LoopRouter, bounds: TxBounds, parallelism: int):
+        self.looper = looper
+        self.bounds = bounds
+        self.parallelism = parallelism
+        self.bounds_map = {}  # map (out:str, in_:str) -> TxBounds. it's a cache so we don't have to try 10 routes every time.
+
+    def pop_job(self) -> LoopJob | LoopJobIdle | LoopJobDone:
+        raise NotImplemented  # abstract method
+
+    def finished_job(self, job: LoopJob, progress: int|LoopError) -> None:
+        raise NotImplemented  # abstract method
+
+    def run_to_completion(self, exit_on_any_completed:bool = False) -> None:
+        self.exiting = False
+        self.exit_on_any_completed = exit_on_any_completed
+        if self.parallelism == 1:
+            # run inline to aid debugging
+            self._worker_thread()
+        else:
+            with ThreadPoolExecutor(max_workers=self.parallelism) as executor:
+                _ = list(executor.map(lambda _i: self._try_invoke(self._worker_thread), range(self.parallelism)))
+
+    def drop_caches(self) -> None:
+        logger.info("AbstractLoopRunner.drop_caches()")
+        self.looper.drop_caches()
+        self.bounds_map = {}
+
+
+    def _try_invoke(self, f, *args) -> None:
+        """
+        try to invoke `f` with the provided `args`, and log if it fails.
+        this overcomes the issue that background tasks which fail via Exception otherwise do so silently.
+        """
+        try:
+            f(*args)
+        except Exception as e:
+            logger.error(f"task failed: {e}")
+
+
+    def _worker_thread(self) -> None:
+        while not self.exiting:
+            job = self.pop_job()
+            logger.debug(f"popped job: {job}")
+            if isinstance(job, LoopJobDone):
+                return self._worker_finished(job)
+
+            if isinstance(job, LoopJobIdle):
+                logger.debug(f"idling for {job.sec}")
+                time.sleep(job.sec)
+                continue
+
+            result = self._execute_job(job)
+            logger.debug(f"finishing job {job} with {result}")
+            self.finished_job(job, result)
+
+    def _execute_job(self, job: LoopJob) -> LoopError|int:
+        bounds = self.bounds_map.get((job.out, job.in_), self.bounds)
+        bounds = bounds.intersect(TxBounds(max_msat=job.amount))
+        if not bounds.is_satisfiable():
+            logger.debug(f"TxBounds for job are unsatisfiable; skipping: {bounds} {job}")
+            return LoopError.NO_ROUTE
+
+        amt_looped = self.looper.loop_once(job.out, job.in_, bounds)
+        if amt_looped in (0, LoopError.NO_ROUTE, LoopError.TRANSIENT):
+            return amt_looped
+
+        logger.info(f"looped {amt_looped} from {job.out} -> {job.in_}")
+        bounds = bounds.intersect(TxBounds(max_msat=amt_looped))
+
+        self.bounds_map[(job.out, job.in_)] = bounds
+        return amt_looped
+
+    def _worker_finished(self, job: LoopJobDone) -> None:
+        if job == LoopJobDone.COMPLETED and self.exit_on_any_completed:
+            logger.debug(f"worker completed -> exiting pool")
+            self.exiting = True
+
+class LoopPairState:
+    # TODO: use this in MultiLoopBalancer, or stop shoving state in here and put it on LoopBalancer instead.
+    def __init__(self, out: str, in_: str, amount: int):
+        self.out = out
+        self.in_ = in_
+        self.amount_target = amount
+        self.amount_looped = 0
+        self.amount_outstanding = 0
+        self.tx_fail_count = 0
+        self.route_fail_count = 0
+        self.last_job_start_time = None
+        self.failed_tx_throttler = 0  # increase by one every time we fail, decreases more gradually, when we succeed
+
+class LoopBalancer(AbstractLoopRunner):
+    def __init__(self, out: str, in_: str, amount: int, looper: LoopRouter, bounds: TxBounds, parallelism: int=1):
+        super().__init__(looper, bounds, parallelism)
+        self.state = LoopPairState(out, in_, amount)
+
+    def pop_job(self) -> LoopJob | LoopJobIdle | LoopJobDone:
+        if self.state.tx_fail_count + 10*self.state.route_fail_count >= MAX_SEQUENTIAL_JOB_FAILURES:
+            logger.info(f"giving up ({self.state.out} -> {self.state.in_}): {self.state.tx_fail_count} tx failures, {self.state.route_fail_count} route failures")
+            return LoopJobDone.ABORTED
+
+        if self.state.tx_fail_count + self.state.route_fail_count > 0:
+            # N.B.: last_job_start_time is guaranteed to have been set by now
+            idle_until = self.state.last_job_start_time + TX_FAIL_BACKOFF*self.state.failed_tx_throttler
+            idle_for = idle_until - time.time()
+            if self.state.amount_outstanding != 0 or idle_for > 0:
+                # when we hit transient failures, restrict to just one job in flight at a time.
+                # this is aimed for the initial route building, where multiple jobs in flight is just useless,
+                # but it's not a bad idea for network blips, etc, either.
+                logger.info(f"throttling ({self.state.out} -> {self.state.in_}) for {idle_for:.0f}: {self.state.tx_fail_count} tx failures, {self.state.route_fail_count} route failures")
+                return LoopJobIdle(idle_for) if idle_for > 0 else LoopJobIdle()
+
+        amount_avail = self.state.amount_target - self.state.amount_looped - self.state.amount_outstanding
+        if amount_avail < self.bounds.min_msat:
+            if self.state.amount_outstanding == 0: return LoopJobDone.COMPLETED
+            return LoopJobIdle()  # sending out another job would risk over-transferring
+        amount_this_job = min(amount_avail, self.bounds.max_msat)
+
+        self.state.amount_outstanding += amount_this_job
+        self.state.last_job_start_time = time.time()
+        return LoopJob(out=self.state.out, in_=self.state.in_, amount=amount_this_job)
+
+    def finished_job(self, job: LoopJob, progress: int) -> None:
+        self.state.amount_outstanding -= job.amount
+        if progress == LoopError.NO_ROUTE:
+            self.state.route_fail_count += 1
+            self.state.failed_tx_throttler += 10
+        elif progress == LoopError.TRANSIENT:
+            self.state.tx_fail_count += 1
+            self.state.failed_tx_throttler += 1
+        else:
+            self.state.amount_looped += progress
+            self.state.tx_fail_count = 0
+            self.state.route_fail_count = 0
+            self.state.failed_tx_throttler = max(0, self.state.failed_tx_throttler - 0.2)
+        logger.info(f"loop progressed ({job.out} -> {job.in_}) {progress}: {self.state.amount_looped} of {self.state.amount_target}")
+
+class MultiLoopBalancer(AbstractLoopRunner):
+    """
+    multiplexes jobs between multiple LoopBalancers.
+    note that the child LoopBalancers don't actually execute the jobs -- just produce them.
+    """
+    def __init__(self, looper: LoopRouter, bounds: TxBounds, parallelism: int=1):
+        super().__init__(looper, bounds, parallelism)
+        self.loops = []
+        # job_index: increments on every job so we can grab jobs evenly from each LoopBalancer.
+        # in that event that producers are idling, it can actually increment more than once,
+        # so don't take this too literally
+        self.job_index = 0
+
+    def add_loop(self, out: LocalChannel, in_: LocalChannel, amount: int) -> None:
+        """
+        start looping sats from out -> in_
+        """
+        assert not any(l.state.out == out.scid and l.state.in_ == in_.scid for l in self.loops), f"tried to add duplicate loops from {out} -> {in_}"
+        logger.info(f"looping from ({out}) to ({in_})")
+        self.loops.append(LoopBalancer(out.scid, in_.scid, amount, self.looper, self.bounds, self.parallelism))
+
+    def pop_job(self) -> LoopJob | LoopJobIdle | LoopJobDone:
+        # N.B.: this can be called in parallel, so try to be consistent enough to not crash
+
+        idle_job = None
+        abort_job = None
+        for i, _ in enumerate(self.loops):
+            loop = self.loops[(self.job_index + i) % len(self.loops)]
+            self.job_index += 1
+            job = loop.pop_job()
+            if isinstance(job, LoopJob):
+                return job
+            if isinstance(job, LoopJobIdle):
+                idle_job = LoopJobIdle(min(job.sec, idle_job.sec)) if idle_job is not None else job
+            if job == LoopJobDone.ABORTED:
+                abort_job = job
+
+        # either there's a task to idle, or we have to terminate.
+        # if terminating, terminate ABORTED if any job aborted, else COMPLETED
+        if idle_job is not None: return idle_job
+        if abort_job is not None: return abort_job
+        return LoopJobDone.COMPLETED
+
+    def finished_job(self, job: LoopJob, progress: int) -> None:
+        # this assumes (enforced externally) that we have only one loop for a given out/in_ pair
+        for l in self.loops:
+            if l.state.out == job.out and l.state.in_ == job.in_:
+                l.finished_job(job, progress)
+
+        logger.info(f"total: {self.looper.metrics}")
+
+
+def balance_loop(rpc: RpcHelper, out: str, in_: str, amount_msat: int, min_msat: int, max_msat: int, parallelism: int):
+    looper = LoopRouter(rpc)
+    bounds = TxBounds(min_msat=min_msat, max_msat=max_msat)
+    balancer = LoopBalancer(out, in_, amount_msat, looper, bounds, parallelism)
+
+    balancer.run_to_completion()
+
+def autobalance_once(rpc: RpcHelper, metrics: Metrics, bounds: TxBounds, parallelism: int) -> bool:
+    """
+    autobalances all channels.
+    returns True if channels are balanced (or as balanced as can be); False if in need of further balancing
+    """
+    looper = LoopRouter(rpc, metrics)
+    balancer = MultiLoopBalancer(looper, bounds, parallelism)
+
+    channels = []
+    for peerch in rpc.rpc.listpeerchannels()["channels"]:
+        try:
+            channels.append(rpc.localchannel(peerch["short_channel_id"]))
+        except:
+            logger.info(f"NO CHANNELS for {peerch['peer_id']}")
+
+    channels = [ch for ch in channels if ch.online and ch.base_fee_to_me == 0]
+    give_to = [ ch for ch in channels if ch.send_ratio > 0.95 ]
+    take_from = [ ch for ch in channels if ch.send_ratio < 0.20 ]
+
+    if give_to == [] and take_from == []:
+        return True
+
+    for to in give_to:
+        for from_ in take_from:
+            balancer.add_loop(to, from_, 10000000)
+
+    balancer.run_to_completion(exit_on_any_completed=True)
+    return False
+
+
+def autobalance(rpc: RpcHelper, min_msat: int, max_msat: int, parallelism: int):
+    bounds = TxBounds(min_msat=min_msat, max_msat=max_msat)
+    metrics = Metrics()
+    while not autobalance_once(rpc, metrics, bounds, parallelism):
+        pass
+
+def show_status(rpc: RpcHelper, full: bool=False):
+    """
+    show a table of channel balances between peers.
+    """
+    for peerch in rpc.rpc.listpeerchannels()["channels"]:
+        try:
+            ch = rpc.localchannel(peerch["short_channel_id"])
+        except:
+            print(f"{peerch['peer_id']} scid:{peerch['short_channel_id']} state:{peerch['state']} NO CHANNELS")
+        else:
+            print(ch.to_str(with_scid=True, with_bal_ratio=True, with_payments=True, with_cost=full, with_ppm_theirs=True, with_ppm_mine=True, with_peer_id=full))
+
+def main():
+    logging.basicConfig()
+    logger.setLevel(logging.INFO)
+
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--verbose", action="store_true", help="more logging")
+    parser.add_argument("--min-msat", default="999", help="min transaction size")
+    parser.add_argument("--max-msat", default="1000000", help="max transaction size")
+    parser.add_argument("--jobs", default="1", help="how many HTLCs to keep in-flight at once")
+    subparsers = parser.add_subparsers(help="action")
+
+    status_parser = subparsers.add_parser("status")
+    status_parser.set_defaults(action="status")
+    status_parser.add_argument("--full", action="store_true", help="more info per channel")
+
+    loop_parser = subparsers.add_parser("loop")
+    loop_parser.set_defaults(action="loop")
+    loop_parser.add_argument("out", help="peer id to send tx through")
+    loop_parser.add_argument("in_", help="peer id to receive tx through")
+    loop_parser.add_argument("amount", help="total amount of msat to loop")
+
+    autobal_parser = subparsers.add_parser("autobalance")
+    autobal_parser.set_defaults(action="autobalance")
+
+    args = parser.parse_args()
+
+    if args.verbose:
+        logger.setLevel(logging.DEBUG)
+
+    rpc = RpcHelper(LightningRpc(RPC_FILE))
+
+    if args.action == "status":
+        show_status(rpc, full=args.full)
+
+    if args.action == "loop":
+        balance_loop(rpc, out=args.out, in_=args.in_, amount_msat=int(args.amount), min_msat=int(args.min_msat), max_msat=int(args.max_msat), parallelism=int(args.jobs))
+
+    if args.action == "autobalance":
+        autobalance(rpc, min_msat=int(args.min_msat), max_msat=int(args.max_msat), parallelism=int(args.jobs))
+
+if __name__ == '__main__':
+    main()

hosts/by-name/servo/services/cryptocurrencies/clightning.nix

@@ -0,0 +1,135 @@
+# clightning is an implementation of Bitcoin's Lightning Network.
+# as such, this assumes that `services.bitcoin` is enabled.
+# docs:
+# - tor clightning config: <https://docs.corelightning.org/docs/tor>
+# - `lightning-cli` and subcommands: <https://docs.corelightning.org/reference/lightning-cli>
+# - `man lightningd-config`
+#
+# management/setup/use:
+# - guide: <https://github.com/ElementsProject/lightning>
+#
+# debugging:
+# - `lightning-cli getlog debug`
+# - `lightning-cli listpays`  -> show payments this node sent
+# - `lightning-cli listinvoices`  -> show payments this node received
+#
+# first, acquire peers:
+# - `lightning-cli connect id@host`
+#   where `id` is the node's pubkey, and `host` is perhaps an ip:port tuple, or a hash.onion:port tuple.
+#   for testing, choose any node listed on <https://1ml.com>
+# - `lightning-cli listpeers`
+#   should show the new peer, with `connected: true`
+#
+# then, fund the clightning wallet
+# - `lightning-cli newaddr`
+#
+# then, open channels
+# - `lightning-cli connect ...`
+# - `lightning-cli fundchannel <node_id> <amount_in_satoshis>`
+#
+# who to federate with?
+# - a lot of the larger nodes allow hands-free channel creation
+#   - either inbound or outbound, sometimes paid
+# - find nodes on:
+#   - <https://terminal.lightning.engineering/>
+#   - <https://1ml.com>
+#     - tor nodes: <https://1ml.com/node?order=capacity&iponionservice=true>
+#   - <https://lightningnetwork.plus>
+#   - <https://mempool.space/lightning>
+#   - <https://amboss.space>
+# - a few tor-capable nodes which allow channel creation:
+#   - <https://c-otto.de/>
+#   - <https://cyberdyne.sh/>
+#   - <https://yalls.org/about/>
+#   - <https://coincept.com/>
+# - more resources: <https://www.lopp.net/lightning-information.html>
+#   - node routability: https://hashxp.org/lightning/node/<id>
+# - especially, acquire inbound liquidity via lightningnetwork.plus's swap feature
+#   - most of the opportunities are gated behind a minimum connection or capacity requirement
+#
+# tune payment parameters
+# - `lightning-cli setchannel <id> [feebase] [feeppm] [htlcmin] [htlcmax] [enforcedelay] [ignorefeelimits]`
+#   - e.g. `lightning-cli setchannel all 0 10`
+#   - it's suggested that feebase=0 simplifies routing.
+#
+# teardown:
+# - `lightning-cli withdraw <bc1... dest addr> <amount in satoshis> [feerate]`
+#
+# sanity:
+# - `lightning-cli listfunds`
+#
+# to receive a payment (do as `clightning` user):
+# - `lightning-cli invoice <amount in millisatoshi> <label> <description>`
+#   - specify amount as `any` if undetermined
+#   - then give the resulting bolt11 URI to the payer
+# to send a payment:
+# - `lightning-cli pay <bolt11 URI>`
+#   - or `lightning-cli pay <bolt11 URI> [amount_msat] [label] [riskfactor] [maxfeepercent] ...`
+#   - amount_msat must be "null" if the bolt11 URI specifies a value
+#   - riskfactor defaults to 10
+#   - maxfeepercent defaults to 0.5
+#   - label is a human-friendly label for my records
+
+{ config, pkgs, ... }:
+{
+  sane.persist.sys.byStore.ext = [
+    { user = "clightning"; group = "clightning"; mode = "0710"; path = "/var/lib/clightning"; method = "bind"; }
+  ];
+
+  # `lightning-cli` finds its RPC file via `~/.lightning/bitcoin/lightning-rpc`, to message the daemon
+  sane.user.fs.".lightning".symlink.target = "/var/lib/clightning";
+
+  # see bitcoin.nix for how to generate this
+  services.bitcoind.mainnet.rpc.users.clightning.passwordHMAC =
+    "befcb82d9821049164db5217beb85439$2c31ac7db3124612e43893ae13b9527dbe464ab2d992e814602e7cb07dc28985";
+
+  sane.services.clightning.enable = true;
+  sane.services.clightning.proxy = "127.0.0.1:9050";  # proxy outgoing traffic through tor
+  # sane.services.clightning.publicAddress = "statictor:127.0.0.1:9051";
+  sane.services.clightning.getPublicAddressCmd = "cat /var/lib/tor/onion/clightning/hostname";
+
+  services.tor.relay.onionServices.clightning = {
+    version = 3;
+    map = [{
+      # by default tor will route public tor port P to 127.0.0.1:P.
+      # so if this port is the same as clightning would natively use, then no further config is needed here.
+      # see: <https://2019.www.torproject.org/docs/tor-manual.html.en#HiddenServicePort>
+      port = 9735;
+      # target.port; target.addr;  #< set if tor port != clightning port
+    }];
+    # allow "tor" group (i.e. clightning) to read /var/lib/tor/onion/clightning/hostname
+    settings.HiddenServiceDirGroupReadable = true;
+  };
+
+  # must be in "tor" group to read /var/lib/tor/onion/*/hostname
+  users.users.clightning.extraGroups = [ "tor" ];
+
+  systemd.services.clightning.after = [ "tor.service" ];
+
+  # lightning-config contains fields from here:
+  # - <https://docs.corelightning.org/docs/configuration>
+  # secret config includes:
+  # - bitcoin-rpcpassword
+  # - alias=nodename
+  # - rgb=rrggbb
+  # - fee-base=<millisatoshi>
+  # - fee-per-satoshi=<ppm>
+  # - feature configs (i.e. experimental-xyz options)
+  sane.services.clightning.extraConfig = ''
+    log-level=debug:lightningd
+    # peerswap:
+    # - config example: <https://github.com/fort-nix/nix-bitcoin/pull/462/files#diff-b357d832705b8ce8df1f41934d613f79adb77c4cd5cd9e9eb12a163fca3e16c6>
+    # XXX: peerswap crashes clightning on launch. stacktrace is useless.
+    # plugin=${pkgs.peerswap}/bin/peerswap
+    # peerswap-db-path=/var/lib/clightning/peerswap/swaps
+    # peerswap-policy-path=...
+  '';
+  sane.services.clightning.extraConfigFiles = [ config.sops.secrets."lightning-config".path ];
+  sops.secrets."lightning-config" = {
+    mode = "0640";
+    owner = "clightning";
+    group = "clightning";
+  };
+
+  sane.programs.clightning.enableFor.user.colin = true;  # for debugging/admin: `lightning-cli`
+}

hosts/by-name/servo/services/cryptocurrencies/default.nix

@@ -0,0 +1,10 @@
+{ ... }:
+{
+  imports = [
+    ./bitcoin.nix
+    ./clightning.nix
+    ./i2p.nix
+    ./monero.nix
+    ./tor.nix
+  ];
+}

hosts/by-name/servo/services/cryptocurrencies/i2p.nix

@@ -0,0 +1,4 @@
+{ ... }:
+{
+  services.i2p.enable = true;
+}

hosts/by-name/servo/services/cryptocurrencies/monero.nix

@@ -0,0 +1,31 @@
+# as of 2023/11/26: complete downloaded blockchain should be 200GiB on disk, give or take.
+{ ... }:
+{
+  sane.persist.sys.byStore.ext = [
+    # /var/lib/monero/lmdb is what consumes most of the space
+    { user = "monero"; group = "monero"; path = "/var/lib/monero"; method = "bind"; }
+  ];
+
+  services.monero.enable = true;
+  services.monero.limits.upload = 5000;  # in kB/s
+  services.monero.extraConfig = ''
+    # see: monero doc/ANONYMITY_NETWORKS.md
+    #
+    # "If any anonymity network is enabled, transactions being broadcast that lack a valid 'context'
+    # (i.e. the transaction did not come from a P2P connection) will only be sent to peers on anonymity networks."
+    #
+    # i think this means that setting tx-proxy here ensures any transactions sent locally to my node (via RPC)
+    # will be sent over an anonymity network.
+    tx-proxy=i2p,127.0.0.1:9000
+    tx-proxy=tor,127.0.0.1:9050
+  '';
+
+  # monero ports: <https://monero.stackexchange.com/questions/604/what-ports-does-monero-use-rpc-p2p-etc>
+  # - 18080 = "P2P" monero node <-> monero node connections
+  # - 18081 = "RPC" monero client -> monero node connections
+  sane.ports.ports."18080" = {
+    protocol = [ "tcp" ];
+    visibleTo.wan = true;
+    description = "colin-monero-p2p";
+  };
+}

hosts/by-name/servo/services/cryptocurrencies/tor.nix

@@ -0,0 +1,25 @@
+# tor settings: <https://2019.www.torproject.org/docs/tor-manual.html.en>
+{ lib, ... }:
+{
+  # tor hidden service hostnames aren't deterministic, so persist.
+  # might be able to get away with just persisting /var/lib/tor/onion, not sure.
+  sane.persist.sys.byStore.plaintext = [
+    { user = "tor"; group = "tor"; mode = "0710"; path = "/var/lib/tor"; method = "bind"; }
+  ];
+
+  # tor: `tor.enable` doesn't start a relay, exit node, proxy, etc. it's minimal.
+  # tor.client.enable configures a torsocks proxy, accessible *only* to localhost.
+  #   at 127.0.0.1:9050
+  services.tor.enable = true;
+  services.tor.client.enable = true;
+
+  # in order for services to read /var/lib/tor/onion/*/hostname, they must be able to traverse /var/lib/tor,
+  # and /var/lib/tor must have g+x.
+  # DataDirectoryGroupReadable causes tor to use g+rx, technically more than we need, but all the files are 600 so it's fine.
+  services.tor.settings.DataDirectoryGroupReadable = true;
+  # StateDirectoryMode defaults to 0700, and thereby prevents the onion hostnames from being group readable
+  systemd.services.tor.serviceConfig.StateDirectoryMode = lib.mkForce "0710";
+  users.users.tor.homeMode = "0710";  # home mode defaults to 0700, causing readability problems, enforced by nixos "users" activation script
+
+  services.tor.settings.SafeLogging = false;  # show actual .onion names in the syslog, else debugging is impossible
+}

hosts/by-name/servo/services/ddns-afraid.nix

@@ -1,27 +0,0 @@
-{ config, lib, pkgs, ... }:
-
-# using manual ddns now
-lib.mkIf false
-{
-  systemd.services.ddns-afraid = {
-    description = "update dynamic DNS entries for freedns.afraid.org";
-    serviceConfig = {
-      EnvironmentFile = config.sops.secrets.ddns_afraid.path;
-      # TODO: ProtectSystem = "strict";
-      # TODO: ProtectHome = "full";
-      # TODO: PrivateTmp = true;
-    };
-    script = let
-      curl = "${pkgs.curl}/bin/curl -4";
-    in ''
-      ${curl} "https://freedns.afraid.org/dynamic/update.php?$AFRAID_KEY"
-    '';
-  };
-  systemd.timers.ddns-afraid = {
-    wantedBy = [ "multi-user.target" ];
-    timerConfig = {
-      OnStartupSec = "2min";
-      OnUnitActiveSec = "10min";
-    };
-  };
-}

hosts/by-name/servo/services/ddns-he.nix

@@ -1,30 +0,0 @@
-{ config, lib, pkgs, ... }:
-
-# we use manual DDNS now
-lib.mkIf false
-{
-  systemd.services.ddns-he = {
-    description = "update dynamic DNS entries for HurricaneElectric";
-    serviceConfig = {
-      EnvironmentFile = config.sops.secrets.ddns_he.path;
-      # TODO: ProtectSystem = "strict";
-      # TODO: ProtectHome = "full";
-      # TODO: PrivateTmp = true;
-    };
-    # HE DDNS API is documented: https://dns.he.net/docs.html
-    script = let
-      crl = "${pkgs.curl}/bin/curl -4";
-    in ''
-      ${crl} "https://he.uninsane.org:$HE_PASSPHRASE@dyn.dns.he.net/nic/update?hostname=he.uninsane.org"
-      ${crl} "https://native.uninsane.org:$HE_PASSPHRASE@dyn.dns.he.net/nic/update?hostname=native.uninsane.org"
-      ${crl} "https://uninsane.org:$HE_PASSPHRASE@dyn.dns.he.net/nic/update?hostname=uninsane.org"
-    '';
-  };
-  systemd.timers.ddns-he = {
-    wantedBy = [ "multi-user.target" ];
-    timerConfig = {
-      OnStartupSec = "2min";
-      OnUnitActiveSec = "10min";
-    };
-  };
-}

hosts/by-name/servo/services/default.nix

@@ -1,24 +1,31 @@
 { ... }:
 {
   imports = [
-    ./ddns-afraid.nix
-    ./ddns-he.nix
+    ./calibre.nix
+    ./coturn.nix
+    ./cryptocurrencies
+    ./email
     ./ejabberd.nix
     ./freshrss.nix
+    ./export
     ./gitea.nix
     ./goaccess.nix
     ./ipfs.nix
     ./jackett.nix
     ./jellyfin.nix
     ./kiwix-serve.nix
+    ./komga.nix
+    ./lemmy.nix
     ./matrix
     ./navidrome.nix
-    ./nixserve.nix
     ./nginx.nix
+    ./nixos-prebuild.nix
+    ./ntfy
+    ./pict-rs.nix
     ./pleroma.nix
-    ./postfix.nix
     ./postgres.nix
-    ./prosody.nix
+    ./prosody
+    ./slskd.nix
     ./transmission.nix
     ./trust-dns.nix
     ./wikipedia.nix

hosts/by-name/servo/services/ejabberd.nix

@@ -14,36 +14,111 @@
 #
 # compliance tests:
 # - <https://compliance.conversations.im/server/uninsane.org/#xep0352>
+#
+# administration:
+# - `sudo -u ejabberd ejabberdctl help`
+#
+# federation/support matrix:
+# - avatars
+#   - nixnet.services + dino: works in MUCs but not DMs  (as of 2023 H1)
+#   - movim.eu + dino: works in DMs, MUCs untested  (as of 2023/08/29)
+# - calls
+#   - local + dino: audio, video, works in DMs  (as of 2023/08/29)
+#   - movim.eu + dino: audio, video, works in DMs, no matter which side initiates  (as of 2023/08/30)
+#   - +native-cell-number@cheogram.com + dino: audio works in DMs, no matter which side initiates  (as of 2023/09/01)
+#     - can receive calls even if sender isn't in my roster
+#     - this is presumably using JMP.chat's SIP servers, which then convert it to XMPP call
+#
+# bugs:
+# - 2023/09/01: will randomly stop federating. `systemctl restart ejabberd` fixes, but takes 10 minutes.
 { config, lib, pkgs, ... }:
 
-# XXX: avatar support works in MUCs but not DMs
-# lib.mkIf false
+let
+  # TODO: this range could be larger, but right now that's costly because each element is its own UPnP forward
+  # TURN port range (inclusive)
+  turnPortLow = 49152;
+  turnPortHigh = 49167;
+  turnPortRange = lib.range turnPortLow turnPortHigh;
+in
+# XXX(2023/10/15): disabled in favor of Prosody.
+# everything configured below was fine: used ejabberd for several months.
+lib.mkIf false
 {
-  sane.persist.sys.plaintext = [
-    { user = "ejabberd"; group = "ejabberd"; directory = "/var/lib/ejabberd"; }
+  sane.persist.sys.byStore.plaintext = [
+    { user = "ejabberd"; group = "ejabberd"; path = "/var/lib/ejabberd"; method = "bind"; }
   ];
-  networking.firewall.allowedTCPPorts = [
-    3478  # STUN/TURN
-    5222  # XMPP  client -> server
-    5223  # XMPPS client -> server (XMPP over TLS)
-    5269  # XMPP  server -> server
-    5270  # XMPPS server -> server (XMPP over TLS)
-    5280  # bosh
-    5281  # bosh (https) ??
-    5349  # STUN/TURN (TLS)
-    5443  # web services (file uploads, websockets, admin)
-  ];
-  networking.firewall.allowedUDPPorts = [
-    3478  # STUN/TURN
-  ];
-  networking.firewall.allowedTCPPortRanges = [{
-    from = 49152;  # TURN
-    to = 65535;
-  }];
-  networking.firewall.allowedUDPPortRanges = [{
-    from = 49152;  # TURN
-    to = 65535;
-  }];
+  sane.ports.ports = lib.mkMerge ([
+    {
+      "3478" = {
+        protocol = [ "tcp" "udp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-xmpp-stun-turn";
+      };
+      "5222" = {
+        protocol = [ "tcp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-xmpp-client-to-server";
+      };
+      "5223" = {
+        protocol = [ "tcp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-xmpps-client-to-server";  # XMPP over TLS
+      };
+      "5269" = {
+        protocol = [ "tcp" ];
+        visibleTo.wan = true;
+        description = "colin-xmpp-server-to-server";
+      };
+      "5270" = {
+        protocol = [ "tcp" ];
+        visibleTo.wan = true;
+        description = "colin-xmpps-server-to-server";  # XMPP over TLS
+      };
+      "5280" = {
+        protocol = [ "tcp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-xmpp-bosh";
+      };
+      "5281" = {
+        protocol = [ "tcp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-xmpp-bosh-https";
+      };
+      "5349" = {
+        protocol = [ "tcp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-xmpp-stun-turn-over-tls";
+      };
+      "5443" = {
+        protocol = [ "tcp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-xmpp-web-services";  # file uploads, websockets, admin
+      };
+    }
+  ] ++ (builtins.map
+    (port: {
+      "${builtins.toString port}" = let
+        count = port - turnPortLow + 1;
+        numPorts = turnPortHigh - turnPortLow + 1;
+      in {
+        protocol = [ "tcp" "udp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-xmpp-turn-${builtins.toString count}-of-${builtins.toString numPorts}";
+      };
+    })
+    turnPortRange
+  ));
+
+  # this ejabberd config uses builtin STUN/TURN server, so hack to ensure no other implementation fights for ports
+  services.coturn.enable = false;
 
   # provide access to certs
   # TODO: this should just be `acme`. then we also add nginx to the `acme` group.
@@ -75,9 +150,9 @@
     useACMEHost = "uninsane.org";
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet = {
+  sane.dns.zones."uninsane.org".inet = {
     # XXX: SRV records have to point to something with a A/AAAA record; no CNAMEs
-    A."xmpp" =                "%NATIVE%";
+    A."xmpp" =                "%ANATIVE%";
     CNAME."muc.xmpp" =        "xmpp";
     CNAME."pubsub.xmpp" =     "xmpp";
     CNAME."upload.xmpp" =     "xmpp";
@@ -110,284 +185,285 @@
   services.ejabberd.enable = true;
   services.ejabberd.configFile = "/var/lib/ejabberd/ejabberd.yaml";
   systemd.services.ejabberd.preStart = let
-    config-in = pkgs.writeTextFile {
-      name = "ejabberd.yaml.in";
-      text = ''
-        hosts:
-          - uninsane.org
+    config-in = pkgs.writeText "ejabberd.yaml.in" (lib.generators.toYAML {} {
+      hosts = [ "uninsane.org" ];
+      # none | emergency | alert | critical | error | warning | notice | info | debug
+      loglevel = "debug";
+      acme.auto = false;
+      certfiles = [ "/var/lib/acme/uninsane.org/full.pem" ];
+      # ca_file = "${pkgs.cacert.unbundled}/etc/ssl/certs/";
+      # ca_file = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";
 
-        # none | emergency | alert | critical | error | warning | notice | info | debug
-        loglevel: debug
-        # loglevel: info
-        # loglevel: notice
+      pam_userinfotype = "jid";
+      acl = {
+        admin.user = [ "colin@uninsane.org" ];
+        local.user_regexp = "";
+        loopback.ip = [ "127.0.0.0/8" "::1/128" ];
+      };
 
-        acme:
-          auto: false
-        certfiles:
-        - /var/lib/acme/uninsane.org/full.pem
-        # ca_file: ${pkgs.cacert.unbundled}/etc/ssl/certs/
-        # ca_file: ${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt
+      access_rules = {
+        local.allow = "local";
+        c2s_access.allow = "all";
+        announce.allow = "admin";
+        configure.allow = "admin";
+        muc_create.allow = "local";
+        pubsub_createnode_access.allow = "all";
+        trusted_network.allow = "loopback";
+      };
 
-        pam_userinfotype: jid
+      # docs: <https://docs.ejabberd.im/admin/configuration/basic/#shaper-rules>
+      shaper_rules = {
+        # setting this to above 1 may break outgoing messages
+        # - maybe some servers rate limit? or just don't understand simultaneous connections?
+        max_s2s_connections = 1;
+        max_user_sessions = 10;
+        max_user_offline_messages = 5000;
+        c2s_shaper.fast = "all";
+        s2s_shaper.med = "all";
+      };
 
-        acl:
-          admin:
-            user:
-              - "colin@uninsane.org"
-          local:
-            user_regexp: ""
-          loopback:
-            ip:
-              - 127.0.0.0/8
-              - ::1/128
+      # docs: <https://docs.ejabberd.im/admin/configuration/basic/#shapers>
+      # this limits the bytes/sec.
+      # for example, burst: 3_000_000 and rate: 100_000 means:
+      # - each client has a BW budget that accumulates 100kB/sec and is capped at 3 MB
+      shaper.fast = 1000000;
+      shaper.med  =  500000;
+      # shaper.fast.rate       =  1000000;
+      # shaper.fast.burst_size = 10000000;
+      # shaper.med.rate        =   500000;
+      # shaper.med.burst_size  =  5000000;
 
-        access_rules:
-          local:
-            allow: local
-          c2s_access:
-            allow: all
-          announce:
-            allow: admin
-          configure:
-            allow: admin
-          muc_create:
-            allow: local
-          pubsub_createnode_access:
-            allow: all
-          trusted_network:
-            allow: loopback
+      # see: <https://docs.ejabberd.im/admin/configuration/listen/>
+      # s2s_use_starttls = true;
+      s2s_use_starttls = "optional";
+      # lessens 504: remote-server-timeout errors
+      # see: <https://github.com/processone/ejabberd/issues/3105#issuecomment-562182967>
+      negotiation_timeout = 60;
 
-        # docs: <https://docs.ejabberd.im/admin/configuration/basic/#shaper-rules>
-        shaper_rules:
-          # setting this to above 1 may break outgoing messages
-          # - maybe some servers rate limit? or just don't understand simultaneous connections?
-          max_s2s_connections: 1
-          max_user_sessions: 10
-          max_user_offline_messages: 5000
-          c2s_shaper:
-            fast: all
-          s2s_shaper:
-            med: all
+      listen = [
+        {
+          port = 5222;
+          module = "ejabberd_c2s";
+          shaper = "c2s_shaper";
+          starttls = true;
+          access = "c2s_access";
+        }
+        {
+          port = 5223;
+          module = "ejabberd_c2s";
+          shaper = "c2s_shaper";
+          tls = true;
+          access = "c2s_access";
+        }
+        {
+          port = 5269;
+          module = "ejabberd_s2s_in";
+          shaper = "s2s_shaper";
+        }
+        {
+          port = 5270;
+          module = "ejabberd_s2s_in";
+          shaper = "s2s_shaper";
+          tls = true;
+        }
+        {
+          port = 5443;
+          module = "ejabberd_http";
+          tls = true;
+          request_handlers = {
+            "/admin" = "ejabberd_web_admin";  # TODO: ensure this actually works
+            "/api" = "mod_http_api";  # ejabberd API endpoint (to control server)
+            "/bosh" = "mod_bosh";
+            "/upload" = "mod_http_upload";
+            "/ws" = "ejabberd_http_ws";
+            # "/.well-known/host-meta" = "mod_host_meta";
+            # "/.well-known/host-meta.json" = "mod_host_meta";
+          };
+        }
+        {
+          # STUN+TURN TCP
+          # note that the full port range should be forwarded ("not NAT'd")
+          # `use_turn=true` enables both TURN *and* STUN
+          port = 3478;
+          module = "ejabberd_stun";
+          transport = "tcp";
+          use_turn = true;
+          turn_min_port = turnPortLow;
+          turn_max_port = turnPortHigh;
+          turn_ipv4_address = "%ANATIVE%";
+        }
+        {
+          # STUN+TURN UDP
+          port = 3478;
+          module = "ejabberd_stun";
+          transport = "udp";
+          use_turn = true;
+          turn_min_port = turnPortLow;
+          turn_max_port = turnPortHigh;
+          turn_ipv4_address = "%ANATIVE%";
+        }
+        {
+          # STUN+TURN TLS over TCP
+          port = 5349;
+          module = "ejabberd_stun";
+          transport = "tcp";
+          tls = true;
+          certfile = "/var/lib/acme/uninsane.org/full.pem";
+          use_turn = true;
+          turn_min_port = turnPortLow;
+          turn_max_port = turnPortHigh;
+          turn_ipv4_address = "%ANATIVE%";
+        }
+      ];
 
-        # docs: <https://docs.ejabberd.im/admin/configuration/basic/#shapers>
-        # this limits the bytes/sec.
-        # for example, burst: 3_000_000 and rate: 100_000 means:
-        # - each client has a BW budget that accumulates 100kB/sec and is capped at 3 MB
-        shaper:
-          fast: 1000000
-          med:   500000
-        #   fast:
-        #     - rate:        1000000
-        #     - burst_size: 10000000
-        #   med:
-        #     - rate:         500000
-        #     - burst_size:  5000000
+      # TODO: enable mod_fail2ban
+      # TODO(low): look into mod_http_fileserver for serving macros?
+      modules = {
+        # mod_adhoc = {};
+        # mod_announce = {
+        #   access = "admin";
+        # };
+        # allows users to set avatars in vCard
+        # - <https://docs.ejabberd.im/admin/configuration/modules/#mod-avatar>
+        mod_avatar = {};
+        mod_caps = {};  # for mod_pubsub
+        mod_carboncopy = {};  # allows multiple clients to receive a user's message
+        # queues messages when recipient is offline, including PEP and presence messages.
+        # compliance test suggests this be enabled
+        mod_client_state = {};
 
-        # see: <https://docs.ejabberd.im/admin/configuration/listen/>
-        # s2s_use_starttls: true
-        s2s_use_starttls: optional
-        # lessens 504: remote-server-timeout errors
-        # see: <https://github.com/processone/ejabberd/issues/3105#issuecomment-562182967>
-        negotiation_timeout: 60
-
-        listen:
-          -
-            port: 5222
-            module: ejabberd_c2s
-            shaper: c2s_shaper
-            starttls: true
-            access: c2s_access
-          -
-            port: 5223
-            module: ejabberd_c2s
-            shaper: c2s_shaper
-            tls: true
-            access: c2s_access
-          -
-            port: 5269
-            module: ejabberd_s2s_in
-            shaper: s2s_shaper
-          -
-            port: 5270
-            module: ejabberd_s2s_in
-            shaper: s2s_shaper
-            tls: true
-          -
-            port: 5443
-            module: ejabberd_http
-            tls: true
-            request_handlers:
-              /admin: ejabberd_web_admin  # TODO: ensure this actually works
-              /api: mod_http_api  # ejabberd API endpoint (to control server)
-              /bosh: mod_bosh
-              /upload: mod_http_upload
-              /ws: ejabberd_http_ws
-              # /.well-known/host-meta: mod_host_meta
-              # /.well-known/host-meta.json: mod_host_meta
-          -
-            # STUN+TURN TCP
-            # note that the full port range should be forwarded ("not NAT'd")
-            # `use_turn=true` enables both TURN *and* STUN
-            port: 3478
-            module: ejabberd_stun
-            transport: tcp
-            use_turn: true
-            turn_min_port: 49152
-            turn_max_port: 65535
-            turn_ipv4_address: %NATIVE%
-          -
-            # STUN+TURN UDP
-            port: 3478
-            module: ejabberd_stun
-            transport: udp
-            use_turn: true
-            turn_min_port: 49152
-            turn_max_port: 65535
-            turn_ipv4_address: %NATIVE%
-          -
-            # STUN+TURN TLS over TCP
-            port: 5349
-            module: ejabberd_stun
-            transport: tcp
-            tls: true
-            certfile: /var/lib/acme/uninsane.org/full.pem
-            use_turn: true
-            turn_min_port: 49152
-            turn_max_port: 65535
-            turn_ipv4_address: %NATIVE%
-
-        # TODO: enable mod_fail2ban
-        # TODO(low): look into mod_http_fileserver for serving macros?
-        modules:
-          # mod_adhoc: {}
-          # mod_announce:
-          #   access: admin
-          # allows users to set avatars in vCard
-          # - <https://docs.ejabberd.im/admin/configuration/modules/#mod-avatar>
-          mod_avatar: {}
-          mod_caps: {}  # for mod_pubsub
-          mod_carboncopy: {}  # allows multiple clients to receive a user's message
-          # queues messages when recipient is offline, including PEP and presence messages.
-          # compliance test suggests this be enabled
-          mod_client_state: {}
-          # mod_conversejs: TODO: enable once on 21.12
-          # allows clients like Dino to discover where to upload files
-          mod_disco:
-            server_info:
-              -
-                modules: all
-                name: abuse-addresses
-                urls:
-                  - "mailto:admin.xmpp@uninsane.org"
-                  - "xmpp:colin@uninsane.org"
-              -
-                modules: all
-                name: admin-addresses
-                urls:
-                  - "mailto:admin.xmpp@uninsane.org"
-                  - "xmpp:colin@uninsane.org"
-          mod_http_upload:
-            host: upload.xmpp.uninsane.org
-            hosts:
-              - upload.xmpp.uninsane.org
-            put_url: "https://@HOST@:5443/upload"
-            dir_mode: "0750"
-            file_mode: "0750"
-            rm_on_unregister: false
-          # allow discoverability of BOSH and websocket endpoints
-          # TODO: enable once on ejabberd 22.05  (presently 21.04)
-          # mod_host_meta: {}
-          mod_jidprep: {}  # probably not needed: lets clients normalize jids
-          mod_last: {}  # allow other users to know when i was last online
-          mod_mam:
-            # Mnesia is limited to 2GB, better to use an SQL backend
-            # For small servers SQLite is a good fit and is very easy
-            # to configure. Uncomment this when you have SQL configured:
-            # db_type: sql
-            assume_mam_usage: true
-            default: always
-          mod_muc:
-            access:
-              - allow
-            access_admin:
-              - allow: admin
-            access_create: muc_create
-            access_persistent: muc_create
-            access_mam:
-              - allow
-            history_size: 100  # messages to show new participants
-            host: muc.xmpp.uninsane.org
-            hosts:
-              - muc.xmpp.uninsane.org
-            default_room_options:
-              anonymous: false
-              lang: en
-              persistent: true
-              mam: true
-          mod_muc_admin: {}
-          mod_offline:  # store messages for a user when they're offline (TODO: understand multi-client workflow?)
-            access_max_user_messages: max_user_offline_messages
-            store_groupchat: true
-          mod_ping: {}
-          mod_privacy: {}  # deprecated, but required for `ejabberctl export_piefxis`
-          mod_private: {}  # allow local clients to persist arbitrary data on my server
-          # push notifications to services integrated with e.g. Apple/Android.
-          # default is for a maximum amount of PII to be withheld, since these push notifs
-          # generally traverse 3rd party services. can opt to include message body, etc, though.
-          mod_push: {}
-          # i don't fully understand what this does, but it seems aimed at making push notifs more reliable.
-          mod_push_keepalive: {}
-          mod_roster:
-            versioning: true
-          # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-s2s-dialback>
-          # s2s dialback to verify inbound messages
-          # unclear to what degree the XMPP network requires this
-          mod_s2s_dialback: {}
-          mod_shared_roster: {}  # creates groups for @all, @online, and anything manually administered?
-          mod_stream_mgmt:
-            resend_on_timeout: if_offline  # resend undelivered messages if the origin client is offline
-          # fallback for when DNS-based STUN discovery is unsupported.
-          # - see: <https://xmpp.org/extensions/xep-0215.html>
-          # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-stun-disco>
-          # people say to just keep this defaulted (i guess ejabberd knows to return its `host` option of uninsane.org?)
-          mod_stun_disco: {}
-          # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-vcard>
-          mod_vcard:
-            allow_return_all: true  # all users are discoverable (?)
-            host: vjid.xmpp.uninsane.org
-            hosts:
-              - vjid.xmpp.uninsane.org
-            search: true
-          mod_vcard_xupdate: {}  # needed for avatars
-          # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-pubsub>
-          mod_pubsub:  # needed for avatars
-            access_createnode: pubsub_createnode_access
-            host: pubsub.xmpp.uninsane.org
-            hosts:
-              - pubsub.xmpp.uninsane.org
-            ignore_pep_from_offline: false
-            last_item_cache: true
-            plugins:
-              - pep
-              - flat
-            force_node_config:
-              # ensure client bookmarks are private
-              storage:bookmarks:
-                access_model: whitelist
-              urn:xmpp:avatar:data:
-                access_model: open
-              urn:xmpp:avatar:metadata:
-                access_model: open
-          mod_version: {}
-      '';
-    };
+        # mod_conversejs: TODO: enable once on 21.12
+        # allows clients like Dino to discover where to upload files
+        mod_disco.server_info = [
+          {
+            modules = "all";
+            name = "abuse-addresses";
+            urls = [
+              "mailto:admin.xmpp@uninsane.org"
+              "xmpp:colin@uninsane.org"
+            ];
+          }
+          {
+            modules = "all";
+            name = "admin-addresses";
+            urls = [
+              "mailto:admin.xmpp@uninsane.org"
+              "xmpp:colin@uninsane.org"
+            ];
+          }
+        ];
+        mod_http_upload = {
+          host = "upload.xmpp.uninsane.org";
+          hosts = [ "upload.xmpp.uninsane.org" ];
+          put_url = "https://@HOST@:5443/upload";
+          dir_mode = "0750";
+          file_mode = "0750";
+          rm_on_unregister = false;
+        };
+        # allow discoverability of BOSH and websocket endpoints
+        # TODO: enable once on ejabberd 22.05  (presently 21.04)
+        # mod_host_meta = {};
+        mod_jidprep = {};  # probably not needed: lets clients normalize jids
+        mod_last = {};  # allow other users to know when i was last online
+        mod_mam = {
+          # Mnesia is limited to 2GB, better to use an SQL backend
+          # For small servers SQLite is a good fit and is very easy
+          # to configure. Uncomment this when you have SQL configured:
+          # db_type: sql
+          assume_mam_usage = true;
+          default = "always";
+        };
+        mod_muc = {
+          access = [ "allow" ];
+          access_admin = { allow = "admin"; };
+          access_create = "muc_create";
+          access_persistent = "muc_create";
+          access_mam = [ "allow" ];
+          history_size = 100;  # messages to show new participants
+          host = "muc.xmpp.uninsane.org";
+          hosts = [ "muc.xmpp.uninsane.org" ];
+          default_room_options = {
+            anonymous = false;
+            lang = "en";
+            persistent = true;
+            mam = true;
+          };
+        };
+        mod_muc_admin = {};
+        mod_offline = {
+          # store messages for a user when they're offline (TODO: understand multi-client workflow?)
+          access_max_user_messages = "max_user_offline_messages";
+          store_groupchat = true;
+        };
+        mod_ping = {};
+        mod_privacy = {};  # deprecated, but required for `ejabberctl export_piefxis`
+        mod_private = {};  # allow local clients to persist arbitrary data on my server
+        # push notifications to services integrated with e.g. Apple/Android.
+        # default is for a maximum amount of PII to be withheld, since these push notifs
+        # generally traverse 3rd party services. can opt to include message body, etc, though.
+        mod_push = {};
+        # i don't fully understand what this does, but it seems aimed at making push notifs more reliable.
+        mod_push_keepalive = {};
+        mod_roster = {
+          versioning = true;
+        };
+        # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-s2s-dialback>
+        # s2s dialback to verify inbound messages
+        # unclear to what degree the XMPP network requires this
+        mod_s2s_dialback = {};
+        mod_shared_roster = {};  # creates groups for @all, @online, and anything manually administered?
+        mod_stream_mgmt = {
+          # resend undelivered messages if the origin client is offline
+          resend_on_timeout = "if_offline";
+        };
+        # fallback for when DNS-based STUN discovery is unsupported.
+        # - see: <https://xmpp.org/extensions/xep-0215.html>
+        # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-stun-disco>
+        # people say to just keep this defaulted (i guess ejabberd knows to return its `host` option of uninsane.org?)
+        mod_stun_disco = {};
+        # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-vcard>
+        mod_vcard = {
+          allow_return_all = true;  # all users are discoverable (?)
+          host = "vjid.xmpp.uninsane.org";
+          hosts = [ "vjid.xmpp.uninsane.org" ];
+          search = true;
+        };
+        mod_vcard_xupdate = {};  # needed for avatars
+        # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-pubsub>
+        mod_pubsub = {
+          #^ needed for avatars
+          access_createnode = "pubsub_createnode_access";
+          host = "pubsub.xmpp.uninsane.org";
+          hosts = [ "pubsub.xmpp.uninsane.org" ];
+          ignore_pep_from_offline = false;
+          last_item_cache = true;
+          plugins = [
+            "pep"
+            "flat"
+          ];
+          force_node_config = {
+            # ensure client bookmarks are private
+            "storage:bookmarks:" = {
+              "access_model" = "whitelist";
+            };
+            "urn:xmpp:avatar:data" = {
+              "access_model" = "open";
+            };
+            "urn:xmpp:avatar:metadata" = {
+              "access_model" = "open";
+            };
+          };
+        };
+        mod_version = {};
+      };
+    });
     sed = "${pkgs.gnused}/bin/sed";
   in ''
     ip=$(cat '${config.sane.services.dyn-dns.ipPath}')
     # config is 444 (not 644), so we want to write out-of-place and then atomically move
     # TODO: factor this out into `sane-woop` helper?
     rm -f /var/lib/ejabberd/ejabberd.yaml.new
-    ${sed} "s/%NATIVE%/$ip/" ${config-in} > /var/lib/ejabberd/ejabberd.yaml.new
+    ${sed} "s/%ANATIVE%/$ip/g" ${config-in} > /var/lib/ejabberd/ejabberd.yaml.new
     mv /var/lib/ejabberd/ejabberd.yaml{.new,}
   '';
 

hosts/by-name/servo/services/email/default.nix

@@ -0,0 +1,44 @@
+# nix configs to reference:
+# - <https://gitlab.com/simple-nixos-mailserver/nixos-mailserver>
+# - <https://github.com/nix-community/nur-combined/-/tree/master/repos/eh5/machines/srv-m/mail-rspamd.nix>
+#   - postfix / dovecot / rspamd / stalwart-jmap / sogo
+#
+# rspamd:
+# - nixos: <https://nixos.wiki/wiki/Rspamd>
+# - guide: <https://rspamd.com/doc/quickstart.html>
+# - non-nixos example: <https://dataswamp.org/~solene/2021-07-13-smtpd-rspamd.html>
+#
+#
+# my rough understanding of the pieces:
+# - postfix handles SMTP protocol with the rest of the world.
+# - dovecot implements IMAP protocol.
+#   - client auth (i.e. validate that user@uninsane.org is who they claim)
+#   - "folders" (INBOX, JUNK) are internal to dovecot?
+#     or where do folders live, on-disk?
+#
+# - non-local clients (i.e. me) interact with BOTH postfix and dovecot, but primarily dovecot:
+#   - mail reading is done via IMAP (so, dovecot)
+#   - mail sending is done via SMTP/submission port (so, postfix)
+#     - but postfix delegates authorization of that outgoing mail to dovecot, on the server side
+#
+# - local clients (i.e. sendmail) interact only with postfix
+#
+# debugging: general connectivity issues
+# - test that inbound port 25 is unblocked:
+#   - `curl https://canyouseeme.org/ --data 'port=25&IP=185.157.162.178' | grep 'see your service'`
+#     - and retry with port 465, 587
+#     - i think this API requires the queried IP match the source IP
+#   - if necessary, `systemctl stop postfix` and `sudo nc -l 185.157.162.178 25`, then try https://canyouseeme.org
+
+{ ... }:
+{
+  imports = [
+    ./dovecot.nix
+    ./postfix.nix
+  ];
+
+
+  #### SPAM FILTERING
+  # services.rspamd.enable = true;
+  # services.rspamd.postfix.enable = true;
+}

hosts/by-name/servo/services/email/dovecot.nix

@@ -0,0 +1,145 @@
+# dovecot config options: <https://doc.dovecot.org/configuration_manual/>
+#
+# sieve docs:
+# - sieve language examples: <https://doc.dovecot.org/configuration_manual/sieve/examples/>
+# - sieve protocol/language: <https://proton.me/support/sieve-advanced-custom-filters>
+
+{ config, lib, pkgs, ... }:
+{
+  sane.ports.ports."143" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-imap-imap.uninsane.org";
+  };
+  sane.ports.ports."993" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-imaps-imap.uninsane.org";
+  };
+
+  # exists only to manage certs for dovecot
+  services.nginx.virtualHosts."imap.uninsane.org" = {
+    enableACME = true;
+  };
+
+  sane.dns.zones."uninsane.org".inet = {
+    CNAME."imap" = "native";
+  };
+
+  sops.secrets."dovecot_passwd" = {
+    owner = config.users.users.dovecot2.name;
+    # TODO: debug why mail can't be sent without this being world-readable
+    mode = "0444";
+  };
+
+  # inspired by https://gitlab.com/simple-nixos-mailserver/nixos-mailserver/
+  services.dovecot2.enable = true;
+  # services.dovecot2.enableLmtp = true;
+  services.dovecot2.sslServerCert = "/var/lib/acme/imap.uninsane.org/fullchain.pem";
+  services.dovecot2.sslServerKey = "/var/lib/acme/imap.uninsane.org/key.pem";
+  services.dovecot2.enablePAM = false;
+
+  # sieve scripts require me to set a user for... idk why?
+  services.dovecot2.mailUser = "colin";
+  services.dovecot2.mailGroup = "users";
+  users.users.colin.isSystemUser = lib.mkForce false;
+
+  services.dovecot2.extraConfig =
+  let
+    passwdFile = config.sops.secrets.dovecot_passwd.path;
+  in
+    ''
+    passdb {
+      driver = passwd-file
+      args = ${passwdFile}
+    }
+    userdb {
+      driver = passwd-file
+      args = ${passwdFile}
+    }
+
+    # allow postfix to query our auth db
+    service auth {
+      unix_listener auth {
+        mode = 0660
+        user = postfix
+        group = postfix
+      }
+    }
+    auth_mechanisms = plain login
+
+    # accept incoming messaging from postfix
+    # service lmtp {
+    #   unix_listener dovecot-lmtp {
+    #     mode = 0600
+    #     user = postfix
+    #     group = postfix
+    #   }
+    # }
+
+    # plugin {
+    #   sieve_plugins = sieve_imapsieve
+    # }
+
+    mail_debug = yes
+    auth_debug = yes
+    # verbose_ssl = yes
+  '';
+
+  services.dovecot2.mailboxes = {
+    # special-purpose mailboxes: "All" "Archive" "Drafts" "Flagged" "Junk" "Sent" "Trash"
+    # RFC6154 describes these special mailboxes: https://www.ietf.org/rfc/rfc6154.html
+    # how these boxes are treated is 100% up to the client and server to decide.
+    # client behavior:
+    # iOS
+    #   - Drafts: ?
+    #   - Sent: works
+    #   - Trash: works
+    #   - Junk: works ("mark" -> "move to Junk")
+    # aerc
+    #   - Drafts: works
+    #   - Sent: works
+    #   - Trash: no; deleted messages are actually deleted
+    #       use `:move trash` instead
+    #   - Junk: ?
+    # Sent mailbox: all sent messages are copied to it. unclear if this happens server-side or client-side.
+    Drafts = { specialUse = "Drafts"; auto = "create"; };
+    Sent = { specialUse = "Sent"; auto = "create"; };
+    Trash = { specialUse = "Trash"; auto = "create"; };
+    Junk = { specialUse = "Junk"; auto = "create"; };
+  };
+
+  services.dovecot2.mailPlugins = {
+    perProtocol = {
+      # imap.enable = [
+      #   "imap_sieve"
+      # ];
+      lda.enable = [
+        "sieve"
+      ];
+      # lmtp.enable = [
+      #   "sieve"
+      # ];
+    };
+  };
+  services.dovecot2.modules = [
+    pkgs.dovecot_pigeonhole  # enables sieve execution (?)
+  ];
+  services.dovecot2.sieve = {
+    extensions = [ "fileinto" ];
+    # if any messages fail to pass (or lack) DKIM, move them to Junk
+    # XXX the key name ("after") is only used to order sieve execution/ordering
+    scripts.after = builtins.toFile "ensuredkim.sieve" ''
+      require "fileinto";
+
+      if not header :contains "Authentication-Results" "dkim=pass" {
+        fileinto "Junk";
+        stop;
+      }
+    '';
+  };
+
+  systemd.services.dovecot2.serviceConfig.RestartSec = lib.mkForce "15s";  # nixos defaults this to 1s
+}

hosts/by-name/servo/services/email/postfix.nix

@@ -1,7 +1,6 @@
-# DOCS:
-# - dovecot config: <https://doc.dovecot.org/configuration_manual/>
+# postfix config options: <https://www.postfix.org/postconf.5.html>
 
-{ config, lib, ... }:
+{ lib, pkgs, ... }:
 
 let
   submissionOptions = {
@@ -19,39 +18,46 @@ let
   };
 in
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
-    { user = "opendkim"; group = "opendkim"; directory = "/var/lib/opendkim"; }
-    { user = "root"; group = "root"; directory = "/var/lib/postfix"; }
-    { user = "root"; group = "root"; directory = "/var/spool/mail"; }
+    { user = "opendkim"; group = "opendkim"; path = "/var/lib/opendkim"; method = "bind"; }
+    { user = "root"; group = "root"; path = "/var/lib/postfix"; method = "bind"; }
+    { user = "root"; group = "root"; path = "/var/spool/mail"; method = "bind"; }
     # *probably* don't need these dirs:
     # "/var/lib/dhparams"          # https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/security/dhparams.nix
     # "/var/lib/dovecot"
   ];
 
-  networking.firewall.allowedTCPPorts = [
-    25   # SMTP
-    143  # IMAP
-    465  # SMTPS
-    587  # SMTPS/submission
-    993  # IMAPS
-  ];
+  # XXX(2023/10/20): opening these ports in the firewall has the OPPOSITE effect as intended.
+  # these ports are only routable so long as they AREN'T opened.
+  # probably some cursed interaction with network namespaces introduced after 2023/10/10.
+  # sane.ports.ports."25" = {
+  #   protocol = [ "tcp" ];
+  #   # XXX visibleTo.lan effectively means "open firewall, but don't configure any NAT/forwarding"
+  #   visibleTo.lan = true;
+  #   description = "colin-smtp-mx.uninsane.org";
+  # };
+  # sane.ports.ports."465" = {
+  #   protocol = [ "tcp" ];
+  #   visibleTo.lan = true;
+  #   description = "colin-smtps-mx.uninsane.org";
+  # };
+  # sane.ports.ports."587" = {
+  #   protocol = [ "tcp" ];
+  #   visibleTo.lan = true;
+  #   description = "colin-smtps-submission-mx.uninsane.org";
+  # };
 
-  # exists only to manage certs for dovecot
-  services.nginx.virtualHosts."imap.uninsane.org" = {
-    enableACME = true;
-  };
   # exists only to manage certs for Postfix
   services.nginx.virtualHosts."mx.uninsane.org" = {
     enableACME = true;
   };
 
 
-  sane.services.trust-dns.zones."uninsane.org".inet = {
+  sane.dns.zones."uninsane.org".inet = {
     MX."@" = "10 mx.uninsane.org.";
     # XXX: RFC's specify that the MX record CANNOT BE A CNAME
     A."mx" = "185.157.162.178";
-    CNAME."imap" = "native";
 
     # Sender Policy Framework:
     #   +mx     => mail passes if it originated from the MX
@@ -62,7 +68,7 @@ in
 
     # DKIM public key:
     TXT."mx._domainkey" =
-      "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCkSyMufc2KrRx3j17e/LyB+3eYSBRuEFT8PUka8EDX04QzCwDPdkwgnj3GNDvnB5Ktb05Cf2SJ/S1OLqNsINxJRWtkVfZd/C339KNh9wrukMKRKNELL9HLUw0bczOI4gKKFqyrRE9qm+4csCMAR79Te9FCjGV/jVnrkLdPT0GtFwIDAQAB"
+      "v=DKIM1;k=rsa;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCkSyMufc2KrRx3j17e/LyB+3eYSBRuEFT8PUka8EDX04QzCwDPdkwgnj3GNDvnB5Ktb05Cf2SJ/S1OLqNsINxJRWtkVfZd/C339KNh9wrukMKRKNELL9HLUw0bczOI4gKKFqyrRE9qm+4csCMAR79Te9FCjGV/jVnrkLdPT0GtFwIDAQAB"
     ;
 
     # DMARC fields <https://datatracker.ietf.org/doc/html/rfc7489>:
@@ -95,18 +101,40 @@ in
     @uninsane.org colin
   '';
 
-  services.postfix.extraConfig = ''
+  services.postfix.config = {
     # smtpd_milters = local:/run/opendkim/opendkim.sock
     # milter docs: http://www.postfix.org/MILTER_README.html
-    # mail filters for receiving email and authorized SMTP clients
+    # mail filters for receiving email and from authorized SMTP clients (i.e. via submission)
     # smtpd_milters = inet:185.157.162.190:8891
-    smtpd_milters = unix:/run/opendkim/opendkim.sock
+    # opendkim.sock will add a Authentication-Results header, with `dkim=pass|fail|...` value to received messages
+    smtpd_milters = "unix:/run/opendkim/opendkim.sock";
     # mail filters for sendmail
-    non_smtpd_milters = $smtpd_milters
-    milter_default_action = accept
-    inet_protocols = ipv4
-    smtp_tls_security_level = may
-  '';
+    non_smtpd_milters = "$smtpd_milters";
+
+    # what to do when a milter exits unexpectedly:
+    milter_default_action = "accept";
+
+    inet_protocols = "ipv4";
+    smtp_tls_security_level = "may";
+
+    # hand received mail over to dovecot so that it can run sieves & such
+    mailbox_command = ''${pkgs.dovecot}/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"'';
+
+    # hand received mail over to dovecot
+    # virtual_alias_maps = [
+    #   "hash:/etc/postfix/virtual"
+    # ];
+    # mydestination = "";
+    # virtual_mailbox_domains = [ "localhost" "uninsane.org" ];
+    # # virtual_mailbox_maps = "hash:/etc/postfix/virtual";
+    # virtual_transport = "lmtp:unix:/run/dovecot2/dovecot-lmtp";
+
+    # anti-spam options: <https://www.postfix.org/SMTPD_ACCESS_README.html>
+    # reject_unknown_sender_domain: causes postfix to `dig <sender> MX` and make sure that exists.
+    # but may cause problems receiving mail from google & others who load-balance?
+    # - <https://unix.stackexchange.com/questions/592131/how-to-reject-email-from-unknown-domains-with-postfix-on-centos>
+    # smtpd_sender_restrictions = reject_unknown_sender_domain
+  };
 
   services.postfix.enableSubmission = true;
   services.postfix.submissionOptions = submissionOptions;
@@ -121,6 +149,8 @@ in
   };
 
 
+  #### OPENDKIM
+
   services.opendkim.enable = true;
   # services.opendkim.domains = "csl:uninsane.org";
   services.opendkim.domains = "uninsane.org";
@@ -144,59 +174,6 @@ in
     UMask = lib.mkForce "0011";
   };
 
-  # inspired by https://gitlab.com/simple-nixos-mailserver/nixos-mailserver/
-  services.dovecot2.enable = true;
-  services.dovecot2.mailboxes = {
-    # special-purpose mailboxes: "All" "Archive" "Drafts" "Flagged" "Junk" "Sent" "Trash"
-    # RFC6154 describes these special mailboxes: https://www.ietf.org/rfc/rfc6154.html
-    # how these boxes are treated is 100% up to the client and server to decide.
-    # client behavior:
-    # iOS
-    #   - Drafts: ?
-    #   - Sent: works
-    #   - Trash: works
-    # aerc
-    #   - Drafts: works
-    #   - Sent: works
-    #   - Trash: no; deleted messages are actually deleted
-    #       use `:move trash` instead
-    # Sent mailbox: all sent messages are copied to it. unclear if this happens server-side or client-side.
-    Drafts = { specialUse = "Drafts"; auto = "create"; };
-    Sent = { specialUse = "Sent"; auto = "create"; };
-    Trash = { specialUse = "Trash"; auto = "create"; };
-  };
-  services.dovecot2.sslServerCert = "/var/lib/acme/imap.uninsane.org/fullchain.pem";
-  services.dovecot2.sslServerKey = "/var/lib/acme/imap.uninsane.org/key.pem";
-  services.dovecot2.enablePAM = false;
-  services.dovecot2.extraConfig =
-  let
-    passwdFile = config.sops.secrets.dovecot_passwd.path;
-  in
-    ''
-    passdb {
-      driver = passwd-file
-      args = ${passwdFile}
-    }
-    userdb {
-      driver = passwd-file
-      args = ${passwdFile}
-    }
-
-    # allow postfix to query our auth db
-    service auth {
-      unix_listener auth {
-        mode = 0660
-        user = postfix
-        group = postfix
-      }
-    }
-    auth_mechanisms = plain login
-
-
-    mail_debug = yes
-    auth_debug = yes
-    # verbose_ssl = yes
-  '';
 
   #### OUTGOING MESSAGE REWRITING:
   services.postfix.enableHeaderChecks = true;
@@ -218,10 +195,4 @@ in
     #   pattern = "/^Subject:.*activate your account/";
     # }
   ];
-
-  sops.secrets."dovecot_passwd" = {
-    owner = config.users.users.dovecot2.name;
-    # TODO: debug why mail can't be sent without this being world-readable
-    mode = "0444";
-  };
 }

hosts/by-name/servo/services/export/default.nix

@@ -0,0 +1,53 @@
+{ config, ... }:
+{
+  imports = [
+    ./nfs.nix
+    ./sftpgo
+  ];
+
+  users.groups.export = {};
+
+  fileSystems."/var/export/media" = {
+    # everything in here could be considered publicly readable (based on the viewer's legal jurisdiction)
+    device = "/var/media";
+    options = [ "rbind" ];
+  };
+  # fileSystems."/var/export/playground" = {
+  #   device = config.fileSystems."/mnt/persist/ext".device;
+  #   fsType = "btrfs";
+  #   options = [
+  #     "subvol=export-playground"
+  #     "compress=zstd"
+  #     "defaults"
+  #   ];
+  # };
+  # N.B.: the backing directory should be manually created here **as a btrfs subvolume** and with a quota.
+  # - `sudo btrfs subvolume create /mnt/persist/ext/persist/var/export/playground`
+  # - `sudo btrfs quota enable /mnt/persist/ext/persist/var/export/playground`
+  # - `sudo btrfs quota rescan -sw /mnt/persist/ext/persist/var/export/playground`
+  # to adjust the limits (which apply at the block layer, i.e. post-compression):
+  # - `sudo btrfs qgroup limit 20G /mnt/persist/ext/persist/var/export/playground`
+  # to query the quota/status:
+  # - `sudo btrfs qgroup show -re /var/export/playground`
+  sane.persist.sys.byStore.ext = [
+    { user = "root"; group = "export"; mode = "0775"; path = "/var/export/playground"; method = "bind"; }
+  ];
+
+  sane.fs."/var/export/README.md" = {
+    wantedBy = [ "nfs.service" "sftpgo.service" ];
+    file.text = ''
+      - media/         read-only:  Videos, Music, Books, etc
+      - playground/    read-write: use it to share files with other users of this server
+    '';
+  };
+
+  sane.fs."/var/export/playground/README.md" = {
+    wantedBy = [ "nfs.service" "sftpgo.service" ];
+    file.text = ''
+      this directory is intentionally read+write by anyone with access (i.e. on the LAN).
+      - share files
+      - write poetry
+      - be a friendly troll
+    '';
+  };
+}

hosts/by-name/servo/services/export/nfs.nix

@@ -0,0 +1,135 @@
+# docs:
+# - <https://nixos.wiki/wiki/NFS>
+# - <https://wiki.gentoo.org/wiki/Nfs-utils>
+# system files:
+# - /etc/exports
+# system services:
+# - nfs-server.service
+# - nfs-idmapd.service
+# - nfs-mountd.service
+# - nfsdcld.service
+# - rpc-statd.service
+# - rpcbind.service
+#
+# TODO: force files to be 755, or 750.
+# - could maybe be done with some mount option?
+
+{ config, lib, ... }:
+lib.mkIf false  #< TODO: remove nfs altogether! it's not exactly the most secure
+{
+  services.nfs.server.enable = true;
+
+  # see which ports NFS uses with:
+  # - `rpcinfo -p`
+  sane.ports.ports."111" = {
+    protocol = [ "tcp" "udp" ];
+    visibleTo.lan = true;
+    description = "NFS server portmapper";
+  };
+  sane.ports.ports."2049" = {
+    protocol = [ "tcp" "udp" ];
+    visibleTo.lan = true;
+    description = "NFS server";
+  };
+  sane.ports.ports."4000" = {
+    protocol = [ "udp" ];
+    visibleTo.lan = true;
+    description = "NFS server status daemon";
+  };
+  sane.ports.ports."4001" = {
+    protocol = [ "tcp" "udp" ];
+    visibleTo.lan = true;
+    description = "NFS server lock daemon";
+  };
+  sane.ports.ports."4002" = {
+    protocol = [ "tcp" "udp" ];
+    visibleTo.lan = true;
+    description = "NFS server mount daemon";
+  };
+
+  # NFS4 allows these to float, but NFS3 mandates specific ports, so fix them for backwards compat.
+  services.nfs.server.lockdPort = 4001;
+  services.nfs.server.mountdPort = 4002;
+  services.nfs.server.statdPort = 4000;
+
+  services.nfs.extraConfig = ''
+    [nfsd]
+    # XXX: NFS over UDP REQUIRES SPECIAL CONFIG TO AVOID DATA LOSS.
+    # see `man 5 nfs`: "Using NFS over UDP on high-speed links".
+    # it's actually just a general property of UDP over IPv4 (IPv6 fixes it).
+    # both the client and the server should configure a shorter-than-default IPv4 fragment reassembly window to mitigate.
+    # OTOH, tunneling NFS over Wireguard also bypasses this weakness, because a mis-assembled packet would not have a valid signature.
+    udp=y
+
+    [exports]
+    # all export paths are relative to rootdir.
+    # for NFSv4, the export with fsid=0 behaves as `/` publicly,
+    # but NFSv3 implements no such feature.
+    # using `rootdir` instead of relying on `fsid=0` allows consistent export paths regardless of NFS proto version
+    rootdir=/var/export
+  '';
+
+  # format:
+  #   fspoint	visibility(options)
+  # options:
+  # - see: <https://wiki.gentoo.org/wiki/Nfs-utils#Exports>
+  # - see [man 5 exports](https://linux.die.net/man/5/exports)
+  # - insecure:  require clients use src port > 1024
+  # - rw, ro (default)
+  # - async, sync (default)
+  # - no_subtree_check (default), subtree_check: verify not just that files requested by the client live
+  #     in the expected fs, but also that they live under whatever subdirectory of that fs is exported.
+  # - no_root_squash, root_squash (default): map requests from uid 0 to user `nobody`.
+  # - crossmnt:  reveal filesystems that are mounted under this endpoint
+  # - fsid:  must be zero for the root export
+  #   - fsid=root  is alias for fsid=0
+  # - mountpoint[=/path]:  only export the directory if it's a mountpoint. used to avoid exporting failed mounts.
+  # - all_squash: rewrite all client requests such that they come from anonuid/anongid
+  #   - any files a user creates are owned by local anonuid/anongid.
+  #   - users can read any local file which anonuid/anongid would be able to read.
+  #   - users can't chown to/away from anonuid/anongid.
+  #   - users can chmod files they own, to anything (making them unreadable to non-`nfsuser` export users, like FTP).
+  #   - `stat` remains unchanged, returning the real UIDs/GIDs to the client.
+  #     - thus programs which check `uid` or `gid` before trying an operation may incorrectly conclude they can't perform some op.
+  #
+  # 10.0.0.0/8 to export both to LAN (readonly, unencrypted) and wg vpn (read-write, encrypted)
+  services.nfs.server.exports =
+  let
+    fmtExport = { export, baseOpts, extraLanOpts ? [], extraVpnOpts ? [] }:
+      let
+        always = [ "subtree_check" ];
+        lanOpts = always ++ baseOpts ++ extraLanOpts;
+        vpnOpts = always ++ baseOpts ++ extraVpnOpts;
+      in "${export} 10.78.79.0/22(${lib.concatStringsSep "," lanOpts}) 10.0.10.0/24(${lib.concatStringsSep "," vpnOpts})";
+  in lib.concatStringsSep "\n" [
+    (fmtExport {
+      export = "/";
+      baseOpts = [ "crossmnt" "fsid=root" ];
+      extraLanOpts = [ "ro" ];
+      extraVpnOpts = [ "rw" "no_root_squash" ];
+    })
+    (fmtExport {
+      # provide /media as an explicit export. NFSv4 can transparently mount a subdir of an export, but NFSv3 can only mount paths which are exports.
+      export = "/media";
+      baseOpts = [ "crossmnt" ];  # TODO: is crossmnt needed here?
+      extraLanOpts = [ "ro" ];
+      extraVpnOpts = [ "rw" "no_root_squash" ];
+    })
+    (fmtExport {
+      export = "/playground";
+      baseOpts = [
+        "mountpoint"
+        "all_squash"
+        "rw"
+        "anonuid=${builtins.toString config.users.users.nfsuser.uid}"
+        "anongid=${builtins.toString config.users.groups.export.gid}"
+      ];
+    })
+  ];
+
+  users.users.nfsuser = {
+    description = "virtual user for anonymous NFS operations";
+    group = "export";
+    isSystemUser = true;
+  };
+}

hosts/by-name/servo/services/export/sftpgo/default.nix

@@ -0,0 +1,158 @@
+# docs:
+# - <https://github.com/drakkan/sftpgo>
+# - config options: <https://github.com/drakkan/sftpgo/blob/main/docs/full-configuration.md>
+# - config defaults: <https://github.com/drakkan/sftpgo/blob/main/sftpgo.json>
+# - nixos options: <repo:nixos/nixpkgs:nixos/modules/services/web-apps/sftpgo.nix>
+# - nixos example: <repo:nixos/nixpkgs:nixos/tests/sftpgo.nix>
+#
+# sftpgo is a FTP server that also supports WebDAV, SFTP, and web clients.
+
+{ config, lib, pkgs, sane-lib, ... }:
+let
+  external_auth_hook = pkgs.static-nix-shell.mkPython3Bin {
+    pname = "external_auth_hook";
+    srcRoot = ./.;
+    pyPkgs = [ "passlib" ];
+  };
+  # Client initiates a FTP "control connection" on port 21.
+  # - this handles the client -> server commands, and the server -> client status, but not the actual data
+  # - file data, directory listings, etc need to be transferred on an ephemeral "data port".
+  # - 50000-50100 is a common port range for this.
+  #   50000 is used by soulseek.
+  passiveStart = 50050;
+  passiveEnd   = 50070;
+in
+{
+  sane.ports.ports = {
+    "21" = {
+      protocol = [ "tcp" ];
+      visibleTo.lan = true;
+      # visibleTo.wan = true;
+      description = "colin-FTP server";
+    };
+    "990" = {
+      protocol = [ "tcp" ];
+      visibleTo.lan = true;
+      visibleTo.wan = true;
+      description = "colin-FTPS server";
+    };
+  } // (sane-lib.mapToAttrs
+    (port: {
+      name = builtins.toString port;
+      value = {
+        protocol = [ "tcp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-FTP server data port range";
+      };
+    })
+    (lib.range passiveStart passiveEnd)
+  );
+
+  # use nginx/acme to produce a cert for FTPS
+  services.nginx.virtualHosts."ftp.uninsane.org" = {
+    addSSL = true;
+    enableACME = true;
+  };
+  sane.dns.zones."uninsane.org".inet.CNAME."ftp" = "native";
+
+  services.sftpgo = {
+    enable = true;
+    group = "export";
+
+    package = pkgs.sftpgo.overrideAttrs (upstream: {
+      patches = (upstream.patches or []) ++ [
+        # fix for compatibility with kodi:
+        # ftp LIST operation returns entries over-the-wire like:
+        # - dgrwxrwxr-x 1 ftp ftp            9 Apr  9 15:05 Videos
+        # however not all clients understand all mode bits (like that `g`, indicating SGID / group sticky bit).
+        # instead, only send mode bits which are well-understood.
+        # the full set of bits, from which i filter, is found here: <https://pkg.go.dev/io/fs#FileMode>
+        ./safe_fileinfo.patch
+      ];
+    });
+
+    settings = {
+      ftpd = {
+        bindings = [
+          {
+            # binding this means any wireguard client can connect
+            address = "10.0.10.5";
+            port = 21;
+            debug = true;
+          }
+          {
+            # binding this means any LAN client can connect (also WAN traffic forwarded from the gateway)
+            address = "10.78.79.51";
+            port = 21;
+            debug = true;
+          }
+          {
+            # binding this means any wireguard client can connect
+            address = "10.0.10.5";
+            port = 990;
+            debug = true;
+            tls_mode = 2;  # 2 = "implicit FTPS": client negotiates TLS before any FTP command.
+          }
+          {
+            # binding this means any LAN client can connect (also WAN traffic forwarded from the gateway)
+            address = "10.78.79.51";
+            port = 990;
+            debug = true;
+            tls_mode = 2;  # 2 = "implicit FTPS": client negotiates TLS before any FTP command.
+          }
+        ];
+
+        # active mode is susceptible to "bounce attacks", without much benefit over passive mode
+        disable_active_mode = true;
+        hash_support = true;
+        passive_port_range = {
+          start = passiveStart;
+          end = passiveEnd;
+        };
+
+        certificate_file = "/var/lib/acme/ftp.uninsane.org/full.pem";
+        certificate_key_file = "/var/lib/acme/ftp.uninsane.org/key.pem";
+
+        banner = ''
+          Welcome, friends, to Colin's FTP server! Also available via NFS on the same host, but LAN-only.
+
+          Read-only access (LAN-restricted):
+          Username: "anonymous"
+          Password: "anonymous"
+
+          CONFIGURE YOUR CLIENT FOR "PASSIVE" MODE, e.g. `ftp --passive ftp.uninsane.org`.
+          Please let me know if anything's broken or not as it should be. Otherwise, browse and transfer freely :)
+        '';
+
+      };
+      data_provider = {
+        driver = "memory";
+        external_auth_hook = "${external_auth_hook}/bin/external_auth_hook";
+        # track_quota:
+        # - 0: disable quota tracking
+        # - 1: quota is updated on every upload/delete, even if user has no quota restriction
+        # - 2: quota is updated on every upload/delete, but only if user/folder has a quota restriction  (default, i think)
+        # track_quota = 2;
+      };
+    };
+  };
+
+  users.users.sftpgo.extraGroups = [
+    "export"
+    "media"
+    "nginx"  # to access certs
+  ];
+
+  systemd.services.sftpgo = {
+    after = [ "network-online.target" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      ReadWritePaths = [ "/var/export" ];
+
+      Restart = "always";
+      RestartSec = "20s";
+      UMask = lib.mkForce "0002";
+    };
+  };
+}

hosts/by-name/servo/services/export/sftpgo/external_auth_hook

@@ -0,0 +1,155 @@
+#!/usr/bin/env nix-shell
+#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.passlib ])"
+# vim: set filetype=python :
+#
+# available environment variables:
+# - SFTPGO_AUTHD_USERNAME
+# - SFTPGO_AUTHD_USER
+# - SFTPGO_AUTHD_IP
+# - SFTPGO_AUTHD_PROTOCOL  = { "DAV", "FTP", "HTTP", "SSH" }
+# - SFTPGO_AUTHD_PASSWORD
+# - SFTPGO_AUTHD_PUBLIC_KEY
+# - SFTPGO_AUTHD_KEYBOARD_INTERACTIVE
+# - SFTPGO_AUTHD_TLS_CERT
+#
+# user permissions:
+# - see <repo:drakkan/sftpgo:internal/dataprovider/user.go>
+# - "*" = grant all permissions
+# - read-only perms:
+#   - "list" = list files and directories
+#   - "download"
+# - rw perms:
+#   - "upload"
+#   - "overwrite" = allow uploads to replace existing files
+#   - "delete" = delete files and directories
+#     - "delete_files"
+#     - "delete_dirs"
+#   - "rename" = rename files and directories
+#     - "rename_files"
+#     - "rename_dirs"
+#   - "create_dirs"
+#   - "create_symlinks"
+#   - "chmod"
+#   - "chown"
+#   - "chtimes" = change atime/mtime (access and modification times)
+#
+# home_dir:
+# - it seems (empirically) that a user can't cd above their home directory.
+#   though i don't have a reference for that in the docs.
+
+import json
+import os
+import passlib.hosts
+
+from hmac import compare_digest
+
+authFail = dict(username="")
+
+PERM_RO = [ "list", "download" ]
+PERM_RW = [
+  # read-only:
+  "list",
+  "download",
+  # write:
+  "upload",
+  "overwrite",
+  "delete",
+  "rename",
+  "create_dirs",
+  "create_symlinks",
+  # intentionally omitted:
+  # "chmod",
+  # "chown",
+  # "chtimes",
+]
+
+TRUSTED_CREDS = [
+  # /etc/shadow style creds.
+  # mkpasswd -m sha-512
+  # $<method>$<salt>$<hash>
+  "$6$Zq3c2u4ghUH4S6EP$pOuRt13sEKfX31OqPbbd1LuhS21C9MICMc94iRdTAgdAcJ9h95gQH/6Jf6Ie4Obb0oxQtojRJ1Pd/9QHOlFMW."  #< m. rocket boy
+]
+
+def mkAuthOk(username: str, permissions: dict[str, list[str]]) -> dict:
+  return dict(
+    status = 1,
+    username = username,
+    expiration_date = 0,
+    home_dir = "/var/export",
+    # uid/gid 0 means to inherit sftpgo uid.
+    # - i.e. users can't read files which Linux user `sftpgo` can't read
+    # - uploaded files belong to Linux user `sftpgo`
+    # other uid/gid values aren't possible for localfs backend, unless i let sftpgo use `sudo`.
+    uid = 0,
+    gid = 0,
+    # uid = 65534,
+    # gid = 65534,
+    max_sessions = 0,
+    # quota_*: 0 means to not use SFTP's quota system
+    quota_size = 0,
+    quota_files = 0,
+    permissions = permissions,
+    upload_bandwidth = 0,
+    download_bandwidth = 0,
+    filters = dict(
+      allowed_ip = [],
+      denied_ip = [],
+    ),
+    public_keys = [],
+    # other fields:
+    # ? groups
+    # ? virtual_folders
+  )
+
+def isLan(ip: str) -> bool:
+  return ip.startswith("10.78.76.") \
+    or ip.startswith("10.78.77.") \
+    or ip.startswith("10.78.78.") \
+    or ip.startswith("10.78.79.")
+
+def isWireguard(ip: str) -> bool:
+  return ip.startswith("10.0.10.")
+
+def isTrustedCred(password: str) -> bool:
+  for cred in TRUSTED_CREDS:
+    if passlib.hosts.linux_context.verify(password, cred):
+      return True
+
+  return False
+
+def getAuthResponse(ip: str, username: str, password: str) -> dict:
+  """
+  return a sftpgo auth response either denying the user or approving them
+  with a set of permissions.
+  """
+  if isTrustedCred(password) and username != "colin":
+    # allow r/w access from those with a special token
+    return mkAuthOk(username, permissions = {
+      "/": PERM_RW,
+      "/playground": PERM_RW,
+    })
+  if isWireguard(ip):
+    # allow any user from wireguard
+    return mkAuthOk(username, permissions = {
+      "/": PERM_RW,
+      "/playground": PERM_RW,
+    })
+  if isLan(ip):
+    if username == "anonymous":
+      # allow anonymous users on the LAN
+      return mkAuthOk("anonymous", permissions = {
+        "/": PERM_RO,
+        "/playground": PERM_RW,
+      })
+
+  return authFail
+
+def main():
+  ip = os.environ.get("SFTPGO_AUTHD_IP", "")
+  username = os.environ.get("SFTPGO_AUTHD_USERNAME", "")
+  password = os.environ.get("SFTPGO_AUTHD_PASSWORD", "")
+  resp = getAuthResponse(ip, username, password)
+  print(json.dumps(resp))
+
+if __name__ == "__main__":
+  main()

hosts/by-name/servo/services/export/sftpgo/safe_fileinfo.patch

@@ -0,0 +1,32 @@
+diff --git a/internal/ftpd/handler.go b/internal/ftpd/handler.go
+index 036c3977..33211261 100644
+--- a/internal/ftpd/handler.go
++++ b/internal/ftpd/handler.go
+@@ -169,7 +169,7 @@ func (c *Connection) Stat(name string) (os.FileInfo, error) {
+ 		}
+ 		return nil, err
+ 	}
+-	return fi, nil
++	return vfs.NewFileInfo(name, fi.IsDir(), fi.Size(), fi.ModTime(), false), nil
+ }
+
+ // Name returns the name of this connection
+@@ -315,7 +315,17 @@ func (c *Connection) ReadDir(name string) (ftpserver.DirLister, error) {
+ 		}, nil
+ 	}
+
+-	return c.ListDir(name)
++	lister, err := c.ListDir(name)
++	if err != nil {
++		return nil, err
++	}
++	return &patternDirLister{
++		DirLister:      lister,
++		pattern:        "*",
++		lastCommand:    c.clientContext.GetLastCommand(),
++		dirName:        name,
++		connectionPath: c.clientContext.Path(),
++	}, nil
+ }
+
+ // GetHandle implements ClientDriverExtentionFileTransfer

hosts/by-name/servo/services/freshrss.nix

@@ -15,8 +15,8 @@
     owner = config.users.users.freshrss.name;
     mode = "0400";
   };
-  sane.persist.sys.plaintext = [
-    { user = "freshrss"; group = "freshrss"; directory = "/var/lib/freshrss"; }
+  sane.persist.sys.byStore.plaintext = [
+    { user = "freshrss"; group = "freshrss"; path = "/var/lib/freshrss"; method = "bind"; }
   ];
 
   services.freshrss.enable = true;
@@ -59,5 +59,5 @@
     # the routing is handled by services.freshrss.virtualHost
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."rss" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."rss" = "native";
 }

hosts/by-name/servo/services/gitea.nix

@@ -1,20 +1,22 @@
+# config options: <https://docs.gitea.io/en-us/administration/config-cheat-sheet/>
 { config, pkgs, lib, ... }:
 
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
-    { user = "git"; group = "gitea"; directory = "/var/lib/gitea"; }
+    { user = "git"; group = "gitea"; path = "/var/lib/gitea"; method = "bind"; }
   ];
   services.gitea.enable = true;
   services.gitea.user = "git";  # default is 'gitea'
   services.gitea.database.type = "postgres";
   services.gitea.database.user = "git";
   services.gitea.appName = "Perfectly Sane Git";
-  services.gitea.domain = "git.uninsane.org";
-  services.gitea.rootUrl = "https://git.uninsane.org/";
-  services.gitea.settings.session.COOKIE_SECURE = true;
   # services.gitea.disableRegistration = true;
 
+  services.gitea.database.createDatabase = false;  #< silence warning which wants db user and name to be equal
+  # TODO: remove this after merge: <https://github.com/NixOS/nixpkgs/pull/268849>
+  services.gitea.database.socket = "/run/postgresql";  #< would have been set if createDatabase = true
+
   # gitea doesn't create the git user
   users.users.git = {
     description = "Gitea Service";
@@ -27,9 +29,13 @@
   };
 
   services.gitea.settings = {
+    # options: "Trace", "Debug", "Info", "Warn", "Error", "Critical"
+    log.LEVEL = "Warn";
     server = {
       # options: "home", "explore", "organizations", "login" or URL fragment (or full URL)
       LANDING_PAGE = "explore";
+      DOMAIN = "git.uninsane.org";
+      ROOT_URL = "https://git.uninsane.org/";
     };
     service = {
       # timeout for email approval. 5760 = 4 days
@@ -44,6 +50,7 @@
       ENABLE_CAPTCHA = true;
       NOREPLY_ADDRESS = "noreply.anonymous.git@uninsane.org";
     };
+    session.COOKIE_SECURE = true;
     repository = {
       DEFAULT_BRANCH = "master";
     };
@@ -58,6 +65,8 @@
     };
     #"ui.meta" = ... to customize html author/description/etc
     mailer = {
+      # alternative is to use nixos-level config:
+      # services.gitea.mailerPasswordFile = ...
       ENABLED = true;
       MAILER_TYPE = "sendmail";
       FROM = "notify.git@uninsane.org";
@@ -69,8 +78,6 @@
       FORMAT = "RFC3339";
     };
   };
-  # options: "Trace", "Debug", "Info", "Warn", "Error", "Critical"
-  services.gitea.settings.log.LEVEL = "Warn";
 
   systemd.services.gitea.serviceConfig = {
     # nix default is AF_UNIX AF_INET AF_INET6.
@@ -83,6 +90,8 @@
     ];
   };
 
+  services.openssh.settings.UsePAM = true;  #< required for `git` user to authenticate
+
   # hosted git (web view and for `git <cmd>` use
   # TODO: enable publog?
   services.nginx.virtualHosts."git.uninsane.org" = {
@@ -93,7 +102,32 @@
     locations."/" = {
       proxyPass = "http://127.0.0.1:3000";
     };
+    # gitea serves all `raw` files as content-type: plain, but i'd like to serve them as their actual content type.
+    # or at least, enough to make specific pages viewable (serving unoriginal content as arbitrary content type is dangerous).
+    locations."~ ^/colin/phone-case-cq/raw/.*.html" = {
+      proxyPass = "http://127.0.0.1:3000";
+      extraConfig = ''
+        proxy_hide_header Content-Type;
+        default_type text/html;
+        add_header Content-Type text/html;
+      '';
+    };
+    locations."~ ^/colin/phone-case-cq/raw/.*.js" = {
+      proxyPass = "http://127.0.0.1:3000";
+      extraConfig = ''
+        proxy_hide_header Content-Type;
+        default_type text/html;
+        add_header Content-Type text/javascript;
+      '';
+    };
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."git" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."git" = "native";
+
+  sane.ports.ports."22" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-git@git.uninsane.org";
+  };
 }

hosts/by-name/servo/services/goaccess.nix

@@ -20,7 +20,7 @@
           --ignore-panel=HOSTS \
           --ws-url=wss://sink.uninsane.org:443/ws \
           --port=7890 \
-          -o /var/lib/uninsane/sink/index.html
+          -o /var/lib/goaccess/index.html
       '';
       ExecReload = "${pkgs.coreutils}/bin/kill -HUP $MAINPID";
       Type = "simple";
@@ -28,17 +28,19 @@
       RestartSec = "10s";
 
       # hardening
-      WorkingDirectory = "/tmp";
+      # TODO: run as `goaccess` user and add `goaccess` user to group `nginx`.
       NoNewPrivileges = true;
+      PrivateDevices = "yes";
       PrivateTmp = true;
       ProtectHome = "read-only";
-      ProtectSystem = "strict";
-      SystemCallFilter = "~@clock @cpu-emulation @debug @keyring @memlock @module @mount @obsolete @privileged @reboot @resources @setuid @swap @raw-io";
-      ReadOnlyPaths = "/";
-      ReadWritePaths = [ "/proc/self" "/var/lib/uninsane/sink" ];
-      PrivateDevices = "yes";
       ProtectKernelModules = "yes";
       ProtectKernelTunables = "yes";
+      ProtectSystem = "strict";
+      ReadOnlyPaths = [ "/var/log/nginx" ];
+      ReadWritePaths = [ "/proc/self" "/var/lib/goaccess" ];
+      StateDirectory = "goaccess";
+      SystemCallFilter = "~@clock @cpu-emulation @debug @keyring @memlock @module @mount @obsolete @privileged @reboot @resources @setuid @swap @raw-io";
+      WorkingDirectory = "/var/lib/goaccess";
     };
     after = [ "network.target" ];
     wantedBy = [ "multi-user.target" ];
@@ -49,7 +51,7 @@
     addSSL = true;
     enableACME = true;
     # inherit kTLS;
-    root = "/var/lib/uninsane/sink";
+    root = "/var/lib/goaccess";
 
     locations."/ws" = {
       proxyPass = "http://127.0.0.1:7890";
@@ -64,5 +66,5 @@
     };
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."sink" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."sink" = "native";
 }

hosts/by-name/servo/services/ipfs.nix

@@ -10,9 +10,9 @@
 
 lib.mkIf false # i don't actively use ipfs anymore
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
-    { user = "261"; group = "261"; directory = "/var/lib/ipfs"; }
+    { user = "261"; group = "261"; path = "/var/lib/ipfs"; method = "bind"; }
   ];
 
   networking.firewall.allowedTCPPorts = [ 4001 ];
@@ -34,7 +34,7 @@ lib.mkIf false # i don't actively use ipfs anymore
     };
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."ipfs" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."ipfs" = "native";
 
   # services.ipfs.enable = true;
   services.kubo.localDiscovery = true;

hosts/by-name/servo/services/jackett.nix

@@ -1,9 +1,10 @@
-{ ... }:
+{ lib, pkgs, ... }:
 
+lib.mkIf false  #< TODO: re-enable once confident of sandboxing
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? we only need this to save Indexer creds ==> migrate to config?
-    { user = "root"; group = "root"; directory = "/var/lib/jackett"; }
+    { user = "root"; group = "root"; path = "/var/lib/jackett"; method = "bind"; }
   ];
   services.jackett.enable = true;
 
@@ -12,6 +13,8 @@
   systemd.services.jackett.serviceConfig = {
     # run this behind the OVPN static VPN
     NetworkNamespacePath = "/run/netns/ovpns";
+    ExecStartPre = [ "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect 185.157.162.178" ];  # abort if public IP is not as expected
+
     # patch jackett to listen on the public interfaces
     # ExecStart = lib.mkForce "${pkgs.jackett}/bin/Jackett --NoUpdates --DataFolder /var/lib/jackett/.config/Jackett --ListenPublic";
   };
@@ -24,9 +27,10 @@
     locations."/" = {
       # proxyPass = "http://ovpns.uninsane.org:9117";
       proxyPass = "http://10.0.1.6:9117";
+      recommendedProxySettings = true;
     };
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."jackett" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."jackett" = "native";
 }
 

hosts/by-name/servo/services/jellyfin.nix

@@ -1,21 +1,81 @@
+# configuration options (today i don't store my config in nix):
+#
+# - jellyfin-web can be statically configured (result/share/jellyfin-web/config.json)
+#   - <https://jellyfin.org/docs/general/clients/web-config>
+#   - configure server list, plugins, "menuLinks", colors
+#
+# - jellfyin server is configured in /var/lib/jellfin/
+#   - root/default/<LibraryType>/
+#     - <LibraryName>.mblink: contains the directory name where this library lives
+#     - options.xml: contains preferences which were defined in the web UI during import
+#       - e.g. `EnablePhotos`, `EnableChapterImageExtraction`, etc.
+#   - config/encoding.xml: transcoder settings
+#   - config/system.xml: misc preferences like log file duration, audiobook resume settings, etc.
+#   - data/jellyfin.db: maybe account definitions? internal state?
+
 { config, lib, ... }:
 
-# TODO: re-enable after migrating media dir to /var/lib/uninsane/media
-# else it's too spammy
-lib.mkIf false
 {
-  networking.firewall.allowedUDPPorts = [
-    1900 7359 # DLNA: https://jellyfin.org/docs/general/networking/index.html
-  ];
-  sane.persist.sys.plaintext = [
-    # TODO: mode? could be more granular
-    { user = "jellyfin"; group = "jellyfin"; directory = "/var/lib/jellyfin"; }
+  # https://jellyfin.org/docs/general/networking/index.html
+  sane.ports.ports."1900" = {
+    protocol = [ "udp" ];
+    visibleTo.lan = true;
+    description = "colin-upnp-for-jellyfin";
+  };
+  sane.ports.ports."7359" = {
+    protocol = [ "udp" ];
+    visibleTo.lan = true;
+    description = "colin-jellyfin-specific-client-discovery";
+    # ^ not sure if this is necessary: copied this port from nixos jellyfin.openFirewall
+  };
+  # not sure if 8096/8920 get used either:
+  sane.ports.ports."8096" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    description = "colin-jellyfin-http-lan";
+  };
+  sane.ports.ports."8920" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    description = "colin-jellyfin-https-lan";
+  };
+
+  sane.persist.sys.byStore.plaintext = [
+    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin"; method = "bind"; }
   ];
+  sane.fs."/var/lib/jellyfin/config/logging.json" = {
+    # "Emby.Dlna" logging: <https://jellyfin.org/docs/general/networking/dlna>
+    symlink.text = ''
+      {
+        "Serilog": {
+          "MinimumLevel": {
+            "Default": "Information",
+            "Override": {
+              "Microsoft": "Warning",
+              "System": "Warning",
+              "Emby.Dlna": "Debug",
+              "Emby.Dlna.Eventing": "Debug"
+            }
+          },
+          "WriteTo": [
+            {
+              "Name": "Console",
+              "Args": {
+                "outputTemplate": "[{Timestamp:HH:mm:ss}] [{Level:u3}] [{ThreadId}] {SourceContext}: {Message:lj}{NewLine}{Exception}"
+              }
+            }
+          ],
+          "Enrich": [ "FromLogContext", "WithThreadId" ]
+        }
+      }
+    '';
+    wantedBeforeBy = [ "jellyfin.service" ];
+  };
 
   # Jellyfin multimedia server
   # this is mostly taken from the official jellfin.org docs
   services.nginx.virtualHosts."jelly.uninsane.org" = {
-    addSSL = true;
+    forceSSL = true;
     enableACME = true;
     # inherit kTLS;
 
@@ -61,7 +121,7 @@ lib.mkIf false
     };
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."jelly" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."jelly" = "native";
 
   services.jellyfin.enable = true;
 }

hosts/by-name/servo/services/kiwix-serve.nix

@@ -1,9 +1,19 @@
+# how to update wikipedia snapshot:
+# - browse for later snapshots:
+#   - <https://mirror.accum.se/mirror/wikimedia.org/other/kiwix/zim/wikipedia>
+# - DL directly, or via rsync (resumable):
+#   - `rsync --progress --append-verify rsync://mirror.accum.se/mirror/wikimedia.org/other/kiwix/zim/wikipedia/wikipedia_en_all_maxi_2022-05.zim .`
+
 { ... }:
 {
+  sane.persist.sys.byStore.ext = [
+    { user = "colin"; group = "users"; path = "/var/lib/kiwix"; method = "bind"; }
+  ];
+
   sane.services.kiwix-serve = {
     enable = true;
     port = 8013;
-    zimPaths = [ "/var/lib/uninsane/www-archive/wikipedia_en_all_maxi_2022-05.zim" ];
+    zimPaths = [ "/var/lib/kiwix/wikipedia_en_all_maxi_2023-11.zim" ];
   };
 
   services.nginx.virtualHosts."w.uninsane.org" = {
@@ -13,5 +23,5 @@
     locations."/".proxyPass = "http://127.0.0.1:8013";
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."w" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."w" = "native";
 }

hosts/by-name/servo/services/komga.nix

@@ -0,0 +1,22 @@
+{ config, ... }:
+let
+  svc-cfg = config.services.komga;
+  inherit (svc-cfg) user group port stateDir;
+in
+{
+  sane.persist.sys.byStore.plaintext = [
+    { inherit user group; mode = "0700"; path = stateDir; method = "bind"; }
+  ];
+
+  services.komga.enable = true;
+  services.komga.port = 11319;  # chosen at random
+
+  services.nginx.virtualHosts."komga.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+    locations."/" = {
+      proxyPass = "http://127.0.0.1:${builtins.toString port}";
+    };
+  };
+  sane.dns.zones."uninsane.org".inet.CNAME."komga" = "native";
+}

hosts/by-name/servo/services/lemmy.nix

@@ -0,0 +1,90 @@
+# docs:
+# - <repo:LemmyNet/lemmy:docker/federation/nginx.conf>
+# - <repo:LemmyNet/lemmy:docker/nginx.conf>
+# - <repo:LemmyNet/lemmy-ansible:templates/nginx.conf>
+
+{ config, lib, pkgs, ... }:
+let
+  inherit (builtins) toString;
+  inherit (lib) mkForce;
+  uiPort = 1234;  # default ui port is 1234
+  backendPort = 8536; # default backend port is 8536
+  #^ i guess the "backend" port is used for federation?
+  pict-rs = pkgs.pict-rs;
+  # pict-rs = pkgs.pict-rs.overrideAttrs (upstream: {
+  #   # as of v0.4.2, all non-GIF video is forcibly transcoded.
+  #   # that breaks lemmy, because of the request latency.
+  #   # and it eats up hella CPU.
+  #   # pict-rs is iffy around video altogether: mp4 seems the best supported.
+  #   # XXX: this patch no longer applies after 0.5.10 -> 0.5.11 update.
+  #   #      git log is hard to parse, but *suggests* that video is natively supported
+  #   #      better than in the 0.4.2 days, e.g. 5fd59fc5b42d31559120dc28bfef4e5002fb509e
+  #   #      "Change commandline flag to allow disabling video, since it is enabled by default"
+  #   postPatch = (upstream.postPatch or "") + ''
+  #     substituteInPlace src/validate.rs \
+  #       --replace-fail 'if transcode_options.needs_reencode() {' 'if false {'
+  #   '';
+  # });
+in {
+  services.lemmy = {
+    enable = true;
+    settings.hostname = "lemmy.uninsane.org";
+    # federation.debug forces outbound federation queries to be run synchronously
+    # N.B.: this option might not be read for 0.17.0+? <https://github.com/LemmyNet/lemmy/blob/c32585b03429f0f76d1e4ff738786321a0a9df98/RELEASES.md#upgrade-instructions>
+    # settings.federation.debug = true;
+    settings.port = backendPort;
+    ui.port = uiPort;
+    database.createLocally = true;
+    nginx.enable = true;
+  };
+
+  systemd.services.lemmy.serviceConfig = {
+    # fix to use a normal user so we can configure perms correctly
+    DynamicUser = mkForce false;
+    User = "lemmy";
+    Group = "lemmy";
+  };
+  systemd.services.lemmy.environment = {
+    RUST_BACKTRACE = "full";
+    # RUST_LOG = "debug";
+    # RUST_LOG = "trace";
+    # upstream defaults LEMMY_DATABASE_URL = "postgres:///lemmy?host=/run/postgresql";
+    # - Postgres complains that we didn't specify a user
+    # lemmy formats the url as:
+    # - postgres://{user}:{password}@{host}:{port}/{database}
+    # SO suggests (https://stackoverflow.com/questions/3582552/what-is-the-format-for-the-postgresql-connection-string-url):
+    # - postgresql://[user[:password]@][netloc][:port][/dbname][?param1=value1&...]
+    # LEMMY_DATABASE_URL = "postgres://lemmy@/run/postgresql";  # connection to server on socket "/run/postgresql/.s.PGSQL.5432" failed: FATAL:  database "run/postgresql" does not exist
+    # LEMMY_DATABASE_URL = "postgres://lemmy?host=/run/postgresql";  # no PostgreSQL user name specified in startup packet
+    # LEMMY_DATABASE_URL = mkForce "postgres://lemmy@?host=/run/postgresql";  # WORKS
+    LEMMY_DATABASE_URL = mkForce "postgres://lemmy@/lemmy?host=/run/postgresql";
+  };
+  users.groups.lemmy = {};
+  users.users.lemmy = {
+    group = "lemmy";
+    isSystemUser = true;
+  };
+
+  services.nginx.virtualHosts."lemmy.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+  };
+
+  sane.dns.zones."uninsane.org".inet.CNAME."lemmy" = "native";
+
+  #v DO NOT REMOVE: defaults to 0.3, instead of latest, so always need to explicitly set this.
+  services.pict-rs.package = pict-rs;
+
+  # pict-rs configuration is applied in this order:
+  # - via toml
+  # - via env vars (overrides everything above)
+  # - via CLI flags (overrides everything above)
+  # some of the CLI flags have defaults, making it the only actual way to configure certain things even when docs claim otherwise.
+  # CLI args: <https://git.asonix.dog/asonix/pict-rs#user-content-running>
+  systemd.services.pict-rs.serviceConfig.ExecStart = lib.mkForce (lib.concatStringsSep " " [
+    "${lib.getBin pict-rs}/bin/pict-rs run"
+    "--media-video-max-frame-count" (builtins.toString (30*60*60))
+    "--media-process-timeout 120"
+    "--media-video-allow-audio"  # allow audio
+  ]);
+}

hosts/by-name/servo/services/matrix/default.nix

@@ -1,75 +1,92 @@
-# docs: https://nixos.wiki/wiki/Matrix
-# docs: https://nixos.org/manual/nixos/stable/index.html#module-services-matrix-synapse
+# docs: <https://nixos.wiki/wiki/Matrix>
+# docs: <https://nixos.org/manual/nixos/stable/index.html#module-services-matrix-synapse>
+# example config: <https://github.com/matrix-org/synapse/blob/develop/docs/sample_config.yaml>
+#
+# ENABLING PUSH NOTIFICATIONS (with UnifiedPush/ntfy):
+# - Matrix "pushers" API spec: <https://spec.matrix.org/latest/client-server-api/#post_matrixclientv3pushersset>
+# - first, view notification settings:
+#   - obtain your client's auth token. e.g. Element -> profile -> help/about -> access token.
+#   - `curl --header 'Authorization: Bearer <your_access_token>' localhost:8008/_matrix/client/v3/pushers | jq .`
+# - enable a new notification destination:
+#   - `curl --header "Authorization: Bearer <your_access_token>" --data '{ "app_display_name": "<topic>", "app_id": "ntfy.uninsane.org", "data": { "url": "https://ntfy.uninsane.org/_matrix/push/v1/notify", "format": "event_id_only" }, "device_display_name": "<topic>", "kind": "http", "lang": "en-US", "profile_tag": "", "pushkey": "<topic>" }' localhost:8008/_matrix/client/v3/pushers/set`
+# - delete a notification destination by setting `kind` to `null` (otherwise, request is identical to above)
+#
 { config, lib, pkgs, ... }:
 
 {
   imports = [
     ./discord-puppet.nix
-    # ./irc.nix
+    ./irc.nix
     ./signal.nix
   ];
 
-  # allow synapse to read the registration files of its appservices
-  users.users.matrix-synapse.extraGroups = [ "mautrix-signal" ];
-
-  sane.persist.sys.plaintext = [
-    { user = "matrix-synapse"; group = "matrix-synapse"; directory = "/var/lib/matrix-synapse"; }
+  sane.persist.sys.byStore.plaintext = [
+    { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/matrix-synapse"; method = "bind"; }
   ];
   services.matrix-synapse.enable = true;
-  # this changes the default log level from INFO to WARN.
-  # maybe there's an easier way?
-  services.matrix-synapse.settings.log_config = ./synapse-log_level.yaml;
-  services.matrix-synapse.settings.server_name = "uninsane.org";
+  services.matrix-synapse.settings = {
+    # this changes the default log level from INFO to WARN.
+    # maybe there's an easier way?
+    log_config = ./synapse-log_level.yaml;
+    server_name = "uninsane.org";
 
-  # services.matrix-synapse.enable_registration_captcha = true;
-  # services.matrix-synapse.enable_registration_without_verification = true;
-  services.matrix-synapse.settings.enable_registration = true;
-  # services.matrix-synapse.registration_shared_secret = "<shared key goes here>";
+    # services.matrix-synapse.enable_registration_captcha = true;
+    # services.matrix-synapse.enable_registration_without_verification = true;
+    enable_registration = true;
+    # services.matrix-synapse.registration_shared_secret = "<shared key goes here>";
 
-  # default for listeners is port = 8448, tls = true, x_forwarded = false.
-  # we change this because the server is situated behind nginx.
-  services.matrix-synapse.settings.listeners = [
-    {
-      port = 8008;
-      bind_addresses = [ "127.0.0.1" ];
-      type = "http";
-      tls = false;
-      x_forwarded = true;
-      resources = [
-        {
-          names = [ "client" "federation" ];
-          compress = false;
-        }
-      ];
-    }
-  ];
+    # default for listeners is port = 8448, tls = true, x_forwarded = false.
+    # we change this because the server is situated behind nginx.
+    listeners = [
+      {
+        port = 8008;
+        bind_addresses = [ "127.0.0.1" ];
+        type = "http";
+        tls = false;
+        x_forwarded = true;
+        resources = [
+          {
+            names = [ "client" "federation" ];
+            compress = false;
+          }
+        ];
+      }
+    ];
 
-  services.matrix-synapse.settings.admin_contact = "admin.matrix@uninsane.org";
-  services.matrix-synapse.settings.registrations_require_3pid = [ "email" ];
+    ip_range_whitelist = [
+      # to communicate with ntfy.uninsane.org push notifs.
+      # TODO: move this to some non-shared loopback device: we don't want Matrix spouting http requests to *anything* on this machine
+      "10.78.79.51"
+    ];
+
+    x_forwarded = true;  # because we proxy matrix behind nginx
+    max_upload_size = "100M";  # default is "50M"
+
+    admin_contact = "admin.matrix@uninsane.org";
+    registrations_require_3pid = [ "email" ];
+  };
 
   services.matrix-synapse.extraConfigFiles = [
-    config.sops.secrets.matrix_synapse_secrets.path
+    config.sops.secrets."matrix_synapse_secrets.yaml".path
   ];
 
-  # services.matrix-synapse.extraConfigFiles = [builtins.toFile "matrix-synapse-extra-config" ''
-  #   admin_contact: "admin.matrix@uninsane.org"
-  #   registrations_require_3pid:
-  #     - email
-  #   email:
-  #     smtp_host: "mx.uninsane.org"
-  #     smtp_port: 587
-  #     smtp_user: "matrix-synapse"
-  #     smtp_pass: "${secrets.matrix-synapse.smtp_pass}"
-  #     require_transport_security: true
-  #     enable_tls: true
-  #     notif_from: "%(app)s <notify.matrix@uninsane.org>"
-  #     app_name: "Uninsane Matrix"
-  #     enable_notifs: true
-  #     validation_token_lifetime: 96h
-  #     invite_client_location: "https://web.matrix.uninsane.org"
-  #     subjects:
-  #       email_validation: "[%(server_name)s] Validate your email"
-  # ''];
+  systemd.services.matrix-synapse.postStart = ''
+    ACCESS_TOKEN=$(${pkgs.coreutils}/bin/cat ${config.sops.secrets.matrix_access_token.path})
+    TOPIC=$(${pkgs.coreutils}/bin/cat ${config.sops.secrets.ntfy-sh-topic.path})
+
+    echo "ensuring ntfy push gateway"
+    ${pkgs.curl}/bin/curl \
+      --header "Authorization: Bearer $ACCESS_TOKEN" \
+      --data "{ \"app_display_name\": \"ntfy-adapter\", \"app_id\": \"ntfy.uninsane.org\", \"data\": { \"url\": \"https://ntfy.uninsane.org/_matrix/push/v1/notify\", \"format\": \"event_id_only\" }, \"device_display_name\": \"ntfy-adapter\", \"kind\": \"http\", \"lang\": \"en-US\", \"profile_tag\": \"\", \"pushkey\": \"$TOPIC\" }" \
+      localhost:8008/_matrix/client/v3/pushers/set
+
+    echo "registered push gateways:"
+    ${pkgs.curl}/bin/curl \
+      --header "Authorization: Bearer $ACCESS_TOKEN" \
+      localhost:8008/_matrix/client/v3/pushers \
+      | ${pkgs.jq}/bin/jq .
+  '';
+
 
   # new users may be registered on the CLI:
   #   register_new_matrix_user -c /nix/store/8n6kcka37jhmi4qpd2r03aj71pkyh21s-homeserver.yaml http://localhost:8008
@@ -97,6 +114,10 @@
 
     locations."/" = {
       proxyPass = "http://127.0.0.1:8008";
+      extraConfig = ''
+        # allow uploading large files (matrix enforces a separate limit, downstream)
+        client_max_body_size  512m;
+      '';
     };
     # redirect browsers to the web client.
     # i don't think native matrix clients ever fetch the root.
@@ -127,13 +148,18 @@
     };
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet = {
+  sane.dns.zones."uninsane.org".inet = {
     CNAME."matrix" = "native";
     CNAME."web.matrix" = "native";
   };
 
 
-  sops.secrets."matrix_synapse_secrets" = {
+  sops.secrets."matrix_synapse_secrets.yaml" = {
     owner = config.users.users.matrix-synapse.name;
   };
+  sops.secrets."matrix_access_token" = {
+    owner = config.users.users.matrix-synapse.name;
+  };
+  # provide access to ntfy-sh-topic secret
+  users.users.matrix-synapse.extraGroups = [ "ntfy-sh" ];
 }

hosts/by-name/servo/services/matrix/discord-puppet.nix

@@ -1,7 +1,12 @@
 { lib, ... }:
+
+# XXX mx-discord-puppet uses nodejs_14 which is EOL
+# - mx-discord-puppet is abandoned upstream _and_ in nixpkgs
+# - recommended to use mautrix-discord: <https://github.com/NixOS/nixpkgs/pull/200462>
+lib.mkIf false
 {
-  sane.persist.sys.plaintext = [
-    { user = "matrix-synapse"; group = "matrix-synapse"; directory = "/var/lib/mx-puppet-discord"; }
+  sane.persist.sys.byStore.plaintext = [
+    { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/mx-puppet-discord"; method = "bind"; }
   ];
 
   services.matrix-synapse.settings.app_service_config_files = [

hosts/by-name/servo/services/matrix/irc-no-reveal-bridge.patch

@@ -0,0 +1,13 @@
+diff --git a/src/irc/ConnectionInstance.ts b/src/irc/ConnectionInstance.ts
+index 688036ca..3373fa27 100644
+--- a/src/irc/ConnectionInstance.ts
++++ b/src/irc/ConnectionInstance.ts
+@@ -149,7 +149,7 @@ export class ConnectionInstance {
+         if (this.dead) {
+             return Promise.resolve();
+         }
+-        ircReason = ircReason || reason;
++        ircReason = "bye"; // don't reveal through the IRC quit message that we're a bridge
+         log.info(
+             "disconnect()ing %s@%s - %s", this.nick, this.domain, reason
+         );

hosts/by-name/servo/services/matrix/irc-no-reveal-mxid.patch

@@ -0,0 +1,50 @@
+diff --git a/config.schema.yml b/config.schema.yml
+index 2e71c8d6..42ba8ba1 100644
+--- a/config.schema.yml
++++ b/config.schema.yml
+@@ -433,7 +433,7 @@ properties:
+                                     type: "boolean"
+                                 realnameFormat:
+                                     type: "string"
+-                                    enum: ["mxid","reverse-mxid"]
++                                    enum: ["mxid","reverse-mxid","localpart"]
+                                 ipv6:
+                                     type: "object"
+                                     properties:
+diff --git a/src/irc/IdentGenerator.ts b/src/irc/IdentGenerator.ts
+index 7a2b5cf1..50f7815a 100644
+--- a/src/irc/IdentGenerator.ts
++++ b/src/irc/IdentGenerator.ts
+@@ -74,6 +74,9 @@ export class IdentGenerator {
+         else if (server.getRealNameFormat() === "reverse-mxid") {
+             realname = IdentGenerator.sanitiseRealname(IdentGenerator.switchAroundMxid(matrixUser));
+         }
++        else if (server.getRealNameFormat() == "localpart") {
++            realname = IdentGenerator.sanitiseRealname(matrixUser.localpart);
++        }
+         else {
+             throw Error('Invalid value for realNameFormat');
+         }
+diff --git a/src/irc/IrcServer.ts b/src/irc/IrcServer.ts
+index 2af73ab4..895b9783 100644
+--- a/src/irc/IrcServer.ts
++++ b/src/irc/IrcServer.ts
+@@ -101,7 +101,7 @@ export interface IrcServerConfig {
+         };
+         lineLimit: number;
+         userModes?: string;
+-        realnameFormat?: "mxid"|"reverse-mxid";
++        realnameFormat?: "mxid"|"reverse-mxid"|"localpart";
+         pingTimeoutMs: number;
+         pingRateMs: number;
+         kickOn: {
+@@ -289,7 +289,7 @@ export class IrcServer {
+         return this.config.ircClients.userModes || "";
+     }
+ 
+-    public getRealNameFormat(): "mxid"|"reverse-mxid" {
++    public getRealNameFormat(): "mxid"|"reverse-mxid"|"localpart" {
+         return this.config.ircClients.realnameFormat || "mxid";
+     }
+ 
+

hosts/by-name/servo/services/matrix/irc.nix

@@ -1,21 +1,123 @@
+# config docs:
+# - <https://github.com/matrix-org/matrix-appservice-irc/blob/develop/config.sample.yaml>
+#       probably want to remove that.
 { config, lib, ... }:
 
+let
+  ircServer = { name, additionalAddresses ? [], sasl ? true, port ? 6697 }: let
+    lowerName = lib.toLower name;
+  in {
+    # XXX sasl: appservice doesn't support NickServ identification (only SASL, or PASS if sasl = false)
+    inherit name additionalAddresses sasl port;
+    ssl = true;
+    botConfig = {
+      # bot has no presence in IRC channel; only real Matrix users
+      enabled = false;
+      # this is the IRC username/nickname *of the bot* (not visible in channels): not of the end-user.
+      # the irc username/nick of a mapped Matrix user is determined further down in `ircClients` section.
+      # if `enabled` is false, then this name probably never shows up on the IRC side (?)
+      nick = "uninsane";
+      username = "uninsane";
+      joinChannelsIfNoUsers = false;
+    };
+    dynamicChannels = {
+      enabled = true;
+      aliasTemplate = "#irc_${lowerName}_$CHANNEL";
+      published = false;  # false => irc rooms aren't listed in homeserver public rooms list
+      federate = false;  # false => Matrix users from other homeservers can't join IRC channels
+    };
+    ircClients = {
+      nickTemplate = "$LOCALPARTsane";  # @colin:uninsane.org (Matrix) -> colinsane (IRC)
+      realnameFormat = "reverse-mxid";  # @colin:uninsane.org (Matrix) -> org.uninsane:colin (IRC)
+      # realnameFormat = "localpart";  # @colin:uninsane.org (Matrix) -> colin (IRC)  -- but requires the mxid patch below
+      # by default, Matrix will convert messages greater than (3) lines into a pastebin-like URL to send to IRC.
+      lineLimit = 20;
+      # Rizon in particular allows only 4 connections from one IP before a 30min ban.
+      # that's effectively reduced to 2 during a netsplit, or maybe during a restart.
+      #   - https://wiki.rizon.net/index.php?title=Connection/Session_Limit_Exemptions
+      # especially, misconfigurations elsewhere in this config may cause hundreds of connections
+      # so this is a safeguard.
+      maxClients = 2;
+      # don't have the bridge disconnect me from IRC when idle.
+      idleTimeout = 0;
+      concurrentReconnectLimit = 2;
+      reconnectIntervalMs = 60000;
+      kickOn = {
+        # remove Matrix user from room when...
+        channelJoinFailure = false;
+        ircConnectionFailure = false;
+        userQuit = true;
+      };
+    };
+    matrixClients = {
+      userTemplate = "@irc_${lowerName}_$NICK";  # the :uninsane.org part is appended automatically
+    };
+
+    # this will let this user message the appservice with `!join #<IRCChannel>` and the rest "Just Works"
+    "@colin:uninsane.org" = "admin";
+
+    membershipLists = {
+      enabled = true;
+      global = {
+        ircToMatrix = {
+          initial = true;
+          incremental = true;
+          requireMatrixJoined = false;
+        };
+        matrixToIrc = {
+          initial = true;
+          incremental = true;
+        };
+      };
+      ignoreIdleUsersOnStartup = {
+        enabled = false;  # false => always bridge users, even if idle
+      };
+    };
+    # sync room description?
+    bridgeInfoState = {
+      enabled = true;
+      initial = true;
+    };
+
+    # for per-user IRC password:
+    # - invite @irc_${lowerName}_NickServ:uninsane.org to a DM and type `help`  => register
+    # - invite the matrix-appservice-irc user to a DM and type `!help`   => add PW to database
+    # to validate that i'm authenticated on the IRC network, DM @irc_${lowerName}_NickServ:uninsane.org:
+    # - send: `STATUS colinsane`
+    # - response should be `3`: "user recognized as owner via password identification"
+    # passwordEncryptionKeyPath = "/path/to/privkey";  # appservice will generate its own if unspecified
+  };
+in
 {
-  sane.persist.sys.plaintext = [
-    # TODO: mode?
-    # user and group are both "matrix-appservice-irc"
-    { user = "993"; group = "992"; directory = "/var/lib/matrix-appservice-irc"; }
+
+  nixpkgs.overlays = [
+    (next: prev: {
+      matrix-appservice-irc = prev.matrix-appservice-irc.overrideAttrs (super: {
+        patches = super.patches or [] ++ [
+          ./irc-no-reveal-bridge.patch
+          # ./irc-no-reveal-mxid.patch
+        ];
+      });
+    })
   ];
 
+  sane.persist.sys.byStore.plaintext = [
+    # TODO: mode?
+    { user = "matrix-appservice-irc"; group = "matrix-appservice-irc"; path = "/var/lib/matrix-appservice-irc"; method = "bind"; }
+  ];
+
+  # XXX: matrix-appservice-irc PreStart tries to chgrp the registration.yml to matrix-synapse,
+  # which requires matrix-appservice-irc to be of that group
+  users.users.matrix-appservice-irc.extraGroups = [ "matrix-synapse" ];
+  # weird race conditions around registration.yml mean we want matrix-synapse to be of matrix-appservice-irc group too.
+  users.users.matrix-synapse.extraGroups = [ "matrix-appservice-irc" ];
+
   services.matrix-synapse.settings.app_service_config_files = [
     "/var/lib/matrix-appservice-irc/registration.yml"  # auto-created by irc appservice
   ];
 
-  # note: Rizon allows only FOUR simultaneous IRC connections per IP: https://wiki.rizon.net/index.php?title=Connection/Session_Limit_Exemptions
-  # Rizon supports CertFP for auth: https://wiki.rizon.net/index.php?title=CertFP
   services.matrix-appservice-irc.enable = true;
   services.matrix-appservice-irc.registrationUrl = "http://127.0.0.1:8009";
-  # settings documented here: https://github.com/matrix-org/matrix-appservice-irc/blob/develop/config.sample.yaml
   services.matrix-appservice-irc.settings = {
     homeserver = {
       url = "http://127.0.0.1:8008";
@@ -28,70 +130,39 @@
 
     ircService = {
       servers = {
-        "irc.rizon.net" = {
-          name = "Rizon";
-          port = 6697;  # SSL port
-          ssl = true;
-          sasl = true;  # appservice doesn't support NickServ identification
-          botConfig = {
-            # bot has no presence in IRC channel; only real Matrix users
-            enabled = false;
-            # nick = "UninsaneDotOrg";
-            nick = "uninsane";
-            username = "uninsane";
-          };
-          dynamicChannels = {
-            enabled = true;
-            aliasTemplate = "#irc_rizon_$CHANNEL";
-          };
-          ircClients = {
-            nickTemplate = "$LOCALPARTsane";
-            # by default, Matrix will convert messages greater than (3) lines into a pastebin-like URL to send to IRC.
-            lineLimit = 20;
-          };
-          matrixClients = {
-            userTemplate = "@irc_rizon_$NICK";  # the :uninsane.org part is appended automatically
-          };
-
-          # this will let this user message the appservice with `!join #<IRCChannel>` and the rest "Just Works"
-          "@colin:uninsane.org" = "admin";
-
-          membershipLists = {
-            enabled = true;
-            global = {
-              ircToMatrix = {
-                initial = true;
-                incremental = true;
-                requireMatrixJoined = false;
-              };
-              matrixToIrc = {
-                initial = true;
-                incremental = true;
-              };
-            };
-          };
-          # sync room description?
-          bridgeInfoState = {
-            enabled = true;
-            initial = true;
-          };
-
-          # hardcoded mappings, for when dynamicChannels fails us. TODO: probably safe to remove these.
-          # mappings = {
-          #   "#chat" = {
-          #     roomIds = [ "!GXJSOTdbtxRboGtDep:uninsane.org" ];
-          #   };
-          #   # BakaBT requires account registration, which i think means my user needs to be added before the appservice user
-          #   "#BakaBT" = {
-          #     roomIds = [ "!feZKttuYuHilqPFSkD:uninsane.org" ];
-          #   };
-          # };
-          # for per-user IRC password:
-          #   invite @irc_rizon_NickServ:uninsane.org to a DM and type `help`  => register
-          #   invite the matrix-appservice-irc user to a DM and type `!help`   => add PW to database
-          # passwordEncryptionKeyPath = "/path/to/privkey";  # appservice will generate its own if unspecified
+        "irc.esper.net" = ircServer {
+          name = "esper";
+          sasl = false;
+          # notable channels:
+          # - #merveilles
         };
+        "irc.libera.chat" = ircServer {
+          name = "libera";
+          sasl = false;
+          # notable channels:
+          # - #hare
+          # - #mnt-reform
+        };
+        "irc.myanonamouse.net" = ircServer {
+          name = "MyAnonamouse";
+          additionalAddresses = [ "irc2.myanonamouse.net" ];
+          sasl = false;
+        };
+        "irc.oftc.net" = ircServer {
+          name = "oftc";
+          sasl = false;
+          # notable channels:
+          # - #sxmo
+          # - #sxmo-offtopic
+        };
+        "irc.rizon.net" = ircServer { name = "Rizon"; };
       };
     };
   };
+
+  systemd.services.matrix-appservice-irc.serviceConfig = {
+    # XXX 2023/06/20: nixos specifies this + @aio and @memlock as forbidden
+    # the service actively uses at least one of these, and both of them are fairly innocuous
+    SystemCallFilter = lib.mkForce "~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io @setuid @swap";
+  };
 }

hosts/by-name/servo/services/matrix/signal.nix

@@ -1,12 +1,17 @@
 # config options:
 # - <https://github.com/mautrix/signal/blob/master/mautrix_signal/example-config.yaml>
-{ config, pkgs, ... }:
+{ config, lib, pkgs, ... }:
+
+lib.mkIf false  # disabled 2024/01/11: i don't use it, and pkgs.mautrix-signal had some API changes
 {
-  sane.persist.sys.plaintext = [
-    { user = "mautrix-signal"; group = "mautrix-signal"; directory = "/var/lib/mautrix-signal"; }
-    { user = "signald"; group = "signald"; directory = "/var/lib/signald"; }
+  sane.persist.sys.byStore.plaintext = [
+    { user = "mautrix-signal"; group = "mautrix-signal"; path = "/var/lib/mautrix-signal"; method = "bind"; }
+    { user = "signald"; group = "signald"; path = "/var/lib/signald"; method = "bind"; }
   ];
 
+  # allow synapse to read the registration file
+  users.users.matrix-synapse.extraGroups = [ "mautrix-signal" ];
+
   services.signald.enable = true;
   services.mautrix-signal.enable = true;
   services.mautrix-signal.environmentFile =
@@ -27,7 +32,6 @@
   };
 
   sops.secrets."mautrix_signal_env" = {
-    format = "binary";
     mode = "0440";
     owner = config.users.users.mautrix-signal.name;
     group = config.users.users.matrix-synapse.name;

hosts/by-name/servo/services/navidrome.nix

@@ -1,15 +1,16 @@
 { lib, ... }:
 
+lib.mkIf false  #< i don't actively use navidrome
 {
-  sane.persist.sys.plaintext = [
-    { user = "navidrome"; group = "navidrome"; directory = "/var/lib/navidrome"; }
+  sane.persist.sys.byStore.plaintext = [
+    { user = "navidrome"; group = "navidrome"; path = "/var/lib/navidrome"; method = "bind"; }
   ];
   services.navidrome.enable = true;
   services.navidrome.settings = {
     # docs: https://www.navidrome.org/docs/usage/configuration-options/
     Address = "127.0.0.1";
     Port = 4533;
-    MusicFolder = "/var/lib/uninsane/media/Music";
+    MusicFolder = "/var/media/Music";
     CovertArtPriority = "*.jpg, *.JPG, *.png, *.PNG, embedded";
     AutoImportPlaylists = false;
     ScanSchedule = "@every 1h";
@@ -36,5 +37,5 @@
     locations."/".proxyPass = "http://127.0.0.1:4533";
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."music" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."music" = "native";
 }

hosts/by-name/servo/services/nginx.nix

@@ -1,4 +1,5 @@
-# docs: https://nixos.wiki/wiki/Nginx
+# docs: <https://nixos.wiki/wiki/Nginx>
+# docs: <https://nginx.org/en/docs/>
 { config, lib, pkgs, ... }:
 
 let
@@ -13,7 +14,19 @@ let
 in
 {
 
-  networking.firewall.allowedTCPPorts = [ 80 443 ];
+  sane.ports.ports."80" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    visibleTo.ovpn = true;  # so that letsencrypt can procure a cert for the mx record
+    description = "colin-http-uninsane.org";
+  };
+  sane.ports.ports."443" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-https-uninsane.org";
+  };
 
   services.nginx.enable = true;
   services.nginx.appendConfig = ''
@@ -41,8 +54,10 @@ in
   services.nginx.recommendedOptimisation = true;
 
   # web blog/personal site
+  # alternative way to link stuff into the share:
+  # sane.fs."/var/www/sites/uninsane.org/share/Ubunchu".mount.bind = "/var/media/Books/Visual/HiroshiSeo/Ubunchu";
+  # sane.fs."/var/media/Books/Visual/HiroshiSeo/Ubunchu".dir = {};
   services.nginx.virtualHosts."uninsane.org" = publog {
-    root = "${pkgs.uninsane-dot-org}/share/uninsane-dot-org";
     # a lot of places hardcode https://uninsane.org,
     # and then when we mix http + non-https, we get CORS violations
     # and things don't look right. so force SSL.
@@ -52,9 +67,38 @@ in
     # for OCSP stapling
     sslTrustedCertificate = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";
 
-    # uninsane.org/share/foo => /var/lib/uninsane/root/share/foo.
-    # yes, nginx does not strip the prefix when evaluating against the root.
-    locations."/share".root = "/var/lib/uninsane/root";
+    locations."/" = {
+      root = "${pkgs.uninsane-dot-org}/share/uninsane-dot-org";
+      tryFiles = "$uri $uri/ @fallback";
+    };
+
+    # unversioned files
+    locations."@fallback" = {
+      root = "/var/www/sites/uninsane.org";
+    };
+
+    # uninsane.org/share/foo => /var/www/sites/uninsane.org/share/foo.
+    # special-cased to enable directory listings
+    locations."/share" = {
+      root = "/var/www/sites/uninsane.org";
+      extraConfig = ''
+        # autoindex => render directory listings
+        autoindex on;
+        # don't follow any symlinks when serving files
+        # otherwise it allows a directory escape
+        disable_symlinks on;
+      '';
+    };
+    locations."/share/Milkbags/" = {
+      alias = "/var/media/Videos/Milkbags/";
+      extraConfig = ''
+        # autoindex => render directory listings
+        autoindex on;
+        # don't follow any symlinks when serving files
+        # otherwise it allows a directory escape
+        disable_symlinks on;
+      '';
+    };
 
     # allow matrix users to discover that @user:uninsane.org is reachable via matrix.uninsane.org
     locations."= /.well-known/matrix/server".extraConfig =
@@ -89,11 +133,25 @@ in
     };
 
     # allow ActivityPub clients to discover how to reach @user@uninsane.org
-    # TODO: waiting on https://git.pleroma.social/pleroma/pleroma/-/merge_requests/3361/
+    # see: https://git.pleroma.social/pleroma/pleroma/-/merge_requests/3361/
+    # not sure this makes sense while i run multiple AP services (pleroma, lemmy)
     # locations."/.well-known/nodeinfo" = {
     #   proxyPass = "http://127.0.0.1:4000";
     #   extraConfig = pleromaExtraConfig;
     # };
+
+    # redirect common feed URIs to the canonical feed
+    locations."= /atom".extraConfig = "return 301 /atom.xml;";
+    locations."= /feed".extraConfig = "return 301 /atom.xml;";
+    locations."= /feed.xml".extraConfig = "return 301 /atom.xml;";
+    locations."= /rss".extraConfig = "return 301 /atom.xml;";
+    locations."= /rss.xml".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/atom".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/atom.xml".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/feed".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/feed.xml".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/rss".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/rss.xml".extraConfig = "return 301 /atom.xml;";
   };
 
 
@@ -120,10 +178,9 @@ in
   security.acme.acceptTerms = true;
   security.acme.defaults.email = "admin.acme@uninsane.org";
 
-  sane.persist.sys.plaintext = [
-    # TODO: mode?
-    { user = "acme"; group = "acme"; directory = "/var/lib/acme"; }
-    { user = "colin"; group = "users"; directory = "/var/www/sites"; }
+  sane.persist.sys.byStore.plaintext = [
+    { user = "acme"; group = "acme"; path = "/var/lib/acme"; method = "bind"; }
+    { user = "colin"; group = "users"; path = "/var/www/sites"; method = "bind"; }
   ];
 
   # let's encrypt default chain looks like:

hosts/by-name/servo/services/nixos-prebuild.nix

@@ -0,0 +1,26 @@
+{ lib, pkgs, ... }:
+
+lib.optionalAttrs false  # disabled until i can be sure it's not gonna OOM my server in the middle of the night
+{
+  systemd.services.nixos-prebuild = {
+    description = "build a nixos image with all updated deps";
+    path = with pkgs; [ coreutils git nix ];
+    script = ''
+      working=$(mktemp -d /tmp/nixos-prebuild.XXXXXX)
+      pushd "$working"
+      git clone https://git.uninsane.org/colin/nix-files.git \
+        && cd nix-files \
+        && nix flake update \
+        || true
+      RC=$(nix run "$working/nix-files#check" -- -j1 --cores 5 --builders "")
+      popd
+      rm -rf "$working"
+      exit "$RC"
+    '';
+  };
+
+  systemd.timers.nixos-prebuild = {
+    wantedBy = [ "multi-user.target" ];
+    timerConfig.OnCalendar = "11,23:00:00";
+  };
+}

hosts/by-name/servo/services/nixserve.nix

@@ -1,21 +0,0 @@
-{ config, ... }:
-
-{
-  services.nginx.virtualHosts."nixcache.uninsane.org" = {
-    addSSL = true;
-    enableACME = true;
-    # inherit kTLS;
-    # serverAliases = [ "nixcache" ];
-    locations."/".extraConfig = ''
-      proxy_pass http://localhost:${toString config.services.nix-serve.port};
-      proxy_set_header Host $host;
-      proxy_set_header X-Real-IP $remote_addr;
-      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-    '';
-  };
-
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."nixcache" = "native";
-
-  sane.services.nixserve.enable = true;
-  sane.services.nixserve.sopsFile = ../../../../secrets/servo.yaml;
-}

hosts/by-name/servo/services/ntfy/default.nix

@@ -0,0 +1,14 @@
+# ntfy: UnifiedPush notification delivery system
+# - used to get push notifications out of Matrix and onto a Phone (iOS, Android, or a custom client)
+{ config, ... }:
+{
+  imports = [
+    ./ntfy-waiter.nix
+    ./ntfy-sh.nix
+  ];
+  sops.secrets."ntfy-sh-topic" = {
+    mode = "0440";
+    owner = config.users.users.ntfy-sh.name;
+    group = config.users.users.ntfy-sh.name;
+  };
+}

hosts/by-name/servo/services/ntfy/ntfy-sh.nix

@@ -0,0 +1,92 @@
+# ntfy: UnifiedPush notification delivery system
+# - used to get push notifications out of Matrix and onto a Phone (iOS, Android, or a custom client)
+#
+# config options:
+# - <https://docs.ntfy.sh/config/#config-options>
+#
+# usage:
+# - ntfy sub https://ntfy.uninsane.org/TOPIC
+# - ntfy pub https://ntfy.uninsane.org/TOPIC "my message"
+# in production, TOPIC is a shared secret between the publisher (Matrix homeserver) and the subscriber (phone)
+#
+# administering:
+# - sudo -u ntfy-sh ntfy access
+#
+# debugging:
+# - make sure that the keepalives are good:
+#   - on the subscriber machine, run `lsof -i4` to find the port being used
+#   - `sudo tcpdump tcp port <p>`
+#     - shouldn't be too spammy
+#
+# matrix integration:
+# - the user must manually point synapse to the ntfy endpoint:
+#   - `curl --header "Authorization: <your_token>" --data '{ "app_display_name": "sane-nix moby", "app_id": "ntfy.uninsane.org", "data": { "url": "https://ntfy.uninsane.org/_matrix/push/v1/notify", "format": "event_id_only" }, "device_display_name": "sane-nix moby", "kind": "http", "lang": "en-US", "profile_tag": "", "pushkey": "https://ntfy.uninsane.org/TOPIC" }' localhost:8008/_matrix/client/v3/pushers/set`
+#     where the token is grabbed from Element's help&about page when logged in
+#   - to remove, send this `curl` with `"kind": null`
+{ config, lib, pkgs, ... }:
+let
+  # subscribers need a non-443 public port to listen on as a way to easily differentiate this traffic
+  # at the IP layer, to enable e.g. wake-on-lan.
+  altPort = 2587;
+in
+{
+  sane.persist.sys.byStore.plaintext = [
+    # not 100% necessary to persist this, but ntfy does keep a 12hr (by default) cache
+    # for pushing notifications to users who become offline.
+    # ACLs also live here.
+    { user = "ntfy-sh"; group ="ntfy-sh"; path = "/var/lib/ntfy-sh"; method = "bind"; }
+  ];
+
+  services.ntfy-sh.enable = true;
+  services.ntfy-sh.settings = {
+    base-url = "https://ntfy.uninsane.org";
+    behind-proxy = true;  # not sure if needed
+    # keepalive interval is a ntfy-specific keepalive thing, where it sends actual data down the wire.
+    # it's not simple TCP keepalive.
+    # defaults to 45s.
+    # note that the client may still do its own TCP-level keepalives, typically every 30s
+    keepalive-interval = "15m";
+    log-level = "trace";  # trace, debug, info (default), warn, error
+    auth-default-access = "deny-all";
+  };
+  systemd.services.ntfy-sh.serviceConfig.DynamicUser = lib.mkForce false;
+  systemd.services.ntfy-sh.preStart = ''
+    # make this specific topic read-write by world
+    # it would be better to use the token system, but that's extra complexity for e.g.
+    # how do i plumb a secret into the Matrix notification pusher
+    #
+    # note that this will fail upon first run, i.e. before ntfy has created its db.
+    # just restart the service.
+    topic=$(cat ${config.sops.secrets.ntfy-sh-topic.path})
+    ${pkgs.ntfy-sh}/bin/ntfy access everyone "$topic" read-write
+  '';
+
+
+  services.nginx.virtualHosts."ntfy.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+    listen = [
+      { addr = "0.0.0.0"; port = altPort; ssl = true; }
+      { addr = "0.0.0.0"; port = 443;  ssl = true; }
+      { addr = "0.0.0.0"; port = 80;   ssl = false; }
+    ];
+    locations."/" = {
+      proxyPass = "http://127.0.0.1:2586";
+      proxyWebsockets = true;  #< support websocket upgrades. without that, `ntfy sub` hangs silently
+      recommendedProxySettings = true; #< adds headers so ntfy logs include the real IP
+      extraConfig = ''
+        # absurdly long timeout (86400s=24h) so that we never hang up on clients.
+        # make sure the client is smart enough to detect a broken proxy though!
+        proxy_read_timeout 86400s;
+      '';
+    };
+  };
+  sane.dns.zones."uninsane.org".inet.CNAME."ntfy" = "native";
+
+  sane.ports.ports."${builtins.toString altPort}" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-ntfy.uninsane.org";
+  };
+}

hosts/by-name/servo/services/ntfy/ntfy-waiter

@@ -0,0 +1,151 @@
+#!/usr/bin/env nix-shell
+#!nix-shell -i python3 -p "python3.withPackages (ps: [  ])" -p ntfy-sh
+
+import argparse
+import logging
+import os
+import socket
+import subprocess
+import sys
+import threading
+import time
+
+logger = logging.getLogger(__name__)
+
+LISTEN_QUEUE = 3
+WAKE_MESSAGE = b'notification\n'
+
+class Client:
+    def __init__(self, sock, addr_info, live_after: float):
+        self.live_after = live_after
+        self.sock = sock
+        self.addr_info = addr_info
+
+    def __cmp__(self, other: 'Client'):
+        return cmp(self.addr_info, other.addr_info)
+
+    def try_notify(self, message: bytes) -> bool:
+        """
+        returns true if we send a packet to notify client.
+        fals otherwise (e.g. the socket is dead).
+        """
+        ttl = self.live_after - time.time()
+        if ttl > 0:
+            logger.debug(f"sleeping {ttl:.2f}s until client {self.addr_info} is ready to receive notification")
+            time.sleep(ttl)
+
+        try:
+            self.sock.sendall(message)
+        except Exception as e:
+            logger.warning(f"failed to notify client {self.addr_info} {e}")
+            return False
+        else:
+            logger.info(f"successfully notified {self.addr_info}: {message}")
+            return True
+
+class Adapter:
+    def __init__(self, host: str, port: int, silence: int, topic: str):
+        self.host = host
+        self.port = port
+        self.silence = silence
+        self.topic = topic
+        self.clients = set()
+
+    def log_clients(self):
+        clients_str = '\n'.join(f'  {c.addr_info}' for c in self.clients)
+        logger.debug(f"clients alive ({len(self.clients)}):\n{clients_str}")
+
+    def add_client(self, client: Client):
+        # it's a little bit risky to keep more than one client at the same IP address,
+        # because it's possible a notification comes in and we ring the old connection,
+        # even when the new connection says "don't ring yet".
+        for c in set(self.clients):
+            if c.addr_info[0] == client.addr_info[0]:
+                logger.info(f"purging old client before adding new one at same address: {c.addr_info} -> {client.addr_info}")
+                self.clients.remove(c)
+
+        logger.info(f"accepted client at {client.addr_info}")
+        self.clients.add(client)
+
+    def listener_loop(self):
+        logger.info(f"listening for connections on {self.host}:{self.port}")
+        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        s.bind((self.host, self.port))
+        s.listen(LISTEN_QUEUE)
+        while True:
+            conn, addr_info = s.accept()
+            self.add_client(Client(conn, addr_info, live_after = time.time() + self.silence))
+
+    def notify_clients(self, message: bytes = WAKE_MESSAGE):
+        # notify every client, and drop any which have disconnected.
+        # note that we notify based on age (oldest -> youngest)
+        # because notifying young clients might entail sleeping until they're ready.
+        clients = sorted(self.clients, key=lambda c: (c.live_after, c.addr_info))
+
+        dead_clients = [
+            c for c in clients if not c.try_notify(message)
+        ]
+        for c in dead_clients:
+            self.clients.remove(c)
+
+        self.log_clients()
+
+    def notify_loop(self):
+        logger.info("waiting for notification events")
+        ntfy_proc = subprocess.Popen(
+            [
+                "ntfy",
+                "sub",
+                f"https://ntfy.uninsane.org/{self.topic}"
+            ],
+            stdout=subprocess.PIPE
+        )
+        for line in iter(ntfy_proc.stdout.readline, b''):
+            logger.debug(f"received notification: {line}")
+            self.notify_clients()
+
+def get_topic() -> str:
+    return open('/run/secrets/ntfy-sh-topic', 'rt').read().strip()
+
+def run_forever(callable):
+    try:
+        callable()
+    except Exception as e:
+        logger.error(f"{callable} failed: {e}")
+    else:
+        logger.error(f"{callable} unexpectedly returned")
+    # sys.exit(1)
+    os._exit(1)  # sometimes `sys.exit()` doesn't actually exit...
+
+def main():
+    logging.basicConfig()
+    logging.getLogger().setLevel(logging.DEBUG)
+
+    parser = argparse.ArgumentParser(description="accept connections and notify the other end upon ntfy activity, with a guaranteed amount of silence")
+    parser.add_argument('--verbose', action='store_true')
+    parser.add_argument('--host', type=str, default='')
+    parser.add_argument('--port', type=int)
+    parser.add_argument('--silence', type=int, help="number of seconds to remain silent upon accepting a connection")
+
+    args = parser.parse_args()
+
+    if args.verbose:
+        logging.getLogger().setLevel(logging.DEBUG)
+    else:
+        logging.getLogger().setLevel(logging.INFO)
+
+    adapter = Adapter(args.host, args.port, args.silence, get_topic())
+
+    listener_loop = threading.Thread(target=run_forever, name="listener_loop", args=(adapter.listener_loop,))
+    notify_loop = threading.Thread(target=run_forever, name="notify_loop", args=(adapter.notify_loop,))
+
+    # TODO: this method of exiting seems to sometimes leave the listener behind (?)
+    # preventing anyone else from re-binding the port.
+    listener_loop.start()
+    notify_loop.start()
+    listener_loop.join()
+    notify_loop.join()
+
+
+if __name__ == '__main__':
+    main()

hosts/by-name/servo/services/ntfy/ntfy-waiter.nix

@@ -0,0 +1,72 @@
+# service which adapts ntfy-sh into something suitable specifically for the Pinephone's
+# wake-on-lan (WoL) feature.
+# notably, it provides a mechanism by which the caller can be confident of an interval in which
+# zero traffic will occur on the TCP connection, thus allowing it to enter sleep w/o fear of hitting
+# race conditions in the Pinephone WoL feature.
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.sane.ntfy-waiter;
+  portLow = 5550;
+  portHigh = 5559;
+  portRange = lib.range portLow portHigh;
+  numPorts = portHigh - portLow + 1;
+  mkService = port: let
+    silence = port - portLow;
+    flags = lib.optional cfg.verbose "--verbose";
+    cli = [
+      "${cfg.package}/bin/ntfy-waiter"
+      "--port"
+      "${builtins.toString port}"
+      "--silence"
+      "${builtins.toString silence}"
+    ] ++ flags;
+  in {
+    "ntfy-waiter-${builtins.toString silence}" = {
+      # TODO: run not as root (e.g. as ntfy-sh)
+      description = "wait for notification, with ${builtins.toString silence} seconds of guaranteed silence";
+      serviceConfig = {
+        Type = "simple";
+        Restart = "always";
+        RestartSec = "5s";
+        ExecStart = lib.concatStringsSep " " cli;
+      };
+      after = [ "network.target" ];
+      wantedBy = [ "default.target" ];
+    };
+  };
+in
+{
+  options = with lib; {
+    sane.ntfy-waiter.enable = mkOption {
+      type = types.bool;
+      default = true;
+    };
+    sane.ntfy-waiter.verbose = mkOption {
+      type = types.bool;
+      default = true;
+    };
+    sane.ntfy-waiter.package = mkOption {
+      type = types.package;
+      default = pkgs.static-nix-shell.mkPython3Bin {
+        pname = "ntfy-waiter";
+        srcRoot = ./.;
+        pkgs = [ "ntfy-sh" ];
+      };
+      description = ''
+        exposed to provide an attr-path by which one may build the package for manual testing.
+      '';
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+    sane.ports.ports = lib.mkMerge (lib.forEach portRange (port: {
+      "${builtins.toString port}" = {
+        protocol = [ "tcp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-notification-waiter-${builtins.toString (port - portLow + 1)}-of-${builtins.toString numPorts}";
+      };
+    }));
+    systemd.services = lib.mkMerge (builtins.map mkService portRange);
+  };
+}

hosts/by-name/servo/services/pict-rs.nix

@@ -0,0 +1,23 @@
+# pict-rs is an image database/store used by Lemmy.
+# i don't explicitly activate it here -- just adjust its defaults to be a bit friendlier
+{ config, lib, ... }:
+let
+  cfg = config.services.pict-rs;
+in
+{
+  sane.persist.sys.byStore.plaintext = lib.mkIf cfg.enable [
+    { user = "pict-rs"; group = "pict-rs"; path = cfg.dataDir; method = "bind"; }
+  ];
+
+  systemd.services.pict-rs.serviceConfig = {
+    # fix to use a normal user so we can configure perms correctly
+    DynamicUser = lib.mkForce false;
+    User = "pict-rs";
+    Group = "pict-rs";
+  };
+  users.groups.pict-rs = {};
+  users.users.pict-rs = {
+    group = "pict-rs";
+    isSystemUser = true;
+  };
+}

hosts/by-name/servo/services/pleroma.nix

@@ -1,14 +1,21 @@
 # docs:
-# - https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/networking/pleroma.nix
-# - https://docs.pleroma.social/backend/configuration/cheatsheet/
+# - <https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/networking/pleroma.nix>
+# - <https://docs.pleroma.social/backend/configuration/cheatsheet/>
+# example config:
+# - <https://git.pleroma.social/pleroma/pleroma/-/blob/develop/config/config.exs>
 #
-# to run it in a oci-container: https://github.com/barrucadu/nixfiles/blob/master/services/pleroma.nix
+# to run it in a oci-container: <https://github.com/barrucadu/nixfiles/blob/master/services/pleroma.nix>
+#
+# admin frontend: <https://fed.uninsane.org/pleroma/admin>
 { config, pkgs, ... }:
 
+let
+  logLevel = "warn";
+  # logLevel = "debug";
+in
 {
-  sane.persist.sys.plaintext = [
-    # TODO: mode? could be more granular
-    { user = "pleroma"; group = "pleroma"; directory = "/var/lib/pleroma"; }
+  sane.persist.sys.byStore.plaintext = [
+    { user = "pleroma"; group = "pleroma"; path = "/var/lib/pleroma"; method = "bind"; }
   ];
   services.pleroma.enable = true;
   services.pleroma.secretConfigFile = config.sops.secrets.pleroma_secrets.path;
@@ -18,7 +25,7 @@
 
     config :pleroma, Pleroma.Web.Endpoint,
       url: [host: "fed.uninsane.org", scheme: "https", port: 443],
-      http: [ip: {127, 0, 0, 1}, port: 4000]
+      http: [ip: {127, 0, 0, 1}, port: 4040]
     #   secret_key_base: "{secrets.pleroma.secret_key_base}",
     #   signing_salt: "{secrets.pleroma.signing_salt}"
 
@@ -56,6 +63,7 @@
       database: "pleroma",
       hostname: "localhost",
       pool_size: 10,
+      prepare: :named,
       parameters: [
           plan_cache_mode: "force_custom_plan"
       ]
@@ -96,10 +104,22 @@
       backends: [{ExSyslogger, :ex_syslogger}]
 
     config :logger, :ex_syslogger,
-      level: :warn
-    #  level: :debug
+      level: :${logLevel}
+
+    # policies => list of message rewriting facilities to be enabled
+    # transparence => whether to publish these rules in node_info (and /about)
+    config :pleroma, :mrf,
+      policies: [Pleroma.Web.ActivityPub.MRF.SimplePolicy],
+      transparency: true
+
+    # reject => { host, reason }
+    config :pleroma, :mrf_simple,
+      reject: [ {"threads.net", "megacorp"}, {"*.threads.net", "megacorp"} ]
+      # reject: [ [host: "threads.net", reason: "megacorp"], [host: "*.threads.net", reason: "megacorp"] ]
 
     # XXX colin: not sure if this actually _does_ anything
+    # better to steal emoji from other instances?
+    # - <https://docs.pleroma.social/backend/configuration/cheatsheet/#mrf_steal_emoji>
     config :pleroma, :emoji,
       shortcode_globs: ["/emoji/**/*.png"],
       groups: [
@@ -147,7 +167,8 @@
     enableACME = true;
     # inherit kTLS;
     locations."/" = {
-      proxyPass = "http://127.0.0.1:4000";
+      proxyPass = "http://127.0.0.1:4040";
+      recommendedProxySettings = true;
       # documented: https://git.pleroma.social/pleroma/pleroma/-/blob/develop/installation/pleroma.nginx
       extraConfig = ''
         # XXX colin: this block is in the nixos examples: i don't understand all of it
@@ -166,23 +187,24 @@
         add_header Referrer-Policy same-origin;
         add_header X-Download-Options noopen;
 
-        proxy_http_version 1.1;
-        proxy_set_header Upgrade $http_upgrade;
-        proxy_set_header Connection "upgrade";
-        # proxy_set_header Host $http_host;
-        proxy_set_header Host $host;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        # proxy_http_version 1.1;
+        # proxy_set_header Upgrade $http_upgrade;
+        # proxy_set_header Connection "upgrade";
+        # # proxy_set_header Host $http_host;
+        # proxy_set_header Host $host;
+        # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
         # colin: added this due to Pleroma complaining in its logs
         # proxy_set_header X-Real-IP $remote_addr;
         # proxy_set_header X-Forwarded-Proto $scheme;
 
+        # NB: this defines the maximum upload size
         client_max_body_size 16m;
       '';
     };
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."fed" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."fed" = "native";
 
   sops.secrets."pleroma_secrets" = {
     owner = config.users.users.pleroma.name;

hosts/by-name/servo/services/postgres.nix

@@ -1,12 +1,39 @@
-{ ... }:
+{ pkgs, ... }:
 
+let
+  GiB = n: MiB 1024*n;
+  MiB = n: KiB 1024*n;
+  KiB = n: 1024*n;
+in
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode?
-    { user = "postgres"; group = "postgres"; directory = "/var/lib/postgresql"; }
+    { user = "postgres"; group = "postgres"; path = "/var/lib/postgresql"; method = "bind"; }
   ];
   services.postgresql.enable = true;
-  # services.postgresql.dataDir = "/opt/postgresql/13";
+
+  # HOW TO UPDATE:
+  # postgres version updates are manual and require intervention.
+  # - `sane-stop-all-servo`
+  # - `systemctl start postgresql`
+  # - as `sudo su postgres`:
+  #   - `cd /var/log/postgresql`
+  #   - `pg_dumpall > state.sql`
+  #   - `echo placeholder > <new_version>`  # to prevent state from being created earlier than we want
+  # - then, atomically:
+  #   - update the `services.postgresql.package` here
+  #   - `dataDir` is atomically updated to match package; don't touch
+  #   - `nixos-rebuild --flake . switch ; sane-stop-all-servo`
+  # - `sudo rm -rf /var/lib/postgresql/<new_version>`
+  # - `systemctl start postgresql`
+  # - as `sudo su postgres`:
+  #   - `cd /var/lib/postgreql`
+  #   - `psql -f state.sql`
+  # - restart dependent services (maybe test one at a time)
+
+  services.postgresql.package = pkgs.postgresql_15;
+
+
   # XXX colin: for a proper deploy, we'd want to include something for Pleroma here too.
   # services.postgresql.initialScript = pkgs.writeText "synapse-init.sql" ''
   #   CREATE ROLE "matrix-synapse" WITH LOGIN PASSWORD '<password goes here>';
@@ -17,10 +44,33 @@
   #     LC_CTYPE = "C";
   # '';
 
-  # TODO: perf tuning
+  # perf tuning
   # - for recommended values see: <https://pgtune.leopard.in.ua/>
   # - for official docs (sparse), see: <https://www.postgresql.org/docs/11/config-setting.html#CONFIG-SETTING-CONFIGURATION-FILE>
-  # services.postgresql.settings = { ... }
+  services.postgresql.settings = {
+    # DB Version: 15
+    # OS Type: linux
+    # DB Type: web
+    # Total Memory (RAM): 32 GB
+    # CPUs num: 12
+    # Data Storage: ssd
+    max_connections = 200;
+    shared_buffers = "8GB";
+    effective_cache_size = "24GB";
+    maintenance_work_mem = "2GB";
+    checkpoint_completion_target = 0.9;
+    wal_buffers = "16MB";
+    default_statistics_target = 100;
+    random_page_cost = 1.1;
+    effective_io_concurrency = 200;
+    work_mem = "10485kB";
+    min_wal_size = "1GB";
+    max_wal_size = "4GB";
+    max_worker_processes = 12;
+    max_parallel_workers_per_gather = 4;
+    max_parallel_workers = 12;
+    max_parallel_maintenance_workers = 4;
+  };
 
   # daily backups to /var/backup
   services.postgresqlBackup.enable = true;

hosts/by-name/servo/services/prosody.nix

@@ -1,64 +0,0 @@
-# example configs:
-# - <https://github.com/kittywitch/nixfiles/blob/main/services/prosody.nix>
-# create users with:
-# - `sudo -u prosody prosodyctl adduser colin@uninsane.org`
-
-{ lib, ... }:
-
-# XXX disabled: doesn't send messages to nixnet.social (only receives them).
-# nixnet runs ejabberd, so revisiting that.
-lib.mkIf false
-{
-  sane.persist.sys.plaintext = [
-    { user = "prosody"; group = "prosody"; directory = "/var/lib/prosody"; }
-  ];
-  networking.firewall.allowedTCPPorts = [
-    5222  # XMPP client -> server
-    5269  # XMPP server -> server
-    5280  # bosh
-    5281  # Prosody HTTPS port  (necessary?)
-  ];
-
-  # provide access to certs
-  users.users.prosody.extraGroups = [ "nginx" ];
-
-  security.acme.certs."uninsane.org".extraDomainNames = [
-    "conference.xmpp.uninsane.org"
-    "upload.xmpp.uninsane.org"
-  ];
-
-  services.prosody = {
-    enable = true;
-    admins = [ "colin@uninsane.org" ];
-    # allowRegistration = false;
-    # extraConfig = ''
-    #   s2s_require_encryption = true
-    #   c2s_require_encryption = true
-    # '';
-
-    extraModules = [ "private" "vcard" "privacy" "compression" "component" "muc" "pep" "adhoc" "lastactivity" "admin_adhoc" "blocklist"];
-
-    ssl.cert = "/var/lib/acme/uninsane.org/fullchain.pem";
-    ssl.key = "/var/lib/acme/uninsane.org/key.pem";
-
-    muc = [
-      {
-        domain = "conference.xmpp.uninsane.org";
-      }
-    ];
-    uploadHttp.domain = "upload.xmpp.uninsane.org";
-
-    virtualHosts = {
-      localhost = {
-        domain = "localhost";
-        enabled = true;
-      };
-      "xmpp.uninsane.org" = {
-        domain = "uninsane.org";
-        enabled = true;
-        ssl.cert = "/var/lib/acme/uninsane.org/fullchain.pem";
-        ssl.key = "/var/lib/acme/uninsane.org/key.pem";
-      }; 
-    };
-  };
-}

hosts/by-name/servo/services/prosody/default.nix

@@ -0,0 +1,289 @@
+# example configs:
+# - official: <https://prosody.im/doc/example_config>
+# - nixos: <https://github.com/kittywitch/nixfiles/blob/main/services/prosody.nix>
+# config options:
+# - <https://prosody.im/doc/configure>
+#
+# modules:
+# - main: <https://prosody.im/doc/modules>
+# - community: <https://modules.prosody.im/index.html>
+#
+# debugging:
+# - logging:
+#   - enable `stanza_debug` module
+#   - enable `log.debug = "*syslog"` in extraConfig
+# - interactive:
+#   - `telnet localhost 5582` (this is equal to `prosodyctl shell` -- but doesn't hang)
+#     - `watch:stanzas(target_spec, filter)` -> to log stanzas, for version > 0.12
+#   - console docs: <https://prosody.im/doc/console>
+#   - can modify/inspect arbitrary internals (lua) by prefixing line with `> `
+#     - e.g. `> _G` to print all globals
+#
+# sanity checks:
+# - `sudo -u prosody -g prosody prosodyctl check connectivity`
+# - `sudo -u prosody -g prosody prosodyctl check turn`
+# - `sudo -u prosody -g prosody prosodyctl check turn -v --ping=stun.conversations.im`
+#   - checks that my stun/turn server is usable by clients of conversations.im (?)
+# - `sudo -u prosody -g prosody prosodyctl check`  (dns, config, certs)
+#
+#
+# create users with:
+# - `sudo -u prosody prosodyctl adduser colin@uninsane.org`
+#
+#
+# federation/support matrix:
+# - nixnet.services (runs ejabberd):
+#   - WORKS: sending and receiving PMs and calls (2023/10/15)
+#     - N.B.: it didn't originally work; was solved by disabling the lua-unbound DNS option & forcing the system/local resolver
+# - cheogram (XMPP <-> SMS gateway):
+#   - WORKS: sending and receiving PMs, images (2023/10/15)
+#   - PARTIAL: calls (xmpp -> tel works; tel -> xmpp fails)
+#     - maybe i need to setup stun/turn
+#
+# TODO:
+# - enable push notifications (mod_cloud_notify)
+# - optimize coturn (e.g. move off of the VPN!)
+# - ensure muc is working
+# - enable file uploads
+#   - "upload.xmpp.uninsane.org:http_upload: URL: <https://upload.xmpp.uninsane.org:5281/upload> - Ensure this can be reached by users"
+# - disable or fix bosh (jabber over http):
+#   - "certmanager: No certificate/key found for client_https port 0"
+
+{ lib, pkgs, ... }:
+
+let
+  # enables very verbose logging
+  enableDebug = false;
+in
+{
+  sane.persist.sys.byStore.plaintext = [
+    { user = "prosody"; group = "prosody"; path = "/var/lib/prosody"; method = "bind"; }
+  ];
+  sane.ports.ports."5000" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpp-prosody-fileshare-proxy65";
+  };
+  sane.ports.ports."5222" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpp-client-to-server";
+  };
+  sane.ports.ports."5223" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpps-client-to-server";  # XMPP over TLS
+  };
+  sane.ports.ports."5269" = {
+    protocol = [ "tcp" ];
+    visibleTo.wan = true;
+    description = "colin-xmpp-server-to-server";
+  };
+  sane.ports.ports."5270" = {
+    protocol = [ "tcp" ];
+    visibleTo.wan = true;
+    description = "colin-xmpps-server-to-server";  # XMPP over TLS
+  };
+  sane.ports.ports."5280" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpp-bosh";
+  };
+  sane.ports.ports."5281" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpp-prosody-https";  # necessary?
+  };
+
+  users.users.prosody.extraGroups = [
+    "nginx"  # provide access to certs
+    "ntfy-sh"  # access to secret ntfy topic
+  ];
+
+  security.acme.certs."uninsane.org".extraDomainNames = [
+    "xmpp.uninsane.org"
+    "conference.xmpp.uninsane.org"
+    "upload.xmpp.uninsane.org"
+  ];
+
+  # exists so the XMPP server's cert can obtain altNames for all its resources
+  services.nginx.virtualHosts."xmpp.uninsane.org" = {
+    useACMEHost = "uninsane.org";
+  };
+  services.nginx.virtualHosts."conference.xmpp.uninsane.org" = {
+    useACMEHost = "uninsane.org";
+  };
+  services.nginx.virtualHosts."upload.xmpp.uninsane.org" = {
+    useACMEHost = "uninsane.org";
+  };
+
+  sane.dns.zones."uninsane.org".inet = {
+    # XXX: SRV records have to point to something with a A/AAAA record; no CNAMEs
+    A."xmpp" =                "%ANATIVE%";
+    CNAME."conference.xmpp" = "xmpp";
+    CNAME."upload.xmpp" =     "xmpp";
+
+    # _Service._Proto.Name    TTL Class SRV    Priority Weight Port Target
+    # - <https://xmpp.org/extensions/xep-0368.html>
+    # something's requesting the SRV records for conference.xmpp, so let's include it
+    # nothing seems to request XMPP SRVs for the other records (except @)
+    # lower numerical priority field tells clients to prefer this method
+    SRV."_xmpps-client._tcp.conference.xmpp" =       "3 50 5223 xmpp";
+    SRV."_xmpps-server._tcp.conference.xmpp" =       "3 50 5270 xmpp";
+    SRV."_xmpp-client._tcp.conference.xmpp" =        "5 50 5222 xmpp";
+    SRV."_xmpp-server._tcp.conference.xmpp" =        "5 50 5269 xmpp";
+
+    SRV."_xmpps-client._tcp" =                "3 50 5223 xmpp";
+    SRV."_xmpps-server._tcp" =                "3 50 5270 xmpp";
+    SRV."_xmpp-client._tcp" =                 "5 50 5222 xmpp";
+    SRV."_xmpp-server._tcp" =                 "5 50 5269 xmpp";
+  };
+
+  # help Prosody find its certificates.
+  # pointing it to /var/lib/acme doesn't quite work because it expects the private key
+  # to be named `privkey.pem` instead of acme's `key.pem`
+  # <https://prosody.im/doc/certificates#automatic_location>
+  sane.fs."/etc/prosody/certs/uninsane.org/fullchain.pem" = {
+    symlink.target = "/var/lib/acme/uninsane.org/fullchain.pem";
+    wantedBeforeBy = [ "prosody.service" ];
+  };
+  sane.fs."/etc/prosody/certs/uninsane.org/privkey.pem" = {
+    symlink.target = "/var/lib/acme/uninsane.org/key.pem";
+    wantedBeforeBy = [ "prosody.service" ];
+  };
+
+  services.prosody = {
+    enable = true;
+    package = pkgs.prosody.override {
+      # XXX(2023/10/15): build without lua-unbound support.
+      # this forces Prosody to fall back to the default Lua DNS resolver, which seems more reliable.
+      # fixes errors like "unbound.queryXYZUV: Resolver error: out of memory"
+      # related: <https://issues.prosody.im/1737#comment-11>
+      lua.withPackages = selector: pkgs.lua.withPackages (p:
+        selector (p // { luaunbound = null; })
+      );
+      # withCommunityModules = [ "turncredentials" ];
+    };
+    admins = [ "colin@uninsane.org" ];
+    # allowRegistration = false;  # defaults to false
+
+    muc = [
+      {
+        domain = "conference.xmpp.uninsane.org";
+      }
+    ];
+    uploadHttp.domain = "upload.xmpp.uninsane.org";
+
+    virtualHosts = {
+      # "Prosody requires at least one enabled VirtualHost to function. You can
+      # safely remove or disable 'localhost' once you have added another."
+      # localhost = {
+      #   domain = "localhost";
+      #   enabled = true;
+      # };
+      "xmpp.uninsane.org" = {
+        domain = "uninsane.org";
+        enabled = true;
+      };
+    };
+
+    ## modules:
+    # these are enabled by default, via <repo:nixos/nixpkgs:/pkgs/servers/xmpp/prosody/default.nix>
+    # - cloud_notify
+    # - http_upload
+    # - vcard_muc
+    # these are enabled by the module defaults (services.prosody.modules.<foo>)
+    # - admin_adhoc
+    # - blocklist
+    # - bookmarks
+    # - carbons
+    # - cloud_notify
+    # - csi
+    # - dialback
+    # - disco
+    # - http_files
+    # - mam
+    # - pep
+    # - ping
+    # - private
+    #   - XEP-0049: let clients store arbitrary (private) data on the server
+    # - proxy65
+    #   - XEP-0065: allow server to proxy file transfers between two clients who are behind NAT
+    # - register
+    # - roster
+    # - saslauth
+    # - smacks
+    # - time
+    # - tls
+    # - uptime
+    # - vcard_legacy
+    # - version
+
+    extraPluginPaths = [ ./modules ];
+
+    extraModules = [
+      # admin_shell: allows `prosodyctl shell` to work
+      # see: <https://prosody.im/doc/modules/mod_admin_shell>
+      # see: <https://prosody.im/doc/console>
+      "admin_shell"
+      "admin_telnet"  #< needed by admin_shell
+      # lastactivity: XEP-0012: allow users to query how long another user has been idle for
+      # - not sure why i enabled this; think it was in someone's config i referenced
+      "lastactivity"
+      # allows prosody to share TURN/STUN secrets with XMPP clients to provide them access to the coturn server.
+      # see: <https://prosody.im/doc/coturn>
+      "turn_external"
+      # legacy coturn integration
+      # see: <https://modules.prosody.im/mod_turncredentials.html>
+      # "turncredentials"
+      "sane_ntfy"
+    ] ++ lib.optionals enableDebug [
+      "stanza_debug"  #< logs EVERY stanza as debug: <https://prosody.im/doc/modules/mod_stanza_debug>
+    ];
+
+    extraConfig = ''
+      local function readAll(file)
+        local f = assert(io.open(file, "rb"))
+        local content = f:read("*all")
+        f:close()
+        -- remove trailing newline
+        return string.gsub(content, "%s+", "")
+      end
+
+      -- logging docs:
+      -- - <https://prosody.im/doc/logging>
+      -- - <https://prosody.im/doc/advanced_logging>
+      -- levels: debug, info, warn, error
+      log = {
+        ${if enableDebug then "debug" else "info"} = "*syslog";
+      }
+
+      -- see: <https://prosody.im/doc/certificates#automatic_location>
+      -- try to solve: "certmanager: Error indexing certificate directory /etc/prosody/certs: cannot open /etc/prosody/certs: No such file or directory"
+      -- only, this doesn't work because prosody doesn't like acme's naming scheme
+      -- certificates = "/var/lib/acme"
+
+      c2s_direct_tls_ports = { 5223 }
+      s2s_direct_tls_ports = { 5270 }
+
+      turn_external_host = "turn.uninsane.org"
+      turn_external_secret = readAll("/var/lib/coturn/shared_secret.bin")
+      -- turn_external_user = "prosody"
+
+      -- legacy mod_turncredentials integration
+      -- turncredentials_host = "turn.uninsane.org"
+      -- turncredentials_secret = readAll("/var/lib/coturn/shared_secret.bin")
+
+      ntfy_binary = "${pkgs.ntfy-sh}/bin/ntfy"
+      ntfy_topic = readAll("/run/secrets/ntfy-sh-topic")
+
+      -- s2s_require_encryption = true
+      -- c2s_require_encryption = true
+    '';
+  };
+}

hosts/by-name/servo/services/prosody/modules/mod_sane_ntfy.lua

@@ -0,0 +1,52 @@
+-- simple proof-of-concept Prosody module
+-- module development guide: <https://prosody.im/doc/developers/modules>
+-- module API docs: <https://prosody.im/doc/developers/moduleapi>
+--
+-- much of this code is lifted from Prosody's own `mod_cloud_notify`
+
+local jid = require"util.jid";
+
+local ntfy = module:get_option_string("ntfy_binary", "ntfy");
+local ntfy_topic = module:get_option_string("ntfy_topic", "xmpp");
+
+module:log("info", "initialized");
+
+local function is_urgent(stanza)
+  if stanza.name == "message" then
+    if stanza:get_child("propose", "urn:xmpp:jingle-message:0") then
+      return true, "jingle call";
+    end
+  end
+end
+
+local function publish_ntfy(message)
+  -- message should be the message to publish
+  local ntfy_url = string.format("https://ntfy.uninsane.org/%s", ntfy_topic)
+  local cmd = string.format("%s pub %q %q", ntfy, ntfy_url, message)
+  module.log("debug", "invoking ntfy: %s", cmd)
+  local success, reason, code = os.execute(cmd)
+  if not success then
+    module:log("warn", "ntfy failed: %s => %s %d", cmd, reason, code)
+  end
+end
+
+
+local function archive_message_added(event)
+  -- event is: { origin = origin, stanza = stanza, for_user = store_user, id = id }
+  local stanza = event.stanza;
+  local to = stanza.attr.to;
+  to = to and jid.split(to) or event.origin.username;
+
+  -- only notify if the stanza destination is the mam user we store it for
+  if event.for_user == to then
+    local is_urgent_stanza, urgent_reason = is_urgent(event.stanza);
+
+    if is_urgent_stanza then
+      module:log("info", "urgent push for %s (%s)", to, urgent_reason);
+      publish_ntfy(urgent_reason)
+    end
+  end
+end
+
+
+module:hook("archive-message-added", archive_message_added);

hosts/by-name/servo/services/slskd.nix

@@ -0,0 +1,80 @@
+# Soulseek daemon (p2p file sharing with an emphasis on Music)
+# docs: <https://github.com/slskd/slskd/blob/master/docs/config.md>
+#
+# config precedence (higher precedence overrules lower precedence):
+# - Default Values < Environment Variables < YAML Configuraiton File < Command Line Arguments
+#
+# debugging:
+# - soulseek is just *flaky*. if you see e.g. DNS errors, even though you can't replicate them via `dig` or `getent ahostsv4`, just give it 10 minutes to work out:
+#   - "Soulseek.AddressException: Failed to resolve address 'vps.slsknet.org': Resource temporarily unavailable"
+{ config, lib, pkgs, ... }:
+
+{
+  sane.persist.sys.byStore.plaintext = [
+    { user = "slskd"; group = "media"; path = "/var/lib/slskd"; method = "bind"; }
+  ];
+  sops.secrets."slskd_env" = {
+    owner = config.users.users.slskd.name;
+    mode = "0400";
+  };
+
+  users.users.slskd.extraGroups = [ "media" ];
+
+  sane.ports.ports."50300" = {
+    protocol = [ "tcp" ];
+    # not visible to WAN: i run this in a separate netns
+    visibleTo.ovpn = true;
+    description = "colin-soulseek";
+  };
+
+  sane.dns.zones."uninsane.org".inet.CNAME."soulseek" = "native";
+
+  services.nginx.virtualHosts."soulseek.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+    locations."/" = {
+      proxyPass = "http://10.0.1.6:5030";
+      proxyWebsockets = true;
+    };
+  };
+
+  services.slskd.enable = true;
+  services.slskd.domain = null;  # i'll manage nginx for it
+  services.slskd.group = "media";
+  # env file, for auth (SLSKD_SLSK_PASSWORD, SLSKD_SLSK_USERNAME)
+  services.slskd.environmentFile = config.sops.secrets.slskd_env.path;
+  services.slskd.settings = {
+    soulseek.diagnostic_level = "Debug";  # one of "None"|"Warning"|"Info"|"Debug"
+    shares.directories = [
+      # folders to share
+      # syntax: <https://github.com/slskd/slskd/blob/master/docs/config.md#directories>
+      # [Alias]/path/on/disk
+      # NOTE: Music library is quick to scan; videos take a solid 10min to scan.
+      # TODO: re-enable the other libraries
+      # "[Audioooks]/var/media/Books/Audiobooks"
+      # "[Books]/var/media/Books/Books"
+      # "[Manga]/var/media/Books/Visual"
+      # "[games]/var/media/games"
+      "[Music]/var/media/Music"
+      # "[Film]/var/media/Videos/Film"
+      # "[Shows]/var/media/Videos/Shows"
+    ];
+    # directories.downloads = "..." # TODO
+    # directories.incomplete = "..." # TODO
+    # what unit is this? kbps??
+    global.upload.speed_limit = 32000;
+    web.logging = true;
+    # debug = true;
+    flags.no_logo = true;  # don't show logo at start
+    # flags.volatile = true;  # store searches and active transfers in RAM (completed transfers still go to disk). rec for btrfs/zfs
+  };
+
+  systemd.services.slskd.serviceConfig = {
+    # run this behind the OVPN static VPN
+    NetworkNamespacePath = "/run/netns/ovpns";
+    ExecStartPre = [ "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect 185.157.162.178" ];  # abort if public IP is not as expected
+
+    Restart = lib.mkForce "always";  # exits "success" when it fails to connect to soulseek server
+    RestartSec = "60s";
+  };
+}

hosts/by-name/servo/services/transmission.nix

@@ -1,13 +1,113 @@
-{ pkgs, ... }:
+{ config, lib, pkgs, ... }:
 
+let
+  # 2023/09/06: nixpkgs `transmission` defaults to old 3.00
+  # 2024/02/15: some torrent trackers whitelist clients; everyone is still on 3.00 for some reason :|
+  #             some do this via peer-id (e.g. baka); others via user-agent (e.g. MAM).
+  #             peer-id format is essentially the same between 3.00 and 4.x (just swap the MAJOR/MINOR/PATCH numbers).
+  #             user-agent format has changed. `Transmission/3.00` (old) v.s. `TRANSMISSION/MAJ.MIN.PATCH` (new).
+  realTransmission = pkgs.transmission_4;
+  realVersion = {
+    major = lib.versions.major realTransmission.version;
+    minor = lib.versions.minor realTransmission.version;
+    patch = lib.versions.patch realTransmission.version;
+  };
+  package = realTransmission.overrideAttrs (upstream: {
+    # `cmakeFlags = [ "-DTR_VERSION_MAJOR=3" ]`, etc, doesn't seem to take effect.
+    postPatch = (upstream.postPatch or "") + ''
+      substituteInPlace CMakeLists.txt \
+        --replace-fail 'TR_VERSION_MAJOR "${realVersion.major}"' 'TR_VERSION_MAJOR "3"' \
+        --replace-fail 'TR_VERSION_MINOR "${realVersion.minor}"' 'TR_VERSION_MINOR "0"' \
+        --replace-fail 'TR_VERSION_PATCH "${realVersion.patch}"' 'TR_VERSION_PATCH "0"' \
+        --replace-fail 'set(TR_USER_AGENT_PREFIX "''${TR_SEMVER}")' 'set(TR_USER_AGENT_PREFIX "3.00")'
+    '';
+  });
+  download-dir = "/var/media/torrents";
+  torrent-done = pkgs.writeShellApplication {
+    name = "torrent-done";
+    runtimeInputs = with pkgs; [
+      acl
+      coreutils
+      findutils
+      rsync
+      util-linux
+    ];
+    text = ''
+      destructive() {
+        if [ -n "''${TR_DRY_RUN-}" ]; then
+          echo "$*"
+        else
+          "$@"
+        fi
+      }
+      if [[ "$TR_TORRENT_DIR" =~ ^.*freeleech.*$ ]]; then
+        # freeleech torrents have no place in my permanent library
+        echo "freeleech: nothing to do"
+        exit 0
+      fi
+      if ! [[ "$TR_TORRENT_DIR" =~ ^${download-dir}/.*$ ]]; then
+        echo "unexpected torrent dir, aborting: $TR_TORRENT_DIR"
+        exit 0
+      fi
+
+      REL_DIR="''${TR_TORRENT_DIR#${download-dir}/}"
+      MEDIA_DIR="/var/media/$REL_DIR"
+
+      destructive mkdir -p "$(dirname "$MEDIA_DIR")"
+      destructive rsync -arv "$TR_TORRENT_DIR/" "$MEDIA_DIR/"
+      # make the media rwx by anyone in the group
+      destructive find "$MEDIA_DIR" -type d -exec setfacl --recursive --modify d:g::rwx,o::rx {} \;
+      destructive find "$MEDIA_DIR" -type d -exec chmod g+rw,a+rx {} \;
+
+      # if there's a single directory inside the media dir, then inline that
+      subdirs=("$MEDIA_DIR"/*)
+      if [ ''${#subdirs} -eq 1 ]; then
+        dirname="''${subdirs[0]}"
+        if [ -d "$dirname" ]; then
+          mv "$dirname"/* "$MEDIA_DIR/" && rmdir "$dirname"
+        fi
+      fi
+
+      # remove noisy files:
+      find "$MEDIA_DIR/" -type f \(\
+           -iname 'www.YTS.*.jpg' \
+        -o -iname 'WWW.YIFY*.COM.jpg' \
+        -o -iname 'YIFY*.com.txt' \
+        -o -iname 'YTS*.com.txt' \
+        \) -exec rm {} \;
+
+      # dedupe the whole media library.
+      # yeah, a bit excessive: move this to a cron job if that's problematic.
+      destructive hardlink /var/media --reflink=always --ignore-time --verbose
+    '';
+  };
+in
+lib.mkIf false  #< TODO: re-enable once confident of sandboxing
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? we need this specifically for the stats tracking in .config/
-    { user = "transmission"; group = "transmission"; directory = "/var/lib/transmission"; }
+    { user = "transmission"; group = config.users.users.transmission.group; path = "/var/lib/transmission"; method = "bind"; }
   ];
+  users.users.transmission.extraGroups = [ "media" ];
+
   services.transmission.enable = true;
+  services.transmission.package = package;
+  #v setting `group` this way doesn't tell transmission to `chown` the files it creates
+  #  it's a nixpkgs setting which just runs the transmission daemon as this group
+  services.transmission.group = "media";
+
+  # transmission will by default not allow the world to read its files.
+  services.transmission.downloadDirPermissions = "775";
+  services.transmission.extraFlags = [
+    # "--log-level=debug"
+  ];
+
   services.transmission.settings = {
-    rpc-bind-address = "0.0.0.0";
+    # DOCUMENTATION/options list: <https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md#options>
+
+    # message-level = 3;  #< enable for debug logging. 0-3, default is 2.
+    # 10.0.1.6 => allow rpc only from the root servo ns. it'll tunnel things to the net, if need be.
+    rpc-bind-address = "10.0.1.6";
     #rpc-host-whitelist = "bt.uninsane.org";
     #rpc-whitelist = "*.*.*.*";
     rpc-authentication-required = true;
@@ -17,35 +117,54 @@
     rpc-password = "{503fc8928344f495efb8e1f955111ca5c862ce0656SzQnQ5";
     rpc-whitelist-enabled = false;
 
-    # download-dir = "/opt/uninsane/media/";
+    # force behind ovpns in case the NetworkNamespace fails somehow
+    bind-address-ipv4 = "185.157.162.178";
+    port-forwarding-enabled = false;
+
     # hopefully, make the downloads world-readable
-    umask = 0;
+    # umask = 0;  #< default is 2: i.e. deny writes from world
 
     # force peer connections to be encrypted
     encryption = 2;
 
     # units in kBps
-    speed-limit-down = 3000;
+    speed-limit-down = 12000;
     speed-limit-down-enabled = true;
-    speed-limit-up = 300;
+    speed-limit-up = 800;
     speed-limit-up-enabled = true;
 
     # see: https://git.zknt.org/mirror/transmission/commit/cfce6e2e3a9b9d31a9dafedd0bdc8bf2cdb6e876?lang=bg-BG
     anti-brute-force-enabled = false;
 
-    download-dir = "/var/lib/uninsane/media";
-    incomplete-dir = "/var/lib/uninsane/media/incomplete";
+    inherit download-dir;
+    incomplete-dir = "${download-dir}/incomplete";
+    # transmission regularly fails to move stuff from the incomplete dir to the main one, so disable:
+    incomplete-dir-enabled = false;
 
+    # env vars available in script:
+    # - TR_APP_VERSION - Transmission's short version string, e.g. `4.0.0`
+    # - TR_TIME_LOCALTIME
+    # - TR_TORRENT_BYTES_DOWNLOADED - Number of bytes that were downloaded for this torrent
+    # - TR_TORRENT_DIR - Location of the downloaded data
+    # - TR_TORRENT_HASH - The torrent's info hash
+    # - TR_TORRENT_ID
+    # - TR_TORRENT_LABELS - A comma-delimited list of the torrent's labels
+    # - TR_TORRENT_NAME - Name of torrent (not filename)
+    # - TR_TORRENT_TRACKERS - A comma-delimited list of the torrent's trackers' announce URLs
+    script-torrent-done-enabled = true;
+    script-torrent-done-filename = "${torrent-done}/bin/torrent-done";
   };
-  # transmission will by default not allow the world to read its files.
-  services.transmission.downloadDirPermissions = "775";
 
   systemd.services.transmission.after = [ "wireguard-wg-ovpns.service" ];
   systemd.services.transmission.partOf = [ "wireguard-wg-ovpns.service" ];
   systemd.services.transmission.serviceConfig = {
     # run this behind the OVPN static VPN
     NetworkNamespacePath = "/run/netns/ovpns";
-    LogLevelMax = "warning";
+    ExecStartPre = [ "${lib.getExe pkgs.sane-scripts.ip-check} --no-upnp --expect 185.157.162.178" ];  # abort if public IP is not as expected
+
+    Restart = "on-failure";
+    RestartSec = "30s";
+    BindPaths = [ "/var/media" ];  #< so it can move completed torrents into the media library
   };
 
   # service to automatically backup torrents i add to transmission
@@ -75,6 +194,11 @@
     };
   };
 
-  sane.services.trust-dns.zones."uninsane.org".inet.CNAME."bt" = "native";
+  sane.dns.zones."uninsane.org".inet.CNAME."bt" = "native";
+  sane.ports.ports."51413" = {
+    protocol = [ "tcp" "udp" ];
+    visibleTo.ovpn = true;
+    description = "colin-bittorrent";
+  };
 }
 

hosts/by-name/servo/services/trust-dns.nix

@@ -1,17 +1,21 @@
-{ config, pkgs, ... }:
+# TODO: split this file apart into smaller files to make it easier to understand
+{ config, lib, pkgs, ... }:
 
+let
+  dyn-dns = config.sane.services.dyn-dns;
+  nativeAddrs = lib.mapAttrs (_name: builtins.head) config.sane.dns.zones."uninsane.org".inet.A;
+  bindOvpn = "10.0.1.5";
+in
 {
-  sane.services.trust-dns.enable = true;
+  sane.ports.ports."53" = {
+    protocol = [ "udp" "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    visibleTo.ovpn = true;
+    description = "colin-dns-hosting";
+  };
 
-  sane.services.trust-dns.listenAddrsIPv4 = [
-    # specify each address explicitly, instead of using "*".
-    # this ensures responses are sent from the address at which the request was received.
-    "192.168.0.5"
-    "10.0.1.5"
-  ];
-  sane.services.trust-dns.quiet = true;
-
-  sane.services.trust-dns.zones."uninsane.org".TTL = 900;
+  sane.dns.zones."uninsane.org".TTL = 900;
 
   # SOA record structure: <https://en.wikipedia.org/wiki/SOA_record#Structure>
   # SOA MNAME RNAME (... rest)
@@ -21,26 +25,30 @@
   # Refresh = how frequently secondary NS should query master
   # Retry = how long secondary NS should wait until re-querying master after a failure (must be < Refresh)
   # Expire = how long secondary NS should continue to reply to queries after master fails (> Refresh + Retry)
-  sane.services.trust-dns.zones."uninsane.org".inet = {
+  sane.dns.zones."uninsane.org".inet = {
     SOA."@" = ''
       ns1.uninsane.org. admin-dns.uninsane.org. (
-                                      2022122101 ; Serial
+                                      2023092101 ; Serial
                                       4h         ; Refresh
                                       30m        ; Retry
                                       7d         ; Expire
                                       5m)        ; Negative response TTL
     '';
-    TXT."rev" = "2022122101";
+    TXT."rev" = "2023092101";
+
+    CNAME."native" = "%CNAMENATIVE%";
+    A."@" =      "%ANATIVE%";
+    A."servo.wan" = "%AWAN%";
+    A."servo.lan" = config.sane.hosts.by-name."servo".lan-ip;
+    A."servo.hn" = config.sane.hosts.by-name."servo".wg-home.ip;
 
     # XXX NS records must also not be CNAME
     # it's best that we keep this identical, or a superset of, what org. lists as our NS.
     # so, org. can specify ns2/ns3 as being to the VPN, with no mention of ns1. we provide ns1 here.
-    A."ns1" =    "%NATIVE%";
+    A."ns1" =    "%ANATIVE%";
     A."ns2" =    "185.157.162.178";
     A."ns3" =    "185.157.162.178";
     A."ovpns" =  "185.157.162.178";
-    A."native" = "%NATIVE%";
-    A."@" =      "%NATIVE%";
     NS."@" = [
       "ns1.uninsane.org."
       "ns2.uninsane.org."
@@ -48,20 +56,104 @@
     ];
   };
 
-  sane.services.trust-dns.zones."uninsane.org".file =
-    "/var/lib/trust-dns/uninsane.org.zone";
+  services.trust-dns.settings.zones = [ "uninsane.org" ];
 
-  systemd.services.trust-dns.preStart = let
-    sed = "${pkgs.gnused}/bin/sed";
-    zone-dir = "/var/lib/trust-dns";
-    zone-out = "${zone-dir}/uninsane.org.zone";
-    zone-template = pkgs.writeText "uninsane.org.zone.in" config.sane.services.trust-dns.generatedZones."uninsane.org";
-  in ''
-    # make WAN records available to trust-dns
-    mkdir -p ${zone-dir}
-    ip=$(cat '${config.sane.services.dyn-dns.ipPath}')
-    ${sed} s/%NATIVE%/$ip/ ${zone-template} > ${zone-out}
+
+  networking.nat.enable = true;
+  networking.nat.extraCommands = ''
+    # redirect incoming DNS requests from LAN addresses
+    #   to the LAN-specialized DNS service
+    # N.B.: use the `nixos-*` chains instead of e.g. PREROUTING
+    #   because they get cleanly reset across activations or `systemctl restart firewall`
+    #   instead of accumulating cruft
+    iptables -t nat -A nixos-nat-pre -p udp --dport 53 \
+      -m iprange --src-range 10.78.76.0-10.78.79.255 \
+      -j DNAT --to-destination :1053
+    iptables -t nat -A nixos-nat-pre -p tcp --dport 53 \
+      -m iprange --src-range 10.78.76.0-10.78.79.255 \
+      -j DNAT --to-destination :1053
   '';
+  sane.ports.ports."1053" = {
+    # because the NAT above redirects in nixos-nat-pre, LAN requests behave as though they arrived on the external interface at the redirected port.
+    # TODO: try nixos-nat-post instead?
+    # TODO: or, don't NAT from port 53 -> port 1053, but rather nat from LAN addr to a loopback addr.
+    # - this is complicated in that loopback is a different interface than eth0, so rewriting the destination address would cause the packets to just be dropped by the interface
+    protocol = [ "udp" "tcp" ];
+    visibleTo.lan = true;
+    description = "colin-redirected-dns-for-lan-namespace";
+  };
 
-  sane.services.dyn-dns.restartOnChange = [ "trust-dns.service" ];
+
+  sane.services.trust-dns.enable = true;
+  sane.services.trust-dns.instances = let
+    mkSubstitutions = flavor: {
+      "%AWAN%" = "$(cat '${dyn-dns.ipPath}')";
+      "%CNAMENATIVE%" = "servo.${flavor}";
+      "%ANATIVE%" = nativeAddrs."servo.${flavor}";
+      "%AOVPNS%" = "185.157.162.178";
+    };
+  in
+  {
+    wan = {
+      substitutions = mkSubstitutions "wan";
+      listenAddrsIpv4 = [
+        nativeAddrs."servo.lan"
+        bindOvpn
+      ];
+    };
+    lan = {
+      substitutions = mkSubstitutions "lan";
+      listenAddrsIpv4 = [ nativeAddrs."servo.lan" ];
+      port = 1053;
+    };
+    hn = {
+      substitutions = mkSubstitutions "hn";
+      listenAddrsIpv4 = [ nativeAddrs."servo.hn" ];
+      port = 1053;
+    };
+    # hn-resolver = {
+    #   # don't need %AWAN% here because we forward to the hn instance.
+    #   listenAddrsIpv4 = [ nativeAddrs."servo.hn" ];
+    #   extraConfig = {
+    #     zones = [
+    #       {
+    #         zone = "uninsane.org";
+    #         zone_type = "Forward";
+    #         stores = {
+    #           type = "forward";
+    #           name_servers = [
+    #             {
+    #               socket_addr = "${nativeAddrs."servo.hn"}:1053";
+    #               protocol = "udp";
+    #               trust_nx_responses = true;
+    #             }
+    #           ];
+    #         };
+    #       }
+    #       {
+    #         # forward the root zone to the local DNS resolver
+    #         zone = ".";
+    #         zone_type = "Forward";
+    #         stores = {
+    #           type = "forward";
+    #           name_servers = [
+    #             {
+    #               socket_addr = "127.0.0.53:53";
+    #               protocol = "udp";
+    #               trust_nx_responses = true;
+    #             }
+    #           ];
+    #         };
+    #       }
+    #     ];
+    #   };
+    # };
+  };
+
+  sane.services.dyn-dns.restartOnChange = [
+    "trust-dns-wan.service"
+    "trust-dns-lan.service"
+    "trust-dns-hn.service"
+    # "trust-dns-hn-resolver.service"  # doesn't need restart because it doesn't know about WAN IP
+  ];
 }

hosts/common/cross.nix

@@ -1,22 +0,0 @@
-{ config, ... }:
-
-let
-  mkCrossFrom = localSystem: pkgs: import pkgs.path {
-    inherit localSystem;
-    crossSystem = pkgs.stdenv.hostPlatform.system;
-    inherit (config.nixpkgs) config overlays;
-  };
-in
-{
-  # the configuration of which specific package set `pkgs.cross` refers to happens elsewhere;
-  # here we just define them all.
-  nixpkgs.overlays = [
-    (next: prev: {
-      # non-emulated packages build *from* local *for* target.
-      # for large packages like the linux kernel which are expensive to build under emulation,
-      # the config can explicitly pull such packages from `pkgs.cross` to do more efficient cross-compilation.
-      crossFrom."x86_64-linux" = mkCrossFrom "x86_64-linux" next;
-      crossFrom."aarch64-linux" = mkCrossFrom "aarch64-linux" next;
-    })
-  ];
-}

hosts/common/default.nix

@@ -1,86 +1,49 @@
-{ pkgs, ... }:
+{ config, lib, pkgs, ... }:
 {
   imports = [
-    ./cross.nix
     ./feeds.nix
     ./fs.nix
-    ./hardware.nix
+    ./hardware
     ./home
-    ./i2p.nix
+    ./hosts.nix
     ./ids.nix
     ./machine-id.nix
-    ./net.nix
+    ./net
+    ./nix
     ./persist.nix
+    ./polyunfill.nix
+    ./programs
     ./secrets.nix
     ./ssh.nix
-    ./users.nix
-    ./vpn.nix
+    ./systemd.nix
+    ./users
   ];
 
   sane.nixcache.enable-trusted-keys = true;
-  sane.packages.enableConsolePkgs = true;
-  sane.packages.enableSystemPkgs = true;
+  sane.nixcache.enable = lib.mkDefault true;
+  sane.persist.enable = lib.mkDefault true;
+  sane.root-on-tmpfs = lib.mkDefault true;
+  sane.programs.sysadminUtils.enableFor.system = lib.mkDefault true;
+  sane.programs.consoleUtils.enableFor.user.colin = lib.mkDefault true;
 
-  # some services which use private directories error if the parent (/var/lib/private) isn't 700.
-  sane.fs."/var/lib/private".dir.acl.mode = "0700";
-
-  nixpkgs.config.allowUnfree = true;
+  nixpkgs.config.allowUnfree = true;  # NIXPKGS_ALLOW_UNFREE=1
+  nixpkgs.config.allowBroken = true;  # NIXPKGS_ALLOW_BROKEN=1
 
   # time.timeZone = "America/Los_Angeles";
   time.timeZone = "Etc/UTC";  # DST is too confusing for me => use a stable timezone
 
-  # allow `nix flake ...` command
-  nix.extraOptions = ''
-    experimental-features = nix-command flakes
-  '';
-  # allow `nix-shell` (and probably nix-index?) to locate our patched and custom packages
-  nix.nixPath = [
-    "nixpkgs=${pkgs.path}"
-    "nixpkgs-overlays=${../..}/overlays"
-  ];
-
-  fonts = {
-    enableDefaultFonts = true;
-    fonts = with pkgs; [ font-awesome twitter-color-emoji hack-font ];
-    fontconfig.enable = true;
-    fontconfig.defaultFonts = {
-      emoji = [ "Font Awesome 6 Free" "Twitter Color Emoji" ];
-      monospace = [ "Hack" ];
-      serif = [ "DejaVu Serif" ];
-      sansSerif = [ "DejaVu Sans" ];
-    };
+  system.activationScripts.nixClosureDiff = {
+    supportsDryActivation = true;
+    text = ''
+      # show which packages changed versions or are new/removed in this upgrade
+      # source: <https://github.com/luishfonseca/dotfiles/blob/32c10e775d9ec7cc55e44592a060c1c9aadf113e/modules/upgrade-diff.nix>
+      # modified to not error on boot (when /run/current-system doesn't exist)
+      if [ -d /run/current-system ]; then
+        ${pkgs.nvd}/bin/nvd --nix-bin-dir=${pkgs.nix}/bin diff /run/current-system "$systemConfig"
+      fi
+    '';
   };
 
-  # disable non-required packages like nano, perl, rsync, strace
-  environment.defaultPackages = [];
-
-  # programs.vim.defaultEditor = true;
-  environment.variables = {
-    EDITOR = "vim";
-    # git claims it should use EDITOR, but it doesn't!
-    GIT_EDITOR = "vim";
-    # TODO: these should be moved to `home.sessionVariables` (home-manager)
-    # Electron apps should use native wayland backend:
-    #   https://nixos.wiki/wiki/Slack#Wayland
-    # Discord under sway crashes with this.
-    # NIXOS_OZONE_WL = "1";
-    # LIBGL_ALWAYS_SOFTWARE = "1";
-  };
-
-  # dconf docs: <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/desktop_migration_and_administration_guide/profiles>
-  # find keys/values with `dconf dump /`
-  programs.dconf.enable = true;
-  programs.dconf.packages = [
-    (pkgs.writeTextFile {
-      name = "dconf-user-profile";
-      destination = "/etc/dconf/profile/user";
-      text = ''
-        user-db:user
-        system-db:site
-      '';
-    })
-  ];
-
   # link debug symbols into /run/current-system/sw/lib/debug
   # hopefully picked up by gdb automatically?
   environment.enableDebugInfo = true;

hosts/common/feeds.nix

@@ -1,3 +1,14 @@
+# where to find good stuff?
+# - universal search/directory: <https://podcastindex.org>
+# - podcasts w/ a community: <https://lemmyverse.net/communities?query=podcast>
+# - podcast rec thread: <https://lemmy.ml/post/1565858>
+#
+# candidates:
+# - The Nonlinear Library (podcast): <https://forum.effectivealtruism.org/posts/JTZTBienqWEAjGDRv/listen-to-more-ea-content-with-the-nonlinear-library>
+#   - has ~10 posts per day, text-to-speech; i would need better tagging before adding this
+# - <https://www.metaculus.com/questions/11102/introducing-the-metaculus-journal-podcast/>
+#   - dead since 2022/10 - 2023/03
+
 { lib, sane-data, ... }:
 let
   hourly = { freq = "hourly"; };
@@ -40,6 +51,8 @@ let
         else
           "infrequent"
       ));
+    } // lib.optionalAttrs (lib.hasPrefix "https://www.youtube.com/" raw.url) {
+      format = "video";
     } // lib.optionalAttrs (raw.is_podcast or false) {
       format = "podcast";
     } // lib.optionalAttrs (raw.title or "" != "") {
@@ -47,152 +60,203 @@ let
     };
 
   podcasts = [
-    (fromDb "lexfridman.com/podcast" // rat)
-    ## Astral Codex Ten
-    (fromDb "sscpodcast.libsyn.com" // rat)
-    ## Econ Talk
-    (fromDb "feeds.simplecast.com/wgl4xEgL" // rat)
-    ## Cory Doctorow -- both podcast & text entries
-    (fromDb "craphound.com" // pol)
-    (fromDb "congressionaldish.libsyn.com" // pol)
-    ## Civboot -- https://anchor.fm/civboot
-    (fromDb "anchor.fm/s/34c7232c/podcast/rss" // tech)
-    ## Emerge: making sense of what's next -- <https://www.whatisemerging.com/emergepodcast>
-    (mkPod "https://anchor.fm/s/21bc734/podcast/rss" // pol // infrequent)
-    (fromDb "feeds.feedburner.com/80000HoursPodcast" // rat)
+    (fromDb "acquiredlpbonussecretsecret.libsyn.com" // tech)  # ACQ2 - more "Acquired" episodes
     (fromDb "allinchamathjason.libsyn.com" // pol)
-    (fromDb "acquired.libsyn.com" // tech)
-    # The Intercept - Deconstructed; also available: <rss.acast.com/deconstructed>
-    (fromDb "rss.prod.firstlook.media/deconstructed/podcast.rss" // pol)
-    ## The Daily
-    (mkPod "https://feeds.simplecast.com/54nAGcIl" // pol // daily)
-    # The Intercept - Intercepted; also available: <https://rss.acast.com/intercepted-with-jeremy-scahill>
-    (fromDb "rss.prod.firstlook.media/intercepted/podcast.rss" // pol)
-    (fromDb "podcast.posttv.com/itunes/post-reports.xml" // pol)
-    ## Eric Weinstein
-    (fromDb "rss.art19.com/the-portal" // rat)
+    (fromDb "anchor.fm/s/34c7232c/podcast/rss" // tech)  # Civboot -- https://anchor.fm/civboot
+    (fromDb "anchor.fm/s/2da69154/podcast/rss" // tech)  # POD OF JAKE -- https://podofjake.com/
+    (fromDb "cast.postmarketos.org" // tech)
+    (fromDb "congressionaldish.libsyn.com" // pol)  # Jennifer Briney
+    (fromDb "craphound.com" // pol)  # Cory Doctorow -- both podcast & text entries
     (fromDb "darknetdiaries.com" // tech)
-    ## Radiolab -- also available here, but ONLY OVER HTTP: <http://feeds.wnyc.org/radiolab>
-    (fromDb "feeds.feedburner.com/radiolab" // pol)
-    ## Sam Harris
-    (fromDb "wakingup.libsyn.com" // pol)
-    ## 99% Invisible -- also available here: <https://feeds.simplecast.com/BqbsxVfO>
-    (fromDb "feeds.99percentinvisible.org/99percentinvisible" // pol)
-    (fromDb "rss.acast.com/ft-tech-tonic" // tech)
+    (fromDb "feed.podbean.com/matrixlive/feed.xml" // tech)  # Matrix (chat) Live
+    (fromDb "feeds.99percentinvisible.org/99percentinvisible" // pol)  # 99% Invisible -- also available here: <https://feeds.simplecast.com/BqbsxVfO>
+    (fromDb "feeds.feedburner.com/80000HoursPodcast" // rat)
     (fromDb "feeds.feedburner.com/dancarlin/history" // rat)
+    (fromDb "feeds.feedburner.com/radiolab" // pol)  # Radiolab -- also available here, but ONLY OVER HTTP: <http://feeds.wnyc.org/radiolab>
+    (fromDb "feeds.megaphone.fm/behindthebastards" // pol)  # also Maggie Killjoy
+    (fromDb "feeds.megaphone.fm/recodedecode" // tech)  # The Verge - Decoder
+    (fromDb "feeds.simplecast.com/54nAGcIl" // pol)  # The Daily
+    (fromDb "feeds.simplecast.com/82FI35Px" // pol)  # Ezra Klein Show
+    (fromDb "feeds.simplecast.com/wgl4xEgL" // rat)  # Econ Talk
+    (fromDb "feeds.simplecast.com/xKJ93w_w" // uncat)  # Atlas Obscura
+    (fromDb "feeds.transistor.fm/acquired" // tech)
+    (fromDb "fulltimenix.com" // tech)
+    (fromDb "futureofcoding.org/episodes" // tech)
+    (fromDb "hackerpublicradio.org" // tech)
+    (fromDb "lexfridman.com/podcast" // rat)
+    (fromDb "mapspodcast.libsyn.com" // uncat)  # Multidisciplinary Association for Psychedelic Studies
+    (fromDb "microarch.club" // tech)
+    (fromDb "mintcast.org" // tech)
+    (fromDb "omegataupodcast.net" // tech)  # 3/4 German; 1/4 eps are English
+    (fromDb "omny.fm/shows/cool-people-who-did-cool-stuff" // pol)  # Maggie Killjoy -- referenced by Cory Doctorow
+    (fromDb "omny.fm/shows/money-stuff-the-podcast")  # Matt Levine
+    (fromDb "omny.fm/shows/the-dollop-with-dave-anthony-and-gareth-reynolds")  # The Dollop history/comedy
+    (fromDb "originstories.libsyn.com" // uncat)
+    (fromDb "podcast.posttv.com/itunes/post-reports.xml" // pol)
+    (fromDb "politicalorphanage.libsyn.com" // pol)
+    (fromDb "reverseengineering.libsyn.com/rss" // tech)  # UnNamed Reverse Engineering Podcast
+    (fromDb "rss.acast.com/deconstructed")  # The Intercept - Deconstructed
+    (fromDb "rss.acast.com/ft-tech-tonic" // tech)
+    (fromDb "rss.acast.com/intercepted-with-jeremy-scahill")  # The Intercept - Intercepted
     (fromDb "rss.art19.com/60-minutes" // pol)
-    ## The Verge - Decoder
-    (fromDb "feeds.megaphone.fm/recodedecode" // tech)
-    ## Matrix (chat) Live
-    (fromDb "feed.podbean.com/matrixlive/feed.xml" // tech)
-    ## Michael Malice - Your Welcome -- also available here: <https://origin.podcastone.com/podcast?categoryID2=2232>
-    (fromDb "rss.art19.com/your-welcome" // pol)
+    (fromDb "rss.art19.com/the-portal" // rat)  # Eric Weinstein
     (fromDb "seattlenice.buzzsprout.com" // pol)
-    ## Sci-Fi? has Peter Watts; author of No Moods, Ads or Cutesy Fucking Icons (rifters.com)
-    (fromDb "talesfromthebridge.buzzsprout.com" // tech)
+    (fromDb "srslywrong.com" // pol)
+    (fromDb "sharkbytes.transistor.fm" // tech)  # Wireshark Podcast o_0
+    (fromDb "sscpodcast.libsyn.com" // rat)  # Astral Codex Ten
+    (fromDb "talesfromthebridge.buzzsprout.com" // tech)  # Sci-Fi? has Peter Watts; author of No Moods, Ads or Cutesy Fucking Icons (rifters.com)
+    (fromDb "theamphour.com" // tech)
+    (fromDb "techtalesshow.com" // tech)  # Corbin Davenport
+    (fromDb "techwontsave.us" // pol)  # rec by Cory Doctorow
+    (fromDb "wakingup.libsyn.com" // pol)  # Sam Harris
+    (fromDb "werenotwrong.fireside.fm" // pol)
+    (mkPod "https://sfconservancy.org/casts/the-corresponding-source/feeds/ogg/" // tech)
+
+    # (fromDb "feeds.libsyn.com/421877" // rat)  # Less Wrong Curated
+    # (fromDb "feeds.megaphone.fm/hubermanlab" // uncat)  # Daniel Huberman on sleep
+    # (fromDb "feeds.simplecast.com/l2i9YnTd" // tech // pol)  # Hard Fork (NYtimes tech)
+    # (fromDb "podcast.thelinuxexp.com" // tech)  # low-brow linux/foss PR announcements
+    # (fromDb "rss.art19.com/your-welcome" // pol)  # Michael Malice - Your Welcome -- also available here: <https://origin.podcastone.com/podcast?categoryID2=2232>
+    # (fromDb "rss.prod.firstlook.media/deconstructed/podcast.rss" // pol)  #< possible URL rot
+    # (fromDb "rss.prod.firstlook.media/intercepted/podcast.rss" // pol)  #< possible URL rot
+    # (fromDb "trashfuturepodcast.podbean.com" // pol)  # rec by Cory Doctorow, but way rambly
+    # (mkPod "https://anchor.fm/s/21bc734/podcast/rss" // pol // infrequent)  # Emerge: making sense of what's next -- <https://www.whatisemerging.com/emergepodcast>
+    # (mkPod "https://audioboom.com/channels/5097784.rss" // tech)  # Lateral with Tom Scott
+    # (mkPod "https://feeds.megaphone.fm/RUNMED9919162779" // pol // infrequent)  # The Witch Trials of J.K. Rowling: <https://www.thefp.com/witchtrials>
+    # (mkPod "https://podcasts.la.utexas.edu/this-is-democracy/feed/podcast/" // pol // weekly)
   ];
 
   texts = [
-    # AGGREGATORS (> 1 post/day)
-    (fromDb "lwn.net" // tech)
-    (fromDb "lesswrong.com" // rat)
-    (fromDb "econlib.org" // pol)
-
-    # AGGREGATORS (< 1 post/day)
-    (fromDb "palladiummag.com" // uncat)
-    (fromDb "profectusmag.com" // uncat)
-    (fromDb "semiaccurate.com" // tech)
-    (mkText "https://linuxphoneapps.org/blog/atom.xml" // tech // infrequent)
-    (fromDb "spectrum.ieee.org" // tech)
-    ## n.b.: quality RSS list here: <https://forum.merveilles.town/thread/57/share-your-rss-feeds%21-6/>
-    (mkText "https://forum.merveilles.town/rss.xml" // pol // infrequent)
-
-    ## No Moods, Ads or Cutesy Fucking Icons
-    (fromDb "rifters.com/crawl" // uncat)
-
-    # DEVELOPERS
-    (fromDb "uninsane.org" // tech)
-    (fromDb "mg.lol" // tech)
-    (fromDb "drewdevault.com" // tech)
-    ## Ken Shirriff
-    (fromDb "righto.com" // tech)
-    ## shared blog by a few NixOS devs, notably onny
-    (fromDb "project-insanity.org" // tech)
-    ## Vitalik Buterin
-    (fromDb "vitalik.ca" // tech)
-    ## ian (Sanctuary)
-    (fromDb "sagacioussuricata.com" // tech)
-    ## Bunnie Juang
-    (fromDb "bunniestudios.com" // tech)
-    (fromDb "blog.danieljanus.pl" // tech)
-    (fromDb "ianthehenry.com" // tech)
-    (fromDb "bitbashing.io" // tech)
-    (fromDb "idiomdrottning.org" // uncat)
-    (mkText "https://anish.lakhwara.com/home.html" // tech // weekly)
-    (fromDb "jefftk.com" // tech)
-    (fromDb "pomeroyb.com" // tech)
-
-    # (TECH; POL) COMMENTATORS
-    ## Matt Webb -- engineering-ish, but dreamy
-    (fromDb "interconnected.org/home/feed" // rat)
-    (fromDb "edwardsnowden.substack.com" // pol // text)
-    ## Julia Evans
-    (mkText "https://jvns.ca/atom.xml" // tech // weekly)
-    (mkText "http://benjaminrosshoffman.com/feed" // pol // weekly)
-    ## Ben Thompson
-    (mkText "https://www.stratechery.com/rss" // pol // weekly)
-    ## Balaji
-    (fromDb "balajis.com" // pol)
-    (fromDb "ben-evans.com/benedictevans" // pol)
-    (fromDb "lynalden.com" // pol)
-    (fromDb "austinvernon.site" // tech)
-    (mkSubstack "oversharing" // pol // daily)
-    (mkSubstack "doomberg" // tech // weekly)
-    ## David Rosenthal
-    (fromDb "blog.dshr.org" // pol)
-    ## Matt Levine
-    (mkText "https://www.bloomberg.com/opinion/authors/ARbTQlRLRjE/matthew-s-levine.rss" // pol // weekly)
-    (fromDb "stpeter.im/atom.xml" // pol)
-    ## Peter Saint-Andre -- side project of stpeter.im
-    (fromDb "philosopher.coach" // rat)
-
-    # RATIONALITY/PHILOSOPHY/ETC
-    (mkSubstack "samkriss" // humor // infrequent)
-    (fromDb "unintendedconsequenc.es" // rat)
+    (fromDb "acoup.blog/feed")  # history, states. author: <https://historians.social/@bretdevereaux/following>
+    (fromDb "amosbbatto.wordpress.com" // tech)
+    (fromDb "anish.lakhwara.com" // tech)
+    (fromDb "apenwarr.ca/log/rss.php" // tech)  # CEO of tailscale
     (fromDb "applieddivinitystudies.com" // rat)
-    (fromDb "slimemoldtimemold.com" // rat)
-    (fromDb "richardcarrier.info" // rat)
+    (fromDb "artemis.sh" // tech)
+    (fromDb "ascii.textfiles.com" // tech)  # Jason Scott
+    (fromDb "austinvernon.site" // tech)
+    (fromDb "ben-evans.com/benedictevans" // pol)
+    (fromDb "bitbashing.io" // tech)
+    (fromDb "bitsaboutmoney.com" // uncat)
+    (fromDb "blog.danieljanus.pl" // tech)
+    (fromDb "blog.dshr.org" // pol)  # David Rosenthal
+    (fromDb "blog.jmp.chat" // tech)
+    (fromDb "blog.rust-lang.org" // tech)
+    (fromDb "blog.thalheim.io" // tech)  # Mic92
+    (fromDb "bunniestudios.com" // tech)  # Bunnie Juang
+    (fromDb "capitolhillseattle.com" // pol)
+    (fromDb "edwardsnowden.substack.com" // pol // text)
+    (fromDb "fasterthanli.me" // tech)
     (fromDb "gwern.net" // rat)
-    ## Jason Crawford
-    (fromDb "rootsofprogress.org" // rat)
-    ## Robin Hanson
-    (fromDb "overcomingbias.com" // rat)
-    ## Scott Alexander
-    (mkSubstack "astralcodexten" // rat // daily)
-    ## Paul Christiano
-    (fromDb "sideways-view.com" // rat)
-    ## Sean Carroll
-    (fromDb "preposterousuniverse.com" // rat)
+    (fromDb "hardcoresoftware.learningbyshipping.com" // tech)  # Steven Sinofsky
+    (fromDb "harihareswara.net" // tech // pol)  # rec by Cory Doctorow
+    (fromDb "ianthehenry.com" // tech)
+    (fromDb "idiomdrottning.org" // uncat)
+    (fromDb "interconnected.org/home/feed" // rat)  # Matt Webb -- engineering-ish, but dreamy
+    (fromDb "jeffgeerling.com" // tech)
+    (fromDb "jefftk.com" // tech)
+    (fromDb "jwz.org/blog" // tech // pol)  # DNA lounge guy, loooong-time blogger
+    (fromDb "kill-the-newsletter.com/feeds/joh91bv7am2pnznv.xml" // pol)  # Matt Levine - Money Stuff
+    (fromDb "kosmosghost.github.io/index.xml" // tech)
+    (fromDb "linmob.net" // tech)
+    (fromDb "lwn.net" // tech)
+    (fromDb "lynalden.com" // pol)
+    (fromDb "mako.cc/copyrighteous" // tech // pol)  # rec by Cory Doctorow
+    (fromDb "mg.lol" // tech)
+    (fromDb "mindingourway.com" // rat)
+    (fromDb "morningbrew.com/feed" // pol)
+    (fromDb "nixpkgs.news" // tech)
+    (fromDb "overcomingbias.com" // rat)  # Robin Hanson
+    (fromDb "palladiummag.com" // uncat)
+    (fromDb "philosopher.coach" // rat)  # Peter Saint-Andre -- side project of stpeter.im
+    (fromDb "pomeroyb.com" // tech)
+    (fromDb "postmarketos.org/blog" // tech)
+    (fromDb "preposterousuniverse.com" // rat)  # Sean Carroll
+    (fromDb "project-insanity.org" // tech)  # shared blog by a few NixOS devs, notably onny
+    (fromDb "putanumonit.com" // rat)  # mostly dating topics. not advice, or humor, but looking through a social lens
+    (fromDb "richardcarrier.info" // rat)
+    (fromDb "rifters.com/crawl" // uncat)  # No Moods, Ads or Cutesy Fucking Icons
+    (fromDb "righto.com" // tech)  # Ken Shirriff
+    (fromDb "rootsofprogress.org" // rat)  # Jason Crawford
+    (fromDb "samuel.dionne-riel.com" // tech)  # SamuelDR
+    (fromDb "sagacioussuricata.com" // tech)  # ian (Sanctuary)
+    (fromDb "semiaccurate.com" // tech)
+    (fromDb "sideways-view.com" // rat)  # Paul Christiano
+    (fromDb "slatecave.net" // tech)
+    (fromDb "slimemoldtimemold.com" // rat)
+    (fromDb "spectrum.ieee.org" // tech)
+    (fromDb "stpeter.im/atom.xml" // pol)
+    (fromDb "thediff.co" // pol)  # Byrne Hobart
+    (fromDb "thisweek.gnome.org" // tech)
+    (fromDb "tuxphones.com" // tech)
+    (fromDb "uninsane.org" // tech)
+    (fromDb "unintendedconsequenc.es" // rat)
+    (fromDb "vitalik.eth.limo" // tech)  # Vitalik Buterin
+    (fromDb "weekinethereumnews.com" // tech)
+    (fromDb "willow.phantoma.online")  # wizard@xyzzy.link
+    (fromDb "xn--gckvb8fzb.com" // tech)
+    (mkSubstack "astralcodexten" // rat // daily)  # Scott Alexander
+    (mkSubstack "eliqian" // rat // weekly)
+    (mkSubstack "oversharing" // pol // daily)
+    (mkSubstack "samkriss" // humor // infrequent)
+    (mkText "http://benjaminrosshoffman.com/feed" // pol // weekly)
+    (mkText "http://boginjr.com/feed" // tech // infrequent)
+    (mkText "https://forum.merveilles.town/rss.xml" // pol // infrequent)  #quality RSS list here: <https://forum.merveilles.town/thread/57/share-your-rss-feeds%21-6/>
+    (mkText "https://jvns.ca/atom.xml" // tech // weekly)  # Julia Evans
+    (mkText "https://linuxphoneapps.org/blog/atom.xml" // tech // infrequent)
+    (mkText "https://nixos.org/blog/announcements-rss.xml" // tech // infrequent)  # more nixos stuff here, but unclear how to subscribe: <https://nixos.org/blog/categories.html>
+    (mkText "https://nixos.org/blog/stories-rss.xml" // tech // weekly)
+    (mkText "https://solar.lowtechmagazine.com/posts/index.xml" // tech // weekly)
+    (mkText "https://www.stratechery.com/rss" // pol // weekly)  # Ben Thompson
 
-    ## mostly dating topics. not advice, or humor, but looking through a social lens
-    (fromDb "putanumonit.com" // rat)
-
-    # CODE
+    # (fromDb "balajis.com" // pol)  # Balaji
+    # (fromDb "drewdevault.com" // tech)
+    # (fromDb "econlib.org" // pol)
+    # (fromDb "lesswrong.com" // rat)
+    # (fromDb "profectusmag.com" // pol)  # some conservative/libertarian think tank
+    # (fromDb "thesideview.co" // uncat)  # spiritual journal; RSS items are stubs
+    # (fromDb "theregister.com" // tech)
+    # (fromDb "vitalik.ca" // tech)  # moved to vitalik.eth.limo
+    # (fromDb "webcurious.co.uk" // uncat)  # link aggregator; defunct?
+    # (mkSubstack "doomberg" // tech // weekly)  # articles are all pay-walled
     # (mkText "https://github.com/Kaiteki-Fedi/Kaiteki/commits/master.atom" // tech // infrequent)
+    # (mkText "https://til.simonwillison.net/tils/feed.atom" // tech // weekly)
+    # (mkText "https://www.bloomberg.com/opinion/authors/ARbTQlRLRjE/matthew-s-levine.rss" // pol // weekly)  # Matt Levine (preview/paywalled)
+  ];
+
+  videos = [
+    (fromDb "youtube.com/@Channel5YouTube" // pol)
+    (fromDb "youtube.com/@ColdFusion")
+    (fromDb "youtube.com/@ContraPoints" // pol)
+    (fromDb "youtube.com/@Exurb1a")
+    (fromDb "youtube.com/@hbomberguy")
+    (fromDb "youtube.com/@JackStauber")
+    (fromDb "youtube.com/@NativLang")
+    (fromDb "youtube.com/@PolyMatter")
+    (fromDb "youtube.com/@TechnologyConnections" // tech)
+    (fromDb "youtube.com/@TheB1M")
+    (fromDb "youtube.com/@TomScottGo")
+    (fromDb "youtube.com/@Vihart")
+    (fromDb "youtube.com/@Vox")
+    (fromDb "youtube.com/@Vsauce")
+
+    # (fromDb "youtube.com/@rossmanngroup" // pol // tech)  # Louis Rossmann
   ];
 
   images = [
-    (fromDb "smbc-comics.com" // img // humor)
-    (fromDb "xkcd.com" // img // humor)
-    (fromDb "pbfcomics.com" // img // humor)
-    # (mkImg "http://dilbert.com/feed" // humor // daily)
-
-    # ART
+    (fromDb "catandgirl.com" // img // humor)
+    (fromDb "davidrevoy.com" // img // art)
+    (fromDb "grumpy.website" // img // humor)
     (fromDb "miniature-calendar.com" // img // art // daily)
+    (fromDb "pbfcomics.com" // img // humor)
+    (fromDb "poorlydrawnlines.com/feed" // img // humor)
+    (fromDb "smbc-comics.com" // img // humor)
+    (fromDb "turnoff.us" // img // humor)
+    (fromDb "xkcd.com" // img // humor)
   ];
 in
 {
-  sane.feeds = texts ++ images ++ podcasts;
+  sane.feeds = texts ++ images ++ podcasts ++ videos;
 
   assertions = builtins.map
     (p: {

hosts/common/fs.nix

@@ -1,74 +1,236 @@
-{ pkgs, ... }:
+# docs
+# - x-systemd options: <https://www.freedesktop.org/software/systemd/man/systemd.mount.html>
+# - fuse options: `man mount.fuse`
 
-let sshOpts = rec {
-  fsType = "fuse.sshfs";
-  optionsBase = [
-    "x-systemd.automount"
-    "_netdev"
-    "user"
-    "identityfile=/home/colin/.ssh/id_ed25519"
-    "allow_other"
-    "default_permissions"
-  ];
-  optionsColin = optionsBase ++ [
-    "transform_symlinks"
-    "idmap=user"
-    "uid=1000"
-    "gid=100"
-  ];
+{ config, lib, pkgs, sane-lib, utils, ... }:
 
-  optionsRoot = optionsBase ++ [
-    # we don't transform_symlinks because that breaks the validity of remote /nix stores
-    "sftp_server=/run/wrappers/bin/sudo\\040/run/current-system/sw/libexec/sftp-server"
-  ];
-};
+let
+  fsOpts = rec {
+    common = [
+      "_netdev"
+      "noatime"
+      # user: allow any user with access to the device to mount the fs.
+      #       note that this requires a suid `mount` binary; see: <https://zameermanji.com/blog/2022/8/5/using-fuse-without-root-on-linux/>
+      "user"
+      "x-systemd.requires=network-online.target"
+      "x-systemd.after=network-online.target"
+      "x-systemd.mount-timeout=10s"  # how long to wait for mount **and** how long to wait for unmount
+    ];
+    # x-systemd.automount: mount the fs automatically *on first access*.
+    # creates a `path-to-mount.automount` systemd unit.
+    automount = [ "x-systemd.automount" ];
+    # noauto: don't mount as part of remote-fs.target.
+    # N.B.: `remote-fs.target` is a dependency of multi-user.target, itself of graphical.target.
+    # hence, omitting `noauto` can slow down boots.
+    noauto = [ "noauto" ];
+    # lazyMount: defer mounting until first access from userspace.
+    # see: `man systemd.automount`, `man automount`, `man autofs`
+    lazyMount = noauto ++ automount;
+    wg = [
+      "x-systemd.requires=wireguard-wg-home.service"
+      "x-systemd.after=wireguard-wg-home.service"
+    ];
+
+    fuse = [
+      "allow_other"  # allow users other than the one who mounts it to access it. needed, if systemd is the one mounting this fs (as root)
+      # allow_root: allow root to access files on this fs (if mounted by non-root, else it can always access them).
+      #             N.B.: if both allow_root and allow_other are specified, then only allow_root takes effect.
+      # "allow_root"
+      # default_permissions: enforce local permissions check. CRUCIAL if using `allow_other`.
+      # w/o this, permissions mode of sshfs is like:
+      # - sshfs runs all remote commands as the remote user.
+      # - if a local user has local permissions to the sshfs mount, then their file ops are sent blindly across the tunnel.
+      # - `allow_other` allows *any* local user to access the mount, and hence any local user can now freely become the remote mapped user.
+      # with default_permissions, sshfs doesn't tunnel file ops from users until checking that said user could perform said op on an equivalent local fs.
+      "default_permissions"
+    ];
+    fuseColin = fuse ++ [
+      "uid=1000"
+      "gid=100"
+    ];
+
+    ssh = common ++ fuse ++ [
+      "identityfile=/home/colin/.ssh/id_ed25519"
+      # i *think* idmap=user means that `colin` on `localhost` and `colin` on the remote are actually treated as the same user, even if their uid/gid differs?
+      # i.e., local colin's id is translated to/from remote colin's id on every operation?
+      "idmap=user"
+    ];
+    sshColin = ssh ++ fuseColin ++ [
+      # follow_symlinks: remote files which are symlinks are presented to the local system as ordinary files (as the target of the symlink).
+      #                  if the symlink target does not exist, the presentation is unspecified.
+      #                  symlinks which point outside the mount ARE followed. so this is more capable than `transform_symlinks`
+      "follow_symlinks"
+      # symlinks on the remote fs which are absolute paths are presented to the local system as relative symlinks pointing to the expected data on the remote fs.
+      # only symlinks which would point inside the mountpoint are translated.
+      "transform_symlinks"
+    ];
+    # sshRoot = ssh ++ [
+    #   # we don't transform_symlinks because that breaks the validity of remote /nix stores
+    #   "sftp_server=/run/wrappers/bin/sudo\\040/run/current-system/sw/libexec/sftp-server"
+    # ];
+    # in the event of hunt NFS mounts, consider:
+    # - <https://unix.stackexchange.com/questions/31979/stop-broken-nfs-mounts-from-locking-a-directory>
+
+    # NFS options: <https://linux.die.net/man/5/nfs>
+    # actimeo=n  = how long (in seconds) to cache file/dir attributes (default: 3-60s)
+    # bg         = retry failed mounts in the background
+    # retry=n    = for how many minutes `mount` will retry NFS mount operation
+    # intr       = allow Ctrl+C to abort I/O (it will error with `EINTR`)
+    # soft       = on "major timeout", report I/O error to userspace
+    # softreval  = on "major timeout", service the request using known-stale cache results instead of erroring -- if such cache data exists
+    # retrans=n  = how many times to retry a NFS request before giving userspace a "server not responding" error (default: 3)
+    # timeo=n    = number of *deciseconds* to wait for a response before retrying it (default: 600)
+    #              note: client uses a linear backup, so the second request will have double this timeout, then triple, etc.
+    # proto=udp  = encapsulate protocol ops inside UDP packets instead of a TCP session.
+    #              requires `nfsvers=3` and a kernel compiled with `NFS_DISABLE_UDP_SUPPORT=n`.
+    #              UDP might be preferable to TCP because the latter is liable to hang for ~100s (kernel TCP timeout) after a link drop.
+    #              however, even UDP has issues with `umount` hanging.
+    #
+    # N.B.: don't change these without first testing the behavior of sandboxed apps on a flaky network.
+    nfs = common ++ [
+      # "actimeo=5"
+      # "bg"
+      "retrans=1"
+      "retry=0"
+      # "intr"
+      "soft"
+      "softreval"
+      "timeo=30"
+      "nofail"  # don't fail remote-fs.target when this mount fails  (not an option for sshfs else would be common)
+      # "proto=udp"  # default kernel config doesn't support NFS over UDP: <https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1964093> (see comment 11).
+      # "nfsvers=3"  # NFSv4+ doesn't support UDP at *all*. it's ok to omit nfsvers -- server + client will negotiate v3 based on udp requirement. but omitting causes confusing mount errors when the server is *offline*, because the client defaults to v4 and thinks the udp option is a config error.
+      # "x-systemd.idle-timeout=10"  # auto-unmount after this much inactivity
+    ];
+
+    # manually perform a ftp mount via e.g.
+    #   curlftpfs -o ftpfs_debug=2,user=anonymous:anonymous,connect_timeout=10 -f -s ftp://servo-hn /mnt/my-ftp
+    ftp = common ++ fuseColin ++ [
+      # "ftpfs_debug=2"
+      "user=colin:ipauth"
+      # connect_timeout=10: casting shows to T.V. fails partway through about half the time
+      "connect_timeout=20"
+    ];
+  };
+  remoteHome = host: {
+    sane.programs.sshfs-fuse.enableFor.system = true;
+    fileSystems."/mnt/${host}/home" = {
+      device = "colin@${host}:/home/colin";
+      fsType = "fuse.sshfs";
+      options = fsOpts.sshColin ++ fsOpts.lazyMount;
+      noCheck = true;
+    };
+    sane.fs."/mnt/${host}/home" = sane-lib.fs.wanted {
+      dir.acl.user = "colin";
+      dir.acl.group = "users";
+      dir.acl.mode = "0700";
+    };
+  };
+  remoteServo = subdir: {
+    sane.programs.curlftpfs.enableFor.system = true;
+    sane.fs."/mnt/servo/${subdir}" = sane-lib.fs.wanted {
+      dir.acl.user = "colin";
+      dir.acl.group = "users";
+      dir.acl.mode = "0750";
+    };
+    fileSystems."/mnt/servo/${subdir}" = {
+      device = "ftp://servo-hn:/${subdir}";
+      noCheck = true;
+      fsType = "fuse.curlftpfs";
+      options = fsOpts.ftp ++ fsOpts.noauto ++ fsOpts.wg;
+      # fsType = "nfs";
+      # options = fsOpts.nfs ++ fsOpts.lazyMount ++ fsOpts.wg;
+    };
+    systemd.services."automount-servo-${utils.escapeSystemdPath subdir}" = let
+      fs = config.fileSystems."/mnt/servo/${subdir}";
+    in {
+      # this is a *flaky* network mount, especially on moby.
+      # if done as a normal autofs mount, access will eternally block when network is dropped.
+      # notably, this would block *any* sandboxed app which allows media access, whether they actually try to use that media or not.
+      # a practical solution is this: mount as a service -- instead of autofs -- and unmount on timeout error, in a restart loop.
+      # until the ftp handshake succeeds, nothing is actually mounted to the vfs, so this doesn't slow down any I/O when network is down.
+      description = "automount /mnt/servo/${subdir} in a fault-tolerant and non-blocking manner";
+      after = [ "network-online.target" ];
+      requires = [ "network-online.target" ];
+      wantedBy = [ "default.target" ];
+
+      serviceConfig.Type = "simple";
+      serviceConfig.ExecStart = lib.escapeShellArgs [
+        "/usr/bin/env"
+        "PATH=/run/current-system/sw/bin"
+        "mount.${fs.fsType}"
+        "-f"  # foreground (i.e. don't daemonize)
+        "-s"  # single-threaded (TODO: it's probably ok to disable this?)
+        "-o"
+        (lib.concatStringsSep "," (lib.filter (o: !lib.hasPrefix "x-systemd." o) fs.options))
+        fs.device
+        "/mnt/servo/${subdir}"
+      ];
+      # not sure if this configures a linear, or exponential backoff.
+      # but the first restart will be after `RestartSec`, and the n'th restart (n = RestartSteps) will be RestartMaxDelaySec after the n-1'th exit.
+      serviceConfig.Restart = "always";
+      serviceConfig.RestartSec = "10s";
+      serviceConfig.RestartMaxDelaySec = "120s";
+      serviceConfig.RestartSteps = "5";
+    };
+  };
 in
-{
-  environment.pathsToLink = [
-    # needed to achieve superuser access for user-mounted filesystems (see optionsRoot above)
-    # we can only link whole directories here, even though we're only interested in pkgs.openssh
-    "/libexec"
-  ];
+lib.mkMerge [
+  {
+    # some services which use private directories error if the parent (/var/lib/private) isn't 700.
+    sane.fs."/var/lib/private".dir.acl.mode = "0700";
 
-  fileSystems."/mnt/servo-media-wan" = {
-    device = "colin@uninsane.org:/var/lib/uninsane/media";
-    inherit (sshOpts) fsType;
-    options = sshOpts.optionsColin;
-    noCheck = true;
-  };
-  fileSystems."/mnt/servo-media-lan" = {
-    device = "colin@servo:/var/lib/uninsane/media";
-    inherit (sshOpts) fsType;
-    options = sshOpts.optionsColin;
-    noCheck = true;
-  };
-  fileSystems."/mnt/servo-root-wan" = {
-    device = "colin@uninsane.org:/";
-    inherit (sshOpts) fsType;
-    options = sshOpts.optionsRoot;
-    noCheck = true;
-  };
-  fileSystems."/mnt/servo-root-lan" = {
-    device = "colin@servo:/";
-    inherit (sshOpts) fsType;
-    options = sshOpts.optionsRoot;
-    noCheck = true;
-  };
-  fileSystems."/mnt/desko-home" = {
-    device = "colin@desko:/home/colin";
-    inherit (sshOpts) fsType;
-    options = sshOpts.optionsColin;
-    noCheck = true;
-  };
-  fileSystems."/mnt/desko-root" = {
-    device = "colin@desko:/";
-    inherit (sshOpts) fsType;
-    options = sshOpts.optionsRoot;
-    noCheck = true;
-  };
+    # in-memory compressed RAM
+    # defaults to compressing at most 50% size of RAM
+    # claimed compression ratio is about 2:1
+    # - but on moby w/ zstd default i see 4-7:1  (ratio lowers as it fills)
+    # note that idle overhead is about 0.05% of capacity (e.g. 2B per 4kB page)
+    # docs: <https://www.kernel.org/doc/Documentation/blockdev/zram.txt>
+    #
+    # to query effectiveness:
+    # `cat /sys/block/zram0/mm_stat`. whitespace separated fields:
+    # - *orig_data_size*  (bytes)
+    # - *compr_data_size* (bytes)
+    # - mem_used_total    (bytes)
+    # - mem_limit         (bytes)
+    # - mem_used_max      (bytes)
+    # - *same_pages*  (pages which are e.g. all zeros (consumes no additional mem))
+    # - *pages_compacted*  (pages which have been freed thanks to compression)
+    # - huge_pages  (incompressible)
+    #
+    # see also:
+    # - `man zramctl`
+    zramSwap.enable = true;
+    # how much ram can be swapped into the zram device.
+    # this shouldn't be higher than the observed compression ratio.
+    # the default is 50% (why?)
+    # 100% should be "guaranteed" safe so long as the data is even *slightly* compressible.
+    # but it decreases working memory under the heaviest of loads by however much space the compressed memory occupies (e.g. 50% if 2:1; 25% if 4:1)
+    zramSwap.memoryPercent = 100;
 
-  environment.systemPackages = [
-    pkgs.sshfs-fuse
-  ];
-}
+    # environment.pathsToLink = [
+    #   # needed to achieve superuser access for user-mounted filesystems (see sshRoot above)
+    #   # we can only link whole directories here, even though we're only interested in pkgs.openssh
+    #   "/libexec"
+    # ];
+
+    programs.fuse.userAllowOther = true;  #< necessary for `allow_other` or `allow_root` options.
+  }
+
+  (remoteHome "desko")
+  (remoteHome "lappy")
+  (remoteHome "moby")
+  # this granularity of servo media mounts is necessary to support sandboxing:
+  # for flaky mounts, we can only bind the mountpoint itself into the sandbox,
+  # so it's either this or unconditionally bind all of media/.
+  (remoteServo "media/archive")
+  (remoteServo "media/Books")
+  (remoteServo "media/collections")
+  # (remoteServo "media/datasets")
+  (remoteServo "media/games")
+  (remoteServo "media/Music")
+  (remoteServo "media/Pictures/macros")
+  (remoteServo "media/torrents")
+  (remoteServo "media/Videos")
+  (remoteServo "playground")
+]
 

hosts/common/hardware.nix

@@ -1,40 +0,0 @@
-{ lib, pkgs, ... }:
-
-{
-  boot.initrd.supportedFilesystems = [ "ext4" "btrfs" "ext2" "ext3" "vfat" ];
-  # useful emergency utils
-  boot.initrd.extraUtilsCommands = ''
-    copy_bin_and_libs ${pkgs.btrfs-progs}/bin/btrfstune
-  '';
-  boot.kernelParams = [ "boot.shell_on_fail" ];
-  # other kernelParams:
-  #   "boot.trace"
-  #   "systemd.log_level=debug"
-  #   "systemd.log_target=console"
-
-  # hack in the `boot.shell_on_fail` arg since that doesn't always seem to work.
-  boot.initrd.preFailCommands = "allowShell=1";
-
-  # default: 4 (warn). 7 is debug
-  boot.consoleLogLevel = 7;
-
-  boot.loader.grub.enable = lib.mkDefault false;
-  boot.loader.generic-extlinux-compatible.enable = lib.mkDefault true;
-
-  # non-free firmware
-  hardware.enableRedistributableFirmware = true;
-  services.fwupd.enable = true;
-
-  # powertop will default to putting USB devices -- including HID -- to sleep after TWO SECONDS
-  powerManagement.powertop.enable = false;
-
-  # services.snapper.configs = {
-  #   root = {
-  #     subvolume = "/";
-  #     extraConfig = {
-  #       ALLOW_USERS = "colin";
-  #     };
-  #   };
-  # };
-  # services.snapper.snapshotInterval = "daily";
-}

hosts/common/hardware/default.nix

@@ -0,0 +1,99 @@
+{ config, lib, pkgs, ... }:
+
+{
+  imports = [
+    ./x86_64.nix
+  ];
+
+  boot.initrd.supportedFilesystems = [ "ext4" "btrfs" "ext2" "ext3" "vfat" ];
+  # useful emergency utils
+  boot.initrd.extraUtilsCommands = ''
+    copy_bin_and_libs ${pkgs.btrfs-progs}/bin/btrfstune
+    copy_bin_and_libs ${pkgs.util-linux}/bin/{cfdisk,lsblk,lscpu}
+    copy_bin_and_libs ${pkgs.gptfdisk}/bin/{cgdisk,gdisk}
+    copy_bin_and_libs ${pkgs.smartmontools}/bin/smartctl
+    copy_bin_and_libs ${pkgs.e2fsprogs}/bin/resize2fs
+  '' + lib.optionalString pkgs.stdenv.hostPlatform.isx86_64 ''
+    copy_bin_and_libs ${pkgs.nvme-cli}/bin/nvme  # doesn't cross compile
+  '';
+  boot.kernelParams = [
+    "boot.shell_on_fail"
+    #v experimental full pre-emption for hopefully better call/audio latency on moby.
+    # also toggleable at runtime via /sys/kernel/debug/sched/preempt
+    # defaults to preempt=voluntary
+    # "preempt=full"
+  ];
+  # other kernelParams:
+  #   "boot.trace"
+  #   "systemd.log_level=debug"
+  #   "systemd.log_target=console"
+
+  # moby has to run recent kernels (defined elsewhere).
+  # meanwhile, kernel variation plays some minor role in things like sandboxing (landlock) and capabilities.
+  # simpler to keep near the latest kernel on all devices,
+  # and also makes certain that any weird system-level bugs i see aren't likely to be stale kernel bugs.
+  # servo needs zfs though, which doesn't support every kernel.
+  boot.kernelPackages = lib.mkDefault pkgs.zfs.latestCompatibleLinuxPackages;
+
+  # TODO: remove after linux 6.9. see: <https://github.com/axboe/liburing/issues/1113>
+  # - <https://github.com/neovim/neovim/issues/28149>
+  # - <https://git.kernel.dk/cgit/linux/commit/?h=io_uring-6.9&id=e5444baa42e545bb929ba56c497e7f3c73634099>
+  # when removing, try starting and suspending (ctrl+z) two instances of neovim simultaneously.
+  # if the system doesn't freeze, then this is safe to remove.
+  # added 2024-04-04
+  sane.user.fs.".profile".symlink.text = lib.mkBefore ''
+    export UV_USE_IO_URING=0
+  '';
+
+  # hack in the `boot.shell_on_fail` arg since that doesn't always seem to work.
+  boot.initrd.preFailCommands = "allowShell=1";
+
+  # default: 4 (warn). 7 is debug
+  boot.consoleLogLevel = 7;
+
+  boot.loader.grub.enable = lib.mkDefault false;
+  boot.loader.generic-extlinux-compatible.enable = lib.mkDefault true;
+
+  # non-free firmware
+  hardware.enableRedistributableFirmware = true;
+
+  # default is 252274, which is too low particularly for servo.
+  # manifests as spurious "No space left on device" when trying to install watches,
+  # e.g. in dyn-dns by `systemctl start dyn-dns-watcher.path`.
+  # see: <https://askubuntu.com/questions/828779/failed-to-add-run-systemd-ask-password-to-directory-watch-no-space-left-on-dev>
+  boot.kernel.sysctl."fs.inotify.max_user_watches" = 1048576;
+
+  # powertop will default to putting USB devices -- including HID -- to sleep after TWO SECONDS
+  powerManagement.powertop.enable = false;
+  # linux CPU governor: <https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt>
+  # - options:
+  #   - "powersave" => force CPU to always run at lowest supported frequency
+  #   - "performance" => force CPU to always run at highest frequency
+  #   - "ondemand" => adjust frequency based on load
+  #   - "conservative"  (ondemand but slower to adjust)
+  #   - "schedutil"
+  #   - "userspace"
+  # - not all options are available for all platforms
+  #   - intel (intel_pstate) appears to manage scaling w/o intervention/control from the OS.
+  #   - AMD (acpi-cpufreq) appears to manage scaling via the OS *or* HW. but the ondemand defaults never put it to max hardware frequency.
+  #   - qualcomm (cpufreq-dt) appears to manage scaling *only* via the OS. ondemand governor exercises the full range.
+  # - query details with `sudo cpupower frequency-info`
+  powerManagement.cpuFreqGovernor = "ondemand";
+
+  # see: `man logind.conf`
+  # don’t shutdown when power button is short-pressed (commonly done an accident, or by cats).
+  #   but do on long-press: useful to gracefully power-off server.
+  services.logind.powerKey = "lock";
+  services.logind.powerKeyLongPress = "poweroff";
+  services.logind.lidSwitch = "lock";
+
+  # services.snapper.configs = {
+  #   root = {
+  #     subvolume = "/";
+  #     extraConfig = {
+  #       ALLOW_USERS = "colin";
+  #     };
+  #   };
+  # };
+  # services.snapper.snapshotInterval = "daily";
+}

hosts/common/hardware/x86_64.nix

@@ -0,0 +1,26 @@
+{ lib, pkgs, ... }:
+
+{
+  config = lib.mkIf (pkgs.system == "x86_64-linux") {
+    boot.initrd.availableKernelModules = [
+      "xhci_pci" "ahci" "sd_mod" "sdhci_pci"  # nixos-generate-config defaults
+      "usb_storage"   # rpi needed this to boot from usb storage, i think.
+      "nvme"  # to boot from nvme devices
+      # efi_pstore evivars
+
+      # added (speculatively) 2024/05/21; these were implicitly being added by nixos/modules/system/boot/kernel.nix
+      # i've copied not all of them, but most
+      "mmc_block"
+      "dm_mod"
+      # USB keyboards
+      "uhci_hcd" "ehci_hcd" "ehci_pci" "ohci_hcd" "ohci_pci" "xhci_hcd" "xhci_pci" "usbhid" "hid_generic" "hid_lenovo" "hid_apple" "hid_roccat" "hid_logitech_hidpp" "hid_logitech_dj" "hid_microsoft" "hid_cherry" "hid_corsair"
+      # x86 keyboard stuff
+      "pcips2" "atkbd" "i8042"
+      # stage-2 init needs rtc?
+      "rtc_cmos"
+    ];
+
+    hardware.cpu.amd.updateMicrocode = true;    # desktop
+    hardware.cpu.intel.updateMicrocode = true;  # laptop
+  };
+}

hosts/common/home/aerc.nix

@@ -1,11 +0,0 @@
-# Terminal UI mail client
-{ config, sane-lib, ... }:
-
-{
-  sops.secrets."aerc_accounts" = {
-    owner = config.users.users.colin.name;
-    sopsFile = ../../../secrets/universal/aerc_accounts.conf;
-    format = "binary";
-  };
-  sane.user.fs.".config/aerc/accounts.conf" = sane-lib.fs.wantedSymlinkTo config.sops.secrets.aerc_accounts.path;
-}

hosts/common/home/default.nix

@@ -1,23 +1,9 @@
 { ... }:
 {
   imports = [
-    ./aerc.nix
-    ./firefox.nix
-    ./gfeeds.nix
-    ./git.nix
-    ./gpodder.nix
-    ./keyring.nix
-    ./kitty.nix
-    ./libreoffice.nix
+    ./fs.nix
     ./mime.nix
-    ./mpv.nix
-    ./neovim.nix
-    ./newsflash.nix
-    ./splatmoji.nix
     ./ssh.nix
-    ./sublime-music.nix
-    ./vlc.nix
     ./xdg-dirs.nix
-    ./zsh
   ];
 }

hosts/common/home/firefox.nix

@@ -1,199 +0,0 @@
-# common settings to toggle (at runtime, in about:config):
-#   > security.ssl.require_safe_negotiation
-
-# librewolf is a forked firefox which patches firefox to allow more things
-# (like default search engines) to be configurable at runtime.
-# many of the settings below won't have effect without those patches.
-# see: https://gitlab.com/librewolf-community/settings/-/blob/master/distribution/policies.json
-
-{ config, lib, pkgs, sane-lib, ...}:
-with lib;
-let
-  cfg = config.sane.web-browser;
-  # allow easy switching between firefox and librewolf with `defaultSettings`, below
-  librewolfSettings = {
-    browser = pkgs.librewolf-unwrapped;
-    # browser = pkgs.librewolf-unwrapped.overrideAttrs (drv: {
-    #   # this allows side-loading unsigned addons
-    #   MOZ_REQUIRE_SIGNING = false;
-    # });
-    libName = "librewolf";
-    dotDir = ".librewolf";
-    cacheDir = ".cache/librewolf";  # TODO: is it?
-    desktop = "librewolf.desktop";
-  };
-  firefoxSettings = {
-    browser = pkgs.firefox-esr-unwrapped;
-    libName = "firefox";
-    dotDir = ".mozilla/firefox";
-    cacheDir = ".cache/mozilla";
-    desktop = "firefox.desktop";
-  };
-  defaultSettings = firefoxSettings;
-  # defaultSettings = librewolfSettings;
-
-  addon = name: extid: hash: pkgs.fetchFirefoxAddon {
-    inherit name hash;
-    url = "https://addons.mozilla.org/firefox/downloads/latest/${name}/latest.xpi";
-    # extid can be found by unar'ing the above xpi, and copying browser_specific_settings.gecko.id field
-    fixedExtid = extid;
-  };
-  localAddon = pkg: pkgs.fetchFirefoxAddon {
-    inherit (pkg) name;
-    src = "${pkg}/share/mozilla/extensions/\\{ec8030f7-c20a-464f-9b0e-13a3a9e97384\\}/${pkg.extid}.xpi";
-    fixedExtid = pkg.extid;
-  };
-
-  package = pkgs.wrapFirefox cfg.browser.browser {
-    # inherit the default librewolf.cfg
-    # it can be further customized via ~/.librewolf/librewolf.overrides.cfg
-    inherit (pkgs.librewolf-unwrapped) extraPrefsFiles;
-    inherit (cfg.browser) libName;
-
-    extraNativeMessagingHosts = [ pkgs.browserpass ];
-    # extraNativeMessagingHosts = [ pkgs.gopass-native-messaging-host ];
-
-    nixExtensions = concatMap (ext: optional ext.enable ext.package) (attrValues cfg.addons);
-
-    extraPolicies = {
-      NoDefaultBookmarks = true;
-      SearchEngines = {
-        Default = "DuckDuckGo";
-      };
-      AppUpdateURL = "https://localhost";
-      DisableAppUpdate = true;
-      OverrideFirstRunPage = "";
-      OverridePostUpdatePage = "";
-      DisableSystemAddonUpdate = true;
-      DisableFirefoxStudies = true;
-      DisableTelemetry = true;
-      DisableFeedbackCommands = true;
-      DisablePocket = true;
-      DisableSetDesktopBackground = false;
-
-      # remove many default search providers
-      # XXX this seems to prevent the `nixExtensions` from taking effect
-      # Extensions.Uninstall = [
-      #   "google@search.mozilla.org"
-      #   "bing@search.mozilla.org"
-      #   "amazondotcom@search.mozilla.org"
-      #   "ebay@search.mozilla.org"
-      #   "twitter@search.mozilla.org"
-      # ];
-      # XXX doesn't seem to have any effect...
-      # docs: https://github.com/mozilla/policy-templates#homepage
-      # Homepage = {
-      #   HomepageURL = "https://uninsane.org/";
-      #   StartPage = "homepage";
-      # };
-      # NewTabPage = true;
-    };
-  };
-
-  addonOpts = types.submodule {
-    options = {
-      package = mkOption {
-        type = types.package;
-      };
-      enable = mkOption {
-        type = types.bool;
-      };
-    };
-  };
-in
-{
-  options = {
-    sane.web-browser.browser = mkOption {
-      default = defaultSettings;
-      type = types.attrs;
-    };
-    sane.web-browser.persistData = mkOption {
-      description = "optional store name to which persist browsing data (like history)";
-      type = types.nullOr types.str;
-      default = null;
-    };
-    sane.web-browser.persistCache = mkOption {
-      description = "optional store name to which persist browser cache";
-      type = types.nullOr types.str;
-      default = "cryptClearOnBoot";
-    };
-    sane.web-browser.addons = mkOption {
-      type = types.attrsOf addonOpts;
-      default = {
-        # get names from:
-        # - ~/ref/nix-community/nur-combined/repos/rycee/pkgs/firefox-addons/generated-firefox-addons.nix
-        # `wget ...xpi`; `unar ...xpi`; `cat */manifest.json | jq '.browser_specific_settings.gecko.id'`
-        # browserpass-ce.package = addon "browserpass-ce" "browserpass@maximbaz.com" "sha256-sXgUBbRvMnRpeIW1MTkmTcoqtW/8RDXAkxAq1evFkpc=";
-        browserpass-extension.package = localAddon pkgs.browserpass-extension;
-        bypass-paywalls-clean.package = addon "bypass-paywalls-clean" "{d133e097-46d9-4ecc-9903-fa6a722a6e0e}" "sha256-JOj5P7c2JTTReHCRZXm4BscaGr3i+9Y4Ey/y621x8PI=";
-        ether-metamask.package = addon "ether-metamask" "webextension@metamask.io" "sha256-G+MwJDOcsaxYSUXjahHJmkWnjLeQ0Wven8DU/lGeMzA=";
-        i2p-in-private-browsing.package = addon "i2p-in-private-browsing" "i2ppb@eyedeekay.github.io" "sha256-dJcJ3jxeAeAkRvhODeIVrCflvX+S4E0wT/PyYzQBQWs=";
-        sidebery.package = addon "sidebery" "{3c078156-979c-498b-8990-85f7987dd929}" "sha256-YONfK/rIjlsrTgRHIt3km07Q7KnpIW89Z9r92ZSCc6w=";
-        sponsorblock.package = addon "sponsorblock" "sponsorBlocker@ajay.app" "sha256-d2K3ufvurWnYVzqLbyR//MgejybkY9exitAf9RdLNRo=";
-        ublacklist.package = addon "ublacklist" "@ublacklist" "sha256-RqY5iHzbL2qizth7aguyOKWPyINXmrwOlf/OsfqAS48=";
-        ublock-origin.package = addon "ublock-origin" "uBlock0@raymondhill.net" "sha256-a/ivUmY1P6teq9x0dt4CbgHt+3kBsEMMXlOfZ5Hx7cg=";
-
-        browserpass-extension.enable = lib.mkDefault true;
-        bypass-paywalls-clean.enable = lib.mkDefault true;
-        ether-metamask.enable = lib.mkDefault true;
-        i2p-in-private-browsing.enable = lib.mkDefault config.services.i2p.enable;
-        sidebery.enable = lib.mkDefault true;
-        sponsorblock.enable = lib.mkDefault true;
-        ublacklist.enable = lib.mkDefault true;
-        ublock-origin.enable = lib.mkDefault true;
-      };
-    };
-  };
-
-  config = {
-
-    # uBlock filter list configuration.
-    # specifically, enable the GDPR cookie prompt blocker.
-    # data.toOverwrite.filterLists is additive (i.e. it supplements the default filters)
-    # this configuration method is documented here:
-    # - <https://github.com/gorhill/uBlock/issues/2986#issuecomment-364035002>
-    # the specific attribute path is found via scraping ublock code here:
-    # - <https://github.com/gorhill/uBlock/blob/master/src/js/storage.js>
-    # - <https://github.com/gorhill/uBlock/blob/master/assets/assets.json>
-    sane.user.fs."${cfg.browser.dotDir}/managed-storage/uBlock0@raymondhill.net.json" = sane-lib.fs.wantedText ''
-      {
-       "name": "uBlock0@raymondhill.net",
-       "description": "ignored",
-       "type": "storage",
-       "data": {
-          "toOverwrite": "{\"filterLists\": [\"fanboy-cookiemonster\"]}"
-       }
-      }
-    '';
-    sane.user.fs."${cfg.browser.dotDir}/${cfg.browser.libName}.overrides.cfg" = sane-lib.fs.wantedText ''
-      // if we can't query the revocation status of a SSL cert because the issuer is offline,
-      // treat it as unrevoked.
-      // see: <https://librewolf.net/docs/faq/#im-getting-sec_error_ocsp_server_error-what-can-i-do>
-      defaultPref("security.OCSP.require", false);
-    '';
-
-    sane.packages.extraGuiPkgs = [ package ];
-    # flush the cache to disk to avoid it taking up too much tmp
-    sane.user.persist.byPath."${cfg.browser.cacheDir}" = lib.mkIf (cfg.persistCache != null) {
-      store = cfg.persistCache;
-    };
-
-    sane.user.persist.byPath."${cfg.browser.dotDir}/default" = lib.mkIf (cfg.persistData != null) {
-      store = cfg.persistData;
-    };
-    sane.user.fs."${cfg.browser.dotDir}/default" = sane-lib.fs.wantedDir;
-    # instruct Firefox to put the profile in a predictable directory (so we can do things like persist just it).
-    # XXX: the directory *must* exist, even if empty; Firefox will not create the directory itself.
-    sane.user.fs."${cfg.browser.dotDir}/profiles.ini" = sane-lib.fs.wantedText ''
-      [Profile0]
-      Name=default
-      IsRelative=1
-      Path=default
-      Default=1
-
-      [General]
-      StartWithLastProfile=1
-    '';
-
-  };
-}

hosts/common/home/fs.nix

@@ -0,0 +1,45 @@
+{ config, lib, ... }:
+{
+  sane.user.persist.byStore.plaintext = [
+    "archive"
+    "dev"
+    # TODO: records should be private
+    "records"
+    "ref"
+    "tmp"
+    "use"
+    "Books/local"
+    "Music"
+    "Pictures/albums"
+    "Pictures/cat"
+    "Pictures/from"
+    "Pictures/Screenshots"  #< XXX: something is case-sensitive about this?
+    "Pictures/Photos"
+    "Videos/local"
+
+    # these are persisted simply to save on RAM.
+    # ~/.cache/nix can become several GB.
+    # mesa_shader_cache is < 10 MB.
+    # TODO: integrate with sane.programs.sandbox?
+    ".cache/mesa_shader_cache"
+    ".cache/nix"
+  ];
+  sane.user.persist.byStore.private = [
+    "knowledge"
+  ];
+
+  # convenience
+  sane.user.fs = let
+    persistEnabled = config.sane.persist.enable;
+  in {
+    ".persist/private" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.private.origin; };
+    ".persist/plaintext" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.plaintext.origin; };
+    ".persist/ephemeral" = lib.mkIf persistEnabled { symlink.target = config.sane.persist.stores.cryptClearOnBoot.origin; };
+
+    "nixos".symlink.target = "dev/nixos";
+
+    "Books/servo".symlink.target = "/mnt/servo/media/Books";
+    "Videos/servo".symlink.target = "/mnt/servo/media/Videos";
+    "Pictures/servo-macros".symlink.target = "/mnt/servo/media/Pictures/macros";
+  };
+}

hosts/common/home/gfeeds.nix

@@ -1,42 +0,0 @@
-# gnome feeds RSS viewer
-{ config, lib, sane-lib, ... }:
-
-let
-  feeds = sane-lib.feeds;
-  all-feeds = config.sane.feeds;
-  wanted-feeds = feeds.filterByFormat ["text" "image"] all-feeds;
-in {
-  sane.user.fs.".config/org.gabmus.gfeeds.json" = sane-lib.fs.wantedText (
-    builtins.toJSON {
-      # feed format is a map from URL to a dict,
-      #   with dict["tags"] a list of string tags.
-      feeds = sane-lib.mapToAttrs (feed: {
-        name = feed.url;
-        value.tags = [ feed.cat feed.freq ];
-      }) wanted-feeds;
-      dark_reader = false;
-      new_first = true;
-      # windowsize = {
-      #   width = 350;
-      #   height = 650;
-      # };
-      max_article_age_days = 90;
-      enable_js = false;
-      max_refresh_threads = 3;
-      # saved_items = {};
-      # read_items = [];
-      show_read_items = true;
-      full_article_title = true;
-      # views: "webview", "reader", "rsscont"
-      default_view = "rsscont";
-      open_links_externally = true;
-      full_feed_name = false;
-      refresh_on_startup = true;
-      tags = lib.unique (
-        (builtins.catAttrs "cat" wanted-feeds) ++ (builtins.catAttrs "freq" wanted-feeds)
-      );
-      open_youtube_externally = false;
-      media_player = "vlc";  # default: mpv
-    }
-  );
-}

hosts/common/home/git.nix

@@ -1,18 +0,0 @@
-{ lib, pkgs, sane-lib, ... }:
-
-let
-  mkCfg = lib.generators.toINI { };
-in
-{
-  sane.user.fs.".config/git/config" = sane-lib.fs.wantedText (mkCfg {
-    user.name = "Colin";
-    user.email = "colin@uninsane.org";
-    alias.co = "checkout";
-    # difftastic docs:
-    # - <https://difftastic.wilfred.me.uk/git.html>
-    diff.tool = "difftastic";
-    difftool.prompt = false;
-    "difftool \"difftastic\"".cmd = ''${pkgs.difftastic}/bin/difft "$LOCAL" "$REMOTE"'';
-    # now run `git difftool` to use difftastic git
-  });
-}

hosts/common/home/gpodder.nix

@@ -1,12 +0,0 @@
-# gnome feeds RSS viewer
-{ config, sane-lib, ... }:
-
-let
-  feeds = sane-lib.feeds;
-  all-feeds = config.sane.feeds;
-  wanted-feeds = feeds.filterByFormat ["podcast"] all-feeds;
-in {
-  sane.user.fs.".config/gpodderFeeds.opml" = sane-lib.fs.wantedText (
-    feeds.feedsToOpml wanted-feeds
-  );
-}

hosts/common/home/keyring.nix

@@ -1,11 +0,0 @@
-{ config, sane-lib, ... }:
-
-{
-  sane.user.persist.private = [ ".local/share/keyrings" ];
-
-  sane.user.fs."private/.local/share/keyrings/default" = {
-    generated.script.script = builtins.readFile ../../../scripts/init-keyring;
-    # TODO: is this `wantedBy` needed? can we inherit it?
-    wantedBy = [ config.sane.fs."/home/colin/private".unit ];
-  };
-}

hosts/common/home/kitty.nix

@@ -1,68 +0,0 @@
-{ pkgs, sane-lib, ... }:
-
-{
-  sane.user.fs.".config/kitty/kitty.conf" = sane-lib.fs.wantedText ''
-    # docs: https://sw.kovidgoyal.net/kitty/conf/
-    # disable terminal bell (when e.g. you backspace too many times)
-    enable_audio_bell no
-
-    map ctrl+n new_os_window_with_cwd
-
-    include ${pkgs.kitty-themes}/themes/PaperColor_dark.conf
-  '';
-  # THEME CHOICES:
-  #   docs: https://github.com/kovidgoyal/kitty-themes
-  # theme = "1984 Light";  # dislike: awful, harsh blues/teals
-  # theme = "Adventure Time";  # dislike: harsh (dark)
-  # theme = "Atom One Light";  # GOOD: light theme. all color combos readable. not a huge fan of the blue.
-  # theme = "Belafonte Day";  # dislike: too low contrast for text colors
-  # theme = "Belafonte Night";  # better: dark theme that's easy on the eyes. all combos readable. low contrast.
-  # theme = "Catppuccin";  # dislike: a bit pale/low-contrast (dark)
-  # theme = "Desert";  # mediocre: colors are harsh
-  # theme = "Earthsong";  # BEST: dark theme. readable, good contrast. unique, but decent colors.
-  # theme = "Espresso Libre";  # better: dark theme. readable, but meh colors
-  # theme = "Forest Night";  # decent: very pastel. it's workable, but unconventional and muted/flat.
-  # theme = "Gruvbox Material Light Hard";  # mediocre light theme.
-  # theme = "kanagawabones";  # better: dark theme. colors are too background-y
-  # theme = "Kaolin Dark";  # dislike: too dark
-  # theme = "Kaolin Breeze";  # mediocre: not-too-harsh light theme, but some parts are poor contrast
-  # theme = "Later This Evening";  # mediocre: not-too-harsh dark theme, but cursor is poor contrast
-  # theme = "Material"; # decent: light theme, few colors.
-  # theme = "Mayukai";  # decent: not-too-harsh dark theme. the teal is a bit straining
-  # theme = "Nord";  # mediocre: pale background, low contrast
-  # theme = "One Half Light";  # better: not-too-harsh light theme. contrast could be better
-  # theme = "PaperColor Dark";  # BEST: dark theme, very readable still the colors are background-y
-  # theme = "Parasio Dark";  # dislike: too low contrast
-  # theme = "Pencil Light";  # better: not-too-harsh light theme. decent contrast.
-  # theme = "Pnevma";  # dislike: too low contrast
-  # theme = "Piatto Light";  # better: readable light theme. pleasing colors. powerline prompt is hard to read.
-  # theme = "Rosé Pine Dawn";  # GOOD: light theme. all color combinations are readable. it is very mild -- may need to manually tweak contrast. tasteful colors
-  # theme = "Rosé Pine Moon";  # GOOD: dark theme. tasteful colors. but background is a bit intense
-  # theme = "Sea Shells";  # mediocre. not all color combos are readable
-  # theme = "Solarized Light";  # mediocre: not-too-harsh light theme; GREAT background; but some colors are low contrast
-  # theme = "Solarized Dark Higher Contrast";  # better: dark theme, decent colors
-  # theme = "Sourcerer";  # mediocre: ugly colors
-  # theme = "Space Gray";  # mediocre: too muted
-  # theme = "Space Gray Eighties";  # better: all readable, decent colors
-  # theme = "Spacemacs";  # mediocre: too muted
-  # theme = "Spring";  # mediocre: readable light theme, but the teal is ugly.
-  # theme = "Srcery";  # better: highly readable. colors are ehhh
-  # theme = "Substrata";  # decent: nice colors, but a bit flat.
-  # theme = "Sundried";  # mediocre: the solar text makes me squint
-  # theme = "Symfonic";  # mediocre: the dark purple has low contrast to the black bg.
-  # theme = "Tango Light";  # dislike: teal is too grating
-  # theme = "Tokyo Night Day";  # medicore: too muted
-  # theme = "Tokyo Night";  # better: tasteful. a bit flat
-  # theme = "Tomorrow";  # GOOD: all color combinations are readable. contrast is slightly better than Rose. on the blander side
-  # theme = "Treehouse";  # dislike: the orange is harsh on my eyes.
-  # theme = "Urple";  # dislike: weird palette
-  # theme = "Warm Neon";  # decent: not-too-harsh dark theme. the green is a bit unattractive
-  # theme = "Wild Cherry";  # GOOD: dark theme: nice colors. a bit flat
-  # theme = "Xcodedark";  # dislike: bad palette
-  # theme = "citylights";  # decent: dark theme. some parts have just a bit low contrast
-  # theme = "neobones_light"; # better light theme. the background is maybe too muted
-  # theme = "vimbones";
-  # theme = "zenbones_dark";  # mediocre: readable, but meh colors
-  # theme = "zenbones_light";  # decent: light theme. all colors are readable. contrast is passable but not excellent. highlight color is BAD
-  # theme = "zenwritten_dark";  # mediocre: looks same as zenbones_dark
-}

hosts/common/home/libreoffice.nix

@@ -1,14 +0,0 @@
-{ sane-lib, ... }:
-
-{
-  # libreoffice: disable first-run stuff
-  sane.user.fs.".config/libreoffice/4/user/registrymodifications.xcu" = sane-lib.fs.wantedText ''
-    <?xml version="1.0" encoding="UTF-8"?>
-    <oor:items xmlns:oor="http://openoffice.org/2001/registry" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
-      <item oor:path="/org.openoffice.Office.Common/Misc"><prop oor:name="FirstRun" oor:op="fuse"><value>false</value></prop></item>
-      <item oor:path="/org.openoffice.Office.Common/Misc"><prop oor:name="ShowTipOfTheDay" oor:op="fuse"><value>false</value></prop></item>
-    </oor:items>
-  '';
-  # <item oor:path="/org.openoffice.Setup/Product"><prop oor:name="LastTimeDonateShown" oor:op="fuse"><value>1667693880</value></prop></item>
-  # <item oor:path="/org.openoffice.Setup/Product"><prop oor:name="LastTimeGetInvolvedShown" oor:op="fuse"><value>1667693880</value></prop></item>
-}

hosts/common/home/mime.nix

@@ -1,42 +1,94 @@
-{ config, sane-lib, ...}:
+# TODO: move into modules/users.nix
+{ config, lib, pkgs, ...}:
 
 let
-  www = config.sane.web-browser.browser.desktop;
-  pdf = "org.gnome.Evince.desktop";
-  md = "obsidian.desktop";
-  thumb = "org.gnome.gThumb.desktop";
-  video = "vlc.desktop";
-  # audio = "mpv.desktop";
-  audio = "vlc.desktop";
+  # [ ProgramConfig ]
+  enabledPrograms = builtins.filter
+    (p: p.enabled)
+    (builtins.attrValues config.sane.programs);
+
+  # [ ProgramConfig ]
+  enabledProgramsWithPackage = builtins.filter (p: p.package != null) enabledPrograms;
+
+  # [ { "<mime-type>" = { prority, desktop } ]
+  enabledWeightedMimes = builtins.map weightedMimes enabledPrograms;
+
+  # ProgramConfig -> { "<mime-type>" = { priority, desktop }; }
+  weightedMimes = prog: builtins.mapAttrs
+    (_key: desktop: {
+      priority = prog.mime.priority; desktop = desktop;
+    })
+    prog.mime.associations;
+
+  # [ { "<mime-type>" = { priority, desktop } ]; } ] -> { "<mime-type>" = [ { priority, desktop } ... ]; }
+  mergeMimes = mimes: lib.foldAttrs (item: acc: [item] ++ acc) [] mimes;
+
+  # [ { priority, desktop } ... ] -> Self
+  sortOneMimeType = associations: builtins.sort
+    (l: r: lib.throwIf
+      (l.priority == r.priority)
+      "${l.desktop} and ${r.desktop} share a preferred mime type with identical priority ${builtins.toString l.priority} (and so the desired association is ambiguous)"
+      (l.priority < r.priority)
+    )
+    associations;
+  sortMimes = mimes: builtins.mapAttrs (_k: sortOneMimeType) mimes;
+  # { "<mime-type>"} = [ { priority, desktop } ... ]; } -> { "<mime-type>" = [ "<desktop>" ... ]; }
+  removePriorities = mimes: builtins.mapAttrs
+    (_k: associations: builtins.map (a: a.desktop) associations)
+    mimes;
+  # { "<mime-type>" = [ "<desktop>" ... ]; } -> { "<mime-type>" = "<desktop1>;<desktop2>;..."; }
+  formatDesktopLists = mimes: builtins.mapAttrs
+    (_k: desktops: lib.concatStringsSep ";" desktops)
+    mimes;
+
+  mimeappsListPkg = pkgs.writeTextDir "share/applications/mimeapps.list" (
+    lib.generators.toINI { } {
+      "Default Applications" = formatDesktopLists (removePriorities (sortMimes (mergeMimes enabledWeightedMimes)));
+    }
+  );
+
+  localShareApplicationsPkg = (pkgs.symlinkJoin {
+    name = "user-local-share-applications";
+    paths = builtins.map
+      (p: builtins.toString p.package)
+      (enabledProgramsWithPackage ++ [ { package=mimeappsListPkg; } ]);
+  }).overrideAttrs (orig: {
+    # like normal symlinkJoin, but don't error if the path doesn't exist
+    buildCommand = ''
+      mkdir -p $out/share/applications
+      for i in $(cat $pathsPath); do
+        if [ -e "$i/share/applications" ]; then
+          ${pkgs.buildPackages.xorg.lndir}/bin/lndir -silent $i/share/applications $out/share/applications
+        fi
+      done
+      runHook postBuild
+    '';
+    postBuild = ''
+      # rebuild `mimeinfo.cache`, used by file openers to show the list of *all* apps, not just the user's defaults.
+      ${pkgs.buildPackages.desktop-file-utils}/bin/update-desktop-database $out/share/applications
+    '';
+  });
+
 in
 {
-
   # the xdg mime type for a file can be found with:
   # - `xdg-mime query filetype path/to/thing.ext`
+  # the default handler for a mime type can be found with:
+  # - `xdg-mime query default <mimetype>`  (e.g. x-scheme-handler/http)
+  # the nix-configured handler can be found `nix-repl > :lf . > hostConfigs.desko.xdg.mime.defaultApplications`
+  #
+  # glib/gio is queried via glib.bin output:
+  # - `gio mime x-scheme-handler/https`
+  # - `gio open <path_or_url>`
+  # - `gio launch </path/to/app.desktop>`
+  #
   # we can have single associations or a list of associations.
   # there's also options to *remove* [non-default] associations from specific apps
-  xdg.mime.enable = true;
-  xdg.mime.defaultApplications = {
-    # AUDIO
-    "audio/flac" = audio;
-    "audio/mpeg" = audio;
-    "audio/x-vorbis+ogg" = audio;
-    # IMAGES
-    "image/heif" = thumb;  # apple codec
-    "image/png" = thumb;
-    "image/jpeg" = thumb;
-    # VIDEO
-    "video/mp4" = video;
-    "video/quicktime" = video;
-    "video/x-matroska" = video;
-    # HTML
-    "text/html" = www;
-    "x-scheme-handler/http" = www;
-    "x-scheme-handler/https" = www;
-    "x-scheme-handler/about" = www;
-    "x-scheme-handler/unknown" = www;
-    # RICH-TEXT DOCUMENTS
-    "application/pdf" = pdf;
-    "text/markdown" = md;
-  };
+  # N.B.: don't use nixos' `xdg.mime` option becaue that caues `/share/applications` to be linked into the whole system,
+  # which limits what i can do around sandboxing. getting the default associations to live in ~/ makes it easier to expose
+  # the associations to apps selectively.
+  # xdg.mime.enable = true;
+  # xdg.mime.defaultApplications = removePriorities (sortMimes (mergeMimes enabledWeightedMimes));
+
+  sane.user.fs.".local/share/applications".symlink.target = "${localShareApplicationsPkg}/share/applications";
 }

hosts/common/home/mpv.nix

@@ -1,10 +0,0 @@
-{ sane-lib, ... }:
-
-{
-  # format is <key>=%<length>%<value>
-  sane.user.fs.".config/mpv/mpv.conf" = sane-lib.fs.wantedText ''
-    save-position-on-quit=%3%yes
-    keep-open=%3%yes
-  '';
-}
-