modules/programs: require manual definition; don't auto-populate attrset

this greatly decreases nix eval time

README.md

@@ -1,3 +1,7 @@
+![hello](doc/hello.gif)
+
+# .❄️≡We|_c0m3 7o m`/ f14k≡❄️.
+
 ## What's Here
 
 this is the top-level repo from which i configure/deploy all my NixOS machines:
@@ -6,18 +10,18 @@ this is the top-level repo from which i configure/deploy all my NixOS machines:
 - server
 - mobile phone (Pinephone)
 
-everything outside of <./hosts/> and <./secrets/> is intended for export, to be importable for use by 3rd parties.
+everything outside of [hosts/](./hosts/) and [secrets/](./secrets/) is intended for export, to be importable for use by 3rd parties.
 the only hard dependency for my exported pkgs/modules should be [nixpkgs][nixpkgs].
-building <./hosts/> will require [sops][sops].
+building [hosts/](./hosts/) will require [sops][sops].
 
 you might specifically be interested in these files (elaborated further in #key-points-of-interest):
 - [`sxmo-utils`](./pkgs/additional/sxmo-utils/default.nix)
   - [example SXMO deployment](./hosts/modules/gui/sxmo/default.nix)
 - [my implementation of impermanence](./modules/persist/default.nix)
 - my way of deploying dotfiles/configuring programs per-user:
-  - <./modules/fs/default.nix>
+  - [modules/fs/](./modules/fs/default.nix)
-  - <./modules/programs.nix>
+  - [modules/programs/](./modules/programs/default.nix)
-  - <./modules/users.nix>
+  - [modules/users.nix](./modules/users.nix)
 
 [nixpkgs]: https://github.com/NixOS/nixpkgs
 [sops]: https://github.com/Mic92/sops-nix
@@ -35,37 +39,37 @@ or follow the instructions [here][NUR] to use it via the Nix User Repositories.
 
 ## Layout
 - `doc/`
-    - instructions for tasks i find myself doing semi-occasionally in this repo.
+  - instructions for tasks i find myself doing semi-occasionally in this repo.
 - `hosts/`
-    - the bulk of config which isn't factored with external use in mind.
+  - the bulk of config which isn't factored with external use in mind.
-    - that is, if you were to add this repo to a flake.nix for your own use,
+  - that is, if you were to add this repo to a flake.nix for your own use,
-      you won't likely be depending on anything in this directory.
+    you won't likely be depending on anything in this directory.
 - `integrations/`
-    - code intended for consumption by external tools (e.g. the Nix User Repos)
+  - code intended for consumption by external tools (e.g. the Nix User Repos)
 - `modules/`
-    - config which is gated behind `enable` flags, in similar style to nixpkgs'
+  - config which is gated behind `enable` flags, in similar style to nixpkgs'
-      `nixos/` directory.
+    `nixos/` directory.
-    - if you depend on this repo, it's most likely for something in this directory.
+  - if you depend on this repo, it's most likely for something in this directory.
 - `nixpatches/`
-    - literally, diffs i apply atop upstream nixpkgs before performing further eval.
+  - literally, diffs i apply atop upstream nixpkgs before performing further eval.
 - `overlays/`
-    - exposed via the `overlays` output in `flake.nix`.
+  - exposed via the `overlays` output in `flake.nix`.
-    - predominantly a list of `callPackage` directives.
+  - predominantly a list of `callPackage` directives.
 - `pkgs/`
-    - derivations for things not yet packaged in nixpkgs.
+  - derivations for things not yet packaged in nixpkgs.
-    - derivations for things from nixpkgs which i need to `override` for some reason.
+  - derivations for things from nixpkgs which i need to `override` for some reason.
-    - inline code for wholly custom packages (e.g. `pkgs/additional/sane-scripts/` for CLI tools
+  - inline code for wholly custom packages (e.g. `pkgs/additional/sane-scripts/` for CLI tools
-      that are highly specific to my setup).
+    that are highly specific to my setup).
 - `scripts/`
-    - scripts which aren't reachable on a deployed system, but may aid manual deployments
+  - scripts which aren't reachable on a deployed system, but may aid manual deployments
 - `secrets/`
-    - encrypted keys, API tokens, anything which one or more of my machines needs
+  - encrypted keys, API tokens, anything which one or more of my machines needs
-      read access to but shouldn't be world-readable.
+    read access to but shouldn't be world-readable.
-    - not much to see here
+  - not much to see here
 - `templates/`
-    - exposed via the `templates` output in `flake.nix`.
+  - exposed via the `templates` output in `flake.nix`.
-    - used to instantiate short-lived environments.
+  - used to instantiate short-lived environments.
-    - used to auto-fill the boiler-plate portions of new packages.
+  - used to auto-fill the boiler-plate portions of new packages.
 
 
 ## Key Points of Interest
@@ -73,35 +77,40 @@ or follow the instructions [here][NUR] to use it via the Nix User Repositories.
 i.e. you might find value in using these in your own config:
 
 - `modules/fs/`
-    - use this to statically define leafs and nodes anywhere in the filesystem,
+  - use this to statically define leafs and nodes anywhere in the filesystem,
-      not just inside `/nix/store`.
+    not just inside `/nix/store`.
-    - e.g. specify that `/var/www` should be:
+  - e.g. specify that `/var/www` should be:
-        - owned by a specific user/group
+    - owned by a specific user/group
-        - set to a specific mode
+    - set to a specific mode
-        - symlinked to some other path
+    - symlinked to some other path
-        - populated with some statically-defined data
+    - populated with some statically-defined data
-        - populated according to some script
+    - populated according to some script
-        - created as a dependency of some service (e.g. `nginx`)
+    - created as a dependency of some service (e.g. `nginx`)
-    - values defined here are applied neither at evaluation time _nor_ at activation time.
+  - values defined here are applied neither at evaluation time _nor_ at activation time.
-        - rather, they become systemd services.
+    - rather, they become systemd services.
-        - systemd manages dependencies
+    - systemd manages dependencies
-        - e.g. link `/var/www -> /mnt/my-drive/www` only _after_ `/mnt/my-drive/www` appears)
+    - e.g. link `/var/www -> /mnt/my-drive/www` only _after_ `/mnt/my-drive/www` appears)
-    - this is akin to using [Home Manager's][home-manager] file API -- the part which lets you
+  - this is akin to using [Home Manager's][home-manager] file API -- the part which lets you
-      statically define `~/.config` files -- just with a different philosophy.
+    statically define `~/.config` files -- just with a different philosophy.
 - `modules/persist/`
-    - my alternative to the Impermanence module.
+  - my alternative to the Impermanence module.
-    - this builds atop `modules/fs/` to achieve things stock impermanence can't:
+  - this builds atop `modules/fs/` to achieve things stock impermanence can't:
-        - persist things to encrypted storage which is unlocked at login time (pam_mount).
+    - persist things to encrypted storage which is unlocked at login time (pam_mount).
-        - "persist" cache directories -- to free up RAM -- but auto-wipe them on mount
+    - "persist" cache directories -- to free up RAM -- but auto-wipe them on mount
-          and encrypt them to ephemeral keys so they're unreadable post shutdown/unmount.
+      and encrypt them to ephemeral keys so they're unreadable post shutdown/unmount.
-- `modules/programs.nix`
+- `modules/programs/`
-    - like nixpkgs' `programs` options, but allows both system-wide or per-user deployment.
+  - like nixpkgs' `programs` options, but allows both system-wide or per-user deployment.
-    - allows `fs` and `persist` config values to be gated behind program deployment:
+  - allows `fs` and `persist` config values to be gated behind program deployment:
-        - e.g. `/home/<user>/.mozilla/firefox` is persisted only for users who
+    - e.g. `/home/<user>/.mozilla/firefox` is persisted only for users who
-          `sane.programs.firefox.enableFor.user."<user>" = true;`
+      `sane.programs.firefox.enableFor.user."<user>" = true;`
+  - allows aggressive sandboxing any program:
+    - `sane.programs.firefox.sandbox.method = "bwrap";  # sandbox with bubblewrap`
+    - `sane.programs.firefox.sandbox.whitelistWayland = true;  # allow it to render a wayland window`
+    - `sane.programs.firefox.sandbox.extraHomePaths = [ "Downloads" ];  # allow it read/write access to ~/Downloads`
+    - integrated with `fs` and `persist` modules so that programs' config files and persisted data stores are linked into the sandbox w/o any extra involvement.
 - `modules/users.nix`
-    - convenience layer atop the above modules so that you can just write
+  - convenience layer atop the above modules so that you can just write
-      `fs.".config/git"` instead of `fs."/home/colin/.config/git"`
+    `fs.".config/git"` instead of `fs."/home/colin/.config/git"`
 
 some things in here could easily find broader use. if you would find benefit in
 them being factored out of my config, message me and we could work to make that happen.

TODO.md

@@ -1,16 +1,14 @@
 ## BUGS
 - ringer (i.e. dino incoming call) doesn't prevent moby from sleeping
-- Fractal opens links with non-preferred web browser
 - `nix` operations from lappy hang when `desko` is unreachable
   - could at least direct the cache to `http://desko-hn:5001`
-- waybar isn't visible on moby until after `swaymsg reload`
 
 ## REFACTORING:
-
+- consolidate ~/dev and ~/ref
+  - ~/dev becomes a link to ~/ref/cat/mine
 - fold hosts/common/home/ssh.nix -> hosts/common/users/colin.nix
 
 ### sops/secrets
-- attach secrets to the thing they're used by (sane.programs)
 - rework secrets to leverage `sane.fs`
 - remove sops activation script as it's covered by my systemd sane.fs impl
 
@@ -23,12 +21,10 @@
 - bump nodejs version in lemmy-ui
 - add updateScripts to all my packages in nixpkgs
 - fix lightdm-mobile-greeter for newer libhandy
-- port zecwallet-lite to a from-source build
 - REVIEW/integrate jellyfin dataDir config: <https://github.com/NixOS/nixpkgs/pull/233617>
 
 #### upstreaming to non-nixpkgs repos
 - gtk: build schemas even on cross compilation: <https://github.com/NixOS/nixpkgs/pull/247844>
-- sxmo: add new app entries
 
 
 ## IMPROVEMENTS:
@@ -36,23 +32,27 @@
 - validate duplicity backups!
 - encrypt more ~ dirs (~/archives, ~/records, ..?)
   - best to do this after i know for sure i have good backups
-- have `sane.programs` be wrapped such that they run in a cgroup?
+- /mnt/desko/home, etc, shouldn't include secrets (~/private)
-  - at least, only give them access to the portion of the fs they *need*.
+  - 95% of its use is for remote media access and stuff which isn't in VCS (~/records)
-  - Android takes approach of giving each app its own user: could hack that in here.
+- port all sane.programs to be sandboxed
-  - **systemd-run** takes a command and runs it in a temporary scope (cgroup)
+  - enforce that all `environment.packages` has a sandbox profile (or explicitly opts out)
-    - presumably uses the same options as systemd services
+  - revisit "non-sandboxable" apps and check that i'm not actually just missing mountpoints
-    - see e.g. <https://github.com/NixOS/nixpkgs/issues/113903#issuecomment-857296349>
+    - LL_FS_RW=/ isn't enough -- need all mount points like `=/:/proc:/sys:...`.
-  - flatpak does this, somehow
+  - ensure non-bin package outputs are linked for sandboxed apps
-  - apparmor?  SElinux?  (desktop) "portals"?
+    - i.e. `outputs.man`, `outputs.debug`, `outputs.doc`, ...
-  - see Spectrum OS; Alyssa Ross; etc
+  - lock down dbus calls within the sandbox
-  - bubblewrap-based sandboxing: <https://github.com/nixpak/nixpak>
+    - otherwise anyone can `systemd-run --user ...` to potentially escape a sandbox
+    - <https://github.com/flatpak/xdg-dbus-proxy>
+  - remove `.ssh` access from Firefox!
+    - limit access to `~/knowledge/secrets` through an agent that requires GUI approval, so a firefox exploit can't steal all my logins
+  - port sane-sandboxed to a compiled language (hare?)
+    - it adds like 50-70ms launch time _on my laptop_. i'd hate to know how much that is on the pinephone.
+- make dconf stuff less monolithic
+  - i.e. per-app dconf profiles for those which need it. possible static config.
 - canaries for important services
   - e.g. daily email checks; daily backup checks
   - integrate `nix check` into Gitea actions?
 
-### faster/better deployments
-- remove audacity's dependency on webkitgtk (via wxwidgets)
-
 ### user experience
 - install apps:
   - display QR codes for WiFi endpoints: <https://linuxphoneapps.org/apps/noappid.wisperwind.wifi2qr/>
@@ -70,14 +70,15 @@
   - UnCiv (Civ V clone; nixpkgs `unciv`; doesn't cross-compile):  <https://github.com/yairm210/UnCiv>
   - Simon Tatham's Puzzle Collection (not in nixpkgs) <https://git.tartarus.org/?p=simon/puzzles.git>
   - Shootin Stars  (Godot; not in nixpkgs) <https://gitlab.com/greenbeast/shootin-stars>
+  - numberlink (generic name for Flow Free). not packaged in Nix
+  - Neverball (https://neverball.org/screenshots.php). nix: as `neverball`
+  - blurble (https://linuxphoneapps.org/games/app.drey.blurble/). nix: not as of 2024-02-05
 
 #### moby
 - fix cpuidle (gets better power consumption): <https://xnux.eu/log/077.html>
 - SwayNC:
   - don't show MPRIS if no players detected
     - this is a problem of playerctld, i guess
-    - also, the album icon when "Not playing" doesn't follow the size we give in the config
-      - that means mpris always takes up excessive space on moby
   - add option to change audio output
   - fix colors (red alert) to match overall theme
 - moby: tune GPS
@@ -88,15 +89,9 @@
   - manually do smoothing, as some layer between mepo and geoclue/gpsd?
 - moby: show battery state on ssh login
 - moby: improve gPodder launch time
-- sxmo: port to swaybar like i use on desktop
-  - users in #sxmo claim it's way better perf
-- sxmo: fix youtube scripts (package youtube-cli)
 - moby: theme GTK apps (i.e. non-adwaita styles)
-  - combine multiple icon themes to get one which has the full icon set?
-  - get adwaita-icon-theme to ship everything even when cross-compiled?
   - especially, make the menubar collapsible
   - try Gradience tool specifically for theming adwaita? <https://linuxphoneapps.org/apps/com.github.gradienceteam.gradience/>
-- phog: remove the gnome-shell runtime dependency to save hella closure size
 
 #### non-moby
 - RSS: integrate a paywall bypass
@@ -118,16 +113,13 @@
   - could change junk filter from "no DKIM success" to explicit "DKIM failed"
 
 ### perf
+- debug nixos-rebuild times
+  - i bet sane.programs adds a LOT of time, with how it automatically creates an attrs for EVERY package in nixpkgs.
 - add `pkgs.impure-cached.<foo>` package set to build things with ccache enabled
   - every package here can be auto-generated, and marked with some env var so that it doesn't pollute the pure package set
   - would be super handy for package prototyping!
-- get moby to build without binfmt emulation (i.e. make all emulation explicit)
-  - then i can distribute builds across servo + desko, and also allow servo to pull packages from desko w/o worrying about purity
-
 
 ## NEW FEATURES:
 - migrate MAME cabinet to nix
   - boot it from PXE from servo?
-- deploy to new server, and use it as a remote builder
 - enable IPv6
-- package lemonade lemmy app: <https://linuxphoneapps.org/apps/ml.mdwalters.lemonade/>

flake.lock

@@ -1,20 +1,20 @@
 {
   "nodes": {
-    "flake-utils": {
+    "flake-parts": {
       "inputs": {
-        "systems": "systems"
+        "nixpkgs-lib": "nixpkgs-lib"
       },
       "locked": {
-        "lastModified": 1694529238,
+        "lastModified": 1698882062,
-        "narHash": "sha256-zsNZZGTGnMOf9YpHKJqMSsa0dXbfmxeoJ7xHlrt+xmY=",
+        "narHash": "sha256-HkhafUayIqxXyHH1X8d9RDl1M2CkFgZLjKD3MzabiEo=",
-        "owner": "numtide",
+        "owner": "hercules-ci",
-        "repo": "flake-utils",
+        "repo": "flake-parts",
-        "rev": "ff7b65b44d01cf9ba6a71320833626af21126384",
+        "rev": "8c9fa2545007b49a5db5f650ae91f227672c3877",
         "type": "github"
       },
       "original": {
-        "owner": "numtide",
+        "owner": "hercules-ci",
-        "repo": "flake-utils",
+        "repo": "flake-parts",
         "type": "github"
       }
     },
@@ -35,29 +35,99 @@
         "type": "github"
       }
     },
-    "nixpkgs-stable": {
+    "nix-fast-build": {
+      "inputs": {
+        "flake-parts": "flake-parts",
+        "nixpkgs": "nixpkgs",
+        "treefmt-nix": "treefmt-nix"
+      },
       "locked": {
-        "lastModified": 1700905716,
+        "lastModified": 1703607026,
-        "narHash": "sha256-w1vHn2MbGfdC+CrP3xLZ3scsI06N0iQLU7eTHIVEFGw=",
+        "narHash": "sha256-Emh0BPoqlS4ntp2UJrwydXfIP4qIMF0VBB2FUE3/M/E=",
+        "owner": "Mic92",
+        "repo": "nix-fast-build",
+        "rev": "4376b8a33b217ee2f78ba3dcff01a3e464d13a46",
+        "type": "github"
+      },
+      "original": {
+        "owner": "Mic92",
+        "repo": "nix-fast-build",
+        "type": "github"
+      }
+    },
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1698890957,
+        "narHash": "sha256-DJ+SppjpPBoJr0Aro9TAcP3sxApCSieY6BYBCoWGUX8=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "dfb95385d21475da10b63da74ae96d89ab352431",
+        "rev": "c082856b850ec60cda9f0a0db2bc7bd8900d708c",
         "type": "github"
       },
       "original": {
         "owner": "NixOS",
-        "ref": "release-23.05",
+        "ref": "nixpkgs-unstable",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-lib": {
+      "locked": {
+        "dir": "lib",
+        "lastModified": 1698611440,
+        "narHash": "sha256-jPjHjrerhYDy3q9+s5EAsuhyhuknNfowY6yt6pjn9pc=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "0cbe9f69c234a7700596e943bfae7ef27a31b735",
+        "type": "github"
+      },
+      "original": {
+        "dir": "lib",
+        "owner": "NixOS",
+        "ref": "nixos-unstable",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-next-unpatched": {
+      "locked": {
+        "lastModified": 1708992120,
+        "narHash": "sha256-t/8QV+lEroW5fK44w5oEUalIM0eYYVGs833AHDCIl4s=",
+        "owner": "nixos",
+        "repo": "nixpkgs",
+        "rev": "6daf4de0662e1d895d220a4a4ddb356eb000abe9",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nixos",
+        "ref": "staging-next",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "nixpkgs-stable": {
+      "locked": {
+        "lastModified": 1708819810,
+        "narHash": "sha256-1KosU+ZFXf31GPeCBNxobZWMgHsSOJcrSFA6F2jhzdE=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "89a2a12e6c8c6a56c72eb3589982c8e2f89c70ea",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "release-23.11",
         "repo": "nixpkgs",
         "type": "github"
       }
     },
     "nixpkgs-unpatched": {
       "locked": {
-        "lastModified": 1701180790,
+        "lastModified": 1708995544,
-        "narHash": "sha256-kYWcHsk2A1VUpiOvSo7Pq175WnSVeltspTGM2q+Cr3U=",
+        "narHash": "sha256-YJgLopKOKVTggnKzjX4OiAS22hx/vNv397DcsAyTZgY=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "c9702bf40b036c0f1d3d5b0aaf3eee2bf920124c",
+        "rev": "5bd8df40204f47a12263f3614c72cd5b6832a9a0",
         "type": "github"
       },
       "original": {
@@ -70,6 +140,8 @@
     "root": {
       "inputs": {
         "mobile-nixos": "mobile-nixos",
+        "nix-fast-build": "nix-fast-build",
+        "nixpkgs-next-unpatched": "nixpkgs-next-unpatched",
         "nixpkgs-unpatched": "nixpkgs-unpatched",
         "sops-nix": "sops-nix",
         "uninsane-dot-org": "uninsane-dot-org"
@@ -83,11 +155,11 @@
         "nixpkgs-stable": "nixpkgs-stable"
       },
       "locked": {
-        "lastModified": 1701127353,
+        "lastModified": 1708987867,
-        "narHash": "sha256-qVNX0wOl0b7+I35aRu78xUphOyELh+mtUp1KBx89K1Q=",
+        "narHash": "sha256-k2lDaDWNTU5sBVHanYzjDKVDmk29RHIgdbbXu5sdzBA=",
         "owner": "Mic92",
         "repo": "sops-nix",
-        "rev": "b1edbf5c0464b4cced90a3ba6f999e671f0af631",
+        "rev": "a1c8de14f60924fafe13aea66b46157f0150f4cf",
         "type": "github"
       },
       "original": {
@@ -96,34 +168,39 @@
         "type": "github"
       }
     },
-    "systems": {
+    "treefmt-nix": {
+      "inputs": {
+        "nixpkgs": [
+          "nix-fast-build",
+          "nixpkgs"
+        ]
+      },
       "locked": {
-        "lastModified": 1681028828,
+        "lastModified": 1698438538,
-        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "narHash": "sha256-AWxaKTDL3MtxaVTVU5lYBvSnlspOS0Fjt8GxBgnU0Do=",
-        "owner": "nix-systems",
+        "owner": "numtide",
-        "repo": "default",
+        "repo": "treefmt-nix",
-        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "rev": "5deb8dc125a9f83b65ca86cf0c8167c46593e0b1",
         "type": "github"
       },
       "original": {
-        "owner": "nix-systems",
+        "owner": "numtide",
-        "repo": "default",
+        "repo": "treefmt-nix",
         "type": "github"
       }
     },
     "uninsane-dot-org": {
       "inputs": {
-        "flake-utils": "flake-utils",
         "nixpkgs": [
           "nixpkgs-unpatched"
         ]
       },
       "locked": {
-        "lastModified": 1699515935,
+        "lastModified": 1707981105,
-        "narHash": "sha256-cJIuVrYorhIzG5pRFZb+ZtaKhTFD92ThC42SaxvSe/E=",
+        "narHash": "sha256-YCU1eNslBHabjP+OCY+BxPycEFO9SRUts10MrN9QORE=",
         "ref": "refs/heads/master",
-        "rev": "8a4273489d945f21d7e0ca6aac952460c7d4c391",
+        "rev": "bb10cd8853d05191e4d62947d93687c462e92c30",
-        "revCount": 216,
+        "revCount": 235,
         "type": "git",
         "url": "https://git.uninsane.org/colin/uninsane"
       },

flake.nix

@@ -29,7 +29,7 @@
     # - daily:
     #   - nixos-unstable cut from master after enough packages have been built in caches.
     # - every 6 hours:
-    #   - master auto-merged into staging.
+    #   - master auto-merged into staging and staging-next
     #   - staging-next auto-merged into staging.
     # - manually, approximately once per month:
     #   - staging-next is cut from staging.
@@ -44,8 +44,9 @@
     # <https://github.com/nixos/nixpkgs/tree/nixos-unstable>
     # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-unstable";
     nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=master";
-    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=staging-next";
+    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-staging";
-    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=staging";
+    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-staging-next";
+    nixpkgs-next-unpatched.url = "github:nixos/nixpkgs?ref=staging-next";
 
     mobile-nixos = {
       # <https://github.com/nixos/mobile-nixos>
@@ -56,6 +57,10 @@
       url = "github:nixos/mobile-nixos?ref=d25d3b87e7f300d8066e31d792337d9cd7ecd23b";
       flake = false;
     };
+    nix-fast-build = {
+      # https://github.com/Mic92/nix-fast-build
+      url = "github:Mic92/nix-fast-build";
+    };
     sops-nix = {
       # <https://github.com/Mic92/sops-nix>
       # used to distribute secrets to my hosts
@@ -74,7 +79,9 @@
   outputs = {
     self,
     nixpkgs-unpatched,
+    nixpkgs-next-unpatched ? nixpkgs-unpatched,
     mobile-nixos,
+    nix-fast-build,
     sops-nix,
     uninsane-dot-org,
     ...
@@ -92,39 +99,49 @@
       # rather than apply our nixpkgs patches as a flake input, do that here instead.
       # this (temporarily?) resolves the bad UX wherein a subflake residing in the same git
       # repo as the main flake causes the main flake to have an unstable hash.
-      nixpkgs = (import ./nixpatches/flake.nix).outputs {
+      patchNixpkgs = variant: nixpkgs: (import ./nixpatches/flake.nix).outputs {
-        self = nixpkgs;
+        inherit variant nixpkgs;
-        nixpkgs = nixpkgs-unpatched;
+        self = patchNixpkgs variant nixpkgs;
       } // {
-        # provide values that nixpkgs ordinarily sources from the flake.lock file,
+        # sourceInfo includes fields (square brackets for the ones which are not always present):
-        # inaccessible to it here because of the import-from-derivation.
+        # - [dirtyRev]
-        # rev and shortRev seem to not always exist (e.g. if the working tree is dirty),
+        # - [dirtyShortRev]
-        # so those are made conditional.
+        # - lastModified
+        # - lastModifiedDate
+        # - narHash
+        # - outPath
+        # - [rev]
+        # - [revCount]
+        # - [shortRev]
+        # - submodules
         #
-        # these values impact the name of a produced nixos system. having date/rev in the
+        # these values are used within nixpkgs:
-        # `readlink /run/current-system` store path helps debuggability.
+        # - to give a friendly name to the nixos system (`readlink /run/current-system` -> `...nixos-system-desko-24.05.20240227.dirty`)
-        inherit (self) lastModifiedDate lastModified;
+        # - to alias `import <nixpkgs>` so that nix uses the system's nixpkgs when called externally (supposedly).
-      } // optionalAttrs (self ? rev) {
+        #
-        inherit (self) rev;
+        # these values seem to exist both within the `sourceInfo` attrset and at the top-level.
-      } // optionalAttrs (self ? shortRev) {
+        # for a list of all implicit flake outputs (which is what these seem to be):
-        inherit (self) shortRev;
+        # $ nix-repl
-      };
+        # > lf .
+        # > <tab>
+        inherit (self) sourceInfo;
+      } // self.sourceInfo;
 
-      nixpkgsCompiledBy = system: nixpkgs.legacyPackages."${system}";
+      nixpkgs' = patchNixpkgs "master" nixpkgs-unpatched;
+      nixpkgsCompiledBy = system: nixpkgs'.legacyPackages."${system}";
 
-      evalHost = { name, local, target, light ? false }: nixpkgs.lib.nixosSystem {
+      evalHost = { name, local, target, light ? false, nixpkgs ? nixpkgs' }: nixpkgs.lib.nixosSystem {
         system = target;
         modules = [
           {
-            nixpkgs = (if (local != null) then {
+            nixpkgs.buildPlatform.system = local;
-              buildPlatform = local;
-            } else {}) // {
-              # TODO: does the earlier `system` arg to nixosSystem make its way here?
-              hostPlatform.system = target;
-            };
-            # nixpkgs.buildPlatform = local;  # set by instantiate.nix instead
             # nixpkgs.config.replaceStdenv = { pkgs }: pkgs.ccacheStdenv;
           }
+          (optionalAttrs (local != target) {
+            # XXX(2023/12/11): cache.nixos.org uses `system = ...` instead of `hostPlatform.system`, and that choice impacts the closure of every package.
+            # so avoid specifying hostPlatform.system on non-cross builds, so i can use upstream caches.
+            nixpkgs.hostPlatform.system = target;
+          })
           (optionalAttrs light {
             sane.enableSlowPrograms = false;
           })
@@ -140,39 +157,24 @@
         ];
       };
     in {
-      nixosConfigurations =
+      nixosConfigurations = let
-        let
+        hosts = {
-          hosts = {
+          servo       = { name = "servo";  local = "x86_64-linux"; target = "x86_64-linux";  };
-            servo       = { name = "servo";  local = "x86_64-linux"; target = "x86_64-linux";  };
+          desko       = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  };
-            desko       = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  };
+          desko-light = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  light = true; };
-            desko-light = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  light = true; };
+          lappy       = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  };
-            lappy       = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  };
+          lappy-light = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  light = true; };
-            lappy-light = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  light = true; };
+          moby        = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; };
-            moby        = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; };
+          moby-light  = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; light = true; };
-            moby-light  = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; light = true; };
+          rescue      = { name = "rescue"; local = "x86_64-linux"; target = "x86_64-linux";  };
-            rescue      = { name = "rescue"; local = "x86_64-linux"; target = "x86_64-linux";  };
+        };
-          };
+        hostsNext = mapAttrs' (h: v: {
-          # cross-compiled builds: instead of emulating the host, build using a cross-compiler.
+          name = "${h}-next";
-          # - these are faster to *build* than the emulated variants (useful when tweaking packages),
+          value = v // { nixpkgs = patchNixpkgs "staging-next" nixpkgs-next-unpatched; };
-          # - but fewer of their packages can be found in upstream caches.
+        }) hosts;
-          cross = mapAttrValues evalHost hosts;
+      in mapAttrValues evalHost (
-          emulated = mapAttrValues
+        hosts // hostsNext
-            (args: evalHost (args // { local = null; }))
+      );
-            hosts;
-          prefixAttrs = prefix: attrs: mapAttrs'
-            (name: value: {
-              name = prefix + name;
-              inherit value;
-            })
-            attrs;
-        in
-          (prefixAttrs "cross-" cross) //
-          (prefixAttrs "emulated-" emulated) // {
-            # prefer native builds for these machines:
-            inherit (emulated) servo desko desko-light lappy lappy-light rescue;
-            # prefer cross-compiled builds for these machines:
-            inherit (cross) moby moby-light;
-          };
 
       # unofficial output
       # this produces a EFI-bootable .img file (GPT with a /boot partition and a system (/ or /nix) partition).
@@ -192,25 +194,29 @@
 
       # unofficial output
       hostConfigs = mapAttrValues (host: host.config) self.nixosConfigurations;
+      hostSystems = mapAttrValues (host: host.config.system.build.toplevel) self.nixosConfigurations;
       hostPkgs = mapAttrValues (host: host.config.system.build.pkgs) self.nixosConfigurations;
       hostPrograms = mapAttrValues (host: mapAttrValues (p: p.package) host.config.sane.programs) self.nixosConfigurations;
 
+      patched.nixpkgs = nixpkgs';
+
       overlays = {
         # N.B.: `nix flake check` requires every overlay to take `final: prev:` at defn site,
         #   hence the weird redundancy.
         default = final: prev: self.overlays.pkgs final prev;
         sane-all = final: prev: import ./overlays/all.nix final prev;
-        disable-flakey-tests = final: prev: import ./overlays/disable-flakey-tests.nix final prev;
         pkgs = final: prev: import ./overlays/pkgs.nix final prev;
         pins = final: prev: import ./overlays/pins.nix final prev;
         preferences = final: prev: import ./overlays/preferences.nix final prev;
-        optimizations = final: prev: import ./overlays/optimizations.nix final prev;
         passthru = final: prev:
           let
             mobile = (import "${mobile-nixos}/overlay/overlay.nix");
-            uninsane = uninsane-dot-org.overlay;
+            uninsane = uninsane-dot-org.overlays.default;
+            # TODO: why do i have to use `self.inputs.nix-fast-build` instead of just `nix-fast-build` here?
+            nix-fast-build = (_: prev: self.inputs.nix-fast-build.packages."${prev.stdenv.system}" or {});
           in
             (mobile final prev)
+            // (nix-fast-build final prev)
             // (uninsane final prev)
           ;
       };
@@ -239,23 +245,27 @@
       # extract only our own packages from the full set.
       # because of `nix flake check`, we flatten the package set and only surface x86_64-linux packages.
       packages = mapAttrs
-        (system: allPkgs:
+        (system: passthruPkgs: passthruPkgs.lib.filterAttrs
-          allPkgs.lib.filterAttrs (name: pkg:
+          (name: pkg:
             # keep only packages which will pass `nix flake check`, i.e. keep only:
             # - derivations (not package sets)
             # - packages that build for the given platform
             (! elem name [ "feeds" "pythonPackagesExtensions" ])
-            && (allPkgs.lib.meta.availableOn allPkgs.stdenv.hostPlatform pkg)
+            && (passthruPkgs.lib.meta.availableOn passthruPkgs.stdenv.hostPlatform pkg)
           )
           (
             # expose sane packages and chosen inputs (uninsane.org)
-            (import ./pkgs { pkgs = allPkgs; }) // {
+            (import ./pkgs { pkgs = passthruPkgs; }) // {
-              inherit (allPkgs) uninsane-dot-org;
+              inherit (passthruPkgs) uninsane-dot-org;
             }
           )
         )
         # self.legacyPackages;
-        { inherit (self.legacyPackages) x86_64-linux; }
+        {
+          x86_64-linux = (nixpkgsCompiledBy "x86_64-linux").appendOverlays [
+            self.overlays.passthru
+          ];
+        }
       ;
 
       apps."x86_64-linux" =
@@ -263,13 +273,43 @@
           pkgs = self.legacyPackages."x86_64-linux";
           sanePkgs = import ./pkgs { inherit pkgs; };
           deployScript = host: addr: action: pkgs.writeShellScript "deploy-${host}" ''
-            nix build '.#nixosConfigurations.${host}.config.system.build.toplevel' --out-link ./result-${host} $@
+            host="${host}"
-            sudo nix sign-paths -r -k /run/secrets/nix_serve_privkey $(readlink ./result-${host})
+            addr="${addr}"
+            action="${if action != null then action else ""}"
+            runOnTarget() {
+              # run the command ($@) on the machine we're deploying to.
+              # if that's a remote machine, then do it via ssh, else local shell.
+              if [ -n "$addr" ]; then
+                ssh "$addr" "$@"
+              else
+                "$@"
+              fi
+            }
 
-            # XXX: this triggers another config eval & (potentially) build.
+            nix build ".#nixosConfigurations.$host.config.system.build.toplevel" --out-link "./result-$host" "$@"
-            # if the config changed between these invocations, the above signatures might not apply to the deployed config.
+            storePath="$(readlink ./result-$host)"
-            # let the user handle that edge case by re-running this whole command
+
-            nixos-rebuild --flake '.#${host}' ${action} --target-host colin@${addr} --use-remote-sudo $@
+            # mimic `nixos-rebuild --target-host`, in effect:
+            # - nix-copy-closure ...
+            # - nix-env --set ...
+            # - switch-to-configuration <boot|dry-activate|switch|test|>
+            # avoid the actual `nixos-rebuild` for a few reasons:
+            # - fewer nix evals
+            # - more introspectability and debuggability
+            # - sandbox friendliness (especially: `git` doesn't have to be run as root)
+
+            if [ -n "$addr" ]; then
+              sudo nix store sign -r -k /run/secrets/nix_serve_privkey "$storePath"
+              # add more `-v` for more verbosity (up to 5).
+              # builders-use-substitutes false: optimizes so that the remote machine doesn't try to get paths from its substituters.
+              #   we already have all paths here, and the remote substitution is slow to check and SERIOUSLY flaky on moby in particular.
+              nix copy -vv --option builders-use-substitutes false --to "ssh-ng://$addr" "$storePath"
+            fi
+
+            if [ -n "$action" ]; then
+              runOnTarget sudo nix-env -p /nix/var/nix/profiles/system --set "$storePath"
+              runOnTarget sudo "$storePath/bin/switch-to-configuration" "$action"
+            fi
           '';
           deployApp = host: addr: action: {
             type = "app";
@@ -343,7 +383,11 @@
                 - `nix run '.#update.feeds'`
                   - updates metadata for all feeds
                 - `nix run '.#init-feed' <url>`
-                - `nix run '.#deploy.{desko,lappy,moby,servo}[-light][.test]' [nixos-rebuild args ...]`
+                - `nix run '.#deploy.{desko,lappy,moby,servo}[-light|-test]' [nix args ...]`
+                  - build and deploy the host
+                - `nix run '.#preDeploy.{desko,lappy,moby,servo}[-light]' [nix args ...]`
+                  - copy closures to a host, but don't activate it
+                  - or `nix run '.#preDeploy'` to target all hosts
                 - `nix run '.#check'`
                   - make sure all systems build; NUR evaluates
 
@@ -372,31 +416,92 @@
           };
 
           deploy = {
+            desko       = deployApp "desko"       "desko" "switch";
+            desko-light = deployApp "desko-light" "desko" "switch";
             lappy       = deployApp "lappy"       "lappy" "switch";
             lappy-light = deployApp "lappy-light" "lappy" "switch";
             moby        = deployApp "moby"        "moby"  "switch";
             moby-light  = deployApp "moby-light"  "moby"  "switch";
             moby-test   = deployApp "moby"        "moby"  "test";
             servo       = deployApp "servo"       "servo" "switch";
-          };
 
-          sync-moby = {
+            # like `nixos-rebuild --flake . switch`
-            # copy music from the current device to moby
+            self        = deployApp "$(hostname)"       ""      "switch";
-            # TODO: should i actually sync from /mnt/servo-media/Music instead of the local drive?
+            self-light  = deployApp "$(hostname)-light" ""      "switch";
+
             type = "app";
-            program = builtins.toString (pkgs.writeShellScript "sync-to-moby" ''
+            program = builtins.toString (pkgs.writeShellScript "deploy-all" ''
-              sudo mount /mnt/moby-home
+              nix run '.#deploy.lappy'
-              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music ~/Music /mnt/moby-home/Music
+              nix run '.#deploy.moby'
+              nix run '.#deploy.desko'
+              nix run '.#deploy.servo'
+            '');
+          };
+          preDeploy = {
+            # build the host and copy the runtime closure to that host, but don't activate it.
+            desko       = deployApp "desko"       "desko" null;
+            desko-light = deployApp "desko-light" "desko" null;
+            lappy       = deployApp "lappy"       "lappy" null;
+            lappy-light = deployApp "lappy-light" "lappy" null;
+            moby        = deployApp "moby"        "moby"  null;
+            moby-light  = deployApp "moby-light"  "moby"  null;
+            servo       = deployApp "servo"       "servo" null;
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "predeploy-all" ''
+              # copy the -light variants first; this might be run while waiting on a full build. or the full build failed.
+              nix run '.#preDeploy.moby-light' -- "$@"
+              nix run '.#preDeploy.lappy-light' -- "$@"
+              nix run '.#preDeploy.desko-light' -- "$@"
+              nix run '.#preDeploy.lappy' -- "$@"
+              nix run '.#preDeploy.servo' -- "$@"
+              nix run '.#preDeploy.moby' -- "$@"
+              nix run '.#preDeploy.desko' -- "$@"
             '');
           };
 
-          sync-lappy = {
+          sync = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-all" ''
+              RC_lappy=$(nix run '.#sync.lappy' -- "$@")
+              RC_moby=$(nix run '.#sync.moby' -- "$@")
+              RC_desko=$(nix run '.#sync.desko' -- "$@")
+
+              echo "lappy: $RC_lappy"
+              echo "moby: $RC_moby"
+              echo "desko: $RC_desko"
+            '');
+          };
+
+          sync.desko = {
+            # copy music from servo to desko
+            # can run this from any device that has ssh access to desko and servo
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-to-desko" ''
+              sudo mount /mnt/desko/home
+              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music --compat /mnt/servo/media/Music /mnt/desko/home/Music "$@"
+            '');
+          };
+
+          sync.lappy = {
             # copy music from servo to lappy
-            # can run this from any device that has ssh access to lappy
+            # can run this from any device that has ssh access to lappy and servo
             type = "app";
             program = builtins.toString (pkgs.writeShellScript "sync-to-lappy" ''
-              sudo mount /mnt/lappy-home
+              sudo mount /mnt/lappy/home
-              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music /mnt/servo-media/Music /mnt/lappy-home/Music
+              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music --compress --compat /mnt/servo/media/Music /mnt/lappy/home/Music "$@"
+            '');
+          };
+
+          sync.moby = {
+            # copy music from servo to moby
+            # can run this from any device that has ssh access to moby and servo
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "sync-to-moby" ''
+              sudo mount /mnt/moby/home
+              sudo mount /mnt/desko/home
+              ${pkgs.rsync}/bin/rsync -arv --exclude servo-macros /mnt/moby/home/Pictures/ /mnt/desko/home/Pictures/moby/
+              # N.B.: limited by network/disk -> reduce job count to improve pause/resume behavior
+              ${pkgs.sane-scripts.sync-music}/bin/sane-sync-music --compress --compat --jobs 4 /mnt/servo/media/Music /mnt/moby/home/Music "$@"
             '');
           };
 
@@ -405,12 +510,12 @@
             program = builtins.toString (pkgs.writeShellScript "check-all" ''
               nix run '.#check.nur'
               RC0=$?
-              nix run '.#check.host-configs'
+              nix run '.#check.hostConfigs'
               RC1=$?
               nix run '.#check.rescue'
               RC2=$?
               echo "nur: $RC0"
-              echo "host-configs: $RC1"
+              echo "hostConfigs: $RC1"
               echo "rescue: $RC2"
               exit $(($RC0 | $RC1 | $RC2))
             '');
@@ -427,19 +532,19 @@
                 --option restrict-eval true \
                 --option allow-import-from-derivation true \
                 --drv-path --show-trace \
-                -I nixpkgs=$(nix-instantiate --find-file nixpkgs) \
+                -I nixpkgs=${nixpkgs-unpatched} \
                 -I ../../ \
                 | tee  # tee to prevent interactive mode
             '');
           };
 
-          check.host-configs = {
+          check.hostConfigs = {
             type = "app";
             program = let
               checkHost = host: let
                 shellHost = pkgs.lib.replaceStrings [ "-" ] [ "_" ] host;
               in ''
-                nix build -v '.#nixosConfigurations.${host}.config.system.build.toplevel' --out-link ./result-${host} -j2 $@
+                nix build -v '.#nixosConfigurations.${host}.config.system.build.toplevel' --out-link ./result-${host} -j2 "$@"
                 RC_${shellHost}=$?
               '';
             in builtins.toString (pkgs.writeShellScript
@@ -457,11 +562,29 @@
                 ${checkHost "moby"}
                 ${checkHost "rescue"}
 
+                # still want to build the -light variants first so as to avoid multiple simultaneous webkitgtk builds
+                ${checkHost "desko-light-next"}
+                ${checkHost "moby-light-next"}
+
+                ${checkHost "desko-next"}
+                ${checkHost "lappy-next"}
+                ${checkHost "servo-next"}
+                ${checkHost "moby-next"}
+                ${checkHost "rescue-next"}
+
                 echo "desko: $RC_desko"
                 echo "lappy: $RC_lappy"
                 echo "servo: $RC_servo"
                 echo "moby: $RC_moby"
                 echo "rescue: $RC_rescue"
+
+                echo "desko-next: $RC_desko_next"
+                echo "lappy-next: $RC_lappy_next"
+                echo "servo-next: $RC_servo_next"
+                echo "moby-next: $RC_moby_next"
+                echo "rescue-next: $RC_rescue_next"
+
+                # i don't really care if the -next hosts fail. i build them mostly to keep the cache fresh/ready
                 exit $(($RC_desko | $RC_lappy | $RC_servo | $RC_moby | $RC_rescue))
               ''
             );

hosts/by-name/desko/default.nix

@@ -9,6 +9,9 @@
   # services.distccd.enable = true;
   # sane.programs.distcc.enableFor.user.guest = true;
 
+  # TODO: remove emulation, but need to fix nixos-rebuild to moby for that.
+  # sane.roles.build-machine.emulation = true;
+
   sops.secrets.colin-passwd.neededForUsers = true;
 
   sane.ports.openFirewall = true;  # for e.g. nix-serve
@@ -25,14 +28,14 @@
   sane.nixcache.substituters.desko = false;
   sane.nixcache.remote-builders.desko = false;
 
-  sane.gui.sway.enable = true;
+  sane.programs.sway.enableFor.user.colin = true;
   sane.programs.iphoneUtils.enableFor.user.colin = true;
   sane.programs.steam.enableFor.user.colin = true;
 
   # sane.programs.devPkgs.enableFor.user.colin = true;
 
-  sane.programs.signal-desktop.config.autostart = true;
   sane.programs."gnome.geary".config.autostart = true;
+  sane.programs.signal-desktop.config.autostart = true;
 
   boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];

hosts/by-name/desko/fs.nix

@@ -6,7 +6,6 @@
   fileSystems."/tmp".options = [ "size=64G" ];
 
   fileSystems."/nix" = {
-    # device = "/dev/disk/by-uuid/0ab0770b-7734-4167-88d9-6e4e20bb2a56";
     device = "/dev/disk/by-uuid/845d85bf-761d-431b-a406-e6f20909154f";
     fsType = "btrfs";
     options = [
@@ -16,7 +15,6 @@
   };
 
   fileSystems."/boot" = {
-    # device = "/dev/disk/by-uuid/41B6-BAEF";
     device = "/dev/disk/by-uuid/5049-9AFD";
     fsType = "vfat";
   };

hosts/by-name/lappy/default.nix

@@ -12,10 +12,12 @@
   sane.services.wg-home.ip = config.sane.hosts.by-name."lappy".wg-home.ip;
 
   # sane.guest.enable = true;
-  sane.gui.sway.enable = true;
   boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
 
+  sane.programs.sway.enableFor.user.colin = true;
+  sane.programs."gnome.geary".config.autostart = true;
+  sane.programs.signal-desktop.config.autostart = true;
   sane.programs.stepmania.enableFor.user.colin = true;
 
   sops.secrets.colin-passwd.neededForUsers = true;

hosts/by-name/lappy/fs.nix

@@ -14,24 +14,4 @@
     device = "/dev/disk/by-uuid/BD79-D6BB";
     fsType = "vfat";
   };
-
-  # fileSystems."/nix" = {
-  #   device = "/dev/disk/by-uuid/5a7fa69c-9394-8144-a74c-6726048b129f";
-  #   fsType = "btrfs";
-  # };
-
-  # fileSystems."/boot" = {
-  #   device = "/dev/disk/by-uuid/4302-1685";
-  #   fsType = "vfat";
-  # };
-
-  # fileSystems."/" = {
-  #   device = "none";
-  #   fsType = "tmpfs";
-  #   options = [
-  #     "mode=755"
-  #     "size=1G"
-  #     "defaults"
-  #   ];
-  # };
 }

hosts/by-name/lappy/polyfill.nix

@@ -3,7 +3,6 @@
 { pkgs, ... }:
 {
   sane.gui.sxmo = {
-    greeter = "greetd-sway-gtkgreet";
     noidle = true;  #< power button requires 1s hold, which makes it impractical to be dealing with.
     settings = {
       # XXX: make sure the user is part of the `input` group!
@@ -22,7 +21,6 @@
       # the device type informs (at least):
       # - SXMO_WIFI_MODULE
       # - SXMO_RTW_SCAN_INTERVAL
-      # - SXMO_SYS_FILES
       # - SXMO_TOUCHSCREEN_ID
       # - SXMO_MONITOR
       # - SXMO_ALSA_CONTROL_NAME

hosts/by-name/moby/default.nix

@@ -25,10 +25,14 @@
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."moby".wg-home.ip;
 
+  # for some reason desko -> moby deploys are super flaky when desko is also a nixcache (not true of desko -> lappy deploys, though!)
+  # > unable to download 'http://desko:5001/<hash>.narinfo': Server returned nothing (no headers, no data) (52)
+  sane.nixcache.substituters.desko = false;
+
   # XXX colin: phosh doesn't work well with passwordless login,
   # so set this more reliable default password should anything go wrong
   users.users.colin.initialPassword = "147147";
-  services.getty.autologinUser = "root";  # allows for emergency maintenance?
+  # services.getty.autologinUser = "root";  # allows for emergency maintenance?
 
   sops.secrets.colin-passwd.neededForUsers = true;
 
@@ -36,7 +40,6 @@
   # sane.programs.consoleUtils.enableFor.user.colin = false;
   # sane.programs.guiApps.enableFor.user.colin = false;
   sane.programs.blueberry.enableFor.user.colin = false;  # bluetooth manager: doesn't cross compile!
-  sane.programs.dialect.enableFor.user.colin = false;  # drags in 700MB of x86 dependencies (e.g. gtk4)
   sane.programs.mercurial.enableFor.user.colin = false;  # does not cross compile
   sane.programs.nvme-cli.enableFor.system = false;  # does not cross compile (libhugetlbfs)
 
@@ -46,7 +49,7 @@
 
   # sane.programs.ntfy-sh.config.autostart = true;
   sane.programs.dino.config.autostart = true;
-  sane.programs.signal-desktop.config.autostart = true;
+  # sane.programs.signal-desktop.config.autostart = true;  # TODO: enable once electron stops derping.
   # sane.programs."gnome.geary".config.autostart = true;
   # sane.programs.calls.config.autostart = true;
   sane.programs.mpv.config.vo = "wlshm";  #< see hosts/common/programs/mpv.nix for details

hosts/by-name/moby/kernel.nix

@@ -74,6 +74,7 @@ in
     # without this some GUI apps fail: `DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory`
     # this is because they can't allocate enough video ram.
     # see related nixpkgs issue: <https://github.com/NixOS/nixpkgs/issues/260222>
+    # TODO(2023/12/03): remove once mesa 23.3.1 lands: <https://github.com/NixOS/nixpkgs/pull/265740>
     #
     # the default CMA seems to be 32M.
     # i was running fine with 256MB from 2022/07-ish through 2022/12-ish, but then the phone quit reliably coming back from sleep (phosh): maybe a memory leak?

hosts/by-name/servo/default.nix

@@ -15,9 +15,9 @@
   };
 
   sane.roles.build-machine.enable = true;
-  sane.roles.build-machine.emulation = false;
   sane.zsh.showDeadlines = false;  # ~/knowledge doesn't always exist
   sane.programs.consoleUtils.suggestedPrograms = [
+    "consoleMediaUtils"  # notably, for go2tv / casting
     "pcConsoleUtils"
     "sane-scripts.stop-all-servo"
   ];

hosts/by-name/servo/fs.nix

@@ -1,6 +1,58 @@
+# zfs docs:
+# - <https://nixos.wiki/wiki/ZFS>
+# - <repo:nixos/nixpkgs:nixos/modules/tasks/filesystems/zfs.nix>
+#
+# zfs check health: `zpool status`
+#
+# zfs pool creation (requires `boot.supportedFilesystems = [ "zfs" ];`
+# - 1. identify disk IDs: `ls -l /dev/disk/by-id`
+# - 2. pool these disks: `zpool create -f -m legacy pool raidz ata-ST4000VN008-2DR166_WDH0VB45 ata-ST4000VN008-2DR166_WDH17616 ata-ST4000VN008-2DR166_WDH0VC8Q ata-ST4000VN008-2DR166_WDH17680`
+#   - legacy documented: <https://superuser.com/questions/790036/what-is-a-zfs-legacy-mount-point>
+#
+# import pools: `zpool import pool`
+# show zfs datasets: `zfs list` (will be empty if haven't imported)
+# show zfs properties (e.g. compression): `zfs get all pool`
+# set zfs properties: `zfs set compression=on pool`
 { ... }:
 
 {
+  # hostId: not used for anything except zfs guardrail?
+  #   [hex(ord(x)) for x in 'serv']
+  networking.hostId = "73657276";
+  boot.supportedFilesystems = [ "zfs" ];
+  # boot.zfs.enabled = true;
+  boot.zfs.forceImportRoot = false;
+  # scrub all zfs pools weekly:
+  services.zfs.autoScrub.enable = true;
+  boot.extraModprobeConfig = ''
+    # ZFS likes to use half the ram for its own cache and let the kernel push everything else to swap.
+    # so, reduce its cache size
+    # see: <https://askubuntu.com/a/1290387>
+    # see: <https://serverfault.com/a/1119083>
+    # see: <https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-arc-max>
+    # for all tunables, see: `man 4 zfs`
+    # to update these parameters without rebooting:
+    # - `echo '4294967296' | sane-sudo-redirect /sys/module/zfs/parameters/zfs_arc_max`
+    options zfs zfs_arc_max=4294967296
+  '';
+  # to be able to mount the pool like this, make sure to tell zfs to NOT manage it itself.
+  # otherwise local-fs.target will FAIL and you will be dropped into a rescue shell.
+  # - `zfs set mountpoint=legacy pool`
+  # if done correctly, the pool can be mounted before this `fileSystems` entry is created:
+  # - `sudo mount -t zfs pool /mnt/persist/pool`
+  fileSystems."/mnt/pool" = {
+    device = "pool";
+    fsType = "zfs";
+  };
+  # services.zfs.zed = ... # TODO: zfs can send me emails when disks fail
+  sane.programs.sysadminUtils.suggestedPrograms = [ "zfs" ];
+
+  sane.persist.stores."ext" = {
+    origin = "/mnt/pool/persist";
+    storeDescription = "external HDD storage";
+    defaultMethod = "bind";  #< TODO: change to "symlink"?
+  };
+
   # increase /tmp space (defaults to 50% of RAM) for building large nix things.
   # even the stock `nixpkgs.linux` consumes > 16 GB of tmp
   fileSystems."/tmp".options = [ "size=32G" ];
@@ -20,7 +72,7 @@
   };
 
   # slow, external storage (for archiving, etc)
-  fileSystems."/mnt/persist/ext" = {
+  fileSystems."/mnt/usb-hdd" = {
     device = "/dev/disk/by-uuid/aa272cff-0fcc-498e-a4cb-0d95fb60631b";
     fsType = "btrfs";
     options = [
@@ -28,22 +80,18 @@
       "defaults"
     ];
   };
-
+  sane.fs."/mnt/usb-hdd".mount = {};
-  sane.persist.stores."ext" = {
-    origin = "/mnt/persist/ext/persist";
-    storeDescription = "external HDD storage";
-  };
-  sane.fs."/mnt/persist/ext".mount = {};
 
   sane.persist.sys.byStore.plaintext = [
     # TODO: this is overly broad; only need media and share directories to be persisted
-    { user = "colin"; group = "users"; path = "/var/lib/uninsane"; }
+    { user = "colin"; group = "users"; path = "/var/lib/uninsane"; method = "bind"; }
   ];
   # force some problematic directories to always get correct permissions:
   sane.fs."/var/lib/uninsane/media".dir.acl = {
     user = "colin"; group = "media"; mode = "0775";
   };
   sane.fs."/var/lib/uninsane/media/archive".dir = {};
+  # this is file.text instead of symlink.text so that it may be read over a remote mount (where consumers might not have any /nix/store/.../README.md path)
   sane.fs."/var/lib/uninsane/media/archive/README.md".file.text = ''
     this directory is for media i wish to remove from my library,
     but keep for a short time in case i reverse my decision.
@@ -62,6 +110,7 @@
   sane.fs."/var/lib/uninsane/media/Videos/Film".dir = {};
   sane.fs."/var/lib/uninsane/media/Videos/Shows".dir = {};
   sane.fs."/var/lib/uninsane/media/Videos/Talks".dir = {};
+  # this is file.text instead of symlink.text so that it may be read over a remote mount (where consumers might not have any /nix/store/.../README.md path)
   sane.fs."/var/lib/uninsane/datasets/README.md".file.text = ''
     this directory may seem redundant with ../media/datasets. it isn't.
     this directory exists on SSD, allowing for speedy access to specific datasets when necessary.

hosts/by-name/servo/net.nix

@@ -24,61 +24,12 @@ in
     sane.ports.openFirewall = true;
     sane.ports.openUpnp = true;
 
-    # view refused packets with: `sudo journalctl -k`
-    # networking.firewall.logRefusedPackets = true;
-
-    # The global useDHCP flag is deprecated, therefore explicitly set to false here.
-    # Per-interface useDHCP will be mandatory in the future, so this generated config
-    # replicates the default behaviour.
-    networking.useDHCP = false;
-    networking.interfaces.eth0.useDHCP = true;
-    # XXX colin: probably don't need this. wlan0 won't be populated unless i touch a value in networking.interfaces.wlan0
-    networking.wireless.enable = false;
-
-    # this is needed to forward packets from the VPN to the host
-    boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
-
     # unless we add interface-specific settings for each VPN, we have to define nameservers globally.
     # networking.nameservers = [
     #   "1.1.1.1"
     #   "9.9.9.9"
     # ];
 
-    # use systemd's stub resolver.
-    # /etc/resolv.conf isn't sophisticated enough to use different servers per net namespace (or link).
-    # instead, running the stub resolver on a known address in the root ns lets us rewrite packets
-    # in the ovnps namespace to use the provider's DNS resolvers.
-    # a weakness is we can only query 1 NS at a time (unless we were to clone the packets?)
-    # there also seems to be some cache somewhere that's shared between the two namespaces.
-    # i think this is a libc thing. might need to leverage proper cgroups to _really_ kill it.
-    # - getent ahostsv4 www.google.com
-    # - try fix: <https://serverfault.com/questions/765989/connect-to-3rd-party-vpn-server-but-dont-use-it-as-the-default-route/766290#766290>
-    services.resolved.enable = true;
-    # without DNSSEC:
-    # - dig matrix.org => works
-    # - curl https://matrix.org => works
-    # with default DNSSEC:
-    # - dig matrix.org => works
-    # - curl https://matrix.org => fails
-    # i don't know why. this might somehow be interfering with the DNS run on this device (trust-dns)
-    services.resolved.dnssec = "false";
-    networking.nameservers = [
-      # use systemd-resolved resolver
-      # full resolver (which understands /etc/hosts) lives on 127.0.0.53
-      # stub resolver (just forwards upstream) lives on 127.0.0.54
-      "127.0.0.53"
-    ];
-
-    # nscd -- the Name Service Caching Daemon -- caches DNS query responses
-    # in a way that's unaware of my VPN routing, so routes are frequently poor against
-    # services which advertise different IPs based on geolocation.
-    # nscd claims to be usable without a cache, but in practice i can't get it to not cache!
-    # nsncd is the Name Service NON-Caching Daemon. it's a drop-in that doesn't cache;
-    # this is OK on the host -- because systemd-resolved caches. it's probably sub-optimal
-    # in the netns and we query upstream DNS more often than needed. hm.
-    # TODO: run a separate recursive resolver in each namespace.
-    services.nscd.enableNsncd = true;
-
     # services.resolved.extraConfig = ''
     #   # docs: `man resolved.conf`
     #   # DNS servers to use via the `wg-ovpns` interface.

hosts/by-name/servo/services/calibre.nix

@@ -13,7 +13,7 @@ in
 lib.mkIf false
 {
   sane.persist.sys.byStore.plaintext = [
-    { inherit user group; mode = "0700"; path = svc-dir; }
+    { inherit user group; mode = "0700"; path = svc-dir; method = "bind"; }
   ];
 
   services.calibre-web.enable = true;

hosts/by-name/servo/services/cryptocurrencies/bitcoin.nix

@@ -0,0 +1,83 @@
+# as of 2023/12/02: complete blockchain is 530 GiB (on-disk size may be larger)
+#
+# ports:
+# - 8333: for node-to-node communications
+# - 8332: rpc (client-to-node)
+#
+# rpc setup:
+# - generate a password
+#   - use: <https://github.com/bitcoin/bitcoin/blob/master/share/rpcauth/rpcauth.py>
+#     (rpcauth.py is not included in the `'.#bitcoin'` package result)
+#   - `wget https://raw.githubusercontent.com/bitcoin/bitcoin/master/share/rpcauth/rpcauth.py`
+#   - `python ./rpcauth.py colin`
+#   - copy the hash here. it's SHA-256, so safe to be public.
+#   - add "rpcuser=colin" and "rpcpassword=<output>" to secrets/servo/bitcoin.conf  (i.e. ~/.bitcoin/bitcoin.conf)
+#     - bitcoin.conf docs: <https://github.com/bitcoin/bitcoin/blob/master/doc/bitcoin-conf.md>
+# - validate with `bitcoin-cli -netinfo`
+{ config, lib, pkgs, sane-lib, ... }:
+let
+  # wrapper to run bitcoind with the tor onion address as externalip (computed at runtime)
+  _bitcoindWithExternalIp = with pkgs; writeShellScriptBin "bitcoind" ''
+    externalip="$(cat /var/lib/tor/onion/bitcoind/hostname)"
+    exec ${bitcoind}/bin/bitcoind "-externalip=$externalip" "$@"
+  '';
+  # the package i provide to services.bitcoind ends up on system PATH, and used by other tools like clightning.
+  # therefore, even though services.bitcoind only needs `bitcoind` binary, provide all the other bitcoin-related binaries (notably `bitcoin-cli`) as well:
+  bitcoindWithExternalIp = with pkgs; symlinkJoin {
+    name = "bitcoind-with-external-ip";
+    paths = [ _bitcoindWithExternalIp bitcoind ];
+  };
+in
+{
+  sane.persist.sys.byStore.ext = [
+    { user = "bitcoind-mainnet"; group = "bitcoind-mainnet"; path = "/var/lib/bitcoind-mainnet"; method = "bind"; }
+  ];
+
+  # sane.ports.ports."8333" = {
+  #   # this allows other nodes and clients to download blocks from me.
+  #   protocol = [ "tcp" ];
+  #   visibleTo.wan = true;
+  #   description = "colin-bitcoin";
+  # };
+
+  services.tor.relay.onionServices.bitcoind = {
+    version = 3;
+    map = [{
+      # by default tor will route public tor port P to 127.0.0.1:P.
+      # so if this port is the same as clightning would natively use, then no further config is needed here.
+      # see: <https://2019.www.torproject.org/docs/tor-manual.html.en#HiddenServicePort>
+      port = 8333;
+      # target.port; target.addr;  #< set if tor port != clightning port
+    }];
+    # allow "tor" group (i.e. bitcoind-mainnet) to read /var/lib/tor/onion/bitcoind/hostname
+    settings.HiddenServiceDirGroupReadable = true;
+  };
+
+  services.bitcoind.mainnet = {
+    enable = true;
+    package = bitcoindWithExternalIp;
+    rpc.users.colin = {
+      # see docs at top of file for how to generate this
+      passwordHMAC = "30002c05d82daa210550e17a182db3f3$6071444151281e1aa8a2729f75e3e2d224e9d7cac3974810dab60e7c28ffaae4";
+    };
+    extraConfig = ''
+      # don't load the wallet, and disable wallet RPC calls
+      disablewallet=1
+      # proxy all outbound traffic through Tor
+      proxy=127.0.0.1:9050
+    '';
+  };
+
+  users.users.bitcoind-mainnet.extraGroups = [ "tor" ];
+
+  systemd.services.bitcoind-mainnet.serviceConfig.RestartSec = "30s";  #< default is 0
+
+  sane.users.colin.fs.".bitcoin/bitcoin.conf" = sane-lib.fs.wantedSymlinkTo config.sops.secrets."bitcoin.conf".path;
+  sops.secrets."bitcoin.conf" = {
+    mode = "0600";
+    owner = "colin";
+    group = "users";
+  };
+
+  sane.programs.bitcoind.enableFor.user.colin = true;  # for debugging/administration: `bitcoin-cli`
+}

hosts/by-name/servo/services/cryptocurrencies/clightning-sane/clightning-sane

@@ -0,0 +1,766 @@
+#!/usr/bin/env nix-shell
+#!nix-shell -i python3 -p "python3.withPackages (ps: [ ps.pyln-client ])"
+
+# pyln-client docs: <https://github.com/ElementsProject/lightning/tree/master/contrib/pyln-client>
+# terminology:
+# - "scid": "Short Channel ID", e.g. 123456x7890x0
+#   from this id, we can locate the actual channel, its peers, and its parameters
+
+import argparse
+import logging
+import math
+import sys
+import time
+
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import dataclass
+from enum import Enum
+
+from pyln.client import LightningRpc, Millisatoshi, RpcError
+
+logger = logging.getLogger(__name__)
+
+RPC_FILE = "/var/lib/clightning/bitcoin/lightning-rpc"
+# CLTV (HLTC delta) of the final hop
+# set this too low and you might get inadvertent channel closures (?)
+CLTV = 18
+
+# for every sequentally failed transaction, delay this much before trying again.
+# note that the initial route building process can involve 10-20 "transient" failures, as it discovers dead channels.
+TX_FAIL_BACKOFF = 0.8
+MAX_SEQUENTIAL_JOB_FAILURES = 200
+
+class LoopError(Enum):
+    """ error when trying to loop sats, or when unable to calculate a route for the loop """
+    TRANSIENT = "TRANSIENT"  # try again, we'll maybe find a different route
+    NO_ROUTE = "NO_ROUTE"
+
+class RouteError(Enum):
+    """ error when calculated a route """
+    HAS_BASE_FEE = "HAS_BASE_FEE"
+    NO_ROUTE = "NO_ROUTE"
+
+class Metrics:
+    looped_msat: int = 0
+    sendpay_fail: int = 0
+    sendpay_succeed: int = 0
+    own_bad_channel: int = 0
+    no_route: int = 0
+    in_ch_unsatisfiable: int = 0
+
+    def __repr__(self) -> str:
+        return f"looped:{self.looped_msat}, tx:{self.sendpay_succeed}, tx_fail:{self.sendpay_fail}, own_bad_ch:{self.own_bad_channel}, no_route:{self.no_route}, in_ch_restricted:{self.in_ch_unsatisfiable}"
+
+@dataclass
+class TxBounds:
+    max_msat: int
+    min_msat: int = 0
+
+    def __repr__(self) -> str:
+        return f"TxBounds({self.min_msat} <= msat <= {self.max_msat})"
+
+    def is_satisfiable(self) -> bool:
+        return self.min_msat <= self.max_msat
+
+    def raise_max_to_be_satisfiable(self) -> "Self":
+        if self.max_msat < self.min_msat:
+            logger.debug(f"raising max_msat to be consistent: {self.max_msat} -> {self.min_msat}")
+            return TxBounds(self.min_msat, self.min_msat)
+        return TxBounds(min_msat=self.min_msat, max_msat=self.max_msat)
+
+    def intersect(self, other: "TxBounds") -> "Self":
+        return TxBounds(
+            min_msat=max(self.min_msat, other.min_msat),
+            max_msat=min(self.max_msat, other.max_msat),
+        )
+
+    def restrict_to_htlc(self, ch: "LocalChannel", why: str = "") -> "Self":
+        """
+        apply min/max HTLC size restrictions of the given channel.
+        """
+        if ch:
+            why = why or ch.directed_scid_to_me
+        if why: why = f"{why}: "
+
+        new_min, new_max = self.min_msat, self.max_msat
+        if ch.htlc_minimum_to_me > self.min_msat:
+            new_min = ch.htlc_minimum_to_me
+            logger.debug(f"{why}raising min_msat due to HTLC requirements: {self.min_msat} -> {new_min}")
+        if ch.htlc_maximum_to_me < self.max_msat:
+            new_max = ch.htlc_maximum_to_me
+            logger.debug(f"{why}lowering max_msat due to HTLC requirements: {self.max_msat} -> {new_max}")
+        return TxBounds(min_msat=new_min, max_msat=new_max)
+
+    def restrict_to_zero_fees(self, ch: "LocalChannel"=None, base: int=0, ppm: int=0, why:str = "") -> "Self":
+        """
+        restrict tx size such that PPM fees are zero.
+        if the channel has a base fee, then `max_msat` is forced to 0.
+        """
+        if ch:
+            why = why or ch.directed_scid_to_me
+            self = self.restrict_to_zero_fees(base=ch.to_me["base_fee_millisatoshi"], ppm=ch.to_me["fee_per_millionth"], why=why)
+
+        if why: why = f"{why}: "
+
+        new_max = self.max_msat
+        ppm_max = math.ceil(1000000 / ppm) - 1 if ppm != 0 else new_max
+        if ppm_max < new_max:
+            logger.debug(f"{why}decreasing max_msat due to fee ppm: {new_max} -> {ppm_max}")
+            new_max = ppm_max
+
+        if base != 0:
+            logger.debug(f"{why}free route impossible: channel has base fees")
+            new_max = 0
+
+        return TxBounds(min_msat=self.min_msat, max_msat=new_max)
+
+
+class LocalChannel:
+    def __init__(self, channels: list, rpc: "RpcHelper"):
+        assert 0 < len(channels) <= 2, f"unexpected: channel count: {channels}"
+        out = None
+        in_ = None
+        for c in channels:
+            if c["source"] == rpc.self_id:
+                assert out is None, f"unexpected: multiple channels from self: {channels}"
+                out = c
+            if c["destination"] == rpc.self_id:
+                assert in_ is None, f"unexpected: multiple channels to self: {channels}"
+                in_ = c
+
+        # assert out is not None, f"no channel from self: {channels}"
+        # assert in_ is not None, f"no channel to self: {channels}"
+
+        if out and in_:
+            assert out["destination"] == in_["source"], f"channel peers are asymmetric?! {channels}"
+            assert out["short_channel_id"] == in_["short_channel_id"], f"channel ids differ?! {channels}"
+
+        self.from_me = out
+        self.to_me = in_
+        self.remote_node = rpc.node(self.remote_peer)
+        self.peer_ch = rpc.peerchannel(self.scid, self.remote_peer)
+        self.forwards_from_me = rpc.rpc.listforwards(out_channel=self.scid, status="settled")["forwards"]
+
+    def __repr__(self) -> str:
+        return self.to_str(with_scid=True, with_bal_ratio=True, with_cost=False, with_ppm_theirs=False)
+
+    def to_str(
+        self,
+        with_peer_id:bool = False,
+        with_scid:bool = False,
+        with_bal_msat:bool = False,
+        with_bal_ratio:bool = False,
+        with_cost:bool = False,
+        with_ppm_theirs:bool = False,
+        with_ppm_mine:bool = False,
+        with_profits:bool = True,
+        with_payments:bool = False,
+    ) -> str:
+        base_flag = "*" if not self.online or self.base_fee_to_me != 0 else ""
+        alias = f"({self.remote_alias}){base_flag}"
+        peerid = f" {self.remote_peer}" if with_peer_id else ""
+        scid = f"  scid:{self.scid:>13}" if with_scid else ""
+        bal = f"  S:{int(self.sendable):11}/R:{int(self.receivable):11}" if with_bal_msat else ""
+        ratio = f"  MINE:{(100*self.send_ratio):>8.4f}%" if with_bal_ratio else ""
+        payments = f"  OUT:{int(self.out_fulfilled_msat):>11}/IN:{int(self.in_fulfilled_msat):>11}" if with_payments else ""
+        profits = f"  P$:{int(self.fees_lifetime_mine):>8}" if with_profits else ""
+        cost = f"  COST:{self.opportunity_cost_lent:>8}" if with_cost else ""
+        ppm_theirs = self.ppm_to_me if self.to_me else "N/A"
+        ppm_theirs = f"  PPM_THEIRS:{ppm_theirs:>6}" if with_ppm_theirs else ""
+        ppm_mine = self.ppm_from_me if self.from_me else "N/A"
+        ppm_mine = f"  PPM_MINE:{ppm_mine:>6}" if with_ppm_mine else ""
+        return f"channel{alias:30}{peerid}{scid}{bal}{ratio}{payments}{profits}{cost}{ppm_theirs}{ppm_mine}"
+
+
+    @property
+    def online(self) -> bool:
+        return self.from_me and self.to_me
+
+    @property
+    def remote_peer(self) -> str:
+        if self.from_me:
+            return self.from_me["destination"]
+        else:
+            return self.to_me["source"]
+
+    @property
+    def remote_alias(self) -> str:
+        return self.remote_node["alias"]
+
+    @property
+    def scid(self) -> str:
+        if self.from_me:
+            return self.from_me["short_channel_id"]
+        else:
+            return self.to_me["short_channel_id"]
+
+    @property
+    def htlc_minimum_to_me(self) -> Millisatoshi:
+        return self.to_me["htlc_minimum_msat"]
+
+    @property
+    def htlc_minimum_from_me(self) -> Millisatoshi:
+        return self.from_me["htlc_minimum_msat"]
+
+    @property
+    def htlc_minimum(self) -> Millisatoshi:
+        return max(self.htlc_minimum_to_me, self.htlc_minimum_from_me)
+
+    @property
+    def htlc_maximum_to_me(self) -> Millisatoshi:
+        return self.to_me["htlc_maximum_msat"]
+
+    @property
+    def htlc_maximum_from_me(self) -> Millisatoshi:
+        return self.from_me["htlc_maximum_msat"]
+
+    @property
+    def htlc_maximum(self) -> Millisatoshi:
+        return min(self.htlc_maximum_to_me, self.htlc_maximum_from_me)
+
+    @property
+    def direction_to_me(self) -> int:
+        return self.to_me["direction"]
+
+    @property
+    def direction_from_me(self) -> int:
+        return self.from_me["direction"]
+
+    @property
+    def directed_scid_to_me(self) -> str:
+        return f"{self.scid}/{self.direction_to_me}"
+
+    @property
+    def directed_scid_from_me(self) -> str:
+        return f"{self.scid}/{self.direction_from_me}"
+
+    @property
+    def delay_them(self) -> str:
+        return self.to_me["delay"]
+
+    @property
+    def delay_me(self) -> str:
+        return self.from_me["delay"]
+
+    @property
+    def ppm_to_me(self) -> int:
+        return self.to_me["fee_per_millionth"]
+
+    @property
+    def ppm_from_me(self) -> int:
+        return self.from_me["fee_per_millionth"]
+        # return self.peer_ch["fee_proportional_millionths"]
+
+    @property
+    def base_fee_to_me(self) -> int:
+        return self.to_me["base_fee_millisatoshi"]
+
+    @property
+    def receivable(self) -> int:
+        return self.peer_ch["receivable_msat"]
+
+    @property
+    def sendable(self) -> int:
+        return self.peer_ch["spendable_msat"]
+
+    @property
+    def in_fulfilled_msat(self) -> Millisatoshi:
+        return self.peer_ch["in_fulfilled_msat"]
+
+    @property
+    def out_fulfilled_msat(self) -> Millisatoshi:
+        return self.peer_ch["out_fulfilled_msat"]
+
+    @property
+    def fees_lifetime_mine(self) -> Millisatoshi:
+        return sum(fwd["fee_msat"] for fwd in self.forwards_from_me)
+
+    @property
+    def send_ratio(self) -> float:
+        cap = self.receivable + self.sendable
+        return self.sendable / cap
+
+    @property
+    def opportunity_cost_lent(self) -> int:
+        """ how much msat did we gain by pushing their channel to its current balance? """
+        return int(self.receivable * self.ppm_from_me / 1000000)
+
+class RpcHelper:
+    def __init__(self, rpc: LightningRpc):
+        self.rpc = rpc
+        self.self_id = rpc.getinfo()["id"]
+
+    def localchannel(self, scid: str) -> LocalChannel:
+        listchan = self.rpc.listchannels(scid)
+        # this assertion would probably indicate a typo in the scid
+        assert listchan and listchan.get("channels", []) != [], f"bad listchannels for {scid}: {listchan}"
+        return LocalChannel(listchan["channels"], self)
+
+    def node(self, id: str) -> dict:
+        nodes = self.rpc.listnodes(id)["nodes"]
+        assert len(nodes) == 1, f"unexpected: multiple nodes for {id}: {nodes}"
+        return nodes[0]
+
+    def peerchannel(self, scid: str, peer_id: str) -> dict:
+        peerchannels = self.rpc.listpeerchannels(peer_id)["channels"]
+        channels = [c for c in peerchannels if c["short_channel_id"] == scid]
+        assert len(channels) == 1, f"expected exactly 1 channel, got: {channels}"
+        return channels[0]
+
+    def try_getroute(self, *args, **kwargs) -> dict | None:
+        """ wrapper for getroute which returns None instead of error if no route exists """
+        try:
+            route = self.rpc.getroute(*args, **kwargs)
+        except RpcError as e:
+            logger.debug(f"rpc failed: {e}")
+            return None
+        else:
+            route = route["route"]
+            if route == []: return None
+            return route
+
+class LoopRouter:
+    def __init__(self, rpc: RpcHelper, metrics: Metrics = None):
+        self.rpc = rpc
+        self.metrics = metrics or Metrics()
+        self.bad_channels = []  # list of directed scid
+        self.nonzero_base_channels = []  # list of directed scid
+
+    def drop_caches(self) -> None:
+        logger.info("LoopRouter.drop_caches()")
+        self.bad_channels = []
+
+    def _get_directed_scid(self, scid: str, direction: int) -> dict:
+        channels = self.rpc.rpc.listchannels(scid)["channels"]
+        channels = [c for c in channels if c["direction"] == direction]
+        assert len(channels) == 1, f"expected exactly 1 channel: {channels}"
+        return channels[0]
+
+    def loop_once(self, out_scid: str, in_scid: str, bounds: TxBounds) -> LoopError|int:
+        out_ch = self.rpc.localchannel(out_scid)
+        in_ch = self.rpc.localchannel(in_scid)
+
+        if out_ch.directed_scid_from_me in self.bad_channels or in_ch.directed_scid_to_me in self.bad_channels:
+            logger.info(f"loop {out_scid} -> {in_scid} failed in our own channel")
+            self.metrics.own_bad_channel += 1
+            return LoopError.TRANSIENT
+
+        # bounds = bounds.restrict_to_htlc(out_ch)  # htlc bounds seem to be enforced only in the outward direction
+        bounds = bounds.restrict_to_htlc(in_ch)
+        bounds = bounds.restrict_to_zero_fees(in_ch)
+        if not bounds.is_satisfiable():
+            self.metrics.in_ch_unsatisfiable += 1
+            return LoopError.NO_ROUTE
+
+        logger.debug(f"route with bounds {bounds}")
+        route = self.route(out_ch, in_ch, bounds)
+        logger.debug(f"route: {route}")
+        if route == RouteError.NO_ROUTE:
+            self.metrics.no_route += 1
+            return LoopError.NO_ROUTE
+        elif route == RouteError.HAS_BASE_FEE:
+            # try again with a different route
+            return LoopError.TRANSIENT
+
+        amount_msat = route[0]["amount_msat"]
+        invoice_id = f"loop-{time.time():.6f}".replace(".", "_")
+        invoice_desc = f"bal {out_scid}:{in_scid}"
+        invoice = self.rpc.rpc.invoice("any", invoice_id, invoice_desc)
+        logger.debug(f"invoice: {invoice}")
+
+        payment = self.rpc.rpc.sendpay(route, invoice["payment_hash"], invoice_id, amount_msat, invoice["bolt11"], invoice["payment_secret"])
+        logger.debug(f"sent: {payment}")
+
+        try:
+            wait = self.rpc.rpc.waitsendpay(invoice["payment_hash"])
+            logger.debug(f"result: {wait}")
+        except RpcError as e:
+            self.metrics.sendpay_fail += 1
+            err_data = e.error["data"]
+            err_scid, err_dir = err_data["erring_channel"], err_data["erring_direction"]
+            err_directed_scid = f"{err_scid}/{err_dir}"
+            logger.debug(f"ch failed, adding to excludes: {err_directed_scid}; {e.error}")
+            self.bad_channels.append(err_directed_scid)
+            return LoopError.TRANSIENT
+        else:
+            self.metrics.sendpay_succeed += 1
+            self.metrics.looped_msat += int(amount_msat)
+            return int(amount_msat)
+
+    def route(self, out_ch: LocalChannel, in_ch: LocalChannel, bounds: TxBounds) -> list[dict] | RouteError:
+        exclude = [
+            # ensure the payment doesn't cross either channel in reverse.
+            # note that this doesn't preclude it from taking additional trips through self, with other peers.
+            # out_ch.directed_scid_to_me,
+            # in_ch.directed_scid_from_me,
+
+            # alternatively, never route through self. this avoids a class of logic error, like what to do with fees i charge "myself".
+            self.rpc.self_id
+        ] + self.bad_channels + self.nonzero_base_channels
+
+        out_peer = out_ch.remote_peer
+        in_peer = in_ch.remote_peer
+
+        route_or_bounds = bounds
+        while isinstance(route_or_bounds, TxBounds):
+            old_bounds = route_or_bounds
+            route_or_bounds = self._find_partial_route(out_peer, in_peer, old_bounds, exclude=exclude)
+            if route_or_bounds == old_bounds:
+                return RouteError.NO_ROUTE
+
+        if isinstance(route_or_bounds, RouteError):
+            return route_or_bounds
+
+        route = self._add_route_endpoints(route_or_bounds, out_ch, in_ch)
+        return route
+
+    def _find_partial_route(self, out_peer: str, in_peer: str, bounds: TxBounds, exclude: list[str]=[]) -> list[dict] | RouteError | TxBounds:
+        route = self.rpc.try_getroute(in_peer, amount_msat=bounds.max_msat, riskfactor=0, fromid=out_peer, exclude=exclude, cltv=CLTV)
+        if route is None:
+            logger.debug(f"no route for {bounds.max_msat}msat {out_peer} -> {in_peer}")
+            return RouteError.NO_ROUTE
+
+        send_msat = route[0]["amount_msat"]
+        if send_msat != Millisatoshi(bounds.max_msat):
+            logger.debug(f"found route with non-zero fee: {send_msat} -> {bounds.max_msat}. {route}")
+
+            error = None
+            for hop in route:
+                hop_scid = hop["channel"]
+                hop_dir = hop["direction"]
+                directed_scid = f"{hop_scid}/{hop_dir}"
+                ch = self._get_directed_scid(hop_scid, hop_dir)
+                if ch["base_fee_millisatoshi"] != 0:
+                    self.nonzero_base_channels.append(directed_scid)
+                    error = RouteError.HAS_BASE_FEE
+                bounds = bounds.restrict_to_zero_fees(ppm=ch["fee_per_millionth"], why=directed_scid)
+
+            return bounds.raise_max_to_be_satisfiable() if error is None else error
+
+        return route
+
+    def _add_route_endpoints(self, route, out_ch: LocalChannel, in_ch: LocalChannel):
+        inbound_hop = dict(
+          id=self.rpc.self_id,
+          channel=in_ch.scid,
+          direction=in_ch.direction_to_me,
+          amount_msat=route[-1]["amount_msat"],
+          delay=route[-1]["delay"],
+          style="tlv",
+        )
+        route = self._add_route_delay(route, in_ch.delay_them) + [ inbound_hop ]
+
+        outbound_hop = dict(
+            id=out_ch.remote_peer,
+            channel=out_ch.scid,
+            direction=out_ch.direction_from_me,
+            amount_msat=route[0]["amount_msat"],
+            delay=route[0]["delay"] + out_ch.delay_them,
+            style="tlv",
+        )
+        route = [ outbound_hop ] + route
+        return route
+
+    def _add_route_delay(self, route: list[dict], delay: int) -> list[dict]:
+        return [ dict(hop, delay=hop["delay"] + delay) for hop in route ]
+
+@dataclass
+class LoopJob:
+    out: str  # scid
+    in_: str  # scid
+    amount: int
+
+@dataclass
+class LoopJobIdle:
+    sec: int = 10
+
+class LoopJobDone(Enum):
+    COMPLETED = "COMPLETED"
+    ABORTED = "ABORTED"
+
+class AbstractLoopRunner:
+    def __init__(self, looper: LoopRouter, bounds: TxBounds, parallelism: int):
+        self.looper = looper
+        self.bounds = bounds
+        self.parallelism = parallelism
+        self.bounds_map = {}  # map (out:str, in_:str) -> TxBounds. it's a cache so we don't have to try 10 routes every time.
+
+    def pop_job(self) -> LoopJob | LoopJobIdle | LoopJobDone:
+        raise NotImplemented  # abstract method
+
+    def finished_job(self, job: LoopJob, progress: int|LoopError) -> None:
+        raise NotImplemented  # abstract method
+
+    def run_to_completion(self, exit_on_any_completed:bool = False) -> None:
+        self.exiting = False
+        self.exit_on_any_completed = exit_on_any_completed
+        if self.parallelism == 1:
+            # run inline to aid debugging
+            self._worker_thread()
+        else:
+            with ThreadPoolExecutor(max_workers=self.parallelism) as executor:
+                _ = list(executor.map(lambda _i: self._try_invoke(self._worker_thread), range(self.parallelism)))
+
+    def drop_caches(self) -> None:
+        logger.info("AbstractLoopRunner.drop_caches()")
+        self.looper.drop_caches()
+        self.bounds_map = {}
+
+
+    def _try_invoke(self, f, *args) -> None:
+        """
+        try to invoke `f` with the provided `args`, and log if it fails.
+        this overcomes the issue that background tasks which fail via Exception otherwise do so silently.
+        """
+        try:
+            f(*args)
+        except Exception as e:
+            logger.error(f"task failed: {e}")
+
+
+    def _worker_thread(self) -> None:
+        while not self.exiting:
+            job = self.pop_job()
+            logger.debug(f"popped job: {job}")
+            if isinstance(job, LoopJobDone):
+                return self._worker_finished(job)
+
+            if isinstance(job, LoopJobIdle):
+                logger.debug(f"idling for {job.sec}")
+                time.sleep(job.sec)
+                continue
+
+            result = self._execute_job(job)
+            logger.debug(f"finishing job {job} with {result}")
+            self.finished_job(job, result)
+
+    def _execute_job(self, job: LoopJob) -> LoopError|int:
+        bounds = self.bounds_map.get((job.out, job.in_), self.bounds)
+        bounds = bounds.intersect(TxBounds(max_msat=job.amount))
+        if not bounds.is_satisfiable():
+            logger.debug(f"TxBounds for job are unsatisfiable; skipping: {bounds} {job}")
+            return LoopError.NO_ROUTE
+
+        amt_looped = self.looper.loop_once(job.out, job.in_, bounds)
+        if amt_looped in (0, LoopError.NO_ROUTE, LoopError.TRANSIENT):
+            return amt_looped
+
+        logger.info(f"looped {amt_looped} from {job.out} -> {job.in_}")
+        bounds = bounds.intersect(TxBounds(max_msat=amt_looped))
+
+        self.bounds_map[(job.out, job.in_)] = bounds
+        return amt_looped
+
+    def _worker_finished(self, job: LoopJobDone) -> None:
+        if job == LoopJobDone.COMPLETED and self.exit_on_any_completed:
+            logger.debug(f"worker completed -> exiting pool")
+            self.exiting = True
+
+class LoopPairState:
+    # TODO: use this in MultiLoopBalancer, or stop shoving state in here and put it on LoopBalancer instead.
+    def __init__(self, out: str, in_: str, amount: int):
+        self.out = out
+        self.in_ = in_
+        self.amount_target = amount
+        self.amount_looped = 0
+        self.amount_outstanding = 0
+        self.tx_fail_count = 0
+        self.route_fail_count = 0
+        self.last_job_start_time = None
+        self.failed_tx_throttler = 0  # increase by one every time we fail, decreases more gradually, when we succeed
+
+class LoopBalancer(AbstractLoopRunner):
+    def __init__(self, out: str, in_: str, amount: int, looper: LoopRouter, bounds: TxBounds, parallelism: int=1):
+        super().__init__(looper, bounds, parallelism)
+        self.state = LoopPairState(out, in_, amount)
+
+    def pop_job(self) -> LoopJob | LoopJobIdle | LoopJobDone:
+        if self.state.tx_fail_count + 10*self.state.route_fail_count >= MAX_SEQUENTIAL_JOB_FAILURES:
+            logger.info(f"giving up ({self.state.out} -> {self.state.in_}): {self.state.tx_fail_count} tx failures, {self.state.route_fail_count} route failures")
+            return LoopJobDone.ABORTED
+
+        if self.state.tx_fail_count + self.state.route_fail_count > 0:
+            # N.B.: last_job_start_time is guaranteed to have been set by now
+            idle_until = self.state.last_job_start_time + TX_FAIL_BACKOFF*self.state.failed_tx_throttler
+            idle_for = idle_until - time.time()
+            if self.state.amount_outstanding != 0 or idle_for > 0:
+                # when we hit transient failures, restrict to just one job in flight at a time.
+                # this is aimed for the initial route building, where multiple jobs in flight is just useless,
+                # but it's not a bad idea for network blips, etc, either.
+                logger.info(f"throttling ({self.state.out} -> {self.state.in_}) for {idle_for:.0f}: {self.state.tx_fail_count} tx failures, {self.state.route_fail_count} route failures")
+                return LoopJobIdle(idle_for) if idle_for > 0 else LoopJobIdle()
+
+        amount_avail = self.state.amount_target - self.state.amount_looped - self.state.amount_outstanding
+        if amount_avail < self.bounds.min_msat:
+            if self.state.amount_outstanding == 0: return LoopJobDone.COMPLETED
+            return LoopJobIdle()  # sending out another job would risk over-transferring
+        amount_this_job = min(amount_avail, self.bounds.max_msat)
+
+        self.state.amount_outstanding += amount_this_job
+        self.state.last_job_start_time = time.time()
+        return LoopJob(out=self.state.out, in_=self.state.in_, amount=amount_this_job)
+
+    def finished_job(self, job: LoopJob, progress: int) -> None:
+        self.state.amount_outstanding -= job.amount
+        if progress == LoopError.NO_ROUTE:
+            self.state.route_fail_count += 1
+            self.state.failed_tx_throttler += 10
+        elif progress == LoopError.TRANSIENT:
+            self.state.tx_fail_count += 1
+            self.state.failed_tx_throttler += 1
+        else:
+            self.state.amount_looped += progress
+            self.state.tx_fail_count = 0
+            self.state.route_fail_count = 0
+            self.state.failed_tx_throttler = max(0, self.state.failed_tx_throttler - 0.2)
+        logger.info(f"loop progressed ({job.out} -> {job.in_}) {progress}: {self.state.amount_looped} of {self.state.amount_target}")
+
+class MultiLoopBalancer(AbstractLoopRunner):
+    """
+    multiplexes jobs between multiple LoopBalancers.
+    note that the child LoopBalancers don't actually execute the jobs -- just produce them.
+    """
+    def __init__(self, looper: LoopRouter, bounds: TxBounds, parallelism: int=1):
+        super().__init__(looper, bounds, parallelism)
+        self.loops = []
+        # job_index: increments on every job so we can grab jobs evenly from each LoopBalancer.
+        # in that event that producers are idling, it can actually increment more than once,
+        # so don't take this too literally
+        self.job_index = 0
+
+    def add_loop(self, out: LocalChannel, in_: LocalChannel, amount: int) -> None:
+        """
+        start looping sats from out -> in_
+        """
+        assert not any(l.state.out == out.scid and l.state.in_ == in_.scid for l in self.loops), f"tried to add duplicate loops from {out} -> {in_}"
+        logger.info(f"looping from ({out}) to ({in_})")
+        self.loops.append(LoopBalancer(out.scid, in_.scid, amount, self.looper, self.bounds, self.parallelism))
+
+    def pop_job(self) -> LoopJob | LoopJobIdle | LoopJobDone:
+        # N.B.: this can be called in parallel, so try to be consistent enough to not crash
+
+        idle_job = None
+        abort_job = None
+        for i, _ in enumerate(self.loops):
+            loop = self.loops[(self.job_index + i) % len(self.loops)]
+            self.job_index += 1
+            job = loop.pop_job()
+            if isinstance(job, LoopJob):
+                return job
+            if isinstance(job, LoopJobIdle):
+                idle_job = LoopJobIdle(min(job.sec, idle_job.sec)) if idle_job is not None else job
+            if job == LoopJobDone.ABORTED:
+                abort_job = job
+
+        # either there's a task to idle, or we have to terminate.
+        # if terminating, terminate ABORTED if any job aborted, else COMPLETED
+        if idle_job is not None: return idle_job
+        if abort_job is not None: return abort_job
+        return LoopJobDone.COMPLETED
+
+    def finished_job(self, job: LoopJob, progress: int) -> None:
+        # this assumes (enforced externally) that we have only one loop for a given out/in_ pair
+        for l in self.loops:
+            if l.state.out == job.out and l.state.in_ == job.in_:
+                l.finished_job(job, progress)
+
+        logger.info(f"total: {self.looper.metrics}")
+
+
+def balance_loop(rpc: RpcHelper, out: str, in_: str, amount_msat: int, min_msat: int, max_msat: int, parallelism: int):
+    looper = LoopRouter(rpc)
+    bounds = TxBounds(min_msat=min_msat, max_msat=max_msat)
+    balancer = LoopBalancer(out, in_, amount_msat, looper, bounds, parallelism)
+
+    balancer.run_to_completion()
+
+def autobalance_once(rpc: RpcHelper, metrics: Metrics, bounds: TxBounds, parallelism: int) -> bool:
+    """
+    autobalances all channels.
+    returns True if channels are balanced (or as balanced as can be); False if in need of further balancing
+    """
+    looper = LoopRouter(rpc, metrics)
+    balancer = MultiLoopBalancer(looper, bounds, parallelism)
+
+    channels = []
+    for peerch in rpc.rpc.listpeerchannels()["channels"]:
+        try:
+            channels.append(rpc.localchannel(peerch["short_channel_id"]))
+        except:
+            logger.info(f"NO CHANNELS for {peerch['peer_id']}")
+
+    channels = [ch for ch in channels if ch.online and ch.base_fee_to_me == 0]
+    give_to = [ ch for ch in channels if ch.send_ratio > 0.95 ]
+    take_from = [ ch for ch in channels if ch.send_ratio < 0.20 ]
+
+    if give_to == [] and take_from == []:
+        return True
+
+    for to in give_to:
+        for from_ in take_from:
+            balancer.add_loop(to, from_, 10000000)
+
+    balancer.run_to_completion(exit_on_any_completed=True)
+    return False
+
+
+def autobalance(rpc: RpcHelper, min_msat: int, max_msat: int, parallelism: int):
+    bounds = TxBounds(min_msat=min_msat, max_msat=max_msat)
+    metrics = Metrics()
+    while not autobalance_once(rpc, metrics, bounds, parallelism):
+        pass
+
+def show_status(rpc: RpcHelper, full: bool=False):
+    """
+    show a table of channel balances between peers.
+    """
+    for peerch in rpc.rpc.listpeerchannels()["channels"]:
+        try:
+            ch = rpc.localchannel(peerch["short_channel_id"])
+        except:
+            print(f"{peerch['peer_id']} scid:{peerch['short_channel_id']} state:{peerch['state']} NO CHANNELS")
+        else:
+            print(ch.to_str(with_scid=True, with_bal_ratio=True, with_payments=True, with_cost=full, with_ppm_theirs=True, with_ppm_mine=True, with_peer_id=full))
+
+def main():
+    logging.basicConfig()
+    logger.setLevel(logging.INFO)
+
+    parser = argparse.ArgumentParser(description="rebalance lightning channel balances")
+    parser.add_argument("--verbose", action="store_true", help="more logging")
+    parser.add_argument("--min-msat", default="999", help="min transaction size")
+    parser.add_argument("--max-msat", default="1000000", help="max transaction size")
+    parser.add_argument("--jobs", default="1", help="how many HTLCs to keep in-flight at once")
+    subparsers = parser.add_subparsers(help="action")
+
+    status_parser = subparsers.add_parser("status")
+    status_parser.set_defaults(action="status")
+    status_parser.add_argument("--full", action="store_true", help="more info per channel")
+
+    loop_parser = subparsers.add_parser("loop")
+    loop_parser.set_defaults(action="loop")
+    loop_parser.add_argument("out", help="peer id to send tx through")
+    loop_parser.add_argument("in_", help="peer id to receive tx through")
+    loop_parser.add_argument("amount", help="total amount of msat to loop")
+
+    autobal_parser = subparsers.add_parser("autobalance")
+    autobal_parser.set_defaults(action="autobalance")
+
+    args = parser.parse_args()
+
+    if args.verbose:
+        logger.setLevel(logging.DEBUG)
+
+    rpc = RpcHelper(LightningRpc(RPC_FILE))
+
+    if args.action == "status":
+        show_status(rpc, full=args.full)
+
+    if args.action == "loop":
+        balance_loop(rpc, out=args.out, in_=args.in_, amount_msat=int(args.amount), min_msat=int(args.min_msat), max_msat=int(args.max_msat), parallelism=int(args.jobs))
+
+    if args.action == "autobalance":
+        autobalance(rpc, min_msat=int(args.min_msat), max_msat=int(args.max_msat), parallelism=int(args.jobs))
+
+if __name__ == '__main__':
+    main()

hosts/by-name/servo/services/cryptocurrencies/clightning.nix

@@ -0,0 +1,135 @@
+# clightning is an implementation of Bitcoin's Lightning Network.
+# as such, this assumes that `services.bitcoin` is enabled.
+# docs:
+# - tor clightning config: <https://docs.corelightning.org/docs/tor>
+# - `lightning-cli` and subcommands: <https://docs.corelightning.org/reference/lightning-cli>
+# - `man lightningd-config`
+#
+# management/setup/use:
+# - guide: <https://github.com/ElementsProject/lightning>
+#
+# debugging:
+# - `lightning-cli getlog debug`
+# - `lightning-cli listpays`  -> show payments this node sent
+# - `lightning-cli listinvoices`  -> show payments this node received
+#
+# first, acquire peers:
+# - `lightning-cli connect id@host`
+#   where `id` is the node's pubkey, and `host` is perhaps an ip:port tuple, or a hash.onion:port tuple.
+#   for testing, choose any node listed on <https://1ml.com>
+# - `lightning-cli listpeers`
+#   should show the new peer, with `connected: true`
+#
+# then, fund the clightning wallet
+# - `lightning-cli newaddr`
+#
+# then, open channels
+# - `lightning-cli connect ...`
+# - `lightning-cli fundchannel <node_id> <amount_in_satoshis>`
+#
+# who to federate with?
+# - a lot of the larger nodes allow hands-free channel creation
+#   - either inbound or outbound, sometimes paid
+# - find nodes on:
+#   - <https://terminal.lightning.engineering/>
+#   - <https://1ml.com>
+#     - tor nodes: <https://1ml.com/node?order=capacity&iponionservice=true>
+#   - <https://lightningnetwork.plus>
+#   - <https://mempool.space/lightning>
+#   - <https://amboss.space>
+# - a few tor-capable nodes which allow channel creation:
+#   - <https://c-otto.de/>
+#   - <https://cyberdyne.sh/>
+#   - <https://yalls.org/about/>
+#   - <https://coincept.com/>
+# - more resources: <https://www.lopp.net/lightning-information.html>
+#   - node routability: https://hashxp.org/lightning/node/<id>
+# - especially, acquire inbound liquidity via lightningnetwork.plus's swap feature
+#   - most of the opportunities are gated behind a minimum connection or capacity requirement
+#
+# tune payment parameters
+# - `lightning-cli setchannel <id> [feebase] [feeppm] [htlcmin] [htlcmax] [enforcedelay] [ignorefeelimits]`
+#   - e.g. `lightning-cli setchannel all 0 10`
+#   - it's suggested that feebase=0 simplifies routing.
+#
+# teardown:
+# - `lightning-cli withdraw <bc1... dest addr> <amount in satoshis> [feerate]`
+#
+# sanity:
+# - `lightning-cli listfunds`
+#
+# to receive a payment (do as `clightning` user):
+# - `lightning-cli invoice <amount in millisatoshi> <label> <description>`
+#   - specify amount as `any` if undetermined
+#   - then give the resulting bolt11 URI to the payer
+# to send a payment:
+# - `lightning-cli pay <bolt11 URI>`
+#   - or `lightning-cli pay <bolt11 URI> [amount_msat] [label] [riskfactor] [maxfeepercent] ...`
+#   - amount_msat must be "null" if the bolt11 URI specifies a value
+#   - riskfactor defaults to 10
+#   - maxfeepercent defaults to 0.5
+#   - label is a human-friendly label for my records
+
+{ config, pkgs, ... }:
+{
+  sane.persist.sys.byStore.ext = [
+    { user = "clightning"; group = "clightning"; mode = "0710"; path = "/var/lib/clightning"; method = "bind"; }
+  ];
+
+  # `lightning-cli` finds its RPC file via `~/.lightning/bitcoin/lightning-rpc`, to message the daemon
+  sane.user.fs.".lightning".symlink.target = "/var/lib/clightning";
+
+  # see bitcoin.nix for how to generate this
+  services.bitcoind.mainnet.rpc.users.clightning.passwordHMAC =
+    "befcb82d9821049164db5217beb85439$2c31ac7db3124612e43893ae13b9527dbe464ab2d992e814602e7cb07dc28985";
+
+  sane.services.clightning.enable = true;
+  sane.services.clightning.proxy = "127.0.0.1:9050";  # proxy outgoing traffic through tor
+  # sane.services.clightning.publicAddress = "statictor:127.0.0.1:9051";
+  sane.services.clightning.getPublicAddressCmd = "cat /var/lib/tor/onion/clightning/hostname";
+
+  services.tor.relay.onionServices.clightning = {
+    version = 3;
+    map = [{
+      # by default tor will route public tor port P to 127.0.0.1:P.
+      # so if this port is the same as clightning would natively use, then no further config is needed here.
+      # see: <https://2019.www.torproject.org/docs/tor-manual.html.en#HiddenServicePort>
+      port = 9735;
+      # target.port; target.addr;  #< set if tor port != clightning port
+    }];
+    # allow "tor" group (i.e. clightning) to read /var/lib/tor/onion/clightning/hostname
+    settings.HiddenServiceDirGroupReadable = true;
+  };
+
+  # must be in "tor" group to read /var/lib/tor/onion/*/hostname
+  users.users.clightning.extraGroups = [ "tor" ];
+
+  systemd.services.clightning.after = [ "tor.service" ];
+
+  # lightning-config contains fields from here:
+  # - <https://docs.corelightning.org/docs/configuration>
+  # secret config includes:
+  # - bitcoin-rpcpassword
+  # - alias=nodename
+  # - rgb=rrggbb
+  # - fee-base=<millisatoshi>
+  # - fee-per-satoshi=<ppm>
+  # - feature configs (i.e. experimental-xyz options)
+  sane.services.clightning.extraConfig = ''
+    log-level=debug:lightningd
+    # peerswap:
+    # - config example: <https://github.com/fort-nix/nix-bitcoin/pull/462/files#diff-b357d832705b8ce8df1f41934d613f79adb77c4cd5cd9e9eb12a163fca3e16c6>
+    # XXX: peerswap crashes clightning on launch. stacktrace is useless.
+    # plugin=${pkgs.peerswap}/bin/peerswap
+    # peerswap-db-path=/var/lib/clightning/peerswap/swaps
+    # peerswap-policy-path=...
+  '';
+  sane.services.clightning.extraConfigFiles = [ config.sops.secrets."lightning-config".path ];
+  sops.secrets."lightning-config" = {
+    mode = "0640";
+    owner = "clightning";
+    group = "clightning";
+  };
+
+  sane.programs.clightning.enableFor.user.colin = true;  # for debugging/admin: `lightning-cli`
+}

hosts/by-name/servo/services/cryptocurrencies/default.nix

@@ -0,0 +1,10 @@
+{ ... }:
+{
+  imports = [
+    ./bitcoin.nix
+    ./clightning.nix
+    ./i2p.nix
+    ./monero.nix
+    ./tor.nix
+  ];
+}

hosts/by-name/servo/services/cryptocurrencies/i2p.nix

@@ -0,0 +1,4 @@
+{ ... }:
+{
+  services.i2p.enable = true;
+}

hosts/by-name/servo/services/cryptocurrencies/monero.nix

@@ -3,7 +3,7 @@
 {
   sane.persist.sys.byStore.ext = [
     # /var/lib/monero/lmdb is what consumes most of the space
-    { user = "monero"; group = "monero"; path = "/var/lib/monero"; }
+    { user = "monero"; group = "monero"; path = "/var/lib/monero"; method = "bind"; }
   ];
 
   services.monero.enable = true;
@@ -20,12 +20,6 @@
     tx-proxy=tor,127.0.0.1:9050
   '';
 
-  services.i2p.enable = true;
-  # tor: `tor.enable` doesn't start a relay, exit node, proxy, etc. it's minimal.
-  # tor.client.enable configures a torsocks proxy, accessible *only* to localhost.
-  services.tor.enable = true;
-  services.tor.client.enable = true;
-
   # monero ports: <https://monero.stackexchange.com/questions/604/what-ports-does-monero-use-rpc-p2p-etc>
   # - 18080 = "P2P" monero node <-> monero node connections
   # - 18081 = "RPC" monero client -> monero node connections

hosts/by-name/servo/services/cryptocurrencies/tor.nix

@@ -0,0 +1,25 @@
+# tor settings: <https://2019.www.torproject.org/docs/tor-manual.html.en>
+{ lib, ... }:
+{
+  # tor hidden service hostnames aren't deterministic, so persist.
+  # might be able to get away with just persisting /var/lib/tor/onion, not sure.
+  sane.persist.sys.byStore.plaintext = [
+    { user = "tor"; group = "tor"; mode = "0710"; path = "/var/lib/tor"; method = "bind"; }
+  ];
+
+  # tor: `tor.enable` doesn't start a relay, exit node, proxy, etc. it's minimal.
+  # tor.client.enable configures a torsocks proxy, accessible *only* to localhost.
+  #   at 127.0.0.1:9050
+  services.tor.enable = true;
+  services.tor.client.enable = true;
+
+  # in order for services to read /var/lib/tor/onion/*/hostname, they must be able to traverse /var/lib/tor,
+  # and /var/lib/tor must have g+x.
+  # DataDirectoryGroupReadable causes tor to use g+rx, technically more than we need, but all the files are 600 so it's fine.
+  services.tor.settings.DataDirectoryGroupReadable = true;
+  # StateDirectoryMode defaults to 0700, and thereby prevents the onion hostnames from being group readable
+  systemd.services.tor.serviceConfig.StateDirectoryMode = lib.mkForce "0710";
+  users.users.tor.homeMode = "0710";  # home mode defaults to 0700, causing readability problems, enforced by nixos "users" activation script
+
+  services.tor.settings.SafeLogging = false;  # show actual .onion names in the syslog, else debugging is impossible
+}

hosts/by-name/servo/services/ddns-afraid.nix

@@ -1,27 +0,0 @@
-{ config, lib, pkgs, ... }:
-
-# using manual ddns now
-lib.mkIf false
-{
-  systemd.services.ddns-afraid = {
-    description = "update dynamic DNS entries for freedns.afraid.org";
-    serviceConfig = {
-      EnvironmentFile = config.sops.secrets."ddns_afraid.env".path;
-      # TODO: ProtectSystem = "strict";
-      # TODO: ProtectHome = "full";
-      # TODO: PrivateTmp = true;
-    };
-    script = let
-      curl = "${pkgs.curl}/bin/curl -4";
-    in ''
-      ${curl} "https://freedns.afraid.org/dynamic/update.php?$AFRAID_KEY"
-    '';
-  };
-  systemd.timers.ddns-afraid = {
-    wantedBy = [ "multi-user.target" ];
-    timerConfig = {
-      OnStartupSec = "2min";
-      OnUnitActiveSec = "10min";
-    };
-  };
-}

hosts/by-name/servo/services/ddns-he.nix

@@ -1,30 +0,0 @@
-{ config, lib, pkgs, ... }:
-
-# we use manual DDNS now
-lib.mkIf false
-{
-  systemd.services.ddns-he = {
-    description = "update dynamic DNS entries for HurricaneElectric";
-    serviceConfig = {
-      EnvironmentFile = config.sops.secrets."ddns_he.env".path;
-      # TODO: ProtectSystem = "strict";
-      # TODO: ProtectHome = "full";
-      # TODO: PrivateTmp = true;
-    };
-    # HE DDNS API is documented: https://dns.he.net/docs.html
-    script = let
-      crl = "${pkgs.curl}/bin/curl -4";
-    in ''
-      ${crl} "https://he.uninsane.org:$HE_PASSPHRASE@dyn.dns.he.net/nic/update?hostname=he.uninsane.org"
-      ${crl} "https://native.uninsane.org:$HE_PASSPHRASE@dyn.dns.he.net/nic/update?hostname=native.uninsane.org"
-      ${crl} "https://uninsane.org:$HE_PASSPHRASE@dyn.dns.he.net/nic/update?hostname=uninsane.org"
-    '';
-  };
-  systemd.timers.ddns-he = {
-    wantedBy = [ "multi-user.target" ];
-    timerConfig = {
-      OnStartupSec = "2min";
-      OnUnitActiveSec = "10min";
-    };
-  };
-}

hosts/by-name/servo/services/default.nix

@@ -3,8 +3,7 @@
   imports = [
     ./calibre.nix
     ./coturn.nix
-    ./ddns-afraid.nix
+    ./cryptocurrencies
-    ./ddns-he.nix
     ./email
     ./ejabberd.nix
     ./freshrss.nix
@@ -18,9 +17,9 @@
     ./komga.nix
     ./lemmy.nix
     ./matrix
-    ./monero.nix
     ./navidrome.nix
     ./nginx.nix
+    ./nixos-prebuild.nix
     ./nixserve.nix
     ./ntfy
     ./pict-rs.nix

hosts/by-name/servo/services/ejabberd.nix

@@ -45,7 +45,7 @@ in
 lib.mkIf false
 {
   sane.persist.sys.byStore.plaintext = [
-    { user = "ejabberd"; group = "ejabberd"; path = "/var/lib/ejabberd"; }
+    { user = "ejabberd"; group = "ejabberd"; path = "/var/lib/ejabberd"; method = "bind"; }
   ];
   sane.ports.ports = lib.mkMerge ([
     {

hosts/by-name/servo/services/email/dovecot.nix

@@ -127,10 +127,11 @@
   services.dovecot2.modules = [
     pkgs.dovecot_pigeonhole  # enables sieve execution (?)
   ];
-  services.dovecot2.sieveScripts = {
+  services.dovecot2.sieve = {
+    extensions = [ "fileinto" ];
     # if any messages fail to pass (or lack) DKIM, move them to Junk
     # XXX the key name ("after") is only used to order sieve execution/ordering
-    after = builtins.toFile "ensuredkim.sieve" ''
+    scripts.after = builtins.toFile "ensuredkim.sieve" ''
       require "fileinto";
 
       if not header :contains "Authentication-Results" "dkim=pass" {
@@ -139,4 +140,6 @@
       }
     '';
   };
+
+  systemd.services.dovecot2.serviceConfig.RestartSec = lib.mkForce "15s";  # nixos defaults this to 1s
 }

hosts/by-name/servo/services/email/postfix.nix

@@ -20,9 +20,9 @@ in
 {
   sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
-    { user = "opendkim"; group = "opendkim"; path = "/var/lib/opendkim"; }
+    { user = "opendkim"; group = "opendkim"; path = "/var/lib/opendkim"; method = "bind"; }
-    { user = "root"; group = "root"; path = "/var/lib/postfix"; }
+    { user = "root"; group = "root"; path = "/var/lib/postfix"; method = "bind"; }
-    { user = "root"; group = "root"; path = "/var/spool/mail"; }
+    { user = "root"; group = "root"; path = "/var/spool/mail"; method = "bind"; }
     # *probably* don't need these dirs:
     # "/var/lib/dhparams"          # https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/security/dhparams.nix
     # "/var/lib/dovecot"

hosts/by-name/servo/services/export/default.nix

@@ -30,7 +30,7 @@
   # to query the quota/status:
   # - `sudo btrfs qgroup show -re /var/export/playground`
   sane.persist.sys.byStore.ext = [
-    { user = "root"; group = "export"; mode = "0775"; path = "/var/export/playground"; }
+    { user = "root"; group = "export"; mode = "0775"; path = "/var/export/playground"; method = "bind"; }
   ];
 
   sane.fs."/var/export/README.md" = {

hosts/by-name/servo/services/export/sftpgo.nix

@@ -91,7 +91,7 @@ let
   authFailJson = pkgs.writeText "sftp-auth-fail.json" (builtins.toJSON authResponseFail);
   unwrappedAuthProgram = pkgs.static-nix-shell.mkBash {
     pname = "sftpgo_external_auth_hook";
-    src = ./.;
+    srcRoot = ./.;
     pkgs = [ "coreutils" ];
   };
   authProgram = pkgs.writeShellScript "sftpgo-auth-hook" ''
@@ -172,13 +172,15 @@ in
 
   users.users.sftpgo.extraGroups = [ "export" ];
 
-  systemd.services.sftpgo.serviceConfig = {
+  systemd.services.sftpgo = {
-    ReadOnlyPaths = [ "/var/export" ];
-    ReadWritePaths = [ "/var/export/playground" ];
     after = [ "network-online.target" ];
     wants = [ "network-online.target" ];
+    serviceConfig = {
+      ReadOnlyPaths = [ "/var/export" ];
+      ReadWritePaths = [ "/var/export/playground" ];
 
-    Restart = "always";
+      Restart = "always";
-    RestartSec = "20s";
+      RestartSec = "20s";
+    };
   };
 }

hosts/by-name/servo/services/freshrss.nix

@@ -16,7 +16,7 @@
     mode = "0400";
   };
   sane.persist.sys.byStore.plaintext = [
-    { user = "freshrss"; group = "freshrss"; path = "/var/lib/freshrss"; }
+    { user = "freshrss"; group = "freshrss"; path = "/var/lib/freshrss"; method = "bind"; }
   ];
 
   services.freshrss.enable = true;

hosts/by-name/servo/services/gitea.nix

@@ -4,7 +4,7 @@
 {
   sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
-    { user = "git"; group = "gitea"; path = "/var/lib/gitea"; }
+    { user = "git"; group = "gitea"; path = "/var/lib/gitea"; method = "bind"; }
   ];
   services.gitea.enable = true;
   services.gitea.user = "git";  # default is 'gitea'
@@ -100,6 +100,24 @@
     locations."/" = {
       proxyPass = "http://127.0.0.1:3000";
     };
+    # gitea serves all `raw` files as content-type: plain, but i'd like to serve them as their actual content type.
+    # or at least, enough to make specific pages viewable (serving unoriginal content as arbitrary content type is dangerous).
+    locations."~ ^/colin/phone-case-cq/raw/.*.html" = {
+      proxyPass = "http://127.0.0.1:3000";
+      extraConfig = ''
+        proxy_hide_header Content-Type;
+        default_type text/html;
+        add_header Content-Type text/html;
+      '';
+    };
+    locations."~ ^/colin/phone-case-cq/raw/.*.js" = {
+      proxyPass = "http://127.0.0.1:3000";
+      extraConfig = ''
+        proxy_hide_header Content-Type;
+        default_type text/html;
+        add_header Content-Type text/javascript;
+      '';
+    };
   };
 
   sane.dns.zones."uninsane.org".inet.CNAME."git" = "native";

hosts/by-name/servo/services/ipfs.nix

@@ -12,7 +12,7 @@ lib.mkIf false # i don't actively use ipfs anymore
 {
   sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
-    { user = "261"; group = "261"; path = "/var/lib/ipfs"; }
+    { user = "261"; group = "261"; path = "/var/lib/ipfs"; method = "bind"; }
   ];
 
   networking.firewall.allowedTCPPorts = [ 4001 ];

hosts/by-name/servo/services/jackett.nix

@@ -3,7 +3,7 @@
 {
   sane.persist.sys.byStore.plaintext = [
     # TODO: mode? we only need this to save Indexer creds ==> migrate to config?
-    { user = "root"; group = "root"; path = "/var/lib/jackett"; }
+    { user = "root"; group = "root"; path = "/var/lib/jackett"; method = "bind"; }
   ];
   services.jackett.enable = true;
 

hosts/by-name/servo/services/jellyfin.nix

@@ -41,7 +41,7 @@
   };
 
   sane.persist.sys.byStore.plaintext = [
-    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin"; }
+    { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin"; method = "bind"; }
   ];
   sane.fs."/var/lib/jellyfin/config/logging.json" = {
     # "Emby.Dlna" logging: <https://jellyfin.org/docs/general/networking/dlna>

hosts/by-name/servo/services/kiwix-serve.nix

@@ -1,9 +1,19 @@
+# how to update wikipedia snapshot:
+# - browse for later snapshots:
+#   - <https://mirror.accum.se/mirror/wikimedia.org/other/kiwix/zim/wikipedia>
+# - DL directly, or via rsync (resumable):
+#   - `rsync --progress --append-verify rsync://mirror.accum.se/mirror/wikimedia.org/other/kiwix/zim/wikipedia/wikipedia_en_all_maxi_2022-05.zim .`
+
 { ... }:
 {
+  sane.persist.sys.byStore.ext = [
+    { user = "colin"; group = "users"; path = "/var/lib/kiwix"; method = "bind"; }
+  ];
+
   sane.services.kiwix-serve = {
     enable = true;
     port = 8013;
-    zimPaths = [ "/var/lib/uninsane/www-archive/wikipedia_en_all_maxi_2022-05.zim" ];
+    zimPaths = [ "/var/lib/kiwix/wikipedia_en_all_maxi_2023-11.zim" ];
   };
 
   services.nginx.virtualHosts."w.uninsane.org" = {

hosts/by-name/servo/services/komga.nix

@@ -5,7 +5,7 @@ let
 in
 {
   sane.persist.sys.byStore.plaintext = [
-    { inherit user group; mode = "0700"; path = stateDir; }
+    { inherit user group; mode = "0700"; path = stateDir; method = "bind"; }
   ];
 
   services.komga.enable = true;

hosts/by-name/servo/services/lemmy.nix

@@ -78,8 +78,8 @@ in {
   # CLI args: <https://git.asonix.dog/asonix/pict-rs#user-content-running>
   systemd.services.pict-rs.serviceConfig.ExecStart = lib.mkForce (lib.concatStringsSep " " [
     "${lib.getBin pict-rs}/bin/pict-rs run"
-    "--media-max-frame-count" (builtins.toString (30*60*60))
+    "--media-video-max-frame-count" (builtins.toString (30*60*60))
     "--media-process-timeout 120"
-    "--media-enable-full-video true"  # allow audio
+    "--media-video-allow-audio"  # allow audio
   ]);
 }

hosts/by-name/servo/services/matrix/default.nix

@@ -21,7 +21,7 @@
   ];
 
   sane.persist.sys.byStore.plaintext = [
-    { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/matrix-synapse"; }
+    { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/matrix-synapse"; method = "bind"; }
   ];
   services.matrix-synapse.enable = true;
   services.matrix-synapse.settings = {

hosts/by-name/servo/services/matrix/discord-puppet.nix

@@ -6,7 +6,7 @@
 lib.mkIf false
 {
   sane.persist.sys.byStore.plaintext = [
-    { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/mx-puppet-discord"; }
+    { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/mx-puppet-discord"; method = "bind"; }
   ];
 
   services.matrix-synapse.settings.app_service_config_files = [

hosts/by-name/servo/services/matrix/irc.nix

@@ -103,7 +103,7 @@ in
 
   sane.persist.sys.byStore.plaintext = [
     # TODO: mode?
-    { user = "matrix-appservice-irc"; group = "matrix-appservice-irc"; path = "/var/lib/matrix-appservice-irc"; }
+    { user = "matrix-appservice-irc"; group = "matrix-appservice-irc"; path = "/var/lib/matrix-appservice-irc"; method = "bind"; }
   ];
 
   # XXX: matrix-appservice-irc PreStart tries to chgrp the registration.yml to matrix-synapse,

hosts/by-name/servo/services/matrix/signal.nix

@@ -1,10 +1,12 @@
 # config options:
 # - <https://github.com/mautrix/signal/blob/master/mautrix_signal/example-config.yaml>
-{ config, pkgs, ... }:
+{ config, lib, pkgs, ... }:
+
+lib.mkIf false  # disabled 2024/01/11: i don't use it, and pkgs.mautrix-signal had some API changes
 {
   sane.persist.sys.byStore.plaintext = [
-    { user = "mautrix-signal"; group = "mautrix-signal"; path = "/var/lib/mautrix-signal"; }
+    { user = "mautrix-signal"; group = "mautrix-signal"; path = "/var/lib/mautrix-signal"; method = "bind"; }
-    { user = "signald"; group = "signald"; path = "/var/lib/signald"; }
+    { user = "signald"; group = "signald"; path = "/var/lib/signald"; method = "bind"; }
   ];
 
   # allow synapse to read the registration file

hosts/by-name/servo/services/navidrome.nix

@@ -2,7 +2,7 @@
 
 {
   sane.persist.sys.byStore.plaintext = [
-    { user = "navidrome"; group = "navidrome"; path = "/var/lib/navidrome"; }
+    { user = "navidrome"; group = "navidrome"; path = "/var/lib/navidrome"; method = "bind"; }
   ];
   services.navidrome.enable = true;
   services.navidrome.settings = {

hosts/by-name/servo/services/nginx.nix

@@ -54,8 +54,10 @@ in
   services.nginx.recommendedOptimisation = true;
 
   # web blog/personal site
+  # alternative way to link stuff into the share:
+  # sane.fs."/var/lib/uninsane/share/Ubunchu".mount.bind = "/var/lib/uninsane/media/Books/Visual/HiroshiSeo/Ubunchu";
+  # sane.fs."/var/lib/uninsane/media/Books/Visual/HiroshiSeo/Ubunchu".dir = {};
   services.nginx.virtualHosts."uninsane.org" = publog {
-    root = "${pkgs.uninsane-dot-org}/share/uninsane-dot-org";
     # a lot of places hardcode https://uninsane.org,
     # and then when we mix http + non-https, we get CORS violations
     # and things don't look right. so force SSL.
@@ -65,9 +67,28 @@ in
     # for OCSP stapling
     sslTrustedCertificate = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";
 
-    # uninsane.org/share/foo => /var/lib/uninsane/root/share/foo.
+    locations."/" = {
-    # yes, nginx does not strip the prefix when evaluating against the root.
+      root = "${pkgs.uninsane-dot-org}/share/uninsane-dot-org";
-    locations."/share".root = "/var/lib/uninsane/root";
+      tryFiles = "$uri $uri/ @fallback";
+    };
+
+    # unversioned files
+    locations."@fallback" = {
+      root = "/var/www/sites/uninsane.org";
+    };
+
+    # uninsane.org/share/foo => /var/www/sites/uninsane.org/share/foo.
+    # special-cased to enable directory listings
+    locations."/share" = {
+      root = "/var/www/sites/uninsane.org";
+      extraConfig = ''
+        # autoindex => render directory listings
+        autoindex on;
+        # don't follow any symlinks when serving files
+        # otherwise it allows a directory escape
+        disable_symlinks on;
+      '';
+    };
 
     # allow matrix users to discover that @user:uninsane.org is reachable via matrix.uninsane.org
     locations."= /.well-known/matrix/server".extraConfig =
@@ -108,6 +129,19 @@ in
     #   proxyPass = "http://127.0.0.1:4000";
     #   extraConfig = pleromaExtraConfig;
     # };
+
+    # redirect common feed URIs to the canonical feed
+    locations."= /atom".extraConfig = "return 301 /atom.xml;";
+    locations."= /feed".extraConfig = "return 301 /atom.xml;";
+    locations."= /feed.xml".extraConfig = "return 301 /atom.xml;";
+    locations."= /rss".extraConfig = "return 301 /atom.xml;";
+    locations."= /rss.xml".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/atom".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/atom.xml".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/feed".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/feed.xml".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/rss".extraConfig = "return 301 /atom.xml;";
+    locations."= /blog/rss.xml".extraConfig = "return 301 /atom.xml;";
   };
 
 
@@ -135,9 +169,8 @@ in
   security.acme.defaults.email = "admin.acme@uninsane.org";
 
   sane.persist.sys.byStore.plaintext = [
-    # TODO: mode?
+    { user = "acme"; group = "acme"; path = "/var/lib/acme"; method = "bind"; }
-    { user = "acme"; group = "acme"; path = "/var/lib/acme"; }
+    { user = "colin"; group = "users"; path = "/var/www/sites"; method = "bind"; }
-    { user = "colin"; group = "users"; path = "/var/www/sites"; }
   ];
 
   # let's encrypt default chain looks like:

hosts/by-name/servo/services/nixos-prebuild.nix

@@ -0,0 +1,26 @@
+{ lib, pkgs, ... }:
+
+lib.optionalAttrs false  # disabled until i can be sure it's not gonna OOM my server in the middle of the night
+{
+  systemd.services.nixos-prebuild = {
+    description = "build a nixos image with all updated deps";
+    path = with pkgs; [ coreutils git nix ];
+    script = ''
+      working=$(mktemp -d /tmp/nixos-prebuild.XXXXXX)
+      pushd "$working"
+      git clone https://git.uninsane.org/colin/nix-files.git \
+        && cd nix-files \
+        && nix flake update \
+        || true
+      RC=$(nix run "$working/nix-files#check" -- -j1 --cores 5 --builders "")
+      popd
+      rm -rf "$working"
+      exit "$RC"
+    '';
+  };
+
+  systemd.timers.nixos-prebuild = {
+    wantedBy = [ "multi-user.target" ];
+    timerConfig.OnCalendar = "11,23:00:00";
+  };
+}

hosts/by-name/servo/services/ntfy/ntfy-sh.nix

@@ -34,7 +34,7 @@ in
     # not 100% necessary to persist this, but ntfy does keep a 12hr (by default) cache
     # for pushing notifications to users who become offline.
     # ACLs also live here.
-    { user = "ntfy-sh"; group ="ntfy-sh"; path = "/var/lib/ntfy-sh"; }
+    { user = "ntfy-sh"; group ="ntfy-sh"; path = "/var/lib/ntfy-sh"; method = "bind"; }
   ];
 
   services.ntfy-sh.enable = true;

hosts/by-name/servo/services/ntfy/ntfy-waiter.nix

@@ -49,7 +49,7 @@ in
       type = types.package;
       default = pkgs.static-nix-shell.mkPython3Bin {
         pname = "ntfy-waiter";
-        src = ./.;
+        srcRoot = ./.;
         pkgs = [ "ntfy-sh" ];
       };
       description = ''
@@ -64,7 +64,7 @@ in
         protocol = [ "tcp" ];
         visibleTo.lan = true;
         visibleTo.wan = true;
-        description = "colin-notification-waiter-${builtins.toString (port+1)}-of-${builtins.toString numPorts}";
+        description = "colin-notification-waiter-${builtins.toString (port - portLow + 1)}-of-${builtins.toString numPorts}";
       };
     }));
     systemd.services = lib.mkMerge (builtins.map mkService portRange);

hosts/by-name/servo/services/pict-rs.nix

@@ -6,7 +6,7 @@ let
 in
 {
   sane.persist.sys.byStore.plaintext = lib.mkIf cfg.enable [
-    { user = "pict-rs"; group = "pict-rs"; path = cfg.dataDir; }
+    { user = "pict-rs"; group = "pict-rs"; path = cfg.dataDir; method = "bind"; }
   ];
 
   systemd.services.pict-rs.serviceConfig = {

hosts/by-name/servo/services/pleroma.nix

@@ -15,7 +15,7 @@ let
 in
 {
   sane.persist.sys.byStore.plaintext = [
-    { user = "pleroma"; group = "pleroma"; path = "/var/lib/pleroma"; }
+    { user = "pleroma"; group = "pleroma"; path = "/var/lib/pleroma"; method = "bind"; }
   ];
   services.pleroma.enable = true;
   services.pleroma.secretConfigFile = config.sops.secrets.pleroma_secrets.path;

hosts/by-name/servo/services/postgres.nix

@@ -8,7 +8,7 @@ in
 {
   sane.persist.sys.byStore.plaintext = [
     # TODO: mode?
-    { user = "postgres"; group = "postgres"; path = "/var/lib/postgresql"; }
+    { user = "postgres"; group = "postgres"; path = "/var/lib/postgresql"; method = "bind"; }
   ];
   services.postgresql.enable = true;
 

hosts/by-name/servo/services/prosody/default.nix

@@ -57,7 +57,7 @@ let
 in
 {
   sane.persist.sys.byStore.plaintext = [
-    { user = "prosody"; group = "prosody"; path = "/var/lib/prosody"; }
+    { user = "prosody"; group = "prosody"; path = "/var/lib/prosody"; method = "bind"; }
   ];
   sane.ports.ports."5000" = {
     protocol = [ "tcp" ];

hosts/by-name/servo/services/slskd.nix

@@ -6,7 +6,7 @@
 { config, lib, ... }:
 {
   sane.persist.sys.byStore.plaintext = [
-    { user = "slskd"; group = "slskd"; path = "/var/lib/slskd"; }
+    { user = "slskd"; group = "slskd"; path = "/var/lib/slskd"; method = "bind"; }
   ];
   sops.secrets."slskd_env" = {
     owner = config.users.users.slskd.name;
@@ -57,7 +57,7 @@
     # what unit is this? kbps??
     global.upload.speed_limit = 32000;
     web.logging = true;
-    debug = true;
+    # debug = true;
     flags.no_logo = true;  # don't show logo at start
     # flags.volatile = true;  # store searches and active transfers in RAM (completed transfers still go to disk). rec for btrfs/zfs
   };
@@ -66,8 +66,8 @@
     serviceConfig = {
       # run this behind the OVPN static VPN
       NetworkNamespacePath = "/run/netns/ovpns";
-      Restart = "on-failure";
+      Restart = lib.mkForce "always";  # exits "success" when it fails to connect to soulseek server
-      RestartSec = "30s";
+      RestartSec = "60s";
       Group = "media";
     };
   };

hosts/by-name/servo/services/transmission.nix

@@ -1,14 +1,37 @@
-{ config, pkgs, ... }:
+{ config, lib, pkgs, ... }:
 
+let
+  # 2023/09/06: nixpkgs `transmission` defaults to old 3.00
+  # 2024/02/15: some torrent trackers whitelist clients; everyone is still on 3.00 for some reason :|
+  #             some do this via peer-id (e.g. baka); others via user-agent (e.g. MAM).
+  #             peer-id format is essentially the same between 3.00 and 4.x (just swap the MAJOR/MINOR/PATCH numbers).
+  #             user-agent format has changed. `Transmission/3.00` (old) v.s. `TRANSMISSION/MAJ.MIN.PATCH` (new).
+  realTransmission = pkgs.transmission_4;
+  realVersion = {
+    major = lib.versions.major realTransmission.version;
+    minor = lib.versions.minor realTransmission.version;
+    patch = lib.versions.patch realTransmission.version;
+  };
+  package = realTransmission.overrideAttrs (upstream: {
+    # `cmakeFlags = [ "-DTR_VERSION_MAJOR=3" ]`, etc, doesn't seem to take effect.
+    postPatch = (upstream.postPatch or "") + ''
+      substituteInPlace CMakeLists.txt \
+        --replace-fail 'TR_VERSION_MAJOR "${realVersion.major}"' 'TR_VERSION_MAJOR "3"' \
+        --replace-fail 'TR_VERSION_MINOR "${realVersion.minor}"' 'TR_VERSION_MINOR "0"' \
+        --replace-fail 'TR_VERSION_PATCH "${realVersion.patch}"' 'TR_VERSION_PATCH "0"' \
+        --replace-fail 'set(TR_USER_AGENT_PREFIX "''${TR_SEMVER}")' 'set(TR_USER_AGENT_PREFIX "3.00")'
+    '';
+  });
+in
 {
   sane.persist.sys.byStore.plaintext = [
     # TODO: mode? we need this specifically for the stats tracking in .config/
-    { user = "transmission"; group = config.users.users.transmission.group; path = "/var/lib/transmission"; }
+    { user = "transmission"; group = config.users.users.transmission.group; path = "/var/lib/transmission"; method = "bind"; }
   ];
   users.users.transmission.extraGroups = [ "media" ];
 
   services.transmission.enable = true;
-  services.transmission.package = pkgs.transmission_4;  #< 2023/09/06: nixpkgs `transmission` defaults to old 3.00
+  services.transmission.package = package;
   #v setting `group` this way doesn't tell transmission to `chown` the files it creates
   #  it's a nixpkgs setting which just runs the transmission daemon as this group
   services.transmission.group = "media";
@@ -16,10 +39,12 @@
   # transmission will by default not allow the world to read its files.
   services.transmission.downloadDirPermissions = "775";
   services.transmission.extraFlags = [
-    "--log-level=debug"
+    # "--log-level=debug"
   ];
 
   services.transmission.settings = {
+    # DOCUMENTATION/options list: <https://github.com/transmission/transmission/blob/main/docs/Editing-Configuration-Files.md#options>
+
     # message-level = 3;  #< enable for debug logging. 0-3, default is 2.
     # 0.0.0.0 => allow rpc from any host: we gate it via firewall and auth requirement
     rpc-bind-address = "0.0.0.0";
@@ -39,9 +64,9 @@
     encryption = 2;
 
     # units in kBps
-    speed-limit-down = 3000;
+    speed-limit-down = 12000;
     speed-limit-down-enabled = true;
-    speed-limit-up = 600;
+    speed-limit-up = 800;
     speed-limit-up-enabled = true;
 
     # see: https://git.zknt.org/mirror/transmission/commit/cfce6e2e3a9b9d31a9dafedd0bdc8bf2cdb6e876?lang=bg-BG

hosts/common/default.nix

@@ -1,22 +1,22 @@
-{ lib, pkgs, ... }:
+{ config, lib, pkgs, ... }:
 {
   imports = [
     ./feeds.nix
     ./fs.nix
     ./hardware
     ./home
-    ./hostnames.nix
     ./hosts.nix
     ./ids.nix
     ./machine-id.nix
-    ./net.nix
+    ./net
-    ./nix-path
+    ./nix
     ./persist.nix
+    ./polyunfill.nix
     ./programs
     ./secrets.nix
     ./ssh.nix
+    ./systemd.nix
     ./users
-    ./vpn.nix
   ];
 
   sane.nixcache.enable-trusted-keys = true;
@@ -32,91 +32,28 @@
   # time.timeZone = "America/Los_Angeles";
   time.timeZone = "Etc/UTC";  # DST is too confusing for me => use a stable timezone
 
-  nix.extraOptions = ''
-    # see: `man nix.conf`
-    # useful when a remote builder has a faster internet connection than me
-    builders-use-substitutes = true  # default: false
-    # maximum seconds to wait when connecting to binary substituter
-    connect-timeout = 3  # default: 0
-    # download-attempts = 5  # default: 5
-    # allow `nix flake ...` command
-    experimental-features = nix-command flakes
-    # whether to build from source when binary substitution fails
-    fallback = true  # default: false
-    # whether to keep building dependencies if any other one fails
-    keep-going = true  # default: false
-    # whether to keep build-only dependencies of GC roots (e.g. C compiler) when doing GC
-    keep-outputs = true  # default: false
-    # how many lines to show from failed build
-    log-lines = 30  # default: 10
-    # narinfo-cache-negative-ttl = 3600  # default: 3600
-    # whether to use ~/.local/state/nix/profile instead of ~/.nix-profile, etc
-    use-xdg-base-directories = true  # default: false
-    # whether to warn if repository has uncommited changes
-    warn-dirty = false  # default: true
-  '';
-  # hardlinks identical files in the nix store to save 25-35% disk space.
-  # unclear _when_ this occurs. it's not a service.
-  # does the daemon continually scan the nix store?
-  # does the builder use some content-addressed db to efficiently dedupe?
-  nix.settings.auto-optimise-store = true;
-  # TODO: see if i can remove this?
-  nix.settings.trusted-users = [ "root" ];
-
-  services.journald.extraConfig = ''
-    # docs: `man journald.conf`
-    # merged journald config is deployed to /etc/systemd/journald.conf
-    [Journal]
-    # disable journal compression because the underlying fs is compressed
-    Compress=no
-  '';
-
-  systemd.services.nix-daemon.serviceConfig = {
-    # the nix-daemon manages nix builders
-    # kill nix-daemon subprocesses when systemd-oomd detects an out-of-memory condition
-    # see:
-    # - nixos PR that enabled systemd-oomd: <https://github.com/NixOS/nixpkgs/pull/169613>
-    # - systemd's docs on these properties: <https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#ManagedOOMSwap=auto%7Ckill>
-    #
-    # systemd's docs warn that without swap, systemd-oomd might not be able to react quick enough to save the system.
-    # see `man oomd.conf` for further tunables that may help.
-    #
-    # alternatively, apply this more broadly with `systemd.oomd.enableSystemSlice = true` or `enableRootSlice`
-    # TODO: also apply this to the guest user's slice (user-1100.slice)
-    # TODO: also apply this to distccd
-    ManagedOOMMemoryPressure = "kill";
-    ManagedOOMSwap = "kill";
-  };
-
-
   system.activationScripts.nixClosureDiff = {
     supportsDryActivation = true;
     text = ''
       # show which packages changed versions or are new/removed in this upgrade
       # source: <https://github.com/luishfonseca/dotfiles/blob/32c10e775d9ec7cc55e44592a060c1c9aadf113e/modules/upgrade-diff.nix>
-      ${pkgs.nvd}/bin/nvd --nix-bin-dir=${pkgs.nix}/bin diff /run/current-system "$systemConfig"
+      # modified to not error on boot (when /run/current-system doesn't exist)
+      if [ -d /run/current-system ]; then
+        ${pkgs.nvd}/bin/nvd --nix-bin-dir=${pkgs.nix}/bin diff /run/current-system "$systemConfig"
+      fi
+    '';
+  };
+  system.activationScripts.notifyActive = {
+    text = ''
+      # send a notification to any sway users logged in, that the system has been activated/upgraded.
+      # this probably doesn't work if more than one sway session exists on the system.
+      _notifyActiveSwaySock="$(echo /run/user/*/sway-ipc*.sock)"
+      if [ -e "$_notifyActiveSwaySock" ]; then
+        SWAYSOCK="$_notifyActiveSwaySock" ${config.sane.programs.sway.packageUnwrapped}/bin/swaymsg -- exec \
+          "${pkgs.libnotify}/bin/notify-send 'nixos activated' 'version: $(cat $systemConfig/nixos-version)'"
+      fi
     '';
   };
-
-  # disable non-required packages like nano, perl, rsync, strace
-  environment.defaultPackages = [];
-
-  # dconf docs: <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/desktop_migration_and_administration_guide/profiles>
-  # this lets programs temporarily write user-level dconf settings (aka gsettings).
-  # they're written to ~/.config/dconf/user, unless `DCONF_PROFILE` is set to something other than the default of /etc/dconf/profile/user
-  # find keys/values with `dconf dump /`
-  programs.dconf.enable = true;
-  programs.dconf.packages = [
-    (pkgs.writeTextFile {
-      name = "dconf-user-profile";
-      destination = "/etc/dconf/profile/user";
-      text = ''
-        user-db:user
-        system-db:site
-      '';
-    })
-  ];
-  # sane.programs.glib.enableFor.user.colin = true;  # for `gsettings`
 
   # link debug symbols into /run/current-system/sw/lib/debug
   # hopefully picked up by gdb automatically?

hosts/common/feeds.nix

@@ -1,4 +1,5 @@
 # where to find good stuff?
+# - universal search/directory: <https://podcastindex.org>
 # - podcasts w/ a community: <https://lemmyverse.net/communities?query=podcast>
 # - podcast rec thread: <https://lemmy.ml/post/1565858>
 #
@@ -50,6 +51,8 @@ let
         else
           "infrequent"
       ));
+    } // lib.optionalAttrs (lib.hasPrefix "https://www.youtube.com/" raw.url) {
+      format = "video";
     } // lib.optionalAttrs (raw.is_podcast or false) {
       format = "podcast";
     } // lib.optionalAttrs (raw.title or "" != "") {
@@ -60,6 +63,7 @@ let
     (fromDb "acquiredlpbonussecretsecret.libsyn.com" // tech)  # ACQ2 - more "Acquired" episodes
     (fromDb "allinchamathjason.libsyn.com" // pol)
     (fromDb "anchor.fm/s/34c7232c/podcast/rss" // tech)  # Civboot -- https://anchor.fm/civboot
+    (fromDb "anchor.fm/s/2da69154/podcast/rss" // tech)  # POD OF JAKE -- https://podofjake.com/
     (fromDb "cast.postmarketos.org" // tech)
     (fromDb "congressionaldish.libsyn.com" // pol)  # Jennifer Briney
     (fromDb "craphound.com" // pol)  # Cory Doctorow -- both podcast & text entries
@@ -69,22 +73,21 @@ let
     (fromDb "feeds.feedburner.com/80000HoursPodcast" // rat)
     (fromDb "feeds.feedburner.com/dancarlin/history" // rat)
     (fromDb "feeds.feedburner.com/radiolab" // pol)  # Radiolab -- also available here, but ONLY OVER HTTP: <http://feeds.wnyc.org/radiolab>
-    (fromDb "feeds.libsyn.com/421877" // rat)  # Less Wrong Curated
     (fromDb "feeds.megaphone.fm/behindthebastards" // pol)  # also Maggie Killjoy
-    (fromDb "feeds.megaphone.fm/hubermanlab" // uncat)  # Daniel Huberman on sleep
     (fromDb "feeds.megaphone.fm/recodedecode" // tech)  # The Verge - Decoder
     (fromDb "feeds.simplecast.com/54nAGcIl" // pol)  # The Daily
     (fromDb "feeds.simplecast.com/82FI35Px" // pol)  # Ezra Klein Show
     (fromDb "feeds.simplecast.com/wgl4xEgL" // rat)  # Econ Talk
     (fromDb "feeds.simplecast.com/xKJ93w_w" // uncat)  # Atlas Obscura
-    (fromDb "feeds.simplecast.com/l2i9YnTd" // tech // pol)  # Hard Fork (NYtimes tech)
     (fromDb "feeds.transistor.fm/acquired" // tech)
+    (fromDb "fulltimenix.com" // tech)
     (fromDb "lexfridman.com/podcast" // rat)
     (fromDb "mapspodcast.libsyn.com" // uncat)  # Multidisciplinary Association for Psychedelic Studies
     (fromDb "omegataupodcast.net" // tech)  # 3/4 German; 1/4 eps are English
     (fromDb "omny.fm/shows/cool-people-who-did-cool-stuff" // pol)  # Maggie Killjoy -- referenced by Cory Doctorow
+    (fromDb "omny.fm/shows/the-dollop-with-dave-anthony-and-gareth-reynolds")  # The Dollop history/comedy
+    (fromDb "originstories.libsyn.com" // uncat)
     (fromDb "podcast.posttv.com/itunes/post-reports.xml" // pol)
-    (fromDb "podcast.thelinuxexp.com" // tech)
     (fromDb "politicalorphanage.libsyn.com" // pol)
     (fromDb "reverseengineering.libsyn.com/rss" // tech)  # UnNamed Reverse Engineering Podcast
     (fromDb "rss.acast.com/deconstructed")  # The Intercept - Deconstructed
@@ -93,17 +96,22 @@ let
     (fromDb "rss.art19.com/60-minutes" // pol)
     (fromDb "rss.art19.com/the-portal" // rat)  # Eric Weinstein
     (fromDb "seattlenice.buzzsprout.com" // pol)
+    (fromDb "srslywrong.com" // pol)
     (fromDb "sharkbytes.transistor.fm" // tech)  # Wireshark Podcast o_0
     (fromDb "sscpodcast.libsyn.com" // rat)  # Astral Codex Ten
     (fromDb "talesfromthebridge.buzzsprout.com" // tech)  # Sci-Fi? has Peter Watts; author of No Moods, Ads or Cutesy Fucking Icons (rifters.com)
     (fromDb "techwontsave.us" // pol)  # rec by Cory Doctorow
-    # (fromDb "trashfuturepodcast.podbean.com" // pol)  # rec by Cory Doctorow, but way rambly
     (fromDb "wakingup.libsyn.com" // pol)  # Sam Harris
     (fromDb "werenotwrong.fireside.fm" // pol)
 
+    # (fromDb "feeds.libsyn.com/421877" // rat)  # Less Wrong Curated
+    # (fromDb "feeds.megaphone.fm/hubermanlab" // uncat)  # Daniel Huberman on sleep
+    # (fromDb "feeds.simplecast.com/l2i9YnTd" // tech // pol)  # Hard Fork (NYtimes tech)
+    # (fromDb "podcast.thelinuxexp.com" // tech)  # low-brow linux/foss PR announcements
     # (fromDb "rss.art19.com/your-welcome" // pol)  # Michael Malice - Your Welcome -- also available here: <https://origin.podcastone.com/podcast?categoryID2=2232>
     # (fromDb "rss.prod.firstlook.media/deconstructed/podcast.rss" // pol)  #< possible URL rot
     # (fromDb "rss.prod.firstlook.media/intercepted/podcast.rss" // pol)  #< possible URL rot
+    # (fromDb "trashfuturepodcast.podbean.com" // pol)  # rec by Cory Doctorow, but way rambly
     # (mkPod "https://anchor.fm/s/21bc734/podcast/rss" // pol // infrequent)  # Emerge: making sense of what's next -- <https://www.whatisemerging.com/emergepodcast>
     # (mkPod "https://audioboom.com/channels/5097784.rss" // tech)  # Lateral with Tom Scott
     # (mkPod "https://feeds.megaphone.fm/RUNMED9919162779" // pol // infrequent)  # The Witch Trials of J.K. Rowling: <https://www.thefp.com/witchtrials>
@@ -111,137 +119,120 @@ let
   ];
 
   texts = [
-    # AGGREGATORS (> 1 post/day)
+    (fromDb "acoup.blog/feed")  # history, states. author: <https://historians.social/@bretdevereaux/following>
-    (fromDb "lwn.net" // tech)
+    (fromDb "amosbbatto.wordpress.com" // tech)
-    # (fromDb "lesswrong.com" // rat)
+    (fromDb "applieddivinitystudies.com" // rat)
+    (fromDb "artemis.sh" // tech)
+    (fromDb "ascii.textfiles.com" // tech)  # Jason Scott
+    (fromDb "austinvernon.site" // tech)
+    # (fromDb "balajis.com" // pol)  # Balaji
+    (fromDb "ben-evans.com/benedictevans" // pol)
+    (fromDb "bitbashing.io" // tech)
+    (fromDb "bitsaboutmoney.com" // uncat)
+    (fromDb "blog.danieljanus.pl" // tech)
+    (fromDb "blog.dshr.org" // pol)  # David Rosenthal
+    (fromDb "blog.jmp.chat" // tech)
+    (fromDb "blog.rust-lang.org" // tech)
+    (fromDb "blog.thalheim.io" // tech)  # Mic92
+    (fromDb "bunniestudios.com" // tech)  # Bunnie Juang
+    (fromDb "capitolhillseattle.com" // pol)
+    # (fromDb "drewdevault.com" // tech)
     # (fromDb "econlib.org" // pol)
-
+    (fromDb "edwardsnowden.substack.com" // pol // text)
-    # AGGREGATORS (< 1 post/day)
+    (fromDb "fasterthanli.me" // tech)
+    (fromDb "gwern.net" // rat)
+    (fromDb "harihareswara.net" // tech // pol)  # rec by Cory Doctorow
+    (fromDb "ianthehenry.com" // tech)
+    (fromDb "idiomdrottning.org" // uncat)
+    (fromDb "interconnected.org/home/feed" // rat)  # Matt Webb -- engineering-ish, but dreamy
+    (fromDb "jeffgeerling.com" // tech)
+    (fromDb "jefftk.com" // tech)
+    (fromDb "kill-the-newsletter.com/feeds/joh91bv7am2pnznv.xml" // pol)  # Matt Levine - Money Stuff
+    (fromDb "kosmosghost.github.io/index.xml" // tech)
+    # (fromDb "lesswrong.com" // rat)
+    (fromDb "linmob.net" // tech)
+    (fromDb "lwn.net" // tech)
+    (fromDb "lynalden.com" // pol)
+    (fromDb "mako.cc/copyrighteous" // tech // pol)  # rec by Cory Doctorow
+    (fromDb "mg.lol" // tech)
+    (fromDb "mindingourway.com" // rat)
+    (fromDb "morningbrew.com/feed" // pol)
+    (fromDb "overcomingbias.com" // rat)  # Robin Hanson
     (fromDb "palladiummag.com" // uncat)
+    (fromDb "philosopher.coach" // rat)  # Peter Saint-Andre -- side project of stpeter.im
+    (fromDb "pomeroyb.com" // tech)
+    (fromDb "preposterousuniverse.com" // rat)  # Sean Carroll
     (fromDb "profectusmag.com" // uncat)
+    (fromDb "project-insanity.org" // tech)  # shared blog by a few NixOS devs, notably onny
+    (fromDb "putanumonit.com" // rat)  # mostly dating topics. not advice, or humor, but looking through a social lens
+    (fromDb "richardcarrier.info" // rat)
+    (fromDb "rifters.com/crawl" // uncat)  # No Moods, Ads or Cutesy Fucking Icons
+    (fromDb "righto.com" // tech)  # Ken Shirriff
+    (fromDb "rootsofprogress.org" // rat)  # Jason Crawford
+    (fromDb "sagacioussuricata.com" // tech)  # ian (Sanctuary)
     (fromDb "semiaccurate.com" // tech)
-    (mkText "https://linuxphoneapps.org/blog/atom.xml" // tech // infrequent)
+    (fromDb "sideways-view.com" // rat)  # Paul Christiano
-    (fromDb "tuxphones.com" // tech)
+    (fromDb "slimemoldtimemold.com" // rat)
     (fromDb "spectrum.ieee.org" // tech)
+    (fromDb "stpeter.im/atom.xml" // pol)
     # (fromDb "theregister.com" // tech)
     (fromDb "thisweek.gnome.org" // tech)
-    # more nixos stuff here, but unclear how to subscribe: <https://nixos.org/blog/categories.html>
+    (fromDb "tuxphones.com" // tech)
-    (mkText "https://nixos.org/blog/announcements-rss.xml" // tech // infrequent)
-    (mkText "https://nixos.org/blog/stories-rss.xml" // tech // weekly)
-    ## n.b.: quality RSS list here: <https://forum.merveilles.town/thread/57/share-your-rss-feeds%21-6/>
-    (mkText "https://forum.merveilles.town/rss.xml" // pol // infrequent)
-
-    ## No Moods, Ads or Cutesy Fucking Icons
-    (fromDb "rifters.com/crawl" // uncat)
-
-    # DEVELOPERS
-    (fromDb "blog.jmp.chat" // tech)
     (fromDb "uninsane.org" // tech)
-    (fromDb "blog.thalheim.io" // tech)  # Mic92
+    (fromDb "unintendedconsequenc.es" // rat)
-    (fromDb "ascii.textfiles.com" // tech)  # Jason Scott
+    # (fromDb "vitalik.ca" // tech)  # moved to vitalik.eth.limo
+    (fromDb "vitalik.eth.limo" // tech)  # Vitalik Buterin
+    (fromDb "webcurious.co.uk" // uncat)
     (fromDb "xn--gckvb8fzb.com" // tech)
-    (fromDb "amosbbatto.wordpress.com" // tech)
+    (mkSubstack "astralcodexten" // rat // daily)  # Scott Alexander
-    (fromDb "fasterthanli.me" // tech)
-    (fromDb "jeffgeerling.com" // tech)
-    (fromDb "mg.lol" // tech)
-    # (fromDb "drewdevault.com" // tech)
-    ## Ken Shirriff
-    (fromDb "righto.com" // tech)
-    ## shared blog by a few NixOS devs, notably onny
-    (fromDb "project-insanity.org" // tech)
-    ## Vitalik Buterin
-    (fromDb "vitalik.ca" // tech)
-    ## ian (Sanctuary)
-    (fromDb "sagacioussuricata.com" // tech)
-    (fromDb "artemis.sh" // tech)
-    ## Bunnie Juang
-    (fromDb "bunniestudios.com" // tech)
-    (fromDb "blog.danieljanus.pl" // tech)
-    (fromDb "ianthehenry.com" // tech)
-    (fromDb "bitbashing.io" // tech)
-    (fromDb "idiomdrottning.org" // uncat)
-    (mkText "http://boginjr.com/feed" // tech // infrequent)
-    (mkText "https://anish.lakhwara.com/home.html" // tech // weekly)
-    (fromDb "jefftk.com" // tech)
-    (fromDb "pomeroyb.com" // tech)
-    (fromDb "harihareswara.net" // tech // pol)  # rec by Cory Doctorow
-    (fromDb "mako.cc/copyrighteous" // tech // pol)  # rec by Cory Doctorow
-    # (mkText "https://til.simonwillison.net/tils/feed.atom" // tech // weekly)
-
-    # TECH PROJECTS
-    (fromDb "blog.rust-lang.org" // tech)
-
-    # (TECH; POL) COMMENTATORS
-    ## Matt Webb -- engineering-ish, but dreamy
-    (fromDb "interconnected.org/home/feed" // rat)
-    (fromDb "edwardsnowden.substack.com" // pol // text)
-    ## Julia Evans
-    (mkText "https://jvns.ca/atom.xml" // tech // weekly)
-    (mkText "http://benjaminrosshoffman.com/feed" // pol // weekly)
-    ## Ben Thompson
-    (mkText "https://www.stratechery.com/rss" // pol // weekly)
-    ## Balaji
-    (fromDb "balajis.com" // pol)
-    (fromDb "ben-evans.com/benedictevans" // pol)
-    (fromDb "lynalden.com" // pol)
-    (fromDb "austinvernon.site" // tech)
-    (mkSubstack "oversharing" // pol // daily)
     (mkSubstack "byrnehobart" // pol // infrequent)
     # (mkSubstack "doomberg" // tech // weekly)  # articles are all pay-walled
-    ## David Rosenthal
-    (fromDb "blog.dshr.org" // pol)
-    ## Matt Levine
-    (mkText "https://www.bloomberg.com/opinion/authors/ARbTQlRLRjE/matthew-s-levine.rss" // pol // weekly)
-    (fromDb "stpeter.im/atom.xml" // pol)
-    ## Peter Saint-Andre -- side project of stpeter.im
-    (fromDb "philosopher.coach" // rat)
-    (fromDb "morningbrew.com/feed" // pol)
-
-    # RATIONALITY/PHILOSOPHY/ETC
-    (mkSubstack "samkriss" // humor // infrequent)
-    (fromDb "unintendedconsequenc.es" // rat)
-    (fromDb "applieddivinitystudies.com" // rat)
-    (fromDb "slimemoldtimemold.com" // rat)
-    (fromDb "richardcarrier.info" // rat)
-    (fromDb "gwern.net" // rat)
-    ## Jason Crawford
-    (fromDb "rootsofprogress.org" // rat)
-    ## Robin Hanson
-    (fromDb "overcomingbias.com" // rat)
-    ## Scott Alexander
-    (mkSubstack "astralcodexten" // rat // daily)
-    ## Paul Christiano
-    (fromDb "sideways-view.com" // rat)
-    ## Sean Carroll
-    (fromDb "preposterousuniverse.com" // rat)
     (mkSubstack "eliqian" // rat // weekly)
-    (mkText "https://acoup.blog/feed" // rat // weekly)
+    (mkSubstack "oversharing" // pol // daily)
-    (fromDb "mindingourway.com" // rat)
+    (mkSubstack "samkriss" // humor // infrequent)
-
+    (mkText "http://benjaminrosshoffman.com/feed" // pol // weekly)
-    ## mostly dating topics. not advice, or humor, but looking through a social lens
+    (mkText "http://boginjr.com/feed" // tech // infrequent)
-    (fromDb "putanumonit.com" // rat)
+    (mkText "https://anish.lakhwara.com/home.html" // tech // weekly)
-
+    (mkText "https://forum.merveilles.town/rss.xml" // pol // infrequent)  #quality RSS list here: <https://forum.merveilles.town/thread/57/share-your-rss-feeds%21-6/>
-    # LOCAL
-    (fromDb "capitolhillseattle.com" // pol)
-
-    # CODE
     # (mkText "https://github.com/Kaiteki-Fedi/Kaiteki/commits/master.atom" // tech // infrequent)
+    (mkText "https://jvns.ca/atom.xml" // tech // weekly)  # Julia Evans
+    (mkText "https://linuxphoneapps.org/blog/atom.xml" // tech // infrequent)
+    (mkText "https://nixos.org/blog/announcements-rss.xml" // tech // infrequent)  # more nixos stuff here, but unclear how to subscribe: <https://nixos.org/blog/categories.html>
+    (mkText "https://nixos.org/blog/stories-rss.xml" // tech // weekly)
+    # (mkText "https://til.simonwillison.net/tils/feed.atom" // tech // weekly)
+    # (mkText "https://www.bloomberg.com/opinion/authors/ARbTQlRLRjE/matthew-s-levine.rss" // pol // weekly)  # Matt Levine (preview/paywalled)
+    (mkText "https://www.stratechery.com/rss" // pol // weekly)  # Ben Thompson
+  ];
+
+  videos = [
+    (fromDb "youtube.com/@Channel5YouTube" // pol)
+    (fromDb "youtube.com/@ColdFusion")
+    (fromDb "youtube.com/@ContraPoints" // pol)
+    (fromDb "youtube.com/@Exurb1a")
+    (fromDb "youtube.com/@hbomberguy")
+    (fromDb "youtube.com/@JackStauber")
+    (fromDb "youtube.com/@PolyMatter")
+    # (fromDb "youtube.com/@rossmanngroup" // pol // tech)  # Louis Rossmann
+    (fromDb "youtube.com/@TechnologyConnections" // tech)
+    (fromDb "youtube.com/@TheB1M")
+    (fromDb "youtube.com/@TomScottGo")
+    (fromDb "youtube.com/@Vihart")
+    (fromDb "youtube.com/@Vox")
+    (fromDb "youtube.com/@Vsauce")
   ];
 
   images = [
-    (fromDb "smbc-comics.com" // img // humor)
+    (fromDb "catandgirl.com" // img // humor)
-    (fromDb "xkcd.com" // img // humor)
-    (fromDb "turnoff.us" // img // humor)
-    (fromDb "pbfcomics.com" // img // humor)
-    # (mkImg "http://dilbert.com/feed" // humor // daily)
-    (fromDb "poorlydrawnlines.com/feed" // img // humor)
-
-    # ART
     (fromDb "miniature-calendar.com" // img // art // daily)
+    (fromDb "pbfcomics.com" // img // humor)
+    (fromDb "poorlydrawnlines.com/feed" // img // humor)
+    (fromDb "smbc-comics.com" // img // humor)
+    (fromDb "turnoff.us" // img // humor)
+    (fromDb "xkcd.com" // img // humor)
   ];
 in
 {
-  sane.feeds = texts ++ images ++ podcasts;
+  sane.feeds = texts ++ images ++ podcasts ++ videos;
 
   assertions = builtins.map
     (p: {

hosts/common/fs.nix

@@ -1,5 +1,6 @@
 # docs
 # - x-systemd options: <https://www.freedesktop.org/software/systemd/man/systemd.mount.html>
+# - fuse options: `man mount.fuse`
 
 { lib, pkgs, sane-lib, ... }:
 
@@ -8,13 +9,22 @@ let
     common = [
       "_netdev"
       "noatime"
-      "user"  # allow any user with access to the device to mount the fs
+      # user: allow any user with access to the device to mount the fs.
+      #       note that this requires a suid `mount` binary; see: <https://zameermanji.com/blog/2022/8/5/using-fuse-without-root-on-linux/>
+      "user"
       "x-systemd.requires=network-online.target"
       "x-systemd.after=network-online.target"
       "x-systemd.mount-timeout=10s"  # how long to wait for mount **and** how long to wait for unmount
     ];
-    auto = [ "x-systemd.automount" ];
+    # x-systemd.automount: mount the fs automatically *on first access*.
-    noauto = [ "noauto" ];  # don't mount as part of remote-fs.target
+    # creates a `path-to-mount.automount` systemd unit.
+    automount = [ "x-systemd.automount" ];
+    # noauto: don't mount as part of remote-fs.target.
+    # N.B.: `remote-fs.target` is a dependency of multi-user.target, itself of graphical.target.
+    # hence, omitting `noauto` can slow down boots.
+    noauto = [ "noauto" ];
+    # lazyMount: defer mounting until first access from userspace
+    lazyMount = noauto ++ automount;
     wg = [
       "x-systemd.requires=wireguard-wg-home.service"
       "x-systemd.after=wireguard-wg-home.service"
@@ -22,19 +32,34 @@ let
 
     ssh = common ++ [
       "identityfile=/home/colin/.ssh/id_ed25519"
-      "allow_other"
+      "allow_other"  # allow users other than the one who mounts it to access it. needed, if systemd is the one mounting this fs (as root)
+      # allow_root: allow root to access files on this fs (if mounted by non-root, else it can always access them).
+      #             N.B.: if both allow_root and allow_other are specified, then only allow_root takes effect.
+      # "allow_root"
+      # default_permissions: enforce local permissions check. CRUCIAL if using `allow_other`.
+      # w/o this, permissions mode of sshfs is like:
+      # - sshfs runs all remote commands as the remote user.
+      # - if a local user has local permissions to the sshfs mount, then their file ops are sent blindly across the tunnel.
+      # - `allow_other` allows *any* local user to access the mount, and hence any local user can now freely become the remote mapped user.
+      # with default_permissions, sshfs doesn't tunnel file ops from users until checking that said user could perform said op on an equivalent local fs.
       "default_permissions"
     ];
     sshColin = ssh ++ [
+      # follow_symlinks: remote files which are symlinks are presented to the local system as ordinary files (as the target of the symlink).
+      #                  if the symlink target does not exist, the presentation is unspecified.
+      #                  symlinks which point outside the mount ARE followed. so this is more capable than `transform_symlinks`
+      "follow_symlinks"
+      # symlinks on the remote fs which are absolute paths are presented to the local system as relative symlinks pointing to the expected data on the remote fs.
+      # only symlinks which would point inside the mountpoint are translated.
       "transform_symlinks"
       "idmap=user"
       "uid=1000"
       "gid=100"
     ];
-    sshRoot = ssh ++ [
+    # sshRoot = ssh ++ [
-      # we don't transform_symlinks because that breaks the validity of remote /nix stores
+    #   # we don't transform_symlinks because that breaks the validity of remote /nix stores
-      "sftp_server=/run/wrappers/bin/sudo\\040/run/current-system/sw/libexec/sftp-server"
+    #   "sftp_server=/run/wrappers/bin/sudo\\040/run/current-system/sw/libexec/sftp-server"
-    ];
+    # ];
     # in the event of hunt NFS mounts, consider:
     # - <https://unix.stackexchange.com/questions/31979/stop-broken-nfs-mounts-from-locking-a-directory>
 
@@ -57,13 +82,17 @@ let
     ];
   };
   remoteHome = host: {
-    fileSystems."/mnt/${host}-home" = {
+    fileSystems."/mnt/${host}/home" = {
       device = "colin@${host}:/home/colin";
       fsType = "fuse.sshfs";
-      options = fsOpts.sshColin ++ fsOpts.noauto;
+      options = fsOpts.sshColin ++ fsOpts.lazyMount;
       noCheck = true;
     };
-    sane.fs."/mnt/${host}-home" = sane-lib.fs.wantedDir;
+    sane.fs."/mnt/${host}/home" = sane-lib.fs.wanted {
+      dir.acl.user = "colin";
+      dir.acl.group = "users";
+      dir.acl.mode = "0700";
+    };
   };
 in
 lib.mkMerge [
@@ -103,34 +132,38 @@ lib.mkMerge [
     #   device = "servo-hn:/";
     #   noCheck = true;
     #   fsType = "nfs";
-    #   options = fsOpts.nfs ++ fsOpts.auto ++ fsOpts.wg;
+    #   options = fsOpts.nfs ++ fsOpts.automount ++ fsOpts.wg;
     # };
-    fileSystems."/mnt/servo-nfs/media" = {
+    fileSystems."/mnt/servo/media" = {
       device = "servo-hn:/media";
       noCheck = true;
       fsType = "nfs";
-      options = fsOpts.nfs ++ fsOpts.auto ++ fsOpts.wg;
+      options = fsOpts.nfs ++ fsOpts.lazyMount ++ fsOpts.wg;
     };
-    fileSystems."/mnt/servo-nfs/playground" = {
+    sane.fs."/mnt/servo/media" = sane-lib.fs.wanted {
+      dir.acl.user = "colin";
+      dir.acl.group = "users";
+      dir.acl.mode = "0750";
+    };
+    fileSystems."/mnt/servo/playground" = {
       device = "servo-hn:/playground";
       noCheck = true;
       fsType = "nfs";
-      options = fsOpts.nfs ++ fsOpts.auto ++ fsOpts.wg;
+      options = fsOpts.nfs ++ fsOpts.lazyMount ++ fsOpts.wg;
+    };
+    sane.fs."/mnt/servo/playground" = sane-lib.fs.wanted {
+      dir.acl.user = "colin";
+      dir.acl.group = "users";
+      dir.acl.mode = "0750";
     };
-    # fileSystems."/mnt/servo-media-nfs" = {
-    #   device = "servo-hn:/media";
-    #   noCheck = true;
-    #   fsType = "nfs";
-    #   options = fsOpts.common ++ fsOpts.auto;
-    # };
-    sane.fs."/mnt/servo-media" = sane-lib.fs.wantedSymlinkTo "/mnt/servo-nfs/media";
 
-    environment.pathsToLink = [
+    # environment.pathsToLink = [
-      # needed to achieve superuser access for user-mounted filesystems (see optionsRoot above)
+    #   # needed to achieve superuser access for user-mounted filesystems (see sshRoot above)
-      # we can only link whole directories here, even though we're only interested in pkgs.openssh
+    #   # we can only link whole directories here, even though we're only interested in pkgs.openssh
-      "/libexec"
+    #   "/libexec"
-    ];
+    # ];
 
+    programs.fuse.userAllowOther = true;  #< necessary for `allow_other` or `allow_root` options.
     environment.systemPackages = [
       pkgs.sshfs-fuse
     ];

hosts/common/hardware/default.nix

@@ -1,4 +1,4 @@
-{ lib, pkgs, ... }:
+{ config, lib, pkgs, ... }:
 
 {
   imports = [
@@ -28,6 +28,13 @@
   #   "systemd.log_level=debug"
   #   "systemd.log_target=console"
 
+  # moby has to run recent kernels (defined elsewhere).
+  # meanwhile, kernel variation plays some minor role in things like sandboxing (landlock) and capabilities.
+  # simpler to keep near the latest kernel on all devices,
+  # and also makes certain that any weird system-level bugs i see aren't likely to be stale kernel bugs.
+  # servo needs zfs though, which doesn't support every kernel.
+  boot.kernelPackages = lib.mkDefault pkgs.zfs.latestCompatibleLinuxPackages;
+
   # hack in the `boot.shell_on_fail` arg since that doesn't always seem to work.
   boot.initrd.preFailCommands = "allowShell=1";
 
@@ -40,6 +47,12 @@
   # non-free firmware
   hardware.enableRedistributableFirmware = true;
 
+  # default is 252274, which is too low particularly for servo.
+  # manifests as spurious "No space left on device" when trying to install watches,
+  # e.g. in dyn-dns by `systemctl start dyn-dns-watcher.path`.
+  # see: <https://askubuntu.com/questions/828779/failed-to-add-run-systemd-ask-password-to-directory-watch-no-space-left-on-dev>
+  boot.kernel.sysctl."fs.inotify.max_user_watches" = 1048576;
+
   # powertop will default to putting USB devices -- including HID -- to sleep after TWO SECONDS
   powerManagement.powertop.enable = false;
   # linux CPU governor: <https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt>
@@ -58,10 +71,19 @@
   powerManagement.cpuFreqGovernor = "ondemand";
 
   services.logind.extraConfig = ''
-    # don’t shutdown when power button is short-pressed
+    # see: `man logind.conf`
-    HandlePowerKey=ignore
+    # don’t shutdown when power button is short-pressed (commonly done an accident, or by cats).
+    #   but do on long-press: useful to gracefully power-off server.
+    HandlePowerKey=lock
+    HandlePowerKeyLongPress=poweroff
+    HandleLidSwitch=lock
   '';
 
+  # some packages build only if binfmt *isn't* present
+  nix.settings.system-features = lib.mkIf (config.boot.binfmt.emulatedSystems == []) [
+    "no-binfmt"
+  ];
+
   # services.snapper.configs = {
   #   root = {
   #     subvolume = "/";

hosts/common/home/default.nix

@@ -1,7 +1,7 @@
 { ... }:
 {
   imports = [
-    ./keyring
+    ./fs.nix
     ./mime.nix
     ./ssh.nix
     ./xdg-dirs.nix

hosts/common/home/fs.nix

@@ -0,0 +1,42 @@
+{ config, ... }:
+{
+  sane.user.persist.byStore.plaintext = [
+    "archive"
+    "dev"
+    # TODO: records should be private
+    "records"
+    "ref"
+    "tmp"
+    "use"
+    "Books/local"
+    "Music"
+    "Pictures/albums"
+    "Pictures/cat"
+    "Pictures/from"
+    "Pictures/Screenshots"  #< XXX: something is case-sensitive about this?
+    "Pictures/Photos"
+    "Videos/local"
+
+    # these are persisted simply to save on RAM.
+    # ~/.cache/nix can become several GB.
+    # mesa_shader_cache is < 10 MB.
+    # TODO: integrate with sane.programs.sandbox?
+    ".cache/mesa_shader_cache"
+    ".cache/nix"
+  ];
+  sane.user.persist.byStore.private = [
+    "knowledge"
+  ];
+
+  # convenience
+  sane.user.fs.".persist/private".symlink.target = config.sane.persist.stores.private.origin;
+  sane.user.fs.".persist/plaintext".symlink.target = config.sane.persist.stores.plaintext.origin;
+  sane.user.fs.".persist/ephemeral".symlink.target = config.sane.persist.stores.cryptClearOnBoot.origin;
+
+  sane.user.fs."nixos".symlink.target = "dev/nixos";
+
+  sane.user.fs."Books/servo".symlink.target = "/mnt/servo/media/Books";
+  sane.user.fs."Videos/servo".symlink.target = "/mnt/servo/media/Videos";
+  # sane.user.fs."Music/servo".symlink.target = "/mnt/servo/media/Music";
+  sane.user.fs."Pictures/servo-macros".symlink.target = "/mnt/servo/media/Pictures/macros";
+}

hosts/common/home/keyring/default.nix

@@ -1,17 +0,0 @@
-{ config, pkgs, sane-lib, ... }:
-
-let
-  init-keyring = pkgs.static-nix-shell.mkBash {
-    pname = "init-keyring";
-    src = ./.;
-  };
-in
-{
-  sane.user.persist.byStore.private = [ ".local/share/keyrings" ];
-
-  sane.user.fs."private/.local/share/keyrings/default" = {
-    generated.command = [ "${init-keyring}/bin/init-keyring" ];
-    wantedBy = [ config.sane.fs."/home/colin/private".unit ];
-    wantedBeforeBy = [ ];  # don't created this as part of `multi-user.target`
-  };
-}

hosts/common/home/keyring/init-keyring

@@ -1,21 +0,0 @@
-#!/usr/bin/env nix-shell
-#!nix-shell -i bash
-# initializes the default libsecret keyring (used by gnome-keyring) if not already initialized.
-# this initializes it to be plaintext/unencrypted.
-
-ringdir=/home/colin/private/.local/share/keyrings
-if test -f "$ringdir/default"
-then
-  echo 'keyring already initialized: not doing anything'
-else
-  keyring="$ringdir/Default_keyring.keyring"
-
-  echo 'initializing default user keyring:' "$keyring.new"
-  echo '[keyring]' > "$keyring.new"
-  echo 'display-name=Default keyring' >> "$keyring.new"
-  echo 'lock-on-idle=false' >> "$keyring.new"
-  echo 'lock-after=false' >> "$keyring.new"
-  chown colin:users "$keyring.new"
-  # closest to an atomic update we can achieve
-  mv "$keyring.new" "$keyring" && echo -n "Default_keyring" > "$ringdir/default"
-fi

hosts/common/home/mime.nix

@@ -1,28 +1,94 @@
-{ config, lib, ...}:
+# TODO: move into modules/users.nix
+{ config, lib, pkgs, ...}:
 
 let
-  # ProgramConfig -> { "<mime-type>" = { priority, desktop }; }
+  # [ ProgramConfig ]
-  weightedMimes = prog: builtins.mapAttrs (_key: desktop: { priority = prog.mime.priority; desktop = desktop; }) prog.mime.associations;
+  enabledPrograms = builtins.filter
-  # [ { "<mime-type>" = { priority, desktop } ]; } ] -> { "<mime-type>" = [ { priority, desktop } ... ]; }
+    (p: p.enabled)
-  mergeMimes = mimes: lib.foldAttrs (item: acc: [item] ++ acc) [] mimes;
+    (builtins.attrValues config.sane.programs);
-  # [ { priority, desktop } ... ] -> Self
-  sortOneMimeType = associations: builtins.sort (l: r: assert l.priority != r.priority; l.priority < r.priority) associations;
-  sortMimes = mimes: builtins.mapAttrs (_k: sortOneMimeType) mimes;
-  removePriorities = mimes: builtins.mapAttrs (_k: associations: builtins.map (a: a.desktop) associations) mimes;
 
   # [ ProgramConfig ]
-  enabledPrograms = builtins.filter (p: p.enabled) (builtins.attrValues config.sane.programs);
+  enabledProgramsWithPackage = builtins.filter (p: p.package != null) enabledPrograms;
+
   # [ { "<mime-type>" = { prority, desktop } ]
   enabledWeightedMimes = builtins.map weightedMimes enabledPrograms;
+
+  # ProgramConfig -> { "<mime-type>" = { priority, desktop }; }
+  weightedMimes = prog: builtins.mapAttrs
+    (_key: desktop: {
+      priority = prog.mime.priority; desktop = desktop;
+    })
+    prog.mime.associations;
+
+  # [ { "<mime-type>" = { priority, desktop } ]; } ] -> { "<mime-type>" = [ { priority, desktop } ... ]; }
+  mergeMimes = mimes: lib.foldAttrs (item: acc: [item] ++ acc) [] mimes;
+
+  # [ { priority, desktop } ... ] -> Self
+  sortOneMimeType = associations: builtins.sort
+    (l: r: lib.throwIf
+      (l.priority == r.priority)
+      "${l.desktop} and ${r.desktop} share a preferred mime type with identical priority ${builtins.toString l.priority} (and so the desired association is ambiguous)"
+      (l.priority < r.priority)
+    )
+    associations;
+  sortMimes = mimes: builtins.mapAttrs (_k: sortOneMimeType) mimes;
+  # { "<mime-type>"} = [ { priority, desktop } ... ]; } -> { "<mime-type>" = [ "<desktop>" ... ]; }
+  removePriorities = mimes: builtins.mapAttrs
+    (_k: associations: builtins.map (a: a.desktop) associations)
+    mimes;
+  # { "<mime-type>" = [ "<desktop>" ... ]; } -> { "<mime-type>" = "<desktop1>;<desktop2>;..."; }
+  formatDesktopLists = mimes: builtins.mapAttrs
+    (_k: desktops: lib.concatStringsSep ";" desktops)
+    mimes;
+
+  mimeappsListPkg = pkgs.writeTextDir "share/applications/mimeapps.list" (
+    lib.generators.toINI { } {
+      "Default Applications" = formatDesktopLists (removePriorities (sortMimes (mergeMimes enabledWeightedMimes)));
+    }
+  );
+
+  localShareApplicationsPkg = (pkgs.symlinkJoin {
+    name = "user-local-share-applications";
+    paths = builtins.map
+      (p: "${p.package}")
+      (enabledProgramsWithPackage ++ [ { package=mimeappsListPkg; } ]);
+  }).overrideAttrs (orig: {
+    # like normal symlinkJoin, but don't error if the path doesn't exist
+    buildCommand = ''
+      mkdir -p $out/share/applications
+      for i in $(cat $pathsPath); do
+        if [ -e "$i/share/applications" ]; then
+          ${pkgs.buildPackages.xorg.lndir}/bin/lndir -silent $i/share/applications $out/share/applications
+        fi
+      done
+      runHook postBuild
+    '';
+    postBuild = ''
+      # rebuild `mimeinfo.cache`, used by file openers to show the list of *all* apps, not just the user's defaults.
+      ${pkgs.buildPackages.desktop-file-utils}/bin/update-desktop-database $out/share/applications
+    '';
+  });
+
 in
 {
   # the xdg mime type for a file can be found with:
   # - `xdg-mime query filetype path/to/thing.ext`
   # the default handler for a mime type can be found with:
   # - `xdg-mime query default <mimetype>`  (e.g. x-scheme-handler/http)
+  # the nix-configured handler can be found `nix-repl > :lf . > hostConfigs.desko.xdg.mime.defaultApplications`
+  #
+  # glib/gio is queried via glib.bin output:
+  # - `gio mime x-scheme-handler/https`
+  # - `gio open <path_or_url>`
+  # - `gio launch </path/to/app.desktop>`
   #
   # we can have single associations or a list of associations.
   # there's also options to *remove* [non-default] associations from specific apps
-  xdg.mime.enable = true;
+  # N.B.: don't use nixos' `xdg.mime` option becaue that caues `/share/applications` to be linked into the whole system,
-  xdg.mime.defaultApplications = removePriorities (sortMimes (mergeMimes enabledWeightedMimes));
+  # which limits what i can do around sandboxing. getting the default associations to live in ~/ makes it easier to expose
+  # the associations to apps selectively.
+  # xdg.mime.enable = true;
+  # xdg.mime.defaultApplications = removePriorities (sortMimes (mergeMimes enabledWeightedMimes));
+
+  sane.user.fs.".local/share/applications".symlink.target = "${localShareApplicationsPkg}/share/applications";
 }

hosts/common/hosts.nix

@@ -19,7 +19,7 @@
   };
 
   sane.hosts.by-name."moby" = {
-    ssh.authorized = lib.mkDefault false;  # moby's too easy to hijack: don't let it ssh places
+    # ssh.authorized = lib.mkDefault false;  # moby's too easy to hijack: don't let it ssh places
     ssh.user_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICrR+gePnl0nV/vy7I5BzrGeyVL+9eOuXHU1yNE3uCwU";
     ssh.host_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO1N/IT3nQYUD+dBlU1sTEEVMxfOyMkrrDeyHcYgnJvw";
     wg-home.pubkey = "I7XIR1hm8bIzAtcAvbhWOwIAabGkuEvbWH/3kyIB1yA=";
@@ -36,10 +36,4 @@
     wg-home.endpoint = "uninsane.org:51820";
     lan-ip = "10.78.79.51";
   };
-
-  sane.hosts.by-name."supercap" = {
-    ssh.authorized = false;
-    ssh.host_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHf/mqqkX45EWAcquV04MC3SUljTApdclH1gjI19F+PA";
-    lan-ip = "10.78.79.232";
-  };
 }

hosts/common/ids.nix

@@ -53,6 +53,12 @@
   sane.ids.monero.gid = 2416;
   sane.ids.slskd.uid = 2417;
   sane.ids.slskd.gid = 2417;
+  sane.ids.bitcoind-mainnet.uid = 2418;
+  sane.ids.bitcoind-mainnet.gid = 2418;
+  sane.ids.clightning.uid = 2419;
+  sane.ids.clightning.gid = 2419;
+  sane.ids.nix-serve.uid = 2420;
+  sane.ids.nix-serve.gid = 2420;
 
   sane.ids.colin.uid = 1000;
   sane.ids.guest.uid = 1100;

hosts/common/net/default.nix

@@ -1,6 +1,30 @@
 { lib, ... }:
 
 {
+  imports = [
+    ./dns.nix
+    ./hostnames.nix
+    ./upnp.nix
+    ./vpn.nix
+  ];
+
+  systemd.network.enable = true;
+  networking.useNetworkd = true;
+
+  # view refused/dropped packets with: `sudo journalctl -k`
+  # networking.firewall.logRefusedPackets = true;
+  # networking.firewall.logRefusedUnicastsOnly = false;
+  networking.firewall.logReversePathDrops = true;
+  # linux will drop inbound packets if it thinks a reply to that packet wouldn't exit via the same interface (rpfilter).
+  #   that heuristic fails for complicated VPN-style routing, especially with SNAT.
+  # networking.firewall.checkReversePath = false;  # or "loose" to keep it partially.
+  # networking.firewall.enable = false;  #< set false to debug
+
+  # this is needed to forward packets from the VPN to the host.
+  # this is required separately by servo and by any `sane-vpn` users,
+  # however Nix requires this be set centrally, in only one location (i.e. here)
+  boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
+
   # the default backend is "wpa_supplicant".
   # wpa_supplicant reliably picks weak APs to connect to.
   # see: <https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/474>
@@ -35,10 +59,6 @@
   # e.g. openconnect drags in webkitgtk (for SSO)!
   networking.networkmanager.plugins = lib.mkForce [];
 
-  networking.firewall.allowedUDPPorts = [
-    1900  # to received UPnP advertisements. required by sane-ip-check-upnp
-  ];
-
   # keyfile.path = where networkmanager should look for connection credentials
   networking.networkmanager.extraConfig = ''
     [keyfile]

hosts/common/net/dns.nix

@@ -0,0 +1,67 @@
+# things to consider when changing these parameters:
+# - temporary VPN access (`sane-vpn up ...`)
+# - servo `ovpns` namespace (it *relies* on /etc/resolv.conf mentioning 127.0.0.53)
+# - jails: `firejail --net=br-ovpnd-us --noprofile --dns=46.227.67.134 ping 1.1.1.1`
+#
+# components:
+# - /etc/nsswitch.conf:
+#   - glibc uses this to provide `getaddrinfo`, i.e. host -> ip address lookup
+#     call directly with `getent ahostsv4 www.google.com`
+#   - `nss` (a component of glibc) is modular: names mentioned in that file are `dlopen`'d (i think that's the mechanism)
+#     in NixOS, that means _they have to be on LDPATH_.
+#   - `nscd` is used by NixOS simply to proxy nss requests.
+#      here, /etc/nsswitch.conf consumers contact nscd via /var/run/nscd/socket.
+#      in this way, only `nscd` needs to have the nss modules on LDPATH.
+# - /etc/resolv.conf
+#   - contains the DNS servers for a system.
+#   - historically, NetworkManager would update this file as you switch networks.
+#   - modern implementations hardcodes `127.0.0.53` and then systemd-resolved proxies everything (and caches).
+#
+# namespacing:
+# - each namespace can use a different /etc/resolv.conf to specify different DNS servers (see `firejail --dns=...`)
+# - nscd breaks namespacing: the host nscd is unaware of the guest's /etc/resolv.conf, and so direct's the guest's DNS requests to the host's servers.
+#   - this is fixed by either `firejail --blacklist=/var/run/nscd/socket`, or disabling nscd altogether.
+{ lib, ... }:
+{
+  # use systemd's stub resolver.
+  # /etc/resolv.conf isn't sophisticated enough to use different servers per net namespace (or link).
+  # instead, running the stub resolver on a known address in the root ns lets us rewrite packets
+  # in servo's ovnps namespace to use the provider's DNS resolvers.
+  # a weakness is we can only query 1 NS at a time (unless we were to clone the packets?)
+  # TODO: rework servo's netns to use `firejail`, which is capable of spoofing /etc/resolv.conf.
+  services.resolved.enable = true;  #< to disable, set ` = lib.mkForce false`, as other systemd features default to enabling `resolved`.
+  # without DNSSEC:
+  # - dig matrix.org => works
+  # - curl https://matrix.org => works
+  # with default DNSSEC:
+  # - dig matrix.org => works
+  # - curl https://matrix.org => fails
+  # i don't know why. this might somehow be interfering with the DNS run on this device (trust-dns)
+  services.resolved.dnssec = "false";
+  networking.nameservers = [
+    # use systemd-resolved resolver
+    # full resolver (which understands /etc/hosts) lives on 127.0.0.53
+    # stub resolver (just forwards upstream) lives on 127.0.0.54
+    "127.0.0.53"
+  ];
+
+  # nscd -- the Name Service Caching Daemon -- caches DNS query responses
+  # in a way that's unaware of my VPN routing, so routes are frequently poor against
+  # services which advertise different IPs based on geolocation.
+  # nscd claims to be usable without a cache, but in practice i can't get it to not cache!
+  # nsncd is the Name Service NON-Caching Daemon. it's a drop-in that doesn't cache;
+  # this is OK on the host -- because systemd-resolved caches. it's probably sub-optimal
+  # in the netns and we query upstream DNS more often than needed. hm.
+  # services.nscd.enableNsncd = true;
+
+  # disabling nscd LOSES US SOME FUNCTIONALITY. in particular, only the glibc-builtin modules are accessible via /etc/resolv.conf.
+  # - dns: glibc-bultin
+  # - files: glibc-builtin
+  # - myhostname: systemd
+  # - mymachines: systemd
+  # - resolve: systemd
+  # in practice, i see no difference with nscd disabled.
+  # disabling nscd VASTLY simplifies netns and process isolation. see explainer at top of file.
+  services.nscd.enable = false;
+  system.nssModules = lib.mkForce [];
+}

hosts/common/net/hostnames.nix

@@ -1,4 +1,3 @@
-# TODO: move to hosts/common/
 { config, lib, ... }:
 
 {

hosts/common/net/upnp.nix

@@ -0,0 +1,20 @@
+{ pkgs, ... }:
+{
+  networking.firewall.allowedUDPPorts = [
+    # to receive UPnP advertisements. required by sane-ip-check.
+    # N.B. sane-ip-check isn't query/response based. it needs to receive on port 1900 -- not receive responses FROM port 1900.
+    1900
+  ];
+
+  networking.firewall.extraCommands = with pkgs; ''
+    # after an outgoing SSDP query to the multicast address, open FW for incoming responses.
+    # necessary for anything DLNA, especially go2tv
+    # source: <https://serverfault.com/a/911286>
+    # context: <https://github.com/alexballas/go2tv/issues/72>
+
+    # ipset -! means "don't fail if set already exists"
+    ${ipset}/bin/ipset create -! upnp hash:ip,port timeout 10
+    ${iptables}/bin/iptables -A OUTPUT -d 239.255.255.250/32 -p udp -m udp --dport 1900 -j SET --add-set upnp src,src --exist
+    ${iptables}/bin/iptables -A INPUT -p udp -m set --match-set upnp dst,dst -j ACCEPT
+  '';
+}

hosts/common/net/vpn.nix

@@ -0,0 +1,56 @@
+# to add a new OVPN VPN:
+# - generate a privkey `wg genkey`
+# - add this key to `sops secrets/universal.yaml`
+# - upload pubkey to OVPN.com (`cat wg.priv | wg pubkey`)
+# - generate config @ OVPN.com
+# - copy the Address, PublicKey, Endpoint from OVPN's config
+
+{ config, lib, pkgs, ... }:
+let
+  def-ovpn = name: { endpoint, publicKey, addrV4, id }: {
+    sane.vpn."ovpnd-${name}" = {
+      inherit endpoint publicKey addrV4 id;
+      privateKeyFile = config.sops.secrets."wg/ovpnd_${name}_privkey".path;
+      dns = [
+        "46.227.67.134"
+        "192.165.9.158"
+      ];
+    };
+
+    sops.secrets."wg/ovpnd_${name}_privkey" = {
+      # needs to be readable by systemd-network or else it says "Ignoring network device" and doesn't expose it to networkctl.
+      owner = "systemd-network";
+    };
+  };
+in lib.mkMerge [
+  (def-ovpn "us" {
+    endpoint = "vpn31.prd.losangeles.ovpn.com:9929";
+    publicKey = "VW6bEWMOlOneta1bf6YFE25N/oMGh1E1UFBCfyggd0k=";
+    id = 1;
+    addrV4 = "172.27.237.218";
+    # addrV6 = "fd00:0000:1337:cafe:1111:1111:ab00:4c8f";
+  })
+  # TODO: us-atl disabled until i can give it a different link-local address and wireguard key than us-mi
+  # (def-ovpn "us-atl" {
+  #   endpoint = "vpn18.prd.atlanta.ovpn.com:9929";
+  #   publicKey = "Dpg/4v5s9u0YbrXukfrMpkA+XQqKIFpf8ZFgyw0IkE0=";
+  #   address = [
+  #     "172.21.182.178/32"
+  #     "fd00:0000:1337:cafe:1111:1111:cfcb:27e3/128"
+  #   ];
+  # })
+  (def-ovpn "us-mi" {
+    endpoint = "vpn34.prd.miami.ovpn.com:9929";
+    publicKey = "VtJz2irbu8mdkIQvzlsYhU+k9d55or9mx4A2a14t0V0=";
+    id = 2;
+    addrV4 = "172.21.182.178";
+    # addrV6 = "fd00:0000:1337:cafe:1111:1111:cfcb:27e3";
+  })
+  (def-ovpn "ukr" {
+    endpoint = "vpn96.prd.kyiv.ovpn.com:9929";
+    publicKey = "CjZcXDxaaKpW8b5As1EcNbI6+42A6BjWahwXDCwfVFg=";
+    id = 3;
+    addrV4 = "172.18.180.159";
+    # addrV6 = "fd00:0000:1337:cafe:1111:1111:ec5c:add3";
+  })
+]

hosts/common/nix-path/default.nix

@@ -1,16 +0,0 @@
-{ pkgs, sane-lib, ... }:
-
-{
-  # allow `nix-shell` (and probably nix-index?) to locate our patched and custom packages
-  nix.nixPath = [
-    "nixpkgs=${pkgs.path}"
-    # note the import starts at repo root: this allows `./overlay/default.nix` to access the stuff at the root
-    # "nixpkgs-overlays=${../../..}/hosts/common/nix-path/overlay"
-    # as long as my system itself doesn't rely on NIXPKGS at runtime, we can point the overlays to git
-    # to avoid switching so much during development
-    "nixpkgs-overlays=/home/colin/dev/nixos/hosts/common/nix-path/overlay"
-  ];
-
-  # ensure new deployments have a source of this repo with which they can bootstrap.
-  environment.etc."nixos".source = ../../..;
-}

hosts/common/nix/default.nix

@@ -0,0 +1,84 @@
+{ config, lib, pkgs, ... }:
+
+{
+  nix.settings = {
+    # see: `man nix.conf`
+
+    # useful when a remote builder has a faster internet connection than me.
+    # note that this also applies to `nix copy --to`, though.
+    # i think any time a remote machine wants a path, this means we ask them to try getting it themselves before we supply it.
+    builders-use-substitutes = true;  # default: false
+
+    # maximum seconds to wait when connecting to binary substituter
+    connect-timeout = 3;  # default: 0
+
+    # download-attempts = 5;  # default: 5
+
+    # allow `nix flake ...` command
+    experimental-features = [ "nix-command" "flakes "];
+
+    # whether to build from source when binary substitution fails
+    fallback = true;  # default: false
+
+    # whether to keep building dependencies if any other one fails
+    keep-going = true;  # default: false
+
+    # whether to keep build-only dependencies of GC roots (e.g. C compiler) when doing GC
+    keep-outputs = true;  # default: false
+
+    # how many lines to show from failed build
+    log-lines = 30;  # default: 10
+
+    # how many substitution downloads to perform in parallel.
+    # i wonder if parallelism is causing moby's substitutions to fail?
+    max-substitution-jobs = 6;  # default: 16
+
+    # narinfo-cache-negative-ttl = 3600  # default: 3600
+    # whether to use ~/.local/state/nix/profile instead of ~/.nix-profile, etc
+    use-xdg-base-directories = true;  # default: false
+
+    # whether to warn if repository has uncommited changes
+    warn-dirty = false;  # default: true
+
+    # hardlinks identical files in the nix store to save 25-35% disk space.
+    # unclear _when_ this occurs. it's not a service.
+    # does the daemon continually scan the nix store?
+    # does the builder use some content-addressed db to efficiently dedupe?
+    auto-optimise-store = true;
+
+    # allow #!nix-shell scripts to locate my patched nixpkgs & custom packages.
+    # this line might become unnecessary: see <https://github.com/NixOS/nixpkgs/pull/273170>
+    nix-path = config.nix.nixPath;
+  };
+
+  # allow `nix-shell` (and probably nix-index?) to locate our patched and custom packages.
+  # this is actually a no-op, and the real action happens in assigning `nix.settings.nix-path`.
+  nix.nixPath = [
+    "nixpkgs=${pkgs.path}"
+    # note the import starts at repo root: this allows `./overlay/default.nix` to access the stuff at the root
+    # "nixpkgs-overlays=${../../..}/hosts/common/nix-path/overlay"
+    # as long as my system itself doesn't rely on NIXPKGS at runtime, we can point the overlays to git
+    # to avoid switching so much during development
+    "nixpkgs-overlays=/home/colin/dev/nixos/hosts/common/nix/overlay"
+  ];
+
+  # ensure new deployments have a source of this repo with which they can bootstrap.
+  environment.etc."nixos".source = ../../..;
+
+  systemd.services.nix-daemon.serviceConfig = {
+    # the nix-daemon manages nix builders
+    # kill nix-daemon subprocesses when systemd-oomd detects an out-of-memory condition
+    # see:
+    # - nixos PR that enabled systemd-oomd: <https://github.com/NixOS/nixpkgs/pull/169613>
+    # - systemd's docs on these properties: <https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#ManagedOOMSwap=auto%7Ckill>
+    #
+    # systemd's docs warn that without swap, systemd-oomd might not be able to react quick enough to save the system.
+    # see `man oomd.conf` for further tunables that may help.
+    #
+    # alternatively, apply this more broadly with `systemd.oomd.enableSystemSlice = true` or `enableRootSlice`
+    # TODO: also apply this to the guest user's slice (user-1100.slice)
+    # TODO: also apply this to distccd
+    ManagedOOMMemoryPressure = "kill";
+    ManagedOOMSwap = "kill";
+  };
+}

hosts/common/persist.nix

@@ -1,13 +1,14 @@
 { ... }:
 
 {
-  sane.persist.stores.private.origin = "/home/colin/private";
+  # store /home/colin/a/b in /mnt/persist/private/a/b instead of /mnt/persist/private/home/colin/a/b
-  # store /home/colin/a/b in /home/private/a/b instead of /home/private/home/colin/a/b
   sane.persist.stores.private.prefix = "/home/colin";
 
+  sane.persist.sys.byStore.initrd = [
+    "/var/log"
+  ];
   sane.persist.sys.byStore.plaintext = [
     # TODO: these should be private.. somehow
-    "/var/log"
     "/var/backup"  # for e.g. postgres dumps
   ];
   sane.persist.sys.byStore.cryptClearOnBoot = [

hosts/common/polyunfill.nix

@@ -0,0 +1,45 @@
+# strictly *decrease* the scope of the default nixos installation/config
+
+{ lib, ... }:
+{
+  # disable non-required packages like nano, perl, rsync, strace
+  environment.defaultPackages = [];
+
+  # remove all the non-existent default directories from XDG_DATA_DIRS, XDG_CONFIG_DIRS to simplify debugging.
+  # this is defaulted in <repo:nixos/nixpkgs:nixos/modules/programs/environment.nix>,
+  # without being gated by any higher config.
+  environment.profiles = lib.mkForce [
+    "/etc/profiles/per-user/$USER"
+    "/run/current-system/sw"
+  ];
+
+  # NIXPKGS_CONFIG defaults to "/etc/nix/nixpkgs-config.nix", for idfk why.
+  # that's never existed on my system and everything does fine without it set empty (no nixpkgs API to forcibly *unset* it).
+  environment.variables.NIXPKGS_CONFIG = lib.mkForce "";
+  # XDG_CONFIG_DIRS defaults to "/etc/xdg", which doesn't exist.
+  # in practice, pam appends the values i want to XDG_CONFIG_DIRS, though this approach causes an extra leading `:`
+  environment.sessionVariables.XDG_CONFIG_DIRS = lib.mkForce [];
+  # XCURSOR_PATH: defaults to `[ "$HOME/.icons" "$HOME/.local/share/icons" ]`, neither of which i use, just adding noise.
+  # see: <repo:nixos/nixpkgs:nixos/modules/config/xdg/icons.nix>
+  environment.sessionVariables.XCURSOR_PATH = lib.mkForce [];
+
+  # disable nixos' portal module, otherwise /share/applications gets linked into the system and complicates things (sandboxing).
+  # instead, i manage portals myself via the sane.programs API (e.g. sane.programs.xdg-desktop-portal).
+  xdg.portal.enable = false;
+  xdg.menus.enable = false;  #< links /share/applications, and a bunch of other empty (i.e. unused) dirs
+
+  # xdg.autostart.enable defaults to true, and links /etc/xdg/autostart into the environment, populated with .desktop files.
+  # see: <repo:nixos/nixpkgs:nixos/modules/config/xdg/autostart.nix>
+  # .desktop files are a questionable way to autostart things: i generally prefer a service manager for that.
+  xdg.autostart.enable = false;
+
+  # nix.channel.enable: populates `/nix/var/nix/profiles/per-user/root/channels`, `/root/.nix-channels`, `$HOME/.nix-defexpr/channels`
+  #   <repo:nixos/nixpkgs:nixos/modules/config/nix-channel.nix>
+  # TODO: may want to recreate NIX_PATH, nix.settings.nix-path
+  nix.channel.enable = false;
+
+  # environment.stub-ld: populate /lib/ld-linux.so with an object that unconditionally errors on launch,
+  # so as to inform when trying to run a non-nixos binary?
+  # IMO that's confusing: i thought /lib/ld-linux.so was some file actually required by nix.
+  environment.stub-ld.enable = false;
+}

hosts/common/programs/abaddon.nix

@@ -10,26 +10,18 @@ in
       type = types.submodule {
         options.autostart = mkOption {
           type = types.bool;
-          default = true;
+          default = false;
         };
       };
     };
 
-    package = pkgs.abaddon.overrideAttrs (upstream: {
+    packageUnwrapped = pkgs.abaddon.overrideAttrs (upstream: {
       patches = (upstream.patches or []) ++ [
         (pkgs.fetchpatch {
           url = "https://git.uninsane.org/colin/abaddon/commit/eb551f188d34679f75adcbc83cb8d5beb4d19fd6.patch";
           name = ''"view members" default to false'';
           hash = "sha256-9BX8iO86CU1lNrKS1G2BjDR+3IlV9bmhRNTsLrxChwQ=";
         })
-        (pkgs.fetchpatch {
-          # this makes it so Abaddon reports its app_name in notifications.
-          # not 100% necessary; just a nice-to-have. maybe don't rely on it until it's merged upstream.
-          # upstream PR: <https://github.com/uowuo/abaddon/pull/247>
-          url = "https://git.uninsane.org/colin/abaddon/commit/18cd863fdbb5e6b1e9aaf9394dbd673d51839f30.patch";
-          name = "set glib application name";
-          hash = "sha256-IFYxf1D8hIsxgZehGd6hL3zJiBkPZfWGm+Faaa5ZFl4=";
-        })
       ];
     });
 
@@ -95,7 +87,7 @@ in
 
     services.abaddon = {
       description = "unofficial Discord chat client";
-      wantedBy = lib.mkIf cfg.config.autostart [ "default.target" ];
+      wantedBy = lib.mkIf cfg.config.autostart [ "graphical-session.target" ];
       serviceConfig = {
         ExecStart = "${cfg.package}/bin/abaddon";
         Type = "simple";

hosts/common/programs/aerc.nix

@@ -3,6 +3,9 @@
 
 {
   sane.programs.aerc = {
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "inplace";
+    sandbox.net = "clearnet";
     secrets.".config/aerc/accounts.conf" = ../../../secrets/common/aerc_accounts.conf.bin;
     mime.associations."x-scheme-handler/mailto" = "aerc.desktop";
   };

hosts/common/programs/alacritty.nix

@@ -6,19 +6,36 @@
 { lib, ... }:
 {
   sane.programs.alacritty = {
+    sandbox.enable = false;
     env.TERMINAL = lib.mkDefault "alacritty";
-    # note: alacritty will switch to .toml config in 13.0 release
+    fs.".config/alacritty/alacritty.toml".symlink.text = ''
-    # - run `alacritty migrate` to convert the yaml to toml
+      [font]
-    fs.".config/alacritty/alacritty.yml".symlink.text = ''
+      size = 14
-      font:
-        size: 14
 
-      key_bindings:
+      [[keyboard.bindings]]
-        - { key: N,         mods: Control,        action: CreateNewWindow }
+      mods = "Control"
-        - { key: PageUp,    mods: Control,        action: ScrollPageUp }
+      key = "N"
-        - { key: PageDown,  mods: Control,        action: ScrollPageDown }
+      action = "CreateNewWindow"
-        - { key: PageUp,    mods: Control|Shift,  action: ScrollPageUp }
+
-        - { key: PageDown,  mods: Control|Shift,  action: ScrollPageDown }
+      [[keyboard.bindings]]
+      mods = "Control"
+      key = "PageUp"
+      action = "ScrollPageUp"
+
+      [[keyboard.bindings]]
+      mods = "Control"
+      key = "PageDown"
+      action = "ScrollPageDown"
+
+      [[keyboard.bindings]]
+      mods = "Control|Shift"
+      key = "PageUp"
+      action = "ScrollPageUp"
+
+      [[keyboard.bindings]]
+      mods = "Control|Shift"
+      key = "PageDown"
+      action = "ScrollPageDown"
     '';
   };
 }

hosts/common/programs/animatch.nix

@@ -1,10 +1,42 @@
-{ ... }:
+# debug with:
+# - `animatch --debug`
+# - `gdb animatch`
+# try:
+# - `animatch --fullscreen`
+# - `animatch --windowed`
+# the other config options (e.g. verbose logging -- which doesn't seem to do anything) have to be configured via .ini file
+# ```ini
+# # ~/.config/Holy Pangolin/Animatch/SuperDerpy.ini
+# [SuperDerpy]
+# debug=1
+# disableTouch=1
+# [game]
+# verbose=1
+# ```
+{ pkgs, ... }:
 {
   sane.programs.animatch = {
+    packageUnwrapped = with pkgs; animatch.override {
+      # allegro has no native wayland support, and so by default crashes when run without Xwayland.
+      # enable the allegro SDL backend, and achieve Wayland support via SDL's Wayland support.
+      # TODO: see about upstreaming this to nixpkgs?
+      allegro5 = allegro5.overrideAttrs (upstream: {
+        buildInputs = upstream.buildInputs ++ [
+          SDL2
+        ];
+        cmakeFlags = upstream.cmakeFlags ++ [
+          "-DALLEGRO_SDL=on"
+        ];
+      });
+    };
+
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "wrappedDerivation";
+    sandbox.whitelistWayland = true;
+
     persist.byStore.plaintext = [
-      # game progress
+      # ".config/Holy Pangolin/Animatch"  #< used for SuperDerpy config (e.g. debug, disableTouch, fullscreen, enable sound, etc). SuperDerpy.ini
-      ".config/Holy Pangolin/Animatch"
+      ".local/share/Holy Pangolin/Animatch"  #< used for game state (level clears). SuperDerpy.ini
-      ".local/share/Holy Pangolin/Animatch"  # i think this one might be wrong
     ];
   };
 }

hosts/common/programs/audacity.nix

@@ -0,0 +1,35 @@
+{ pkgs, ... }:
+{
+  sane.programs.audacity = {
+    packageUnwrapped = pkgs.audacity.override {
+      # wxGTK32 uses webkitgtk-4.0.
+      # audacity doesn't actually need webkit though, so diable to reduce closure
+      wxGTK32 = pkgs.wxGTK32.override {
+        withWebKit = false;
+      };
+    };
+
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "wrappedDerivation";
+    sandbox.whitelistAudio = true;
+    sandbox.whitelistWayland = true;
+    sandbox.autodetectCliPaths = true;
+    sandbox.extraHomePaths = [
+      # support media imports via file->open dir to some common media directories
+      "tmp"
+      "Music"
+      # audacity needs the entire config dir mounted if running in a sandbox
+      ".config/audacity"
+    ];
+
+    # disable first-run splash screen
+    fs.".config/audacity/audacity.cfg".file.text = ''
+      PrefsVersion=1.1.1r1
+      [GUI]
+      ShowSplashScreen=0
+      [Version]
+      Major=3
+      Minor=4
+    '';
+  };
+}

hosts/common/programs/bemenu.nix

@@ -87,7 +87,14 @@ let
 in
 {
   sane.programs.bemenu = {
-    package = pkgs.bemenu.overrideAttrs (upstream: {
+    sandbox.method = "bwrap";  # landlock works, but requires *all* of /run/user/$ID to be granted.
+    sandbox.wrapperType = "wrappedDerivation";
+    sandbox.whitelistWayland = true;
+    sandbox.extraHomePaths = [
+      ".cache/fontconfig"  #< else it complains, and is *way* slower
+    ];
+
+    packageUnwrapped = pkgs.bemenu.overrideAttrs (upstream: {
       nativeBuildInputs = (upstream.nativeBuildInputs or []) ++ [
         pkgs.makeWrapper
       ];

hosts/common/programs/bonsai.nix

@@ -1,7 +1,7 @@
 # bonsai docs: <https://sr.ht/~stacyharper/bonsai/>
 { config, lib, pkgs, ... }:
 let
-  cfg = config.sane.gui.sxmo.bonsaid;
+  cfg = config.sane.programs.bonsai;
 
   delayType = with lib; types.submodule {
     options = {
@@ -88,74 +88,43 @@ let
         mergeOneOption loc defs
     ;
   };
-
-  # transitionType = with lib; types.submodule {
-  #   options = {
-  #     type = mkOption {
-  #       type = types.enum [ "delay" "event" "exec" ];
-  #     };
-  #     delay_duration = mkOption {
-  #       type = types.nullOr types.int;
-  #       default = null;
-  #       description = ''
-  #         used for "delay" types only.
-  #         nanoseconds until the event is finalized.
-  #       '';
-  #     };
-  #     event_name = mkOption {
-  #       type = types.nullOr types.str;
-  #       default = null;
-  #       description = ''
-  #         name of event which this transition applies to.
-  #       '';
-  #     };
-  #     transitions = mkOption {
-  #       type = types.nullOr (types.listOf transitionType);
-  #       default = null;
-  #       description = ''
-  #         list of transitions out of this state.
-  #       '';
-  #     };
-  #     command = mkOption {
-  #       type = types.nullOr (types.listOf types.str);
-  #       default = null;
-  #       description = ''
-  #         used for "exec" types only.
-  #         command to run when the event is triggered.
-  #       '';
-  #     };
-  #   };
-  # };
 in
 {
-  options = with lib; {
+  sane.programs.bonsai = {
-    sane.gui.sxmo.bonsaid.package = mkOption {
+    configOption = with lib; mkOption {
-      type = types.package;
+      default = {};
-      default = pkgs.bonsai;
+      type = types.submodule {
+        options = {
+          transitions = mkOption {
+            type = types.listOf transitionType;
+            default = [];
+          };
+          configFile = mkOption {
+            type = types.path;
+            default = pkgs.writeText "bonsai_tree.json" (builtins.toJSON cfg.config.transitions);
+            description = ''
+              configuration file to pass to bonsai.
+              usually auto-generated from the sibling options; exposed mainly for debugging or convenience.
+            '';
+          };
+        };
+      };
     };
-    sane.gui.sxmo.bonsaid.transitions = mkOption {
+
-      type = types.listOf transitionType;
+    services.bonsaid = {
-      default = [];
+      description = "bonsai: programmable input dispatcher";
-    };
+      after = [ "graphical-session.target" ];
-    sane.gui.sxmo.bonsaid.configFile = mkOption {
+      wantedBy = [ "graphical-session.target" ];
-      type = types.path;
+
-      default = pkgs.writeText "bonsai_tree.json" (builtins.toJSON cfg.transitions);
-      description = ''
-        configuration file to pass to bonsai.
-        usually auto-generated from the sibling options; exposed mainly for debugging or convenience.
-      '';
-    };
-  };
-  config = lib.mkIf config.sane.gui.sxmo.enable {
-    sane.user.services.bonsaid = {
-      description = "programmable input dispatcher";
       script = ''
         ${pkgs.coreutils}/bin/rm -f $XDG_RUNTIME_DIR/bonsai
-        exec ${cfg.package}/bin/bonsaid -t ${cfg.configFile}
+        exec ${cfg.package}/bin/bonsaid -t ${cfg.config.configFile}
       '';
-      serviceConfig.Type = "simple";
+      serviceConfig = {
-      serviceConfig.Restart = "always";
+        Type = "simple";
-      serviceConfig.RestartSec = "5s";
+        Restart = "always";
+        RestartSec = "5s";
+      };
     };
   };
 }

hosts/common/programs/brave.nix

@@ -1,6 +1,17 @@
 { ... }:
 {
   sane.programs.brave = {
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "inplace";  # /opt/share/brave.com vendor-style packaging
+    sandbox.net = "all";
+    sandbox.extraHomePaths = [
+      "dev"  # for developing anything web-related
+      "tmp"
+    ];
+    sandbox.whitelistAudio = true;
+    sandbox.whitelistDri = true;
+    sandbox.whitelistWayland = true;
+
     persist.byStore.cryptClearOnBoot = [
       ".cache/BraveSoftware"
       ".config/BraveSoftware"

hosts/common/programs/bubblewrap.nix

@@ -0,0 +1,35 @@
+{ pkgs, ... }:
+{
+  sane.programs.bubblewrap = {
+    sandbox.enable = false;  # don't sandbox the sandboxer :)
+    packageUnwrapped = pkgs.bubblewrap.overrideAttrs (base: {
+      # patches = (base.patches or []) ++ [
+      #   (pkgs.fetchpatch {
+      #     url = "https://git.uninsane.org/colin/bubblewrap/commit/9843f9b2b5f086563fd37250658d69350a2939be.patch";
+      #     name = "enable debug logging and add a bunch more tracing";
+      #     hash = "sha256-AlDsqddaBahhqGibZlCjgmChuK7mmxDt0aYHNgY05OI=";
+      #   })
+      # ];
+      postPatch = (base.postPatch or "") + ''
+        # bwrap doesn't like to be invoked with any capabilities, which is troublesome if i
+        # want to do things like ship CAP_NET_ADMIN,CAP_NET_RAW in the ambient set for tools like Wireshark.
+        # but this limitation of bwrap is artificial and at first look is just a scenario the author probably
+        # never expected: patch out the guard check.
+        #
+        # see: <https://github.com/containers/bubblewrap/issues/397>
+        #
+        # note that invoking bwrap with capabilities in the 'init' namespace does NOT grant the sandboxed process
+        # capabilities in the 'init' namespace. it's a limitation of namespaces that namespaced processes can
+        # never receive capabilities in their parent namespace.
+        substituteInPlace bubblewrap.c --replace \
+          'die ("Unexpected capabilities but not setuid, old file caps config?");' \
+          '// die ("Unexpected capabilities but not setuid, old file caps config?");'
+
+        # enable debug printing
+        # substituteInPlace utils.h --replace \
+        #   '#define __debug__(x)' \
+        #   '#define __debug__(x) printf x'
+      '';
+    });
+  };
+}

hosts/common/programs/calls.nix

@@ -44,7 +44,7 @@ in
     services.gnome-calls = {
       # TODO: prevent gnome-calls from daemonizing when started manually
       description = "gnome-calls daemon to monitor incoming SIP calls";
-      wantedBy = lib.mkIf cfg.config.autostart [ "default.target" ];
+      wantedBy = lib.mkIf cfg.config.autostart [ "graphical-session.target" ];
       serviceConfig = {
         # add --verbose for more debugging
         ExecStart = "${cfg.package}/bin/gnome-calls --daemon";

hosts/common/programs/catt.nix

@@ -0,0 +1,29 @@
+# use like:
+# - catt -d lgtv_chrome cast ./path/to.mp4
+#
+# support matrix:
+# - webm: audio only
+# - mp4: audio + video
+{ config, lib, ... }:
+let
+  cfg = config.sane.programs.catt;
+in
+{
+  sane.programs.catt = {
+    fs.".config/catt/catt.cfg".symlink.text = ''
+      [options]
+      device = lgtv_chrome
+
+      [aliases]
+      lgtv_chrome = 10.78.79.106
+    '';
+  };
+
+  # necessary to cast local files
+  networking.firewall.allowedTCPPortRanges = lib.mkIf cfg.enabled [
+    {
+      from = 45000;
+      to = 47000;
+    }
+  ];
+}

hosts/common/programs/chatty.nix

@@ -1,40 +1,8 @@
 { pkgs, ... }:
-let
-  chattyNoOauth = pkgs.chatty.override {
-    # the OAuth feature (presumably used for web-based logins) pulls a full webkitgtk.
-    # especially when using the gtk3 version of evolution-data-server, it's an ancient webkitgtk_4_1.
-    # disable OAuth for a faster build & smaller closure
-    evolution-data-server = pkgs.evolution-data-server.override {
-      enableOAuth2 = false;
-      gnome-online-accounts = pkgs.gnome-online-accounts.override {
-        # disables the upstream "goabackend" feature -- presumably "Gnome Online Accounts Backend"
-        # frees us from webkit_4_1, in turn.
-        enableBackend = false;
-        gvfs = pkgs.gvfs.override {
-          # saves 20 minutes of build time, for unused feature
-          samba = null;
-        };
-      };
-    };
-  };
-  chatty-latest = pkgs.chatty-latest.override {
-    evolution-data-server-gtk4 = pkgs.evolution-data-server-gtk4.override {
-      gnome-online-accounts = pkgs.gnome-online-accounts.override {
-        # disables the upstream "goabackend" feature -- presumably "Gnome Online Accounts Backend"
-        # frees us from webkit_4_1, in turn.
-        enableBackend = false;
-        gvfs = pkgs.gvfs.override {
-          # saves 20 minutes of build time and cross issues, for unused feature
-          samba = null;
-        };
-      };
-    };
-  };
-in
 {
   sane.programs.chatty = {
-    # package = chattyNoOauth;
+    # packageUnwrapped = chattyNoOauth;
-    package = chatty-latest;
+    packageUnwrapped = pkgs.chatty-latest;
     suggestedPrograms = [ "gnome-keyring" ];
     persist.byStore.private = [
       ".local/share/chatty"  # matrix avatars and files

hosts/common/programs/conky/battery_estimate

@@ -1,52 +1,184 @@
+#!/bin/sh
 #!/usr/bin/env nix-shell
 #!nix-shell -i bash
 
+usage() {
+  echo "usage: battery_estimate [options...]"
+  echo
+  echo "pretty-prints a battery estimate (icon to indicate state, and a duration estimate)"
+  echo
+  echo "options:"
+  echo "  --debug: output additional information, to stderr"
+  echo "  --minute-suffix <string>:  use the provided string as a minutes suffix"
+  echo "  --hour-suffix <string>:  use the provided string as an hours suffix"
+  echo "  --icon-suffix <string>:  use the provided string as an icon suffix"
+  echo "  --percent-suffix <string>:  use the provided string when displaying percents"
+}
+
 # these icons come from sxmo; they only render in nerdfonts
-bat_dis="󱊢"
+icon_bat_chg=("󰢟" "󱊤" "󱊥" "󰂅")
-bat_chg="󱊥"
+icon_bat_dis=("󰂎" "󱊡" "󱊢" "󱊣")
+suffix_icon=" "  # thin space
+suffix_percent="%"
+# suffix_icon=" "
+
+# render time like: 2ʰ08ᵐ
+# unicode sub/super-scripts: <https://en.wikipedia.org/wiki/Unicode_subscripts_and_superscripts>
+# symbol_hr="ʰ"
+# symbol_min="ᵐ"
+
+# render time like: 2ₕ08ₘ
+# symbol_hr="ₕ"
+# symbol_min="ₘ"
+
+# render time like: 2h08m
+# symbol_hr="h"
+# symbol_min="m"
+
+# render time like: 2:08
+# symbol_hr=":"
+# symbol_min=
+
+# render time like: 2꞉08⧗
+symbol_hr="꞉"
+symbol_min="⧗"
+# variants:
+# symbol_hr=":"
+# symbol_min="⧖"
+# symbol_min="⌛"
+
+# render time like: 2'08"
+# symbol_hr="'"
+# symbol_min='"'
+
+log() {
+  if [ "$BATTERY_ESTIMATE_DEBUG" = "1" ]; then
+    printf "$@" >&2
+    echo >&2
+  fi
+}
+
+render_icon() {
+  # args:
+  # 1: "chg" or "dis"
+  # 2: current battery percentage
+  level=$(($2 / 25))
+  level=$(($level > 3 ? 3 : $level))
+  level=$(($level < 0 ? 0 : $level))
+  log "icon: %s %d" "$1" "$level"
+  if [ "$1" = "dis" ]; then
+    printf "%s" "${icon_bat_dis[$level]}"
+  elif [ "$1" = "chg" ]; then
+    printf "%s" "${icon_bat_chg[$level]}"
+  fi
+}
 
 try_path() {
-  # returns:
+  # assigns output variables:
-  # - perc, perc_left  (0-100)
+  # - perc, perc_from_full  (0-100)
   # - full, rate (pos means charging)
   if [ -f "$1/capacity" ]; then
+    log "perc, perc_from_full from %s" "$1/capacity"
     perc=$(cat "$1/capacity")
-    perc_left=$((100 - $perc))
+    perc_from_full=$((100 - $perc))
   fi
 
   if [ -f "$1/charge_full_design" ] && [ -f "$1/current_now" ]; then
+    log "full, rate from %s and %s" "$1/charge_full_design" "$1/current_now"
     # current is positive when charging
     full=$(cat "$1/charge_full_design")
     rate=$(cat "$1/current_now")
-  fi
+  elif [ -f "$1/energy_full" ] && [ -f "$1/power_now" ]; then
-  if [ -f "$1/energy_full" ] && [ -f "$1/energy_now" ]; then
+    log "full, rate from %s and %s" "$1/energy_full" "$1/power_now"
-    # energy is positive when discharging
+    # power_now is positive when discharging
+    full=$(cat "$1/energy_full")
+    rate=-$(cat "$1/power_now")
+  elif [ -f "$1/energy_full" ] && [ -f "$1/energy_now" ]; then
+    log "full, rate from %s and %s" "$1/energy_full" "$1/energy_now"
+    log "  this is a compatibility path for legacy Thinkpad batteries which do not populate the 'power_now' field, and incorrectly populate 'energy_now' with power info"
+    # energy_now is positive when discharging
     full=$(cat "$1/energy_full")
     rate=-$(cat "$1/energy_now")
   fi
 }
 
-try_path "/sys/class/power_supply/axp20x-battery"  # Pinephone
+try_all_paths() {
-try_path "/sys/class/power_supply/BAT0"  # Thinkpad
+  try_path "/sys/class/power_supply/axp20x-battery"  # Pinephone
+  try_path "/sys/class/power_supply/BAT0"  # Thinkpad
+  log "perc: %d, perc_from_full: %d" "$perc" "$perc_from_full"
+  log "full: %f, rate: %f" "$full" "$rate"
+  log "  rate > 0 means charging, else discharging"
+}
 
 fmt_minutes() {
+  # args:
+  # 1: icon to render
+  # 2: string to show if charge/discharge time is indefinite
+  # 3: minutes to stable state (i.e. to full charge or full discharge)
+  #    - we work in minutes instead of hours for precision: bash math is integer-only
+  log "charge/discharge time: %f min" "$3"
   # args: <battery symbol> <text if ludicrous estimate> <estimated minutes to full/empty>
-  if [[ $3 -gt 1440 ]]; then
+  if [ -n "$3" ] && [ "$3" -lt 1440 ]; then
-    printf "%s %s" "$1" "$2"  # more than 1d
-  else
     hr=$(($3 / 60))
     hr_in_min=$(($hr * 60))
     min=$(($3 - $hr_in_min))
-    printf "%s %dh%02dm" "$1" "$hr" "$min"
+    printf "%s%s%d%s%02d%s" "$1" "$suffix_icon" "$hr" "$symbol_hr" "$min" "$symbol_min"
+  else
+    log "charge/discharge duration > 1d"
+    printf "%s%s%s" "$1" "$suffix_icon" "$2"  # more than 1d
   fi
 }
 
-if [[ $rate -lt 0 ]]; then
+pretty_output() {
-  # discharging
+  if [ -n "$perc" ]; then
-  fmt_minutes "$bat_dis" '∞' "$(($full * 60 * $perc / (-100 * $rate)))"
+    duration=""
-elif [[ $rate -gt 0 ]]; then
+    if [ "$rate" -gt 0 ]; then
-  # charging
+      log "charging"
-  fmt_minutes "$bat_chg" '100%' "$(($full * 60 * $perc_left / (100 * $rate)))"
+      icon="$(render_icon chg $perc)"
-elif [[ "$perc" != "" ]]; then
+      duration="$(($full * 60 * $perc_from_full / (100 * $rate)))"
-  echo "$bat_dis $perc%"
+    else
-fi
+      log "discharging"
+      icon="$(render_icon dis $perc)"
+      if [ "$rate" -lt 0 ]; then
+        duration="$(($full * 60 * $perc / (-100 * $rate)))"
+      fi
+    fi
+    fmt_minutes "$icon" "$perc$suffix_percent" "$duration"
+  fi
+}
+
+while [ "$#" -gt 0 ]; do
+  case "$1" in
+    "--debug")
+      shift
+      BATTERY_ESTIMATE_DEBUG=1
+      ;;
+    "--icon-suffix")
+      shift
+      suffix_icon="$1"
+      shift
+      ;;
+    "--hour-suffix")
+      shift
+      symbol_hr="$1"
+      shift
+      ;;
+    "--minute-suffix")
+      shift
+      symbol_min="$1"
+      shift
+      ;;
+    "--percent-suffix")
+      shift
+      suffix_percent="$1"
+      shift
+      ;;
+    *)
+      usage
+      exit 1
+      ;;
+  esac
+done
+
+try_all_paths
+pretty_output

hosts/common/programs/conky/conky.conf

@@ -3,48 +3,80 @@
 -- - can also use #rrggbb syntax
 -- example configs: <https://forum.manjaro.org/t/conky-showcase-2022/97123>
 -- example configs: <https://www.reddit.com/r/Conkyporn/>
+--
+-- exec options:
+-- `exec <cmd>` => executes the command, synchronously, renders its output as text
+-- `texeci <interval_sec> <cmd>` => executes the command periodically, async (to not block render), renders as text
+-- `pexec <cmd>` => executes the command, synchronously, parses its output
 
 conky.config = {
-    out_to_wayland = true,
+  out_to_wayland = true,
-    update_interval = 10,
+  update_interval = 10,
 
-    alignment = 'middle_middle',
+  alignment = 'middle_middle',
-    own_window_type = 'desktop',
+  own_window_type = 'desktop',
-    -- own_window_argb_value: opacity of the background (0-255)
+  -- own_window_argb_value: opacity of the background (0-255)
-    own_window_argb_value = 0,
+  own_window_argb_value = 0,
-    -- own_window_argb_value = 92,
+  -- own_window_argb_value = 92,
-    -- own_window_colour = '#beebe5',  -- beebe5 matches nixos flake bg color
+  -- own_window_colour = '#beebe5',  -- beebe5 matches nixos flake bg color
 
-    -- "border" pads the entire conky window
+  -- "border" pads the entire conky window
-    -- this can be used to control the extent of the own_window background
+  -- this can be used to control the extent of the own_window background
-    border_inner_margin = 8,
+  border_inner_margin = 8,
-    -- optionally, actually draw borders
+  -- optionally, actually draw borders
-    -- draw_borders = true,
+  -- draw_borders = true,
 
-    -- shades are drop-shadows, outline is the centered version. both apply to text only
+  -- shades are drop-shadows, outline is the centered version. both apply to text only
-    draw_shades = true,
+  draw_shades = true,
-    draw_outline = false,
+  draw_outline = false,
-    default_shade_color = '#beebe5',
+  default_shade_color = '#beebe5',
-    default_outline_color = '#beebe5',
+  default_outline_color = '#beebe5',
 
-    font = 'sans-serif:size=8',
+  font = 'sans-serif:size=8',
-    use_xft = true,
+  use_xft = true,
 
-    default_color = '#ffffff',
+  default_color = '#ffffff',
-    color1 = '000000',
+  color1 = '000000',
-    color2 = '404040',
+  color2 = '404040',
 }
 
--- texeci <interval_sec> <cmd>: run the command periodically, _in a separate thread_ so as not to block rendering
+vars = {
+  -- kBps = 'K/s',
+  kBps = 'ᴷᐟˢ',
+  -- percent = '%',
+  -- percent = '﹪',
+  percent = '٪',
+  -- percent = '⁒',
+  -- percent = '％',
+  icon_suffix = nil,
+  hour_suffix = nil,
+  minute_suffix = '${font sans-serif:size=14}${color2}⧗',
+}
+
+bat_args = ""
+if vars.icon_suffix ~= nil then
+  bat_args = bat_args .. " --icon-suffix '" .. vars.icon_suffix .. "'"
+end
+if vars.hour_suffix ~= nil then
+  bat_args = bat_args .. " --hour-suffix '" .. vars.hour_suffix .. "'"
+end
+if vars.minute_suffix ~= nil then
+  bat_args = bat_args .. " --minute-suffix '" .. vars.minute_suffix .. "'"
+end
+if vars.percent ~= nil then
+  bat_args = bat_args .. " --percent-suffix '" .. vars.percent .. "'"
+end
+
+-- N.B.: `[[ <text> ]]` is Lua's multiline string literal
 conky.text = [[
 ${color1}${shadecolor 707070}${font sans-serif:size=50:style=Bold}${alignc}${exec date +"%H:%M"}${font}
 ${color2}${shadecolor a4d7d0}${font sans-serif:size=20}${alignc}${exec date +"%a %d %b"}${font}
 
 
-${color1}${shadecolor}${font sans-serif:size=22:style=Bold}${alignc}${exec @bat@ }${font}
+${color1}${shadecolor}${font sans-serif:size=22:style=Bold}${alignc}${execp @bat@ ]] .. bat_args .. [[ }${font}
 ${color1}${shadecolor}${font sans-serif:size=20:style=Bold}${alignc}${texeci 600 @weather@ }${font}
 
 
-${color2}${shadecolor a4d7d0}${font sans-serif:size=16}${alignc}⇅ ${downspeedf wlan0}K/s${font}
+${color2}${shadecolor a4d7d0}${font sans-serif:size=16}${alignc}⇅ ${downspeedf wlan0}]] .. vars.kBps .. [[${font}
-${font sans-serif:size=16}${alignc}☵ $memperc%  $cpu%${font}
+${font sans-serif:size=16}${alignc}☵ $memperc]] .. vars.percent .. [[  $cpu]] .. vars.percent .. [[${font}
 ]]

hosts/common/programs/conky/default.nix

@@ -1,11 +1,22 @@
 { config, pkgs, ... }:
 {
   sane.programs.conky = {
+    # TODO: non-sandboxed `conky` still ships via `sxmo-utils`, but unused
+    sandbox.method = "bwrap";
+    sandbox.net = "clearnet";  #< for the scripts it calls (weather)
+    sandbox.extraPaths = [
+      "/sys/class/power_supply"
+      "/sys/devices"  # needed by battery_estimate
+      # "/sys/devices/cpu"
+      # "/sys/devices/system"
+    ];
+    sandbox.whitelistWayland = true;
+
     fs.".config/conky/conky.conf".symlink.target =
       let
         battery_estimate = pkgs.static-nix-shell.mkBash {
           pname = "battery_estimate";
-          src = ./.;
+          srcRoot = ./.;
         };
       in pkgs.substituteAll {
         src = ./conky.conf;
@@ -15,20 +26,14 @@
 
     services.conky = {
       description = "conky dynamic desktop background";
-      wantedBy = [ "default.target" ];
+      after = [ "graphical-session.target" ];
-      # XXX: should be part of graphical-session.target, but whatever mix of greetd/sway
+      # partOf = [ "graphical-session.target" ];  # propagate stop/restart signal from graphical-session to this unit
-      # i'm using means that target's never reached...
+      wantedBy = [ "graphical-session.target" ];
-      # wantedBy = [ "graphical-session.target" ];
-      # partOf = [ "graphical-session.target" ];
 
       serviceConfig.ExecStart = "${config.sane.programs.conky.package}/bin/conky";
       serviceConfig.Type = "simple";
       serviceConfig.Restart = "on-failure";
       serviceConfig.RestartSec = "10s";
-      # serviceConfig.Slice = "session.slice";
-
-      # don't start conky until after sway
-      preStart = ''test -n "$SWAYSOCK"'';
     };
   };
 }

hosts/common/programs/cozy.nix

@@ -2,6 +2,16 @@
 
 {
   sane.programs.cozy = {
+    sandbox.method = "bwrap";  # landlock gives: _multiprocessing.SemLock: Permission Denied
+    sandbox.wrapperType = "wrappedDerivation";
+    sandbox.whitelistAudio = true;
+    sandbox.whitelistDbus = [ "user" ];  # mpris
+    sandbox.whitelistWayland = true;
+    sandbox.extraHomePaths = [
+      "Books/local"
+      "Books/servo"
+    ];
+
     # cozy uses a sqlite db for its config and exposes no CLI options other than --help and --debug
     persist.byStore.plaintext = [
       ".local/share/cozy"  # sqlite db (config & index?)

hosts/common/programs/dconf.nix

@@ -0,0 +1,33 @@
+# dconf docs: <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/desktop_migration_and_administration_guide/profiles>
+# this lets programs temporarily write user-level dconf settings (aka gsettings).
+# they're written to ~/.config/dconf/user, unless `DCONF_PROFILE` is set to something other than the default of /etc/dconf/profile/user
+# find keys/values with `dconf dump /`
+{ config, lib, pkgs, ... }:
+
+let
+  cfg = config.sane.programs.dconf;
+in
+{
+  sane.programs.dconf = {
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "wrappedDerivation";
+    persist.byStore.private = [
+      ".config/dconf"
+    ];
+  };
+
+  programs.dconf = lib.mkIf cfg.enabled {
+    # note that `programs.dconf` doesn't allow specifying the dconf package.
+    enable = true;
+    packages = [
+      (pkgs.writeTextFile {
+        name = "dconf-user-profile";
+        destination = "/etc/dconf/profile/user";
+        text = ''
+          user-db:user
+          system-db:site
+        '';
+      })
+    ];
+  };
+}

hosts/common/programs/default.nix

@@ -7,13 +7,18 @@
     ./alacritty.nix
     ./animatch.nix
     ./assorted.nix
+    ./audacity.nix
     ./bemenu.nix
+    ./bonsai.nix
     ./brave.nix
+    ./bubblewrap.nix
     ./calls.nix
     ./cantata.nix
+    ./catt.nix
     ./chatty.nix
     ./conky
     ./cozy.nix
+    ./dconf.nix
     ./dialect.nix
     ./dino.nix
     ./element-desktop.nix
@@ -21,43 +26,62 @@
     ./evince.nix
     ./feedbackd.nix
     ./firefox.nix
+    ./firejail.nix
     ./flare-signal.nix
     ./fontconfig.nix
     ./fractal.nix
+    ./frozen-bubble.nix
     ./fwupd.nix
     ./g4music.nix
     ./gajim.nix
+    ./gdbus.nix
     ./geary.nix
     ./git.nix
     ./gnome-feeds.nix
-    ./gnome-keyring.nix
+    ./gnome-keyring
+    ./gnome-maps.nix
     ./gnome-weather.nix
+    ./go2tv.nix
     ./gpodder.nix
+    ./grimshot.nix
     ./gthumb.nix
     ./gtkcord4.nix
+    ./handbrake.nix
     ./helix.nix
     ./imagemagick.nix
     ./jellyfin-media-player.nix
+    ./kdenlive.nix
     ./komikku.nix
     ./koreader
     ./libreoffice.nix
     ./lemoa.nix
+    ./loupe.nix
     ./mako.nix
     ./megapixels.nix
     ./mepo.nix
+    ./mimeo
     ./mopidy.nix
     ./mpv.nix
     ./msmtp.nix
+    ./nautilus.nix
     ./neovim.nix
     ./newsflash.nix
     ./nheko.nix
+    ./nicotine-plus.nix
     ./nix-index.nix
+    ./notejot.nix
     ./ntfy-sh.nix
     ./obsidian.nix
     ./offlineimap.nix
+    ./open-in-mpv.nix
+    ./pipewire.nix
+    ./planify.nix
+    ./portfolio-filemanager.nix
     ./playerctl.nix
     ./rhythmbox.nix
     ./ripgrep.nix
+    ./rofi
+    ./sane-scripts.nix
     ./sfeed.nix
     ./signal-desktop.nix
     ./splatmoji.nix
@@ -65,19 +89,32 @@
     ./spotify.nix
     ./steam.nix
     ./stepmania.nix
+    ./strings.nix
     ./sublime-music.nix
     ./supertuxkart.nix
+    ./sway
     ./sway-autoscaler
+    ./swaylock.nix
     ./swaynotificationcenter.nix
     ./tangram.nix
-    ./tor-browser-bundle-bin.nix
+    ./tor-browser.nix
     ./tuba.nix
+    ./unl0kr
     ./vlc.nix
+    ./waybar
+    ./waylock.nix
     ./wike.nix
     ./wine.nix
+    ./wireplumber.nix
     ./wireshark.nix
+    ./wob
     ./xarchiver.nix
+    ./xdg-desktop-portal.nix
+    ./xdg-desktop-portal-gtk.nix
+    ./xdg-desktop-portal-wlr.nix
+    ./xdg-utils.nix
     ./zeal.nix
+    ./zecwallet-lite.nix
     ./zsh
   ];
 

hosts/common/programs/dialect.nix

@@ -1,7 +1,13 @@
 { pkgs, ... }:
 {
   sane.programs.dialect = {
-    package = pkgs.dialect.overrideAttrs (upstream: {
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "inplace";  # share/search_providers/ calls back into the binary, weird wrap semantics
+    sandbox.whitelistWayland = true;
+    sandbox.net = "clearnet";
+    suggestedPrograms = [ "dconf" ];  #< to persist settings
+
+    packageUnwrapped = pkgs.dialect.overrideAttrs (upstream: {
       # TODO: send upstream
       # TODO: figure out how to get audio working
       # TODO: move to runtimeDependencies?

hosts/common/programs/dino.nix

@@ -40,16 +40,39 @@ in
       type = types.submodule {
         options.autostart = mkOption {
           type = types.bool;
-          default = false;
+          default = true;
         };
       };
     };
 
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "wrappedDerivation";
+    sandbox.net = "clearnet";
+    sandbox.whitelistAudio = true;
+    sandbox.whitelistDbus = [ "user" ];  # notifications
+    sandbox.whitelistDri = true;  #< not strictly necessary, but we need all the perf we can get on moby
+    sandbox.whitelistWayland = true;
+    sandbox.extraHomePaths = [
+      "Music"
+      "Pictures/albums"
+      "Pictures/cat"
+      "Pictures/from"
+      "Pictures/Photos"
+      "Pictures/Screenshots"
+      "Pictures/servo-macros"
+      "Videos/local"
+      "Videos/servo"
+      "tmp"
+    ];
+
     persist.byStore.private = [ ".local/share/dino" ];
 
     services.dino = {
       description = "dino XMPP client";
-      wantedBy = lib.mkIf cfg.config.autostart [ "default.target" ];
+      after = [ "graphical-session.target" ];
+      # partOf = [ "graphical-session.target" ];
+      wantedBy = lib.mkIf cfg.config.autostart [ "graphical-session.target" ];
+
       serviceConfig = {
         ExecStart = "${cfg.package}/bin/dino";
         Type = "simple";

hosts/common/programs/element-desktop.nix

@@ -7,14 +7,36 @@
 { pkgs, ... }:
 {
   sane.programs.element-desktop = {
-    package = pkgs.element-desktop.override {
+    packageUnwrapped = pkgs.element-desktop.override {
       # use pre-build electron because otherwise it takes 4 hrs to build from source.
       electron = pkgs.electron-bin;
     };
+    suggestedPrograms = [
+      "gnome-keyring"
+      "xwayland"
+    ];
+
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "wrappedDerivation";
+    sandbox.net = "clearnet";
+    sandbox.whitelistAudio = true;
+    sandbox.whitelistDbus = [ "user" ];  # notifications
+    sandbox.whitelistDri = true;
+    sandbox.whitelistWayland = true;
+    sandbox.extraHomePaths = [
+      "Music"
+      "Pictures/albums"
+      "Pictures/cat"
+      "Pictures/from"
+      "Pictures/Photos"
+      "Pictures/Screenshots"
+      "Pictures/servo-macros"
+      "Videos/local"
+      "Videos/servo"
+      "tmp"
+    ];
 
     # creds/session keys, etc
     persist.byStore.private = [ ".config/Element" ];
-
-    suggestedPrograms = [ "gnome-keyring" ];
   };
 }

hosts/common/programs/epiphany.nix

@@ -8,6 +8,19 @@
 { pkgs, ... }:
 {
   sane.programs.epiphany = {
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "inplace";  # /share/epiphany/default-bookmarks.rdf refers back to /share; dbus files to /libexec
+    sandbox.net = "clearnet";
+    sandbox.whitelistAudio = true;
+    # default sandboxing breaks rendering in weird ways. sites are super zoomed in / not scaled.
+    # enabling DRI/DRM (as below) seems to fix that.
+    sandbox.whitelistDri = true;
+    sandbox.whitelistWayland = true;
+    sandbox.extraHomePaths = [
+      ".config/epiphany"  #< else it gets angry at launch
+      "tmp"
+    ];
+
     # XXX(2023/07/08): running on moby without `WEBKIT_DISABLE_SANDBOX...` fails, with:
     # - `bwrap: Can't make symlink at /var/run: File exists`
     # this could be due to:
@@ -22,13 +35,14 @@
     #
     # TODO: consider `WEBKIT_USE_SINGLE_WEB_PROCESS=1` for better perf
     # - this runs all tabs in 1 process. which is fine, if i'm not a heavy multi-tabber
-    package = pkgs.epiphany.overrideAttrs (upstream: {
+    packageUnwrapped = pkgs.epiphany.overrideAttrs (upstream: {
       preFixup = ''
         gappsWrapperArgs+=(
           --set WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS "1"
         );
       '' + (upstream.preFixup or "");
     });
+    suggestedPrograms = [ "dconf" ];  #< for persisting e.g. "Set as Default Browser" prompt question
     persist.byStore.private = [
       ".cache/epiphany"
       ".local/share/epiphany"

hosts/common/programs/evince.nix

@@ -1,4 +1,10 @@
 { ... }:
 {
-  sane.programs.evince.mime.associations."application/pdf" = "org.gnome.Evince.desktop";
+  sane.programs.evince = {
+    sandbox.method = "bwrap";
+    sandbox.autodetectCliPaths = true;
+    sandbox.whitelistWayland = true;
+
+    mime.associations."application/pdf" = "org.gnome.Evince.desktop";
+  };
 }

hosts/common/programs/feedbackd.nix

@@ -1,10 +1,13 @@
+# test with e.g.
+# - `fbcli --event proxied-message-new-instant`
+
 { config, lib, pkgs, ... }:
 let
   cfg = config.sane.programs.feedbackd;
 in
 {
   sane.programs.feedbackd = {
-    package = pkgs.rmDbusServices pkgs.feedbackd;
+    packageUnwrapped = pkgs.rmDbusServices pkgs.feedbackd;
 
     configOption = with lib; mkOption {
       type = types.submodule {
@@ -21,6 +24,11 @@ in
       default = {};
     };
 
+    sandbox.method = "bwrap";
+    sandbox.wrapperType = "wrappedDerivation";
+    sandbox.whitelistDbus = [ "user" ];
+    sandbox.whitelistAudio = true;
+
     # N.B.: feedbackd will load ~/.config/feedbackd/themes/default.json by default
     # - but using that would forbid `parent-theme = "default"`
     # the default theme ships support for these events:
@@ -89,7 +97,7 @@ in
 
     services.feedbackd = {
       description = "feedbackd audio/vibration/led controller";
-      wantedBy = [ "default.target" ];
+      wantedBy = [ "default.target" ];  #< should technically be `sound.target`, but that doesn't seem to get auto-started?
       serviceConfig = {
         ExecStart = "${cfg.package}/libexec/feedbackd";
         Type = "simple";