WIP: mootube: init at 2022-08-28

note that this doesn't cross compile, because of stupid fucking rust build.rs scripts

thanks, openssl-sys

.gitignore

@@ -1,2 +1,4 @@
+.working
 result
+result-*
 /secrets/local.nix

README.md

@@ -4,17 +4,35 @@ this is the top-level repo from which i configure/deploy all my NixOS machines:
 - desktop
 - laptop
 - server
-- mobile phone
+- mobile phone (Pinephone)
 
-i enjoy a monorepo approach. this repo references [nixpkgs][nixpkgs], a couple 3rd party
-nix modules like [sops][sops], the sources for [uninsane.org][uninsane-org], and that's
-about it. custom derivations and modules (some of which i try to upstream) live
-directly here; even the sources for those packages is often kept here too.
+everything outside of <./hosts/> and <./secrets/> is intended for export, to be importable for use by 3rd parties.
+the only hard dependency for my exported pkgs/modules should be [nixpkgs][nixpkgs].
+building <./hosts/> will require [sops][sops].
+
+you might specifically be interested in these files (elaborated further in #key-points-of-interest):
+- [`sxmo-utils`](./pkgs/additional/sxmo-utils/default.nix)
+  - [example SXMO deployment](./hosts/modules/gui/sxmo/default.nix)
+- [my implementation of impermanence](./modules/persist/default.nix)
+- my way of deploying dotfiles/configuring programs per-user:
+  - <./modules/fs/default.nix>
+  - <./modules/programs.nix>
+  - <./modules/users.nix>
 
 [nixpkgs]: https://github.com/NixOS/nixpkgs
 [sops]: https://github.com/Mic92/sops-nix
 [uninsane-org]: https://uninsane.org
 
+## Using This Repo In Your Own Config
+
+this should be a pretty "standard" flake. just reference it, and import either
+- `nixosModules.sane` (for the modules)
+- `overlays.pkgs` (for the packages)
+
+or follow the instructions [here][NUR] to use it via the Nix User Repositories.
+
+[NUR]: https://nur.nix-community.org/
+
 ## Layout
 - `doc/`
     - instructions for tasks i find myself doing semi-occasionally in this repo.
@@ -90,12 +108,6 @@ them being factored out of my config, message me and we could work to make that
 
 [home-manager]: https://github.com/nix-community/home-manager
 
-## Using This Repo In Your Own Config
-
-this should be a pretty "standard" flake. just reference it, and import either
-- `nixosModules.sane` (for the modules)
-- `overlays.pkgs` (for the packages)
-
 ## Mirrors
 
 this repo exists in a few known locations:

TODO.md

@@ -1,13 +1,14 @@
 ## BUGS
-- mpv UI is sometimes blank for audio/podcasts?
-  - i think it's when the audio file has no thumbnail?
-- why i need to manually restart `wireguard-wg-ovpns` on servo periodically
-  - else DNS fails
-- fix epiphany URL bar input on moby
-- sxmo: wvkbd: missing font for icons on the 3rd page
+- ringer (i.e. dino incoming call) doesn't prevent moby from sleeping
+- Fractal opens links with non-preferred web browser
+- `nix` operations from lappy hang when `desko` is unreachable
+  - could at least direct the cache to `http://desko-hn:5001`
+- waybar isn't visible on moby until after `swaymsg reload`
 
 ## REFACTORING:
 
+- fold hosts/common/home/ssh.nix -> hosts/common/users/colin.nix
+
 ### sops/secrets
 - attach secrets to the thing they're used by (sane.programs)
 - rework secrets to leverage `sane.fs`
@@ -24,7 +25,6 @@
 - fix lightdm-mobile-greeter for newer libhandy
 - port zecwallet-lite to a from-source build
 - REVIEW/integrate jellyfin dataDir config: <https://github.com/NixOS/nixpkgs/pull/233617>
-- remove `libsForQt5.callPackage` broadly: <https://github.com/NixOS/nixpkgs/issues/180841>
 
 #### upstreaming to non-nixpkgs repos
 - gtk: build schemas even on cross compilation: <https://github.com/NixOS/nixpkgs/pull/247844>
@@ -50,47 +50,77 @@
   - e.g. daily email checks; daily backup checks
   - integrate `nix check` into Gitea actions?
 
+### faster/better deployments
+- remove audacity's dependency on webkitgtk (via wxwidgets)
+
 ### user experience
-- moby: sxmo: fix youtube scripts (package youtube-cli)
+- install apps:
+  - display QR codes for WiFi endpoints: <https://linuxphoneapps.org/apps/noappid.wisperwind.wifi2qr/>
+  - shopping list (not in nixpkgs): <https://linuxphoneapps.org/apps/ro.hume.cosmin.shoppinglist/>
+  - offline Wikipedia (or, add to `wike`)
+  - offline docs viewer (gtk): <https://github.com/workbenchdev/Biblioteca>
+  - some type of games manager/launcher
+    - Gnome Highscore (retro games)?: <https://gitlab.gnome.org/World/highscore>
+  - better maps for mobile (Osmin (QtQuick)? Pure Maps (Qt/Kirigami)? Gnome Maps is improved in 45)
+  - note-taking app: <https://linuxphoneapps.org/categories/note-taking/>
+  - OSK overlay specifically for mobile gaming
+    - i.e. mock joysticks, for use with SuperTux and SuperTuxKart
+- install mobile-friendly games:
+  - Shattered Pixel Dungeon (nixpkgs `shattered-pixel-dungeon`; doesn't cross-compile b/c openjdk/libIDL) <https://github.com/ebolalex/shattered-pixel-dungeon>
+  - UnCiv (Civ V clone; nixpkgs `unciv`; doesn't cross-compile):  <https://github.com/yairm210/UnCiv>
+  - Simon Tatham's Puzzle Collection (not in nixpkgs) <https://git.tartarus.org/?p=simon/puzzles.git>
+  - Shootin Stars  (Godot; not in nixpkgs) <https://gitlab.com/greenbeast/shootin-stars>
+
+#### moby
+- fix cpuidle (gets better power consumption): <https://xnux.eu/log/077.html>
+- SwayNC:
+  - don't show MPRIS if no players detected
+    - this is a problem of playerctld, i guess
+    - also, the album icon when "Not playing" doesn't follow the size we give in the config
+      - that means mpris always takes up excessive space on moby
+  - add option to change audio output
+  - fix colors (red alert) to match overall theme
 - moby: tune GPS
   - run only geoclue, and not gpsd, to save power?
   - tune QGPS setting in eg25-control, for less jitter?
   - direct mepo to prefer gpsd, with fallback to geoclue, for better accuracy?
   - configure geoclue to do some smoothing?
   - manually do smoothing, as some layer between mepo and geoclue/gpsd?
+- moby: show battery state on ssh login
+- moby: improve gPodder launch time
+- sxmo: port to swaybar like i use on desktop
+  - users in #sxmo claim it's way better perf
+- sxmo: fix youtube scripts (package youtube-cli)
+- moby: theme GTK apps (i.e. non-adwaita styles)
+  - combine multiple icon themes to get one which has the full icon set?
+  - get adwaita-icon-theme to ship everything even when cross-compiled?
+  - especially, make the menubar collapsible
+  - try Gradience tool specifically for theming adwaita? <https://linuxphoneapps.org/apps/com.github.gradienceteam.gradience/>
+- phog: remove the gnome-shell runtime dependency to save hella closure size
+
+#### non-moby
+- RSS: integrate a paywall bypass
+  - e.g. self-hosted [ladder](https://github.com/everywall/ladder) (like 12ft.io)
 - neovim: set up language server (lsp; rnix-lsp; nvim-lspconfig)
 - Helix: make copy-to-system clipboard be the default
 - firefox/librewolf: persist history
   - just not cookies or tabs
-- moby: improve gPodder launch time
-- moby: theme GTK apps (i.e. non-adwaita styles)
-  - especially, make the menubar collapsible
-  - try Gradience tool specifically for theming adwaita? <https://linuxphoneapps.org/apps/com.github.gradienceteam.gradience/>
 - package Nix/NixOS docs for Zeal
   - install [doc-browser](https://github.com/qwfy/doc-browser)
   - this supports both dash (zeal) *and* the datasets from <https://devdocs.io> (which includes nix!)
   - install [devhelp](https://wiki.gnome.org/Apps/Devhelp)  (gnome)
 - have xdg-open parse `<repo:...> URIs (or adjust them so that it _can_ parse)
 - sane-bt-search: show details like 5.1 vs stereo, h264 vs h265
+  - maybe just color these "keywords" in all search results?
 - uninsane.org: make URLs relative to allow local use (and as offline homepage)
 - email: fix so that local mail doesn't go to junk
   - git sendmail flow adds the DKIM signatures, but gets delivered locally w/o having the sig checked, so goes into Junk
   - could change junk filter from "no DKIM success" to explicit "DKIM failed"
-- sxmo: don't put all deps on PATH
-  - maybe: use resholve to hard-code them
-    - this is the most "correct", but least patchable
-  - maybe: express each invocation as a function in sxmo_common.sh
-    - this will require some patching to handle `exec <foo>` style
-  - maybe: save original PATH and reset it before invoking user files
 
 ### perf
 - add `pkgs.impure-cached.<foo>` package set to build things with ccache enabled
   - every package here can be auto-generated, and marked with some env var so that it doesn't pollute the pure package set
   - would be super handy for package prototyping!
-- why does nixos-rebuild switch take 5 minutes when net is flakey?
-  - trying to auto-mount servo?
-  - something to do with systemd services restarting/stalling
-  - maybe wireguard & its refresh operation, specifically?
 - get moby to build without binfmt emulation (i.e. make all emulation explicit)
   - then i can distribute builds across servo + desko, and also allow servo to pull packages from desko w/o worrying about purity
 

default.nix

@@ -0,0 +1,9 @@
+# limited, non-flake interface to this repo.
+# this file exposes the same view into `pkgs` which the flake would see when evaluated.
+#
+# the primary purpose of this file is so i can run `updateScript`s which expect
+# the root to be `default.nix`
+{ pkgs ? import <nixpkgs> {} }:
+pkgs.appendOverlays [
+  (import ./overlays/all.nix)
+]

flake.lock

@@ -5,11 +5,11 @@
         "systems": "systems"
       },
       "locked": {
-        "lastModified": 1687709756,
-        "narHash": "sha256-Y5wKlQSkgEK2weWdOu4J3riRd+kV/VCgHsqLNTTWQ/0=",
+        "lastModified": 1694529238,
+        "narHash": "sha256-zsNZZGTGnMOf9YpHKJqMSsa0dXbfmxeoJ7xHlrt+xmY=",
         "owner": "numtide",
         "repo": "flake-utils",
-        "rev": "dbabf0ca0c0c4bce6ea5eaf65af5cb694d2082c7",
+        "rev": "ff7b65b44d01cf9ba6a71320833626af21126384",
         "type": "github"
       },
       "original": {
@@ -21,59 +21,27 @@
     "mobile-nixos": {
       "flake": false,
       "locked": {
-        "lastModified": 1690059310,
-        "narHash": "sha256-4zcoDp8wwZVfGSzXltC5x+eH4kDWC/eJpyQNgr7shAA=",
+        "lastModified": 1694749521,
+        "narHash": "sha256-MiVokKlpcJmfoGuWAMeW1En7gZ5hk0rCQArYm6P9XCc=",
         "owner": "nixos",
         "repo": "mobile-nixos",
-        "rev": "56fc9f9619f305f0865354975a98d22410eed127",
+        "rev": "d25d3b87e7f300d8066e31d792337d9cd7ecd23b",
         "type": "github"
       },
       "original": {
         "owner": "nixos",
+        "ref": "d25d3b87e7f300d8066e31d792337d9cd7ecd23b",
         "repo": "mobile-nixos",
         "type": "github"
       }
     },
-    "nix-serve": {
-      "inputs": {
-        "nixpkgs": "nixpkgs"
-      },
-      "locked": {
-        "lastModified": 1687251388,
-        "narHash": "sha256-E9cVlgeCvzPbA/G3mCDCzz8TdRwXyGYzIjmwcvIfghg=",
-        "owner": "edolstra",
-        "repo": "nix-serve",
-        "rev": "d6df5bd8584f37e22cff627db2fc4058a4aab5ee",
-        "type": "github"
-      },
-      "original": {
-        "owner": "edolstra",
-        "repo": "nix-serve",
-        "type": "github"
-      }
-    },
-    "nixpkgs": {
-      "locked": {
-        "lastModified": 1606086654,
-        "narHash": "sha256-VFl+3eGIMqNp7cyOMJ6TjM/+UcsLKtodKoYexrlTJMI=",
-        "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "19db3e5ea2777daa874563b5986288151f502e27",
-        "type": "github"
-      },
-      "original": {
-        "id": "nixpkgs",
-        "ref": "nixos-20.09",
-        "type": "indirect"
-      }
-    },
     "nixpkgs-stable": {
       "locked": {
-        "lastModified": 1693097136,
-        "narHash": "sha256-fBZSMdBaoZ0INFbyZ5s0DOF7zDNcLsLxgkwdDh3l9Pc=",
+        "lastModified": 1700905716,
+        "narHash": "sha256-w1vHn2MbGfdC+CrP3xLZ3scsI06N0iQLU7eTHIVEFGw=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "9117c4e9dc117a6cd0319cca40f2349ed333669d",
+        "rev": "dfb95385d21475da10b63da74ae96d89ab352431",
         "type": "github"
       },
       "original": {
@@ -85,16 +53,16 @@
     },
     "nixpkgs-unpatched": {
       "locked": {
-        "lastModified": 1693663421,
-        "narHash": "sha256-ImMIlWE/idjcZAfxKK8sQA7A1Gi/O58u5/CJA+mxvl8=",
+        "lastModified": 1701180790,
+        "narHash": "sha256-kYWcHsk2A1VUpiOvSo7Pq175WnSVeltspTGM2q+Cr3U=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "e56990880811a451abd32515698c712788be5720",
+        "rev": "c9702bf40b036c0f1d3d5b0aaf3eee2bf920124c",
         "type": "github"
       },
       "original": {
         "owner": "nixos",
-        "ref": "nixos-unstable",
+        "ref": "master",
         "repo": "nixpkgs",
         "type": "github"
       }
@@ -102,7 +70,6 @@
     "root": {
       "inputs": {
         "mobile-nixos": "mobile-nixos",
-        "nix-serve": "nix-serve",
         "nixpkgs-unpatched": "nixpkgs-unpatched",
         "sops-nix": "sops-nix",
         "uninsane-dot-org": "uninsane-dot-org"
@@ -116,11 +83,11 @@
         "nixpkgs-stable": "nixpkgs-stable"
       },
       "locked": {
-        "lastModified": 1693404499,
-        "narHash": "sha256-cx/7yvM/AP+o/3wPJmA9W9F+WHemJk5t+Xcr+Qwkqhg=",
+        "lastModified": 1701127353,
+        "narHash": "sha256-qVNX0wOl0b7+I35aRu78xUphOyELh+mtUp1KBx89K1Q=",
         "owner": "Mic92",
         "repo": "sops-nix",
-        "rev": "d9c5dc41c4b1f74c77f0dbffd0f3a4ebde447b7a",
+        "rev": "b1edbf5c0464b4cced90a3ba6f999e671f0af631",
         "type": "github"
       },
       "original": {
@@ -152,11 +119,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1691106178,
-        "narHash": "sha256-3mZ9gTvMpbZA9ea9ovoQpn2wKuQY0QZ7MDdEjArYdAQ=",
+        "lastModified": 1699515935,
+        "narHash": "sha256-cJIuVrYorhIzG5pRFZb+ZtaKhTFD92ThC42SaxvSe/E=",
         "ref": "refs/heads/master",
-        "rev": "f4d91aa201b6e49af690f250d4786bd1d8b4dcfd",
-        "revCount": 205,
+        "rev": "8a4273489d945f21d7e0ca6aac952460c7d4c391",
+        "revCount": 216,
         "type": "git",
         "url": "https://git.uninsane.org/colin/uninsane"
       },

flake.nix

@@ -4,6 +4,8 @@
 #   - this is marginally the case with schemes like `github:nixos/nixpkgs`.
 #   - given the *existing* `git+https://` scheme, i propose expressing github URLs similarly:
 #     - `github+https://github.com/nixos/nixpkgs/tree/nixos-22.11`
+#     - this would allow for the same optimizations as today's `github:nixos/nixpkgs`, but without obscuring the source.
+#       a code reader could view the source being referenced simply by clicking the https:// portion of that URI.
 # - need some way to apply local patches to inputs.
 #
 #
@@ -34,36 +36,39 @@
     #   - staging-next merged into master.
     #
     # which branch to source from?
-    # - for everyday development, prefer `nixos-unstable` branch, as it provides good caching.
-    # - if need to test bleeding updates (e.g. if submitting code into staging):
-    #   - use `staging-next` if it's been cut (i.e. if there's an active staging-next -> master PR)
-    #   - use `staging` if no staging-next branch has been cut.
+    # - nixos-unstable: for everyday development; it provides good caching
+    # - master: temporarily if i'm otherwise cherry-picking lots of already-applied patches
+    # - staging-next: if testing stuff that's been PR'd into staging, i.e. base library updates.
+    # - staging: maybe if no staging-next -> master PR has been cut yet?
     #
     # <https://github.com/nixos/nixpkgs/tree/nixos-unstable>
-    nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-unstable";
+    # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=nixos-unstable";
+    nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=master";
     # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=staging-next";
     # nixpkgs-unpatched.url = "github:nixos/nixpkgs?ref=staging";
 
     mobile-nixos = {
       # <https://github.com/nixos/mobile-nixos>
-      url = "github:nixos/mobile-nixos";
+      # only used for building disk images, not relevant after deployment
+      # TODO: replace with something else. commit `0f3ac0bef1aea70254a3bae35e3cc2561623f4c1`
+      # replaces the imageBuilder with a "new implementation from celun" and wildly breaks my use.
+      # pinning to d25d3b... is equivalent to holding at 2023-09-15
+      url = "github:nixos/mobile-nixos?ref=d25d3b87e7f300d8066e31d792337d9cd7ecd23b";
       flake = false;
     };
     sops-nix = {
       # <https://github.com/Mic92/sops-nix>
+      # used to distribute secrets to my hosts
       url = "github:Mic92/sops-nix";
       # inputs.nixpkgs.follows = "nixpkgs";
       inputs.nixpkgs.follows = "nixpkgs-unpatched";
     };
     uninsane-dot-org = {
+      # provides the package to deploy <https://uninsane.org>, used only when building the servo host
       url = "git+https://git.uninsane.org/colin/uninsane";
       # inputs.nixpkgs.follows = "nixpkgs";
       inputs.nixpkgs.follows = "nixpkgs-unpatched";
     };
-    nix-serve = {
-      # <https://github.com/edolstra/nix-serve>
-      url = "github:edolstra/nix-serve";
-    };
   };
 
   outputs = {
@@ -72,26 +77,42 @@
     mobile-nixos,
     sops-nix,
     uninsane-dot-org,
-    nix-serve,
     ...
   }@inputs:
     let
       inherit (builtins) attrNames elem listToAttrs map mapAttrs;
+      # redefine some nixpkgs `lib` functions to avoid the infinite recursion
+      # of if we tried to use patched `nixpkgs.lib` as part of the patching process.
       mapAttrs' = f: set:
         listToAttrs (map (attr: f attr set.${attr}) (attrNames set));
+      optionalAttrs = cond: attrs: if cond then attrs else {};
       # mapAttrs but without the `name` argument
       mapAttrValues = f: mapAttrs (_: f);
+
       # rather than apply our nixpkgs patches as a flake input, do that here instead.
       # this (temporarily?) resolves the bad UX wherein a subflake residing in the same git
       # repo as the main flake causes the main flake to have an unstable hash.
       nixpkgs = (import ./nixpatches/flake.nix).outputs {
         self = nixpkgs;
         nixpkgs = nixpkgs-unpatched;
+      } // {
+        # provide values that nixpkgs ordinarily sources from the flake.lock file,
+        # inaccessible to it here because of the import-from-derivation.
+        # rev and shortRev seem to not always exist (e.g. if the working tree is dirty),
+        # so those are made conditional.
+        #
+        # these values impact the name of a produced nixos system. having date/rev in the
+        # `readlink /run/current-system` store path helps debuggability.
+        inherit (self) lastModifiedDate lastModified;
+      } // optionalAttrs (self ? rev) {
+        inherit (self) rev;
+      } // optionalAttrs (self ? shortRev) {
+        inherit (self) shortRev;
       };
 
       nixpkgsCompiledBy = system: nixpkgs.legacyPackages."${system}";
 
-      evalHost = { name, local, target }: nixpkgs.lib.nixosSystem {
+      evalHost = { name, local, target, light ? false }: nixpkgs.lib.nixosSystem {
         system = target;
         modules = [
           {
@@ -104,6 +125,9 @@
             # nixpkgs.buildPlatform = local;  # set by instantiate.nix instead
             # nixpkgs.config.replaceStdenv = { pkgs }: pkgs.ccacheStdenv;
           }
+          (optionalAttrs light {
+            sane.enableSlowPrograms = false;
+          })
           (import ./hosts/instantiate.nix { hostName = name; })
           self.nixosModules.default
           self.nixosModules.passthru
@@ -121,8 +145,11 @@
           hosts = {
             servo       = { name = "servo";  local = "x86_64-linux"; target = "x86_64-linux";  };
             desko       = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  };
+            desko-light = { name = "desko";  local = "x86_64-linux"; target = "x86_64-linux";  light = true; };
             lappy       = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  };
+            lappy-light = { name = "lappy";  local = "x86_64-linux"; target = "x86_64-linux";  light = true; };
             moby        = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; };
+            moby-light  = { name = "moby";   local = "x86_64-linux"; target = "aarch64-linux"; light = true; };
             rescue      = { name = "rescue"; local = "x86_64-linux"; target = "x86_64-linux";  };
           };
           # cross-compiled builds: instead of emulating the host, build using a cross-compiler.
@@ -130,10 +157,7 @@
           # - but fewer of their packages can be found in upstream caches.
           cross = mapAttrValues evalHost hosts;
           emulated = mapAttrValues
-            ({name, local, target}: evalHost {
-              inherit name target;
-              local = null;
-            })
+            (args: evalHost (args // { local = null; }))
             hosts;
           prefixAttrs = prefix: attrs: mapAttrs'
             (name: value: {
@@ -145,9 +169,9 @@
           (prefixAttrs "cross-" cross) //
           (prefixAttrs "emulated-" emulated) // {
             # prefer native builds for these machines:
-            inherit (emulated) servo desko lappy rescue;
+            inherit (emulated) servo desko desko-light lappy lappy-light rescue;
             # prefer cross-compiled builds for these machines:
-            inherit (cross) moby;
+            inherit (cross) moby moby-light;
           };
 
       # unofficial output
@@ -167,8 +191,9 @@
       imgs = mapAttrValues (host: host.config.system.build.img) self.nixosConfigurations;
 
       # unofficial output
-      host-pkgs = mapAttrValues (host: host.config.system.build.pkgs) self.nixosConfigurations;
-      host-programs = mapAttrValues (host: mapAttrValues (p: p.package) host.config.sane.programs) self.nixosConfigurations;
+      hostConfigs = mapAttrValues (host: host.config) self.nixosConfigurations;
+      hostPkgs = mapAttrValues (host: host.config.system.build.pkgs) self.nixosConfigurations;
+      hostPrograms = mapAttrValues (host: mapAttrValues (p: p.package) host.config.sane.programs) self.nixosConfigurations;
 
       overlays = {
         # N.B.: `nix flake check` requires every overlay to take `final: prev:` at defn site,
@@ -184,17 +209,9 @@
           let
             mobile = (import "${mobile-nixos}/overlay/overlay.nix");
             uninsane = uninsane-dot-org.overlay;
-            # nix-serve' = nix-serve.overlay;
-            nix-serve' = next: prev: {
-              # XXX(2023/03/02): upstream isn't compatible with modern `nix`. probably the perl bindings.
-              # - we use the package built against `nixpkgs` specified in its flake rather than use its overlay,
-              #   to get around this.
-              inherit (nix-serve.packages."${next.system}") nix-serve;
-            };
           in
             (mobile final prev)
             // (uninsane final prev)
-            // (nix-serve' final prev)
           ;
       };
 
@@ -244,6 +261,7 @@
       apps."x86_64-linux" =
         let
           pkgs = self.legacyPackages."x86_64-linux";
+          sanePkgs = import ./pkgs { inherit pkgs; };
           deployScript = host: addr: action: pkgs.writeShellScript "deploy-${host}" ''
             nix build '.#nixosConfigurations.${host}.config.system.build.toplevel' --out-link ./result-${host} $@
             sudo nix sign-paths -r -k /run/secrets/nix_serve_privkey $(readlink ./result-${host})
@@ -253,6 +271,65 @@
             # let the user handle that edge case by re-running this whole command
             nixos-rebuild --flake '.#${host}' ${action} --target-host colin@${addr} --use-remote-sudo $@
           '';
+          deployApp = host: addr: action: {
+            type = "app";
+            program = ''${deployScript host addr action}'';
+          };
+
+          # pkg updating.
+          # a cleaner alternative lives here: <https://discourse.nixos.org/t/how-can-i-run-the-updatescript-of-personal-packages/25274/2>
+          # mkUpdater :: [ String ] -> { type = "app"; program = path; }
+          mkUpdater = attrPath: {
+            type = "app";
+            program = let
+              pkg = pkgs.lib.getAttrFromPath attrPath sanePkgs;
+              strAttrPath = pkgs.lib.concatStringsSep "." attrPath;
+              commandArgv = pkg.updateScript.command or pkg.updateScript;
+              command = pkgs.lib.escapeShellArgs commandArgv;
+            in builtins.toString (pkgs.writeShellScript "update-${strAttrPath}" ''
+              export UPDATE_NIX_NAME=${pkg.name}
+              export UPDATE_NIX_PNAME=${pkg.pname}
+              export UPDATE_NIX_OLD_VERSION=${pkg.version}
+              export UPDATE_NIX_ATTR_PATH=${strAttrPath}
+              ${command}
+            '');
+          };
+          mkUpdatersNoAliases = opts: basePath: pkgs.lib.concatMapAttrs
+            (name: pkg:
+              if pkg.recurseForDerivations or false then {
+                "${name}" = mkUpdaters opts (basePath ++ [ name ]);
+              } else if pkg.updateScript or null != null then {
+                "${name}" = mkUpdater (basePath ++ [ name ]);
+              } else {}
+            )
+            (pkgs.lib.getAttrFromPath basePath sanePkgs);
+          mkUpdaters = { ignore ? [], flakePrefix ? [] }@opts: basePath:
+            let
+              updaters = mkUpdatersNoAliases opts basePath;
+              invokeUpdater = name: pkg:
+                let
+                  fullPath = basePath ++ [ name ];
+                  doUpdateByDefault = !builtins.elem fullPath ignore;
+
+                  # in case `name` has a `.` in it, we have to quote it
+                  escapedPath = builtins.map (p: ''"${p}"'') fullPath;
+                  updatePath = builtins.concatStringsSep "." (flakePrefix ++ escapedPath);
+                in pkgs.lib.optionalString doUpdateByDefault (
+                  pkgs.lib.escapeShellArgs [
+                    "nix" "run" ".#${updatePath}"
+                  ]
+                );
+            in {
+              type = "app";
+              # top-level app just invokes the updater of everything one layer below it
+              program = builtins.toString (pkgs.writeShellScript
+                (builtins.concatStringsSep "-" (flakePrefix ++ basePath))
+                (builtins.concatStringsSep
+                  "\n"
+                  (pkgs.lib.mapAttrsToList invokeUpdater updaters)
+                )
+              );
+            } // updaters;
         in {
           help = {
             type = "app";
@@ -261,41 +338,46 @@
                 commands:
                 - `nix run '.#help'`
                   - show this message
-                - `nix run '.#update-feeds'`
+                - `nix run '.#update.pkgs'`
+                  - updates every package
+                - `nix run '.#update.feeds'`
                   - updates metadata for all feeds
                 - `nix run '.#init-feed' <url>`
-                - `nix run '.#deploy-{lappy,moby,moby-test,servo}' [nixos-rebuild args ...]`
-                - `nix run '.#check-nur'`
+                - `nix run '.#deploy.{desko,lappy,moby,servo}[-light][.test]' [nixos-rebuild args ...]`
+                - `nix run '.#check'`
+                  - make sure all systems build; NUR evaluates
+
+                specific build targets of interest:
+                - `nix build '.#imgs.rescue'`
               '';
             in builtins.toString (pkgs.writeShellScript "nixos-config-help" ''
               cat ${helpMsg}
+              echo ""
+              echo "complete flake structure:"
+              nix flake show --option allow-import-from-derivation true
             '');
           };
-          update-feeds = {
-            type = "app";
-            program = "${pkgs.feeds.updateScript}";
+          # wrangle some names to get package updaters which refer back into the flake, but also conditionally ignore certain paths (e.g. sane.feeds).
+          # TODO: better design
+          update = rec {
+            _impl.pkgs.sane = mkUpdaters { flakePrefix = [ "update" "_impl" "pkgs" ]; ignore = [ [ "sane" "feeds" ] ]; } [ "sane" ];
+            pkgs = _impl.pkgs.sane;
+            _impl.feeds.sane.feeds = mkUpdaters { flakePrefix = [ "update" "_impl" "feeds" ]; } [ "sane" "feeds" ];
+            feeds = _impl.feeds.sane.feeds;
           };
 
           init-feed = {
             type = "app";
-            program = "${pkgs.feeds.initFeedScript}";
+            program = "${pkgs.feeds.init-feed}";
           };
 
-          deploy-lappy = {
-            type = "app";
-            program = ''${deployScript "lappy" "lappy" "switch"}'';
-          };
-          deploy-moby-test = {
-            type = "app";
-            program = ''${deployScript "moby" "moby-hn" "test"}'';
-          };
-          deploy-moby = {
-            type = "app";
-            program = ''${deployScript "moby" "moby-hn" "switch"}'';
-          };
-          deploy-servo = {
-            type = "app";
-            program = ''${deployScript "servo" "servo" "switch"}'';
+          deploy = {
+            lappy       = deployApp "lappy"       "lappy" "switch";
+            lappy-light = deployApp "lappy-light" "lappy" "switch";
+            moby        = deployApp "moby"        "moby"  "switch";
+            moby-light  = deployApp "moby-light"  "moby"  "switch";
+            moby-test   = deployApp "moby"        "moby"  "test";
+            servo       = deployApp "servo"       "servo" "switch";
           };
 
           sync-moby = {
@@ -318,7 +400,23 @@
             '');
           };
 
-          check-nur = {
+          check = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "check-all" ''
+              nix run '.#check.nur'
+              RC0=$?
+              nix run '.#check.host-configs'
+              RC1=$?
+              nix run '.#check.rescue'
+              RC2=$?
+              echo "nur: $RC0"
+              echo "host-configs: $RC1"
+              echo "rescue: $RC2"
+              exit $(($RC0 | $RC1 | $RC2))
+            '');
+          };
+
+          check.nur = {
             # `nix run '.#check-nur'`
             # validates that my repo can be included in the Nix User Repository
             type = "app";
@@ -330,22 +428,51 @@
                 --option allow-import-from-derivation true \
                 --drv-path --show-trace \
                 -I nixpkgs=$(nix-instantiate --find-file nixpkgs) \
-                -I ../../
+                -I ../../ \
+                | tee  # tee to prevent interactive mode
             '');
           };
 
-          check-host-configs = {
+          check.host-configs = {
             type = "app";
-            program = builtins.toString (pkgs.writeShellScript
+            program = let
+              checkHost = host: let
+                shellHost = pkgs.lib.replaceStrings [ "-" ] [ "_" ] host;
+              in ''
+                nix build -v '.#nixosConfigurations.${host}.config.system.build.toplevel' --out-link ./result-${host} -j2 $@
+                RC_${shellHost}=$?
+              '';
+            in builtins.toString (pkgs.writeShellScript
               "check-host-configs"
-              (builtins.concatStringsSep "\n" (builtins.map
-                (host: "nix build '.#nixosConfigurations.${host}.config.system.build.toplevel' --out-link ./result-${host} -j1 $@ &")
-                [ "desko" "lappy" "servo" "moby" "rescue" ]
-                # not part of the `map`. wait for all builds to complete
-                ++ [ "wait" ]
-              ))
+              ''
+                # build minimally-usable hosts first, then their full image.
+                # this gives me a minimal image i can deploy or copy over, early.
+                ${checkHost "desko-light"}
+                ${checkHost "moby-light"}
+                ${checkHost "lappy-light"}
+
+                ${checkHost "desko"}
+                ${checkHost "lappy"}
+                ${checkHost "servo"}
+                ${checkHost "moby"}
+                ${checkHost "rescue"}
+
+                echo "desko: $RC_desko"
+                echo "lappy: $RC_lappy"
+                echo "servo: $RC_servo"
+                echo "moby: $RC_moby"
+                echo "rescue: $RC_rescue"
+                exit $(($RC_desko | $RC_lappy | $RC_servo | $RC_moby | $RC_rescue))
+              ''
             );
           };
+
+          check.rescue = {
+            type = "app";
+            program = builtins.toString (pkgs.writeShellScript "check-rescue" ''
+              nix build -v '.#imgs.rescue' --out-link ./result-rescue-img -j2
+            '');
+          };
         };
 
       templates = {

hosts/by-name/desko/default.nix

@@ -11,23 +11,29 @@
 
   sops.secrets.colin-passwd.neededForUsers = true;
 
+  sane.ports.openFirewall = true;  # for e.g. nix-serve
+
   sane.roles.build-machine.enable = true;
-  sane.roles.ac = true;
   sane.roles.client = true;
   sane.roles.dev-machine = true;
+  sane.roles.pc = true;
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."desko".wg-home.ip;
   sane.services.duplicity.enable = true;
   sane.services.nixserve.secretKeyFile = config.sops.secrets.nix_serve_privkey.path;
 
+  sane.nixcache.substituters.desko = false;
+  sane.nixcache.remote-builders.desko = false;
+
   sane.gui.sway.enable = true;
   sane.programs.iphoneUtils.enableFor.user.colin = true;
   sane.programs.steam.enableFor.user.colin = true;
 
-  sane.programs.guiApps.suggestedPrograms = [ "desktopGuiApps" ];
-  sane.programs.consoleUtils.suggestedPrograms = [ "consoleMediaUtils" "desktopConsoleUtils" ];
   # sane.programs.devPkgs.enableFor.user.colin = true;
 
+  sane.programs.signal-desktop.config.autostart = true;
+  sane.programs."gnome.geary".config.autostart = true;
+
   boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
 

hosts/by-name/desko/fs.nix

@@ -1,14 +1,13 @@
 { ... }:
 
 {
-  sane.persist.root-on-tmpfs = true;
   # increase /tmp space (defaults to 50% of RAM) for building large nix things.
   # a cross-compiled kernel, particularly, will easily use 30+GB of tmp
   fileSystems."/tmp".options = [ "size=64G" ];
 
   fileSystems."/nix" = {
-    # device = "/dev/disk/by-uuid/985a0a32-da52-4043-9df7-615adec2e4ff";
-    device = "/dev/disk/by-uuid/0ab0770b-7734-4167-88d9-6e4e20bb2a56";
+    # device = "/dev/disk/by-uuid/0ab0770b-7734-4167-88d9-6e4e20bb2a56";
+    device = "/dev/disk/by-uuid/845d85bf-761d-431b-a406-e6f20909154f";
     fsType = "btrfs";
     options = [
       "compress=zstd"
@@ -17,8 +16,8 @@
   };
 
   fileSystems."/boot" = {
-    # device = "/dev/disk/by-uuid/CAA7-E7D2";
-    device = "/dev/disk/by-uuid/41B6-BAEF";
+    # device = "/dev/disk/by-uuid/41B6-BAEF";
+    device = "/dev/disk/by-uuid/5049-9AFD";
     fsType = "vfat";
   };
 }

hosts/by-name/lappy/default.nix

@@ -7,19 +7,16 @@
 
   sane.roles.client = true;
   sane.roles.dev-machine = true;
+  sane.roles.pc = true;
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."lappy".wg-home.ip;
 
   # sane.guest.enable = true;
-  sane.gui.sxmo.enable = true;
+  sane.gui.sway.enable = true;
   boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
 
-  sane.programs.guiApps.suggestedPrograms = [
-    "desktopGuiApps"
-    "stepmania"
-  ];
-  sane.programs.consoleUtils.suggestedPrograms = [ "consoleMediaUtils" "desktopConsoleUtils" ];
+  sane.programs.stepmania.enableFor.user.colin = true;
 
   sops.secrets.colin-passwd.neededForUsers = true;
 

hosts/by-name/lappy/fs.nix

@@ -1,8 +1,6 @@
 { ... }:
 
 {
-  sane.persist.root-on-tmpfs = true;
-
   fileSystems."/nix" = {
     device = "/dev/disk/by-uuid/75230e56-2c69-4e41-b03e-68475f119980";
     fsType = "btrfs";

hosts/by-name/lappy/polyfill.nix

@@ -4,12 +4,14 @@
 {
   sane.gui.sxmo = {
     greeter = "greetd-sway-gtkgreet";
+    noidle = true;  #< power button requires 1s hold, which makes it impractical to be dealing with.
     settings = {
       # XXX: make sure the user is part of the `input` group!
       SXMO_LISGD_INPUT_DEVICE = "/dev/input/by-id/usb-Wacom_Co._Ltd._Pen_and_multitouch_sensor-event-if00";
       # these identifiers are from `swaymsg -t get_inputs`
       SXMO_VOLUME_BUTTON = "1:1:AT_Translated_Set_2_keyboard";
       # SXMO_VOLUME_BUTTON = "none";
+      # N.B.: thinkpad's power button requires a full second press to do anything
       SXMO_POWER_BUTTON = "0:1:Power_Button";
       # SXMO_POWER_BUTTON = "none";
       SXMO_DISABLE_LEDS = "1";
@@ -31,7 +33,7 @@
       #   and so it just wouldn't handle any button inputs (sxmo_hook_inputhandler.sh not on path)
       SXMO_DEVICE_NAME = "three_button_touchscreen";
     };
-    package = pkgs.sxmo-utils-latest.overrideAttrs (base: {
+    package = (pkgs.sxmo-utils.override { preferSystemd = true; }).overrideAttrs (base: {
       postPatch = (base.postPatch or "") + ''
         # after volume-button navigation mode, restore full keyboard functionality
         cp ${./xkb_mobile_normal_buttons} ./configs/xkb/xkb_mobile_normal_buttons

hosts/by-name/moby/default.nix

@@ -20,6 +20,7 @@
   ];
 
   sane.roles.client = true;
+  sane.roles.handheld = true;
   sane.zsh.showDeadlines = false;  # unlikely to act on them when in shell
   sane.services.wg-home.enable = true;
   sane.services.wg-home.ip = config.sane.hosts.by-name."moby".wg-home.ip;
@@ -31,28 +32,56 @@
 
   sops.secrets.colin-passwd.neededForUsers = true;
 
-  sane.user.persist.plaintext = [
-    # TODO: make this just generally conditional upon pulse being enabled?
-    ".config/pulse"  # persist pulseaudio volume
-  ];
-
   sane.gui.sxmo.enable = true;
-  sane.programs.guiApps.suggestedPrograms = [ "handheldGuiApps" ];
   # sane.programs.consoleUtils.enableFor.user.colin = false;
   # sane.programs.guiApps.enableFor.user.colin = false;
-  sane.programs.sequoia.enableFor.user.colin = false;
-  sane.programs.tuiApps.enableFor.user.colin = false;  # visidata, others, don't compile well
-  # disabled for faster deploys
-  sane.programs.soundconverter.enableFor.user.colin = false;
+  sane.programs.blueberry.enableFor.user.colin = false;  # bluetooth manager: doesn't cross compile!
+  sane.programs.dialect.enableFor.user.colin = false;  # drags in 700MB of x86 dependencies (e.g. gtk4)
+  sane.programs.mercurial.enableFor.user.colin = false;  # does not cross compile
+  sane.programs.nvme-cli.enableFor.system = false;  # does not cross compile (libhugetlbfs)
+
+  # enabled for easier debugging
   sane.programs.eg25-control.enableFor.user.colin = true;
+  sane.programs.rtl8723cs-wowlan.enableFor.user.colin = true;
 
-  # sane.programs.firefox.mime.priority = 300;  # prefer other browsers when possible
+  # sane.programs.ntfy-sh.config.autostart = true;
+  sane.programs.dino.config.autostart = true;
+  sane.programs.signal-desktop.config.autostart = true;
+  # sane.programs."gnome.geary".config.autostart = true;
+  # sane.programs.calls.config.autostart = true;
+  sane.programs.mpv.config.vo = "wlshm";  #< see hosts/common/programs/mpv.nix for details
+
+  sane.programs.firefox.mime.priority = 300;  # prefer other browsers when possible
   # HACK/TODO: make `programs.P.env.VAR` behave according to `mime.priority`
-  # sane.programs.firefox.env = lib.mkForce {};
-  # sane.programs.epiphany.env.BROWSER = "epiphany";
-  # sane.programs.firefox.enableFor.user.colin = false;  # use epiphany instead
+  sane.programs.firefox.env = lib.mkForce {};
+  sane.programs.epiphany.env.BROWSER = "epiphany";
 
-  # sane.programs.mpv.enableFor.user.colin = true;
+  # note the .conf.d approach: using ~/.config/pipewire/pipewire.conf directly breaks all audio,
+  # presumably because that deletes the defaults entirely whereas the .conf.d approach selectively overrides defaults
+  sane.user.fs.".config/pipewire/pipewire.conf.d/10-fix-dino-mic-cutout.conf".symlink.text = ''
+    # config docs: <https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/Config-PipeWire#properties>
+    # useful to run `pw-top` to see that these settings are actually having effect,
+    # and `pw-metadata` to see if any settings conflict (e.g. max-quantum < min-quantum)
+    #
+    # restart pipewire after editing these files:
+    # - `systemctl --user restart pipewire`
+    # - pipewire users will likely stop outputting audio until they are also restarted
+    #
+    # there's seemingly two buffers for the mic (see: <https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/FAQ#pipewire-buffering-explained>)
+    # 1. Pipewire buffering out of the driver and into its own member.
+    # 2. Pipewire buffering into Dino.
+    # the latter is fixed at 10ms by Dino, difficult to override via runtime config.
+    # the former defaults low (e.g. 512 samples)
+    # this default configuration causes the mic to regularly drop out entirely for a couple seconds at a time during a call,
+    # presumably because the system can't keep up (pw-top shows incrementing counter in ERR column).
+    # `pw-metadata -n settings 0 clock.force-quantum 1024` reduces to about 1 error per second.
+    # `pw-metadata -n settings 0 clock.force-quantum 2048` reduces to 1 error every < 10s.
+    # pipewire default config includes `clock.power-of-two-quantum = true`
+    context.properties = {
+      default.clock.min-quantum = 2048
+      default.clock.max-quantum = 8192
+    }
+  '';
 
   boot.loader.efi.canTouchEfiVariables = false;
   # /boot space is at a premium. default was 20.
@@ -62,23 +91,24 @@
   # mobile.boot.stage-1.enable = false;
   # boot.initrd.systemd.enable = false;
   # boot.initrd.services.swraid.enable = false;  # attempt to fix dm_mod stuff
-  # disable proximity sensor.
-  # the filtering/calibration is bad that it causes the screen to go fully dark at times.
-  boot.blacklistedKernelModules = [ "stk3310" ];
-
-  # without this some GUI apps fail: `DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory`
-  # this is because they can't allocate enough video ram.
-  # the default CMA seems to be 32M.
-  # i was running fine with 256MB from 2022/07-ish through 2022/12-ish, but then the phone quit reliably coming back from sleep: maybe a memory leak?
-  # `cat /proc/meminfo` to see CmaTotal/CmaFree if interested in tuning this.
-  boot.kernelParams = [ "cma=512M" ];
 
+  # hardware.firmware makes the referenced files visible to the kernel, for whenever a driver explicitly asks for them.
+  # these files are visible from userspace by following `/sys/module/firmware_class/parameters/path`
+  #
   # mobile-nixos' /lib/firmware includes:
   #   rtl_bt          (bluetooth)
-  #   anx7688-fw.bin  (USB-C -> HDMI bridge)
+  #   anx7688-fw.bin  (USB-C chip: power negotiation, HDMI/dock)
   #   ov5640_af.bin   (camera module)
   # hardware.firmware = [ config.mobile.device.firmware ];
-  hardware.firmware = [ pkgs.rtl8723cs-firmware ];
+  # hardware.firmware = [ pkgs.rtl8723cs-firmware ];
+  hardware.firmware = [
+    (pkgs.linux-firmware-megous.override {
+      # rtl_bt = false probably means no bluetooth connectivity.
+      # N.B.: DON'T RE-ENABLE without first confirming that wake-on-lan works during suspend (rtcwake).
+      # it seems the rtl_bt stuff ("bluetooth coexist") might make wake-on-LAN radically more flaky.
+      rtl_bt = false;
+    })
+  ];
 
   system.stateVersion = "21.11";
 
@@ -96,16 +126,17 @@
   # see pkgs/patched/alsa-ucm-conf for more info.
   environment.variables.ALSA_CONFIG_UCM2 = "/run/current-system/sw/share/alsa/ucm2";
   environment.pathsToLink = [ "/share/alsa/ucm2" ];
-  environment.systemPackages = [ pkgs.alsa-ucm-conf-sane ];
-  systemd =
-    let ucm-env = config.environment.variables.ALSA_CONFIG_UCM2;
+  environment.systemPackages = [
+    (pkgs.alsa-ucm-conf-sane.override {
+      # internal speaker has a tendency to break :(
+      preferEarpiece = true;
+    })
+  ];
+  systemd = let
+    ucm-env = config.environment.variables.ALSA_CONFIG_UCM2;
   in {
     # cribbed from <repo:nixos/mobile-nixos:modules/quirks/audio.nix>
 
-    # pulseaudio
-    user.services.pulseaudio.environment.ALSA_CONFIG_UCM2 = ucm-env;
-    services.pulseaudio.environment.ALSA_CONFIG_UCM2      = ucm-env;
-
     # pipewire
     user.services.pipewire.environment.ALSA_CONFIG_UCM2       = ucm-env;
     user.services.pipewire-pulse.environment.ALSA_CONFIG_UCM2 = ucm-env;
@@ -114,6 +145,10 @@
     services.pipewire-pulse.environment.ALSA_CONFIG_UCM2      = ucm-env;
     services.wireplumber.environment.ALSA_CONFIG_UCM2         = ucm-env;
 
+    # pulseaudio
+    # user.services.pulseaudio.environment.ALSA_CONFIG_UCM2 = ucm-env;
+    # services.pulseaudio.environment.ALSA_CONFIG_UCM2      = ucm-env;
+
 
     # TODO: move elsewhere...
     services.ModemManager.serviceConfig = {
@@ -135,6 +170,4 @@
     # make Pinephone front LEDs writable by user.
     SUBSYSTEM=="leds", DEVPATH=="*/*:indicator", RUN+="${chmod} g+w /sys%p/brightness", RUN+="${chown} :video /sys%p/brightness"
   '';
-
-  hardware.opengl.driSupport = true;
 }

hosts/by-name/moby/fs.nix

@@ -1,7 +1,6 @@
 { ... }:
 
 {
-  sane.persist.root-on-tmpfs = true;
   fileSystems."/nix" = {
     device = "/dev/disk/by-uuid/1f1271f8-53ce-4081-8a29-60a4a6b5d6f9";
     fsType = "btrfs";

hosts/by-name/moby/gps.nix

@@ -7,15 +7,38 @@
 # - `screen /dev/ttyUSB2 115200`
 #   - `AT+QGPSCFG="nmeasrc",1`
 #   - `AT+QGPS=1`
+# this process is automated by my `eg25-control` program and services (`eg25-control-powered`, `eg25-control-gps`)
+# - see the `modules/` directory further up this repository.
 #
-# now, something like `gpsd` can directly read from /dev/ttyUSB1.
+# now, something like `gpsd` can directly read from /dev/ttyUSB1,
+# or geoclue can query the GPS directly through modem-manager
 #
 # initial GPS fix can take 15+ minutes.
-# meanwhile, services like eg25-manager can speed this up by uploading assisted GPS data to the modem.
+# meanwhile, services like eg25-manager or eg25-control-freshen-agps can speed this up by uploading assisted GPS data to the modem.
 #
-# geoclue somehow fits in here as a geospatial provider that leverages GPS and also other sources like radio towers
+# support/help:
+# - geoclue, gnome-maps
+#   - irc: #gnome-maps on irc.gimp.org
+#   - Matrix: #gnome-maps:gnome.org  (unclear if bridged to IRC)
+#
+# programs to pair this with:
+# - `satellite-gtk`: <https://codeberg.org/tpikonen/satellite>
+#   - shows/tracks which satellites the GPS is connected to; useful to understand fix characteristics
+# - `gnome-maps`: uses geoclue, has route planning
+# - `mepo`: uses gpsd, minimalist, flaky, and buttons are kinda hard to activate on mobile
+# - puremaps?
+# - osmin?
+#
+# known/outstanding bugs:
+# - `systemctl start eg25-control-gps` can the hang the whole system (2023/10/06)
+#   - i think it's actually `eg25-control-powered` which does this (started by the gps)
+#   - best guess is modem draws so much power at launch that other parts of the system see undervoltage
+#   - workaround is to hard power-cycle the system. the modem may not bring up after reboot: leave unpowered for 60s and boot again.
+#
+# future work:
+# - integrate with [wigle](https://www.wigle.net/) for offline equivalent to Mozilla Location Services
 
-{ lib, ... }:
+{ config, lib, ... }:
 {
   # test gpsd with `gpspipe -w -n 10 2> /dev/null | grep -m 1 TPV | jq '.lat, .lon' | tr '\n' ' '`
   # ^ should return <lat> <long>
@@ -23,20 +46,24 @@
   services.gpsd.devices = [ "/dev/ttyUSB1" ];
 
   # test geoclue2 by building `geoclue2-with-demo-agent`
-  # and running "${geoclue2-with-demo-agent}/libexec/geoclue-2/demos/where-am-i"
+  # and running "${geoclue2-with-demo-agent}/libexec/geoclue-2.0/demos/where-am-i"
+  # note that geoclue is dbus-activated, and auto-stops after 60s with no caller
   services.geoclue2.enable = true;
   services.geoclue2.appConfig.where-am-i = {
     # this is the default "agent", shipped by geoclue package: allow it to use location
     isAllowed = true;
     isSystem = false;
     # XXX: setting users != [] might be causing `where-am-i` to time out
-    # users = [
-    #   # restrict to only one set of users. empty array (default) means "allow any user to access geolocation".
-    #   (builtins.toString config.users.users.colin.uid)
-    # ];
+    users = [
+      # restrict to only one set of users. empty array (default) means "allow any user to access geolocation".
+      (builtins.toString config.users.users.colin.uid)
+    ];
   };
   systemd.services.geoclue.after = lib.mkForce [];  #< defaults to network-online, but not all my sources require network
-
+  users.users.geoclue.extraGroups = [
+    "dialout"  # TODO: figure out if dialout is required. that's for /dev/ttyUSB1, but geoclue probably doesn't read that?
+  ];
 
   sane.services.eg25-control.enable = true;
+  sane.programs.where-am-i.enableFor.user.colin = true;
 }

hosts/by-name/moby/kernel.nix

@@ -66,6 +66,24 @@ in
     # target = "zImage";  # <-- confuses other parts of nixos :-(
   };
 
+  # disable proximity sensor.
+  # the filtering/calibration is bad that it causes the screen to go fully dark at times.
+  boot.blacklistedKernelModules = [ "stk3310" ];
+
+  boot.kernelParams = [
+    # without this some GUI apps fail: `DRM_IOCTL_MODE_CREATE_DUMB failed: Cannot allocate memory`
+    # this is because they can't allocate enough video ram.
+    # see related nixpkgs issue: <https://github.com/NixOS/nixpkgs/issues/260222>
+    #
+    # the default CMA seems to be 32M.
+    # i was running fine with 256MB from 2022/07-ish through 2022/12-ish, but then the phone quit reliably coming back from sleep (phosh): maybe a memory leak?
+    # `cat /proc/meminfo` to see CmaTotal/CmaFree if interested in tuning this.
+    "cma=512M"
+    # 2023/10/20: potential fix for the lima (GPU) timeout bugs:
+    # - <https://gitlab.com/postmarketOS/pmaports/-/issues/805#note_890467824>
+    "lima.sched_timeout_ms=2000"
+  ];
+
   services.xserver.displayManager.job.preStart = ensureHWReady;
   systemd.services.greetd.preStart = ensureHWReady;
 }

hosts/by-name/moby/polyfill.nix

@@ -9,13 +9,6 @@
 # - <https://itsfoss.com/content/images/2023/04/nixos-tutorials.png>
 
 { lib, pkgs, sane-lib, ... }:
-let
-  # TODO: generate this from the .svg
-  # bg = ./nixos-bg-02.png;
-  bg = pkgs.runCommand "nixos-bg.png" { nativeBuildInputs = [ pkgs.inkscape ]; } ''
-    inkscape ${./nixos-bg-02.svg} -o $out
-  '';
-in
 {
   sane.programs.firefox.config = {
     # compromise impermanence for the sake of usability
@@ -27,6 +20,9 @@ in
     # sidebery UX doesn't make sense on small screen
     addons.sidebery.enable = false;
   };
+  sane.programs.swaynotificationcenter.config = {
+    backlight = "backlight";  # /sys/class/backlight/*backlight*/brightness
+  };
 
   sane.gui.sxmo = {
     nogesture = true;
@@ -35,94 +31,11 @@ in
       SXMO_LISGD_INPUT_DEVICE = "/dev/input/by-path/platform-1c2ac00.i2c-event";
       # vol and power are detected correctly by upstream
 
-
       ### preferences
-      # notable bemenu options:
-      #   - see `bemenu --help` for all
-      #   -P, --prefix          text to show before highlighted item.
-      #   --scrollbar           display scrollbar. (none (default), always, autohide)
-      #   -H, --line-height     defines the height to make each menu line (0 = default height). (wx)
-      #   -M, --margin          defines the empty space on either side of the menu. (wx)
-      #   -W, --width-factor    defines the relative width factor of the menu (from 0 to 1). (wx)
-      #   -B, --border          defines the width of the border in pixels around the menu. (wx)
-      #   -R  --border-radius   defines the radius of the border around the menu (0 = no curved borders).
-      #   --ch                  defines the height of the cursor (0 = scales with line height). (wx)
-      #   --cw                  defines the width of the cursor. (wx)
-      #   --hp                  defines the horizontal padding for the entries in single line mode. (wx)
-      #   --fn                  defines the font to be used ('name [size]'). (wx)
-      #   --tb                  defines the title background color. (wx)
-      #   --tf                  defines the title foreground color. (wx)
-      #   --fb                  defines the filter background color. (wx)
-      #   --ff                  defines the filter foreground color. (wx)
-      #   --nb                  defines the normal background color. (wx)
-      #   --nf                  defines the normal foreground color. (wx)
-      #   --hb                  defines the highlighted background color. (wx)
-      #   --hf                  defines the highlighted foreground color. (wx)
-      #   --fbb                 defines the feedback background color. (wx)
-      #   --fbf                 defines the feedback foreground color. (wx)
-      #   --sb                  defines the selected background color. (wx)
-      #   --sf                  defines the selected foreground color. (wx)
-      #   --ab                  defines the alternating background color. (wx)
-      #   --af                  defines the alternating foreground color. (wx)
-      #   --scb                 defines the scrollbar background color. (wx)
-      #   --scf                 defines the scrollbar foreground color. (wx)
-      #   --bdr                 defines the border color. (wx)
-      #
-      # colors are specified as `#RRGGBB`
-      # defaults:
-      # --ab "#222222"
-      # --af "#bbbbbb"
-      # --bdr "#005577"
-      # --border 3
-      # --cb "#222222"
-      # --center
-      # --cf "#bbbbbb"
-      # --fb "#222222"
-      # --fbb "#eeeeee"
-      # --fbf "#222222"
-      # --ff "#bbbbbb"
-      # --fixed-height
-      # --fn 'Sxmo 14'
-      # --hb "#005577"
-      # --hf "#eeeeee"
-      # --line-height 20
-      # --list 16
-      # --margin 40
-      # --nb "#222222"
-      # --nf "#bbbbbb"
-      # --no-overlap
-      # --no-spacing
-      # --sb "#323232"
-      # --scb "#005577"
-      # --scf "#eeeeee"
-      # --scrollbar autohide
-      # --tb "#005577"
-      # --tf "#eeeeee"
-      # --wrap
-      BEMENU_OPTS = let
-        bg = "#1d1721";       # slight purple
-        fg0 = "#d8d8d8";      # inactive text (light grey)
-        fg1 = "#ffffff";      # active text   (white)
-        accent0 = "#1f5e54";  # darker but saturated teal
-        accent1 = "#418379";  # teal (matches nixos-bg)
-        accent2 = "#5b938a";  # brighter but muted teal
-      in lib.concatStringsSep " " [
-        "--wrap --scrollbar autohide --fixed-height"
-        "--center --margin 45"
-        "--no-spacing"
-        # XXX: font size doesn't seem to take effect (would prefer larger)
-        "--fn 'Sxmo 14' --line-height 22 --border 3"
-        "--bdr '${accent0}'"                     # border
-        "--scf '${accent2}' --scb '${accent0}'"  # scrollbar
-        "--tb '${accent0}' --tf '${fg0}'"        # title
-        "--fb '${accent0}' --ff '${fg1}'"        # filter (i.e. text that's been entered)
-        "--hb '${accent1}' --hf '${fg1}'"        # selected item
-        "--nb '${bg}' --nf '${fg0}'"             # normal lines (even)
-        "--ab '${bg}' --af '${fg0}'"             # alternated lines (odd)
-        "--cf '${accent0}' --cb '${accent0}'"    # cursor (not very useful)
-      ];
       DEFAULT_COUNTRY = "US";
 
+      SXMO_AUTOROTATE = "1";  # enable auto-rotation at launch. has no meaning in stock/upstream sxmo-utils
+
       # BEMENU lines (wayland DMENU):
       # - camera is 9th entry
       # - flashlight is 10th entry
@@ -134,7 +47,6 @@ in
       # - close is 16th entry
       SXMO_BEMENU_LANDSCAPE_LINES = "11";  # default 8
       SXMO_BEMENU_PORTRAIT_LINES = "16";  # default 16
-      SXMO_BG_IMG = "${bg}";
       SXMO_LOCK_IDLE_TIME = "15";  # how long between screenoff -> lock -> back to screenoff (default: 8)
       # gravity: how far to tilt the device before the screen rotates
       # for a given setting, normal <-> invert requires more movement then left <-> right
@@ -144,9 +56,22 @@ in
       # SXMO_ROTATION_GRAVITY = "12500";    # kinda uncomfortable when walking
       SXMO_ROTATION_GRAVITY = "12000";
       SXMO_SCREENSHOT_DIR = "/home/colin/Pictures";  # default: "$HOME"
-      # test new scales by running `swaymsg -- output DSI-1 scale x.y`
+
+      # sway/wayland scaling:
+      # - conflicting info out there on how scaling actually works
+      #   at the least, for things where it matters (mpv), it seems like scale settings have 0 effect on perf
+      # ways to enforce scaling:
+      # - <https://wiki.archlinux.org/title/HiDPI>
+      # - `swaymsg -- output DSI-1 scale 2.0`  (scales everything)
+      # - `dconf write /org/gnome/desktop/interface/text-scaling-factor 2.0`  (scales ONLY TEXT)
+      # - `GDK_DPI_SCALE=2.0`  (scales ONLY TEXT)
+      #
+      # application notes:
+      # - cozy: in landscape, playback position is not visible unless scale <= 1.7
+      #   - if in a tab, then scale 1.6 is the max
       # SXMO_SWAY_SCALE = "1.5";  # hard to press gPodder icons
-      SXMO_SWAY_SCALE = "1.8";
+      SXMO_SWAY_SCALE = "1.6";
+      # SXMO_SWAY_SCALE = "1.8";
       # SXMO_SWAY_SCALE = "2";
       SXMO_WORKSPACE_WRAPPING = "5";  # how many workspaces. default: 4
 
@@ -167,13 +92,5 @@ in
       WVKBD_LANDSCAPE_LAYERS = "landscape,special,emoji";
       WVKBD_LAYERS = "full,special,emoji";
     };
-    package = pkgs.sxmo-utils-latest.overrideAttrs (base: {
-      postPatch = (base.postPatch or "") + ''
-        cat <<EOF >> ./configs/default_hooks/sxmo_hook_start.sh
-        # rotate UI based on physical display angle by default
-        sxmo_daemons.sh start autorotate sxmo_autorotate.sh
-        EOF
-      '';
-    });
   };
 }

hosts/by-name/rescue/default.nix

@@ -4,12 +4,15 @@
     ./fs.nix
   ];
 
-  boot.loader.generic-extlinux-compatible.enable = true;
   boot.loader.efi.canTouchEfiVariables = false;
   sane.image.extraBootFiles = [ pkgs.bootpart-uefi-x86_64 ];
-  # sane.persist.enable = false;  # TODO: disable (but run `nix flake check` to ensure it works!)
+  sane.persist.enable = false;
   sane.nixcache.enable = false;  # don't want to be calling out to dead machines that we're *trying* to rescue
 
+  # auto-login at shell
+  services.getty.autologinUser = "colin";
+  # users.users.colin.initialPassword = "colin";
+
   # docs: https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion
   system.stateVersion = "21.05";
 }

hosts/by-name/rescue/fs.nix

@@ -1,7 +1,7 @@
 { ... }:
 
 {
-  fileSystems."/" = {
+  fileSystems."/nix" = {
     device = "/dev/disk/by-uuid/44445555-6666-7777-8888-999900001111";
     fsType = "ext4";
   };

hosts/by-name/servo/default.nix

@@ -14,20 +14,23 @@
     signaldctl.enableFor.user.colin = true;
   };
 
-  sane.roles.ac = true;
   sane.roles.build-machine.enable = true;
   sane.roles.build-machine.emulation = false;
   sane.zsh.showDeadlines = false;  # ~/knowledge doesn't always exist
   sane.programs.consoleUtils.suggestedPrograms = [
-    "desktopConsoleUtils"
+    "pcConsoleUtils"
     "sane-scripts.stop-all-servo"
   ];
   sane.services.dyn-dns.enable = true;
   sane.services.wg-home.enable = true;
-  sane.services.wg-home.enableWan = true;
+  sane.services.wg-home.visibleToWan = true;
+  sane.services.wg-home.forwardToWan = true;
+  sane.services.wg-home.routeThroughServo = false;
   sane.services.wg-home.ip = config.sane.hosts.by-name."servo".wg-home.ip;
   sane.nixcache.substituters.servo = false;
   sane.nixcache.substituters.desko = false;
+  sane.nixcache.remote-builders.desko = false;
+  sane.nixcache.remote-builders.servo = false;
   # sane.services.duplicity.enable = true;  # TODO: re-enable after HW upgrade
 
   # automatically log in at the virtual consoles.

hosts/by-name/servo/fs.nix

@@ -1,7 +1,6 @@
 { ... }:
 
 {
-  sane.persist.root-on-tmpfs = true;
   # increase /tmp space (defaults to 50% of RAM) for building large nix things.
   # even the stock `nixpkgs.linux` consumes > 16 GB of tmp
   fileSystems."/tmp".options = [ "size=32G" ];
@@ -36,12 +35,40 @@
   };
   sane.fs."/mnt/persist/ext".mount = {};
 
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: this is overly broad; only need media and share directories to be persisted
     { user = "colin"; group = "users"; path = "/var/lib/uninsane"; }
   ];
+  # force some problematic directories to always get correct permissions:
+  sane.fs."/var/lib/uninsane/media".dir.acl = {
+    user = "colin"; group = "media"; mode = "0775";
+  };
+  sane.fs."/var/lib/uninsane/media/archive".dir = {};
+  sane.fs."/var/lib/uninsane/media/archive/README.md".file.text = ''
+    this directory is for media i wish to remove from my library,
+    but keep for a short time in case i reverse my decision.
+    treat it like a system trash can.
+  '';
+  sane.fs."/var/lib/uninsane/media/Books".dir = {};
+  sane.fs."/var/lib/uninsane/media/Books/Audiobooks".dir = {};
+  sane.fs."/var/lib/uninsane/media/Books/Books".dir = {};
+  sane.fs."/var/lib/uninsane/media/Books/Visual".dir = {};
+  sane.fs."/var/lib/uninsane/media/collections".dir = {};
+  sane.fs."/var/lib/uninsane/media/datasets".dir = {};
+  sane.fs."/var/lib/uninsane/media/freeleech".dir = {};
+  sane.fs."/var/lib/uninsane/media/Music".dir = {};
+  sane.fs."/var/lib/uninsane/media/Pictures".dir = {};
+  sane.fs."/var/lib/uninsane/media/Videos".dir = {};
+  sane.fs."/var/lib/uninsane/media/Videos/Film".dir = {};
+  sane.fs."/var/lib/uninsane/media/Videos/Shows".dir = {};
+  sane.fs."/var/lib/uninsane/media/Videos/Talks".dir = {};
+  sane.fs."/var/lib/uninsane/datasets/README.md".file.text = ''
+    this directory may seem redundant with ../media/datasets. it isn't.
+    this directory exists on SSD, allowing for speedy access to specific datasets when necessary.
+    the contents should be a subset of what's in ../media/datasets.
+  '';
   # make sure large media is stored to the HDD
-  sane.persist.sys.ext = [
+  sane.persist.sys.byStore.ext = [
     {
       user = "colin";
       group = "users";

hosts/by-name/servo/net.nix

@@ -1,6 +1,24 @@
-{ config, pkgs, ... }:
+{ config, lib, pkgs, ... }:
 
+let
+  portOpts = with lib; types.submodule {
+    options = {
+      visibleTo.ovpn = mkOption {
+        type = types.bool;
+        default = false;
+      };
+    };
+  };
+in
 {
+  options = with lib; {
+    sane.ports.ports = mkOption {
+      # add the `visibleTo.ovpn` option
+      type = types.attrsOf portOpts;
+    };
+  };
+
+  config = {
     networking.domain = "uninsane.org";
 
     sane.ports.openFirewall = true;
@@ -83,6 +101,14 @@
       vpn-ip = "185.157.162.178";
       # DNS = 46.227.67.134, 192.165.9.158, 2a07:a880:4601:10f0:cd45::1, 2001:67c:750:1:cafe:cd45::1
       vpn-dns = "46.227.67.134";
+      bridgePort = port: proto: ''
+        ${in-ns} ${iptables} -A PREROUTING -t nat -p ${proto} --dport ${port} -m iprange --dst-range ${vpn-ip} \
+          -j DNAT --to-destination ${veth-host-ip}
+      '';
+      bridgeStatements = lib.foldlAttrs
+        (acc: port: portCfg: acc ++ (builtins.map (bridgePort port) portCfg.protocol))
+        []
+        config.sane.ports.ports;
     in {
       privateKeyFile = config.sops.secrets.wg_ovpns_privkey.path;
       # wg is active only in this namespace.
@@ -109,10 +135,10 @@
           # dynamicEndpointRefreshRestartSeconds = 5;
         }
       ];
-    preSetup = "" + ''
+      preSetup = ''
         ${ip} netns add ovpns || echo "ovpns already exists"
       '';
-    postShutdown = "" + ''
+      postShutdown = ''
         ${in-ns} ip link del ovpns-veth-b || echo "couldn't delete ovpns-veth-b"
         ${ip} link del ovpns-veth-a || echo "couldn't delete ovpns-veth-a"
         ${ip} netns delete ovpns || echo "couldn't delete ovpns"
@@ -122,7 +148,7 @@
         ${ip} rule add from all lookup local pref 0
         ${ip} rule del from all lookup local pref 100
       '';
-    postSetup = "" + ''
+      postSetup = ''
         # DOCS:
         # - some of this approach is described here: <https://josephmuia.ca/2018-05-16-net-namespaces-veth-nat/>
         # - iptables primer: <https://danielmiessler.com/study/iptables/>
@@ -146,25 +172,11 @@
         ${ip} rule add from all lookup local pref 100
         ${ip} rule del from all lookup local pref 0
 
-      # bridge HTTP traffic:
-      # any external port-80 request sent to the VPN addr will be forwarded to the rootns.
-      # this exists so LetsEncrypt can procure a cert for the MX over http.
-      # TODO: we could use _acme_challence.mx.uninsane.org CNAME to avoid this forwarding
-      # - <https://community.letsencrypt.org/t/where-does-letsencrypt-resolve-dns-from/37607/8>
-      ${in-ns} ${iptables} -A PREROUTING -t nat -p tcp --dport 80 -m iprange --dst-range ${vpn-ip} \
-        -j DNAT --to-destination ${veth-host-ip}:80
-
-      # we also bridge DNS traffic
-      ${in-ns} ${iptables} -A PREROUTING -t nat -p udp --dport 53 -m iprange --dst-range ${vpn-ip} \
-        -j DNAT --to-destination ${veth-host-ip}
-      ${in-ns} ${iptables} -A PREROUTING -t nat -p tcp --dport 53 -m iprange --dst-range ${vpn-ip} \
-        -j DNAT --to-destination ${veth-host-ip}
-
         # in order to access DNS in this netns, we need to route it to the VPN's nameservers
         # - alternatively, we could fix DNS servers like 1.1.1.1.
         ${in-ns} ${iptables} -A OUTPUT -t nat -p udp --dport 53 -m iprange --dst-range 127.0.0.53 \
           -j DNAT --to-destination ${vpn-dns}:53
-    '';
+      '' + (lib.concatStringsSep "\n" bridgeStatements);
     };
 
     # create a new routing table that we can use to proxy traffic out of the root namespace
@@ -217,4 +229,5 @@
     # # test with:
     # #   curl --interface hurricane http://[2607:f8b0:400a:80b::2004]
     # #   ping 2607:f8b0:400a:80b::2004
+  };
 }

hosts/by-name/servo/services/calibre.nix

@@ -12,7 +12,7 @@ in
 # > AttributeError: 'NoneType' object has no attribute 'query'
 lib.mkIf false
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { inherit user group; mode = "0700"; path = svc-dir; }
   ];
 

hosts/by-name/servo/services/coturn.nix

@@ -0,0 +1,124 @@
+# TURN/STUN NAT traversal service
+# commonly used to establish realtime calls with prosody, or possibly matrix/synapse
+#
+# - <https://github.com/coturn/coturn/>
+# - `man turnserver`
+# - config docs: <https://github.com/coturn/coturn/blob/master/examples/etc/turnserver.conf>
+#
+# N.B. during operation it's NORMAL to see "error 401".
+# during session creation:
+# - client sends Allocate request
+# - server replies error 401, providing a realm and nonce
+# - client uses realm + nonce + shared secret to construct an auth key & call Allocate again
+# - server replies Allocate Success Response
+# - source: <https://stackoverflow.com/a/66643135>
+#
+# N.B. this safest implementation routes all traffic THROUGH A VPN
+# - that adds a lot of latency, but in practice turns out to be inconsequential.
+#   i guess ICE allows clients to prefer the other party's lower-latency server, in practice?
+# - still, this is the "safe" implementation because STUN works with IP addresses instead of domain names:
+#   1. client A queries the STUN server to determine its own IP address/port.
+#   2. client A tells client B which IP address/port client A is visible on.
+#   3. client B contacts that IP address/port
+#   this only works so long as the IP address/port which STUN server sees client A on is publicly routable.
+#   that is NOT the case when the STUN server and client A are on the same LAN
+#   even if client A contacts the STUN server via its WAN address with port reflection enabled.
+#   hence, there's no obvious way to put the STUN server on the same LAN as either client and expect the rest to work.
+{ lib, ... }:
+let
+  # TODO: this range could be larger, but right now that's costly because each element is its own UPnP forward
+  # TURN port range (inclusive)
+  turnPortLow = 49152;
+  turnPortHigh = 49167;
+  turnPortRange = lib.range turnPortLow turnPortHigh;
+in
+{
+  sane.ports.ports = lib.mkMerge ([
+    {
+      "3478" = {
+        # this is the "control" port.
+        # i.e. no client data is forwarded through it, but it's where clients request tunnels.
+        protocol = [ "tcp" "udp" ];
+        # visibleTo.lan = true;
+        # visibleTo.wan = true;
+        visibleTo.ovpn = true;
+        description = "colin-stun-turn";
+      };
+      "5349" = {
+        # the other port 3478 also supports TLS/DTLS, but presumably clients wanting TLS will default 5349
+        protocol = [ "tcp" ];
+        # visibleTo.lan = true;
+        # visibleTo.wan = true;
+        visibleTo.ovpn = true;
+        description = "colin-stun-turn-over-tls";
+      };
+    }
+  ] ++ (builtins.map
+    (port: {
+      "${builtins.toString port}" = let
+        count = port - turnPortLow + 1;
+        numPorts = turnPortHigh - turnPortLow + 1;
+      in {
+        protocol = [ "tcp" "udp" ];
+        # visibleTo.lan = true;
+        # visibleTo.wan = true;
+        visibleTo.ovpn = true;
+        description = "colin-turn-${builtins.toString count}-of-${builtins.toString numPorts}";
+      };
+    })
+    turnPortRange
+  ));
+
+  services.nginx.virtualHosts."turn.uninsane.org" = {
+    # allow ACME to procure a cert via nginx for this domain
+    enableACME = true;
+  };
+  sane.dns.zones."uninsane.org".inet = {
+    # CNAME."turn" = "servo.wan";
+    # CNAME."turn" = "ovpns";
+    # CNAME."turn" = "native";
+    # XXX: SRV records have to point to something with a A/AAAA record; no CNAMEs
+    A."turn" = "%AOVPNS%";
+    # A."turn" = "%AWAN%";
+
+    SRV."_stun._udp" =                        "5 50 3478 turn";
+    SRV."_stun._tcp" =                        "5 50 3478 turn";
+    SRV."_stuns._tcp" =                       "5 50 5349 turn";
+    SRV."_turn._udp" =                        "5 50 3478 turn";
+    SRV."_turn._tcp" =                        "5 50 3478 turn";
+    SRV."_turns._tcp" =                       "5 50 5349 turn";
+  };
+
+  sane.derived-secrets."/var/lib/coturn/shared_secret.bin" = {
+    encoding = "base64";
+    # TODO: make this not globally readable
+    acl.mode = "0644";
+  };
+  sane.fs."/var/lib/coturn/shared_secret.bin".wantedBeforeBy = [ "coturn.service" ];
+
+  # provide access to certs
+  users.users.turnserver.extraGroups = [ "nginx" ];
+
+  services.coturn.enable = true;
+  services.coturn.realm = "turn.uninsane.org";
+  services.coturn.cert = "/var/lib/acme/turn.uninsane.org/fullchain.pem";
+  services.coturn.pkey = "/var/lib/acme/turn.uninsane.org/key.pem";
+  services.coturn.use-auth-secret = true;
+  services.coturn.static-auth-secret-file = "/var/lib/coturn/shared_secret.bin";
+  services.coturn.lt-cred-mech = true;
+  services.coturn.min-port = turnPortLow;
+  services.coturn.max-port = turnPortHigh;
+  # services.coturn.secure-stun = true;
+  services.coturn.extraConfig = lib.concatStringsSep "\n" [
+    "verbose"
+    # "Verbose"  #< even MORE verbosity than "verbose"
+    # "no-multicast-peers"  # disables sending to IPv4 broadcast addresses (e.g. 224.0.0.0/3)
+    "listening-ip=10.0.1.5"
+    # "external-ip=185.157.162.178/10.0.1.5"
+    "external-ip=185.157.162.178"
+    # "listening-ip=10.78.79.51"  # can be specified multiple times; omit for *
+    # "external-ip=97.113.128.229/10.78.79.51"
+    # "external-ip=97.113.128.229"
+    # "mobility"  # "mobility with ICE (MICE) specs support" (?)
+  ];
+}

hosts/by-name/servo/services/default.nix

@@ -2,6 +2,7 @@
 {
   imports = [
     ./calibre.nix
+    ./coturn.nix
     ./ddns-afraid.nix
     ./ddns-he.nix
     ./email
@@ -17,13 +18,16 @@
     ./komga.nix
     ./lemmy.nix
     ./matrix
+    ./monero.nix
     ./navidrome.nix
-    ./nixserve.nix
     ./nginx.nix
+    ./nixserve.nix
+    ./ntfy
     ./pict-rs.nix
     ./pleroma.nix
     ./postgres.nix
-    ./prosody.nix
+    ./prosody
+    ./slskd.nix
     ./transmission.nix
     ./trust-dns.nix
     ./wikipedia.nix

hosts/by-name/servo/services/ejabberd.nix

@@ -40,8 +40,11 @@ let
   turnPortHigh = 49167;
   turnPortRange = lib.range turnPortLow turnPortHigh;
 in
+# XXX(2023/10/15): disabled in favor of Prosody.
+# everything configured below was fine: used ejabberd for several months.
+lib.mkIf false
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "ejabberd"; group = "ejabberd"; path = "/var/lib/ejabberd"; }
   ];
   sane.ports.ports = lib.mkMerge ([
@@ -114,6 +117,9 @@ in
     turnPortRange
   ));
 
+  # this ejabberd config uses builtin STUN/TURN server, so hack to ensure no other implementation fights for ports
+  services.coturn.enable = false;
+
   # provide access to certs
   # TODO: this should just be `acme`. then we also add nginx to the `acme` group.
   #   why is /var/lib/acme/* owned by `nginx` group??
@@ -179,284 +185,285 @@ in
   services.ejabberd.enable = true;
   services.ejabberd.configFile = "/var/lib/ejabberd/ejabberd.yaml";
   systemd.services.ejabberd.preStart = let
-    config-in = pkgs.writeTextFile {
-      name = "ejabberd.yaml.in";
-      text = ''
-        hosts:
-          - uninsane.org
-
+    config-in = pkgs.writeText "ejabberd.yaml.in" (lib.generators.toYAML {} {
+      hosts = [ "uninsane.org" ];
       # none | emergency | alert | critical | error | warning | notice | info | debug
-        loglevel: debug
-        # loglevel: info
-        # loglevel: notice
+      loglevel = "debug";
+      acme.auto = false;
+      certfiles = [ "/var/lib/acme/uninsane.org/full.pem" ];
+      # ca_file = "${pkgs.cacert.unbundled}/etc/ssl/certs/";
+      # ca_file = "${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt";
 
-        acme:
-          auto: false
-        certfiles:
-        - /var/lib/acme/uninsane.org/full.pem
-        # ca_file: ${pkgs.cacert.unbundled}/etc/ssl/certs/
-        # ca_file: ${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt
+      pam_userinfotype = "jid";
+      acl = {
+        admin.user = [ "colin@uninsane.org" ];
+        local.user_regexp = "";
+        loopback.ip = [ "127.0.0.0/8" "::1/128" ];
+      };
 
-        pam_userinfotype: jid
-
-        acl:
-          admin:
-            user:
-              - "colin@uninsane.org"
-          local:
-            user_regexp: ""
-          loopback:
-            ip:
-              - 127.0.0.0/8
-              - ::1/128
-
-        access_rules:
-          local:
-            allow: local
-          c2s_access:
-            allow: all
-          announce:
-            allow: admin
-          configure:
-            allow: admin
-          muc_create:
-            allow: local
-          pubsub_createnode_access:
-            allow: all
-          trusted_network:
-            allow: loopback
+      access_rules = {
+        local.allow = "local";
+        c2s_access.allow = "all";
+        announce.allow = "admin";
+        configure.allow = "admin";
+        muc_create.allow = "local";
+        pubsub_createnode_access.allow = "all";
+        trusted_network.allow = "loopback";
+      };
 
       # docs: <https://docs.ejabberd.im/admin/configuration/basic/#shaper-rules>
-        shaper_rules:
+      shaper_rules = {
         # setting this to above 1 may break outgoing messages
         # - maybe some servers rate limit? or just don't understand simultaneous connections?
-          max_s2s_connections: 1
-          max_user_sessions: 10
-          max_user_offline_messages: 5000
-          c2s_shaper:
-            fast: all
-          s2s_shaper:
-            med: all
+        max_s2s_connections = 1;
+        max_user_sessions = 10;
+        max_user_offline_messages = 5000;
+        c2s_shaper.fast = "all";
+        s2s_shaper.med = "all";
+      };
 
       # docs: <https://docs.ejabberd.im/admin/configuration/basic/#shapers>
       # this limits the bytes/sec.
       # for example, burst: 3_000_000 and rate: 100_000 means:
       # - each client has a BW budget that accumulates 100kB/sec and is capped at 3 MB
-        shaper:
-          fast: 1000000
-          med:   500000
-        #   fast:
-        #     - rate:        1000000
-        #     - burst_size: 10000000
-        #   med:
-        #     - rate:         500000
-        #     - burst_size:  5000000
+      shaper.fast = 1000000;
+      shaper.med  =  500000;
+      # shaper.fast.rate       =  1000000;
+      # shaper.fast.burst_size = 10000000;
+      # shaper.med.rate        =   500000;
+      # shaper.med.burst_size  =  5000000;
 
       # see: <https://docs.ejabberd.im/admin/configuration/listen/>
-        # s2s_use_starttls: true
-        s2s_use_starttls: optional
+      # s2s_use_starttls = true;
+      s2s_use_starttls = "optional";
       # lessens 504: remote-server-timeout errors
       # see: <https://github.com/processone/ejabberd/issues/3105#issuecomment-562182967>
-        negotiation_timeout: 60
+      negotiation_timeout = 60;
 
-        listen:
-          -
-            port: 5222
-            module: ejabberd_c2s
-            shaper: c2s_shaper
-            starttls: true
-            access: c2s_access
-          -
-            port: 5223
-            module: ejabberd_c2s
-            shaper: c2s_shaper
-            tls: true
-            access: c2s_access
-          -
-            port: 5269
-            module: ejabberd_s2s_in
-            shaper: s2s_shaper
-          -
-            port: 5270
-            module: ejabberd_s2s_in
-            shaper: s2s_shaper
-            tls: true
-          -
-            port: 5443
-            module: ejabberd_http
-            tls: true
-            request_handlers:
-              /admin: ejabberd_web_admin  # TODO: ensure this actually works
-              /api: mod_http_api  # ejabberd API endpoint (to control server)
-              /bosh: mod_bosh
-              /upload: mod_http_upload
-              /ws: ejabberd_http_ws
-              # /.well-known/host-meta: mod_host_meta
-              # /.well-known/host-meta.json: mod_host_meta
-          -
+      listen = [
+        {
+          port = 5222;
+          module = "ejabberd_c2s";
+          shaper = "c2s_shaper";
+          starttls = true;
+          access = "c2s_access";
+        }
+        {
+          port = 5223;
+          module = "ejabberd_c2s";
+          shaper = "c2s_shaper";
+          tls = true;
+          access = "c2s_access";
+        }
+        {
+          port = 5269;
+          module = "ejabberd_s2s_in";
+          shaper = "s2s_shaper";
+        }
+        {
+          port = 5270;
+          module = "ejabberd_s2s_in";
+          shaper = "s2s_shaper";
+          tls = true;
+        }
+        {
+          port = 5443;
+          module = "ejabberd_http";
+          tls = true;
+          request_handlers = {
+            "/admin" = "ejabberd_web_admin";  # TODO: ensure this actually works
+            "/api" = "mod_http_api";  # ejabberd API endpoint (to control server)
+            "/bosh" = "mod_bosh";
+            "/upload" = "mod_http_upload";
+            "/ws" = "ejabberd_http_ws";
+            # "/.well-known/host-meta" = "mod_host_meta";
+            # "/.well-known/host-meta.json" = "mod_host_meta";
+          };
+        }
+        {
           # STUN+TURN TCP
           # note that the full port range should be forwarded ("not NAT'd")
           # `use_turn=true` enables both TURN *and* STUN
-            port: 3478
-            module: ejabberd_stun
-            transport: tcp
-            use_turn: true
-            turn_min_port: ${builtins.toString turnPortLow}
-            turn_max_port: ${builtins.toString turnPortHigh}
-            turn_ipv4_address: %ANATIVE%
-          -
+          port = 3478;
+          module = "ejabberd_stun";
+          transport = "tcp";
+          use_turn = true;
+          turn_min_port = turnPortLow;
+          turn_max_port = turnPortHigh;
+          turn_ipv4_address = "%ANATIVE%";
+        }
+        {
           # STUN+TURN UDP
-            port: 3478
-            module: ejabberd_stun
-            transport: udp
-            use_turn: true
-            turn_min_port: ${builtins.toString turnPortLow}
-            turn_max_port: ${builtins.toString turnPortHigh}
-            turn_ipv4_address: %ANATIVE%
-          -
+          port = 3478;
+          module = "ejabberd_stun";
+          transport = "udp";
+          use_turn = true;
+          turn_min_port = turnPortLow;
+          turn_max_port = turnPortHigh;
+          turn_ipv4_address = "%ANATIVE%";
+        }
+        {
           # STUN+TURN TLS over TCP
-            port: 5349
-            module: ejabberd_stun
-            transport: tcp
-            tls: true
-            certfile: /var/lib/acme/uninsane.org/full.pem
-            use_turn: true
-            turn_min_port: ${builtins.toString turnPortLow}
-            turn_max_port: ${builtins.toString turnPortHigh}
-            turn_ipv4_address: %ANATIVE%
+          port = 5349;
+          module = "ejabberd_stun";
+          transport = "tcp";
+          tls = true;
+          certfile = "/var/lib/acme/uninsane.org/full.pem";
+          use_turn = true;
+          turn_min_port = turnPortLow;
+          turn_max_port = turnPortHigh;
+          turn_ipv4_address = "%ANATIVE%";
+        }
+      ];
 
       # TODO: enable mod_fail2ban
       # TODO(low): look into mod_http_fileserver for serving macros?
-        modules:
-          # mod_adhoc: {}
-          # mod_announce:
-          #   access: admin
+      modules = {
+        # mod_adhoc = {};
+        # mod_announce = {
+        #   access = "admin";
+        # };
         # allows users to set avatars in vCard
         # - <https://docs.ejabberd.im/admin/configuration/modules/#mod-avatar>
-          mod_avatar: {}
-          mod_caps: {}  # for mod_pubsub
-          mod_carboncopy: {}  # allows multiple clients to receive a user's message
+        mod_avatar = {};
+        mod_caps = {};  # for mod_pubsub
+        mod_carboncopy = {};  # allows multiple clients to receive a user's message
         # queues messages when recipient is offline, including PEP and presence messages.
         # compliance test suggests this be enabled
-          mod_client_state: {}
+        mod_client_state = {};
+
         # mod_conversejs: TODO: enable once on 21.12
         # allows clients like Dino to discover where to upload files
-          mod_disco:
-            server_info:
-              -
-                modules: all
-                name: abuse-addresses
-                urls:
-                  - "mailto:admin.xmpp@uninsane.org"
-                  - "xmpp:colin@uninsane.org"
-              -
-                modules: all
-                name: admin-addresses
-                urls:
-                  - "mailto:admin.xmpp@uninsane.org"
-                  - "xmpp:colin@uninsane.org"
-          mod_http_upload:
-            host: upload.xmpp.uninsane.org
-            hosts:
-              - upload.xmpp.uninsane.org
-            put_url: "https://@HOST@:5443/upload"
-            dir_mode: "0750"
-            file_mode: "0750"
-            rm_on_unregister: false
+        mod_disco.server_info = [
+          {
+            modules = "all";
+            name = "abuse-addresses";
+            urls = [
+              "mailto:admin.xmpp@uninsane.org"
+              "xmpp:colin@uninsane.org"
+            ];
+          }
+          {
+            modules = "all";
+            name = "admin-addresses";
+            urls = [
+              "mailto:admin.xmpp@uninsane.org"
+              "xmpp:colin@uninsane.org"
+            ];
+          }
+        ];
+        mod_http_upload = {
+          host = "upload.xmpp.uninsane.org";
+          hosts = [ "upload.xmpp.uninsane.org" ];
+          put_url = "https://@HOST@:5443/upload";
+          dir_mode = "0750";
+          file_mode = "0750";
+          rm_on_unregister = false;
+        };
         # allow discoverability of BOSH and websocket endpoints
         # TODO: enable once on ejabberd 22.05  (presently 21.04)
-          # mod_host_meta: {}
-          mod_jidprep: {}  # probably not needed: lets clients normalize jids
-          mod_last: {}  # allow other users to know when i was last online
-          mod_mam:
+        # mod_host_meta = {};
+        mod_jidprep = {};  # probably not needed: lets clients normalize jids
+        mod_last = {};  # allow other users to know when i was last online
+        mod_mam = {
           # Mnesia is limited to 2GB, better to use an SQL backend
           # For small servers SQLite is a good fit and is very easy
           # to configure. Uncomment this when you have SQL configured:
           # db_type: sql
-            assume_mam_usage: true
-            default: always
-          mod_muc:
-            access:
-              - allow
-            access_admin:
-              - allow: admin
-            access_create: muc_create
-            access_persistent: muc_create
-            access_mam:
-              - allow
-            history_size: 100  # messages to show new participants
-            host: muc.xmpp.uninsane.org
-            hosts:
-              - muc.xmpp.uninsane.org
-            default_room_options:
-              anonymous: false
-              lang: en
-              persistent: true
-              mam: true
-          mod_muc_admin: {}
-          mod_offline:  # store messages for a user when they're offline (TODO: understand multi-client workflow?)
-            access_max_user_messages: max_user_offline_messages
-            store_groupchat: true
-          mod_ping: {}
-          mod_privacy: {}  # deprecated, but required for `ejabberctl export_piefxis`
-          mod_private: {}  # allow local clients to persist arbitrary data on my server
+          assume_mam_usage = true;
+          default = "always";
+        };
+        mod_muc = {
+          access = [ "allow" ];
+          access_admin = { allow = "admin"; };
+          access_create = "muc_create";
+          access_persistent = "muc_create";
+          access_mam = [ "allow" ];
+          history_size = 100;  # messages to show new participants
+          host = "muc.xmpp.uninsane.org";
+          hosts = [ "muc.xmpp.uninsane.org" ];
+          default_room_options = {
+            anonymous = false;
+            lang = "en";
+            persistent = true;
+            mam = true;
+          };
+        };
+        mod_muc_admin = {};
+        mod_offline = {
+          # store messages for a user when they're offline (TODO: understand multi-client workflow?)
+          access_max_user_messages = "max_user_offline_messages";
+          store_groupchat = true;
+        };
+        mod_ping = {};
+        mod_privacy = {};  # deprecated, but required for `ejabberctl export_piefxis`
+        mod_private = {};  # allow local clients to persist arbitrary data on my server
         # push notifications to services integrated with e.g. Apple/Android.
         # default is for a maximum amount of PII to be withheld, since these push notifs
         # generally traverse 3rd party services. can opt to include message body, etc, though.
-          mod_push: {}
+        mod_push = {};
         # i don't fully understand what this does, but it seems aimed at making push notifs more reliable.
-          mod_push_keepalive: {}
-          mod_roster:
-            versioning: true
+        mod_push_keepalive = {};
+        mod_roster = {
+          versioning = true;
+        };
         # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-s2s-dialback>
         # s2s dialback to verify inbound messages
         # unclear to what degree the XMPP network requires this
-          mod_s2s_dialback: {}
-          mod_shared_roster: {}  # creates groups for @all, @online, and anything manually administered?
-          mod_stream_mgmt:
-            resend_on_timeout: if_offline  # resend undelivered messages if the origin client is offline
+        mod_s2s_dialback = {};
+        mod_shared_roster = {};  # creates groups for @all, @online, and anything manually administered?
+        mod_stream_mgmt = {
+          # resend undelivered messages if the origin client is offline
+          resend_on_timeout = "if_offline";
+        };
         # fallback for when DNS-based STUN discovery is unsupported.
         # - see: <https://xmpp.org/extensions/xep-0215.html>
         # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-stun-disco>
         # people say to just keep this defaulted (i guess ejabberd knows to return its `host` option of uninsane.org?)
-          mod_stun_disco: {}
+        mod_stun_disco = {};
         # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-vcard>
-          mod_vcard:
-            allow_return_all: true  # all users are discoverable (?)
-            host: vjid.xmpp.uninsane.org
-            hosts:
-              - vjid.xmpp.uninsane.org
-            search: true
-          mod_vcard_xupdate: {}  # needed for avatars
-          # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-pubsub>
-          mod_pubsub:  # needed for avatars
-            access_createnode: pubsub_createnode_access
-            host: pubsub.xmpp.uninsane.org
-            hosts:
-              - pubsub.xmpp.uninsane.org
-            ignore_pep_from_offline: false
-            last_item_cache: true
-            plugins:
-              - pep
-              - flat
-            force_node_config:
-              # ensure client bookmarks are private
-              storage:bookmarks:
-                access_model: whitelist
-              urn:xmpp:avatar:data:
-                access_model: open
-              urn:xmpp:avatar:metadata:
-                access_model: open
-          mod_version: {}
-      '';
+        mod_vcard = {
+          allow_return_all = true;  # all users are discoverable (?)
+          host = "vjid.xmpp.uninsane.org";
+          hosts = [ "vjid.xmpp.uninsane.org" ];
+          search = true;
         };
+        mod_vcard_xupdate = {};  # needed for avatars
+        # docs: <https://docs.ejabberd.im/admin/configuration/modules/#mod-pubsub>
+        mod_pubsub = {
+          #^ needed for avatars
+          access_createnode = "pubsub_createnode_access";
+          host = "pubsub.xmpp.uninsane.org";
+          hosts = [ "pubsub.xmpp.uninsane.org" ];
+          ignore_pep_from_offline = false;
+          last_item_cache = true;
+          plugins = [
+            "pep"
+            "flat"
+          ];
+          force_node_config = {
+            # ensure client bookmarks are private
+            "storage:bookmarks:" = {
+              "access_model" = "whitelist";
+            };
+            "urn:xmpp:avatar:data" = {
+              "access_model" = "open";
+            };
+            "urn:xmpp:avatar:metadata" = {
+              "access_model" = "open";
+            };
+          };
+        };
+        mod_version = {};
+      };
+    });
     sed = "${pkgs.gnused}/bin/sed";
   in ''
     ip=$(cat '${config.sane.services.dyn-dns.ipPath}')
     # config is 444 (not 644), so we want to write out-of-place and then atomically move
     # TODO: factor this out into `sane-woop` helper?
     rm -f /var/lib/ejabberd/ejabberd.yaml.new
-    ${sed} "s/%ANATIVE%/$ip/" ${config-in} > /var/lib/ejabberd/ejabberd.yaml.new
+    ${sed} "s/%ANATIVE%/$ip/g" ${config-in} > /var/lib/ejabberd/ejabberd.yaml.new
     mv /var/lib/ejabberd/ejabberd.yaml{.new,}
   '';
 

hosts/by-name/servo/services/email/default.nix

@@ -22,6 +22,13 @@
 #     - but postfix delegates authorization of that outgoing mail to dovecot, on the server side
 #
 # - local clients (i.e. sendmail) interact only with postfix
+#
+# debugging: general connectivity issues
+# - test that inbound port 25 is unblocked:
+#   - `curl https://canyouseeme.org/ --data 'port=25&IP=185.157.162.178' | grep 'see your service'`
+#     - and retry with port 465, 587
+#     - i think this API requires the queried IP match the source IP
+#   - if necessary, `systemctl stop postfix` and `sudo nc -l 185.157.162.178 25`, then try https://canyouseeme.org
 
 { ... }:
 {

hosts/by-name/servo/services/email/postfix.nix

@@ -18,7 +18,7 @@ let
   };
 in
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
     { user = "opendkim"; group = "opendkim"; path = "/var/lib/opendkim"; }
     { user = "root"; group = "root"; path = "/var/lib/postfix"; }
@@ -28,21 +28,25 @@ in
     # "/var/lib/dovecot"
   ];
 
-  sane.ports.ports."25" = {
-    protocol = [ "tcp" ];
-    visibleTo.ovpn = true;
-    description = "colin-smtp-mx.uninsane.org";
-  };
-  sane.ports.ports."465" = {
-    protocol = [ "tcp" ];
-    visibleTo.ovpn = true;
-    description = "colin-smtps-mx.uninsane.org";
-  };
-  sane.ports.ports."587" = {
-    protocol = [ "tcp" ];
-    visibleTo.ovpn = true;
-    description = "colin-smtps-submission-mx.uninsane.org";
-  };
+  # XXX(2023/10/20): opening these ports in the firewall has the OPPOSITE effect as intended.
+  # these ports are only routable so long as they AREN'T opened.
+  # probably some cursed interaction with network namespaces introduced after 2023/10/10.
+  # sane.ports.ports."25" = {
+  #   protocol = [ "tcp" ];
+  #   # XXX visibleTo.lan effectively means "open firewall, but don't configure any NAT/forwarding"
+  #   visibleTo.lan = true;
+  #   description = "colin-smtp-mx.uninsane.org";
+  # };
+  # sane.ports.ports."465" = {
+  #   protocol = [ "tcp" ];
+  #   visibleTo.lan = true;
+  #   description = "colin-smtps-mx.uninsane.org";
+  # };
+  # sane.ports.ports."587" = {
+  #   protocol = [ "tcp" ];
+  #   visibleTo.lan = true;
+  #   description = "colin-smtps-submission-mx.uninsane.org";
+  # };
 
   # exists only to manage certs for Postfix
   services.nginx.virtualHosts."mx.uninsane.org" = {

hosts/by-name/servo/services/export/default.nix

@@ -29,7 +29,7 @@
   # - `sudo btrfs qgroup limit 20G /mnt/persist/ext/persist/var/export/playground`
   # to query the quota/status:
   # - `sudo btrfs qgroup show -re /var/export/playground`
-  sane.persist.sys.ext = [
+  sane.persist.sys.byStore.ext = [
     { user = "root"; group = "export"; mode = "0775"; path = "/var/export/playground"; }
   ];
 

hosts/by-name/servo/services/export/sftpgo.nix

@@ -175,5 +175,10 @@ in
   systemd.services.sftpgo.serviceConfig = {
     ReadOnlyPaths = [ "/var/export" ];
     ReadWritePaths = [ "/var/export/playground" ];
+    after = [ "network-online.target" ];
+    wants = [ "network-online.target" ];
+
+    Restart = "always";
+    RestartSec = "20s";
   };
 }

hosts/by-name/servo/services/freshrss.nix

@@ -15,7 +15,7 @@
     owner = config.users.users.freshrss.name;
     mode = "0400";
   };
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "freshrss"; group = "freshrss"; path = "/var/lib/freshrss"; }
   ];
 

hosts/by-name/servo/services/gitea.nix

@@ -2,7 +2,7 @@
 { config, pkgs, lib, ... }:
 
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
     { user = "git"; group = "gitea"; path = "/var/lib/gitea"; }
   ];
@@ -13,6 +13,10 @@
   services.gitea.appName = "Perfectly Sane Git";
   # services.gitea.disableRegistration = true;
 
+  services.gitea.database.createDatabase = false;  #< silence warning which wants db user and name to be equal
+  # TODO: remove this after merge: <https://github.com/NixOS/nixpkgs/pull/268849>
+  services.gitea.database.socket = "/run/postgresql";  #< would have been set if createDatabase = true
+
   # gitea doesn't create the git user
   users.users.git = {
     description = "Gitea Service";

hosts/by-name/servo/services/ipfs.nix

@@ -10,7 +10,7 @@
 
 lib.mkIf false # i don't actively use ipfs anymore
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? could be more granular
     { user = "261"; group = "261"; path = "/var/lib/ipfs"; }
   ];

hosts/by-name/servo/services/jackett.nix

@@ -1,7 +1,7 @@
 { ... }:
 
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? we only need this to save Indexer creds ==> migrate to config?
     { user = "root"; group = "root"; path = "/var/lib/jackett"; }
   ];

hosts/by-name/servo/services/jellyfin.nix

@@ -40,7 +40,7 @@
     description = "colin-jellyfin-https-lan";
   };
 
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "jellyfin"; group = "jellyfin"; mode = "0700"; path = "/var/lib/jellyfin"; }
   ];
   sane.fs."/var/lib/jellyfin/config/logging.json" = {

hosts/by-name/servo/services/komga.nix

@@ -4,7 +4,7 @@ let
   inherit (svc-cfg) user group port stateDir;
 in
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { inherit user group; mode = "0700"; path = stateDir; }
   ];
 

hosts/by-name/servo/services/matrix/default.nix

@@ -1,6 +1,16 @@
 # docs: <https://nixos.wiki/wiki/Matrix>
 # docs: <https://nixos.org/manual/nixos/stable/index.html#module-services-matrix-synapse>
 # example config: <https://github.com/matrix-org/synapse/blob/develop/docs/sample_config.yaml>
+#
+# ENABLING PUSH NOTIFICATIONS (with UnifiedPush/ntfy):
+# - Matrix "pushers" API spec: <https://spec.matrix.org/latest/client-server-api/#post_matrixclientv3pushersset>
+# - first, view notification settings:
+#   - obtain your client's auth token. e.g. Element -> profile -> help/about -> access token.
+#   - `curl --header 'Authorization: Bearer <your_access_token>' localhost:8008/_matrix/client/v3/pushers | jq .`
+# - enable a new notification destination:
+#   - `curl --header "Authorization: Bearer <your_access_token>" --data '{ "app_display_name": "<topic>", "app_id": "ntfy.uninsane.org", "data": { "url": "https://ntfy.uninsane.org/_matrix/push/v1/notify", "format": "event_id_only" }, "device_display_name": "<topic>", "kind": "http", "lang": "en-US", "profile_tag": "", "pushkey": "<topic>" }' localhost:8008/_matrix/client/v3/pushers/set`
+# - delete a notification destination by setting `kind` to `null` (otherwise, request is identical to above)
+#
 { config, lib, pkgs, ... }:
 
 {
@@ -10,23 +20,24 @@
     ./signal.nix
   ];
 
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/matrix-synapse"; }
   ];
   services.matrix-synapse.enable = true;
+  services.matrix-synapse.settings = {
     # this changes the default log level from INFO to WARN.
     # maybe there's an easier way?
-  services.matrix-synapse.settings.log_config = ./synapse-log_level.yaml;
-  services.matrix-synapse.settings.server_name = "uninsane.org";
+    log_config = ./synapse-log_level.yaml;
+    server_name = "uninsane.org";
 
     # services.matrix-synapse.enable_registration_captcha = true;
     # services.matrix-synapse.enable_registration_without_verification = true;
-  services.matrix-synapse.settings.enable_registration = true;
+    enable_registration = true;
     # services.matrix-synapse.registration_shared_secret = "<shared key goes here>";
 
     # default for listeners is port = 8448, tls = true, x_forwarded = false.
     # we change this because the server is situated behind nginx.
-  services.matrix-synapse.settings.listeners = [
+    listeners = [
       {
         port = 8008;
         bind_addresses = [ "127.0.0.1" ];
@@ -42,35 +53,40 @@
       }
     ];
 
-  services.matrix-synapse.settings.x_forwarded = true;  # because we proxy matrix behind nginx
-  services.matrix-synapse.settings.max_upload_size = "100M";  # default is "50M"
+    ip_range_whitelist = [
+      # to communicate with ntfy.uninsane.org push notifs.
+      # TODO: move this to some non-shared loopback device: we don't want Matrix spouting http requests to *anything* on this machine
+      "10.78.79.51"
+    ];
 
-  services.matrix-synapse.settings.admin_contact = "admin.matrix@uninsane.org";
-  services.matrix-synapse.settings.registrations_require_3pid = [ "email" ];
+    x_forwarded = true;  # because we proxy matrix behind nginx
+    max_upload_size = "100M";  # default is "50M"
+
+    admin_contact = "admin.matrix@uninsane.org";
+    registrations_require_3pid = [ "email" ];
+  };
 
   services.matrix-synapse.extraConfigFiles = [
     config.sops.secrets."matrix_synapse_secrets.yaml".path
   ];
 
-  # services.matrix-synapse.extraConfigFiles = [builtins.toFile "matrix-synapse-extra-config" ''
-  #   admin_contact: "admin.matrix@uninsane.org"
-  #   registrations_require_3pid:
-  #     - email
-  #   email:
-  #     smtp_host: "mx.uninsane.org"
-  #     smtp_port: 587
-  #     smtp_user: "matrix-synapse"
-  #     smtp_pass: "${secrets.matrix-synapse.smtp_pass}"
-  #     require_transport_security: true
-  #     enable_tls: true
-  #     notif_from: "%(app)s <notify.matrix@uninsane.org>"
-  #     app_name: "Uninsane Matrix"
-  #     enable_notifs: true
-  #     validation_token_lifetime: 96h
-  #     invite_client_location: "https://web.matrix.uninsane.org"
-  #     subjects:
-  #       email_validation: "[%(server_name)s] Validate your email"
-  # ''];
+  systemd.services.matrix-synapse.postStart = ''
+    ACCESS_TOKEN=$(${pkgs.coreutils}/bin/cat ${config.sops.secrets.matrix_access_token.path})
+    TOPIC=$(${pkgs.coreutils}/bin/cat ${config.sops.secrets.ntfy-sh-topic.path})
+
+    echo "ensuring ntfy push gateway"
+    ${pkgs.curl}/bin/curl \
+      --header "Authorization: Bearer $ACCESS_TOKEN" \
+      --data "{ \"app_display_name\": \"ntfy-adapter\", \"app_id\": \"ntfy.uninsane.org\", \"data\": { \"url\": \"https://ntfy.uninsane.org/_matrix/push/v1/notify\", \"format\": \"event_id_only\" }, \"device_display_name\": \"ntfy-adapter\", \"kind\": \"http\", \"lang\": \"en-US\", \"profile_tag\": \"\", \"pushkey\": \"$TOPIC\" }" \
+      localhost:8008/_matrix/client/v3/pushers/set
+
+    echo "registered push gateways:"
+    ${pkgs.curl}/bin/curl \
+      --header "Authorization: Bearer $ACCESS_TOKEN" \
+      localhost:8008/_matrix/client/v3/pushers \
+      | ${pkgs.jq}/bin/jq .
+  '';
+
 
   # new users may be registered on the CLI:
   #   register_new_matrix_user -c /nix/store/8n6kcka37jhmi4qpd2r03aj71pkyh21s-homeserver.yaml http://localhost:8008
@@ -141,4 +157,9 @@
   sops.secrets."matrix_synapse_secrets.yaml" = {
     owner = config.users.users.matrix-synapse.name;
   };
+  sops.secrets."matrix_access_token" = {
+    owner = config.users.users.matrix-synapse.name;
+  };
+  # provide access to ntfy-sh-topic secret
+  users.users.matrix-synapse.extraGroups = [ "ntfy-sh" ];
 }

hosts/by-name/servo/services/matrix/discord-puppet.nix

@@ -5,7 +5,7 @@
 # - recommended to use mautrix-discord: <https://github.com/NixOS/nixpkgs/pull/200462>
 lib.mkIf false
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "matrix-synapse"; group = "matrix-synapse"; path = "/var/lib/mx-puppet-discord"; }
   ];
 

hosts/by-name/servo/services/matrix/irc.nix

@@ -101,7 +101,7 @@ in
     })
   ];
 
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode?
     { user = "matrix-appservice-irc"; group = "matrix-appservice-irc"; path = "/var/lib/matrix-appservice-irc"; }
   ];
@@ -141,6 +141,7 @@ in
           sasl = false;
           # notable channels:
           # - #hare
+          # - #mnt-reform
         };
         "irc.myanonamouse.net" = ircServer {
           name = "MyAnonamouse";

hosts/by-name/servo/services/matrix/signal.nix

@@ -2,7 +2,7 @@
 # - <https://github.com/mautrix/signal/blob/master/mautrix_signal/example-config.yaml>
 { config, pkgs, ... }:
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "mautrix-signal"; group = "mautrix-signal"; path = "/var/lib/mautrix-signal"; }
     { user = "signald"; group = "signald"; path = "/var/lib/signald"; }
   ];

hosts/by-name/servo/services/monero.nix

@@ -0,0 +1,37 @@
+# as of 2023/11/26: complete downloaded blockchain should be 200GiB on disk, give or take.
+{ ... }:
+{
+  sane.persist.sys.byStore.ext = [
+    # /var/lib/monero/lmdb is what consumes most of the space
+    { user = "monero"; group = "monero"; path = "/var/lib/monero"; }
+  ];
+
+  services.monero.enable = true;
+  services.monero.limits.upload = 5000;  # in kB/s
+  services.monero.extraConfig = ''
+    # see: monero doc/ANONYMITY_NETWORKS.md
+    #
+    # "If any anonymity network is enabled, transactions being broadcast that lack a valid 'context'
+    # (i.e. the transaction did not come from a P2P connection) will only be sent to peers on anonymity networks."
+    #
+    # i think this means that setting tx-proxy here ensures any transactions sent locally to my node (via RPC)
+    # will be sent over an anonymity network.
+    tx-proxy=i2p,127.0.0.1:9000
+    tx-proxy=tor,127.0.0.1:9050
+  '';
+
+  services.i2p.enable = true;
+  # tor: `tor.enable` doesn't start a relay, exit node, proxy, etc. it's minimal.
+  # tor.client.enable configures a torsocks proxy, accessible *only* to localhost.
+  services.tor.enable = true;
+  services.tor.client.enable = true;
+
+  # monero ports: <https://monero.stackexchange.com/questions/604/what-ports-does-monero-use-rpc-p2p-etc>
+  # - 18080 = "P2P" monero node <-> monero node connections
+  # - 18081 = "RPC" monero client -> monero node connections
+  sane.ports.ports."18080" = {
+    protocol = [ "tcp" ];
+    visibleTo.wan = true;
+    description = "colin-monero-p2p";
+  };
+}

hosts/by-name/servo/services/navidrome.nix

@@ -1,7 +1,7 @@
 { lib, ... }:
 
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "navidrome"; group = "navidrome"; path = "/var/lib/navidrome"; }
   ];
   services.navidrome.enable = true;

hosts/by-name/servo/services/nginx.nix

@@ -1,4 +1,5 @@
-# docs: https://nixos.wiki/wiki/Nginx
+# docs: <https://nixos.wiki/wiki/Nginx>
+# docs: <https://nginx.org/en/docs/>
 { config, lib, pkgs, ... }:
 
 let
@@ -133,7 +134,7 @@ in
   security.acme.acceptTerms = true;
   security.acme.defaults.email = "admin.acme@uninsane.org";
 
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode?
     { user = "acme"; group = "acme"; path = "/var/lib/acme"; }
     { user = "colin"; group = "users"; path = "/var/www/sites"; }

hosts/by-name/servo/services/ntfy/default.nix

@@ -0,0 +1,14 @@
+# ntfy: UnifiedPush notification delivery system
+# - used to get push notifications out of Matrix and onto a Phone (iOS, Android, or a custom client)
+{ config, ... }:
+{
+  imports = [
+    ./ntfy-waiter.nix
+    ./ntfy-sh.nix
+  ];
+  sops.secrets."ntfy-sh-topic" = {
+    mode = "0440";
+    owner = config.users.users.ntfy-sh.name;
+    group = config.users.users.ntfy-sh.name;
+  };
+}

hosts/by-name/servo/services/ntfy/ntfy-sh.nix

@@ -0,0 +1,92 @@
+# ntfy: UnifiedPush notification delivery system
+# - used to get push notifications out of Matrix and onto a Phone (iOS, Android, or a custom client)
+#
+# config options:
+# - <https://docs.ntfy.sh/config/#config-options>
+#
+# usage:
+# - ntfy sub https://ntfy.uninsane.org/TOPIC
+# - ntfy pub https://ntfy.uninsane.org/TOPIC "my message"
+# in production, TOPIC is a shared secret between the publisher (Matrix homeserver) and the subscriber (phone)
+#
+# administering:
+# - sudo -u ntfy-sh ntfy access
+#
+# debugging:
+# - make sure that the keepalives are good:
+#   - on the subscriber machine, run `lsof -i4` to find the port being used
+#   - `sudo tcpdump tcp port <p>`
+#     - shouldn't be too spammy
+#
+# matrix integration:
+# - the user must manually point synapse to the ntfy endpoint:
+#   - `curl --header "Authorization: <your_token>" --data '{ "app_display_name": "sane-nix moby", "app_id": "ntfy.uninsane.org", "data": { "url": "https://ntfy.uninsane.org/_matrix/push/v1/notify", "format": "event_id_only" }, "device_display_name": "sane-nix moby", "kind": "http", "lang": "en-US", "profile_tag": "", "pushkey": "https://ntfy.uninsane.org/TOPIC" }' localhost:8008/_matrix/client/v3/pushers/set`
+#     where the token is grabbed from Element's help&about page when logged in
+#   - to remove, send this `curl` with `"kind": null`
+{ config, lib, pkgs, ... }:
+let
+  # subscribers need a non-443 public port to listen on as a way to easily differentiate this traffic
+  # at the IP layer, to enable e.g. wake-on-lan.
+  altPort = 2587;
+in
+{
+  sane.persist.sys.byStore.plaintext = [
+    # not 100% necessary to persist this, but ntfy does keep a 12hr (by default) cache
+    # for pushing notifications to users who become offline.
+    # ACLs also live here.
+    { user = "ntfy-sh"; group ="ntfy-sh"; path = "/var/lib/ntfy-sh"; }
+  ];
+
+  services.ntfy-sh.enable = true;
+  services.ntfy-sh.settings = {
+    base-url = "https://ntfy.uninsane.org";
+    behind-proxy = true;  # not sure if needed
+    # keepalive interval is a ntfy-specific keepalive thing, where it sends actual data down the wire.
+    # it's not simple TCP keepalive.
+    # defaults to 45s.
+    # note that the client may still do its own TCP-level keepalives, typically every 30s
+    keepalive-interval = "15m";
+    log-level = "trace";  # trace, debug, info (default), warn, error
+    auth-default-access = "deny-all";
+  };
+  systemd.services.ntfy-sh.serviceConfig.DynamicUser = lib.mkForce false;
+  systemd.services.ntfy-sh.preStart = ''
+    # make this specific topic read-write by world
+    # it would be better to use the token system, but that's extra complexity for e.g.
+    # how do i plumb a secret into the Matrix notification pusher
+    #
+    # note that this will fail upon first run, i.e. before ntfy has created its db.
+    # just restart the service.
+    topic=$(cat ${config.sops.secrets.ntfy-sh-topic.path})
+    ${pkgs.ntfy-sh}/bin/ntfy access everyone "$topic" read-write
+  '';
+
+
+  services.nginx.virtualHosts."ntfy.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+    listen = [
+      { addr = "0.0.0.0"; port = altPort; ssl = true; }
+      { addr = "0.0.0.0"; port = 443;  ssl = true; }
+      { addr = "0.0.0.0"; port = 80;   ssl = false; }
+    ];
+    locations."/" = {
+      proxyPass = "http://127.0.0.1:2586";
+      proxyWebsockets = true;  #< support websocket upgrades. without that, `ntfy sub` hangs silently
+      recommendedProxySettings = true; #< adds headers so ntfy logs include the real IP
+      extraConfig = ''
+        # absurdly long timeout (86400s=24h) so that we never hang up on clients.
+        # make sure the client is smart enough to detect a broken proxy though!
+        proxy_read_timeout 86400s;
+      '';
+    };
+  };
+  sane.dns.zones."uninsane.org".inet.CNAME."ntfy" = "native";
+
+  sane.ports.ports."${builtins.toString altPort}" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-ntfy.uninsane.org";
+  };
+}

hosts/by-name/servo/services/ntfy/ntfy-waiter

@@ -0,0 +1,151 @@
+#!/usr/bin/env nix-shell
+#!nix-shell -i python3 -p "python3.withPackages (ps: [  ])" -p ntfy-sh
+
+import argparse
+import logging
+import os
+import socket
+import subprocess
+import sys
+import threading
+import time
+
+logger = logging.getLogger(__name__)
+
+LISTEN_QUEUE = 3
+WAKE_MESSAGE = b'notification\n'
+
+class Client:
+    def __init__(self, sock, addr_info, live_after: float):
+        self.live_after = live_after
+        self.sock = sock
+        self.addr_info = addr_info
+
+    def __cmp__(self, other: 'Client'):
+        return cmp(self.addr_info, other.addr_info)
+
+    def try_notify(self, message: bytes) -> bool:
+        """
+        returns true if we send a packet to notify client.
+        fals otherwise (e.g. the socket is dead).
+        """
+        ttl = self.live_after - time.time()
+        if ttl > 0:
+            logger.debug(f"sleeping {ttl:.2f}s until client {self.addr_info} is ready to receive notification")
+            time.sleep(ttl)
+
+        try:
+            self.sock.sendall(message)
+        except Exception as e:
+            logger.warning(f"failed to notify client {self.addr_info} {e}")
+            return False
+        else:
+            logger.info(f"successfully notified {self.addr_info}: {message}")
+            return True
+
+class Adapter:
+    def __init__(self, host: str, port: int, silence: int, topic: str):
+        self.host = host
+        self.port = port
+        self.silence = silence
+        self.topic = topic
+        self.clients = set()
+
+    def log_clients(self):
+        clients_str = '\n'.join(f'  {c.addr_info}' for c in self.clients)
+        logger.debug(f"clients alive ({len(self.clients)}):\n{clients_str}")
+
+    def add_client(self, client: Client):
+        # it's a little bit risky to keep more than one client at the same IP address,
+        # because it's possible a notification comes in and we ring the old connection,
+        # even when the new connection says "don't ring yet".
+        for c in set(self.clients):
+            if c.addr_info[0] == client.addr_info[0]:
+                logger.info(f"purging old client before adding new one at same address: {c.addr_info} -> {client.addr_info}")
+                self.clients.remove(c)
+
+        logger.info(f"accepted client at {client.addr_info}")
+        self.clients.add(client)
+
+    def listener_loop(self):
+        logger.info(f"listening for connections on {self.host}:{self.port}")
+        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        s.bind((self.host, self.port))
+        s.listen(LISTEN_QUEUE)
+        while True:
+            conn, addr_info = s.accept()
+            self.add_client(Client(conn, addr_info, live_after = time.time() + self.silence))
+
+    def notify_clients(self, message: bytes = WAKE_MESSAGE):
+        # notify every client, and drop any which have disconnected.
+        # note that we notify based on age (oldest -> youngest)
+        # because notifying young clients might entail sleeping until they're ready.
+        clients = sorted(self.clients, key=lambda c: (c.live_after, c.addr_info))
+
+        dead_clients = [
+            c for c in clients if not c.try_notify(message)
+        ]
+        for c in dead_clients:
+            self.clients.remove(c)
+
+        self.log_clients()
+
+    def notify_loop(self):
+        logger.info("waiting for notification events")
+        ntfy_proc = subprocess.Popen(
+            [
+                "ntfy",
+                "sub",
+                f"https://ntfy.uninsane.org/{self.topic}"
+            ],
+            stdout=subprocess.PIPE
+        )
+        for line in iter(ntfy_proc.stdout.readline, b''):
+            logger.debug(f"received notification: {line}")
+            self.notify_clients()
+
+def get_topic() -> str:
+    return open('/run/secrets/ntfy-sh-topic', 'rt').read().strip()
+
+def run_forever(callable):
+    try:
+        callable()
+    except Exception as e:
+        logger.error(f"{callable} failed: {e}")
+    else:
+        logger.error(f"{callable} unexpectedly returned")
+    # sys.exit(1)
+    os._exit(1)  # sometimes `sys.exit()` doesn't actually exit...
+
+def main():
+    logging.basicConfig()
+    logging.getLogger().setLevel(logging.DEBUG)
+
+    parser = argparse.ArgumentParser(description="accept connections and notify the other end upon ntfy activity, with a guaranteed amount of silence")
+    parser.add_argument('--verbose', action='store_true')
+    parser.add_argument('--host', type=str, default='')
+    parser.add_argument('--port', type=int)
+    parser.add_argument('--silence', type=int, help="number of seconds to remain silent upon accepting a connection")
+
+    args = parser.parse_args()
+
+    if args.verbose:
+        logging.getLogger().setLevel(logging.DEBUG)
+    else:
+        logging.getLogger().setLevel(logging.INFO)
+
+    adapter = Adapter(args.host, args.port, args.silence, get_topic())
+
+    listener_loop = threading.Thread(target=run_forever, name="listener_loop", args=(adapter.listener_loop,))
+    notify_loop = threading.Thread(target=run_forever, name="notify_loop", args=(adapter.notify_loop,))
+
+    # TODO: this method of exiting seems to sometimes leave the listener behind (?)
+    # preventing anyone else from re-binding the port.
+    listener_loop.start()
+    notify_loop.start()
+    listener_loop.join()
+    notify_loop.join()
+
+
+if __name__ == '__main__':
+    main()

hosts/by-name/servo/services/ntfy/ntfy-waiter.nix

@@ -0,0 +1,72 @@
+# service which adapts ntfy-sh into something suitable specifically for the Pinephone's
+# wake-on-lan (WoL) feature.
+# notably, it provides a mechanism by which the caller can be confident of an interval in which
+# zero traffic will occur on the TCP connection, thus allowing it to enter sleep w/o fear of hitting
+# race conditions in the Pinephone WoL feature.
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.sane.ntfy-waiter;
+  portLow = 5550;
+  portHigh = 5559;
+  portRange = lib.range portLow portHigh;
+  numPorts = portHigh - portLow + 1;
+  mkService = port: let
+    silence = port - portLow;
+    flags = lib.optional cfg.verbose "--verbose";
+    cli = [
+      "${cfg.package}/bin/ntfy-waiter"
+      "--port"
+      "${builtins.toString port}"
+      "--silence"
+      "${builtins.toString silence}"
+    ] ++ flags;
+  in {
+    "ntfy-waiter-${builtins.toString silence}" = {
+      # TODO: run not as root (e.g. as ntfy-sh)
+      description = "wait for notification, with ${builtins.toString silence} seconds of guaranteed silence";
+      serviceConfig = {
+        Type = "simple";
+        Restart = "always";
+        RestartSec = "5s";
+        ExecStart = lib.concatStringsSep " " cli;
+      };
+      after = [ "network.target" ];
+      wantedBy = [ "default.target" ];
+    };
+  };
+in
+{
+  options = with lib; {
+    sane.ntfy-waiter.enable = mkOption {
+      type = types.bool;
+      default = true;
+    };
+    sane.ntfy-waiter.verbose = mkOption {
+      type = types.bool;
+      default = true;
+    };
+    sane.ntfy-waiter.package = mkOption {
+      type = types.package;
+      default = pkgs.static-nix-shell.mkPython3Bin {
+        pname = "ntfy-waiter";
+        src = ./.;
+        pkgs = [ "ntfy-sh" ];
+      };
+      description = ''
+        exposed to provide an attr-path by which one may build the package for manual testing.
+      '';
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+    sane.ports.ports = lib.mkMerge (lib.forEach portRange (port: {
+      "${builtins.toString port}" = {
+        protocol = [ "tcp" ];
+        visibleTo.lan = true;
+        visibleTo.wan = true;
+        description = "colin-notification-waiter-${builtins.toString (port+1)}-of-${builtins.toString numPorts}";
+      };
+    }));
+    systemd.services = lib.mkMerge (builtins.map mkService portRange);
+  };
+}

hosts/by-name/servo/services/pict-rs.nix

@@ -5,7 +5,7 @@ let
   cfg = config.services.pict-rs;
 in
 {
-  sane.persist.sys.plaintext = lib.mkIf cfg.enable [
+  sane.persist.sys.byStore.plaintext = lib.mkIf cfg.enable [
     { user = "pict-rs"; group = "pict-rs"; path = cfg.dataDir; }
   ];
 

hosts/by-name/servo/services/pleroma.nix

@@ -14,7 +14,7 @@ let
   # logLevel = "debug";
 in
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     { user = "pleroma"; group = "pleroma"; path = "/var/lib/pleroma"; }
   ];
   services.pleroma.enable = true;

hosts/by-name/servo/services/postgres.nix

@@ -6,7 +6,7 @@ let
   KiB = n: 1024*n;
 in
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode?
     { user = "postgres"; group = "postgres"; path = "/var/lib/postgresql"; }
   ];

hosts/by-name/servo/services/prosody.nix

@@ -1,81 +0,0 @@
-# example configs:
-# - <https://github.com/kittywitch/nixfiles/blob/main/services/prosody.nix>
-# create users with:
-# - `sudo -u prosody prosodyctl adduser colin@uninsane.org`
-
-{ lib, ... }:
-
-# XXX disabled: doesn't send messages to nixnet.social (only receives them).
-# nixnet runs ejabberd, so revisiting that.
-lib.mkIf false
-{
-  sane.persist.sys.plaintext = [
-    { user = "prosody"; group = "prosody"; path = "/var/lib/prosody"; }
-  ];
-  sane.ports.ports."5222" = {
-    protocol = [ "tcp" ];
-    visibleTo.lan = true;
-    visibleTo.wan = true;
-    description = "colin-xmpp-client-to-server";
-  };
-  sane.ports.ports."5269" = {
-    protocol = [ "tcp" ];
-    visibleTo.wan = true;
-    description = "colin-xmpp-server-to-server";
-  };
-  sane.ports.ports."5280" = {
-    protocol = [ "tcp" ];
-    visibleTo.lan = true;
-    visibleTo.wan = true;
-    description = "colin-xmpp-bosh";
-  };
-  sane.ports.ports."5281" = {
-    protocol = [ "tcp" ];
-    visibleTo.lan = true;
-    visibleTo.wan = true;
-    description = "colin-xmpp-prosody-https";  # necessary?
-  };
-
-  # provide access to certs
-  users.users.prosody.extraGroups = [ "nginx" ];
-
-  security.acme.certs."uninsane.org".extraDomainNames = [
-    "conference.xmpp.uninsane.org"
-    "upload.xmpp.uninsane.org"
-  ];
-
-  services.prosody = {
-    enable = true;
-    admins = [ "colin@uninsane.org" ];
-    # allowRegistration = false;
-    # extraConfig = ''
-    #   s2s_require_encryption = true
-    #   c2s_require_encryption = true
-    # '';
-
-    extraModules = [ "private" "vcard" "privacy" "compression" "component" "muc" "pep" "adhoc" "lastactivity" "admin_adhoc" "blocklist"];
-
-    ssl.cert = "/var/lib/acme/uninsane.org/fullchain.pem";
-    ssl.key = "/var/lib/acme/uninsane.org/key.pem";
-
-    muc = [
-      {
-        domain = "conference.xmpp.uninsane.org";
-      }
-    ];
-    uploadHttp.domain = "upload.xmpp.uninsane.org";
-
-    virtualHosts = {
-      localhost = {
-        domain = "localhost";
-        enabled = true;
-      };
-      "xmpp.uninsane.org" = {
-        domain = "uninsane.org";
-        enabled = true;
-        ssl.cert = "/var/lib/acme/uninsane.org/fullchain.pem";
-        ssl.key = "/var/lib/acme/uninsane.org/key.pem";
-      }; 
-    };
-  };
-}

hosts/by-name/servo/services/prosody/default.nix

@@ -0,0 +1,289 @@
+# example configs:
+# - official: <https://prosody.im/doc/example_config>
+# - nixos: <https://github.com/kittywitch/nixfiles/blob/main/services/prosody.nix>
+# config options:
+# - <https://prosody.im/doc/configure>
+#
+# modules:
+# - main: <https://prosody.im/doc/modules>
+# - community: <https://modules.prosody.im/index.html>
+#
+# debugging:
+# - logging:
+#   - enable `stanza_debug` module
+#   - enable `log.debug = "*syslog"` in extraConfig
+# - interactive:
+#   - `telnet localhost 5582` (this is equal to `prosodyctl shell` -- but doesn't hang)
+#     - `watch:stanzas(target_spec, filter)` -> to log stanzas, for version > 0.12
+#   - console docs: <https://prosody.im/doc/console>
+#   - can modify/inspect arbitrary internals (lua) by prefixing line with `> `
+#     - e.g. `> _G` to print all globals
+#
+# sanity checks:
+# - `sudo -u prosody -g prosody prosodyctl check connectivity`
+# - `sudo -u prosody -g prosody prosodyctl check turn`
+# - `sudo -u prosody -g prosody prosodyctl check turn -v --ping=stun.conversations.im`
+#   - checks that my stun/turn server is usable by clients of conversations.im (?)
+# - `sudo -u prosody -g prosody prosodyctl check`  (dns, config, certs)
+#
+#
+# create users with:
+# - `sudo -u prosody prosodyctl adduser colin@uninsane.org`
+#
+#
+# federation/support matrix:
+# - nixnet.services (runs ejabberd):
+#   - WORKS: sending and receiving PMs and calls (2023/10/15)
+#     - N.B.: it didn't originally work; was solved by disabling the lua-unbound DNS option & forcing the system/local resolver
+# - cheogram (XMPP <-> SMS gateway):
+#   - WORKS: sending and receiving PMs, images (2023/10/15)
+#   - PARTIAL: calls (xmpp -> tel works; tel -> xmpp fails)
+#     - maybe i need to setup stun/turn
+#
+# TODO:
+# - enable push notifications (mod_cloud_notify)
+# - optimize coturn (e.g. move off of the VPN!)
+# - ensure muc is working
+# - enable file uploads
+#   - "upload.xmpp.uninsane.org:http_upload: URL: <https://upload.xmpp.uninsane.org:5281/upload> - Ensure this can be reached by users"
+# - disable or fix bosh (jabber over http):
+#   - "certmanager: No certificate/key found for client_https port 0"
+
+{ lib, pkgs, ... }:
+
+let
+  # enables very verbose logging
+  enableDebug = false;
+in
+{
+  sane.persist.sys.byStore.plaintext = [
+    { user = "prosody"; group = "prosody"; path = "/var/lib/prosody"; }
+  ];
+  sane.ports.ports."5000" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpp-prosody-fileshare-proxy65";
+  };
+  sane.ports.ports."5222" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpp-client-to-server";
+  };
+  sane.ports.ports."5223" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpps-client-to-server";  # XMPP over TLS
+  };
+  sane.ports.ports."5269" = {
+    protocol = [ "tcp" ];
+    visibleTo.wan = true;
+    description = "colin-xmpp-server-to-server";
+  };
+  sane.ports.ports."5270" = {
+    protocol = [ "tcp" ];
+    visibleTo.wan = true;
+    description = "colin-xmpps-server-to-server";  # XMPP over TLS
+  };
+  sane.ports.ports."5280" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpp-bosh";
+  };
+  sane.ports.ports."5281" = {
+    protocol = [ "tcp" ];
+    visibleTo.lan = true;
+    visibleTo.wan = true;
+    description = "colin-xmpp-prosody-https";  # necessary?
+  };
+
+  users.users.prosody.extraGroups = [
+    "nginx"  # provide access to certs
+    "ntfy-sh"  # access to secret ntfy topic
+  ];
+
+  security.acme.certs."uninsane.org".extraDomainNames = [
+    "xmpp.uninsane.org"
+    "conference.xmpp.uninsane.org"
+    "upload.xmpp.uninsane.org"
+  ];
+
+  # exists so the XMPP server's cert can obtain altNames for all its resources
+  services.nginx.virtualHosts."xmpp.uninsane.org" = {
+    useACMEHost = "uninsane.org";
+  };
+  services.nginx.virtualHosts."conference.xmpp.uninsane.org" = {
+    useACMEHost = "uninsane.org";
+  };
+  services.nginx.virtualHosts."upload.xmpp.uninsane.org" = {
+    useACMEHost = "uninsane.org";
+  };
+
+  sane.dns.zones."uninsane.org".inet = {
+    # XXX: SRV records have to point to something with a A/AAAA record; no CNAMEs
+    A."xmpp" =                "%ANATIVE%";
+    CNAME."conference.xmpp" = "xmpp";
+    CNAME."upload.xmpp" =     "xmpp";
+
+    # _Service._Proto.Name    TTL Class SRV    Priority Weight Port Target
+    # - <https://xmpp.org/extensions/xep-0368.html>
+    # something's requesting the SRV records for conference.xmpp, so let's include it
+    # nothing seems to request XMPP SRVs for the other records (except @)
+    # lower numerical priority field tells clients to prefer this method
+    SRV."_xmpps-client._tcp.conference.xmpp" =       "3 50 5223 xmpp";
+    SRV."_xmpps-server._tcp.conference.xmpp" =       "3 50 5270 xmpp";
+    SRV."_xmpp-client._tcp.conference.xmpp" =        "5 50 5222 xmpp";
+    SRV."_xmpp-server._tcp.conference.xmpp" =        "5 50 5269 xmpp";
+
+    SRV."_xmpps-client._tcp" =                "3 50 5223 xmpp";
+    SRV."_xmpps-server._tcp" =                "3 50 5270 xmpp";
+    SRV."_xmpp-client._tcp" =                 "5 50 5222 xmpp";
+    SRV."_xmpp-server._tcp" =                 "5 50 5269 xmpp";
+  };
+
+  # help Prosody find its certificates.
+  # pointing it to /var/lib/acme doesn't quite work because it expects the private key
+  # to be named `privkey.pem` instead of acme's `key.pem`
+  # <https://prosody.im/doc/certificates#automatic_location>
+  sane.fs."/etc/prosody/certs/uninsane.org/fullchain.pem" = {
+    symlink.target = "/var/lib/acme/uninsane.org/fullchain.pem";
+    wantedBeforeBy = [ "prosody.service" ];
+  };
+  sane.fs."/etc/prosody/certs/uninsane.org/privkey.pem" = {
+    symlink.target = "/var/lib/acme/uninsane.org/key.pem";
+    wantedBeforeBy = [ "prosody.service" ];
+  };
+
+  services.prosody = {
+    enable = true;
+    package = pkgs.prosody.override {
+      # XXX(2023/10/15): build without lua-unbound support.
+      # this forces Prosody to fall back to the default Lua DNS resolver, which seems more reliable.
+      # fixes errors like "unbound.queryXYZUV: Resolver error: out of memory"
+      # related: <https://issues.prosody.im/1737#comment-11>
+      lua.withPackages = selector: pkgs.lua.withPackages (p:
+        selector (p // { luaunbound = null; })
+      );
+      # withCommunityModules = [ "turncredentials" ];
+    };
+    admins = [ "colin@uninsane.org" ];
+    # allowRegistration = false;  # defaults to false
+
+    muc = [
+      {
+        domain = "conference.xmpp.uninsane.org";
+      }
+    ];
+    uploadHttp.domain = "upload.xmpp.uninsane.org";
+
+    virtualHosts = {
+      # "Prosody requires at least one enabled VirtualHost to function. You can
+      # safely remove or disable 'localhost' once you have added another."
+      # localhost = {
+      #   domain = "localhost";
+      #   enabled = true;
+      # };
+      "xmpp.uninsane.org" = {
+        domain = "uninsane.org";
+        enabled = true;
+      };
+    };
+
+    ## modules:
+    # these are enabled by default, via <repo:nixos/nixpkgs:/pkgs/servers/xmpp/prosody/default.nix>
+    # - cloud_notify
+    # - http_upload
+    # - vcard_muc
+    # these are enabled by the module defaults (services.prosody.modules.<foo>)
+    # - admin_adhoc
+    # - blocklist
+    # - bookmarks
+    # - carbons
+    # - cloud_notify
+    # - csi
+    # - dialback
+    # - disco
+    # - http_files
+    # - mam
+    # - pep
+    # - ping
+    # - private
+    #   - XEP-0049: let clients store arbitrary (private) data on the server
+    # - proxy65
+    #   - XEP-0065: allow server to proxy file transfers between two clients who are behind NAT
+    # - register
+    # - roster
+    # - saslauth
+    # - smacks
+    # - time
+    # - tls
+    # - uptime
+    # - vcard_legacy
+    # - version
+
+    extraPluginPaths = [ ./modules ];
+
+    extraModules = [
+      # admin_shell: allows `prosodyctl shell` to work
+      # see: <https://prosody.im/doc/modules/mod_admin_shell>
+      # see: <https://prosody.im/doc/console>
+      "admin_shell"
+      "admin_telnet"  #< needed by admin_shell
+      # lastactivity: XEP-0012: allow users to query how long another user has been idle for
+      # - not sure why i enabled this; think it was in someone's config i referenced
+      "lastactivity"
+      # allows prosody to share TURN/STUN secrets with XMPP clients to provide them access to the coturn server.
+      # see: <https://prosody.im/doc/coturn>
+      "turn_external"
+      # legacy coturn integration
+      # see: <https://modules.prosody.im/mod_turncredentials.html>
+      # "turncredentials"
+      "sane_ntfy"
+    ] ++ lib.optionals enableDebug [
+      "stanza_debug"  #< logs EVERY stanza as debug: <https://prosody.im/doc/modules/mod_stanza_debug>
+    ];
+
+    extraConfig = ''
+      local function readAll(file)
+        local f = assert(io.open(file, "rb"))
+        local content = f:read("*all")
+        f:close()
+        -- remove trailing newline
+        return string.gsub(content, "%s+", "")
+      end
+
+      -- logging docs:
+      -- - <https://prosody.im/doc/logging>
+      -- - <https://prosody.im/doc/advanced_logging>
+      -- levels: debug, info, warn, error
+      log = {
+        ${if enableDebug then "debug" else "info"} = "*syslog";
+      }
+
+      -- see: <https://prosody.im/doc/certificates#automatic_location>
+      -- try to solve: "certmanager: Error indexing certificate directory /etc/prosody/certs: cannot open /etc/prosody/certs: No such file or directory"
+      -- only, this doesn't work because prosody doesn't like acme's naming scheme
+      -- certificates = "/var/lib/acme"
+
+      c2s_direct_tls_ports = { 5223 }
+      s2s_direct_tls_ports = { 5270 }
+
+      turn_external_host = "turn.uninsane.org"
+      turn_external_secret = readAll("/var/lib/coturn/shared_secret.bin")
+      -- turn_external_user = "prosody"
+
+      -- legacy mod_turncredentials integration
+      -- turncredentials_host = "turn.uninsane.org"
+      -- turncredentials_secret = readAll("/var/lib/coturn/shared_secret.bin")
+
+      ntfy_binary = "${pkgs.ntfy-sh}/bin/ntfy"
+      ntfy_topic = readAll("/run/secrets/ntfy-sh-topic")
+
+      -- s2s_require_encryption = true
+      -- c2s_require_encryption = true
+    '';
+  };
+}

hosts/by-name/servo/services/prosody/modules/mod_sane_ntfy.lua

@@ -0,0 +1,52 @@
+-- simple proof-of-concept Prosody module
+-- module development guide: <https://prosody.im/doc/developers/modules>
+-- module API docs: <https://prosody.im/doc/developers/moduleapi>
+--
+-- much of this code is lifted from Prosody's own `mod_cloud_notify`
+
+local jid = require"util.jid";
+
+local ntfy = module:get_option_string("ntfy_binary", "ntfy");
+local ntfy_topic = module:get_option_string("ntfy_topic", "xmpp");
+
+module:log("info", "initialized");
+
+local function is_urgent(stanza)
+  if stanza.name == "message" then
+    if stanza:get_child("propose", "urn:xmpp:jingle-message:0") then
+      return true, "jingle call";
+    end
+  end
+end
+
+local function publish_ntfy(message)
+  -- message should be the message to publish
+  local ntfy_url = string.format("https://ntfy.uninsane.org/%s", ntfy_topic)
+  local cmd = string.format("%s pub %q %q", ntfy, ntfy_url, message)
+  module.log("debug", "invoking ntfy: %s", cmd)
+  local success, reason, code = os.execute(cmd)
+  if not success then
+    module:log("warn", "ntfy failed: %s => %s %d", cmd, reason, code)
+  end
+end
+
+
+local function archive_message_added(event)
+  -- event is: { origin = origin, stanza = stanza, for_user = store_user, id = id }
+  local stanza = event.stanza;
+  local to = stanza.attr.to;
+  to = to and jid.split(to) or event.origin.username;
+
+  -- only notify if the stanza destination is the mam user we store it for
+  if event.for_user == to then
+    local is_urgent_stanza, urgent_reason = is_urgent(event.stanza);
+
+    if is_urgent_stanza then
+      module:log("info", "urgent push for %s (%s)", to, urgent_reason);
+      publish_ntfy(urgent_reason)
+    end
+  end
+end
+
+
+module:hook("archive-message-added", archive_message_added);

hosts/by-name/servo/services/slskd.nix

@@ -0,0 +1,74 @@
+# Soulseek daemon (p2p file sharing with an emphasis on Music)
+# docs: <https://github.com/slskd/slskd/blob/master/docs/config.md>
+#
+# config precedence (higher precedence overrules lower precedence):
+# - Default Values < Environment Variables < YAML Configuraiton File < Command Line Arguments
+{ config, lib, ... }:
+{
+  sane.persist.sys.byStore.plaintext = [
+    { user = "slskd"; group = "slskd"; path = "/var/lib/slskd"; }
+  ];
+  sops.secrets."slskd_env" = {
+    owner = config.users.users.slskd.name;
+    mode = "0400";
+  };
+
+  users.users.slskd.extraGroups = [ "media" ];
+
+  sane.ports.ports."50000" = {
+    protocol = [ "tcp" ];
+    # not visible to WAN: i run this in a separate netns
+    visibleTo.ovpn = true;
+    description = "colin-soulseek";
+  };
+
+  sane.dns.zones."uninsane.org".inet.CNAME."soulseek" = "native";
+
+  services.nginx.virtualHosts."soulseek.uninsane.org" = {
+    forceSSL = true;
+    enableACME = true;
+    locations."/" = {
+      proxyPass = "http://10.0.1.6:5001";
+      proxyWebsockets = true;
+    };
+  };
+
+  services.slskd.enable = true;
+  # env file, for auth (SLSKD_SLSK_PASSWORD, SLSKD_SLSK_USERNAME)
+  services.slskd.environmentFile = config.sops.secrets.slskd_env.path;
+  services.slskd.settings = {
+    soulseek.diagnostic_level = "Debug";  # one of "None"|"Warning"|"Info"|"Debug"
+    shares.directories = [
+      # folders to share
+      # syntax: <https://github.com/slskd/slskd/blob/master/docs/config.md#directories>
+      # [Alias]/path/on/disk
+      # NOTE: Music library is quick to scan; videos take a solid 10min to scan.
+      # TODO: re-enable the other libraries
+      # "[Audioooks]/var/lib/uninsane/media/Books/Audiobooks"
+      # "[Books]/var/lib/uninsane/media/Books/Books"
+      # "[Manga]/var/lib/uninsane/media/Books/Visual"
+      # "[games]/var/lib/uninsane/media/games"
+      "[Music]/var/lib/uninsane/media/Music"
+      # "[Film]/var/lib/uninsane/media/Videos/Film"
+      # "[Shows]/var/lib/uninsane/media/Videos/Shows"
+    ];
+    # directories.downloads = "..." # TODO
+    # directories.incomplete = "..." # TODO
+    # what unit is this? kbps??
+    global.upload.speed_limit = 32000;
+    web.logging = true;
+    debug = true;
+    flags.no_logo = true;  # don't show logo at start
+    # flags.volatile = true;  # store searches and active transfers in RAM (completed transfers still go to disk). rec for btrfs/zfs
+  };
+
+  systemd.services.slskd = {
+    serviceConfig = {
+      # run this behind the OVPN static VPN
+      NetworkNamespacePath = "/run/netns/ovpns";
+      Restart = "on-failure";
+      RestartSec = "30s";
+      Group = "media";
+    };
+  };
+}

hosts/by-name/servo/services/transmission.nix

@@ -1,12 +1,27 @@
-{ pkgs, ... }:
+{ config, pkgs, ... }:
 
 {
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: mode? we need this specifically for the stats tracking in .config/
-    { user = "transmission"; group = "transmission"; path = "/var/lib/transmission"; }
+    { user = "transmission"; group = config.users.users.transmission.group; path = "/var/lib/transmission"; }
   ];
+  users.users.transmission.extraGroups = [ "media" ];
+
   services.transmission.enable = true;
+  services.transmission.package = pkgs.transmission_4;  #< 2023/09/06: nixpkgs `transmission` defaults to old 3.00
+  #v setting `group` this way doesn't tell transmission to `chown` the files it creates
+  #  it's a nixpkgs setting which just runs the transmission daemon as this group
+  services.transmission.group = "media";
+
+  # transmission will by default not allow the world to read its files.
+  services.transmission.downloadDirPermissions = "775";
+  services.transmission.extraFlags = [
+    "--log-level=debug"
+  ];
+
   services.transmission.settings = {
+    # message-level = 3;  #< enable for debug logging. 0-3, default is 2.
+    # 0.0.0.0 => allow rpc from any host: we gate it via firewall and auth requirement
     rpc-bind-address = "0.0.0.0";
     #rpc-host-whitelist = "bt.uninsane.org";
     #rpc-whitelist = "*.*.*.*";
@@ -17,9 +32,8 @@
     rpc-password = "{503fc8928344f495efb8e1f955111ca5c862ce0656SzQnQ5";
     rpc-whitelist-enabled = false;
 
-    # download-dir = "/opt/uninsane/media/";
     # hopefully, make the downloads world-readable
-    umask = 0;
+    # umask = 0;  #< default is 2: i.e. deny writes from world
 
     # force peer connections to be encrypted
     encryption = 2;
@@ -35,17 +49,18 @@
 
     download-dir = "/var/lib/uninsane/media";
     incomplete-dir = "/var/lib/uninsane/media/incomplete";
-
+    # transmission regularly fails to move stuff from the incomplete dir to the main one, so disable:
+    # TODO: uncomment this line!
+    incomplete-dir-enabled = false;
   };
-  # transmission will by default not allow the world to read its files.
-  services.transmission.downloadDirPermissions = "775";
 
   systemd.services.transmission.after = [ "wireguard-wg-ovpns.service" ];
   systemd.services.transmission.partOf = [ "wireguard-wg-ovpns.service" ];
   systemd.services.transmission.serviceConfig = {
     # run this behind the OVPN static VPN
     NetworkNamespacePath = "/run/netns/ovpns";
-    LogLevelMax = "warning";
+    Restart = "on-failure";
+    RestartSec = "30s";
   };
 
   # service to automatically backup torrents i add to transmission
@@ -76,5 +91,10 @@
   };
 
   sane.dns.zones."uninsane.org".inet.CNAME."bt" = "native";
+  sane.ports.ports."51413" = {
+    protocol = [ "tcp" "udp" ];
+    visibleTo.ovpn = true;
+    description = "colin-bittorrent";
+  };
 }
 

hosts/by-name/servo/services/trust-dns.nix

@@ -1,23 +1,25 @@
+# TODO: split this file apart into smaller files to make it easier to understand
 { config, lib, pkgs, ... }:
 
+let
+  nativeAddrs = lib.mapAttrs (_name: builtins.head) config.sane.dns.zones."uninsane.org".inet.A;
+  bindOvpn = "10.0.1.5";
+in lib.mkMerge [
 {
   services.trust-dns.enable = true;
 
-  services.trust-dns.settings.listen_addrs_ipv4 = [
-    # specify each address explicitly, instead of using "*".
-    # this ensures responses are sent from the address at which the request was received.
-    config.sane.hosts.by-name."servo".lan-ip
-    "10.0.1.5"
-  ];
   # don't bind to IPv6 until i explicitly test that stack
   services.trust-dns.settings.listen_addrs_ipv6 = [];
   services.trust-dns.quiet = true;
+  # FIXME(2023/11/26): services.trust-dns.debug doesn't log requests: use RUST_LOG=debug env for that.
+  # - see: <https://github.com/hickory-dns/hickory-dns/issues/2082>
   # services.trust-dns.debug = true;
 
   sane.ports.ports."53" = {
     protocol = [ "udp" "tcp" ];
     visibleTo.lan = true;
     visibleTo.wan = true;
+    visibleTo.ovpn = true;
     description = "colin-dns-hosting";
   };
 
@@ -34,18 +36,19 @@
   sane.dns.zones."uninsane.org".inet = {
     SOA."@" = ''
       ns1.uninsane.org. admin-dns.uninsane.org. (
-                                      2022122101 ; Serial
+                                      2023092101 ; Serial
                                       4h         ; Refresh
                                       30m        ; Retry
                                       7d         ; Expire
                                       5m)        ; Negative response TTL
     '';
-    TXT."rev" = "2023052901";
+    TXT."rev" = "2023092101";
 
     CNAME."native" = "%CNAMENATIVE%";
     A."@" =      "%ANATIVE%";
-    A."wan" = "%AWAN%";
+    A."servo.wan" = "%AWAN%";
     A."servo.lan" = config.sane.hosts.by-name."servo".lan-ip;
+    A."servo.hn" = config.sane.hosts.by-name."servo".wg-home.ip;
 
     # XXX NS records must also not be CNAME
     # it's best that we keep this identical, or a superset of, what org. lists as our NS.
@@ -63,55 +66,23 @@
 
   services.trust-dns.settings.zones = [ "uninsane.org" ];
 
-  services.trust-dns.package =
-    let
-      sed = "${pkgs.gnused}/bin/sed";
-      zone-dir = "/var/lib/trust-dns";
-      zone-wan = "${zone-dir}/wan/uninsane.org.zone";
-      zone-lan = "${zone-dir}/lan/uninsane.org.zone";
-      zone-template = pkgs.writeText "uninsane.org.zone.in" config.sane.dns.zones."uninsane.org".rendered;
-    in pkgs.writeShellScriptBin "named" ''
-      # compute wan/lan values
-      mkdir -p ${zone-dir}/{ovpn,wan,lan}
-      wan=$(cat '${config.sane.services.dyn-dns.ipPath}')
-      lan=${config.sane.hosts.by-name."servo".lan-ip}
-
-      # create specializations that resolve native.uninsane.org to different CNAMEs
-      ${sed} s/%AWAN%/$wan/ ${zone-template} \
-        | ${sed} s/%CNAMENATIVE%/wan/ \
-        | ${sed} s/%ANATIVE%/$wan/ \
-        > ${zone-wan}
-      ${sed} s/%AWAN%/$wan/ ${zone-template} \
-        | ${sed} s/%CNAMENATIVE%/servo.lan/ \
-        | ${sed} s/%ANATIVE%/$lan/ \
-        > ${zone-lan}
-
-      # launch the different interfaces, separately
-      ${pkgs.trust-dns}/bin/named --port 53 --zonedir ${zone-dir}/wan/ $@ &
-      WANPID=$!
-      ${pkgs.trust-dns}/bin/named --port 1053 --zonedir ${zone-dir}/lan/ $@ &
-      LANPID=$!
-
-      # wait until any of the processes exits, then kill them all and exit error
-      while kill -0 $WANPID $LANPID ; do
-        sleep 5
-      done
-      kill $WANPID $LANPID
-      exit 1
-    '';
-
+  # TODO: can i transform this into some sort of service group?
+  # have `systemctl restart trust-dns.service` restart all the individual services?
   systemd.services.trust-dns.serviceConfig = {
     DynamicUser = lib.mkForce false;
     User = "trust-dns";
     Group = "trust-dns";
+    wantedBy = lib.mkForce [];
   };
+  systemd.services.trust-dns.enable = false;
+
   users.groups.trust-dns = {};
   users.users.trust-dns = {
     group = "trust-dns";
     isSystemUser = true;
   };
 
-  sane.services.dyn-dns.restartOnChange = [ "trust-dns.service" ];
+  # sane.services.dyn-dns.restartOnChange = [ "trust-dns.service" ];
 
   networking.nat.enable = true;
   networking.nat.extraCommands = ''
@@ -127,12 +98,113 @@
       -m iprange --src-range 10.78.76.0-10.78.79.255 \
       -j DNAT --to-destination :1053
   '';
-
   sane.ports.ports."1053" = {
     # because the NAT above redirects in nixos-nat-pre, LAN requests behave as though they arrived on the external interface at the redirected port.
     # TODO: try nixos-nat-post instead?
+    # TODO: or, don't NAT from port 53 -> port 1053, but rather nat from LAN addr to a loopback addr.
+    # - this is complicated in that loopback is a different interface than eth0, so rewriting the destination address would cause the packets to just be dropped by the interface
     protocol = [ "udp" "tcp" ];
     visibleTo.lan = true;
     description = "colin-redirected-dns-for-lan-namespace";
   };
 }
+{
+  systemd.services =
+    let
+      sed = "${pkgs.gnused}/bin/sed";
+      stateDir = "/var/lib/trust-dns";
+      zoneTemplate = pkgs.writeText "uninsane.org.zone.in" config.sane.dns.zones."uninsane.org".rendered;
+
+      zoneDirFor = flavor: "${stateDir}/${flavor}";
+      zoneFor = flavor: "${zoneDirFor flavor}/uninsane.org.zone";
+      mkTrustDnsService = opts: flavor: let
+        flags = let baseCfg = config.services.trust-dns; in
+          (lib.optional baseCfg.debug "--debug") ++ (lib.optional baseCfg.quiet "--quiet");
+        flagsStr = builtins.concatStringsSep " " flags;
+
+        anative = nativeAddrs."servo.${flavor}";
+
+        toml = pkgs.formats.toml { };
+        configTemplate = opts.config or (toml.generate "trust-dns-${flavor}.toml" (
+          (
+            lib.filterAttrsRecursive (_: v: v != null) config.services.trust-dns.settings
+          ) // {
+            listen_addrs_ipv4 = opts.listen or [ anative ];
+          }
+        ));
+        configFile = "${stateDir}/${flavor}-config.toml";
+
+        port = opts.port or 53;
+      in {
+        description = "trust-dns Domain Name Server (serving ${flavor})";
+        unitConfig.Documentation = "https://trust-dns.org/";
+
+        preStart = ''
+          wan=$(cat '${config.sane.services.dyn-dns.ipPath}')
+          ${sed} s/%AWAN%/$wan/ ${configTemplate} > ${configFile}
+        '' + lib.optionalString (!opts ? config) ''
+          mkdir -p ${zoneDirFor flavor}
+          ${sed} \
+            -e s/%CNAMENATIVE%/servo.${flavor}/ \
+            -e s/%ANATIVE%/${anative}/ \
+            -e s/%AWAN%/$wan/ \
+            -e s/%AOVPNS%/185.157.162.178/ \
+            ${zoneTemplate} > ${zoneFor flavor}
+        '';
+        serviceConfig = config.systemd.services.trust-dns.serviceConfig // {
+          ExecStart = ''
+            ${pkgs.trust-dns}/bin/${pkgs.trust-dns.meta.mainProgram} \
+            --port ${builtins.toString port} \
+            --zonedir ${zoneDirFor flavor}/ \
+            --config ${configFile} ${flagsStr}
+          '';
+        };
+
+        after = [ "network.target" ];
+        wantedBy = [ "multi-user.target" ];
+      };
+    in {
+      trust-dns-wan = mkTrustDnsService { listen = [ nativeAddrs."servo.lan" bindOvpn ]; } "wan";
+      trust-dns-lan = mkTrustDnsService { port = 1053; } "lan";
+      trust-dns-hn = mkTrustDnsService { port = 1053; } "hn";
+      trust-dns-hn-resolver = mkTrustDnsService {
+        config = pkgs.writeText "hn-resolver-config.toml" ''
+          # i host a resolver in the wireguard VPN so that clients can resolve DNS through the VPN.
+          # (that's what this file achieves).
+          #
+          # one would expect this resolver could host the authoritative zone for `uninsane.org`, and then forward everything else to the system resolver...
+          # and while that works for `dig`, it breaks for `nslookup` (and so `ssh`, etc).
+          #
+          # DNS responses include a flag for if the responding server is the authority of the zone queried.
+          # it seems that default Linux stub resolvers either:
+          # - expect DNSSEC when the response includes that bit, or
+          # - expect A records to be in the `answer` section instead of `additional` section.
+          # or perhaps something more nuanced. but for `nslookup` to be reliable, it has to talk to an
+          # instance of trust-dns which is strictly a resolver, with no authority.
+          # hence, this config: a resolver which forwards to the actual authority.
+
+          listen_addrs_ipv4 = ["${nativeAddrs."servo.hn"}"]
+          listen_addrs_ipv6 = []
+
+          [[zones]]
+          zone = "uninsane.org"
+          zone_type = "Forward"
+          stores = { type = "forward", name_servers = [{ socket_addr = "${nativeAddrs."servo.hn"}:1053", protocol = "udp", trust_nx_responses = true }] }
+
+          [[zones]]
+          # forward the root zone to the local DNS resolver
+          zone = "."
+          zone_type = "Forward"
+          stores = { type = "forward", name_servers = [{ socket_addr = "127.0.0.53:53", protocol = "udp", trust_nx_responses = true }] }
+        '';
+      } "hn-resolver";
+    };
+
+  sane.services.dyn-dns.restartOnChange = [
+    "trust-dns-wan.service"
+    "trust-dns-lan.service"
+    "trust-dns-hn.service"
+    # "trust-dns-hn-resolver.service"  # doesn't need restart because it doesn't know about WAN IP
+  ];
+}
+]

hosts/common/default.nix

@@ -3,8 +3,10 @@
   imports = [
     ./feeds.nix
     ./fs.nix
-    ./hardware.nix
+    ./hardware
     ./home
+    ./hostnames.nix
+    ./hosts.nix
     ./ids.nix
     ./machine-id.nix
     ./net.nix
@@ -20,25 +22,54 @@
   sane.nixcache.enable-trusted-keys = true;
   sane.nixcache.enable = lib.mkDefault true;
   sane.persist.enable = lib.mkDefault true;
+  sane.root-on-tmpfs = lib.mkDefault true;
   sane.programs.sysadminUtils.enableFor.system = lib.mkDefault true;
   sane.programs.consoleUtils.enableFor.user.colin = lib.mkDefault true;
 
-  nixpkgs.config.allowUnfree = true;
-  nixpkgs.config.allowBroken = true;  # NIXPKGS_ALLOW_BROKEN
+  nixpkgs.config.allowUnfree = true;  # NIXPKGS_ALLOW_UNFREE=1
+  nixpkgs.config.allowBroken = true;  # NIXPKGS_ALLOW_BROKEN=1
 
   # time.timeZone = "America/Los_Angeles";
   time.timeZone = "Etc/UTC";  # DST is too confusing for me => use a stable timezone
 
-  # allow `nix flake ...` command
-  # TODO: is this still required?
   nix.extraOptions = ''
+    # see: `man nix.conf`
+    # useful when a remote builder has a faster internet connection than me
+    builders-use-substitutes = true  # default: false
+    # maximum seconds to wait when connecting to binary substituter
+    connect-timeout = 3  # default: 0
+    # download-attempts = 5  # default: 5
+    # allow `nix flake ...` command
     experimental-features = nix-command flakes
+    # whether to build from source when binary substitution fails
+    fallback = true  # default: false
+    # whether to keep building dependencies if any other one fails
+    keep-going = true  # default: false
+    # whether to keep build-only dependencies of GC roots (e.g. C compiler) when doing GC
+    keep-outputs = true  # default: false
+    # how many lines to show from failed build
+    log-lines = 30  # default: 10
+    # narinfo-cache-negative-ttl = 3600  # default: 3600
+    # whether to use ~/.local/state/nix/profile instead of ~/.nix-profile, etc
+    use-xdg-base-directories = true  # default: false
+    # whether to warn if repository has uncommited changes
+    warn-dirty = false  # default: true
   '';
   # hardlinks identical files in the nix store to save 25-35% disk space.
   # unclear _when_ this occurs. it's not a service.
   # does the daemon continually scan the nix store?
   # does the builder use some content-addressed db to efficiently dedupe?
   nix.settings.auto-optimise-store = true;
+  # TODO: see if i can remove this?
+  nix.settings.trusted-users = [ "root" ];
+
+  services.journald.extraConfig = ''
+    # docs: `man journald.conf`
+    # merged journald config is deployed to /etc/systemd/journald.conf
+    [Journal]
+    # disable journal compression because the underlying fs is compressed
+    Compress=no
+  '';
 
   systemd.services.nix-daemon.serviceConfig = {
     # the nix-daemon manages nix builders

hosts/common/feeds.nix

@@ -57,84 +57,57 @@ let
     };
 
   podcasts = [
-    (fromDb "lexfridman.com/podcast" // rat)
-    ## Astral Codex Ten
-    (fromDb "sscpodcast.libsyn.com" // rat)
-    ## Less Wrong Curated
-    (fromDb "feeds.libsyn.com/421877" // rat)
-    ## Econ Talk
-    (fromDb "feeds.simplecast.com/wgl4xEgL" // rat)
-    ## Cory Doctorow -- both podcast & text entries
-    (fromDb "craphound.com" // pol)
-    ## Maggie Killjoy -- referenced by Cory Doctorow
-    (fromDb "omny.fm/shows/cool-people-who-did-cool-stuff" // pol)
-    ## also Maggie Killjoy
-    (fromDb "feeds.megaphone.fm/behindthebastards" // pol)
-    ## Jennifer Briney
-    (fromDb "congressionaldish.libsyn.com" // pol)
-    (fromDb "werenotwrong.fireside.fm" // pol)
-    (fromDb "usefulidiots.substack.com" // pol)
-    # (mkPod "https://podcasts.la.utexas.edu/this-is-democracy/feed/podcast/" // pol // weekly)
-    ## Civboot -- https://anchor.fm/civboot
-    (fromDb "anchor.fm/s/34c7232c/podcast/rss" // tech)
-    ## Emerge: making sense of what's next -- <https://www.whatisemerging.com/emergepodcast>
-    (mkPod "https://anchor.fm/s/21bc734/podcast/rss" // pol // infrequent)
-    (fromDb "feeds.feedburner.com/80000HoursPodcast" // rat)
-    ## Daniel Huberman on sleep
-    (fromDb "feeds.megaphone.fm/hubermanlab" // uncat)
-    ## Multidisciplinary Association for Psychedelic Studies
-    (fromDb "mapspodcast.libsyn.com" // uncat)
+    (fromDb "acquiredlpbonussecretsecret.libsyn.com" // tech)  # ACQ2 - more "Acquired" episodes
     (fromDb "allinchamathjason.libsyn.com" // pol)
-    (fromDb "feeds.transistor.fm/acquired" // tech)
-    ## ACQ2 - more "Acquired" episodes
-    (fromDb "acquiredlpbonussecretsecret.libsyn.com" // tech)
-    # The Intercept - Deconstructed
-    (fromDb "rss.acast.com/deconstructed")
-    # (fromDb "rss.prod.firstlook.media/deconstructed/podcast.rss" // pol)  #< possible URL rot
-    ## The Daily
-    (mkPod "https://feeds.simplecast.com/54nAGcIl" // pol // daily)
-    # The Intercept - Intercepted
-    (fromDb "rss.acast.com/intercepted-with-jeremy-scahill")
-    # (fromDb "rss.prod.firstlook.media/intercepted/podcast.rss" // pol)  #< possible URL rot
-    (fromDb "podcast.posttv.com/itunes/post-reports.xml" // pol)
-    ## Eric Weinstein
-    (fromDb "rss.art19.com/the-portal" // rat)
-    (fromDb "darknetdiaries.com" // tech)
-    ## Radiolab -- also available here, but ONLY OVER HTTP: <http://feeds.wnyc.org/radiolab>
-    (fromDb "feeds.feedburner.com/radiolab" // pol)
-    ## Sam Harris
-    (fromDb "wakingup.libsyn.com" // pol)
-    ## 99% Invisible -- also available here: <https://feeds.simplecast.com/BqbsxVfO>
-    (fromDb "feeds.99percentinvisible.org/99percentinvisible" // pol)
-    (fromDb "rss.acast.com/ft-tech-tonic" // tech)
-    (fromDb "feeds.feedburner.com/dancarlin/history" // rat)
-    (fromDb "rss.art19.com/60-minutes" // pol)
-    ## The Verge - Decoder
-    (fromDb "feeds.megaphone.fm/recodedecode" // tech)
-    ## Matrix (chat) Live
-    (fromDb "feed.podbean.com/matrixlive/feed.xml" // tech)
+    (fromDb "anchor.fm/s/34c7232c/podcast/rss" // tech)  # Civboot -- https://anchor.fm/civboot
     (fromDb "cast.postmarketos.org" // tech)
+    (fromDb "congressionaldish.libsyn.com" // pol)  # Jennifer Briney
+    (fromDb "craphound.com" // pol)  # Cory Doctorow -- both podcast & text entries
+    (fromDb "darknetdiaries.com" // tech)
+    (fromDb "feed.podbean.com/matrixlive/feed.xml" // tech)  # Matrix (chat) Live
+    (fromDb "feeds.99percentinvisible.org/99percentinvisible" // pol)  # 99% Invisible -- also available here: <https://feeds.simplecast.com/BqbsxVfO>
+    (fromDb "feeds.feedburner.com/80000HoursPodcast" // rat)
+    (fromDb "feeds.feedburner.com/dancarlin/history" // rat)
+    (fromDb "feeds.feedburner.com/radiolab" // pol)  # Radiolab -- also available here, but ONLY OVER HTTP: <http://feeds.wnyc.org/radiolab>
+    (fromDb "feeds.libsyn.com/421877" // rat)  # Less Wrong Curated
+    (fromDb "feeds.megaphone.fm/behindthebastards" // pol)  # also Maggie Killjoy
+    (fromDb "feeds.megaphone.fm/hubermanlab" // uncat)  # Daniel Huberman on sleep
+    (fromDb "feeds.megaphone.fm/recodedecode" // tech)  # The Verge - Decoder
+    (fromDb "feeds.simplecast.com/54nAGcIl" // pol)  # The Daily
+    (fromDb "feeds.simplecast.com/82FI35Px" // pol)  # Ezra Klein Show
+    (fromDb "feeds.simplecast.com/wgl4xEgL" // rat)  # Econ Talk
+    (fromDb "feeds.simplecast.com/xKJ93w_w" // uncat)  # Atlas Obscura
+    (fromDb "feeds.simplecast.com/l2i9YnTd" // tech // pol)  # Hard Fork (NYtimes tech)
+    (fromDb "feeds.transistor.fm/acquired" // tech)
+    (fromDb "lexfridman.com/podcast" // rat)
+    (fromDb "mapspodcast.libsyn.com" // uncat)  # Multidisciplinary Association for Psychedelic Studies
+    (fromDb "omegataupodcast.net" // tech)  # 3/4 German; 1/4 eps are English
+    (fromDb "omny.fm/shows/cool-people-who-did-cool-stuff" // pol)  # Maggie Killjoy -- referenced by Cory Doctorow
+    (fromDb "podcast.posttv.com/itunes/post-reports.xml" // pol)
     (fromDb "podcast.thelinuxexp.com" // tech)
-    ## Michael Malice - Your Welcome -- also available here: <https://origin.podcastone.com/podcast?categoryID2=2232>
-    (fromDb "rss.art19.com/your-welcome" // pol)
+    (fromDb "politicalorphanage.libsyn.com" // pol)
+    (fromDb "reverseengineering.libsyn.com/rss" // tech)  # UnNamed Reverse Engineering Podcast
+    (fromDb "rss.acast.com/deconstructed")  # The Intercept - Deconstructed
+    (fromDb "rss.acast.com/ft-tech-tonic" // tech)
+    (fromDb "rss.acast.com/intercepted-with-jeremy-scahill")  # The Intercept - Intercepted
+    (fromDb "rss.art19.com/60-minutes" // pol)
+    (fromDb "rss.art19.com/the-portal" // rat)  # Eric Weinstein
     (fromDb "seattlenice.buzzsprout.com" // pol)
-    ## Sci-Fi? has Peter Watts; author of No Moods, Ads or Cutesy Fucking Icons (rifters.com)
-    (fromDb "talesfromthebridge.buzzsprout.com" // tech)
-    ## UnNamed Reverse Engineering Podcast
-    (fromDb "reverseengineering.libsyn.com/rss" // tech)
-    ## The Witch Trials of J.K. Rowling
-    ## - <https://www.thefp.com/witchtrials>
-    (mkPod "https://feeds.megaphone.fm/RUNMED9919162779" // pol // infrequent)
-    ## Atlas Obscura
-    (fromDb "feeds.simplecast.com/xKJ93w_w" // uncat)
-    ## Ezra Klein Show
-    (fromDb "feeds.simplecast.com/82FI35Px" // pol)
-    ## Wireshark Podcast o_0
-    (fromDb "sharkbytes.transistor.fm" // tech)
-    ## 3/4 German; 1/4 eps are English
-    (fromDb "omegataupodcast.net" // tech)
-    ## Lateral with Tom Scott
-    (mkPod "https://audioboom.com/channels/5097784.rss" // tech)
+    (fromDb "sharkbytes.transistor.fm" // tech)  # Wireshark Podcast o_0
+    (fromDb "sscpodcast.libsyn.com" // rat)  # Astral Codex Ten
+    (fromDb "talesfromthebridge.buzzsprout.com" // tech)  # Sci-Fi? has Peter Watts; author of No Moods, Ads or Cutesy Fucking Icons (rifters.com)
+    (fromDb "techwontsave.us" // pol)  # rec by Cory Doctorow
+    # (fromDb "trashfuturepodcast.podbean.com" // pol)  # rec by Cory Doctorow, but way rambly
+    (fromDb "wakingup.libsyn.com" // pol)  # Sam Harris
+    (fromDb "werenotwrong.fireside.fm" // pol)
+
+    # (fromDb "rss.art19.com/your-welcome" // pol)  # Michael Malice - Your Welcome -- also available here: <https://origin.podcastone.com/podcast?categoryID2=2232>
+    # (fromDb "rss.prod.firstlook.media/deconstructed/podcast.rss" // pol)  #< possible URL rot
+    # (fromDb "rss.prod.firstlook.media/intercepted/podcast.rss" // pol)  #< possible URL rot
+    # (mkPod "https://anchor.fm/s/21bc734/podcast/rss" // pol // infrequent)  # Emerge: making sense of what's next -- <https://www.whatisemerging.com/emergepodcast>
+    # (mkPod "https://audioboom.com/channels/5097784.rss" // tech)  # Lateral with Tom Scott
+    # (mkPod "https://feeds.megaphone.fm/RUNMED9919162779" // pol // infrequent)  # The Witch Trials of J.K. Rowling: <https://www.thefp.com/witchtrials>
+    # (mkPod "https://podcasts.la.utexas.edu/this-is-democracy/feed/podcast/" // pol // weekly)
   ];
 
   texts = [
@@ -150,7 +123,7 @@ let
     (mkText "https://linuxphoneapps.org/blog/atom.xml" // tech // infrequent)
     (fromDb "tuxphones.com" // tech)
     (fromDb "spectrum.ieee.org" // tech)
-    (fromDb "theregister.com" // tech)
+    # (fromDb "theregister.com" // tech)
     (fromDb "thisweek.gnome.org" // tech)
     # more nixos stuff here, but unclear how to subscribe: <https://nixos.org/blog/categories.html>
     (mkText "https://nixos.org/blog/announcements-rss.xml" // tech // infrequent)
@@ -164,8 +137,12 @@ let
     # DEVELOPERS
     (fromDb "blog.jmp.chat" // tech)
     (fromDb "uninsane.org" // tech)
+    (fromDb "blog.thalheim.io" // tech)  # Mic92
     (fromDb "ascii.textfiles.com" // tech)  # Jason Scott
     (fromDb "xn--gckvb8fzb.com" // tech)
+    (fromDb "amosbbatto.wordpress.com" // tech)
+    (fromDb "fasterthanli.me" // tech)
+    (fromDb "jeffgeerling.com" // tech)
     (fromDb "mg.lol" // tech)
     # (fromDb "drewdevault.com" // tech)
     ## Ken Shirriff
@@ -176,6 +153,7 @@ let
     (fromDb "vitalik.ca" // tech)
     ## ian (Sanctuary)
     (fromDb "sagacioussuricata.com" // tech)
+    (fromDb "artemis.sh" // tech)
     ## Bunnie Juang
     (fromDb "bunniestudios.com" // tech)
     (fromDb "blog.danieljanus.pl" // tech)
@@ -186,6 +164,8 @@ let
     (mkText "https://anish.lakhwara.com/home.html" // tech // weekly)
     (fromDb "jefftk.com" // tech)
     (fromDb "pomeroyb.com" // tech)
+    (fromDb "harihareswara.net" // tech // pol)  # rec by Cory Doctorow
+    (fromDb "mako.cc/copyrighteous" // tech // pol)  # rec by Cory Doctorow
     # (mkText "https://til.simonwillison.net/tils/feed.atom" // tech // weekly)
 
     # TECH PROJECTS
@@ -251,6 +231,7 @@ let
   images = [
     (fromDb "smbc-comics.com" // img // humor)
     (fromDb "xkcd.com" // img // humor)
+    (fromDb "turnoff.us" // img // humor)
     (fromDb "pbfcomics.com" // img // humor)
     # (mkImg "http://dilbert.com/feed" // humor // daily)
     (fromDb "poorlydrawnlines.com/feed" // img // humor)

hosts/common/hardware.nix

@@ -1,44 +0,0 @@
-{ lib, pkgs, ... }:
-
-{
-  boot.initrd.supportedFilesystems = [ "ext4" "btrfs" "ext2" "ext3" "vfat" ];
-  # useful emergency utils
-  boot.initrd.extraUtilsCommands = ''
-    copy_bin_and_libs ${pkgs.btrfs-progs}/bin/btrfstune
-  '';
-  boot.kernelParams = [ "boot.shell_on_fail" ];
-  # other kernelParams:
-  #   "boot.trace"
-  #   "systemd.log_level=debug"
-  #   "systemd.log_target=console"
-
-  # hack in the `boot.shell_on_fail` arg since that doesn't always seem to work.
-  boot.initrd.preFailCommands = "allowShell=1";
-
-  # default: 4 (warn). 7 is debug
-  boot.consoleLogLevel = 7;
-
-  boot.loader.grub.enable = lib.mkDefault false;
-  boot.loader.generic-extlinux-compatible.enable = lib.mkDefault true;
-
-  # non-free firmware
-  hardware.enableRedistributableFirmware = true;
-
-  # powertop will default to putting USB devices -- including HID -- to sleep after TWO SECONDS
-  powerManagement.powertop.enable = false;
-
-  services.logind.extraConfig = ''
-    # don’t shutdown when power button is short-pressed
-    HandlePowerKey=ignore
-  '';
-
-  # services.snapper.configs = {
-  #   root = {
-  #     subvolume = "/";
-  #     extraConfig = {
-  #       ALLOW_USERS = "colin";
-  #     };
-  #   };
-  # };
-  # services.snapper.snapshotInterval = "daily";
-}

hosts/common/hardware/default.nix

@@ -0,0 +1,74 @@
+{ lib, pkgs, ... }:
+
+{
+  imports = [
+    ./x86_64.nix
+  ];
+
+  boot.initrd.supportedFilesystems = [ "ext4" "btrfs" "ext2" "ext3" "vfat" ];
+  # useful emergency utils
+  boot.initrd.extraUtilsCommands = ''
+    copy_bin_and_libs ${pkgs.btrfs-progs}/bin/btrfstune
+    copy_bin_and_libs ${pkgs.util-linux}/bin/{cfdisk,lsblk,lscpu}
+    copy_bin_and_libs ${pkgs.gptfdisk}/bin/{cgdisk,gdisk}
+    copy_bin_and_libs ${pkgs.smartmontools}/bin/smartctl
+    copy_bin_and_libs ${pkgs.e2fsprogs}/bin/resize2fs
+  '' + lib.optionalString pkgs.stdenv.hostPlatform.isx86_64 ''
+    copy_bin_and_libs ${pkgs.nvme-cli}/bin/nvme  # doesn't cross compile
+  '';
+  boot.kernelParams = [
+    "boot.shell_on_fail"
+    #v experimental full pre-emption for hopefully better call/audio latency on moby.
+    # also toggleable at runtime via /sys/kernel/debug/sched/preempt
+    # defaults to preempt=voluntary
+    # "preempt=full"
+  ];
+  # other kernelParams:
+  #   "boot.trace"
+  #   "systemd.log_level=debug"
+  #   "systemd.log_target=console"
+
+  # hack in the `boot.shell_on_fail` arg since that doesn't always seem to work.
+  boot.initrd.preFailCommands = "allowShell=1";
+
+  # default: 4 (warn). 7 is debug
+  boot.consoleLogLevel = 7;
+
+  boot.loader.grub.enable = lib.mkDefault false;
+  boot.loader.generic-extlinux-compatible.enable = lib.mkDefault true;
+
+  # non-free firmware
+  hardware.enableRedistributableFirmware = true;
+
+  # powertop will default to putting USB devices -- including HID -- to sleep after TWO SECONDS
+  powerManagement.powertop.enable = false;
+  # linux CPU governor: <https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt>
+  # - options:
+  #   - "powersave" => force CPU to always run at lowest supported frequency
+  #   - "performance" => force CPU to always run at highest frequency
+  #   - "ondemand" => adjust frequency based on load
+  #   - "conservative"  (ondemand but slower to adjust)
+  #   - "schedutil"
+  #   - "userspace"
+  # - not all options are available for all platforms
+  #   - intel (intel_pstate) appears to manage scaling w/o intervention/control from the OS.
+  #   - AMD (acpi-cpufreq) appears to manage scaling via the OS *or* HW. but the ondemand defaults never put it to max hardware frequency.
+  #   - qualcomm (cpufreq-dt) appears to manage scaling *only* via the OS. ondemand governor exercises the full range.
+  # - query details with `sudo cpupower frequency-info`
+  powerManagement.cpuFreqGovernor = "ondemand";
+
+  services.logind.extraConfig = ''
+    # don’t shutdown when power button is short-pressed
+    HandlePowerKey=ignore
+  '';
+
+  # services.snapper.configs = {
+  #   root = {
+  #     subvolume = "/";
+  #     extraConfig = {
+  #       ALLOW_USERS = "colin";
+  #     };
+  #   };
+  # };
+  # services.snapper.snapshotInterval = "daily";
+}

hosts/common/hardware/x86_64.nix

@@ -9,12 +9,7 @@
       # efi_pstore evivars
     ];
 
-    powerManagement.cpuFreqGovernor = "powersave";
     hardware.cpu.amd.updateMicrocode = true;    # desktop
     hardware.cpu.intel.updateMicrocode = true;  # laptop
-
-    hardware.opengl.driSupport = true;
-    # For 32 bit applications
-    hardware.opengl.driSupport32Bit = true;
   };
 }

hosts/common/home/keyring/default.nix

@@ -7,7 +7,7 @@ let
   };
 in
 {
-  sane.user.persist.private = [ ".local/share/keyrings" ];
+  sane.user.persist.byStore.private = [ ".local/share/keyrings" ];
 
   sane.user.fs."private/.local/share/keyrings/default" = {
     generated.command = [ "${init-keyring}/bin/init-keyring" ];

hosts/common/home/ssh.nix

@@ -13,7 +13,7 @@ let
 in
 {
   # ssh key is stored in private storage
-  sane.user.persist.private = [
+  sane.user.persist.byStore.private = [
     { type = "file"; path = ".ssh/id_ed25519"; }
   ];
   sane.user.fs.".ssh/id_ed25519.pub" = lib.mkIf (user-pubkey != null) {

hosts/common/hostnames.nix

@@ -0,0 +1,16 @@
+# TODO: move to hosts/common/
+{ config, lib, ... }:
+
+{
+  # give each host a shortname that all the other hosts know, to allow easy comms.
+  networking.hosts = lib.mkMerge (builtins.map
+    (host: let
+      cfg = config.sane.hosts.by-name."${host}";
+    in {
+      "${cfg.lan-ip}" = [ host ];
+    } // lib.optionalAttrs (cfg.wg-home.ip != null) {
+      "${cfg.wg-home.ip}" = [ "${host}-hn" ];
+    })
+    (builtins.attrNames config.sane.hosts.by-name)
+  );
+}

hosts/common/hosts.nix

@@ -0,0 +1,45 @@
+{ lib, ... }:
+
+{
+  # TODO: this should be populated per-host
+  sane.hosts.by-name."desko" = {
+    ssh.user_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPU5GlsSfbaarMvDA20bxpSZGWviEzXGD8gtrIowc1pX";
+    ssh.host_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFw9NoRaYrM6LbDd3aFBc4yyBlxGQn8HjeHd/dZ3CfHk";
+    wg-home.pubkey = "17PMZssYi0D4t2d0vbmhjBKe1sGsE8kT8/dod0Q2CXc=";
+    wg-home.ip = "10.0.10.22";
+    lan-ip = "10.78.79.52";
+  };
+
+  sane.hosts.by-name."lappy" = {
+    ssh.user_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDpmFdNSVPRol5hkbbCivRhyeENzb9HVyf9KutGLP2Zu";
+    ssh.host_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILSJnqmVl9/SYQ0btvGb0REwwWY8wkdkGXQZfn/1geEc";
+    wg-home.pubkey = "FTUWGw2p4/cEcrrIE86PWVnqctbv8OYpw8Gt3+dC/lk=";
+    wg-home.ip = "10.0.10.20";
+    lan-ip = "10.78.79.53";
+  };
+
+  sane.hosts.by-name."moby" = {
+    ssh.authorized = lib.mkDefault false;  # moby's too easy to hijack: don't let it ssh places
+    ssh.user_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICrR+gePnl0nV/vy7I5BzrGeyVL+9eOuXHU1yNE3uCwU";
+    ssh.host_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO1N/IT3nQYUD+dBlU1sTEEVMxfOyMkrrDeyHcYgnJvw";
+    wg-home.pubkey = "I7XIR1hm8bIzAtcAvbhWOwIAabGkuEvbWH/3kyIB1yA=";
+    wg-home.ip = "10.0.10.48";
+    lan-ip = "10.78.79.54";
+  };
+
+  sane.hosts.by-name."servo" = {
+    ssh.authorized = lib.mkDefault false;  # servo presents too many services to the internet: easy atack vector
+    ssh.user_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPS1qFzKurAdB9blkWomq8gI1g0T3sTs9LsmFOj5VtqX";
+    ssh.host_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfdSmFkrVT6DhpgvFeQKm3Fh9VKZ9DbLYOPOJWYQ0E8";
+    wg-home.pubkey = "roAw+IUFVtdpCcqa4khB385Qcv9l5JAB//730tyK4Wk=";
+    wg-home.ip = "10.0.10.5";
+    wg-home.endpoint = "uninsane.org:51820";
+    lan-ip = "10.78.79.51";
+  };
+
+  sane.hosts.by-name."supercap" = {
+    ssh.authorized = false;
+    ssh.host_pubkey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHf/mqqkX45EWAcquV04MC3SUljTApdclH1gjI19F+PA";
+    lan-ip = "10.78.79.232";
+  };
+}

hosts/common/ids.nix

@@ -46,6 +46,13 @@
   sane.ids.trust-dns.gid = 2411;
   sane.ids.export.gid = 2412;
   sane.ids.nfsuser.uid = 2413;
+  sane.ids.media.gid = 2414;
+  sane.ids.ntfy-sh.uid = 2415;
+  sane.ids.ntfy-sh.gid = 2415;
+  sane.ids.monero.uid = 2416;
+  sane.ids.monero.gid = 2416;
+  sane.ids.slskd.uid = 2417;
+  sane.ids.slskd.gid = 2417;
 
   sane.ids.colin.uid = 1000;
   sane.ids.guest.uid = 1100;
@@ -60,6 +67,8 @@
   sane.ids.systemd-oom.uid = 2005;
   sane.ids.systemd-oom.gid = 2005;
   sane.ids.wireshark.gid = 2006;
+  sane.ids.nixremote.uid = 2007;
+  sane.ids.nixremote.gid = 2007;
 
   # found on graphical hosts
   sane.ids.nm-iodine.uid = 2101;  # desko/moby/lappy
@@ -83,4 +92,8 @@
   sane.ids.rtkit.gid = 2307;
   # phosh
   sane.ids.feedbackd.gid = 2308;
+
+  # new moby users
+  sane.ids.eg25-control.uid = 2309;
+  sane.ids.eg25-control.gid = 2309;
 }

hosts/common/nix-path/default.nix

@@ -1,4 +1,4 @@
-{ pkgs, ... }:
+{ pkgs, sane-lib, ... }:
 
 {
   # allow `nix-shell` (and probably nix-index?) to locate our patched and custom packages
@@ -10,4 +10,7 @@
     # to avoid switching so much during development
     "nixpkgs-overlays=/home/colin/dev/nixos/hosts/common/nix-path/overlay"
   ];
+
+  # ensure new deployments have a source of this repo with which they can bootstrap.
+  environment.etc."nixos".source = ../../..;
 }

hosts/common/persist.nix

@@ -5,12 +5,12 @@
   # store /home/colin/a/b in /home/private/a/b instead of /home/private/home/colin/a/b
   sane.persist.stores.private.prefix = "/home/colin";
 
-  sane.persist.sys.plaintext = [
+  sane.persist.sys.byStore.plaintext = [
     # TODO: these should be private.. somehow
     "/var/log"
     "/var/backup"  # for e.g. postgres dumps
   ];
-  sane.persist.sys.cryptClearOnBoot = [
+  sane.persist.sys.byStore.cryptClearOnBoot = [
     "/var/lib/systemd/coredump"
   ];
 }

hosts/common/programs/abaddon.nix

@@ -0,0 +1,107 @@
+# discord gtk3 client
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.sane.programs.abaddon;
+in
+{
+  sane.programs.abaddon = {
+    configOption = with lib; mkOption {
+      default = {};
+      type = types.submodule {
+        options.autostart = mkOption {
+          type = types.bool;
+          default = true;
+        };
+      };
+    };
+
+    package = pkgs.abaddon.overrideAttrs (upstream: {
+      patches = (upstream.patches or []) ++ [
+        (pkgs.fetchpatch {
+          url = "https://git.uninsane.org/colin/abaddon/commit/eb551f188d34679f75adcbc83cb8d5beb4d19fd6.patch";
+          name = ''"view members" default to false'';
+          hash = "sha256-9BX8iO86CU1lNrKS1G2BjDR+3IlV9bmhRNTsLrxChwQ=";
+        })
+        (pkgs.fetchpatch {
+          # this makes it so Abaddon reports its app_name in notifications.
+          # not 100% necessary; just a nice-to-have. maybe don't rely on it until it's merged upstream.
+          # upstream PR: <https://github.com/uowuo/abaddon/pull/247>
+          url = "https://git.uninsane.org/colin/abaddon/commit/18cd863fdbb5e6b1e9aaf9394dbd673d51839f30.patch";
+          name = "set glib application name";
+          hash = "sha256-IFYxf1D8hIsxgZehGd6hL3zJiBkPZfWGm+Faaa5ZFl4=";
+        })
+      ];
+    });
+
+    suggestedPrograms = [ "gnome-keyring" ];
+
+    fs.".config/abaddon/abaddon.ini".symlink.text = ''
+      # see abaddon README.md for options.
+      # at time of writing:
+      # | Setting       | Type    | Default | Description                                                                                      |
+      # |[discord]------|---------|---------|--------------------------------------------------------------------------------------------------|
+      # | `gateway`     | string  |         | override url for Discord gateway. must be json format and use zlib stream compression            |
+      # | `api_base`    | string  |         | override base url for Discord API                                                                |
+      # | `memory_db`   | boolean | false   | if true, Discord data will be kept in memory as opposed to on disk                               |
+      # | `token`       | string  |         | Discord token used to login, this can be set from the menu                                       |
+      # | `prefetch`    | boolean | false   | if true, new messages will cause the avatar and image attachments to be automatically downloaded |
+      # | `autoconnect` | boolean | false   | autoconnect to discord                                                                           |
+      # |[http]--------|--------|---------|---------------------------------------------------------------------------------------------|
+      # | `user_agent` | string |         | sets the user-agent to use in HTTP requests to the Discord API (not including media/images) |
+      # | `concurrent` | int    | 20      | how many images can be concurrently retrieved                                               |
+      # |[gui}------------------------|---------|---------|----------------------------------------------------------------------------------------------------------------------------|
+      # | `member_list_discriminator` | boolean | true    | show user discriminators in the member list                                                                                |
+      # | `stock_emojis`              | boolean | true    | allow abaddon to substitute unicode emojis with images from emojis.bin, must be false to allow GTK to render emojis itself |
+      # | `custom_emojis`             | boolean | true    | download and use custom Discord emojis                                                                                     |
+      # | `css`                       | string  |         | path to the main CSS file                                                                                                  |
+      # | `animations`                | boolean | true    | use animated images where available (e.g. server icons, emojis, avatars). false means static images will be used           |
+      # | `animated_guild_hover_only` | boolean | true    | only animate guild icons when the guild is being hovered over                                                              |
+      # | `owner_crown`               | boolean | true    | show a crown next to the owner                                                                                             |
+      # | `unreads`                   | boolean | true    | show unread indicators and mention badges                                                                                  |
+      # | `save_state`                | boolean | true    | save the state of the gui (active channels, tabs, expanded channels)                                                       |
+      # | `alt_menu`                  | boolean | false   | keep the menu hidden unless revealed with alt key                                                                          |
+      # | `hide_to_tray`              | boolean | false   | hide abaddon to the system tray on window close                                                                            |
+      # | `show_deleted_indicator`    | boolean | true    | show \[deleted\] indicator next to deleted messages instead of actually deleting the message                               |
+      # | `font_scale`                | double  |         | scale font rendering. 1 is unchanged                                                                                       |
+      # |[style]------------------|--------|-----------------------------------------------------|
+      # | `linkcolor`             | string | color to use for links in messages                  |
+      # | `expandercolor`         | string | color to use for the expander in the channel list   |
+      # | `nsfwchannelcolor`      | string | color to use for NSFW channels in the channel list  |
+      # | `channelcolor`          | string | color to use for SFW channels in the channel list   |
+      # | `mentionbadgecolor`     | string | background color for mention badges                 |
+      # | `mentionbadgetextcolor` | string | color to use for number displayed on mention badges |
+      # | `unreadcolor`           | string | color to use for the unread indicator               |
+      # |[notifications]|---------|--------------------------|-------------------------------------------------------------------------------|
+      # | `enabled`     | boolean | true (if not on Windows) | Enable desktop notifications                                                  |
+      # | `playsound`   | boolean | true                     | Enable notification sounds. Requires ENABLE_NOTIFICATION_SOUNDS=TRUE in CMake |
+      # |[voice]--|--------|------------------------------------|------------------------------------------------------------|
+      # | `vad`   | string | rnnoise if enabled, gate otherwise | Method used for voice activity detection. Changeable in UI |
+      # |[windows]|---------|---------|-------------------------|
+      # | `hideconsole` | boolean | true    | Hide console on startup |
+
+      # N.B.: abaddon writes this file itself (and even when i don't change anything internally).
+      # it prefers no spaces around the equal sign.
+      [discord]
+      autoconnect=true
+
+      [notifications]
+      # playsound: i manage sounds via swaync
+      playsound=false
+    '';
+
+    persist.byStore.private = [
+      ".cache/abaddon"
+    ];
+
+    services.abaddon = {
+      description = "unofficial Discord chat client";
+      wantedBy = lib.mkIf cfg.config.autostart [ "default.target" ];
+      serviceConfig = {
+        ExecStart = "${cfg.package}/bin/abaddon";
+        Type = "simple";
+        Restart = "always";
+        RestartSec = "20s";
+      };
+    };
+  };
+}

hosts/common/programs/animatch.nix

@@ -0,0 +1,10 @@
+{ ... }:
+{
+  sane.programs.animatch = {
+    persist.byStore.plaintext = [
+      # game progress
+      ".config/Holy Pangolin/Animatch"
+      ".local/share/Holy Pangolin/Animatch"  # i think this one might be wrong
+    ];
+  };
+}

hosts/common/programs/assorted.nix

@@ -1,4 +1,4 @@
-{ pkgs, ... }:
+{ config, lib, pkgs, ... }:
 
 let
   declPackageSet = pkgs: {
@@ -20,6 +20,7 @@ in
       "sane-scripts.bt-show"
     ];
     "sane-scripts.dev" = declPackageSet [
+      "sane-scripts.clone"
       "sane-scripts.dev-cargo-loop"
       "sane-scripts.git-init"
     ];
@@ -41,13 +42,13 @@ in
       "sane-scripts.secrets-unlock"
       "sane-scripts.secrets-update-keys"
       "sane-scripts.shutdown"
-      "sane-scripts.ssl-dump"
       "sane-scripts.sudo-redirect"
       "sane-scripts.sync-from-servo"
-      "sane-scripts.vpn-down"
-      "sane-scripts.vpn-up"
+      "sane-scripts.vpn"
       "sane-scripts.which"
       "sane-scripts.wipe-browser"
+      "sane-scripts.wipe-flare"
+      "sane-scripts.wipe-fractal"
     ];
     "sane-scripts.sys-utils" = declPackageSet [
       "sane-scripts.ip-port-forward"
@@ -59,21 +60,27 @@ in
       "btrfs-progs"
       "cacert.unbundled"  # some services require unbundled /etc/ssl/certs
       "cryptsetup"
+      "ddrescue"
       "dig"
+      "dtc"  # device tree [de]compiler
+      "e2fsprogs"  # resize2fs
       "efibootmgr"
+      "ethtool"
       "fatresize"
       "fd"
       "file"
       # "fwupd"
       "gawk"
+      "gdb"  # to debug segfaults
       "git"
-      "gptfdisk"
+      "gptfdisk"  # gdisk
       "hdparm"
       "htop"
       "iftop"
       "inetutils"  # for telnet
       "iotop"
       "iptables"
+      "iw"
       "jq"
       "killall"
       "lsof"
@@ -84,6 +91,7 @@ in
       "netcat"
       "nethogs"
       "nmap"
+      "nvme-cli"  # nvme
       "openssl"
       "parted"
       "pciutils"
@@ -91,15 +99,17 @@ in
       "pstree"
       "ripgrep"
       "screen"
-      "smartmontools"
+      "smartmontools"  # smartctl
       "socat"
       "strace"
       "subversion"
       "tcpdump"
       "tree"
       "usbutils"
+      "util-linux"  # lsblk, lscpu, etc
       "wget"
       "wirelesstools"  # iwlist
+      "xq"  # jq for XML
     ];
     sysadminExtraUtils = declPackageSet [
       "backblaze-b2"
@@ -115,11 +125,13 @@ in
     #   - debugging?
     consoleUtils = declPackageSet [
       "alsaUtils"  # for aplay, speaker-test
+      "binutils-unwrapped"  # for strings; though this brings 80MB of unrelated baggage too
       # "cdrtools"
       "clinfo"
       "dmidecode"
       "dtrx"  # `unar` alternative, "Do The Right eXtraction"
       "efivar"
+      "eza"  # a better 'ls'
       # "flashrom"
       "git"  # needed as a user package, for config.
       # "gnupg"
@@ -127,11 +139,11 @@ in
       # "gopass"
       # "gopass-jsonapi"
       "helix"  # text editor
-      # "kitty"  # XXX needs to be in consolueUtils because `ssh servo` from kitty sets `TERM=xterm-kitty` in the remote and breaks things
       "libsecret"  # for managing user keyrings. TODO: what needs this? lift into the consumer
       "lm_sensors"  # for sensors-detect. TODO: what needs this? lift into the consumer
       "lshw"
       # "memtester"
+      "mercurial"  # hg
       "neovim"  # needed as a user package, for swap persistence
       # "nettools"
       # "networkmanager"
@@ -141,7 +153,7 @@ in
       # "oathToolkit"  # for oathtool
       # "ponymix"
       "pulsemixer"
-      "python3"
+      "python3-repl"
       # "python3Packages.eyeD3"  # music tagging
       "ripgrep"  # needed as a user package so that its user-level config file can be installed
       "rsync"
@@ -154,14 +166,14 @@ in
       "sudo"
       # "tageditor"  # music tagging
       # "unar"
+      "unzip"
       "wireguard-tools"
-      "xdg-terminal-exec"
       "xdg-utils"  # for xdg-open
       # "yarn"
       "zsh"
     ];
 
-    desktopConsoleUtils = declPackageSet [
+    pcConsoleUtils = declPackageSet [
       "gh"  # MS GitHub cli
       "nix-index"
       "nixpkgs-review"
@@ -176,11 +188,11 @@ in
       "yt-dlp"
     ];
 
-    tuiApps = declPackageSet [
+    pcTuiApps = declPackageSet [
       "aerc"  # email client
       "msmtp"  # sendmail
       "offlineimap"  # email mailbox sync
-      "sfeed"  # RSS fetcher
+      # "sfeed"  # RSS fetcher
       "visidata"  # TUI spreadsheet viewer/editor
       "w3m"  # web browser
     ];
@@ -193,61 +205,83 @@ in
     ];
 
     devPkgs = declPackageSet [
+      "cargo"
       "clang"
+      "lua"
       "nodejs"
+      "patchelf"
+      "rustc"
       "tree-sitter"
     ];
 
 
     # INDIVIDUAL PACKAGE DEFINITIONS
 
+    cargo.persist.byStore.plaintext = [ ".cargo" ];
+
     # creds, but also 200 MB of node modules, etc
-    discord.persist.private = [ ".config/discord" ];
+    discord.persist.byStore.private = [ ".config/discord" ];
+
+    endless-sky.persist.byStore.plaintext = [ ".local/share/endless-sky" ];
 
     # `emote` will show a first-run dialog based on what's in this directory.
     # mostly, it just keeps a LRU of previously-used emotes to optimize display order.
     # TODO: package [smile](https://github.com/mijorus/smile) for probably a better mobile experience.
-    emote.persist.plaintext = [ ".local/share/Emote" ];
+    emote.persist.byStore.plaintext = [ ".local/share/Emote" ];
 
-    fluffychat-moby.persist.plaintext = [ ".local/share/chat.fluffy.fluffychat" ];
+    fluffychat-moby.persist.byStore.plaintext = [ ".local/share/chat.fluffy.fluffychat" ];
+
+    font-manager.package = pkgs.font-manager.override {
+      # build without the "Google Fonts" integration feature, to save closure / avoid webkitgtk_4_0
+      withWebkit = false;
+    };
 
     # MS GitHub stores auth token in .config
     # TODO: we can populate gh's stuff statically; it even lets us use the same oauth across machines
-    gh.persist.private = [ ".config/gh" ];
+    gh.persist.byStore.private = [ ".config/gh" ];
+
+    gnome-2048.persist.byStore.plaintext = [ ".local/share/gnome-2048/scores" ];
+
+    "gnome.gnome-maps".persist.byStore.plaintext = [ ".cache/shumate" ];
+    "gnome.gnome-maps".persist.byStore.private = [ ".local/share/maps-places.json" ];
 
     # actual monero blockchain (not wallet/etc; safe to delete, just slow to regenerate)
     # XXX: is it really safe to persist this? it doesn't have info that could de-anonymize if captured?
-    monero-gui.persist.plaintext = [ ".bitmonero" ];
+    monero-gui.persist.byStore.plaintext = [ ".bitmonero" ];
 
-    mumble.persist.private = [ ".local/share/Mumble" ];
+    mumble.persist.byStore.private = [ ".local/share/Mumble" ];
 
     # settings (electron app)
-    obsidian.persist.plaintext = [ ".config/obsidian" ];
+    obsidian.persist.byStore.plaintext = [ ".config/obsidian" ];
 
-    # creds, media
-    signal-desktop.persist.private = [ ".config/Signal" ];
+    python3-repl.package = pkgs.python3.withPackages (ps: with ps; [
+      requests
+    ]);
+
+    shattered-pixel-dungeon.persist.byStore.plaintext = [ ".local/share/.shatteredpixel/shattered-pixel-dungeon" ];
 
     # printer/filament settings
-    slic3r.persist.plaintext = [ ".Slic3r" ];
+    slic3r.persist.byStore.plaintext = [ ".Slic3r" ];
 
-    # creds, widevine .so download. TODO: could easily manage these statically.
-    spotify.persist.plaintext = [ ".config/spotify" ];
+    space-cadet-pinball.persist.byStore.plaintext = [ ".local/share/SpaceCadetPinball" ];
 
-    tdesktop.persist.private = [ ".local/share/TelegramDesktop" ];
+    superTux.persist.byStore.plaintext = [ ".local/share/supertux2" ];
 
-    tokodon.persist.private = [ ".cache/KDE/tokodon" ];
+    tdesktop.persist.byStore.private = [ ".local/share/TelegramDesktop" ];
 
-    # hardenedMalloc solves an "unable to connect to Tor" error when pressing the "connect" button
-    # - still required as of 2023/07/14
-    tor-browser-bundle-bin.package = pkgs.tor-browser-bundle-bin.override {
-      useHardenedMalloc = false;
-    };
+    tokodon.persist.byStore.private = [ ".cache/KDE/tokodon" ];
 
-    whalebird.persist.private = [ ".config/Whalebird" ];
+    vvvvvv.persist.byStore.plaintext = [ ".local/share/VVVVVV" ];
 
-    yarn.persist.plaintext = [ ".cache/yarn" ];
+    whalebird.persist.byStore.private = [ ".config/Whalebird" ];
+
+    yarn.persist.byStore.plaintext = [ ".cache/yarn" ];
 
     # zcash coins. safe to delete, just slow to regenerate (10-60 minutes)
-    zecwallet-lite.persist.private = [ ".zcash" ];
+    zecwallet-lite.persist.byStore.private = [ ".zcash" ];
+  };
+
+  programs.feedbackd = lib.mkIf config.sane.programs.feedbackd.enabled {
+    enable = true;
   };
 }

hosts/common/programs/bemenu.nix

@@ -0,0 +1,103 @@
+{ lib, pkgs, ... }:
+# notable bemenu options:
+#   - see `bemenu --help` for all
+#   -P, --prefix          text to show before highlighted item.
+#   --scrollbar           display scrollbar. (none (default), always, autohide)
+#   -H, --line-height     defines the height to make each menu line (0 = default height). (wx)
+#   -M, --margin          defines the empty space on either side of the menu. (wx)
+#   -W, --width-factor    defines the relative width factor of the menu (from 0 to 1). (wx)
+#   -B, --border          defines the width of the border in pixels around the menu. (wx)
+#   -R  --border-radius   defines the radius of the border around the menu (0 = no curved borders).
+#   --ch                  defines the height of the cursor (0 = scales with line height). (wx)
+#   --cw                  defines the width of the cursor. (wx)
+#   --hp                  defines the horizontal padding for the entries in single line mode. (wx)
+#   --fn                  defines the font to be used ('name [size]'). (wx)
+#   --tb                  defines the title background color. (wx)
+#   --tf                  defines the title foreground color. (wx)
+#   --fb                  defines the filter background color. (wx)
+#   --ff                  defines the filter foreground color. (wx)
+#   --nb                  defines the normal background color. (wx)
+#   --nf                  defines the normal foreground color. (wx)
+#   --hb                  defines the highlighted background color. (wx)
+#   --hf                  defines the highlighted foreground color. (wx)
+#   --fbb                 defines the feedback background color. (wx)
+#   --fbf                 defines the feedback foreground color. (wx)
+#   --sb                  defines the selected background color. (wx)
+#   --sf                  defines the selected foreground color. (wx)
+#   --ab                  defines the alternating background color. (wx)
+#   --af                  defines the alternating foreground color. (wx)
+#   --scb                 defines the scrollbar background color. (wx)
+#   --scf                 defines the scrollbar foreground color. (wx)
+#   --bdr                 defines the border color. (wx)
+#
+# colors are specified as `#RRGGBB`
+# defaults:
+# --ab "#222222"
+# --af "#bbbbbb"
+# --bdr "#005577"
+# --border 3
+# --cb "#222222"
+# --center
+# --cf "#bbbbbb"
+# --fb "#222222"
+# --fbb "#eeeeee"
+# --fbf "#222222"
+# --ff "#bbbbbb"
+# --fixed-height
+# --fn 'Sxmo 14'
+# --hb "#005577"
+# --hf "#eeeeee"
+# --line-height 20
+# --list 16
+# --margin 40
+# --nb "#222222"
+# --nf "#bbbbbb"
+# --no-overlap
+# --no-spacing
+# --sb "#323232"
+# --scb "#005577"
+# --scf "#eeeeee"
+# --scrollbar autohide
+# --tb "#005577"
+# --tf "#eeeeee"
+# --wrap
+let
+  bg = "#1d1721";       # slight purple
+  fg0 = "#d8d8d8";      # inactive text (light grey)
+  fg1 = "#ffffff";      # active text   (white)
+  accent0 = "#1f5e54";  # darker but saturated teal
+  accent1 = "#418379";  # teal (matches nixos-bg)
+  accent2 = "#5b938a";  # brighter but muted teal
+  bemenuArgs = [
+    "--wrap --scrollbar autohide --fixed-height"
+    "--center --margin 45"
+    "--no-spacing"
+    # XXX: font size doesn't seem to take effect (would prefer larger)
+    "--fn 'monospace 14' --line-height 22 --border 3"
+    "--bdr '${accent0}'"                     # border
+    "--scf '${accent2}' --scb '${accent0}'"  # scrollbar
+    "--tb '${accent0}' --tf '${fg0}'"        # title
+    "--fb '${accent0}' --ff '${fg1}'"        # filter (i.e. text that's been entered)
+    "--hb '${accent1}' --hf '${fg1}'"        # selected item
+    "--nb '${bg}' --nf '${fg0}'"             # normal lines (even)
+    "--ab '${bg}' --af '${fg0}'"             # alternated lines (odd)
+    "--cf '${accent0}' --cb '${accent0}'"    # cursor (not very useful)
+  ];
+  bemenuOpts = lib.concatStringsSep " " bemenuArgs;
+in
+{
+  sane.programs.bemenu = {
+    package = pkgs.bemenu.overrideAttrs (upstream: {
+      nativeBuildInputs = (upstream.nativeBuildInputs or []) ++ [
+        pkgs.makeWrapper
+      ];
+      # can alternatively be specified as CLI flags
+      postInstall = (upstream.postInstall or "") + ''
+        wrapProgram $out/bin/bemenu \
+          --set BEMENU_OPTS "${bemenuOpts}"
+        wrapProgram $out/bin/bemenu-run \
+          --set BEMENU_OPTS "${bemenuOpts}"
+      '';
+    });
+  };
+}

hosts/common/programs/brave.nix

@@ -0,0 +1,9 @@
+{ ... }:
+{
+  sane.programs.brave = {
+    persist.byStore.cryptClearOnBoot = [
+      ".cache/BraveSoftware"
+      ".config/BraveSoftware"
+    ];
+  };
+}

hosts/common/programs/calls.nix

@@ -0,0 +1,61 @@
+# GNOME calls
+# - <https://gitlab.gnome.org/GNOME/calls>
+# - both a dialer and a call handler.
+# - uses callaudiod dbus package.
+#
+# initial JMP.chat configuration:
+# - message @cheogram.com "reset sip account"  (this is not destructive, despite the name)
+# - the bot will reply with auto-generated username/password plus a SIP server endpoint.
+#   just copy those into gnome-calls' GUI configurator
+# - now gnome-calls can do outbound calls. inbound calls requires more chatting with the help bot
+#
+# my setup here is still very WIP.
+# open questions:
+# - can i receive calls even with GUI closed?
+#   - e.g. activated by callaudiod?
+#   - looks like `gnome-calls --daemon` does that?
+{ config, lib, ... }:
+let
+  cfg = config.sane.programs.calls;
+in
+{
+  sane.programs.calls = {
+    configOption = with lib; mkOption {
+      default = {};
+      type = types.submodule {
+        options.autostart = mkOption {
+          type = types.bool;
+          default = false;
+        };
+      };
+    };
+
+    persist.byStore.private = [
+      # ".cache/folks"      # contact avatars?
+      # ".config/calls"
+      ".local/share/calls"  # call "records"
+      # .local/share/folks  # contacts?
+    ];
+    secrets.".config/calls/sip-account.cfg" = ../../../secrets/common/gnome_calls_sip-account.cfg.bin;
+    suggestedPrograms = [
+      "feedbackd"  # needs `phone-incoming-call`, in particular
+    ];
+
+    services.gnome-calls = {
+      # TODO: prevent gnome-calls from daemonizing when started manually
+      description = "gnome-calls daemon to monitor incoming SIP calls";
+      wantedBy = lib.mkIf cfg.config.autostart [ "default.target" ];
+      serviceConfig = {
+        # add --verbose for more debugging
+        ExecStart = "${cfg.package}/bin/gnome-calls --daemon";
+        Type = "simple";
+        Restart = "always";
+        RestartSec = "10s";
+      };
+      environment.G_MESSAGES_DEBUG = "all";
+    };
+  };
+  programs.calls = lib.mkIf cfg.enabled {
+    enable = true;
+  };
+}

hosts/common/programs/cantata.nix

@@ -4,7 +4,7 @@
 { ... }:
 {
   sane.programs.cantata = {
-    persist.plaintext = [
+    persist.byStore.plaintext = [
       ".cache/cantata"  # album art
       ".local/share/cantata/library"  # library index (?)
     ];

hosts/common/programs/chatty.nix

@@ -24,7 +24,7 @@ let
         # frees us from webkit_4_1, in turn.
         enableBackend = false;
         gvfs = pkgs.gvfs.override {
-          # saves 20 minutes of build time, for unused feature
+          # saves 20 minutes of build time and cross issues, for unused feature
           samba = null;
         };
       };
@@ -36,7 +36,7 @@ in
     # package = chattyNoOauth;
     package = chatty-latest;
     suggestedPrograms = [ "gnome-keyring" ];
-    persist.private = [
+    persist.byStore.private = [
       ".local/share/chatty"  # matrix avatars and files
       # not just XMPP; without this Chatty will regenerate its device-id every boot.
       # .purple/ contains XMPP *and* Matrix auth, logs, avatar cache, and a bit more

hosts/common/programs/conky/battery_estimate

@@ -1,14 +1,34 @@
 #!/usr/bin/env nix-shell
 #!nix-shell -i bash
 
-full=$(cat /sys/class/power_supply/axp20x-battery/charge_full_design)
-rate=$(cat /sys/class/power_supply/axp20x-battery/current_now)
-perc=$(cat /sys/class/power_supply/axp20x-battery/capacity)
-perc_left=$((100 - $perc))
 # these icons come from sxmo; they only render in nerdfonts
 bat_dis="󱊢"
 bat_chg="󱊥"
 
+try_path() {
+  # returns:
+  # - perc, perc_left  (0-100)
+  # - full, rate (pos means charging)
+  if [ -f "$1/capacity" ]; then
+    perc=$(cat "$1/capacity")
+    perc_left=$((100 - $perc))
+  fi
+
+  if [ -f "$1/charge_full_design" ] && [ -f "$1/current_now" ]; then
+    # current is positive when charging
+    full=$(cat "$1/charge_full_design")
+    rate=$(cat "$1/current_now")
+  fi
+  if [ -f "$1/energy_full" ] && [ -f "$1/energy_now" ]; then
+    # energy is positive when discharging
+    full=$(cat "$1/energy_full")
+    rate=-$(cat "$1/energy_now")
+  fi
+}
+
+try_path "/sys/class/power_supply/axp20x-battery"  # Pinephone
+try_path "/sys/class/power_supply/BAT0"  # Thinkpad
+
 fmt_minutes() {
   # args: <battery symbol> <text if ludicrous estimate> <estimated minutes to full/empty>
   if [[ $3 -gt 1440 ]]; then
@@ -27,6 +47,6 @@ if [[ $rate -lt 0 ]]; then
 elif [[ $rate -gt 0 ]]; then
   # charging
   fmt_minutes "$bat_chg" '100%' "$(($full * 60 * $perc_left / (100 * $rate)))"
-else
+elif [[ "$perc" != "" ]]; then
   echo "$bat_dis $perc%"
 fi

hosts/common/programs/conky/conky.conf

@@ -27,7 +27,7 @@ conky.config = {
     default_shade_color = '#beebe5',
     default_outline_color = '#beebe5',
 
-    font = 'Sxmo:size=8',
+    font = 'sans-serif:size=8',
     use_xft = true,
 
     default_color = '#ffffff',
@@ -37,14 +37,14 @@ conky.config = {
 
 -- texeci <interval_sec> <cmd>: run the command periodically, _in a separate thread_ so as not to block rendering
 conky.text = [[
-${color1}${shadecolor 707070}${font Sxmo:size=50:style=Bold}${alignc}${exec date +"%H:%M"}${font}
-${color2}${shadecolor a4d7d0}${font Sxmo:size=20}${alignc}${exec date +"%a %d %b"}${font}
+${color1}${shadecolor 707070}${font sans-serif:size=50:style=Bold}${alignc}${exec date +"%H:%M"}${font}
+${color2}${shadecolor a4d7d0}${font sans-serif:size=20}${alignc}${exec date +"%a %d %b"}${font}
 
 
-${color1}${shadecolor}${font Sxmo:size=22:style=Bold}${alignc}${exec @bat@ }${font}
-${color1}${shadecolor}${font Sxmo:size=20:style=Bold}${alignc}${texeci 600 @weather@ }${font}
+${color1}${shadecolor}${font sans-serif:size=22:style=Bold}${alignc}${exec @bat@ }${font}
+${color1}${shadecolor}${font sans-serif:size=20:style=Bold}${alignc}${texeci 600 @weather@ }${font}
 
 
-${color2}${shadecolor a4d7d0}${font Sxmo:size=16}${alignc}⇅ ${downspeedf wlan0}K/s${font}
-${font Sxmo:size=16}${alignc}☵ $memperc%  $cpu%${font}
+${color2}${shadecolor a4d7d0}${font sans-serif:size=16}${alignc}⇅ ${downspeedf wlan0}K/s${font}
+${font sans-serif:size=16}${alignc}☵ $memperc%  $cpu%${font}
 ]]

hosts/common/programs/conky/default.nix

@@ -0,0 +1,34 @@
+{ config, pkgs, ... }:
+{
+  sane.programs.conky = {
+    fs.".config/conky/conky.conf".symlink.target =
+      let
+        battery_estimate = pkgs.static-nix-shell.mkBash {
+          pname = "battery_estimate";
+          src = ./.;
+        };
+      in pkgs.substituteAll {
+        src = ./conky.conf;
+        bat = "${battery_estimate}/bin/battery_estimate";
+        weather = "timeout 20 ${pkgs.sane-weather}/bin/sane-weather";
+      };
+
+    services.conky = {
+      description = "conky dynamic desktop background";
+      wantedBy = [ "default.target" ];
+      # XXX: should be part of graphical-session.target, but whatever mix of greetd/sway
+      # i'm using means that target's never reached...
+      # wantedBy = [ "graphical-session.target" ];
+      # partOf = [ "graphical-session.target" ];
+
+      serviceConfig.ExecStart = "${config.sane.programs.conky.package}/bin/conky";
+      serviceConfig.Type = "simple";
+      serviceConfig.Restart = "on-failure";
+      serviceConfig.RestartSec = "10s";
+      # serviceConfig.Slice = "session.slice";
+
+      # don't start conky until after sway
+      preStart = ''test -n "$SWAYSOCK"'';
+    };
+  };
+}

hosts/common/programs/cozy.nix

@@ -3,7 +3,7 @@
 {
   sane.programs.cozy = {
     # cozy uses a sqlite db for its config and exposes no CLI options other than --help and --debug
-    persist.plaintext = [
+    persist.byStore.plaintext = [
       ".local/share/cozy"  # sqlite db (config & index?)
       ".cache/cozy"  # offline cache
     ];

hosts/common/programs/default.nix

@@ -2,36 +2,47 @@
 
 {
   imports = [
+    ./abaddon.nix
     ./aerc.nix
     ./alacritty.nix
+    ./animatch.nix
     ./assorted.nix
+    ./bemenu.nix
+    ./brave.nix
+    ./calls.nix
     ./cantata.nix
     ./chatty.nix
+    ./conky
     ./cozy.nix
+    ./dialect.nix
     ./dino.nix
     ./element-desktop.nix
     ./epiphany.nix
     ./evince.nix
+    ./feedbackd.nix
     ./firefox.nix
+    ./flare-signal.nix
     ./fontconfig.nix
     ./fractal.nix
     ./fwupd.nix
     ./g4music.nix
     ./gajim.nix
+    ./geary.nix
     ./git.nix
     ./gnome-feeds.nix
     ./gnome-keyring.nix
     ./gnome-weather.nix
     ./gpodder.nix
     ./gthumb.nix
+    ./gtkcord4.nix
     ./helix.nix
     ./imagemagick.nix
     ./jellyfin-media-player.nix
-    ./kitty
     ./komikku.nix
     ./koreader
     ./libreoffice.nix
     ./lemoa.nix
+    ./mako.nix
     ./megapixels.nix
     ./mepo.nix
     ./mopidy.nix
@@ -41,17 +52,29 @@
     ./newsflash.nix
     ./nheko.nix
     ./nix-index.nix
+    ./ntfy-sh.nix
     ./obsidian.nix
     ./offlineimap.nix
+    ./playerctl.nix
     ./rhythmbox.nix
     ./ripgrep.nix
     ./sfeed.nix
+    ./signal-desktop.nix
     ./splatmoji.nix
+    ./spot.nix
+    ./spotify.nix
     ./steam.nix
+    ./stepmania.nix
     ./sublime-music.nix
+    ./supertuxkart.nix
+    ./sway-autoscaler
+    ./swaynotificationcenter.nix
     ./tangram.nix
+    ./tor-browser-bundle-bin.nix
     ./tuba.nix
     ./vlc.nix
+    ./wike.nix
+    ./wine.nix
     ./wireshark.nix
     ./xarchiver.nix
     ./zeal.nix

hosts/common/programs/dialect.nix

@@ -0,0 +1,13 @@
+{ pkgs, ... }:
+{
+  sane.programs.dialect = {
+    package = pkgs.dialect.overrideAttrs (upstream: {
+      # TODO: send upstream
+      # TODO: figure out how to get audio working
+      # TODO: move to runtimeDependencies?
+      buildInputs = upstream.buildInputs ++ [
+        pkgs.glib-networking  # for TLS
+      ];
+    });
+  };
+}

hosts/common/programs/dino.nix

@@ -9,7 +9,63 @@
 #   - ensure the other party is in your roster
 #   - open a DM with the party
 #   - click the phone icon at top (only visible if other party is in your roster)
-{ ... }:
+#
+# dino can be autostarted on login -- useful to ensure that i always receive calls and notifications --
+# but at present it has no "start in tray" type of option: it must render a window.
+#
+# outstanding bugs:
+# - mic is sometimes disabled at call start despite presenting as enabled
+#   - fix is to toggle it off -> on in the Dino UI
+# - default mic gain is WAY TOO MUCH (heavily distorted)
+# - TODO: dino should have more optimal niceness/priority to ensure it can process its buffers
+#   - possibly this is solved by enabling RealtimeKit (rtkit)
+# - TODO: see if Dino calls work better with `echo full > /sys/kernel/debug/sched/preempt`
+#
+# probably fixed:
+# - once per 1-2 minutes dino will temporarily drop mic input:
+#   - `rtp-WRNING: plugin.vala:148: Warning in pipeline: Can't record audio fast enough
+#   - this was *partially* fixed by bumping the pipewire mic buffer to 2048 samples (from ~512)
+#   - this was further fixed by setting PULSE_LATENCY_MSEC=20.
+#   - possibly Dino should be updated internally: `info.rate / 100` -> `info.rate / 50`.
+#     - i think that affects the batching for echo cancellation, adaptive gain control, etc.
+#
+{ config, lib, ... }:
+let
+  cfg = config.sane.programs.dino;
+in
 {
-  sane.programs.dino.persist.private = [ ".local/share/dino" ];
+  sane.programs.dino = {
+    configOption = with lib; mkOption {
+      default = {};
+      type = types.submodule {
+        options.autostart = mkOption {
+          type = types.bool;
+          default = false;
+        };
+      };
+    };
+
+    persist.byStore.private = [ ".local/share/dino" ];
+
+    services.dino = {
+      description = "dino XMPP client";
+      wantedBy = lib.mkIf cfg.config.autostart [ "default.target" ];
+      serviceConfig = {
+        ExecStart = "${cfg.package}/bin/dino";
+        Type = "simple";
+        Restart = "always";
+        RestartSec = "20s";
+      };
+
+      # audio buffering; see: <https://gitlab.freedesktop.org/pipewire/pipewire/-/wikis/FAQ#pipewire-buffering-explained>
+      # dino defaults to 10ms mic buffer, which causes underruns, which Dino handles *very* poorly
+      # as in, the other end of the call will just not receive sound from us for a couple seconds.
+      # pipewire uses power-of-two buffering for the mic itself. that would put us at 21.33 ms, but this env var supports only whole numbers (21ms ends up not power-of-two).
+      # also, Dino's likely still doing things in 10ms batches internally anyway.
+      environment.PULSE_LATENCY_MSEC = "20";
+
+      # note that debug logging during calls produces so much journal spam that it pegs the CPU and causes dropped audio
+      # environment.G_MESSAGES_DEBUG = "all";
+    };
+  };
 }

hosts/common/programs/element-desktop.nix

@@ -1,8 +1,19 @@
-{ ... }:
+# debugging tips:
+# - if element opens but does not render:
+#   - `element-desktop --disable-gpu --in-process-gpu`
+#     - <https://github.com/vector-im/element-desktop/issues/1029#issuecomment-1632688224>
+#   - `rm -rf ~/.config/Element/GPUCache`
+#     - <https://github.com/NixOS/nixpkgs/issues/244486>
+{ pkgs, ... }:
 {
   sane.programs.element-desktop = {
+    package = pkgs.element-desktop.override {
+      # use pre-build electron because otherwise it takes 4 hrs to build from source.
+      electron = pkgs.electron-bin;
+    };
+
     # creds/session keys, etc
-    persist.private = [ ".config/Element" ];
+    persist.byStore.private = [ ".config/Element" ];
 
     suggestedPrograms = [ "gnome-keyring" ];
   };

hosts/common/programs/epiphany.nix

@@ -8,7 +8,7 @@
 { pkgs, ... }:
 {
   sane.programs.epiphany = {
-    # XXX(2023/07/08): running on moby without this hack fails, with:
+    # XXX(2023/07/08): running on moby without `WEBKIT_DISABLE_SANDBOX...` fails, with:
     # - `bwrap: Can't make symlink at /var/run: File exists`
     # this could be due to:
     # - epiphany is somewhere following a symlink into /var/run instead of /run
@@ -19,6 +19,9 @@
     # - <https://gitlab.gnome.org/GNOME/gnome-builder/-/issues/1164>
     # - <https://github.com/flatpak/flatpak/issues/3477>
     # - <https://github.com/NixOS/nixpkgs/issues/197085>
+    #
+    # TODO: consider `WEBKIT_USE_SINGLE_WEB_PROCESS=1` for better perf
+    # - this runs all tabs in 1 process. which is fine, if i'm not a heavy multi-tabber
     package = pkgs.epiphany.overrideAttrs (upstream: {
       preFixup = ''
         gappsWrapperArgs+=(
@@ -26,7 +29,7 @@
         );
       '' + (upstream.preFixup or "");
     });
-    persist.private = [
+    persist.byStore.private = [
       ".cache/epiphany"
       ".local/share/epiphany"
       # also .config/epiphany, but appears empty

hosts/common/programs/feedbackd.nix

@@ -0,0 +1,114 @@
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.sane.programs.feedbackd;
+in
+{
+  sane.programs.feedbackd = {
+    package = pkgs.rmDbusServices pkgs.feedbackd;
+
+    configOption = with lib; mkOption {
+      type = types.submodule {
+        options.proxied = mkOption {
+          type = types.bool;
+          default = false;
+          description = ''
+          whether to use a sound theme in which common application events are muted
+          with the intent that a proxy (notification daemon) with knowledge of this
+          modification will "speak" on behalf of all applications.
+          '';
+        };
+      };
+      default = {};
+    };
+
+    # N.B.: feedbackd will load ~/.config/feedbackd/themes/default.json by default
+    # - but using that would forbid `parent-theme = "default"`
+    # the default theme ships support for these events:
+    # - alarm-clock-elapsed
+    # - battery-caution
+    # - bell-terminal
+    # - button-pressed
+    # - button-released
+    # - camera-focus
+    # - camera-shutter
+    # - message-missed-email
+    # - message-missed-instant
+    # - message-missed-notification
+    # - message-missed-sms
+    # - message-new-email
+    # - message-new-instant
+    # - message-new-sms
+    # - message-sent-instant
+    # - phone-failure
+    # - phone-hangup
+    # - phone-incoming-call
+    # - phone-missed-call
+    # - phone-outgoing-busy
+    # - screen-capture
+    # - theme-demo
+    # - timeout-completed
+    # - window-close
+    fs.".config/feedbackd/themes/proxied.json".symlink.text = builtins.toJSON {
+      name = "proxied";
+      parent-theme = "default";
+      profiles = [
+        {
+          name = "full";
+          feedbacks = [
+            # forcibly disable normal events which we'd prefer for the notification daemon (e.g. swaync) to handle
+            {
+              event-name = "message-new-instant";
+              type = "Dummy";
+            }
+            {
+              event-name = "proxied-message-new-instant";
+              type = "Sound";
+              effect = "message-new-instant";
+            }
+            # re-define sounds from the default theme which we'd like to pass through w/o proxying.
+            # i guess this means i'm not inheriting the default theme :|
+            {
+              event-name = "phone-incoming-call";
+              type = "Sound";
+              effect = "phone-incoming-call";
+            }
+            {
+              event-name = "alarm-clock-elapsed";
+              type = "Sound";
+              effect = "alarm-clock-elapsed";
+            }
+            {
+              event-name = "timeout-completed";
+              type = "Sound";
+              effect = "complete";
+            }
+          ];
+        }
+      ];
+    };
+
+    services.feedbackd = {
+      description = "feedbackd audio/vibration/led controller";
+      wantedBy = [ "default.target" ];
+      serviceConfig = {
+        ExecStart = "${cfg.package}/libexec/feedbackd";
+        Type = "simple";
+        Restart = "on-failure";
+        RestartSec = "10s";
+      };
+      environment = {
+        G_MESSAGES_DEBUG = "all";
+      } // (lib.optionalAttrs cfg.config.proxied {
+        FEEDBACK_THEME = "/home/colin/.config/feedbackd/themes/proxied.json";
+      });
+    };
+  };
+
+  services.udev.packages = lib.mkIf cfg.enabled [
+    # ships udev rules for `feedbackd` group to be able to control vibrator and LEDs
+    cfg.package
+  ];
+  users.groups = lib.mkIf cfg.enabled {
+    feedbackd = {};
+  };
+}

hosts/common/programs/firefox.nix

@@ -13,7 +13,14 @@ let
   mobile-prefs = lib.optionals false pkgs.librewolf-pmos-mobile.extraPrefsFiles;
   # allow easy switching between firefox and librewolf with `defaultSettings`, below
   librewolfSettings = {
-    browser = pkgs.librewolf-unwrapped;
+    browser = pkgs.librewolf-unwrapped.overrideAttrs (upstream: {
+      # TEMP(2023/11/21): fix eval bug in wrapFirefox
+      # see: <https://github.com/NixOS/nixpkgs/pull/244591>
+      passthru = upstream.passthru // {
+        requireSigning = false;
+        allowAddonSideload = true;
+      };
+    });
     extraPrefsFiles = pkgs.librewolf-unwrapped.extraPrefsFiles ++ mobile-prefs;
     libName = "librewolf";
     dotDir = ".librewolf";
@@ -225,6 +232,13 @@ in
           // treat it as unrevoked.
           // see: <https://librewolf.net/docs/faq/#im-getting-sec_error_ocsp_server_error-what-can-i-do>
           defaultPref("security.OCSP.require", false);
+
+          // scrollbar configuration, see: <https://artemis.sh/2023/10/12/scrollbars.html>
+          // style=4 gives rectangular scrollbars
+          // could also enable "always show scrollbars" in about:preferences -- not sure what the actual pref name for that is
+          // note that too-large scrollbars (like 50px wide) tend to obscure content (and make buttons unclickable)
+          defaultPref("widget.non-native-theme.scrollbar.size.override", 20);
+          defaultPref("widget.non-native-theme.scrollbar.style", 4);
         '';
         fs."${cfg.browser.dotDir}/default".dir = {};
         # instruct Firefox to put the profile in a predictable directory (so we can do things like persist just it).

hosts/common/programs/flare-signal.nix

@@ -0,0 +1,47 @@
+# Flare is a 3rd-party GTK4 Signal app.
+# UI is effectively a clone of Fractal.
+#
+# compatibility:
+# - desko: works fine. pairs, and exchanges contact list (but not message history) with the paired device. exchanges future messages fine.
+# - moby (cross compiled flare-signal-nixified): nope. it pairs, but can only *receive* messages and never *send* them.
+#   - even `rsync`ing the data and keyrings from desko -> moby, still fails in that same manner.
+#   - console shows error messages. quite possibly an endianness mismatch somewhere
+# - moby (partially-emulated flare-signal): works! pairs and can send/receive messages, same as desko.
+#
+# error signatures (to reset, run `sane-wipe-fractal`):
+# - upon sending a message, the other side receives it, but Signal desktop gets "A message from Colin could not be delivered" and the local CLI shows:
+#   ```
+#   ERROR libsignal_service::websocket] SignalWebSocket: Websocket error: SignalWebSocket: end of application request stream; socket closing
+#   ERROR presage::manager] Error opening envelope: ProtobufDecodeError(DecodeError { description: "invalid tag value: 0", stack: [("Content", "data_message")] }), message will be skipped!
+#   ERROR presage::manager] Error opening envelope: ProtobufDecodeError(DecodeError { description: "invalid tag value: 0", stack: [("Content", "data_message")] }), message will be skipped!
+#   ```
+# - this occurs on moby, desko, `flare-signal` and `flare-signal-nixified`
+# - the Websocket error seems to be unrelated, occurs during normal/good operation
+# - related issues: <https://github.com/whisperfish/presage/issues/152>
+# error when sending from Flare to other Flare device:
+# - ```
+#   ERROR libsignal_protocol::session_cipher] Message from <UUID>.3 failed to decrypt; sender ratchet public key <key> message counter 1
+#       No current session
+#   ERROR presage::manager] Error opening envelope: SignalProtocolError(InvalidKyberPreKeyId), message will be skipped!
+#   ```
+# - but signal iOS will still read it.
+#
+# well, seems to have unpredictable errors particularly when being used on multiple devices.
+# desktop _seems_ more reliable than on mobile, but not confident.
+
+{ pkgs, ... }:
+{
+  sane.programs.flare-signal = {
+    package = pkgs.flare-signal-nixified;
+    # package = pkgs.flare-signal;
+    persist.byStore.private = [
+      # everything: conf, state, files, all opaque
+      ".local/share/flare"
+      # also persists a secret in ~/.local/share/keyrings. reset with:
+      # - `secret-tool search --all --unlock 'xdg:schema' 'de.schmidhuberj.Flare'`
+      # - `secret-tool clear 'xdg:schema' 'de.schmidhuberj.Flare'`
+      # and it persists some dconf settings (e.g. device name). reset with:
+      # - `dconf reset -f /de/schmidhuberj/Flare/`.
+    ];
+  };
+}

hosts/common/programs/fontconfig.nix

@@ -1,15 +1,68 @@
+# to preview fonts:
+# - `font-manager` (gui)
+#   - useful to determine official name; codepoint support
 { config, lib, pkgs, ... }:
+let
+  # nerdfonts takes popular open fonts and patches them to support a wider range of glyphs, notably emoji.
+  # any nerdfonts font includes icons such as these:
+  # - 󱊥 (battery charging)
+  # - 󰃝 (brightness)
+  # -  (gps / crosshairs)
+  # - 󰎈 (music note)
+  # - 󰍦 (message bubble)
+  # - 󰏲 (phone)
+  # -  (weather/sun-behind-clouds)
+  # used particularly by sxmo utilities, but also a few of my own (e.g. conky)
+  #
+  # nerdfonts is very heavy. each font is 20-900 MiB (2 MiB per "variation")
+  # lots of redundant data inside there, but no deduplication except whatever nix or the fs does implicitly.
+  wantedNerdfonts = [
+    # used explicitly by SXMO
+    # "DejaVuSansMono"  # 25 MiB
+    # good terminal/coding font. grab via nerdfonts for more emoji/unicode support
+    "Hack"  # 26 MiB
+    "Noto"  # 861 MiB
+  ];
+  nerdfontPkgs = builtins.map
+    (f: pkgs.nerdfonts.override { fonts = [ f ]; })
+    wantedNerdfonts;
+in
 {
   fonts = lib.mkIf config.sane.programs.fontconfig.enabled {
     fontconfig.enable = true;
     fontconfig.defaultFonts = {
-      emoji = [ "Font Awesome 6 Free" "Noto Color Emoji" ];
-      monospace = [ "Hack" ];
-      serif = [ "DejaVu Serif" ];
-      sansSerif = [ "DejaVu Sans" ];
+      emoji = [
+        "Noto Color Emoji"
+        "Font Awesome 6 Free"
+        "Font Awesome 6 Brands"
+      ];
+      monospace = [
+        "Hack Nerd Font Propo"
+        # "DejaVuSansM Nerd Font Propo"
+        "NotoMono Nerd Font Propo"
+      ];
+      serif = [
+        "NotoSerif Nerd Font"
+        "DejaVu Serif"
+      ];
+      sansSerif = [
+        "NotoSans Nerd Font"
+        "DejaVu Sans"
+      ];
     };
     #vvv enables dejavu_fonts, freefont_ttf, gyre-fonts, liberation_ttf, unifont, noto-fonts-emoji
-    enableDefaultPackages = true;
-    packages = with pkgs; [ font-awesome noto-fonts-emoji hack-font ];
+    enableDefaultPackages = false;
+    packages = with pkgs; [
+      # TODO: reduce this font set.
+      # - probably need only one of dejavu/freefont/liberation
+      dejavu_fonts  # 10 MiB; DejaVu {Sans,Serif,Sans Mono,Math TeX Gyre}; also available as a NerdFonts (Sans Mono only)
+      font-awesome  #  2 MiB; Font Awesome 6 {Free,Brands}
+      freefont_ttf  # 11 MiB; Free{Mono,Sans,Serif}
+      gyre-fonts    #  4 MiB; Tex Gyre *; ttf substitutes for standard PostScript fonts
+      # hack-font     #  1 MiB; Hack; also available as a NerdFonts
+      liberation_ttf # 4 MiB; Liberation {Mono,Sans,Serif}; also available as a NerdFonts
+      noto-fonts-color-emoji  # 10 Mib; Noto Color Emoji
+      unifont       # 16 MiB; Unifont; provides LOTS of unicode coverage
+    ] ++ nerdfontPkgs;
   };
 }

hosts/common/programs/fractal.nix

@@ -1,12 +1,61 @@
-{ pkgs, ... }:
+# Fractal: GTK4 instant messenger client for the Matrix protocol
+#
+# very susceptible to state corruption during hard power-cycles.
+# if it stalls while launching, especially with a brief message at bottom
+# "unable to open store"
+# then:
+# - remove ~/.local/share/stable/*
+#   - this might give I/O error, in which case remove the corresponding path under
+#     /nix/persist/home/colin/private (which can be found by correlating timestamps/sizes with that in ~/private/.local/share/stable).
+# - reboot (maybe necessary).
+# - now you can send messages, and read messages in unencrypted rooms, but not read messages from encrypted rooms.
+# to fix encrypted message receipt:
+# - start from above (fractal closed, no ~/.local/share/stable/*)
+# - in ~/.local/share/keyrings/Default_keyring.keyring:
+#   - find the entry that says "display-name=Fractal: Matrix credentials for <mxid>"
+#   - remove that entry and all associated entries (i.e. ones with same number but different :attributeN)
+#   - REBOOT. otherwise keyring stuff seems to stay cached in RAM
+#   - login to Fractal. give an hour to sync.
+#   - it'll kick you back to a page asking you to cross-sign. open FluffyChat and do the emoji compare. success!
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.sane.programs.fractal;
+in
 {
   sane.programs.fractal = {
+    package = pkgs.fractal-nixified;
     # package = pkgs.fractal-latest;
-    package = pkgs.fractal-next;
+    # package = pkgs.fractal-next;
 
-    # XXX by default fractal stores its state in ~/.local/share/stable/<UUID>.
-    persist.private = [ ".local/share/stable" ];
+    configOption = with lib; mkOption {
+      default = {};
+      type = types.submodule {
+        options.autostart = mkOption {
+          type = types.bool;
+          default = true;
+        };
+      };
+    };
+
+    persist.byStore.private = [
+      # XXX by default fractal stores its state in ~/.local/share/<build-profile>/<UUID>.
+      ".local/share/hack"    # for debug-like builds
+      ".local/share/stable"  # for normal releases
+      ".local/share/fractal" # for version 5+, i think?
+    ];
 
     suggestedPrograms = [ "gnome-keyring" ];
+
+    services.fractal = {
+      description = "auto-start and maintain fractal Matrix connection";
+      wantedBy = lib.mkIf cfg.config.autostart [ "default.target" ];
+      serviceConfig = {
+        ExecStart = "${cfg.package}/bin/fractal";
+        Type = "simple";
+        Restart = "always";
+        RestartSec = "20s";
+      };
+      # environment.G_MESSAGES_DEBUG = "all";
+    };
   };
 }

hosts/common/programs/g4music.nix

@@ -8,7 +8,7 @@
 { ... }:
 {
   sane.programs.g4music = {
-    persist.plaintext = [
+    persist.byStore.plaintext = [
       # index?
       ".cache/com.github.neithern.g4music"
     ];

hosts/common/programs/gajim.nix

@@ -1,7 +1,7 @@
 { ... }:
 {
   sane.programs.gajim = {
-    persist.private = [
+    persist.byStore.private = [
       # avatars, thumbnails...
       ".cache/gajim"
       # sqlite database labeled "settings". definitely includes UI theming

hosts/common/programs/geary.nix

@@ -0,0 +1,82 @@
+# geary is a gtk3 email client.
+# outstanding issues:
+# - it uses webkitgtk_4_1, which is expensive to build.
+#   could be upgraded to webkitgtk latest if upgraded to gtk4
+#   <https://gitlab.gnome.org/GNOME/geary/-/issues/1212>
+{ config, lib, ... }:
+let
+  cfg = config.sane.programs."gnome.geary";
+in
+{
+  sane.programs."gnome.geary" = {
+    configOption = with lib; mkOption {
+      default = {};
+      type = types.submodule {
+        options.autostart = mkOption {
+          type = types.bool;
+          default = false;
+        };
+      };
+    };
+
+    slowToBuild = true;  # uses webkitgtk 4.1
+    persist.byStore.private = [
+      # attachments, and email -- contained in a sqlite db
+      ".local/share/geary"
+      # also `.cache/geary/web-resources`, which tends to stay << 1 MiB
+    ];
+    fs.".config/geary/account_01/geary.ini".symlink.text = ''
+      [Metadata]
+      version=1
+      status=enabled
+
+      [Account]
+      ordinal=2
+      label=
+      # 14 = "fetch last 14d of mail every time i connect"
+      # -1 = "fetch *all* mail"
+      prefetch_days=-1
+      save_drafts=true
+      save_sent=true
+      use_signature=false
+      signature=
+      sender_mailboxes=colin@uninsane.org;
+      service_provider=other
+
+      [Folders]
+      archive_folder=Archive;
+      drafts_folder=
+      sent_folder=
+      junk_folder=
+      trash_folder=
+
+      [Incoming]
+      login=colin
+      remember_password=true
+      host=imap.uninsane.org
+      port=993
+      transport_security=transport
+      credentials=custom
+
+      [Outgoing]
+      remember_password=true
+      host=mx.uninsane.org
+      port=465
+      transport_security=transport
+      credentials=use-incoming
+    '';
+    secrets.".config/geary/account_02/geary.ini" = ../../../secrets/common/geary_account_02.ini.bin;
+
+    services.geary = {
+      description = "Geary email client";
+      wantedBy = lib.mkIf cfg.config.autostart [ "default.target" ];
+      serviceConfig = {
+        ExecStart = "${cfg.package}/bin/geary";
+        Type = "simple";
+        Restart = "always";
+        RestartSec = "20s";
+      };
+    };
+  };
+
+}

hosts/common/programs/gnome-weather.nix

@@ -3,7 +3,7 @@
 { ... }:
 {
   sane.programs.gnome-weather = {
-    persist.plaintext = [
+    persist.byStore.plaintext = [
       ".cache/libgweather"
     ];
   };

hosts/common/programs/gpodder.nix

@@ -1,4 +1,5 @@
-# gnome feeds RSS viewer
+# help:
+# - #gpodder on irc.libera.chat
 { config, pkgs, sane-lib, ... }:
 
 let
@@ -7,12 +8,22 @@ let
   wanted-feeds = feeds.filterByFormat ["podcast"] all-feeds;
 in {
   sane.programs.gpodder = {
-    package = pkgs.gpodder-adaptive-configured;
+    package = pkgs.gpodder-adaptive-configured.overrideAttrs (base: {
+      # environment variables:
+      # - GPODDER_HOME (defaults to "~/gPodder")
+      # - GPODDER_DOWNLOAD_DIR (defaults to "$GPODDER_HOME/Downloads")
+      # - GPODDER_WRITE_LOGS ("yes" or "no")
+      # - GPODDER_EXTENSIONS
+      # - GPODDER_DISABLE_EXTENSIONS ("yes" or "no")
+      extraMakeWrapperArgs = (base.extraMakeWrapperArgs or []) ++ [
+        "--set" "GPODDER_HOME" "~/.local/share/gPodder"
+      ];
+    });
     # package = pkgs.gpodder-configured;
     fs.".config/gpodderFeeds.opml".symlink.text = feeds.feedsToOpml wanted-feeds;
 
     # XXX: we preserve the whole thing because if we only preserve gPodder/Downloads
     #   then startup is SLOW during feed import, and we might end up with zombie eps in the dl dir.
-    persist.plaintext = [ "gPodder" ];
+    persist.byStore.plaintext = [ ".local/share/gPodder" ];
   };
 }

hosts/common/programs/gtkcord4.nix

@@ -0,0 +1,33 @@
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.sane.programs.gtkcord4;
+in
+{
+  sane.programs.gtkcord4 = {
+    configOption = with lib; mkOption {
+      default = {};
+      type = types.submodule {
+        options.autostart = mkOption {
+          type = types.bool;
+          default = false;
+        };
+      };
+    };
+
+    persist.byStore.private = [
+      ".cache/gtkcord4"
+      ".config/gtkcord4"  # empty?
+    ];
+
+    services.gtkcord4 = {
+      description = "unofficial Discord chat client";
+      wantedBy = lib.mkIf cfg.config.autostart [ "default.target" ];
+      serviceConfig = {
+        ExecStart = "${cfg.package}/bin/gtkcord4";
+        Type = "simple";
+        Restart = "always";
+        RestartSec = "20s";
+      };
+    };
+  };
+}

hosts/common/programs/helix.nix

@@ -7,7 +7,7 @@
     # grammars need to be persisted when developing them
     # - `hx --grammar fetch` and `hx --grammar build`
     # but otherwise, they ship as part of HELIX_RUNTIME, in the nix store
-    # persist.plaintext = [ ".config/helix/runtime/grammars" ];
+    # persist.byStore.plaintext = [ ".config/helix/runtime/grammars" ];
     fs.".config/helix/config.toml".symlink.text = ''
       # docs: <https://docs.helix-editor.com/configuration.html>
       [editor.soft-wrap]

hosts/common/programs/jellyfin-media-player.nix

@@ -10,6 +10,6 @@
     # jellyfin stores things in a bunch of directories: this one persists auth info.
     # it *might* be possible to populate this externally (it's Qt stuff), but likely to
     # be fragile and take an hour+ to figure out.
-    persist.plaintext = [ ".local/share/Jellyfin Media Player" ];
+    persist.byStore.plaintext = [ ".local/share/Jellyfin Media Player" ];
   };
 }

hosts/common/programs/kitty/PaperColor_dark.conf

@@ -1,47 +0,0 @@
-# vim:ft=kitty
-
-## name: PaperColor Dark
-## author: Nikyle Nguyen
-## license: MIT
-## blurb: Dark color scheme inspired by Google's Material Design 
-
-# special
-foreground		#d0d0d0
-background		#1c1c1c
-cursor			#d0d0d0
-cursor_text_color	background
-
-# black
-color0			#1c1c1c
-color8			#585858
-
-# red
-color1			#af005f
-color9			#5faf5f
-
-# green
-# "color2" is the green color used by ls to indicate executability
-#   both as text color
-#   or as bg color when the text is blue (color4)
-color2			#246a28
-color10			#2df200
-
-# yellow
-color3			#d7af5f
-color11			#af87d7
-
-# blue
-color4			#78c6ef
-color12			#ffaf00
-
-# magenta
-color5			#808080
-color13			#ff5faf
-
-# cyan
-color6			#d7875f
-color14			#00afaf
-
-# white
-color7			#d0d0d0
-color15			#5f8787

hosts/common/programs/kitty/default.nix

@@ -1,73 +0,0 @@
-{ lib, ... }:
-
-{
-  sane.programs.kitty = {
-    fs.".config/kitty/kitty.conf".symlink.text = ''
-      # docs: https://sw.kovidgoyal.net/kitty/conf/
-      # disable terminal bell (when e.g. you backspace too many times)
-      enable_audio_bell no
-
-      map ctrl+n new_os_window_with_cwd
-      include ${./PaperColor_dark.conf}
-    '';
-    env.TERMINAL = lib.mkDefault "kitty";
-  };
-
-  #  include ${pkgs.kitty-themes}/themes/PaperColor_dark.conf
-
-  # THEME CHOICES:
-  #   docs: https://github.com/kovidgoyal/kitty-themes
-  # theme = "1984 Light";  # dislike: awful, harsh blues/teals
-  # theme = "Adventure Time";  # dislike: harsh (dark)
-  # theme = "Atom One Light";  # GOOD: light theme. all color combos readable. not a huge fan of the blue.
-  # theme = "Belafonte Day";  # dislike: too low contrast for text colors
-  # theme = "Belafonte Night";  # better: dark theme that's easy on the eyes. all combos readable. low contrast.
-  # theme = "Catppuccin";  # dislike: a bit pale/low-contrast (dark)
-  # theme = "Desert";  # mediocre: colors are harsh
-  # theme = "Earthsong";  # BEST: dark theme. readable, good contrast. unique, but decent colors.
-  # theme = "Espresso Libre";  # better: dark theme. readable, but meh colors
-  # theme = "Forest Night";  # decent: very pastel. it's workable, but unconventional and muted/flat.
-  # theme = "Gruvbox Material Light Hard";  # mediocre light theme.
-  # theme = "kanagawabones";  # better: dark theme. colors are too background-y
-  # theme = "Kaolin Dark";  # dislike: too dark
-  # theme = "Kaolin Breeze";  # mediocre: not-too-harsh light theme, but some parts are poor contrast
-  # theme = "Later This Evening";  # mediocre: not-too-harsh dark theme, but cursor is poor contrast
-  # theme = "Material"; # decent: light theme, few colors.
-  # theme = "Mayukai";  # decent: not-too-harsh dark theme. the teal is a bit straining
-  # theme = "Nord";  # mediocre: pale background, low contrast
-  # theme = "One Half Light";  # better: not-too-harsh light theme. contrast could be better
-  # theme = "PaperColor Dark";  # BEST: dark theme, very readable still the colors are background-y
-  # theme = "Parasio Dark";  # dislike: too low contrast
-  # theme = "Pencil Light";  # better: not-too-harsh light theme. decent contrast.
-  # theme = "Pnevma";  # dislike: too low contrast
-  # theme = "Piatto Light";  # better: readable light theme. pleasing colors. powerline prompt is hard to read.
-  # theme = "Rosé Pine Dawn";  # GOOD: light theme. all color combinations are readable. it is very mild -- may need to manually tweak contrast. tasteful colors
-  # theme = "Rosé Pine Moon";  # GOOD: dark theme. tasteful colors. but background is a bit intense
-  # theme = "Sea Shells";  # mediocre. not all color combos are readable
-  # theme = "Solarized Light";  # mediocre: not-too-harsh light theme; GREAT background; but some colors are low contrast
-  # theme = "Solarized Dark Higher Contrast";  # better: dark theme, decent colors
-  # theme = "Sourcerer";  # mediocre: ugly colors
-  # theme = "Space Gray";  # mediocre: too muted
-  # theme = "Space Gray Eighties";  # better: all readable, decent colors
-  # theme = "Spacemacs";  # mediocre: too muted
-  # theme = "Spring";  # mediocre: readable light theme, but the teal is ugly.
-  # theme = "Srcery";  # better: highly readable. colors are ehhh
-  # theme = "Substrata";  # decent: nice colors, but a bit flat.
-  # theme = "Sundried";  # mediocre: the solar text makes me squint
-  # theme = "Symfonic";  # mediocre: the dark purple has low contrast to the black bg.
-  # theme = "Tango Light";  # dislike: teal is too grating
-  # theme = "Tokyo Night Day";  # medicore: too muted
-  # theme = "Tokyo Night";  # better: tasteful. a bit flat
-  # theme = "Tomorrow";  # GOOD: all color combinations are readable. contrast is slightly better than Rose. on the blander side
-  # theme = "Treehouse";  # dislike: the orange is harsh on my eyes.
-  # theme = "Urple";  # dislike: weird palette
-  # theme = "Warm Neon";  # decent: not-too-harsh dark theme. the green is a bit unattractive
-  # theme = "Wild Cherry";  # GOOD: dark theme: nice colors. a bit flat
-  # theme = "Xcodedark";  # dislike: bad palette
-  # theme = "citylights";  # decent: dark theme. some parts have just a bit low contrast
-  # theme = "neobones_light"; # better light theme. the background is maybe too muted
-  # theme = "vimbones";
-  # theme = "zenbones_dark";  # mediocre: readable, but meh colors
-  # theme = "zenbones_light";  # decent: light theme. all colors are readable. contrast is passable but not excellent. highlight color is BAD
-  # theme = "zenwritten_dark";  # mediocre: looks same as zenbones_dark
-}

hosts/common/programs/komikku.nix

@@ -3,6 +3,6 @@
   sane.programs.komikku = {
     secrets.".local/share/komikku/keyrings/plaintext.keyring" = ../../../secrets/common/komikku_accounts.json.bin;
     # downloads end up here, and without the toplevel database komikku doesn't know they exist.
-    persist.plaintext = [ ".local/share/komikku" ];
+    persist.byStore.plaintext = [ ".local/share/komikku" ];
   };
 }