colin
/
uninsane
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

blog: nixos-upstreaming: finish a first draft

This commit is contained in:
colin 2022-10-11 07:08:30 -07:00
parent ea196acf40
commit 25df079540
1 changed files with 70 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

85
content/blog/DRAFT-2022-07-09-nixos-upstreaming.md
View File

@ -5,7 +5,7 @@ description = "remove DRAFT prefix and add a date field to link this from the in
as a [nix/nixOS](https://nixos.org) user, you'll inherently find yourself writing code, and some of this code might be valuable to others: be it new packages, improvements upon existing packages, or new services/options. part of what makes nix so successful (IMO) is the ease of sharing work and upstreaming, so i'm setting out here to show some easy workflows for doing so.
the number one thing you can do -- if you're not already -- is host your nix config publicly. for example, mine is [here](https://git.uninsane.org/colin/nix-files), and even without advertising it i occasionally hear from people in my orbit that they've copied some approach. as a beginner i had some public configs i found on google or [wiby](https://wiby.me) which i routinely consulted, and the authors are likely unaware since i never even contacted them.
the number one thing you can do -- if you're not already -- is host your nix config publicly. for example, mine is [here](https://git.uninsane.org/colin/nix-files), and even without advertising it i occasionally hear from people in my orbit that they've copied some approach. as a beginner i had some public configs i found on google or [wiby](https://wiby.me) which i routinely consulted, even though many of those authors probably never heard a word from the people like me who found use in their work.
## Authoring New Packages
let's say you find a package you want but isn't yet packaged. first, confirm it doesn't exist by searching the package name (and likely variants) on github. remove all filters, so that you're searching both PRs and issues: maybe somebody opened a [package request] (TODO: link docs) and there's already something to work off of. for [example](https://github.com/NixOS/nixpkgs/issues?q=zecwallet-lite).
@ -87,7 +87,7 @@ $ cp -R /etc/nixos/zecwallet-lite ~/nixpkgs/pkgs/applications/misc/
then take your `callPackage` invocation from your overlay and place it in `pkgs/top-level/all-packages.nix` instead. `git add` the file, and then make sure the package builds:
```sh
~/nixpkgs$ nix build './#zecwallet-lite'
~/nixpkgs$ nix build '.#zecwallet-lite'
~/nixpkgs$ ./result/bin/zecwallet-lite # make sure it runs
```
@ -97,24 +97,79 @@ commit the changes, push to a github account, and open a PR. github should pre-p
so you could leave the duplicate package definition both in your `/etc/nixos` repo _and_ in your `nixpkgs` repo, and wait until the next release of nixos/nixpkgs before cleaning up, or you can reference commits in the nixpkgs repository _or in a PR_ from your nixos configuration directly, and consolidate this now.
TODO: show how to apply patches to nixpkgs within a flake.
if you're using a flake for your system config, it probably looks something like this right now:
```nix
{
inputs = {
nixpkgs.url = "nixpkgs/nixos-22.05";
};
outputs = { self, nixpkgs }: {
nixosConfigurations.mySystem = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
./mySystem.nix
];
};
};
}
```
we want to somehow patch that `inputs.nixpkgs` so that it includes whatever changes we're trying to upstream (our new package, a package update, etc). the simplest option is to point `inputs.nixpkgs.url` directly at our open PR (add `&rev=<git-commit>` to do this), but that doesn't work if we have multiple PRs.
## OLD: TODO: cleanup
instead, we use a trick shown [here](https://github.com/NixOS/nix/issues/3920#issuecomment-1168041777):
```nix
{
inputs = {
nixpkgs.url = "nixpkgs/nixos-22.05";
};
outputs = { self, nixpkgs }:
let
patchedPkgs = nixpkgs.legacyPackages.x86_64-linux.applyPatches {
name = "nixpkgs-patched";
src = nixpkgs;
patches = [ ./patch1.patch ./patch2.patch ];
};
nixosSystem = import (patchedPkgs + "/nixos/lib/eval-config.nix");
in {
nixosConfigurations.mySystem = nixosSystem {
system = "x86_64-linux";
modules = [
./mySystem.nix
];
};
};
}
```
first, i create a package in `~/nixos/pkgs/my-package`, add it to the `~/nixos/pkgs/overlay.nix`, and rebuild.
that is, we take the base nixpkgs repo, apply patches to it (creating a new derivation), and then import the result of that derivation so that the rest of our flake refers to this patched nixpkgs.
after i know the package is working reliably i:
- add the package to `~/dev/nixpkgs`
- `git diff HEAD~1.. > ~/nixos/nixpatches/my-package.patch`
- link this `my-package.patch` into the nixpatches and remove `~/nixos/pkgs/my-package`.
the version above expects your patches to live in your config repo, but if the goal is to avoid duplication we'd rather point these at the the PRs live on github. we can use `fetchpatch` for this. if our PR is [180960](https://github.com/NixOS/nixpkgs/pull/180960), we can append `.diff` to the URL to get a patch. the `let` block above becomes this:
after my overlay is working reliably i:
- `git push colinsane new-pkg-my-package` and open a PR.
```nix
let
fetchpatch = nixpkgs.legacyPackages.x86_64-linux.fetchpatch;
patchedPkgs = nixpkgs.legacyPackages.x86_64-linux.applyPatches {
name = "nixpkgs-patched";
src = nixpkgs;
patches = [
(fetchpatch {
url = "https://github.com/NixOS/nixpkgs/pull/180960.diff";
sha256 = "sha256-HVVj/T3yQtjYBoxXpoPiG9Zar/eik9IoDVDhTOehBdY=";
})
];
};
nixosSystem = import (patchedPkgs + "/nixos/lib/eval-config.nix");
```
as the PR is worked, i'll repeat this process (`git diff` into a patch, then commit & push once i see it's working).
and we're set! as before, if you don't know the hash then set `sha256` to any value not used elsewhere and nix will tell you the real hash when you build. you can freely push to your open PR: to have the changes reflect on your system just update the hash and `nixos-rebuild switch`. once your PR is merged, the next `nix flake update` might include your patch already in which case nix will error when building `nixpkgs-patched`. at that point you can safely remove this entry from `patches`.
once the PR is merged i:
- replace `~/nixos/nixpatches/my-package.patch` with a link to the PR's diff instead.
## Upstreaming Other Changes
since this approach is ultimately just applying patches onto `nixpkgs`, it's not limited to only new packages. if you want to update a package version, add a new config option, etc, you can follow effectively the same process: clone `nixpkgs`, checkout the master branch, make a change and `nix build` the result, commit & open a PR, `fetchpatch` the PR into your flake.
if you're making changes in a hot area of the codebase, the diff from your PR (which targets master) might not apply cleanly to your flake (which pulls from a release or nixos-unstable). in that case you can `git log` the files you're changing and use the same `fetchpatch` trick to cherry-pick whatever you need so that your patch applies cleanly.
## Closing Thoughts
i described here the workflow i adopted as i first began contributing. with something as configurable as nix, many people use many different workflows. some prefer maintaining their own long-running fork of nixpkgs wherein they cherry-pick all these PRs and periodically rebase against nixos-unstable, or a release branch, and have their flake point directly to their fork of nixpkgs. your workflow will surely evolve as you settle in, but hopefully this gives you enough inspiration to get started :^)
finally as new nixos versions are released my `~/nixos/nixpatches` directory shrinks.