colin/nixpkgs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
d9414a70d8
nixpkgs/maintainers/scripts/bootstrap-files/README.md
Sergei Trofimovich 8a256ed0a7 maintainers/scripts/bootstrap-files: documentation and a script to update tarballs 
This script attempts to document the exact procedure used to upload
bootstrap binaries used previously. I modeled it after most recent
https://github.com/NixOS/nixpkgs/pull/282517 upload.

There is one deviation from it to make it easier to handle mass updates
for https://github.com/NixOS/nixpkgs/issues/253713:

The binaries are expected to be stored in `stdenv/$target` (and not
something like `stdenv-linux/i686`.

The script handles both native and cross- linux targets. `darwin` will
need a bit more work to fin into this scheme, but it should be easy.

Example run to generate `i686-linux` update:

    $ maintainers/scripts/bootstrap-files/refresh-tarballs.bash --commit --targets=i686-unknown-linux-gnu
2024-01-28 14:48:18 +00:00

2.9 KiB
Raw Blame History

Bootstrap files

Currently nixpkgs builds most of it's packages using bootstrap seed binaries (without the reliance on external inputs):

  • bootstrap-tools: an archive with the compiler toolchain and other helper tools enough to build the rest of the nixpkgs.
  • initial binaries needed to unpack bootstrap-tools.*. On linux it's just busybox, on darwin it's sh, bzip2, mkdir and cpio. These binaries can be executed directly from the store.

These are called "bootstrap files".

Bootstrap files should always be fetched from hydra and uploaded to tarballs.nixos.org to guarantee that all the binaries were built from the code committed into nixpkgs repository.

The uploads to tarballs.nixos.org are done by @lovesegfault today.

This document describes the procedure of updating bootstrap files in nixpkgs.

How to request the bootstrap seed update

To get the tarballs updated let's use an example i686-unknown-linux-gnu target:

  1. Create a local update:

    $ maintainers/scripts/bootstrap-files/refresh-tarballs.bash --commit --targets=i686-unknown-linux-gnu

  2. Test the update locally. I'll build local hello derivation with the result:

    $ nix-build -A hello --argstr system i686-linux

    To validate cross-targets binfmt NixOS helper can be useful. For riscv64-unknown-linux-gnu the /etc/nixox/configuraqtion.nix entry would be boot.binfmt.emulatedSystems = [ "riscv64-linux" ].

  3. Propose the commit as a PR to update bootstrap tarballs, tag people who can help you test the updated architecture and once reviewed tag @lovesegfault to upload the tarballs.

Bootstrap files job definitions

There are two types of bootstrap files:

  • natively built stdenvBootstrapTools.build hydra jobs in nixpkgs:trunk jobset. Incomplete list of examples is:

    • aarch64-unknown-linux-musl.nix
    • i686-unknown-linux-gnu.nix

    These are Tier 1 hydra platforms.

  • cross-built by bootstrapTools.build hydra jobs in nixpkgs:cross-trunk jobset. Incomplete list of examples is:

    • mips64el-unknown-linux-gnuabi64.nix
    • mips64el-unknown-linux-gnuabin32.nix
    • mipsel-unknown-linux-gnu.nix
    • powerpc64le-unknown-linux-gnu.nix
    • riscv64-unknown-linux-gnu.nix

    These are usually Tier 2 and lower targets.

The .build job contains /on-server/ subdirectory with binaries to be uploaded to tarballs.nixos.org. The files are uploaded to tarballs.nixos.org by writers to S3 store.

TODOs

  • pkgs/stdenv/darwin file layout is slightly different from pkgs/stdenv/linux. Once linux seed update becomes a routine we can bring darwin in sync if it's feasible.
  • darwin definition of .build on-server/ directory layout differs and should be updated.