The nixpkgs-unstable channel's programs.sqlite was used to identify
packages producing exactly one binary, and these automatically added
to their package definitions wherever possible.
`glibc-2.39` improved `--enable-cet` detection on unsupported platforms
and `./configure` will start failing as:
glibc> configure: error: "CET is only supported on x86_64 or x32"
This is an alternative resolution of the problem identified in
https://github.com/NixOS/nixpkgs/pull/259964
which stated that "glibc depends on buildPackages.glibc for locale
things. buildPackages.glibc depended on buildPackages.libgcc,
which, since it's GCC, depends on the target's bintools, which
depend on the target's glibc, which, again, depends on
buildPackages.glibc, causing an infinute recursion when evaluating
buildPackages.glibc when glibc hasn't come from stdenv (e.g. on
musl)."
The fact that we use pkgsBuildHost.glibc instead of
pkgsBuildBuild.glibc to generate the locales has always been a gross
hack. If we simply remove the gross hack the circularity goes away.
When specifying the `builder` attribute in `stdenv.mkDerivation`, this
will be effectively transformed into
builtins.derivation {
builder = stdenv.shell;
args = [ "-e" builder ];
}
This also means that `default-builder.sh` is never sourced and as a
result it's not guaranteed that `$NIX_ATTRS_SH_FILE` is set to a correct
location[1].
Also, we need to source `.attrs.sh` to source `$stdenv`. So, the
following is done now:
* If `$NIX_ATTRS_SH_FILE` points to a correct location, then use it.
Directly using `.attrs.sh` is problematic for `nix-shell(1)` usage
(see previous commit for more context), so prefer the environment
variable if possible.
* Otherwise, if `.attrs.sh` exists, then use it. See [1] for when this
can happen.
* If neither applies, it can be assumed that `__structuredAttrs` is
turned off and thus nothing needs to be done.
[1] It's possible that it doesn't exist at all - in case of Nix 2.3 or
it can point to a wrong location on older Nix versions with a bug in
`__structuredAttrs`.
Announcement: https://sourceware.org/pipermail/libc-alpha/2023-July/150524.html
So far this looks surprisingly good, I managed to build the stdenv
on `aarch64-linux` and got up to building `zfs` and `nix` on `x86_64-linux`.
The patchset is still empty because the latest commit on the release branch is
the one the 2.38 tag points to. I added an empty file though to keep
things consistent.
Also applied the new version of the DT_HASH fix from ArchLinux[1]. This
one's a way easier version than before because it doesn't contain the
autoconf changes, but only hardcodes the desired ld flags. It was
already confirmed that this patch is sufficient to fix the underlying
problem[2].
[1] e54d98e2d1 (7b1bfda0391ff4c2662e04a5e193c37e233a0738)
[2] https://github.com/ValveSoftware/Proton/issues/6051#issuecomment-1666055553
On some platforms (e.g. loongarch64-linux), libutil.so.1 is not built in
glibc (see glibc login/Makefile), only a dummy libutil.a is provided.
Do not symlink libutil.so but copy libutil.a to glibc.out if
libutil.so.1 does not exist.
### Summary
This PR completely and finally solves the gcc<->glibc circular
`buildInputs` problem, for cross compilation. The same technique
can be applied to native builds in the future.
Closes#213453
### Motivation
Prior to this PR, we had the following circular `buildInputs` problem:
1. gcc has glibc in its `buildInputs`
- a compiled copy of glibc must be present before building gcc;
if it isn't, gcc cripples itself (`inhibit_libc`) and refuses
to build libgcc_s.so
2. glibc has libgcc_s.so in its `buildInputs`
- glibc `dlopen()`s libgcc_s.so in order to implement POSIX
thread cancellation. For security reasons `glibc` requires
that the path to `libgcc_s.so` is [hardwired] into `glibc` at
compile time, so it's technically not a true dynamic link -- it
just pretends to be one.
3. libgcc_s.so is built in the same derivation as gcc
- libgcc_s.so is built as part of the gcc build process
We must cut one of these three links in the loop.
### Previous Attempts
Previously https://github.com/NixOS/nixpkgs/pull/238154 had
attempted to cut link (1) by building `gcc` without `glibc`, and
using the `libgcc_s` which emerges from that build. Unfortunately
this just doesn't work. GCC's configure script extracts quite a lot
of information from the glibc headers (which are a build artifact --
you can't just copy them out of the source tarball) and various
`./configure`-driven linking attempts. If `glibc` isn't around at
build time you wind up with a `libgcc_s.so` that is missing various
unwinder features (see https://github.com/NixOS/nixpkgs/issues/213453
for the most problematic one).
Musl "cuts" link (2), or rather never creates it in the first place.
["Cancellation cleanup handling in musl has no relationship to C++
exceptions and unwinding... glibc implements cancellation as an
exception"](https://wiki.musl-libc.org/functional-differences-from-glibc.html#Thread-cancellation).
IMHO Musl made the smarter decision here. It is incredibly rare to
find a codebase that uses both POSIX thread cancellation *and* C++
exceptions. I have never seen a codebase that uses both *and*
expects them to be aware of each other, and I would be astonished if
one existed. Glibc paid an immense cost in complexity for something
nobody has ever used.
### Changes Made
This PR cuts link (3): instead of building libgcc_s.so as part of
gcc, we build it separately from gcc. Now there is a strict acyclic
graph of `buildInputs`:
```
gccWithoutTargetLibc
|
+--->glibc-nolibgcc
| |
| v
+--->libgcc
| |
| v
+--->glibc
| |
| v
+--->gcc
```
In other words, there's a simple linear `buildInputs` chain
`glibc-nolibgcc` `->` `libgcc` `->` `glibc` `->` `gcc` where all
four packages are compiled by (and therefore have as a
`(native)BuildInput`) `gccWithoutTargetLibc`.
`gccWithoutTargetLibc` and `glibc-nolibgcc` are strictly
bootstrapping artifacts; nothing else has them as a `buildInput` and
they shouldn't appear in the closure of any final deployment
packages. `glibc-nolibgcc` lacks `libgcc_s.so`, so it will segfault
if you try to use it with POSIX thread cancellation. Fortunately
all we need from it is (a) its headers (`lib.getDev`) and (b) to use
it in the `./configure` script for `libgcc`.
When translated over to the native bootstrap, `xgcc` takes the place
of `gccWithoutTargetLibc`, and the "first `glibc`" (we build two of
them) takes the place of `glibc-nolibgcc`. At that point our native
and cross bootstrap have the same overall architecture, and it
becomes possible to merge them (at last!)
[213453]: https://github.com/NixOS/nixpkgs/issues/213453
[238154]: https://github.com/NixOS/nixpkgs/pull/238154
[hardwired]: 7553d0fe29/pkgs/development/libraries/glibc/default.nix (L69-L88)
The way `nixpkgs` runs parallel `localedef` instances shares
`--prefix=/build` flag. As a result `localedef` processes
non-deterministically extend the file with new locales (hopefully
without data corruption):
https://github.com/NixOS/nixpkgs/issues/245360
Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
This commit causes glibc/info.nix to use the standard builder's
`buildPhase` so things like `preBuild`, `postBuild`,
`enableParallelBuilding`, etc work correctly.
This commit massively reduces the build latency for glibcLocales by
allowing it to build in parallel. This requires passing
`-j$NIX_BUILD_CORES` via the glibc-specific make variable
`PARALLELMFLAGS`.
This commit also fixes a preexisting bug where the glibcLocales
package would ignore `preBuild` and `postBuild`.
Our gcc_multi and glibc_multi expressions merge together a
32-bit-targeted and 64-bit-targeted gcc. However they do not thread
through the passthru.libgcc from these merged gccs.
This commit corrects that.
It also extends passthru.libgcc to allow a *list* rather than just a
single outpath.
Resolves part of #221891 (at least getting it back to the error
message it gave before).
with structuredAttrs lists will be bash arrays which cannot be exported
which will be a issue with some patches and some wrappers like cc-wrapper
this makes it clearer that NIX_CFLAGS_COMPILE must be a string as lists
in env cause a eval failure
Without the change a copy from freshly built `gcc` still retains a
reference even after `patchelf --remove-rpath` because `patchelf` does
not cleanup dynamic sprintgs section.
The change stubs the reference out and fixes build on bootstraps where
`gcc` is built before `glibc`.
Otherwise copy it from the default output. The difference is visible
when we build `glibc` with:
- `bootstrapTools` `gcc`: ${stdenv.cc.cc.out}/lib/libgcc_s.so.1 is used
- `nixpkgs` `gcc`: ${stdenv.cc.cc.lib}/lib/libgcc_s.so.1 is used
Noticed when experimented with multiple `gcc` rebuilds in bootstrap.
While at it killing `RUNPATH` reference to bootstrap `glibc`.
I don't think anyone sane is really using kernel < 3.10 nowadays.
(At least in a use case with glibc from nixpkgs.)
As another suspect with old kernels, Ubuntu seems to start at 3.13 now.
