Commit Graph

2583 Commits

Author SHA1 Message Date
Artturin
42c94d06fd stdenv: fix overriding with attrset when finalAttrs isn't used
```
nix-repl> (pkgs.htop.overrideAttrs { pname = "hello-overriden"; }).pname
error:
       … while evaluating a branch condition

         at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/lib/customisation.nix:86:7:

           85|     in
           86|       if builtins.isAttrs result then
             |       ^
           87|         result // {

       … while calling the 'isAttrs' builtin

         at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/lib/customisation.nix:86:10:

           85|     in
           86|       if builtins.isAttrs result then
             |          ^
           87|         result // {

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: attempt to call something which is not a function but a set

       at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/pkgs/stdenv/generic/make-derivation.nix:58:21:

           57|           f = self: super:
           58|             let x = f0 super;
             |                     ^
           59|             in
```
2023-07-03 21:34:50 +03:00
Randy Eckenrode
ebc1bcf409
swift-corelibs: don’t link against libcurl
swift-corelibs uses libcurl to implement `NSURLSession` in Foundation
via the symbols exported by CF. Foundation is not build on Darwin, and
these symbols are not exported by the system CoreFoundation.

By not linking against libcurl, this breaks a cycle between CF and
libcurl. That should allow libcurl to drop the patch disabling
linking against the SystemConfiguration and restore NAT64 support.

Unfortunately, the Darwin stdenv bootstrap still needs to build
dependencies that use `fetchFromGitHub`. While it can drop curl from the
final stdenv, it still needs to use it during the stdenv bootstrap.
2023-07-02 17:56:25 -04:00
Randy Eckenrode
a845397040
darwin.stdenv: refactor stdenv definition
In preparation for bumping the LLVM used by Darwin, this change
refactors and reworks the stdenv build process. When it made sense,
existing behaviors were kept to avoid causing any unwanted breakage.
However, there are some differences. The reasoning and differences are
discussed below.

- Improved cycle times - Working on the Darwin stdenv was a tedious
  process because `allowedRequisites` determined what was allowed
  between stages. If you made a mistake, you might have to wait a
  considerable amount of time for the build to fail. Using assertions
  makes many errors fail at evaluation time and makes moving things
  around safer and easier to do.
- Decoupling from bootstrap tools - The stdenv build process builds as
  much as it can in the early stages to remove the requirement that the
  bootstrap tools need bumped in order to bump the stdenv itself. This
  should lower the barrier to updates and make it easier to bump in the
  future. It also allows changes to be made without requiring additional
  tools be added to the bootstrap tools.
- Patterned after the Linux stdenv - I tried to follow the patterns
  established in the Linux stdenv with adaptations made to Darwin’s
  needs. My hope is this makes the Darwin stdenv more approable for
  non-Darwin developers who made need to interact with it. It also
  allowed some of the hacks to be removed.
- Documentation - Comments were added explaining what was happening and
  why things were being done. This is particular important for some
  stages that might not be obvious (such as the sysctl stage).
- Cleanup - Converting the intermediate `allowedRequisites` to
  assertions revealed that many packages were being referenced that no
  longer exist or have been renamed. Removing them reduces clutter and
  should help make the stdenv bootstrap process be more understandable.
2023-07-02 17:56:24 -04:00
uri
f829b24045 stdenv: fix stagesNative 2023-07-02 19:07:23 +10:00
Robert Hensing
1f5a0b0135
Merge pull request #239005 from Artturin/setorfunoverr
stdenv: let overrideAttrs accept attrset OR function
2023-07-01 15:27:23 +02:00
Artturin
0fdae31531 stdenv: let overrideAttrs accept attrset OR function
Makes overrideAttrs usable in the same way that `override` can be used.
It allows the first argument of `overrideAttrs` to be either a function
or an attrset, instead of only a function:

hello.overrideAttrs (old: { postBuild = "echo hello"; })
hello.overrideAttrs { postBuild = "echo hello"; }

Previously only the first example was possible.

Co-authored-by: adisbladis <adisbladis@gmail.com>
Co-authored-by: matthewcroughan <matt@croughan.sh>
2023-06-30 23:28:00 +03:00
github-actions[bot]
ffc0f1526c
Merge master into staging-next 2023-06-26 06:01:14 +00:00
Uri Baghin
6e51b1a92e
Merge pull request #235990 from uri-canva/fix-static
nixStatic: Fix darwin.
2023-06-26 15:22:01 +10:00
Robert Scott
db3e94c3b2 hardening flags: enable fortify3 by default 2023-06-25 11:50:46 +01:00
github-actions[bot]
ea69de970d
Merge master into staging-next 2023-06-23 18:01:07 +00:00
Artturin
d88b362808 stdenv: finish removal of deprecated string/null Flags
forgot to do it before 23.05 but it's ok
2023-06-22 20:37:06 +03:00
github-actions[bot]
fc5ef80184
Merge staging-next into staging 2023-06-22 12:02:08 +00:00
Artturin
ac031478cb stdenv: use lib.isX instead of typeOf Y == X
more readable
2023-06-22 12:31:52 +03:00
Uri Baghin
fd8e2ca0df stdenv: fix makeStaticDarwin not composing with stdenvNoCC 2023-06-22 15:33:54 +10:00
Adam Joseph
b9c1ae2a5f stdenv: eliminate duplicate gnu-config in extraNativeBuildInputs
Accidentally omitted from #237968.
2023-06-16 02:33:57 -07:00
Adam Joseph
3b8e3c1f3d stdenv: updateAutotoolsGnuConfigScriptsHook unconditionally
Unlike autoreconfHook, updateAutotoolsGnuConfigScriptsHook adds
almost no compilations.  Therefore, in the interest of building the
same source code on every platform wherever possible, let's
eliminate the conditional guards around
updateAutotoolsGnuConfigScriptsHook in stdenv.
2023-06-15 19:47:38 -07:00
toonn
b7d4899ad5
Merge pull request #234861 from reckenrode/corefoundation-rpath
swift-corelibs: set NIX_COREFOUNDATION_RPATH in a hook
2023-06-11 17:13:54 +02:00
toonn
782dbaf30f
Merge pull request #234859 from reckenrode/cctools-llvm
cctools-llvm: init at 11.1.0-973.0.1
2023-06-08 19:23:56 +02:00
Randy Eckenrode
df7487ff17
cctools-llvm: init at 11.1.0-973.0.1
cctools-llvm is a replacement for cctools that replaces as much of cctools with equivalents from LLVM that it can reasonably do. This was motivated by wanting to reduce dependencies on cctools, which are updated infrequently by upstream.

To provide a motivating example, the version of `strip` included in cctools cannot properly strip the archives in compiler-rt in LLVM 15. Paths are left to bootstrap tools, resulting in failed requisites checks in the final stdenv build. Since `strip` needs replaced, the opportunity was taken to replace other provided they are functional replacements.

Note: This has to be done in cctools (or some equivalent) because some derivations (noteably LLVM) use the bintools of the stdenv directly instead of going through the wrapper.

The following tools from LLVM are not used in this derivation:

* LLD - not fully compatible with ld64 yet and potentially too big of a change;
* libtool - not a drop-in replacement yet because it does not support linker passthrough, which is needed by xcbuild;
* lipo - crashes when running the LLVM test suite;
* install_name_tool - fails when trying to build swift-corefoundation; and.
* randlib - not completely a drop-in replacement, so leaving it out for now.

If other incompatabilities are found, the tools can be reverted or made conditional. For example, cctools `strip` is preferred on older versions of LLVM (which lack the compiler-rt issue) or when cctools itself is a new enough version because `llvm-strip` on LLVM 11 produces files that older verions of `codesign_allocate` cannot process correctly.

One final caveat/note: Some tools are not duplicated or linked from cctools-port. The names of the tools and which ones were linked was determined based on what is provided upstream in Xcode and is installed on macOS system.
2023-06-06 19:05:22 -04:00
Robert Hensing
dae6b98a90
Merge pull request #234883 from sternenseemann/nix-structured-attrs
stdenv/setup.sh: deal with Nix < 2.4 structured attrs
2023-05-31 19:41:58 +02:00
Randy Eckenrode
7c043234ec
darwin.stdenv: drop NIX_COREFOUNDATION_RPATH from preHook
Rely on swift-corefoundation to set this in its hook. That way, it will
be dropped from the `extraBuildInputs` during a cross build.
2023-05-30 13:28:34 -04:00
sternenseemann
1a29857b8a stdenv/setup.sh: deal with Nix < 2.4 structured attrs
Nix does not (as far it is documented) guarantee that NIX_ATTRS_*_FILE
is set, the only [documented] guarantee seems to be:

> […] made available to the builder via the file .attrs.json in the
> builder’s temporary directory.

This guarantee is of course affected by https://github.com/NixOS/nix/issues/6736,
so it seems to be prudent to fall back to the Nix 2.3 style ATTRS_*_FILE
env vars before defaulting to the expected location in case neither is
available.

See also:

- https://github.com/NixOS/nixpkgs/pull/214937#discussion_r1178101895
- https://github.com/nixos/nixpkgs/commit/afef6588e250

[documented]: https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-structuredAttrs
2023-05-29 23:12:24 +02:00
Weijia Wang
91b69f8555
Merge pull request #228714 from helsinki-systems/upd/gnugrep
[staging] gnugrep: 3.7 -> 3.11
2023-05-24 12:47:48 +03:00
github-actions[bot]
2881fdf82f
Merge staging-next into staging 2023-05-20 12:01:51 +00:00
figsoda
98b9e41f61 pkgs: fix typos 2023-05-19 22:31:04 -04:00
Sergei Trofimovich
c3bdae6ce5
Merge pull request #223861 from eliasnaur/no-nix-lib64-in-self-rpath
stdenv: don't set NIX_LIB*_IN_SELF_RPATH by default
2023-05-17 07:08:57 +01:00
Janne Heß
11c8f46850
gnugrep/stdenv: Fix PCRE support by replacing PCRE lib 2023-05-15 10:21:34 +02:00
Robert Hensing
9524f57dd5
Merge pull request #230666 from alyssais/inputDerivation-passAsFile
stdenv: fix inputDerivation with passAsFile
2023-05-09 20:36:29 +02:00
Alyssa Ross
a295c4566e
stdenv: fix inputDerivation with passAsFile
passAsFile passes the values of Nix bindings to the builder as
files, so if those values contained references, they wouldn't end up
in the inputDerivation output.  To fix that, append the contents of
every such passed file to the output.

We only have shell builtins in this derivation, so we can't use cat.
The only way I know of appending the contents of one file to another
using only shell builtins is as I've done here, but it requires
putting the contents of the file on echo's argv.  This might end up
causing problems with large files.  Regardless, I think we should try
this, as a failure is better than silently producing an incorrect
result like the previous behavior.
2023-05-09 14:00:57 +00:00
Alyssa Ross
cd0bccbbe0
stdenv: fix inputDerivation with allowed refs
Same motivation as for the disallowed references.
2023-05-08 12:02:59 +00:00
github-actions[bot]
bf19e21376
Merge master into staging-next 2023-05-04 12:01:18 +00:00
Weijia Wang
e2adc3a3a2 bootstrap-tools-cross: add loongarch64-linux 2023-05-04 09:51:11 +03:00
github-actions[bot]
489a4937eb
Merge master into staging-next 2023-05-03 10:25:04 +00:00
Sergei Trofimovich
ba63ea9268
Merge pull request #229547 from trofi/stdenv-ca-false
stdenv: do not pass `__contentAddressed = false` as environment variable
2023-05-03 11:23:56 +01:00
github-actions[bot]
e7d9c8250f
Merge master into staging-next 2023-05-03 00:02:12 +00:00
Sergei Trofimovich
72d71dd641 stdenv: do not pass __contentAddressed = false as environment variable
`nix-2.4+` automatically filters `__contentAddressed` out of the
environment. But not `nix-2.3`. This make `.drv` to differ between
unset and `__contentAddressed = false` derivations.

This change makes them equal by filtering out `__contentAddressed`
unless it's set to `true`.
2023-05-02 22:57:54 +01:00
Weijia Wang
658f507622
Merge pull request #229289 from wegank/loongarch64-cross-bootstrap
pkgsCross.loongarch64-linux.freshBootstrapTools.bootstrapFiles: fix build
2023-05-02 22:39:14 +03:00
github-actions[bot]
bdb79914d6
Merge master into staging-next 2023-05-02 18:01:20 +00:00
Weijia Wang
c78d0d9d39 pkgsCross.loongarch64-linux.freshBootstrapTools.bootstrapFiles: fix build 2023-05-02 20:25:05 +03:00
Weijia Wang
e6e3e4da34
Merge pull request #229372 from wegank/loongarch64-native-bootstrap
stdenv: always update config script on loongarch64-linux
2023-05-02 18:49:43 +03:00
github-actions[bot]
0ee4180a08
Merge staging-next into staging 2023-05-02 00:03:00 +00:00
Weijia Wang
b017e29705 stdenv: always update config script on loongarch64-linux 2023-05-02 01:02:57 +03:00
Artturi
ea8e21f9fa
Merge pull request #228759 from Artturin/splitfactoroutmeta 2023-05-02 00:46:31 +03:00
Emily Trau
2ad68a7d90 stdenv: factor out meta attr augmentation for reusability
stdenv: check if separating commonMeta from assertValidity fixes laziness issue

```
error: cannot coerce null to a string

at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/mr-est/ofborg-evaluator-3/pkgs/development/coq-modules/coq-lsp/default.nix:34:60:

    33|     homepage = "https://github.com/ejgallego/coq-lsp";
    34|     changelog = "https://github.com/ejgallego/coq-lsp/blob/${defaultVersion}/CHANGES.md";
```
2023-05-01 20:18:11 +03:00
github-actions[bot]
cff4be2a2b
Merge staging-next into staging 2023-05-01 12:01:54 +00:00
Alyssa Ross
5130c4f4ef freshBootstrapTools: enable musl on RISC-V
musl now supports RISC-V.  Let's centralise musl availability checks
in musl.meta.platforms, so we don't have to keep cleaning up ad-hoc
checks like this all over the tree.
2023-05-01 11:25:04 +00:00
Alyssa Ross
2839c94f1e pkgsLLVM.stdenv: use clangNoLibc when libc is null
clangNoLibc always uses LLVM bintools, so it still has the useLLVM
semantics.
2023-04-28 10:01:22 +00:00
Adam Joseph
bc024e10ef
Merge pull request #228281 from amjoseph-nixpkgs/pr/bootstrapFiles/cross/fix
stdenvBootstrapTools: drop no-longer-needed, now-broken cross clause
2023-04-27 03:58:30 +00:00
Adam Joseph
38f41a27b4 stdenvBootstrapTools: drop no-longer-needed, now-broken cross clause
The comment stating that "GCC has certain things built in
statically" is no longer true as of
https://github.com/NixOS/nixpkgs/pull/209870.  Moreover, as pointed
out in https://github.com/NixOS/nixpkgs/pull/228130 this
optionalString block no longer builds for
pkgsCross.*.freshBootstrapTools.

It is safe to drop this clause now.
2023-04-26 00:02:00 -07:00
Weijia Wang
91bc06b6d7 stdenv: avoid -p flag for strip on aarch64-darwin 2023-04-24 10:33:11 +03:00
Sergei Trofimovich
e7eb209fed
stdenv: avoid -p flag for strip when boostrapping x86_64-darwin
The stdenv wouldn't build with it, as
compiler-rt-libc-11.1.0/lib/darwin/libclang_rt.*_osx.a
retained reference to SDKs (which we forbid for final stdenv).

Assigned authorship to Trofi; I just bisected and added condition.
https://github.com/NixOS/nixpkgs/pull/224669#issuecomment-1518225496
2023-04-22 16:05:35 +02:00
github-actions[bot]
6176f16de2
Merge staging-next into staging 2023-04-14 12:02:03 +00:00
Sandro
b04d4bad27
Merge pull request #216992 from SuperSandro2000/stdenvNative-fix-eval
{bintools,cc}-wrapper: don't fallback to version = null
2023-04-14 11:22:20 +02:00
github-actions[bot]
ae00b7fb14
Merge staging-next into staging 2023-04-13 00:02:47 +00:00
Sandro Jäckel
b0ee79f9fd
mkDerivation: show meaningful error when version is set to null
instead of `Cannot coerce null to string`
2023-04-12 22:08:36 +02:00
Artturi
c88044fe0b
Merge pull request #225929 from Artturin/fixadapters2 2023-04-12 23:04:31 +03:00
Artturin
67a84d1428 stdenvAdapters: fix preservation of env.NIX_CFLAGS_COMPILE
we have managed to migrate to NIX_CFLAGS_COMPILE to the env attrset well
enough that we don't need to support having it toplevel. mkDerivation
will throw if there's a attr in both env and toplevel so no need to
worry about that
2023-04-12 21:27:34 +03:00
github-actions[bot]
18530e57da
Merge staging-next into staging 2023-04-12 00:02:52 +00:00
github-actions[bot]
47157337bf
Merge master into staging-next 2023-04-12 00:02:18 +00:00
Artturin
9adf83737f stdenvAdapters: fix preservation of env 2023-04-11 21:25:52 +03:00
github-actions[bot]
9dd2031c08
Merge staging-next into staging 2023-04-06 16:11:47 +00:00
Adam Joseph
3778e07994 stdenv: fix pkgsMusl
I broke `pkgsMusl` with #209870.

Something odd is happening with `xgcc` (the temporary compiler that
should be used only to compile `gcc`, although we are using it to
compile a temporary `patchelf` too) and `libstdc++`.

The temporary fix in this commit is to use `-static-libstdc++` for
the ephemeral `patchelf` built by `xgcc`.  It isn't pretty, but it
appears to work.

Incorporates:

- https://github.com/NixOS/nixpkgs/pull/224945
2023-04-06 03:16:31 -07:00
Adam Joseph
1847b1bc15 stdenv/linux: fix patchelf confusion
The stage before `xgcc` creates the first compiled patchelf
(i.e. not from bootstrapFiles).

The `xgcc` stage was inadvertently switching *back* to using the
patchelf *from* the bootstrapFiles.

The first commit in this PR adds self-checking comments (assertions)
to make it clear where each stage's patchelf comes from.

The second commit fixes the bug, and updates the self-checking
comments.
2023-04-05 22:18:27 -07:00
Adam Joseph
2abf285e2b stdenv/linux: add is{From,BuiltBy} assertions for patchelf 2023-04-05 22:18:22 -07:00
Sergei Trofimovich
c2e14a73e6 xgcc: avoid libc.so mix between gcc and binutils-ld linker plugin
Without the change when I attempt to built `nixpkgs` with weekly
`gcc-13` (it pulls in `flex` as a build input`) I am getting build
failure related to glibc mix caused by glibc loading:

    ...-binutils-patchelfed-ld-2.40/bin/ld: ...-xgcc-13.0.0/libexec/gcc/x86_64-unknown-linux-gnu/13.0.1/liblto_plugin.so:
    error loading plugin: ...-bootstrap-tools/lib/libpthread.so.0: undefined symbol: __libc_vfork, version GLIBC_PRIVATE

The change disables LTO plugin entirely to avoid loading of `glibc` mix.
2023-04-05 07:43:22 +01:00
Bernardo Meurer
f1f6ca8bcd
Merge pull request #209870 from amjoseph-nixpkgs/pr/stdenv/external-gcc-bootstrap 2023-04-03 08:19:03 -07:00
Adam Joseph
96588eb3de gcc: add common/checksum.nix
This commit adds `gcc/common/checksum.nix`, which contains code
common to both gcc11 and gcc12, implementing the `enableChecksum`
feature.

When gcc's built-in bootstrap (`--enable-bootstrap`) is used, gcc
compiles itself three times and compares a hash of the unlinked `.o`
files from the second and third compilation.  The
`enableChecksum=true` parameter performs the same comparison as part
of the `postInstall` phase.

Notably, `enableChecksum=true` can be used with `enableBootstrap=false`.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2023-04-02 13:49:53 -07:00
Adam Joseph
fed2300bea unpack-bootstrap-tools.sh: patchelf libgcc_s.so.1 2023-04-02 13:49:53 -07:00
Adam Joseph
d7fe0a5548 make-bootstrap-tools.nix: use a patchelf built with -static-{libgcc,libstdc++}
Our bootstrap-files unpacker has always relied on a lot of unstated
assumptions, one of them being that every library has a DT_NEEDED
for librt.so, so patchelf'ing something into the RUNPATH into
librt.so means that it will be searched for every library load in
all of the bootstrap-files.

Unfortunately that assumption is not true for libgcc.

This causes problems, because patchelf links against libgcc (and
against libstdc++, which links against libgcc).  So we can't use
patchelf on libgcc, because it needs libgcc, so patchelf doesn't
work until libgcc is patchelfed.

The robust solution here is to use static linking for the copy of
patchelf that is shipped with the bootstrap-files.  We don't have to
go all the way to a statically linked libc; just -static-libgcc and
-static-libstdc++ are enough to break the circular dependency.
2023-04-02 13:49:53 -07:00
Adam Joseph
86ca0faff7 make-bootstrap-tools.nix: cp libgcc_s without -d
We do not want to preserve the symlinks from libgcc_s, since they
point to another outpath.  We want to copy from that outpath rather
than link to it.
2023-04-02 13:49:53 -07:00
Adam Joseph
c6bd37a691 make-bootstrap-tools.nix: ship libisl.so
Now that we've dropped the
gcc-links-statically-to-lib{isl,mpfr,mpc,gmp} hack, our gcc needs
libisl.so.  Let's add it to the bootstrap-files.
2023-04-02 13:49:53 -07:00
Adam Joseph
7553d0fe29 stdenv: Nix-driven bootstrap of gcc
#### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like https://github.com/NixOS/nixpkgs/pull/132343

 #### Incorporates

- https://github.com/NixOS/nixpkgs/pull/210004
- https://github.com/NixOS/nixpkgs/pull/36948 (unreverted)
- https://github.com/NixOS/nixpkgs/pull/210325
- https://github.com/NixOS/nixpkgs/pull/210118
- https://github.com/NixOS/nixpkgs/pull/210132
- https://github.com/NixOS/nixpkgs/pull/210109
- https://github.com/NixOS/nixpkgs/pull/213909
- https://github.com/NixOS/nixpkgs/pull/216136
- https://github.com/NixOS/nixpkgs/pull/216237
- https://github.com/NixOS/nixpkgs/pull/210019
- https://github.com/NixOS/nixpkgs/pull/216232
- https://github.com/NixOS/nixpkgs/pull/216016
- https://github.com/NixOS/nixpkgs/pull/217977
- https://github.com/NixOS/nixpkgs/pull/217995

 #### Closes

- Closes #108305
- Closes #108111
- Closes #201254
- Closes #208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: https://github.com/NixOS/nixpkgs/pull/210004
[versioned directory]: https://github.com/NixOS/nixpkgs/pull/209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: 5e4ec82181/gnu/packages/gcc.scm (L253)
[mentioned]: https://github.com/NixOS/nixpkgs/pull/210112#issuecomment-1379608483
[crisis]: https://github.com/NixOS/nixpkgs/issues/108305
[foreign]: https://github.com/NixOS/nixpkgs/pull/170857#issuecomment-1170558348
[static lib{mpfr,mpc,gmp,isl}.a hack]: 2f1948af9c/pkgs/stdenv/linux/default.nix (L380)
2023-04-02 13:49:41 -07:00
Adam Joseph
70d34589bd stdenv/linux: factor out commonGccOverrides
This commit has no effect on eval.  It simply factors out a common
subexpression.
2023-04-02 03:04:20 -07:00
Adam Joseph
e8b10284f3 linux/default.nix: use mipsel.nix instead of longson.nix for mips32
Right now our bootstrapFiles-selecting algorithm uses the
`loongson2f.nix` bootstrapFiles (which were not built by Hydra).

These bootstrapFiles don't work anymore.  They were added in 2010 by
40405d03ac.

This commit causes mipsel-linux native builds to use the Hydra-built
bootstrap files from this PR instead:

  https://github.com/NixOS/nixpkgs/pull/183487
2023-04-01 23:25:25 -07:00
Elias Naur
88fd6601d8 stdenv: remove the NIX_LIB*_IN_SELF_RPATH environment variables
The NIX_LIB64|32_IN_SELF_RPATH environment variables control whether
to add lib64 and lib32 to rpaths. However, they're set depending
on the build paltform, not the target platform and thus their values
are incorrect for for cross-builds.

On the other hand, setting them according to the build platform introduce
pointless differences in build outputs; see #221350 for details.

This change fixes the issues by boldly removes the NIX_LIB*_IN_SELF_RPATH
facility altogether, in the hope that it is no longer necessary. They
were introduced in 2009, long before nixpkgs had good support for
cross-builds.

Fixes #221350
2023-03-31 10:47:16 -06:00
Vladimír Čunát
12dd95fbb1
Merge branch 'master' into staging-next 2023-03-24 09:07:41 +01:00
Adam Joseph
13507da345 check-meta.nix: fix self-contradictory error messages
See https://github.com/NixOS/nixpkgs/pull/222792#pullrequestreview-1356114111

You can't just `lib.filter _ lib.systems.all` -- that throws away
important information, leading to nixpkgs disagreeing with itself
like this:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is only supported on ... x86_64-linux but not on requested x86_64-linux, refusing to evaluate.
```

After:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is not available on the requested hostPlatform:
         hostPlatform.config = "x86_64-unknown-linux-musl"
         package.meta.platforms = [
           "aarch64-linux"
           "armv5tel-linux"
           "armv6l-linux"
           "armv7a-linux"
           "armv7l-linux"
           "i686-linux"
           "m68k-linux"
           "microblaze-linux"
           "microblazeel-linux"
           "mipsel-linux"
           "mips64el-linux"
           "powerpc64-linux"
           "powerpc64le-linux"
           "riscv32-linux"
           "riscv64-linux"
           "s390-linux"
           "s390x-linux"
           "x86_64-linux"
         ]
         package.meta.badPlatforms = [
           {
             isStatic = true;
             parsed = { };
           }
         ]
       , refusing to evaluate.
```
2023-03-24 00:15:57 -07:00
Weijia Wang
b46c97e1dc stdenv: generalise showPlatforms 2023-03-23 23:47:11 +02:00
github-actions[bot]
307b719414
Merge master into staging-next 2023-03-23 18:01:20 +00:00
Weijia Wang
2a7e1e0228 stdenv: fix error with patterned platforms 2023-03-23 17:27:49 +02:00
Sandro
979c6c1fe4
Merge pull request #213780 from SuperSandro2000/check-meta-platform
stdenv: show supported and requested platforms when check meta fails
2023-03-23 15:01:34 +01:00
github-actions[bot]
455127ad5e
Merge master into staging-next 2023-03-16 18:01:20 +00:00
Bernardo Meurer
6e55733359
Merge pull request #219747 from Stunkymonkey/deprecate-isNull 2023-03-16 11:10:22 -03:00
Martin Weinelt
19680e9902
Merge pull request #217568 from trofi/stdenv-parallel-install
stdenv/generic/setup.sh: enable parallel installs for parallel builds
2023-03-15 17:10:19 +00:00
Martin Weinelt
ef91384e6f Merge remote-tracking branch 'origin/master' into staging-next 2023-03-10 13:09:25 +01:00
Bernardo Meurer
e9cf2d1a41
Merge pull request #188334 from amjoseph-nixpkgs/pr/bootstrapFiles/mips64n32 2023-03-09 21:21:15 -05:00
Adam Joseph
49878856e6 https://github.com/NixOS/nixpkgs/pull/188334#issuecomment-1445425412 2023-03-07 02:11:30 -08:00
Felix Buehler
d10e69c86b treewide: deprecate isNull
https://nixos.org/manual/nix/stable/language/builtins.html#builtins-isNull
2023-03-06 22:40:04 +01:00
Martin Weinelt
5aeab34845
Merge pull request #218301 from rrbutani/fix/separate-debuginfo-with-lld
Fix `separate-debug-info` with lld
2023-03-04 00:43:26 +00:00
Weijia Wang
5028ec96c9
Merge pull request #202347 from stephank/feat/bootstrap-aarch64-darwin
stdenvBootstrapTools: native aarch64-darwin build
2023-03-03 20:05:28 +02:00
Vladimír Čunát
763470bbb0
Merge #207135: stdenv: aarch64-linux: gcc9 -> gcc12
...into staging
Also merge the commit that was referenced as the base for the build
of the new bootstrap tools (although others would give the same).
2023-03-01 12:54:52 +01:00
Sergei Trofimovich
69cf5181c3 stdenv/generic/setup.sh: enable parallel installs by default
The primary motivating example is openssl:

Before the change full package build took 1m54s minutes.
After the change full package build takes 59s.

About a 2x speedup.

The difference is visible because openssl builds hundreds of manpages
spawning a perl process per manual in `install` phase. Such a workload
is very easy to parallelize.

Another example would be `autotools`+`libtool` based build system where
install step requires relinking. The more binaries there are to relink
the more gain it will be to do it in parallel.

The change enables parallel installs by default only for buiilds that
already have parallel builds enabled. There is a high chance those build
systems already handle parallelism well but some packages will fail.

Consistently propagated the enableParallelBuilding to:
- cmake (enabled by default, similar to builds)
- ninja (set parallelism explicitly, don't rely on default)
- bmake (enable when requested)
- scons (enable when requested)
- meson (set parallelism explicitly, don't rely on default)
- waf (set parallelism explicitly, don't rely on default)
- qmake-4/5/6 (enable by default, similar to builds)
- xorg (always enable, similar to builds)
2023-02-26 22:02:09 +00:00
Weijia Wang
f05b5d4054 stdenv: aarch64-linux: gcc9 -> gcc12
Hydra job building them: https://hydra.nixos.org/build/208909151

The bootstrap files can be reproduced on the commit 21ec906463, e.g. by:
  cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
    file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
  nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
  nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
    sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=

You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
2023-02-26 23:26:30 +02:00
Rahul Butani
d364ee8d13
mkDerivation: do not disable separateDebugInfo on LLVM stdenvs
This was disabled here: b86e62d30d (diff-282a02cc3871874f16401347d8fadc90d59d7ab11f6a99eaa5173c3867e1a160)

h/t to @teh: b86e62d30d (commitcomment-77916294)
for pointing out that the failure that @matthewbauer was
seeing was caused by the `separate-debug-info.sh` `build-id` length
requirement that #146275 will relax

`lld` has had `--build-id` support dating back to LLVM4: https://reviews.llvm.org/D18091

This predates every `llvmPackages_` version currently in nixpkgs (and
certainly every version actually still used in `useLLVM` stdenvs) so
with the previous commit (asking `ld` for sufficiently long SHA1 hashes)
I think we can safely enable `separateDebugInfo` when using LLVM
bintools.
2023-02-25 12:49:40 -06:00
github-actions[bot]
3cdd771820
Merge staging-next into staging 2023-02-23 18:01:49 +00:00
Alyssa Ross
52c286ee5b
Merge remote-tracking branch 'origin/master' into staging-next
Conflicts:
	pkgs/development/libraries/pmdk/default.nix
2023-02-23 13:51:34 +00:00
Artturin
f9fdf2d402 treewide: move NIX_CFLAGS_COMPILE to the env attrset
with structuredAttrs lists will be bash arrays which cannot be exported
which will be a issue with some patches and some wrappers like cc-wrapper

this makes it clearer that NIX_CFLAGS_COMPILE must be a string as lists
in env cause a eval failure
2023-02-22 21:23:04 +02:00
Sergei Trofimovich
3057968396
Merge pull request #210019 from amjoseph-nixpkgs/pr/stdenv/verify-comments
express #208478 as assertions
2023-02-21 19:38:58 +00:00
Adam Joseph
d7aad24531 express #208478 as assertions
PR #208478 added a lot of documentation about which packages were
rebuilt in each stage of the stdenv bootstrap.  However nothing
checks that these comments agree with reality; they can bitrot over
time.  This PR rewrites those comments as assertions, so they cannot
bitrot.

This conversion did expose some ambiguity in our scheme for naming
the stages.   Suppose that `pkgs.stdenv.name=="stdenv-stage4", then
which of these is "the stage4 coreutils"?

```
pkgs.coreutils
pkgs.stdenv.__bootPackages.coreutils
```

The choice is arbitrary, and both choices have confusing corner
cases.  We should revisit this at some point.
2023-02-20 23:20:30 -08:00
Artturi
ee54eb7d21
Merge pull request #216383 from Artturin/bintoolswrappermold 2023-02-17 19:32:06 +02:00
Artturin
299a7bd35e stdenvAdapters: add useMoldLinker 2023-02-17 06:35:24 +02:00
Robert Scott
0eedcfc3f4
Merge pull request #212498 from risicle/ris-fortify3
hardening flags: add `FORTIFY_SOURCE=3` support
2023-02-16 21:19:30 +00:00
github-actions[bot]
c4fe2133de
Merge staging-next into staging 2023-02-15 06:01:44 +00:00
github-actions[bot]
58cfebde3e
Merge staging-next into staging 2023-02-15 00:02:50 +00:00
Felix Buehler
bc3d5934d7 treewide: use lib.optionals 2023-02-14 19:11:59 +01:00
Felix Buehler
cdb39a86e0 treewide: use optionalString 2023-02-13 21:52:34 +01:00
github-actions[bot]
29c2637dda
Merge staging-next into staging 2023-02-13 12:02:05 +00:00
github-actions[bot]
fd8daee7c2
Merge master into staging-next 2023-02-13 12:01:30 +00:00
Weijia Wang
7c026bae62
stdenv: aarch64-linux: update the bootstrap tools (incl. busybox)
Hydra job building them: https://hydra.nixos.org/build/208909151

The bootstrap files can be reproduced on the parent commit, e.g. by:
  cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
    file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
  nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
  nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
    sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=

You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
2023-02-12 10:09:35 +01:00
Stéphan Kochen
c3693fbfd5 stdenvBootstrapTools: native aarch64-darwin build 2023-02-11 20:11:55 +01:00
Artturin
84e37a10ec stdenv: allow propagating propagated dependencies separately from the
fixup phase

for makeSetupHook
2023-02-07 21:00:18 +02:00
Robert Hensing
059bd43546 make-derivation.nix: Support inputDerivation on disallowedReferences 2023-02-07 18:58:13 +01:00
github-actions[bot]
99cce0e1f1
Merge staging-next into staging 2023-02-07 06:02:00 +00:00
github-actions[bot]
41e5bd55d5
Merge master into staging-next 2023-02-07 06:01:28 +00:00
figsoda
42d1d60a92
Merge pull request #206773 from SuperSandro2000/cleanup-unused-bindings
treewide: cleanup some unused bindings
2023-02-06 20:07:50 -05:00
Sandro Jäckel
50e0012f9d
treewide: cleanup some unused bindings 2023-02-07 01:36:15 +01:00
Artturi
aa7d2fe7dc
Merge pull request #214937 from hercules-ci/support-NIX_ATTRS-envs 2023-02-06 20:59:10 +02:00
github-actions[bot]
36822caa75
Merge staging-next into staging 2023-02-06 18:02:09 +00:00
github-actions[bot]
f876e1f1e9
Merge master into staging-next 2023-02-06 18:01:37 +00:00
John Ericson
6d0b3086f7
Merge pull request #214304 from obsidiansystems/pkg-config-meta
meta.pkgConfigModules: Init convention
2023-02-06 11:44:29 -05:00
Robert Hensing
afef6588e2 stdenv/setup.sh: Allow NIX_ATTRS_{JSON,SH}_FILE to be set correctly by Nix 2023-02-06 14:17:59 +01:00
github-actions[bot]
dfee1a3150
Merge staging-next into staging 2023-02-05 00:03:15 +00:00
github-actions[bot]
bc833a50cc
Merge master into staging-next 2023-02-05 00:02:31 +00:00
superherointj
2c77d453e3
Merge pull request #213633 from alyssais/mkDerivation-meson-exotic
stdenv.mkDerivation: fix meson for some archs
2023-02-04 18:43:43 -03:00
John Ericson
6e4a1b18d9 meta.pkgConfigModules: Init convention
See docs.

Follow-up work:

- Existing packages should be converted

- `defaultPkgConfigPackages` should assert on `meta.pkgConfigModules`
  and let `tests.pkg-config` alone test the build results.

CC @sternenseemann

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-02-03 09:37:31 -05:00
Artturi
dcc7df7fe6
Merge pull request #211685 from Artturin/splicingstuff1-split 2023-02-03 12:49:08 +02:00
Artturi
fa5f0d6bd0
Merge pull request #211783 from R-VdP/fix_disallowed_references
stdenv: don't include drvs in disallowedRefs as build-time deps.
2023-01-31 20:51:03 +02:00
Sandro Jäckel
0a8e692d29
stdenv: show supported and requested platforms when check meta fails 2023-01-31 14:43:47 +01:00
github-actions[bot]
33d4318fcc
Merge staging-next into staging 2023-01-31 00:03:07 +00:00
github-actions[bot]
dd1ff149da
Merge master into staging-next 2023-01-31 00:02:31 +00:00
Alyssa Ross
b6304bf11d
stdenv.mkDerivation: fix meson for some archs
platform.uname.processor seems to be what we want in many more cases
than what we were using before — it does the right thing for aarch64,
x86_64, riscv32, riscv64, mips, mips64, powerpc, and powerpc64 (the
latter three of which were broken before).

This fixes cross-compilation of systemd for PowerPC/POWER platforms.
2023-01-30 19:42:19 +00:00
R-VdP
51acc6245e
stdenv: don't include drvs in disallowedRefs as build-time deps.
Derivations listed as disallowedReferences or disallowedRequisites,
currently end up as build-time dependencies.
This is problematic since the disallowed derivations will be built by nix as
build-time dependencies, while those derivations might take a very long time
to build, or might not even build successfully on the platform used.
However, in order to scan for disallowed references in the final output,
knowing the out path is sufficient, and the out path can be calculated from
the derivation without needing to build it, saving time and resources.

While the problem is less severe for allowedReferences and allowedRequisites,
since we want the derivation to be built eventually, we would still like to
get the error early and without having to wait while nix builds a derivation
that might not be used (e.g. if we listed the wrong one).
2023-01-30 17:58:05 +01:00
Sergei Trofimovich
b3a97d7eb4
Merge pull request #211910 from tejing1/srcs-name-collision
stdenv: Improve error from `stdenv` when `srcs` is used with colliding directories
2023-01-29 12:04:42 +00:00
Sergei Trofimovich
0ba9da441d
Merge pull request #211126 from trofi/binutils-update
binutils: 2.39 -> 2.40
2023-01-29 09:50:57 +00:00
Jeff Huffman
183939da54
improve error when srcs is used with directories with the same post-hash name 2023-01-29 04:27:24 -05:00
Sergei Trofimovich
e1ef521cff binutils: 2.39 -> 2.40
A few potentially disruptive changes:

- binutils does not embed ${binutils-unwrapped}/lib as a default library
  search path anymore. This will cause link failures for -lbfd -lopcodes
  users that did not declare their dependency on those libraries. They
  will need to add `libbfd` and `libopcodes` attributes to build inputs.

- `libbfd` and `libopcodes` attributes now just reference
  `binutils-unwrapped.{dev,lib}` pair of attributes without patching
  `binutils` build system.

We don't patch build system anymore and use multiple outputs out of
existing `binutils` build. That makes the result more maintainable: no
need to handle ever growing list of dependencied of `libbfd`. This time
new addition was `libsframe`.

To accomodate `out` / `lib` output split I had to remove `lib` -> `bin`
backreference by removing legacy lookup path for plugins.

I also did not enable `zstd` just yet as `nixpkgs` version of `zstd`
package pulls in `cmake` into bootstrap sequence.

Changes: https://lists.gnu.org/archive/html/info-gnu/2023-01/msg00003.html
2023-01-27 23:16:45 +00:00
Adam Joseph
778419b9e6 Revert "lib/meta.nix: platformMatch: allow predicate functions"
This reverts commit b7d097438b.
2023-01-27 01:49:39 -08:00
github-actions[bot]
c47e4b69a3
Merge master into staging-next 2023-01-25 00:02:20 +00:00
Robert Scott
c09e1fa406 gcc: mark hardeningUnsupportedFlags fortify3 for all but gcc 12 2023-01-24 21:52:12 +00:00
Robert Scott
3d453e2aee mkDerivation: add support for fortify3 hardening flag 2023-01-24 21:52:10 +00:00
Alyssa Ross
2ae30c9f45 llvmPackages: use libcxxrt on FreeBSD
FreeBSD doesn't use LLVM's cxxabi implementation, for backwards
compatibility reasons.  Software expects the libcxxrt API when
building on FreeBSD.  This fixes the build of
pkgsCross.x86_64-freebsd.boost.
2023-01-24 21:18:39 +00:00
github-actions[bot]
feb2240b37
Merge master into staging-next 2023-01-22 18:01:03 +00:00
Adam Joseph
b7d097438b lib/meta.nix: platformMatch: allow predicate functions 2023-01-22 00:27:19 -08:00
Guillaume Girol
4ce9749f74 Merge remote-tracking branch 'origin/staging' into staging-nativeCheckInputs 2023-01-21 17:18:51 +01:00
Artturin
d29b1ecb89 make-bootstrap-tools.nix: fix for wrapped gzip 2023-01-21 17:19:38 +02:00
Guillaume Girol
f39abbc350 mkDerivation: introduce native checkInputs and nativeCheckInputs
When strictDeps is set, only nativeCheckInputs are added to PATH and
only checkInputs can be linked against. See https://github.com/NixOS/nixpkgs/issues/161570
2023-01-21 16:42:09 +01:00
Guillaume Girol
90c78aee6c Merge branch 'nativeCheckInputs' into staging-nativeCheckInputs 2023-01-21 12:00:00 +00:00
Sergei Trofimovich
b9b1d958d0 Merge remote-tracking branch 'origin/staging-next' into staging
Conflicts:
	pkgs/development/libraries/qt-6/modules/qtbase.nix
	pkgs/stdenv/linux/make-bootstrap-tools.nix
2023-01-20 21:56:57 +00:00
Alyssa Ross
0ae87d514f treewide: add names to all setup hooks 2023-01-19 15:00:36 +00:00
Andrew Childs
68f1182b65 stdenv: don't clobber useArray and type in {prepend,append}ToVar
Some other packages, for example ruby gems via buildRubyGem, use a
variable called "type" internally, which is overwritten here and
causes failures like:

    failure: $gempkg path unspecified

Fix for changes in 11c3127e38.
2023-01-18 11:59:47 +09:00
Artturin
e525ae1e1e stdenv: disable shellcheck rule SC2068 & SC1091
this is intentional to support both structuredAttrs and non

In pkgs/stdenv/generic/setup.sh line 614:
for pkg in ${depsBuildBuild[@]} ${depsBuildBuildPropagated[@]}; do
           ^------------------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.

In pkgs/stdenv/generic/setup.sh line 521:
    local varRef="$varVar[$((targetOffset - hostOffset))]"
                  ^-- SC1087 (error): Use braces when expanding arrays, e.g. ${array[idx]} (or ${var}[.. to quiet).
2023-01-16 23:54:14 +02:00
Artturin
f2c27018f0 stdenv: fix SC2242
exit -1 == exit 255 but we don't have a reason to use 255

In pkgs/stdenv/generic/setup.sh line 518:
    (( hostOffset <= targetOffset )) || exit -1
                                             ^-- SC2242 (error): Can only exit with status 0-255. Other data should be wri
tten to stdout/stderr.
2023-01-16 23:03:21 +02:00
sternenseemann
8e35002bc2
make-bootstrap-tools.nix: don't pull in pkgs.glibc in test (#210038)
`builtins.baseNameOf` retains any string context, causing the test
derivation to incorrectly depend on `pkgs.glibc`. All we really want is
to know what the dynamicLinker is called, but we don't need it to be
present in store.

Thanks to Adam Joseph for spotting this.
2023-01-16 19:29:40 +01:00
Artturin
e58785bf41 stdenv: disable shellcheck rule SC2048
we use [*] to support structuredAttrs and non

In pkgs/stdenv/generic/setup.sh line 1542:
    for curPhase in ${phases[*]}; do
                    ^----------^ SC2048 (warning): Use "${array[@]}" (with quotes) to prevent whitespace problems.
2023-01-16 08:23:04 +02:00
Artturin
4db439c599 stdenv: disable shellcheck rules
In pkgs/stdenv/generic/setup.sh line 101:
        source "$hookName"
               ^---------^ SC1090 (warning): ShellCheck can't follow non-constant source. Use a directive to specify location.

In pkgs/stdenv/generic/setup.sh line 166:
            mkdir -p "$out/nix-support"
                      ^--^ SC2154 (warning): out is referenced but not assigned.

In pkgs/stdenv/generic/setup.sh line 407:
PATH=
^--^ SC2123 (warning): PATH is the shell search path. Use another name.

In pkgs/stdenv/generic/setup.sh line 452:
declare -a pkgBuildAccumVars=(pkgsBuildBuild pkgsBuildHost pkgsBuildTarget)
           ^---------------^ SC2034 (warning): pkgBuildAccumVars appears unused. Verify use (or export if used e
xternally).
because pkgBuildAccumVars is used

In pkgs/stdenv/generic/setup.sh line 235:
        nameref="$* ${nameref-}"
        ^-----^ SC2178 (warning): Variable was used as an array but is now assigned a string.
because we theres a useArray conditional
2023-01-16 08:01:38 +02:00
Artturin
0417f953e2 stdenv: fix SC2004 & SC2086
SC2004 (style): $/${} is unnecessary on arithmetic variables.
SC2086 (info): Double quote to prevent globbing and word splitting.
2023-01-16 07:54:19 +02:00
Artturin
0e8263ce73 stdenv: fix SC2223
In pkgs/stdenv/generic/setup.sh line 36:
    : ${outputs:=out}
      ^-------------^ SC2223 (info): This default assignment may cause DoS due to globbing. Quote it.
2023-01-16 07:54:19 +02:00
Artturin
856f3a46b2 stdenv: drop remove unnecessary env var
and the associated obsolete functions

support for log nesting was removed in 2017 6669a3b477
2023-01-15 23:56:23 +02:00
Sergei Trofimovich
9c0068c40e
Merge pull request #210752 from trofi/make-bootstrap-tools-libstdcxx-rpath
make-bootstrap-tools: fix test to include libstdc++ -rpath
2023-01-15 07:01:34 +00:00
Robert Hensing
bff126ffbc stdenv.mkDerivation: Make overrideAttrs overridable
(cherry picked from commit 43c8b43f808f48fd5600afcad5503eaeaf6d71b7)
2023-01-15 04:37:14 +02:00
Artturi
f338f1422e
Merge pull request #209371 from Artturin/gziprepro
gzip: make reproducible when GZIP_NO_TIMESTAMPS is set
2023-01-15 00:04:51 +02:00
Artturi
5fae6b5c74
Merge pull request #210109 from amjoseph-nixpkgs/pr/stdenv/build-gettext-only-once 2023-01-14 21:55:22 +02:00
Sergei Trofimovich
497033467f make-bootstrap-tools: fix test to include libstdc++ -rpath
After https://github.com/NixOS/nixpkgs/pull/209054 we started moving
libstdc++.so out of default glibc's paths. This exposed bootstrap
tools build failure as:

    $ nix build --no-link -f ./pkgs/stdenv/linux/make-bootstrap-tools.nix
    ...
    >
    .../bin/bar: error while loading shared libraries: libstdc++.so.6:
       cannot open shared object file: No such file or directory

Note that bootstrap itself did not break. The change only expands
handcrafted `-rpath` entries.
2023-01-14 19:42:50 +00:00
Adam Joseph
58857196f1
stdenv: mark binutils-patchelfed (#209600)
The stdenv bootstrap creates several wrappers around binutils, and
gives them the exact same drvName as the binutils package itself.

These wrappers cost almost nothing to create (they are just file
copies and patchelf runs, not builds), so we should distinguish them
from expensive binutils builds with a unique pname.  This commit
does that.
2023-01-14 10:07:44 +00:00
Sergei Trofimovich
4433093854
Merge pull request #209054 from trofi/stdcxx-out-of-bootstrap-lib
linux/bootstrap-tools: move libstdc++ out of default library search path
2023-01-12 18:27:58 +00:00
Artturin
88f36d2694 gzip: make reproducible when GZIP_NO_TIMESTAMPS is set
the logic can be tested with

```
$ GZIP_NO_TIMESTAMPS=1 && echo "${GZIP_NO_TIMESTAMPS:+-n }"
-n
$ unset GZIP_NO_TIMESTAMPS && echo "${GZIP_NO_TIMESTAMPS:+-n }"

```
2023-01-12 01:30:29 +02:00
github-actions[bot]
b3d2428df4
Merge staging-next into staging 2023-01-11 20:59:01 +00:00
github-actions[bot]
a113c6f06f
Merge master into staging-next 2023-01-11 20:58:29 +00:00
Adam Joseph
098c6b0bec check-meta(hasUnsupportedPlatform): use lib.meta.availableOn
`hasUnsupportedPlatform` was not updated with #37395, so it does not
understand attrsets in `meta.[bad]platforms`.  In particular,
attrsets in `meta.badPlatforms` will "fail open" and be ignored.

Let's use `lib.meta.availableOn` instead of duplicating its logic.

Thanks to @alyssais for [noticing][1].

[1][https://github.com/NixOS/nixpkgs/pull/194148#discussion_r990817610]

Co-authored-by: sternenseemann <sternenseemann@systemli.org>
2023-01-11 19:31:52 +00:00
Adam Joseph
74df5ad72b stdenv: build gettext only once
Right now we build gettext several times during the bootstrap.
Gettext's build process is "embarrassingly serial", so avoiding
rebuilding it speeds things up considerably.
2023-01-10 13:29:02 -08:00
Artturi
386f2bb148
Merge pull request #209255 from Artturin/stdenvupdate1
stdenv: don't fail installPhase on missing makefile
2023-01-10 03:48:17 +02:00
Sergei Trofimovich
8a4e6b7e1b stdenv/darwin: rebuild gawk earlier: in stage4 instead of final stage
`gawk-5.1.1 -> 5.2.1` update (https://github.com/NixOS/nixpkgs/pull/207478)
started failing `stdenv` reference checks as `gawk` now leaks
`bootstrapTools` reference:

    `gawk` -> `gettext` -> `libiconv` -> `bootstrapTools`.

The change rebuild `gawk` in `stage4` to pull rebuilt tools.
2023-01-06 06:48:35 +00:00
Artturin
314b03125f stdenv: don't fail installPhase on missing makefile
otherwise the build just fails with 'make: *** No rule to make target 'install'.  Stop.'

and update buildPhase message

i don't know if the 'makefile may have been created in buildPhase' is
true but i guess it might be possible
2023-01-06 03:01:39 +02:00
github-actions[bot]
789ae2c93d
Merge staging-next into staging 2023-01-04 18:01:36 +00:00
Sergei Trofimovich
b470a6b212 linux/bootstrap-tools: move libstdc++ out of default library search path
This change allows building new gcc during bootstrap without fear of
pulling in outdated libstdc++.so after g++ switched from bootstrapTools
to freshly built g++.

Noticed when tried to add early bootstrap stage to rebuild `gcc` before
`glibc` is fully untangled from `bootstrapTools` as a failure to built
`binutils`:

    ld: dwp.o: in function `__gnu_cxx::new_allocator<gold::Dwp_output_file::Contribution>::allocate(unsigned long, void const*)':
    /nix/store/...-gcc-11.3.0/include/c++/11.3.0/ext/new_allocator.h:116: undefined reference to `std::__throw_bad_array_new_length()'

The change survives existing bootstrap and unblockes early `gcc` bootstrap.
2023-01-04 16:02:22 +00:00
piegames
aa8ead30fd
Merge pull request #175495 from amjoseph-nixpkgs/pr/sourceProvenance/correct-meta-typecheck
check-meta.nix: make non-source consistent with documentation
2023-01-04 15:31:51 +01:00
github-actions[bot]
03e830eb69
Merge staging-next into staging 2023-01-03 00:02:53 +00:00
John Ericson
a6b1de7902
Merge pull request #208478 from trofi/comment-stdenv-bootstrap
stdenv/linux: document some tips in debugging stdenv bootstrap tower
2023-01-02 17:28:43 -05:00
Sergei Trofimovich
062e14b561 stdenv/linux: document some tips in debugging stdenv bootstrap tower
Just a few comments added:

- added a few one-liners to explore which tools are rebuilt at each
  stdenv iteration during bootstrap
- explicitly listed available toolchains and their sources for on each
  bootstrap step: glibc, binutils, gcc, coreutils.
- added mention of static libraries linked into gcc

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
2023-01-02 21:55:32 +00:00
Adam Joseph
607d59fa9e check-meta.nix: make non-source consistent with documentation
The documentation for `meta.sourceProvenance` in
`doc/stdenv/meta.chapter.md` says: "the `meta.sourceProvenance`
attribute should be a list containing one or more value..."

Let's update check-meta.nix to require that `meta.sourceProvenance` is
a list, as the documentation says, rather than a single element.

Adding two extra keystrokes `[` and `]` when filling out this field is
an insignificant burden for package authors, and being able to assume
that the `meta.sourceProvenance` field is always a list greatly
simplifies any code that acts on the value of this field.

Since `meta.sourceProvenance` was just merged a few hours ago now is
the easiest time to fix this: nobody is using the feature yet.
2023-01-01 18:21:11 -08:00
Jan Tojnar
5810109b42 Merge branch 'staging-next' into staging
- readline6 attribute removed from all-packages.nix in d879125d61
- readline attribute was bumped to readline82 in 50adabdd60
2023-01-02 03:04:32 +01:00
Naïm Favier
4af22aab8e
stdenv/check-meta: do deep type checks
Use a wrapper around `mergeDefinitions` to type-check values deeply, so
that e.g. `maintainers = [ 42 ];` is an error.
2023-01-01 14:10:42 +01:00
github-actions[bot]
dc7ebb0163
Merge staging-next into staging 2022-12-18 18:01:41 +00:00
figsoda
ec8cb34358 treewide: fix typos 2022-12-17 19:39:44 -05:00
Sergei Trofimovich
428107f837 stdenv: set enableParallelBuilding explicitly if enableParallelBuildingByDefault is set
Without the change we don't propagate `enableParallelBuilding = true`
and leave most builds sequential.

Noticed on `mythtv` package which did not specify parallelism and
`config.enableParallelBuildingByDefault = true` had no effect.
2022-12-17 22:23:24 +00:00
Naïm Favier
0b067316d4
stdenv: use intersectAttrs instead of intersectLists
Better complexity.
2022-12-17 12:55:44 +01:00
Naïm Favier
84eebc0fe4
Merge pull request #205944 from ncfavier/structured-attrs-env 2022-12-15 13:27:54 +01:00
Naïm Favier
e14de22618
stdenv: handle env gracefully
Derivations not using `__structuredAttrs` should not attempt to set
environment variables from `env`.

Derivations using `__structuredAttrs` should fail if `env` is not
exportable.
2022-12-15 13:27:11 +01:00
Artturin
630bb71ac5 stdenv: sort defaultNativeBuildInputs alphabetically 2022-12-13 01:12:20 +02:00
Artturin
b3717f6c14 stdenv: remove now unneeded linux conditional 2022-12-13 01:04:29 +02:00
Artturin
7866db71cc stdenv/generic: fix todo 2022-12-13 00:34:48 +02:00
Artturin
11c3127e38 stdenv: detect the type of variable in {prepend,append}ToVar
stdenv: error if using {prepend,append}ToVar on associative array

i don't know how to prepend to associative array
2022-12-10 04:42:36 +02:00
Artturin
89dc806f13 what to do about attrs.env or {} maybe have a empty env attrset always
so no need to use `or`
2022-12-08 21:11:35 +02:00
Artturin
02e3f51d27 darwin: use // for binutils-unwrapped and cctools to preserve the other
attributes
2022-12-08 21:09:02 +02:00
Artturin
bb914d8676 stdenv: export system pname name version for substituteAll
so we don't have to add these to the env attrset
2022-12-08 21:09:02 +02:00
Artturin
734d7df235 allow derivation attributes in env
derivations can be coerced to their output paths
2022-12-08 06:13:19 +02:00
Artturin
8ad0103a34 config.structuredAttrsByDefault: add option 2022-12-08 06:13:19 +02:00
Artturin
1c4820efdd work around a nix bug 2022-12-08 06:13:19 +02:00
Artturin
238a6053c4 stdenv: support opt-in __structuredAttrs
Co-authored-by: Robin Gloster <mail@glob.in>

stdenv: print message if structuredAttrs is enabled

stdenv: add _append

reduces the chance of a user doing it wrong

fix nix develop issue

output hooks don't work yet in nix develop though

making $outputs be the same on non-structuredAttrs and structuredAttrs
is too much trouble.

lets instead make a function that gets the output names

reading environment file '/nix/store/2x7m69a2sm2kh0r6v0q5s9z1dh41m4xf-xz-5.2.5-env-bin'
nix: src/nix/develop.cc:299: std::string Common::makeRcScript(nix::ref<nix::Store>, const BuildEnvironment&, const Path&): Assertion `outputs != buildEnvironment.vars.end()' failed.

use a function to get all output names instead of using $outputs

copy env functionality from https://github.com/NixOS/nixpkgs/pull/76732/commits
2022-12-08 06:13:19 +02:00
Artturi
df3056bfac
Merge pull request #176597 from amjoseph-nixpkgs/pr/remove-redundant-file-buildInput
stdenv/cross: remove now-redundant `file` nativeBuildInput on mingw
2022-11-21 19:22:16 +02:00