Commit Graph

2673 Commits

Author SHA1 Message Date
K900
b40035f472
Merge pull request #267058 from toonn/bootstrap-tools-specify-llvm
freshBootstrapTools: Overlay the package set with the desired LLVM
2023-11-17 14:06:37 +03:00
Adam Joseph
985ff1cb88
Update pkgs/stdenv/darwin/make-bootstrap-tools.nix 2023-11-17 08:26:41 +00:00
Ivan Mincik
3b4b805561 stdenv: consistent phases header
Make phases header consistent for all phases.

`Running phase:` is from an old nix ux doc from 2020

https://github.com/tweag/nix-ux/blob/master/first_steps_with_nix_v2.md

Co-authored-by: Artturin <Artturin@artturin.com>
2023-11-17 03:11:48 +02:00
Artturin
249c6d04b1 check-meta: don't use rec
possible performance improvement
2023-11-14 16:00:42 +02:00
Artturin
0527676bd9 check-meta: don't use with 2023-11-13 23:47:25 +02:00
Artturin
215951e4c8 check-meta: Improve performance
See c3c31aa798
2023-11-13 23:43:03 +02:00
Weijia Wang
74d102ce50 Merge branch 'master' into staging-next 2023-11-12 19:45:20 +01:00
toonn
887d33a089 freshBootstrapTools: Overlay the package set with the desired LLVM
As reported in #241692, since the `llvmPackages` bump the
bootstrap-tools started failing to build due to a mismatch in LLVM
versions used to build certain tools.

By overlaying the imported package set to specify `llvmPackages`, we get
everything built with the expected LLVM version.
2023-11-12 15:08:11 +01:00
Artturin
c3c31aa798 stdenv: Improve performance
| stat                   | before          | after           | Δ               | Δ%      |
|------------------------|-----------------|-----------------|-----------------|---------|
| cpuTime                | 513.67          | 507.77          | ↘ 5.90          | -1.15%  |
| envs-bytes             | 20,682,847,968  | 20,628,961,616  | ↘ 53,886,352    | -0.26%  |
| envs-elements          | 1,054,735,104   | 1,051,395,620   | ↘ 3,339,484     | -0.32%  |
| envs-number            | 765,310,446     | 763,612,291     | ↘ 1,698,155     | -0.22%  |
| gc-heapSize            | 53,439,602,688  | 51,711,545,344  | ↘ 1,728,057,344 | -3.23%  |
| gc-totalBytes          | 113,062,066,672 | 112,139,998,240 | ↘ 922,068,432   | -0.82%  |
| list-bytes             | 3,118,249,784   | 3,118,249,784   | 0               |         |
| list-concats           | 52,834,140      | 52,834,140      | 0               |         |
| list-elements          | 389,781,223     | 389,781,223     | 0               |         |
| nrAvoided              | 968,097,988     | 991,889,795     | ↗ 23,791,807    | 2.46%   |
| nrFunctionCalls        | 697,259,792     | 697,259,792     | 0               |         |
| nrLookups              | 510,257,062     | 338,275,331     | ↘ 171,981,731   | -33.70% |
| nrOpUpdateValuesCopied | 1,446,690,216   | 1,446,690,216   | 0               |         |
| nrOpUpdates            | 68,504,034      | 68,504,034      | 0               |         |
| nrPrimOpCalls          | 429,464,805     | 429,464,805     | 0               |         |
| nrThunks               | 1,009,240,391   | 982,109,100     | ↘ 27,131,291    | -2.69%  |
| sets-bytes             | 33,524,722,928  | 33,524,722,928  | 0               |         |
| sets-elements          | 1,938,309,212   | 1,938,309,212   | 0               |         |
| sets-number            | 156,985,971     | 156,985,971     | 0               |         |
| sizes-Attr             | 16              | 16              | 0               |         |
| sizes-Bindings         | 16              | 16              | 0               |         |
| sizes-Env              | 16              | 16              | 0               |         |
| sizes-Value            | 24              | 24              | 0               |         |
| symbols-bytes          | 2,151,298       | 2,151,298       | 0               |         |
| symbols-number         | 159,707         | 159,707         | 0               |         |
| values-bytes           | 30,218,194,248  | 29,567,043,264  | ↘ 651,150,984   | -2.15%  |
| values-number          | 1,259,091,427   | 1,231,960,136   | ↘ 27,131,291    | -2.15%  |

> Accessing the lexical scope directly should be more efficient, yes, because it changes from a binary search (many lookups) to just two memory accesses
> correction: one short linked list + one array access
> oh and you had to do the lexical scope lookup anyway for lib itself
> so it really does save a binary search at basically no extra cost

- roberth

after seeing the stats

> Oooh nice. I did not consider that more of the maybeThunk optimization becomes effective (nrAvoided). Those lookups also caused allocations!

- roberth

Left `lib.generators` and `lib.strings` alone because they're only used
once.
2023-11-12 10:42:49 +02:00
Randy Eckenrode
940180423a
Merge pull request #265102 from reckenrode/darwin-corefoundation
darwin.stdenv: use CoreFoundation instead of CF
2023-11-11 12:10:10 -05:00
Alyssa Ross
506dcaabb4 stdenv: fix meson rust cross compilation 2023-11-09 10:02:24 +01:00
github-actions[bot]
fecd99b105
Merge staging-next into staging 2023-11-09 06:01:40 +00:00
github-actions[bot]
4aad65bfdb
Merge master into staging-next 2023-11-09 06:01:11 +00:00
Artturi
71f42b7f1a
Merge pull request #230874 from Ninlives/runPhase 2023-11-07 20:00:47 +02:00
Weijia Wang
8012034692 Merge branch 'staging-next' into staging 2023-11-07 03:41:49 +01:00
Artturi
fbb3ff5e12
Merge pull request #243161 from nh2/more-enableStatic 2023-11-07 03:49:21 +02:00
mlatus
707a5c09db stdenv: wrap phase running actions of genericBuild
Provide a `runPhase` function which wraps the phase running action of
genericBuild. The new function can be used as an interface by `nix
develop`, i.e. `nix develop some#flake --build` may just call `runPhase
build`, which makes its behavior more consistent with `nix build`.
In preparation of fixing https://github.com/NixOS/nix/issues/6202
2023-11-07 03:31:26 +02:00
Weijia Wang
023d20ae64
Merge pull request #265307 from reckenrode/clang16-fixes-batch2
rubyPackages.iconv, v8: fix build with clang 16
2023-11-04 17:14:36 +01:00
Randy Eckenrode
98b8d3474b
stdenvAdapters.overrideSDK: special case the 10.12 Libsystem
The 10.12 Libsystem is not located as a sub-attribute of
`darwin.apple_sdk_10_12`. This will be fixed as part of the SDK changes
planned for post-23.11. In the meantime, special case it so the adapter
can be used to change the deployment target.
2023-11-04 01:47:58 -04:00
Randy Eckenrode
cc4fcc147b
stdenvAdapters: add overrideLibcxx
This was taken from #264091 to use in the interim before that PR lands
(sometime after the release of 23.11). It allows different versions of
clang to link the same libc++, allowing dependencies to be linked when
they are built with a different version of clang than the stdenv.
2023-11-03 23:11:32 -04:00
Randy Eckenrode
daa79a1d2d
darwin.stdenv: use CoreFoundation instead of CF
This patch switches the CoreFoundation on x86_64-darwin from the open
source swift-corelibs-foundation (CF) to the system CoreFoundation.

This change was motivated by failures building packages for the current
staging-next cycle #263535 due to an apparent incompatibility with the
rpath-based approach to choosing CF or CoreFoundation and macOS 14. This
error often manifests as a crash with an Illegal Instruction.

For example, building aws-sdk-cpp for building Nix will fail this way.

https://hydra.nixos.org/build/239459417/nixlog/1

    Application Specific Information:
    CF objects must have a non-zero isa

    Error Formulating Crash Report:
    PC register does not match crashing frame (0x0 vs 0x7FF8094DD640)

    Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
    0   CoreFoundation                	    0x7ff8094dd640 CF_IS_OBJC.cold.1 + 14
    1   CoreFoundation                	    0x7ff8094501d0 CF_IS_OBJC + 60
    2   CoreFoundation                	    0x7ff8093155e8 CFRelease + 40
    3   ???                           	       0x10c7a2c61 s_aws_secure_transport_ctx_destroy + 65
    4   ???                           	       0x10c87ba32 aws_ref_count_release + 34
    5   ???                           	       0x10c7b7adb aws_tls_connection_options_clean_up + 27
    6   ???                           	       0x10c596db4 Aws::Crt::Io::TlsConnectionOptions::~TlsConnectionOptions() + 20
    7   ???                           	       0x10c2d249c Aws::CleanupCrt() + 92
    8   ???                           	       0x10c2d1ff0 Aws::ShutdownAPI(Aws::SDKOptions const&) + 64
    9   ???                           	       0x102d9bc6f main + 335
    10  dyld                          	       0x202f333a6 start + 1942

According to a [post][1] on the Apple developer forums, hardening was
added to CoreFoundation, and this particular message occurs when you
attempt to release an object it does not recognize as a valid CF object.
(Thank you to @lilyinstarlight for finding this post).

When I switched aws-sdk-cpp to link against CoreFoundation instead of
CF, the error went away. Somehow both libraries were being used. To
prevent dependent packages from linking the wrong CoreFoundation, it
would need to be added as a propagated build input.

Note that there are other issues related to mixing CF and CoreFoundation
frameworks. #264503 fixes an issue with abseil-cpp where it propagates
CF, causing issues when using a different SDK version. Mixing versions
can also cause crashes with Python when a shared object is loaded that
is linked to the “wrong” CoreFoundation.

`NIX_COREFOUNDATION_RPATH` is supposed to make sure the right
CoreFoundation is being used, but it does not appear to be enough on
macOS 14 (presumably due to the hardening). While it is possible to
propagate CoreFoundation manually, the cleaner solution is to make it
the default. CF remains available as `darwin.swift-corelibs-foundation`.

[1]: https://developer.apple.com/forums/thread/739355
2023-11-02 21:20:55 -04:00
Randy Eckenrode
08e1c95e9c
stdenvAdapters.overrideSDK: handle non-drv nativeBuildInputs 2023-11-01 12:30:26 -04:00
Randy Eckenrode
b6a14a3a84
stdenvAdapters.overrideSDK: also remap propagatedBuildInputs 2023-11-01 09:06:44 -04:00
Randy Eckenrode
50d046c32a
stdenvAdapters.overrideSDK: update the SDK version recursively
Update all propagated build inputs recursively, so that propagated
inputs with propagated inputs use the correct SDK.
2023-11-01 09:06:44 -04:00
emilylange
80627bfe8f
stdenv: enable multithreading for xz decompression
(only if the implementation supports it)

This speeds up the decompression of huge tar.xz tarballs (e.g. chromium)
on high core count systems.
2023-10-31 16:14:08 +01:00
Robert Hensing
e143a933f6 check-meta.nix: Fix flake note
- These new-cli commands can be used with `-f`, in which case they're
  evaluated with pure evaluation disabled.
- Nix 2.4+ is not part of the condition; "flakes" is fully descriptive
  and more relatable.
- Don't suggest that it only enables this variable.
- Just don't say too much.
2023-10-31 12:44:36 +01:00
Randy Eckenrode
3440df992e
stdenvAdapters.overrideSDK: override xcodebuild to use the requested SDK 2023-10-31 00:18:54 -04:00
toonn
f65ccb3163
Merge pull request #263598 from reckenrode/curl-propagation-fix
curl: fix build failures due to needing to propagate frameworks
2023-10-27 18:25:39 +02:00
github-actions[bot]
96132a216a
Merge master into staging-next 2023-10-26 18:01:02 +00:00
Randy Eckenrode
c2a623b52a
stdenvAdapters: add overrideSDK
This is a replacement for using `darwin.apple_sdk_<ver>.callPackage`.
Instead of injecting the required packages, it provides a stdenv adapter
that modifies the derivation’s build inputs to use the requested SDK
versions. This modification extends to any build inputs propagated to it
as well. The `callPackage` approach is not deprecated yet, but it is
expected that it will be eventually.

Note that this is an MVP. It should work with most packages, but it only
handles build inputs and also only handles frameworks. Once more SDKs
are added (after #229210 is merged) and the SDK structure is normalized,
it can be extended to handle any package in the SDK namespace.

Cross-compilation may or may not work. Any cross-related issues can be
addressed after #256590 is merged.
2023-10-26 13:26:39 -04:00
nicoo
87c22100a6 stdenv.mkDerivation: Reject MD5 hashes
While there is no fetcher or builder (in nixpkgs) that takes an `md5` parameter,
for some inscrutable reason the nix interpreter accepts the following:
```nix
fetchurl {
  url = "https://www.perdu.com";
  hash = "md5-rrdBU2a35b2PM2ZO+n/zGw==";
}
```

Note that neither MD5 nor SHA1 are allowed by the syntax of SRI hashes.
2023-10-25 23:14:15 +00:00
toonn
b525407492
Merge pull request #260963 from reckenrode/curl-fix
curl: fix infinite recursion in staging-next
2023-10-21 19:12:14 +02:00
github-actions[bot]
e5968ce788
Merge staging-next into staging 2023-10-21 06:01:24 +00:00
Artturin
1b3831f1e6 stdenv: fix typo 2023-10-21 00:26:57 +03:00
Niklas Hambüchen
e73b7f8d63 bzip2: Add enableStatic. See #61575 2023-10-20 19:59:30 +00:00
github-actions[bot]
34db746ff9
Merge staging-next into staging 2023-10-20 06:35:22 +00:00
Pavel Sobolev
17acba4cd1
stdenvAdapters.useMoldLinker: set allowedRequisites to null 2023-10-19 15:48:42 +03:00
Randy Eckenrode
df14e86d9c
stdenv.darwin: fix infinite recursion after curl update
curl needs to link against several frameworks, but building the
frameworks (directly or indirectly) depends on curl via fetchurl and
fetchFromGitHub. Break the infinite recursion by building the SDKs’
dependencies in the last stage of the stdenv bootstrap using the prior
stage’s fetchurl and fetchFromGitHub.
2023-10-13 18:51:05 -04:00
github-actions[bot]
c68d98e9d4
Merge staging-next into staging 2023-10-13 18:01:31 +00:00
Artturin
e64d24fff4
stdenvAdapters.useMoldLinker: automatically add allowedRequisites
Co-Authored-By: Pavel Sobolev <paveloom@riseup.net>
2023-10-13 10:14:40 +03:00
github-actions[bot]
49523dec15
Merge staging-next into staging 2023-10-13 06:01:45 +00:00
Artturin
77bf1395d6 mold: add tests for mold-wrapped and useMoldLinker adapter
Make both ready for cross with prefixes

Currently
`pkgsCross.aarch64-multiplatform.mold.passthru.tests.{wrapped,adapter}`
fail with

```
Testing running the 'hello' binary which should be linked with 'mold'
Hello, world!
Checking for mold in the '.comment' section
No mention of 'mold' detected in the '.comment' section
The command was:
aarch64-unknown-linux-gnu-readelf -p .comment ...bin/hello
The output was:
String dump of section '.comment':
  [     0]  GCC: (GNU) 12.3.0
```
2023-10-13 05:23:32 +03:00
github-actions[bot]
f1a87e41ca
Merge staging-next into staging 2023-10-09 00:03:06 +00:00
Alyssa Ross
1da4b5c99e
release.nix: namespace bootstrap tools with triples
This will allow buliding bootstrap tools for platforms with
non-default libcs, like *-unknown-linux-musl.

This gets rid of limitedSupportSystems/systemsWithAnySupport.  There
was no need to use systemsWithAnySupport for supportDarwin, because it
was always equivalent to supportedSystems for that purpose, and the
only other way it was used was for determining which platforms to
build the bootstrap tools for, so we might as well use a more explicit
parameter for that, and then we can change how it works without
affecting the rest of the Hydra jobs.

Not affecting the rest of the Hydra jobs is important, because if we
changed all jobs to use config triples, we'd end up renaming every
Hydra job.  That might still be worth thinking about at some point,
but it's unnecessary at this point (and would be a lot of work).

I've checked by running

    nix-eval-jobs --force-recurse pkgs/top-level/release.nix

that the actual bootstrap tools derivations are unaffected by this
change, and that the only other jobs that change are ones that depend
on the hash of all of Nixpkgs.  Of the other jobset entrypoints that
end up importing pkgs/top-level/release.nix, none used the
limitedSupportedSystems parameter, so they should all be unaffected as
well.
2023-10-07 19:38:01 +00:00
Linus Heckemann
81f6dc0864
Merge pull request #257919 from Ma27/structured-attrs-env-vars
structured attrs: prefer `NIX_ATTRS_*_FILE` over `.attrs.*`
2023-10-05 12:48:00 +02:00
github-actions[bot]
d4ed9dd5cd
Merge staging-next into staging 2023-10-05 00:02:58 +00:00
Maximilian Bosch
8bc5104a6e
treewide: refactor .attrs.sh detection
When specifying the `builder` attribute in `stdenv.mkDerivation`, this
will be effectively transformed into

    builtins.derivation {
      builder = stdenv.shell;
      args = [ "-e" builder ];
    }

This also means that `default-builder.sh` is never sourced and as a
result it's not guaranteed that `$NIX_ATTRS_SH_FILE` is set to a correct
location[1].

Also, we need to source `.attrs.sh` to source `$stdenv`. So, the
following is done now:

* If `$NIX_ATTRS_SH_FILE` points to a correct location, then use it.
  Directly using `.attrs.sh` is problematic for `nix-shell(1)` usage
  (see previous commit for more context), so prefer the environment
  variable if possible.

* Otherwise, if `.attrs.sh` exists, then use it. See [1] for when this
  can happen.

* If neither applies, it can be assumed that `__structuredAttrs` is
  turned off and thus nothing needs to be done.

[1] It's possible that it doesn't exist at all - in case of Nix 2.3 or
    it can point to a wrong location on older Nix versions with a bug in
    `__structuredAttrs`.
2023-10-04 18:36:57 +02:00
Maximilian Bosch
53c5f08415
stdenv: refactor .attrs.sh detection
Relying on `.attrs.sh` to exist in `$NIX_BUILD_TOP` is problematic
because that's not compatible with how `nix-shell(1)` behaves. It places
`.attrs.{json,sh}` into a temporary directory and makes them accessible via
`$NIX_ATTRS_{SH,JSON}_FILE` in the environment[1]. The sole reason that
`nix-shell(1)` still works with structured-attrs enabled derivations
is that the contents of `.attrs.sh` are sourced into the
shell before sourcing `$stdenv/setup` (if `$stdenv` exists) by `nix-shell`.

However, the assumption that two files called `.attrs.sh` and
`.attrs.json` exist in `$NIX_BUILD_TOP` is wrong in an interactive shell
session and thus an inconsistency between shell debug session and actual
builds which can lead to unexpected problems.

To be precise, we currently have the following problem: an expression
like

  with import ./. {};
  runCommand "foo" { __structuredAttrs = true; foo.bar = [ 1 2 3 ]; }
    ''
      echo "''${__structuredAttrs@Q}"
      touch $out
    ''

prints `1` in its build-log. However when building interactively in a
`nix-shell`, it doesn't.

Because of that, I'm considering to propose a full deprecation of
`$NIX_BUILD_TOP/.attrs.{json,sh}`. A first step is to only mention the
environment variables, but not the actual paths anymore in Nix's
manual[2]. The second step - this patch - is to fix nixpkgs' stdenv
accordingly.

Please note that we cannot check for `-e "$NIX_ATTRS_JSON_FILE"` because
certain outdated Nix minors (that are still in the range of supported
Nix versions in `nixpkgs`) have a bug where `NIX_ATTRS_JSON_FILE` points
to the wrong file while building[3].

Also, for compatibility with Nix 2.3 which doesn't provide these
environment variables at all we still need to check for the existence of
.attrs.json/.attrs.sh here. As soon as we bump nixpkgs' minver to 2.4,
this can be dropped.

Finally, dropped the check for ATTRS_SH_FILE because that was never
relevant. In nix#4770 the ATTRS_SH_FILE variable was introduced[4] and
in a review iteration prefixed with NIX_[5]. In other words, these
variables were never part of a release and you'd only have this problem
if you'd use a Nix from a git revision of my branch from back then. In
other words, that's dead code.

[1] https://github.com/nixos/nix/pull/4770#issuecomment-834718851
[2] https://github.com/NixOS/nix/pull/9032
[3] https://github.com/NixOS/nix/issues/6736
[4] 3944a120ec
[5] 27ce722638
2023-10-04 18:35:30 +02:00
Artturin
1270ef59e9 bootstrap-tools-cross: Rename bootstrap file jobs to be more descriptive
Rename files in `bootstrap-files/` to match jobs in `make-bootstrap-tools-cross`.

Should make automating the bootstrap files easier.

```nix
$ nix repl --file ./pkgs/top-level/release-cross.nix
nix-repl> bootstrapTools
{ aarch64-unknown-linux-gnu = { ... }; aarch64-unknown-linux-musl = { ... }; armv5tel-unknown-linux-gnueabi = { ... }; armv6l-unknown-linux-gnueabihf = { ... }; armv6l-unknown-linux-musleabihf = { ... }; armv7l-unknown-linux-gnueabihf = { ... }; mips64el-unknown-linux-gnuabi64 = {... }; mips64el-unknown-linux-gnuabin32 = { ... }; mipsel-unknown-linux-gnu = { ... }; powerpc64le-unknown-linux-gnu = { ... }; riscv64-unknown-linux-gnu = { ... }; x86_64-unknown-linux-musl = { ... }; }
```

Additionally if non-linux bootstrap files are added to
`make-bootstrap-tools-cross` then there won't be any renaming needed.
2023-10-03 23:34:40 +03:00
github-actions[bot]
a3a27f01d5
Merge staging-next into staging 2023-10-02 18:01:40 +00:00
Artturi
ece37b8aa4
Merge pull request #258476 from Artturin/bootstrapthingies2 2023-10-02 09:10:31 +03:00
Artturin
0848edf7a0 bootstrap-tools-cross: Add a note about what should be here 2023-10-02 01:30:11 +03:00
Artturin
46ecf91843 stdenv: remove unused loongson2f bootstrap files
The last use of it was removed in (linux/default.nix: use mipsel.nix instead of longson.nix for mips32)[e8b10284f3].
2023-10-02 00:35:57 +03:00
Artturin
a1c2c50cbc bootstrap-tools-cross: sort 2023-10-01 23:54:45 +03:00
Artturin
df7bf17643 bootstrap-tools-cross: remove bootstrap files for systems which are not used in stdenv/linux/default.nix 2023-10-01 23:53:15 +03:00
Artturi
3bf47679b7
Merge pull request #253160 from pwaller/static-skip-install-rpath
makeStaticBinaries: Set -DCMAKE_SKIP_INSTALL_RPATH
2023-10-01 21:36:02 +03:00
Artturin
d0efa70d81 Revert "Revert "Merge pull request #253760 from chivay/bootstrap-scrt""
This reverts commit 3f46cdcb5b.
2023-09-30 15:17:47 -04:00
github-actions[bot]
93a166b92d
Merge staging-next into staging 2023-09-07 06:01:25 +00:00
Artturin
93fb394638 freshBootstrapTools.bootstrapTools: fix eval on darwin
```
error: undefined variable 'system'
       at /home/artturin/nixgits/my-nixpkgs/.worktree/1/pkgs/stdenv/darwin/make-bootstrap-tools.nix:213:32:

          212|
          213|   bootstrapTools = derivation {
             |                                ^
          214|     inherit system;

```

fixes eval of `freshBootstrapTools.bootstrapTools` in darwin repl
2023-09-07 04:24:05 +03:00
github-actions[bot]
1a5c2acd74
Merge staging-next into staging 2023-09-05 00:02:47 +00:00
Martin Zacho
bfdccb6298 pkgs/stdenv: fix typo in booter.nix docs 2023-09-04 22:24:12 +02:00
Artturi
d5139e3017
Merge pull request #245909 from Artturin/setupshchanges2 2023-09-04 20:41:16 +03:00
Peter Waller
9fdf395cc7 makeStaticBinaries: Set -DCMAKE_SKIP_INSTALL_RPATH
Without this, you get error messages during the install phase along the
lines of: "file RPATH_CHANGE could not write new RPATH:".

This is unsurprising because the static binaries do not have any dynamic
linker and thus, no runpath to rewrite either.

Tell cmake it doesn't need to do RPATH manipulation by passing
cmakeFlags.

While we're here, I also renamed `finalAttrs` to `args` and fixed the
indentation; this improves consistency with the surrounding code and
eliminates a point of confusion: because it was named `finalAttrs` I
presumed I should be able to influence it with an overrideAttrs setting
dontAddStaticConfigureFlags, but this turns out not to be possible;
adding prevAttrs as well doesn't work because of a limitation of
overrideAttrs whereby it gives an infinite recursion if the set of
attribute keys being returned depends on finalAttrs.

Signed-off-by: Peter Waller <p@pwaller.net>
2023-09-03 18:30:59 +01:00
Artturin
1db1e3c467 stdenv: Fix possible issues discovered with
```
set -o errexit -o nounset -o pipefail
shopt -s inherit_errexit
```

in `stdenv/default-builder.sh`
2023-09-02 03:25:36 +03:00
Artturin
c25f7f894f stdenv: Add hack to fix cmake canExecute cross-compilation
Fixes `pkgsCross.musl64.llvmPackages_16.clang.cc` on `x86_64-linux`,
which used to fail with `/bin/sh: clang-tblgen: not found`.

Same hack is used in other projects:
https://github.com/search?q=%2FCMAKE_CROSSCOMPILING_EMULATOR.%2B%5C%2Fusr%5C%2Fbin%5C%2Fenv%2F+NOT+is%3Afork&type=code

Comment from 30435a9d0f/build/cmake/HostLinuxToolchain.cmake (L64)

> Required to run host Linux executables during the build itself.
> An example would be https://gitub.com/KhronosGroup/Vulkan-Loader and
> its "asm_offset" program.
>
> NOTE: Alternatives have been tried unsuccessfully, i.e.:
>
>  With $(set CMAKE_CROSSCOMPILING_EMULATOR), the build fails because
>  the CMake ninja/Make script tries to find the executable in the current
>  path, as in:
>
>    [3/16] Generating gen_defines.asm
>    FAILED: loader/gen_defines.asm
>    cd /tmp/cc/build-Vulkan-Loader/loader && asm_offset GAS
>    /bin/sh: asm_offset: command not found
>    ninja: build stopped: subcommand failed.
>
> With $(set CMAKE_CROSSCOMPILING_EMULATOR ""), the build fails because
> the shell cannot find the "" program as in:
>
>    [3/16] Generating gen_defines.asm
>    FAILED: loader/gen_defines.asm
>    cd /tmp/cc/build-Vulkan-Loader/loader && "" /tmp/cc/build-Vulkan-Loader/loader/asm_offset GAS
>    /bin/sh: : command not found
>    ninja: build stopped: subcommand failed.
>
> It seems that the root of the problem comes from how the CMake function
> cmCustomCommandGenerator::GetArgc0Location() computes the target
> executable's location. At this point it's unclear whether this is a CMake
> bug or a feature.

Risicle discovered this hack.

Co-authored-by: Robert Scott <code@humanleg.org.uk>
2023-08-29 22:12:06 +03:00
Artturin
87db45704f stdenv: Print _allFlags debug output to stderr
In the default `fixupPhase` the output of `substituteAllStream` is
streamed to setup-hook.

`stdenv.cc.bintools.overrideAttrs { NIX_DEBUG = 6; }`

With `NIX_DEBUG` contains:

```
@expandResponseParams@ -> /nix/store/yl01rd58vp4m8bbhkihpk132cprfmx6f-expand-response-params/bin/expand-response-params
...
```
2023-08-10 17:35:07 +03:00
Robert Scott
0865f1f4d7
Merge pull request #219421 from risicle/ris-fortify-headers-auto
cc-wrapper: include fortify-headers before libc includes for musl
2023-08-06 22:50:48 +01:00
Robert Scott
95c4a1fe96 cc-wrapper: include fortify-headers before libc includes for musl 2023-08-06 17:52:28 +01:00
github-actions[bot]
0031ae7742
Merge staging-next into staging 2023-08-06 00:03:36 +00:00
github-actions[bot]
c90f3611dc
Merge master into staging-next 2023-08-06 00:02:21 +00:00
Jan Malakhovski
85d4248a4f
treewide: fix some comments (#247365)
* pkgs/top-level/splice.nix: fix broken comment

* stdenv: use a value that actually works in the comment there
2023-08-05 23:01:26 +02:00
Simon Žlender
8e912feb29 codesign_allocate: reference cctools 2023-08-01 21:27:09 +02:00
Weijia Wang
e870890000
Merge pull request #245640 from reckenrode/revert-stdenv-no-response
darwin-stdenv: revert `NIX_CC_NO_RESPONSE_FILE` logic
2023-08-01 19:16:54 +02:00
Vladimír Čunát
2141d9879a
Revert "stdenv: use improved strip.sh for aarch64-linux"
This reverts commit 39919b8f21.
The parent merge resolved this more properly.
2023-07-31 11:20:15 +02:00
Vladimír Čunát
39919b8f21
stdenv: use improved strip.sh for aarch64-linux
Adapted from PR #246164
TODO: clean up / use it everywhere on the next rebuild.
2023-07-30 21:18:30 +02:00
Artturin
9816ca55c2 stdenv: Make condition clearer
-z checks for a empty string
-n checks for a not empty string

It makes more sense to run the chmod if the string is not empty
2023-07-28 04:20:26 +03:00
Randy Eckenrode
2e45100c5c
darwin-stdenv: revert NIX_CC_NO_RESPONSE_FILE logic
To work around intermitent build failures with clang 16, the stdenv
attempted to pass arguments on the command-line on newer versions of
macOS. Unfortunately, the larger `ARG_MAX` is still not large enough to
build qtwebengine. This commit reverts the `NIX_CC_NO_RESPONSE_FILE`
logic in the stdenv. The changes to cc-wrapper in #245282 are needed for
clang 16 to prevent the above-mentioned build failures.
2023-07-25 21:25:46 -04:00
github-actions[bot]
c05c2c2f5c
Merge master into staging-next 2023-07-22 00:02:13 +00:00
Artturin
29b4a92458 stdenvAdapters: Remove 6 year old comment
Comment was added in 3c36dcc and hasn't been needed because makeStaticBinaries already adds glibc.static
2023-07-21 00:49:44 +03:00
Artturin
cb29bc6ace stdenvAdapters: Fix condition
Fixes issue 244232
2023-07-21 00:49:40 +03:00
Uri Baghin
0275cf0e6d
Merge pull request #241025 from uri-canva/fix-native
stdenv: fix stagesNative
2023-07-17 13:18:40 +10:00
Weijia Wang
2b692b1d9d freshBootstrapTools: fix build on darwin 2023-07-15 19:54:50 +02:00
Theodore Ni
bb219736e4
stdenvBootstrapTools: in darwin, only run install_name_tool on Mach-O files 2023-07-14 10:22:55 -07:00
Randy Eckenrode
856ebe6fec
darwin.stdenv: allow patchShebangs during the bootstrap
This fixes pyicu (and any other package that uses `icu-config` instead
of the CMake or some other module to get the build flags).

What happened here is the bootstrap disables `patchShebangs` to avoid
propagating the bootstrap tools to the final stdenv (due to `sh` and
`bash` being on the `PATH` from the bootstrap tools). Because of that,
the `#!/bin/sh` line in `icu-config` was not updated, causing it to
invoke the system bash on Darwin. While that is undesirable in its own
right, when the system bash is invoked as `sh`, `echo -n` will print
`-n`, resulting in the breakage see in https://github.com/NixOS/nixpkgs/pull/241951#issuecomment-1627604354.

The fix is to build bash earlier in the bootstrap while making sure it
is picked up over the one in the bootstrap tools. That allows
`patchShebangs` to be enabled during the bootstrap. Any package with
scripts that is included in the final stdenv should now have its
scripts’ shebang lines properly patched.
2023-07-09 16:53:55 -06:00
github-actions[bot]
c60d48234a
Merge master into staging-next 2023-07-09 12:01:18 +00:00
Randy Eckenrode
6f6687ee24
darwin.stdenv: only run install_name_tool on files 2023-07-08 20:49:50 -06:00
Randy Eckenrode
feb89eed00
darwin.stdenv: fix portable libsystem hook with sandboxing
When sandboxing is enabled, the hook tries to run `install_name_tool`
and fails because the system one is inaccessible. Having it use
`targetPrefix` allows it to find and use the cross-install_name_tool.
2023-07-08 20:49:49 -06:00
Vladimír Čunát
58392652f0
Merge #224822: hardening flags: enable fortify3 by default
...into staging
2023-07-06 10:33:24 +02:00
Weijia Wang
f0a11a54be
Merge pull request #240433 from reckenrode/darwin-stdenv-rework
darwin.stdenv: Darwin stdenv rework
2023-07-05 13:03:24 +03:00
figsoda
34181be024 stdenvAdapters: don't use lib.optional with a list 2023-07-03 21:20:17 -04:00
Artturin
42c94d06fd stdenv: fix overriding with attrset when finalAttrs isn't used
```
nix-repl> (pkgs.htop.overrideAttrs { pname = "hello-overriden"; }).pname
error:
       … while evaluating a branch condition

         at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/lib/customisation.nix:86:7:

           85|     in
           86|       if builtins.isAttrs result then
             |       ^
           87|         result // {

       … while calling the 'isAttrs' builtin

         at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/lib/customisation.nix:86:10:

           85|     in
           86|       if builtins.isAttrs result then
             |          ^
           87|         result // {

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: attempt to call something which is not a function but a set

       at /nix/store/phn5cahwacv9wjgalygw62x8l4xbl6x3-source/pkgs/stdenv/generic/make-derivation.nix:58:21:

           57|           f = self: super:
           58|             let x = f0 super;
             |                     ^
           59|             in
```
2023-07-03 21:34:50 +03:00
Randy Eckenrode
ebc1bcf409
swift-corelibs: don’t link against libcurl
swift-corelibs uses libcurl to implement `NSURLSession` in Foundation
via the symbols exported by CF. Foundation is not build on Darwin, and
these symbols are not exported by the system CoreFoundation.

By not linking against libcurl, this breaks a cycle between CF and
libcurl. That should allow libcurl to drop the patch disabling
linking against the SystemConfiguration and restore NAT64 support.

Unfortunately, the Darwin stdenv bootstrap still needs to build
dependencies that use `fetchFromGitHub`. While it can drop curl from the
final stdenv, it still needs to use it during the stdenv bootstrap.
2023-07-02 17:56:25 -04:00
Randy Eckenrode
a845397040
darwin.stdenv: refactor stdenv definition
In preparation for bumping the LLVM used by Darwin, this change
refactors and reworks the stdenv build process. When it made sense,
existing behaviors were kept to avoid causing any unwanted breakage.
However, there are some differences. The reasoning and differences are
discussed below.

- Improved cycle times - Working on the Darwin stdenv was a tedious
  process because `allowedRequisites` determined what was allowed
  between stages. If you made a mistake, you might have to wait a
  considerable amount of time for the build to fail. Using assertions
  makes many errors fail at evaluation time and makes moving things
  around safer and easier to do.
- Decoupling from bootstrap tools - The stdenv build process builds as
  much as it can in the early stages to remove the requirement that the
  bootstrap tools need bumped in order to bump the stdenv itself. This
  should lower the barrier to updates and make it easier to bump in the
  future. It also allows changes to be made without requiring additional
  tools be added to the bootstrap tools.
- Patterned after the Linux stdenv - I tried to follow the patterns
  established in the Linux stdenv with adaptations made to Darwin’s
  needs. My hope is this makes the Darwin stdenv more approable for
  non-Darwin developers who made need to interact with it. It also
  allowed some of the hacks to be removed.
- Documentation - Comments were added explaining what was happening and
  why things were being done. This is particular important for some
  stages that might not be obvious (such as the sysctl stage).
- Cleanup - Converting the intermediate `allowedRequisites` to
  assertions revealed that many packages were being referenced that no
  longer exist or have been renamed. Removing them reduces clutter and
  should help make the stdenv bootstrap process be more understandable.
2023-07-02 17:56:24 -04:00
uri
f829b24045 stdenv: fix stagesNative 2023-07-02 19:07:23 +10:00
Robert Hensing
1f5a0b0135
Merge pull request #239005 from Artturin/setorfunoverr
stdenv: let overrideAttrs accept attrset OR function
2023-07-01 15:27:23 +02:00
Artturin
0fdae31531 stdenv: let overrideAttrs accept attrset OR function
Makes overrideAttrs usable in the same way that `override` can be used.
It allows the first argument of `overrideAttrs` to be either a function
or an attrset, instead of only a function:

hello.overrideAttrs (old: { postBuild = "echo hello"; })
hello.overrideAttrs { postBuild = "echo hello"; }

Previously only the first example was possible.

Co-authored-by: adisbladis <adisbladis@gmail.com>
Co-authored-by: matthewcroughan <matt@croughan.sh>
2023-06-30 23:28:00 +03:00
github-actions[bot]
ffc0f1526c
Merge master into staging-next 2023-06-26 06:01:14 +00:00
Uri Baghin
6e51b1a92e
Merge pull request #235990 from uri-canva/fix-static
nixStatic: Fix darwin.
2023-06-26 15:22:01 +10:00
Robert Scott
db3e94c3b2 hardening flags: enable fortify3 by default 2023-06-25 11:50:46 +01:00
github-actions[bot]
ea69de970d
Merge master into staging-next 2023-06-23 18:01:07 +00:00
Artturin
d88b362808 stdenv: finish removal of deprecated string/null Flags
forgot to do it before 23.05 but it's ok
2023-06-22 20:37:06 +03:00
github-actions[bot]
fc5ef80184
Merge staging-next into staging 2023-06-22 12:02:08 +00:00
Artturin
ac031478cb stdenv: use lib.isX instead of typeOf Y == X
more readable
2023-06-22 12:31:52 +03:00
Uri Baghin
fd8e2ca0df stdenv: fix makeStaticDarwin not composing with stdenvNoCC 2023-06-22 15:33:54 +10:00
Adam Joseph
b9c1ae2a5f stdenv: eliminate duplicate gnu-config in extraNativeBuildInputs
Accidentally omitted from #237968.
2023-06-16 02:33:57 -07:00
Adam Joseph
3b8e3c1f3d stdenv: updateAutotoolsGnuConfigScriptsHook unconditionally
Unlike autoreconfHook, updateAutotoolsGnuConfigScriptsHook adds
almost no compilations.  Therefore, in the interest of building the
same source code on every platform wherever possible, let's
eliminate the conditional guards around
updateAutotoolsGnuConfigScriptsHook in stdenv.
2023-06-15 19:47:38 -07:00
toonn
b7d4899ad5
Merge pull request #234861 from reckenrode/corefoundation-rpath
swift-corelibs: set NIX_COREFOUNDATION_RPATH in a hook
2023-06-11 17:13:54 +02:00
toonn
782dbaf30f
Merge pull request #234859 from reckenrode/cctools-llvm
cctools-llvm: init at 11.1.0-973.0.1
2023-06-08 19:23:56 +02:00
Randy Eckenrode
df7487ff17
cctools-llvm: init at 11.1.0-973.0.1
cctools-llvm is a replacement for cctools that replaces as much of cctools with equivalents from LLVM that it can reasonably do. This was motivated by wanting to reduce dependencies on cctools, which are updated infrequently by upstream.

To provide a motivating example, the version of `strip` included in cctools cannot properly strip the archives in compiler-rt in LLVM 15. Paths are left to bootstrap tools, resulting in failed requisites checks in the final stdenv build. Since `strip` needs replaced, the opportunity was taken to replace other provided they are functional replacements.

Note: This has to be done in cctools (or some equivalent) because some derivations (noteably LLVM) use the bintools of the stdenv directly instead of going through the wrapper.

The following tools from LLVM are not used in this derivation:

* LLD - not fully compatible with ld64 yet and potentially too big of a change;
* libtool - not a drop-in replacement yet because it does not support linker passthrough, which is needed by xcbuild;
* lipo - crashes when running the LLVM test suite;
* install_name_tool - fails when trying to build swift-corefoundation; and.
* randlib - not completely a drop-in replacement, so leaving it out for now.

If other incompatabilities are found, the tools can be reverted or made conditional. For example, cctools `strip` is preferred on older versions of LLVM (which lack the compiler-rt issue) or when cctools itself is a new enough version because `llvm-strip` on LLVM 11 produces files that older verions of `codesign_allocate` cannot process correctly.

One final caveat/note: Some tools are not duplicated or linked from cctools-port. The names of the tools and which ones were linked was determined based on what is provided upstream in Xcode and is installed on macOS system.
2023-06-06 19:05:22 -04:00
Robert Hensing
dae6b98a90
Merge pull request #234883 from sternenseemann/nix-structured-attrs
stdenv/setup.sh: deal with Nix < 2.4 structured attrs
2023-05-31 19:41:58 +02:00
Randy Eckenrode
7c043234ec
darwin.stdenv: drop NIX_COREFOUNDATION_RPATH from preHook
Rely on swift-corefoundation to set this in its hook. That way, it will
be dropped from the `extraBuildInputs` during a cross build.
2023-05-30 13:28:34 -04:00
sternenseemann
1a29857b8a stdenv/setup.sh: deal with Nix < 2.4 structured attrs
Nix does not (as far it is documented) guarantee that NIX_ATTRS_*_FILE
is set, the only [documented] guarantee seems to be:

> […] made available to the builder via the file .attrs.json in the
> builder’s temporary directory.

This guarantee is of course affected by https://github.com/NixOS/nix/issues/6736,
so it seems to be prudent to fall back to the Nix 2.3 style ATTRS_*_FILE
env vars before defaulting to the expected location in case neither is
available.

See also:

- https://github.com/NixOS/nixpkgs/pull/214937#discussion_r1178101895
- https://github.com/nixos/nixpkgs/commit/afef6588e250

[documented]: https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-structuredAttrs
2023-05-29 23:12:24 +02:00
Weijia Wang
91b69f8555
Merge pull request #228714 from helsinki-systems/upd/gnugrep
[staging] gnugrep: 3.7 -> 3.11
2023-05-24 12:47:48 +03:00
github-actions[bot]
2881fdf82f
Merge staging-next into staging 2023-05-20 12:01:51 +00:00
figsoda
98b9e41f61 pkgs: fix typos 2023-05-19 22:31:04 -04:00
Sergei Trofimovich
c3bdae6ce5
Merge pull request #223861 from eliasnaur/no-nix-lib64-in-self-rpath
stdenv: don't set NIX_LIB*_IN_SELF_RPATH by default
2023-05-17 07:08:57 +01:00
Janne Heß
11c8f46850
gnugrep/stdenv: Fix PCRE support by replacing PCRE lib 2023-05-15 10:21:34 +02:00
Robert Hensing
9524f57dd5
Merge pull request #230666 from alyssais/inputDerivation-passAsFile
stdenv: fix inputDerivation with passAsFile
2023-05-09 20:36:29 +02:00
Alyssa Ross
a295c4566e
stdenv: fix inputDerivation with passAsFile
passAsFile passes the values of Nix bindings to the builder as
files, so if those values contained references, they wouldn't end up
in the inputDerivation output.  To fix that, append the contents of
every such passed file to the output.

We only have shell builtins in this derivation, so we can't use cat.
The only way I know of appending the contents of one file to another
using only shell builtins is as I've done here, but it requires
putting the contents of the file on echo's argv.  This might end up
causing problems with large files.  Regardless, I think we should try
this, as a failure is better than silently producing an incorrect
result like the previous behavior.
2023-05-09 14:00:57 +00:00
Alyssa Ross
cd0bccbbe0
stdenv: fix inputDerivation with allowed refs
Same motivation as for the disallowed references.
2023-05-08 12:02:59 +00:00
github-actions[bot]
bf19e21376
Merge master into staging-next 2023-05-04 12:01:18 +00:00
Weijia Wang
e2adc3a3a2 bootstrap-tools-cross: add loongarch64-linux 2023-05-04 09:51:11 +03:00
github-actions[bot]
489a4937eb
Merge master into staging-next 2023-05-03 10:25:04 +00:00
Sergei Trofimovich
ba63ea9268
Merge pull request #229547 from trofi/stdenv-ca-false
stdenv: do not pass `__contentAddressed = false` as environment variable
2023-05-03 11:23:56 +01:00
github-actions[bot]
e7d9c8250f
Merge master into staging-next 2023-05-03 00:02:12 +00:00
Sergei Trofimovich
72d71dd641 stdenv: do not pass __contentAddressed = false as environment variable
`nix-2.4+` automatically filters `__contentAddressed` out of the
environment. But not `nix-2.3`. This make `.drv` to differ between
unset and `__contentAddressed = false` derivations.

This change makes them equal by filtering out `__contentAddressed`
unless it's set to `true`.
2023-05-02 22:57:54 +01:00
Weijia Wang
658f507622
Merge pull request #229289 from wegank/loongarch64-cross-bootstrap
pkgsCross.loongarch64-linux.freshBootstrapTools.bootstrapFiles: fix build
2023-05-02 22:39:14 +03:00
github-actions[bot]
bdb79914d6
Merge master into staging-next 2023-05-02 18:01:20 +00:00
Weijia Wang
c78d0d9d39 pkgsCross.loongarch64-linux.freshBootstrapTools.bootstrapFiles: fix build 2023-05-02 20:25:05 +03:00
Weijia Wang
e6e3e4da34
Merge pull request #229372 from wegank/loongarch64-native-bootstrap
stdenv: always update config script on loongarch64-linux
2023-05-02 18:49:43 +03:00
github-actions[bot]
0ee4180a08
Merge staging-next into staging 2023-05-02 00:03:00 +00:00
Weijia Wang
b017e29705 stdenv: always update config script on loongarch64-linux 2023-05-02 01:02:57 +03:00
Artturi
ea8e21f9fa
Merge pull request #228759 from Artturin/splitfactoroutmeta 2023-05-02 00:46:31 +03:00
Emily Trau
2ad68a7d90 stdenv: factor out meta attr augmentation for reusability
stdenv: check if separating commonMeta from assertValidity fixes laziness issue

```
error: cannot coerce null to a string

at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/mr-est/ofborg-evaluator-3/pkgs/development/coq-modules/coq-lsp/default.nix:34:60:

    33|     homepage = "https://github.com/ejgallego/coq-lsp";
    34|     changelog = "https://github.com/ejgallego/coq-lsp/blob/${defaultVersion}/CHANGES.md";
```
2023-05-01 20:18:11 +03:00
github-actions[bot]
cff4be2a2b
Merge staging-next into staging 2023-05-01 12:01:54 +00:00
Alyssa Ross
5130c4f4ef freshBootstrapTools: enable musl on RISC-V
musl now supports RISC-V.  Let's centralise musl availability checks
in musl.meta.platforms, so we don't have to keep cleaning up ad-hoc
checks like this all over the tree.
2023-05-01 11:25:04 +00:00
Alyssa Ross
2839c94f1e pkgsLLVM.stdenv: use clangNoLibc when libc is null
clangNoLibc always uses LLVM bintools, so it still has the useLLVM
semantics.
2023-04-28 10:01:22 +00:00
Adam Joseph
bc024e10ef
Merge pull request #228281 from amjoseph-nixpkgs/pr/bootstrapFiles/cross/fix
stdenvBootstrapTools: drop no-longer-needed, now-broken cross clause
2023-04-27 03:58:30 +00:00
Adam Joseph
38f41a27b4 stdenvBootstrapTools: drop no-longer-needed, now-broken cross clause
The comment stating that "GCC has certain things built in
statically" is no longer true as of
https://github.com/NixOS/nixpkgs/pull/209870.  Moreover, as pointed
out in https://github.com/NixOS/nixpkgs/pull/228130 this
optionalString block no longer builds for
pkgsCross.*.freshBootstrapTools.

It is safe to drop this clause now.
2023-04-26 00:02:00 -07:00
Weijia Wang
91bc06b6d7 stdenv: avoid -p flag for strip on aarch64-darwin 2023-04-24 10:33:11 +03:00
Sergei Trofimovich
e7eb209fed
stdenv: avoid -p flag for strip when boostrapping x86_64-darwin
The stdenv wouldn't build with it, as
compiler-rt-libc-11.1.0/lib/darwin/libclang_rt.*_osx.a
retained reference to SDKs (which we forbid for final stdenv).

Assigned authorship to Trofi; I just bisected and added condition.
https://github.com/NixOS/nixpkgs/pull/224669#issuecomment-1518225496
2023-04-22 16:05:35 +02:00
github-actions[bot]
6176f16de2
Merge staging-next into staging 2023-04-14 12:02:03 +00:00
Sandro
b04d4bad27
Merge pull request #216992 from SuperSandro2000/stdenvNative-fix-eval
{bintools,cc}-wrapper: don't fallback to version = null
2023-04-14 11:22:20 +02:00
github-actions[bot]
ae00b7fb14
Merge staging-next into staging 2023-04-13 00:02:47 +00:00
Sandro Jäckel
b0ee79f9fd
mkDerivation: show meaningful error when version is set to null
instead of `Cannot coerce null to string`
2023-04-12 22:08:36 +02:00
Artturi
c88044fe0b
Merge pull request #225929 from Artturin/fixadapters2 2023-04-12 23:04:31 +03:00
Artturin
67a84d1428 stdenvAdapters: fix preservation of env.NIX_CFLAGS_COMPILE
we have managed to migrate to NIX_CFLAGS_COMPILE to the env attrset well
enough that we don't need to support having it toplevel. mkDerivation
will throw if there's a attr in both env and toplevel so no need to
worry about that
2023-04-12 21:27:34 +03:00
github-actions[bot]
18530e57da
Merge staging-next into staging 2023-04-12 00:02:52 +00:00
github-actions[bot]
47157337bf
Merge master into staging-next 2023-04-12 00:02:18 +00:00
Artturin
9adf83737f stdenvAdapters: fix preservation of env 2023-04-11 21:25:52 +03:00
github-actions[bot]
9dd2031c08
Merge staging-next into staging 2023-04-06 16:11:47 +00:00
Adam Joseph
3778e07994 stdenv: fix pkgsMusl
I broke `pkgsMusl` with #209870.

Something odd is happening with `xgcc` (the temporary compiler that
should be used only to compile `gcc`, although we are using it to
compile a temporary `patchelf` too) and `libstdc++`.

The temporary fix in this commit is to use `-static-libstdc++` for
the ephemeral `patchelf` built by `xgcc`.  It isn't pretty, but it
appears to work.

Incorporates:

- https://github.com/NixOS/nixpkgs/pull/224945
2023-04-06 03:16:31 -07:00
Adam Joseph
1847b1bc15 stdenv/linux: fix patchelf confusion
The stage before `xgcc` creates the first compiled patchelf
(i.e. not from bootstrapFiles).

The `xgcc` stage was inadvertently switching *back* to using the
patchelf *from* the bootstrapFiles.

The first commit in this PR adds self-checking comments (assertions)
to make it clear where each stage's patchelf comes from.

The second commit fixes the bug, and updates the self-checking
comments.
2023-04-05 22:18:27 -07:00
Adam Joseph
2abf285e2b stdenv/linux: add is{From,BuiltBy} assertions for patchelf 2023-04-05 22:18:22 -07:00
Sergei Trofimovich
c2e14a73e6 xgcc: avoid libc.so mix between gcc and binutils-ld linker plugin
Without the change when I attempt to built `nixpkgs` with weekly
`gcc-13` (it pulls in `flex` as a build input`) I am getting build
failure related to glibc mix caused by glibc loading:

    ...-binutils-patchelfed-ld-2.40/bin/ld: ...-xgcc-13.0.0/libexec/gcc/x86_64-unknown-linux-gnu/13.0.1/liblto_plugin.so:
    error loading plugin: ...-bootstrap-tools/lib/libpthread.so.0: undefined symbol: __libc_vfork, version GLIBC_PRIVATE

The change disables LTO plugin entirely to avoid loading of `glibc` mix.
2023-04-05 07:43:22 +01:00
Bernardo Meurer
f1f6ca8bcd
Merge pull request #209870 from amjoseph-nixpkgs/pr/stdenv/external-gcc-bootstrap 2023-04-03 08:19:03 -07:00
Adam Joseph
96588eb3de gcc: add common/checksum.nix
This commit adds `gcc/common/checksum.nix`, which contains code
common to both gcc11 and gcc12, implementing the `enableChecksum`
feature.

When gcc's built-in bootstrap (`--enable-bootstrap`) is used, gcc
compiles itself three times and compares a hash of the unlinked `.o`
files from the second and third compilation.  The
`enableChecksum=true` parameter performs the same comparison as part
of the `postInstall` phase.

Notably, `enableChecksum=true` can be used with `enableBootstrap=false`.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2023-04-02 13:49:53 -07:00
Adam Joseph
fed2300bea unpack-bootstrap-tools.sh: patchelf libgcc_s.so.1 2023-04-02 13:49:53 -07:00
Adam Joseph
d7fe0a5548 make-bootstrap-tools.nix: use a patchelf built with -static-{libgcc,libstdc++}
Our bootstrap-files unpacker has always relied on a lot of unstated
assumptions, one of them being that every library has a DT_NEEDED
for librt.so, so patchelf'ing something into the RUNPATH into
librt.so means that it will be searched for every library load in
all of the bootstrap-files.

Unfortunately that assumption is not true for libgcc.

This causes problems, because patchelf links against libgcc (and
against libstdc++, which links against libgcc).  So we can't use
patchelf on libgcc, because it needs libgcc, so patchelf doesn't
work until libgcc is patchelfed.

The robust solution here is to use static linking for the copy of
patchelf that is shipped with the bootstrap-files.  We don't have to
go all the way to a statically linked libc; just -static-libgcc and
-static-libstdc++ are enough to break the circular dependency.
2023-04-02 13:49:53 -07:00
Adam Joseph
86ca0faff7 make-bootstrap-tools.nix: cp libgcc_s without -d
We do not want to preserve the symlinks from libgcc_s, since they
point to another outpath.  We want to copy from that outpath rather
than link to it.
2023-04-02 13:49:53 -07:00
Adam Joseph
c6bd37a691 make-bootstrap-tools.nix: ship libisl.so
Now that we've dropped the
gcc-links-statically-to-lib{isl,mpfr,mpc,gmp} hack, our gcc needs
libisl.so.  Let's add it to the bootstrap-files.
2023-04-02 13:49:53 -07:00
Adam Joseph
7553d0fe29 stdenv: Nix-driven bootstrap of gcc
#### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like https://github.com/NixOS/nixpkgs/pull/132343

 #### Incorporates

- https://github.com/NixOS/nixpkgs/pull/210004
- https://github.com/NixOS/nixpkgs/pull/36948 (unreverted)
- https://github.com/NixOS/nixpkgs/pull/210325
- https://github.com/NixOS/nixpkgs/pull/210118
- https://github.com/NixOS/nixpkgs/pull/210132
- https://github.com/NixOS/nixpkgs/pull/210109
- https://github.com/NixOS/nixpkgs/pull/213909
- https://github.com/NixOS/nixpkgs/pull/216136
- https://github.com/NixOS/nixpkgs/pull/216237
- https://github.com/NixOS/nixpkgs/pull/210019
- https://github.com/NixOS/nixpkgs/pull/216232
- https://github.com/NixOS/nixpkgs/pull/216016
- https://github.com/NixOS/nixpkgs/pull/217977
- https://github.com/NixOS/nixpkgs/pull/217995

 #### Closes

- Closes #108305
- Closes #108111
- Closes #201254
- Closes #208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: https://github.com/NixOS/nixpkgs/pull/210004
[versioned directory]: https://github.com/NixOS/nixpkgs/pull/209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: 5e4ec82181/gnu/packages/gcc.scm (L253)
[mentioned]: https://github.com/NixOS/nixpkgs/pull/210112#issuecomment-1379608483
[crisis]: https://github.com/NixOS/nixpkgs/issues/108305
[foreign]: https://github.com/NixOS/nixpkgs/pull/170857#issuecomment-1170558348
[static lib{mpfr,mpc,gmp,isl}.a hack]: 2f1948af9c/pkgs/stdenv/linux/default.nix (L380)
2023-04-02 13:49:41 -07:00
Adam Joseph
70d34589bd stdenv/linux: factor out commonGccOverrides
This commit has no effect on eval.  It simply factors out a common
subexpression.
2023-04-02 03:04:20 -07:00
Adam Joseph
e8b10284f3 linux/default.nix: use mipsel.nix instead of longson.nix for mips32
Right now our bootstrapFiles-selecting algorithm uses the
`loongson2f.nix` bootstrapFiles (which were not built by Hydra).

These bootstrapFiles don't work anymore.  They were added in 2010 by
40405d03ac.

This commit causes mipsel-linux native builds to use the Hydra-built
bootstrap files from this PR instead:

  https://github.com/NixOS/nixpkgs/pull/183487
2023-04-01 23:25:25 -07:00
Elias Naur
88fd6601d8 stdenv: remove the NIX_LIB*_IN_SELF_RPATH environment variables
The NIX_LIB64|32_IN_SELF_RPATH environment variables control whether
to add lib64 and lib32 to rpaths. However, they're set depending
on the build paltform, not the target platform and thus their values
are incorrect for for cross-builds.

On the other hand, setting them according to the build platform introduce
pointless differences in build outputs; see #221350 for details.

This change fixes the issues by boldly removes the NIX_LIB*_IN_SELF_RPATH
facility altogether, in the hope that it is no longer necessary. They
were introduced in 2009, long before nixpkgs had good support for
cross-builds.

Fixes #221350
2023-03-31 10:47:16 -06:00
Vladimír Čunát
12dd95fbb1
Merge branch 'master' into staging-next 2023-03-24 09:07:41 +01:00
Adam Joseph
13507da345 check-meta.nix: fix self-contradictory error messages
See https://github.com/NixOS/nixpkgs/pull/222792#pullrequestreview-1356114111

You can't just `lib.filter _ lib.systems.all` -- that throws away
important information, leading to nixpkgs disagreeing with itself
like this:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is only supported on ... x86_64-linux but not on requested x86_64-linux, refusing to evaluate.
```

After:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is not available on the requested hostPlatform:
         hostPlatform.config = "x86_64-unknown-linux-musl"
         package.meta.platforms = [
           "aarch64-linux"
           "armv5tel-linux"
           "armv6l-linux"
           "armv7a-linux"
           "armv7l-linux"
           "i686-linux"
           "m68k-linux"
           "microblaze-linux"
           "microblazeel-linux"
           "mipsel-linux"
           "mips64el-linux"
           "powerpc64-linux"
           "powerpc64le-linux"
           "riscv32-linux"
           "riscv64-linux"
           "s390-linux"
           "s390x-linux"
           "x86_64-linux"
         ]
         package.meta.badPlatforms = [
           {
             isStatic = true;
             parsed = { };
           }
         ]
       , refusing to evaluate.
```
2023-03-24 00:15:57 -07:00
Weijia Wang
b46c97e1dc stdenv: generalise showPlatforms 2023-03-23 23:47:11 +02:00
github-actions[bot]
307b719414
Merge master into staging-next 2023-03-23 18:01:20 +00:00
Weijia Wang
2a7e1e0228 stdenv: fix error with patterned platforms 2023-03-23 17:27:49 +02:00
Sandro
979c6c1fe4
Merge pull request #213780 from SuperSandro2000/check-meta-platform
stdenv: show supported and requested platforms when check meta fails
2023-03-23 15:01:34 +01:00
github-actions[bot]
455127ad5e
Merge master into staging-next 2023-03-16 18:01:20 +00:00
Bernardo Meurer
6e55733359
Merge pull request #219747 from Stunkymonkey/deprecate-isNull 2023-03-16 11:10:22 -03:00
Martin Weinelt
19680e9902
Merge pull request #217568 from trofi/stdenv-parallel-install
stdenv/generic/setup.sh: enable parallel installs for parallel builds
2023-03-15 17:10:19 +00:00
Martin Weinelt
ef91384e6f Merge remote-tracking branch 'origin/master' into staging-next 2023-03-10 13:09:25 +01:00
Bernardo Meurer
e9cf2d1a41
Merge pull request #188334 from amjoseph-nixpkgs/pr/bootstrapFiles/mips64n32 2023-03-09 21:21:15 -05:00
Adam Joseph
49878856e6 https://github.com/NixOS/nixpkgs/pull/188334#issuecomment-1445425412 2023-03-07 02:11:30 -08:00
Felix Buehler
d10e69c86b treewide: deprecate isNull
https://nixos.org/manual/nix/stable/language/builtins.html#builtins-isNull
2023-03-06 22:40:04 +01:00
Martin Weinelt
5aeab34845
Merge pull request #218301 from rrbutani/fix/separate-debuginfo-with-lld
Fix `separate-debug-info` with lld
2023-03-04 00:43:26 +00:00
Weijia Wang
5028ec96c9
Merge pull request #202347 from stephank/feat/bootstrap-aarch64-darwin
stdenvBootstrapTools: native aarch64-darwin build
2023-03-03 20:05:28 +02:00
Vladimír Čunát
763470bbb0
Merge #207135: stdenv: aarch64-linux: gcc9 -> gcc12
...into staging
Also merge the commit that was referenced as the base for the build
of the new bootstrap tools (although others would give the same).
2023-03-01 12:54:52 +01:00
Sergei Trofimovich
69cf5181c3 stdenv/generic/setup.sh: enable parallel installs by default
The primary motivating example is openssl:

Before the change full package build took 1m54s minutes.
After the change full package build takes 59s.

About a 2x speedup.

The difference is visible because openssl builds hundreds of manpages
spawning a perl process per manual in `install` phase. Such a workload
is very easy to parallelize.

Another example would be `autotools`+`libtool` based build system where
install step requires relinking. The more binaries there are to relink
the more gain it will be to do it in parallel.

The change enables parallel installs by default only for buiilds that
already have parallel builds enabled. There is a high chance those build
systems already handle parallelism well but some packages will fail.

Consistently propagated the enableParallelBuilding to:
- cmake (enabled by default, similar to builds)
- ninja (set parallelism explicitly, don't rely on default)
- bmake (enable when requested)
- scons (enable when requested)
- meson (set parallelism explicitly, don't rely on default)
- waf (set parallelism explicitly, don't rely on default)
- qmake-4/5/6 (enable by default, similar to builds)
- xorg (always enable, similar to builds)
2023-02-26 22:02:09 +00:00
Weijia Wang
f05b5d4054 stdenv: aarch64-linux: gcc9 -> gcc12
Hydra job building them: https://hydra.nixos.org/build/208909151

The bootstrap files can be reproduced on the commit 21ec906463, e.g. by:
  cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
    file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
  nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
  nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
    sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=

You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
2023-02-26 23:26:30 +02:00
Rahul Butani
d364ee8d13
mkDerivation: do not disable separateDebugInfo on LLVM stdenvs
This was disabled here: b86e62d30d (diff-282a02cc3871874f16401347d8fadc90d59d7ab11f6a99eaa5173c3867e1a160)

h/t to @teh: b86e62d30d (commitcomment-77916294)
for pointing out that the failure that @matthewbauer was
seeing was caused by the `separate-debug-info.sh` `build-id` length
requirement that #146275 will relax

`lld` has had `--build-id` support dating back to LLVM4: https://reviews.llvm.org/D18091

This predates every `llvmPackages_` version currently in nixpkgs (and
certainly every version actually still used in `useLLVM` stdenvs) so
with the previous commit (asking `ld` for sufficiently long SHA1 hashes)
I think we can safely enable `separateDebugInfo` when using LLVM
bintools.
2023-02-25 12:49:40 -06:00
github-actions[bot]
3cdd771820
Merge staging-next into staging 2023-02-23 18:01:49 +00:00
Alyssa Ross
52c286ee5b
Merge remote-tracking branch 'origin/master' into staging-next
Conflicts:
	pkgs/development/libraries/pmdk/default.nix
2023-02-23 13:51:34 +00:00
Artturin
f9fdf2d402 treewide: move NIX_CFLAGS_COMPILE to the env attrset
with structuredAttrs lists will be bash arrays which cannot be exported
which will be a issue with some patches and some wrappers like cc-wrapper

this makes it clearer that NIX_CFLAGS_COMPILE must be a string as lists
in env cause a eval failure
2023-02-22 21:23:04 +02:00
Sergei Trofimovich
3057968396
Merge pull request #210019 from amjoseph-nixpkgs/pr/stdenv/verify-comments
express #208478 as assertions
2023-02-21 19:38:58 +00:00
Adam Joseph
d7aad24531 express #208478 as assertions
PR #208478 added a lot of documentation about which packages were
rebuilt in each stage of the stdenv bootstrap.  However nothing
checks that these comments agree with reality; they can bitrot over
time.  This PR rewrites those comments as assertions, so they cannot
bitrot.

This conversion did expose some ambiguity in our scheme for naming
the stages.   Suppose that `pkgs.stdenv.name=="stdenv-stage4", then
which of these is "the stage4 coreutils"?

```
pkgs.coreutils
pkgs.stdenv.__bootPackages.coreutils
```

The choice is arbitrary, and both choices have confusing corner
cases.  We should revisit this at some point.
2023-02-20 23:20:30 -08:00
Artturi
ee54eb7d21
Merge pull request #216383 from Artturin/bintoolswrappermold 2023-02-17 19:32:06 +02:00
Artturin
299a7bd35e stdenvAdapters: add useMoldLinker 2023-02-17 06:35:24 +02:00
Robert Scott
0eedcfc3f4
Merge pull request #212498 from risicle/ris-fortify3
hardening flags: add `FORTIFY_SOURCE=3` support
2023-02-16 21:19:30 +00:00
github-actions[bot]
c4fe2133de
Merge staging-next into staging 2023-02-15 06:01:44 +00:00
github-actions[bot]
58cfebde3e
Merge staging-next into staging 2023-02-15 00:02:50 +00:00
Felix Buehler
bc3d5934d7 treewide: use lib.optionals 2023-02-14 19:11:59 +01:00
Felix Buehler
cdb39a86e0 treewide: use optionalString 2023-02-13 21:52:34 +01:00
github-actions[bot]
29c2637dda
Merge staging-next into staging 2023-02-13 12:02:05 +00:00
github-actions[bot]
fd8daee7c2
Merge master into staging-next 2023-02-13 12:01:30 +00:00
Weijia Wang
7c026bae62
stdenv: aarch64-linux: update the bootstrap tools (incl. busybox)
Hydra job building them: https://hydra.nixos.org/build/208909151

The bootstrap files can be reproduced on the parent commit, e.g. by:
  cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
    file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
  nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
  nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
    sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=

You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
2023-02-12 10:09:35 +01:00
Stéphan Kochen
c3693fbfd5 stdenvBootstrapTools: native aarch64-darwin build 2023-02-11 20:11:55 +01:00