Commit Graph

324 Commits

Author SHA1 Message Date
Vladimír Čunát
8c0c955a09
Merge branch 'master' into staging-next 2024-04-19 07:52:28 +02:00
sternenseemann
7be562d046 wrapCC, wrapBintools: move expand-response-params bootstrapping out
The cc and bintools wrapper contained ad hoc bootstrapping logic for
expand-response-params (which was callPackage-ed in a let binding). This
lead to the strange situation that the bootstrapping logic related to
expand-response-params is split between the wrapper derivations (where
it is duplicated) and the actual stdenv bootstrapping.

To clean this up, the wrappers simply should take expand-response-params
as an ordinary input: They need an adjacent expand-response-params (i.e.
one that runs on their host platform), but don't care about the how.
Providing this is only problematic during stdenv bootstrapping where we
have to pull it from the previous stage at times.
2024-04-18 20:49:13 +02:00
sternenseemann
b2a568906a wrapCC, wrapBintools: use runtimeShell instead of stdenv shell
We don't need to artificially make sure that we can execute the wrapper
scripts on the build platform by using stdenv's shell (which comes from
buildPackages) since our cross infrastructure will get us the wrapper
from buildPackages. The upside of this change is that cross-compiled
wrappers (e.g. pkgsCross.aarch64-multiplatform.gcc) will actually work
when executed!

For bootstrapping this is also not a problem, since we have a long
build->build platform chain so runtimeShell is just as good as
stdenvNoCC.shell. We do fall back to old ways, though, by explicitly
using the bootstrap-tools shell in stage2, so the adjacent bash is only
used from stage4 onwards. This is unnecessary in principle (I'll try
removing this hack in the future), but ensures this change causes zero
rebuilds.
2024-04-18 20:49:13 +02:00
annalee
8e7f7b4fa0 stdenv.darwin: bootstrap darwin using updated tools
- update the hashes and tools needed to extract the bootstrap-tools archive
- unify the x64 and aarch64 unpack process
2024-04-07 18:33:42 +02:00
github-actions[bot]
882e0f27d3
Merge staging-next into staging 2024-04-05 18:01:27 +00:00
Sergei Trofimovich
178f1e1f5d pkgs/top-level/release.nix: drop unused dist targets
The PR https://github.com/NixOS/nixpkgs/pull/284090 exposed `build`
target directly on hydra. We are using it now in
https://github.com/NixOS/nixpkgs/tree/master/maintainers/scripts/bootstrap-files
to generate bootstrap tarballs on `linux` and `darwin`.

`dist` was not very useful as it was a bit hard to link back to hydra
jobs that build it. Let's just drop it.
2024-04-04 21:40:08 +01:00
github-actions[bot]
fef3e5b690
Merge staging-next into staging 2024-04-03 00:02:45 +00:00
annalee
39b3d15ad8
freshBootstrapTools.{build,test}: group tools together 2024-04-01 22:19:04 +00:00
annalee
5748bfdd76
freshBootstrapTools.{build,test}: allow exes with libs unpack 2024-04-01 22:19:04 +00:00
annalee
72870b3293
freshBootstrapTools.{build,test}: fix as script. test all executables
as is a wrapper around clang and the path to bash and clang need to be
updated when unpacked

run all executables in bin get run in the test
2024-04-01 22:17:02 +00:00
a-n-n-a-l-e-e
7ad496eb80
Merge pull request #300032 from NixOS/revert-295558-darwin-bootstrap-update
Revert "stdenv.darwin: bootstrap darwin using updated tools"
2024-03-29 11:05:07 -07:00
a-n-n-a-l-e-e
6b44d9d62d
Revert "stdenv.darwin: bootstrap darwin using updated tools" 2024-03-29 09:33:31 -07:00
Randy Eckenrode
71c6ee9295
darwin.stdenv: add sdkRoot to extraNativeBuildInputs
Setting the SDK root by default allows `overrideSDK` to correctly set
the SDK version when using a different SDK. It also allows the correct
SDK version to be set when using an older deployment target. Not setting
the correct SDK version can result in unexpected behavior at runtime.

Examples:

* Automatic dark mode switching requires linking against an SDK version
  of 10.14 or newer. With the current behavior, the only way to do this
  is by using a 10.14+ deployment target even when the application
  supports older platforms when build with a newer SDK.
* MetalD3D checks that the system version is at least 14.0. The API it
  uses returns a compatibility version when the the SDK is older than
  11.0, which causes it to display an error and terminate the
  application even when even when its requirements are all met.
2024-03-29 10:07:50 -04:00
Randy Eckenrode
f61e189ad3
overrideSDK: support all deps attributes
This is effectively a rewrite of `overrideSDK`. It was required because
`wrapGAppsHook` propagates `depsTargetTarget` with the expectation that
it will effectively be `buildInputs` when the hook is itself used as a
`nativeBuildInput`. This propagates Gtk, which itself propagates the
default Dariwn SDK, making it effectively impossible to override the SDK
when a package depends on Gtk and uses `wrapGAppsHook`.

This rewrite implements the following improvements:

* Cross-compilation should be supported correctly (untested);
* Supports public and private frameworks;
* Supports SDK `libs`;
* Remaps instead of replacing extra (native) build inputs in the stdenv;
* Updates any Darwin framework references in `nix-support`; and
* It updates `xcodebuild` regardless of which input its in.

The implementation avoids recursion for performance reasons. Instead, it
enumerates transitive dependencies and walks the list from the leaf
packages backwards to the parent packages.
2024-03-29 10:03:08 -04:00
Randy Eckenrode
1f231173a5
darwin.stdenv: drop curl from assertions 2024-03-29 10:00:30 -04:00
Randy Eckenrode
017eda24ae
darwin.stdenv: use wrapBintoolsWith instead of a direct import 2024-03-29 10:00:28 -04:00
annalee
f8ad6442d5
stdenv.darwin: bootstrap darwin using updated tools
- update the hashes and tools needed to extract the bootstrap-tools archive
- unify the x64 and aarch64 unpack process
- unpin libxml for python-minimal
2024-03-22 13:27:27 +00:00
Randy Eckenrode
b7d77e743c
Merge pull request #296750 from reckenrode/darwin-nocurl
darwin.stdenv: drop curl from the bootstrap
2024-03-21 23:42:15 -04:00
annalee
3cdd8d05f0
stdenv.darwin.make-boostrap-tools: include unpack script in the archive 2024-03-19 17:28:32 +00:00
annalee
b310830359
stdenv.darwin.make-boostrap-tools: update tools and unpin LLVM11
- unpin LLVM11. fix discrepancy with freshBootstrapTools and the tools
  built on hydra. pinning the stdenv for the hydra build doesn't pin the
  tools as the included packages are able to change.

- remove unused LLVM tools & libs which reduces the uncompressed and
  compressed file sizes by more than 1/2. compressed tarball is now 40M
  and uncompressed is around 200M

- add @loader_path/. to dylibs that reference other libs in the archive.
  this is needed for libraries with re-exports.

- validate shared objects with @rpath references contain the reference
  in lib

- add a test to verify that the @loader_path/ works for libc++ as it
  re-exports libc++abi
2024-03-19 17:28:32 +00:00
annalee
88f192b247
stdenv.darwin.make-boostrap-tools: refactor to use getBin, getDev and getLib 2024-03-19 17:28:32 +00:00
annalee
7c4b2a599c
freshBootstrapTools: LLVM11 unpin 2024-03-19 14:54:23 +00:00
Randy Eckenrode
07451425e7
darwin.stdenv: drop curl from the bootstrap
The Darwin bootstrap currently requires curl, but it is not strictly
required. The bootstrap requires it for two things:

* Fetchers; and
* As a transitive dependency of llvm-manpages (via Sphinx).

For the fetchers, the bootstrap curl can be used. For hatch-vcs, the
dependency, its tests can be disabled. Doing this allows curl to be
dropped from the Darwin stdenv bootstrap.
2024-03-17 16:48:35 -04:00
a-n-n-a-l-e-e
ce789e7e35
llvmPackages_{12,13,14,15,16,17,git}.{libcxx,libcxxabi}: merge libcxxabi into libcxx (#292043)
- merge libcxxabi into libcxx for LLVM 12, 13, 14, 15, 16, 17, and git.
- remove the link time workaround `-lc++ -lc++abi` from 58 packages as it is no longer required.
- fixes https://github.com/NixOS/nixpkgs/issues/166205
- provides alternative fixes for. https://github.com/NixOS/nixpkgs/issues/269548 https://github.com/NixOS/nix/issues/9640
- pkgsCross.x86_64-freebsd builds work again

This change can be represented in 3 stages
1. merge libcxxabi into libcxx -- files: pkgs/development/compilers/llvm/[12, git]/{libcxx, libcxxabi}
2. update stdenv to account for merge -- files: stdenv.{adapters, cc.wrapper, darwin}
3. remove all references to libcxxabi outside of llvm (about 58 packages modified)

### merging libcxxabi into libcxx
- take the union of the libcxxabi and libcxx cmake flags
- eliminate the libcxx-headers-only package - it was only needed to break libcxx <-> libcxxabi circular dependency
- libcxx.cxxabi is removed. external cxxabi (freebsd) will symlink headers / libs into libcxx.
- darwin will re-export the libcxxabi symbols into libcxx so linking `-lc++` is sufficient.
- linux/freebsd `libc++.so` is a linker script `LINK(libc++.so.1, -lc++abi)` making `-lc++` sufficient.
- libcxx/default.nix [12, 17] are identical except for patches and `LIBCXX_ADDITIONAL_LIBRARIES` (only used in 16+)
- git/libcxx/defaul.nix  does not link with -nostdlib when useLLVM is true so flag is removed. this is not much different than before as libcxxabi used -nostdlib where libcxx did not, so libc was linked in anyway.

### stdenv changes
- darwin bootstrap, remove references to libcxxabi and cxxabi
- cc-wrapper: remove c++ link workaround when libcxx.cxxabi doesn't exist (still exists for LLVM pre 12)
- adapter: update overrideLibcxx to account for a pkgs.stdenv that only has libcxx

### 58 package updates
- remove `NIX_LDFLAGS = "-l${stdenv.cc.libcxx.cxxabi.libName}` as no longer needed
- swift, nodejs_v8 remove libcxxabi references in the clang override

https://github.com/NixOS/nixpkgs/pull/292043
2024-03-11 03:53:37 -07:00
github-actions[bot]
2f68b418f1
Merge staging-next into staging 2024-01-30 06:01:22 +00:00
Sergei Trofimovich
43695b8db4 pkgs/stdenv/darwin: move bootstrap files definitions to bootstrap-files/ directory
The change moves definition of bootstrap files slightly closer to
`linux` structure to eventually allow those to update in bulk:
https://github.com/NixOS/nixpkgs/issues/253713
2024-01-28 22:41:09 +00:00
Robert Scott
40868719b0 cc-wrapper: add zerocallusedregs hardening flag
this uses the value `used-gpr` which seems to be a commonly
chosen value for general use
2024-01-20 13:48:33 +00:00
Sergei Trofimovich
fccb48062f darwin/make-bootstrap-tools.nix: set -headerpad_max_install_names for coreutils
Without the change bootstrapTools build fails as:
https://cache.nixos.org/log/g5wyq9xqshan6m3kl21bjn1z88hx48rh-stdenv-bootstrap-tools.drv

    error: install_name_tool: changing install names or rpaths can't be redone for: /nix/store/0hxg356h7lnl2hck23wrdpbib3cckx41-stdenv-bootstrap-tools/bin/tac (for architecture x86_64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)
2023-12-13 17:57:25 +00:00
Peder Bergebakken Sundt
f41aba3739 treewide: remove unreferenced patch files
Found with `fd \\.patch$ pkgs/ -x bash -c 'rg -F "{/}" pkgs/ -q || echo {}'`
2023-12-01 06:11:20 +01:00
K900
b40035f472
Merge pull request #267058 from toonn/bootstrap-tools-specify-llvm
freshBootstrapTools: Overlay the package set with the desired LLVM
2023-11-17 14:06:37 +03:00
Adam Joseph
985ff1cb88
Update pkgs/stdenv/darwin/make-bootstrap-tools.nix 2023-11-17 08:26:41 +00:00
toonn
887d33a089 freshBootstrapTools: Overlay the package set with the desired LLVM
As reported in #241692, since the `llvmPackages` bump the
bootstrap-tools started failing to build due to a mismatch in LLVM
versions used to build certain tools.

By overlaying the imported package set to specify `llvmPackages`, we get
everything built with the expected LLVM version.
2023-11-12 15:08:11 +01:00
Randy Eckenrode
940180423a
Merge pull request #265102 from reckenrode/darwin-corefoundation
darwin.stdenv: use CoreFoundation instead of CF
2023-11-11 12:10:10 -05:00
Artturi
fbb3ff5e12
Merge pull request #243161 from nh2/more-enableStatic 2023-11-07 03:49:21 +02:00
Randy Eckenrode
daa79a1d2d
darwin.stdenv: use CoreFoundation instead of CF
This patch switches the CoreFoundation on x86_64-darwin from the open
source swift-corelibs-foundation (CF) to the system CoreFoundation.

This change was motivated by failures building packages for the current
staging-next cycle #263535 due to an apparent incompatibility with the
rpath-based approach to choosing CF or CoreFoundation and macOS 14. This
error often manifests as a crash with an Illegal Instruction.

For example, building aws-sdk-cpp for building Nix will fail this way.

https://hydra.nixos.org/build/239459417/nixlog/1

    Application Specific Information:
    CF objects must have a non-zero isa

    Error Formulating Crash Report:
    PC register does not match crashing frame (0x0 vs 0x7FF8094DD640)

    Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
    0   CoreFoundation                	    0x7ff8094dd640 CF_IS_OBJC.cold.1 + 14
    1   CoreFoundation                	    0x7ff8094501d0 CF_IS_OBJC + 60
    2   CoreFoundation                	    0x7ff8093155e8 CFRelease + 40
    3   ???                           	       0x10c7a2c61 s_aws_secure_transport_ctx_destroy + 65
    4   ???                           	       0x10c87ba32 aws_ref_count_release + 34
    5   ???                           	       0x10c7b7adb aws_tls_connection_options_clean_up + 27
    6   ???                           	       0x10c596db4 Aws::Crt::Io::TlsConnectionOptions::~TlsConnectionOptions() + 20
    7   ???                           	       0x10c2d249c Aws::CleanupCrt() + 92
    8   ???                           	       0x10c2d1ff0 Aws::ShutdownAPI(Aws::SDKOptions const&) + 64
    9   ???                           	       0x102d9bc6f main + 335
    10  dyld                          	       0x202f333a6 start + 1942

According to a [post][1] on the Apple developer forums, hardening was
added to CoreFoundation, and this particular message occurs when you
attempt to release an object it does not recognize as a valid CF object.
(Thank you to @lilyinstarlight for finding this post).

When I switched aws-sdk-cpp to link against CoreFoundation instead of
CF, the error went away. Somehow both libraries were being used. To
prevent dependent packages from linking the wrong CoreFoundation, it
would need to be added as a propagated build input.

Note that there are other issues related to mixing CF and CoreFoundation
frameworks. #264503 fixes an issue with abseil-cpp where it propagates
CF, causing issues when using a different SDK version. Mixing versions
can also cause crashes with Python when a shared object is loaded that
is linked to the “wrong” CoreFoundation.

`NIX_COREFOUNDATION_RPATH` is supposed to make sure the right
CoreFoundation is being used, but it does not appear to be enough on
macOS 14 (presumably due to the hardening). While it is possible to
propagate CoreFoundation manually, the cleaner solution is to make it
the default. CF remains available as `darwin.swift-corelibs-foundation`.

[1]: https://developer.apple.com/forums/thread/739355
2023-11-02 21:20:55 -04:00
Niklas Hambüchen
e73b7f8d63 bzip2: Add enableStatic. See #61575 2023-10-20 19:59:30 +00:00
Randy Eckenrode
df14e86d9c
stdenv.darwin: fix infinite recursion after curl update
curl needs to link against several frameworks, but building the
frameworks (directly or indirectly) depends on curl via fetchurl and
fetchFromGitHub. Break the infinite recursion by building the SDKs’
dependencies in the last stage of the stdenv bootstrap using the prior
stage’s fetchurl and fetchFromGitHub.
2023-10-13 18:51:05 -04:00
Alyssa Ross
1da4b5c99e
release.nix: namespace bootstrap tools with triples
This will allow buliding bootstrap tools for platforms with
non-default libcs, like *-unknown-linux-musl.

This gets rid of limitedSupportSystems/systemsWithAnySupport.  There
was no need to use systemsWithAnySupport for supportDarwin, because it
was always equivalent to supportedSystems for that purpose, and the
only other way it was used was for determining which platforms to
build the bootstrap tools for, so we might as well use a more explicit
parameter for that, and then we can change how it works without
affecting the rest of the Hydra jobs.

Not affecting the rest of the Hydra jobs is important, because if we
changed all jobs to use config triples, we'd end up renaming every
Hydra job.  That might still be worth thinking about at some point,
but it's unnecessary at this point (and would be a lot of work).

I've checked by running

    nix-eval-jobs --force-recurse pkgs/top-level/release.nix

that the actual bootstrap tools derivations are unaffected by this
change, and that the only other jobs that change are ones that depend
on the hash of all of Nixpkgs.  Of the other jobset entrypoints that
end up importing pkgs/top-level/release.nix, none used the
limitedSupportedSystems parameter, so they should all be unaffected as
well.
2023-10-07 19:38:01 +00:00
Artturin
93fb394638 freshBootstrapTools.bootstrapTools: fix eval on darwin
```
error: undefined variable 'system'
       at /home/artturin/nixgits/my-nixpkgs/.worktree/1/pkgs/stdenv/darwin/make-bootstrap-tools.nix:213:32:

          212|
          213|   bootstrapTools = derivation {
             |                                ^
          214|     inherit system;

```

fixes eval of `freshBootstrapTools.bootstrapTools` in darwin repl
2023-09-07 04:24:05 +03:00
Simon Žlender
8e912feb29 codesign_allocate: reference cctools 2023-08-01 21:27:09 +02:00
Randy Eckenrode
2e45100c5c
darwin-stdenv: revert NIX_CC_NO_RESPONSE_FILE logic
To work around intermitent build failures with clang 16, the stdenv
attempted to pass arguments on the command-line on newer versions of
macOS. Unfortunately, the larger `ARG_MAX` is still not large enough to
build qtwebengine. This commit reverts the `NIX_CC_NO_RESPONSE_FILE`
logic in the stdenv. The changes to cc-wrapper in #245282 are needed for
clang 16 to prevent the above-mentioned build failures.
2023-07-25 21:25:46 -04:00
Weijia Wang
2b692b1d9d freshBootstrapTools: fix build on darwin 2023-07-15 19:54:50 +02:00
Theodore Ni
bb219736e4
stdenvBootstrapTools: in darwin, only run install_name_tool on Mach-O files 2023-07-14 10:22:55 -07:00
Randy Eckenrode
856ebe6fec
darwin.stdenv: allow patchShebangs during the bootstrap
This fixes pyicu (and any other package that uses `icu-config` instead
of the CMake or some other module to get the build flags).

What happened here is the bootstrap disables `patchShebangs` to avoid
propagating the bootstrap tools to the final stdenv (due to `sh` and
`bash` being on the `PATH` from the bootstrap tools). Because of that,
the `#!/bin/sh` line in `icu-config` was not updated, causing it to
invoke the system bash on Darwin. While that is undesirable in its own
right, when the system bash is invoked as `sh`, `echo -n` will print
`-n`, resulting in the breakage see in https://github.com/NixOS/nixpkgs/pull/241951#issuecomment-1627604354.

The fix is to build bash earlier in the bootstrap while making sure it
is picked up over the one in the bootstrap tools. That allows
`patchShebangs` to be enabled during the bootstrap. Any package with
scripts that is included in the final stdenv should now have its
scripts’ shebang lines properly patched.
2023-07-09 16:53:55 -06:00
github-actions[bot]
c60d48234a
Merge master into staging-next 2023-07-09 12:01:18 +00:00
Randy Eckenrode
6f6687ee24
darwin.stdenv: only run install_name_tool on files 2023-07-08 20:49:50 -06:00
Randy Eckenrode
feb89eed00
darwin.stdenv: fix portable libsystem hook with sandboxing
When sandboxing is enabled, the hook tries to run `install_name_tool`
and fails because the system one is inaccessible. Having it use
`targetPrefix` allows it to find and use the cross-install_name_tool.
2023-07-08 20:49:49 -06:00
Randy Eckenrode
ebc1bcf409
swift-corelibs: don’t link against libcurl
swift-corelibs uses libcurl to implement `NSURLSession` in Foundation
via the symbols exported by CF. Foundation is not build on Darwin, and
these symbols are not exported by the system CoreFoundation.

By not linking against libcurl, this breaks a cycle between CF and
libcurl. That should allow libcurl to drop the patch disabling
linking against the SystemConfiguration and restore NAT64 support.

Unfortunately, the Darwin stdenv bootstrap still needs to build
dependencies that use `fetchFromGitHub`. While it can drop curl from the
final stdenv, it still needs to use it during the stdenv bootstrap.
2023-07-02 17:56:25 -04:00
Randy Eckenrode
a845397040
darwin.stdenv: refactor stdenv definition
In preparation for bumping the LLVM used by Darwin, this change
refactors and reworks the stdenv build process. When it made sense,
existing behaviors were kept to avoid causing any unwanted breakage.
However, there are some differences. The reasoning and differences are
discussed below.

- Improved cycle times - Working on the Darwin stdenv was a tedious
  process because `allowedRequisites` determined what was allowed
  between stages. If you made a mistake, you might have to wait a
  considerable amount of time for the build to fail. Using assertions
  makes many errors fail at evaluation time and makes moving things
  around safer and easier to do.
- Decoupling from bootstrap tools - The stdenv build process builds as
  much as it can in the early stages to remove the requirement that the
  bootstrap tools need bumped in order to bump the stdenv itself. This
  should lower the barrier to updates and make it easier to bump in the
  future. It also allows changes to be made without requiring additional
  tools be added to the bootstrap tools.
- Patterned after the Linux stdenv - I tried to follow the patterns
  established in the Linux stdenv with adaptations made to Darwin’s
  needs. My hope is this makes the Darwin stdenv more approable for
  non-Darwin developers who made need to interact with it. It also
  allowed some of the hacks to be removed.
- Documentation - Comments were added explaining what was happening and
  why things were being done. This is particular important for some
  stages that might not be obvious (such as the sysctl stage).
- Cleanup - Converting the intermediate `allowedRequisites` to
  assertions revealed that many packages were being referenced that no
  longer exist or have been renamed. Removing them reduces clutter and
  should help make the stdenv bootstrap process be more understandable.
2023-07-02 17:56:24 -04:00
toonn
b7d4899ad5
Merge pull request #234861 from reckenrode/corefoundation-rpath
swift-corelibs: set NIX_COREFOUNDATION_RPATH in a hook
2023-06-11 17:13:54 +02:00