defaultHardeningFlags is set to enable pie for Musl, but is not
actually used because the default is never put into
NIX_HARDENING_ENABLE. That still works for cases other than Musl
only because NIX_HARDENING_ENABLE is defaulted in the binutils and
cc-wrapper setup-hook.sh scripts.
This introduces the .inputDerivation attribute on all derivations
created with mkDerivation. This is another derivation that can always
build successfully and whose runtime dependencies are the build time
dependencies of the original derivation.
This allows easy building and distributing of all derivations needed to
enter a nix-shell with
nix-build shell.nix -A inputDerivation
The cross file is added in the `mkDerivation`. It isn't nice putting
build tool-specific stuff here, but our current architecture gives us
little alternative.
Currently it's not possible to determine the reason why a package is
unavailable without evaluating nixpkgs multiple times with different
settings. eg.
nix-repl> :p android-studio.meta
{ available = false; broken = false; unfree = true; unsupported = true; ... }
The following snippet is an example that uses this information to query
the availability information of all packages in nixpkgs, giving an
overview of all the packages currently marked as broken, etc.
{ pkgs }:
with import <nixpkgs/lib>;
let
mapPkgs =
let mapPkgs' = path: f: mapAttrs (n: v:
let result = builtins.tryEval (v ? meta); in
if !result.success then {} else
if isDerivation v then f (path ++ [n]) v else
if isAttrs v && v.recurseForDerivations or false then mapPkgs' (path ++ [n]) f v else
{}
);
in mapPkgs' [];
getMeta = path: drv:
if drv.meta ? available then
let meta = {
pkg = concatStringsSep "." path;
inherit (drv.meta) broken unfree unsupported insecure;
};
in builtins.trace meta.pkg meta
else {};
metaToList = attrs: flatten (map (v: if v ? pkg then v else metaToList v) (attrValues attrs));
in metaToList (mapPkgs getMeta pkgs)
This puts patches in all derivations even if it unspecified by the
derivation. By default it will be an empty list. This simplifies
overrides, as we can now assume that patches is a valid name so that
this works:
self: super: {
mypkg = super.pkg.overrideAttrs (o: {
patches = o.patches ++ [ ./my-very-own.patch ];
});
}
That is, you don’t need to provide a default "or []", make-derivation
provides one for you.
Unfortunately, this is a mass rebuild.
You can build (partially) with LLVM toolchain using the useLLVM flag.
This works like so:
nix-build -A hello --arg crossSystem '{ system =
"aarch64-unknown-linux-musl"; useLLVM = true }'
also don’t separate debug info in lldClang
It doesn’t work currently with that setup hook. Missing build-id?
Comments on conflicts:
- llvm: d6f401e1 vs. 469ecc70 - docs for 6 and 7 say the default is
to build all targets, so we should be fine
- some pypi hashes: they were equivalent, just base16 vs. base32
For a long time now, tracing has been broken in Nixpkgs. So when you
have an eval error you would get something like this:
error: while evaluating the attribute 'buildInputs' of the derivation 'hello-2.10' at /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:11:
while evaluating 'chooseDevOutputs' at /home/mbauer/nixpkgs/lib/attrsets.nix:474:22, called from undefined position:
while evaluating 'optionals' at /home/mbauer/nixpkgs/lib/lists.nix:257:5, called from /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:132:17:
This is coming from how Nix handles string context and how
make-derivation messes with the "name" attribute. This commit should
restore the old behavior so you get a nice line number like:
error: while evaluating the attribute 'buildInputs' of the derivation 'hello-2.10' at /home/mbauer/nixpkgs/pkgs/applications/misc/hello/default.nix:4:3:
while evaluating 'chooseDevOutputs' at /home/mbauer/nixpkgs/lib/attrsets.nix:474:22, called from undefined position:
while evaluating 'optionals' at /home/mbauer/nixpkgs/lib/lists.nix:257:5, called from /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:132:17:
NOTE: This will still be broken for cross compilation due to the
prefixes we are adding to name.
This behavior ended up breaking the handleEvalIssue functionality by hiding those packages. So something like this:
$ nix-env -iA nixpkgs.zoom-us
would silently fail, without telling the user how to fix it! Regardless, this "bug" should be handled in Nix - not Nixpkgs.
Fixes#38952.
We can't run the checkPhase when build != host, so we may as well make
the checkInputs native.
This signicantly improves the situation of Python packages when enabling
strictDeps.
Some trivial builders use the name attr to choose the exec name
produced. For example nixos-install,
nixos-install = makeProg {
name = "nixos-install";
src = ./nixos-install.sh;
nix = config.nix.package.out;
path = makeBinPath [ nixos-enter ];
};
When cross compiling, this puts the prog in,
/bin/nixos-install-powerpc64le-unknown-linux-gnu
When separateDebugInfo = true & !stdenv.hostPlatform.isLinux, we
always gave an error. There is no reason to do this. Instead, just
don’t add separate debug info when we aren’t on Linux.
propagateNativeBuildInputs will end up going in the output derivation.
This case is allowed to end up in references because of that. Sorry
for the disruption!
Fixes#50865
Fixes#49071
On ld.gold, we produce broken executables when linking with the Musl
libc. This appears to be a known bug when using ld.gold and Musl. This
thread describes the workaround as enabling PIE when using ld.gold and
Musl:
https://www.openwall.com/lists/musl/2015/05/01/5
By default we don’t enable PIE to avoid breaking things. But in the
Musl case we are breaking things by not enabling PIE. So this adds a
special case for defaultHardeningFlags which keeps the pie hardening
for everything. Any packages that break with PIE can add the pie flag
to disableHardeningFlags array (a no-op for now on anything but Musl).
When strictDeps = true, we don’t want native build inputs to end up in
the output. For instance gcc is a builtin native build input and
should only show up in an output if it is also listed in buildInputs.
/cc @ericson2314
Uses uname data to find what to set these variables:
- CMAKE_SYSTEM_NAME
- CMAKE_SYSTEM_PROCESSOR
- CMAKE_SYSTEM_VERSION
- CMAKE_HOST_SYSTEM_NAME
- CMAKE_HOST_SYSTEM_PROCESSOR
- CMAKE_HOST_SYSTEM_VERSION
Works similarly to `enableParallelBuilding`, but is set by default when
`enableParallelBuilding` is set. In my experience most packages that build
fine in parallel also check fine in parallel.
Derivations where drawing their `system` attribute from `hostPlatform`
instead of `buildPlatform`. Fix that, and add an explanatory commment.
Fixes#45993
`depsHostBuild` is not a thing, would never be a thing per the rules,
and isn't used anywhere. This is just my typo, hitherto unnoticed
because "host -> host" dependencies are by far the most obscure form.
Not only does the suffix unnecessarily reduce sharing, but it also breaks
unpacker setup hooks (e.g. that of `unzip`) which identify interesting tarballs
using the file extension.
This also means we can get rid of the splicing hacks for fetchers.
The hack of using `crossConfig` to enforce stricter handling of
dependencies is replaced with a dedicated `strictDeps` for that purpose.
(Experience has shown that my punning was a terrible idea that made more
difficult and embarrising to teach teach.)
Now that is is clear, a few packages now use `strictDeps`, to fix
various bugs:
- bintools-wrapper and cc-wrapper
Note that a bunch of non-python packages use this attribute already.
Some of those are clearly unaware of the fact that this attribute does
not exists in stdenv because they define it but don't to add it to
their `bulidInputs` :)
Also note that I use `buildInputs` here and only handle regular
builds because python and haskell builders do it this way and I'm not
sure how to properly handle the cross-compilation case.
First, we need check against the host platform, not the build platform.
That's simple enough.
Second, we move away from exahustive finite case analysis (i.e.
exhaustively listing all platforms the package builds on). That only
work in a closed-world setting, where we know all platforms we might
build one. But with cross compilation, we may be building for arbitrary
platforms, So we need fancier filters. This is the closed world to open
world change.
The solution is instead of having a list of systems (strings in the form
"foo-bar"), we have a list of of systems or "patterns", i.e. attributes
that partially match the output of the parsers in `lib.systems.parse`.
The "check meta" logic treats the systems strings as an exact whitelist
just as before, but treats the patterns as a fuzzy whitelist,
intersecting the actual `hostPlatform` with the pattern and then
checking for equality. (This is done using `matchAttrs`).
The default convenience lists for `meta.platforms` are now changed to be
lists of patterns (usually a single pattern) in
`lib/systems/for-meta.nix` for maximum flexibility under this new
system.
Fixes#30902
This is a temporary workaround to make `nix-env -qa` and `nix search` ignore
broken packages as they they did before this patchset.
This patch should be reverted after `nix` gets a proper fix for this.
See NixOS/nix#1771.
This option makes `meta.evaluate` into a close approximation of the result of
evaluating `.outPath` by checking all the dependencies recursively at a cost of
2x slowdown. Note that actually evaluating `.outPath` costs some
5x-7x more because `.outPath` also computes all the hashes.
I hope this will be a temporary measure. If there is consensus around
issue #33599, then we can follow an explicit `dontCheck`, but default to
not checking during cross builds when none is given.
This pushes check-meta evaluation to derivation evaluation step, leaving all other
attributes accessible.
Before this commit:
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.
as expected
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.name --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.meta.description --argstr system aarch64-linux
> Package ‘xen-4.5.5’ in pkgs/applications/virtualization/xen/generic.nix:226 is not supported on ‘aarch64-linux’, refusing to evaluate.
which is unfortunate since its impossible to use packages in autogenerated
documentation on all platforms.
After this commit:
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen --argstr system aarch64-linux
still fails
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.name --argstr system aarch64-linux
> "xen-4.5.5"
> $ HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate --eval --strict ./default.nix -A xen.meta.description --argstr system aarch64-linux
> "Xen hypervisor and related components (vanilla)"
4 far-reaching changes: Smaller PATH, New vars, different propagation
logic, and different hook logic
Smaller PATH
------------
`buildInputs` no longer go on the PATH at build time, as they cannot be
run when cross compiling and we don't want to special case. Simply make
a `nativeBuildInput` too if one needs them on the PATH. Fixes#21191.
Many new depedendency variables
-------------------------------
See the stdenv chapter of the nixpkgs manual. I pulled out the existing
documentation of dependency specification into a new section, and added
language for these two (and their propagated equivalents) along side
the others'.
More complex propagation logic
------------------------------
Before a propagated*XXX*Input always acted as if it was specified
directly as a *XXX*Input downstream. That's simple enough, but violates
the intended roles of each sort of dep, which has functional and not
just stylistic consequences.
The new algorithm is detailed in the manual, and ensures everything
ends up in the right place. I tried to give both an informal and formal
description, but I suspect in practice it will not make much sense
until one tries cross compiling, after which it will immediately make
sense as the only sane option.
Simplified hook logic
---------------------
Rather than `envHook` and `crossEnvHook`, whose behavior differs
depending on whether we are cross compiling or not, there is now one
hook per sort (or rather non-propagated and propagated pair of sorts)
of dependency. These new hooks have the same meaning regardless of
cross compilation. See the setup hook section of stdenv chapter of the
Nixpkgs manual for more details.
This becomes necessary if more wrappers besides cc-wrapper start
supporting hardening flags. Also good to make the warning into an
error.
Also ensure interface is being used right: Not as a string, not just in
bash.
This reverts commit eeabf85780.
This change suddenly makes tons of stdenv internals visible in
nativeBuildInputs of every derivation, which doesn't seem desirable.
E.g:
````
nix-repl> hello.nativeBuildInputs
[ «derivation /nix/store/bcfkyf6bhssxd2vzwgzmsbn7b5b9rpxc-patchelf-0.9.drv»
«derivation /nix/store/4wnshnz9wwanpfzcrdd76rri7pyqn9sk-paxctl-0.9.drv»
<< snip 10+ lines >>
«derivation /nix/store/d35pgh1lcg5nm0x28d899pxj30b8c9b2-gcc-wrapper-6.4.0.drv»
]
````
Additionally, instead of pulling them from `setup.sh`, route them via
Nix. This gets us one step closer to making stdenv be a plain attribute
set instead of a derivation.
`nix-build pkgs/top-level/release.nix -A tarball` now succeeds.
`configureFlags = null` lead to a type error, and one overrideDrv
needed to be converted to to append a configureFlags list instead of
string due to the normalization.
Thanks @vcunat for alerting me to the issues---sorry I did not catch
them before merging my own PR.
Eventually the adapter will be removed. Moved is
- Name suffix from hostPlatform
- configurePlatforms
To not cause more breakage, the default is currently [], but
eventually it will be [ "build" "host" ], as the cross adapter makes
it today.
