@vcunat and others rightly point out that it's easier to quote always,
than learn Bash's idiosyncrasies enough to know when it doesn't make a
difference.
This reverts commit 2743078f66, which
removes quotes that don't do anything, and then goes further adding
even more quotes.
I took some liberties with the flags-echoing code to make it more
concise and correct. Also, a few warnings in findInputs and friends I
skipped because I am going to rewrite those anyways.
Thanks @grahamc for telling me about this great linter!
This makes those files a bit easier to read. Also, for what it's worth,
it brings us one baby step closer to handling spaces in store paths.
Also, I optimized handling of many transitive deps with read. Probably,
not very beneficial, but nice to enforce the pkg-per-line structure.
Doing so let me find much dubious code and fix it.
Two misc notes:
- `propagated-user-env-packages` also needed to be adjusted as
sometimes it is copied to/from the propagated input files.
- `local fd` should ensure that file descriptors aren't clobbered
during recursion.
When not cross compiling, nativeBuildInputs and buildInputs have
identical behaviour. Currently that is implemented by having
mkDerivation do a concatenation of those variables in Nix code and pass
that to the builder via the nativeBuildInputs attribute.
However, that has some annoying side effects, like `foo.buildInputs`
evaluating to `[ ]` even if buildInputs were specified in the nix
expression for foo.
Instead, pass buildInputs and nativeBuildInputs in separate variables as
usual, and move the logic of cross compilation vs. native compilation to
the stdenv builder script. This is probably a tiny bit uglier but
fixes the previous problem.
Issue #4855.
`stripHash` documentation states that it prints out the stripped name to
the stdout, but the function stored the value in `strippedName`
instead.
Basically all usages did something like
`$(stripHash $foo | echo $strippedName)` which is just braindamaged.
Fixed the implementation and all invocations.
Close#15803. This avoids the error:
while setting up the build environment: executing
‘/nix/store/7sb42axk5lrxqz45nldrb2pchlys14s1-bash-4.3-p42/bin/bash’:
Argument list too long
Note: I wanted to make it optional based on buildCommand length,
but that seems pointless as I'm sure it's less performant.
Amended by vcunat:
https://github.com/NixOS/nixpkgs/pull/15803#issuecomment-224841225
On Linux, paxctl's setup hook should overwrite the paxmark stub, but the
stub is defined after the setup hooks are sourced, so the stub ends up
overwriting the real function. The result is that paxmark fails to do
anything. The fix is to define the stub before any setup hooks are
sourced. Thanks to @vcunat for figuring this out.
Closes#15492
I'm giving this up. Feel free to find some reasonable variant that works
at least on Linux and Darwin. Problems encountered:
- During bootstrap of Darwin stdenv `env -0` and some bash features
don't work.
- Without `env -0` the contents of some multi-line phases is taken as
variable declarations, which wouldn't typically matter, but the PR
wanted to refuse bash-invalid names which would be occasionally
triggered. This commit dowgrades that to a warning with explanation.
It turned out that process substitution fed into a while-cycle
isn't recognized during darwin bootstrap:
http://hydra.nixos.org/build/35382446/nixlog/1/raw
Also fix broken NIX_DEBUG output, noticed by abbradar.
The set/env fix in #14907 wasn't very good, so let's use a null-delimited
approach. Suggested by Aszlig.
In particular, this should fix a mass-breakage on Darwin, though I was
unable to test that.
bash variable names may only contain alphanumeric ASCII-symbols and _,
and must not start with a number. Nix expression attribute names however
might contain nearly every character (in particular spaces and dashes).
Previously, a substitution that was not a valid bash name would be
expanded to an empty string. This commit introduce a check that throws
a (hopefully) helpful error when a wrong name is used in a substitution.
Close#14335.
Since 89036ef76a, when a package doesn't include a configure script,
the build complains with:
grep: : No such file or directory
grep: : No such file or directory
This prevents that.
Otherwise, when building glibc and other packages, the "strip" from
bootstrapTools is used, which doesn't recognise some tags produced by
the newer "ld" from binutils.
Fixes#12632.
I think it's better to quote this variable in general, because it is
common and even documented to pass space-separated commands in there.
The greps should just fail in that case and `if` won't proceed
which seems fine for such cases, and it's certainly better than
passing additional unintended parameters to grep
(which was happening all the time before).
Doing it in an openssl setup hook only works if packages have openssl
as a build input - it doesn't work if they're using a program linked
against openssl.
Commit 6d928ab684 changed this to not
preserve timestamps. However, that results in non-determinism; in
particular, it gives us a broken $SOURCE_DATE_EPOCH (especially for
everything using fetchFromGitHub). Builds affected by timestamps <
1980 should be fixed in some other way (e.g. changing the timestamp to
some fixed date > 1980).
This is used by some build tools to provide reproducible builds. See
https://reproducible-builds.org/specs/source-date-epoch/
for more info.
Later, we'll want to set this to a more intelligent value (such as the
most recent mtime of any source file).
So far if no configure script is found or no makefile,
the rest of the phase is skipped, *including* post-hooks.
I find that behavior unexpected/unintuitive.
Earlier version of this patch had problems due to me assuming
that $configureScript is always a simple path, but that turned out
to be false in many cases, e.g. perl.
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
Now development stuff is propagated from the first output,
and userEnvPkgs from the one with binaries.
Also don't move *.la files (yet). It causes problems, and they're small.
- there were many easy merge conflicts
- cc-wrapper needed nontrivial changes
Many other problems might've been created by interaction of the branches,
but stdenv and a few other packages build fine now.
- IMO using a temporary is not needed here (anymore),
- temporary at that location can cause a problem (in a specific case):
for example, when using the substituteAll function from nixpkgs
on a single file directly under /nix/store/ (or ./foo-file),
the stdenv's substitute tries to create a temporary directly under
/nix/store, which causes problems on chrooted darwin
(according to @copumpkin earlier today on IRC)
Getting the names of all environment variables is tricky. The previous
implementation easily got confused by multi-line variables. The new
one is more reliable but not still not perfect.
This works around a segfault in Bash 4.3, where the expression
"${!var}" (where var="-9") crashes under certain conditions.
http://hydra.nixos.org/build/16693445
Otherwise, stdenv won't have a reference to e.g. patchelf on Linux
(because it was passed in by mkDerivation). This causes the installer
tests to fail, because having "stdenv" in the installation CD closure
is not enough to pull in all stdenv packages.
http://hydra.nixos.org/build/16546643
Now it should contain *all* information from stdenv/setup.sh of
the original mutiple-output branch.
However, the configurability of the output paths is much greater.
This is needed for multiple-output derivations,
where it is desirable to propagate deps and setup-hooks into $dev instead of $out.
Also drop an unused simple function which will not even make sense.
Now gcc is just another build input, making it possible in the future
to have a stdenv that doesn't depend on a C compiler. This is very
useful on NixOS, since it would allow trivial builders like
writeTextFile to work without pulling in the C compiler.
If $src refers to a directory, then always copy it. Previously, we
checked the extension first, so if the directory had an extension like
.tar, unpackPhase would fail.
If a build input is a regular file, use it as a setup hook. This makes
setup hooks more efficient to create: you don't need a derivation that
copies them to $out/nix-support/setup-hook, instead you can use the
file as is.
You can now register multiple values per named hook, e.g.
addHook preConfigure "echo foo"
addHook preConfigure "echo bar"
will cause ‘runHook preConfigure’ to run both ‘echo foo’ and ‘echo
bar’ (in that order). It will also call the shell function
preConfigure() or eval the shell variable $preConfigure, if
defined. Thus, if you don't call addHook, it works like the old hook
mechanism.
Allowing multiple hooks makes stdenv more modular and extensible. For
instance, multiple setup hooks can define a preFixup hook, and all of
these will be executed.
setup.sh uses the anti-pattern `for f in $(find ...); do` in several
places. `find` returns one path per line, but `for` splits its arguments
by words, so paths which contain spaces are incorrectly split! The
correct way is `find ... | while read f; do`
Treating fixupPhase specially is really ugly. Also, it collides with
the work in the multiple-outputs branch (which already has support for
fixing up all outputs).
Partial revert of 0a44a09121.
Some programs, e.g. guile-config, has a shebang that ends in '\':
#!/usr/bin/guile-1.8 \
-e main -s
!#
;;;; guile-config --- utility for linking programs with Guile
;;;; Jim Blandy <jim@red-bean.com> --- September 1997
This currently breaks patchShebangs:
$ read oldPath arg0 args <<< 'shebang \'; echo $?
1
$ echo $oldPath
shebang
$ echo $arg0
$ echo $args
(And setup.sh/patchShebangs is run with 'set -e' so any command that
return non-zero aborts the build.)
Fix by telling 'read' to not interpret backslashes (with the -r flag):
$ read -r oldPath arg0 args <<< 'shebang \'; echo $?
0
$ echo $oldPath
shebang
$ echo $arg0
\
$ echo $args
Also needed: escape the escape characters so that sed doesn't interpret
them.
patchShebangs has a bug that shows itself on files that have the
executable bit set but have no shebang (i.e. a blank/empty first line).
The shell would then evaluate this:
if [ != '#!' ]; then
# not evaluated
fi
With proper quoting we get the correct behaviour:
if [ "" != '#!' ]; then
# this will be evaluated
fi
When building e.g. perl for the first time there is no perl in PATH yet,
so command -v perl will fail.
This brings back the previous behaviour of silently not patching
shebangs for which there is no available command in PATH.
Currently "/usr/bin/env python" is rewritten to "/nix/store/.../env
python". That doesn't really improve anything because the interpreter
still have to be located in $PATH at runtime. The result is that many
nix package expressions do .../bin/env fixup themselves.
Instead of everyone having to do this patching locally, add the
functionality to the standard environment patchShebangs function so that
everyone can benefit.
set CMAKE_LIBRARY_PATH, CMAKE_INCLUDE_PATH based on NIX_CFLAGS_COMPILE and
NIX_LDFLAGS so that cmake's find_library like functions find all the libraries
gcc knows about thanks to the gcc wrapper
This is particular useful with myEnvFun which then also sets those CMAKE_* env
variables.`
Because setup.sh has to change this causes many rebuilds - thus it should be
included in a stdenv-update like branch
Also cmake builds in parallel perfectly fine
update cmake to latest minor number, I didn't change the patches,
just reapplied them manually recordin a new patch
I'm not sure whether this was by intention, but so far postPatch hooks were
silently skipped whenever the patches list was empty. This change could possibly
change the build results of the following packages:
* gcc
* cmake (264)
* systemtap
* quemu-kvm
These packages all have in common that they have a postPatch hook and the
patches list can be empty when certain conditions are met.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This is for consistency with terminology in stdenv (and the terms
"hostDrv" and "buildDrv" are not very intuitive, even if they're
consistent with GNU terminology).
This allows various applications. It allows users to set global
optimisation flags, e.g.
stdenv.userHook = ''NIX_CFLAGS_COMPILE+=" -funroll-loops"'';
But the impetus is as an alternative to issue #229, allowing impure
stdenv setup for people who want to use distcc:
stdenv.userHook = "source /my/impure/setup-script.sh";
This is probably a bad idea, but at least now it's a bad idea in
people's configuration and not in Nixpkgs. :-)
bash's pattern replacement feature. "replace-literal" is an
uncommon command so it was a headache during the bootstrap.
svn path=/nixpkgs/branches/stdenv-updates/; revision=31681
These markers follow the format of those of `nix-store --print-build-trace',
which allows extraction of, say, phase durations in a similar way.
svn path=/nixpkgs/branches/stdenv-updates/; revision=26115
If a build expressions has set "enableParallelBuilding = true", then the
generic builder may utilize more than one CPU core to build that particular
expression. This feature works out of the box for GNU Make. Expressions that
use other build drivers like Boost.Jam or SCons have to specify appropriate
flags such as "-j${NIX_BUILD_CORES}" themselves.
svn path=/nixpkgs/branches/stdenv-updates/; revision=22399
inputs did not get in)
Updating the xorg builder script to support cross building (in fact, support
for propagating the required build inputs).
svn path=/nixpkgs/branches/stdenv-updates/; revision=18569
- Before this changes, cflags and ldflags for the native and the cross compiler
got mixed. Not all the gcc-wrapper/gcc-cross-wrapper variables are
independant now, but enough, I think.
- Fixed the generic stdenv expression, which did a big mess on buildInputs and
buildNativeInputs. Now it distinguishes when there is a stdenvCross or not.
Maybe we should have a single stdenv and forget about the stdenvCross
adapter - this could end in a stdenv a bit complex, but simpler than the
generic stdenv + adapter.
- Added basic support in pkgconfig for cross-builds: a single PKG_CONFIG_PATH
now works for both the cross and the native compilers, but I think this
should work well for most cases I can think of.
- I tried to fix the guile expression to cross-biuld; guile is built, but not
its manual, so the derivation still fails. Guile requires patching to
cross-build, as far as I understnad.
- Made the glibcCross build to be done through the usage of a
gcc-cross-wrapper over the gcc-cross-stage-static, instead of using it
directly.
- Trying to make physfs (a neverball dependency) cross build.
- Updated the gcc expression to support building a cross compiler without getting
derivation variables mixed with those of the stdenvCross.
svn path=/nixpkgs/branches/stdenv-updates/; revision=18534
derivation, the "buildInputs" in every stdenv mkDerivation don't map now
directly to the environment
variable "buildInputs" in the builder, but "buildNativeInputs". So, the inputs
build by the native compiler.
When cross compiling, they will map to the environment variable "buildInputs"
(yes, now the same name), which means does to be built with the cross compiler.
I think I improved the naming of variables a bit. There was a big mess,
specially in the stdenv adapter for cross building, and also in the default
builder script.
I also tried to add proper manager of propagatedInputBuilds, these being
propagated considering the host or build origin of that input build (so, at the
end, being those propagatedInputBuilds being propagated properly to the native
or the cross compiler.
svn path=/nixpkgs/branches/stdenv-updates/; revision=18477
I made the stdenvCross adapter simpler, according to Nicolas Pierron comments,
and I commented it a bit.
There are still jobs to do. At least:
- Plan for the general renaming from buildInputs to hostInputs
- We should not break merges from trunk.
- Make the generic stdenv understand about host/buildInputs, at least for
native builds, because it is used in the always-native building of
stdenvLinux. This should allow us to remove all duplications of "*NoCross" in
nixpkgs.
svn path=/nixpkgs/branches/stdenv-updates/; revision=18449
* setup.sh: removed some obsolete features, specifically some that
were only used by the old build farm.
* addToSearchPath: removed some parameters that weren't used
anywhere.
svn path=/nixpkgs/branches/stdenv-updates/; revision=15136
instead of "gcc-4.3.3". This fixed the long-standing annoyance that
you can't distinguish the two in (say) nix-store -qR.
* On x86_64-linux, put $out/lib64 in the RPATH in addition to
$out/lib, because some packages (in particular GCC) put libraries in
$out/lib64 and ended up linking against the wrong library.
* Strip $out/lib64.
* Removed g77_42 because it's exactly the same as gfortran.
svn path=/nixpkgs/branches/stdenv-updates/; revision=14708
variables. This is so you don't have to write things like
preBuild=preBuild
preBuild() {
bla
}
svn path=/nixpkgs/branches/stdenv-updates/; revision=13941
paths. E.g. /usr/bin/perl is rewritten to /nix/store/<whatever Perl
is in $PATH>. Paths in the Nix store are left unchanged.
Contributed by Nicolas Pierron.
svn path=/nixpkgs/branches/stdenv-updates/; revision=12036
the fail() function unnecessary.
* Flag showBuildStats to print execution times on builder exit.
svn path=/nixpkgs/branches/stdenv-updates/; revision=11688
* Updated the kernel headers.
* ghc: setup hooks are now generated using substituteAll.
svn path=/nixpkgs/branches/stdenv-updates-merge/; revision=10838
stack traces impossible.
* When stripping all symbols on Darwin, don't use the "-s" flag
since it has a completely different meaning (it takes an argument
specifying a file containing a list of symbols).
svn path=/nixpkgs/branches/stdenv-updates-merge/; revision=10809
merge cleanly right away (kde-4, kernel stuff) so it should be
merged later. But the stdenv stuff is all there.
svn path=/nixpkgs/branches/stdenv-updates-merge/; revision=10793
in genericBuild. This also means that you can override a phase
function by redefining it, like
buildPhase() {
... do something ...
}
and you still get the log nesting stuff.
* GNUmakefile is also a valid makefile name.
* Evaluate the variables $preHook and $postHook to allow more stdenv
overrideability.
svn path=/nixpkgs/branches/stdenv-updates/; revision=10786
is skipped when there is no configure script). So you don't need to
set
buildPhase = "true";
for derivations that don't have any build action.
* Variable $makefile to allow the makefile to be specified.
svn path=/nixpkgs/branches/stdenv-updates/; revision=10784
* Removed substitute, it's part of the generic builder now.
* stdenv-initial (Linux): use the real generic builder script. This
does require that sed is in the path of the builder of the initial
stdenv.
svn path=/nixpkgs/trunk/; revision=7498
This has a major advantage: you can write hooks directly in Nix
expressions. For instance, rather than write a builder like this:
source $stdenv/setup
postInstall=postInstall
postInstall() {
ln -sf gzip $out/bin/gunzip
ln -sf gzip $out/bin/zcat
}
genericBuild
(the gzip builder), you can just add this attribute to the
derivation:
postInstall = "ln -sf gzip $out/bin/gunzip; ln -sf gzip $out/bin/zcat";
and so a separate build script becomes unnecessary. This should
allow us to get rid of most builders in Nixpkgs.
* Allow configure and make arguments to contain whitespace.
Previously, you could say, for instance
configureFlags="CFLAGS=-O0"
but not
configureFlags="CFLAGS=-O0 -g"
since the `-g' would be interpreted as a separate argument to
configure. Now you can say
configureFlagsArray=("CFLAGS=-O0 -g")
or similarly
configureFlagsArray=("CFLAGS=-O0 -g" "LDFLAGS=-L/foo -L/bar")
which does the right thing. Idem for makeFlags, installFlags,
checkFlags and distFlags.
Unfortunately you can't pass arrays to Bash through the environment,
so you can't put the array above in a Nix expression, e.g.,
configureFlagsArray = ["CFLAGS=-O0 -g"];
since it would just be flattened to a since string. However, you
can use the inline hooks described above:
preConfigure = "configureFlagsArray=(\"CFLAGS=-O0 -g\")";
svn path=/nixpkgs/trunk/; revision=6863
