Deeply-curried functions are pretty error-prone in untyped languages
like Nix. This is a particularly bad case because
`top-level/splice.nix` *also* declares a makeScopeWithSplicing, but
it takes *two fewer arguments*.
Let's switch to attrset-passing form, to provide some minimal level
of sanity-checking.
`lib.extends hooks pythonPackagesFun` includes python which we want to
splice
what this fixes:
```
nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0
«derivation /nix/store/39dkb51rciw6zwg0c2c44gpmpjapddxc-python3-aarch64-unknown-linux-gnu-3.10.12.drv»
```
to
```
nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0
«derivation /nix/store/21ldw2dp26xvv9iyxn9x77a8yh4waqz5-python3-3.10.12.drv»
```
Before https://github.com/NixOS/nixpkgs/pull/194205 The keep python used to work by accident because self was passed from
__splicedPackages https://github.com/NixOS/nixpkgs/pull/196052/files#diff-44ce3495c4f983ce64dd47c86a9d3e77bad210b2709c098a3806998dcd9b000bR213
But now it does not work because overrideAttrs is used on python in
python-packages-base.nix
Preferably we would [ splice the hooks too ](https://github.com/NixOS/nixpkgs/pull/228139) but we cannot do that until [Package sets within derivations (i.e. python3.pkgs) are not spliced](https://github.com/NixOS/nixpkgs/issues/211340) is fixed, because people often use `python3.pkgs.wrapPython` in `nativeBuildInputs` (it's correct but python3.pkgs should be python3Packages to get splicing.
The setuptools-rust requires some environment variables to really
perform cross build, otherwise it just builds for build platform.
This adds setup hook that introduces these environment variables.
There are three variables.
The PYO3_CROSS_LIB_DIR has to point to the target's Python library
directory. This has to be directory for the target not for the build or
host. We have to choose the correct target Python. I am unsure how to do
that simply in nixpkgs and this this implementations just delays this
and waits for the correct Python when package using this hook is build.
The CARGO_BUILD_TARGET triggers cross compilation in setuptools-rust.
This is simply the Rust target specification.
The CARGO_TARGET_*_LINKER variable should not be essentially required
but setuptools-rust probably mangles the Rust build environment somewhat
and that results to the missing linker. By explicitly specifying it
using the environment variable we force the correct linker.
Since python documentation requires Sphinx to build, documentation is built in
separate derivation to avoid hard recursion/bootstrapping questions and plugged
into main derivation. Care is taken that "python3.doc" derivation feels and
looks as close as possible to as if it were main derivation were multi-output
one.
This change does not cause rebuild of python3 itself.
Add a marker file to the python outputs that tells pip and other tooling
following PEP 668 that they should not be installing things system-wide.
This provides better feedback to the user and also potentially avoids issues in
single-user installs where the /nix/store is owned by the user.
For more details, see <https://peps.python.org/pep-0668/>
Here is how it currently looks like:
$ pip install requests
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
mkPythonDerivation: apply checks to buildInputs as well
fixup: add a suggestion about nativeBuildInputs
Clean up the wording, add reasons for the suggestions
switch to throw
Adjust error to clarify build-time vs. run-time
- Prune unused imports
- Collect imports at the top
- Removed unused exception assignments
- Fill bare except clause with
- Expand overly long check_output lines
Retrieve the current package hash and use it for a more precise match,
which allows us to update a derivation in situation where multiple
hashes are present.
By settings `passthru.skipBulkUpdate = true` in a python package, it
will be excluded from bulk update runs.
This is useful in case of fragile packages, that require special
attention during updates.
Previously when packages that required the git fetcher were updated, we
would wrongly rely on `nix-prefetch-url`, which would reliable break the
hash.
Instead we need to use `nix-prefetch-git` to determine the proper hash,
when the relevant attributes are present.
When PyPy introduced Python 3.8 support with version 7.3.6, they also
migrated to using CPython's directory layout [0]:
> The 3.8 package now uses the same layout as CPython, and many of the
PyPy-specific changes to `sysconfig`, `distutils.sysconfig`, and
`distutils.commands.install.py` have been removed. The stdlib now is
located in `<base>/lib/pypy3.8` on `posix` systems...
When we upgraded past this version and added Python 3.8 support [1], the
`sitePackages` value was never updated, leading `bootstrapped-pip` to fail
to build, because wheel was trying to be located in `$out/site-packages`,
when it was actually installed to `$out/lib/pypy3.8/site-packages`.
[0]: https://www.pypy.org/posts/2021/10/pypy-v736-release.html
[1]: eec28b8cfd
with structuredAttrs lists will be bash arrays which cannot be exported
which will be a issue with some patches and some wrappers like cc-wrapper
this makes it clearer that NIX_CFLAGS_COMPILE must be a string as lists
in env cause a eval failure
The virtualenv mode is trying to be smart and checks whether it can
write to $out, which for some reason fails on darwin.
> virtualenv: error: argument dest: the destination . is not write-able at /nix/store
But what does work is creating it in the pwd and moving it to $out,
because mv just does what it is told.
Allow python3.tests to eval on other platforms than linux, by excluding
conda tests when not on linux, because they include alsa-lib, which is
linux-only.
Previously, unless unsupported platforms were allowed, the following
would fail to evaluate (from an "x86_64-linux" system):
pkgsCross.x86_64-freebsd.__splicedPackages.docutils.__spliced.buildHost
It shouldn't have, because the buildHost package ends up being for
Linux. This broke evaluation of e.g. pkgsCross.x86_64-freebsd.libdrm,
because it has docutils in nativeBuildInputs. mkDerivation would try
to go through __spliced.buildHost on docutils to get to the Linux
version, but the check in ensurePythonModules would kick in first,
triggering the meta check because of the equality check in the
implementation of hasPythonModule, which would fail because Python is
not marked as supported on FreeBSD in Nixpkgs at the moment. Thus,
even though they're not supposed to be, the meta checks would be
triggered even though the only attribute being accessed on the
unsupported derivation was __spliced.
We can fix this by using the same mechanism used to implement the meta
checks themselves: lib.extendDerivation. Now, attempting to access
drvPath or outPath on an attribute that fails the validity check will
produce the same error as before, but other accesses will be allowed
through, fixing splicing.
I've tested evaluation of packages that pass and fail the validity
check, and confirmed that the behaviour is still correct.
It wasn't clear before, but it turns out that this is just four
separate conditions, and an attribute is valid if any of the
conditions is true, so we don't need a sea of parentheses to express
it.
checkInputs used to be added to nativeBuildInputs. Now we have
nativeCheckInputs to do that instead. Doing this treewide change allows
to keep hashes identical to before the introduction of
nativeCheckInputs.
fetchPypi doesn't use python under the hood and doesn't need to be tied
to a specific version of python. Moving it to top level makes it more
consistent with other fetchers and makes code generation easier.
- Remove code related to i686 archs (not supported anymore since #209100)
- Split non-install steps in installPhase in installCheckPhase/preFixup
- Add missing pre/post hooks
Also build it with pythonInterpreters.pypy27_prebuilt instead of
python27, to avoid it being broken after #201859 is merged.
Keep in mind that the above change will make this derivation only work
with x86_64-linux for now (since pypy27_prebuilt is only packaged for
that platform). However support for newer platforms can be added later,
and a working package for one platform is better than a broken package
in 4 platforms.
The package set hasn't been working for a long time now, due to infinite
recursions, that nobody was going to fix.
The release is going to go EOL in 2023/06 and we don't want to ship it
in NixOS 23.05 anyway.
Include full $name of the derivation into docdir, so documentation for
python package "foo" is installed into $out/share/doc/python3.10-foo-1.2.3
instead of just $out/share/doc/foo, where it may conflict with some
other package named "foo" and not-coinstallable with different
versions of the same python package.
Change from $name to $pname was introduced in [1ee5fca], probably as
unindented side-effect of adding support for formats other than html.
Without `--with-system-{ffi,expat}` flags, Python will use its own
embedded libraries that are out-of-date. Thanks to it, they can be a
security issue. So let's use our own libraries instead.
This is already what Python 3.x does, so should be safe.
This PR strips down the modified `python27` derivation used by `resholve`. The
idea is to reduce the possible security issues, and also to make it easier to
bootstrap.
ActiveState is a company that is maintaining a fork of Python 2 to fixes
its security issues. Their support is paid, however the code is
open-source. See the details here:
https://www.activestate.com/products/python/python-2-end-of-life-security-updates/
This enable us to drop a bunch of CVE's patches for Python 2.7 and also
it should be easier to maintain, since we can just bump the version once
ActiveState tags a new version.
Prior to this commit, pythonRelaxDeps would only support removing
version constraints from "Requires-Dist" lines formatted in a particular
way ("foo (>= 1.2.3)"). This way is deprecated as per PyPA Core Metadata
Specs v2.1 [1]:
> Tools parsing the format should accept optional parentheses around
> this, but tools generating it should not use parentheses.
Additionally, a "Requires-Dist" dependency specification can contain
other metadata than just package name and version (extra names,
environment marker). These were being silently dropped by the prior
version of pythonRelaxDeps, or the version could not be relaxed.
The actual grammar is defined in PEP 508 [2]. Our tool of choice here is
sed extended regexps, so there's only so much we can do to be correct
with this parser. The regexp implemented in this commit makes an attempt
at supporting [extra] names, ; env_markers, as well as version specs
without parentheses. There are still unsupported features (URL specs) as
well as unhandled edge cases, but at some point trying to make the
regexp better is bound to awake ZALGO [3].
[1] https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use
[2] https://peps.python.org/pep-0508/#grammar
[3] https://stackoverflow.com/a/1732454/179806
stdenv is included in the package set. It does not provide a Python
module however and was therefore catched by the test. Because we do need
it we provide an exception for it.
This adds a test to ensure no new uses of `buildPythonApplication` can
be added to `python-packages.nix`.
Python packages can be grouped into two groups: 1) applications and 2)
packages providing importable modules. In `python-packages.nix` we only
want to have 2). 1) should be in the top-level package set.
To achieve this, all setup hooks need to be marked as being a setup hook.
For the setup hooks in the Python packages set this is done by creating
a new builder, `makePythonHook`.
Because there were issues with splicing, the file importing all the hooks
is converted to an extension. All non-packages were moved out of `python-packages.nix`
into `python-packages-base.nix`. The `keep` argument to `makeScopeWithSplicing
was cleaned up as well; there is no need to keep this one manually in sync
reducing the risk of breaking cross-compilation.
Before packages did not had to set `format = "other";` if had a custom
installPhase and would not produce a dist folder.
The current error message is hard to understand by new users:
> Executing pythonOutputDistPhase
> mv: cannot stat 'dist': No such file or directory
This commit improves the error message.
BEFORE:
the python derivation did not contain .nativeDrv and .crossDrv because
it was not from the __splicedPackages set
the python used in mk-python-derivation.nix was for host
```
nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs
[ «derivation /nix/store/bhz39ds4v02hn6x4py4mzjyilw4a589h-python3-aarch64-unknown-linux-gnu-3.10.7.drv» «derivation /nix/store/v880cnh4ml7czmivfbk3cdh93hz9yvbn-hook.drv» «derivation /nix/store/f243ab7wv92gqsmc9h7gr0qcnj0xcgdb-hook.drv» «derivation /nix/store/880lf8895bzn8d94lrr2y7ilgkxq0lc4-python-remove-tests-dir-hook.drv» «derivation /nix/store/fzjnhawfs1wpw58hcd1vxd9y750dc08y-python-remove-bin-bytecode-hook.drv» «derivation /nix/store/n8l59iparx98yfw8g5ydqmzmk3fdic75-setuptools-setup-hook.drv» «deri→
```
AFTER:
the python derivation does contain .nativeDrv and .crossDrv because
it is from the __splicedPackages set
those 2 are what makes nativeBuildInputs and buildInputs function
properly
the python used in mk-python-derivation.nix is for build
```
nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs
[ «derivation /nix/store/hvb9yxgv1133cfhxxd869sibldvv2vdx-python3-3.10.7.drv» «derivation /nix/store/v880cnh4ml7czmivfbk3cdh93hz9yvbn-hook.drv» «derivation /nix/store/f243ab7wv92gqsmc9h7gr0qcnj0xcgdb-hook.drv» «derivation /nix/store/880lf8895bzn8d94lrr2y7ilgkxq0lc4-python-remove-tests-dir-hook.drv» «derivation /nix/store/fzjnhawfs1wpw58hcd1vxd9y750dc08y-python-remove-bin-bytecode-hook.drv» «derivation /nix/store/n8l59iparx98yfw8g5ydqmzmk3fdic75-setuptools-setup-hook.drv» «derivation /nix/store/7vyhynla→
```
Since the Nix command used is an implementation detail of the script,
it's the script that will have to be updated if the command changes,
it's the script that should be opting into this, rather than making
the user do it globally.
Since c0972c16dc ("update-python-libraries: add missing dependency nix"),
Nix will be provided as a dependency, so we know we'll always be using
a modern version and no longer need the fallback.
*Flags implies a list
slightly relevant:
> stdenv: start deprecating non-list configureFlags https://github.com/NixOS/nixpkgs/pull/173172
the makeInstalledTests function in `nixos/tests/installed-tests/default.nix` isn't available outside of nixpkgs so
it's not a breaking change
By default buildPython* runs a hook for detecting conflicting packages.
This hook needs pkg_resources which is part of setuptools.
Before this commit, setuptools was simply added to the build. This meant
that when setuptools was forgotten to be added to the build, the build
and installation would still succeed because of this package from the
hook. During runtime (and cross-compilation) one would notice the
missing package.
Store the intermediate artifacts. In time, we should build, install and
test in separate derivations as that reduces circular dependencies,
avoids rebuilds when fixing tests, and makes it possible to use the
wheels for creating say virtualenv's.
