From 8cd56a3b7a2c4bbb7a9b8b117fdaab08db39ea30 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alejandro=20S=C3=A1nchez=20Medina?= Date: Thu, 17 Aug 2023 14:45:29 +0200 Subject: [PATCH] Documentation: nixpkgs manual: move Python reference to the top of Python chapter (#247117) * nixpkgs manual, doc Python: move Reference/Optimizations to FAQ See https://github.com/NixOS/nixpkgs/issues/246234. * nixpkgs manual, doc Python: move Reference/python-optional-dependencies to FAQ See https://github.com/NixOS/nixpkgs/issues/246234. * nixpkgs manual, doc Python: move Reference/Tools to FAQ See https://github.com/NixOS/nixpkgs/issues/246234. * nixpkgs manual, doc Python: move Reference/deterministic-builds to FAQ See https://github.com/NixOS/nixpkgs/issues/246234. * nixpkgs manual, doc Python: move Reference/automatic-tests to FAQ See https://github.com/NixOS/nixpkgs/issues/246234. * nixpkgs manual, doc Python: move Reference to top section See https://github.com/NixOS/nixpkgs/issues/246234. --- doc/languages-frameworks/python.section.md | 1218 ++++++++++---------- 1 file changed, 609 insertions(+), 609 deletions(-) diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md index d171e8419188..78b3559ac014 100644 --- a/doc/languages-frameworks/python.section.md +++ b/doc/languages-frameworks/python.section.md @@ -1,5 +1,512 @@ # Python {#python} +## Reference {#reference} + +### Interpreters {#interpreters} + +| Package | Aliases | Interpreter | +|------------|-----------------|-------------| +| python27 | python2, python | CPython 2.7 | +| python38 | | CPython 3.8 | +| python39 | | CPython 3.9 | +| python310 | python3 | CPython 3.10 | +| python311 | | CPython 3.11 | +| python312 | | CPython 3.12 | +| pypy27 | pypy2, pypy | PyPy2.7 | +| pypy39 | pypy3 | PyPy 3.9 | + +The Nix expressions for the interpreters can be found in +`pkgs/development/interpreters/python`. + +All packages depending on any Python interpreter get appended +`out/{python.sitePackages}` to `$PYTHONPATH` if such directory +exists. + +#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library} + +To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`. + +#### Attributes on interpreters packages {#attributes-on-interpreters-packages} + +Each interpreter has the following attributes: + +- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter. +- `interpreter`. Alias for `${python}/bin/${executable}`. +- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation. +- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation. +- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`. +- `executable`. Name of the interpreter executable, e.g. `python3.10`. +- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`. + +### Building packages and applications {#building-packages-and-applications} + +Python libraries and applications that use `setuptools` or +`distutils` are typically built with respectively the `buildPythonPackage` and +`buildPythonApplication` functions. These two functions also support installing a `wheel`. + +All Python packages reside in `pkgs/top-level/python-packages.nix` and all +applications elsewhere. In case a package is used as both a library and an +application, then the package should be in `pkgs/top-level/python-packages.nix` +since only those packages are made available for all interpreter versions. The +preferred location for library expressions is in +`pkgs/development/python-modules`. It is important that these packages are +called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee +the right version of the package is built. + +Based on the packages defined in `pkgs/top-level/python-packages.nix` an +attribute set is created for each available Python interpreter. The available +sets are + +* `pkgs.python27Packages` +* `pkgs.python3Packages` +* `pkgs.python38Packages` +* `pkgs.python39Packages` +* `pkgs.python310Packages` +* `pkgs.python311Packages` +* `pkgs.pypyPackages` + +and the aliases + +* `pkgs.python2Packages` pointing to `pkgs.python27Packages` +* `pkgs.python3Packages` pointing to `pkgs.python310Packages` +* `pkgs.pythonPackages` pointing to `pkgs.python2Packages` + +#### `buildPythonPackage` function {#buildpythonpackage-function} + +The `buildPythonPackage` function is implemented in +`pkgs/development/interpreters/python/mk-python-derivation.nix` +using setup hooks. + +The following is an example: + +```nix +{ lib +, buildPythonPackage +, fetchPypi + +# build-system +, setuptools-scm + +# dependencies +, attrs +, pluggy +, py +, setuptools +, six + +# tests +, hypothesis + }: + +buildPythonPackage rec { + pname = "pytest"; + version = "3.3.1"; + format = "setuptools"; + + src = fetchPypi { + inherit pname version; + hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M="; + }; + + postPatch = '' + # don't test bash builtins + rm testing/test_argcomplete.py + ''; + + nativeBuildInputs = [ + setuptools-scm + ]; + + propagatedBuildInputs = [ + attrs + py + setuptools + six + pluggy + ]; + + nativeCheckInputs = [ + hypothesis + ]; + + meta = with lib; { + changelog = "https://github.com/pytest-dev/pytest/releases/tag/${version}"; + description = "Framework for writing tests"; + homepage = "https://github.com/pytest-dev/pytest"; + license = licenses.mit; + maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ]; + }; +} +``` + +The `buildPythonPackage` mainly does four things: + +* In the `buildPhase`, it calls `${python.pythonForBuild.interpreter} setup.py bdist_wheel` to + build a wheel binary zipfile. +* In the `installPhase`, it installs the wheel file using `pip install *.whl`. +* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to + wrap all programs in the `$out/bin/*` directory to include `$PATH` + environment variable and add dependent libraries to script's `sys.path`. +* In the `installCheck` phase, `${python.interpreter} setup.py test` is run. + +By default tests are run because `doCheck = true`. Test dependencies, like +e.g. the test runner, should be added to `nativeCheckInputs`. + +By default `meta.platforms` is set to the same value +as the interpreter unless overridden otherwise. + +##### `buildPythonPackage` parameters {#buildpythonpackage-parameters} + +All parameters from `stdenv.mkDerivation` function are still supported. The +following are specific to `buildPythonPackage`: + +* `catchConflicts ? true`: If `true`, abort package build if a package name + appears more than once in dependency tree. Default is `true`. +* `disabled ? false`: If `true`, package is not built for the particular Python + interpreter version. +* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs. +* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment + variable in wrapped programs. +* `format ? "setuptools"`: Format of the source. Valid options are + `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`. + `"setuptools"` is for when the source has a `setup.py` and `setuptools` is + used to build a wheel, `flit`, in case `flit` should be used to build a wheel, + and `wheel` in case a wheel is provided. Use `other` when a custom + `buildPhase` and/or `installPhase` is needed. +* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to + `makeWrapper`, which wraps generated binaries. By default, the arguments to + `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling + the binary. Additional arguments here can allow a developer to set environment + variables which will be available when the binary is run. For example, + `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`. +* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this + defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`. +* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip + install`. To pass options to `python setup.py install`, use + `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`. +* `pipBuildFlags ? []`: A list of strings. Arguments to be passed to `pip wheel`. +* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`. +* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages + in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`). +* `preShellHook`: Hook to execute commands before `shellHook`. +* `postShellHook`: Hook to execute commands after `shellHook`. +* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only + created when the filenames end with `.py`. +* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command. +* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command. + +The `stdenv.mkDerivation` function accepts various parameters for describing +build inputs (see "Specifying dependencies"). The following are of special +interest for Python packages, either because these are primarily used, or +because their behaviour is different: + +* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables + as well as the items listed in `setup_requires`. +* `buildInputs ? []`: Build and/or run-time dependencies that need to be + compiled for the host machine. Typically non-Python libraries which are being + linked. +* `nativeCheckInputs ? []`: Dependencies needed for running the `checkPhase`. These + are added to `nativeBuildInputs` when `doCheck = true`. Items listed in + `tests_require` go here. +* `propagatedBuildInputs ? []`: Aside from propagating dependencies, + `buildPythonPackage` also injects code into and wraps executables with the + paths included in this list. Items listed in `install_requires` go here. + +##### Overriding Python packages {#overriding-python-packages} + +The `buildPythonPackage` function has a `overridePythonAttrs` method that can be +used to override the package. In the following example we create an environment +where we have the `blaze` package using an older version of `pandas`. We +override first the Python interpreter and pass `packageOverrides` which contains +the overrides for packages in the package set. + +```nix +with import {}; + +(let + python = let + packageOverrides = self: super: { + pandas = super.pandas.overridePythonAttrs(old: rec { + version = "0.19.1"; + src = fetchPypi { + pname = "pandas"; + inherit version; + hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE="; + }; + }); + }; + in pkgs.python3.override {inherit packageOverrides; self = python;}; + +in python.withPackages(ps: [ ps.blaze ])).env +``` + +The next example shows a non trivial overriding of the `blas` implementation to +be used through out all of the Python package set: + +```nix +python3MyBlas = pkgs.python3.override { + packageOverrides = self: super: { + # We need toPythonModule for the package set to evaluate this + blas = super.toPythonModule(super.pkgs.blas.override { + blasProvider = super.pkgs.mkl; + }); + lapack = super.toPythonModule(super.pkgs.lapack.override { + lapackProvider = super.pkgs.mkl; + }); + }; +}; +``` + +This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations. +Note that using simply `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in +compilation issues, because scipy dependencies need to use the same blas implementation as well. + +#### `buildPythonApplication` function {#buildpythonapplication-function} + +The `buildPythonApplication` function is practically the same as +`buildPythonPackage`. The main purpose of this function is to build a Python +package where one is interested only in the executables, and not importable +modules. For that reason, when adding this package to a `python.buildEnv`, the +modules won't be made available. + +Another difference is that `buildPythonPackage` by default prefixes the names of +the packages with the version of the interpreter. Because this is irrelevant for +applications, the prefix is omitted. + +When packaging a Python application with `buildPythonApplication`, it should be +called with `callPackage` and passed `python` or `pythonPackages` (possibly +specifying an interpreter version), like this: + +```nix +{ lib +, python3 +, fetchPypi +}: + +python3.pkgs.buildPythonApplication rec { + pname = "luigi"; + version = "2.7.9"; + format = "setuptools"; + + src = fetchPypi { + inherit pname version; + hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw="; + }; + + propagatedBuildInputs = with python3.pkgs; [ + tornado + python-daemon + ]; + + meta = with lib; { + ... + }; +} +``` + +This is then added to `all-packages.nix` just as any other application would be. + +```nix +luigi = callPackage ../applications/networking/cluster/luigi { }; +``` + +Since the package is an application, a consumer doesn't need to care about +Python versions or modules, which is why they don't go in `pythonPackages`. + +#### `toPythonApplication` function {#topythonapplication-function} + +A distinction is made between applications and libraries, however, sometimes a +package is used as both. In this case the package is added as a library to +`python-packages.nix` and as an application to `all-packages.nix`. To reduce +duplication the `toPythonApplication` can be used to convert a library to an +application. + +The Nix expression shall use `buildPythonPackage` and be called from +`python-packages.nix`. A reference shall be created from `all-packages.nix` to +the attribute in `python-packages.nix`, and the `toPythonApplication` shall be +applied to the reference: + +```nix +youtube-dl = with pythonPackages; toPythonApplication youtube-dl; +``` + +#### `toPythonModule` function {#topythonmodule-function} + +In some cases, such as bindings, a package is created using +`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python +bindings should be made available from `python-packages.nix`. The +`toPythonModule` function takes a derivation and makes certain Python-specific +modifications. + +```nix +opencv = toPythonModule (pkgs.opencv.override { + enablePython = true; + pythonPackages = self; +}); +``` + +Do pay attention to passing in the right Python version! + +#### `python.buildEnv` function {#python.buildenv-function} + +Python environments can be created using the low-level `pkgs.buildEnv` function. +This example shows how to create an environment that has the Pyramid Web Framework. +Saving the following as `default.nix` + +```nix +with import {}; + +python.buildEnv.override { + extraLibs = [ pythonPackages.pyramid ]; + ignoreCollisions = true; +} +``` + +and running `nix-build` will create + +``` +/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env +``` + +with wrapped binaries in `bin/`. + +You can also use the `env` attribute to create local environments with needed +packages installed. This is somewhat comparable to `virtualenv`. For example, +running `nix-shell` with the following `shell.nix` + +```nix +with import {}; + +(python3.buildEnv.override { + extraLibs = with python3Packages; [ + numpy + requests + ]; +}).env +``` + +will drop you into a shell where Python will have the +specified packages in its path. + +##### `python.buildEnv` arguments {#python.buildenv-arguments} + + +* `extraLibs`: List of packages installed inside the environment. +* `postBuild`: Shell command executed after the build of environment. +* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`). +* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in + wrapped binaries in the environment. + +#### `python.withPackages` function {#python.withpackages-function} + +The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality. +It takes a function as an argument that is passed the set of python packages and returns the list +of the packages to be included in the environment. Using the `withPackages` function, the previous +example for the Pyramid Web Framework environment can be written like this: + +```nix +with import {}; + +python.withPackages (ps: [ ps.pyramid ]) +``` + +`withPackages` passes the correct package set for the specific interpreter +version as an argument to the function. In the above example, `ps` equals +`pythonPackages`. But you can also easily switch to using python3: + +```nix +with import {}; + +python3.withPackages (ps: [ ps.pyramid ]) +``` + +Now, `ps` is set to `python3Packages`, matching the version of the interpreter. + +As `python.withPackages` simply uses `python.buildEnv` under the hood, it also +supports the `env` attribute. The `shell.nix` file from the previous section can +thus be also written like this: + +```nix +with import {}; + +(python3.withPackages (ps: with ps; [ + numpy + requests +])).env +``` + +In contrast to `python.buildEnv`, `python.withPackages` does not support the +more advanced options such as `ignoreCollisions = true` or `postBuild`. If you +need them, you have to use `python.buildEnv`. + +Python 2 namespace packages may provide `__init__.py` that collide. In that case +`python.buildEnv` should be used with `ignoreCollisions = true`. + +#### Setup hooks {#setup-hooks} + +The following are setup hooks specifically for Python packages. Most of these +are used in `buildPythonPackage`. + +- `eggUnpackhook` to move an egg to the correct folder so it can be installed + with the `eggInstallHook` +- `eggBuildHook` to skip building for eggs. +- `eggInstallHook` to install eggs. +- `flitBuildHook` to build a wheel using `flit`. +- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system + (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`. +- `pypaBuildHook` to build a wheel using + [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and + PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still + be added as `nativeBuildInput`. +- `pipInstallHook` to install wheels. +- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook). +- `pythonCatchConflictsHook` to check whether a Python package is not already existing. +- `pythonImportsCheckHook` to check whether importing the listed modules works. +- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package. + See [example usage](#using-pythonrelaxdepshook). +- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder. +- `setuptoolsBuildHook` to build a wheel using `setuptools`. +- `setuptoolsCheckHook` to run tests with `python setup.py test`. +- `sphinxHook` to build documentation and manpages using Sphinx. +- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A + `venv` is created if it does not yet exist. `postVenvCreation` can be used to + to run commands only after venv is first created. +- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed + with the `pipInstallHook`. +- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook). + +### Development mode {#development-mode} + +Development or editable mode is supported. To develop Python packages +`buildPythonPackage` has additional logic inside `shellPhase` to run `pip +install -e . --prefix $TMPDIR/`for the package. + +Warning: `shellPhase` is executed only if `setup.py` exists. + +Given a `default.nix`: + +```nix +with import {}; + +pythonPackages.buildPythonPackage { + name = "myproject"; + buildInputs = with pythonPackages; [ pyramid ]; + + src = ./.; +} +``` + +Running `nix-shell` with no arguments should give you the environment in which +the package would be built with `nix-build`. + +Shortcut to setup environments with C headers/libraries and Python packages: + +```shell +nix-shell -p pythonPackages.pyramid zlib libjpeg git +``` + +Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked. + ## User Guide {#user-guide} ### Using Python {#using-python} @@ -993,615 +1500,6 @@ don't explicitly define which `python` derivation should be used. In the above example we use `buildPythonPackage` that is part of the set `python3Packages`, and in this case the `python3` interpreter is automatically used. -## Reference {#reference} - -### Interpreters {#interpreters} - -| Package | Aliases | Interpreter | -|------------|-----------------|-------------| -| python27 | python2, python | CPython 2.7 | -| python38 | | CPython 3.8 | -| python39 | | CPython 3.9 | -| python310 | python3 | CPython 3.10 | -| python311 | | CPython 3.11 | -| python312 | | CPython 3.12 | -| pypy27 | pypy2, pypy | PyPy2.7 | -| pypy39 | pypy3 | PyPy 3.9 | - -The Nix expressions for the interpreters can be found in -`pkgs/development/interpreters/python`. - -All packages depending on any Python interpreter get appended -`out/{python.sitePackages}` to `$PYTHONPATH` if such directory -exists. - -#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library} - -To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`. - -#### Attributes on interpreters packages {#attributes-on-interpreters-packages} - -Each interpreter has the following attributes: - -- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter. -- `interpreter`. Alias for `${python}/bin/${executable}`. -- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation. -- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation. -- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`. -- `executable`. Name of the interpreter executable, e.g. `python3.10`. -- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`. - -### Optimizations {#optimizations} - -The Python interpreters are by default not built with optimizations enabled, because -the builds are in that case not reproducible. To enable optimizations, override the -interpreter of interest, e.g using - -``` -let - pkgs = import ./. {}; - mypython = pkgs.python3.override { - enableOptimizations = true; - reproducibleBuild = false; - self = mypython; - }; -in mypython -``` - -### Building packages and applications {#building-packages-and-applications} - -Python libraries and applications that use `setuptools` or -`distutils` are typically built with respectively the `buildPythonPackage` and -`buildPythonApplication` functions. These two functions also support installing a `wheel`. - -All Python packages reside in `pkgs/top-level/python-packages.nix` and all -applications elsewhere. In case a package is used as both a library and an -application, then the package should be in `pkgs/top-level/python-packages.nix` -since only those packages are made available for all interpreter versions. The -preferred location for library expressions is in -`pkgs/development/python-modules`. It is important that these packages are -called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee -the right version of the package is built. - -Based on the packages defined in `pkgs/top-level/python-packages.nix` an -attribute set is created for each available Python interpreter. The available -sets are - -* `pkgs.python27Packages` -* `pkgs.python3Packages` -* `pkgs.python38Packages` -* `pkgs.python39Packages` -* `pkgs.python310Packages` -* `pkgs.python311Packages` -* `pkgs.pypyPackages` - -and the aliases - -* `pkgs.python2Packages` pointing to `pkgs.python27Packages` -* `pkgs.python3Packages` pointing to `pkgs.python310Packages` -* `pkgs.pythonPackages` pointing to `pkgs.python2Packages` - -#### `buildPythonPackage` function {#buildpythonpackage-function} - -The `buildPythonPackage` function is implemented in -`pkgs/development/interpreters/python/mk-python-derivation.nix` -using setup hooks. - -The following is an example: - -```nix -{ lib -, buildPythonPackage -, fetchPypi - -# build-system -, setuptools-scm - -# dependencies -, attrs -, pluggy -, py -, setuptools -, six - -# tests -, hypothesis - }: - -buildPythonPackage rec { - pname = "pytest"; - version = "3.3.1"; - format = "setuptools"; - - src = fetchPypi { - inherit pname version; - hash = "sha256-z4Q23FnYaVNG/NOrKW3kZCXsqwDWQJbOvnn7Ueyy65M="; - }; - - postPatch = '' - # don't test bash builtins - rm testing/test_argcomplete.py - ''; - - nativeBuildInputs = [ - setuptools-scm - ]; - - propagatedBuildInputs = [ - attrs - py - setuptools - six - pluggy - ]; - - nativeCheckInputs = [ - hypothesis - ]; - - meta = with lib; { - changelog = "https://github.com/pytest-dev/pytest/releases/tag/${version}"; - description = "Framework for writing tests"; - homepage = "https://github.com/pytest-dev/pytest"; - license = licenses.mit; - maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ]; - }; -} -``` - -The `buildPythonPackage` mainly does four things: - -* In the `buildPhase`, it calls `${python.pythonForBuild.interpreter} setup.py bdist_wheel` to - build a wheel binary zipfile. -* In the `installPhase`, it installs the wheel file using `pip install *.whl`. -* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to - wrap all programs in the `$out/bin/*` directory to include `$PATH` - environment variable and add dependent libraries to script's `sys.path`. -* In the `installCheck` phase, `${python.interpreter} setup.py test` is run. - -By default tests are run because `doCheck = true`. Test dependencies, like -e.g. the test runner, should be added to `nativeCheckInputs`. - -By default `meta.platforms` is set to the same value -as the interpreter unless overridden otherwise. - -##### `buildPythonPackage` parameters {#buildpythonpackage-parameters} - -All parameters from `stdenv.mkDerivation` function are still supported. The -following are specific to `buildPythonPackage`: - -* `catchConflicts ? true`: If `true`, abort package build if a package name - appears more than once in dependency tree. Default is `true`. -* `disabled ? false`: If `true`, package is not built for the particular Python - interpreter version. -* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs. -* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment - variable in wrapped programs. -* `format ? "setuptools"`: Format of the source. Valid options are - `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`. - `"setuptools"` is for when the source has a `setup.py` and `setuptools` is - used to build a wheel, `flit`, in case `flit` should be used to build a wheel, - and `wheel` in case a wheel is provided. Use `other` when a custom - `buildPhase` and/or `installPhase` is needed. -* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to - `makeWrapper`, which wraps generated binaries. By default, the arguments to - `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling - the binary. Additional arguments here can allow a developer to set environment - variables which will be available when the binary is run. For example, - `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`. -* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this - defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications to `""`. -* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip - install`. To pass options to `python setup.py install`, use - `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`. -* `pipBuildFlags ? []`: A list of strings. Arguments to be passed to `pip wheel`. -* `pypaBuildFlags ? []`: A list of strings. Arguments to be passed to `python -m build --wheel`. -* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages - in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`). -* `preShellHook`: Hook to execute commands before `shellHook`. -* `postShellHook`: Hook to execute commands after `shellHook`. -* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only - created when the filenames end with `.py`. -* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command. -* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command. - -The `stdenv.mkDerivation` function accepts various parameters for describing -build inputs (see "Specifying dependencies"). The following are of special -interest for Python packages, either because these are primarily used, or -because their behaviour is different: - -* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables - as well as the items listed in `setup_requires`. -* `buildInputs ? []`: Build and/or run-time dependencies that need to be - compiled for the host machine. Typically non-Python libraries which are being - linked. -* `nativeCheckInputs ? []`: Dependencies needed for running the `checkPhase`. These - are added to `nativeBuildInputs` when `doCheck = true`. Items listed in - `tests_require` go here. -* `propagatedBuildInputs ? []`: Aside from propagating dependencies, - `buildPythonPackage` also injects code into and wraps executables with the - paths included in this list. Items listed in `install_requires` go here. - -##### Overriding Python packages {#overriding-python-packages} - -The `buildPythonPackage` function has a `overridePythonAttrs` method that can be -used to override the package. In the following example we create an environment -where we have the `blaze` package using an older version of `pandas`. We -override first the Python interpreter and pass `packageOverrides` which contains -the overrides for packages in the package set. - -```nix -with import {}; - -(let - python = let - packageOverrides = self: super: { - pandas = super.pandas.overridePythonAttrs(old: rec { - version = "0.19.1"; - src = fetchPypi { - pname = "pandas"; - inherit version; - hash = "sha256-JQn+rtpy/OA2deLszSKEuxyttqBzcAil50H+JDHUdCE="; - }; - }); - }; - in pkgs.python3.override {inherit packageOverrides; self = python;}; - -in python.withPackages(ps: [ ps.blaze ])).env -``` - -The next example shows a non trivial overriding of the `blas` implementation to -be used through out all of the Python package set: - -```nix -python3MyBlas = pkgs.python3.override { - packageOverrides = self: super: { - # We need toPythonModule for the package set to evaluate this - blas = super.toPythonModule(super.pkgs.blas.override { - blasProvider = super.pkgs.mkl; - }); - lapack = super.toPythonModule(super.pkgs.lapack.override { - lapackProvider = super.pkgs.mkl; - }); - }; -}; -``` - -This is particularly useful for numpy and scipy users who want to gain speed with other blas implementations. -Note that using simply `scipy = super.scipy.override { blas = super.pkgs.mkl; };` will likely result in -compilation issues, because scipy dependencies need to use the same blas implementation as well. - -#### Optional extra dependencies {#python-optional-dependencies} - -Some packages define optional dependencies for additional features. With -`setuptools` this is called `extras_require` and `flit` calls it -`extras-require`, while PEP 621 calls these `optional-dependencies`. A -method for supporting this is by declaring the extras of a package in its -`passthru`, e.g. in case of the package `dask` - -```nix -passthru.optional-dependencies = { - complete = [ distributed ]; -}; -``` - -and letting the package requiring the extra add the list to its dependencies - -```nix -propagatedBuildInputs = [ - ... -] ++ dask.optional-dependencies.complete; -``` - -Note this method is preferred over adding parameters to builders, as that can -result in packages depending on different variants and thereby causing -collisions. - -#### `buildPythonApplication` function {#buildpythonapplication-function} - -The `buildPythonApplication` function is practically the same as -`buildPythonPackage`. The main purpose of this function is to build a Python -package where one is interested only in the executables, and not importable -modules. For that reason, when adding this package to a `python.buildEnv`, the -modules won't be made available. - -Another difference is that `buildPythonPackage` by default prefixes the names of -the packages with the version of the interpreter. Because this is irrelevant for -applications, the prefix is omitted. - -When packaging a Python application with `buildPythonApplication`, it should be -called with `callPackage` and passed `python` or `pythonPackages` (possibly -specifying an interpreter version), like this: - -```nix -{ lib -, python3 -, fetchPypi -}: - -python3.pkgs.buildPythonApplication rec { - pname = "luigi"; - version = "2.7.9"; - format = "setuptools"; - - src = fetchPypi { - inherit pname version; - hash = "sha256-Pe229rT0aHwA98s+nTHQMEFKZPo/yw6sot8MivFDvAw="; - }; - - propagatedBuildInputs = with python3.pkgs; [ - tornado - python-daemon - ]; - - meta = with lib; { - ... - }; -} -``` - -This is then added to `all-packages.nix` just as any other application would be. - -```nix -luigi = callPackage ../applications/networking/cluster/luigi { }; -``` - -Since the package is an application, a consumer doesn't need to care about -Python versions or modules, which is why they don't go in `pythonPackages`. - -#### `toPythonApplication` function {#topythonapplication-function} - -A distinction is made between applications and libraries, however, sometimes a -package is used as both. In this case the package is added as a library to -`python-packages.nix` and as an application to `all-packages.nix`. To reduce -duplication the `toPythonApplication` can be used to convert a library to an -application. - -The Nix expression shall use `buildPythonPackage` and be called from -`python-packages.nix`. A reference shall be created from `all-packages.nix` to -the attribute in `python-packages.nix`, and the `toPythonApplication` shall be -applied to the reference: - -```nix -youtube-dl = with pythonPackages; toPythonApplication youtube-dl; -``` - -#### `toPythonModule` function {#topythonmodule-function} - -In some cases, such as bindings, a package is created using -`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python -bindings should be made available from `python-packages.nix`. The -`toPythonModule` function takes a derivation and makes certain Python-specific -modifications. - -```nix -opencv = toPythonModule (pkgs.opencv.override { - enablePython = true; - pythonPackages = self; -}); -``` - -Do pay attention to passing in the right Python version! - -#### `python.buildEnv` function {#python.buildenv-function} - -Python environments can be created using the low-level `pkgs.buildEnv` function. -This example shows how to create an environment that has the Pyramid Web Framework. -Saving the following as `default.nix` - -```nix -with import {}; - -python.buildEnv.override { - extraLibs = [ pythonPackages.pyramid ]; - ignoreCollisions = true; -} -``` - -and running `nix-build` will create - -``` -/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env -``` - -with wrapped binaries in `bin/`. - -You can also use the `env` attribute to create local environments with needed -packages installed. This is somewhat comparable to `virtualenv`. For example, -running `nix-shell` with the following `shell.nix` - -```nix -with import {}; - -(python3.buildEnv.override { - extraLibs = with python3Packages; [ - numpy - requests - ]; -}).env -``` - -will drop you into a shell where Python will have the -specified packages in its path. - -##### `python.buildEnv` arguments {#python.buildenv-arguments} - - -* `extraLibs`: List of packages installed inside the environment. -* `postBuild`: Shell command executed after the build of environment. -* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`). -* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in - wrapped binaries in the environment. - -#### `python.withPackages` function {#python.withpackages-function} - -The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality. -It takes a function as an argument that is passed the set of python packages and returns the list -of the packages to be included in the environment. Using the `withPackages` function, the previous -example for the Pyramid Web Framework environment can be written like this: - -```nix -with import {}; - -python.withPackages (ps: [ ps.pyramid ]) -``` - -`withPackages` passes the correct package set for the specific interpreter -version as an argument to the function. In the above example, `ps` equals -`pythonPackages`. But you can also easily switch to using python3: - -```nix -with import {}; - -python3.withPackages (ps: [ ps.pyramid ]) -``` - -Now, `ps` is set to `python3Packages`, matching the version of the interpreter. - -As `python.withPackages` simply uses `python.buildEnv` under the hood, it also -supports the `env` attribute. The `shell.nix` file from the previous section can -thus be also written like this: - -```nix -with import {}; - -(python3.withPackages (ps: with ps; [ - numpy - requests -])).env -``` - -In contrast to `python.buildEnv`, `python.withPackages` does not support the -more advanced options such as `ignoreCollisions = true` or `postBuild`. If you -need them, you have to use `python.buildEnv`. - -Python 2 namespace packages may provide `__init__.py` that collide. In that case -`python.buildEnv` should be used with `ignoreCollisions = true`. - -#### Setup hooks {#setup-hooks} - -The following are setup hooks specifically for Python packages. Most of these -are used in `buildPythonPackage`. - -- `eggUnpackhook` to move an egg to the correct folder so it can be installed - with the `eggInstallHook` -- `eggBuildHook` to skip building for eggs. -- `eggInstallHook` to install eggs. -- `flitBuildHook` to build a wheel using `flit`. -- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system - (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`. -- `pypaBuildHook` to build a wheel using - [`pypa/build`](https://pypa-build.readthedocs.io/en/latest/index.html) and - PEP 517/518. Note a build system (e.g. `setuptools` or `flit`) should still - be added as `nativeBuildInput`. -- `pipInstallHook` to install wheels. -- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook). -- `pythonCatchConflictsHook` to check whether a Python package is not already existing. -- `pythonImportsCheckHook` to check whether importing the listed modules works. -- `pythonRelaxDepsHook` will relax Python dependencies restrictions for the package. - See [example usage](#using-pythonrelaxdepshook). -- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder. -- `setuptoolsBuildHook` to build a wheel using `setuptools`. -- `setuptoolsCheckHook` to run tests with `python setup.py test`. -- `sphinxHook` to build documentation and manpages using Sphinx. -- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A - `venv` is created if it does not yet exist. `postVenvCreation` can be used to - to run commands only after venv is first created. -- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed - with the `pipInstallHook`. -- `unittestCheckHook` will run tests with `python -m unittest discover`. See [example usage](#using-unittestcheckhook). - -### Development mode {#development-mode} - -Development or editable mode is supported. To develop Python packages -`buildPythonPackage` has additional logic inside `shellPhase` to run `pip -install -e . --prefix $TMPDIR/`for the package. - -Warning: `shellPhase` is executed only if `setup.py` exists. - -Given a `default.nix`: - -```nix -with import {}; - -pythonPackages.buildPythonPackage { - name = "myproject"; - buildInputs = with pythonPackages; [ pyramid ]; - - src = ./.; -} -``` - -Running `nix-shell` with no arguments should give you the environment in which -the package would be built with `nix-build`. - -Shortcut to setup environments with C headers/libraries and Python packages: - -```shell -nix-shell -p pythonPackages.pyramid zlib libjpeg git -``` - -Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked. - -### Tools {#tools} - -Packages inside nixpkgs must use the `buildPythonPackage` or `buildPythonApplication` function directly, -because we can only provide security support for non-vendored dependencies. - -We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs, -as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`. - -### Deterministic builds {#deterministic-builds} - -The Python interpreters are now built deterministically. Minor modifications had -to be made to the interpreters in order to generate deterministic bytecode. This -has security implications and is relevant for those using Python in a -`nix-shell`. - -When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will -have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1` -and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED). -Both are also exported in `nix-shell`. - -### Automatic tests {#automatic-tests} - -It is recommended to test packages as part of the build process. -Source distributions (`sdist`) often include test files, but not always. - -By default the command `python setup.py test` is run as part of the -`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An -example of such a situation is when `py.test` is used. - -#### Common issues {#common-issues} - -* Non-working tests can often be deselected. By default `buildPythonPackage` - runs `python setup.py test`. which is deprecated. Most Python modules however - do follow the standard test protocol where the pytest runner can be used - instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test - methods or classes as well as whole files. For `pytestCheckHook` these are - conveniently exposed as `disabledTests` and `disabledTestPaths` respectively. - - ```nix - buildPythonPackage { - # ... - nativeCheckInputs = [ - pytestCheckHook - ]; - - disabledTests = [ - "function_name" - "other_function" - ]; - - disabledTestPaths = [ - "this/file.py" - ]; - } - ``` - -* Tests that attempt to access `$HOME` can be fixed by using the following - work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)` - ## FAQ {#faq} ### How to solve circular dependencies? {#how-to-solve-circular-dependencies} @@ -1950,6 +1848,108 @@ In a `setup.py` or `setup.cfg` it is common to declare dependencies: * `install_requires` corresponds to `propagatedBuildInputs` * `tests_require` corresponds to `nativeCheckInputs` +### How to enable interpreter optimizations? {#optimizations} + +The Python interpreters are by default not built with optimizations enabled, because +the builds are in that case not reproducible. To enable optimizations, override the +interpreter of interest, e.g using + +``` +let + pkgs = import ./. {}; + mypython = pkgs.python3.override { + enableOptimizations = true; + reproducibleBuild = false; + self = mypython; + }; +in mypython +``` + +### How to add optional dependencies? {#python-optional-dependencies} + +Some packages define optional dependencies for additional features. With +`setuptools` this is called `extras_require` and `flit` calls it +`extras-require`, while PEP 621 calls these `optional-dependencies`. A +method for supporting this is by declaring the extras of a package in its +`passthru`, e.g. in case of the package `dask` + +```nix +passthru.optional-dependencies = { + complete = [ distributed ]; +}; +``` + +and letting the package requiring the extra add the list to its dependencies + +```nix +propagatedBuildInputs = [ + ... +] ++ dask.optional-dependencies.complete; +``` + +Note this method is preferred over adding parameters to builders, as that can +result in packages depending on different variants and thereby causing +collisions. + +### How to contribute a Python package to nixpkgs? {#tools} + +Packages inside nixpkgs must use the `buildPythonPackage` or `buildPythonApplication` function directly, +because we can only provide security support for non-vendored dependencies. + +We recommend [nix-init](https://github.com/nix-community/nix-init) for creating new python packages within nixpkgs, +as it already prefetches the source, parses dependencies for common formats and prefills most things in `meta`. + +### Are Python interpreters built deterministically? {#deterministic-builds} + +The Python interpreters are now built deterministically. Minor modifications had +to be made to the interpreters in order to generate deterministic bytecode. This +has security implications and is relevant for those using Python in a +`nix-shell`. + +When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will +have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1` +and [PYTHONHASHSEED=0](https://docs.python.org/3.11/using/cmdline.html#envvar-PYTHONHASHSEED). +Both are also exported in `nix-shell`. + +### How to provide automatic tests to Python packages? {#automatic-tests} + +It is recommended to test packages as part of the build process. +Source distributions (`sdist`) often include test files, but not always. + +By default the command `python setup.py test` is run as part of the +`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An +example of such a situation is when `py.test` is used. + +#### Common issues {#common-issues} + +* Non-working tests can often be deselected. By default `buildPythonPackage` + runs `python setup.py test`. which is deprecated. Most Python modules however + do follow the standard test protocol where the pytest runner can be used + instead. `pytest` supports the `-k` and `--ignore` parameters to ignore test + methods or classes as well as whole files. For `pytestCheckHook` these are + conveniently exposed as `disabledTests` and `disabledTestPaths` respectively. + + ```nix + buildPythonPackage { + # ... + nativeCheckInputs = [ + pytestCheckHook + ]; + + disabledTests = [ + "function_name" + "other_function" + ]; + + disabledTestPaths = [ + "this/file.py" + ]; + } + ``` + +* Tests that attempt to access `$HOME` can be fixed by using the following + work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)` + ## Contributing {#contributing} ### Contributing guidelines {#contributing-guidelines}