Commit Graph

46 Commits

Author SHA1 Message Date
David Arnold
db614e11d6
nixos/tests/test-driver: better control test env symbols
Previous to this commit, the entire test driver environment was shared
with the actual python test environment.

This is a hefty api surface. This commit selectively exposes only those
symbols to the test environment that are actually meant to be used by
tests.
2021-08-19 23:55:26 -05:00
Maximilian Bosch
1ba3f7927f
nixos/testing: only create nixos-run-vms for nixos-build-vms(8)
This is the case when the test-script is empty. `nixos-build-vms(8)` is
primarily supposed to be used as tool to test changes or to reproduce
bugs (IMHO) where "just spinning up a few VMs" is the primary use-case.

In the ongoing discussion about these changes[1] it was suggested to
only expose it when needed (i.e. in the case I described above) to keep
the API surface as slim as possible.

[1] https://github.com/NixOS/nixpkgs/pull/133675#discussion_r688112485
2021-08-13 17:39:49 +02:00
Maximilian Bosch
260d9cc7e1
nixos/testing: re-add nixos-run-vms script
Originally removed in 926fb93968. This
one is actually quite useful for `nixos-build-vms(8)`.
2021-08-12 22:50:29 +02:00
lewo
7aa78642c5
Merge pull request #125979 from blaggacao/nixos-test-ref/03-normalse-the-python-entrypoint
nixos/test-driver: normalize the python entrypoint
2021-08-08 10:24:47 +02:00
David Arnold
926fb93968
nixos/tests/test-driver: normalise test driver entrypoint(s)
Previously the driver was configured exclusively through convoluted
environment variables.

Now the driver's defaults are configured through env variables.

Some additional concerns are in the github comments of this PR.
2021-08-05 19:07:11 -05:00
David Arnold
dc060ff8b2
nixos/testing: fix invalid node names detection
the use of python further restricts possible RFC1035 host labels since
dash is not allowed for use in python identifiers.

The previous implementation of this check was flawed, since it did not
check the `hostName` value that is actually used to construe the
identifier, but the node name, which can be anything, e.g. just `machine`.

The previous implementation, by further restricting RFC1035 labels, only
for the sake of testing seems to be an unacceptable restriction and should
be addressed separately.
2021-07-28 15:37:57 -05:00
Jörg Thalheim
48aec70ea1
nixos/test-driver: replace termlib with socat
telnetlib does not handle unicode, which is annoying when using systemctl.
Also this gives us a nice readline with history.
2021-06-22 09:08:57 +02:00
aszlig
bc0997489b
nixos/test-driver: Fix passing passthru attribute
Apparently this looks like it was forgotten when doing commit
3884ff70ba, which refactored the test
runner and driver a bit.

The passthru argument actually was correctly reintroduced in
setupDriverForTest, but the actual makeTest function didn't use it.

This fixes the nixpkgs tarball job, which previously failed with:

  attribute 'elkPackages' missing, at /build/source/pkgs/tools/misc/logstash/6.x.nix:58:30

Signed-off-by: aszlig <aszlig@nix.build>
Acked-by: David Arnold <dar@xoe.solutions>
Fixes: https://github.com/NixOS/nixpkgs/issues/127274
Merges: https://github.com/NixOS/nixpkgs/pull/127346
2021-06-18 14:50:15 +02:00
David Arnold
3884ff70ba
nixos/tests/test-driver: cleanup nix expression
Less nesting, where that improves readability. More nesteing, where
that improves readability, but most importantly:

Expose individual functions separately so that they can be more easily
built directly, eg.:

`nix build --impure --expr '(import ./testing-python.nix {system = builtins.currentSystem;}).mkTestDriver'`
2021-06-13 15:47:51 -05:00
David Arnold
03bfa29891
nixos/tests/test-driver: cleanup "dead" code (USE_SERIAL)
At nixpkgs root:

`rg redirectSerial ./` does not result in any other match
nor does
`rg USE_SERIAL ./` except for an unrelated match in:

pkgs/tools/graphics/argyllcms/default.nix
2021-06-06 12:58:58 -05:00
Frederik Rietdijk
3edde6562e make-test-python: disallow aliases
When importing Nixpkgs within Nixpkgs, we should not consider aliases
to ensure we don't rely on them internally.

There are probably more places that need to be converted.
2021-06-03 11:03:31 -07:00
David Arnold
8ee31be5dd
nixos/testing: add interactive serial stdout logs switch and dim them 2021-05-14 20:36:00 -04:00
Robert Hensing
75c4fc1c8b nixos/testing-python.nix: Move makeWrapper to nativeBuildInputs 2021-05-09 03:04:03 +02:00
Robert Hensing
56d9637119
nixos/testing: Set up scope for testScript linter
Our test driver exposes a bunch of variables and functions, which
pyflakes doesn't recognise by default because it assumes that the test
script is executed standalone. In reality however the test driver script
is using exec() on the testScript.

Fortunately pyflakes has $PYFLAKES_BUILTINS, which are the attributes
that are globally available on all modules to be checked. Since we only
have one module, using this environment variable is fine as opposed to
my first approach to this, which tried to use the unstable internal API
of pyflakes.

The attributes are gathered by the main derivation of the test driver,
because we don't want to end up defining a new attribute in the test
driver module just to being confused why using it in a test will result
in an error.

Another way we could have gathered these attributes would be in
mkDriver, which is where the linting takes place. However, we do have a
different set of Python dependencies in scope and duplicating these will
again just cause confusion over having it at one location only.

Signed-off-by: aszlig <aszlig@nix.build>
Co-Authored-By: aszlig <aszlig@nix.build>
2021-05-09 02:26:51 +02:00
Robert Hensing
71087b2bc4
nixos/testing-python.nix: Expose driver
(cherry picked from commit a2c9220568648b4528154ebd8e657add243ed0b4)
2021-05-09 02:26:40 +02:00
aszlig
c362a28fcf
nixos/testing: Switch from black to pyflakes
So far, we have used "black" for formatting the test code, which is
rather strict and opinionated and when used inline in Nix expressions it
creates all sorts of trouble.

One of the main annoyances is that when using strings coming from Nix
expressions (eg. store paths or option definitions from NixOS modules),
completely unrelated changes could cause tests to fail, since eg. black
wants lines to be broken.

Another downside of enforcing a certain kind of formatting is that it
makes the Nix expression code inconsistent because we're mixing two
spaces of indentation (common in nixpkgs) with four spaces of
indentation as defined in PEP-8. While this is perfectly fine for
standalone Python files, it really looks ugly and inconsistent IMO when
used within Nix strings.

What we actually want though is a linter that catches problems early on
before actually running the test, because this is *actually* helping in
development because running the actual VM test takes much longer.

This is the reason why I switched from black to pyflakes, because the
latter actually has useful checks, eg. usage of undefined variables,
invalid format arguments, duplicate arguments, shadowed loop vars and
more.

Signed-off-by: aszlig <aszlig@nix.build>
Closes: https://github.com/NixOS/nixpkgs/issues/72964
2021-05-09 02:26:37 +02:00
Robert Hensing
162b8fba12 testing-python.nix: document runTests pos argument 2021-05-03 13:33:41 +02:00
Robert Hensing
b2f44e9aeb
Merge pull request #112504 from hercules-ci/fix-nixosTest-meta-position
nixosTest: fix meta.position
2021-05-03 11:50:57 +02:00
Alyssa Ross
a8afbb45c1 treewide: use lib.warnIf where appropriate 2021-04-28 21:44:21 +00:00
Robert Hensing
cda907d895 nixosTest: fix meta.position 2021-03-23 15:14:31 +01:00
Robert Hensing
72a5301610 nixos/lib/testing-python.nix: Wire up passthru 2020-12-09 13:24:03 +01:00
Andreas Rammhold
eba8f5425f
nixos/tests: fix wrong inherit that passes on the nodes attrs
The hydra tarball step would fail due to the nodes attribute not being
properly inherited. Since we can't execute all the tests and release
steps locally anymore (thanks to the JSONification and faster hydra
eval) these errors will probably keep in appearing.

This is hopefully the last of those introduced by me test runner
refactoring.

Error was seen on hydra (https://hydra.nixos.org/build/129282411):
> unpacking sources
> unpacking source archive /nix/store/bp95x52h6nv3j8apxrryyj2rviw682k1-source
> source root is source
> patching sources
> autoconfPhase
> No bootstrap, bootstrap.sh, configure.in or configure.ac. Assuming this is not an GNU Autotools package.
> configuring
> release name is nixpkgs-21.03pre249116.1088f059401
> git-revision is 1088f05940
> building
> no Makefile, doing nothing
> running tests
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> warning: you did not specify '--add-root'; the result might be removed by the garbage collector
> checking Nixpkgs on i686-linux
> checking Nixpkgs on x86_64-linux
> checking Nixpkgs on x86_64-darwin
> checking eval-release.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> checking find-tarballs.nix
> trace: `mkStrict' is obsolete; use `mkOverride 0' instead.
> trace: `lib.nixpkgsVersion` is deprecated, use `lib.version` instead!
> trace: warning: lib.readPathsFromFile is deprecated, use a list instead
> trace: Warning: `showVal` is deprecated and will be removed in the next release, please use `traceSeqN`
> trace: lib.zip is deprecated, use lib.zipAttrsWith instead
> error: while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:6:1, called from undefined position:
> while evaluating 'operator' at /build/source/maintainers/scripts/find-tarballs.nix:27:16, called from undefined position:
> while evaluating 'immediateDependenciesOf' at /build/source/maintainers/scripts/find-tarballs.nix:39:29, called from /build/source/maintainers/scripts/find-tarballs.nix:27:44:
> while evaluating anonymous function at /build/source/lib/attrsets.nix:234:10, called from undefined position:
> while evaluating anonymous function at /build/source/maintainers/scripts/find-tarballs.nix:40:37, called from /build/source/lib/attrsets.nix:234:16:
> while evaluating 'derivationsIn' at /build/source/maintainers/scripts/find-tarballs.nix:42:19, called from /build/source/maintainers/scripts/find-tarballs.nix:40:40:
> while evaluating 'canEval' at /build/source/maintainers/scripts/find-tarballs.nix:48:13, called from /build/source/maintainers/scripts/find-tarballs.nix:43:9:
> while evaluating the attribute 'nodes' at /build/source/nixos/lib/testing-python.nix:195:23:
> attribute 'nodes' missing, at /build/source/nixos/lib/testing-python.nix:193:16
> build time elapsed:  0m0.122s 0m0.043s 17m51.526s 0m56.668s
> builder for '/nix/store/96rk3c74vrk6m3snm7n6jhis3j640pn4-nixpkgs-tarball-21.03pre249116.1088f059401.drv' failed with exit code 1
2020-10-27 00:10:31 +01:00
Andreas Rammhold
73635b859d
nixos/tests: fix runInMachine
In 5500dc8 we introduced the --keep-vm-state flag and defaulted to that
flag not being set. This lead to the `runInMachine` tests not longer
working and that going unnoticed for quite some time now.
2020-10-25 20:09:33 +01:00
Andreas Rammhold
fa25d84d13
nixos/tests: fix testDriver reference in runInMachine function
In a previous commit I broke this as there is no longer one testDriver
but only a function to generate one based on some QEMU inputs.
2020-10-25 20:09:33 +01:00
Andreas Rammhold
61b09f552c
nixos/tests: format the testing-python.nix file more consistenly 2020-10-25 20:09:33 +01:00
Andreas Rammhold
04100cd281
nixos/tests: restructure test driver so that QEMU is actually overriden
Previously you would be able to override only the QEMU package to be
used in the test runner. Frankly that doesn't help a lot if you are
trying to get a graphical session. The graphical session requires the
option in the NixOS module system to bet set to the correct QEMU
package.

In this commit I moved most of the test node configuration and
transformations into the `mkDriver` function (previously called
`driver`). The motivation was to be able to create a `driver` instance
with a given QEMU package that will be used consistently througout the
test expression.
2020-10-25 20:09:33 +01:00
Andreas Rammhold
20893b3a70
nixos/tests: expose both the interactive and non-interactive driver
For a lot of the work the non-interactive drivers are enough and it is
probably a good idea to keep it accessible for debugging without
touching the Nix expression.
2020-10-19 17:39:48 +02:00
Andreas Rammhold
c096880d46
nixos/tests: make the driver attribute use a rich qemu
Since we previously stripped down the features of `qemu_test` some of
the features users are used to while running tests through the (impure)
driver didn't work anymore. Most notably we lost support for graphical
output and audio. With this change the `driver` attribute uses are more
feature complete version of QEmu compared to the one used in the pure
Nix builds.

This gives us the best of both worlds. Users are able to see the
graphical windows of VMs while CI and regular nix builds do not have to
download all the (unnecessary) dependencies.
2020-10-19 17:39:48 +02:00
Janne Heß
9f33ab62d9
nixos/testing: Add support for specialArgs
Since using flakes disallows the usage of <unstable> (which I use in
some tests), this adds an alternative. By setting specialArgs, all VMs
can get the `unstable` flake input as an arg. This is not possible with
extraConfigurations, as that would lead to infinite recursions.
2020-09-14 00:10:21 +02:00
Florian Klink
98d6b55fdc nixos/testing: remove remaining coverage-data logic
This isn't used anymore as per
https://github.com/NixOS/nixpkgs/pull/72354#discussion_r451031449.
2020-09-05 16:07:59 +02:00
Anders Kaseorg
59b6664f15 Revert "Merge pull request #96254 from Mic92/logging"
This reverts commit 4fc708567f, reversing
changes made to 0e54f3a6d8.

Fixes #96699.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-08-30 15:46:39 -07:00
Jörg Thalheim
392415c285
nixos/test-driver: switch to pythons' logging lib
- Less code
- more thread-safe according to @flokli
2020-08-25 10:13:27 +01:00
Alexandre Esteves
e10e7d6a8b
testing-python: fix typo 2020-06-30 22:31:32 -05:00
Eelco Dolstra
0ffc85d64b
Remove unused files 2020-05-14 14:34:50 +02:00
Eelco Dolstra
3f80fadec4 testing{-python}.nix: Remove log pretty-printing cruft
This completes the removal of the nested log feature, which previously
got removed from Nix, Hydra, stdenv and GNU Make. In particular, this
means that the output of VM builds no longer contains a copy of
jQuery.
2020-05-07 15:56:30 +02:00
Maximilian Bosch
6d14bac048
nixos/python-test-driver: allow single char as hostName and fix misleading error-message
In case of invalid chars, the error-message references "perl variables"
which is not the case here as the python-based framework is used.
2020-03-13 01:06:12 +01:00
worldofpeace
88f76812f2 testing-python: readd auto displayManager
we import it for the runInMachineWithX
2020-02-09 16:23:35 -05:00
worldofpeace
fa9af83e96 testing-python: fix runInMachine
The test script's were unported.
It's unclear whether the preBuild or
postBuild will work as expect, due to
the linting of the test scripts.
2020-02-09 16:21:52 -05:00
Erik Arvstedt
caa435fd1d test-driver.py: specify coreutils dependency
Otherwise the driver script fails when coreutils are not in PATH.
2020-01-14 15:52:44 -08:00
Maximilian Bosch
b72661749b
nixos/python-test-driver: add an option to disable python linter
While it's a good idea to automate the linting of the python code used
for our tests, I think that it can be quite distracting when hacking on
a NixOS test.

I figured that it might be more convenient to add an option as a
shortcut for this to avoid that everyone needs to dig into the test
driver again.
2019-12-22 19:27:43 +01:00
Jan Tojnar
8dc5ff7dcf
nixos/displayManager: deprecate separate options for default wm/dm
The upstream session files display managers use have no concept of sessions being composed from
desktop manager and window manager. To be able to set upstream session files as default
session, we need a single option. Having two different ways to set default session would be confusing,
though, so we decided to deprecate the old method.

We also created separate script for each session, just like we already had a separate desktop
file for each one, and started using displayManager.sessionPackages mechanism to make the
session handling more uniform.
2019-12-15 04:16:20 +01:00
worldofpeace
4583e293fd
Merge pull request #73251 from worldofpeace/remove-slim
nixos/slim: remove
2019-11-15 15:56:48 +00:00
worldofpeace
ce26b3eaf0 nixos/slim: remove
The SLIM project is abandoned and their last release was in 2013.
Because of this it poses a security risk to systems, no one is working
on it or picked up maintenance. It also lacks compatibility with systemd
and logind sessions. For users, there liikely isn't anything like slim
that's as lightweight in terms of dependencies.
2019-11-11 17:10:41 -05:00
Jörg Thalheim
03e6ca15e2
test-driver: add mypy support
It's a good idea to expand this in future to test code as well,
so we get type checking there as well.
2019-11-11 13:49:48 +00:00
Florian Klink
ac97edf013 nixos/test: use ptpython as repl 2019-11-04 23:50:27 +01:00
Jacek Galowicz
3a28fefe7d nixos/test: Port test driver to python
Thanks @blitz and @jtraue for help with implementing machine methods
2019-11-04 23:50:27 +01:00