Commit Graph

967 Commits

Author SHA1 Message Date
Robert Hensing
124b0c4abc nixos/testing/network.nix: Avoid deprecated .config 2022-09-21 10:55:11 +01:00
Robert Hensing
b3de22483c nixos/testing-python.nix: Add evalTest
This is a decomposition of the testing-python.nix and build-vms.nix
files into modules.

By refactoring the glue, we accomplish the following:

 - NixOS tests can now use `imports` and other module system features.
    - Network-wide test setup can now be reusable; example:
       - A setup with all VMs configured to use a DNS server
       - Split long, slow tests into multiple tests that import a
         common module that has most of the setup.
    - Type checking for the test arguments
    - (TBD) "generated" options reference docs
 - Aspects that had to be wired through all the glue are now in their
   own files.
    - Chief example: interactive.nix.
    - Also: network.nix

In rewriting this, I've generally stuck as close as possible to the
existing code; copying pieces of logic and rewiring them, without
changing the logic itself.

I've made two exceptions to this rule

 - Introduction of `extraDriverArgs` instead of hardcoded
   interactivity logic.

 - Incorporation of https://github.com/NixOS/nixpkgs/pull/144110
   in testScript.nix.

I might revert the latter and split it into a new commit.
2022-09-21 10:55:11 +01:00
Robert Hensing
5b52d552cb nixos/make-options-doc: Explain docbook to markdown migration 2022-09-17 13:38:23 +01:00
pennae
767485a0de lib/options: deprecate docbook text and literalDocBook
deprecate literalDocBook by adding a warning (that will not fire yet) to
its uses and other docbook literal strings by adding optional warning
message to mergeJSON.
2022-09-10 18:23:13 +02:00
Jacek Galowicz
8ae3e986a7
Merge pull request #189999 from sbruder/test-driver-formatting-closed-port
nixos/test-driver: fix formatting of closed port
2022-09-06 14:48:13 +02:00
Simon Bruder
9caf5f6ba9
nixos/test-driver: fix formatting of closed port
Co-authored-by: a-kenji <aks.kenji@protonmail.com>
2022-09-06 14:32:56 +02:00
Alyssa Ross
488412a1db
nixos/test-driver: add wait_for_unit() timeout arg
For example, the wait_for_unit() call in the Moodle test times out for
myself and others[1], so it would be good to be able to increase it to
something less likely to be hit by a test that would otherwise pass.

[1]: https://github.com/NixOS/nixpkgs/pull/177052#issue-1266336706
2022-09-02 18:06:03 +00:00
pennae
3bddcf5f90
Merge branch 'master' into option-docs-md 2022-09-01 16:10:09 +02:00
pennae
9547123258 nixos/*: convert internal option descriptions to MD
we'll have to do it eventually, may as well be now.
2022-08-31 16:32:54 +02:00
pennae
ef176dcf7e nixos/*: automatically convert option descriptions
conversions were done using https://github.com/pennae/nix-doc-munge
using (probably) rev f34e145 running

    nix-doc-munge nixos/**/*.nix
    nix-doc-munge --import nixos/**/*.nix

the tool ensures that only changes that could affect the generated
manual *but don't* are committed, other changes require manual review
and are discarded.
2022-08-31 16:32:53 +02:00
pennae
5a643387ec nixos/*: normalize more docbook for auto-conversion 2022-08-31 16:32:53 +02:00
pennae
9c3c13b50d nixos/make-options-doc: add inline roles for varname/envar
both of these render distinctly from plain literals in the manpage, and
manpages even semantically distinguish between the two.
2022-08-31 16:21:10 +02:00
Silvan Mosberger
6eb620ace7
Merge pull request #180222 from YorikSar/qemu-vm-darwin-pkgs
nixos/qemu-vm: Allow to build and run VMs on Darwin
2022-08-30 21:54:22 +02:00
pennae
65fd6f0774 nixos/make-options-doc: eat newlines in MD admonitions
leaving some newlines around after an admonition was closed causes the
newline rule to match, which in turn inserts literallayout newlines into
te xml output. that's not what we want.
2022-08-27 19:18:29 +02:00
Yuriy Taraday
e0fd131310 nixos/qemu: Add QEMU arguments for HVF on Darwin
Only support the same architechture for now, no emulation.

Related to #108984.
2022-08-25 17:00:14 +04:00
Yuriy Taraday
8a25d56a6d nixos/qemu: nixpkgs-fmt nixos/lib/qemu-common.nix 2022-08-25 17:00:11 +04:00
Ivan Shapovalov
ac18c55b56 make-system-tarball: use pixz -t
Pass `-t` to pixz to prevent it from appending an index to the end of
the uncompressed stream, confusing tools such as `machinectl import-tar`.

Fixes: #187816
2022-08-22 05:28:36 +03:00
Christian Kögler
5e7662d848
Merge pull request #167692 from peperunas/onsuccess
nixos/systemd: Add onSuccess module option for units
2022-08-19 09:17:14 +02:00
Robert Schütz
243053e521 python310Packages.mistune: 0.8.4 -> 2.0.4
Remove mistune_0_8 because it's insecure.
2022-08-15 06:53:01 +00:00
pennae
af98bacbe0 Revert "nixos/docs: cache mergeJSON md conversion on baseOptionsJSON"
This reverts commit 52b0ad17e3.

we only needed this because mergeJSON was slow, but in the interim we
found a better solution to the slowness.
2022-08-05 17:13:47 +02:00
pennae
645cfa59ac nixos/make-option-docs: add xref support to markdown conversion 2022-08-03 22:01:14 +02:00
pennae
7a091b2686 nixos/make-options-doc: reuse markdown instance in mergeJSON
this doesn't construct a new (expensive) parser for every option, making
rendering about 30x faster.
2022-08-03 22:00:24 +02:00
pennae
6b13dd0e9e
Merge pull request #183491 from pennae/automatic-md-conversions
treewide: automatically md-convert option descriptions
2022-08-02 02:15:30 +02:00
Silvan Mosberger
fe2c9ae814
Merge pull request #177012 from hercules-ci/nixos-modular-system
`flake.lib.nixosSystem`: Allow `nixpkgs.system` to be set modularly; improve error message
2022-08-01 15:05:16 +02:00
Alyssa Ross
5330c0a1af
treewide: use isAarch where appropriate 2022-07-30 16:18:27 +00:00
pennae
52b0ad17e3 nixos/docs: cache mergeJSON md conversion on baseOptionsJSON
with ever more options being markdown rather than docbook the conversion
time is starting to become a significant factor of doc build time.
luckily we can pre-convert all nixos option docs to MD and cache the
result of this conversion, then merge the already-converted json file
with user option docs. we leave options.json unconverted to keep it as
close to the actual nix code as possible.
2022-07-28 23:20:02 +02:00
pennae
18be724a58 nixos/make-options-doc: give MD conversion error locations
during docs conversion it can be very useful to know exactly *where* the
error the script complained about is. the name of the option should be
sufficient since option merging is rather rare, and won't merge doc
attributes anyway.
2022-07-28 20:03:02 +02:00
Philippe Schaaf
5ae6580474 add hub mode comment
Signed-off-by: Philippe Schaaf <philippe.schaaf@secunet.com>
2022-07-21 16:34:09 +02:00
Philippe Schaaf
f6a290932e use vde switch in hubmode by default
Within a dual VM test-setup a strange behaviour was observed.
The two VMs are connected via one vde_switch instance
(instancevirtualisation.vlans = [ 1 ]; IMO a bad attribute name for
switch instances, has nothing to do with VLANs in sense of 802.1Q).

A ping on the base interface (eth1) works, but not on VLAN
subinterfaces (vlan1@eth1). A tcpdump of eth1 includes the ARP requests
tagged with the subinterfaces VLAN ID, but responses seems not to pass
the vde_switch. This works fine if performed on the base interface.

Putting the vde_switch in hub mode results in flooding
traffic to all vde_switch ports. This results in a expected behaviour
and a ping on a VLAN subinterface works as expected.

Signed-off-by: Philippe Schaaf <philippe.schaaf@secunet.com>
2022-07-21 11:38:03 +02:00
Robert Hensing
711e653a65 nixos/eval-config: Allow system to be set modularly when arg is null 2022-07-10 13:35:54 +02:00
Martin Weinelt
c0d1de14b3
Merge pull request #179291 from dotlambda/utillinux-alias
treewide: don't use utillinux alias
2022-07-02 00:05:54 +02:00
Robert Hensing
7c81905344 nixos/make-options-doc: Support Nix-provided declaration locations
Feature was introduced in https://github.com/NixOS/nixpkgs/pull/174460,
but wasn't supported in `mergeJSON.py` yet.
2022-06-27 22:07:20 +02:00
Robert Hensing
8bff3fef40 nixos/make-options-doc: Support block quotes
Our tooling would trip without the inner <para>, despite the docbook
docs suggesting that <para> occurs in <blockquote> and vice versa.
2022-06-27 17:41:32 +02:00
Robert Hensing
e04aa1bcd9 nixos/make-options-doc: Escape inline code and code blocks 2022-06-27 17:41:32 +02:00
Robert Hensing
f900ed1749 nixos/make-options-doc: Support newline md node
This occurs in the ast generated for blockquotes.
2022-06-27 17:41:32 +02:00
Robert Hensing
bccc3e747b nixos/make-options-doc: Fix exception handler for arity /= 1 methods 2022-06-27 17:41:32 +02:00
Robert Hensing
aff2dbbc82 make-options-doc: Make variablelist id configurable
I've tried XInclude set-xml-id first, but our tooling did not pick up on it.
2022-06-27 17:41:32 +02:00
Robert Schütz
40990c3586 treewide: don't use utillinux alias 2022-06-27 04:55:23 +00:00
Robert Hensing
fa9c83ca7f
Merge pull request #174460 from hercules-ci/module-docs-Nix-driven-location-links
make-options-doc: Support Nix-provided declaration links
2022-06-22 15:48:46 +02:00
Lassulus
773cbfbd28
Merge pull request #151633 from wucke13/sd-image-experiment
honor `sdImage.compressImage` also in the build process
2022-06-22 14:48:21 +02:00
Robert Hensing
e2c261f2c0
Merge pull request #176146 from pennae/module-docs-markdown
treewide: markdown option docs
2022-06-21 13:16:02 +02:00
Jörg Thalheim
f5ca531054
Merge pull request #158602 from hercules-ci/systemd-unit-mixed-list-error
nixos/systemd: Custom error when mixing list/non-list defs
2022-06-19 07:05:04 +01:00
Robert Hensing
2c8bbf33fd nixos/test-driver: Support mypy through regular mechanisms 2022-06-17 12:22:57 +02:00
Robert Hensing
cee66a8cd5 make-options-doc: Support Nix-provided declaration links
Previously, the location logic was hardcoded, supporting only
Nixpkgs and NixOps properly, leaving other uses of the module
system without good location support.
2022-06-15 00:45:05 +02:00
pennae
320aa2a791 treewide: attempt at markdown option docs 2022-06-12 12:44:38 +02:00
Robert Hensing
9970c0b874
Merge pull request #174470 from hercules-ci/module-docs-visibility-override
make-options-doc: Allow transformOptions to override visibility
2022-06-10 14:27:58 +02:00
Robert Hensing
bad676c7ed
Merge pull request #174441 from tljuniper/172325-nixostest-override-python-pkgs-additional-param
nixos/test-driver: additional Python packages in test driver
2022-06-05 20:46:01 +02:00
Bobby Rong
9d2a8e3190
nixos/test-driver: fix type hint for send_chars 2022-06-04 12:05:47 +08:00
Robert Hensing
793180cf55
Merge branch 'master' into 172325-nixostest-override-python-pkgs-additional-param 2022-06-03 14:22:13 +02:00
Michael Schneider
fd4ebd8990 add opt out flag for type checking 2022-06-03 13:37:04 +02:00
Michael Schneider
814027378b nixos/test-driver: Typecheck TestScript 2022-06-02 10:05:45 +02:00
Robert Hensing
53326189ba
Merge pull request #174829 from blaggacao/fix/cleanup-nixpkgs-initial-system
nixos: remove effect-less nixpgks.initialSystem
2022-05-31 16:26:25 +02:00
tljuniper
a99736e399 nixos/test-driver: add option to add extra python packages to test code 2022-05-29 11:22:23 +02:00
Alyssa Ross
1e396f9802
nixos/testing-python.nix: fix typo in warning
"pyton" -> "python"
2022-05-27 13:00:25 +00:00
David Arnold
646e214e11
nixos: remove effect-less nixpgks.initialSystem
- initialSystem was keeping track of the evaluating system
- it had been used by `nesting.children`
- since, 20.09, `nesting.children` has been replaced with named
  specializations

It appears that this option was left over and not cleand up properly.
2022-05-26 20:00:05 -05:00
Robert Hensing
75bc6da237 make-options-doc: Filter options after transformOptions
This allows the user-supplied function to change the visibility
of options.
2022-05-25 12:50:07 +02:00
Robert Hensing
90344e9f40
Merge pull request #171163 from hercules-ci/nixpkgs-config-doc
Add generated `nixpkgs.config` doc to Nixpkgs manual
2022-05-19 20:51:38 +02:00
Robert Hensing
0b02135d3b nixosOptionsDoc: refactor
Thanks to Infinisil for pointing this out.
2022-05-16 22:53:23 +02:00
Robert Hensing
062bc5e74a lib.types.functionTo: Add pseudo-attr to generated docs 2022-05-13 09:01:05 +02:00
Robert Hensing
ae172a2bb4 treewide: nixosTest -> testers.nixosTest 2022-05-09 14:16:06 +02:00
Robert Hensing
28f99aad31 nixos/testing-python.nix: Set meta.mainProgram 2022-05-05 12:33:57 +02:00
Robert Hensing
bb2c5a3684 nixosOptionsDoc: Make appendix tag optional 2022-05-01 21:51:19 +02:00
Patrick Hilhorst
976f00963f
nixos/test-driver: rewrite wait_before_entry to wait 2022-04-29 15:36:03 +02:00
Will Fancher
69d8047516 nixos: Fix up systemd shutdown ramfs 2022-04-28 14:12:14 -04:00
Patrick Hilhorst
1db0dcdfec
nixos/test-driver: add wait_before_entry to polling_condition 2022-04-27 16:23:33 +02:00
Patrick Hilhorst
57beeedbdf
nixos/test-driver: allow multiple entry of polling condition 2022-04-27 16:21:10 +02:00
Anna Gillert
dbc95f15b8 nixos/test-driver: Avoid shell injection in machine.execute() 2022-04-21 10:54:00 +02:00
Anna Gillert
f7e89a59da nixos/test-driver: fix missing shellopts in execute
Without this fix, setting the shellopts in `machine.execute` is
inconsitent.  When no timeout is used, shellopts `set -euo pipefail` are
applied to the command as expected. When a timeout is specified, the
shellopts are not applied to the command itself (which is called inside
a `sh -c` that doesn't inherit the shellopts) but rather to the
`timeout` command, leading to the following full command:

```bash
(set -euo pipefail; timeout 900 sh -c 'cmd') | (base64 --wrap 0; echo)\n
```

With this fix, this is the command we get:

```bash
timeout 900 sh -c 'set -euo pipefail; false | true') | (base64 --wrap 0; echo)\n
```
2022-04-21 10:54:00 +02:00
Graham Christensen
d65eff4fb6
Merge pull request #167902 from DeterminateSystems/openstack-zfs-cleanup
openstack-image: init, make-single-disk-zfs-image: init
2022-04-16 19:44:53 -04:00
Jacek Galowicz
6b414c5b66
Merge pull request #168609 from rnhmjoj/pr-test-driver
nixos/test-driver: highlight driver log lines
2022-04-14 13:09:36 +02:00
rnhmjoj
ca1b1f6dc0
nixos/test-driver: highlight driver log lines
There is a whole lot of noise in a NixOS test log due to journal,
commands, and test driver messages all being mixing together.
With this commit the test driver messages are highlighted so you don't
have to squint too much to see where a subtest starts and ends or what
was the last command being run.

Here's an excerpt from a highlighted log:

    machine # [    6.647826] dhcpcd[668]: eth0: adding default route via 10.0.2.2
    machine # I've had a great day!
    (finished: must succeed: curl -sfL https://example.com/how-are-you? | tee /dev/stderr, in 0.19 seconds)
    (finished: subtest: Privoxy can filter https requests, in 0.22 seconds)
    subtest: Blocks are working
    machine: waiting for TCP port 443
    machine # Connection to localhost (::1) 443 port [tcp/https] succeeded!
    (finished: waiting for TCP port 443, in 0.03 seconds)
    machine: must fail: curl -f https://example.com/ads 1>&2
    machine # [    6.730608] nscd[742]: 742 monitored file `/etc/resolv.conf` was written to
    machine # [    6.733071] privoxy[814]: 2022-04-14 09:41:01.695 7fa995c39640 Actions: +block{Fake ads} ...
    machine # curl: (22) The requested URL returned error: 403
    machine # [    6.753098] systemd[1]: Stopping Name Service Cache Daemon...
    machine # [    6.755632] systemd[1]: nscd.service: Deactivated successfully.
    machine # [    6.757069] systemd[1]: Stopped Name Service Cache Daemon.
    (finished: must fail: curl -f https://example.com/ads 1>&2, in 0.07 seconds)
    machine: must succeed: curl -f https://example.com/PRIVOXY-FORCE/ads 1>&2
    machine # [    6.765099] systemd[1]: Starting Name Service Cache Daemon...
    machine # [    6.781519] nscd[925]: 925 monitoring file `/etc/passwd` (1)
    machine # [    6.844105] systemd[1]: Reached target Network is Online.
    machine # [    6.848860] privoxy[814]: 2022-04-14 09:41:01.805 7fa995c39640 Actions: +block{Fake ads} ...
    machine # [    6.856068] nscd[925]: 925 monitoring file `/etc/nsswitch.conf` (8)
    machine # [    6.858229] systemd[1]: Reached target Multi-User System.
    machine # [    6.858934] nscd[925]: 925 monitoring directory `/etc` (2)
    machine # [    6.861315] systemd[1]: Startup finished in 2.374s (kernel) + 4.451s (userspace) = 6.826s.
    machine # Hot Nixpkgs PRs in your area. Click here!
    machine # [    6.873055] nscd[925]: 925 monitoring directory `/etc` (2)
    machine # [    6.874489] nscd[925]: 925 monitoring file `/etc/nsswitch.conf` (8)
    machine # [    6.875599] nscd[925]: 925 monitoring directory `/etc` (2)
    (finished: must succeed: curl -f https://example.com/PRIVOXY-FORCE/ads 1>&2, in 0.12 seconds)
    (finished: subtest: Blocks are working, in 0.22 seconds)
    subtest: Temporary certificates are cleaned
    machine: must succeed: test $(ls /run/privoxy/certs | wc -l) -gt 0
    (finished: must succeed: test $(ls /run/privoxy/certs | wc -l) -gt 0, in 0.02 seconds)
    machine: must succeed: date -s "$(date --date '12 days')"
    (finished: must succeed: date -s "$(date --date '12 days')", in 0.02 seconds)
    machine # [    6.959589] systemd[1]: Started Logrotate Service.
    machine # [    6.966685] systemd[1]: Starting Cleanup of Temporary Directories...
    machine # [    6.974783] systemd[1]: logrotate.service: Deactivated successfully.
    machine # [    7.004493] systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
    machine # [    7.005457] systemd[1]: Finished Cleanup of Temporary Directories.
    machine: must succeed: test $(ls /run/privoxy/certs | wc -l) -eq 0
    (finished: must succeed: test $(ls /run/privoxy/certs | wc -l) -eq 0, in 0.01 seconds)
    (finished: subtest: Temporary certificates are cleaned, in 0.15 seconds)
    (finished: run the VM test script, in 7.44 seconds)
    test script finished in 7.48s
    cleanup
    kill machine (pid 8)
2022-04-14 11:54:21 +02:00
Sandro
a39694b41f
Merge pull request #168194 from SuperSandro2000/xserver-excludePackages
Xserver exclude packages
2022-04-13 11:37:32 +02:00
Sandro Jäckel
28539842d8
nixos/utils: move removePackagesByName to here from gnome 2022-04-11 15:42:49 +02:00
Lassulus
f0848eeb05
Merge pull request #167316 from helsinki-systems/feat/opportunistic-kvm
nixos/lib/qemu-common: Opportunistically use KVM
2022-04-10 18:54:59 +01:00
Graham Christensen
8a5bdce566 make-single-disk-zfs-image: init 2022-04-07 15:28:28 -04:00
Graham Christensen
a8f41adbb7 amazon-image: use make-multi-disk-zfs-image 2022-04-07 15:28:28 -04:00
Giulio De Pasquale
fbc8dbb075 nixos/systemd: Add onSuccess module option for units 2022-04-07 13:25:52 +02:00
Robert Hensing
c705953267
Merge pull request #165540 from Infinisil/module-args-docs
lib/modules: Document `_module.args`
2022-04-05 21:51:46 +02:00
Silvan Mosberger
25de2935ef lib/modules: Document _module.args
Documents the _module.args option, motivated by many usages in Flakes,
especially with the deprecation of extraArgs
(78ada83361)

The documentation rendering for this option had to be handled a bit
specially, since it's not declared in nixos/modules like all the other
NixOS options.

Co-Authored-By: pennae <github@quasiparticle.net>
Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
2022-04-05 18:26:40 +02:00
lassulus
ed18db011f make-disk-image: add systemdMinimal to PATH
this is needed to setup /tmp inside the chroot. which is needed by
systemd-boot to sucessfully install
2022-04-05 11:57:53 +02:00
Max Hausch
a2465483e8
nixos/lib/qemu-common: Opportunistically use KVM
This will try to use KVM if available and fallback to using TCG if not.
Useful for nested virtualisation of arm on x86.
2022-04-05 11:49:42 +02:00
Janne Heß
beb02229fb
nixos/systemd-unit-options: Fix indentation 2022-04-04 13:53:58 +01:00
Janne Heß
4b9efea255
nixos/stage-1-systemd: Implement job scripts 2022-04-04 11:44:31 +01:00
Janne Heß
7cdc4dd5d1
Merge pull request #164943 from ElvishJerricco/systemd-initrd-reuse-systemd-module
initrd: Opt-in bare bones systemd-based initrd
2022-04-03 15:53:02 +02:00
Janne Heß
b7c62b8238
nixos/systemd-initrd: Remove unit options that don't work 2022-04-01 11:58:30 +02:00
Janne Heß
1e5261f31c
nixos/systemd-lib: Use module composition 2022-04-01 09:57:19 +02:00
Robert Hensing
ecbf5ae27a nixosTest: Simplify doc by deprecating syntax sugar 2022-03-28 14:11:58 +02:00
Silvan Mosberger
5a67e9db61
Merge pull request #133532 from Infinisil/systemd-unit-dirs 2022-03-25 19:05:35 +01:00
Silvan Mosberger
c70a466d21 nixos/systemd: Allow creation of unit directories
This patch allows creation of files like
/etc/systemd/system/user-.slice.d/limits.conf with

    systemd.units."user-.slice.d/limits.conf" = {
      text = ''
        [Slice]
        CPUAccounting=yes
        CPUQuota=50%
      '';
    };

which previously threw an error

Also renames the systemd-unit-path test to sytsemd-misc, and extends it to
test that `systemd.units` can handle directories. In this case we make
sure that resource limits specified in user slices apply.
2022-03-25 17:47:44 +01:00
Janne Heß
7d0e21c77e
nixos/test-runner: Allow writing to qemu stdin 2022-03-25 12:08:21 +01:00
Florian Klink
fc91cdb5bc nixos/lib/systemd-lib.nix: move comment back down to packages
https://github.com/NixOS/nixpkgs/pull/164943#discussion_r833215903
2022-03-24 19:03:09 +01:00
Florian Klink
e3083decc4 systemd-initrd, systemd-lib: drop initrdServiceToUnit
We can just use serviceToUnit here.
2022-03-24 18:57:16 +01:00
Robert Hensing
ce5a33e62b
Merge pull request #164660 from ncfavier/tests-restrict-arguments
nixos/testing: restrict arguments to makeTest
2022-03-24 17:01:47 +01:00
Will Fancher
9828446608 systemd-initrd: Fix Environment= and PATH 2022-03-22 21:28:51 -04:00
Will Fancher
2d4ebf1259 initrd: Optional systemd-based initrd 2022-03-22 21:28:43 -04:00
Viktor Kleen
6d766ae8b7
nixos/test-driver: deduplicate VLANs
Naively deduplicate VLANs in the python driver for NixOS tests. The
current implementation accidentally works, since the VLan class mutates
the environment. On construction it sets QEMU_VDE_SOCKET_${id} and this
environment variable gets overwritten once a second VLAN with the same
id is constructed. Because the NIC flags passed to qemu just use the
QEMU_VDE_SOCKET_${id} environment variable, this implicitly chooses a
single vde_switch process for each VLAN.

However, this leads to unusable vde_switch processes being spawned in
each test run and as a side effect makes it impossible to access the
correct VLan objects in the interactive test driver. It also makes it
remarkably hard to understand why the current implementation ever
worked.
2022-03-22 13:16:28 +00:00
Will Fancher
25113740a5 nixos: systemd-lib: Make generateUnits general with default args 2022-03-22 07:02:23 -04:00
Robert Hensing
6c469679f6 Merge remote-tracking branch 'upstream/master' into tests-restrict-arguments 2022-03-21 23:17:17 +01:00
Will Fancher
52c98fc3e9 nixos: systemd: Split unit types into separate module 2022-03-21 10:25:27 -04:00
Naïm Favier
8f57dc38d9
fixup! nixos/testing: restrict arguments to makeTest 2022-03-19 11:51:39 +01:00
Naïm Favier
79a234567c
nixos/testing: restrict arguments to makeTest
Disallow passing arbitrary arguments to makeTest since they are not
used; this can help catch mistakes.
2022-03-18 02:44:11 +01:00
Robert Hensing
0395086d0c
Merge pull request #162271 from Infinisil/warn-no-type
Throw an error for options without a type
2022-03-16 22:58:45 +01:00
Bob van der Linden
9bc093b30a
nixos: systemd: split off helper functions into systemd-lib 2022-03-15 21:52:13 +01:00
pennae
aa7b129708
Merge pull request #154113 from pennae/systemd-escaping
nixos: add functions and documentation for escaping systemd Exec* directives
2022-03-13 19:57:32 +00:00
pennae
40a35299fa nixos: add functions and documentation for escaping systemd Exec* directives
it's really easy to accidentally write the wrong systemd Exec* directive, ones
that works most of the time but fails when users include systemd metacharacters
in arguments that are interpolated into an Exec* directive. add a few functions
analogous to escapeShellArg{,s} and some documentation on how and when to use them.
2022-03-12 00:48:52 +01:00
Janne Heß
0c766a100e lib/options: Throw error for options without a type
Makes all options rendered in the manual throw an error if they don't
have a type specified.

This is a follow-up to #76184

Co-Authored-By: Silvan Mosberger <contact@infinisil.com>
2022-02-28 22:51:41 +01:00
talyz
b549e7bbb5
genJqSecretsReplacementSnippet: Fix error handling
With the previous change that enabled error propagation through
`inherit_errexit`, the script would fail if `errexit` was set, but
`inherit_errexit` was not. This is due to `shopt -p` exiting with an
error if the option is disabled. To work around this, use the exit
code instead of the text value returned by `shopt -p`.

Fixes #160869.
2022-02-23 12:48:51 +01:00
Daniel Fullmer
24fb872079 nixos/test-driver: fix regression when using legacy start command
The aarch64-linux versions of the boot.uefiUsb and boot.uefiCdrom tests
were broken by b0fc9da879.
That commit was a refactor which omitted the qemuBinary option, which was
previously available in the legacy start command. This restores that
option and fixes the tests previously mentioned.
2022-02-17 18:12:24 -08:00
Janne Heß
fa3c756621
Merge pull request #157329 from helsinki-systems/feat/nixos-reload-triggers
nixos/systemd: Implement reload triggers
2022-02-11 23:59:26 +01:00
Janne Heß
b5b3ee4f78
nixos/systemd: Add reloadTriggers to services 2022-02-09 15:14:37 +01:00
Robert Hensing
f86de05ca6 nixos/systemd: Custom error when mixing list/non-list defs 2022-02-08 12:50:06 +01:00
Jonathan Ringer
46fd0afcb3
Merge remote-tracking branch 'origin/master' into staging-next
Conflicts:
	pkgs/development/python-modules/pysdl2/default.nix
	pkgs/top-level/aliases.nix
2022-02-06 18:36:59 -08:00
Robert Hensing
8403e02a5d
Merge pull request #126769 from ncfavier/nixosSystem-lib
nixos: move default module location logic to `eval-config.nix`
2022-02-06 22:49:17 +01:00
Jonathan Ringer
c6c7162876
Merge remote-tracking branch 'origin/staging-next' into staging 2022-02-03 21:34:17 -08:00
Jonathan Ringer
9a16965c75
Merge remote-tracking branch 'origin/staging-next' into staging
Conflicts:
	pkgs/development/python-modules/awesomeversion/default.nix
	pkgs/development/python-modules/md2gemini/default.nix
	pkgs/development/python-modules/mill-local/default.nix
	pkgs/development/python-modules/pyatv/default.nix
	pkgs/development/python-modules/pyezviz/default.nix
	pkgs/development/python-modules/pyinsteon/default.nix
	pkgs/development/python-modules/pylitterbot/default.nix
	pkgs/development/python-modules/pynuki/default.nix
	pkgs/development/python-modules/pypck/default.nix
	pkgs/development/python-modules/types-requests/default.nix
	pkgs/top-level/aliases.nix
2022-02-03 21:29:24 -08:00
Kim Lindberger
415b9c3b5a
Merge pull request #157519 from talyz/gitlab-error-handling
nixos/gitlab: Implement better script error handling
2022-02-03 22:19:03 +01:00
Lassulus
b80607af67
Merge pull request #154804 from Mic92/tests-prompt
nixos/tests: add prompt to shell_interact()
2022-02-02 18:50:46 +01:00
github-actions[bot]
aee8ca6639
Merge staging-next into staging 2022-02-01 00:02:21 +00:00
Guillaume Girol
d6f45aa6c3
Merge pull request #152372 from symphorien/systemd-validate-after
nixos/systemd: validate the values of systemd.services.<name>.after
2022-01-31 20:57:13 +00:00
talyz
95069d76d3
genJqSecretsReplacementSnippet: Propagate secret file read errors
If an error occurs while trying to read a secret file, we want that
error to propagate to the main shell context. That means we have to
set the `inherit_errexit` option, which allows errors from subshells
to propagate to the outer shell. Also, the subshell cannot run as part
of another command, such as `export`, since that will simply ignore
the subshell exit status and only respect `export`s exit status; first
assigning the value to a variable and then exporting it solves issue.
2022-01-31 12:44:54 +01:00
Jan Tojnar
2da5ce4ab3 Merge branch 'staging-next' into staging
; Conflicts:
;	pkgs/development/python-modules/fakeredis/default.nix
2022-01-28 14:05:11 +01:00
Naïm Favier
5ab62e17b5
nixos: move default module location logic to eval-config.nix 2022-01-27 22:13:35 +01:00
Daniel Olsen
68e9cd0f7e nixos/lib: Use SingleLineStr in systemd description 2022-01-27 12:56:36 -08:00
Dmitry Kalinkin
0693fd77f7
Merge branch 'staging-next' into staging
Conflicts:
	nixos/doc/manual/from_md/release-notes/rl-2205.section.xml
	nixos/doc/manual/release-notes/rl-2205.section.md
	pkgs/development/python-modules/aioesphomeapi/default.nix
	pkgs/development/python-modules/mat2/default.nix
	pkgs/development/python-modules/pydevccu/default.nix
	pkgs/development/python-modules/pywlroots/default.nix
	pkgs/development/python-modules/rokuecp/default.nix
2022-01-27 00:54:10 -05:00
Patrick Hilhorst
23c87e7991
nixos/testing-python: actually copy build artefacts
This broke in #156858.
2022-01-26 22:26:19 +01:00
Marijan Petričević
411bb3c35f add --no-interactive flag 2022-01-26 11:27:01 +01:00
Marijan Petričević
44830dc048 use lib.optionalString 2022-01-26 10:46:46 +01:00
Marijan Petričević
75de3397fd nixos/tests: fix #146169 2022-01-26 10:46:45 +01:00
Marijan Petričević
446c21fdc7 factor out tmp-dir checks 2022-01-25 11:31:25 +01:00
Marijan Petričević
03c90974a7 add tmp_dir access check 2022-01-25 11:14:57 +01:00
Marijan Petričević
e17fcbc966 introduce writeable_dir argparse type 2022-01-25 10:21:33 +01:00
Marijan Petričević
7765670c8a make output_directory an optional parameter 2022-01-25 01:18:52 +01:00
Marijan Petričević
f9b5f9dba7 nixos/test-driver: use an argument instead of the out env-var 2022-01-25 01:17:29 +01:00
Alyssa Ross
1a60dd2adc
Merge remote-tracking branch 'nixpkgs/staging-next' into staging
Conflicts:
	pkgs/development/python-modules/cupy/default.nix
	pkgs/development/python-modules/staticjinja/default.nix
2022-01-24 15:39:34 +00:00
Samuel Dionne-Riel
18ebd1c330
Merge pull request #153409 from tpwrules/fix-nonx86-iso-reproducibility
installer/cd-dvd/iso-image: avoid leaking build timestamps on non-x86
2022-01-23 21:15:26 -05:00
github-actions[bot]
aeb6db8171
Merge staging-next into staging 2022-01-20 18:01:55 +00:00
Alyssa Ross
464d6108b0 Revert "qemu, runInLinuxVM: change default cpu to qemu64"
This reverts commit af180d554b.

The bug we were working around with this change has been fixed in QEMU
6.2.0.
2022-01-20 15:29:33 +00:00
Robert Hensing
98ae5a9298
Merge pull request #153211 from hercules-ci/minimal-nixos
Add minimal NixOS entrypoint
2022-01-20 14:42:46 +01:00
Jörg Thalheim
f472e52aa6 nixos/tests: add prompt to shell_interact()
Example session:

>>> start_all()
>>> machines[0].shell_interact()
client: waiting for the VM to finish booting
client: connected to guest root shell
client: (connecting took 0.00 seconds)
(finished: waiting for the VM to finish booting, in 0.05 seconds)
client: Terminal is ready (there is no initial prompt):
$ ls -la
total 39
drwxrwxrwt 11 root root 4096 Jan 15 06:06 .
drwxr-xr-x 17 root root 4096 Jan 15 06:06 ..
drwxrwxrwt  2 root root 4096 Jan 15 06:06 .font-unix
drwxrwxrwt  2 root root 4096 Jan 15 06:06 .ICE-unix
drwx------  2 root root    2 Jan 12 20:19 shared
drwx------  3 root root 4096 Jan 15 06:06 systemd-private-b8f21699ea684491926859758de41975-nscd.service-txgYVZ
drwx------  3 root root 4096 Jan 15 06:06 systemd-private-b8f21699ea684491926859758de41975-systemd-logind.service-U3GmlL
drwxrwxrwt  2 root root 4096 Jan 15 06:06 .Test-unix
drwxrwxrwt  2 root root 4096 Jan 15 06:06 .X11-unix
drwxr-xr-x  2 root root    2 Jan 15 06:06 xchg
drwxrwxrwt  2 root root 4096 Jan 15 06:06 .XIM-unix
2022-01-15 07:07:06 +01:00
Robert Hensing
2bf5958169
Merge pull request #151082 from hercules-ci/nixos-cleanup-vmWithBootLoader
nixos: turn vmWithBootLoader into option (`nixos-rebuild build-vm`)
2022-01-14 18:49:27 +01:00
Robert Hensing
3168017b90 nixos/lib: Clarify that nixos.evalModules impl is NOT experimental 2022-01-10 12:36:57 +01:00
Robert Hensing
d3f956aba3 nixos/lib: Add featureFlags, use it for minimal modules 2022-01-10 12:36:57 +01:00
Robert Hensing
25caf736d5 nixos/eval-config: Layer on top of nixos/eval-config-minimal 2022-01-10 12:26:59 +01:00
Robert Hensing
e31e096b66 nixos/lib: Move evalModules into its own file 2022-01-10 12:26:59 +01:00
Zhaofeng Li
9e97ef794a lib/qemu-common: Add serial device name for RISC-V 2022-01-09 15:29:52 -08:00
github-actions[bot]
b40a01817b
Merge master into staging-next 2022-01-08 18:01:00 +00:00
Jacek Galowicz
048fd95f10
Merge pull request #146905 from Synthetica9/failure_mode
nixos/test-driver: add polling_condition
2022-01-08 18:28:25 +01:00
Jonathan Ringer
86f9a32c4c
Merge remote-tracking branch 'origin/master' into staging-next
Conflicts:
	pkgs/development/python-modules/hydra/default.nix
2022-01-07 10:31:29 -08:00
Patrick Hilhorst
0c3f1cf420
Merge pull request #153273 from Synthetica9/passthru-driverInteractive 2022-01-07 16:04:43 +01:00
Robert Hensing
fbd038eca2 nixos/lib: init (experimental) 2022-01-07 01:21:07 +01:00
Colin L Rice
fee76f189c lib: Create make-btrfs-fs library to match make-ext4-fs.
I wrote this up when I was playing with a btrfs sd card, and
figured that it would be useful for others.
2022-01-05 22:44:22 -05:00
Thomas Watson
450ce00ec4 installer/cd-dvd/iso-image: avoid leaking build timestamps on non-x86 2022-01-05 12:05:15 -06:00
github-actions[bot]
89a0a771b6
Merge master into staging-next 2022-01-05 00:01:52 +00:00
Patrick Hilhorst
d4dc638d77
nixos/test-driver: also passthru driverInteractive 2022-01-04 14:52:37 +01:00
Patrick Hilhorst
793a2f50f1
nixos/test-driver: remove unused imports, add pylint unused-import check 2022-01-02 23:12:21 +01:00
Patrick Hilhorst
a2f5092867
nixos/test-driver: simplify logic, reduce interaction surface 2022-01-02 22:52:17 +01:00
Patrick Hilhorst
7830f000c5
nixos/test-driver: simplify coopmulti 2022-01-02 22:20:04 +01:00
pennae
50954ad1c5 nixos/make-options-doc: treat missing descriptions as errors by default
this partially solves the problem of "missing description" warnings of the
options doc build being lost by nix build, at the cost of failing builds that
previously ran. an option to disable this behaviour is provided.
2022-01-02 19:46:13 +01:00
pennae
1301bdb185 nixos/make-options-doc: turn relatedPackages into links
link to search.nixos.org instead of pulling package metadata out of pkgs. this
lets us cache docs of a few more modules and provides easier access to package
info from the HTML manual, but makes the manpage slightly less useful since
package description are no longer rendered.
2022-01-02 19:46:13 +01:00
pennae
b92a47c87c nixos/make-options-doc: add type annotations to mergeJSON.py 2022-01-02 19:46:13 +01:00
pennae
fc614c37c6 nixos/documentation: split options doc build
most modules can be evaluated for their documentation in a very
restricted environment that doesn't include all of nixpkgs. this
evaluation can then be cached and reused for subsequent builds, merging
only documentation that has changed into the cached set. since nixos
ships with a large number of modules of which only a few are used in any
given config this can save evaluation a huge percentage of nixos
options available in any given config.

in tests of this caching, despite having to copy most of nixos/, saves
about 80% of the time needed to build the system manual, or about two
second on the machine used for testing. build time for a full system
config shrank from 9.4s to 7.4s, while turning documentation off
entirely shortened the build to 7.1s.
2022-01-02 19:46:13 +01:00
Patrick Hilhorst
ac6c06c549
nixos/test-driver: bump version 2022-01-01 23:17:37 +01:00
Patrick Hilhorst
4e1556ed4d
nixos/test-driver: add polling_condition 2022-01-01 23:17:32 +01:00
github-actions[bot]
129083edcf
Merge master into staging-next 2021-12-29 06:01:11 +00:00
Julien Moutinho
8b842173d0 nixos/make-options-doc: fix invalid ':' in XML NCName (non-colonized name) 2021-12-28 22:18:16 -05:00
Guillaume Girol
2a37dd8ab3 nixos/systemd: validate the values of systemd.services.<name>.after
and similar option. Notably check that they end with one of .service,
.target, etc.
2021-12-27 12:00:00 +00:00
wucke13
121e2f7e15 honor sdImage.compressImage in intermediate build steps 2021-12-21 21:31:44 +01:00
Robert Hensing
9fd9c617a9 nixos/lib/eval-config.nix: Return all of evalModules return attrs
We were exposing everything pointwise anyway.
If any new attrs are added, there's a good chance we'll want to
expose them anyway.
2021-12-17 14:07:47 +01:00
Robert Hensing
8fd49c116b nixos/default.nix: Use extendModules 2021-12-17 13:17:37 +01:00
github-actions[bot]
a292b5fe68
Merge staging-next into staging 2021-12-14 12:02:00 +00:00
pennae
a70b1eb630 nixos/lib/make-options-doc: fix with nix 2.3 2021-12-14 03:41:09 +01:00
Dmitry Kalinkin
07a8ae0c5a
Merge branch 'staging-next' into staging
Conflicts:
	pkgs/development/libraries/log4cplus/default.nix
2021-12-11 14:01:19 -05:00
Jacek Galowicz
b6bf1ca717
Merge pull request #149329 from marijanp/test-driver-restructuring
nixos/test-driver: make the test-driver a python package
2021-12-10 18:41:49 +00:00
Marijan Petričević
ab693de868 nixos/test-driver: make the test-driver a python package 2021-12-10 12:27:45 +00:00
Silvan Mosberger
2f2b60dd87 lib/nixos/eval-config.nix: Fix extraArgs
Fixes a mistake in https://github.com/NixOS/nixpkgs/pull/148315 that
caused https://github.com/NixOS/nixpkgs/issues/148343#issuecomment-990881216
2021-12-10 12:55:30 +01:00
Silvan Mosberger
15c41e1d54
Merge pull request #147265 from pennae/option-docs-build
put all option docs build flavors on equal footing
2021-12-08 21:27:02 +01:00
Artturi
3fe92b45cb
Merge pull request #149665 from Artturin/squashfscores 2021-12-08 20:20:32 +02:00
Artturin
eea6baad50 make-squashfs: use $NIX_BUILD_CORES or 48 cores if above 48
by default all cores are used

hoping this will fix the hydra i686 squashfs build issues as all the
failures were using 64 cores

Parallel mksquashfs: Using 64 processors
Creating 4.0 filesystem on ..., block size 1048576.
FATAL ERROR: mangle2:: xz compress failed with error code 5
2021-12-08 19:23:31 +02:00
ajs124
eee45bb295
Merge pull request #146815 from ElvishJerricco/systemd-utils-expressions
Move systemd-lib.nix and systemd-unit-options.nix into utils
2021-12-08 15:07:28 +00:00
Silvan Mosberger
490d46f044
Merge pull request #148315 from hercules-ci/nixos-evalModules-legacy-cleanup
NixOS/evalModules legacy cleanup
2021-12-07 18:30:52 +01:00
pennae
027f7e1b7f nixos/lib/make-options-doc: generate options.xml from options.json
to do this we must replace derivations with attrsets in make-options-doc, since
xml can represent derivations differently from attrset but json cannot. this
also given asciidoc and mddoc the ability to handle derivation differently,
which they previously didn't have.
2021-12-06 16:12:32 +01:00
pennae
9b97a2ea88 nix/lib/make-options-doc: remove nix-level sorting
there are no remaining users of sorted option lists except the docbook build,
which sorts its input separately.
2021-12-06 16:12:32 +01:00
pennae
4670400309 nixos/lib/make-options-doc: generate asciidoc/md in derivations
use the json file derivation we already have to also generate the asciidoc and
md options docs instead of formatting the options in nix. docbook docs are
already produced in derivations.

the new script produce the exact same output as the old in-nix generation.
2021-12-06 16:12:30 +01:00
Artturin
60422ba2ea nixos/test-driver: add 10ms delay to send_key
attempt to fix https://github.com/NixOS/nixpkgs/issues/147294
2021-12-03 20:04:56 +02:00
Robert Hensing
78ada83361 nixos/eval-config: Deprecate extraArgs and check parameters 2021-12-02 18:23:43 +00:00
Robert Hensing
0b5aea2b27 nixos/eval-config: Remove a rec 2021-12-02 18:23:43 +00:00
Robert Hensing
59c4a35aab nixos/eval-config: Avoid evalModules args and check parameters 2021-12-02 18:23:43 +00:00
Jacek Galowicz
58371472fe
Merge pull request #146512 from DeterminateSystems/better-visibility
nixos/test-driver: more context when step finishes, give more functions nested labels
2021-12-02 09:26:21 +00:00
Jacek Galowicz
57dc1085dc
Merge pull request #146271 from DeterminateSystems/better-timeouts
nixos/test-driver: add execute_with_timeout, add (functional) timeouts to more functions
2021-12-01 19:01:05 +00:00
Cole Helbling
af765f3abd nixos/test-driver: give more functions nested labels
This will make it easier to trace through the test execution without
having to scroll through the entire kernel output.
2021-12-01 10:29:56 -08:00
Cole Helbling
c6ee63259a nixos/test-driver: more context when step finishes
When displaying the amount of time some step took, with no other
context, it becomes nigh impossible (especially in longer tests) to see
when specific steps finished.
2021-12-01 10:29:56 -08:00
Robert Hensing
8a129f8cf0
Merge pull request #144094 from hercules-ci/nixos-specialisations-use-extendModules
nixos/specialisation: Rephrase in terms of extendModules, noUserModules
2021-12-01 11:03:36 +01:00
Martin Schwaighofer
af180d554b qemu, runInLinuxVM: change default cpu to qemu64
The flag -cpu max leaves QEMU 6.1.0 stuck on some systems,
for example when /dev/kvm is not read-writable.
This does not happen with -cpu qemu64.

Getting stuck like that is a regression in 6.1.0 not yet present in 6.0.0
and should be fixed with 6.2.0 according to early testing with rc1.

We should consider reverting this change when we merge QEMU 6.2.0.
See #146526.

fixes #141596
2021-11-30 13:06:22 +00:00
talyz
e8cc900eae
make-disk-image: Make additionalPaths work with Nix 2.4
The `nix` command is marked as experimental since 2.4, so an extra
flag is required to unlock it.
2021-11-29 21:11:15 +01:00
Ryan Burns
2175b157ac treewide: refactor isi686 && isx86_64 -> isx86 2021-11-20 17:50:41 -08:00
Will Fancher
851495a752 Move systemd-lib.nix and systemd-unit-options.nix into utils 2021-11-20 17:52:29 -05:00
Patrick Hilhorst
1334a62539
test-driver.py: directly import pathlib.Path 2021-11-20 01:37:08 +01:00
Patrick Hilhorst
9e71014ede
test-driver.py: always export single machine as 'machine' 2021-11-19 23:02:30 +01:00
Patrick Hilhorst
a8f693ed48
test-driver.py: fix weird non-pythonism 2021-11-19 22:28:32 +01:00
Cole Helbling
e62b8020f3 nixos/test-driver: add (functional) timeouts to more functions
A retry timeout doesn't really help if the thing it's retrying may block
forever.
2021-11-17 16:11:10 -08:00
Cole Helbling
363d7f3ae8 nixos/test-driver: add timeout parameter to execute 2021-11-17 16:11:03 -08:00
Artturin
83f892c511 nixos/lib: add /usr to pathsNeededForBoot
if /usr is split from root then it has to be mounted at boot
See https://github.com/NixOS/nixpkgs/issues/146115
2021-11-15 16:28:22 +02:00
Jacek Galowicz
2ba0732988
Merge pull request #142747 from helsinki-systems/fix/test-runner-execute
nixos/test-runner: Fix execute() flakiness
2021-11-02 14:51:50 +01:00
Robert Hensing
b5681a7a40 nixos/specialisation: Rephrase in terms of extendModules, noUserModules
By using the new extendModules function to produce the specialisations,
we avoid reimplementing the eval-config.nix logic in reverse and fix
cross compilation support for specialisations in the process.
2021-11-01 10:59:47 +01:00
Robert Hensing
0b0d2637fc Revert "nixos/specialisation: Rephrase in terms of extendModules, noUserModules"
This reverts commit e2bea4427b.

While this commit was probably fine, I want to be conservative
with changes right before the release branch-off.
So far the extendModules commits have been adding and refactoring
expressions that did not affect derivation hashes, whereas this
commit changes the module ordering. I will open a separate PR for
it.
2021-11-01 09:41:55 +01:00
Robert Hensing
e2bea4427b nixos/specialisation: Rephrase in terms of extendModules, noUserModules 2021-11-01 09:40:16 +01:00
Robert Hensing
22584ce667 nixos/eval-config.nix: Expose type
Also works for (pkgs.nixos {}).type.
2021-11-01 09:38:49 +01:00
Robert Hensing
56c4f9d005 nixosTest: Fix infinite recursion involving hasContext testScript when useNixStoreImage = true
The involved test was nixosTests.nextcloud.basic21.

It has a testScript that is strict in nodes.nextcloud.config.system.build.vm,
in assertions about imagemagick being in the system closure.

The recursion was introduced in 329a4461a7 from
https://github.com/NixOS/nixpkgs/pull/140792
2021-10-31 18:25:19 +01:00
talyz
46f7521144 make-disk-image: Add an argument to only build a Nix store image
Add the `onlyNixStore` argument which enables building images
containing the contents of the Nix store at the root, instead of a
full system.
2021-10-28 12:55:01 +02:00
talyz
329a4461a7 nixos/testing-python: Copy test script derivations to nodes
Make sure the all derivations referenced by the test script are
available on the nodes. Accessing these derivations works just fine
without this change when using 9p to mount the host's store, but when
an image is built (virtualisation.buildRootImage), the dependencies
need to be copied to the image. We don't want to copy the script
itself, though, since that would trigger unnecessary image rebuilds.
2021-10-28 12:55:01 +02:00
talyz
87d8eec069 make-disk-image: Add additionalPaths argument
Add an argument which makes it possible to add additional paths to the
image's nix store.
2021-10-28 12:55:01 +02:00
talyz
398a73ac98 make-disk-image: Add copyChannel argument
Add a copyChannel argument which controls whether the current source
tree will be made available as a nix channel in the image or
not. Previously, it always was. Making it available is useful for
interactive use of nix utils, but changes the hash of the image when
the sources are updated.
2021-10-28 12:55:01 +02:00
talyz
56c5efa25b make-disk-image: Reintroduce the installBootLoader argument
installBootLoader was removed in
f1708a9d7d79e2bf2961fc648625578b23b3460f; probably by mistake, since
only the logic was removed, not the argument.
2021-10-28 12:55:01 +02:00
Janne Heß
1640359f33
nixos/test-runner: Fix execute() flakiness
Instead of using the magic string, we now just base64-encode everything
and check for a newline.
2021-10-28 11:51:20 +02:00
Janne Heß
1ed5606d8a
nixos/test-runner: Fix thread cleanup 2021-10-24 15:11:46 +02:00
K900
9611041ef5 nixos/lib/test-driver: set defaults in __init__ instead of class level
Just a minor code style nit to avoid potential footguns
2021-10-19 16:29:19 +03:00
K900
a874235dff nixos/lib/test-driver: clean up threads correctly
The current implementation just forks off a thread to read
QEMU's stdout and lets it exist forever. This, however,
makes the interpreter shutdown racy, as the thread could
still be running and writing out buffered stdout when the
main thread exits (and since it's using the low level API,
the worker thread does not get cleaned up by the atexit hooks
installed by `threading`, either). So, instead of doing that,
let's create a real `threading.Thread` object, and also
explicitly `join` it along with the other stuff when cleaning up.
2021-10-19 16:23:51 +03:00
Domen Kožar
8fdc8fe5ce
Merge pull request #142040 from pennae/optimize-module-docs
make-options-docs: don't sort the options XML file
2021-10-18 16:38:32 -05:00
pennae
24eb353907 make-options-docs: don't sort the options XML file
we need the file itself as a dependency for the docbook build, but we don't need
it to be properly sorted at the nix level. push the sort out to a python script
instead to save eval time. on the machine used to write this `nix-instantiate
<nixos/nixos> -A system` went down from 7.1s to 5.4s and GC heap size decreased
by 50MB (or 70MB max RSS).
2021-10-18 03:45:33 +02:00
Robert Hensing
c247827654 nixosTest: Provide system.build.networkConfig 2021-10-17 23:56:52 +02:00
Janne Heß
540dc908ca
nixos/test-runner: Print exceptions that happen 2021-10-14 00:15:53 +02:00
Kim Lindberger
4e97bedf93
Merge pull request #112823 from deviant/nixos-boot-paths
nixos/boot: add /var/lib/nixos to pathsNeededForBoot
2021-10-07 17:38:54 +02:00
Jacek Galowicz
d68f1e43df
Merge pull request #126713 from blaggacao/nixos-test-ref/05-class-ify-test-driver
nixos test ref/05 class ify test driver
2021-10-07 14:37:01 +02:00
Jacek Galowicz
5c666cdf62 Re-RAII-ify the NixOS integration test driver's VLAN class.
We have no usecase for manually/selectively starting or stopping VLANs
in integration tests.
By starting and stopping the VLANs with the constructor and destructor
of VLAN objects, we remove the obligation and complexity to maintain
network lifetime separately.
2021-10-05 14:38:48 -05:00
Maximilian Bosch
3f63e3ce65 nixos/test-driver: fix graphics for VM
`-nographic` should only be set if no display is available. Otherwise,
tools such as `nixos-build-vms(8)` will never open up a graphical
display.
2021-10-05 14:38:48 -05:00
David Arnold
b0fc9da879 nixos/test/test-driver: Class-ify the test driver
This commit encapsulates the involved domain into classes and
defines explicit and typed arguments where untyped dicts where used.

It preserves backwards compatibility through legacy wrappers.
2021-10-05 14:38:48 -05:00
Naïm Favier
330b1e08b8
nixos/lib/make-options-doc: implement literalDocBook 2021-10-03 17:59:44 +02:00
rnhmjoj
b29c2f97c3
nixos/lib/qemu-flags: rename to qemu-common
The current name is misleading: it doesn't contain cli arguments,
but several constants and utility functions related to qemu.
This commit also removes the use of `with import ...` for clarity.
2021-09-18 16:58:16 +02:00
rnhmjoj
30f0faac22
nixos/qemu: add types to all options 2021-09-18 16:47:11 +02:00
rnhmjoj
c1fd51f490
nixos/qemu: fix several shellcheck issues
- Fix shell quoting issues
- Fix unsafe cd in run-machine-vm script
2021-09-18 16:46:20 +02:00
Matthew Kenigsberg
b00d2a8f3e
nixos/testing: remove unused function runInMachine
runInMachine and runInMachineWithX are not used for any tests and can be
removed
2021-09-08 06:25:20 -05:00
Graham Christensen
076f6e2d94 nixos/make-zfs-image: init
This is a private interface for internal NixOS  use. It is similar
to `make-disk-image` except it is much more opinionated about what
kind of disk image it'll make.

Specifically, it will always create *two* disks:

1. a `boot` disk formatted with FAT in a hybrid GPT mode.
2. a `root` disk which is completely owned by a single zpool.

The partitioning and FAT decisions should make the resulting images
bootable under EFI or BIOS, with systemd-boot or grub.

The root disk's zpools options are highly customizable, including
fully customizable datasets and their options.

Because the boot disk and partition are highly opinionated, it is
expected that the `boot` disk will be mounted at `/boot`. It is
always labeled ESP even on BIOS boot systems.

In order for the datasets to be mounted properly, the `datasets`
passed in to `make-zfs-image` are turned in to NixOS configuration
stored at /etc/nixos/configuration.nix inside the VM.
NOTE: The function accepts a system configuration in the `config`
argument. The *caller* must manually configure the system
in `config` to have each specified `dataset` be represented
by a corresponding `fileSystems` entry.

One way to test the resulting images is with qemu:

```sh
boot=$(find ./result/ -name '*.boot.*');
root=$(find ./result/ -name '*.root.*');

echo '`Ctrl-a h` to get help on the monitor';
echo '`Ctrl-a x` to exit';

qemu-kvm \
    -nographic \
    -cpu max \
    -m 16G \
    -drive file=$boot,snapshot=on,index=0,media=disk \
    -drive file=$root,snapshot=on,index=1,media=disk \
    -boot c \
    -net user \
    -net nic \
    -msg timestamp=on
```
2021-08-25 10:42:35 -04:00
David Arnold
2832896c5b
nixos/test: some test fixes in succession of #125992 2021-08-20 13:38:22 -05:00
David Arnold
db614e11d6
nixos/tests/test-driver: better control test env symbols
Previous to this commit, the entire test driver environment was shared
with the actual python test environment.

This is a hefty api surface. This commit selectively exposes only those
symbols to the test environment that are actually meant to be used by
tests.
2021-08-19 23:55:26 -05:00
Maximilian Bosch
0362d572b2
Merge pull request #133675 from Ma27/test-driver-compat
nixos/test-driver: minor fixes for `nixos-build-vms(8)`
2021-08-16 14:07:01 +02:00
Robert Hensing
fbafeb7ad5 treewide: runCommandNoCC -> runCommand
This has been synonymous for ~5y.
2021-08-15 17:36:41 +02:00
Maximilian Bosch
1ba3f7927f
nixos/testing: only create nixos-run-vms for nixos-build-vms(8)
This is the case when the test-script is empty. `nixos-build-vms(8)` is
primarily supposed to be used as tool to test changes or to reproduce
bugs (IMHO) where "just spinning up a few VMs" is the primary use-case.

In the ongoing discussion about these changes[1] it was suggested to
only expose it when needed (i.e. in the case I described above) to keep
the API surface as slim as possible.

[1] https://github.com/NixOS/nixpkgs/pull/133675#discussion_r688112485
2021-08-13 17:39:49 +02:00
Maximilian Bosch
e1ec5acd31
nixos/test-driver: start interactive mode if testScript is empty
This is relevant for `nixos-build-vms(8)` which doesn't have a
test-script. In that case it's more intuitive to directly go into the
interactive mode which is IMHO more intuitive.
2021-08-12 23:01:03 +02:00
Maximilian Bosch
260d9cc7e1
nixos/testing: re-add nixos-run-vms script
Originally removed in 926fb93968. This
one is actually quite useful for `nixos-build-vms(8)`.
2021-08-12 22:50:29 +02:00
lewo
7aa78642c5
Merge pull request #125979 from blaggacao/nixos-test-ref/03-normalse-the-python-entrypoint
nixos/test-driver: normalize the python entrypoint
2021-08-08 10:24:47 +02:00
David Arnold
926fb93968
nixos/tests/test-driver: normalise test driver entrypoint(s)
Previously the driver was configured exclusively through convoluted
environment variables.

Now the driver's defaults are configured through env variables.

Some additional concerns are in the github comments of this PR.
2021-08-05 19:07:11 -05:00
David Arnold
dc060ff8b2
nixos/testing: fix invalid node names detection
the use of python further restricts possible RFC1035 host labels since
dash is not allowed for use in python identifiers.

The previous implementation of this check was flawed, since it did not
check the `hostName` value that is actually used to construe the
identifier, but the node name, which can be anything, e.g. just `machine`.

The previous implementation, by further restricting RFC1035 labels, only
for the sake of testing seems to be an unacceptable restriction and should
be addressed separately.
2021-07-28 15:37:57 -05:00
Ingo Blechschmidt
5143ab9f74 tigervnc, tightvnc: add basic tests
Co-Authored-By: Ingo Blechschmidt <iblech@web.de>
2021-07-20 15:22:31 +02:00
misuzu
702a1c23a4 nixos/test-driver: allow overriding qemu binary in create_startcommand 2021-07-14 12:47:12 +03:00
Robert Hensing
c1a75084ea
Merge pull request #127744 from cransom/disk-image-faster
make-disk-image: Reduce forking
2021-06-26 08:31:12 +02:00
Jörg Thalheim
48aec70ea1
nixos/test-driver: replace termlib with socat
telnetlib does not handle unicode, which is annoying when using systemctl.
Also this gives us a nice readline with history.
2021-06-22 09:08:57 +02:00
Casey Ransom
c3f28aed94 make-disk-image: Reduce forking
The disk image calculator was using find + exec forking du for every
file in the disk image, making it very slow. Change du to accept files,
nul delimeted on stdin to speed it back up.

Before change:
nix-build nixos/tests/image-contents.nix  9.71s user 1.06s system 8% cpu 2:13.11 total

After change:
nix-build nixos/tests/image-contents.nix  9.93s user 1.23s system 21% cpu 51.601 total
2021-06-21 22:58:07 -04:00
David Arnold
8bbdff4581
nixosTest: Force system.nixos.revision constant
nixos tests are blended with other system configurations, hence
their settings must be either enforced or defaulted.

This particular setting is set via lib.nixosSystem as
`system.nixos.revision = final.mkIf (self ? rev) self.rev;` which would
mean that without this change no flake generated nixos could be blended
with nixos testing.

This setting was made previously constant in 
169c6b4b14 in order to avoid pointless
rebuilds of the testing VMs, but was set without enforcing it.
2021-06-21 11:08:26 -05:00
Kim Lindberger
410496d0f3
Merge pull request #127309 from jakobrs/nixos-normalise-null
Resolve evaluation failure when filesystem device option is null
2021-06-18 17:15:08 +02:00
aszlig
bc0997489b
nixos/test-driver: Fix passing passthru attribute
Apparently this looks like it was forgotten when doing commit
3884ff70ba, which refactored the test
runner and driver a bit.

The passthru argument actually was correctly reintroduced in
setupDriverForTest, but the actual makeTest function didn't use it.

This fixes the nixpkgs tarball job, which previously failed with:

  attribute 'elkPackages' missing, at /build/source/pkgs/tools/misc/logstash/6.x.nix:58:30

Signed-off-by: aszlig <aszlig@nix.build>
Acked-by: David Arnold <dar@xoe.solutions>
Fixes: https://github.com/NixOS/nixpkgs/issues/127274
Merges: https://github.com/NixOS/nixpkgs/pull/127346
2021-06-18 14:50:15 +02:00
jakobrs
7a0c1728ac nixos/lib: Handle null device correctly 2021-06-18 08:32:45 +02:00
Jacek Galowicz
a6d8488c58
Merge pull request #125973 from blaggacao/nixos-test-ref/02-normalise-test-driver-nix-expressions
nixos test ref/02 normalise test driver nix expressions
2021-06-16 18:59:23 +02:00
Kim Lindberger
26706834a5
Merge pull request #86967 from jakobrs/more-general-fsbefore
nixos/lib/utils: Add `fileSystems.<name>.depends` option and generalise fsBefore (fixes #86955)
2021-06-14 10:50:07 +02:00
David Arnold
3884ff70ba
nixos/tests/test-driver: cleanup nix expression
Less nesting, where that improves readability. More nesteing, where
that improves readability, but most importantly:

Expose individual functions separately so that they can be more easily
built directly, eg.:

`nix build --impure --expr '(import ./testing-python.nix {system = builtins.currentSystem;}).mkTestDriver'`
2021-06-13 15:47:51 -05:00
jakobrs
b07602a604 nixos/lib, nixos/filesystems: Make fsBefore more stable, and add depends option 2021-06-08 18:51:31 +02:00
David Arnold
03bfa29891
nixos/tests/test-driver: cleanup "dead" code (USE_SERIAL)
At nixpkgs root:

`rg redirectSerial ./` does not result in any other match
nor does
`rg USE_SERIAL ./` except for an unrelated match in:

pkgs/tools/graphics/argyllcms/default.nix
2021-06-06 12:58:58 -05:00
talyz
b7749c7671
nixos/test-driver: Run commands with error handling
Bash's standard behavior of not propagating non-zero exit codes
through a pipeline is unexpected and almost universally
unwanted. Default to setting `pipefail` for the command being run;
it can still be turned off by prefixing the pipeline with
`set +o pipefail` if needed.

Also, set `errexit` and `nonunset` options to make the first command
of consecutive commands separated by `;` fail, and disallow
dereferencing unset variables respectively.
2021-06-05 18:44:42 +02:00
Frederik Rietdijk
3edde6562e make-test-python: disallow aliases
When importing Nixpkgs within Nixpkgs, we should not consider aliases
to ensure we don't rely on them internally.

There are probably more places that need to be converted.
2021-06-03 11:03:31 -07:00
Domen Kožar
f88be6e55c
Merge pull request #125372 from Synthetica9/shell_interact
nixos/tests/test-driver: add shell_interact
2021-06-03 11:46:01 +02:00
Patrick Hilhorst
fd739c4dee
nixos/tests/test-driver: make it clear when shell is ready
Co-authored-by: Domen Kožar <domen@enlambda.com>
2021-06-03 11:20:26 +02:00
Patrick Hilhorst
9469433e34
nixos/tests/test-driver: document shell_interact 2021-06-02 14:49:59 +02:00
Patrick Hilhorst
5a589b5ba8
nixos/tests/test-driver: add shell_interact 2021-06-02 14:20:04 +02:00
Damien Diederen
7c2adb1d5c nixos/lib/make-ext4-fs: Fix: `resize2fs -M' can leave insufficient slack
The root filesystem resizing step, `resize2fs -M', does not provide any
control over the amount of slack left in the result.  It can produce an
arbitrarily tight fit, depending on how well the payload aligns with
ext4 data structures.

This is problematic, as NixOS must create a few files and directories
during its first boot, before the root is enlarged to match the size of
the containing SD card.

An overly tight fit can cause failures in the first stage:

    mkdir: can't create directory '/mnt-root/proc': No space left on device

or in the second stage:

    install: cannot create directory '/var': No space left on device

A previous version of `make-ext4-fs' (before PR #79368) was explicitly
"reserving" 16 MiB of free space in the final filesystem.  Manually
calculating the size of an ext4 filesystem is a perilous endeavor,
however, and the method it employed was apparently unreliable.

Reverting is consequently not a good option.

A solution would be to create some sort of "balloon" occupying inodes
and blocks in the image prior to invoking `resize2fs -M', and to remove
these temporary files/directories before the compression step.

This changeset takes the simpler approach of simply dropping the
resizing step.

Note that this does *not* result in a larger image in general, as the
current procedure does not truncate the `.img' file anyway.  In fact, it
has been observed to yield *smaller* compressed images---probably
because of some "noise" left after resizing.  E.g., before-vs-after:

    -r--r--r-- 2 root root 607M  1. Jan 1970  nixos-sd-image-21.11pre-git-x86_64-linux.img.zst

    -r--r--r-- 2 root root 606M  1. Jan 1970  nixos-sd-image-21.11pre-git-x86_64-linux.img.zst
2021-05-31 21:35:51 +02:00
Andreas Rammhold
d07f52bf81
nixos/test-driver: mention the elapsed time when it times out
For now you had to know that the actions are retried for 900s when
seeing an error like

> Traceback (most recent call last):
>   File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 927, in run_tests
>     exec(tests, globals())
>   File "<string>", line 1, in <module>
>   File "<string>", line 31, in <module>
>   File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 565, in wait_for_file
>     retry(check_file)
>   File "/nix/store/dbvmxk60sv87xsxm7kwzzjm7a4fhgy6y-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 142, in retry
>     raise Exception("action timed out")
> Exception: action timed out

in your (hydra) build failure. Due to the absence of timestamps you were
left guessing if the machine was just slow, someone passed a low timeout
value (which they couldn't until now) or whatever might have happened.

By making this error a bit more descriptive (by including the elapsed
time) these hopefully become more useful.
2021-05-30 17:26:13 +02:00
Jacek Galowicz
5d05391846
Merge pull request #123823 from misuzu/test-driver-usb-boot-speedup
nixos/test-driver: use usb-ehci controller instead of piix3-usb-uhci
2021-05-21 20:47:47 +02:00
misuzu
b2319b086c nixos/test-driver: use usb-ehci controller instead of piix3-usb-uhci
On my system this change offers ~5X speed up of
nixosTests.boot.biosUsb and nixosTests.boot.uefiUsb tests.
2021-05-20 22:33:08 +03:00
Robert Hensing
a37d157601
Merge pull request #123052 from xoe-labs/da-test-vm-innteractive-log-switch
nixos/testing: add interactive serial stdout logs switch and dim them
2021-05-20 15:32:54 +02:00
Robert Hensing
76a7840f5f
Merge pull request #117275 from hercules-ci/nixosTest-remove-nixpkgs-commit-hash
nixosTest: Make system.nixos.revision constant
2021-05-20 10:40:59 +02:00
David Arnold
8ee31be5dd
nixos/testing: add interactive serial stdout logs switch and dim them 2021-05-14 20:36:00 -04:00
Robert Hensing
75c4fc1c8b nixos/testing-python.nix: Move makeWrapper to nativeBuildInputs 2021-05-09 03:04:03 +02:00
Robert Hensing
56d9637119
nixos/testing: Set up scope for testScript linter
Our test driver exposes a bunch of variables and functions, which
pyflakes doesn't recognise by default because it assumes that the test
script is executed standalone. In reality however the test driver script
is using exec() on the testScript.

Fortunately pyflakes has $PYFLAKES_BUILTINS, which are the attributes
that are globally available on all modules to be checked. Since we only
have one module, using this environment variable is fine as opposed to
my first approach to this, which tried to use the unstable internal API
of pyflakes.

The attributes are gathered by the main derivation of the test driver,
because we don't want to end up defining a new attribute in the test
driver module just to being confused why using it in a test will result
in an error.

Another way we could have gathered these attributes would be in
mkDriver, which is where the linting takes place. However, we do have a
different set of Python dependencies in scope and duplicating these will
again just cause confusion over having it at one location only.

Signed-off-by: aszlig <aszlig@nix.build>
Co-Authored-By: aszlig <aszlig@nix.build>
2021-05-09 02:26:51 +02:00
Robert Hensing
71087b2bc4
nixos/testing-python.nix: Expose driver
(cherry picked from commit a2c9220568648b4528154ebd8e657add243ed0b4)
2021-05-09 02:26:40 +02:00
aszlig
c362a28fcf
nixos/testing: Switch from black to pyflakes
So far, we have used "black" for formatting the test code, which is
rather strict and opinionated and when used inline in Nix expressions it
creates all sorts of trouble.

One of the main annoyances is that when using strings coming from Nix
expressions (eg. store paths or option definitions from NixOS modules),
completely unrelated changes could cause tests to fail, since eg. black
wants lines to be broken.

Another downside of enforcing a certain kind of formatting is that it
makes the Nix expression code inconsistent because we're mixing two
spaces of indentation (common in nixpkgs) with four spaces of
indentation as defined in PEP-8. While this is perfectly fine for
standalone Python files, it really looks ugly and inconsistent IMO when
used within Nix strings.

What we actually want though is a linter that catches problems early on
before actually running the test, because this is *actually* helping in
development because running the actual VM test takes much longer.

This is the reason why I switched from black to pyflakes, because the
latter actually has useful checks, eg. usage of undefined variables,
invalid format arguments, duplicate arguments, shadowed loop vars and
more.

Signed-off-by: aszlig <aszlig@nix.build>
Closes: https://github.com/NixOS/nixpkgs/issues/72964
2021-05-09 02:26:37 +02:00
Michael Weiss
3c1a76611e
nixos/test-driver: Allow interactive testing on Wayland-only setups
On my system I have XWayland disabled and therefore only WAYLAND_DISPLAY
is set. This ensures that the graphical output will still be enabled on
such setups (both Wayland and X11 are supported by the viewer).
2021-05-04 16:23:02 +02:00
Robert Hensing
162b8fba12 testing-python.nix: document runTests pos argument 2021-05-03 13:33:41 +02:00
Robert Hensing
b2f44e9aeb
Merge pull request #112504 from hercules-ci/fix-nixosTest-meta-position
nixosTest: fix meta.position
2021-05-03 11:50:57 +02:00
Alyssa Ross
a8afbb45c1 treewide: use lib.warnIf where appropriate 2021-04-28 21:44:21 +00:00
Lassulus
ee04d772e4
Merge pull request #120489 from samueldr/fix/make-disk-image-auto-size
Fix make disk image automatic size
2021-04-26 10:34:15 +02:00
Lassulus
cdddbf59ea
Merge pull request #120251 from mschwaig/fix-make-disk-image-for-efi-2
make-disk-image: fix broken EFI image builds
2021-04-26 10:04:00 +02:00
Martin Schwaighofer
f20ae954d5 make-disk-image: fix broken EFI image builds
Work around missing /dev files inside runInLinuxVM by creating a
symlink before calling nixos-enter.

This fixes https://github.com/NixOS/nixpkgs/issues/93381.
I ran into this issue when trying to create a VMware image that boots from EFI.

Thanks @colemickens for reporting this and @danielfullmer for fixing the same thing in in qemu-vm.nix (37676e77cb) and explaining what the issue was.
2021-04-26 01:12:10 +02:00
Samuel Dionne-Riel
7b8b3fab6d make-disk-image: Round image size to the next mebibyte
This ensures the following gptfdisk warning won't happen:

```
Warning: File size is not a multiple of 512 bytes! Misbehavior is likely!
```

Additionally, helps towards aligning the partition to be more optimal
for the underlying storage.

It is actually impossible to align for the actual underlying storage
optimally because we don't know what the block device will be!

But aligning on 1MiB should help.
2021-04-25 15:24:45 -04:00
Samuel Dionne-Riel
ba666011a6 make-disk-image: Account for reserved disk space
This is a bit of a thorny issue. See, the actual `diskSize` variable is
for the *total* disk size, not for the filesystem!

The automatic numbers are meant to compute the *filesystem* required
space. So we have to add any other reserved space!

We have different requirements for reserved space. E.g. there could be
none (when it's actually a filesystem image). There could also be 1MiB
for alignment for an MBR image, legacy+gpt needs 2MiB, then GPT with an
ESP ("bootSize") needs to take the boot partition and GPT size into
account too!

Though luckily(?) for this latter situation we can cheat! As noted in the
change, `bootSize` is NOT the boot partition size. It is actually the
offset where the target filesystem starts.
2021-04-24 14:49:05 -04:00
Samuel Dionne-Riel
9b18a78c73 make-disk-image: Account for the ext4 reserved space
Reserved space includes:

 - inodes space in use (2 blocks per)
 - about 5.2% of the space

The 5.2% reserved space was computed empirically when working on a
previous EXT4 image builder. It seems to stabilize around 5% even for
much larger filesystems.
2021-04-24 14:49:04 -04:00
Samuel Dionne-Riel
05c13a03e2 make-disk-image: Get proper size for automatic size
On some filesystems, `du` without `--apparent-size` will not give the
actual size for a file. Using `--apparent-size` will give us the actual
file size.

Though, this is not actually correct still. 1000 × 1 bytes is not 1000
bytes. It is 1000 × ceil(filesize/blockSize)*blockSize.

So instead of adding up the actual file sizes. We are adding up the
block sizes.

Note that this also changes the builder to work with *bytes*, rather
than with any other units. Doing maths on bytes is less likely to go
awry than doing it on other units.
2021-04-24 14:49:04 -04:00
Luke Granger-Brown
4de343cccf nixos/test-driver: use a variety of different Tesseract settings for OCR
When performing OCR, some of the Tesseract settings perform better than
others on a variety of different workloads, but they mostly take
~negligible incremental time to run compared to the overhead of running
the ImageMagick filters.

After this commit, we try using all three of the current Tesseract
models (classic, LSTM, and classic+LSTM) to generate output text. This
fixes chromium-90's tests at release-20.09, and should make cases where
you're looking for *specific* text better, with the tradeoff of running
Tesseract multiple times.

To make it sensible to cherrypick this into release-20.09, this doesn't
change the existing API surface for the test driver. In particular,
get_screen_text continues to have the existing behaviour.
2021-04-23 18:42:35 +00:00
Pacman99
d12aba5406 nixosOptionsDoc: add markdown formatting 2021-04-13 10:10:40 -07:00
Robert Hensing
cda907d895 nixosTest: fix meta.position 2021-03-23 15:14:31 +01:00
Robert Hensing
169c6b4b14 nixosTest: Make system.nixos.revision constant
Make the revision metadata constant, in order to avoid needless retesting.

The human version (e.g. 21.05-pre) is left as is, because it is useful
for external modules that test with e.g. nixosTest and rely on that
version number.
2021-03-22 21:39:43 +01:00
Ryan Burns
0ebf63de08 nixos/qemu-flags: add ppc64
* PPC uses ttyAMA0 as its serial device, similar to ARM.
* PowerNV is a typical platform for testing and distro development
2021-03-05 01:57:54 -08:00
Arnout Engelen
0aeba64fb2
squashfs: use -no-hardlinks for reproducible squashfs images (#114454)
the nix store may contain hardlinks: derivations may output them
directly, or users may be using store optimization which automatically
hardlinks identical files in the nix store.

The presence of these links are intended to be a 'transparent'
optimization. However, when creating a squashfs image, the image
will be different depending on whether hard links were present
on the filesystem, leading to reproducibility problems.

By passing '-no-hardlinks' to mksquashfs the files are stored
as duplicates in the squashfs image. Since squashfs has support
for duplicate files this does not lead to a larger image.

For more details see
https://github.com/NixOS/nixpkgs/issues/114331
2021-02-28 18:03:50 +00:00
V
ededd308a8 nixos/boot: add /var/lib/nixos to pathsNeededForBoot
/var/lib/nixos is used by update-users-groups.pl in the activation
script for storing uid/gid mappings. If this has its own mountpoint
(as is the case in some setups with fine-grained bind mounts pointing
into persistent storage), the mappings are written to /var/lib, /var,
or /. These may be backed by a tmpfs or (otherwise ephemeral storage),
resulting in the mappings not persisting between reboots.
2021-02-11 22:59:05 +01:00
Jörg Thalheim
e5ce05cc1e
Merge pull request #67493 from clefru/make-disk-image 2021-01-29 07:18:50 +00:00
clefru
60c8cf4e02
Update nixos/lib/make-disk-image.nix
Co-authored-by: Alyssa Ross <hi@alyssa.is>
2021-01-22 19:58:23 +01:00
Léo Gaspard
a5a819e059
Merge pull request #104292 from fgaz/image-contents
nixos/lib/make-disk-image.nix: support content mode and ownership
2020-12-21 19:58:49 +01:00
Francesco Gazzetta
6f21ae7a62 nixos/lib/make-disk-image.nix: support content mode and ownership 2020-12-20 21:22:47 +01:00
Robert Hensing
72a5301610 nixos/lib/testing-python.nix: Wire up passthru 2020-12-09 13:24:03 +01:00
Graham Christensen
bc49a0815a
utillinux: rename to util-linux 2020-11-24 12:42:06 -05:00
Frederik Rietdijk
f36b838e2a nixos test-driver: fix single line docstrings, fixes #104467
Single line docstrings should have the """ on a single line according to PEP 8.
It seems support for this landed in the latest version of Black.
2020-11-21 09:51:31 +01:00
Emery Hemingway
7e25b71132 nixos: use nativeBuildInputs in make- iso9660-image and system-tarball
The tools used to create iso9660 images and tarballs are independent of
the platform of the closure contained within.
2020-11-18 14:05:30 +01:00
Graham Christensen
d78aa080f5
make-disk-image: support legacy+gpt 2020-10-30 15:50:24 -04:00