We spent a whole afternoon debugging this, because upstream has very
bad software quality and the error messages were incredibly
misleading.
So let’s document it for the sanity of other people.
Btw, I think the implementation of our module is pretty brittle,
especially the part about diffing tokens to check whether they
changed. We should rather just request a new builder registration
every time, it’s not that much overhead, and always set `replace` so
it is idempotent.
The `nix.*` options, apart from options for setting up the
daemon itself, currently provide a lot of setting mappings
for the Nix daemon configuration. The scope of the mapping yields
convience, but the line where an option is considered essential
is blurry. For instance, the `extra-sandbox-paths` mapping is
provided without its primary consumer, and the corresponding
`sandbox-paths` option is also not mapped.
The current system increases the maintenance burden as maintainers have to
closely follow upstream changes. In this case, there are two state versions
of Nix which have to be maintained collectively, with different options
avaliable.
This commit aims to following the standard outlined in RFC 42[1] to
implement a structural setting pattern. The Nix configuration is encoded
at its core as key-value pairs which maps nicely to attribute sets, making
it feasible to express in the Nix language itself. Some existing options are
kept such as `buildMachines` and `registry` which present a simplified interface
to managing the respective settings. The interface is exposed as `nix.settings`.
Legacy configurations are mapped to their corresponding options under `nix.settings`
for backwards compatibility.
Various options settings in other nixos modules and relevant tests have been
updated to use structural setting for consistency.
The generation and validation of the configration file has been modified to
use `writeTextFile` instead of `runCommand` for clarity. Note that validation
is now mandatory as strict checking of options has been pushed down to the
derivation level due to freeformType consuming unmatched options. Furthermore,
validation can not occur when cross-compiling due to current limitations.
A new option `publicHostKey` was added to the `buildMachines`
submodule corresponding to the base64 encoded public host key settings
exposed in the builder syntax. The build machine generation was subsequently
rewritten to use `concatStringsSep` for better performance by grouping
concatenations.
[1] - https://github.com/NixOS/rfcs/blob/master/rfcs/0042-config-option.md
adds defaultText for all options that use `cfg.*` values in their
defaults, but only for interpolations with no extra processing (other
than toString where necessary)
This commit changes how we deal with the current token, i.e., the token
which may exist from a previous runner registration, and the configured
token, i.e., the path set for the respective NixOS configuration option.
Until now, we copied the configured and the current token (if any) to
the runtime directory to compare them. The path of the current token may
reference a file which is only accessible to specific users (even only
root). Therefore, we ran the copying of credentials with elevated
privileges by prefixing the `ExecStartPre=` script with a `+` (see
systemd.service(5)). In this script, we also changed the owner of the
files to the service user. Apparently, however, the user/group pair
sometimes did not exist because we use `DynamicUser=`.
To address this issue, we no longer change the owner of the file.
Instead, we change the file permissions to 0666 to allow the runner
configuration script (runs with full sandboxing) to read-write the file.
Due to the current permissions of the runtime directory (0755), this
would expose the token. Therefore, we process the tokens in the state
directory, which is only accessible to the service user.
If a new token file exists in the state directory, the configuration
script should trigger a new runner registration. Afterward, it deletes
the new token file. The token is still available using the path of the
current token which is inaccessible within the service's sandbox.
This addresses #120263 in part, by allowing users to override the
github-runner derivation that is bound to turn non-functional via the
self-update mechanism. (And it'll allow using a buildFHSUserEnv-based
derivation, if someone ends up building that!)
Add support for folder jobs
(https://plugins.jenkins.io/cloudbees-folder/) by reworking the service
to support nested jobs.
This also fixes this deprecation warning (as a happy side effect):
WARNING:jenkins_jobs.cli.subcommand.test:(Deprecated) The default output behavior of `jenkins-jobs test` when given the --output flag will change in JJB 3.0. Instead of writing jobs to OUTPUT/jobname; they will be written to OUTPUT/jobname/config.xml. The new behavior can be enabled by the passing `--config-xml` parameter
While looking at the sphinx package I noticed it was heavily
undermaintained, which is when we noticed nand0p has been inactive for
roughly 18 months. It is therefore prudent to assume they will not be
maintaining their packages, modules and tests.
- Their last contribution to nixpkgs was in 2019/12
- On 2021/05/08 I wrote them an email to the address listed in the
maintainer-list, which they didn't reply to.
The buildkite agent supports multiple tags with the same key. This
functionality is used to have a [single agent listen on multiple
queues](https://buildkite.com/docs/agent/v3/queues#setting-an-agents-queue).
However, having the tags be of type `attrsOf str` means that
we cannot suport this use case. This commit modifies the type
of tags to be `attrsOf (either str (listOf str))` where the list
is expanded into multiple tags with the same key.
Example:
```
{tags = {queue = ["default", "testing"];};}
```
generates
```
tags="queue=default,queue=testing"
```
in the buildkite agent configuration.
* github-runner: init at 2.277.1
* nixos/github-runner: initial version
* nixos/github-runner: add warning if tokenFile in Nix store
* github-runner: don't accept unexpected attrs
* github-runner: formatting nits
* github-runner: add pre and post hooks to checkPhase
* nixos/github-runner: update ExecStartPre= comment
* nixos/github-runner: adapt tokenFile option description
Also note that not only a change to the option value will trigger a
reconfiguration but also modifications to the file's content.
* nixos/github-runner: remove mkDefault for DynamicUser=
* nixos/github-runner: create a parent for systemd dirs
Adds a parent directory "github-runner/" to all of the systemd lifecycle
directories StateDirectory=, RuntimeDirectory= and LogDirectory=.
Doing this has two motivations:
1. Something like this would required if we want to support multiple
runners configurations. Please note that this is already possible
using NixOS containers.
2. Having an additional parent directory makes it easier to remap
any of the directories. Without a parent, systemd is going to
complain if, for example, the given StateDirectory= is a symlink.
* nixos/github-runner: use specifier to get abs runtime path
* nixos/github-runner: use hostname as default for option `name`
Until now, the runner registration did not set the `--name` argument if
the configuration option was `null`, the default for the option.
According to GitHub's documentation, this instructs the registration
script to use the machine's hostname.
This commit causes the registration to always pass the `--name` argument
to the runner configuration script. The option now defaults to
`networking.hostName` which should be always set on NixOS.
This change becomes necessary as the systemd service name includes the
name of the runner since fcfa809 and, hence, expects it to be set. Thus,
an unset `name` option leads to an error.
* nixos/github-runner: use types.str for `name` option
Forcing a `name` option to comply with a pattern which could also be
used as a hostname is probably not required by GitHub.
* nixos/github-runner: pass dir paths explicitly for ExecStartPre=
* nixos/github-runner: update variable and script naming
* nixos/github-runner: let systemd choose the user/group
User and group naming restrictions are a complex topic [1] that I don't
even want to touch. Let systemd figure out the username and group and
reference it in our scripts through the USER environment variable.
[1] https://systemd.io/USER_NAMES/
* Revert "nixos/github-runner: use types.str for `name` option"
The escaping applied to the subdirectory paths given to StateDirectory=,
RuntimeDirectory= and LogsDirectory= apparently doesn't use the same
strategy that is used to escape unit names (cf. systemd-escape(1)). This
makes it unreasonably hard to construct reliable paths which work for
StateDirectory=/RuntimeDirectory=/LogsDirectory= and ExecStartPre=.
Against this background, I decided to (re-)apply restrictions to the
name a user might give for the GitHub runner. The pattern for
`networking.hostName` seems like a reasonable choice, also as its value
is the default if the `name` option isn't set.
This reverts commit 193ac67ba337990c22126da24a775c497dbc7e7d.
* nixos/github-runner: use types.path for `tokenFile` option
* nixos/github-runner: escape options used as shell arguments
* nixos/github-runner: wait for network-online.target
* github-runner: ignore additional online tests
This will make it easier to track specifically where queries are being
made from (assuming a `log_line_prefix` that includes `%a` in the
postgres configuration).
These were broken since 2016:
f0367da7d1
since StartLimitIntervalSec got moved into [Unit] from [Service].
StartLimitBurst has also been moved accordingly, so let's fix that one
too.
NixOS systems have been producing logs such as:
/nix/store/wf98r55aszi1bkmln1lvdbp7znsfr70i-unit-caddy.service/caddy.service:31:
Unknown key name 'StartLimitIntervalSec' in section 'Service', ignoring.
I have also removed some unnecessary duplication in units disabling
rate limiting since setting either interval or burst to zero disables it
(ad16158c10/src/basic/ratelimit.c (L16))
This should NOT be backported to 20.09!
When 21.03 is released, the DB changes are about a year old and
operators had two release cycles for the upgrade. At this point it
should be fair to remove the compat layer to reduce the complexity of
the module itself.
In the current implementation, there's no possibility to modify the default
parameter for keepalive. This is a number that indicates how frequently
keepalive messages should be sent from the worker to the buildmaster,
expressed in seconds. The default (600) causes a message to be sent to
the buildmaster at least once every 10 minutes.
If the worker is behind a NAT box or stateful firewall, these messages
may help to keep the connection alive: some NAT boxes tend to forget about
a connection if it has not been used in a while. When this happens, the
buildmaster will think that the worker has disappeared, and builds will
time out. Meanwhile the worker will not realize than anything is wrong.
Since Buildbot 0.9.0, status targets were deprecated and ignored.
There's a very small line on startup explaining that, and status simply
isn't reported. Avoid others the same headaches, and do it right in the
NixOS module.
As there might have been changes in the way reporters are organized, and
configuration might need to be migrated remove the old option, and not
just provide an alias.
It's pbPort, and it's also a connection string, meaning
listen-on-localhost is also possible. Provide an alias for the old
option name, so old configs still work.
We already set the relevant env vars in the systemd services. That does
not help one when executing any of the executables outside a service,
e.g. when creating a new user.
Also removed `pkgs.hydra-flakes` since flake-support has been merged
into master[1]. Because of that, `pkgs.hydra-unstable` is now compiled
against `pkgs.nixFlakes` and currently requires a patch since Hydra's
master doesn't compile[2] atm.
[1] https://github.com/NixOS/hydra/pull/730
[2] https://github.com/NixOS/hydra/pull/732
Upgrades Hydra to the latest master/flake branch. To perform this
upgrade, it's needed to do a non-trivial db-migration which provides a
massive performance-improvement[1].
The basic ideas behind multi-step upgrades of services between NixOS versions
have been gathered already[2]. For further context it's recommended to
read this first.
Basically, the following steps are needed:
* Upgrade to a non-breaking version of Hydra with the db-changes
(columns are still nullable here). If `system.stateVersion` is set to
something older than 20.03, the package will be selected
automatically, otherwise `pkgs.hydra-migration` needs to be used.
* Run `hydra-backfill-ids` on the server.
* Deploy either `pkgs.hydra-unstable` (for Hydra master) or
`pkgs.hydra-flakes` (for flakes-support) to activate the optimization.
The steps are also documented in the release-notes and in the module
using `warnings`.
`pkgs.hydra` has been removed as latest Hydra doesn't compile with
`pkgs.nixStable` and to ensure a graceful migration using the newly
introduced packages.
To verify the approach, a simple vm-test has been added which verifies
the migration steps.
[1] https://github.com/NixOS/hydra/pull/711
[2] https://github.com/NixOS/nixpkgs/pull/82353#issuecomment-598269471
* nixos/buildkite: drop user option
This reverts 8c6b1c3eaa.
Turns out, buildkite-agent has logic to write .ssh/known_hosts files and
only really works when $HOME and the user homedir are in sync.
On top of that, we provision ssh keys in /var/lib/buildkite-agent, which
doesn't work if that other users' homedir points elsewhere (we can cheat
by setting $HOME, but then getent and $HOME provide conflicting
results).
So after all, it's better to only run the system-wide buildkite agent as
the "buildkite-agent" user only - if one wants to run buildkite as
different users, systemd user services might be a better fit.
* nixosTests.buildkite-agent: add node with separate user and no ssh key
This gets passed to BUILDKITE_SHELL, which will specify the shell being
used to executes script in.
Defaults to `${pkgs.bash}/bin/bash -e -c`, matching how buildkite
behaves on other distros.
SSH public keys aren't needed to clone private repos, and if we only
need to configure a single attribute, there's no need for the "openssh"
attrset anymore.
This applies [hydra PR #432](https://github.com/NixOS/hydra/pull/432)
to the NixOS module in nixpkgs:
```
commit 4efd078977e5ea20e1104783efc324cba11690bc
Author: Bas van Dijk <v.dijk.bas@gmail.com>
Date: Sun Dec 11 15:35:38 2016 +0100
Only set buildMachinesFiles when nix.buildMachines is defined
```
The following commit from 2016 in hydra removed the `--option
build-use-substitutes` from the hydra-queue-runner service:
```
commit ee2e9f5335c8c0288c102975b506f6b275793cfe
Author: Eelco Dolstra <edolstra@gmail.com>
Date: Fri Oct 7 20:23:05 2016 +0200
Update to reflect BinaryCacheStore changes
BinaryCacheStore no longer implements buildPaths() and ensurePath(),
so we need to use copyPath() / copyClosure().
```
It would be better if the hydra module in NixOS matches the upstream
module.
During the last update, `hydra-notify` was rewritten as a daemon which
listens to postgresql notifications for each build[1]. The module
uses the `hydra-notify.service` unit from upstream's Hydra module and
the VM test ensures that email notifications are sent properly.
Also updated `hydra-init.service` to install `pg_trgm` on a local
database if needed[2].
[1] c7861b85c4
[2] 8a0a5ec3a3
Currently there are two calls to curl in the reloadScript, neither which
check for errors. If something is misconfigured (like wrong authToken),
the only trace that something wrong happened is this log message:
Asking Jenkins to reload config
<h1>Bad Message 400</h1><pre>reason: Illegal character VCHAR='<'</pre>
The service isn't marked as failed, so it's easy to miss.
Fix it by passing --fail to curl.
While at it:
* Add $curl_opts and $jenkins_url variables to keep the curl command
lines DRY.
* Add --show-error to curl to show short error message explanation when
things go wrong (like HTTP 401 error).
* Lower-case the $CRUMB variable as upper case is for exported environment
variables.
The new behaviour, when having wrong accessToken:
Asking Jenkins to reload config
curl: (22) The requested URL returned error: 401
And the service is clearly marked as failed in `systemctl --failed`.
@cleverca found this bug in the declarative hooks config. Any shell
variables referenced in a hook script would get expanded by the hooks
directory builder.
Prevent variable expansion by quoting the here doc limit string.
Pass the -L flag to curl to make it follow redirects. This fixes an
issue I found when setting up reverse proxy for Jenkins. Without this
fix, the returned HTTP code was stuck at 302, making postStart fail the
service (it expects 200 or 403).
Also removes configText, functionality is now provided more conveniently by configOptions.
Keep in mind that this breaks compatibility with previous configurations,
configFile provides a means to protect the CI token from being written into the nix store.
The current behavior was for gitlab-runner is to immediately terminate when there
was a restart required. This can lead to aborted builds and is annoying to users.
By enabling graceful mode gitlab-runner will wait for all builds to finish before
terminating. The disadvantage is that a nixos-rebuild switch needs to wait till
all jobs are done. Because of that it is not enabled by default.
- adds distro dependency
- buildbot nodaemon in service module
- fakerepo for module tests
- service module parameter fixup
- tested on nixos
- tested on darwin
1) The forking behavior of `buildbot start` is temporarily broken for
mysterious reasons that I'm still looking into
2) Let systemd do the forking: no point in using two different process
startup wait loops
The nixbld group belongs to nix-daemon and you really don't want to be
in it. If you are in it, nix-daemon will kill your processes when you
least expect it :)
The extraOptions option has default values which seems surprising. This
moves those values to startupOptions (which is what gocd-agent uses) and
empties out the default extraOptions.
The gocd-agent startupOptions description was also changed to remove the
mention of the example (given there isn't one).
- Agent now takes a full URL to the Go.CD server
- Instruct the agent to attempt restart every 30s upon failure
- Test's Accept header did not match the server's expectation
- Replace the tests' complex Awk matches with calls to `jq`
Update gocd-agent package version to 16.6.0-3590 including new sha. Modify heapSize
and maxMemory mkOption to accurately reflect their intended purpose of configuring
initial java heap sizes.
Update gocd-server package version to 16.6.0-3590 including new sha. Modify heapSize
and maxMemory mkOption to accurately reflect their intended purpose of configuring
initial java heap sizes.
GoCD is an open source continuous delivery server specializing in advanced workflow
modeling and visualization. Update maintainers list to include swarren83. Update
module list to include gocd agent and server module. Update packages list to include
gocd agent and server package. Update version, revision and checksum for GoCD
release 16.5.0.
As pointed out by @danbst, the tomcat NixOS module expects packages
listed in services.tomcat.webapps to either be direct .war file paths or
have .war files inside a "webapps" directory.
Commit 4075c10a59
("jenkins: move .war file from $out to $out/lib/jenkins.war") broke
jenkins + tomcat. Fix it by moving jenkins.war to $out/webapps/.
Fixes#14137, also known as:
$ nix-shell -p jenkins
bash: source: /nix/store/ln1yw6c2v8bb2cjqfr1z5aqcssw054wa-jenkins-2.3:
cannot execute binary file
[nix-shell exited with error]
The problem is that jenkins.war is not installed inside the directory
$out, but rather _as the file_ $out. Fix it by moving the file to
$out/lib/jenkins.war.
While at it, move buildCommand so that the "meta" section is at the end
of the expression (standard style), and quote shell variables.
This was originally removed in d4d0e449d7.
The intent was not to maintain hydra expression at two places.
Nowadays we have enough devs to maintain this despite copy/pasta.
This should encourage more people to use Hydra, which is a really
great piece of software together with Nix.
Tested a deploy using https://github.com/peti/hydra-tutorial
* Perform HTTP HEAD request instead of full GET (lighter weight)
* Don't log output of curl to the journal (it's noise/debug)
* Use explicit http:// URL scheme
* Reduce poll interval from 10s to 2s (respond to state changes
quicker). Probably not relevant on boot (lots of services compete for
the CPU), but online service restarts/reloads should be quicker.
* Pass --fail to curl (should be more robust against false positives)
* Use 4 space indent for shell code.
The current postStart code holds Jenkins off the "started" state until
Jenkins becomes idle. But it should be enough to wait until Jenkins
start handling HTTP requests to consider it "started".
More reasons why the current approach is bad and we should remove it,
from @coreyoconnor in
https://github.com/NixOS/nixpkgs/issues/14991#issuecomment-216572571:
1. Repeatedly curling for a specific human-readable string to
determine "Active" is fragile. For instance, what happens when jenkins
is localized?
2. The time jenkins takes to initializes is variable. This (at least
used to) depend on the number of jobs and any plugin upgrades requested.
3. Jenkins can be requested to restart from the UI. Which will not
affect the status of the service. This means that the service being
"active" does not imply jenkins is initialized. Downstream services
cannot assume jenkins is initialized if the service is active. Might
as well accept that and remove the initialized test from service
startup.
Fixes#14991.
- add missing types in module definitions
- add missing 'defaultText' in module definitions
- wrap example with 'literalExample' where necessary in module definitions
I think the name 'listenAddress' is more descriptive. Other NixOS
modules that define 'host' either use it as listen address or as address
a client connects to. listenAddress is unambiguous.
The addition of 'host' was added earlier today[1], so not bothering with
./nixos/modules/rename.nix.
[1]: 44ea184997 ("jenkins ci enhancement: add port and prefix option")
As named these options enable to specify a bind host and url prefix
to be used by jenkins. Adding these options in the config rather than
using extra arguments allows us to re-use those information in other
services using jenkins such as jenkins-job-builder or a reverse proxy.
The most complex problems were from dealing with switches reverted in
the meantime (gcc5, gmp6, ncurses6).
It's likely that darwin is (still) broken nontrivially.
This option allows to define (declarative) Jenkins jobs, using Jenkins
Job Builder (JJB) as backend.
Example:
services.jenkins = {
enable = true;
jobBuilder = {
enable = true;
yamlJobs = ''
- job:
name: jenkins-job-test
builders:
- shell: echo 'Hello world!'
'';
};
};
Jobs can be defined using YAML, JSON and Nix.
Note that it really is declarative configuration; if you remove a
previously defined job, the module will remove the jobdir under
$JENKINS_HOME.
Jobs managed through the Jenkins WebUI (or by other means) are not
touched by this module.
Changes v1 -> v2:
* add nixJobs
* let jsonJobs take a list of strings (allows merge)
* 4 space indent in shell code
Jenkins gets (by default) an additional environment of
{ NIX_REMOTE = "daemon"; }
This has the following problems:
1. NIX_REMOTE disappears when users specify additional environment
variables, because defaults have low merge priority.
2. nix cannot be used without additional NIX_PATH envvar, which is
currently missing.
3. If you try to use HTTPS, you'll see that jenkins lacks
SSL_CERT_FILE envvar, causing it to fail.
This commit adds config.environment.sessionVariables and NIX_REMOTE to
the set of variables that are always there for jenkins, making nix and
HTTPS work out of the box.
services.jenkins.environment is now empty by default.
Using pkgs.lib on the spine of module evaluation is problematic
because the pkgs argument depends on the result of module
evaluation. To prevent an infinite recursion, pkgs and some of the
modules are evaluated twice, which is inefficient. Using ‘with lib’
prevents this problem.
Uses standard NixOS user config merging.
Work in progress: The slave config does not actually start the slave agent. This just configures a
jenkins user if required. Bare minimum to enable a nice jenkins SSH slave.
By default the jenkins server is executed under the user "jenkins". Which can be configured using
users.jenkins.* options. If a different user is requested by changing services.jenkins.user then
none of the users.jenkins options apply.
This patch does not include jenkins slave configuration. Some config options will probably change
when this is implemented.
Aspects like the user and environment are typically identical between slave and master. The service
configs are different. The design is for users.jenkins to cover the shared aspects while
services.jenkins and services.jenkins-slave cover the master and slave specific aspects,
respectively.
Another option would be to place everything under services.jenkins and have a config that selects
master vs slave.
