logrotate.timer is enough for rotating logs. Enabling logrotate.service would
make the service start on every configuration switch, leading to tests failure when
logrotate is enabled.
Also update test to make sure the timer is active and runs the service
on date change.
logrotate global options only affect rules following them - as such,
services.logrotate.extraConfig being added last makes the option only
useful for adding new paths but not for setting global options (e.g.
'dateext' so all logs are rotate with a date suffix).
Moving this first solves this problem, and we can then use this instead
of default paths config to append missingok/notifempty.
wtmp and btmp are created by systemd, so the rules are more appropriate there.
They can be disabled explicitly with something like
services.ogrotate.paths = {
"/var/log/btmp".enable = false;
"/var/log/wtmp".enable = false;
};
if required.
* elk7: 7.11.1 -> 7.16.1
* nixosTests.elk: Improve reliability and compatibility with ELK 7.x
- Use comparisons in jq instead of grepping
- Match for `.hits.total.value` if version >= 7, otherwise it always
passes
- Make curl fail if requests fails
* nixos/filebeat: Add initial module and test
Filebeat is an open source file harvester, mostly used to fetch logs
files and feed them into logstash.
This module can be used instead of journalbeat if used with
`filebeat7` and configured with the `journald` input.
* python3Packages.parsedmarc.tests: Fix breakage
- Don't use the deprecated elasticsearch7-oss package
- Improve jq query robustness and add tracing
* rl-2205: Note the addition of the filebeat service
* elk6: 6.8.3 -> 6.8.21
The latest version includes a fix for CVE-2021-44228.
* nixos/journalbeat: Add a loose dependency on elasticsearch
Avoid unnecssary back-off when elasticsearch is running on the same
host.
On reboots and shutdowns promtail blocks for at least 90 seconds,
because it would still try to deliver log messages for loki, which isn't
possible when the network has already gone down.
Upstreams example unit also uses a ten seconds timeout, something which
has worked pretty well for me as well.
Since the upstream graylogctl script will prefer finding its java
executable based on JAVA_HOME, we now set this instead of PATH in
order to allow it to find the JRE. By setting it conditionally on it
not already being set, we allow selecting a different JRE at runtime.
We also explicitly use openjdk11, which supports the
UseConcMarkSweepGC option which graylog insists on using.
A centralized list for these renames is not good because:
- It breaks disabledModules for modules that have a rename defined
- Adding/removing renames for a module means having to find them in the
central file
- Merge conflicts due to multiple people editing the central file
Until now the startup failed with an error like this:
```
com.github.joschi.jadconfig.ValidationException: Parent path /var/lib/graylog/server for Node ID file at /var/lib/graylog/server/node-id is not a directory
```
This happens since `graylog.service` ensures that `/var/lib/graylog`
exists, however it doesn't take care of the directory for
`cfg.nodeIdFile`.
Included changes:
* upstream repository has moved, URLs changed accordingly
* journaldriver bumped to new upstream release
The new release includes an important workaround for an issue that
could cause log-forwarding to fail after service restarts due to
invalid journal cursors being persisted.
Adds a module for running the journaldriver log forwarding agent via
systemd.
The agent can be deployed on both GCP instances and machines hosted
elsewhere to forward all logs from journald to Stackdriver Logging.
Consult the module options and upstream documentation for more
information.
Implementation notes:
* The service unit is configured to use systemd's dynamic user feature
which will let systemd set up the state directory and appropriate
user configuration at unit launch time instead of hardcoding it.
* The module depends on `network-online.target` to prevent a situation
where journaldriver is failing and restarting multiple times before
the network is online.
* Add kibana5 and logstash5
* Upgrade the elastic beats to 5.4
* Make sure all elastic products use the same version
(see elk5Version)
* Add a test for the ELK stack
Note: the option to configure the watchdog timeout seems to be gone
in the 2.3 series of Logstash. It complains about an unknown option
and it is not in the source anymore. I am thus removing this
configuration option to adjust the service to these changes, too.
