Script to do some anonymization to NetworkManager logs. It does
very basic stuff so it shouldn't be trusted without manually
reviewing the logs, but it can still be useful to replace lot
of potentially sensitive data.
What it masks by default:
- MAC addresses
- Public IP addresses
- Hostnames detected from `hostname` command and some known
log messages from NM.
- Hostnames ending in some common domains such as .com or .org
- Hostnames specified via --hostname argument
What it can mask but it doesn't by default:
- Private IPs
Options like --show-macs and --hide-private-ips can override the default
behaviour.
Note that masking IP addresses can make difficult to analyze routing
problems, and trying to be smart analyzing the defined routes from the
logs or from `ip route` can lead to even worse results. Because of this,
if routing problems need to be analyzed, --show-public-ips need to be
passed.
Instead of doing the broken `podman run` and `podman start` approach,
build an image ("nm-code-format:f38"), cache it, and use it to run
"nm-code-format.sh" via `podman run`. We should build and keep a
container image, not a container.
The benefit is that this allows to hand over the command line arguments
to "nm-code-format.sh". In particular the "-u" and "-F" options, which
are life savers.
This means,
$ contrib/scripts/nm-code-format-container.sh -u
works.
Try also
$ contrib/scripts/nm-code-format-container.sh -h
which tells you that you are running inside the container, and how to
delete/renew the container image.
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1798
From time to time we bump the used clang-format (and Fedora) version.
Previously, we had to change more than one places.
Instead, let the "nm-code-format-container.sh" parse it from ".gitlab-ci/config.yml".
This is the version shipped in Fedora 38. As Fedora 38 is now out, the
core developers switch to it. Our gitlab-ci will also use that as base
image for the check-{patch.tree} tests and to generate the pages. There
is a need that everybody agrees on which clang-format version to use,
and that version should be the one of the currently used Fedora release.
Also update the used Fedora image in "contrib/scripts/nm-code-format-container.sh"
script.
The gitlab-ci still needs update in the following commit. This change
in isolation will break the "check-tree" test.
The actual formatting depends on the version of clang-format. Print the
used version, which is in particular interesting when we get an error in
our gitlab-ci check (which uses the correct version).
Every branch (for example "nm-1-40") has exactly one next branch, from
which patches should be backported (in that example that branch is
"nm-1-42").
While "find-backports" searches all newer branches for patches, it does
not make it clear form where the patch should come from.
That means, if you run the script `contrib/scripts/find-backports origin/nm-1-40`
it will check nm-1-42 and main branch, and might suggest to backport
patches that are only on main, but not "nm-1-42". That would be wrong,
because patches need to first go into nm-1-42, and then backported (from
there) further to nm-1-40.
Print a warning to highlight that.
By default, podman bind mounts a "/etc/resolv.conf" file. That prevents
NetworkManager (inside the container) to update the file, which leads to
warnings in the log and certain NM-ci tests won't pass due to that.
Disable handling of "/etc/resolv.conf" in podman. But also pre-deploy a
default resolv.conf, with the google name server 8.8.8.8. I don't
understand why, but even with "--dns=none", writing "/etc/resolv.conf"
while building the container doesn't take effect. Instead, write a
usable "/etc/resolv.conf" from "/etc/rc.d/rc.local".
On the main branch, we commonly rebase our WIP branches to latest HEAD,
before merging them with "--no-ff". The effect is to have a merge commit
that acts as a parentheses around the set of patches.
When backporting such a branch, we should preserve that structure and
take the merge commit too. We should must use `git cherry-pick -x` to
record the commit IDs of the original patch.
This script helps with that.
Also hook it up in "contrib/scripts/nm-setup-git.sh" to create an alias
for it. This alias has the advantage, of fetching the latest version of
the script from "main" or "origin/main", so it also works on older
branches.
It seems that `meson test` is preferred over `ninja test`. Also, pass
"--print-errorlogs" to meson, and pass "-v" to the build steps.
Note that `ninja test` already ends up calling `meson test
--print-errorlogs`, but it doesn't use "-v", so the logs are truncated.
Like done for autotools. First we run the test without debugging option.
If it fails, we run it again to possibly trigger the failure again and
get better logs.
"debug" is implied when setting NMTST_DEBUG, but not specifying
"no-debug". This change has thus no effect, but it seems clearer to be
explicit.
The "debug" flag affects nmtst_is_debug(). Note that tests *must* not
result in different code paths based on debug, they may only
1) print more debug logging
2) do more assertion checks.
Having more assertion checks can result in different outcome of the
test, that is, that the additional assertion fails first. That is
acceptable, because failing earlier is possibly closer to the issue and
helps debugging. Also, when the additional failure is fixed and passes,
we still will fail at the assertion we are trying to debug.
In particular, an access to nmtst_get_rand*()/nmtst_rand*() must not
depend on nmtst_is_debug(), because then different randomized paths
are taken based on whether debugging is enabled.
The code was just wrong. Usually in gitlab-ci, NMTST_SEED_RANDOM is
unset, so the previous code would not have set it. Which means that our
tests run with NMTST_SEED_RANDOM="0".
Fuzzing (or randomizing tests) is very useful, we should do that for the
unit tests that run in gitlab-ci. Fix this.
But don't let the test choose a random number. Instead, let the calling
script choose it. That is, because we might run the tests more than once
(without debugging and no valgrind; in case of failure return with
debugging; with valgrind). Those runs should use the same seed.
This fixes commit 70487d9ff8 ('ci: randomize tests during our CI'),
but as fixing randomization can break previously running tests, we may
only want to backport this commit after careful evaluation.
When we build a copr image, we run the "nm-copr-build.sh" script.
That script, should honor "LTO=0|1|" to explicitly enable/disable
LTO. Since the copr script only builds a SRPM, which then gets build
we need that the default LTO flag in the SRPM is templated.
Fixes: 0566e9dc63 ('contrib: support disabling "LTO" in "nm-copr-build.sh"')
The "nm-copr-build.sh" script is run by our copr to generate the SRPM of
NetworkManager (via `curl ... | bash`).
Building with LTO takes a long time, for testing it can be nice to disable
that. Add an environment variable for that. It can be used when manually
building an RPM in copr.
NM-ci wants to install a lot of packages when running the first test.
In particular, NM-ci has no nice script that lists all the dependencies,
so it's not immediately clear which packages are required.
Still, install some of those packages so that they are already present
when running the first NM-ci test.
This is the version shipped in Fedora 37. As Fedora 37 is now out, the
core developers switch to it. Our gitlab-ci will also use that as base
image for the check-{patch.tree} tests and to generate the pages. There
is a need that everybody agrees on which clang-format version to use,
and that version should be the one of the currently used Fedora release.
Also update the used Fedora image in "contrib/scripts/nm-code-format-container.sh"
script.
The gitlab-ci still needs update in the following commit. The change
in isolation will break the "check-tree" test.
Cleanup the handling of close().
First of all, closing an invalid (non-negative) file descriptor (EBADF) is
always a serious bug. We want to catch that. Hence, we should use nm_close()
(or nm_close_with_error()) which asserts against such bugs. Don't ever use
close() directly, to get that additional assertion.
Also, our nm_close() handles EINTR internally and correctly. Recent
POSIX defines that on EINTR the close should be retried. On Linux,
that is never correct. After close() returns, the file descriptor is
always closed (or invalid). nm_close() gets this right, and pretends
that EINTR is a success (without retrying).
The majority of our file descriptors are sockets, etc. That means,
often an error from close isn't something that we want to handle. Adjust
nm_close() to return no error and preserve the caller's errno. That is
the appropriate reaction to error (ignoring it) in most of our cases.
And error from close may mean that there was an IO error (except EINTR
and EBADF). In a few cases, we may want to handle that. For those
cases we have nm_close_with_error().
TL;DR: use almost always nm_close(). Unless you want to handle the error
code, then use nm_close_with_error(). Never use close() directly.
There is much reading on the internet about handling errors of close and
in particular EINTR. See the following links:
https://lwn.net/Articles/576478/https://askcodes.net/coding/what-to-do-if-a-posix-close-call-fails-https://www.austingroupbugs.net/view.php?id=529https://sourceware.org/bugzilla/show_bug.cgi?id=14627https://news.ycombinator.com/item?id=3363819https://peps.python.org/pep-0475/
Our GObject structs should be internal API. In which case, we should
embed the private data in the struct themselves (`_priv`) and use the
_NM_GET_PRIVATE() macro. The advantage is better debugability because
following G_TYPE_INSTANCE_GET_PRIVATE() in the debugger is very
cumbersome. Another (less relevant) advantage is better performance.
Thus, warn about uses of g_type_class_add_private() and
G_TYPE_INSTANCE_GET_PRIVATE().
Note that if the struct and is in a header file (which is usually only
necessary when subclassing the type), then the private data should be
an opaque pointer `_priv` instead, and we should use the _NM_GET_PRIVATE_PTR()
macro. In that case, the use of g_type_class_add_private() and
G_TYPE_INSTANCE_GET_PRIVATE() is correct and the warning is false. But
this is only a warning, for the unusual case where we have deep object
hierarchies.
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1396
"-u" calls `git diff -name-only $UPSTREAM` to get a list of files.
However, if $UPSTREAM contains a file that doesn't exist in the current
checkout, we get a non-existing file name and clang-format will fail.
Avoid that, by filtering only files that exist.
Also, pass "--no-renames" option to git-diff.
Reformatting the entire source tree takes quite long. For fast
development it's useful to only check the files that changes on the
current checkout.
For that there was already the "-F|--fast" option, but that only
compared the files that changed compared to HEAD^.
What actually would be useful is to check the files that changed on the
current branch, compared to some upstream commit. Add "-u|--upstream"
option to specify the upstream commit (usually "main").
As a special twist,
./contrib/scripts/nm-code-format.sh -u
is the same as
./contrib/scripts/nm-code-format.sh -u main
We need to mount sysfs, so that `ip netns exec` works.
Do that automatically when starting the system container, via rc.local.
While at it, use `podman build --squash-all` to speedup the building of
the container image.
