Python 3 doesn't like this:
======================================================================
ERROR: test_001 (__main__.TestNmcli)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./clients/tests/test-client.py", line 785, in f
self._nm_test_post()
File "./clients/tests/test-client.py", line 767, in _nm_test_post
content_new = ''.join([r['content'] for r in results])
TypeError: sequence item 0: expected str instance, bytes found
Recent Python versions warn about this:
./clients/tests/test-client.py:569: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif lang is de:
./clients/tests/test-client.py:572: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif lang is pl:
And rightly so: https://bugs.python.org/issue34850
Instead of straight-out rejecting extra parameters for various nmcli
sub-commands (such as "nmcli dev status", "nmcli dev wifi rescan" or
"nmcli dev wifi connect", etc.), we just print a warning and go ahead.
This is unhelpful. In case the user makes a typo, we'll do the wrong
thing and possibly even drown the warning in the output.
While at that, let's make the error message consistent. One less string
to translate.
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/merge_requests/217
I saw this timeout reached in our gitlab-ci. I think it was due to the machine
being busy and taking more than 2 seconds. Assuming the timeout was just too short,
increase it to 4 seconds.
"ipv4.dns-options" and "ipv6.dns-options" are special, in that they can
be either unset/default or an empty list of options. That means, nmcli
must treat these two options differently.
For the (terse) getter, it returns "" for default and " " for empty.
The setter must likewise support and distingish between these two cases.
Cleanup the handling of such options. This only applies to properties of
type "multilist". Hence, add multilist.clear_emptyunset_fcn() handler
that indicates that the property is of that kind.
Try:
nmcli connection modify "$PROFILE" ipv4.dns-options ''
nmcli connection modify "$PROFILE" ipv4.dns-options ' '
and compare the output:
nmcli connection show "$PROFILE" | sed -n '/ipv4.dns-options/ s/.*/<\0>/p'
nmcli -t connection show "$PROFILE" | sed -n '/ipv4.dns-options/ s/.*/<\0>/p'
nmcli -o connection show "$PROFILE" | sed -n '/ipv4.dns-options/ s/.*/<\0>/p'
Leak detection adds unhelpful messages to the stderr of nmcli, making
tests fail. For example:
=================================================================
==17156==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 256 byte(s) in 2 object(s) allocated from:
#0 0x7f08c7e27c88 in realloc (/lib64/libasan.so.5+0xefc88)
#1 0x7f08c7546e7d in g_realloc (/lib64/libglib-2.0.so.0+0x54e7d)
On failure the self.fail() message often appears in the middle of the
diff between expected/actual output, making it hard to read. Since
print() output goes to stdout (which is buffered) and self.fail() to
stderr (which is not), flush stdout before printing the failure
message to ensure the two don't mix.
Instead of re-fetching the UUID each time with a D-Bus call,
cache/memoize it.
On my system, this improves
$ make check-local-clients-tests-test-client
from 20.9 to 20.4 seconds (- 2.6%). That is not stellar, but noticible
enough to warrant this simple change.
These nmcli calls are inside a Util.iter_nmcli_output_modes()
loop, hence, --terse will passed to them already.
Specifying --terse more than once, causes nmcli to fail.
We didn't actually want to test how nmcli rejects such
duplicate arguments. Adjust the test.
Add more tests for various output modes of nmcli.
This most interestingly includes a terse output for
$ nmcli device status
Which was not tested previously (but a later commit will change
behavior here).
This blows up the number of tests even further.
Previously, the test invoked nmcli 850 times (taking ~15 seconds
ony my machine), afterwards, it is 1334 times (taking ~20 seconds).
No doubt, this seems excessive and questionable.
However, I maintain that the computer doesn't mind running a lot of
redundant tests. The important thing is, that we cover as many cases
as possible. Trying at the same time to minimize the number of tests
by avoiding duplicates and redundant tests, is just a lot of manual
labor. Manual labor that must be repeated whenever new tests are added, to
ensure that this test case is not yet covered. So, I am fine
with the large number of tests. Let the computer do the work.
Instead of letting each nmcli run write an individual .expected file,
combine the output of multiple runs in one file (per test).
Advantages:
- since there is a very large number of tests, having a file for each
tests is cumbersome. For example, since they are all added to
$(EXTRA_DIST) (and since we use non-recursive make), autoconf can
easily hit a length limit when processing all the file names.
- previously, whenever we added tests, all .expected files shifted
and the diff was large. Now, there is a chance that the diff is
smaller and more accurate.
Commands that fail with G_DEBUG=fatal-warnings produce unstable
output like
(process:10743): GLib-CRITICAL **: 16:29:13.635: g_ptr_array_add: assertion 'rarray' failed
To workaround that, add a new option for marking the output as unstable.
An alternative might be to extend the replace_stdout, replace_stderr
arguments to support more powerful matching, like by specifying regular
expression for replacing. However, it's just too complicated. Add a simpler
workaround by passing _UNSTABLE_OUTPUT.
How does `nmcli -f ALL dev show $DEV` look, if it references
a connection that is invisible to the user?
Note in the output:
CONNECTIONS.AVAILABLE-CONNECTIONS[1]: (null) | (null)
At several places, "test-networkmanager-service.py" uses generated numbers
with a defined seed. For example, generated connection's UUID is
generated in a predictable, but randomized way (if you forgive the
inprecise use of the word "random" in context of using a deterministic
seed).
Aside the connection's UUID, this becomes more interesting in the next commit
where the stub server generates a list of IP and DHCP settings in a predictable
randomized way.
For "clients/tests" we spawn the test service multiple times, but also
create similar environments by calling init_001(). This is done for
convenience, where out of lazyness all the tests share one setup. But it's
still a good idea that these tests generate slightly different setups,
wherever applicable. this increases the possible setups which get tested.
For example, the number of static IPv4 addresses (the following commit) is
interested to explicitly test for zero or a non-zero number of
addresses. If all tests happen to use the same seed, the tests are expected
to also generate the same number of addresses, and we miss an opportunity to
hit interesting test cases.
There is still no guarantee that all interesting cases are hit, the chances are just
better. The approach of generating the setup randomly, does not preclude that
the stub-server allows to explicitly configure the setup. However, due to the
sheer number of combinations that might be interesting to test, it's much simpler
to rely on some randomization and have the justifid hope we catch interesting cases.
Also in terms of runtime of the test, the cli unit tests should complete within
few seconds. Testing every combination would result in huge tests and long runtimes.
Also, the patch refactors generating random numbers in
"test-networkmanager-service.py". For example, it introduces
Util.RandomSeed(), which can be used to generate a sequence of different
random numbers. It works by having an internal state and a counter which is
combined to chain the seed and generate different numbers on each call.
Just to give it some variety. Also, note how the message from the
server cannot be translated. Which is the case with real NetworkManager
as well, and is a major usability issue.
The real NetworkManager service has a clear understanding how a valid
connection looks like. This is what nm_connection_verify() returns.
Let also our stub-service verify connections the same way.
Note that this is cumbersome, because the stub service uses python's
dbus module, while libnm only accepts creating NMConnection instances
from GVariant. Thus, we need to a cumbersome conversion first.
It would be better if test-networkmanager-service.py would also expose
normalized connections on D-Bus. But that requires the inverse converion
from GVariant to python dbus.
Most nmcli calls from clients/tests don't change the server's state.
Hence, they can easily run in parallel.
Run tests in parallel. No longer handle one nmcli invocation after the other.
Instead, spawn groups of processes in parallel, and track the pending jobs.
Only at certain synchronization points we call self.async_wait() to
wait for all previous jobs to complete.
This reduces the test time on my machine from 7 to 3 seconds. Arguably,
that matters less during a full `make check -j 8`, because the entire
set of tests anyway takes longer than 7 seconds. So when running the
entire test suite, the machine is kept busy anyway. It matters however
for manual invocations.
The developer can re-generate .expected files with
$ NM_TEST_REGENERATE=1 ./clients/tests/test-client.py
Note that these files are also dist-ed, so that the tests also work
from a source-tarball. For that, we need to add them to EXTRA_DIST.
Previously, this was done manually in the base Makefile.am file. This
was cumbersome, because when adding a new test, the developer would need
to manually add the files.
Now, let the test (with NM_TEST_REGENERATE=1) also generate a makefile
part.
call_nmcli_l() would test for 3 languages: 'C', 'de', and 'pl'. There
is no fundamental difference between 'de' and 'pl', so there is no need
to test for two languages.
