Iterating hash tables gives an undefined order. Often we want to have
a stable order, for example when printing the content of a hash or
when converting it to a "a{sv}" variant.
How to achieve that best? I think we should only iterate the hash once,
and not require additional lookups. nm_utils_named_values_from_strdict()
achieves that by returning the key and the value together. Also, often
we only need the list for a short time, so we can avoid heap allocating
the list, if it is short enough. This works by allowing the caller to
provide a pre-allocated buffer (usually on the stack) and only as fallback
allocate a new list.
When parsing user input if is often convenient to allow stripping whitespace.
Especially with escaped strings, the user could still escape the whitespace,
if the space should be taken literally.
Add support for that to nm_utils_buf_utf8safe_unescape().
Note that this is not the same as calling g_strstrip() before/after
unescape. That is, because nm_utils_buf_utf8safe_unescape() correctly
preserves escaped whitespace. If you call g_strstrip() before/after
the unescape, you don't know whether the whitespace is escaped.
We want to use the function to unescape (compress) secrets. As such, we want
to be sure that no secrets are leaked in memory due to growing the buffer with
realloc. In fact, reallocation should never happen. Assert for that.
As reallocation cannot happen, we could directly fill a buffer with
API like nm_utils_strbuf_*(). But NMStrBuf has low overhead even in this
case.
Add nm_utils_invoke_on_timeout() beside nm_utils_invoke_on_idle().
They are fundamentally similar, except one schedules an idle handler
and the other a timeout.
Also, use the current g_main_context_get_thread_default() as context
instead of the singleton instance. That is a change in behavior, but
the only caller of nm_utils_invoke_on_idle() is the daemon, which
doesn't use different main contexts. Anyway, to avoid anybody being
tripped up by this also change the order of arguments. It anyway
seems nicer to first pass the cancellable, and the callback and user
data as last arguments. It's more in line with glib's asynchronous
methods.
Also, in the unlikely case that the cancellable is already cancelled
from the start, always schedule an idle action to complete fast.
NMStrBuf is not an opaque structure, so that we can allocate it on the
stack or embed it in a struct.
But most of the fields should not be touched outside of the
implementation.
Also, "len" and "allocated" fields may be accessed directly, but
they should not be modified.
Rename the fields to make that clearer.
Add flags to explicitly escape leading or trailing spaces. Note
that we were already escaping trailing spaces.
This will be used later when supporting backslash escapes for
option parameters for nmcli (vpn.data).
nm_utils_buf_utf8safe_unescape() is almost the same as g_strcompress(),
with the only difference is that if the string contains NUL escapes "\000",
it will be handled correctly.
In other words, g_strcompress() and nm_utils_str_utf8safe_unescape() can only
unescape values, that contain no NUL escapes. That's why we added our
own binary unescape function.
As we already have our g_strcompress() variant, use it. It just gives it more
testing and usage. Also, we have full control over it's behavior. For example,
g_strcompress() issues a g_warning() when encountering a trailing '\\'. I
think this makes it unsuitable to unescape untrusted data. Either the function
should fail, or just make the best of it. Currently, our implementation
does the latter.
Our own implementation of a string buffer like GString.
Advantages (in decreasing relevance):
- Since we are in control, we can easily let it nm_explicit_bzero()
the memory. The regular GString API cannot be used in such a case.
While nm_explicit_bzero() may or may not be of questionable benefit,
the problem is that if the underlying API counteracts the aim of
clearing memory, it gets impossible. As API like NMStrBuf supports
it, clearing memory is a easy as enable the right flag.
This would for example be useful for example when we read passwords
from a file or file descriptor (e.g. try_spawn_vpn_auth_helper()).
- We have API like
nmp_object_to_string (const NMPObject *obj,
NMPObjectToStringMode to_string_mode,
char *buf,
gsize buf_size);
which accept a fixed size output buffer. This has the problem of
how choosing the right sized buffer. With NMStrBuf such API could
be instead
nmp_object_to_string (const NMPObject *obj,
NMPObjectToStringMode to_string_mode,
NMStrBuf *buf);
which can automatically grow (using heap allocation). It would be
easy to extend NMStrBuf to use a fixed buffer or limiting the
maximum string length. The point is, that the to-string API wouldn't
have to change. Depending on the NMStrBuf passed in, you can fill
an unbounded heap allocated string, a heap allocated string up to
a fixed length, or a static string of fixed length. NMStrBuf currently
only implements the unbounded heap allocate string case, but it would
be simple to extend.
Note that we already have API like nm_utils_strbuf_*() to fill a buffer
of fixed size. GString is not useable for that (efficiently), hence
this API exists. NMStrBuf could be easily extended to replace this API
without usability or performance penalty. So, while this adds one new
API, it could replace other APIs.
- GString always requires a heap allocation for the container. In by far
most of the cases where we use GString, we use it to simply construct
a string dynamically. There is zero use for this overhead. If one
really needs a heap allocated buffer, NMStrBuf can easily embedded
in a malloc'ed memory and boxed that way.
- GString API supports inserting and removing range. We almost never
make use of that. We only require append-only, which is simple to
implement.
- GString needs to NUL terminate the buffer on every append. It
has unnecessary overhead for allowing a usage of where intermediate
buffer contents are valid strings too. That is not the case with
NMStrBuf: the API requires the user to call nm_str_buf_get_str() or
nm_str_buf_finalize(). In most cases, you would only access the string
once at the end, and not while constructing it.
- GString always grows the buffer size by doubling it. I don't think
that is optimal. I don't think there is one optimal approach for how
to grow the buffer, it depends on the usage patterns. However, trying
to make an optimal choice here makes a difference. QT also thinks so,
and I adopted their approach in nm_utils_get_next_realloc_size().
When growing a buffer by appending a previously unknown number
of elements, the often preferable strategy is growing it exponentially,
so that the amortized runtime and re-allocation costs scale linearly.
GString just always increases the buffer length to the next power of
two. That works.
I think there is value in trying to find an optimal next size. Because
while it doesn't matter in terms of asymptotic behavior, in practice
a better choice should make a difference. This is inspired by what QT
does ([1]), to take more care when growing the buffers:
- QString allocates 4 characters at a time until it reaches size 20.
- From 20 to 4084, it advances by doubling the size each time. More
precisely, it advances to the next power of two, minus 12. (Some memory
allocators perform worst when requested exact powers of two, because
they use a few bytes per block for book-keeping.)
- From 4084 on, it advances by blocks of 2048 characters (4096 bytes).
This makes sense because modern operating systems don't copy the entire
data when reallocating a buffer; the physical memory pages are simply
reordered, and only the data on the first and last pages actually needs
to be copied.
Note that a QT is talking about 12 characters, so we use 24 bytes
head room.
[1] https://doc.qt.io/qt-5/containers.html#growth-strategies
Sometimes these function may set errno to unexpected values like EAGAIN.
This causes confusion. Avoid that by using our own wrappers that retry
in that case. For example, in rhbz#1797915 we have failures like:
errno = 0;
v = g_ascii_strtoll ("10", 0, &end);
if (errno != 0)
g_assert_not_reached ();
as g_ascii_strtoll() would return 10, but also set errno to EAGAIN.
Work around that by using wrapper functions that retry. This certainly
should be fixed in glib (or glibc), but the issues are severe enough to
warrant a workaround.
Note that our workarounds are very defensive. We only retry 2 times, if
we get an unexpected errno value. This is in the hope to recover from
a spurious EAGAIN. It won't recover from other errors.
https://bugzilla.redhat.com/show_bug.cgi?id=1797915
Depending on the type, OVS interfaces also have a corresponding netdev
in kernel (e.g. type "internal" does, type "patch" does not).
Such a case is neither NMU_IFACE_OVS nor NMU_IFACE_KERNEL (alone). There should
be a special type to represent those cases.
Add NMU_IFACE_OVS_OR_KERNEL for that.
nm_utils_ifname_valid() is to validate "connection.interface-name"
property. But the exact validation depends on the connection type.
Add "NMU_IFACE_ANY" to validate the name to check whether it would be
valid for any connection type.
This is for completeness and for places where the caller might not know
the connection type.
"all" and "default" never works.
"bonding_masters" works if you unload the bonding module. Well,
that should not really be called working...
Reject these names.
Generally, it's dangerous to reject values that were accepted
previously. This will lead to NetworkManager being unable to load
a profile from disk, which was loadable previously.
On the other hand, kernel would not have treated this setting as
it was intended. So, I would argue that the such a setting was not
working (as intended) anyway.
We can only hope that users don't configure arbitrary interface names.
It generally isn't a good idea to do, so "breaking" such things is less
of a concern.
The interface name might come from the command line or from a filename
(like during nm_vpn_wireguard_import()). It's not clear that this
is valid UTF-8. Asserting against that requires the caller to ensure
that the encoding is valid. That is cumbersome, especially since we anyway
check. Just report a regular error.
quoting 'man ovs-vswitchd.conf.db':
"The name must be alphanumeric and must not contain forward or backward
slashes."
OVS actually accepts a wider range of chars (all printable UTF-8 chars),
NetworkManager restricts this to ASCII char as it's a safer option for
now since OVS is not well documented on this matter.
https://bugzilla.redhat.com/show_bug.cgi?id=1788432
Fixes: e7d72a14f6 ('libnm-core: use different ifname validation function for OVS bridges, ports and interfaces')
The only affected caller is nm_modem_manager_get_modems(), which
is used by NMDeviceBt for DUN connections. This is rather bad.
Fixes: 4154d9618c ('bluetooth: refactor BlueZ handling and let NMBluezManager cache ObjectManager data')
Fixes: e688e70b37 ('shared: add nm_utils_hash_values_to_array() helper')
Most callers would pass FALSE to nm_utils_error_is_cancelled(). That's
not very useful. Split the two functions and have nm_utils_error_is_cancelled()
and nm_utils_error_is_cancelled_is_disposing().
There are crashes where this assertion fails, but it's not clear
how that could happen (because the input text seems a usual IPv4 address).
Try to collect some more information about what failed. It's only
enabled with NM_MORE_ASSERTS anyway.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1797915
The abbreviations "ns" and "ms" seem not very clear to me. Spell them
out to nsec/msec. Also, in parts we already used the longer abbreviations,
so it wasn't consistent.
inet_aton() also supports IPv4 addresses in octal (with a leading '0')
or where not all 4 digits of the address are present.
Add nm_utils_parse_inaddr_bin_full() to optionally fallback to
parse the address with inet_aton().
Note taht inet_aton() also supports all crazy formats, including
ignoring trailing garbage after a whitespace. We don't want to accept
that in general.
Note that even in legacy format we:
- accept everything that inet_pton() would accept
- additionally, we also accept some forms which inet_aton() would
accept, but not all.
That means, the legacy format that we accept is a superset of
inet_pton() and a subset of inet_aton(). Which is desirable.
We will rework NMClient entirely. Then, the synchronous initialization will also use
the asynchronous code paths. The difference will be that with synchronous initialization,
all D-Bus interaction will be done with an internal GMainContext as current thread default,
and that internal context will run until initialization completes.
Note that even after initialization completes, it cannot be swapped back to the user's
(outer) GMainContext. That is because contexts are essentially the queue for our
D-Bus events, and we cannot swap from one queue to the other in a race
free manner (or a full resync). In other words, the two contexts are not in sync,
so after using the internal context NMClient needs to stick to that (at least, until
the name owner gets lost, which gives an opportunity to resync and switch back to the
user's main context).
We thus need to hook the internal (inner) GMainContext with the user's (outer) context,
so when the user iterates the outer context, events on the inner context get dispatched.
Add nm_utils_g_main_context_create_integrate_source() to create such a GSource for
integrating two contexts.
Note that the use-case here is limited: the integrated, inner main context must
not be explicitly iterated except from being dispatched by the integrating
source. Otherwise, you'd get recursive runs, possible deadlocks and general
ugliness. NMClient must show restrain how to use the inner context while it is
integrated.
