The function print_times() skips any time elements which are zero, so
output may lack the ms part. Adjust the sed call dropping anything but
the minutes value to not fail in that case.
Reported-by: Pablo Neira Ayuso <pablo@netfilter.org> Fixes: 255ec36a11525 ("tests: shell: Stabilize sets/reset_command_0 test") Signed-off-by: Phil Sutter <phil@nwl.cc>
Phil Sutter [Thu, 28 Sep 2023 16:19:37 +0000 (18:19 +0200)]
tests: shell: Fix for failing nft-f/sample-ruleset
For whatever reason, my system lacks an entry for 'sip' in
/etc/services. Assuming the service name is not relevant to the test,
just replace it by the respective port number.
tests: shell: fix spurious errors in sets/0036add_set_element_expiration_0
A number of changes to fix spurious errors:
- Add seconds as expiration, otherwise 14m59 reports 14m in minute
granularity, this ensures suficient time in a very slow environment with
debugging instrumentation.
- Provide expected output.
- Update sed regular expression to make 'ms' optional and use -E mode.
Fixes: adf38fd84257 ("tests: shell: use minutes granularity in sets/0036add_set_element_expiration_0") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
netlink_linearize: skip set element expression in map statement key
This fix is similar to 22d201010919 ("netlink_linearize: skip set element
expression in set statement key") to fix map statement.
netlink_gen_map_stmt() relies on the map key, that is expressed as a set
element. Use the set element key instead to skip the set element wrap,
otherwise get_register() abort execution:
The dynamic flag is not exported via JSON, this triggers spurious
ENOTSUPP errors when restoring rulesets in JSON with dynamic flags
set on.
Fixes: 6e45b102650a2 ("nft: set: print dynamic flag when set") Acked-by: Phil Sutter <phil@nwl.cc> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Thomas Haller [Wed, 20 Sep 2023 14:26:08 +0000 (16:26 +0200)]
expression: cleanup expr_ops_by_type() and handle u32 input
Make fewer assumptions about the underlying integer type of the enum.
Instead, be clear about where we have an untrusted uint32_t from netlink
and an enum. Rename expr_ops_by_type() to expr_ops_by_type_u32() to make
this clearer. Later we might make the enum as packed, when this starts
to matter more.
Also, only the code path expr_ops() wants strict validation and assert
against valid enum values. Move the assertion out of
__expr_ops_by_type(). Then expr_ops_by_type_u32() does not need to
duplicate the handling of EXPR_INVALID. We still need to duplicate the
check against EXPR_MAX, to ensure that the uint32_t value can be cast to
an enum value.
[ Remove cast on EXPR_MAX. --pablo ]
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Phil Sutter [Wed, 13 Sep 2023 18:53:41 +0000 (20:53 +0200)]
parser_json: Wrong check in json_parse_ct_timeout_policy()
The conditional around json_unpack() was meant to accept a missing
policy attribute. But the accidentally inverted check made the function
either ignore a given policy or access uninitialized memory.
Parts of the code were unsafe (parsing 'I' format into uint32_t), the
rest just plain wrong (parsing 'o' format into char *tmp). Introduce a
temporary int variable to parse into.
Phil Sutter [Wed, 13 Sep 2023 18:32:37 +0000 (20:32 +0200)]
parser_json: Catch wrong "reset" payload
The statement happily accepted any valid expression as payload and
assumed it to be a tcpopt expression (actually, a special case of
exthdr). Add a check to make sure this is the case.
Standard syntax does not provide this flexibility, so no need to have
the check there as well.
tests: shell: add feature probe for sets with more than one element
Kernels < 5.11 can handle only one expression per element, e.g.
its possible to attach a counter per key, or a rate limiter,
or a quota, but not two at the same time.
Add a probe file and skip the relevant tests if the feature is absent.
Thomas Haller [Mon, 18 Sep 2023 20:27:07 +0000 (22:27 +0200)]
tests/shell: honor NFT_TEST_FAIL_ON_SKIP variable to fail on any skipped tests
The test suite should pass with various kernels and build
configurations. Of course, that means, that some tests will be
gracefully skipped, and we don't treat that as an overall failure.
However, it should be possible to run a specific kernel (net-next?) and
build configuration, where we expect that all tests pass.
Add an option to fail the run, if any tests were skipped. This is to
ensure that we don't have broken tests that never pass.
This will make more sense with automated CI is running, to enable on a
test system and ensure that at least on that system, all tests pass.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Wed, 20 Sep 2023 20:09:19 +0000 (22:09 +0200)]
datatype: return const pointer from datatype_get()
"struct datatype" is for the most part immutable, and most callers deal
with const pointers. That's why datatype_get() accepts a const pointer
to increase the reference count (mutating the refcnt field).
It should also return a const pointer. In fact, all callers are fine
with that already.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Thomas Haller [Wed, 20 Sep 2023 14:26:10 +0000 (16:26 +0200)]
proto: add missing proto_definitions for PROTO_DESC_GENEVE
While at it, make proto_definitions const. For global variables, this
allows the linker to mark the memory as read only. It's just good to do
by default.
Fixes: 156d22654003 ("src: add geneve matching support") Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
datatype: initialize TYPE_CT_EVENTBIT slot in datatype array
Matching on ct event makes no sense since this is mostly used as
statement to globally filter out ctnetlink events, but do not crash
if it is used from concatenations.
Add the missing slot in the datatype array so this does not crash.
Fixes: 2595b9ad6840 ("ct: add conntrack event mask support") Reported-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
datatype: initialize TYPE_CT_LABEL slot in datatype array
Otherwise, ct label with concatenations such as:
table ip x {
chain y {
ct label . ct mark { 0x1 . 0x1 }
}
}
crashes:
../include/datatype.h:196:11: runtime error: member access within null pointer of type 'const struct datatype'
AddressSanitizer:DEADLYSIGNAL
=================================================================
==640948==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fc970d3199b bp 0x7fffd1f20560 sp 0x7fffd1f20540 T0)
==640948==The signal is caused by a READ memory access.
==640948==Hint: address points to the zero page.
sudo #0 0x7fc970d3199b in datatype_equal ../include/datatype.h:196
Fixes: 2fcce8b0677b ("ct: connlabel matching support") Reported-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Thomas Haller [Mon, 18 Sep 2023 19:59:24 +0000 (21:59 +0200)]
tests/shell: run `nft --check` on persisted dump files
"nft --check" will trigger a rollback in kernel. The existing dump files
might hit new code paths. Take the opportunity to call the command on
the existing files.
And alternative would be to write a separate tests, that iterates over
all files. However, then we can only run all the commands sequentially
(unless we do something smart). That might be slower than the
opportunity to run the checks in parallel. More importantly, it would be
nice if the check for the dump file is clearly tied to the file's test.
So run it right after the test, from the test wrapper.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
A library should not restrict being used by multiple threads or make
assumptions about how it's being used. Hence a "init_once" pattern
without no locking is racy, a code smell and should be avoided.
Note that libxtables is full of global variables and when linking against
it, libnftables cannot be used from multiple threads either. That is not
easy to fix.
Move the ugliness of "init_once" away from nft_ctx_new(), so that the
problem is concentrated closer to libxtables.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Thomas Haller [Tue, 19 Sep 2023 12:36:16 +0000 (14:36 +0200)]
libnftables: drop gmp_init() and mp_set_memory_functions()
Setting global handles for libgmp via mp_set_memory_functions() is very
ugly. When we don't use mini-gmp, then potentially there are other users
of the library in the same process, and every process fighting about the
allocation functions is not gonna work.
It also means, we must not reset the allocation functions after somebody
already allocated GMP data with them. Which we cannot ensure, as we
don't know what other parts of the process are doing.
It's also unnecessary. The default allocation functions for gmp and
mini-gmp already abort the process on allocation failure ([1], [2]),
just like our xmalloc().
3975430b12d9 ("src: expand table command before evaluation") moved
ruleset expansion before evaluation, except for sets and maps. For
sets and maps there is still a post_expand() phase.
This patch moves sets and map expansion to allocate an independent
CMD_OBJ_SETELEMS command to add elements to named set and maps which is
evaluated, this consolidates the ruleset expansion to happen always
before the evaluation step for all objects, except for anonymous sets
and maps.
This approach avoids an interference with the set interval code which
detects overlaps and merges of adjacents ranges. This set interval
routine uses set->init to maintain a cache of existing elements. Then,
the post_expand() phase incorrectly expands set->init cache and it
triggers a bogus ENOENT errors due to incorrect bytecode (placing
element addition before set creation) in combination with user declared
sets using the flat syntax notation.
Since the evaluation step (coming after the expansion) creates
implicit/anonymous sets and maps, those are not expanded anymore. These
anonymous sets still need to be evaluated from set_evaluate() path and
the netlink bytecode generation path, ie. do_add_set(), needs to deal
with anonymous sets.
Note that, for named sets, do_add_set() does not use set->init. Such
content is part of the existing cache, and the CMD_OBJ_SETELEMS command
is responsible for adding elements to named sets.
Fixes: 3975430b12d9 ("src: expand table command before evaluation") Reported-by: Jann Haber <jannh@selfnet.de> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
evaluate: fix memleak in prefix evaluation with wildcard interface name
The following ruleset:
table ip x {
chain y {
meta iifname { abcde*, xyz }
}
}
triggers the following memleak:
==6871== 16 bytes in 1 blocks are definitely lost in loss record 1 of 1
==6871== at 0x483877F: malloc (vg_replace_malloc.c:307)
==6871== by 0x48AD898: xmalloc (utils.c:37)
==6871== by 0x4BC8B22: __gmpz_init2 (in /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.1)
==6871== by 0x4887E67: constant_expr_alloc (expression.c:424)
==6871== by 0x488EF1F: expr_evaluate_prefix (evaluate.c:1138)
==6871== by 0x488EF1F: expr_evaluate (evaluate.c:2725)
==6871== by 0x488E76D: expr_evaluate_set_elem (evaluate.c:1662)
==6871== by 0x488E76D: expr_evaluate (evaluate.c:2739)
==6871== by 0x4891033: list_member_evaluate (evaluate.c:1454)
==6871== by 0x488E2B6: expr_evaluate_set (evaluate.c:1757)
==6871== by 0x488E2B6: expr_evaluate (evaluate.c:2737)
==6871== by 0x48910D0: elems_evaluate (evaluate.c:4605)
==6871== by 0x4891432: set_evaluate (evaluate.c:4711)
==6871== by 0x48915BC: implicit_set_declaration (evaluate.c:122)
==6871== by 0x488F18A: expr_evaluate_relational (evaluate.c:2503)
==6871== by 0x488F18A: expr_evaluate (evaluate.c:2745)
expr_evaluate_prefix() calls constant_expr_alloc() which have already
called mpz_init2(), the second call to mpz_init2() overlaps the existing
mpz_t data memory area.
Remove extra mpz_init2() call to fix this memleak.
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Thomas Haller [Thu, 14 Sep 2023 14:09:50 +0000 (16:09 +0200)]
netlink: fix leaking typeof_expr_data/typeof_expr_key in netlink_delinearize_set()
There are various code paths that return without freeing typeof_expr_data
and typeof_expr_key. It's not at all obvious, that there isn't a leak
that way. Quite possibly there is a leak. Fix it, or at least make the
code more obviously correct.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Thomas Haller [Mon, 18 Sep 2023 19:59:23 +0000 (21:59 +0200)]
tests/shell: simplify collecting error result in "test-wrapper.sh"
The previous pattern was unnecessarily confusing.
The "$rc_{dump,valgrind,tainted}" variable should only remember whether
that particular check failed, not the overall exit code of the test
wrapper.
Otherwise, if you want to know in which case the wrapper exits with code
122, you have to oddly follow the rc_valgrind variable.
This change will make more sense, when we add another such variable, but
which will be assigned the non-zero value at multiple places. Assigning
there the exit code of the wrapper, duplicates the places where the
condition maps to the exit code.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Mon, 18 Sep 2023 18:45:21 +0000 (20:45 +0200)]
tests/shell: colorize NFT_TEST_HAS_SOCKET_LIMITS
NFT_TEST_HAS_SOCKET_LIMITS= is similar to NFT_TEST_HAVE_* variables and
indicates a feature (or lack thereof), except that it's inverted. Maybe
this should be consolidated, however, NFT_TEST_HAS_SOCKET_LIMITS= is
detected in the root namespace, unlike the shell scripts from features.
So it's unclear how to consolidate them best.
Anyway. Still highlight a lack of the capability, as it can cause tests
to be skipped and we should see that easily.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
This doesn't seem very useful. For one, we have special exit codes like
0 (OK), 77 (SKIPPED), 124 (DUMP FAIL), 123 (TAINTED), 122 (VALGRIND).
Any other exit code is just an arbitrary failure. We don't define any
special codes, and printing them is not useful.
Note that further exit codes (118 - 121) are reserved, and could be
special purposed, when there is a use.
You can find the real exit code from the test in the result data in the
"rc-failed" file.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Mon, 18 Sep 2023 18:45:19 +0000 (20:45 +0200)]
tests/shell: set C locale in "run-tests.sh"
The tests should run always the same, regardless of the user's language
settings. Set LANG=C and LC_ALL=C and unset LANGUAGE. If some part wants
to test a different language, it would set it explicitly. They anyway
wouldn't want to depend on something from the user's environment.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
tests/shell: skip reset tests if kernel lacks support
reset is implemented via flush + extra attribute, so older kernels
perform a flush. This means .nft doesn't work, we need to check
if the individual set contents/sets are still in place post-reset.
Make this generic and permit use of feat.sh in addition to the simpler
foo.nft feature files.
Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Thomas Haller <thaller@redhat.com>
Thomas Haller [Fri, 15 Sep 2023 15:54:00 +0000 (17:54 +0200)]
tests/shell: cleanup creating dummy interfaces in tests
In "tests/shell/testcases/chains/netdev_chain_0", calling "trap ...
EXIT" multiple times does not work. Fix it, by calling one cleanup
function.
Note that we run in separate namespaces, so the cleanup is usually not
necessary. Still do it, we might want to run without unshare (via
NFT_TEST_UNSHARE_CMD=""). Without unshare, it's important that the
cleanup always works. In practice it might not, for example, "trap ...
EXIT" does not run for SIGTERM. A leaked interface might break the
follow up test and tests interfere with each other.
Try to workaround that by first trying to delete the interface.
Also failures to create the interfaces are not considered fatal. I don't
understand under what circumstances this might fail, note that there are
other tests that create dummy interface and don't "exit 77" on failure.
We want to know when something odd is going on.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Fri, 15 Sep 2023 15:54:02 +0000 (17:54 +0200)]
tests/shell: suggest 4Mb /proc/sys/net/core/{wmem_max,rmem_max} for rootless
2Mb was not enough to pass "tests/shell/testcases/sets/0030add_many_elements_interval_0"
in an unprivileged/rootless namespace.
Instead, bump the suggestion to 4Mb, which lets the test pass.
Note that the 4Mb are only the recommended value when running the test
as rootless, and is used to autodetect NFT_TEST_HAS_SOCKET_LIMITS=y.
You can set whatever values are suitable for your environment, and
explicitly indicate whether the limits are appropriate or not via
NFT_TEST_HAS_SOCKET_LIMITS=n|y.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Fri, 15 Sep 2023 15:32:35 +0000 (17:32 +0200)]
tests/shell: add feature probing via "features/*.nft" files
Running selftests on older kernels makes some of them fail very early
because some tests use features that are not available on older kernels,
e.g. -stable releases.
Known examples:
- inner header matching
- anonymous chains
- elem delete from packet path
Also, some test cases might fail because a feature isn't compiled in,
such as netdev chains.
This adds a feature-probing mechanism to shell tests.
Simply drop a 'nft -f' compatible file with a .nft suffix into
"tests/shell/features". "run-tests.sh" will load it via `nft --check`
and will export
NFT_TEST_HAVE_${feature}=y|n
Here ${feature} is the basename of the .nft file without file extension.
It must be all lower-case.
This extends the existing NFT_TEST_HAVE_json= feature detection.
Similarly, NFT_TEST_REQUIRES(NFT_TEST_HAVE_*) tags work to easily skip a
test.
The test script that cannot fully work without the feature should either
skip the test entirely (NFT_TEST_REQUIRES(NFT_TEST_HAVE_*)), or run a
reduced/modified test. If a modified test was run and passes, it is
still a good idea to mark the overall result as skipped (exit 77)
instead of claiming success to the modified test. We want to know when
not the full test was running, while we want to test as much as we can.
This patch is based on Florian's feature probing patch.
Thomas Haller [Thu, 14 Sep 2023 13:14:02 +0000 (15:14 +0200)]
tests/build: capture more output from "tests/build/run-tests.sh" script
Dropping stdout for various build tests makes it hard to understand what
happens, when a build fails. Redirect both stdout and stderr to the log
files for easier debugging.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Wed, 13 Sep 2023 17:11:02 +0000 (19:11 +0200)]
tests/shell: accept $NFT_TEST_TMPDIR_TAG for the result directory
We allow the user to set "$TMPDIR" to affect where the "nft-test.*"
directory is created. However, we don't allow the user to specify the
exact location, so the user doesn't really know which directory was
created.
One remedy is that the test will also create the symlink
"$TMPDIR/nft-test.latest.$USER" to point to the last test result.
However, if you run multiple tests in parallel, that is not reliable to
find the test results.
Accept $NFT_TEST_TMPDIR_TAG and use it as part of the generated
filename. That way, the caller can set it to a unique tag, and find the
directory later based on that. For example
export TMPDIR=/tmp
export NFT_TEST_TMPDIR_TAG=".$(uuidgen)"
./tests/shell/run-tests.sh
ls -lad "$TMPDIR/nft-test."*"$NFT_TEST_TMPDIR_TAG"*/
will work reliably -- as long as the tag is chosen uniquely.
The reason to not allow the user to specify the directory name directly,
is because we want that tests results follow the well-known pattern
"/tmp/nft-test*".
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Wed, 13 Sep 2023 17:11:01 +0000 (19:11 +0200)]
tests/shell: exit 77 from "run-tests.sh" if all tests were skipped
If there are multiple tests and some of them pass and some are skipped,
the overall result should be success (zero). Because likely the user
just selected a bunch of tests (or all of them). So skipping some tests
does not mean that the entire run is not a success.
However, if all tests are skipped, then mark the overall result as
skipped too. The more common case is if you only run one single test,
then we want to know, that the test didn't run.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Wed, 13 Sep 2023 17:05:05 +0000 (19:05 +0200)]
tests/shell: drop unstable dump for "transactions/0051map_0" test
The file "tests/shell/testcases/transactions/dumps/0051map_0.nft" gets
generated differently on Fedora 38 (6.4.14-200.fc38.x86_64) and
CentOS-Stream-9 (5.14.0-354.el9.x86_64). It's not stable.
diff --git c/tests/shell/testcases/transactions/dumps/0051map_0.nft w/tests/shell/testcases/transactions/dumps/0051map_0.nft
index 59d69df70e61..fa7df9f93757 100644
--- c/tests/shell/testcases/transactions/dumps/0051map_0.nft
+++ w/tests/shell/testcases/transactions/dumps/0051map_0.nft
@@ -1,7 +1,11 @@
table ip x {
+ chain w {
+ }
+
chain m {
}
chain y {
+ ip saddr vmap { 1.1.1.1 : jump w, 2.2.2.2 : accept, 3.3.3.3 : goto m }
}
}
Drop it.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Wed, 13 Sep 2023 08:20:25 +0000 (10:20 +0200)]
tests/shell: add option to shuffle execution order of tests
The user can set NFT_TEST_SHUFFLE_TESTS=y|n to have the tests shuffled
randomly. The purpose of shuffling is to find tests that depend on each
other, or would break when run in unexpected order.
If unspecified, by default tests are shuffled if no tests are selected
on the command line.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Wed, 13 Sep 2023 08:20:23 +0000 (10:20 +0200)]
tests/shell: export NFT_TEST_RANDOM_SEED variable for tests
Let "run-tests.sh" export a NFT_TEST_RANDOM_SEED variable, set to
a decimal, random integer (in the range of 0 to 0x7FFFFFFF).
The purpose is to provide a seed to tests for randomization.
Randomizing tests is very useful to increase the coverage while not
testing all combinations (which might not be practical).
The point of NFT_TEST_RANDOM_SEED is that the user can set the
environment variable so that the same series of random events is used.
That is useful for reproducing an issue, that is known to happen with a
certain seed.
- by default, if the user leaves NFT_TEST_RANDOM_SEED unset or empty,
the script generates a number using $SRANDOM.
- if the user sets NFT_TEST_RANDOM_SEED to an integer it is taken
as is (modulo 0x80000000).
- otherwise, calculate a number by hashing the value of
$NFT_TEST_RANDOM_SEED.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Tue, 12 Sep 2023 07:30:54 +0000 (09:30 +0200)]
datatype: fix leak and cleanup reference counting for struct datatype
Test `./tests/shell/run-tests.sh -V tests/shell/testcases/maps/nat_addr_port`
fails:
==118== 195 (112 direct, 83 indirect) bytes in 1 blocks are definitely lost in loss record 3 of 3
==118== at 0x484682C: calloc (vg_replace_malloc.c:1554)
==118== by 0x48A39DD: xmalloc (utils.c:37)
==118== by 0x48A39DD: xzalloc (utils.c:76)
==118== by 0x487BDFD: datatype_alloc (datatype.c:1205)
==118== by 0x487BDFD: concat_type_alloc (datatype.c:1288)
==118== by 0x488229D: stmt_evaluate_nat_map (evaluate.c:3786)
==118== by 0x488229D: stmt_evaluate_nat (evaluate.c:3892)
==118== by 0x488229D: stmt_evaluate (evaluate.c:4450)
==118== by 0x488328E: rule_evaluate (evaluate.c:4956)
==118== by 0x48ADC71: nft_evaluate (libnftables.c:552)
==118== by 0x48AEC29: nft_run_cmd_from_buffer (libnftables.c:595)
==118== by 0x402983: main (main.c:534)
I think the reference handling for datatype is wrong. It was introduced
by commit 01a13882bb59 ('src: add reference counter for dynamic
datatypes').
We don't notice it most of the time, because instances are statically
allocated, where datatype_get()/datatype_free() is a NOP.
Fix and rework.
- Commit 01a13882bb59 comments "The reference counter of any newly
allocated datatype is set to zero". That seems not workable.
Previously, functions like datatype_clone() would have returned the
refcnt set to zero. Some callers would then then set the refcnt to one, but
some wouldn't (set_datatype_alloc()). Calling datatype_free() with a
refcnt of zero will overflow to UINT_MAX and leak:
if (--dtype->refcnt > 0)
return;
While there could be schemes with such asymmetric counting that juggle the
appropriate number of datatype_get() and datatype_free() calls, this is
confusing and error prone. The common pattern is that every
alloc/clone/get/ref is paired with exactly one unref/free.
Let datatype_clone() return references with refcnt set 1 and in
general be always clear about where we transfer ownership (take a
reference) and where we need to release it.
- set_datatype_alloc() needs to consistently return ownership to the
reference. Previously, some code paths would and others wouldn't.
Thomas Haller [Tue, 12 Sep 2023 22:44:50 +0000 (00:44 +0200)]
tests/shell: ensure vgdb-pipe files are deleted from "nft-valgrind-wrapper.sh"
When the valgrind process gets killed, those files can be left over.
They are located in the original $TMPDIR (usually /tmp). They should be
cleaned up.
I tried to cleanup the files from withing "nft-valgrind-wrapper.sh"
itself via a `trap`, but it doesn't work. Instead, let "run-tests.sh"
delete all files with a matching pattern.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Tue, 12 Sep 2023 22:44:49 +0000 (00:44 +0200)]
tests/shell: kill running child processes when aborting "run-tests.sh"
When aborting "run-tests.sh", child processes were left running. Kill
them. It's surprisingly complicated to get this somewhat right. Do it by
enabling monitor mode for each test call, so that they run in separate
process groups and we can kill the entire group.
Note that we cannot just `kill -- -$$`, because it's not clear who is in
this process group. Also, we don't want to kill the `tee` process which
handles our logging.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Fri, 8 Sep 2023 17:32:20 +0000 (19:32 +0200)]
include: include <stdlib.h> in <nft.h>
It provides malloc()/free(), which is so basic that we need it
everywhere. Include via <nft.h>.
The ultimate purpose is to define more things in <nft.h>. While it has
not corresponding C sources, <nft.h> can contain macros and static
inline functions, and is a good place for things that we shall have
everywhere. Since <stdlib.h> provides malloc()/free() and size_t, that
is a very basic dependency, that will be needed for that.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Thomas Haller [Fri, 8 Sep 2023 17:32:19 +0000 (19:32 +0200)]
parser_bison: include <nft.h> for base C environment to "parser_bison.y"
All our C sources should include <nft.h> as first. This prepares an
environment of things that we expect to have available in all our C
sources (and indirectly in our internal header files, because internal
header files are always indirectly from a C source).
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Thomas Haller [Fri, 8 Sep 2023 15:07:25 +0000 (17:07 +0200)]
tests/shell: add "--quick" option to skip slow tests (via NFT_TEST_SKIP_slow=y)
It's important to run (a part) of the tests in a timely manner.
Add an option to skip long running tests.
Thereby, add a more general NFT_TEST_SKIP_* mechanism.
This is related and inverse from "NFT_TEST_HAVE_json", where a test
can require [ "$NFT_TEST_HAVE_json" != n ] to run, but is skipped when
[ "$NFT_TEST_SKIP_slow" = y ].
Currently only NFT_TEST_SKIP_slow is supported. The user can set such
environment variables (or use the -Q|--quick command line option). The
configuration is printed in the test info.
Tests should check for [ "$NFT_TEST_SKIP_slow" = y ] so that the
variable has to be explicitly set to opt-out. For convenience, tests can
also add a
# NFT_TEST_SKIP(NFT_TEST_SKIP_slow)
tag, which is evaluated by test-wrapper.sh. Or they can run a quick, reduced
part of the test, but then should still indicate to be skipped.
Mark 8 tests are as slow, that take longer than 5 seconds on my machine.
With this, a parallel wall time for the non-slow tests is only 7 seconds
(on my machine).
The ultimate point is to integrate a call to "tests/shell/run-tests.sh"
in a `make check` target. For development, you can then export
NFT_TEST_SKIP_slow=y and have a fast `make check`.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Fri, 8 Sep 2023 15:07:24 +0000 (17:07 +0200)]
tests/shell: skip tests if nft does not support JSON mode
We can build nft without JSON support, and some tests will fail without
it. Instead, they should be skipped. Also note, that the test accepts any
nft binary via the "NFT" environment variable. So it's not enough to
make the skipping dependent on build configuration, but on the currently
used $NFT variable.
Let "run-test.sh" detect and export a "NFT_TEST_HAVE_json=y|n" variable. This
is heavily inspired by Florian's feature probing patches.
Tests that require JSON can check that variable, and skip. Note that
they check in the form of [ "$NFT_TEST_HAVE_json" != n ], so the test is
only skipped, if we explicitly detect lack of support. That is, don't
check via [ "$NFT_TEST_HAVE_json" = y ].
Some of the tests still run parts of the tests that don't require JSON.
Only towards the end of such partial run, mark the test as skipped.
Some tests require JSON support throughout. For those, add a mechanism
where tests can add a tag (in their first 10 lines):
# NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)
This will be checked by "test-wrapper.sh", which will skip the test.
The purpose of this is to make it low-effort to skip a test and to print
the reason in the text output as
Test skipped due to NFT_TEST_HAVE_json=n (test has "NFT_TEST_REQUIRES(NFT_TEST_HAVE_json)" tag)
This is intentionally not shortened to NFT_TEST_REQUIRES(json), so that
we can grep for NFT_TEST_HAVE_json to find all relevant places.
Note that while NFT_TEST_HAVE_json is autodetected, the caller can also
force it by setting the environment variable. This allows to see what
would happen to such a test.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Fri, 8 Sep 2023 14:25:31 +0000 (16:25 +0200)]
tests/shell: set valgrind's "--vgdb-prefix=" to orignal TMPDIR
"test-wrapper.sh" sets TMPDIR="$NFT_TEST_TESTTMPDIR". That is useful, so
that temporary files of the tests are placed inside the test result
data.
Sometimes tests miss to delete those files, which would result in piling
up /tmp/tmp.XXXXXXXXXX files. By setting $TMPDIR, those files are
clearly related to the test run that created them, and can be deleted
together.
However, valgrind likes to create files like
"vgdb-pipe-from-vgdb-to-68-by-thom-on-???" inside $TMPDIR. These are
pipes, so if you run `grep -R ^ /tmp/nft-test.latest` while
the test is still running (to inspect the results), then the process
hands reading from the pipe.
Instead, tell valgrind to put those files in the original TMPDIR. For
that purpose, export NFT_TEST_TMPDIR_ORIG from "run-tests.sh".
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
projects / thirdparty / nftables.git / log
