Florian Westphal [Mon, 26 Feb 2024 09:35:16 +0000 (10:35 +0100)]
tests: shell: add more json dumps
Those are expected to be stable, so add them.
Some are not 100% correct, as "typeof" is misprinted as "type" (json output
and input parser lack support for this), but for these files the "type"
is valid too.
This will allow better validation once proper "typeof" support is
added to json.c and json-parser.c.
Florian Westphal [Wed, 14 Feb 2024 10:41:30 +0000 (11:41 +0100)]
tests: shell: permit use of host-endian constant values in set lookup keys
extend an existing test case with the afl input to cover in/output.
A new test case is added to test linearization, delinearization and
matching
Fixes: c0080feb0d03 ("evaluate: permit use of host-endian constant values in set lookup keys") Signed-off-by: Florian Westphal <fw@strlen.de> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
evaluate: permit use of host-endian constant values in set lookup keys
AFL found following crash:
table ip filter {
map ipsec_in {
typeof ipsec in reqid . iif : verdict
flags interval
}
chain INPUT {
type filter hook input priority filter; policy drop;
ipsec in reqid . 100 @ipsec_in
}
}
Which yields:
nft: evaluate.c:1213: expr_evaluate_unary: Assertion `!expr_is_constant(arg)' failed.
All existing test cases with constant values use big endian values, but
"iif" expects host endian values.
As raw values were not supported before, concat byteorder conversion
doesn't handle constants.
Fix this:
1. Add constant handling so that the number is converted in-place,
without unary expression.
2. Add the inverse handling on delinearization for non-interval set
types.
When dissecting the concat data soup, watch for integer constants where
the datatype indicates host endian integer.
Last, extend an existing test case with the afl input to cover
in/output.
A new test case is added to test linearization, delinearization and
matching.
Based on original patch from Florian Westphal, patch subject and
description wrote by him.
Fixes: b422b07ab2f9 ("src: permit use of constant values in set lookup keys") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Phil Sutter [Thu, 8 Feb 2024 01:10:48 +0000 (02:10 +0100)]
cache: Always set NFT_CACHE_TERSE for list cmd with --terse
This fixes at least 'nft -t list table ...' and 'nft -t list set ...'.
Note how --terse handling for 'list sets/maps' remains in place since
setting NFT_CACHE_TERSE does not fully undo NFT_CACHE_SETELEM: setting
both enables fetching of anonymous sets which is pointless for that
command.
evaluate: skip byteorder conversion for selector smaller than 2 bytes
Add unary expression to trigger byteorder conversion for host byteorder
selectors only if selectors length is larger or equal than 2 bytes.
# cat test.nft
table ip x {
set test {
type ipv4_addr . ether_addr . inet_proto
flags interval
}
chain y {
ip saddr . ether saddr . meta l4proto @test counter
}
}
# nft -f test.nft
ip x y
[ meta load iiftype => reg 1 ]
[ cmp eq reg 1 0x00000001 ]
[ payload load 4b @ network header + 12 => reg 1 ]
[ payload load 6b @ link header + 6 => reg 9 ]
[ meta load l4proto => reg 11 ]
[ byteorder reg 11 = hton(reg 11, 2, 1) ] <--- should not be here
[ lookup reg 1 set test ]
[ counter pkts 0 bytes 0 ]
Fixes: 1017d323cafa ("src: support for selectors with different byteorder with interval concatenations") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Phil Sutter [Thu, 8 Feb 2024 13:30:17 +0000 (14:30 +0100)]
tests: shell: Pretty-print all *.json-nft dumps
The problem with single line output as produced by 'nft -j list ruleset'
is its incompatibility to unified diff format as any change in this
single line will produce a diff which contains the old and new lines in
total. This is not just unreadable but will blow up patches which may
exceed mailinglists' mail size limits.
Convert them all at once by feeding their contents to
tests/shell/helpers/json-pretty.sh.
The script "json-diff-pretty.sh" is no longer used. It is kept
however, because it might be a useful for manual comparing files.
Note that "json-sanitize-ruleset.sh" and "json-pretty.sh" are still
two separate scripts and called at different times. They also do
something different. The former mangles the JSON to account for changes
that are not stable (in the JSON data itself), while the latter only
pretty prints it.
- when generating a new .json-nft dump file, the file will be updated to
use the new, prettified format, unless the file is in the old format
and needs no update. This means, with DUMPGEN=y, old style is preserved
unless an update becomes necessary.
This requires "json-pretty.sh" having stable output, as those files are
committed to git. This is probably fine.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Phil Sutter <phil@nwl.cc>
Several tests reports DUMP_FAILED because it was missing the auto-merge
flag. That is, the original json dump was not correct. Update tests
accordingly now that json support provides an automerge flag.
Fixes: a4034c66b03e ("json: Support sets' auto-merge option") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Phil Sutter [Tue, 6 Feb 2024 18:26:57 +0000 (19:26 +0100)]
cache: Optimize caching for 'list tables' command
No point in fetching anything other than existing tables from kernel:
'list tables' merely prints existing table names, no contents.
Also populate filter's family field to reduce overhead when listing
tables in one family with many tables in another one. It works without
further adjustments because nftnl_nlmsg_build_hdr() will use the value
for nfgen_family.
This becomes an error in GCC 14 and Clang 16. It's a common
misconception that these warnings are invalid or simply noise for
Bison/parser files, but even if that were true, we'd need to handle it
somehow anyway. Silencing them does nothing, so stop doing that.
Further, I don't actually get any warnings to fix with bison-3.8.2. This
mirrors changes we've done in other netfilter.org projects.
Signed-off-by: Sam James <sam@gentoo.org> Signed-off-by: Phil Sutter <phil@nwl.cc>
conflict_resolution_gen_dependency() can only handle linklayer
conflicts, hence the assert.
Rename it accordingly. Also rename resolve_protocol_conflict, it doesn't
do anything for != PROTO_BASE_LL_HDR and extend the assertion to that
function too.
Callers now enforce PROTO_BASE_LL_HDR prerequisite.
after:
Error: conflicting transport layer protocols specified: comp vs. udp
ip6 nexthdr comp udp dport 4789
^^^^^^^^^
Florian Westphal [Mon, 15 Jan 2024 13:27:15 +0000 (14:27 +0100)]
rule: fix sym refcount assertion
Scope release must happen last.
afl provided a reproducer where policy is a define, because
scope is released too early we get:
nft: src/rule.c:559: scope_release: Assertion `sym->refcnt == 1' failed.
... because chain->policy is EXPR_SYMBOL.
Fixes: 627c451b2351 ("src: allow variables in the chain priority specification") Signed-off-by: Florian Westphal <fw@strlen.de>
For loads, this is already prevented via expr_evaluate_bits() which has:
if (masklen > NFT_REG_SIZE * BITS_PER_BYTE)
return expr_error(ctx->msgs, expr, "mask length %u exceeds allowed maximum of %u\n",
masklen, NFT_REG_SIZE * BITS_PER_BYTE);
But for the store path this isn't called.
The reproducer asks to store a 128 bit integer at bit offset 1, i.e.
17 bytes would need to be munged, but we can only handle up to 16 bytes
(one pseudo-register).
Fixes: 78936d50f306 ("evaluate: add support to set IPv6 non-byte header fields") Signed-off-by: Florian Westphal <fw@strlen.de>
Florian Westphal [Fri, 12 Jan 2024 12:27:23 +0000 (13:27 +0100)]
parser: reject raw payload expressions with 0 length
Reject this at parser stage. Fix up the json input side too, else
reproducer gives:
nft: src/netlink.c:243: netlink_gen_raw_data: Assertion `len > 0' failed.
Direct leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x7fe7b54a9e8f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x7fe7b538b9a9 in __gmp_default_allocate (/lib/x86_64-linux-gnu/libgmp.so.10+0xc9a9)
Fixes: 3671c4897003 ("evaluate: guard against NULL basetype") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Direct leak of 144 byte(s) in 1 object(s) allocated from:
#0 0x7fde06ca9e8f in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
#1 0x7fde062924af in xmalloc src/utils.c:31
#2 0x7fde0629266c in xzalloc src/utils.c:70
#3 0x7fde06167299 in expr_alloc src/expression.c:46
#4 0x7fde0616b014 in constant_expr_alloc src/expression.c:420
#5 0x7fde06128e43 in expr_evaluate_map src/evaluate.c:2027
#6 0x7fde06137b06 in expr_evaluate src/evaluate.c:2891
#7 0x7fde06132417 in expr_evaluate_relational src/evaluate.c:2497
#8 0x7fde06137b36 in expr_evaluate src/evaluate.c:2895
#9 0x7fde06137d5f in stmt_evaluate_expr src/evaluate.c:2914
#10 0x7fde061524c8 in stmt_evaluate src/evaluate.c:4646
#11 0x7fde0615c9ee in rule_evaluate src/evaluate.c:5202
#12 0x7fde061600c7 in cmd_evaluate_add src/evaluate.c:5422
Fixes: 70054e6e1c87 ("evaluate: catch implicit map expressions without known datatype") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
evaluate: do not fetch next expression on runaway number of concatenation components
If this is the last expression, then the runaway flag is set on and
evaluation bails in the next iteration, do not fetch next list element
which refers to the list head.
I found this by code inspection, I could not trigger any crash with this
one.
Fixes: ae1d54d1343f ("evaluate: do not crash on runaway number of concatenation components") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
evaluate: skip anonymous set optimization for concatenations
Concatenation is only supported with sets. Moreover, stripping of the
set leads to broken ruleset listing, therefore, skip this optimization
for the concatenations.
Fixes: fa17b17ea74a ("evaluate: revisit anonymous set with single element optimization") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
Florian Westphal [Thu, 11 Jan 2024 17:14:15 +0000 (18:14 +0100)]
evaluate: tproxy: move range error checks after arg evaluation
Testing for range before evaluation will still crash us later during
netlink linearization, prefixes turn into ranges, symbolic expression
might hide a range/prefix.
So move this after the argument has been evaluated.
Florian Westphal [Thu, 11 Jan 2024 15:57:28 +0000 (16:57 +0100)]
evaluate: error out when expression has no datatype
add rule ip6 f i rt2 addr . ip6 daddr { dead:: . dead:: }
... will cause a segmentation fault, we assume expr->dtype is always
set.
rt2 support is incomplete, the template is uninitialised.
This could be fixed up, but rt2 (a subset of the deperecated type 0),
like all other routing headers, lacks correct dependency tracking.
Currently such routing headers are always assumed to be segment routing
headers, we would need to add dependency on 'Routing Type' field in the
routing header, similar to icmp type/code.
Quan Tian [Wed, 10 Jan 2024 04:30:59 +0000 (04:30 +0000)]
doc: clarify reject is supported at prerouting stage
It's supported since kernel commit f53b9b0bdc59 ("netfilter: introduce
support for reject at prerouting stage").
Reported-by: Dan Winship <danwinship@redhat.com> Signed-off-by: Quan Tian <tianquan23@gmail.com> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
tests: shell: prefer project nft to system-wide nft
Use $NFT (src/nft, in-tree binary), not the one installed by the distro.
Else we may not find newly added bugs unless user did "make install" or
bug has propagated to release.
Phil Sutter [Fri, 22 Dec 2023 16:00:44 +0000 (17:00 +0100)]
datatype: Describe rt symbol tables
Implement a symbol_table_print() wrapper for the run-time populated
rt_symbol_tables which formats output similar to expr_describe() and
includes the data source.
Since these tables reside in struct output_ctx there is no implicit
connection between data type and therefore providing callbacks for
relevant datat types which feed the data into said wrapper is a simpler
solution than extending expr_describe() itself.
Phil Sutter [Fri, 22 Dec 2023 15:53:14 +0000 (16:53 +0100)]
datatype: Initialize rt_symbol_tables' base field
It is unconditionally accessed in symbol_table_print() so make sure it
is initialized to either BASE_DECIMAL (arbitrary) for empty or
non-existent source files or a proper value depending on entry number
format.
Phil Sutter [Fri, 15 Dec 2023 20:59:44 +0000 (21:59 +0100)]
datatype: rt_symbol_table_init() to search for iproute2 configs
There is an ongoing effort among various distributions to tidy up in
/etc. The idea is to reduce contents to just what the admin manually
inserted to customize the system, anything else shall move out to /usr
(or so). The various files in /etc/iproute2 fall in that category as
they are seldomly modified.
The crux is though that iproute2 project seems not quite sure yet where
the files should go. While v6.6.0 installs them into /usr/lib/iproute2,
current mast^Wmain branch uses /usr/share/iproute2. Assume this is going
to stay as /(usr/)lib does not seem right for such files.
Note that rt_symbol_table_init() is not just used for
iproute2-maintained configs but also for connlabel.conf - so retain the
old behaviour when passed an absolute path.
Florian Westphal [Thu, 21 Dec 2023 10:25:14 +0000 (11:25 +0100)]
src: do not allow to chain more than 16 binops
netlink_linearize.c has never supported more than 16 chained binops.
Adding more is possible but overwrites the stack in
netlink_gen_bitwise().
Add a recursion counter to catch this at eval stage.
Its not enough to just abort once the counter hits
NFT_MAX_EXPR_RECURSION.
This is because there are valid test cases that exceed this.
For example, evaluation of 1 | 2 will merge the constans, so even
if there are a dozen recursive eval calls this will not end up
with large binop chain post-evaluation.
v2: allow more than 16 binops iff the evaluation function
did constant-merging.
Florian Westphal [Wed, 13 Dec 2023 16:41:26 +0000 (17:41 +0100)]
netlink: fix stack buffer overflow with sub-reg sized prefixes
The calculation of the dynamic on-stack array is incorrect,
the scratch space can be too low which gives stack corruption:
AddressSanitizer: dynamic-stack-buffer-overflow on address 0x7ffdb454f064..
#1 0x7fabe92aaac4 in __mpz_export_data src/gmputil.c:108
#2 0x7fabe92d71b1 in netlink_export_pad src/netlink.c:251
#3 0x7fabe92d91d8 in netlink_gen_prefix src/netlink.c:476
div_round_up() cannot be used here, it fails to account for register
padding. A 16 bit prefix will need 2 registers (start, end -- 8 bytes
in total).
Remove the dynamic sizing and add an assertion in case upperlayer
ever passes invalid expr sizes down to us.
After this fix, the combination is rejected by the kernel
because of the maps' wrong data size, before the fix userspace
may crash before.
Florian Westphal [Wed, 13 Dec 2023 16:29:50 +0000 (17:29 +0100)]
evaluate: error out when existing set has incompatible key
Before:
BUG: invalid range expression type symbol
nft: expression.c:1494: range_expr_value_high: Assertion `0' failed.
After:
range_expr_value_high_assert:5:20-27: Error: Could not resolve protocol name
elements = { 100-11.0.0.0, }
^^^^^^^^
range_expr_value_high_assert:7:6-7: Error: set definition has conflicting key (ipv4_addr vs inet_proto)
Florian Westphal [Wed, 13 Dec 2023 10:09:58 +0000 (11:09 +0100)]
parser_bison: close chain scope before chain release
cmd_alloc() will free the chain, so we must close the scope opened
in chain_block_alloc beforehand.
The included test file will cause a use-after-free because nft attempts
to search for an identifier in a scope that has been freed:
AddressSanitizer: heap-use-after-free on address 0x618000000368 at pc 0x7f1cbc0e6959 bp 0x7ffd3ccb7850 sp 0x7ffd3ccb7840
#0 0x7f1cbc0e6958 in symbol_lookup src/rule.c:629
#1 0x7f1cbc0e66a1 in symbol_get src/rule.c:588
#2 0x7f1cbc120d67 in nft_parse src/parser_bison.y:4325
Fixes: a66b5ad9540d ("src: allow for updating devices on existing netdev chain") Signed-off-by: Florian Westphal <fw@strlen.de>
Thomas Haller [Wed, 27 Sep 2023 12:23:28 +0000 (14:23 +0200)]
netlink_linearize: avoid strict-overflow warning in netlink_gen_bitwise()
With gcc-13.2.1-1.fc38.x86_64:
$ gcc -Iinclude -c -o tmp.o src/netlink_linearize.c -Werror -Wstrict-overflow=5 -O3
src/netlink_linearize.c: In function ‘netlink_gen_bitwise’:
src/netlink_linearize.c:1790:1: error: assuming signed overflow does not occur when changing X +- C1 cmp C2 to X cmp C2 -+ C1 [-Werror=strict-overflow]
1790 | }
| ^
cc1: all warnings being treated as errors
It also makes more sense this way, where "n" is the hight of the
"binops" stack, and we check for a non-empty stack with "n > 0" and pop
the last element with "binops[--n]".
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
There is a stack overflow somewhere in this code, we end
up memcpy'ing a way too large expr into a fixed-size on-stack
buffer.
This is hard to diagnose, most of this code gets inlined so
the crash happens later on return from alloc_nftnl_setelem.
Condense the mempy into a helper and add a BUG so we can catch
the overflow before it occurs.
->value is too small (4, should be 16), but for normal
cases (well-formed data must fit into max reg space, i.e.
64 byte) the chain buffer that comes after value in the
structure provides a cushion.
In order to have the new BUG() not trigger on valid data,
bump value to the correct size, this is userspace so the additional
60 bytes of stack usage is no concern.
counter_arg : PACKETS NUM { $<stmt>0->counter.packets = $2; }
[..]
This has 'counter_stmt_alloc' EITHER return counter or objref statement.
Both are the same structure but with different (union'd) trailer content.
counter_stmt permits the 'packet' and 'byte' argument.
But the 'counter_arg' directive only works with a statement
coming from counter_stmt_alloc().
afl++ came up with following input:
table inet x {
chain y {
counter name ip saddr bytes 1.1.1. 1024
}
}
This clobbers $<stmt>->objref.expr pointer, we then crash when
calling expr_evaluate() on it.
Split the objref related statements into their own directive.
After this, the input will fail with:
"syntax error, unexpected bytes, expecting newline or semicolon".
Also split most of the other objref statements into their own blocks.
synproxy seems to have same problem, limit and quota appeared to be ok.
v1 added objref_stmt to stateful_stmt list, this is wrong, we will
assert when generating the 'counter' statement.
Place it in the normal statement list so netlink_gen_stmt_stateful_assert
throws the expected parser error.
tests: py: missing json output in meta.t with vlan mapping
Fix this warning due to missing coverage:
tests/py/any/meta.t.json.got: WARNING: line 2: Wrote JSON equivalent for rule meta mark set vlan id map { 1 : 0x00000001, 4095 : 0x00004095 }
ERROR: did not find JSON equivalent for rule 'meta mark set vlan id map @map1
Fixes: 8d3de823b622 ("evaluate: reset statement length context before evaluating statement") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
evaluate: reset statement length context before evaluating statement
This patch consolidates ctx->stmt_len reset in stmt_evaluate() to avoid
this problem. Note that stmt_evaluate_meta() and stmt_evaluate_ct()
already reset it after the statement evaluation.
Moreover, statement dependency can be generated while evaluating a meta
and ct statement. Payload statement dependency already manually stashes
this before calling stmt_evaluate(). Add a new stmt_dependency_evaluate()
function to stash statement length context when evaluating a new statement
dependency and use it for all of the existing statement dependencies.
Florian also says:
'meta mark set vlan id map { 1 : 0x00000001, 4095 : 0x00004095 }' will
crash. Reason is that the l2 dependency generated here is errounously
expanded to a 32bit-one, so the evaluation path won't recognize this
as a L2 dependency. Therefore, pctx->stacked_ll_count is 0 and
__expr_evaluate_payload() crashes with a null deref when
dereferencing pctx->stacked_ll[0].
nft-test.py gains a fugly hack to tolerate '!map typeof vlan id : meta mark'.
For more generic support we should find something more acceptable, e.g.
!map typeof( everything here is a key or data ) timeout ...
tests/py update and assert(pctx->stacked_ll_count) by Florian Westphal.
Fixes: edecd58755a8 ("evaluate: support shifts larger than the width of the left operand") Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org> Signed-off-by: Florian Westphal <fw@strlen.de>
parser: tcpopt: fix tcp option parsing with NUM + length field
tcp option 254 length ge 4
... will segfault.
The crash bug is that tcpopt_expr_alloc() can return NULL if we cannot
find a suitable template for the requested kind + field combination,
so add the needed error handling in the bison parser.
However, we can handle this. NOP and EOL have templates, all other
options (known or unknown) must also have a length field.
So also add a fallback template to handle both kind and length, even
if only a numeric option is given that nft doesn't recognize.
Don't bother with output, above will be printed via raw syntax, i.e.
tcp option @254,8,8 >= 4.
Fixes: 24d8da308342 ("tcpopt: allow to check for presence of any tcp option") Reported-by: Maciej Żenczykowski <zenczykowski@gmail.com> Signed-off-by: Florian Westphal <fw@strlen.de>
prefer
binop_with_different_basetype_assert:3:29-35: Error: Binary operation (<<) with different base types (string vs integer) is not supported
oifname set ip9dscp << 26 | 0x10
^^^^^^^~~~~~~
to assertion failure.
Thomas Haller [Fri, 24 Nov 2023 12:45:53 +0000 (13:45 +0100)]
tests/shell: use generated ruleset for `nft --check`
The command `nft [-j] list ruleset | nft [-j] --check -f -` should never
fail. "test-wrapper.sh" already checks for that.
However, previously, we would run check against the .nft/.json-nft
files. In most cases, the generated ruleset and the files in git are
identical. However, when they are not, we (also) want to run the check
against the generated one.
This means, we can also run this check every time, regardless whether a
.nft/.json-nft file exists.
If the .nft/.json-nft file is different from the generated one, (because
a test was skipped or because there is a bug), then also check those
files. But this time, any output is ignored as failures are expected
to happen. We still run the check, to get additional coverage for
valgrind or santizers.
Signed-off-by: Thomas Haller <thaller@redhat.com> Signed-off-by: Florian Westphal <fw@strlen.de>
Phil Sutter [Sat, 2 Dec 2023 10:10:34 +0000 (11:10 +0100)]
main: Refer to nft_options in nft_options_check()
Consult the array when determining whether a given option is followed by
an argument or not instead of hard-coding those that do. The array holds
both short and long option name, so one extra pitfall removed.
Signed-off-by: Phil Sutter <phil@nwl.cc> Signed-off-by: Florian Westphal <fw@strlen.de>
projects / thirdparty / nftables.git / log
