Victor Julien [Fri, 18 Feb 2022 09:19:04 +0000 (10:19 +0100)]
output: fix timestamp missing usecs
On ARM 32bit with Musl `tv_usecs` is defined as `int64_t` which lead to
CreateIsoTimeString() printing all zeros on the usecs. Work around this
by first assigning to a `int64_t` and then updating the expected format
string to accept `int64_t`.
Victor Julien [Thu, 17 Feb 2022 12:32:17 +0000 (13:32 +0100)]
radix: improve address range handling
Handle non-exact address ranges from string. This can come directly
from user input, so here it is accepted but the address is converted
to the address range start. A warning will be issued.
Debug validation checks are added to catch this.
This issue could lead to bad input from iprep (with cidr), defrag config
and htp server personalities to produce a bad radix tree.
Victor Julien [Tue, 15 Feb 2022 19:43:27 +0000 (20:43 +0100)]
detect/iponly: fix netmask handling
If the ipaddress was not the address range start, it was not masked to turn
it into that. So 1.2.3.4/24 was not stored as address 1.2.3.0 with netmask 24,
but as 1.2.3.4 with netmask 24. This was then propagated into the radix tree,
where it was used as an exact key in exact lookups, giving unexpected results.
This patch implements the netmask handling for IPv4 and IPv6, and adds a set
of tests for it.
Victor Julien [Fri, 11 Feb 2022 14:50:01 +0000 (15:50 +0100)]
radix: fix FP/FN issue in IP-only
A bug was reported about the IP-only rules not correctly matching. This was
traced to the rules in question not getting recorded into the IP-only radix
tree correctly.
Sequence:
- 100.117.241.0/25 inserted into the tree
- 100.117.241.0/26 inserted into the tree
Both are part of the same radix node, but recorded by their different netmasks
in the user data portion.
The IP-only engine first does a search to get to the user data it may need to
include. It does so for with `SCRadixFindKeyIPV4ExactMatch` for single IPs, or
using `SCRadixFindKeyIPV4Netblock` in case of a netblock. Any "match" from
either of these is considered an "exact match" by the IP-only setup code.
This exact match expectation turned out to be wrong and
`SCRadixFindKeyIPV4Netblock` behaved more like "best match" instead, which is
a non-exact match, but its the next best match if no exact match is found.
The way the look up for 100.117.241.64/26 went wrong, is that it returned
the user data for 100.117.241.0/26. This happens as follows:
- first it would do an exact find, which didn't give a result
- then it removed bits from the keystream until it found a matching node
and explore if any of the netmasks it contained matched. Here the first
step of the bug started:
it considered the netmask (with user data) a match that matched the
number of bits of the matching key, but not of the actual range netmask cidr
value.
So in this case the number of shared bits between `100.117.241.0/25` and
`100.117.241.64/26` was 25, so it assumed that the user data for the
netmask 25 was the match.
To summarize this step, there are 2 problems with this:
1. it returns a match on something that isn't an exact match
2. it considered the wrong netmask value
- the radix code then took the returned node, and did the netmask check
again. This time it did use its own netmask value, so this time
it did find the netmask 26 (+ user data). However because of the node that
was returned, this netmask (+user data) belongs to `100.117.241.0`, not to
`100.117.241.64`.
- the IP-only detection code was satisfied with what it assumed to be
"exact match" and just updated the user data to include the user data that
should have been associated with `100.117.241.64/26` to `100.117.241.0/26`.
This patch addresses the issue as follows:
It makes `SCRadixFindKeyIPV4Netblock` also return an exact match by propagating
the netmask in the search and in the evaluation of the stored netmasks.
It does away with the secondary netmask (+user data) evaluation.
`SCRadixFindKeyIPV4Netblock` is expected to handle this correctly.
The IP-only engine will fall back to the "not found" path, which does an explicit
"best match" lookup and then insert a new entry into the radix tree based on
the user data of the "best match".
With these, the portion of code within the tags should be included
in the related code-snippets (for frame support documentation) w/o
errors, even if the code within changes. The tags can also work as
a reminder that the existing code is being shown elsewhere, so folks
know documentation might need updates, in case of major changes.
With these, the portion of code within the tags should be included
in the related code-snippets (for frame support documentation) w/o
errors, even if the code within changes. The tags can also work as
a reminder that the existing code is being shown elsewhere, so folks
know documentation might need updates, in case of major changes.
Our current rust code isn't always documentation friendly when it
comes to using code snippets. Used rustfmt to apply rust default
formatting on functions that we wanted to show in our documentation
for Frame support
When we want to share our code in our documentation pages, the current
rust formatting isn't so nice to read. Formatted just the portion of
the code that will be shown, for now.
>>> CID 1499365: (UNINIT)
>>> Using uninitialized value "infstream.total_out" when calling "inflate".
98 int result = inflate(&infstream, Z_NO_FLUSH);
99 switch(result) {
100 case Z_STREAM_END:
101 break;
102 case Z_OK:
103 break;
>>> CID 1499365: (UNINIT)
>>> Using uninitialized value "infstream.total_out" when calling "inflate".
98 int result = inflate(&infstream, Z_NO_FLUSH);
99 switch(result) {
100 case Z_STREAM_END:
101 break;
102 case Z_OK:
103 break;
*** CID 1499363: Error handling issues (CHECKED_RETURN)
/src/util-file-swf-decompression.c: 97 in FileSwfZlibDecompression()
91
92 infstream.avail_in = (uInt)compressed_data_len;
93 infstream.next_in = (Bytef *)compressed_data;
94 infstream.avail_out = (uInt)decompressed_data_len;
95 infstream.next_out = (Bytef *)decompressed_data;
96
>>> CID 1499363: Error handling issues (CHECKED_RETURN)
>>> Calling "inflateInit_(&infstream, "1.2.11", 112)" without checking return value. This library function may fail and return an error code.
97 inflateInit(&infstream);
98 int result = inflate(&infstream, Z_NO_FLUSH);
99 switch(result) {
100 case Z_STREAM_END:
101 break;
102 case Z_OK:
Jason Ish [Fri, 11 Feb 2022 15:26:06 +0000 (09:26 -0600)]
build: remove configure check for cargo vendor
cargo vendor has been part of the core cargo command since Rust 1.37,
and are minimum Rust version is not 1.41, so remove the check. Its
always available now.
Jason Ish [Mon, 31 Jan 2022 18:06:55 +0000 (12:06 -0600)]
devguide: move image generation into sphinx setup
While ReadTheDocs can't execute arbitrary scripts, we can install
mscgen in the container that builds the docs (see .readthedocs.yaml).
Then instead of calling generate-images.sh from the Makefile, move this
into the setup function defined on conf.yaml, which will generate the
scripts as part of a normal Sphinx workflow.
This should give us an image generation solution that works on
ReadTheDocs, and locally within anyones build system provided they have
mscgen installed.
Philippe Antoine [Tue, 25 Jan 2022 20:10:37 +0000 (21:10 +0100)]
smtp: check if we have a current transaction
Ticket: 4948
This is not the perfect solution, but it prevents to trigger
the assert, and keep the assert.
A better solution would need to create transaction from
the reponse parsing, in case a later command was buffered and
not answered. But this would not be enough as NoNewTx prevents
the creation of a new transaction for RSET...
Parse extract-url-schemes from the mime config.
e.g. 'extract-urls-schemes: [http, https, ftp, mailto]'
Update MimeDecConfig struct to new url extraction fields.
Change app-layer-smtp.c & util-decode-mime.c to initialize new struct
fields for MimeDecConfig.
Sets the default value for extract-url-schemes if not found in the
config to 'extract-urls-schemes: [http]' for backwards compatibility.
Uses the schemes defined in the mime config value for
extract-urls-schemes to search for URLS starting with those scheme
names followed by "://".
Logs the URLS with the scheme + '://' at the start if the
log-url-scheme is set in the mime config, otherwise the old behaviour
is reverted to and the urls are logged with the schemes stripped.
Removed unused constant URL_STR now that URLS are being searched for
using extract-urls-schemes mime config values instead of just URL's
starting with 'http://'.
Added commented out new options for extract-urls-schemes and
log-url-scheme to suricata.yaml.in.
Update FindUrlStrings comments.
Remove old outdated comments/commented code from FindUrlStrings.
Update test case for mime which now needs schemes list to be set.
Add Test Cases for FindUrlStrings() method.
Philippe Antoine [Tue, 25 Jan 2022 08:01:54 +0000 (09:01 +0100)]
http2: event for variable-length integer overflow
http2_parse_var_uint can overflow the variable-length
integer it is decoding. In this case, it now returns an error
of kind LengthValue.
The new function http2_parse_headers_blocks, which factorizes
the code loop for headers, push promise, and continuation, will
check for this specific error, and instead of erroring itself,
will return the list of so far parsed headers, plus another one
with HTTP2HeaderDecodeStatus::HTTP2HeaderDecodeIntegerOverflow
This status is then checked by process_headers to create an
app-layer event.
Philippe Antoine [Fri, 28 Jan 2022 13:02:32 +0000 (14:02 +0100)]
detect: checking validity of rules with http protocol
We want to check that a rule beginning with alert http
can be valid, that is if either HTTP1 or HTTP2 is enabled.
So, AppLayerProtoDetectGetProtoName will do a more complex
check for this ALPROTO_HTTP (any).
projects / thirdparty / suricata.git / log
