Victor Julien [Mon, 25 Apr 2022 16:00:24 +0000 (18:00 +0200)]
detect: fix rule inspection order
Fix rules from the 'match' list getting added to the tx candidates list
unsorted. In some cases this could lead to the same sid getting inspected
twice leading to a DEBUG_VALIDATION_BUG_ON trigger.
Arne Welzel [Sat, 12 Feb 2022 16:49:07 +0000 (17:49 +0100)]
flow-manager: fix off-by-one in flow_hash row allocation
The current code doesn't cover all rows when more than one flow manager is
used. It leaves a single row between ftd->max and ftd->min of the next
manager orphaned. As an example:
Shivani Bhardwaj [Fri, 28 Jan 2022 20:17:17 +0000 (01:47 +0530)]
detect/dataset: fix space condition in rule lang
If there is a space following a keyword that does not expect a value,
the rule fails to load due to improper value evaluation.
e.g. Space after "set" command
alert http any any -> any any (http.user_agent; dataset:set ,ua-seen,type string,save datasets.csv; sid:1;)
gives error
[ERRCODE: SC_ERR_UNKNOWN_VALUE(129)] - dataset action "" is not supported.
Fix this by handling values correctly for such cases.
Victor Julien [Fri, 22 Apr 2022 16:27:15 +0000 (18:27 +0200)]
stream: improve last_ack validation check
If a packet after the initialization would come with ACK flag set
but a ACK value of 0, the last_ack tracking could get confused. Fix
this by not checking for 0 but instead checking if the ACK flag
has been seen.
Jason Ish [Wed, 6 Apr 2022 21:38:35 +0000 (15:38 -0600)]
ftp: truncate command data that is too long
FTP control commands will be buffered forever until a new line is seen,
this can lead to memory exhaustion in Suricata.
To fix, set an upper bound, 4096 bytes on the size of the command that
is saved in the transaction. The input continues to be parsed to find
the end of the command so the parser can continue to move onto the next
command.
The result is that the command data in the transaction is truncated,
which also shows up in the ftp transaction logs.
This value is configurable with the max-line-length field in the ftp
app-layer.protocols section.
As FTP doesn't have events at this time, add a new fields to eve-log
that specificy if the request, or the response has been truncated.
The input data received in DATA and BDAT command modes can be huge and
could have important data, like a legit huge email. Therefore, exempt
these from the line buffering limits which were introduced to regulate
the size of lines that we buffer at any point in time.
As a part of this patch, anything that comes under DATA or BDAT is
processed early without buffering as and when it arrives. The ways of
processing remain the same as before.
Issue
-----
So far, with the SMTP parser, we would buffer data up until an LF char
was found indicating the end of one line. This would happen in case of
fragmented data where a line might come broken into multiple chunks.
This was problematic if there was a really long line without any LF
character. It would mean that we'd keep buffering data up until we
encounter one such LF char which may be many many bytes of data later.
Fix
---
Fix this issue by setting an upper limit of 4KB on the buffering of
lines. If the limit is reached then we save the data into current line
and process it as if it were a regular request/response up until 4KB
only. Any data after 4KB is discarded up until there is a new LF char in
the received input.
Cases
-----
1. Fragmentation
The limit is enforced for any cases where a line of >= 4KB comes as diff
fragments that are each/some < 4KB.
2. Single too long line
The limit is also enforced for any cases where a single line exceeds the
limit of buffer.
Jason Ish [Mon, 28 Feb 2022 22:48:34 +0000 (16:48 -0600)]
dns: don't parse a full request during probe if not enough data
If there is more data than a header, but not enough for a complete DNS
message, the hostname parser could return an error causing the probe to
fail on valid DNS messages.
So only parse the complete message if we have enough input data. This is
reliable for TCP as DNS messages are prefixed, but for UDP its just
going to be the size of the input buffer presented to the parser, so
incomplete could still happen.
Jason Ish [Tue, 1 Feb 2022 21:44:43 +0000 (15:44 -0600)]
dns: better error handling when parsing names
The DNS name parser will error out with an error even if the
error is incomplete. Instead of manually generating errors,
use '?' to let the nom error ripple up the error handling chain.
The reason this wasn't done in the first place is this code
predates the ? operator, or we were not aware of it at the time.
This prevents the case where probing fails when there is enough data to
parse the header, but not enough to complete name parser. In such a case
a parse error is returned (instead of incomplete) resulting in the
payload not being detected as DNS.
Maximum length of a base64 encoded string can be 33% over the actual
length of the input string. The formula to best cover all the edge cases
is mathematically
(4 * (input_length + 2) / 3) + 1
Add a macro to calculate this for a given input length.
Jason Ish [Thu, 31 Mar 2022 18:45:07 +0000 (12:45 -0600)]
smb: protocol detection on pattern without midstream
To recognize a protocol, Suricata first looks for
patterns, which can be confirmed by a probing parser.
If this does not work, Suricata can try to run
some probing parsers on some ports.
This is the case for SMB.
This commit makes handling the confirming and the probing
paser differently even if they share much code.
The confirmation parser knows that a pattern has been found.
So, it must not do the midstream case of looking for this
pattern in the whole buffer, but only check it at the beginning.
But it must reverse direction if needed.
Jason Ish [Mon, 18 Apr 2022 16:32:25 +0000 (10:32 -0600)]
smb: fix smb2 header flag parsing
The bits were being parsed in the order they're displayed in Wireshark,
rather than the order they were being seen on the wire, resulting in
direction and async being 0 more often than they should be.
Instead of bits, take the 4 bytes as an le_u32 and just use bit masks to
extract what we need into a struct, I think its easier to reason about
this way when comparing to the Microsoft documentation.
Jason Ish [Tue, 22 Mar 2022 15:46:45 +0000 (09:46 -0600)]
detect-content: error on single char hex pairs
Fix parsing of content like "|aa b cc|" which was parsed as "|aa bc|"
without error or warning. This will now fail out, requiring all hex
values to be 2 chars.
Philippe Antoine [Tue, 25 Jan 2022 20:10:37 +0000 (21:10 +0100)]
smtp: check if we have a current transaction
Ticket: 4948
This is not the perfect solution, but it prevents to trigger
the assert, and keep the assert.
A better solution would need to create transaction from
the reponse parsing, in case a later command was buffered and
not answered. But this would not be enough as NoNewTx prevents
the creation of a new transaction for RSET...
Jason Ish [Tue, 15 Feb 2022 21:18:49 +0000 (15:18 -0600)]
smb: check correct buffer for overflow
Fix an error in the checking of an overflow condition.
The first overflow check is only checking the size of the new data, not
the new data + the size of the buffered data. This is due to the buffer
on the state being emptied into a local variable just before the check.
This results in overflows not being caught, but being caught a few lines
down after the copy resulting in increased CPU usage for data that is
just going to be thrown away.
Victor Julien [Fri, 18 Mar 2022 18:02:45 +0000 (12:02 -0600)]
nfs3: fix partial write record handling
Note: This was more of a manual cherry-pick converting some parsers from
named macros into functions in order to handle more arguments than just
the input data -- Jason Ish
Jeff Lucovsky [Fri, 11 Feb 2022 14:02:39 +0000 (09:02 -0500)]
threads: Honor per-thread stack size setting
Issue: 4550
This commit adjusts the per-thread stack size if a size has been
configured. If the setting has not been configured, the default
per-thread stack size provided by the runtime mechanisms are used.
This commit documents the new per-thread stack-size setting. Some
systems have a small default value that is not suitable for Suricata's
multi-threaded architecture and adjustment may be required.
Jeff Lucovsky [Thu, 6 May 2021 13:49:55 +0000 (09:49 -0400)]
proto: Remove dependency on /etc/protocols
This commit eliminates the dependency on /etc/protocols and equivalent
on other platforms by using a static table of IANA assigned protocol
values (names, description).
Victor Julien [Thu, 17 Feb 2022 12:32:17 +0000 (13:32 +0100)]
radix: improve address range handling
Handle non-exact address ranges from string. This can come directly
from user input, so here it is accepted but the address is converted
to the address range start. A warning will be issued.
Debug validation checks are added to catch this.
This issue could lead to bad input from iprep (with cidr), defrag config
and htp server personalities to produce a bad radix tree.
Victor Julien [Tue, 15 Feb 2022 19:43:27 +0000 (20:43 +0100)]
detect/iponly: fix netmask handling
If the ipaddress was not the address range start, it was not masked to turn
it into that. So 1.2.3.4/24 was not stored as address 1.2.3.0 with netmask 24,
but as 1.2.3.4 with netmask 24. This was then propagated into the radix tree,
where it was used as an exact key in exact lookups, giving unexpected results.
This patch implements the netmask handling for IPv4 and IPv6, and adds a set
of tests for it.
Victor Julien [Fri, 11 Feb 2022 14:50:01 +0000 (15:50 +0100)]
radix: fix FP/FN issue in IP-only
A bug was reported about the IP-only rules not correctly matching. This was
traced to the rules in question not getting recorded into the IP-only radix
tree correctly.
Sequence:
- 100.117.241.0/25 inserted into the tree
- 100.117.241.0/26 inserted into the tree
Both are part of the same radix node, but recorded by their different netmasks
in the user data portion.
The IP-only engine first does a search to get to the user data it may need to
include. It does so for with `SCRadixFindKeyIPV4ExactMatch` for single IPs, or
using `SCRadixFindKeyIPV4Netblock` in case of a netblock. Any "match" from
either of these is considered an "exact match" by the IP-only setup code.
This exact match expectation turned out to be wrong and
`SCRadixFindKeyIPV4Netblock` behaved more like "best match" instead, which is
a non-exact match, but its the next best match if no exact match is found.
The way the look up for 100.117.241.64/26 went wrong, is that it returned
the user data for 100.117.241.0/26. This happens as follows:
- first it would do an exact find, which didn't give a result
- then it removed bits from the keystream until it found a matching node
and explore if any of the netmasks it contained matched. Here the first
step of the bug started:
it considered the netmask (with user data) a match that matched the
number of bits of the matching key, but not of the actual range netmask cidr
value.
So in this case the number of shared bits between `100.117.241.0/25` and
`100.117.241.64/26` was 25, so it assumed that the user data for the
netmask 25 was the match.
To summarize this step, there are 2 problems with this:
1. it returns a match on something that isn't an exact match
2. it considered the wrong netmask value
- the radix code then took the returned node, and did the netmask check
again. This time it did use its own netmask value, so this time
it did find the netmask 26 (+ user data). However because of the node that
was returned, this netmask (+user data) belongs to `100.117.241.0`, not to
`100.117.241.64`.
- the IP-only detection code was satisfied with what it assumed to be
"exact match" and just updated the user data to include the user data that
should have been associated with `100.117.241.64/26` to `100.117.241.0/26`.
This patch addresses the issue as follows:
It makes `SCRadixFindKeyIPV4Netblock` also return an exact match by propagating
the netmask in the search and in the evaluation of the stored netmasks.
It does away with the secondary netmask (+user data) evaluation.
`SCRadixFindKeyIPV4Netblock` is expected to handle this correctly.
The IP-only engine will fall back to the "not found" path, which does an explicit
"best match" lookup and then insert a new entry into the radix tree based on
the user data of the "best match".
Victor Julien [Fri, 18 Feb 2022 09:19:04 +0000 (10:19 +0100)]
output: fix timestamp missing usecs
On ARM 32bit with Musl `tv_usecs` is defined as `int64_t` which lead to
CreateIsoTimeString() printing all zeros on the usecs. Work around this
by first assigning to a `int64_t` and then updating the expected format
string to accept `int64_t`.