Arne Welzel [Sat, 17 Feb 2024 17:19:27 +0000 (18:19 +0100)]
stats: Fix non-worker stats missing
Commit b8b8aa69b49ac0dd222446c28d00a50f9fd7d716 used tm_name of the
first StatsRecord of a thread block as key for the "threads" object.
However, depending on the type of thread, tm_name can be NULL and would
result in no entry being included for that thread at all. This caused
non-worker metrics to vanish from the "threads" object in the
dump-counters output.
This patch fixes this by remembering the first occurrence of a valid
tm_name within the per-thread block and adds another unittest to
cover this scenario.
Victor Julien [Thu, 29 Feb 2024 10:02:40 +0000 (11:02 +0100)]
rust: update parser dependencies
Time locked to 0.3.20 to guarantee MSRV of 1.63.
Update snmp-parser to 0.10.0.
Update asn1-rs to 0.6.1.
Update kerberos-parser to 0.8.0.
Update x509-parser 0.16.0.
Update der-parser to 9.0.0.
Remove specific use of der-parser 6.
Shivani Bhardwaj [Wed, 28 Feb 2024 14:29:04 +0000 (19:59 +0530)]
detect/port: remove SigGroupHead* ops
The functions in detect-engine-port.c are only being used at the time of
parsing the ports from rules initially. Since there are no SGHs at that
point, remove the ops related to them too.
Victor Julien [Mon, 26 Feb 2024 16:08:21 +0000 (21:38 +0530)]
detect/port: use qsort instead of insert sort
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.
Shivani Bhardwaj [Wed, 21 Feb 2024 06:42:30 +0000 (12:12 +0530)]
detect/port: merge port ranges for same signatures
To avoid getting multiple entries in the final port list and to also
make the next step more efficient by reducing the size of the items to
traverse over.
Shivani Bhardwaj [Tue, 20 Feb 2024 16:22:38 +0000 (21:52 +0530)]
detect/port: create list of small port ranges
Using the unique port points, create a list of small port ranges which
contain the DetectPort objects and the designated SGHs found by finding
the overlaps with the existing ports and copying the SGHs accordingly.
Shivani Bhardwaj [Fri, 16 Feb 2024 09:18:46 +0000 (14:48 +0530)]
detect/port: create a tree of given ports
After all the SGHs have been appropriately copied to the designated
ports, create an interval tree out of it for a faster lookup when later
a search for overlaps is made.
Shivani Bhardwaj [Fri, 16 Feb 2024 08:57:52 +0000 (14:27 +0530)]
detect/port: find unique port points
In order to create the smallest possible port ranges, it is convenient
to first have a list of unique ports. Then, the work becomes simple. See
below:
Given, a port range P1 = [1, 8]; SGH1
and another, P2 = [3, 94]; SGH2
right now, the code will follow a logic of recursively cutting port
ranges until we create the small ranges. But, with the help of unique
port points, we get, unique_port_points = [1, 3, 8, 94]
So, now, in a later stage, we can create the ranges as
[1, 2], [3, 7], [8, 8], [9, 94] and copy the designated SGHs where they
belong. Note that the intervals are closed which means that the range
is inclusive of both the points.
The final result becomes:
1. [1, 2]; SGH1
2. [3, 7]; SGH1 + SGH2
3. [8, 8]; SGH1 + SGH2
4. [9, 94]; SGH2
There would be 3 unique rule groups made for the case above.
Group 1: [1, 2]
Group 2: [3, 7], [8, 8]
Group 3: [9, 94]
Warning was:
src/util-port-interval-tree.c:50:1: warning: Either the condition 'tmp!=NULL' is redundant or there is possible null pointer dereference: tmp. [nullPointerRedundantCheck]
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Assuming that condition 'tmp!=NULL' is not redundant
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Null pointer dereference
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: warning: Either the condition 'oleft!=NULL' is redundant or there is possible null pointer dereference: oleft. [nullPointerRedundantCheck]
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Assuming that condition 'oleft!=NULL' is not redundant
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Null pointer dereference
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: warning: Either the condition 'oright!=NULL' is redundant or there is possible null pointer dereference: oright. [nullPointerRedundantCheck]
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Assuming that condition 'oright!=NULL' is not redundant
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Null pointer dereference
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: warning: Either the condition 'left!=NULL' is redundant or there is possible null pointer dereference: left. [nullPointerRedundantCheck]
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Assuming that condition 'left!=NULL' is not redundant
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
src/util-port-interval-tree.c:50:1: note: Null pointer dereference
IRB_GENERATE(PI, SCPortIntervalNode, irb, SCPortIntervalCompareAndUpdate);
^
Shivani Bhardwaj [Fri, 16 Feb 2024 08:07:23 +0000 (13:37 +0530)]
util/interval-tree: add utility fns
Add new utility files to deal with the interval trees. These cover the
basic ops:
1. Creation/Destruction of the tree
2. Creation/Destruction of the nodes
It also adds the support for finding overlaps for a given set of ports.
This function is used by the detection engine is the Stage 2 of
signature preparation.
Shivani Bhardwaj [Mon, 29 Jan 2024 06:08:51 +0000 (11:38 +0530)]
interval-tree: add augmentation fns to the tree
An interval tree uses red-black tree as its base data structure and
follows all the properties of a usual red-black tree. The additional
params are:
1. An interval such as [low, high] per node.
2. A max attribute per node. This attribute stores the maximum high
value of any subtree rooted at this node.
At any point in time, an inorder traversal of an interval tree should
give the port ranges sorted by the low key in ascending order.
This commit modifies the IRB_AUGMENT macro and it's call sites to make
sure that on every insertion, the max attribute of the tree is properly
updated.
Victor Julien [Fri, 12 Jan 2024 07:03:06 +0000 (12:33 +0530)]
detect/engine: fix whitelisting check
In the commit 4a00ae607, the whitelisting check was updated in a quest
to make use of the conditional better but it made things worse as every
range would be whitelisted as long as it had any of the default
whitelisted port which is very common.
Jason Ish [Wed, 28 Feb 2024 21:21:57 +0000 (15:21 -0600)]
examples: minimal example capture plugin for ci
Create a mininal capture plugin that injects one packet. While it can
also be a template, we should be able to run this in CI to test the
loading and registration of the capture plugin mechanisms.
Jason Ish [Tue, 27 Feb 2024 22:07:33 +0000 (16:07 -0600)]
thread modules: separate initialization from registration
Move the zero'ing to the thread module array InitGlobal in an effort
to fix capture modules.
At some point device validation moved to a point in startup before
plugins are loaded meaning that capture plugins could not be
used. Moving plugin registration early enough caused some of their
registration to be wiped out as clearing the array was done after.
It matches the rrtype field in DNS
It's an unsigned integer match
valid ranges = [0-65535]
Does not support prefilter
Supports flow in both directions
Feature #6666
We only have more logging to do if the app update was fresh,
ie if p->app_update_direction != 0
If we have data acknowledged in one direction,
and then many packets in the other direction,
the APP_UPDATED flow flags did not get reset because we did not
run detection yet in this direction,
but there is nothing more to do after the first packet in the
other direction.
Giuseppe Longo [Tue, 1 Aug 2023 19:20:58 +0000 (21:20 +0200)]
rust/sip: add direction to transaction
This patch permits to set a direction when a new transaction is created in order
to avoid 'signature shadowing' as reported by Eric Leblond in commit 5aaf50760f546e9047da508f725f43a7ad9b8a35
Jason Ish [Thu, 22 Feb 2024 17:24:52 +0000 (11:24 -0600)]
examples/lib: work with bundled libhtp
The simple example Makefile.am was unconditionally including
$(HTP_LDADD) which might be empty resulting in "../.." ending up in the
Makefile causing the build to fail.
Instead, also make HTP_LDADD a conditional, so we can only include it
when actually set, and its only set when libhtp is bundled.
The reason this Makefile needs to include the path components "../.."
is because the HTP_LDADD value is relative to the top level "src/"
directory.
Lukas Sismis [Tue, 6 Feb 2024 10:38:19 +0000 (11:38 +0100)]
tcp: do not assign TCP flags to pseudopackets
Previously pseudopackets were assigned with ACK flag which falsely turned
"SYN" flows to "SYN/ACK" flows when Suricata ran with raw content-matching
rules. The problem occured during the flow timeout or Suricata shutdown,
essentially, when the flow was being kicked out (with a pseudopacket).
When Suricata ran without raw content-matching rules (the ruleset did not
contain content matching keywords or it only contained keywords that are
app-layer content-matching) then raw stream reassembly tracking is turned off
(SignatureHasStreamContent()).
This in turn disabled a check in StreamNeedsReassembly() and the right edge
was not checked with the raw stream progress. In turn, it did not generate
a pseudopacket that would go through the detection engine. Suricata with
raw content-matching keywords would therefore on a flow with SYN packet only
return STREAM_HAS_UNPROCESSED_SEGMENTS_NEED_ONLY_DETECTION which would generate
the pseudopacket.
In Suricata versions <= 6.0.x, the flow output was correct because
only the commit 1bb6f44ff01363fa29488f1ae83b9368e33c2770 started to
differentiate the right edge calculation between the raw and application
layer streams. The older Suricata versions used only the application layer
right edge equation and therefore did not generate a pseudopacket.
Lukas Sismis [Sat, 10 Feb 2024 19:04:55 +0000 (20:04 +0100)]
hugepages: run hugepage check only on DPDK runmode and on Linux
Previous implementation allowed FreeBSD to enter into the hugepage
analysis. It then failed with an error message because hugepage/
NUMA node paths that are used in the codebase to retrieve info about
the system are not the same with the structure in Linux.
Additionally, the messages were logged on error level. It has been
demoted to info level because the whole hugepage analysis checkup is
only for informational purposes and does not affect Suricata operation.
The hugepage analysis and the hugepage snapshots are now limited to
only run in the DPDK runmode.
Victor Julien [Sat, 17 Feb 2024 09:52:59 +0000 (10:52 +0100)]
runmode/unix-socket: fix cppcheck warnings
src/runmode-unix-socket.c:547:9: warning: %d in format string (no. 1) requires 'int' but the argument type is 'unsigned int'. [invalidPrintfArgType_sint]
snprintf(tstr, sizeof(tstr), "%d", cfile->tenant_id);
^
src/runmode-unix-socket.c:1040:5: warning: %d in format string (no. 1) requires 'int' but the argument type is 'unsigned int'. [invalidPrintfArgType_sint]
snprintf(prefix, sizeof(prefix), "multi-detect.%d", tenant_id);
^
src/runmode-unix-socket.c:1189:5: warning: %d in format string (no. 1) requires 'int' but the argument type is 'unsigned int'. [invalidPrintfArgType_sint]
snprintf(prefix, sizeof(prefix), "multi-detect.%d", tenant_id);
^
Jason Ish [Thu, 15 Feb 2024 15:19:14 +0000 (09:19 -0600)]
build: remove autoconf.h from dist
We were including "autoconf.h" in our dist releases, however this file
is generated by ./configure. This can be problematic in out of tree
builds as the one included in the dist, not the generated one may be
used.
Also using "autoconf.h" instead of <autoconf.h> so it is searched for
relative to the file including it which is necessary if a library/plugin
does something like "#include "suricata/foo.h".
Jason Ish [Mon, 5 Feb 2024 22:49:45 +0000 (16:49 -0600)]
examples: program linking against library
Provide an example of an extremely simple application that links
against Suricata. This provides a Makefile integrated with the
Suricata build system for in-tree building, as well as an example
Makefile for building out of tree.
Currently this application just wraps SuricataMain and does nothing
else.
Jason Ish [Wed, 14 Feb 2024 21:23:44 +0000 (15:23 -0600)]
examples/plugin: simplify Makefile
Simplify the Makefile by avoiding automake and providing our own
Makefile.in that is suitable for in-tree builds of the plugin and can
also serve as an example for standalone plugins.
But the bigger benefit of this is to allow building the example plugin
even with --disable-shared provided to configure, as this is just a
phony limitation imposed by automake/libtool.
Jason Ish [Tue, 13 Feb 2024 19:38:57 +0000 (13:38 -0600)]
cocci/run-check: log if parallel command is not found
If CONCURRENCY_LEVEL was set, the script would log a concurrency level
even if the parallel command was not available. Not log if parallel is
not available and set concurrency to 1.
Arne Welzel [Mon, 5 Feb 2024 16:45:30 +0000 (17:45 +0100)]
stats: Do not expand dots of tm_name
When an interface with dots is used, per worker stats are nested by the
dot-separated-components of the interface due to the usage of
OutputStats2Json().
Prevent this by using OutputStats2Json() on a per-thread specific object
and setting this object into the threads object using the
json_object_set_new() which won't do the dot expansion.
This was tested by creating an interface with dots in the name
and checking the stats.
ip link add name a.b.c type dummy
With Suricata 7.0.2, sniffing on the a.b.c interface results in the
following worker stats format:
Simon Dugas [Fri, 29 Dec 2023 16:58:50 +0000 (11:58 -0500)]
detect-engine-iponly: improve ip list performance
The runtime complexity of insertion sort is approx. O(h*n)^2 where
h is the size of the HOME_NET and n is the number of ip only rules
that use the HOME_NET.
Replacing this with qsort significantly improves rule load time when
a large HOME_NET is used in combination with a moderate amount of ip
only rules.
Philippe Antoine [Thu, 25 Jan 2024 13:26:09 +0000 (14:26 +0100)]
detect: respect directionality for filestore
Ticket: 6617
So that rules with keyword like `filestore:to_server,flow`
only store the files to server and not the ones to client...
Directionality only worked with the default scope, ie the
current file, and not the scope tx or scope flow.
For non-default scope, tx or flow, both directions were stored
whatever the directionality specified.
For these non-default scopes, this commit keeps a default
of both directions, but use only one direction if specified.
Need to split flag FLOWFILE_STORE per direction, so that Suricata
can retain this (optional) directional info from the filestore
keyword.
Fixes: 79499e476979 ("app-layer: move files into transactions")
Victor Julien [Tue, 13 Feb 2024 08:51:15 +0000 (09:51 +0100)]
multi-tenant: fix coverity warning
Rework locking logic to avoid the following coverity warning.
** CID 1591966: Concurrent data access violations (MISSING_LOCK)
/src/detect-engine-loader.c: 475 in DetectLoadersSync()
474 SCCtrlMutexLock(loader->tv->ctrl_mutex);
>>> CID 1591966: Concurrent data access violations (MISSING_LOCK)
>>> Accessing "loader->tv" without holding lock "DetectLoaderControl_.m". Elsewhere, "DetectLoaderControl_.tv" is written to with "DetectLoaderControl_.m" held 1 out of 1 times (1 of these accesses strongly imply that it is necessary).
475 pthread_cond_broadcast(loader->tv->ctrl_cond);
476 SCCtrlMutexUnlock(loader->tv->ctrl_mutex);
Jason Ish [Tue, 13 Feb 2024 16:08:37 +0000 (10:08 -0600)]
dependabot: disable rust checks
As we don't have a Cargo.toml and a Cargo.lock, dependabot for Rust
hasn't been working correctly. Disable, as we now have our own cargo
audit and update workflows.
projects / thirdparty / suricata.git / log
