Jason Ish [Fri, 27 Oct 2023 16:19:31 +0000 (10:19 -0600)]
dns/eve: use default formats if formats is empty
If the configuration field "formats" is empty, DNS response records do
not have any relevant information other than that there was a
response, but not much about the response.
I'm pretty sure the intention here was to log the response details if
no formats were provided, which is what happens when the field is
commented out.
So if no formats are specified, use the default of all.
Make sure Suricata is in the running state before
you attempt to execute commands on the Unix sockets.
UnixMain is being called in an infinite loop where
TmThreadsCheckFlag(th_v, THV_KILL) is checked for the
deinit phase. However, it may take some time between
the start of Suricata's deinitialization and
the receipt of THV_KILL flag in the Unix thread.
In between this time period, the Unix manager can still
perform select() operation on the Unix socket while
the socket being already deinitialized.
Likely with a longer time span between the initial shutdown
command and actual closing of Unix sockets resulted in
an error of invalid file descriptors.
Victor Julien [Fri, 13 Oct 2023 11:47:05 +0000 (13:47 +0200)]
detect: inspect all packets in multi-layer tunneling
When the decoders encounter multiple layers of tunneling, multiple tunnel
packets are created. These are then stored in ThreadVars::decode_pq, where
they are processed after the current thread "slot" is done. However, due
to a logic error, the tunnel packets after the first, where not called
for the correct position in the packet pipeline. This would lead to these
packets not going through the FlowWorker module, so skipping everything
from flow tracking, detection and logging.
This would only happen for single and workers, due to how the pipelines
are constructed.
The "slot" holding the decoder, would contain 2 packets in
ThreadVars::decode_pq. Then it would call the pipeline on the first
packet with the next slot of the pipeline through a indirect call to
TmThreadsSlotVarRun(), so it would be called for the FlowWorker.
However when that first (the most inner) packet was done, the call
to TmThreadsSlotVarRun() would again service the ThreadVars::decode_pq
and process it, again moving the slot pointer forward, so past the
FlowWorker.
This patch addresses the issue by making sure only a "decode" thread
slot will service the ThreadVars::decode_pq, thus never moving the
slot past the FlowWorker.
Lukas Sismis [Wed, 16 Aug 2023 21:51:10 +0000 (23:51 +0200)]
dpdk: add hugepage hint to lower the amount of reserved hugepages
If a user allocates too many hugepages and those are largely not used
then Suricata suggests that the user can lower the amount of hugepages
and therefore save memory for other purposes.
Our documentation was failing to build, seems connected to the new way
of indicating build options (cf
https://readthedocs.org/projects/suricata/builds/22112658/,
https://docs.readthedocs.io/en/stable/config-file/v2.html#build,
and https://docs.readthedocs.io/en/stable/config-file/v2.html#build-os).
Added the build.os required new field, and adjusted the way python
version is passed.
For the new configuration style for read the docs, one of the ways to
pass extra configuration for python is having a requirements file.
email_ctx->fields only gets populated when smtp.custom setting is on.
The fn EveEmailLogJSONCustom is called when either
1. smtp.extended setting is on or,
2. email_ctx->fields is populated which means smtp.custom setting is on
In case neither of these are set in suricata.yaml, no call should
ideally be made to the fn EveEmailLogJSONCustom.
However, it turns out that email_ctx->fields is unset and then set only
after the smtp config was found. This leads to email_ctx->fields
sometimes contain value even when no config was given to the smtp
section and can lead to unexpected output.
Fix this by using SCCalloc while initializing OutputJsonEmailCtx struct
instead of SCMalloc.
Although we have an updated version of instructions for installation
from git, our install guide was only referring to RedMine, which is less
up-to-date.
Kept that reference, since it might still be useful for non-Ubuntu
cases.
Philippe Antoine [Wed, 27 Sep 2023 12:15:18 +0000 (14:15 +0200)]
detect: check if signature uses too many buffers
Ticket: #6104
The approach in master branch is to change the prototype of
SigMatchAppendSMToList so that it allocates itself the new SigMatch
This approach requires to change all the 100-ish calls to
SigMatchAppendSMToList and is thus quite a big change.
For branch 7, we still wanted to avoid the buffer overflow, but
did not want such an intrusive change, and still wanted to make
the signature invalid. Instead of changing the prototype of the
function, we make it return early, and set a flag in the signature
which can be later checked by SigValidate
Jason Ish [Thu, 28 Sep 2023 22:19:51 +0000 (16:19 -0600)]
readme: add a resources section
Consolidate a few items into a resources section, and add few more
items, most importantly the bug tracker as it can't currently be found
from our GitHub presence.
dpdk: reset stats just before the start of packet receive loop
While Suricata initializes, the device must be started to e.g.
apply rte_flow rules on some devices. But in the meantime, the
NIC started receiving packets but accounted those as missed.
Stats reset was added to better represent true packet drop.
dpdk: stop devices immediately after Suricata stop command
To better represent port stats of Suricata stop the device
right after Suricata is deemed to shut down.
While Suricata deinitialization happened, the device kept
receiving packets. But because Suricata was no longer
interested in the packets, the device accounted these packets
as missed and it could have alter true stats of Suricata.
Shivani Bhardwaj [Thu, 10 Aug 2023 15:35:45 +0000 (21:05 +0530)]
af-packet: fetch mtu info once
With the current layout and fn calls, it was seen that once in the
beginning after the MTU was found and displayed to the user, when the
threads spawned, each thread displayed MTU info as a part of
AFPPeersListAdd fn. This happened in AF_PACKET IPS mode and led to
excessive MTU logs.
Save this info in the LiveDevice struct and avoid calling the unneeded
fns later on.
Shivani Bhardwaj [Thu, 10 Aug 2023 15:34:56 +0000 (21:04 +0530)]
util/ioctl: use LiveDevice to retrieve name
The fn GetIfaceMaxPacketSize now uses LiveDevice object as a param
instead of a string. This was done to keep the logic of checking for the
device to this function itself instead of having callers first determine
whether the device exists or not.
This also falls in line with the changes made to avoid excessive MTU
logs in the following commit.
Philippe Antoine [Wed, 30 Aug 2023 19:35:08 +0000 (21:35 +0200)]
smtp: fix null deref with config option body md5
Ticket: #6279
If we have the smtp body beginning without headers, we need to
create the md5 context and right away and supply data to it.
Otherwise, on the next line being processed, md5_ctx will be
NULL but body_begin will have been reset to 0
Jason Ish [Tue, 5 Sep 2023 21:06:24 +0000 (15:06 -0600)]
configure: check for new enough sphinx-build
We need a recent version of Sphinx to build the documentation in
7.0. Check for a minimum version of 3.4.3. If older, do not build the
docs which is the same behavior when sphinx-build is not found.
Arne Welzel [Sun, 20 Aug 2023 15:32:47 +0000 (17:32 +0200)]
community-id: Fix IPv6 address sorting not respecting byte order
When comparing IPv6 addresses based on uint32_t chunks, one needs to
apply ntohl() conversion to the individual parts, otherwise on little
endian systems individual bytes are compared in the wrong order.
Avoid this all and leverage memcmp(), it'll short circuit on the first
differing byte and its return values tells us which address sorts lower.
Ralph Eastwood [Wed, 16 Aug 2023 11:30:01 +0000 (13:30 +0200)]
napatech: print NUMA recommendation early
When thread affinity is set, the NUMA configuration specified in
the napatech.ini configuration could be incorrect and then fail.
This fails before the recommended configuration is printed, which
is pretty unhelpful.
Ralph Eastwood [Thu, 3 Aug 2023 13:32:10 +0000 (15:32 +0200)]
napatech: fix warnings with ByteExtractStringUint8
The WARN_UNUSED attribute has been added to ByteExtractStringUint8
in commit 698816811406572c443ca1e95c309d292f489376. The return
value is now handled and appropriate errors printed.
Ralph Eastwood [Thu, 3 Aug 2023 13:03:41 +0000 (13:03 +0000)]
napatech: fix compilation errors in SCLog calls
Since f8474344cdd00e3d128ffc3ec6d7e465bbe2894d, there is an extra
argument to SCLog which indicates the module and subsystem
identifier. The Napatech vendor code is missing this argument,
which is fixed here.
* Log vendor client identifier (dhcp option 60) if extended dhcp
logging is turned on. This required the `vendor_client_identifier` to
be added to the json schema. Validation done using an SV Test
* Added `requested_ip` to the json schema as well, since it was
missed. My SV test failed without it.
Thomas Winter [Mon, 15 May 2023 02:18:47 +0000 (14:18 +1200)]
iprep: fix parsing ip-rep data with carriage return
Commit e7c0f0ad91fd removed uses of atoi with a new number parsing
functions. This broke parsing ip-reputation data files that contained
trailing carriage returns as it was being included in the number
string to convert.
Victor Julien [Thu, 10 Aug 2023 14:31:29 +0000 (16:31 +0200)]
threading: set min thread stack size; set it early
Multi-tenancy uses loader threads that initialize detection engines. During
this, esp the AC family of MPM implementations, there is significant stack
usage. In most OS' threads have a lower stack size by default. In Linux, when
using the Musl C library, a thread by default gets 128KiB.
This patch does 2 things:
1. it centralizes the handling of the `threading.stack-size`. It it is not
longer handled by the runmodes, but called from the global initialization
logic.
2. it sets a minimum per thread stack size of 512k, unless `threading.stack-size`
is set.
projects / thirdparty / suricata.git / log
