Victor Julien [Fri, 13 Oct 2023 11:47:05 +0000 (13:47 +0200)]
detect: inspect all packets in multi-layer tunneling
When the decoders encounter multiple layers of tunneling, multiple tunnel
packets are created. These are then stored in ThreadVars::decode_pq, where
they are processed after the current thread "slot" is done. However, due
to a logic error, the tunnel packets after the first, where not called
for the correct position in the packet pipeline. This would lead to these
packets not going through the FlowWorker module, so skipping everything
from flow tracking, detection and logging.
This would only happen for single and workers, due to how the pipelines
are constructed.
The "slot" holding the decoder, would contain 2 packets in
ThreadVars::decode_pq. Then it would call the pipeline on the first
packet with the next slot of the pipeline through a indirect call to
TmThreadsSlotVarRun(), so it would be called for the FlowWorker.
However when that first (the most inner) packet was done, the call
to TmThreadsSlotVarRun() would again service the ThreadVars::decode_pq
and process it, again moving the slot pointer forward, so past the
FlowWorker.
This patch addresses the issue by making sure only a "decode" thread
slot will service the ThreadVars::decode_pq, thus never moving the
slot past the FlowWorker.
Lukas Sismis [Wed, 16 Aug 2023 21:51:10 +0000 (23:51 +0200)]
dpdk: add hugepage hint to lower the amount of reserved hugepages
If a user allocates too many hugepages and those are largely not used
then Suricata suggests that the user can lower the amount of hugepages
and therefore save memory for other purposes.
Our documentation was failing to build, seems connected to the new way
of indicating build options (cf
https://readthedocs.org/projects/suricata/builds/22112658/,
https://docs.readthedocs.io/en/stable/config-file/v2.html#build,
and https://docs.readthedocs.io/en/stable/config-file/v2.html#build-os).
Added the build.os required new field, and adjusted the way python
version is passed.
For the new configuration style for read the docs, one of the ways to
pass extra configuration for python is having a requirements file.
email_ctx->fields only gets populated when smtp.custom setting is on.
The fn EveEmailLogJSONCustom is called when either
1. smtp.extended setting is on or,
2. email_ctx->fields is populated which means smtp.custom setting is on
In case neither of these are set in suricata.yaml, no call should
ideally be made to the fn EveEmailLogJSONCustom.
However, it turns out that email_ctx->fields is unset and then set only
after the smtp config was found. This leads to email_ctx->fields
sometimes contain value even when no config was given to the smtp
section and can lead to unexpected output.
Fix this by using SCCalloc while initializing OutputJsonEmailCtx struct
instead of SCMalloc.
Although we have an updated version of instructions for installation
from git, our install guide was only referring to RedMine, which is less
up-to-date.
Kept that reference, since it might still be useful for non-Ubuntu
cases.
Philippe Antoine [Wed, 27 Sep 2023 12:15:18 +0000 (14:15 +0200)]
detect: check if signature uses too many buffers
Ticket: #6104
The approach in master branch is to change the prototype of
SigMatchAppendSMToList so that it allocates itself the new SigMatch
This approach requires to change all the 100-ish calls to
SigMatchAppendSMToList and is thus quite a big change.
For branch 7, we still wanted to avoid the buffer overflow, but
did not want such an intrusive change, and still wanted to make
the signature invalid. Instead of changing the prototype of the
function, we make it return early, and set a flag in the signature
which can be later checked by SigValidate
Jason Ish [Thu, 28 Sep 2023 22:19:51 +0000 (16:19 -0600)]
readme: add a resources section
Consolidate a few items into a resources section, and add few more
items, most importantly the bug tracker as it can't currently be found
from our GitHub presence.
dpdk: reset stats just before the start of packet receive loop
While Suricata initializes, the device must be started to e.g.
apply rte_flow rules on some devices. But in the meantime, the
NIC started receiving packets but accounted those as missed.
Stats reset was added to better represent true packet drop.
dpdk: stop devices immediately after Suricata stop command
To better represent port stats of Suricata stop the device
right after Suricata is deemed to shut down.
While Suricata deinitialization happened, the device kept
receiving packets. But because Suricata was no longer
interested in the packets, the device accounted these packets
as missed and it could have alter true stats of Suricata.
Shivani Bhardwaj [Thu, 10 Aug 2023 15:35:45 +0000 (21:05 +0530)]
af-packet: fetch mtu info once
With the current layout and fn calls, it was seen that once in the
beginning after the MTU was found and displayed to the user, when the
threads spawned, each thread displayed MTU info as a part of
AFPPeersListAdd fn. This happened in AF_PACKET IPS mode and led to
excessive MTU logs.
Save this info in the LiveDevice struct and avoid calling the unneeded
fns later on.
Shivani Bhardwaj [Thu, 10 Aug 2023 15:34:56 +0000 (21:04 +0530)]
util/ioctl: use LiveDevice to retrieve name
The fn GetIfaceMaxPacketSize now uses LiveDevice object as a param
instead of a string. This was done to keep the logic of checking for the
device to this function itself instead of having callers first determine
whether the device exists or not.
This also falls in line with the changes made to avoid excessive MTU
logs in the following commit.
Philippe Antoine [Wed, 30 Aug 2023 19:35:08 +0000 (21:35 +0200)]
smtp: fix null deref with config option body md5
Ticket: #6279
If we have the smtp body beginning without headers, we need to
create the md5 context and right away and supply data to it.
Otherwise, on the next line being processed, md5_ctx will be
NULL but body_begin will have been reset to 0
Jason Ish [Tue, 5 Sep 2023 21:06:24 +0000 (15:06 -0600)]
configure: check for new enough sphinx-build
We need a recent version of Sphinx to build the documentation in
7.0. Check for a minimum version of 3.4.3. If older, do not build the
docs which is the same behavior when sphinx-build is not found.
Arne Welzel [Sun, 20 Aug 2023 15:32:47 +0000 (17:32 +0200)]
community-id: Fix IPv6 address sorting not respecting byte order
When comparing IPv6 addresses based on uint32_t chunks, one needs to
apply ntohl() conversion to the individual parts, otherwise on little
endian systems individual bytes are compared in the wrong order.
Avoid this all and leverage memcmp(), it'll short circuit on the first
differing byte and its return values tells us which address sorts lower.
Ralph Eastwood [Wed, 16 Aug 2023 11:30:01 +0000 (13:30 +0200)]
napatech: print NUMA recommendation early
When thread affinity is set, the NUMA configuration specified in
the napatech.ini configuration could be incorrect and then fail.
This fails before the recommended configuration is printed, which
is pretty unhelpful.
Ralph Eastwood [Thu, 3 Aug 2023 13:32:10 +0000 (15:32 +0200)]
napatech: fix warnings with ByteExtractStringUint8
The WARN_UNUSED attribute has been added to ByteExtractStringUint8
in commit 698816811406572c443ca1e95c309d292f489376. The return
value is now handled and appropriate errors printed.
Ralph Eastwood [Thu, 3 Aug 2023 13:03:41 +0000 (13:03 +0000)]
napatech: fix compilation errors in SCLog calls
Since f8474344cdd00e3d128ffc3ec6d7e465bbe2894d, there is an extra
argument to SCLog which indicates the module and subsystem
identifier. The Napatech vendor code is missing this argument,
which is fixed here.
* Log vendor client identifier (dhcp option 60) if extended dhcp
logging is turned on. This required the `vendor_client_identifier` to
be added to the json schema. Validation done using an SV Test
* Added `requested_ip` to the json schema as well, since it was
missed. My SV test failed without it.
Thomas Winter [Mon, 15 May 2023 02:18:47 +0000 (14:18 +1200)]
iprep: fix parsing ip-rep data with carriage return
Commit e7c0f0ad91fd removed uses of atoi with a new number parsing
functions. This broke parsing ip-reputation data files that contained
trailing carriage returns as it was being included in the number
string to convert.
Victor Julien [Thu, 10 Aug 2023 14:31:29 +0000 (16:31 +0200)]
threading: set min thread stack size; set it early
Multi-tenancy uses loader threads that initialize detection engines. During
this, esp the AC family of MPM implementations, there is significant stack
usage. In most OS' threads have a lower stack size by default. In Linux, when
using the Musl C library, a thread by default gets 128KiB.
This patch does 2 things:
1. it centralizes the handling of the `threading.stack-size`. It it is not
longer handled by the runmodes, but called from the global initialization
logic.
2. it sets a minimum per thread stack size of 512k, unless `threading.stack-size`
is set.
Victor Julien [Wed, 2 Aug 2023 06:37:45 +0000 (08:37 +0200)]
var-names: reimplement var name handling
Implement a new design for handling var name id's. The old logic
was aware of detection engine versions and generally didn't work
well for multi-tenancy cases. Other than memory leaks and crashes,
logging of var names worked or failed based on which tenant was
loaded last.
This patch implements a new approach, where there is a global store
of vars and their id's for the lifetime of the program.
Overall Design:
Base Store: "base"
Used during keyword registration. Operates under lock. Base is shared
between all detect engines, detect engine versions and tenants.
Each variable name is ref counted.
During the freeing of a detect engine / tenant, unregistration decreases
the ref cnt.
Base has both a string to id and a id to string hash table. String to
id is used during parsing/registration. id to string during unregistration.
Active Store Pointer (atomic)
The "active" store atomic pointer points to the active lookup store. The call
to `VarNameStoreActivate` will build a new lookup store and hot swap
the pointer.
Ensuring memory safety. During the hot swap, the pointer is replaced, so
any new call to the lookup functions will automatically use the new store.
This leaves the case of any lookup happening concurrently with the pointer
swap. For this case we add the old store to a free list. It gets a timestamp
before which it cannot be freed.
Free List
The free list contains old stores that are waiting to get removed. They
contain a timestamp that is checked before they are freed.
Add direction indication in SCStreamingBuffer usage example.
This adds documentation for the changes introduced by commit 5b1d8c7e94ef613107870d4d9d9cdde76d4c3438.
Victor Julien [Thu, 3 Aug 2023 14:02:10 +0000 (16:02 +0200)]
dpdk/mlx5: fix shutdown crash in IPS mode
Make sure to first close all ports before freeing device mempools.
Thread 1 "Suricata-Main" received signal SIGSEGV, Segmentation fault.
0x00007ffff456a3fb in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so
(gdb) bt
#0 0x00007ffff456a3fb in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so
#1 0x00007ffff469a948 in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so
#2 0x00007ffff45606aa in ?? () from /usr/lib/x86_64-linux-gnu/dpdk/pmds-20.0/librte_pmd_mlx5.so
#3 0x00007ffff6d4ed8d in rte_eth_dev_close () from /usr/lib/x86_64-linux-gnu/librte_ethdev.so.20.0
#4 0x000000000055fc4c in DPDKCloseDevice (ldev=ldev@entry=0xe3a400) at util-dpdk.c:53
#5 0x000000000055f4eb in LiveDeviceListClean () at util-device.c:331
#6 0x00000000005511c8 in GlobalsDestroy (suri=<optimized out>) at suricata.c:381
#7 0x0000000000550a76 in SuricataMain (argc=<optimized out>, argv=<optimized out>) at suricata.c:3059
#8 0x00007ffff6a24083 in __libc_start_main (main=0x54cca0 <main>, argc=8, argv=0x7fffffffe4c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe4b8) at ../csu/libc-start.c:308
#9 0x000000000054cbde in _start ()
Jeff Lucovsky [Thu, 3 Aug 2023 14:06:47 +0000 (10:06 -0400)]
detect/analysis: Move globals to engine ctx
Issue: 6239
This commit moves the global variables associated with engine analysis
into the detect engine context. Doing so provides encapsulation of the
analysis components as well as thread-safe operation in a multi-tenant
(context) deployment.
Jason Ish [Wed, 2 Aug 2023 16:09:40 +0000 (10:09 -0600)]
doc/userguide: display version on front page
When viewing the docs online at Readthedocs, or similar it might be
immediately apparent what version of the documentation is being
displayed. Display the version on the first line before the table of
contents to make it clear.
projects / thirdparty / suricata.git / log
