Victor Julien [Wed, 22 Nov 2023 08:03:09 +0000 (09:03 +0100)]
flow: fix condition signalling
Signal threads while holding lock. This should make the signalling
more reliable.
From PTHREAD_COND(3):
"Unlocking the mutex and suspending on the condition variable is done
atomically. Thus, if all threads always acquire the mutex before
signaling the condition, this guarantees that the condition cannot be
signaled (and thus ignored) between the time a thread locks the
mutex and the time it waits on the condition variable."
Victor Julien [Wed, 13 Sep 2023 05:01:53 +0000 (07:01 +0200)]
packetpool: dynamic return threshold
Problem:
In pcap autofp mode, there is one threads reading packets (RX). These packets
are then passed on to worker threads. When these workers are done with a
packet, they return packets to the pcap reader threads packet pool, which is
the owner of the packets. Since this requires expensive synchronization between
threads, there is logic in place to batch this operation.
When the reader thread depletes its pool, it notifies the other threads that
it is starving and that a sync needs to happen asap. Then the reader enters
a wait state. During this time no new packets are read.
However, there is a problem with this approach. When the reader encountered
an empty pool, it would set an atomic flag that it needed a sync. The first
worker to return a packet to the pool would then set this flag, sync, and
unset the flag. This forced sync could result in just a single packet being
synchronized, or several. So if unlucky, the reader would just get a single
packet before hitting the same condition again.
Solution:
This patch updates the logic to use a new approach. Instead of using a
binary flag approach where the behavior only changes when the reader is
already starved, it uses a dynamic sync threshold that is controlled by
the reader. The reader keeps a running count of packets it its pool,
and calculates the percentage of available packets. This percentage is
then used to set the sync threshold.
When the pool is starved, it sets the threshold to 1 (sync for each packet).
After each successful get/sync the threshold is adjusted.
And failures should be handled to say that the rule failed to load
Reverts the fix by 299ee6ed5561f01575150b436d5db31485dab146
that was simple, but not complete (memory leak),
to have this bigger API change which simplifies code.
Philippe Antoine [Thu, 11 May 2023 08:02:32 +0000 (10:02 +0200)]
output/alert: rewrite code for app-layer properties
Especially fix setup-app-layer script to not forget this part
This allows, for simple loggers, to have a unique definition
of the actual logging function with the jsonbuilder.
This way, alerts, files, and app-layer event can share the code
to output the same data.
Joseph Reilly [Tue, 1 Aug 2023 12:42:48 +0000 (12:42 +0000)]
af-xdp: detach XDP program early
To mitigate a bug with AF_XDP sockets in high traffic scenarios, the XDP program must be detatched before
the sockets are closed. This issue happens when large ammounts of traffic are sent to suricata and
the XDP program is not removed before AF_XDP sockets are closed. I believe this is a race
condition bug as detailed here: https://bugzilla.kernel.org/show_bug.cgi?id=217712
Further investigation shows this may be a bug exclusive to the driver/AMD processor combination.
This commit addresses the bug by ensuring the first thread to run the deinit function
removes the XDP program, which fixes the bug as detailed in the bugzilla link.
Thomas Winter [Thu, 27 Apr 2023 04:08:46 +0000 (16:08 +1200)]
rule-reload: Release excess memory freed during engine reload
The hot reload results in large chunks of memory being freed as the
as the old signature tables are discarded. Help the memory management
system along by telling to release as much memory as it can at this
point.
Jason Ish [Mon, 30 Oct 2023 23:25:12 +0000 (17:25 -0600)]
examples: add an example plugin of an eve filetype
This is an example of what adding plugin examples to the Suricata repo
could look like.
This plugin is an example plugin for an EVE filetype. It could be
extended to support outputs like Redis, syslog, etc.
There is one issue with adding plugins like this to an autotools
project, the project can't be built with --disable-shared, which is
more of an autotools limitation, and not really a Suricata issue.
Suricata built with --disable-shared will load plugins just fine.
Note that the examples directory was added as DIST_SUBDIRS as we don't
want normal builds to recurse into it and attempt to build the plugin,
its just an example, but we still need to keep distcheck happy.
Until now the implementation would scan the stream, fallback to the
packet payload in exception cases, then keep track of where the match
was and in the flow match logic reject the match if it was in the wrong
buffer.
This patch simplifies this logic, by refusing to inspect the packet
payload when `only_stream` is set.
To do this the `only_stream`/`no_stream` options are now translated
to the pseudo protocols `tcp-stream` and `tcp-pkt` at parsing, so that
the `flow` keyword doesn't have to evaluate these conditions anymore.
Clarify the transform validation step. When a transform indicates that
the content/byte-array is not compatible, validation will stop.
Content is incompatible is some cases -- e.g., following the
to_lowercase transform with content containing uppercase characters.
An alert is not possible since the content contains uppercase and the
transform has converted the buffer into all lowercase.
Jason Ish [Wed, 1 Nov 2023 22:57:39 +0000 (16:57 -0600)]
outputs: call plugin ThreadDeinit, not Deinit
With the change to the hash table for tracking threaded loggers, this
call is now called once per thread, so should be changed to the
ThreadDeinit, as that is not longer being called.
Then call Deinit for the primary logger. In threaded mode this would be
the parent, its just the logger in non-threaded mode.
Victor Julien [Thu, 14 Sep 2023 04:49:31 +0000 (06:49 +0200)]
packetpool: signal waiter within lock
Needed for predictable scheduling. From pthread_cond_signal man page:
"The pthread_cond_signal() or pthread_cond_broadcast() functions may
be called by a thread whether or not it currently owns the mutex that
threads calling pthread_cond_wait() or pthread_cond_timedwait() have
associated with the condition variable during their waits; however, if
predictable scheduling behaviour is required, then that mutex is locked
by the thread calling pthread_cond_signal() or pthread_cond_broadcast()."
Jason Ish [Fri, 27 Oct 2023 16:19:31 +0000 (10:19 -0600)]
dns/eve: use default formats if formats is empty
If the configuration field "formats" is empty, DNS response records do
not have any relevant information other than that there was a
response, but not much about the response.
I'm pretty sure the intention here was to log the response details if
no formats were provided, which is what happens when the field is
commented out.
So if no formats are specified, use the default of all.
Make sure Suricata is in the running state before
you attempt to execute commands on the Unix sockets.
UnixMain is being called in an infinite loop where
TmThreadsCheckFlag(th_v, THV_KILL) is checked for the
deinit phase. However, it may take some time between
the start of Suricata's deinitialization and
the receipt of THV_KILL flag in the Unix thread.
In between this time period, the Unix manager can still
perform select() operation on the Unix socket while
the socket being already deinitialized.
Likely with a longer time span between the initial shutdown
command and actual closing of Unix sockets resulted in
an error of invalid file descriptors.
projects / thirdparty / suricata.git / log
