Thomas Winter [Thu, 27 Apr 2023 04:08:46 +0000 (16:08 +1200)]
rule-reload: Release excess memory freed during engine reload
The hot reload results in large chunks of memory being freed as the
as the old signature tables are discarded. Help the memory management
system along by telling to release as much memory as it can at this
point.
Jason Ish [Mon, 30 Oct 2023 23:25:12 +0000 (17:25 -0600)]
examples: add an example plugin of an eve filetype
This is an example of what adding plugin examples to the Suricata repo
could look like.
This plugin is an example plugin for an EVE filetype. It could be
extended to support outputs like Redis, syslog, etc.
There is one issue with adding plugins like this to an autotools
project, the project can't be built with --disable-shared, which is
more of an autotools limitation, and not really a Suricata issue.
Suricata built with --disable-shared will load plugins just fine.
Note that the examples directory was added as DIST_SUBDIRS as we don't
want normal builds to recurse into it and attempt to build the plugin,
its just an example, but we still need to keep distcheck happy.
Until now the implementation would scan the stream, fallback to the
packet payload in exception cases, then keep track of where the match
was and in the flow match logic reject the match if it was in the wrong
buffer.
This patch simplifies this logic, by refusing to inspect the packet
payload when `only_stream` is set.
To do this the `only_stream`/`no_stream` options are now translated
to the pseudo protocols `tcp-stream` and `tcp-pkt` at parsing, so that
the `flow` keyword doesn't have to evaluate these conditions anymore.
Clarify the transform validation step. When a transform indicates that
the content/byte-array is not compatible, validation will stop.
Content is incompatible is some cases -- e.g., following the
to_lowercase transform with content containing uppercase characters.
An alert is not possible since the content contains uppercase and the
transform has converted the buffer into all lowercase.
Jason Ish [Wed, 1 Nov 2023 22:57:39 +0000 (16:57 -0600)]
outputs: call plugin ThreadDeinit, not Deinit
With the change to the hash table for tracking threaded loggers, this
call is now called once per thread, so should be changed to the
ThreadDeinit, as that is not longer being called.
Then call Deinit for the primary logger. In threaded mode this would be
the parent, its just the logger in non-threaded mode.
Victor Julien [Thu, 14 Sep 2023 04:49:31 +0000 (06:49 +0200)]
packetpool: signal waiter within lock
Needed for predictable scheduling. From pthread_cond_signal man page:
"The pthread_cond_signal() or pthread_cond_broadcast() functions may
be called by a thread whether or not it currently owns the mutex that
threads calling pthread_cond_wait() or pthread_cond_timedwait() have
associated with the condition variable during their waits; however, if
predictable scheduling behaviour is required, then that mutex is locked
by the thread calling pthread_cond_signal() or pthread_cond_broadcast()."
Jason Ish [Fri, 27 Oct 2023 16:19:31 +0000 (10:19 -0600)]
dns/eve: use default formats if formats is empty
If the configuration field "formats" is empty, DNS response records do
not have any relevant information other than that there was a
response, but not much about the response.
I'm pretty sure the intention here was to log the response details if
no formats were provided, which is what happens when the field is
commented out.
So if no formats are specified, use the default of all.
Make sure Suricata is in the running state before
you attempt to execute commands on the Unix sockets.
UnixMain is being called in an infinite loop where
TmThreadsCheckFlag(th_v, THV_KILL) is checked for the
deinit phase. However, it may take some time between
the start of Suricata's deinitialization and
the receipt of THV_KILL flag in the Unix thread.
In between this time period, the Unix manager can still
perform select() operation on the Unix socket while
the socket being already deinitialized.
Likely with a longer time span between the initial shutdown
command and actual closing of Unix sockets resulted in
an error of invalid file descriptors.
Victor Julien [Fri, 13 Oct 2023 11:47:05 +0000 (13:47 +0200)]
detect: inspect all packets in multi-layer tunneling
When the decoders encounter multiple layers of tunneling, multiple tunnel
packets are created. These are then stored in ThreadVars::decode_pq, where
they are processed after the current thread "slot" is done. However, due
to a logic error, the tunnel packets after the first, where not called
for the correct position in the packet pipeline. This would lead to these
packets not going through the FlowWorker module, so skipping everything
from flow tracking, detection and logging.
This would only happen for single and workers, due to how the pipelines
are constructed.
The "slot" holding the decoder, would contain 2 packets in
ThreadVars::decode_pq. Then it would call the pipeline on the first
packet with the next slot of the pipeline through a indirect call to
TmThreadsSlotVarRun(), so it would be called for the FlowWorker.
However when that first (the most inner) packet was done, the call
to TmThreadsSlotVarRun() would again service the ThreadVars::decode_pq
and process it, again moving the slot pointer forward, so past the
FlowWorker.
This patch addresses the issue by making sure only a "decode" thread
slot will service the ThreadVars::decode_pq, thus never moving the
slot past the FlowWorker.
Lukas Sismis [Wed, 16 Aug 2023 21:51:10 +0000 (23:51 +0200)]
dpdk: add hugepage hint to lower the amount of reserved hugepages
If a user allocates too many hugepages and those are largely not used
then Suricata suggests that the user can lower the amount of hugepages
and therefore save memory for other purposes.
Our documentation was failing to build, seems connected to the new way
of indicating build options (cf
https://readthedocs.org/projects/suricata/builds/22112658/,
https://docs.readthedocs.io/en/stable/config-file/v2.html#build,
and https://docs.readthedocs.io/en/stable/config-file/v2.html#build-os).
Added the build.os required new field, and adjusted the way python
version is passed.
For the new configuration style for read the docs, one of the ways to
pass extra configuration for python is having a requirements file.
email_ctx->fields only gets populated when smtp.custom setting is on.
The fn EveEmailLogJSONCustom is called when either
1. smtp.extended setting is on or,
2. email_ctx->fields is populated which means smtp.custom setting is on
In case neither of these are set in suricata.yaml, no call should
ideally be made to the fn EveEmailLogJSONCustom.
However, it turns out that email_ctx->fields is unset and then set only
after the smtp config was found. This leads to email_ctx->fields
sometimes contain value even when no config was given to the smtp
section and can lead to unexpected output.
Fix this by using SCCalloc while initializing OutputJsonEmailCtx struct
instead of SCMalloc.
Although we have an updated version of instructions for installation
from git, our install guide was only referring to RedMine, which is less
up-to-date.
Kept that reference, since it might still be useful for non-Ubuntu
cases.
Philippe Antoine [Wed, 27 Sep 2023 12:15:18 +0000 (14:15 +0200)]
detect: check if signature uses too many buffers
Ticket: #6104
The approach in master branch is to change the prototype of
SigMatchAppendSMToList so that it allocates itself the new SigMatch
This approach requires to change all the 100-ish calls to
SigMatchAppendSMToList and is thus quite a big change.
For branch 7, we still wanted to avoid the buffer overflow, but
did not want such an intrusive change, and still wanted to make
the signature invalid. Instead of changing the prototype of the
function, we make it return early, and set a flag in the signature
which can be later checked by SigValidate
projects / thirdparty / suricata.git / log
