Jeff Lucovsky [Thu, 3 Aug 2023 14:06:47 +0000 (10:06 -0400)]
detect/analysis: Move globals to engine ctx
Issue: 6239
This commit moves the global variables associated with engine analysis
into the detect engine context. Doing so provides encapsulation of the
analysis components as well as thread-safe operation in a multi-tenant
(context) deployment.
Jason Ish [Fri, 27 Oct 2023 16:19:31 +0000 (10:19 -0600)]
dns/eve: use default formats if formats is empty
If the configuration field "formats" is empty, DNS response records do
not have any relevant information other than that there was a
response, but not much about the response.
I'm pretty sure the intention here was to log the response details if
no formats were provided, which is what happens when the field is
commented out.
So if no formats are specified, use the default of all.
Our documentation was failing to build, seems connected to the new way
of indicating build options (cf
https://readthedocs.org/projects/suricata/builds/22112658/,
https://docs.readthedocs.io/en/stable/config-file/v2.html#build,
and https://docs.readthedocs.io/en/stable/config-file/v2.html#build-os).
Added the build.os required new field, and adjusted the way python
version is passed.
For the new configuration style for read the docs, one of the ways to
pass extra configuration for python is having a requirements file.
email_ctx->fields only gets populated when smtp.custom setting is on.
The fn EveEmailLogJSONCustom is called when either
1. smtp.extended setting is on or,
2. email_ctx->fields is populated which means smtp.custom setting is on
In case neither of these are set in suricata.yaml, no call should
ideally be made to the fn EveEmailLogJSONCustom.
However, it turns out that email_ctx->fields is unset and then set only
after the smtp config was found. This leads to email_ctx->fields
sometimes contain value even when no config was given to the smtp
section and can lead to unexpected output.
Fix this by using SCCalloc while initializing OutputJsonEmailCtx struct
instead of SCMalloc.
Victor Julien [Fri, 13 Oct 2023 11:47:05 +0000 (13:47 +0200)]
detect: inspect all packets in multi-layer tunneling
When the decoders encounter multiple layers of tunneling, multiple tunnel
packets are created. These are then stored in ThreadVars::decode_pq, where
they are processed after the current thread "slot" is done. However, due
to a logic error, the tunnel packets after the first, where not called
for the correct position in the packet pipeline. This would lead to these
packets not going through the FlowWorker module, so skipping everything
from flow tracking, detection and logging.
This would only happen for single and workers, due to how the pipelines
are constructed.
The "slot" holding the decoder, would contain 2 packets in
ThreadVars::decode_pq. Then it would call the pipeline on the first
packet with the next slot of the pipeline through a indirect call to
TmThreadsSlotVarRun(), so it would be called for the FlowWorker.
However when that first (the most inner) packet was done, the call
to TmThreadsSlotVarRun() would again service the ThreadVars::decode_pq
and process it, again moving the slot pointer forward, so past the
FlowWorker.
This patch addresses the issue by making sure only a "decode" thread
slot will service the ThreadVars::decode_pq, thus never moving the
slot past the FlowWorker.
Victor Julien [Mon, 15 May 2023 08:02:26 +0000 (10:02 +0200)]
flowworker: simplify pseudo packet use
Pseudo packets originating in the flow worker do not need to leave the
flow worker. Putting those in the ThreadVars::decode_pq will make them
be evaluated by the next steps in the pipeline, but those will all
ignore pseudo packets.
Instead, this patch returns them to the packet pool, while still honoring
the IPS verdict logic.
Philippe Antoine [Wed, 30 Aug 2023 19:35:08 +0000 (21:35 +0200)]
smtp: fix null deref with config option body md5
Ticket: #6279
If we have the smtp body beginning without headers, we need to
create the md5 context and right away and supply data to it.
Otherwise, on the next line being processed, md5_ctx will be
NULL but body_begin will have been reset to 0
Victor Julien [Wed, 2 Aug 2023 06:37:45 +0000 (08:37 +0200)]
var-names: reimplement var name handling
Implement a new design for handling var name id's. The old logic
was aware of detection engine versions and generally didn't work
well for multi-tenancy cases. Other than memory leaks and crashes,
logging of var names worked or failed based on which tenant was
loaded last.
This patch implements a new approach, where there is a global store
of vars and their id's for the lifetime of the program.
Overall Design:
Base Store: "base"
Used during keyword registration. Operates under lock. Base is shared
between all detect engines, detect engine versions and tenants.
Each variable name is ref counted.
During the freeing of a detect engine / tenant, unregistration decreases
the ref cnt.
Base has both a string to id and a id to string hash table. String to
id is used during parsing/registration. id to string during unregistration.
Active Store Pointer (atomic)
The "active" store atomic pointer points to the active lookup store. The call
to `VarNameStoreActivate` will build a new lookup store and hot swap
the pointer.
Ensuring memory safety. During the hot swap, the pointer is replaced, so
any new call to the lookup functions will automatically use the new store.
This leaves the case of any lookup happening concurrently with the pointer
swap. For this case we add the old store to a free list. It gets a timestamp
before which it cannot be freed.
Free List
The free list contains old stores that are waiting to get removed. They
contain a timestamp that is checked before they are freed.
Arne Welzel [Sun, 20 Aug 2023 15:32:47 +0000 (17:32 +0200)]
community-id: Fix IPv6 address sorting not respecting byte order
When comparing IPv6 addresses based on uint32_t chunks, one needs to
apply ntohl() conversion to the individual parts, otherwise on little
endian systems individual bytes are compared in the wrong order.
Avoid this all and leverage memcmp(), it'll short circuit on the first
differing byte and its return values tells us which address sorts lower.
Victor Julien [Sat, 5 Aug 2023 09:46:20 +0000 (11:46 +0200)]
detect/file: correct registration for HTTP
Register file.name and file.magic at correct progress values.
In HTTP1, the files are (part of) the body, so make sure the file
detection logic only runs when the parser has started processing
the body.
Victor Julien [Fri, 16 Jun 2023 13:07:13 +0000 (15:07 +0200)]
detect/filemagic: switch to file.magic implementation
Replace implementation of the legacy `filemagic` keyword by the
implementation for the `file.magic` variant. This leads to better
performance and hooks the rules into the detection engine better.
Victor Julien [Tue, 1 Aug 2023 06:44:53 +0000 (08:44 +0200)]
stream: special handling for RST data
Data on RST packets is not invalid, but also shouldn't be used
in reassembly.
RFC 1122:
4.2.2.12 RST Segment: RFC-793 Section 3.4
A TCP SHOULD allow a received RST segment to include data.
DISCUSSION
It has been suggested that a RST segment could contain
ASCII text that encoded and explained the cause of the
RST. No standard has yet been established for such
data.
RST data will be presented to the detection engine per packet,
but will not be part of stream reassembly.
userguide/eve: format and reorganize alert section
The `field action` portion seemed to be comprised of a more generic
section that followed it. Also formatted the section for lines to be
within the character limit.
So far, if only the starting request was a DCERPC request, it would be
considered DCERPC traffic. Since ALTER_CONTEXT is a valid request type,
it should be accepted too.
Reported and patch proposed in the following Redmine ticket by
InterNALXz.
If an exception policy wasn't set up individually, use the GetDefault
function to pick one. This will check for the master switch option and
handle 'auto' cases.
Instead of deciding what the auto value should be when we are parsing
the master switch, leave that for when some of the other policies is to
be set via the master switch, when since this can change for specific
exception policies - like for midstream, for instance.
Update exceptions policies documentation to clarify that the default
configuration in IPS when midstream is enabled is `ignore`, not
`drop-flow`.
If the master exception policy was set to 'auto' in IDS mode, instead of
just setting the master switch to the default in this case, which is
'ignore', the engine would switch a warning saying that auto wasn't a
valid config and then set the policy to ignore.
This makes 'auto' work for the master switch in IDS, removes function
for setting IPS option and handles the valid IDS options directly from
the function that parses the master policy, as this was the only place
where the function was still called.
We were always setting it to ignore, due to bug 5825.
The engine will now issue an initialization error if an invalid value
is passed in the configuration file for midstream exception policy.
'pass-packet' or 'drop-packet' are never valid, as the midstream policy
concerns the whole flow, not making sense for just a packet.
If midstream is enabled, only two actual config values are allowed:
'ignore' and 'pass-flow', both in IDS and in IPS mode. In default mode
('auto' or if no policy is defined), midstream-policy is set to
'ignore'. All other values will lead to initialization error.
In IDS mode, 'drop-flow' will also lead to initialization error.
Use a mix of SCLogConfig, Warning and Info.
This mix works as follows: when something unnexpected for the user
happens - for instance, the engine ignoring an invalid config value, we
use warning. For indicating the value for the master switch, which
happens only once, we use Info. For all the other cases, we use
SCLogConfig.
It is possible that SCLogConfig isn't showing at the moment, this is a
possible bug to investigate further.
exception: parse config values, don't post process
Get the enum values from the config file. Update the new extracted
functions. Post-process the config values based on runmode and policy.
Also handle 'auto' enum value in these.
As the midstream exception policy has its own specific scenarios, have a
dedicated function to parse and process its config values, and check for
midstream enabled when needed.
Some exception policies can only be applied to the triggering packet or
only make sense considering the whole flow. Highlight such cases in the
table showing each exception policy.
The different interactions between midstream pick-up sessions and the
exception policy can be quite difficult to visualize. Add a section for
that in the userguide.
7a044a99ee14101fbc removed the lines that incremented these defrag
counters, but kept the entities themselves. This commit removes counters
that we judge too complex to maintain, given the current state of the
code, and re-adds incrementing max_hit (memcap related).
In case of 'EXCEPTION_POLICY_REJECT', we were applying the same behavior
regardless of being in IDS or IPS mode.
This meant that (at least) the 'flow.action' was changed to drop when we
hit an exception policy in IDS mode.
This allows all traffic Exception Policies to be set from one
configuration point. All exception policy options are available in IPS
mode. Bypass, pass and auto (disabled) are also available in iDS mode
Exception Policies set up individually will overwrite this setup for the
given traffic exception.
For certain edge case handling for spaces, spaces were handled
particularly in the remainder processing functions. Make sure that now
that as per RFC 2045, util-base64 would skip over any invalid char, the
edge cases in MIME processor also be handled the same way.
Shivani Bhardwaj [Thu, 30 Mar 2023 07:43:08 +0000 (13:13 +0530)]
util/base64: check dest buf size to hold 3Bytes
The destination buffer should be able to hold at least 3 Bytes during
the processing of the last block of data. If it cannot hold at least 3
Bytes, then that may lead to dynamic buffer overflow while decoding.
Shivani Bhardwaj [Thu, 30 Mar 2023 07:41:12 +0000 (13:11 +0530)]
util/base64: check for dest buf size in last block
Just like the check for destination buffer size done previously for
complete data, it should also be done for the trailing data to avoid
goind out of bounds.
Shivani Bhardwaj [Thu, 30 Mar 2023 07:24:29 +0000 (12:54 +0530)]
util/base64: fix padding bytes for trailing data
Padding bytes for the last remainder data should be as follows:
Case | Remainder bytes | Padding
----------------------------------------------
I | 1 | 3
II | 2 | 2
III | 3 | 1
However, we calculate the decoded_bytes with the formula:
decoded_bytes = ASCII_BLOCK - padding
this means for Case I when padding is 3 bytes, the decoded_bytes would
be 0. This is incorrect for any trailing data. In any of the above
cases, if the parsing was successful, there should at least be 1 decoded
byte.
Shivani Bhardwaj [Fri, 17 Mar 2023 12:18:35 +0000 (17:48 +0530)]
util/base64: skip any invalid char for RFC2045
RFC 2045 states that any invalid character should be skipped over, this
is the RFC used by mime handler in Suricata code to deal with base64
encoded data.
So far, only spaces were skipped as a part of implementation of this
RFC, extend it to also skip over any other invalid character. Add
corresponding test.
We only try to parse a small subset of what is possible in
RFB. Currently we only understand some standard auth schemes
and stop parsing when the server-client handshake is complete.
Since in IPS mode returning an error from the parser causes
drops that are likely uncalled for, we do not want to return
errors when we simply do not understand what happens in the
traffic. This addresses Redmine #5912.
projects / thirdparty / suricata.git / log
