pgsql: debug validation on duplicated request msgs
There shouldn't be duplicated messages in the requests Vec. And thus
the parser shouldn't log duplicated keys nor messages. Add debug
validations to ensure this.
With PGSQL's current state machine, most frontend/ client messages will
lead to the creation of a new transaction - which would prevent
duplicated messages being pushed to the requests array and reaching the
logger.
The current exceptions for that are:
- CopyDataIn
- CopyDone
- CopyFail
Thus, debug statements were added for those cases.
CopyDone and CopyFail, per the documentation, shouldn't be seen
duplicated on the wire for the same transaction. CopyDataIn -- yes, but
we consolidate those, so the expectation is that they won't be
duplicated in the requests array or when reaching the logger either.
While this could be considered minor, they were not just bad, but
misleading names, as the variables weren't really `dummy` responses,
but consolidating several messages.
pgsql: add initial support to CopyIn mode/subproto
This sub-protocol inspects messages sent mainly from the frontend to
the backend after a 'COPY FROM STDIN' has been processed by the
backend.
Parses new messages:
- CopyInResponse -- initiates copy-in mode/sub-protocol
- CopyData (In) -- data transfer message, from frontend to backend
- CopyDone -- signals that no more CopyData messages will be seen from
the frontend, for the current transaction
- CopyFail -- used by the frontend to signal some failure to proceed
with sending CopyData messages
Jason Ish [Mon, 2 Jun 2025 21:47:03 +0000 (15:47 -0600)]
lua/streaming: provide streaming buffer as argument
When setting up a Lua output script for streaming data, we're
explicitly requesting stream data. Just pass the streaming data as
arguments, rather than requiring the script to make an extra call to
get the stream data.
The streaming data will be passed in the "stream" field of the args
passed to the log function.
Jason Ish [Mon, 2 Jun 2025 20:37:57 +0000 (14:37 -0600)]
lua/bytevar: convert SCByteVar to Lua lib
Similar to flowvars and flowints, but a byte var cannot be registered
from a Lua script, but it still needs to be setup. Instead provide an
"map" function that sets it up, or errors out if the byte var is
unknown.
This also required passing the signature into the Lua init method, as
the state of the Signature object and the time of loading the Lua
keyword is required.
Jason Ish [Mon, 2 Jun 2025 15:07:20 +0000 (09:07 -0600)]
lua: fix coverity unchecked return
CID 1648351: (#1 of 1): Unchecked return value (CHECKED_RETURN)
1. check_return: Calling lua_getstack without checking return value (as is done elsewhere 9 out of 10 times).
Add section label and doc reference, add another term to Common terms
section.
Tried to also improve readability for the Midstream behavior tables:
- Highlight key-words when differences are only in `do` vs `no`.
- Change order of sentences in certain descriptions, to align with the
steps those happen for the engine.
The terms 'inspection' and 'detection' were being used to signify
different engine actions in this document, while throughout the
documentation and code they're many times interchangeable.
Replace 'inspection' with 'parsing' or even 'decoding and parsing' as
more appropriate.
Add a small glossary to clarify what we mean with those terms.
Shivani Bhardwaj [Thu, 29 May 2025 09:04:57 +0000 (14:34 +0530)]
detect/analyzer: use note instead of warning
It is invalid to issue a warning based purely on pattern matching
between the matched content and certain http methods. Without a check
for which keyword exists in the rule, it makes more sense to issue this
as a general note.
util/flow-rate: fix sum calc on index next to base
When the buffer is wrapped around, for any new index, the calculation
must subtract the previous value stored in the buffer. So far, the code
ended up adding to the existing buffer value on the index unless it was
the first index after wrapping around. This is incorrect and would end
up flagging a flow as elephant a lot before than it should be.
Jason Ish [Wed, 28 May 2025 18:43:05 +0000 (12:43 -0600)]
mdns: add mdns parser, logger and detection
The mDNS support is based heavily on the DNS support, reusing the
existing DNS parser where possible. This meant adding variations on
DNS, as mDNS is a little different. Mainly being that *all* mDNS
traffic is to_server, yet there is still the concept of request and
responses.
They are mostly in-line with the DNS keywords, except
mdns.answers.rdata which is a better than that mdns.response.rrname,
as its actually looking at the rdata, and not rrnames.
mDNS has its own logger that differs from the DNS logger:
- No grouped logging
- In answers/additionals/authorities, the rdata is logged in a field
that is named after the rdata type. For example, "txt" data is no
longer logged in the "rdata" field, but instead a "txt" field. We
currently already did this in DNS for fields that were not a single
buffer, like SOA, SRV, etc. So this makes things more consistent. And
gives query like semantics that the "grouped" object was trying to
provide.
- Types are logged in lower case ("txt" instead of "TXT")
- Flags are logged as an array: "flags": ["aa", "z"]
Jason Ish [Tue, 27 May 2025 17:27:23 +0000 (11:27 -0600)]
dns: parse multiple txt segments into an array
A DNS TXT answer record can actually be made of up multiple TXT
entries in a single record. Suricata currently expands these into
multiple TXT records, however that is not very representative of the
actualy DNS message.
Instead, if a TXT record contains multiple labels, parse them into an
array.
We still expand multiple TXT segements into multiple TXT records at
logging time for compatibility, but this will allow something like
MDNS to log more accurately to the protocol.
Shivani Bhardwaj [Tue, 27 May 2025 13:17:43 +0000 (18:47 +0530)]
doc: move upgrade note to correct section
The PR for the behavior change of dealing with spaces in http URI and
protocol was started in 2019 and merged in 2024. When the PR was
created, it belonged to the correct upgrade section, however, by the
time it was merged, it was 8.0.x branch already.
Move it to upgrade notes from 7 to 8 for correctness.
Jason Ish [Fri, 23 May 2025 15:42:18 +0000 (09:42 -0600)]
clang-format: don't reformat schema on rewrite-branch
Add --extensions c,h flag to git clang-format when reformatting
commits on a branch. This prevents clang-format from modifying
non-C files like JSON schemas.
The extensions is used elsewhere in this script, so I suspect it was
just an oversight.
Also, ./scripts/schema-order.py --in-place will consistently format the
schema, while sorting the keys.
Philippe Antoine [Wed, 21 May 2025 13:34:26 +0000 (15:34 +0200)]
detect/files: increment local_file_id even if buffer is NULL
Ticket: 7579
Otherwise, we will keep on calling again and again GetDataCallback
with the same local_file_id, and we will always get a NULL
buffer even if the next local_file_id would return a non-NULL buffer.
Shivani Bhardwaj [Wed, 21 May 2025 07:18:48 +0000 (12:48 +0530)]
websocket: trigger raw stream inspection
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream inspection which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
Websocket parser creates a new PDU per transaction in each direction. Appropriate
calls to trigger raw stream inspection have been added on succesful parsing of
each PDU.
Shivani Bhardwaj [Wed, 21 May 2025 06:45:11 +0000 (12:15 +0530)]
ssh: trigger raw stream inspection
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream inspection which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
SSH parser creates a new record per request or response. Appropriate calls
to trigger raw stream inspection have been added on succesful parsing of
each request and response.
Shivani Bhardwaj [Mon, 19 May 2025 07:13:25 +0000 (12:43 +0530)]
sip: trigger raw stream inspection
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream inspection which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
SIP parser creates a new transaction per request or response. Appropriate calls
to trigger raw stream inspection have been added on creation of each request and
response.
Shivani Bhardwaj [Mon, 19 May 2025 06:42:18 +0000 (12:12 +0530)]
rfb: trigger raw stream inspection
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream inspection which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
RFB has several different types of requests and responses. Appropriate calls
to trigger raw stream inspection have been added on completion of each type of
request and response.
Shivani Bhardwaj [Mon, 19 May 2025 06:24:12 +0000 (11:54 +0530)]
rdp: trigger raw stream inspection
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream inspection which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
RDP parser creates a transaction per request or response. Appropriate calls
to trigger raw stream inspection have been added on completion of each request
and response.
Shivani Bhardwaj [Mon, 19 May 2025 05:33:49 +0000 (11:03 +0530)]
pop3: trigger raw stream inspection
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream inspection which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
POP3 has a classic request response model where a request is mapped to
a response although not by any ID. Appropriate calls to trigger raw stream
inspection have been added on completion of each request and response.
Shivani Bhardwaj [Sat, 17 May 2025 08:11:32 +0000 (13:41 +0530)]
nfs: trigger raw stream inspection
Internals
---------
Suricata's stream engine returns data for inspection to the detection
engine from the stream when the chunk size is reached.
Bug
---
Inspection triggered only in the specified chunk sizes may be too late
when it comes to inspection of smaller protocol specific data which
could result in delayed inspection, incorrect data logged with a transaction
and logs misindicating the pkt that triggered an alert.
Fix
---
Fix this by making an explicit call from all respective applayer parsers to
trigger raw stream inspection which shall make the data available for inspection
in the following call of the stream engine. This needs to happen per direction
on the completion of an entity like a request or a response.
Important notes
---------------
1. The above mentioned behavior with and without this patch is
affected internally by the following conditions.
- inspection depth
- stream depth
In these special cases, the inspection window will be affected and
Suricata may not consider all the data that could be expected to be
inspected.
2. This only applies to applayer protocols running over TCP.
3. The inspection window is only considered up to the ACK'd data.
4. This entire issue is about IDS mode only.
NFS has a classic request response model where each request is mapped to
a corresponding response. Additionally, there's also file tracking and
handling done as a part of these transactions. Appropriate calls to
trigger raw stream inspection have been added on completion of each
request and response.
Shivani Bhardwaj [Tue, 20 May 2025 13:53:24 +0000 (19:23 +0530)]
stream: rename TriggerRawStreamReassembly
The functions around TriggerRawStreamReassembly are misnomers in the
current layout of the code. The functions were named appropriately when
they were created as per the structural and logical layout of the code
at the time.
These functions in today's code are being used to track, update and
trigger progress of inspection in the raw stream only. Hence, rename them
to TriggerRawStreamInspection.
flow-manager: move time check after RUNNNING state change
When running in pcap-file mode and with a continous directory
reading, the provided directory can be empty.
By having no packets and being in offline mode, the initial packet timestamp
is never set. However, Flow Manager waited until the timestamp was set
to proceed with transferring its state to "RUNNING".
Other pcap-related threads (RX / workers) are set in RUNNING state while
waiting for the PCAP to appear in the directory.
As a result, the main Suricata-Main thread timed out after the default
60 seconds budget for threads to turn from INIT_DONE to RUNNING state.
While it is correct to send this Direction enum as argument to the
sawp_pop3 parser as it expects, it is not correct to use it where the
direction param is obtained from the internal API of Suricata.
The reason is that Suricata's definition of its Direction enum is as
follows.
This can lead to issues like incorrect progress tracking of a transaction in
a direction which could cause inspection on incorrect data and buggy behavior.
Jason Ish [Fri, 16 May 2025 18:58:20 +0000 (12:58 -0600)]
etc/schema: sort properties in alphabetic order
Also:
- Place "additionalProperties" before "properties"
- Place "required" after "additionalProperties"
- Remove "additionalProperties where true, as that is the default
The order should help us spot duplicate keys, and make it easier to
add new keys in their proper place.
Jeff Lucovsky [Thu, 17 Apr 2025 12:30:50 +0000 (08:30 -0400)]
detect/ftp: Add ftp.received_reply
Issue: 7506
Add a (non-sticky buffer) keyword for ftp.reply_received. This is not a
sticky buffer as the keyword relates to protocol state and not bytes
from the actual protocol exchange.
"server_handshake" which logs the following:
1. TLS version used during handshake
2. The chosen cipher suite, excluding GREASE
3. TLS extensions, excluding GREASE
"client_handshake" which logs the following:
1. TLS version used during handshake
2. TLS extensions, excluding GREASE, SNI and ALPN
3. All cipher suites, excluding GREASE
4. All signature algorithms, excluding GREASE
The use-case is for logging TLS handshake parameters in order to survey
them, and so that JA4 hashes can be computed offline (in the case that
they're not already computed for the purposes of rule matching).
This commit is designed in preparation of enabling the handshake object
to log it's own contents rather than being done on the C side.
Moving the tls versions functionality to rust has a couple of uses:
1. Allows both rust and C side to use these fields
2. Moves more of the tls related logic to rust
3. C side can still use these values because of cbindgen
projects / thirdparty / suricata.git / log
