________________________________________________________________________________________________________
*** CID 1646448: Null pointer dereferences (NULL_RETURNS)
/src/detect-engine-prefilter.c: 944 in SetupNonPrefilter()
938 const DetectBufferType *buf = DetectEngineBufferTypeGetById(de_ctx, list_id);
939 /* for now, exclude app-layer-events, as they are not tied to a specific
940 * progress value like other keywords. */
941 SCLogDebug("list_id %d buf %p", list_id, buf);
942 if (list_id == app_events_list_id)
943 continue;
>>> CID 1646448: Null pointer dereferences (NULL_RETURNS)
>>> Dereferencing "buf", which is known to be "NULL".
944 if (buf->packet) {
945 SCLogDebug("packet buf");
946 /* packet is handled below */
947 pkt_non_pf = true;
948 } else if (buf->frame) {
949 for (DetectEngineFrameInspectionEngine *f = de_ctx->frame_inspect_engines;
Alice Akaki [Sun, 2 Mar 2025 23:41:26 +0000 (19:41 -0400)]
detect: add ldap.responses.attribute_type
ldap.responses.attribute_type matches on LDAP attribute type/description
This keyword maps the eve field ldap.responses[].search_result_entry.attributes[].type
It is a sticky buffer
Supports multiple buffer matching
Supports prefiltering
Alice Akaki [Thu, 13 Feb 2025 05:18:37 +0000 (01:18 -0400)]
detect: add ldap.request.attribute_type
ldap.request.attribute_type matches on LDAP attribute type/description
This keyword maps the following eve fields:
ldap.request.search_request.attributes[]
ldap.request.modify_request.changes[].modification.attribute_type
ldap.request.add_request.attributes[].name
ldap.request.compare_request.attribute_value_assertion.description
It is a sticky buffer
Supports multiple buffer matching
Supports prefiltering
Jeff Lucovsky [Fri, 21 Mar 2025 13:57:38 +0000 (09:57 -0400)]
app/ftp: Move FTP response handling to rust
Move handling of FTP responses to Rust to improve support for FTP
keyword matching. Parsing the response line when encountered
simplifies multi-buffer matching and metadata output.
`packet:filter`:
- default policy is `drop:packet`
- rules are process in order
- action scopes are explicit
- `drop` or `accept` is immediate
- `accept:hook` continues to `packet:td`
`packet:td`:
- default policy is `accept:hook`
- rules are ordered by IDS/IPS ordering logic
- action scopes are implicit
- actions are queued
- continues to `app:*:*` or `alert/action finalize`
`app:*:*`:
- default policy is `drop:flow`
- rules are process in order
- action scopes are explicit
- `drop` is immediate
- `accept` is conditional on possible `drop` from `packet:td`
- `accept:hook` continues to `app:td`, `accept:packet` or `accept:flow`
continues to `alert/action finalize`
`app:td`:
- default policy is `accept:hook`
- rules are ordered by IDS/IPS ordering logic
- action scopes are implicit
- actions are queued
- continues to `alert/action finalize`
Implementation:
During sigorder, split into packet:filter, app:*:* and general td.
Allow fw rules to work when in pass:flow mode. When firewall mode is enabled,
`pass:flow` will not skip the detection engine anymore, but instead
process the firewall rules and then apply the pass before inspecting threat
detect rules.
Victor Julien [Tue, 14 Jan 2025 08:41:11 +0000 (09:41 +0100)]
detect: introduce explicit hooks
Generic:
<app_proto>:request_started and <app_proto>:response_started
<app_proto>:request_complete and <app_proto>:response_complete
Per protocol, it uses the registered progress (state) values. E.g.
tls:client_hello_done
A rule ruleset could be:
pass tls:client_hello_done any any -> any any (tls.sni; content:"www.google.com"; sid:21; alert;)
drop tls:client_hello_done any any -> any any (sid:22;)
The pass rule is evaluated when the client hello is parsed, and if it
doesn't match the drop rule will be evaluated.
Registers each generic lists as "<alproto>:<progress state>:generic"
(e.g. "tls:client_hello_done:generic").
Victor Julien [Thu, 28 Dec 2023 21:11:56 +0000 (22:11 +0100)]
detect: move non-pf rules into special prefilter engines
Instead of having a per detection engine list of rule that couldn't be
prefiltered, put those into special "prefilter" engines.
For packet and frame rules this doesn't change much, it just removes
some hard coded logic from the detect engine.
For the packet non-prefilter rules in the "non-prefilter" special prefilter
engine, add additional filtering for the packet variant. It can prefilter on
alproto, dsize and dest port.
The frame non-prefilter rules are added to a single engine, that per
rule checks the alproto and the type.
For app-layer, there is an engine per progress value, per app-layer
protocol and per direction. This hooks app-layer non-prefilter rules
into the app inspect logic at the correct "progress" hook.
e.g. a rule like
dns.query; bsize:1;
Negated MPM rules will also fall into this category:
dns.query; content:!"abc";
Are part of a special "generic list" app engine for dns, at the
same progress hook as `dns.query`.
Dean Balandin [Tue, 27 Jun 2023 12:40:37 +0000 (12:40 +0000)]
stream: decouple stream.bypass dependency from tls bypass
Decouple app.protocols.tls.encryption-handling and stream.bypass.
There's no apparent reason why encrypted TLS bypass traffic should
depend on stream bypass, as these are unrelated features.
Lukas Sismis [Thu, 27 Mar 2025 16:10:12 +0000 (17:10 +0100)]
tls: deprecate "default" as a encryption-handling option
The default value for TLS encryption is generic and not precise.
The new keyword, "track-only" substitues the old keyword, being
more on point what action is actually taken when the TLS flow
reaches an encrypted state.
Jason Ish [Fri, 4 Apr 2025 16:52:51 +0000 (10:52 -0600)]
lua/detect: remove rule sid, rev and gid globals
For some rules the values are always 0, for others they get set, so not
consistent. Also not documented and not tested and should be replaced by
a "suricata.rule" lib.
For example, these globals would be set for this rule:
alert tcp any any -> any any (flow:to_server;
flowbits:isnotset,dataset_added; lua:dataset.lua;
flowbits:set,dataset_added;sid:1;rev:99;)
But not this rule:
alert dns any any -> any any (msg:"TEST DNS LUA dns.rrname"; \
content:"org"; lua:test-rrname.lua; sid:1; rev:1;)
Related to ticket
https://redmine.openinfosecfoundation.org/issues/7490 (but not a fix)
This does not affect the functions with similar names for getting rule
details in output context.
Philippe Antoine [Thu, 16 Jan 2025 07:40:46 +0000 (08:40 +0100)]
app-layer: make some arrays even more dynamic
Ticket: 5053
Do not asume that we know the number of alprotos at the end
of AppLayerNamesSetup, but make arrays allocated by later
AppLayerProtoDetectSetup dynamic so that it can be reallocated
from AppLayerParserRegisterProtocolParsers
This helps have a single entry point for a protocol like SNMP
This sub-protocol inspects messages exchanged between postgresql backend
and frontend after a 'COPY TO STDOUT' has been processed.
Parses new messages:
- CopyOutResponse -- initiates copy-out mode/sub-protocol
- CopyData -- data transfer messages
- CopyDone -- signals that no more CopyData messages will be seen from
the sender for the current transaction
warning: using `contains()` instead of `iter().any()` is more efficient
--> src/http2/http2.rs:267:20
|
267 | if block.value.iter().any(|&x| x == b'@') {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: try: `block.value.contains(&b'@')`
|
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#manual_contains
= note: `#[warn(clippy::manual_contains)]` on by default
Philippe Antoine [Mon, 31 Mar 2025 19:12:22 +0000 (21:12 +0200)]
rust: fix clippy missing_abi warning
warning: extern declarations without an explicit ABI are deprecated
--> src/core.rs:72:1
|
72 | extern {
| ^^^^^^ help: explicitly specify the "C" ABI: `extern "C"`
|
= note: `#[warn(missing_abi)]` on by default
Todd Mortimer [Mon, 26 Jun 2023 20:01:50 +0000 (20:01 +0000)]
http: Use libhtp-rs.
Ticket: #2696
There are a lot of changes here, which are described below.
In general these changes are renaming constants to conform to the
libhtp-rs versions (which are generated by cbindgen); making all htp
types opaque and changing struct->member references to
htp_struct_member() function calls; and a handful of changes to offload
functionality onto libhtp-rs from suricata, such as URI normalization
and transaction cleanup.
Functions introduced to handle opaque htp_tx_t:
- tx->parsed_uri => htp_tx_parsed_uri(tx)
- tx->parsed_uri->path => htp_uri_path(htp_tx_parsed_uri(tx)
- tx->parsed_uri->hostname => htp_uri_hostname(htp_tx_parsed_uri(tx))
- htp_tx_get_user_data() => htp_tx_user_data(tx)
- htp_tx_is_http_2_upgrade(tx) convenience function introduced to detect response status 101
and “Upgrade: h2c" header.
Functions introduced to handle opaque htp_tx_data_t:
- d->len => htp_tx_data_len()
- d->data => htp_tx_data_data()
- htp_tx_data_tx(data) function to get the htp_tx_t from the htp_tx_data_t
- htp_tx_data_is_empty(data) convenience function introduced to test if the data is empty.
Other changes:
Build libhtp-rs as a crate inside rust. Update autoconf to no longer
use libhtp as an external dependency. Remove HAVE_HTP feature defines
since they are no longer needed.
Make function arguments and return values const where possible
htp_tx_destroy(tx) will now free an incomplete transaction
htp_time_t replaced with standard struct timeval
Callbacks from libhtp now provide the htp_connp_t and the htp_tx_data_t
as separate arguments. This means the connection parser is no longer
fetched from the transaction inside callbacks.
SCHTPGenerateNormalizedUri() functionality moved inside libhtp-rs, which
now provides normalized URI values.
The normalized URI is available with accessor function: htp_tx_normalized_uri()
Configuration settings added to control the behaviour of the URI normalization:
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
- htp_config_set_convert_lowercase()
- htp_config_set_double_decode_normalized_query()
- htp_config_set_double_decode_normalized_path()
- htp_config_set_backslash_convert_slashes()
- htp_config_set_bestfit_replacement_byte()
- htp_config_set_convert_lowercase()
- htp_config_set_nul_encoded_terminates()
- htp_config_set_nul_raw_terminates()
- htp_config_set_path_separators_compress()
- htp_config_set_path_separators_decode()
- htp_config_set_u_encoding_decode()
- htp_config_set_url_encoding_invalid_handling()
- htp_config_set_utf8_convert_bestfit()
- htp_config_set_normalized_uri_include_all()
- htp_config_set_plusspace_decode()
Constants related to configuring uri normalization:
- HTP_URL_DECODE_PRESERVE_PERCENT => HTP_URL_ENCODING_HANDLING_PRESERVE_PERCENT
- HTP_URL_DECODE_REMOVE_PERCENT => HTP_URL_ENCODING_HANDLING_REMOVE_PERCENT
- HTP_URL_DECODE_PROCESS_INVALID => HTP_URL_ENCODING_HANDLING_PROCESS_INVALID
htp_config_set_field_limits(soft_limit, hard_limit) changed to
htp_config_set_field_limit(limit) because libhtp didn't implement soft
limits.
libhtp logging API updated to provide HTP_LOG_CODE constants along with
the message. This eliminates the need to perform string matching on
message text to map log messages to HTTP_DECODER_EVENT values, and the
HTP_LOG_CODE values can be used directly. In support of this,
HTP_DECODER_EVENT values are mapped to their corresponding HTP_LOG_CODE
values.
The new htp_log API supports consuming log messages more easily than
walking a list and tracking the current offset. Internally, libhtp-rs
now provides log messages as a queue of htp_log_t, which means the
application can simply call htp_conn_next_log() to fetch the next log
message until the queue is empty. Once the application is done with a
log message, they can call htp_log_free() to dispose of it.
Functions supporting htp_log_t:
htp_conn_next_log(conn) - Get the next log message
htp_log_message(log) - To get the text of the message
htp_log_code(log) - To get the HTP_LOG_CODE value
htp_log_free(log) - To free the htp_log_t
In order to track flow rate and thus determine a course of action or
categorize it as elephant flow, track a flow's byte count per direction
in a ring buffer for a given time interval.
The implementation is simple and keeps overwriting the buffer and
updating the final sum. The sum of all the elements in the ring buffer
at any point in time should reflect the number of bytes for the
respective flow in the last of a given configured interval.
e.g. if the definition says that the flows must be tracked by a rate of
100k bytes in 10 seconds, the ring buffer at any point in time should
carry the total number of bytes seen by the respective flow in the last
10 seconds.
So far, the implementation only supports reading the flow rate
definition from suricata.yaml and using it to track the flows.
This solution adds up a space complexity to the existing Flow struct.
However, the added space complexity should only take effect if the
feature is in use. Since this buffer extends the Flow struct, it does
not impact the usual business logic or complexity of the code.
This implementation is currently limited to defining the time interval
of flow rate in seconds only. However, the number of seconds defined are
directly proportional to the aforementioned added space complexity as
that's the size of the ring buffer.
Victor Julien [Wed, 2 Apr 2025 07:09:48 +0000 (09:09 +0200)]
detect/flowbits: implement prefilter support
Allow for more efficient rules that 'prefilter' on flowbits with 'isset' logic.
This prefilter is enabled by default, which means that if no mpm is present or
no explicit prefilter is used, the flowbits prefilter will be set up for a rule.
flowbits 'isset' prefilter
For rules that have a 'flowbits:isset,<bit>' statement, a "regular" prefilter
facility is created. It means that the rules are removed from the normal
match list(s) and added to a prefilter engine that runs prior to the individual
rule inspection stage.
Implementation: the prefilter is implemented as an RB_TREE of flowbits, with the
rule id's they "enable" stored per tree node. The matching logic is walking the
list of bits set in the flow and looking each of them up in the RB_TREE, adding
the rule ids of each of the matching bits to the list of rule candidates.
The 'isset' prefilter has one important corner case, which is that bits can in
fact be set during the rule evaluation stage. This is different from all other
prefilter engines, that evaluate an immutable state (for the lifetime of the
packet inspection).
flowbits 'set' post-match prefilter
For flowbits 'set' action, special post-match 'prefilter' facilities deal with
this corner case. The high level logic is that these track which 'isset' sigs
depend on them, and add these dependencies to the candidates list when a 'set'
action occurs.
This is implemented in a few steps:
1. flowbits 'set' is flagged
2. when 'set' action occurs the flowbit is added to a "post rule
match work queue"
3. when the rule evaluation ends, the post-match "prefilter" engine is run
on each of the flowbits in the "post rule match work queue"
4. these engines ammend the candidates list with the rule id dependencies
for the flowbit
5. the candidates list is sorted to make sure within the execution for that
packet the inspection order is maintained
Add support for special post-match engines. This allows a rule to enable
other rules when it matches.
Implementation is similar to prefilter engines, however prefilter
engines run before individual rules while this post-match engine runs
after and individual rule match. It will then add the new rules to the
existing rule list.
projects / thirdparty / suricata.git / log
