Victor Julien [Fri, 16 Jun 2023 13:07:13 +0000 (15:07 +0200)]
detect/filemagic: switch to file.magic implementation
Replace implementation of the legacy `filemagic` keyword by the
implementation for the `file.magic` variant. This leads to better
performance and hooks the rules into the detection engine better.
Jason Ish [Mon, 3 Jul 2023 18:04:16 +0000 (12:04 -0600)]
github-ci: replace dist builder with Debian 12
Add new dist builder job based on Debian 12. Debian 12 gives us news
Sphinx that AlmaLinux 8, plus avoids any potential disruption in the
RHEL rebuild ecosystem.
Also make dist building its own job so it finishes quicker, allowing
other jobs to proceed. The new non-dist building Debian 12 job will
still do a complete distcheck, as do other jobs.
If an exception policy wasn't set up individually, use the GetDefault
function to pick one. This will check for the master switch option and
handle 'auto' cases.
Instead of deciding what the auto value should be when we are parsing
the master switch, leave that for when some of the other policies is to
be set via the master switch, when since this can change for specific
exception policies - like for midstream, for instance.
Update exceptions policies documentation to clarify that the default
configuration in IPS when midstream is enabled is `ignore`, not
`drop-flow`.
Philippe Antoine [Fri, 16 Jun 2023 13:28:38 +0000 (15:28 +0200)]
mime: compute full body md5
Previously, the problem was that nested headers/boundaries were not
used to compute the hash
Solution is to move up the call to the hash computation from
ProcessMimeBody to its caller ProcessMimeEntity, and add a set of
conditions to ensure that we are not in the principal headers.
Jason Ish [Tue, 27 Jun 2023 16:52:39 +0000 (10:52 -0600)]
log-pcap: one time errors on file open
If compression was not enabled, the open error was actually suppressed
by default by only being logged at info level, however with
compression it was logged as an error. As opening is retried as long
as it fails to open, make both log as error but wrap in a flag so the
error is logged once until success.
Jason Ish [Tue, 27 Jun 2023 16:25:24 +0000 (10:25 -0600)]
log-pcap: only open dumper after successful file open (lz4)
When LZ4 compression is enabled, open the dumper after successful open
of the file. The dump handle is what forms the check if opening the
file needs to be retried.
We only try to parse a small subset of what is possible in
RFB. Currently we only understand some standard auth schemes
and stop parsing when the server-client handshake is complete.
Since in IPS mode returning an error from the parser causes
drops that are likely uncalled for, we do not want to return
errors when we simply do not understand what happens in the
traffic. This addresses Redmine #5912.
Jason Ish [Mon, 26 Jun 2023 16:57:00 +0000 (10:57 -0600)]
eve/alert: warn on obsolete flags
Log a warning but otherwise ignore obsolete flags under
eve-log.alert. This also prevents accidentially turning off app-layer
logging by attempting to disable a single protocol.
These flags have been deprecated since 5.0, time to stop respecting
them.
liaozhiyuan [Thu, 1 Jun 2023 02:38:04 +0000 (10:38 +0800)]
dpdk: support multiple same EAL arguments
DPDK apps can specify multiple arguments of the same
type. YAML format only allows unique keys within a single
node. This commit adds support for multiple EAL arguments
of the same type to be used within suricata.yaml.
Protocols supporting files are only defined in one place, which
gets used by all keywords, which can handle some exceptions
(like HTTP2 not having file names)
If the master exception policy was set to 'auto' in IDS mode, instead of
just setting the master switch to the default in this case, which is
'ignore', the engine would switch a warning saying that auto wasn't a
valid config and then set the policy to ignore.
This makes 'auto' work for the master switch in IDS, removes function
for setting IPS option and handles the valid IDS options directly from
the function that parses the master policy, as this was the only place
where the function was still called.
Jason Ish [Tue, 6 Jun 2023 22:04:56 +0000 (16:04 -0600)]
datasets: flag to disable "write" actions
Add a new configuration flag, "datasets.rules.allow-write" to control
if rules can contain "save" or "state" rules which allow write access
to the file system.
Jason Ish [Tue, 6 Jun 2023 21:32:18 +0000 (15:32 -0600)]
install: create runtime data directory
On installation, make sure the data directory is created. This will
usually be /var/lib/suricata/data, but otherwise follows the
autoconf/automake instructions.
This directory is for runtime state information, which for now is
datasets but may be expanded in the future. Suricata already expects
this directory to exist for "state" and "save" datasets, but it has
been up to the user to create it.
Jason Ish [Tue, 23 May 2023 21:17:59 +0000 (15:17 -0600)]
datasets: don't allow absolute or paths with directory traversal
For dataset filenames coming from rules, do not allow filenames that
are absolute or contain a directory traversal with "..". This prevents
datasets from escaping the define data-directory which may allow a bad
rule to overwrite any file that Suricata has permission to write to.
Add a new configuration option,
"datasets.rules.allow-absolute-filenames" to allow absolute filenames
in dataset rules. This will be a way to revert back to the pre 6.0.13
behavior where save/state rules could use any filename.
Shivani Bhardwaj [Tue, 23 May 2023 04:41:38 +0000 (10:11 +0530)]
smtp: handle long lines per direction
Issue:
Currently, while handling of long lines, if the line exceeded the limit,
we'd set a variable state->discard_till_lf which will be reset in the
later stages based on the data that arrives. However, because there was
one variable per state, this meant that a later stage in the other
direction could also modify it which is incorrect.
smtp: handle following cmd if LF was found in long line
If a long line had LF post the limit, it should be considered complete
and not wait for the next line to complete it. However, currently, any
following lines were skipped which could sometimes also be important
commands for the entire transaction.
Fix this by setting a flag in case we're truncating a long line but
after having found the LF character.
Before:
If the input was such that we'd enter DATA mode in the middle, the
entire data would be passed through SMTPGetLine fn and be processed with
line limits etc in place.
After:
Since we don't want any limits to be enforced on DATA, we pass it to
SMTPPreProcessCommands fn to take care of it differently from the
commands.
We were always setting it to ignore, due to bug 5825.
The engine will now issue an initialization error if an invalid value
is passed in the configuration file for midstream exception policy.
'pass-packet' or 'drop-packet' are never valid, as the midstream policy
concerns the whole flow, not making sense for just a packet.
If midstream is enabled, only two actual config values are allowed:
'ignore' and 'pass-flow', both in IDS and in IPS mode. In default mode
('auto' or if no policy is defined), midstream-policy is set to
'ignore'. All other values will lead to initialization error.
In IDS mode, 'drop-flow' will also lead to initialization error.
Use a mix of SCLogConfig, Warning and Info.
This mix works as follows: when something unnexpected for the user
happens - for instance, the engine ignoring an invalid config value, we
use warning. For indicating the value for the master switch, which
happens only once, we use Info. For all the other cases, we use
SCLogConfig.
It is possible that SCLogConfig isn't showing at the moment, this is a
possible bug to investigate further.
exceptions: parse config values, don't post process
Get the enum values from the config file. Update the new extracted
functions. Post-process the config values based on runmode and policy.
Also handle 'auto' enum value in these.
As the midstream exception policy has its own specific scenarios, have a
dedicated function to parse and process its config values, and check for
midstream enabled when needed.
Some exception policies can only be applied to the triggering packet or
only make sense considering the whole flow. Highlight such cases in the
table showing each exception policy.
The different interactions between midstream pick-up sessions and the
exception policy can be quite difficult to visualize. Add a section for
that in the userguide.
7a044a99ee14101fbc removed the lines that incremented these defrag
counters, but kept the entities themselves. This commit removes counters
that we judge too complex to maintain, given the current state of the
code, and re-adds incrementing max_hit (memcap related).
Jason Ish [Mon, 12 Jun 2023 16:29:00 +0000 (10:29 -0600)]
rust: fix unit test link error on Rust 1.70
Rust 1.70 appears to now link code on both branches of `if cfg!(test)`
now causing Rust unit tests to fail as that pattern was used to
disable functions only available when linked with the Suricata C code.
To work-around this issue, provide two versions of the `new` function,
one for unit tests and one when running as an application.
Victor Julien [Thu, 8 Jun 2023 16:37:44 +0000 (18:37 +0200)]
app-layer: improve/fix updates logic
In 23323a961fac ("app-layer: reduce app cleanup and output-tx calls"), flag
was set per packet updating the app-layer state. However this was missing a
common pattern: in IDS mode most updates are done in the opposite direction
of the traffic due to updates getting triggered by ACK's. This meant that
file store processing might not happen for a long time, or at all. Also,
app layer cleanup might not be called, which includes file pruning.
This patch sets per flow set of flags to indicate app layer is (potentially)
updated. It sets this per direction, based on how the parsers were invoked.
If an ACK triggers an app update, the flow is tagged for the opposite
direction and the next packet in that direction triggers output and cleanup.
projects / thirdparty / suricata.git / log
