Jason Ish [Tue, 25 Aug 2020 19:52:00 +0000 (13:52 -0600)]
plugins: require registration function SCPluginRegister
Instead of looking for a symbol, "PluginSpec" look for a function
named SCPluginRegister that returns a SCPlugin.
This makes it much easier to create Rust plugins without having
to deal with dlopen constructors and such, which is rather
straight forward in C, but a bit of advanced boilerplate in Rust
that can be eliminated by simply calling a registration function.
Jason Ish [Tue, 25 Aug 2020 18:51:26 +0000 (12:51 -0600)]
rust: plugin bootstrap function
Functions written in Rust will need to suricata::plugin::init()
to bootstrap themselves. This bootstrap process sets the log level
within the Rust address space, and hooks up function pointers
that are expected to be set during normal runs of Suricata.
Jason Ish [Tue, 25 Aug 2020 16:12:04 +0000 (10:12 -0600)]
suricata: expose the SuricataContext with a function
Expose the "SuricataContext" required by Rust as a function. During
normal startup we register this context with the Rust code, but
plugins written in Rust will need to get the same registration
done, but to do this in a plugin, the plugin code must
call and set the context within its address space.
Roland Fischer [Tue, 7 Jul 2020 21:54:36 +0000 (17:54 -0400)]
common: Add clang-format file
clang-format allows to auto-format C code. The settings here are set
up to follow the code style, see
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Coding_Style.
Carl Smith [Sun, 16 Aug 2020 20:41:35 +0000 (08:41 +1200)]
threshold: Change rule parsing to use pcre_copy_substring
Fixes memory leak when parsing threshold rules.
All parsed strings are less than 16 characters except
for the IP address which could be up to 48 characters.
Remove redefinition of MAX_SUBSTRINGS
Jason Ish [Mon, 10 Aug 2020 17:15:45 +0000 (11:15 -0600)]
github-ci: add autoreconf to centos-7 build
The prepared distribution packages are failing autoreconf
due to a missing acsite.m4. Add autoreconf to the centos-7
build which uses the prepared package to test for this
issue.
Jason Ish [Sat, 4 Jul 2020 06:33:08 +0000 (00:33 -0600)]
plugins: support for capture plugins
Allow a plugin to register itself as a capture source. This isn't that
much different than how current sources register, it just happens
a little later on during startup.
One "slot" is reserved for capture plugins, but multiple plugins
implementing a capture can be loaded. The --capture-plugin command
line option must be used to tell Suricata which plugin
to use.
This is still very much a work in progress, but can load
PF_RING as a capture plugin.
Jason Ish [Fri, 3 Jul 2020 22:33:12 +0000 (16:33 -0600)]
plugins: initial support for a filetype plugin
A filetype plugin is a plugin that implements an eve filetype. Most
of the current filetypes could likely be implemented as such a plugin.
Such a plugin must implement Open, Close and Write, where Write
is provided the formatted JSON to be logged.
This commit also includes the plumbing for plugin loading. Example
plugin to come.
Plugins are loaded by the "plugin" section in the configuration
file:
Victor Julien [Thu, 19 Dec 2019 16:26:45 +0000 (17:26 +0100)]
flow: redesign of flow timeout handling
Goals:
- reduce locking
- take advantage of 'hot' caches
- better locality
Locking reduction
New flow spare pool. The global pool is implmented as a list of blocks,
where each block has a 100 spare flows. Worker threads fetch a block at
a time, storing the block in the local thread storage.
Flow Recycler now returns flows to the pool is blocks as well.
Flow Recycler fetches all flows to be processed in one step instead of
one at a time.
Cache 'hot'ness
Worker threads now check the timeout of flows they evaluate during lookup.
The worker will have to read the flow into cache anyway, so the added
overhead of checking the timeout value is minimal. When a flow is considered
timed out, one of 2 things happens:
- if the flow is 'owned' by the thread it is handled locally. Handling means
checking if the flow needs 'timeout' work.
- otherwise, the flow is added to a special 'evicted' list in the flow
bucket where it will be picked up by the flow manager.
Flow Manager timing
By default the flow manager now tries to do passes of the flow hash in
smaller steps, where the goal is to do full pass in 8 x the lowest timeout
value it has to enforce. So if the lowest timeout value is 30s, a full pass
will take 4 minutes. The goal here is to reduce locking overhead and not
get in the way of the workers.
In emergency mode each pass is full, and lower timeouts are used.
Timing of the flow manager is also no longer relying on pthread condition
variables, as these generally cause waking up much quicker than the desired
timout. Instead a simple (u)sleep loop is used.
Both changes reduce the number of hash passes a lot.
Emergency behavior
In emergency mode there a number of changes to the workers. In this scenario
the flow memcap is fully used up and it is unavoidable that some flows won't
be tracked.
1. flow spare pool fetches are reduced to once a second. This avoids locking
overhead, while the chance of success was very low.
2. getting an active flow directly from the hash skips flows that had very
recent activity to avoid the scenario where all flows get only into the
NEW state before getting reused. Rather allow some to have a chance of
completing.
3. TCP packets that are not SYN packets will not get a used flow, unless
stream.midstream is enabled. The goal here is again to avoid evicting
active flows unnecessarily.
Better Localily
Flow Manager injects flows into the worker threads now, instead of one or
two packets. Advantage of this is that the worker threads can get packets
from their local packet pools, avoiding constant overhead of packets returning
to 'foreign' pools.
Counters
A lot of flow counters have been added and some have been renamed.
Overall the worker threads increment 'flow.wrk.*' counters, while the flow
manager increments 'flow.mgr.*'.
Additionally, none of the counters are snapshots anymore, they all increment
over time. The flow.memuse and flow.spare counters are exceptions.
Misc
FlowQueue has been split into a FlowQueuePrivate (unlocked) and FlowQueue.
Flow no longer has 'prev' pointers and used a unified 'next' pointer for
both hash and queue use.
Victor Julien [Tue, 24 Dec 2019 15:12:51 +0000 (16:12 +0100)]
flow-manager: call other timeouts max once a second
Call Defrag and others only once per second. Flow Manager may wake
up (much) more often when flow engine is under resource pressure.
As this does not affect Defrag and others, it only unnecessarily
adds load.
projects / thirdparty / suricata.git / log
