This directory contains classes related to Snort configuration dump.
-Snort supports dumping config into a file for JSON and text formats.
+Snort can dump configuration (it read from all config sources) in JSON or text formats.
-* ConfigData
+==== ConfigData
- The ConfigData structure represents the internal config data of a particular Lua file.
- A unique shell is associated with every base and targeted policy file.
- Every module in a Lua file is represented by the General Tree.
- The Tree node can be one of the following types: TreeConfigNode, ValueConfigNode.
- The TreeConfigNode represents a table or list.
- The ValueConfigNode represents a config value itself.
+ConfigData structure represents the internal config data of a particular Lua file.
+An unique shell is associated with every base and targeted policy files.
+Every module in a Lua file is represented by the General Tree.
+Tree node can be one of the following types: TreeConfigNode, ValueConfigNode.
+TreeConfigNode represents a table or a list.
+ValueConfigNode represents a configured value itself.
-* ConfigOutput
+==== ConfigOutput
- The ConfigOutput class is a base class that encapsulates dumping config into a file.
- Pure virtual function dump() should be overridden by derived classes for particular
- output format.
+ConfigOutput class is a base class that encapsulates dumping config into a file.
+Pure virtual function dump() should be overridden by derived classes for particular
+output format.
-* JsonAllConfigOutput
+==== JsonAllConfigOutput
- The JsonAllConfigOutput class inherits from ConfigOutput and dumps base and targeted
- policy file in JSON format.
+The JsonAllConfigOutput class inherits from ConfigOutput and dumps base and targeted
+policy file in JSON format.
-* TextConfigOutput
+==== TextConfigOutput
- The TextConfigOutput class inherits from ConfigOutput and dumps base and targeted
- policy file in text format.
+The TextConfigOutput class inherits from ConfigOutput and dumps base and targeted
+policy file in text format.
3. JSON
-===== File Layout
+==== File Layout
[options="header"]
|============================================================================
|Records |This is a stream of records. There may be an unlimited number.
|============================================================================
-===== File Header
+==== File Header
[options="header"]
|==========================================================================
|Schema |(schema size) bytes |Schema for parsing records in this file.
|==========================================================================
-===== Record
+==== Record
[options="header"]
|===========================================================================
could easily manipulate, such as header length, chunking, compression, and encodings. The maximum
depth is computed against normalized message body data.
-===== Partial inspection
+==== Partial inspection
include::dev_notes_partial_inspection.txt[]
-===== Message section
+==== Message section
include::dev_notes_message_section.txt[]
-===== Field class
+==== Field class
include::dev_notes_field_class.txt[]
-===== Support for custom xff type headers
+==== Support for custom xff type headers
include::dev_notes_hff_headers.txt[]
-===== URI normalization
+==== URI normalization
include::dev_notes_uri_norm.txt[]
-===== JS normalization
+==== JS normalization
include::dev_notes_js_norm.txt[]
-===== Reassembling chunked message
+==== Reassembling chunked message
include::dev_notes_chunked_processing.txt[]
-===== MIME inspection
+==== MIME inspection
include::dev_notes_mime_inspection.txt[]
-===== HI test tool usage
+==== HI test tool usage
include::dev_notes_test_tool.txt[]
-The wizard uses hexes, spells and curses to determine the most likely service
+The wizard uses hexes, spells and curses to determine the most likely service
on a flow. It does not determine the service with certainty; that is the job of
the service inspector or appId. The goal is to get the most likely service
inspector engaged as quickly as possible.
Hexes and spells differ in the following aspects:
- * spells allow wildcards matching any number of consecutive characters
- whereas hexes allow a single wild char.
- * spells are case insensitive whereas hexes are case sensitive.
- * spells automatically skip leading whitespace (at very start of flow).
+* spells allow wildcards matching any number of consecutive characters
+ whereas hexes allow a single wild char.
+* spells are case insensitive whereas hexes are case sensitive.
+* spells automatically skip leading whitespace (at very start of flow).
To match "`*`" symbol in traffic, put "`**`" in a spell.
+For example:
- For example: traffic "* OK" will match the pattern "** OK".
+ traffic "* OK" will match the pattern "** OK".
A series of asterisks is matched from left to right. '***' is seen as "*<glob>".
==== Concepts
- * `MagicPage` - leaf of a trie. Represents a symbol of a pattern.
- * `MagicBook` - trie itself. Represents a set of patterns for the wizard instance.
- ** `SpellBook` - `MagicBook` implementation for spells.
- ** `HexBook` - `MagicBook` implementation for hexes.
- * `MagicSplitter` - object related to a stream. Applies wizard logic to a stream.
- * `Wand` - contains state of wizard patterns for a stream.
- * `CurseDetails` - settings of a curse. Contains identifiers and algorithm.
- * `CurseTracker` - state of a curse.
- * `CurseBook` - contains all configured curses.
- * `CurseServiceTracker` - instance of a curse. Contains settings and state.
+* `MagicPage` - leaf of a trie. Represents a symbol of a pattern.
+* `MagicBook` - trie itself. Represents a set of patterns for the wizard instance.
+ ** `SpellBook` - `MagicBook` implementation for spells.
+ ** `HexBook` - `MagicBook` implementation for hexes.
+* `MagicSplitter` - object related to a stream. Applies wizard logic to a stream.
+* `Wand` - contains state of wizard patterns for a stream.
+* `CurseDetails` - settings of a curse. Contains identifiers and algorithm.
+* `CurseTracker` - state of a curse.
+* `CurseBook` - contains all configured curses.
+* `CurseServiceTracker` - instance of a curse. Contains settings and state.
==== MagicSplitter
Each flow contains two `MagicSplitter` objects: client-to-server and server-to-client.
Each `MagicSplitter` contains `Wand` that stores pointers unique for the flow:
- 1. MagicPage of Hex
- 2. MagicPage of Spell
- 3. Vector of all curses
+1. MagicPage of Hex
+2. MagicPage of Spell
+3. Vector of all curses
Where 1 and 2 - point to the current page in pattern.
==== Spell matching algorithm
-
-The spell matching algorithm is defined in `SpellBook::find_spell()` method.
+
+The spell matching algorithm is defined in `SpellBook::find_spell()` method.
In general, `MagicPage::next` array is an alphabet (ASCII table),
each element of which can exist or be absent. Thus, if an element exists
in the position of a certain symbol, it means that there is a pattern with
-such a sequence of symbols.
+such a sequence of symbols.
+
+Example:
- Example:
- User configured only one pattern: "ABC"
- MagicPage(root)::next - all elements beside (int)A is nullptr.
- MagicPage(A)::next - all elements beside (int)B is nullptr.
- MagicPage(B)::next - all elements beside (int)C is nullptr.
+ User configured only one pattern: "ABC"
+ MagicPage(root)::next - all elements beside (int)A is nullptr.
+ MagicPage(A)::next - all elements beside (int)B is nullptr.
+ MagicPage(B)::next - all elements beside (int)C is nullptr.
-Wizard iterates over the data from begin to end, checking at each iteration
-if there is a transition from the current character of the pattern to the
+Wizard iterates over the data from begin to end, checking at each iteration
+if there is a transition from the current character of the pattern to the
next character of the data.
`MagicPage::any` reflects a glob (wildcard). If wizard transitioned to a glob of the pattern,
a loop is started, in which wizard is trying to match the pattern from the current symbol of the
-data. If it failed to match the pattern from the current symbol of the data, it moves
-to the next symbol and tries again, and so either until it matches the pattern or the
+data. If it failed to match the pattern from the current symbol of the data, it moves
+to the next symbol and tries again, and so either until it matches the pattern or the
data runs out.
-`MagicPage::value` is not empty only in those positions that are the ends of some pattern.
-Thus, if, after a complete pass through the data, the wizard have reached a position in which this
+`MagicPage::value` is not empty only in those positions that are the ends of some pattern.
+Thus, if, after a complete pass through the data, the wizard have reached a position in which this
field is not empty, means that it has matched the pattern.
It should be mentioned that the matching of spells is case-independent, this is
==== Hex matching algorithm
-The algorithm for matching hexes is defined in `HexBook::find_spell()` and
+The algorithm for matching hexes is defined in `HexBook::find_spell()` and
identical to the algorithm for spells, but lacks:
- 1) converting to an uppercase, since hexes work with raw data;
- 2) loops for working with the glob, since glob in hexes replaces exactly one symbol;
- 3) saving the position of the glob between packets.
+* converting to an uppercase, since hexes work with raw data;
+* loops for working with the glob, since glob in hexes replaces exactly one symbol;
+* saving the position of the glob between packets.
==== TCP traffic processing
-Execution starts from the `MagicSplitter::scan()`.
+Execution starts from the `MagicSplitter::scan()`.
Since we want to be able to match patterns between packets in a stream, wizard need to
-save the state of the pattern at the end of the processing of a particular packet.
+save the state of the pattern at the end of the processing of a particular packet.
The state of the pattern is saved in the `MagicSplitter::wand`. However, for spells,
it needs to keep the presence of a glob between packages. This is implemented with
`MagicSplitter::bookmark`.
-Spells, hexes and curses are called inside the `Wizard::cast_spell()`.
+Spells, hexes and curses are called inside the `Wizard::cast_spell()`.
There wizard determines the search depth and sequentially calls the processing methods.
If wizard matched the pattern in the `Wizard::cast_spell()`, it increments `tcp_hits`.
If it didn't, then it checks whether it reached the limit of `max_search_depth`.
-If wizard has reached the limit of `max_search_depth` and has't matched a pattern,
+If wizard has reached the limit of `max_search_depth` and has't matched a pattern,
then it nullifies `Wand::spell` and `Wand::hex`, thus further in `Wizard::finished()` it'll
know that this flow can be abandoned and raise `tcp_misses` by 1.
Way of processing UDP is similar to TCP but has some differences:
- 1. Instead `MagicSplitter::scan()`, processing starts from `Wizard::eval()`;
- 2. Wizard processes only the first packet of UDP "meta-flow", so for
- every packet amount of previously processed bytes sets at 0;
- 3. There isn't any bookmark - UDP doesn't support wildcard over several packets.
- 4. The wizard don't need to check `Wizard::finished()`, because it processes only the
- first packet of UDP "meta-flow". So, if it hasn't matched anything in
- `Wizard::cast_spell()`, it increments `udp_misses` and unbinds itself from the flow.
+1. Instead `MagicSplitter::scan()`, processing starts from `Wizard::eval()`;
+2. Wizard processes only the first packet of UDP "meta-flow", so for
+ every packet amount of previously processed bytes sets at 0;
+3. There isn't any bookmark - UDP doesn't support wildcard over several packets.
+4. The wizard don't need to check `Wizard::finished()`, because it processes only the
+ first packet of UDP "meta-flow". So, if it hasn't matched anything in
+ `Wizard::cast_spell()`, it increments `udp_misses` and unbinds itself from the flow.
==== Additional info
Every flow gets a context (in `MagicSplitter`), where wizard stores flow's processing state.
Each flow is processed independently from others.
-Currently wizard cannot roll back on the pattern, so if it reaches a certain symbol
-of the pattern, it cannot go back. In some cases this will lead to the fact that
+Currently wizard cannot roll back on the pattern, so if it reaches a certain symbol
+of the pattern, it cannot go back. In some cases this will lead to the fact that
the pattern that could be matched will not be matched.
- For example:
- Patterns: "foobar", "foo*"
- Content: "foobaz"
+For example:
+
+ Patterns: "foobar", "foo*"
+ Content: "foobaz"
Unfortunately, none of the available patterns will match in such case.
- This is due to the fact that the symbols have a higher priority than
+ This is due to the fact that the symbols have a higher priority than
the glob. So from the MagicPage(O) wizard will transit to the MagicPage(B)
of the "foobar" pattern and will not process glob. Further, in MagicPage(А)::next[]
it will not find MagicPage by the symbol "z" and will consider the pattern unmatched.
This directory contains the trace logger framework.
-* TraceLogger
+==== TraceLogger
- The TraceLogger encapsulates the logging method for traces. A particular thread-local logger
- is created per each packet thread and one for the main thread. The logging configuration
- happens in the module. The logger factory is used to init/cleanup loggers.
+The TraceLogger encapsulates the logging method for traces. A particular thread-local logger
+is created per each packet thread and one for the main thread. The logging configuration
+happens in the module. The logger factory is used to init/cleanup loggers.
- Include "trace_logger.h" to get TraceLogger base class.
- Built-in loggers are defined in "trace_loggers.h/trace_loggers.cc".
+Include "trace_logger.h" to get TraceLogger base class.
+Built-in loggers are defined in "trace_loggers.h"/"trace_loggers.cc".
-* TraceLoggerFactory
+==== TraceLoggerFactory
- The base TraceLoggerFactory is used to create a particular TraceLogger instance per each
- thread. One factory instance exists which used to init/cleanup loggers and placed
- into TraceConfig. The factory object instantiates in the module due to configuration.
+The base TraceLoggerFactory is used to create a particular TraceLogger instance per each
+thread. One factory instance exists which used to init/cleanup loggers and placed
+into TraceConfig. The factory object instantiates in the module due to configuration.
- Include "trace_logger.h" to get TraceLoggerFactory base class and template function
- to create particular objects. Built-in factories are defined in "trace_loggers.h/trace_loggers.cc".
+Include "trace_logger.h" to get TraceLoggerFactory base class and template function
+to create particular objects. Built-in factories are defined in "trace_loggers.h/trace_loggers.cc".
-* TraceConfig
+==== TraceConfig
- This is a class that contains pointer on TraceLoggerFactory object and Trace list.
- By default all Traces are off. TraceModule fills Traces from configuration.
+This is a class that contains pointer on TraceLoggerFactory object and Trace list.
+By default all Traces are off. TraceModule fills Traces from configuration.
- TraceConfig placed into "trace_config.h".
+TraceConfig placed into "trace_config.h".
-* TraceModule
+==== TraceModule
- This module provides configuration for trace logs:
- output - create a concrete logger factory based on the output value (stdout/syslog).
- constraints - set packet constraints to use for trace filtering.
- modules - set modules trace level verbosity.
- ntuple - on/off packet n-tuple info logging.
- timestamp - on/off message time stamps.
+This module provides configuration for trace logs:
- This is a built-in module (from coreinit.lua)
+* `output` - create a concrete logger factory based on the output value (stdout/syslog).
+* `constraints` - set packet constraints to use for trace filtering.
+* `modules` - set modules trace level verbosity.
+* `ntuple` - on/off packet n-tuple info logging.
+* `timestamp` - on/off message time stamps.
- The module placed into "trace_module.h/trace_module.cc".
+This is a built-in module (from coreinit.lua)
+The module placed into "trace_module.h"/"trace_module.cc".
- TraceModule ctor should be called after all existed modules due to TraceModule
- dynamic params restriction.
+TraceModule ctor should be called after all existed modules due to TraceModule
+dynamic params restriction.
-* TraceSwap
+==== TraceSwap
- This class extends snort::AnalyzerCommand and represents control channel CLI commands
- for setting/clearing trace configuration from the shell. Commands parameters are encapsulated in
- the TraceSwapParams class.
- Available commands:
- 1) trace.set({ modules = {}, constraints = {}, ntuple = true/false, timestamp = true/false })
- -- set new modules traces, constraints, ntuple and timestamp options;
- 2) trace.clear() -- clear modules traces and constraints.
+This class extends snort::AnalyzerCommand and represents control channel CLI commands
+for setting/clearing trace configuration from the shell. Commands parameters are encapsulated in
+the TraceSwapParams class.
+Available commands:
-* TraceSwapParams
+* `trace.set({ modules = {}, constraints = {}, ntuple = true/false, timestamp = true/false })`
+ -- set new modules traces, constraints, ntuple and timestamp options;
+* `trace.clear()` -- clear modules traces and constraints.
- This is a helper class for TraceSwap which encapsulates dynamic initialization and storing of
- snort::Parameter and snort::Command lists based on TraceModule's parameters.
+==== TraceSwapParams
-* TraceParser
+This is a helper class for TraceSwap which encapsulates dynamic initialization and storing of
+snort::Parameter and snort::Command lists based on TraceModule's parameters.
- This class encapsulates module trace options and packet constraints parsing and setting logic.
+==== TraceParser
-* TraceApi
+This class encapsulates module trace options and packet constraints parsing and setting logic.
- TraceApi is a facade API class used to init/reinit/term thread-local trace logger and module's
- trace pointers and to match packets and flows against trace filtering constraints.
+==== TraceApi
- TraceApi placed into "trace_api.h/trace_api.cc"
+TraceApi is a facade API class used to init/reinit/term thread-local trace logger and module's
+trace pointers and to match packets and flows against trace filtering constraints.
-* TraceOption
+TraceApi placed into "trace_api.h"/"trace_api.cc"
- Represents a targeted module's trace option.
- The targeted module provides a list of its trace options,
- which control the verbosity level of the module's trace messages.
- Since the messages can be present in release-build and/or debug-build,
- one should pay attention which option should be in the options list.
- TraceOptionID indexes (for the module's trace options) must start from zero.
+==== TraceOption
-* Trace
+Represents a targeted module's trace option.
+The targeted module provides a list of its trace options,
+which control the verbosity level of the module's trace messages.
+Since the messages can be present in release-build and/or debug-build,
+one should pay attention which option should be in the options list.
+TraceOptionID indexes (for the module's trace options) must start from zero.
- This class encapsulates trace verbosity functionality.
+==== Trace
-* Configuration override
+This class encapsulates trace verbosity functionality.
- By default, the factory will be initialized based on Snort run mode (stdout or syslog).
- But this can be explicitly overwritten in TraceModule's configuration to specify
- which kind of logger to use.
+==== Configuration override
- Changes will be applied in case of reloading too.
+By default, the factory will be initialized based on Snort run mode (stdout or syslog).
+But this can be explicitly overwritten in TraceModule's configuration to specify
+which kind of logger to use.
-* External dependencies
+Changes will be applied in case of reloading too.
- Include TraceApi header somewhere in the code where it needs to make trace logs.
- Make sure that TraceApi::thread_init()/thread_term() are provided for thread where
- TraceApi::log() is going used.
+==== External dependencies
- TraceConfig should be configured in SnortConfig before TraceApi init.
+Include TraceApi header somewhere in the code where it needs to make trace logs.
+Make sure that TraceApi::thread_init()/thread_term() are provided for thread where
+TraceApi::log() is going used.
- To create specific TraceLogger/TraceLoggerFactory pair just inherit base classes placed
- into "trace_logger.h" and init TraceConfig with a new factory during configuration.
+TraceConfig should be configured in SnortConfig before TraceApi init.
-* Extending the trace logger framework with TraceLogger plugins
+To create specific TraceLogger/TraceLoggerFactory pair just inherit base classes placed
+into "trace_logger.h" and init TraceConfig with a new factory during configuration.
- It's possible to create a trace logger as an inspector plugin to handle a custom logic of trace
- messages printing. The workflow here is to implement the custom logger and logger factory by
- inheriting from the Snort TraceLogger and TraceLoggerFactory classes, put them into a separate
- plugin, and call TraceApi::override_logger_factory() during the plugin configuration to
- initialize the framework with the custom logger factory.
+==== Extending the trace logger framework with TraceLogger plugins
-* Disabling packet constraints matching
+It's possible to create a trace logger as an inspector plugin to handle a custom logic of trace
+messages printing. The workflow here is to implement the custom logger and logger factory by
+inheriting from the Snort TraceLogger and TraceLoggerFactory classes, put them into a separate
+plugin, and call TraceApi::override_logger_factory() during the plugin configuration to
+initialize the framework with the custom logger factory.
- Constraints matching can be disabled by setting "trace.constraints.match = false" during
- configuration. This is effectively saying all packets don't pass the trace filter. It may be
- useful in case of using external packet filtering, such as filtering by the DAQ, or to block
- printing for all trace messages in the context of a packet.
+==== Disabling packet constraints matching
+
+Constraints matching can be disabled by setting "trace.constraints.match = false" during
+configuration. This is effectively saying all packets don't pass the trace filter. It may be
+useful in case of using external packet filtering, such as filtering by the DAQ, or to block
+printing for all trace messages in the context of a packet.
projects / thirdparty / snort3.git / commitdiff
