[people/ms/suricata.git] / doc / devguide / extending / app-layer / transactions.rst

************
Transactions
************

.. contents:: Table of Contents

General Concepts
================

For Suricata, transactions are an abstraction that help with detecting and logging. An example of a complete transaction is
a pair of messages in the form of a request (from client to server) and a response (from server to client) in HTTP.

In order to know when to log an event for a given protocol, the engine tracks the progress of each transaction - that
is, when is it complete, or when it reaches a key intermediate state. They aid during the detection phase,
when dealing with protocols that can have large PDUs (protocol data units), like TCP, in controlling state for partial rule matching -- in case of rules that mention more than one field.

Transactions are implemented and stored in the per-flow state. The engine interacts with them using a set of callbacks the parser registers.

How the engine uses transactions
================================

Logging
~~~~~~~

Suricata controls when logging should happen based on transaction completeness. For simpler protocols, such as ``dns``
or ``ntp``, that will most
likely happen once per transaction, by the time of its completion. In other cases, like with HTTP, this may happen at intermediary states.

In ``OutputTxLog``, the engine will compare current state with the value defined for the logging to happen, per flow
direction (``logger->tc_log_progress``, ``logger->ts_log_progress``). If state is less than that value, the engine skips to
the next logger. Code snippet from: suricata/src/output-tx.c:

.. code-block:: c

    static TmEcode OutputTxLog(ThreadVars *tv, Packet *p, void *thread_data)
    {
        .
        .
        .
            if ((ts_eof && tc_eof) || last_pseudo) {
                SCLogDebug("EOF, so log now");
            } else {
                if (logger->LogCondition) {
                    int r = logger->LogCondition(tv, p, alstate, tx, tx_id);
                    if (r == FALSE) {
                        SCLogDebug("conditions not met, not logging");
                        goto next_logger;
                    }
                } else {
                    if (tx_progress_tc < logger->tc_log_progress) {
                        SCLogDebug("progress not far enough, not logging");
                        goto next_logger;
                    }

                    if (tx_progress_ts < logger->ts_log_progress) {
                        SCLogDebug("progress not far enough, not logging");
                        goto next_logger;
                    }
                }
             }
        .
        .
        .
    }

Rule Matching
~~~~~~~~~~~~~

Transaction progress is also used for certain keywords to know what is the minimum state before we can expect a match: until that, Suricata won't even try to look for the patterns.

As seen in ``DetectAppLayerMpmRegister2`` that has ``int progress`` as parameter, and ``DetectAppLayerInspectEngineRegister2``, which expects ``int tx_min_progress``, for instance. In the code snippet,
``HTTP2StateDataClient``, ``HTTP2StateDataServer`` and ``0`` are the values passed to the functions.


.. code-block:: c

    void DetectFiledataRegister(void)
    {
        .
        .
        DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOSERVER, 2,
                PrefilterMpmFiledataRegister, NULL,
                ALPROTO_HTTP2, HTTP2StateDataClient);
        DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOCLIENT, 2,
                PrefilterMpmFiledataRegister, NULL,
                ALPROTO_HTTP2, HTTP2StateDataServer);
        .
        .
        DetectAppLayerInspectEngineRegister2("file_data",
            ALPROTO_HTTP2, SIG_FLAG_TOCLIENT, HTTP2StateDataServer,
            DetectEngineInspectFiledata, NULL);
        DetectAppLayerInspectEngineRegister2(
                "file_data", ALPROTO_FTPDATA, SIG_FLAG_TOSERVER, 0, DetectEngineInspectFiledata, NULL);
        .
        .
    }

Progress Tracking
=================

As a rule of thumb, transactions will follow a request-response model: if a transaction has had a request and a response, it is complete.

But if a protocol has situations where a request or response won’t expect or generate a message from its counterpart,
it is also possible to have uni-directional transactions. In such cases, transaction is set to complete at the moment of
creation.

For example, DNS responses may be considered as completed transactions, because they also contain the request data, so
all information needed for logging and detection can be found in the response.

In addition, for file transfer protocols, or similar ones where there may be several messages before the file exchange
is completed (NFS, SMB), it is possible to create a level of abstraction to handle such complexity. This could be achieved by adding phases to the model implemented by the protocol (e.g., protocol negotiation phase (SMB), request parsed (HTTP), and so on).

This is controlled by implementing progress states. In Suricata, those will be enums that are incremented as the parsing
progresses. A state will start at 0. The higher its value, the closer the transaction would be to completion.

The engine interacts with transactions' state using a set of callbacks the parser registers. State is defined per flow direction (``STREAM_TOSERVER`` / ``STREAM_TOCLIENT``).

In Summary - Transactions and State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Initial state value: ``0``.
- Simpler scenarios: state is simply an int.  ``1`` represents transaction completion, per direction.
- Complex Transaction State in Suricata: ``enum`` (Rust: ``i32``). Completion is indicated by the highest enum value (some examples are: SSH, HTTP, HTTP2, DNS, SMB).

Examples
========

This section shares some examples from Suricata codebase, to help visualize how Transactions work and are handled by the engine.

Enums
~~~~~

Code snippet from: rust/src/ssh/ssh.rs:

.. code-block:: rust

    pub enum SSHConnectionState {
        SshStateInProgress = 0,
        SshStateBannerWaitEol = 1,
        SshStateBannerDone = 2,
        SshStateFinished = 3,
    }

From src/app-layer-ftp.h:

.. code-block:: c

    enum {
        FTP_STATE_IN_PROGRESS,
        FTP_STATE_PORT_DONE,
        FTP_STATE_FINISHED,
    };

API Callbacks
~~~~~~~~~~~~~

In Rust, this is done via the RustParser struct. As seen in rust/src/applayer.rs:

.. code-block:: rust

    /// Rust parser declaration
    pub struct RustParser {
            .
            .
            .
        /// Progress values at which the tx is considered complete in a direction
        pub tx_comp_st_ts:      c_int,
        pub tx_comp_st_tc:      c_int,
        .
        .
        .
    }

In C, the callback API is:

.. code-block:: c

    void AppLayerParserRegisterStateProgressCompletionStatus(
        AppProto alproto, const int ts, const int tc)

Simple scenario described, in Rust:

rust/src/dhcp/dhcp.rs:

.. code-block:: rust

    tx_comp_st_ts: 1
    tx_comp_st_tc: 1

For SSH, this looks like this:

rust/src/ssh/ssh.rs:

.. code-block:: rust

    tx_comp_st_ts: SSHConnectionState::SshStateFinished as i32,
    tx_comp_st_tc: SSHConnectionState::SshStateFinished as i32,

In C, callback usage would be as follows:

src/app-layer-dcerpc.c:

.. code-block:: c

    AppLayerParserRegisterStateProgressCompletionStatus(ALPROTO_DCERPC, 1, 1);

src/app-layer-ftp.c:

.. code-block:: c

    AppLayerParserRegisterStateProgressCompletionStatus(
        ALPROTO_FTP, FTP_STATE_FINISHED, FTP_STATE_FINISHED);

Sequence Diagrams
~~~~~~~~~~~~~~~~~

A DNS transaction in Suricata can be considered unidirectional:

.. image:: img/DnsUnidirectionalTransactions.png
  :width: 600
  :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server, labeled "DNS Request". After that, there is a dotted line labeled "Transaction Completed".

An HTTP2 transaction is an example of a bidirectional transaction, in Suricata (note that transactions in HTTP2 may
overlap, scenario not shown in this Sequence Diagram):

.. TODO add another example for overlapping HTTP2 transaction

.. image:: img/HTTP2BidirectionalTransaction.png
  :width: 600
  :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server labeled "Request" and below that an arrow going from Server to Client labeled "Response". Below those arrows, a dotted line indicates that the transaction is completed.

A TLS Handshake is a more complex example, where several messages are exchanged before the transaction is considered completed:

.. image:: img/TlsHandshake.png
  :width: 600
  :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server labeled "ClientHello" and below that an arrow going from Server to Client labeled "ServerHello". Below those arrows, several more follow from Server to Client and vice-versa, before a dotted line indicates that the transaction is finally completed.

Template Protocol
~~~~~~~~~~~~~~~~~

Suricata has a template protocol for educational purposes, which has simple bidirectional transactions.

A completed transaction for the template looks like this:

.. image:: img/TemplateTransaction.png
  :width: 600
  :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server, labeled "Request". An arrow below that first one goes from Server to Client.

Following are the functions that check whether a transaction is considered completed, for the Template Protocol. Those are called by the Suricata API. Similar functions exist for each protocol, and may present implementation differences, based on what is considered a transaction for that given protocol.

In C:

.. code-block:: c

    static int TemplateGetStateProgress(void *txv, uint8_t direction)
    {
        TemplateTransaction *tx = txv;

        SCLogNotice("Transaction progress requested for tx ID %"PRIu64
            ", direction=0x%02x", tx->tx_id, direction);

        if (direction & STREAM_TOCLIENT && tx->response_done) {
            return 1;
        }
        else if (direction & STREAM_TOSERVER) {
            /* For the template, just the existence of the transaction means the
             * request is done. */
            return 1;
        }

        return 0;
    }

And in Rust:

.. code-block:: rust

    pub extern "C" fn rs_template_tx_get_alstate_progress(
        tx: *mut std::os::raw::c_void,
        _direction: u8,
    ) -> std::os::raw::c_int {
        let tx = cast_pointer!(tx, TemplateTransaction);

        // Transaction is done if we have a response.
        if tx.response.is_some() {
            return 1;
        }
        return 0;
    }

Work In Progress changes
========================

Currently we are working to have files be part of the transaction instead of the per-flow state, as seen in https://redmine.openinfosecfoundation.org/issues/4444.

Another work in progress is to limit the number of transactions per flow, to prevent Denial of Service (DoS) by quadratic complexity - a type of attack that may happen to protocols which can have multiple transactions at the same time - such as HTTP2 so-called streams (see  https://redmine.openinfosecfoundation.org/issues/4530).

Common words and abbreviations
==============================

- al, applayer: application layer
- alproto: application layer protocol
- alstate: application layer state
- engine: refers to Suricata core detection logic
- flow: a bidirectional flow of packets with the same 5-tuple elements (protocol, source ip, destination ip, source port, destination port. Vlans can be added as well)
- PDU: Protocol Data Unit
- rs: rust
- tc: to client
- ts: to server
- tx: transaction
Commit	Line	Data
a5b344e0 JF	1	************
	2	Transactions
	3	************
	4
	5	.. contents:: Table of Contents
	6
	7	General Concepts
	8	================
	9
84311ab1	10	For Suricata, transactions are an abstraction that help with detecting and logging. An example of a complete transaction is
a5b344e0 JF	11	a pair of messages in the form of a request (from client to server) and a response (from server to client) in HTTP.
	12
	13	In order to know when to log an event for a given protocol, the engine tracks the progress of each transaction - that
	14	is, when is it complete, or when it reaches a key intermediate state. They aid during the detection phase,
	15	when dealing with protocols that can have large PDUs (protocol data units), like TCP, in controlling state for partial rule matching -- in case of rules that mention more than one field.
	16
	17	Transactions are implemented and stored in the per-flow state. The engine interacts with them using a set of callbacks the parser registers.
	18
	19	How the engine uses transactions
	20	================================
	21
	22	Logging
	23	~~~~~~~
	24
	25	Suricata controls when logging should happen based on transaction completeness. For simpler protocols, such as ``dns``
	26	or ``ntp``, that will most
	27	likely happen once per transaction, by the time of its completion. In other cases, like with HTTP, this may happen at intermediary states.
	28
	29	In ``OutputTxLog``, the engine will compare current state with the value defined for the logging to happen, per flow
	30	direction (``logger->tc_log_progress``, ``logger->ts_log_progress``). If state is less than that value, the engine skips to
	31	the next logger. Code snippet from: suricata/src/output-tx.c:
	32
	33	.. code-block:: c
	34
	35	static TmEcode OutputTxLog(ThreadVars tv, Packet p, void *thread_data)
	36	{
	37	.
	38	.
	39	.
	40	if ((ts_eof && tc_eof) \|\| last_pseudo) {
	41	SCLogDebug("EOF, so log now");
	42	} else {
	43	if (logger->LogCondition) {
	44	int r = logger->LogCondition(tv, p, alstate, tx, tx_id);
	45	if (r == FALSE) {
	46	SCLogDebug("conditions not met, not logging");
	47	goto next_logger;
	48	}
	49	} else {
	50	if (tx_progress_tc < logger->tc_log_progress) {
	51	SCLogDebug("progress not far enough, not logging");
	52	goto next_logger;
	53	}
	54
	55	if (tx_progress_ts < logger->ts_log_progress) {
	56	SCLogDebug("progress not far enough, not logging");
	57	goto next_logger;
	58	}
	59	}
	60	}
	61	.
	62	.
	63	.
	64	}
	65
	66	Rule Matching
	67	~~~~~~~~~~~~~
	68
	69	Transaction progress is also used for certain keywords to know what is the minimum state before we can expect a match: until that, Suricata won't even try to look for the patterns.
	70
	71	As seen in ``DetectAppLayerMpmRegister2`` that has ``int progress`` as parameter, and ``DetectAppLayerInspectEngineRegister2``, which expects ``int tx_min_progress``, for instance. In the code snippet,
	72	``HTTP2StateDataClient``, ``HTTP2StateDataServer`` and ``0`` are the values passed to the functions.
	73
	74
75	.. code-block:: c
76
77	void DetectFiledataRegister(void)
78	{
79	.
80	.
81	DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOSERVER, 2,
82	PrefilterMpmFiledataRegister, NULL,
83	ALPROTO_HTTP2, HTTP2StateDataClient);
84	DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOCLIENT, 2,
85	PrefilterMpmFiledataRegister, NULL,
86	ALPROTO_HTTP2, HTTP2StateDataServer);
87	.
88	.
89	DetectAppLayerInspectEngineRegister2("file_data",
90	ALPROTO_HTTP2, SIG_FLAG_TOCLIENT, HTTP2StateDataServer,
91	DetectEngineInspectFiledata, NULL);
92	DetectAppLayerInspectEngineRegister2(
93	"file_data", ALPROTO_FTPDATA, SIG_FLAG_TOSERVER, 0, DetectEngineInspectFiledata, NULL);
94	.
95	.
96	}
97
98	Progress Tracking
99	=================
100
101	As a rule of thumb, transactions will follow a request-response model: if a transaction has had a request and a response, it is complete.
102
103	But if a protocol has situations where a request or response won’t expect or generate a message from its counterpart,
104	it is also possible to have uni-directional transactions. In such cases, transaction is set to complete at the moment of
105	creation.
106
107	For example, DNS responses may be considered as completed transactions, because they also contain the request data, so
108	all information needed for logging and detection can be found in the response.
109
110	In addition, for file transfer protocols, or similar ones where there may be several messages before the file exchange
111	is completed (NFS, SMB), it is possible to create a level of abstraction to handle such complexity. This could be achieved by adding phases to the model implemented by the protocol (e.g., protocol negotiation phase (SMB), request parsed (HTTP), and so on).
112
113	This is controlled by implementing progress states. In Suricata, those will be enums that are incremented as the parsing
114	progresses. A state will start at 0. The higher its value, the closer the transaction would be to completion.
115
84311ab1	116	The engine interacts with transactions' state using a set of callbacks the parser registers. State is defined per flow direction (``STREAM_TOSERVER`` / ``STREAM_TOCLIENT``).
a5b344e0 JF	117
	118	In Summary - Transactions and State
	119	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	120
	121	- Initial state value: ``0``.
	122	- Simpler scenarios: state is simply an int. ``1`` represents transaction completion, per direction.
	123	- Complex Transaction State in Suricata: ``enum`` (Rust: ``i32``). Completion is indicated by the highest enum value (some examples are: SSH, HTTP, HTTP2, DNS, SMB).
	124
	125	Examples
	126	========
	127
	128	This section shares some examples from Suricata codebase, to help visualize how Transactions work and are handled by the engine.
	129
	130	Enums
	131	~~~~~
	132
	133	Code snippet from: rust/src/ssh/ssh.rs:
	134
	135	.. code-block:: rust
	136
	137	pub enum SSHConnectionState {
	138	SshStateInProgress = 0,
	139	SshStateBannerWaitEol = 1,
	140	SshStateBannerDone = 2,
	141	SshStateFinished = 3,
	142	}
	143
	144	From src/app-layer-ftp.h:
	145
	146	.. code-block:: c
	147
	148	enum {
	149	FTP_STATE_IN_PROGRESS,
	150	FTP_STATE_PORT_DONE,
	151	FTP_STATE_FINISHED,
	152	};
	153
	154	API Callbacks
	155	~~~~~~~~~~~~~
	156
	157	In Rust, this is done via the RustParser struct. As seen in rust/src/applayer.rs:
	158
	159	.. code-block:: rust
	160
	161	/// Rust parser declaration
	162	pub struct RustParser {
	163	.
	164	.
	165	.
	166	/// Progress values at which the tx is considered complete in a direction
	167	pub tx_comp_st_ts: c_int,
	168	pub tx_comp_st_tc: c_int,
	169	.
	170	.
	171	.
	172	}
	173
	174	In C, the callback API is:
	175
	176	.. code-block:: c
	177
	178	void AppLayerParserRegisterStateProgressCompletionStatus(
	179	AppProto alproto, const int ts, const int tc)
	180
181	Simple scenario described, in Rust:
182
183	rust/src/dhcp/dhcp.rs:
184
185	.. code-block:: rust
186
187	tx_comp_st_ts: 1
188	tx_comp_st_tc: 1
189
190	For SSH, this looks like this:
191
192	rust/src/ssh/ssh.rs:
193
194	.. code-block:: rust
195
196	tx_comp_st_ts: SSHConnectionState::SshStateFinished as i32,
197	tx_comp_st_tc: SSHConnectionState::SshStateFinished as i32,
198
199	In C, callback usage would be as follows:
200
201	src/app-layer-dcerpc.c:
202
203	.. code-block:: c
204
205	AppLayerParserRegisterStateProgressCompletionStatus(ALPROTO_DCERPC, 1, 1);
206
207	src/app-layer-ftp.c:
208
209	.. code-block:: c
210
211	AppLayerParserRegisterStateProgressCompletionStatus(
212	ALPROTO_FTP, FTP_STATE_FINISHED, FTP_STATE_FINISHED);
213
214	Sequence Diagrams
215	~~~~~~~~~~~~~~~~~
216
217	A DNS transaction in Suricata can be considered unidirectional:
218
d6c5dfac	219	.. image:: img/DnsUnidirectionalTransactions.png
a5b344e0 JF	220	:width: 600
	221	:alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server, labeled "DNS Request". After that, there is a dotted line labeled "Transaction Completed".
	222
	223	An HTTP2 transaction is an example of a bidirectional transaction, in Suricata (note that transactions in HTTP2 may
	224	overlap, scenario not shown in this Sequence Diagram):
	225
	226	.. TODO add another example for overlapping HTTP2 transaction
	227
	228	.. image:: img/HTTP2BidirectionalTransaction.png
	229	:width: 600
	230	:alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server labeled "Request" and below that an arrow going from Server to Client labeled "Response". Below those arrows, a dotted line indicates that the transaction is completed.
	231
	232	A TLS Handshake is a more complex example, where several messages are exchanged before the transaction is considered completed:
	233
	234	.. image:: img/TlsHandshake.png
	235	:width: 600
	236	:alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server labeled "ClientHello" and below that an arrow going from Server to Client labeled "ServerHello". Below those arrows, several more follow from Server to Client and vice-versa, before a dotted line indicates that the transaction is finally completed.
	237
	238	Template Protocol
	239	~~~~~~~~~~~~~~~~~
	240
	241	Suricata has a template protocol for educational purposes, which has simple bidirectional transactions.
	242
	243	A completed transaction for the template looks like this:
	244
	245	.. image:: img/TemplateTransaction.png
	246	:width: 600
	247	:alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server, labeled "Request". An arrow below that first one goes from Server to Client.
	248
	249	Following are the functions that check whether a transaction is considered completed, for the Template Protocol. Those are called by the Suricata API. Similar functions exist for each protocol, and may present implementation differences, based on what is considered a transaction for that given protocol.
	250
	251	In C:
	252
	253	.. code-block:: c
	254
	255	static int TemplateGetStateProgress(void *txv, uint8_t direction)
	256	{
	257	TemplateTransaction *tx = txv;
	258
	259	SCLogNotice("Transaction progress requested for tx ID %"PRIu64
	260	", direction=0x%02x", tx->tx_id, direction);
	261
	262	if (direction & STREAM_TOCLIENT && tx->response_done) {
	263	return 1;
	264	}
	265	else if (direction & STREAM_TOSERVER) {
	266	/* For the template, just the existence of the transaction means the
	267	* request is done. */
	268	return 1;
	269	}
	270
	271	return 0;
	272	}
	273
	274	And in Rust:
	275
	276	.. code-block:: rust
	277
	278	pub extern "C" fn rs_template_tx_get_alstate_progress(
	279	tx: *mut std::os::raw::c_void,
	280	_direction: u8,
	281	) -> std::os::raw::c_int {
	282	let tx = cast_pointer!(tx, TemplateTransaction);
	283
284	// Transaction is done if we have a response.
285	if tx.response.is_some() {
286	return 1;
287	}
288	return 0;
289	}
290
291	Work In Progress changes
292	========================
293
294	Currently we are working to have files be part of the transaction instead of the per-flow state, as seen in https://redmine.openinfosecfoundation.org/issues/4444.
295
84311ab1	296	Another work in progress is to limit the number of transactions per flow, to prevent Denial of Service (DoS) by quadratic complexity - a type of attack that may happen to protocols which can have multiple transactions at the same time - such as HTTP2 so-called streams (see https://redmine.openinfosecfoundation.org/issues/4530).
a5b344e0 JF	297
	298	Common words and abbreviations
	299	==============================
	300
	301	- al, applayer: application layer
	302	- alproto: application layer protocol
	303	- alstate: application layer state
	304	- engine: refers to Suricata core detection logic
	305	- flow: a bidirectional flow of packets with the same 5-tuple elements (protocol, source ip, destination ip, source port, destination port. Vlans can be added as well)
	306	- PDU: Protocol Data Unit
	307	- rs: rust
	308	- tc: to client
	309	- ts: to server
	310	- tx: transaction