]> git.ipfire.org Git - people/ms/suricata.git/blame - doc/devguide/extending/app-layer/transactions.rst
doc/devguide: add few more explanations & details
[people/ms/suricata.git] / doc / devguide / extending / app-layer / transactions.rst
CommitLineData
a5b344e0
JF
1************
2Transactions
3************
4
5.. contents:: Table of Contents
6
7General Concepts
8================
9
84311ab1 10For Suricata, transactions are an abstraction that help with detecting and logging. An example of a complete transaction is
a5b344e0
JF
11a pair of messages in the form of a request (from client to server) and a response (from server to client) in HTTP.
12
13In order to know when to log an event for a given protocol, the engine tracks the progress of each transaction - that
14is, when is it complete, or when it reaches a key intermediate state. They aid during the detection phase,
15when dealing with protocols that can have large PDUs (protocol data units), like TCP, in controlling state for partial rule matching -- in case of rules that mention more than one field.
16
17Transactions are implemented and stored in the per-flow state. The engine interacts with them using a set of callbacks the parser registers.
18
19How the engine uses transactions
20================================
21
22Logging
23~~~~~~~
24
25Suricata controls when logging should happen based on transaction completeness. For simpler protocols, such as ``dns``
26or ``ntp``, that will most
27likely happen once per transaction, by the time of its completion. In other cases, like with HTTP, this may happen at intermediary states.
28
29In ``OutputTxLog``, the engine will compare current state with the value defined for the logging to happen, per flow
30direction (``logger->tc_log_progress``, ``logger->ts_log_progress``). If state is less than that value, the engine skips to
31the next logger. Code snippet from: suricata/src/output-tx.c:
32
33.. code-block:: c
34
35 static TmEcode OutputTxLog(ThreadVars *tv, Packet *p, void *thread_data)
36 {
37 .
38 .
39 .
40 if ((ts_eof && tc_eof) || last_pseudo) {
41 SCLogDebug("EOF, so log now");
42 } else {
43 if (logger->LogCondition) {
44 int r = logger->LogCondition(tv, p, alstate, tx, tx_id);
45 if (r == FALSE) {
46 SCLogDebug("conditions not met, not logging");
47 goto next_logger;
48 }
49 } else {
50 if (tx_progress_tc < logger->tc_log_progress) {
51 SCLogDebug("progress not far enough, not logging");
52 goto next_logger;
53 }
54
55 if (tx_progress_ts < logger->ts_log_progress) {
56 SCLogDebug("progress not far enough, not logging");
57 goto next_logger;
58 }
59 }
60 }
61 .
62 .
63 .
64 }
65
66Rule Matching
67~~~~~~~~~~~~~
68
69Transaction progress is also used for certain keywords to know what is the minimum state before we can expect a match: until that, Suricata won't even try to look for the patterns.
70
71As seen in ``DetectAppLayerMpmRegister2`` that has ``int progress`` as parameter, and ``DetectAppLayerInspectEngineRegister2``, which expects ``int tx_min_progress``, for instance. In the code snippet,
dbeb8bfa
JF
72``HTTP2StateDataClient``, ``HTTP2StateDataServer`` and ``0`` are the values passed to the functions - in the last
73example, for ``FTPDATA``,
74the existence of a transaction implies that a file is being transferred. Hence the ``0`` value.
a5b344e0
JF
75
76
77.. code-block:: c
78
79 void DetectFiledataRegister(void)
80 {
81 .
82 .
83 DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOSERVER, 2,
84 PrefilterMpmFiledataRegister, NULL,
85 ALPROTO_HTTP2, HTTP2StateDataClient);
86 DetectAppLayerMpmRegister2("file_data", SIG_FLAG_TOCLIENT, 2,
87 PrefilterMpmFiledataRegister, NULL,
88 ALPROTO_HTTP2, HTTP2StateDataServer);
89 .
90 .
91 DetectAppLayerInspectEngineRegister2("file_data",
92 ALPROTO_HTTP2, SIG_FLAG_TOCLIENT, HTTP2StateDataServer,
93 DetectEngineInspectFiledata, NULL);
94 DetectAppLayerInspectEngineRegister2(
95 "file_data", ALPROTO_FTPDATA, SIG_FLAG_TOSERVER, 0, DetectEngineInspectFiledata, NULL);
96 .
97 .
98 }
99
100Progress Tracking
101=================
102
103As a rule of thumb, transactions will follow a request-response model: if a transaction has had a request and a response, it is complete.
104
105But if a protocol has situations where a request or response won’t expect or generate a message from its counterpart,
106it is also possible to have uni-directional transactions. In such cases, transaction is set to complete at the moment of
107creation.
108
109For example, DNS responses may be considered as completed transactions, because they also contain the request data, so
110all information needed for logging and detection can be found in the response.
111
112In addition, for file transfer protocols, or similar ones where there may be several messages before the file exchange
113is completed (NFS, SMB), it is possible to create a level of abstraction to handle such complexity. This could be achieved by adding phases to the model implemented by the protocol (e.g., protocol negotiation phase (SMB), request parsed (HTTP), and so on).
114
115This is controlled by implementing progress states. In Suricata, those will be enums that are incremented as the parsing
dbeb8bfa
JF
116progresses. A state will start at 0. The higher its value, the closer the transaction would be to completion. Due to how
117the engine tracks detection accross states, there is an upper limit of 48 to the state progress (it must be < 48).
a5b344e0 118
84311ab1 119The engine interacts with transactions' state using a set of callbacks the parser registers. State is defined per flow direction (``STREAM_TOSERVER`` / ``STREAM_TOCLIENT``).
a5b344e0
JF
120
121In Summary - Transactions and State
122~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
123
dbeb8bfa
JF
124- Initial State value: ``0``.
125- Simpler scenarios: State is simply a bool. ``1`` represents transaction completion, per direction.
a5b344e0
JF
126- Complex Transaction State in Suricata: ``enum`` (Rust: ``i32``). Completion is indicated by the highest enum value (some examples are: SSH, HTTP, HTTP2, DNS, SMB).
127
128Examples
129========
130
131This section shares some examples from Suricata codebase, to help visualize how Transactions work and are handled by the engine.
132
133Enums
134~~~~~
135
136Code snippet from: rust/src/ssh/ssh.rs:
137
138.. code-block:: rust
139
140 pub enum SSHConnectionState {
141 SshStateInProgress = 0,
142 SshStateBannerWaitEol = 1,
143 SshStateBannerDone = 2,
144 SshStateFinished = 3,
145 }
146
147From src/app-layer-ftp.h:
148
149.. code-block:: c
150
151 enum {
152 FTP_STATE_IN_PROGRESS,
153 FTP_STATE_PORT_DONE,
154 FTP_STATE_FINISHED,
155 };
156
f65b3908
JF
157From src/app-layer-ssl.h:
158
159.. code-block:: c
160
161 enum {
162 TLS_STATE_IN_PROGRESS = 0,
163 TLS_STATE_CERT_READY = 1,
164 TLS_HANDSHAKE_DONE = 2,
165 TLS_STATE_FINISHED = 3
166 };
167
a5b344e0
JF
168API Callbacks
169~~~~~~~~~~~~~
170
171In Rust, this is done via the RustParser struct. As seen in rust/src/applayer.rs:
172
173.. code-block:: rust
174
175 /// Rust parser declaration
176 pub struct RustParser {
177 .
178 .
179 .
180 /// Progress values at which the tx is considered complete in a direction
181 pub tx_comp_st_ts: c_int,
182 pub tx_comp_st_tc: c_int,
183 .
184 .
185 .
186 }
187
188In C, the callback API is:
189
190.. code-block:: c
191
192 void AppLayerParserRegisterStateProgressCompletionStatus(
193 AppProto alproto, const int ts, const int tc)
194
195Simple scenario described, in Rust:
196
197rust/src/dhcp/dhcp.rs:
198
199.. code-block:: rust
200
dbeb8bfa
JF
201 tx_comp_st_ts: 1,
202 tx_comp_st_tc: 1,
a5b344e0
JF
203
204For SSH, this looks like this:
205
206rust/src/ssh/ssh.rs:
207
208.. code-block:: rust
209
210 tx_comp_st_ts: SSHConnectionState::SshStateFinished as i32,
211 tx_comp_st_tc: SSHConnectionState::SshStateFinished as i32,
212
213In C, callback usage would be as follows:
214
215src/app-layer-dcerpc.c:
216
217.. code-block:: c
218
219 AppLayerParserRegisterStateProgressCompletionStatus(ALPROTO_DCERPC, 1, 1);
220
221src/app-layer-ftp.c:
222
223.. code-block:: c
224
225 AppLayerParserRegisterStateProgressCompletionStatus(
226 ALPROTO_FTP, FTP_STATE_FINISHED, FTP_STATE_FINISHED);
227
228Sequence Diagrams
229~~~~~~~~~~~~~~~~~
230
231A DNS transaction in Suricata can be considered unidirectional:
232
2cd25e81 233.. image:: diagrams/DnsUnidirectionalTransactions.png
a5b344e0
JF
234 :width: 600
235 :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server, labeled "DNS Request". After that, there is a dotted line labeled "Transaction Completed".
236
dbeb8bfa 237An HTTP2 transaction is an example of a bidirectional transaction, in Suricata (note that, while HTTP2 may have multiple streams, those are mapped to transactions in Suricata. They run in parallel, scenario not shown in this Sequence Diagram - which shows one transaction, only):
a5b344e0 238
2cd25e81 239.. image:: diagrams/HTTP2BidirectionalTransaction.png
a5b344e0
JF
240 :width: 600
241 :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server labeled "Request" and below that an arrow going from Server to Client labeled "Response". Below those arrows, a dotted line indicates that the transaction is completed.
242
243A TLS Handshake is a more complex example, where several messages are exchanged before the transaction is considered completed:
244
2cd25e81 245.. image:: diagrams/TlsHandshake.png
a5b344e0
JF
246 :width: 600
247 :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server labeled "ClientHello" and below that an arrow going from Server to Client labeled "ServerHello". Below those arrows, several more follow from Server to Client and vice-versa, before a dotted line indicates that the transaction is finally completed.
248
249Template Protocol
250~~~~~~~~~~~~~~~~~
251
252Suricata has a template protocol for educational purposes, which has simple bidirectional transactions.
253
254A completed transaction for the template looks like this:
255
2cd25e81 256.. image:: diagrams/TemplateTransaction.png
a5b344e0
JF
257 :width: 600
258 :alt: A sequence diagram with two entities, Client and Server, with an arrow going from the Client to the Server, labeled "Request". An arrow below that first one goes from Server to Client.
259
260Following are the functions that check whether a transaction is considered completed, for the Template Protocol. Those are called by the Suricata API. Similar functions exist for each protocol, and may present implementation differences, based on what is considered a transaction for that given protocol.
261
262In C:
263
264.. code-block:: c
265
266 static int TemplateGetStateProgress(void *txv, uint8_t direction)
267 {
268 TemplateTransaction *tx = txv;
269
270 SCLogNotice("Transaction progress requested for tx ID %"PRIu64
271 ", direction=0x%02x", tx->tx_id, direction);
272
273 if (direction & STREAM_TOCLIENT && tx->response_done) {
274 return 1;
275 }
276 else if (direction & STREAM_TOSERVER) {
277 /* For the template, just the existence of the transaction means the
278 * request is done. */
279 return 1;
280 }
281
282 return 0;
283 }
284
285And in Rust:
286
287.. code-block:: rust
288
289 pub extern "C" fn rs_template_tx_get_alstate_progress(
290 tx: *mut std::os::raw::c_void,
291 _direction: u8,
292 ) -> std::os::raw::c_int {
293 let tx = cast_pointer!(tx, TemplateTransaction);
294
295 // Transaction is done if we have a response.
296 if tx.response.is_some() {
297 return 1;
298 }
299 return 0;
300 }
301
302Work In Progress changes
303========================
304
305Currently we are working to have files be part of the transaction instead of the per-flow state, as seen in https://redmine.openinfosecfoundation.org/issues/4444.
306
84311ab1 307Another work in progress is to limit the number of transactions per flow, to prevent Denial of Service (DoS) by quadratic complexity - a type of attack that may happen to protocols which can have multiple transactions at the same time - such as HTTP2 so-called streams (see https://redmine.openinfosecfoundation.org/issues/4530).
a5b344e0
JF
308
309Common words and abbreviations
310==============================
311
312- al, applayer: application layer
313- alproto: application layer protocol
314- alstate: application layer state
315- engine: refers to Suricata core detection logic
316- flow: a bidirectional flow of packets with the same 5-tuple elements (protocol, source ip, destination ip, source port, destination port. Vlans can be added as well)
317- PDU: Protocol Data Unit
318- rs: rust
319- tc: to client
320- ts: to server
321- tx: transaction