--- /dev/null
+Rule Processing
+===============
+
+.. toctree::
+
+Suricata rules have many elements that influence how they are processed by
+Suricata and matched against network traffic.
+
+This section explains some key aspects of how Suricata handles rules internally,
+so it can be easier to understand/predict how different rules may interact in
+specific scenarios.
+
+**This material is intended for:** rule writers; developers.
+
+Possible questions one should be better equipped to answer after reading this
+document:
+
+- What happens if two rules have the same priority value (the keyword)?
+- What type of rules will be evaluated first, given a set of rules?
+- How does Suricata decide what rule is "more important" when matching traffic?
+
+.. important:: Rules processing is also heavily affected by rule types, as mentioned
+ in this chapter. You may want to read more on :doc:`rule-types`.
+
+.. note::
+
+ Throughout this documentation and for Suricata, the terms "rule" and
+ "signature" are mainly interchangeable, unless context indicates otherwise.
+
+Overview
+--------
+
+Rules are provided to Suricata via rules files. Starting from those, the
+Detection Engine loader will:
+
+#. Load all the signatures;
+#. Check for validity (non-existing keywords; duplicated ``sid`` etc);
+#. Report stats on loaded, good and bad signatures;
+#. Sort all valid signatures and store them in a list in the Detect Engine
+ Context (``DetectEngineCtx``) structure, taking into consideration several
+ rule aspects according to their order of relevance to the Detection Engine;
+#. Attribute internal rule IDs that reflect this `rule prioritization`_;
+#. During inspection, match rules against the inspected traffic, according to rule
+ and traffic type.
+
+.. _rule prioritization:
+
+Rule Prioritization
+-------------------
+
+Suricata registers several different ordering functions (with
+``SCSigRegisterSignatureOrderingFuncs()``), which are then used to compare
+the rules, sort them, and define their priority. The elements taken into
+consideration for such are the signature's:
+
+#. :ref:`Action <actions>`
+#. Usage of :ref:`flowbits`
+#. Usage of :ref:`flowint`
+#. Usage of flowvar
+#. Usage of pktvar
+#. Usage of hostbits
+#. Usage of ippair
+#. :ref:`"Priority" keyword<priority>`
+
+In this order. Once signatures are ordered, they are attributed a unique
+internal ID (``Signature::iid``) which symbolizes their priority (the lower the
+``iid``, the higher the priority). This could mean that a rule with a
+keyword-defined priority of 1 could have lower priority than another rule that
+had flowbits set and a rule action with higher priority, for instance.
+
+.. note:: this list isn't fully comprehensive, in the sense that each item has
+ extra logic for prioritization. For example, considering flowbits, the
+ priority is write (highest) > write + read > read (lowest) > no flowbits.
+
+Another important element when considering rule parsing, processing and matching
+is that the ruleset is optimized into signature group heads based on the signature
+elements (thus, for instance, a TCP rule and an UDP rule would be loaded into
+different groups, and their internal ids will not interfere between one another,
+as they're matched against different traffic). For more on this, see
+:ref:`detection-engine`.
+
+Inspection Process
+------------------
+
+Once it is time to inspect network traffic against the loaded rules, the
+Detect Engine will match against - if applicable:
+
+#. IP Only rules;
+#. Packet/payload-related rules;
+#. Frame keywords;
+#. Application layer protocol transaction rules.
+
+During packet inspection, if the signature uses the last two in this list,
+inspection is left to those steps.
+
+.. tip::
+
+ With the introduction of :doc:`Firewall mode <../firewall/firewall-design>`,
+ it is possible to explicitly control to which step of the detection engine flow
+ a rule will be hooked. This is done with :ref:`Explicit rule hooks <rule-hooks>`.
+
+For each rule that is matched, a ``PacketAlert`` is created. After all matches
+for a packet have been processed, and the :ref:`alert queue limit <alert queue
+overflow impact>` is taken into account, the remaining ``PacketAlerts`` become
+the alerts in Suricata logs.
+
+Considerations on Inspection Steps
+----------------------------------
+
+IP Only rules
+~~~~~~~~~~~~~
+
+Without optimization, IP Only signatures would match on every packet on a flow.
+To improve performance, what Suricata does is to evaluate rules that are
+``ip-only`` only once per flow direction, for the first packet in each direction.
+
+Application layer protocol transactions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each parser has its own state machine, and uses a per-direction parsing state
+"progress". Keywords can be registered for each progress value. So, for
+instance, ``http`` has a value "request line available" for which there are
+keywords like :ref:`http.uri <rules-http-uri-normalization>`, :ref:`http.method
+<http.method>` etc. registered. While parsing the traffic, if the engine reaches
+this state, the signatures with those keywords may be already evaluated, even if
+they have a lower priority than an http body inspecting signature.
+
+Relatedly, a rule with two keywords matching at two different progress stages may
+be evaluated against two different packets.
+
+Implications
+------------
+
+Action precedence and interaction with ``ip-only`` rules
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To illustrate what may be counter-intuitive implications of how inspection
+steps, action prioritization and rule keywords interact and affect the engine
+behavior, we will use a real case example. Consider these three rules:
+
+.. container:: example-rule
+
+ pass tcp 0.0.0.0/0 any <> 0.0.0.0/0 443 (msg:"Allow TCP in port 443"; flow:
+ not_established; sid:1; rev:1;)
+
+.. container:: example-rule
+
+ pass tcp 0.0.0.0/0 any <> 0.0.0.0/0 80 (msg:"Allow TCP in port 80"; flow:
+ not_established; sid:2; rev:1;)
+
+.. container:: example-rule
+
+ drop ip 0.0.0.0/0 any -> 0.0.0.0/0 any (msg:"No outbound internet access
+ from host"; sid:3; rev:1;)
+
+The first two are signatures that analyze individual packets and match only if
+the flow has not been established (``flow:not_established``): the rules grant
+``PASS`` to the matched packet - but not to its flow.
+
+The third signature is considered ``ip-only``. This means it will be evaluated
+for the *first* packet in both directions of a flow, in addition to rules 1 and
+2. By extension, **the other packets in the same flow will not be evaluated
+against this rule.**
+
+With an action order configuration that prioritizes ``PASS`` over ``DROP``, this
+means that rules 1 and 2 will have a higher internal priority over rule 3,
+therefore nullifying the ``DROP`` outcome. The result: a flow for outbound
+internet traffic from the host, expected to be dropped, wouldn't be.
+
+If the expected behavior with those three signatures was to allow traffic on
+ports 80 and 443 only, while dropping everything else, the simplest way to
+achieve this would be to remove the ``flow:not_established`` portion from rules
+sid:1 and sid:2. This ensures that the ``PASS`` action would be applied to the
+whole flow following the match on the first packet and that all other traffic
+would be dropped.
+
+Following that, all three rules will be evaluated on the same step, and if a
+flow isn't flagged with ``pass``, it will be dropped with the third rule.
+
+.. Tip::
+ A more straightforward way to achieve that in Suricata 8 is using the firewall
+ more. See :doc:`../firewall/firewall-design`.
+
+Alerts not seen
+~~~~~~~~~~~~~~~
+
+Another aspect of rule prioritization combined with the alerts queue size is
+that, in corner case scenarios, if a packet matches against too many rules,
+signatures with lower priority could be discarded from the ``PacketAlert`` queue
+(see the section on :ref:`alert queue overflow impact <alert queue overflow impact>`
+for more).
+
+The stats counter ``detect.alert_queue_overflow`` will be higher than zero if
+an alert was discarded due to ``Alert Queue`` overflow (cf. :ref:`alerts stats`).
+
projects / thirdparty / suricata.git / commitdiff
