doc/devguide/codebase/testing.rst

   1 ****************
   2 Testing Suricata
   3 ****************
   4
   5 .. contents:: Table of Contents
   6
   7 General Concepts
   8 ================
   9
  10 There are a few ways of testing Suricata:
  11
  12 - **Unit tests**: for independently checking specific functions or portions of code. This guide has specific sections to
  13   further explain those, for C and Rust;
  14 - `Suricata-Verify <https://github.com/OISF/suricata-verify>`_: those are used to check more complex behavior, like the log output or the alert counts for a given input, where that input is usually comprised of several packets;
  15 - **Static and dynamic analysis tools**: to help in finding bugs, memory leaks and other issues (like     `scan-build <https://clang-analyzer.llvm.org/scan-build.html#scanbuild_basicusage>`_, from `clang`,     which is also used for our C formatting checks; or ASAN, which checks for memory issues);
  16 - **Fuzz testing**: especially good for uncovering existing, often non-trivial bugs. For more on how to fuzz test Suricata, check :doc:`fuzz-testing`;
  17 - **CI checks**: each PR submitted to the project's public repositories will be run against a suit of Continuous Integration
  18   workflows, as part of our QA process. Those cover: formatting and commit checks; fuzz tests (CI Fuzz), and several  builds. See our `github workflows <https://github.com/OISF/suricata/tree/master/.github/workflows>`_ for details and those in
  19   action at `<https://github.com/OISF/suricata/actions>`_.
  20
  21   .. note:: If you can run unit tests or other checks and report failures in our `issue tracker <https://redmine.openinfosecfoundation.org/projects/suricata/issues>`_, that is rather useful and appreciated!
  22
  23 The focus of this document are Unit tests and Suricata-Verify tests, especially on offering some guidance regarding when to use each type of test, and how to prepare input
  24 for them.
  25
  26 Unit tests
  27 ==========
  28
  29 Use these to check that specific functions behave as expected, in success and in failure scenarios. Specially useful
  30 during development, for nom parsers in the Rust codebase, for instance, or for checking that messages
  31 or message parts of a protocol/stream are processed as they should.
  32
  33 To execute all unit tests (both from C and Rust code), as well as ``libhtp`` ones, from the Suricata main directory, run::
  34
  35     make check
  36
  37 .. Check the Suricata Devguide on :doc:`unittests-c` or :doc:`unittests-rust` for more on how to write and run unit tests,
  38 .. considering that the way to do so differs considerably in Suricata, based on the language.
  39
  40 Code Examples
  41 ^^^^^^^^^^^^^
  42
  43 An example from the `DNS parser <https://github.com/OISF/suricata/blob/master/rust/src/dns/parser.rs#L417>`_. This
  44 checks that the given raw input (note the comments indicating what it means), once processed by ``dns_parse_name`` yields
  45 the expected result, including the unparsed portion.
  46
  47 .. code-block:: rust
  48
  49     /// Parse a simple name with no pointers.
  50     #[test]
  51     fn test_dns_parse_name() {
  52         let buf: &[u8] = &[
  53                                                 0x09, 0x63, /* .......c */
  54             0x6c, 0x69, 0x65, 0x6e, 0x74, 0x2d, 0x63, 0x66, /* lient-cf */
  55             0x07, 0x64, 0x72, 0x6f, 0x70, 0x62, 0x6f, 0x78, /* .dropbox */
  56             0x03, 0x63, 0x6f, 0x6d, 0x00, 0x00, 0x01, 0x00, /* .com.... */
  57         ];
  58         let expected_remainder: &[u8] = &[0x00, 0x01, 0x00];
  59         let (remainder,name) = dns_parse_name(buf, buf).unwrap();
  60         assert_eq!("client-cf.dropbox.com".as_bytes(), &name[..]);
  61         assert_eq!(remainder, expected_remainder);
  62     }
  63
  64 From the C side, ``decode-ethernet.c`` offers an good example:
  65
  66 .. code-block:: c
  67
  68     /**
  69      * Test a DCE ethernet frame that is too small.
  70      */
  71     static int DecodeEthernetTestDceTooSmall(void)
  72     {
  73         uint8_t raw_eth[] = {
  74             0x00, 0x10, 0x94, 0x55, 0x00, 0x01, 0x00, 0x10,
  75             0x94, 0x56, 0x00, 0x01, 0x89, 0x03,
  76         };
  77
  78         Packet *p = SCMalloc(SIZE_OF_PACKET);
  79         FAIL_IF_NULL(p);
  80         ThreadVars tv;
  81         DecodeThreadVars dtv;
  82
  83         memset(&dtv, 0, sizeof(DecodeThreadVars));
  84         memset(&tv,  0, sizeof(ThreadVars));
  85         memset(p, 0, SIZE_OF_PACKET);
  86
  87         DecodeEthernet(&tv, &dtv, p, raw_eth, sizeof(raw_eth));
  88
  89         FAIL_IF_NOT(ENGINE_ISSET_EVENT(p, DCE_PKT_TOO_SMALL));
  90
  91         SCFree(p);
  92         PASS;
  93     }
  94
  95 Suricata-Verify
  96 ===============
  97
  98 As mentioned above, these tests are used to check more complex behavior that involve a complete flow, with exchange of requests and responses. This can be done in an easier and more straightforward way,
  99 since one doesn't have to simulate the network traffic and Suricata engine mechanics - one simply runs it, with the desired input packet capture,
 100 configuration and checks.
 101
 102 A Suricata-verify test can help to ensure that code refactoring doesn't affect protocol logs, or signature detection,
 103 for instance, as this could have a major impact to Suricata users and integrators.
 104
 105 For simpler tests, providing the pcap input is enough. But it is also possible to provide Suricata rules to be
 106 inspected, and have Suricata Verify match for alerts and specific events.
 107
 108 Refer to the `Suricata Verify readme <https://github.com/OISF/suricata-verify#readme>`_ for details on how to create
 109 this type of test. It suffices to have a packet capture representative of the behavior one wants to test, and then
 110 follow the steps described there.
 111
 112 The Git repository for the Suricata Verify tests is a great source for examples, like the `app-layer-template <https://github.com/OISF/suricata-verify/tree/master/tests/app-layer-template>`_ one.
 113
 114 Generating Input
 115 ================
 116
 117 Using real traffic
 118 ^^^^^^^^^^^^^^^^^^
 119
 120 Having a packet capture for the desired protocol you want to test, open it in `Wireshark <https://www.wireshark.org/>`_, and select the specific
 121 packet chosen for the test input, then use the Wireshark option ``Follow [TCP/UDP/HTTP/HTTP2/QUIC] Stream``. This allows for inspecting the whole network traffic stream in a different window.
 122 There, it's possible to choose to ``Show and save data as`` ``C Arrays``, as well as to select if one wants to see the whole conversation or just **client** or **server** packets.
 123 It is also possible to reach the same effect by accessing the **Analyze->Follow->TCP Stream** top menu in Wireshark.
 124 (There are other stream options, the available one will depend on the type of network traffic captured).
 125
 126 This option will show the packet data as hexadecimal compatible with C-array style, and easily adapted for Rust,
 127 as well. As shown in the image:
 128
 129 .. image:: img/InputCaptureExample.png
 130
 131 Wireshark can be also used to `capture sample network traffic <https://gitlab.com/wireshark/wireshark/-/wikis/CaptureSetup>`_ and generate pcap files.
 132
 133 Crafting input samples with Scapy
 134 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 135
 136 It is also possible to use Scapy to create specific traffic: `Scapy usage
 137 <https://scapy.readthedocs.io/en/latest/usage.html>`_
 138
 139 Suricata-verify tests have several examples of pcaps generated in such a way. Look for Python scripts like the one used
 140 for the `dce-udp-scapy
 141 <https://github.com/OISF/suricata-verify/blob/master/tests/dcerpc/dcerpc-udp-scapy/dcerpc_udp_scapy.py>`_.
 142
 143 Other examples from our Suricata-Verify tests:
 144 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 145
 146 Going through Suricata-Verify tests `readme` files it is also possible to find an assorted collection of pcap generation possibilities, some with explanation on the how-tos. To list a few:
 147
 148 - `http2-range <https://github.com/OISF/suricata-verify/blob/master/tests/http2-range/README.md>`_
 149 - `http-range <https://github.com/OISF/suricata-verify/blob/master/tests/http-range/README.md>`_
 150 - `smb2-delete <https://github.com/OISF/suricata-verify/blob/master/tests/smb2-delete/README.md>`_
 151 - `smtp-rset <https://github.com/OISF/suricata-verify/blob/master/tests/smtp-rset/README.md>`_
 152 - `http-auth-unrecognized <https://github.com/OISF/suricata-verify/blob/master/tests/http-auth-unrecognized/README.md>`_
 153
 154 Finding Capture Samples
 155 ^^^^^^^^^^^^^^^^^^^^^^^
 156
 157 If you can't capture traffic for the desired protocol from live traffic, or craft something up, you can try finding the type of traffic you
 158 are interested in in public data sets. There's a thread for `Sharing good sources of sample captures
 159 <https://forum.suricata.io/t/sharing-good-sources-of-sample-captures/1766/4>`_ in our forum.