From cefd1d46d78d56e465f7475f8a483084f02cf902 Mon Sep 17 00:00:00 2001 From: "Mike Stepanek (mstepane)" Date: Wed, 17 Jul 2019 10:39:36 -0400 Subject: [PATCH] Merge pull request #1679 in SNORT/snort3 from ~MSTEPANE/snort3:build_258 to master Squashed commit of the following: commit b96f253230c9b1e079a6172cfd4648e64d9b2091 Author: Mike Stepanek Date: Wed Jul 17 09:02:44 2019 -0400 build: 258 --- ChangeLog | 39 ++++++ doc/snort_manual.html | 278 ++++++++++++++++++++++++++++++++---------- doc/snort_manual.pdf | Bin 822496 -> 823527 bytes doc/snort_manual.text | 208 +++++++++++++++++-------------- src/main/build.h | 2 +- 5 files changed, 370 insertions(+), 157 deletions(-) diff --git a/ChangeLog b/ChangeLog index 5ae008865..09b8d8a6e 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,42 @@ +19/07/17 - build 258 + +-- analyzer: 1024 contexts max is a better default until configurable +-- appid: fix header order in appid_session +-- codec: add support of ignore_vlan flag from daq header +-- detection: allocate scratch after configuration +-- detection: immediately onload after offloading when running regression tests +-- detection: on PDUs change search order to set check_ports correctly +-- detection: reduce hard number of contexts to work with pcap default +-- detection: start offload threads before packet threads are pinned +-- detection: use offload_threads = N with -z = 1 +-- flow: Extend stash to support uint32_t and make it SO_PUBLIC +-- flow: Fixes for DAQ-backed HA implementation +-- flow: remove config.h from flow_stash_keys +-- high_availability: high availability support in Snort2Lua +-- host_cache: Adding command and config option to dump hosts +-- host_cache: Closing va_list after usage using va_end +-- http2: decode HPACK uint +-- http2: hpack string decode +-- http_inspect: perf improvements +-- http_inspect: send headers to detection separately +-- ips: add missing non-fast-pattern warning +-- ips: refactor fast pattern searching +-- mpse: api init and print methods are optional +-- no_ack: Purge segment list withouth waiting for ack when using no_ack feature. +-- pcre: cap the pcre_match_limit_recursion based on the stack size available. +-- profiler: convert ips options to use optional profiles +-- profiler: eliminate deep profiling +-- profiler: implement general exclusion +-- profiler: include onload/offload efforts in mpse +-- profiler: refactor +-- profiler: split out paf from stream_tcp +-- profiler: track DAQ message receives and finalizes +-- snort: remove out-of-date Snort 2 version from -V +-- stream: add convenient method for flow deletion +-- stream_tcp: Add no-ack policy to handle flows that have no ACKs for data. +-- stream_tcp: fix non-deep detect profile exclusion +-- talos.lua: various fixes for command line usage + 19/06/19 - build 257 -- analyzer: publish finalize packet event before calling finalize_message. diff --git a/doc/snort_manual.html b/doc/snort_manual.html index ca5ace218..4cf7ccb0c 100644 --- a/doc/snort_manual.html +++ b/doc/snort_manual.html @@ -782,7 +782,7 @@ asciidoc.install(2); 
 ,,_     -*> Snort++ <*-
-o"  )~   Version 3.0.0 (Build 257) from 2.9.11
+o"  )~   Version 3.0.0 (Build 258)
  ''''    By Martin Roesch & The Snort Team
          http://snort.org/contact#team
          Copyright (C) 2014-2019 Cisco and/or its affiliates. All rights reserved.
@@ -4693,17 +4693,17 @@ Delete - A flow has been removed from the cache
 {
     ports = "1",
     enable = true,
-    min_age = 0.0,
-    min_sync = 0.0
+    min_age = 0,
+    min_sync = 0
 }

The ports item maps to the SideChannel port to use for the HA messaging.

The enabled item controls the overall HA operation.

The items min_age and min_sync are used in the stream HA logic. min_age is -the number of seconds that a flow must exist in the flow cache before sending +the number of milliseconds that a flow must exist in the flow cache before sending HA messages to the partner. min_sync is the minimum time between HA status updates. HA messages for a particular flow will not be sent faster than -min_sync. Both are expressed as a floating point number of seconds.

+min_sync. Both are expressed as a number of milliseconds.

HA messages are composed of the base stream information plus any content from additional modules. Modules subscribe HA in order to add message content. The stream HA content is always present in the messages while @@ -5673,36 +5673,15 @@ complicated about that, but suppose we use more than one rule option:

alert tcp any any -> any any ( msg:"combined example"; flow:established,
-to_server; http_uri; content:"chocolate"; file_data;
-content:"sinister POST data"; sid:5; rev:1; )
-
-

This rule requires both the URI and the request message body. That sounds -simple until one considers that the message body may be millions of bytes -long. The headers with the URI may be long gone by that time.

-

Is this rule going to work or do we need to do something different?

-

It is helpful to understand when things happen. All the message headers and -the first few thousand bytes of the body go through detection at the same -time. Commonly this is about 16K bytes but there are several exceptions and -there is no guaranteed minimum amount.

-

That may be all you need. In many cases that will be the entire message. Or -it may be more than your request_depth/response_depth. Or this rule may -simply not care what happens after that in a very long message body.

-

Beyond that the message body will continue to be subdivided into roughly -16K-byte sections and inspected. But the previous rule will not be able to -see the URI and hence will not work unless we rewrite it:

-
-
-
alert tcp any any -> any any ( msg:"URI with_body"; flow:established,
 to_server; http_uri: with_body; content:"chocolate"; file_data;
-content:"sinister POST data"; sid:5; rev:2; )
+content:"sinister POST data"; sid:5; rev:1; )

The with_body option to http_uri causes the URI to be made available with -every body section, not just the first one. These extra inspections have a -performance cost which is why they are not done automatically. with_body is -an option to be used when you actually need it.

+the message body. Use with_body for header-related rule options in rules +that also examine the message body.

The with_trailer option is analogous and causes an earlier message element -to be made available at the end of the message when the trailers following a -chunked body arrive.

+to be made available at the end of the message when the trailers following +a chunked body arrive.

alert tcp any any -> any any ( msg:"double content-language";
@@ -5759,18 +5738,15 @@ detection:

  1. -When the first part of the request message body arrives. The request -line, all of the headers, and the first part of the body all go through -detection at the same time. Of course most requests don’t have a body. In -that case the request line and the headers are the whole message and get -done at the same time. +When the the request headers arrive. The request line and all of the +headers go through detection at the same time.

  2. -When subsequent sections of the request message body arrive. If you want -to combine this with something from the request line or headers you must -use the with_body option. +When sections of the request message body arrive. If you want to combine +this with something from the request line or headers you must use the +with_body option.

  3. @@ -5782,18 +5758,17 @@ option.

  4. -When the first part of the response message body arrives. The status -line, all of the headers, and the first part of the body all go through -detection at the same time. These may be combined with elements from the -request line, request headers, or request trailers. Where ambiguity arises -use the request option. +When the response headers arrive. The status line and all of the headers +go through detection at the same time. These may be combined with elements +from the request line, request headers, or request trailers. Where +ambiguity arises use the request option.

  5. -When subsequent sections of the response message body arrive. These may -be combined with the status line, response headers, request line, request -headers, or request trailers as described above. +When sections of the response message body arrive. These may be combined +with the status line, response headers, request line, request headers, or +request trailers as described above.

  6. @@ -5803,8 +5778,8 @@ described above.

-

Message body data can only go through detection at the time it is received. -Headers may be combined with later items but the body cannot.

+

Message body sections can only go through detection at the time they are +received. Headers may be combined with later items but the body cannot.

@@ -7658,12 +7633,12 @@ bit_list high_availability.ports: side channel message port lis

  • -real high_availability.min_age = 1.0: minimum session life in seconds before HA updates { 0.0:100.0 } +int high_availability.min_age = 0: minimum session life in milliseconds before HA updates { 0:max32 }

  • -real high_availability.min_sync = 0.1: minimum interval in seconds between HA updates { 0.0:100.0 } +int high_availability.min_sync = 0: minimum interval in milliseconds between HA updates { 0:max32 }

    • @@ -7738,14 +7713,27 @@ real high_availability.min_sync = 0.1: minimum interval in seco

    host_cache

    -

    What: configure hosts

    +

    What: global LRU cache of host_tracker data about hosts

    Type: basic

    Usage: global

    Configuration:

    • -int host_cache[].size: size of host cache { 1:max32 } +string host_cache.dump_file: file name to dump host cache on shutdown; won’t dump by default +

      +
      • +
    • +

      +int host_cache.size: size of host cache { 1:max32 } +

      +
      • +
    +

    Commands:

    +
      +
    • +

      +host_cache.dump(file_name): dump host cache

    @@ -8874,7 +8862,7 @@ implied snort.-y: include year in timestamp in the alert and lo

  • -int snort.-z = 1: <count> maximum number of packet threads (same as --max-packet-threads); 0 gets the number of CPU cores reported by the system; default is 1 { 0:max32 } +int snort.-z: <count> maximum number of packet threads (same as --max-packet-threads); 0 gets the number of CPU cores reported by the system; default is 1 { 0:max32 }

  • @@ -9079,7 +9067,7 @@ implied snort.--markup: output help in asciidoc compatible form

  • -int snort.--max-packet-threads = 1: <count> configure maximum number of packet threads (same as -z) { 0:max32 } +int snort.--max-packet-threads: <count> configure maximum number of packet threads (same as -z) { 0:max32 }

  • @@ -9119,7 +9107,7 @@ string snort.--pcap-dir: <dir> a directory to recurse to

  • -string snort.--pcap-filter: <filter> filter to apply when getting pcaps from file or directory +string snort.--pcap-filter = .*cap: <filter> filter to apply when getting pcaps from file or directory

  • @@ -9219,7 +9207,7 @@ implied snort.--stdin-rules: read rules from stdin until EOF or

  • -implied snort.--talos: enable Talos inline rule test mode (same as --tweaks talos -Q -q) +implied snort.--talos: enable Talos tweak (same as --tweaks talos)

  • @@ -12840,6 +12828,23 @@ int gtp_inspect.trace: mask for enabling debug traces in module

    Type: inspector

    Usage: inspect

    Rules:

    +
      +
    • +

      +121:1 (http2_inspect) Error in HPACK integer value +

      +
      • +
    • +

      +121:2 (http2_inspect) Integer value has leading zeros +

      +
      • +
    • +

      +121:3 (http2_inspect) Error in HPACK string value +

      +
      • +

    Peg counts: