From: Mike Stepanek (mstepane) Date: Thu, 6 Aug 2020 12:15:50 +0000 (+0000) Subject: Merge pull request #2367 in SNORT/snort3 from ~MSTEPANE/snort3:3_0_2_build_4 to master X-Git-Tag: 3.0.2-4 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=fbb33198cb68874abda48e9892723f94c1c0baa3;p=thirdparty%2Fsnort3.git Merge pull request #2367 in SNORT/snort3 from ~MSTEPANE/snort3:3_0_2_build_4 to master Squashed commit of the following: commit 70e0c1d9a7e51c6f5edbd3b734bb9b68e36e8523 Author: Mike Stepanek Date: Fri Jul 31 08:12:24 2020 -0400 build: generate and tag 3.0.2 build 4 --- diff --git a/ChangeLog b/ChangeLog index d6b5c1367..24033760c 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,61 @@ +2020/08/06 - 3.0.2 build 4 + +-- appid: Clear service appid entries in dynamic host cache on ODP reload +-- appid: Generate event notification when dns host is set +-- dce_rpc: Fix for smb crash while tcp session pruning +-- dce_rpc: Fix for smb session cleanup issue +-- dce_rpc: Use file name hash as file id +-- doc: Add documentation for dumping consolidated config in text format +-- flow: Fixing free_flow_data logic +-- http_inspect: Code clean up +-- http_inspect: Test tool enhancement +-- main: Dump consolidated config in the text format +-- rna: Fix redefined macro warnings in between unit-test tools +-- rna: TCP fingerprint input and retrieval +-- utils: Keep deprecated attribute table pegcounts + +2020/07/28 - 3.0.2 build 3 + +-- active: Move Active enabled flag into SnortConfig +-- appid: For http traffic, if payload cannot be detected, set it to unknown +-- appid: Move appid data needed by external components to stash +-- appid: Support ODP reload for multiple packet threads and new session +-- dce_rpc: Improve PAF autodetection for heavily segmented TCP traffic +-- doc: Split Snort manual into separate user, reference, and upgrade docs. +-- doc: Update default text manuals +-- doc: Update extending.txt about TraceLogger plugin +-- file_api: Log event generated when lookup timedout +-- ftp_telnet: Remove global config variable shared between multiple threads to prevent data race +-- http2_inpsect: Fix interaction with tool tcpclose +-- http2_inspect: Fix stream_in_hi +-- http2_inspect: General code cleanup +-- http_inspect: Do partial inspections incrementally +-- http_inspect: Reduce memory used by partial inspections +-- main: Rename the config options to ignore flowbits and rules warnings +-- parser: Add support for variables with each ips policy +-- payload_injector: Add HTTP page translation +-- payload_injector: Extend utility to support HTTP/2 (no injection) +-- pub_sub: Added a method in HttpEvent to retrieve true client-ip address from HTTP header based + on priority +-- rna: Fingerprint reader class and lookup table for tcp fingerprints +-- snort_defaults: Remove the NOTIFY, SUBSCRIBE, and UPDATE HTTP methods +-- stream_tcp: Only perform paws validation on real packets, skip this on meta-ack packets +-- stream_tcp: When clearing a session during meta-ack processing pass a nullptr as the Packet* + parameter +-- target_based: Add mutex lock to ensure host service accesses are thread safe +-- target_based: Move host attribute peg counts from the process pegs to stats specific to host + attribute operations +-- target_based: Refactor host attribute to use the LruCacheShared data store class to support + thread safe access +-- target_based: Streamline host attribute table activate and swap logic on startup and reload +-- trace: Add support for extending TraceLogger as a passive inspector plugin +-- wizard: Abandon the wizard on UDP flows after the first packet +-- wizard: Abort the splitter once we've hit the max PDU size +-- wizard: Add peg counts for abandoned searches per protocol +-- wizard: Improve wizard tracing to indicate direction and abandonment +-- wizard: Properly terminate hex matching +-- wizard: Report spell and hex configuration errors and warnings + 2020/07/15 - 3.0.2 build 2 -- appid: Moving thread local ODP stuff to a new class diff --git a/doc/reference/snort_reference.text b/doc/reference/snort_reference.text index d4a510a4a..25ddaebee 100644 --- a/doc/reference/snort_reference.text +++ b/doc/reference/snort_reference.text @@ -8,7 +8,7 @@ Snort 3 Reference Manual The Snort Team Revision History -Revision 3.0.2 (Build 2) 2020-07-23 11:20:26 EDT TST +Revision 3.0.2 (Build 4) 2020-08-06 08:06:49 EDT TST --------------------------------------------------------------------- @@ -824,6 +824,21 @@ Configuration: * enum hosts[].services[].proto = tcp: IP protocol { tcp | udp } * port hosts[].services[].port: port number +Peg counts: + + * hosts.total_hosts: maximum number of entries in the host + attribute table (max) + * hosts.hosts_pruned: number of LRU hosts pruned due to configured + resource limits (sum) + * hosts.dynamic_host_adds: number of host additions after initial + host file load (sum) + * hosts.dynamic_service_adds: number of service additions after + initial host file load (sum) + * hosts.dynamic_service_updates: number of service updates after + initial host file load (sum) + * hosts.service_list_overflows: number of service additions that + failed due to configured resource limits (sum) + 2.14. inspection @@ -874,6 +889,7 @@ Configuration: rules too) * string ips.uuid = 00000000-0000-0000-0000-000000000000: IPS policy uuid + * string ips.variables.$var: IPS policy variable 2.16. latency @@ -892,7 +908,6 @@ Configuration: thresholding (usec) { 0:max53 } * bool latency.packet.fastpath = false: fastpath expensive packets (max_time exceeded) - * bool latency.packet.test_timeout = false: timeout on every packet * int latency.rule.max_time = 500: set timeout for rule evaluation (usec) { 0:max53 } * bool latency.rule.suspend = false: temporarily suspend expensive @@ -902,8 +917,6 @@ Configuration: * int latency.rule.max_suspend_time = 30000: set max time for suspending a rule (ms, 0 means permanently disable rule) { 0:max32 } - * bool latency.rule.test_timeout = false: timeout on every rule - evaluation Rules: @@ -1014,7 +1027,7 @@ Configuration: * bool output.verbose = false: be verbose (same as -v) * bool output.obfuscate = false: obfuscate the logged IP addresses (same as -O) - * bool output.wide_hex_dump = true: output 20 bytes per lines + * bool output.wide_hex_dump = false: output 20 bytes per lines instead of 16 when dumping buffers @@ -1080,6 +1093,8 @@ Peg counts: * payload_injector.http_injects: total number of http injections (sum) + * payload_injector.http2_injects: total number of http2 injections + (sum) 2.23. process @@ -1392,6 +1407,7 @@ Configuration: * implied snort.--dirty-pig: don’t flush packets on shutdown * string snort.--dump-builtin-rules: [] output stub rules for selected modules { (optional) } + * implied snort.--dump-config-text: dump config in text format * implied snort.--dump-dynamic-rules: output stub rules for all loaded rules libraries * string snort.--dump-defaults: [] output module @@ -1432,10 +1448,6 @@ Configuration: logdir instead of instance filename prefix * implied snort.--id-zero: use id prefix / subdirectory even with one packet thread - * implied snort.--ignore-warn-flowbits: ignore warnings about - flowbits that are checked but not set and vice-versa - * implied snort.--ignore-warn-rules: ignore warnings about - duplicate rules and rule parsing issues * string snort.--include-path: where to find Lua and rule included files; searched before current or config directories * implied snort.--list-buffers: output available inspection buffers @@ -1460,10 +1472,12 @@ Configuration: * implied snort.--nostamps: don’t include timestamps in log file names * implied snort.--nolock-pidfile: do not try to lock Snort PID file + * implied snort.--no-warn-flowbits: ignore warnings about flowbits + that are checked but not set and vice-versa + * implied snort.--no-warn-rules: ignore warnings about duplicate + rules and rule parsing issues * implied snort.--pause: wait for resume/quit command before processing packets/terminating - * int snort.--pause-after-n: pause after count packets { - 1:max53 } * string snort.--pcap-file: file that contains a list of pcaps to read - read mode is implied * string snort.--pcap-list: a space separated list of pcaps @@ -1479,7 +1493,6 @@ Configuration: * implied snort.--pcap-show: print a line saying what pcap is currently being read * implied snort.--pedantic: warnings are fatal - * implied snort.--piglet: enable piglet test harness mode * string snort.--plugin-path: a colon separated list of directories or plugin libraries * implied snort.--process-all-events: process all action groups @@ -1511,8 +1524,6 @@ Configuration: * implied snort.--treat-drop-as-ignore: use drop, block, and reset rules to ignore session traffic when not inline * string snort.--tweaks: tune configuration - * string snort.--catch-test: comma separated list of cat unit test - tags or all * implied snort.--version: show version number (same as -V) * implied snort.--warn-all: enable all warnings * implied snort.--warn-conf: warn about configuration issues @@ -1537,7 +1548,6 @@ Configuration: { 0x00:0xFF } * string snort.--x2s: output ASCII string for given byte code (see also --x2c) - * implied snort.--trace: turn on main loop debug trace Commands: @@ -1612,11 +1622,10 @@ Usage: global Configuration: - * int trace.modules.latency.all: enable all trace options { 0:255 } - * int trace.modules.snort.all: enable all trace options { 0:255 } - * int trace.modules.snort.main: enable main trace logging { 0:255 } - * int trace.modules.snort.inspector_manager: enable inspector - manager trace logging { 0:255 } + * int trace.modules.appid.all: enable all trace options { 0:255 } + * int trace.modules.dce_smb.all: enable all trace options { 0:255 } + * int trace.modules.dce_udp.all: enable all trace options { 0:255 } + * int trace.modules.decode.all: enable all trace options { 0:255 } * int trace.modules.detection.all: enable all trace options { 0:255 } * int trace.modules.detection.detect_engine: enable detection @@ -1635,18 +1644,19 @@ Configuration: logging { 0:255 } * int trace.modules.detection.tag: enable tag trace logging { 0:255 } + * int trace.modules.gtp_inspect.all: enable all trace options { + 0:255 } + * int trace.modules.latency.all: enable all trace options { 0:255 } + * int trace.modules.snort.all: enable all trace options { 0:255 } + * int trace.modules.snort.main: enable main trace logging { 0:255 } + * int trace.modules.snort.inspector_manager: enable inspector + manager trace logging { 0:255 } + * int trace.modules.stream.all: enable all trace options { 0:255 } * int trace.modules.stream_ip.all: enable all trace options { 0:255 } * int trace.modules.stream_user.all: enable all trace options { 0:255 } * int trace.modules.wizard.all: enable all trace options { 0:255 } - * int trace.modules.dce_smb.all: enable all trace options { 0:255 } - * int trace.modules.dce_udp.all: enable all trace options { 0:255 } - * int trace.modules.decode.all: enable all trace options { 0:255 } - * int trace.modules.stream.all: enable all trace options { 0:255 } - * int trace.modules.gtp_inspect.all: enable all trace options { - 0:255 } - * int trace.modules.appid.all: enable all trace options { 0:255 } * int trace.constraints.ip_proto: numerical IP protocol ID filter { 0:255 } * string trace.constraints.src_ip: source IP address filter @@ -2365,9 +2375,6 @@ Usage: context Configuration: - * int appid.first_decrypted_packet_debug = 0: the first packet of - an already decrypted SSL flow (debug single session only) { - 0:max32 } * int appid.memcap = 1048576: max size of the service cache before we start pruning the cache { 1024:maxSZ } * bool appid.log_stats = false: enable logging of appid statistics @@ -2389,8 +2396,6 @@ Configuration: on startup * bool appid.log_all_sessions = false: enable logging of all appid sessions - * bool appid.load_odp_detectors_in_ctrl = false: load odp detectors - in control thread Commands: @@ -2398,6 +2403,7 @@ Commands: enable appid debugging * appid.disable_debug(): disable appid debugging * appid.reload_third_party(): reload appid third-party module + * appid.reload_odp(): reload appid open detector package Peg counts: @@ -2413,6 +2419,10 @@ Peg counts: the service cache (sum) * appid.service_cache_removes: number of times an item was removed from the service cache (sum) + * appid.odp_reload_ignored_pkts: count of packets ignored after + open detector package is reloaded (sum) + * appid.tp_reload_ignored_pkts: count of packets ignored after + third-party module is reloaded (sum) 5.2. appid_listener @@ -3467,20 +3477,6 @@ Type: inspector Usage: inspect -Configuration: - - * bool http2_inspect.test_input = false: read HTTP/2 messages from - text file - * bool http2_inspect.test_output = false: print out HTTP section - data - * int http2_inspect.print_amount = 1200: number of characters to - print from a Field { 1:max53 } - * bool http2_inspect.print_hex = false: nonprinting characters - printed in [HH] format instead of using an asterisk - * bool http2_inspect.show_pegs = true: display peg counts with test - output - * bool http2_inspect.show_scan = false: display scanned segments - Rules: * 121:1 (http2_inspect) error in HPACK integer value @@ -3577,17 +3573,6 @@ Configuration: normalizing URIs * bool http_inspect.simplify_path = true: reduce URI directory path to simplest form - * bool http_inspect.test_input = false: read HTTP messages from - text file - * bool http_inspect.test_output = false: print out HTTP section - data - * int http_inspect.print_amount = 1200: number of characters to - print from a Field { 1:max53 } - * bool http_inspect.print_hex = false: nonprinting characters - printed in [HH] format instead of using an asterisk - * bool http_inspect.show_pegs = true: display peg counts with test - output - * bool http_inspect.show_scan = false: display scanned segments Rules: @@ -4785,8 +4770,6 @@ Usage: global Configuration: - * int stream.footprint = 0: use zero for production, non-zero for - testing at given size (for TCP and user) { 0:max32 } * bool stream.ip_frags_only = false: don’t process non-frag flows * int stream.max_flows = 476288: maximum simultaneous flows tracked before pruning { 2:max32 } @@ -7805,6 +7788,7 @@ these libraries see the Getting Started section of the manual. * --dirty-pig don’t flush packets on shutdown * --dump-builtin-rules [] output stub rules for selected modules (optional) + * --dump-config-text dump config in text format * --dump-dynamic-rules output stub rules for all loaded rules libraries * --dump-defaults [] output module defaults in Lua @@ -7840,10 +7824,6 @@ these libraries see the Getting Started section of the manual. of instance filename prefix * --id-zero use id prefix / subdirectory even with one packet thread - * --ignore-warn-flowbits ignore warnings about flowbits that are - checked but not set and vice-versa - * --ignore-warn-rules ignore warnings about duplicate rules and - rule parsing issues * --include-path where to find Lua and rule included files; searched before current or config directories * --list-buffers output available inspection buffers @@ -7865,9 +7845,12 @@ these libraries see the Getting Started section of the manual. string in metadata if set * --nostamps don’t include timestamps in log file names * --nolock-pidfile do not try to lock Snort PID file + * --no-warn-flowbits ignore warnings about flowbits that are + checked but not set and vice-versa + * --no-warn-rules ignore warnings about duplicate rules and rule + parsing issues * --pause wait for resume/quit command before processing packets/ terminating - * --pause-after-n pause after count packets (1:max53) * --pcap-file file that contains a list of pcaps to read - read mode is implied * --pcap-list a space separated list of pcaps to read - read @@ -7882,7 +7865,6 @@ these libraries see the Getting Started section of the manual. file or directory * --pcap-show print a line saying what pcap is currently being read * --pedantic warnings are fatal - * --piglet enable piglet test harness mode * --plugin-path a colon separated list of directories or plugin libraries * --process-all-events process all action groups @@ -7911,7 +7893,6 @@ these libraries see the Getting Started section of the manual. * --treat-drop-as-ignore use drop, block, and reset rules to ignore session traffic when not inline * --tweaks tune configuration - * --catch-test comma separated list of cat unit test tags or all * --version show version number (same as -V) * --warn-all enable all warnings * --warn-conf warn about configuration issues @@ -7931,7 +7912,6 @@ these libraries see the Getting Started section of the manual. * --x2c output ASCII char for given hex (see also --c2x) (0x00:0xFF) * --x2s output ASCII string for given byte code (see also --x2c) - * --trace turn on main loop debug trace 11.4. Configuration @@ -8029,13 +8009,8 @@ these libraries see the Getting Started section of the manual. logging appid statistics { 1:max32 } * int appid.app_stats_rollover_size = 20971520: max file size for appid stats before rolling over the log file { 0:max32 } - * int appid.first_decrypted_packet_debug = 0: the first packet of - an already decrypted SSL flow (debug single session only) { - 0:max32 } * bool appid.list_odp_detectors = false: enable logging of odp detectors statistics - * bool appid.load_odp_detectors_in_ctrl = false: load odp detectors - in control thread * bool appid.log_all_sessions = false: enable logging of all appid sessions * bool appid.log_stats = false: enable logging of appid statistics @@ -8558,17 +8533,6 @@ these libraries see the Getting Started section of the manual. * port host_tracker[].services[].port: port number * enum host_tracker[].services[].proto: IP protocol { ip | tcp | udp } - * int http2_inspect.print_amount = 1200: number of characters to - print from a Field { 1:max53 } - * bool http2_inspect.print_hex = false: nonprinting characters - printed in [HH] format instead of using an asterisk - * bool http2_inspect.show_pegs = true: display peg counts with test - output - * bool http2_inspect.show_scan = false: display scanned segments - * bool http2_inspect.test_input = false: read HTTP/2 messages from - text file - * bool http2_inspect.test_output = false: print out HTTP section - data * implied http_cookie.request: match against the cookie from the request message even when examining the response * implied http_cookie.with_body: parts of this rule examine HTTP @@ -8624,23 +8588,12 @@ these libraries see the Getting Started section of the manual. encodings * bool http_inspect.plus_to_space = true: replace + with when normalizing URIs - * int http_inspect.print_amount = 1200: number of characters to - print from a Field { 1:max53 } - * bool http_inspect.print_hex = false: nonprinting characters - printed in [HH] format instead of using an asterisk * int http_inspect.request_depth = -1: maximum request message body bytes to examine (-1 no limit) { -1:max53 } * int http_inspect.response_depth = -1: maximum response message body bytes to examine (-1 no limit) { -1:max53 } - * bool http_inspect.show_pegs = true: display peg counts with test - output - * bool http_inspect.show_scan = false: display scanned segments * bool http_inspect.simplify_path = true: reduce URI directory path to simplest form - * bool http_inspect.test_input = false: read HTTP messages from - text file - * bool http_inspect.test_output = false: print out HTTP section - data * bool http_inspect.unzip = true: decompress gzip and deflate message bodies * bool http_inspect.utf8_bare_byte = false: when doing UTF-8 @@ -8793,6 +8746,7 @@ these libraries see the Getting Started section of the manual. rules too) * string ips.uuid = 00000000-0000-0000-0000-000000000000: IPS policy uuid + * string ips.variables.$var: IPS policy variable * string isdataat.~length: num | !num * implied isdataat.relative: offset from cursor instead of start of buffer @@ -8802,7 +8756,6 @@ these libraries see the Getting Started section of the manual. (max_time exceeded) * int latency.packet.max_time = 500: set timeout for packet latency thresholding (usec) { 0:max53 } - * bool latency.packet.test_timeout = false: timeout on every packet * int latency.rule.max_suspend_time = 30000: set max time for suspending a rule (ms, 0 means permanently disable rule) { 0:max32 } @@ -8812,8 +8765,6 @@ these libraries see the Getting Started section of the manual. rules * int latency.rule.suspend_threshold = 5: set threshold for number of timeouts before suspending a rule { 1:max32 } - * bool latency.rule.test_timeout = false: timeout on every rule - evaluation * bool log_codecs.file = false: output to log_codecs.txt instead of stdout * bool log_codecs.msg = false: include alert msg @@ -8931,7 +8882,7 @@ these libraries see the Getting Started section of the manual. * int output.tagged_packet_limit = 256: maximum number of packets tagged for non-packet metrics { 0:max32 } * bool output.verbose = false: be verbose (same as -v) - * bool output.wide_hex_dump = true: output 20 bytes per lines + * bool output.wide_hex_dump = false: output 20 bytes per lines instead of 16 when dumping buffers * bool packet_capture.enable = false: initially enable packet dumping @@ -9353,8 +9304,6 @@ these libraries see the Getting Started section of the manual. * string snort.--bpf: are standard BPF options, as seen in TCPDump * string snort.--c2x: output hex for given char (see also --x2c) - * string snort.--catch-test: comma separated list of cat unit test - tags or all * string snort.-c: use this configuration * string snort.--control-socket: to create unix socket * implied snort.-C: print out payloads with character data only (no @@ -9378,6 +9327,7 @@ these libraries see the Getting Started section of the manual. * implied snort.-D: run Snort in background (daemon) mode * string snort.--dump-builtin-rules: [] output stub rules for selected modules { (optional) } + * implied snort.--dump-config-text: dump config in text format * string snort.--dump-defaults: [] output module defaults in Lua format { (optional) } * implied snort.--dump-dynamic-rules: output stub rules for all @@ -9424,10 +9374,6 @@ these libraries see the Getting Started section of the manual. logdir instead of instance filename prefix * implied snort.--id-zero: use id prefix / subdirectory even with one packet thread - * implied snort.--ignore-warn-flowbits: ignore warnings about - flowbits that are checked but not set and vice-versa - * implied snort.--ignore-warn-rules: ignore warnings about - duplicate rules and rule parsing issues * string snort.-i: … list of interfaces * string snort.--include-path: where to find Lua and rule included files; searched before current or config directories @@ -9463,11 +9409,13 @@ these libraries see the Getting Started section of the manual. * implied snort.--nolock-pidfile: do not try to lock Snort PID file * implied snort.--nostamps: don’t include timestamps in log file names + * implied snort.--no-warn-flowbits: ignore warnings about flowbits + that are checked but not set and vice-versa + * implied snort.--no-warn-rules: ignore warnings about duplicate + rules and rule parsing issues * implied snort.-O: obfuscate the logged IP addresses * string snort.-?: