From: Michael Altizer (mialtize) Date: Sun, 13 Sep 2020 19:40:51 +0000 (+0000) Subject: Merge pull request #2472 in SNORT/snort3 from ~MIALTIZE/snort3:3_0_2_build_6 to master X-Git-Tag: 3.0.2-6 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=b9373db13b2b80d2b547b5d7a93f211b617a9def;p=thirdparty%2Fsnort3.git Merge pull request #2472 in SNORT/snort3 from ~MIALTIZE/snort3:3_0_2_build_6 to master Squashed commit of the following: commit fc525e0d3bcaf819080ecc9959fc0f4698052135 Author: Michael Altizer Date: Sun Sep 13 14:44:11 2020 -0400 build: Generate and tag 3.0.2 build 6 --- diff --git a/ChangeLog b/ChangeLog index 176552bdb..19f4911b1 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,87 @@ +2020/09/13 - 3.0.2 build 6 + +-- active: Remove per packet prevent trust action +-- appid: Add check for nullptr before setting tls host +-- appid: Clear services set in host attribute table upon detector reload +-- appid: Detect SMTP after decryption +-- appid: Dump user appid configuration on reload detectors +-- appid: Generate events for service info changes +-- appid: Pass snort protocol id instead of appid while creating future flow +-- appid: Reorder third-party reload to keep only one handle open at a time +-- appid: Send swap response for reload_odp and reload_third_party commands in control thread +-- appid: Set payload to unknown for out-of-order flows +-- appid: Skip detection for existing sessions after detector reload; rename reload_odp command to + reload_detectors +-- appid: Support json logging in appid_listener +-- appid: Update appid stats for decrypted flows +-- appid: Update appid warning messages to print module name in lowercase +-- build: Fix minor cppcheck warnings +-- build: Updates for libdaq changes to interface group field width and naming +-- byte_jump: Fix jump relative to extracted length w/o relative offset +-- cmake: Restore accidentally removed caching of static DAQ modules +-- dce_rpc: Introduce smb2 logs +-- doc: Update the config dump in JSON format (all policies) +-- doc: Update the config dump in JSON format (main policy) +-- doc: Update trace.txt with info about 'trace.modules.all' option +-- dump_config: Add --dump-config="top" to dump the main policy config only +-- dump_config: Dump config in JSON format to stdout +-- file_api: Increase default max_files_per_flow limit to 128 +-- flow: Add a deferred trust class to allow plugins to defer trusting sessions +-- flow: Disabled inspection for FlowState::RESET +-- flow: Reset the flow before removing +-- helpers: Add unit tests for special characters escaping +-- helpers: Fix build on systems without sigaction +-- helpers: Rework DiscoveryFilter to monitor IP lists based on interface rather than group +-- helpers: Use sig_t instead of sighandler_t for better BSD compatibility +-- host_tracker: Fix allocator unit test to work on 32-bit systems again +-- http2_inspect: Convert circular_array to std:vector +-- http2_inspect: Fix continuation frame check +-- http2_inspect: Fix hpack dynamic table init +-- http2_inspect: Prepare http2_inspect and http_inspect for HTTP/2 trailers +-- http2_inspect: Refactor hpack decoding and send trailer to http_inspect for processing +-- http_inspect: Declare get_type_expected const +-- http_inspect: Don't use the URL to cache file verdicts for uploads +-- http_inspect: Script detection +-- http_inspect: Script detection and concurrency fixes +-- http_inspect: Support hyperscan literal search for accelerated blocking +-- http_method: Make available for fast pattern with first body section +-- imap: Publish OPPORTUNISTIC_TLS_EVENT on successfull completion on START_TLS, add a new state to + avoid publishing start_tls events multiple times +-- ips_options: Ensure all options use base class hash and compare methods +-- ips: Use the policies in the flow when creating pseudo packet +-- main: Turn off signal handlers later to catch more during snort shutdown +-- managers: Immediately stop executing inspectors when inspection is disabled +-- mime: Fix off-by-1 error with filename and email id capture +-- mime: Minor code cleanup +-- netflow: Introduce netflow as a service inspector +-- packet_io: Added reason for ActiveStatus WOULD +-- packet_io: Do not allow trust unless the action is allow or trust +-- payload_injector: Assume http1, if packet does not have a gadget +-- payload_injector: Fix warning +-- payload_injector: Support http2 injection +-- payload_injector: Support translation of header field value with length > 127 +-- perf_monitor: Convert the perf_monitor inspector configure warnings to errors +-- pop: Publish start_tls events, support for ssl search abandoned +-- reputation: Change from group-based to interface-based IP lists +-- rna: Add protocols on logging host trackers +-- rna: Implement update_timeout for MAC hosts +-- rna: Remove dependency on uuid library +-- rna: Remove redefinition of USHRT_MAX +-- rna: Removing unused command and exporting swapper +-- rna: Support client discovery from appid event changes +-- rna: Support service discovery from appid event changes +-- rna: Tcp fingerprints configuration, storage, matching and event generation +-- snort2lua: Remove obsolete and unused code +-- snort2lua: Remove unused unit test files +-- snort: Address fatal shutdown stability issues +-- stream_ip: Fix zero fragment built-in rule triggering for some reassembly policies +-- style: Replace some tabs that snuck in with proper spaces +-- tests: Fix the majority of memory leaks in CppUTest unit tests +-- trace: Add support for modules.all option +-- trace: Update loggers to support extended output with n-tuple packet info +-- utils: Add sys/time.h to util.h for struct timeval definition +-- wizard: Fix the error message about invalid pattern + 2020/08/12 - 3.0.2 build 5 -- cip: Fix the trailing parameter for the module diff --git a/doc/reference/snort_reference.text b/doc/reference/snort_reference.text index 73150c1a2..8ee5eef2e 100644 --- a/doc/reference/snort_reference.text +++ b/doc/reference/snort_reference.text @@ -8,7 +8,7 @@ Snort 3 Reference Manual The Snort Team Revision History -Revision 3.0.2 (Build 5) 2020-08-12 08:28:30 EDT TST +Revision 3.0.2 (Build 6) 2020-09-13 14:48:12 EDT TST --------------------------------------------------------------------- @@ -113,30 +113,31 @@ Table of Contents 5.25. imap 5.26. mem_test 5.27. modbus - 5.28. normalizer - 5.29. null_trace_logger - 5.30. packet_capture - 5.31. perf_monitor - 5.32. pop - 5.33. port_scan - 5.34. reputation - 5.35. rna - 5.36. rpc_decode - 5.37. s7commplus - 5.38. sip - 5.39. smtp - 5.40. so_proxy - 5.41. ssh - 5.42. ssl - 5.43. stream - 5.44. stream_file - 5.45. stream_icmp - 5.46. stream_ip - 5.47. stream_tcp - 5.48. stream_udp - 5.49. stream_user - 5.50. telnet - 5.51. wizard + 5.28. netflow + 5.29. normalizer + 5.30. null_trace_logger + 5.31. packet_capture + 5.32. perf_monitor + 5.33. pop + 5.34. port_scan + 5.35. reputation + 5.36. rna + 5.37. rpc_decode + 5.38. s7commplus + 5.39. sip + 5.40. smtp + 5.41. so_proxy + 5.42. ssh + 5.43. ssl + 5.44. stream + 5.45. stream_file + 5.46. stream_icmp + 5.47. stream_ip + 5.48. stream_tcp + 5.49. stream_udp + 5.50. stream_user + 5.51. telnet + 5.52. wizard 6. IPS Action Modules @@ -1096,6 +1097,10 @@ Peg counts: (sum) * payload_injector.http2_injects: total number of http2 injections (sum) + * payload_injector.http2_translate_err: total number of http2 page + translation errors (sum) + * payload_injector.http2_mid_frame: total number of attempts to + inject mid-frame (sum) 2.23. process @@ -1408,6 +1413,8 @@ Configuration: * implied snort.--dirty-pig: don’t flush packets on shutdown * string snort.--dump-builtin-rules: [] output stub rules for selected modules { (optional) } + * select snort.--dump-config: dump config in json format { all | + top } * implied snort.--dump-config-text: dump config in text format * implied snort.--dump-dynamic-rules: output stub rules for all loaded rules libraries @@ -1625,6 +1632,7 @@ Usage: global Configuration: + * int trace.modules.all: enable trace for all modules { 0:255 } * int trace.modules.appid.all: enable all trace options { 0:255 } * int trace.modules.dce_smb.all: enable all trace options { 0:255 } * int trace.modules.dce_udp.all: enable all trace options { 0:255 } @@ -1650,6 +1658,7 @@ Configuration: * int trace.modules.gtp_inspect.all: enable all trace options { 0:255 } * int trace.modules.latency.all: enable all trace options { 0:255 } + * int trace.modules.rna.all: enable all trace options { 0:255 } * int trace.modules.snort.all: enable all trace options { 0:255 } * int trace.modules.snort.main: enable main trace logging { 0:255 } * int trace.modules.snort.inspector_manager: enable inspector @@ -1671,11 +1680,13 @@ Configuration: traces * enum trace.output: output method for trace log messages { stdout | syslog } + * bool trace.log_ntuple = false: use extended trace output with + n-tuple packet info Commands: - * trace.set(modules, constraints): set modules traces and - constraints + * trace.set(modules, constraints, log_ntuple): set modules traces, + constraints and log_ntuple option * trace.clear(): clear modules traces and constraints @@ -2408,7 +2419,7 @@ Commands: enable appid debugging * appid.disable_debug(): disable appid debugging * appid.reload_third_party(): reload appid third-party module - * appid.reload_odp(): reload appid open detector package + * appid.reload_detectors(): reload appid detectors Peg counts: @@ -2442,6 +2453,11 @@ Usage: context Instance Type: global +Configuration: + + * bool appid_listener.json_logging = false: log appid data in json + format + 5.3. arp_spoof @@ -3256,8 +3272,8 @@ Configuration: in bytes { 8:max53 } * int file_id.max_files_cached = 65536: maximal number of files cached in memory { 8:max53 } - * int file_id.max_files_per_flow = 32: maximal number of files able - to be concurrently processed per flow { 1:max53 } + * int file_id.max_files_per_flow = 128: maximal number of files + able to be concurrently processed per flow { 1:max53 } * bool file_id.enable_type = true: enable type ID * bool file_id.enable_signature = false: enable signature calculation @@ -3550,6 +3566,9 @@ Rules: * 121:15 (http2_inspect) invalid HTTP/2 start line * 121:16 (http2_inspect) HTTP/2 padding length is bigger than frame data size + * 121:17 (http2_inspect) HTTP/2 pseudo-header after regular header + * 121:18 (http2_inspect) HTTP/2 pseudo-header in trailers + * 121:19 (http2_inspect) invalid HTTP/2 pseudo-header Peg counts: @@ -3594,6 +3613,8 @@ Configuration: response bodies * bool http_inspect.detained_inspection = false: store-and-forward as necessary to effectively block alerting JavaScript + * bool http_inspect.script_detection = false: inspect JavaScript + immediately upon script end * bool http_inspect.normalize_javascript = false: normalize JavaScript in response bodies * int http_inspect.max_javascript_whitespaces = 200: maximum @@ -3804,6 +3825,8 @@ Peg counts: sessions (max) * http_inspect.detains_requested: packet hold requests for detained inspection (sum) + * http_inspect.script_detections: early inspections of scripts in + HTTP responses (sum) * http_inspect.partial_inspections: pre-inspections for detained inspection (sum) * http_inspect.excess_parameters: repeat parameters exceeding max @@ -3858,6 +3881,10 @@ Peg counts: * imap.concurrent_sessions: total concurrent imap sessions (now) * imap.max_concurrent_sessions: maximum concurrent imap sessions (max) + * imap.start_tls: total STARTTLS events generated (sum) + * imap.ssl_search_abandoned: total SSL search abandoned (sum) + * imap.ssl_srch_abandoned_early: total SSL search abandoned too + soon (sum) * imap.b64_attachments: total base64 attachments decoded (sum) * imap.b64_decoded_bytes: total base64 decoded bytes (sum) * imap.qp_attachments: total quoted-printable attachments decoded @@ -3916,7 +3943,31 @@ Peg counts: sessions (max) -5.28. normalizer +5.28. netflow + +-------------- + +Help: netflow inspection + +Type: inspector + +Usage: inspect + +Instance Type: multiton + +Peg counts: + + * netflow.packets: total packets processed (sum) + * netflow.records: total records found in netflow data (sum) + * netflow.version_5: count of netflow version 5 packets received + (sum) + * netflow.version_9: count of netflow version 9 packets received + (sum) + * netflow.invalid_netflow_pkts: count of invalid netflow packets + (sum) + + +5.29. normalizer -------------- @@ -4054,7 +4105,7 @@ Peg counts: * normalizer.tcp_block: blocked segments (sum) -5.29. null_trace_logger +5.30. null_trace_logger -------------- @@ -4067,7 +4118,7 @@ Usage: global Instance Type: global -5.30. packet_capture +5.31. packet_capture -------------- @@ -4097,7 +4148,7 @@ Peg counts: filter (sum) -5.31. perf_monitor +5.32. perf_monitor -------------- @@ -4157,7 +4208,7 @@ Peg counts: by new flows (sum) -5.32. pop +5.33. pop -------------- @@ -4203,6 +4254,10 @@ Peg counts: * pop.concurrent_sessions: total concurrent pop sessions (now) * pop.max_concurrent_sessions: maximum concurrent pop sessions (max) + * pop.start_tls: total STARTTLS events generated (sum) + * pop.ssl_search_abandoned: total SSL search abandoned (sum) + * pop.ssl_srch_abandoned_early: total SSL search abandoned too soon + (sum) * pop.b64_attachments: total base64 attachments decoded (sum) * pop.b64_decoded_bytes: total base64 decoded bytes (sum) * pop.qp_attachments: total quoted-printable attachments decoded @@ -4215,7 +4270,7 @@ Peg counts: * pop.non_encoded_bytes: total non-encoded extracted bytes (sum) -5.33. port_scan +5.34. port_scan -------------- @@ -4387,7 +4442,7 @@ Peg counts: to reduced memcap (sum) -5.34. reputation +5.35. reputation -------------- @@ -4434,7 +4489,7 @@ Peg counts: * reputation.memory_allocated: total memory allocated (sum) -5.35. rna +5.36. rna -------------- @@ -4450,22 +4505,31 @@ Instance Type: global Configuration: * string rna.rna_conf_path: path to rna configuration - * string rna.fingerprint_dir: directory to fingerprint patterns * bool rna.enable_logger = true: enable or disable writing discovery events into logger * bool rna.log_when_idle = false: enable host update logging when snort is idle * string rna.dump_file: file name to dump RNA mac cache on shutdown; won’t dump by default + * int rna.tcp_fingerprints[].fpid = 0: fingerprint id { 0:max32 } + * int rna.tcp_fingerprints[].type = 0: fingerprint type { 0:max32 } + * string rna.tcp_fingerprints[].uuid: fingerprint uuid + * int rna.tcp_fingerprints[].ttl = 0: fingerprint ttl { 0:256 } + * string rna.tcp_fingerprints[].tcp_window: fingerprint tcp window + * string rna.tcp_fingerprints[].mss = X: fingerprint mss + * string rna.tcp_fingerprints[].id = X: id + * string rna.tcp_fingerprints[].topts: fingerprint tcp options + * string rna.tcp_fingerprints[].ws = X: fingerprint window size + * bool rna.tcp_fingerprints[].df = false: fingerprint don’t + fragment flag Commands: - * rna.reload_fingerprint(): reload rna database of fingerprint - patterns/signatures * rna.dump_macs(): dump rna’s internal MAC trackers Peg counts: + * rna.appid_change: count of appid change events received (sum) * rna.icmp_bidirectional: count of bidirectional ICMP flows received (sum) * rna.icmp_new: count of new ICMP flows received (sum) @@ -4483,7 +4547,7 @@ Peg counts: (sum) -5.36. rpc_decode +5.37. rpc_decode -------------- @@ -4512,7 +4576,7 @@ Peg counts: sessions (max) -5.37. s7commplus +5.38. s7commplus -------------- @@ -4541,7 +4605,7 @@ Peg counts: sessions (max) -5.38. sip +5.39. sip -------------- @@ -4642,7 +4706,7 @@ Peg counts: * sip.code_9xx: 9xx (sum) -5.39. smtp +5.40. smtp -------------- @@ -4751,7 +4815,7 @@ Peg counts: * smtp.non_encoded_bytes: total non-encoded extracted bytes (sum) -5.40. so_proxy +5.41. so_proxy -------------- @@ -4765,7 +4829,7 @@ Usage: global Instance Type: global -5.41. ssh +5.42. ssh -------------- @@ -4805,7 +4869,7 @@ Peg counts: (max) -5.42. ssl +5.43. ssl -------------- @@ -4856,7 +4920,7 @@ Peg counts: (max) -5.43. stream +5.44. stream -------------- @@ -4945,7 +5009,7 @@ Peg counts: deleted by config reloads (sum) -5.44. stream_file +5.45. stream_file -------------- @@ -4962,7 +5026,7 @@ Configuration: * bool stream_file.upload = false: indicate file transfer direction -5.45. stream_icmp +5.46. stream_icmp -------------- @@ -4989,7 +5053,7 @@ Peg counts: * stream_icmp.prunes: icmp session prunes (sum) -5.46. stream_ip +5.47. stream_ip -------------- @@ -5061,7 +5125,7 @@ Peg counts: * stream_ip.fragmented_bytes: total fragmented bytes (sum) -5.47. stream_tcp +5.48. stream_tcp -------------- @@ -5216,7 +5280,7 @@ Peg counts: service stream splitter (sum) -5.48. stream_udp +5.49. stream_udp -------------- @@ -5245,7 +5309,7 @@ Peg counts: * stream_udp.ignored: udp packets ignored (sum) -5.49. stream_user +5.50. stream_user -------------- @@ -5263,7 +5327,7 @@ Configuration: 1:max31 } -5.50. telnet +5.51. telnet -------------- @@ -5299,7 +5363,7 @@ Peg counts: sessions (max) -5.51. wizard +5.52. wizard -------------- @@ -7904,6 +7968,7 @@ these libraries see the Getting Started section of the manual. * --dirty-pig don’t flush packets on shutdown * --dump-builtin-rules [] output stub rules for selected modules (optional) + * --dump-config dump config in json format (all | top) * --dump-config-text dump config in text format * --dump-dynamic-rules output stub rules for all loaded rules libraries @@ -8127,6 +8192,8 @@ these libraries see the Getting Started section of the manual. logging appid statistics { 1:max32 } * int appid.app_stats_rollover_size = 20971520: max file size for appid stats before rolling over the log file { 0:max32 } + * bool appid_listener.json_logging = false: log appid data in json + format * bool appid.list_odp_detectors = false: enable logging of odp detectors statistics * bool appid.log_all_sessions = false: enable logging of all appid @@ -8517,8 +8584,8 @@ these libraries see the Getting Started section of the manual. seconds { 0:max31 } * int file_id.max_files_cached = 65536: maximal number of files cached in memory { 8:max53 } - * int file_id.max_files_per_flow = 32: maximal number of files able - to be concurrently processed per flow { 1:max53 } + * int file_id.max_files_per_flow = 128: maximal number of files + able to be concurrently processed per flow { 1:max53 } * int file_id.qp_decode_depth = -1: Quoted Printable decoding depth (-1 no limit) { -1:65535 } * int file_id.show_data_depth = 100: print this many octets { @@ -8710,6 +8777,8 @@ these libraries see the Getting Started section of the manual. bytes to examine (-1 no limit) { -1:max53 } * int http_inspect.response_depth = -1: maximum response message body bytes to examine (-1 no limit) { -1:max53 } + * bool http_inspect.script_detection = false: inspect JavaScript + immediately upon script end * bool http_inspect.simplify_path = true: reduce URI directory path to simplest form * bool http_inspect.unzip = true: decompress gzip and deflate @@ -9266,10 +9335,20 @@ these libraries see the Getting Started section of the manual. shutdown; won’t dump by default * bool rna.enable_logger = true: enable or disable writing discovery events into logger - * string rna.fingerprint_dir: directory to fingerprint patterns * bool rna.log_when_idle = false: enable host update logging when snort is idle * string rna.rna_conf_path: path to rna configuration + * bool rna.tcp_fingerprints[].df = false: fingerprint don’t + fragment flag + * int rna.tcp_fingerprints[].fpid = 0: fingerprint id { 0:max32 } + * string rna.tcp_fingerprints[].id = X: id + * string rna.tcp_fingerprints[].mss = X: fingerprint mss + * string rna.tcp_fingerprints[].tcp_window: fingerprint tcp window + * string rna.tcp_fingerprints[].topts: fingerprint tcp options + * int rna.tcp_fingerprints[].ttl = 0: fingerprint ttl { 0:256 } + * int rna.tcp_fingerprints[].type = 0: fingerprint type { 0:max32 } + * string rna.tcp_fingerprints[].uuid: fingerprint uuid + * string rna.tcp_fingerprints[].ws = X: fingerprint window size * int rpc.~app: application number { 0:max32 } * string rpc.~proc: procedure number or * for any * string rpc.~ver: version number or * for any @@ -9447,6 +9526,8 @@ these libraries see the Getting Started section of the manual. * implied snort.-D: run Snort in background (daemon) mode * string snort.--dump-builtin-rules: [] output stub rules for selected modules { (optional) } + * select snort.--dump-config: dump config in json format { all | + top } * implied snort.--dump-config-text: dump config in text format * string snort.--dump-defaults: [] output module defaults in Lua format { (optional) } @@ -9804,6 +9885,9 @@ these libraries see the Getting Started section of the manual. traces * string trace.constraints.src_ip: source IP address filter * int trace.constraints.src_port: source port filter { 0:65535 } + * bool trace.log_ntuple = false: use extended trace output with + n-tuple packet info + * int trace.modules.all: enable trace for all modules { 0:255 } * int trace.modules.appid.all: enable all trace options { 0:255 } * int trace.modules.dce_smb.all: enable all trace options { 0:255 } * int trace.modules.dce_udp.all: enable all trace options { 0:255 } @@ -9829,6 +9913,7 @@ these libraries see the Getting Started section of the manual. * int trace.modules.gtp_inspect.all: enable all trace options { 0:255 } * int trace.modules.latency.all: enable all trace options { 0:255 } + * int trace.modules.rna.all: enable all trace options { 0:255 } * int trace.modules.snort.all: enable all trace options { 0:255 } * int trace.modules.snort.inspector_manager: enable inspector manager trace logging { 0:255 } @@ -10419,6 +10504,8 @@ these libraries see the Getting Started section of the manual. * http_inspect.responses: HTTP response messages inspected (sum) * http_inspect.scans: TCP segments scanned looking for HTTP messages (sum) + * http_inspect.script_detections: early inspections of scripts in + HTTP responses (sum) * http_inspect.trace_requests: TRACE requests inspected (sum) * http_inspect.uri_coding: URIs with character coding problems (sum) @@ -10442,6 +10529,10 @@ these libraries see the Getting Started section of the manual. (sum) * imap.qp_decoded_bytes: total quoted-printable decoded bytes (sum) * imap.sessions: total imap sessions (sum) + * imap.ssl_search_abandoned: total SSL search abandoned (sum) + * imap.ssl_srch_abandoned_early: total SSL search abandoned too + soon (sum) + * imap.start_tls: total STARTTLS events generated (sum) * imap.uu_attachments: total uu attachments decoded (sum) * imap.uu_decoded_bytes: total uu decoded bytes (sum) * ipv4.bad_checksum: nonzero ip checksums (sum) @@ -10470,6 +10561,14 @@ these libraries see the Getting Started section of the manual. * modbus.sessions: total sessions processed (sum) * mpls.total_bytes: total mpls labeled bytes processed (sum) * mpls.total_packets: total mpls labeled packets processed (sum) + * netflow.invalid_netflow_pkts: count of invalid netflow packets + (sum) + * netflow.packets: total packets processed (sum) + * netflow.records: total records found in netflow data (sum) + * netflow.version_5: count of netflow version 5 packets received + (sum) + * netflow.version_9: count of netflow version 9 packets received + (sum) * normalizer.icmp4_echo: icmp4 ping normalizations (sum) * normalizer.icmp6_echo: icmp6 echo normalizations (sum) * normalizer.ip4_df: don’t frag bit normalizations (sum) @@ -10553,6 +10652,10 @@ these libraries see the Getting Started section of the manual. * packet_capture.processed: packets processed against filter (sum) * payload_injector.http2_injects: total number of http2 injections (sum) + * payload_injector.http2_mid_frame: total number of attempts to + inject mid-frame (sum) + * payload_injector.http2_translate_err: total number of http2 page + translation errors (sum) * payload_injector.http_injects: total number of http injections (sum) * pcre.pcre_native: total pcre rules compiled by pcre engine (sum) @@ -10582,6 +10685,10 @@ these libraries see the Getting Started section of the manual. (sum) * pop.qp_decoded_bytes: total quoted-printable decoded bytes (sum) * pop.sessions: total pop sessions (sum) + * pop.ssl_search_abandoned: total SSL search abandoned (sum) + * pop.ssl_srch_abandoned_early: total SSL search abandoned too soon + (sum) + * pop.start_tls: total STARTTLS events generated (sum) * pop.total_bytes: total number of bytes processed (sum) * pop.uu_attachments: total uu attachments decoded (sum) * pop.uu_decoded_bytes: total uu decoded bytes (sum) @@ -10599,6 +10706,7 @@ these libraries see the Getting Started section of the manual. * reputation.monitored: number of packets monitored (sum) * reputation.packets: total packets processed (sum) * reputation.whitelisted: number of packets whitelisted (sum) + * rna.appid_change: count of appid change events received (sum) * rna.change_host_update: count number of change host update events (sum) * rna.icmp_bidirectional: count of bidirectional ICMP flows @@ -11334,6 +11442,9 @@ these libraries see the Getting Started section of the manual. * 121:15 (http2_inspect) invalid HTTP/2 start line * 121:16 (http2_inspect) HTTP/2 padding length is bigger than frame data size + * 121:17 (http2_inspect) HTTP/2 pseudo-header after regular header + * 121:18 (http2_inspect) HTTP/2 pseudo-header in trailers + * 121:19 (http2_inspect) invalid HTTP/2 pseudo-header * 122:1 (port_scan) TCP portscan * 122:2 (port_scan) TCP decoy portscan * 122:3 (port_scan) TCP portsweep @@ -11623,7 +11734,7 @@ these libraries see the Getting Started section of the manual. enable appid debugging * appid.disable_debug(): disable appid debugging * appid.reload_third_party(): reload appid third-party module - * appid.reload_odp(): reload appid open detector package + * appid.reload_detectors(): reload appid detectors * host_cache.dump(file_name): dump host cache * packet_capture.enable(filter): dump raw packets * packet_capture.disable(): stop packet dump @@ -11636,8 +11747,6 @@ these libraries see the Getting Started section of the manual. host pairs * perf_monitor.show_flow_ip_profiling(): show status of statistics on host pairs - * rna.reload_fingerprint(): reload rna database of fingerprint - patterns/signatures * rna.dump_macs(): dump rna’s internal MAC trackers * snort.show_plugins(): show available plugins * snort.delete_inspector(inspector): delete an inspector from the @@ -11656,8 +11765,8 @@ these libraries see the Getting Started section of the manual. * snort.detach(): exit shell w/o shutdown * snort.quit(): shutdown and dump-stats * snort.help(): this output - * trace.set(modules, constraints): set modules traces and - constraints + * trace.set(modules, constraints, log_ntuple): set modules traces, + constraints and log_ntuple option * trace.clear(): clear modules traces and constraints @@ -11904,6 +12013,7 @@ and are not applicable elsewhere. * msg (ips_option): rule option summarizing rule purpose output with events * mss (ips_option): detection for TCP maximum segment size + * netflow (inspector): netflow inspection * network (basic): configure basic network parameters * normalizer (inspector): packet scrubbing for inline mode * null_trace_logger (inspector): trace logger with a null printout @@ -12115,6 +12225,7 @@ and are not applicable elsewhere. * inspector::imap: imap inspection * inspector::mem_test: for testing memory management * inspector::modbus: modbus inspection + * inspector::netflow: netflow inspection * inspector::normalizer: packet scrubbing for inline mode * inspector::null_trace_logger: trace logger with a null printout * inspector::packet_capture: raw packet dumping facility diff --git a/doc/upgrade/snort_upgrade.text b/doc/upgrade/snort_upgrade.text index 328a08d9c..940622924 100644 --- a/doc/upgrade/snort_upgrade.text +++ b/doc/upgrade/snort_upgrade.text @@ -8,7 +8,7 @@ Snort 3 Upgrade Manual The Snort Team Revision History -Revision 3.0.2 (Build 5) 2020-08-12 08:28:19 EDT TST +Revision 3.0.2 (Build 6) 2020-09-13 14:48:01 EDT TST --------------------------------------------------------------------- @@ -928,7 +928,6 @@ change -> http_inspect_server: 'utf_8' ==> 'utf8' change -> imap: 'ports' ==> 'bindings' change -> modbus: 'ports' ==> 'bindings' change -> na_policy_mode: 'na_policy_mode' ==> 'mode' -change -> nap_selector: 'nap rules' ==> 'bindings' change -> paf_max: 'paf_max [0:63780]' ==> 'max_pdu [1460:32768]' change -> perfmonitor: 'console' ==> 'format = 'text'' change -> perfmonitor: 'console' ==> 'output = 'console'' @@ -1108,8 +1107,6 @@ deleted -> http_inspect_server: 'unlimited_decompress' deleted -> imap: 'disabled' deleted -> imap: 'max_mime_mem' deleted -> imap: 'memcap' -deleted -> nap_selector: 'fw_required' -deleted -> nap_selector: 'nap_stats_time' deleted -> perfmonitor: 'accumulate' deleted -> perfmonitor: 'atexitonly' deleted -> perfmonitor: 'atexitonly: base-stats' @@ -1133,8 +1130,6 @@ deleted -> sfportscan: 'detect_ack_scans' deleted -> sfportscan: 'disabled' deleted -> sfportscan: 'logfile' deleted -> sfportscan: 'sense_level' -deleted -> sfunified2: 'mpls_event_types' -deleted -> sfunified2: 'vlan_event_types' deleted -> sip: 'disabled' deleted -> sip: 'max_sessions' deleted -> smtp: 'alert_unknown_cmds' diff --git a/doc/user/snort_user.text b/doc/user/snort_user.text index 7828673a8..26aa680dd 100644 --- a/doc/user/snort_user.text +++ b/doc/user/snort_user.text @@ -8,7 +8,7 @@ Snort 3 User Manual The Snort Team Revision History -Revision 3.0.2 (Build 5) 2020-08-12 08:28:19 EDT TST +Revision 3.0.2 (Build 6) 2020-09-13 14:48:01 EDT TST --------------------------------------------------------------------- @@ -2416,64 +2416,302 @@ byte_test:4,>,200,36; -------------- -Using Consolidated Config output enables troubleshooting of -configuration issues. The output contains applied configurations ( -defaults and configured ) and is printed for the main config and all -included policies. So far, Snort supports output in text format. +Config dump mode generates a consolidated dump of the config passed +to Snort. This output consists of the configured values as well as +the module defaults for the values that aren’t configured. + +In the dump mode Snort validates the config (similar to option -T) +and suppresses unrelated messages going to stdout (configuration +warnings and errors are still printed to stderr). + +The dump mode is activated by the following options: +--dump-config-text, --dump-config=all, --dump-config=top. They are +described in detail below. + +The simple configuration is used in examples. The output contains +applied configurations (defaults and configured). To simplify the +output we show a brief list of default options. + +snort.lua + +stream = +{ + max_flows = 2 +} + +stream_tcp = +{ + show_rebuilt_packets = true +} + +binder = +{ + { when = { nets = '10.1.2.0/24' }, use = { inspection_policy = 'http.lua' } }, + { when = { nets = '192.168.2.0/24' }, use = { inspection_policy = 'sip.lua' } }, +} + +http.lua + +wizard = +{ + spells = + { + { service = 'http', proto = 'tcp', client_first = true, to_server = { 'GET' }, to_client = { 'HTTP/' } }, + } +} + +sip.lua + +wizard = +{ + spells = + { + { service = 'sip', to_server = { 'INVITE' } }, + } +} 5.5.1. Text Format The --dump-config-text option verifies the configuration and dumps it -to stdout in text format. +to stdout in text format. The output contains a config of the main +policy and all other included sub-policies. -Example: +Example: snort -c snort.lua --dump-config-text consolidated config for snort.lua +alerts.order="pass reset block drop alert log" +alerts.rate_filter_memcap=1048576 binder[0].when.ips_policy_id=0 -binder[0].when.role='any' -binder[0].when.nets='10.1.2.0/24' -binder[0].use.action='inspect' +binder[0].when.role="any" +binder[0].when.nets="10.1.2.0/24" +binder[0].use.action="inspect" +binder[0].use.inspection_policy="http.lua" binder[1].when.ips_policy_id=0 -binder[1].when.role='any' -binder[1].when.nets='192.168.2.0/24' -binder[1].use.action='inspect' -host_cache.memcap=8.38861e+06 -network.checksum_drop='none' -network.checksum_eval='all' -network.max_ip_layers=0 -process.daemon=false -process.dirty_pig=false -process.utc=false -stream_tcp.flush_factor=0 -stream_tcp.max_window=0 -stream_tcp.overlap_limit=0 -stream_tcp.max_pdu=16384 -stream.footprint=0 -stream.ip_frags_only=false -trace.modules.appid.all=1 -trace.modules.detection.opt_tree=2 -trace.modules.detection.fp_search=4 -trace.modules.detection.rule_eval=1 -trace.modules.wizard.all=1 -trace.constraints.match=true -trace.constraints.dst_ip='10.1.1.2' -trace.constraints.dst_port=200 -trace.constraints.src_port=100 -trace.constraints.ip_proto=17 -trace.output='stdout' -wizard.spells[0].proto='tcp' +binder[1].when.role="any" +binder[1].when.nets="192.168.2.0/24" +binder[1].use.action="inspect" +binder[1].use.inspection_policy="sip.lua" +output.obfuscate=false +output.wide_hex_dump=true +packets.address_space_agnostic=false +packets.limit=0 +search_engine.split_any_any=true +search_engine.queue_limit=128 +stream.file_cache.idle_timeout=180 +stream.file_cache.cap_weight=32 +stream.max_flows=2 +stream_tcp.small_segments.maximum_size=0 +stream_tcp.session_timeout=30 +stream_tcp.track_only=false +stream_tcp.show_rebuilt_packets=true +consolidated config for http.lua +wizard.spells[0].proto="tcp" +wizard.spells[0].client_first=true +wizard.spells[0].service="http" +wizard.spells[0].to_client[0].spell="HTTP/" +wizard.spells[0].to_server[0].spell="GET" +consolidated config for sip.lua +wizard.spells[0].proto="tcp" wizard.spells[0].client_first=true -wizard.spells[0].service='http' -wizard.spells[0].to_client[0].spell='HTTP/' -wizard.spells[0].to_server[0].spell='GET' -wizard.spells[1].proto='tcp' -wizard.spells[1].client_first=true -wizard.spells[1].service='sip' -wizard.spells[1].to_server[0].spell='INVITE' +wizard.spells[0].service="sip" +wizard.spells[0].to_server[0].spell="INVITE" For lists, the index next to the option name designates an element parsing order. +5.5.2. JSON Format + +The --dump-config=all command-line option verifies the configuration +and dumps it to stdout in JSON format. The output contains a config +of the main policy and all other included sub-policies. Snort dumps +output in a one-line format. + +There is 3rd party tool jq for converting to a pretty printed format. + +Example: snort -c snort.lua --dump-config=all | jq . + +[ + { + "filename": "snort.lua", + "config": { + "alerts": { + "order": "pass reset block drop alert log", + "rate_filter_memcap": 1048576 + }, + "binder": [ + { + "when": { + "ips_policy_id": 0, + "role": "any", + "nets": "10.1.2.0/24" + }, + "use": { + "action": "inspect", + "inspection_policy": "http.lua" + } + }, + { + "when": { + "ips_policy_id": 0, + "role": "any", + "nets": "192.168.2.0/24" + }, + "use": { + "action": "inspect", + "inspection_policy": "sip.lua" + } + } + ], + "output": { + "obfuscate": false, + "wide_hex_dump": true + }, + "packets": { + "address_space_agnostic": false, + "limit": 0 + }, + "process": { + "daemon": false, + "dirty_pig": false, + "utc": false + }, + "search_engine": { + "split_any_any": true, + "queue_limit": 128 + }, + "stream": { + "file_cache": { + "idle_timeout": 180, + "cap_weight": 32 + }, + "max_flows": 2 + }, + "stream_tcp": { + "small_segments": { + "maximum_size": 0 + }, + "session_timeout": 30, + "track_only": false, + "show_rebuilt_packets": true + } + } + }, + { + "filename": "http.lua", + "config": { + "wizard": { + "spells": [ + { + "proto": "tcp", + "client_first": true, + "service": "http", + "to_client": [ + { + "spell": "HTTP/" + } + ], + "to_server": [ + { + "spell": "GET" + } + ] + } + ] + } + } + }, + { + "filename": "sip.lua", + "config": { + "wizard": { + "spells": [ + { + "proto": "tcp", + "client_first": true, + "service": "sip", + "to_server": [ + { + "spell": "INVITE" + } + ] + } + ] + } + } + } +] + +The --dump-config=top command-line option is similar to --dump-config +=all, except it produces dump for the main policy only. It verifies +the configuration and dumps the main policy configuration to stdout +in JSON format. + +Example: snort -c snort.lua --dump-config=top | jq . + +{ + "alerts": { + "order": "pass reset block drop alert log", + "rate_filter_memcap": 1048576, + }, + "binder": [ + { + "when": { + "ips_policy_id": 0, + "role": "any", + "nets": "10.1.2.0/24" + }, + "use": { + "action": "inspect", + "inspection_policy": "http.lua" + } + }, + { + "when": { + "ips_policy_id": 0, + "role": "any", + "nets": "192.168.2.0/24" + }, + "use": { + "action": "inspect", + "inspection_policy": "sip.lua" + } + } + ], + "output": { + "obfuscate": false, + "wide_hex_dump": true + }, + "packets": { + "address_space_agnostic": false, + "limit": 0, + }, + "process": { + "daemon": false, + "dirty_pig": false, + "utc": false + }, + "search_engine": { + "split_any_any": true, + "queue_limit": 128 + }, + "stream": { + "file_cache": { + "idle_timeout": 180, + "cap_weight": 32 + } + "max_flows": 2 + }, + "stream_tcp": { + "small_segments": { + "count": 0, + "maximum_size": 0 + }, + "session_timeout": 30, + "track_only": false, + "show_rebuilt_packets": true + }, +} + 5.6. DCE Inspectors @@ -3562,7 +3800,18 @@ mode operation (-Q). This feature is off by default. detained_inspection = true will activate it. -5.10.2.3. gzip +5.10.2.3. script_detection + +Script detection is an alternative to detained inspection. When +http_inspect detects the end of a script it immediately forwards the +available part of the message body for early detection. This enables +malicious Javascripts to be detected more quickly but consumes +somewhat more of the sensor’s resources. + +This feature is off by default. script_detection = true will activate +it. + +5.10.2.4. gzip http_inspect by default decompresses deflate and gzip message bodies before inspecting them. This feature can be turned off by unzip = @@ -3571,14 +3820,14 @@ improvement but at a very high price. It is unlikely that any meaningful inspection of message bodies will be possible. Effectively HTTP processing would be limited to the headers. -5.10.2.4. normalize_utf +5.10.2.5. normalize_utf http_inspect will decode utf-8, utf-7, utf-16le, utf-16be, utf-32le, and utf-32be in response message bodies based on the Content-Type header. This feature is on by default: normalize_utf = false will deactivate it. -5.10.2.5. decompress_pdf +5.10.2.6. decompress_pdf decompress_pdf = true will enable decompression of compressed portions of PDF files encountered in a response body. http_inspect @@ -3587,7 +3836,7 @@ locate PDF streams with a single /FlateDecode filter. The compressed content is decompressed and made available through the file data rule option. -5.10.2.6. decompress_swf +5.10.2.7. decompress_swf decompress_swf = true will enable decompression of compressed SWF (Adobe Flash content) files encountered in a response body. The @@ -3597,7 +3846,7 @@ LZMA. The compressed content is decompressed and made available through the file data rule option. The compressed SWF file signature is converted to FWS to indicate an uncompressed file. -5.10.2.7. normalize_javascript +5.10.2.8. normalize_javascript normalize_javascript = true will enable normalization of JavaScript within the HTTP response body. http_inspect looks for JavaScript by @@ -3609,7 +3858,7 @@ decodeURIComponent are %XX, %uXXXX, XX and uXXXXi. http_inspect also replaces consecutive whitespaces with a single space and normalizes the plus by concatenating the strings. -5.10.2.8. URI processing +5.10.2.9. URI processing Normalization and inspection of the URI in the HTTP request message is a key aspect of what http_inspect does. The best way to normalize @@ -4962,10 +5211,13 @@ the following parameters: output - configure the output method for trace messages modules - trace configuration for specific modules constraints - filter traces by the packet constraints +log_ntuple - on/off packet n-tuple info logging The following lines, added in snort.lua, will enable trace messages for detection and codec modules. The messages will be printed to -syslog if the packet filtering constraints match. +syslog if the packet filtering constraints match. Messages will be in +extended format, including n-tuple packet info at the beginning of +each trace message. trace = { @@ -4981,7 +5233,8 @@ trace = dst_ip = "10.1.1.2", src_port = 100, dst_port = 200 - } + }, + log_ntuple = true } The trace module supports config reloading. Also, it’s possible to @@ -5027,6 +5280,26 @@ trace = } } +Also, it’s possible to enable or disable traces for all modules with +a top-level all option. + +The following configuration states that: + + * all traces are enabled with verbosity level 5 + * traces for the decode module are enabled with level 3 + * rule_eval traces for the detection module are enabled with level + 1 + + trace = + { + modules = + { + all = 5, + decode = { all = 3 }, + detection = { rule_eval = 1 } + } + } + The full list of available trace parameters is placed into the "Basic Modules.trace" chapter. @@ -5145,11 +5418,18 @@ The trace control channel command is a way how to configure module trace options and/or packet filter constraints directly during Snort run and without reloading the entire config. +Control channel also allow adjusting trace output format by setting +log_ntuple switcher. + After entering the Snort shell, there are two commands available for the trace module: trace.set({ modules = {...}, constraints = {...} }) - set modules traces and constraints (should pass a valid Lua-entry) +trace.set({ modules = { all = N } }) - enable traces for all modules with verbosity level N + +trace.set({ log_ntuple = true/false }) - on/off packet n-tuple info logging + trace.clear() - clear modules traces and constraints Also, it’s possible to omit tables in the trace.set() command: @@ -5175,6 +5455,27 @@ the thread type. Possible thread types: C – main (control) thread P – packet thread O – other thread +Setting the option - log_ntuple allows you to change the trace +message format, expanding it with information about the processed +packet. + +It will be added at the beginning, right after the thread type and +instance ID, in the following format: + +src_ip src_port -> dst_ip dst_port ip_proto AS=address_space + +Where: + +src_ip - source IP address +src_port - source port +dst_ip - destination IP address +dst_port - destination port +ip_proto - IP protocol ID +address_space - unique ID of the address space + +Those info can be displayed only for IP packets. Port defaults to +zero if a packet doesn’t have it. + 5.18.7. Example - Debugging rules using detection trace The detection engine is responsible for rule evaluation. Turning on diff --git a/src/main/build.h b/src/main/build.h index 63325a081..8cfb88209 100644 --- a/src/main/build.h +++ b/src/main/build.h @@ -12,7 +12,7 @@ // // //-----------------------------------------------// -#define BUILD_NUMBER 5 +#define BUILD_NUMBER 6 #ifndef EXTRABUILD #define BUILD STRINGIFY_MX(BUILD_NUMBER)