From: Steve Chew (stechew) Date: Thu, 19 May 2022 21:26:26 +0000 (+0000) Subject: Pull request #3436: build: generate and tag 3.1.30.0 X-Git-Tag: 3.1.30.0 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=b59ade8446e287a3b2c57003a5a794e2c843ae84;p=thirdparty%2Fsnort3.git Pull request #3436: build: generate and tag 3.1.30.0 Merge in SNORT/snort3 from ~STECHEW/snort3:build_3.1.30.0 to master Squashed commit of the following: commit 5403acd8e7a4072702b809f0b4302570032728e3 Author: Steve Chew Date: Thu May 19 00:33:25 2022 -0400 build: generate and tag 3.1.30.0 --- diff --git a/CMakeLists.txt b/CMakeLists.txt index e0f5e3f47..94a172124 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -3,7 +3,7 @@ project (snort CXX C) set (VERSION_MAJOR 3) set (VERSION_MINOR 1) -set (VERSION_PATCH 29) +set (VERSION_PATCH 30) set (VERSION_SUBLEVEL 0) set (VERSION "${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}.${VERSION_SUBLEVEL}") diff --git a/ChangeLog b/ChangeLog index 1d07a39d6..59ca03faf 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,28 @@ +2022/05/19 - 3.1.30.0 + +build: Update dependent libdaq version to 3.0.7 +doc: update clone link in README. Thanks to billchenchina. +doc: user documentation update for obfuscate_pii and --help-module +framework: add method to get unquoted string from configuration value +http2_inspect: Templatize variable length integer decoding of integer and string +http_inspect: add ignoring defined object properties for Enchanced JS normalizer +http_inspect: avoid sending compressed data to JS normalizer +http_inspect: check if input available before JavaScript normalization +mime: set partial_header to null after deletion +perf_monitor: remove unused flatbuffers support +piglets: remove unused test harness +smb: handle file context cleanup +snort3: remove SMB detection from service_netbios.cc +stream: refactor flush_queued_segments +stream_tcp: add null check for get_current_wire_packet() in dce too +stream_tcp, pop: add sync_on_start method to StreamSplitter +stream_tcp: provide a context and a wire packet where needed, when calling into reassembly from outside regular processing (handle_timeouts) +utils: add Latin-1 decoding of JavaScript unescape-like functions +utils: allow regex literals after operator +utils: fix regex char classes parsing +utils: turn debug-build assertion into a product-build code +wizard: fix code style + 2022/05/04 - 3.1.29.0 appid: add alpn matchers diff --git a/doc/reference/snort_reference.text b/doc/reference/snort_reference.text index 9e88af41b..c4ec736f4 100644 --- a/doc/reference/snort_reference.text +++ b/doc/reference/snort_reference.text @@ -8,7 +8,7 @@ Snort 3 Reference Manual The Snort Team Revision History -Revision 3.1.29.0 2022-05-04 08:07:08 EDT TST +Revision 3.1.30.0 2022-05-19 00:40:10 EDT TST --------------------------------------------------------------------- @@ -3741,6 +3741,8 @@ Configuration: 1:65535 } * string http_inspect.js_norm_ident_ignore[].ident_name: name of the identifier to ignore + * string http_inspect.js_norm_prop_ignore[].prop_name: name of the + object property to ignore * int http_inspect.max_javascript_whitespaces = 200: maximum consecutive whitespaces allowed within the JavaScript obfuscated data { 1:65535 } @@ -4553,7 +4555,7 @@ Configuration: * string perf_monitor.modules[].pegs: list of statistics to track or empty for all counters * enum perf_monitor.format = csv: output format for stats { csv | - text | json | flatbuffers } + text | json } * bool perf_monitor.summary = false: output summary at shutdown Commands: @@ -8514,7 +8516,6 @@ configure --help. These options are built only if the required libraries and headers are present. There is no need to explicitly enable. - * flatbuffers: for an alternative perf_monitor logging format. * hyperscan >= 4.4.0: for the regex and sd_pattern rule options and the hyperscan search engine. * iconv: for converting UTF16-LE filenames to UTF8 (usually @@ -8534,8 +8535,8 @@ locations, you can use these options: package libraries. These can be used for pcap, luajit, pcre, dnet, daq, lzma, openssl, -flatbuffers, iconv, and hyperscan packages. For more information on -these libraries see the Getting Started section of the manual. +iconv, and hyperscan packages. For more information on these +libraries see the Getting Started section of the manual. 11.2. Environment Variables @@ -9452,6 +9453,8 @@ these libraries see the Getting Started section of the manual. * int http_inspect.js_norm_max_tmpl_nest = 32: maximum depth of template literal nesting that enhanced javascript normalizer will process { 0:255 } + * string http_inspect.js_norm_prop_ignore[].prop_name: name of the + object property to ignore * int http_inspect.maximum_chunk_length = 4294967295: maximum allowed length for a message body chunk { 0:4294967295 } * int http_inspect.maximum_host_length = -1: maximum allowed length @@ -9875,7 +9878,7 @@ these libraries see the Getting Started section of the manual. * int perf_monitor.flow_ports = 1023: maximum ports to track { 0:65535 } * enum perf_monitor.format = csv: output format for stats { csv | - text | json | flatbuffers } + text | json } * int perf_monitor.max_file_size = 1073741824: files will be rolled over if they exceed this size { 4096:max53 } * string perf_monitor.modules[].name: name of the module diff --git a/doc/upgrade/snort_upgrade.text b/doc/upgrade/snort_upgrade.text index f5b35d34e..723947de9 100644 --- a/doc/upgrade/snort_upgrade.text +++ b/doc/upgrade/snort_upgrade.text @@ -8,7 +8,7 @@ Snort 3 Upgrade Manual The Snort Team Revision History -Revision 3.1.29.0 2022-05-04 08:06:54 EDT TST +Revision 3.1.30.0 2022-05-19 00:39:56 EDT TST --------------------------------------------------------------------- @@ -72,7 +72,6 @@ of Snort 3.0 are: * New performance monitor * New time and space profiling * New latency monitoring and enforcement - * Piglets to facilitate component testing * Inspection Events * Autogenerate reference documentation @@ -170,7 +169,7 @@ of Snort 3.0 are: Some things Snort++ can do today that Snort can not do: * regex fast patterns, not just literals - * FlatBuffers and JSON perf monitor logs + * JSON perf monitor logs * LuaJIT scriptable rule options and loggers * pub/sub inspection events (currently used by sip and http_inspect to appid) diff --git a/doc/user/snort_user.text b/doc/user/snort_user.text index 4a0819eac..22c5e746b 100644 --- a/doc/user/snort_user.text +++ b/doc/user/snort_user.text @@ -8,7 +8,7 @@ Snort 3 User Manual The Snort Team Revision History -Revision 3.1.29.0 2022-05-04 08:06:54 EDT TST +Revision 3.1.30.0 2022-05-19 00:39:56 EDT TST --------------------------------------------------------------------- @@ -115,7 +115,6 @@ are: * New performance monitor * New time and space profiling * New latency monitoring and enforcement - * Piglets to facilitate component testing * Inspection Events * Autogenerate reference documentation @@ -621,7 +620,7 @@ them generically and consistently. You can learn quite a lot about any given module from the command line. For example, to see what stream_tcp is all about, do this: -$ snort --help-config stream_tcp +$ snort --help-module stream_tcp Modules are configured using Lua tables with the same name. So the stream_tcp module is configured with defaults like this: @@ -1018,8 +1017,6 @@ Optional: tests with make check * dblatex from http://dblatex.sourceforge.net to build the pdf manual (in addition to asciidoc) - * flatbuffers from https://google.github.io/flatbuffers/ for - enabling the flatbuffers serialization format * hyperscan >= 4.4.0 from https://github.com/01org/hyperscan to build new the regex and sd_pattern rule options and hyperscan search engine. Hyperscan is large so it recommended to follow @@ -3894,7 +3891,7 @@ scripts. It supports scripts over multiple PDUs. It is a stateful JavaScript whitespace and identifiers normalizer. Normalizer concatenates string literals whenever it’s possible to do. This also works with any other normalizations that result in string literals. -All JavaScript identifier names, except those from the ignore list, +All JavaScript identifier names, except those from the ignore lists, will be substituted with unified names in the following format: var_0000 → var_ffff. But the unescape-like function names will be removed from the normalized data. The Normalizer tries to expand an @@ -3905,8 +3902,8 @@ elements. For more information on how additionally configure Enhanced Normalizer check with the following configuration options: js_norm_bytes_depth, js_norm_identifier_depth, js_norm_max_tmpl_nest, js_norm_max_bracket_depth, js_norm_max_scope_depth, -js_norm_ident_ignore. Eventually Enhanced Normalizer will completely -replace Legacy Normalizer. +js_norm_ident_ignore, js_norm_prop_ignore. Eventually Enhanced +Normalizer will completely replace Legacy Normalizer. 5.10.3. Configuration @@ -4120,7 +4117,30 @@ ought to be included in the ignore list. If for some reason the user wants to disable unescape related features, then removing function’s name from the ignore list does the trick. -5.10.3.16. xff_headers +5.10.3.16. js_norm_prop_ignore + +js_norm_prop_ignore = {} is an option of +the enhanced JavaScript normalizer that defines a list of object +properties and methods that will be kept intact during the +identifiers normalization. This list should include methods and +properties of objects that will not be tracked by assignment +substitution functionality, for example, those that can be created +implicitly. + +Subsequent accessors, after dot, in square brackets or after function +call, will not be normalized as well. + +For example: + +http_inspect.js_norm_prop_ignore = { 'split' } + +in: "string".toUpperCase().split("").reverse().join(""); +out: "string".var_0000().split("").reverse().join(""); + +The default list of ignored properties is present in +"snort_defaults.lua". + +5.10.3.17. xff_headers This configuration supports defining custom x-forwarded-for type headers. In a multi-vendor world, it is quite possible that the @@ -4135,7 +4155,7 @@ they are defined, e.g "x-forwarded-for" will be preferred than "true-client-ip" if both headers are present in the stream. The header names should be delimited by a space. -5.10.3.17. maximum_host_length +5.10.3.18. maximum_host_length Setting maximum_host_length causes http_inspect to generate 119:25 if the Host header value including optional white space exceeds the @@ -4143,7 +4163,7 @@ specified length. In the abnormal case of multiple Host headers, the total length of the combined values is used. The default value is -1, meaning do not perform this check. -5.10.3.18. maximum_chunk_length +5.10.3.19. maximum_chunk_length http_inspect strictly limits individual chunks within a chunked message body to be less than four gigabytes. @@ -4151,7 +4171,7 @@ message body to be less than four gigabytes. A lower limit may be configured by setting maximum_chunk_length. Any chunk longer than maximum chunk length will generate a 119:16 alert. -5.10.3.19. URI processing +5.10.3.20. URI processing Normalization and inspection of the URI in the HTTP request message is a key aspect of what http_inspect does. The best way to normalize @@ -5117,14 +5137,7 @@ perf_monitor = { cpu = true } Performance monitor allows statistics to be output in a few formats. Along with human readable text (as seen at shutdown) and csv formats, -a Flatbuffers binary format is also available if Flatbuffers is -present at build. A utility for accessing the statistics generated in -this format has been included for convenience (see fbstreamer in -tools). This tool generates a YAML array of records found, allowing -the data to be read by humans or passed into other analysis tools. -For information on working directly with the Flatbuffers file format -used by Performance monitor, see the developer notes for Performance -monitor or the code provided for fbstreamer. +a JSON format format is also available. 5.15. POP and IMAP @@ -5528,11 +5541,11 @@ occurred 299 times in a packet, you will not see an event. Snort provides discreet logging for the built in patterns "credit_card", "us_social" and "us_social_nodashes". Enabling -output.obfuscate_pii makes Snort obfuscate the suspect packet payload +ips.obfuscate_pii makes Snort obfuscate the suspect packet payload which was matched by the patterns. This configuration is disabled by default. -output = +ips = { obfuscate_pii = true }