From: Mike Stepanek (mstepane) Date: Thu, 7 Apr 2022 17:53:09 +0000 (+0000) Subject: Pull request #3367: build: generate and tag 3.1.27.0 X-Git-Tag: 3.1.27.0 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=1d8096d42b07213c0ead8e023e48db9cc19398eb;p=thirdparty%2Fsnort3.git Pull request #3367: build: generate and tag 3.1.27.0 Merge in SNORT/snort3 from ~MSTEPANE/snort3:build_3.1.27.0 to master Squashed commit of the following: commit 5431b622172ee145af2dbbe6889e87764669d7f1 Author: Mike Stepanek Date: Thu Apr 7 13:27:04 2022 -0400 build: generate and tag 3.1.27.0 --- diff --git a/CMakeLists.txt b/CMakeLists.txt index c76152321..d3b667a5a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -3,7 +3,7 @@ project (snort CXX C) set (VERSION_MAJOR 3) set (VERSION_MINOR 1) -set (VERSION_PATCH 26) +set (VERSION_PATCH 27) set (VERSION_SUBLEVEL 0) set (VERSION "${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}.${VERSION_SUBLEVEL}") diff --git a/ChangeLog b/ChangeLog index 712e8d70d..4c5d8f2de 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,50 @@ +2022/04/07 - 3.1.27.0 + +ac_full: refactor api access +ac_full: remove cruft +ac_std: fix case translation buffer size +alerts: remove obsolete stateful parameter +appid: provide client appid set by encrypted visibility engine to ssl through the ssl appid lookup api +build: compile against libatomic if present. Thanks to W. Michael Petullo +control, shell: add a command to set the network policy to be used by subsequent commands +dce_rpc: handle cleanup path and race conditions for dce traffic +detection: do not check ips policy when builtin events are queued +detection: fixup dump of detection option tree +detection: minor refactoring of rule header access +detection: override match queue limit for offload +detection: remove cruft +detection: skip match deduplication for hyperscan +file_api: handle user_file_data cleanup +hext: change stdin designation from tty to - since the trough uses dash +http2_inspect: reduce holes in objects +http_inspect: add unescape text processing for Enhanced JS Normalizer +http_inspect: decode String.fromCodePoint() JavaScript function +http_inspect: delete alerts 119:279 and 119:280 +http_inspect: provide current packet to trace +http_inspect: support headers Restrict-Access-To-Tenants, Restrict-Access-Context +hyperscan: ensure adequate scratch when deserializing +rate_filter: move to inspection policy +search_engine: add fast pattern only count at startup +search_engine: always build ac_full since it is a hard default case +search_engine: fix .debug = true output +search_engine: fix adjustment for fast_pattern_offset +search_engine: fix fast pattern only eligibility check +search_engine: remove obsolete warning on max_pattern_len change +search_engine: remove search_optimize parameter (always true) +search_engine: truncated patterns not eligible as fast pattern only contents +search_engines: add and refactor unit tests +search_engines: ensure SearchTool with hyperscan gets multi-match mode +search_engines: remove the legacy ac_banded algorithm +search_engines: remove the legacy ac_sparse algorithm +search_engines: remove the legacy ac_sparse_bands algorithm +search_engines: remove the legacy ac_std algorithm +sfip: suppress compiler warning +utils: add string concatenation for Enchanced JS Normalizer +utils: allow opening/closing tags in external scripts +utils: fix JS Normalizer benchmark build +utils: fix tracking variable when the output buffer is reset +utils: harden script opening tag sequence + 2022/03/23 - 3.1.26.0 actions: revert bf62a22d43bb2d15b7425c5ec3e3118ead470e8d diff --git a/doc/reference/snort_reference.text b/doc/reference/snort_reference.text index 417947e45..24eb4c51a 100644 --- a/doc/reference/snort_reference.text +++ b/doc/reference/snort_reference.text @@ -8,7 +8,7 @@ Snort 3 Reference Manual The Snort Team Revision History -Revision 3.1.26.0 2022-03-23 13:19:21 EDT TST +Revision 3.1.27.0 2022-04-07 13:35:35 EDT TST --------------------------------------------------------------------- @@ -423,8 +423,6 @@ Configuration: memory for rate_filters { 0:max32 } * string alerts.reference_net: set the CIDR for homenet (for use with -l or -B, does NOT change $HOME_NET in IDS mode) - * bool alerts.stateful = false: don’t alert w/o established session - (note: rule action still taken) * string alerts.tunnel_verdicts: let DAQ handle non-allow verdicts for gtp|teredo|6in4|4in6|4in4|6in6|gre|mpls|vxlan traffic @@ -1062,6 +1060,11 @@ Configuration: Snort will process for a given packet before raising 116:293 (0 = unlimited) { 0:255 } +Commands: + + * network.set_policy(id): set the network policy for commands given + the user policy id + 2.20. output @@ -1248,7 +1251,7 @@ Help: configure rate filters (which change rule actions) Type: basic -Usage: context +Usage: inspect Configuration: @@ -1322,17 +1325,13 @@ Configuration: * bool search_engine.detect_raw_tcp = false: detect on TCP payload before reassembly * dynamic search_engine.search_method = ac_bnfa: set fast pattern - algorithm - choose available search engine { ac_banded | ac_bnfa - | ac_full | ac_sparse | ac_sparse_bands | ac_std | hyperscan | - lowmem } - * dynamic search_engine.offload_search_method: set fast pattern - offload algorithm - choose available search engine { ac_banded | - ac_bnfa | ac_full | ac_sparse | ac_sparse_bands | ac_std | + algorithm - choose available search engine { ac_bnfa | ac_full | hyperscan | lowmem } + * dynamic search_engine.offload_search_method: set fast pattern + offload algorithm - choose available search engine { ac_bnfa | + ac_full | hyperscan | lowmem } * string search_engine.rule_db_dir: deserialize rule databases from given directory - * bool search_engine.search_optimize = true: tweak state machine - construction for better performance * bool search_engine.show_fast_patterns = false: print fast pattern info for each rule * bool search_engine.split_any_any = true: evaluate any-any rules @@ -3974,10 +3973,6 @@ Rules: * 119:277 (http_inspect) HTTP version in start line is higher than 1 * 119:278 (http_inspect) HTTP gzip body with the FEXTRA flag set - * 119:279 (http_inspect) nested unescape functions in JavaScript - code - * 119:280 (http_inspect) mixing of escape formats in JavaScript - code Peg counts: @@ -8792,8 +8787,6 @@ these libraries see the Getting Started section of the manual. memory for rate_filters { 0:max32 } * string alerts.reference_net: set the CIDR for homenet (for use with -l or -B, does NOT change $HOME_NET in IDS mode) - * bool alerts.stateful = false: don’t alert w/o established session - (note: rule action still taken) * string alerts.tunnel_verdicts: let DAQ handle non-allow verdicts for gtp|teredo|6in4|4in6|4in4|6in6|gre|mpls|vxlan traffic * enum alert_syslog.facility = auth: part of priority applied to @@ -10196,19 +10189,15 @@ these libraries see the Getting Started section of the manual. * int search_engine.max_queue_events = 5: maximum number of matching fast pattern states to queue per packet { 2:100 } * dynamic search_engine.offload_search_method: set fast pattern - offload algorithm - choose available search engine { ac_banded | - ac_bnfa | ac_full | ac_sparse | ac_sparse_bands | ac_std | - hyperscan | lowmem } + offload algorithm - choose available search engine { ac_bnfa | + ac_full | hyperscan | lowmem } * int search_engine.queue_limit = 0: maximum number of fast pattern matches to queue per packet (0 is unlimited) { 0:max32 } * string search_engine.rule_db_dir: deserialize rule databases from given directory * dynamic search_engine.search_method = ac_bnfa: set fast pattern - algorithm - choose available search engine { ac_banded | ac_bnfa - | ac_full | ac_sparse | ac_sparse_bands | ac_std | hyperscan | - lowmem } - * bool search_engine.search_optimize = true: tweak state machine - construction for better performance + algorithm - choose available search engine { ac_bnfa | ac_full | + hyperscan | lowmem } * bool search_engine.show_fast_patterns = false: print fast pattern info for each rule * bool search_engine.split_any_any = true: evaluate any-any rules @@ -12970,7 +12959,8 @@ HTTP response has Content-Type charset=utf-7. 119:109 (http_inspect) more than one level of JavaScript obfuscation More than one level of JavaScript obfuscation. This alert can only be -generated when normalize_javascript configuration option is true. +generated when normalize_javascript configuration option is true or +enhanced JavaScript normalizer is enabled. 119:110 (http_inspect) consecutive JavaScript whitespaces exceed maximum allowed @@ -12984,7 +12974,7 @@ obfuscated data More than one encoding within JavaScript obfuscated data. This alert can only be generated when normalize_javascript configuration option -is true. +is true or enhanced JavaScript normalizer is enabled. 119:112 (http_inspect) SWF file zlib decompression failure @@ -13496,19 +13486,6 @@ traffic. The HTTP message body is gzip encoded and the FEXTRA flag is set in the gzip header. -119:279 (http_inspect) nested unescape functions in JavaScript code - -Detected nesting of unescape functions(unescape, decodeURI, -decodeURIComponent) in JavaScript code. Indicates that this code most -likely has more than one level of obfuscation. This alert is raised -by the enhanced JavaScript normalizer. - -119:280 (http_inspect) mixing of escape formats in JavaScript code - -Detected more than one encoding within unescape function call -arguments in JavaScript code. This alert is raised by the enhanced -JavaScript normalizer. - 121:1 (http2_inspect) invalid flag set on HTTP/2 frame Invalid flag set on HTTP/2 frame header @@ -15185,6 +15162,8 @@ a reserved value * host_cache.delete_client(host_ip, id, service, version): delete client from host * host_cache.get_stats(): get current host cache usage and pegs + * network.set_policy(id): set the network policy for commands given + the user policy id * packet_capture.enable(filter, group): dump raw packets * packet_capture.disable(): stop packet dump * packet_tracer.enable(proto, src_ip, src_port, dst_ip, dst_port): @@ -15965,18 +15944,10 @@ and are not applicable elsewhere. processing based on address space * policy_selector::tenant_selector: configure traffic processing based on tenants - * search_engine::ac_banded: Aho-Corasick Banded (high memory, - moderate performance) * search_engine::ac_bnfa: Aho-Corasick Binary NFA (low memory, high performance) MPSE * search_engine::ac_full: Aho-Corasick Full (high memory, best performance), implements search_all() - * search_engine::ac_sparse: Aho-Corasick Sparse (high memory, - moderate performance) MPSE - * search_engine::ac_sparse_bands: Aho-Corasick Sparse-Banded (high - memory, moderate performance) MPSE - * search_engine::ac_std: Aho-Corasick Full (high memory, best - performance) MPSE * search_engine::hyperscan: intel hyperscan-based mpse with regex support * search_engine::lowmem: Keyword Trie (low memory, moderate diff --git a/doc/upgrade/snort_upgrade.text b/doc/upgrade/snort_upgrade.text index 36c0b44c6..2fc46ebf7 100644 --- a/doc/upgrade/snort_upgrade.text +++ b/doc/upgrade/snort_upgrade.text @@ -8,7 +8,7 @@ Snort 3 Upgrade Manual The Snort Team Revision History -Revision 3.1.26.0 2022-03-23 13:19:07 EDT TST +Revision 3.1.27.0 2022-04-07 13:35:21 EDT TST --------------------------------------------------------------------- @@ -867,17 +867,17 @@ change -> daq: 'config daq:' ==> 'name' change -> daq_mode: 'config daq_mode:' ==> 'mode' change -> daq_var: 'config daq_var:' ==> 'variables' change -> detection: 'ac' ==> 'ac_full' -change -> detection: 'ac-banded' ==> 'ac_banded' +change -> detection: 'ac-banded' ==> 'ac_full' change -> detection: 'ac-bnfa' ==> 'ac_bnfa' change -> detection: 'ac-bnfa-nq' ==> 'ac_bnfa' change -> detection: 'ac-bnfa-q' ==> 'ac_bnfa' change -> detection: 'ac-nq' ==> 'ac_full' change -> detection: 'ac-q' ==> 'ac_full' -change -> detection: 'ac-sparsebands' ==> 'ac_sparse_bands' +change -> detection: 'ac-sparsebands' ==> 'ac_full' change -> detection: 'ac-split' ==> 'ac_full' change -> detection: 'ac-split' ==> 'split_any_any' -change -> detection: 'ac-std' ==> 'ac_std' -change -> detection: 'acs' ==> 'ac_sparse' +change -> detection: 'ac-std' ==> 'ac_full' +change -> detection: 'acs' ==> 'ac_full' change -> detection: 'bleedover-port-limit' ==> 'bleedover_port_limit' change -> detection: 'debug-print-fast-pattern' ==> 'show_fast_patterns' change -> detection: 'intel-cpm' ==> 'hyperscan' @@ -886,7 +886,6 @@ change -> detection: 'lowmem-q' ==> 'lowmem' change -> detection: 'max-pattern-len' ==> 'max_pattern_len' change -> detection: 'no_stream_inserts' ==> 'detect_raw_tcp' change -> detection: 'search-method' ==> 'search_method' -change -> detection: 'search-optimize' ==> 'search_optimize' change -> detection: 'split-any-any' ==> 'split_any_any = true by default' change -> detection: 'split-any-any' ==> 'split_any_any' change -> dnp3: 'ports' ==> 'bindings' @@ -1069,10 +1068,12 @@ deleted -> config 'sfalert_unified2' deleted -> config 'sflog_unified2' deleted -> config 'sidechannel' deleted -> config 'so_rule_memcap' +deleted -> config 'stateful' deleted -> csv: ' can no longer be specific' deleted -> csv: 'default' deleted -> csv: 'trheader' deleted -> detection: 'mwm' +deleted -> detection: 'search-optimize is always true' deleted -> dnp3: 'disabled' deleted -> dnp3: 'memcap' deleted -> dns: 'enable_experimental_types' diff --git a/doc/user/snort_user.text b/doc/user/snort_user.text index d3372ca50..f175785f2 100644 --- a/doc/user/snort_user.text +++ b/doc/user/snort_user.text @@ -8,7 +8,7 @@ Snort 3 User Manual The Snort Team Revision History -Revision 3.1.26.0 2022-03-23 13:19:07 EDT TST +Revision 3.1.27.0 2022-04-07 13:35:21 EDT TST --------------------------------------------------------------------- @@ -3885,13 +3885,18 @@ Normalizer, Legacy Normalizer will be removed. Having ips option js_data in the rules automatically enables Enhanced Normalizer. The Enhanced Normalizer can normalize inline/external scripts. It supports scripts over multiple PDUs. It is a stateful -JavaScript whitespace and identifiers normalizer. All JavaScript -identifier names, except those from the ignore list, will be -substituted with unified names in the following format: var_0000 → -var_ffff. Moreover, Normalizer validates the syntax concerning -ECMA-262 Standard, including scope tracking and restrictions for -script elements. For more information on how additionally configure -Enhanced Normalizer check with the following configuration options: +JavaScript whitespace and identifiers normalizer. Normalizer +concatenates string literals whenever it’s possible to do. This also +works with any other normalizations that result in string literals. +All JavaScript identifier names, except those from the ignore list, +will be substituted with unified names in the following format: +var_0000 → var_ffff. But the unescape-like function names will be +removed from the normalized data. The Normalizer tries to expand an +escaped text, so it will appear in a usual form in the output. +Moreover, Normalizer validates the syntax concerning ECMA-262 +Standard, including scope tracking and restrictions for script +elements. For more information on how additionally configure Enhanced +Normalizer check with the following configuration options: js_norm_bytes_depth, js_norm_identifier_depth, js_norm_max_tmpl_nest, js_norm_max_bracket_depth, js_norm_max_scope_depth, js_norm_ident_ignore. Eventually Enhanced Normalizer will completely