Pull request #3367: build: generate and tag 3.1.27.0

Merge in SNORT/snort3 from ~MSTEPANE/snort3:build_3.1.27.0 to master

Squashed commit of the following:

commit 5431b622172ee145af2dbbe6889e87764669d7f1
Author: Mike Stepanek <mstepane@cisco.com>
Date:   Thu Apr 7 13:27:04 2022 -0400

    build: generate and tag 3.1.27.0

diff --git a/CMakeLists.txt b/CMakeLists.txt
index c761523218d4fa35dadd565a7e1b38d76352eb83..d3b667a5abc643b42e7f1531bacecc9adcdd592d 100644 (file)
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -3,7 +3,7 @@ project (snort CXX C)
 
 set (VERSION_MAJOR 3)
 set (VERSION_MINOR 1)
-set (VERSION_PATCH 26)
+set (VERSION_PATCH 27)
 set (VERSION_SUBLEVEL 0)
 set (VERSION "${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}.${VERSION_SUBLEVEL}")
 

diff --git a/ChangeLog b/ChangeLog
index 712e8d70d35f07a1b865605abe158ca59e4dcb25..4c5d8f2de18c535eea83a04c3822f8452d52e72d 100644 (file)
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,50 @@
+2022/04/07 - 3.1.27.0
+
+ac_full: refactor api access
+ac_full: remove cruft
+ac_std: fix case translation buffer size
+alerts: remove obsolete stateful parameter
+appid: provide client appid set by encrypted visibility engine to ssl through the ssl appid lookup api
+build: compile against libatomic if present. Thanks to W. Michael Petullo <mike@flyn.org>
+control, shell: add a command to set the network policy to be used by subsequent commands
+dce_rpc: handle cleanup path and race conditions for dce traffic
+detection: do not check ips policy when builtin events are queued
+detection: fixup dump of detection option tree
+detection: minor refactoring of rule header access
+detection: override match queue limit for offload
+detection: remove cruft
+detection: skip match deduplication for hyperscan
+file_api: handle user_file_data cleanup
+hext: change stdin designation from tty to - since the trough uses dash
+http2_inspect: reduce holes in objects
+http_inspect: add unescape text processing for Enhanced JS Normalizer
+http_inspect: decode String.fromCodePoint() JavaScript function
+http_inspect: delete alerts 119:279 and 119:280
+http_inspect: provide current packet to trace
+http_inspect: support headers Restrict-Access-To-Tenants, Restrict-Access-Context
+hyperscan: ensure adequate scratch when deserializing
+rate_filter: move to inspection policy
+search_engine: add fast pattern only count at startup
+search_engine: always build ac_full since it is a hard default case
+search_engine: fix .debug = true output
+search_engine: fix adjustment for fast_pattern_offset
+search_engine: fix fast pattern only eligibility check
+search_engine: remove obsolete warning on max_pattern_len change
+search_engine: remove search_optimize parameter (always true)
+search_engine: truncated patterns not eligible as fast pattern only contents
+search_engines: add and refactor unit tests
+search_engines: ensure SearchTool with hyperscan gets multi-match mode
+search_engines: remove the legacy ac_banded algorithm
+search_engines: remove the legacy ac_sparse algorithm
+search_engines: remove the legacy ac_sparse_bands algorithm
+search_engines: remove the legacy ac_std algorithm
+sfip: suppress compiler warning
+utils: add string concatenation for Enchanced JS Normalizer
+utils: allow opening/closing tags in external scripts
+utils: fix JS Normalizer benchmark build
+utils: fix tracking variable when the output buffer is reset
+utils: harden script opening tag sequence
+
 2022/03/23 - 3.1.26.0
 
 actions: revert bf62a22d43bb2d15b7425c5ec3e3118ead470e8d

diff --git a/doc/reference/snort_reference.text b/doc/reference/snort_reference.text
index 417947e4528c4cd2fb0063c8d935da9db47c90b1..24eb4c51a4df6c0e641f17aa656512748544013b 100644 (file)
--- a/doc/reference/snort_reference.text
+++ b/doc/reference/snort_reference.text
@@ -8,7 +8,7 @@ Snort 3 Reference Manual
 The Snort Team
 
 Revision History
-Revision 3.1.26.0 2022-03-23 13:19:21 EDT TST
+Revision 3.1.27.0 2022-04-07 13:35:35 EDT TST
 
 ---------------------------------------------------------------------
 
@@ -423,8 +423,6 @@ Configuration:
     memory for rate_filters { 0:max32 }
   * string alerts.reference_net: set the CIDR for homenet (for use
     with -l or -B, does NOT change $HOME_NET in IDS mode)
-  * bool alerts.stateful = false: don’t alert w/o established session
-    (note: rule action still taken)
   * string alerts.tunnel_verdicts: let DAQ handle non-allow verdicts
     for gtp|teredo|6in4|4in6|4in4|6in6|gre|mpls|vxlan traffic
 
@@ -1062,6 +1060,11 @@ Configuration:
     Snort will process for a given packet before raising 116:293 (0 =
     unlimited) { 0:255 }
 
+Commands:
+
+  * network.set_policy(id): set the network policy for commands given
+    the user policy id
+
 
 2.20. output
 
@@ -1248,7 +1251,7 @@ Help: configure rate filters (which change rule actions)
 
 Type: basic
 
-Usage: context
+Usage: inspect
 
 Configuration:
 
@@ -1322,17 +1325,13 @@ Configuration:
   * bool search_engine.detect_raw_tcp = false: detect on TCP payload
     before reassembly
   * dynamic search_engine.search_method = ac_bnfa: set fast pattern
-    algorithm - choose available search engine { ac_banded | ac_bnfa
-    | ac_full | ac_sparse | ac_sparse_bands | ac_std | hyperscan |
-    lowmem }
-  * dynamic search_engine.offload_search_method: set fast pattern
-    offload algorithm - choose available search engine { ac_banded |
-    ac_bnfa | ac_full | ac_sparse | ac_sparse_bands | ac_std |
+    algorithm - choose available search engine { ac_bnfa | ac_full |
     hyperscan | lowmem }
+  * dynamic search_engine.offload_search_method: set fast pattern
+    offload algorithm - choose available search engine { ac_bnfa |
+    ac_full | hyperscan | lowmem }
   * string search_engine.rule_db_dir: deserialize rule databases from
     given directory
-  * bool search_engine.search_optimize = true: tweak state machine
-    construction for better performance
   * bool search_engine.show_fast_patterns = false: print fast pattern
     info for each rule
   * bool search_engine.split_any_any = true: evaluate any-any rules
@@ -3974,10 +3973,6 @@ Rules:
   * 119:277 (http_inspect) HTTP version in start line is higher than
     1
   * 119:278 (http_inspect) HTTP gzip body with the FEXTRA flag set
-  * 119:279 (http_inspect) nested unescape functions in JavaScript
-    code
-  * 119:280 (http_inspect) mixing of escape formats in JavaScript
-    code
 
 Peg counts:
 
@@ -8792,8 +8787,6 @@ these libraries see the Getting Started section of the manual.
     memory for rate_filters { 0:max32 }
   * string alerts.reference_net: set the CIDR for homenet (for use
     with -l or -B, does NOT change $HOME_NET in IDS mode)
-  * bool alerts.stateful = false: don’t alert w/o established session
-    (note: rule action still taken)
   * string alerts.tunnel_verdicts: let DAQ handle non-allow verdicts
     for gtp|teredo|6in4|4in6|4in4|6in6|gre|mpls|vxlan traffic
   * enum alert_syslog.facility = auth: part of priority applied to
@@ -10196,19 +10189,15 @@ these libraries see the Getting Started section of the manual.
   * int search_engine.max_queue_events = 5: maximum number of
     matching fast pattern states to queue per packet { 2:100 }
   * dynamic search_engine.offload_search_method: set fast pattern
-    offload algorithm - choose available search engine { ac_banded |
-    ac_bnfa | ac_full | ac_sparse | ac_sparse_bands | ac_std |
-    hyperscan | lowmem }
+    offload algorithm - choose available search engine { ac_bnfa |
+    ac_full | hyperscan | lowmem }
   * int search_engine.queue_limit = 0: maximum number of fast pattern
     matches to queue per packet (0 is unlimited) { 0:max32 }
   * string search_engine.rule_db_dir: deserialize rule databases from
     given directory
   * dynamic search_engine.search_method = ac_bnfa: set fast pattern
-    algorithm - choose available search engine { ac_banded | ac_bnfa
-    | ac_full | ac_sparse | ac_sparse_bands | ac_std | hyperscan |
-    lowmem }
-  * bool search_engine.search_optimize = true: tweak state machine
-    construction for better performance
+    algorithm - choose available search engine { ac_bnfa | ac_full |
+    hyperscan | lowmem }
   * bool search_engine.show_fast_patterns = false: print fast pattern
     info for each rule
   * bool search_engine.split_any_any = true: evaluate any-any rules
@@ -12970,7 +12959,8 @@ HTTP response has Content-Type charset=utf-7.
 119:109 (http_inspect) more than one level of JavaScript obfuscation
 
 More than one level of JavaScript obfuscation. This alert can only be
-generated when normalize_javascript configuration option is true.
+generated when normalize_javascript configuration option is true or
+enhanced JavaScript normalizer is enabled.
 
 119:110 (http_inspect) consecutive JavaScript whitespaces exceed
 maximum allowed
@@ -12984,7 +12974,7 @@ obfuscated data
 
 More than one encoding within JavaScript obfuscated data. This alert
 can only be generated when normalize_javascript configuration option
-is true.
+is true or enhanced JavaScript normalizer is enabled.
 
 119:112 (http_inspect) SWF file zlib decompression failure
 
@@ -13496,19 +13486,6 @@ traffic.
 The HTTP message body is gzip encoded and the FEXTRA flag is set in
 the gzip header.
 
-119:279 (http_inspect) nested unescape functions in JavaScript code
-
-Detected nesting of unescape functions(unescape, decodeURI,
-decodeURIComponent) in JavaScript code. Indicates that this code most
-likely has more than one level of obfuscation. This alert is raised
-by the enhanced JavaScript normalizer.
-
-119:280 (http_inspect) mixing of escape formats in JavaScript code
-
-Detected more than one encoding within unescape function call
-arguments in JavaScript code. This alert is raised by the enhanced
-JavaScript normalizer.
-
 121:1 (http2_inspect) invalid flag set on HTTP/2 frame
 
 Invalid flag set on HTTP/2 frame header
@@ -15185,6 +15162,8 @@ a reserved value
   * host_cache.delete_client(host_ip, id, service, version): delete
     client from host
   * host_cache.get_stats(): get current host cache usage and pegs
+  * network.set_policy(id): set the network policy for commands given
+    the user policy id
   * packet_capture.enable(filter, group): dump raw packets
   * packet_capture.disable(): stop packet dump
   * packet_tracer.enable(proto, src_ip, src_port, dst_ip, dst_port):
@@ -15965,18 +15944,10 @@ and are not applicable elsewhere.
     processing based on address space
   * policy_selector::tenant_selector: configure traffic processing
     based on tenants
-  * search_engine::ac_banded: Aho-Corasick Banded (high memory,
-    moderate performance)
   * search_engine::ac_bnfa: Aho-Corasick Binary NFA (low memory, high
     performance) MPSE
   * search_engine::ac_full: Aho-Corasick Full (high memory, best
     performance), implements search_all()
-  * search_engine::ac_sparse: Aho-Corasick Sparse (high memory,
-    moderate performance) MPSE
-  * search_engine::ac_sparse_bands: Aho-Corasick Sparse-Banded (high
-    memory, moderate performance) MPSE
-  * search_engine::ac_std: Aho-Corasick Full (high memory, best
-    performance) MPSE
   * search_engine::hyperscan: intel hyperscan-based mpse with regex
     support
   * search_engine::lowmem: Keyword Trie (low memory, moderate

diff --git a/doc/upgrade/snort_upgrade.text b/doc/upgrade/snort_upgrade.text
index 36c0b44c625c96c0e47bb818c52de7d996d72080..2fc46ebf7a5431bb8a0aedc61df63d5a71f8fab3 100644 (file)
--- a/doc/upgrade/snort_upgrade.text
+++ b/doc/upgrade/snort_upgrade.text
@@ -8,7 +8,7 @@ Snort 3 Upgrade Manual
 The Snort Team
 
 Revision History
-Revision 3.1.26.0 2022-03-23 13:19:07 EDT TST
+Revision 3.1.27.0 2022-04-07 13:35:21 EDT TST
 
 ---------------------------------------------------------------------
 
@@ -867,17 +867,17 @@ change -> daq: 'config daq:' ==> 'name'
 change -> daq_mode: 'config daq_mode:' ==> 'mode'
 change -> daq_var: 'config daq_var:' ==> 'variables'
 change -> detection: 'ac' ==> 'ac_full'
-change -> detection: 'ac-banded' ==> 'ac_banded'
+change -> detection: 'ac-banded' ==> 'ac_full'
 change -> detection: 'ac-bnfa' ==> 'ac_bnfa'
 change -> detection: 'ac-bnfa-nq' ==> 'ac_bnfa'
 change -> detection: 'ac-bnfa-q' ==> 'ac_bnfa'
 change -> detection: 'ac-nq' ==> 'ac_full'
 change -> detection: 'ac-q' ==> 'ac_full'
-change -> detection: 'ac-sparsebands' ==> 'ac_sparse_bands'
+change -> detection: 'ac-sparsebands' ==> 'ac_full'
 change -> detection: 'ac-split' ==> 'ac_full'
 change -> detection: 'ac-split' ==> 'split_any_any'
-change -> detection: 'ac-std' ==> 'ac_std'
-change -> detection: 'acs' ==> 'ac_sparse'
+change -> detection: 'ac-std' ==> 'ac_full'
+change -> detection: 'acs' ==> 'ac_full'
 change -> detection: 'bleedover-port-limit' ==> 'bleedover_port_limit'
 change -> detection: 'debug-print-fast-pattern' ==> 'show_fast_patterns'
 change -> detection: 'intel-cpm' ==> 'hyperscan'
@@ -886,7 +886,6 @@ change -> detection: 'lowmem-q' ==> 'lowmem'
 change -> detection: 'max-pattern-len' ==> 'max_pattern_len'
 change -> detection: 'no_stream_inserts' ==> 'detect_raw_tcp'
 change -> detection: 'search-method' ==> 'search_method'
-change -> detection: 'search-optimize' ==> 'search_optimize'
 change -> detection: 'split-any-any' ==> 'split_any_any = true by default'
 change -> detection: 'split-any-any' ==> 'split_any_any'
 change -> dnp3: 'ports' ==> 'bindings'
@@ -1069,10 +1068,12 @@ deleted -> config 'sfalert_unified2'
 deleted -> config 'sflog_unified2'
 deleted -> config 'sidechannel'
 deleted -> config 'so_rule_memcap'
+deleted -> config 'stateful'
 deleted -> csv: '<filename> can no longer be specific'
 deleted -> csv: 'default'
 deleted -> csv: 'trheader'
 deleted -> detection: 'mwm'
+deleted -> detection: 'search-optimize is always true'
 deleted -> dnp3: 'disabled'
 deleted -> dnp3: 'memcap'
 deleted -> dns: 'enable_experimental_types'

diff --git a/doc/user/snort_user.text b/doc/user/snort_user.text
index d3372ca50e39398116ec424a311847a39235f3e4..f175785f259f05849a2d52ea635c8cf3d9158f1a 100644 (file)
--- a/doc/user/snort_user.text
+++ b/doc/user/snort_user.text
@@ -8,7 +8,7 @@ Snort 3 User Manual
 The Snort Team
 
 Revision History
-Revision 3.1.26.0 2022-03-23 13:19:07 EDT TST
+Revision 3.1.27.0 2022-04-07 13:35:21 EDT TST
 
 ---------------------------------------------------------------------
 
@@ -3885,13 +3885,18 @@ Normalizer, Legacy Normalizer will be removed.
 Having ips option js_data in the rules automatically enables Enhanced
 Normalizer. The Enhanced Normalizer can normalize inline/external
 scripts. It supports scripts over multiple PDUs. It is a stateful
-JavaScript whitespace and identifiers normalizer. All JavaScript
-identifier names, except those from the ignore list, will be
-substituted with unified names in the following format: var_0000 →
-var_ffff. Moreover, Normalizer validates the syntax concerning
-ECMA-262 Standard, including scope tracking and restrictions for
-script elements. For more information on how additionally configure
-Enhanced Normalizer check with the following configuration options:
+JavaScript whitespace and identifiers normalizer. Normalizer
+concatenates string literals whenever it’s possible to do. This also
+works with any other normalizations that result in string literals.
+All JavaScript identifier names, except those from the ignore list,
+will be substituted with unified names in the following format:
+var_0000 → var_ffff. But the unescape-like function names will be
+removed from the normalized data. The Normalizer tries to expand an
+escaped text, so it will appear in a usual form in the output.
+Moreover, Normalizer validates the syntax concerning ECMA-262
+Standard, including scope tracking and restrictions for script
+elements. For more information on how additionally configure Enhanced
+Normalizer check with the following configuration options:
 js_norm_bytes_depth, js_norm_identifier_depth, js_norm_max_tmpl_nest,
 js_norm_max_bracket_depth, js_norm_max_scope_depth,
 js_norm_ident_ignore. Eventually Enhanced Normalizer will completely