Merge pull request #3018 in SNORT/snort3 from ~MSTEPANE/snort3:build_3.1.10.0 to...

Squashed commit of the following:

commit 806d5a1481672022a70738ee17506341f1c09ceb
Author: Mike Stepanek <mstepane@cisco.com>
Date:   Wed Aug 11 07:48:19 2021 -0400

    build: generate and tag 3.1.10.0

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 4c349f5a2e07a7e7a59ef3c323f739418d33d804..cf1f337eac34739173ea59a01c9e4cb62a558201 100644 (file)
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -3,7 +3,7 @@ project (snort CXX C)
 
 set (VERSION_MAJOR 3)
 set (VERSION_MINOR 1)
-set (VERSION_PATCH 9)
+set (VERSION_PATCH 10)
 set (VERSION_SUBLEVEL 0)
 set (VERSION "${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}.${VERSION_SUBLEVEL}")
 

diff --git a/ChangeLog b/ChangeLog
index 160e2198a98d07db4ad2f272ad1b52e08f401712..edcec6ba6dffec31b35fdcbcb84361d23b254e3f 100644 (file)
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,22 @@
+2021/08/11 - 3.1.10.0
+
+appid: update netbios-ss (SMB) detector to extract SMB domain from SMBv2, and more intelligently handle payload appid detection
+appid: use packet thread odp context while creating SIP session
+build: install DAQ modules and Snort plugins in separate folders
+dce_smb: restore file tracker size post deletion
+dns: add DNS splitter
+doc: update user manual for identifier normalization
+file_api: add infra and file debugs to existing debugging framework
+ftp: remove unused defines and crufty comments
+http_inspect: add JavaScript identifiers normalization
+http_inspect: change the default value of request_body_app_detection config parameter to true
+smtp: remove unused defines
+ssh: handle traffic with invalid version string
+ssh: handle version string packets that also contain key exchange data
+stream_tcp: skip unordered segments if last flushed position already moved past
+telnet: correct help for ayt_attack_thresh
+wizard: add wizard max_pattern option and update HTTP/SIP aware methods patterns
+
 2021/07/28 - 3.1.9.0
 
 actions: allow session data to stay accessible for loggers for reject rule action

diff --git a/doc/reference/snort_reference.text b/doc/reference/snort_reference.text
index 9a4d7db3bc98530de798db887634e747138d293f..48e97c58937c70d18b396e736882780e7a87d124 100644 (file)
--- a/doc/reference/snort_reference.text
+++ b/doc/reference/snort_reference.text
@@ -8,7 +8,7 @@ Snort 3 Reference Manual
 The Snort Team
 
 Revision History
-Revision 3.1.9.0 2021-07-28 06:22:26 EDT TST
+Revision 3.1.10.0 2021-08-11 07:53:39 EDT TST
 
 ---------------------------------------------------------------------
 
@@ -1636,6 +1636,7 @@ Configuration:
   * int trace.modules.all: enable trace for all modules { 0:255 }
   * int trace.modules.dce_smb.all: enable all trace options { 0:255 }
   * int trace.modules.dpx.all: enable all trace options { 0:255 }
+  * int trace.modules.file_id.all: enable all trace options { 0:255 }
   * int trace.modules.snort.all: enable all trace options { 0:255 }
   * int trace.modules.snort.inspector_manager: enable inspector
     manager trace logging { 0:255 }
@@ -3667,9 +3668,11 @@ Configuration:
     immediately upon script end
   * bool http_inspect.normalize_javascript = false: use legacy
     normalizer to normalize JavaScript in response bodies
-  * int http_inspect.js_normalization_depth = 0: number of input
-    JavaScript bytes to normalize with enhanced normalizer (-1 max
-    allowed value) (experimental) { -1:max53 }
+  * int http_inspect.js_normalization_depth = 0: enable enhanced
+    normalizer (0 is disabled); number of input JavaScript bytes to
+    normalize (-1 unlimited) (experimental) { -1:max53 }
+  * int http_inspect.js_norm_identifier_depth = 260000: max number of
+    unique JavaScript identifiers to normalize { 0:260000 }
   * int http_inspect.max_javascript_whitespaces = 200: maximum
     consecutive whitespaces allowed within the JavaScript obfuscated
     data { 1:65535 }
@@ -3705,7 +3708,7 @@ Configuration:
   * string http_inspect.xff_headers = x-forwarded-for true-client-ip:
     specifies the xff type headers to parse and consider in the same
     order of preference as defined
-  * bool http_inspect.request_body_app_detection = false: make HTTP/2
+  * bool http_inspect.request_body_app_detection = true: make HTTP/2
     request message bodies available for application detection
     (detection requires AppId)
 
@@ -3850,6 +3853,8 @@ Rules:
   * 119:268 (http_inspect) JavaScript code under the external script
     tags
   * 119:269 (http_inspect) script opening tag in a short form
+  * 119:270 (http_inspect) max number of unique JavaScript
+    identifiers reached
 
 Peg counts:
 
@@ -3905,6 +3910,10 @@ Peg counts:
     JavaScripts processed (sum)
   * http_inspect.js_bytes: total number of JavaScript bytes processed
     (sum)
+  * http_inspect.js_identifiers: total number of unique JavaScript
+    identifiers processed (sum)
+  * http_inspect.js_identifier_overflows: total number of unique
+    JavaScript identifier limit overflows (sum)
 
 
 5.25. iec104
@@ -5681,8 +5690,8 @@ Instance Type: multiton
 
 Configuration:
 
-  * int telnet.ayt_attack_thresh = -1: alert on this number of
-    consecutive Telnet AYT commands { -1:max31 }
+  * int telnet.ayt_attack_thresh = -1: alert beyond this number of
+    consecutive Telnet AYT commands (-1 is disabled) { -1:max31 }
   * bool telnet.check_encrypted = false: check for end of encryption
   * bool telnet.encrypted_traffic = false: check for encrypted Telnet
   * bool telnet.normalize = false: eliminate escape sequences
@@ -5737,6 +5746,8 @@ Configuration:
     wild cards (*)
   * multi wizard.curses: enable service identification based on
     internal algorithm { dce_smb | dce_udp | dce_tcp | sslv2 }
+  * int wizard.max_pattern = 64: maximum scan depth per segment (0 is
+    unlimited) { 0:65535 }
 
 Peg counts:
 
@@ -9133,9 +9144,11 @@ these libraries see the Getting Started section of the manual.
     mapping to normalize characters
   * string http_inspect.iis_unicode_map_file: file containing code
     points for IIS unicode. { (optional) }
-  * int http_inspect.js_normalization_depth = 0: number of input
-    JavaScript bytes to normalize with enhanced normalizer (-1 max
-    allowed value) (experimental) { -1:max53 }
+  * int http_inspect.js_normalization_depth = 0: enable enhanced
+    normalizer (0 is disabled); number of input JavaScript bytes to
+    normalize (-1 unlimited) (experimental) { -1:max53 }
+  * int http_inspect.js_norm_identifier_depth = 260000: max number of
+    unique JavaScript identifiers to normalize { 0:260000 }
   * int http_inspect.max_javascript_whitespaces = 200: maximum
     consecutive whitespaces allowed within the JavaScript obfuscated
     data { 1:65535 }
@@ -9149,7 +9162,7 @@ these libraries see the Getting Started section of the manual.
     encodings
   * bool http_inspect.plus_to_space = true: replace + with <sp> when
     normalizing URIs
-  * bool http_inspect.request_body_app_detection = false: make HTTP/2
+  * bool http_inspect.request_body_app_detection = true: make HTTP/2
     request message bodies available for application detection
     (detection requires AppId)
   * int http_inspect.request_depth = -1: maximum request message body
@@ -10352,8 +10365,8 @@ these libraries see the Getting Started section of the manual.
   * string tcp_connector[].connector: connector name
   * enum tcp_connector[].setup: stream establishment { call | answer
     }
-  * int telnet.ayt_attack_thresh = -1: alert on this number of
-    consecutive Telnet AYT commands { -1:max31 }
+  * int telnet.ayt_attack_thresh = -1: alert beyond this number of
+    consecutive Telnet AYT commands (-1 is disabled) { -1:max31 }
   * bool telnet.check_encrypted = false: check for end of encryption
   * bool telnet.encrypted_traffic = false: check for encrypted Telnet
   * bool telnet.normalize = false: eliminate escape sequences
@@ -10370,6 +10383,7 @@ these libraries see the Getting Started section of the manual.
   * int trace.modules.all: enable trace for all modules { 0:255 }
   * int trace.modules.dce_smb.all: enable all trace options { 0:255 }
   * int trace.modules.dpx.all: enable all trace options { 0:255 }
+  * int trace.modules.file_id.all: enable all trace options { 0:255 }
   * int trace.modules.snort.all: enable all trace options { 0:255 }
   * int trace.modules.snort.inspector_manager: enable inspector
     manager trace logging { 0:255 }
@@ -10407,6 +10421,8 @@ these libraries see the Getting Started section of the manual.
     chars (?)
   * string wizard.hexes[].to_server[].hex: sequence of data with wild
     chars (?)
+  * int wizard.max_pattern = 64: maximum scan depth per segment (0 is
+    unlimited) { 0:65535 }
   * bool wizard.spells[].client_first = true: which end initiates
     data transfer
   * select wizard.spells[].proto = tcp: protocol to scan { tcp | udp
@@ -10966,6 +10982,10 @@ these libraries see the Getting Started section of the manual.
     (sum)
   * http_inspect.js_external_scripts: total number of external
     JavaScripts processed (sum)
+  * http_inspect.js_identifier_overflows: total number of unique
+    JavaScript identifier limit overflows (sum)
+  * http_inspect.js_identifiers: total number of unique JavaScript
+    identifiers processed (sum)
   * http_inspect.js_inline_scripts: total number of inline
     JavaScripts processed (sum)
   * http_inspect.max_concurrent_sessions: maximum concurrent http
@@ -11954,6 +11974,8 @@ these libraries see the Getting Started section of the manual.
   * 119:268 (http_inspect) JavaScript code under the external script
     tags
   * 119:269 (http_inspect) script opening tag in a short form
+  * 119:270 (http_inspect) max number of unique JavaScript
+    identifiers reached
   * 121:1 (http2_inspect) invalid flag set on HTTP/2 frame
   * 121:2 (http2_inspect) HPACK integer value has leading zeros
   * 121:3 (http2_inspect) HTTP/2 stream initiated with invalid stream

diff --git a/doc/upgrade/snort_upgrade.text b/doc/upgrade/snort_upgrade.text
index a00f298b4c70e258905513bf26e0cca6e331e7f4..2759a777db8f32b5dcf5af889e9335f179357fef 100644 (file)
--- a/doc/upgrade/snort_upgrade.text
+++ b/doc/upgrade/snort_upgrade.text
@@ -8,7 +8,7 @@ Snort 3 Upgrade Manual
 The Snort Team
 
 Revision History
-Revision 3.1.9.0 2021-07-28 06:22:15 EDT TST
+Revision 3.1.10.0 2021-08-11 07:53:28 EDT TST
 
 ---------------------------------------------------------------------
 

diff --git a/doc/user/snort_user.text b/doc/user/snort_user.text
index c6b642ae13f81260b34e964e396469e81ed5f677..76bffaa9ac02bc141d5e0e67e0bd103cdd52245c 100644 (file)
--- a/doc/user/snort_user.text
+++ b/doc/user/snort_user.text
@@ -8,7 +8,7 @@ Snort 3 User Manual
 The Snort Team
 
 Revision History
-Revision 3.1.9.0 2021-07-28 06:22:15 EDT TST
+Revision 3.1.10.0 2021-08-11 07:53:28 EDT TST
 
 ---------------------------------------------------------------------
 
@@ -3895,11 +3895,30 @@ means that normalizer is disabled. The enhanced normalizer provides
 more precise whitespace normalization of JavaScript, that removes all
 redundant whitespaces and line terminators from the JavaScript syntax
 point of view (between identifier and punctuator, between identifier
-and operator, etc.) according to ECMAScript 5.1 standard. The
-normalized data is available through the script_data rule option.
-This is currently experimental and still under development.
-
-5.10.2.9. xff_headers
+and operator, etc.) according to ECMAScript 5.1 standard.
+Additionally, it performs normalization of JavaScript identifiers
+making a substitution of unique names with unified names
+representation: a0 → z9999. The identifiers are variables and
+function names. The normalized data is available through the
+script_data rule option. This is currently experimental and still
+under development.
+
+5.10.2.9. js_norm_identifier_depth
+
+js_norm_identifier_depth = N {0 : 260000} will set a number of unique
+JavaScript identifiers to normalize. When the depth is reached, a
+built-in alert is generated. It’s implemented per HTTP transaction
+(request/response), so the context of identifier substitutions is
+shared between all the scripts in the payload. By default, the value
+is set to 260000, which is the max allowed number of unique
+identifiers. The generated names are in the range from a0 to z9999.
+Thus, the number of unique identifiers cannot be greater than 26 *
+10000 = 260000. This option takes effect only if
+js_normalization_depth is set to a non-zero value, enabling the
+enhanced normalizer. This is currently experimental and still under
+development.
+
+5.10.2.10. xff_headers
 
 This configuration supports defining custom x-forwarded-for type
 headers. In a multi-vendor world, it is quite possible that the
@@ -3914,7 +3933,7 @@ they are defined, e.g "x-forwarded-for" will be preferred than
 "true-client-ip" if both headers are present in the stream. The
 header names should be delimited by a space.
 
-5.10.2.10. URI processing
+5.10.2.11. URI processing
 
 Normalization and inspection of the URI in the HTTP request message
 is a key aspect of what http_inspect does. The best way to normalize