From: Russ Combs (rucombs) Date: Sat, 17 Dec 2022 22:13:13 +0000 (+0000) Subject: Pull request #3704: build: generate and tag 3.1.49.0 X-Git-Tag: 3.1.49.0 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=5137fb71d99c5a8ebde3ca652c572d9bc060d152;p=thirdparty%2Fsnort3.git Pull request #3704: build: generate and tag 3.1.49.0 Merge in SNORT/snort3 from ~PRBG/snort3:build_3.1.49.0 to master Squashed commit of the following: commit 98957f0761a73601e6a11f626b8ff975e93c6f7a Author: Priyanka Gurudev Date: Thu Dec 15 16:54:01 2022 -0500 build: generate and tag 3.1.49.0 --- diff --git a/CMakeLists.txt b/CMakeLists.txt index dc1f34934..e27f3578c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -3,7 +3,7 @@ project (snort CXX C) set (VERSION_MAJOR 3) set (VERSION_MINOR 1) -set (VERSION_PATCH 48) +set (VERSION_PATCH 49) set (VERSION_SUBLEVEL 0) set (VERSION "${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}.${VERSION_SUBLEVEL}") diff --git a/ChangeLog.md b/ChangeLog.md index 6ce83ca8e..993cf0286 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,3 +1,19 @@ +2022-12-15: 3.1.49.0 + +* appid: appid_detector_builder.sh addPortPatternService call fixed +* appid: do not reset session data when built-in discovery is not done +* appid: fixed assert condition for odp_ctxt and odp_thread_local_ctxt +* doc: add decompression mention to js_norm reference +* doc: update user/js_norm.txt for PDF in email protocols +* geneve: if daq has the capability, do not bypass geneve tunnel +* ips_options: fix offset related bug in byte_test eval() +* js_norm: add PDF stream processing +* js_norm: add support for email protocols +* js_norm: fix pdf_tokenizer_test on FreeBSD platform +* js_norm: update PDF tokenizer to use glue input streambuf +* stream: ignore PAWS timestamp checks when in no_ack mode +* wizard: remove client_first option + 2022-12-01: 3.1.48.0 * appid: added config for logging alpn service mappings diff --git a/doc/reference/snort_reference.text b/doc/reference/snort_reference.text index 45d0ba95e..0625d9e56 100644 --- a/doc/reference/snort_reference.text +++ b/doc/reference/snort_reference.text @@ -8,7 +8,7 @@ Snort 3 Reference Manual The Snort Team Revision History -Revision 3.1.48.0 2022-12-01 11:51:55 EST TST +Revision 3.1.49.0 2022-12-15 16:32:12 EST TST --------------------------------------------------------------------- @@ -4093,8 +4093,8 @@ Peg counts: JavaScripts processed (sum) * http_inspect.js_external_scripts: total number of external JavaScripts processed (sum) - * http_inspect.js_pdf_scripts: total number of PDF JavaScripts - processed (sum) + * http_inspect.js_pdf_scripts: total number of PDF files processed + (sum) * http_inspect.skip_mime_attach: total number of HTTP requests with too many MIME attachments to inspect (sum) @@ -4292,6 +4292,7 @@ Peg counts: * imap.non_encoded_attachments: total non-encoded attachments extracted (sum) * imap.non_encoded_bytes: total non-encoded extracted bytes (sum) + * imap.js_pdf_scripts: total number of PDF files processed (sum) 5.28. mem_test @@ -4732,6 +4733,7 @@ Peg counts: * pop.non_encoded_attachments: total non-encoded attachments extracted (sum) * pop.non_encoded_bytes: total non-encoded extracted bytes (sum) + * pop.js_pdf_scripts: total number of PDF files processed (sum) 5.37. port_scan @@ -5378,6 +5380,7 @@ Peg counts: * smtp.non_encoded_attachments: total non-encoded attachments extracted (sum) * smtp.non_encoded_bytes: total non-encoded extracted bytes (sum) + * smtp.js_pdf_scripts: total number of PDF files processed (sum) 5.44. so_proxy @@ -5956,8 +5959,6 @@ Configuration: * string wizard.hexes[].service: name of service * select wizard.hexes[].proto = any: protocol to scan { tcp | udp | any } - * bool wizard.hexes[].client_first = true: which end initiates data - transfer (deprecated) * string wizard.hexes[].to_server[].hex: sequence of data with wild chars (?) * string wizard.hexes[].to_client[].hex: sequence of data with wild @@ -5965,8 +5966,6 @@ Configuration: * string wizard.spells[].service: name of service * select wizard.spells[].proto = any: protocol to scan { tcp | udp | any } - * bool wizard.spells[].client_first = true: which end initiates - data transfer (deprecated) * string wizard.spells[].to_server[].spell: sequence of data with wild cards (*) * string wizard.spells[].to_client[].spell: sequence of data with @@ -6188,7 +6187,7 @@ Usage: detect Configuration: * int byte_extract.~count: number of bytes to pick up from the - buffer { 1:10 } + buffer (string can pick less) { 1:10 } * int byte_extract.~offset: number of bytes into the buffer to start processing { -65535:65535 } * string byte_extract.~name: name of the variable that will be used @@ -6223,7 +6222,7 @@ Usage: detect Configuration: * int byte_jump.~count: number of bytes to pick up from the buffer - { 0:10 } + (string can pick less) { 0:10 } * string byte_jump.~offset: variable name or number of bytes into the buffer to start processing * implied byte_jump.relative: offset from cursor instead of start @@ -6262,8 +6261,8 @@ Usage: detect Configuration: - * int byte_math.bytes: number of bytes to pick up from the buffer { - 1:10 } + * int byte_math.bytes: number of bytes to pick up from the buffer + (string can pick less) { 1:10 } * string byte_math.offset: number of bytes into the buffer to start processing * enum byte_math.oper: mathematical operation to perform { +|-|*|/| @@ -6294,7 +6293,7 @@ Usage: detect Configuration: * int byte_test.~count: number of bytes to pick up from the buffer - { 1:10 } + (string can pick less) { 1:10 } * string byte_test.~operator: operation to perform to test the value * string byte_test.~compare: variable name or value to test the @@ -9104,7 +9103,7 @@ libraries see the Getting Started section of the manual. * int byte_extract.bitmask: applies as an AND to the extracted value before storage in name { 0x1:0xFFFFFFFF } * int byte_extract.~count: number of bytes to pick up from the - buffer { 1:10 } + buffer (string can pick less) { 1:10 } * implied byte_extract.dce: dcerpc2 determines endianness * implied byte_extract.dec: convert from decimal string * implied byte_extract.hex: convert from hex string @@ -9125,7 +9124,7 @@ libraries see the Getting Started section of the manual. * int byte_jump.bitmask: applies as an AND prior to evaluation { 0x1:0xFFFFFFFF } * int byte_jump.~count: number of bytes to pick up from the buffer - { 0:10 } + (string can pick less) { 0:10 } * implied byte_jump.dce: dcerpc2 determines endianness * implied byte_jump.dec: convert from decimal string * implied byte_jump.from_beginning: jump from start of buffer @@ -9146,8 +9145,8 @@ libraries see the Getting Started section of the manual. * implied byte_jump.string: convert from string * int byte_math.bitmask: applies as bitwise AND to the extracted value before storage in name { 0x1:0xFFFFFFFF } - * int byte_math.bytes: number of bytes to pick up from the buffer { - 1:10 } + * int byte_math.bytes: number of bytes to pick up from the buffer + (string can pick less) { 1:10 } * implied byte_math.dce: dcerpc2 determines endianness * enum byte_math.endian: specify big/little endian { big|little } * string byte_math.offset: number of bytes into the buffer to start @@ -9167,7 +9166,7 @@ libraries see the Getting Started section of the manual. * string byte_test.~compare: variable name or value to test the converted result against * int byte_test.~count: number of bytes to pick up from the buffer - { 1:10 } + (string can pick less) { 1:10 } * implied byte_test.dce: dcerpc2 determines endianness * implied byte_test.dec: convert from decimal string * implied byte_test.hex: convert from hex string @@ -10962,8 +10961,6 @@ libraries see the Getting Started section of the manual. * multi wizard.curses: enable service identification based on internal algorithm { dce_smb | dce_udp | dce_tcp | mms | s7commplus | sslv2 } - * bool wizard.hexes[].client_first = true: which end initiates data - transfer (deprecated) * select wizard.hexes[].proto = any: protocol to scan { tcp | udp | any } * string wizard.hexes[].service: name of service @@ -10973,8 +10970,6 @@ libraries see the Getting Started section of the manual. chars (?) * int wizard.max_search_depth = 8192: maximum scan depth per flow { 0:65535 } - * bool wizard.spells[].client_first = true: which end initiates - data transfer (deprecated) * select wizard.spells[].proto = any: protocol to scan { tcp | udp | any } * string wizard.spells[].service: name of service @@ -11543,8 +11538,8 @@ libraries see the Getting Started section of the manual. JavaScripts processed (sum) * http_inspect.js_inline_scripts: total number of inline JavaScripts processed (sum) - * http_inspect.js_pdf_scripts: total number of PDF JavaScripts - processed (sum) + * http_inspect.js_pdf_scripts: total number of PDF files processed + (sum) * http_inspect.max_concurrent_sessions: maximum concurrent http sessions (max) * http_inspect.options_requests: OPTIONS requests inspected (sum) @@ -11593,6 +11588,7 @@ libraries see the Getting Started section of the manual. * imap.b64_attachments: total base64 attachments decoded (sum) * imap.b64_decoded_bytes: total base64 decoded bytes (sum) * imap.concurrent_sessions: total concurrent imap sessions (now) + * imap.js_pdf_scripts: total number of PDF files processed (sum) * imap.max_concurrent_sessions: maximum concurrent imap sessions (max) * imap.non_encoded_attachments: total non-encoded attachments @@ -11770,6 +11766,7 @@ libraries see the Getting Started section of the manual. * pop.b64_attachments: total base64 attachments decoded (sum) * pop.b64_decoded_bytes: total base64 decoded bytes (sum) * pop.concurrent_sessions: total concurrent pop sessions (now) + * pop.js_pdf_scripts: total number of PDF files processed (sum) * pop.max_concurrent_sessions: maximum concurrent pop sessions (max) * pop.non_encoded_attachments: total non-encoded attachments @@ -11893,6 +11890,7 @@ libraries see the Getting Started section of the manual. * smtp.b64_attachments: total base64 attachments decoded (sum) * smtp.b64_decoded_bytes: total base64 decoded bytes (sum) * smtp.concurrent_sessions: total concurrent smtp sessions (now) + * smtp.js_pdf_scripts: total number of PDF files processed (sum) * smtp.max_concurrent_sessions: maximum concurrent smtp sessions (max) * smtp.non_encoded_attachments: total non-encoded attachments diff --git a/doc/upgrade/snort_upgrade.text b/doc/upgrade/snort_upgrade.text index 75d97595a..01b2481f4 100644 --- a/doc/upgrade/snort_upgrade.text +++ b/doc/upgrade/snort_upgrade.text @@ -8,7 +8,7 @@ Snort 3 Upgrade Manual The Snort Team Revision History -Revision 3.1.48.0 2022-12-01 11:53:03 EST TST +Revision 3.1.49.0 2022-12-15 16:33:20 EST TST --------------------------------------------------------------------- diff --git a/doc/user/snort_user.text b/doc/user/snort_user.text index 4173ae6f2..6a6b42cb2 100644 --- a/doc/user/snort_user.text +++ b/doc/user/snort_user.text @@ -8,7 +8,7 @@ Snort 3 User Manual The Snort Team Revision History -Revision 3.1.48.0 2022-12-01 11:52:17 EST TST +Revision 3.1.49.0 2022-12-15 16:32:34 EST TST --------------------------------------------------------------------- @@ -2567,6 +2567,11 @@ content:"|00 01 87 88|", offset 12, depth 4; content:"|00 00 00 01 00 00 00 01|", offset 20, depth 8; byte_test:4,>,200,36; +In case of using any byte_* option with "string" parameter, the +amount of bytes to be extracted from payload can be less than +specified by user. This might happen when the buffer has fewer bytes +(from the cursor position) than specified in the option. + 5.5. Consolidated Config @@ -2612,7 +2617,7 @@ wizard = { spells = { - { service = 'http', proto = 'tcp', client_first = true, to_server = { 'GET' }, to_client = { 'HTTP/' } }, + { service = 'http', proto = 'tcp', to_server = { 'GET' }, to_client = { 'HTTP/' } }, } } @@ -2662,13 +2667,11 @@ stream_tcp.track_only=false stream_tcp.show_rebuilt_packets=true consolidated config for http.lua wizard.spells[0].proto="tcp" -wizard.spells[0].client_first=true wizard.spells[0].service="http" wizard.spells[0].to_client[0].spell="HTTP/" wizard.spells[0].to_server[0].spell="GET" consolidated config for sip.lua wizard.spells[0].proto="tcp" -wizard.spells[0].client_first=true wizard.spells[0].service="sip" wizard.spells[0].to_server[0].spell="INVITE" @@ -2759,7 +2762,6 @@ Example: snort -c snort.lua --dump-config=all | jq . "spells": [ { "proto": "tcp", - "client_first": true, "service": "http", "to_client": [ { @@ -2783,7 +2785,6 @@ Example: snort -c snort.lua --dump-config=all | jq . "spells": [ { "proto": "tcp", - "client_first": true, "service": "sip", "to_server": [ { @@ -4883,8 +4884,8 @@ name, or the lowercase function name. One of the improvements in Snort 3 is Enhanced JavaScript Normalizer which has its own module and can be used with any service inspectors -where JavaScript code might occur. Currently it is only used by HTTP -inspector. +where JavaScript code might occur. Currently it is supported for the +following inspectors: HTTP, SMTP, IMAP, POP. 5.13.1. Overview @@ -4900,19 +4901,22 @@ automatically enables Enhanced Normalizer. The Enhanced Normalizer can normalize JavaScript embedded in HTML (inline scripts), in separate .js files (external scripts), and -JavaScript embedded in PDF files sent over HTTP. It supports scripts -over multiple PDUs. It is a stateful JavaScript whitespace and -identifiers normalizer. Normalizer concatenates string literals -whenever it’s possible to do. This also works with any other -normalizations that result in string literals. All JavaScript -identifier names, except those from the ignore lists, will be -substituted with unified names in the following format: var_0000 → -var_ffff. The Normalizer tries to expand escaped text, so it will -appear in a readable form in the output. When such text is a +JavaScript embedded in PDF files sent over HTTP/1, HTTP/2, SMTP, IMAP +and POP3 protocols. It supports scripts over multiple PDUs. It is a +stateful JavaScript whitespace and identifiers normalizer. Normalizer +concatenates string literals whenever it’s possible to do. This also +works with any other normalizations that result in string literals. +All JavaScript identifier names, except those from the ignore lists, +will be substituted with unified names in the following format: +var_0000 → var_ffff. The Normalizer tries to expand escaped text, so +it will appear in a readable form in the output. When such text is a parameter of an unescape function, the entire function call will be replaced by the unescaped string. Moreover, Normalizer validates the syntax concerning ECMA-262 Standard, including scope tracking and -restrictions for script elements. +restrictions for script elements. JavaScript, embedded in PDF files, +has to be decompressed before normalization. For that, decompress_pdf += true option has to be set in configuration of appropriate service +inspectors. Check with the following options for more configurations: bytes_depth, identifier_depth, max_tmpl_nest, max_bracket_depth, @@ -4935,7 +4939,7 @@ conserve resources by doing less. Also, there are default lists of ignored identifiers and object properties provided. To get a complete default configuration, use -default_js_norm from lua/snort_default.lua by adding: +default_js_norm from $SNORT_LUA_PATH/snort_defaults.lua by adding: js_norm = default_js_norm @@ -5076,7 +5080,8 @@ PDU. If later js_data IPS rule matches again, a missed normalization context is detected and 154:8 built-in alert is raised. Further normalization is not possible for the script. For example: -alert http (msg:"JavaScript in HTTP"; js_data; content:"var var_0000=1;"; sid:1;) +alert http (msg:"JS in HTTP"; js_data; content:"var var_0000"; sid:1;) +alert smtp (msg:"JS in SMTP"; js_data; content:"var var_0000"; sid:2;) 5.13.3.1. js_data @@ -6802,8 +6807,6 @@ contain following options: * service - name of the service that would be assigned * proto - protocol to scan - * client_first - indicator of which end initiates data transfer - (deprecated) * to_server - list of text patterns to search in the data sent to the client * to_client - list of text patterns to search in the data sent to @@ -6813,7 +6816,6 @@ contain following options: { service = 'smtp', proto = 'tcp', - client_first = true, to_server = { 'HELO', 'EHLO' }, to_client = { '220*SMTP', '220*MAIL' } } @@ -6838,7 +6840,6 @@ Example of a hex definition in Lua: { service = 'dnp3', proto = 'tcp', - client_first = true, to_server = { '|05 64|' }, to_client = { '|05 64|' } }