From: Adrian Mamolea (admamole) Date: Wed, 25 Jun 2025 17:46:35 +0000 (+0000) Subject: Pull request #4765: http_inspect: add support for partial_depth configuration option X-Git-Tag: 3.9.1.0~2 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=4aa217331d00c8e4232c28599fd4a548f6c9a2fb;p=thirdparty%2Fsnort3.git Pull request #4765: http_inspect: add support for partial_depth configuration option Merge in SNORT/snort3 from ~ADMAMOLE/snort3:cl to master Squashed commit of the following: commit 3e9cdd52035184e38416581e4d5ffb6fd4df0bd1 Author: Adrian Mamolea Date: Fri May 23 15:48:16 2025 -0400 http_inspect: add support for partial_depth configuration option --- diff --git a/doc/reference/builtin_stubs.txt b/doc/reference/builtin_stubs.txt index eb8f21cd3..18fe8d842 100644 --- a/doc/reference/builtin_stubs.txt +++ b/doc/reference/builtin_stubs.txt @@ -1294,6 +1294,11 @@ HTTP request method is not on allowed methods list or is on disallowed methods l HTTP reserved GZIP flags are set +119:289 + +Too many partial flushes. Partial depth is enabled and 20 or more partial flushes are +made before a regular flush. + 121:1 Invalid flag set on HTTP/2 frame header diff --git a/doc/user/http_inspect.txt b/doc/user/http_inspect.txt index 9dee396a2..4557a13c3 100755 --- a/doc/user/http_inspect.txt +++ b/doc/user/http_inspect.txt @@ -148,6 +148,21 @@ more of the sensor's resources. This feature is off by default. script_detection = true will activate it. +===== partial_depth + +Partial depth detection is a feature that enables Snort to more quickly detect +and block malicious requests. It is configured by the partial_depth parameter +which can take values in the range -1-16384 bytes. The feature is enabled by +setting partial_depth to some non zero value. When the feature is enabled and +either, the number of bytes received in the request body is below the value +specified by partial_depth, or partial_depth is set to -1, unlimited; it +immediately forwards the available part of the message body for early detection. +This enables earlier threat detection but consumes somewhat more of the sensor's +resources. + +This feature is turned off by default by setting partial_depth = 0. To activate +it, set partial_depth to the desired value. + ===== gzip http_inspect by default decompresses deflate and gzip message bodies diff --git a/src/service_inspectors/http_inspect/dev_notes.txt b/src/service_inspectors/http_inspect/dev_notes.txt index 9d5a91a7b..d6f0c0035 100755 --- a/src/service_inspectors/http_inspect/dev_notes.txt +++ b/src/service_inspectors/http_inspect/dev_notes.txt @@ -9,17 +9,7 @@ unexpectedly). The nature of splitting allows packets to be forwarded before they are aggregated into a message section and inspected. This may lead to problems when the target consumes a partial message body even though the end of the message body was never received because Snort blocked it. - -Script detection is a feature developed to solve this problem for message bodies containing -Javascripts. The stream splitter scan() method searches its input for the end-of-script tag -"". When necessary this requires scan() to unzip the data. This is an extra unzip as -storage limitations preclude saving the unzipped version of the data for subsequent reassembly. - -Update: the previous sentence has been discovered to be incorrect. The memory requirements of -zlib are very large. It would save a lot of memory and some processing time for script detection -to unzip one time in scan() and store the result for eventual use by reassemble(). The memory -lost by storing partial message sections in HI while waiting for reassemble() would be more than -compensated for by not having two instances of zlib. +Partial inspection was developed to solve this problem. HttpFlowData is a data class representing all HI information relating to a flow. It serves as persistent memory between invocations of HI by the framework. It also glues together the inspector, diff --git a/src/service_inspectors/http_inspect/dev_notes_partial_inspection.txt b/src/service_inspectors/http_inspect/dev_notes_partial_inspection.txt index 981643cb1..ef70bbe0d 100644 --- a/src/service_inspectors/http_inspect/dev_notes_partial_inspection.txt +++ b/src/service_inspectors/http_inspect/dev_notes_partial_inspection.txt @@ -1,16 +1,12 @@ -When the end of a script is found and the normal flush point has not been found, the current TCP -segment and all previous segments for the current message section are flushed using a special -procedure known as partial inspection. From the perspective of Stream (or H2I) a partial inspection -is a regular flush in every respect. - -scan() calls prep_partial_flush() to prepare for the partial inspection. Then it returns a normal -flush point to Stream at the end of the current TCP segment. Partial inspections perform all of the -functions of a regular inspection including forwarding data to file processing and detection. - The difference between a partial inspection and a regular inspection is reassemble() saves the input data for future reuse. Eventually there will be a regular full inspection of the entire message section. reassemble() will accomplish this by combining the input data for the partial -inspection with later data that complete the message section. +inspection with later data that completes the message section. + +scan() calls prep_partial_flush() to prepare for the partial inspection. Then it returns a normal +flush point to Stream at the end of the current TCP segment. Partial inspections perform all of the +functions of a regular inspection including forwarding data to file processing and detection. From +the perspective of Stream (or H2I) a partial inspection is a regular flush in every respect. Correct and efficient execution of a full inspection following a partial inspection requires special handling of certain functions. Unzipping is only done once in reassemble(). The stored @@ -26,12 +22,30 @@ Compared to just doing a full inspection, a partial inspection followed by a ful will not miss anything. The benefits of partial inspection are in addition to the benefits of a full inspection. -The http_inspect partial inspection mechanism is also used by http2_inspect to manage frame -boundaries. When inspecting HTTP/2, a partial inspection by http_inspect may occur because script -detection triggered it, because H2I wanted it, or both. +Partial inspection is used in multiple scenarios, described below, and in combinations of them. + +Script detection uses partial inspection for message bodies containing Javascripts. The stream +splitter scan() method searches its input for the end-of-script tag "". When the end +of a script is found and the normal flush point has not been found, the current TCP segment and +all previous segments for the current message section are flushed using partial inspection. + +Searching for the end-of-script tag may require scan() to unzip the data. This is an extra unzip +as storage limitations preclude saving the unzipped version of the data for subsequent reassembly. + +Update: the previous sentence has been discovered to be incorrect. The memory requirements of +zlib are very large. It would save a lot of memory and some processing time for script detection +to unzip one time in scan() and store the result for eventual use by reassemble(). The memory +lost by storing partial message sections in HI while waiting for reassemble() would be more than +compensated for by not having two instances of zlib. + +For request bodies, when partial_depth parameter is set to a non zero value, a partial body will +be subjected to partial inspection if its length is below partial_depth value. When the partial_depth +parameter is set to -1, the entire body will be subjected to inspection regardless of its length. + +The http_inspect partial inspection mechanism is invoked by http2_inspect on frame boundaries. -Some applications may be affected by blocks too late scenarios related to seeing part of the -zero-length chunk. For example a TCP packet that ends with: +With chunking some applications may be affected by blocks too late scenarios related to seeing part +of the zero-length chunk. For example a TCP packet that ends with: 8abcdefgh0 diff --git a/src/service_inspectors/http_inspect/http_common.h b/src/service_inspectors/http_inspect/http_common.h index 4dac4bd99..1d61cfbf6 100644 --- a/src/service_inspectors/http_inspect/http_common.h +++ b/src/service_inspectors/http_inspect/http_common.h @@ -40,6 +40,11 @@ enum SectionType { SEC_DISCARD = -19, SEC_ABORT = -18, SEC__NOT_COMPUTE=-14, SEC SEC_REQUEST = 2, SEC_STATUS, SEC_HEADER, SEC_BODY_CL, SEC_BODY_CHUNK, SEC_TRAILER, SEC_BODY_OLD, SEC_BODY_HX }; +inline bool is_body(SectionType st) +{ + return (st == SEC_BODY_CL || st == SEC_BODY_CHUNK || st == SEC_BODY_OLD || st == SEC_BODY_HX); +} + // Caters to all extended versions of HTTP, i.e. HTTP/2, HTTP/3 enum HXBodyState { HX_BODY_NOT_COMPLETE, HX_BODY_LAST_SEG, HX_BODY_COMPLETE, HX_BODY_COMPLETE_EXPECT_TRAILERS, HX_BODY_NO_BODY }; diff --git a/src/service_inspectors/http_inspect/http_enum.h b/src/service_inspectors/http_inspect/http_enum.h index 1c8984d9e..fbac0fa09 100755 --- a/src/service_inspectors/http_inspect/http_enum.h +++ b/src/service_inspectors/http_inspect/http_enum.h @@ -444,6 +444,7 @@ enum EventSid EVENT_UNEXPECTED_H2_PREFACE = 286, EVENT_DISALLOWED_METHOD = 287, EVENT_GZIP_RESERVED_FLAGS = 288, + EVENT_MAX_PARTIAL_FLUSH = 289, EVENT__MAX_VALUE }; diff --git a/src/service_inspectors/http_inspect/http_flow_data.h b/src/service_inspectors/http_inspect/http_flow_data.h index 39d774173..c31d054cb 100644 --- a/src/service_inspectors/http_inspect/http_flow_data.h +++ b/src/service_inspectors/http_inspect/http_flow_data.h @@ -120,6 +120,7 @@ private: uint32_t num_good_chunks[2] = { 0, 0 }; uint32_t octets_expected[2] = { 0, 0 }; bool is_broken_chunk[2] = { false, false }; + uint64_t partial_flush_counter = 0; // *** StreamSplitter => Inspector (facts about the most recent message section) HttpCommon::SectionType section_type[2] = { HttpCommon::SEC__NOT_COMPUTE, diff --git a/src/service_inspectors/http_inspect/http_inspect.cc b/src/service_inspectors/http_inspect/http_inspect.cc index d6f69576d..ad8da1c47 100755 --- a/src/service_inspectors/http_inspect/http_inspect.cc +++ b/src/service_inspectors/http_inspect/http_inspect.cc @@ -158,6 +158,7 @@ void HttpInspect::show(const SnortConfig*) const ConfigLogger::log_limit("request_depth", params->request_depth, -1); ConfigLogger::log_limit("response_depth", params->response_depth, -1); + ConfigLogger::log_limit("partial_depth", params->partial_depth, -1, 0); ConfigLogger::log_flag("unzip", params->unzip); ConfigLogger::log_flag("normalize_utf", params->normalize_utf); ConfigLogger::log_flag("decompress_pdf", params->decompress_pdf); diff --git a/src/service_inspectors/http_inspect/http_module.cc b/src/service_inspectors/http_inspect/http_module.cc index f11b8ff6b..a9749d9fa 100755 --- a/src/service_inspectors/http_inspect/http_module.cc +++ b/src/service_inspectors/http_inspect/http_module.cc @@ -52,6 +52,9 @@ const Parameter HttpModule::http_params[] = { "response_depth", Parameter::PT_INT, "-1:max53", "-1", "maximum response message body bytes to examine (-1 no limit)" }, + { "partial_depth", Parameter::PT_INT, "-1:16384", "0", + "maximum request body to send to early detection (0 disabled, -1 no limit)" }, + { "unzip", Parameter::PT_BOOL, nullptr, "true", "decompress gzip and deflate message bodies" }, @@ -209,6 +212,10 @@ bool HttpModule::set(const char*, Value& val, SnortConfig*) { params->response_depth = val.get_int64(); } + else if (val.is("partial_depth")) + { + params->partial_depth = val.get_int64(); + } else if (val.is("unzip")) { params->unzip = val.get_bool(); diff --git a/src/service_inspectors/http_inspect/http_module.h b/src/service_inspectors/http_inspect/http_module.h index fe0976259..c333a424e 100755 --- a/src/service_inspectors/http_inspect/http_module.h +++ b/src/service_inspectors/http_inspect/http_module.h @@ -49,6 +49,7 @@ public: ~HttpParaList(); int64_t request_depth = -1; int64_t response_depth = -1; + int64_t partial_depth = 0; bool unzip = true; bool normalize_utf = true; diff --git a/src/service_inspectors/http_inspect/http_stream_splitter_reassemble.cc b/src/service_inspectors/http_inspect/http_stream_splitter_reassemble.cc index a22f37fd0..eb7c07a53 100644 --- a/src/service_inspectors/http_inspect/http_stream_splitter_reassemble.cc +++ b/src/service_inspectors/http_inspect/http_stream_splitter_reassemble.cc @@ -421,17 +421,11 @@ const StreamBuffer HttpStreamSplitter::reassemble(Flow* flow, unsigned total, HttpModule::increment_peg_counts(PEG_REASSEMBLE); - const bool is_body = - (session_data->section_type[source_id] == SEC_BODY_CHUNK) || - (session_data->section_type[source_id] == SEC_BODY_CL) || - (session_data->section_type[source_id] == SEC_BODY_OLD) || - (session_data->section_type[source_id] == SEC_BODY_HX); - uint8_t*& buffer = session_data->section_buffer[source_id]; if (buffer == nullptr) { // Body sections need extra space to accommodate unzipping - if (is_body) + if (is_body(session_data->section_type[source_id])) buffer = new uint8_t[MAX_OCTETS]; else { diff --git a/src/service_inspectors/http_inspect/http_stream_splitter_scan.cc b/src/service_inspectors/http_inspect/http_stream_splitter_scan.cc index bbc227b30..9db07ecef 100644 --- a/src/service_inspectors/http_inspect/http_stream_splitter_scan.cc +++ b/src/service_inspectors/http_inspect/http_stream_splitter_scan.cc @@ -190,7 +190,7 @@ StreamSplitter::Status HttpStreamSplitter::call_cutter(Flow* flow, HttpFlowData* } const uint32_t max_length = MAX_OCTETS - cutter->get_octets_seen(); - const ScanResult cut_result = cutter->cut(data, (length <= max_length) ? length : + ScanResult cut_result = cutter->cut(data, (length <= max_length) ? length : max_length, session_data->get_infractions(source_id), session_data->events[source_id], session_data->section_size_target[source_id], session_data->stretch_section_to_packet[source_id], @@ -217,6 +217,16 @@ StreamSplitter::Status HttpStreamSplitter::call_cutter(Flow* flow, HttpFlowData* return status_value(StreamSplitter::ABORT); } + if (is_body(type) && source_id == SRC_CLIENT && + (my_inspector->params->partial_depth == -1 || + (cutter->get_octets_seen() < my_inspector->params->partial_depth && cutter->get_num_flush() == 0))) + { + static const uint64_t MAX_PARTIAL_FLUSH_COUNTER = 20; + if (++session_data->partial_flush_counter == MAX_PARTIAL_FLUSH_COUNTER) + session_data->events[source_id]->create_event(HttpEnums::EVENT_MAX_PARTIAL_FLUSH); + cut_result = SCAN_NOT_FOUND_ACCELERATE; + } + if (cut_result == SCAN_NOT_FOUND_ACCELERATE) { prep_partial_flush(flow, length); @@ -252,6 +262,7 @@ StreamSplitter::Status HttpStreamSplitter::call_cutter(Flow* flow, HttpFlowData* case SCAN_FOUND: case SCAN_FOUND_PIECE: { + session_data->partial_flush_counter = 0; const uint32_t flush_octets = cutter->get_num_flush(); prepare_flush(session_data, flush_offset, type, flush_octets, cutter->get_num_excess(), cutter->get_num_head_lines(), cutter->get_is_broken_chunk(), diff --git a/src/service_inspectors/http_inspect/http_tables.cc b/src/service_inspectors/http_inspect/http_tables.cc index c8a963f5b..5b27de7c0 100755 --- a/src/service_inspectors/http_inspect/http_tables.cc +++ b/src/service_inspectors/http_inspect/http_tables.cc @@ -354,6 +354,7 @@ const RuleMap HttpModule::http_events[] = { EVENT_DISALLOWED_METHOD, "HTTP request method is not on allowed methods list or is on " "disallowed methods list" }, { EVENT_GZIP_RESERVED_FLAGS, "HTTP gzip body with reserved flag set" }, + { EVENT_MAX_PARTIAL_FLUSH, "Too many partial flushes" }, { 0, nullptr } };