stream: fix protocol detection issue for GAPs

If the protocol required TOSERVER data first, but the SSN started with
a GAP, then the TOCLIENT side would get stuck in an expensive path:

1. it would run detection on TOCLIENT
2. it would try to force reassembly for TOSERVER
3. it would reset the detected protocol as TOSERVER failed
4. it would not evict any segment

This had 2 consequences:
1. on long running sessions this could lead to using lots of memory
   on segments, denying other sessions resources
2. wasted cycles on protocol detection and segment list management

This patch introduces a fix. It checks in the (2) stage above, whether
the opposing stream (that we depend on) it is a NOREASSEMBLY state. If
so, it gives up on this side of the session as well.

diff --git a/src/app-layer.c b/src/app-layer.c
index c47c2dd6624afedd2f5cd5582cecd7a34f6257fa..aeb69a3d94ba9c187b0247962a5df583feb59b69 100644 (file)
--- a/src/app-layer.c
+++ b/src/app-layer.c
@@ -187,7 +187,14 @@ int AppLayerHandleTCPData(ThreadVars *tv, TcpReassemblyThreadCtx *ra_ctx,
                         p->flowflags |= FLOW_PKT_TOCLIENT;
                     }
                 }
-                int ret = StreamTcpReassembleAppLayer(tv, ra_ctx, ssn,
+
+                int ret = 0;
+                /* if the opposing side is not going to work, then
+                 * we just have to give up. */
+                if (opposing_stream->flags & STREAMTCP_STREAM_FLAG_NOREASSEMBLY)
+                    ret = -1;
+                else
+                    ret = StreamTcpReassembleAppLayer(tv, ra_ctx, ssn,
                                                       opposing_stream, p);
                 if (stream == &ssn->client) {
                     if (StreamTcpInlineMode()) {