From: Vsevolod Stakhov <vsevolod@rspamd.com>
Date: Sat, 8 Nov 2025 14:14:30 +0000 (+0000)
Subject: [Fix] Add batching and forced GC for Redis connections in dmarc_report
X-Git-Tag: 3.14.0~7^2~4
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=352cec1e61bf3c314a186146c13cb550fc9eb811;p=thirdparty%2Frspamd.git

[Fix] Add batching and forced GC for Redis connections in dmarc_report

The coroutine version of lua_redis.request() creates connections that
are only released when Lua garbage collector runs. In tight loops
processing hundreds of domains, GC doesn't run frequently enough,
causing connection exhaustion.

Root cause: lua_redis.connect_sync() + conn:exec() returns connections
to Lua but relies on __gc metamethod for pool release. The connections
pile up until GC runs.

Solution:
- Process reports in batches (reuse existing --batch-size option)
- Force collectgarbage("collect") between batches to trigger __gc
- This ensures connections are properly released back to the pool

Each prepare_report() makes 3-4 Redis requests (EXISTS, RENAME,
ZRANGE, DEL), so batching + forced GC prevents parallel connection
buildup.
---

diff --git a/lualib/rspamadm/dmarc_report.lua b/lualib/rspamadm/dmarc_report.lua
index 62787e7bc0..7f59b96983 100644
--- a/lualib/rspamadm/dmarc_report.lua
+++ b/lualib/rspamadm/dmarc_report.lua
@@ -607,12 +607,27 @@ local function process_report_date(opts, start_time, end_time, date)
     return {}
   end
 
+  -- Process reports in batches to limit Redis connections
   local reports = {}
-  for _, rep in ipairs(results) do
-    local report = prepare_report(opts, start_time, end_time, rep)
+  local batch_size = opts.batch_size or 10
 
-    if report then
-      table.insert(reports, report)
+  for batch_start = 1, #results, batch_size do
+    local batch_end = math.min(batch_start + batch_size - 1, #results)
+    lua_util.debugm(N, 'processing report batch %s to %s (of %s total)',
+        batch_start, batch_end, #results)
+
+    for i = batch_start, batch_end do
+      local rep = results[i]
+      local report = prepare_report(opts, start_time, end_time, rep)
+
+      if report then
+        table.insert(reports, report)
+      end
+    end
+
+    -- Force garbage collection between batches to release Redis connections
+    if batch_end < #results then
+      collectgarbage("collect")
     end
   end