The empty table caused a spurious warning in lua_maps when no
whitelist was configured. Since settings.whitelist defaults to nil,
the else-branch was a no-op. User-configured whitelists via
link_affiliation { whitelist = ... } continue to work as before.
[Minor] lua_maps: handle empty table as static empty map
When map_add_from_ucl receives an empty Lua table, it fell through
to the C map infrastructure, which logged a spurious error-level
message with no map name. Return a lightweight empty map object
directly in Lua, cache it for consistency with other code paths,
and log a warning since an empty table is likely a misconfiguration.
[Minor] maps: include map description in load error messages
Without a map description in the log, users had no way to identify
which map triggered the error, forcing unnecessary investigation.
All 'no urls to be loaded' and 'invalid type' error sites in
rspamd_map_add_from_ucl now include the description; rspamd_printf
handles NULL safely.
[Fix] fuzzy_storage: drop per-command state caching from TCP session (issue #6001)
Each TCP frame allocates a per-frame `cmd_session` whose key, extensions
buffer, ip_stat, and decrypt nm are owned and released by
`fuzzy_session_destroy`. The TCP I/O loop also pre-populated those fields
from a `fuzzy_common_session` cache and "transferred ownership" back at
the end of every frame — but the transfer overwrites without releasing,
so on a persistent TCP connection every frame after the first leaked:
- one g_malloc'd extensions buffer (sent by every scanning request),
- one fuzzy_key REF_RETAIN from the pre-populate path,
- and, for encrypted frames, a second fuzzy_key REF_RETAIN from
rspamd_fuzzy_decrypt_command's own unguarded `s->key = key`.
Stop caching per-command state on the TCP session entirely. cmd_session
starts zeroed each frame, decrypt populates `key`/`nm`, the wire parser
allocates `extensions`, process_command retains `ip_stat`, and
fuzzy_session_destroy frees the lot when the frame is done. The decrypt
overwrite is also no longer a leak because cmd_session->key starts NULL.
Also delete the never-instantiated `fuzzy_udp_session` /
`fuzzy_udp_session_destroy` / `rspamd_fuzzy_udp_write_reply` — UDP has
been using the legacy `fuzzy_session` path all along.
[Test] checkv3: use spam_message for inline-settings regression test
GTUBE short-circuits the Rspamd filter pipeline (forced reject before
SETTINGS_CHECK runs), so the inline metadata.settings test from the
previous commit silently never exercised settings.lua at all. Switch
to messages/spam_message.eml so the prefilter chain actually runs and
apply_settings_side_effects gets a chance to inject the symbol.
Also drop the actions.reject override case: with GTUBE it was a false
positive (GTUBE forces reject regardless of thresholds) and a proper
rewrite needs a second non-GTUBE message wired into the suite, which
can come in a follow-up.
Use the array form `symbols = [..]` matching the v2 INJECT SYMBOL test
in 108_settings.robot - it is the documented apply spelling.
[Fix] protocol: apply inline metadata.settings on /checkv3
Inline metadata.settings on /checkv3 was stashed on task->settings
directly, skipping the apply pipeline. As a result every documented
"apply" effect (action thresholds, symbols / symbols_enabled /
symbols_disabled, subject, variables, add/remove headers) silently
no-op'd, while the same payload sent via the /checkv2 Settings HTTP
header worked. Fixes #5999.
Serialize the inline UCL object to compact JSON and synthesize a
Settings request header so the existing settings.lua
check_query_settings -> apply_settings -> {task:set_settings,
apply_settings_side_effects} pipeline runs uniformly with v2. One
source of truth, no duplicated apply logic in C. Also update the
"settings will be merged" log to "inline settings will take
precedence" since that is what settings.lua actually does when both
settings_id and inline settings are present.
Audit-companion fix: metadata.deliver_to now goes through
rspamd_protocol_escape_braces too, matching the v2 Deliver-To header
which has stripped <...> braces since the beginning.
Add two regression tests in 430_checkv3.robot covering the Lua-side
side effect path (settings.symbols injection) and the C-side action
threshold path (actions.reject override).
[Fix] protocol: populate request headers in /checkv3
/checkv3 was reading only a fixed set of HTTP headers and never passed
them through rspamd_task_add_request_header, so task:get_request_header()
returned nil for arbitrary client-supplied headers under v3 — a regression
vs. v2. Iterate msg->headers at the top of rspamd_protocol_handle_v3_request
and register every HTTP header. Skip Shm / Shm-Offset / Shm-Length, which
at the HTTP level are the proxy body-transfer mechanism and would collide
with the metadata-synthesized "shm" zero-copy message pointer consumed by
rspamd_task_load_message.
[Fix] neural: retarget training vectors to best-known profile
Workers kept writing training data to the last-loaded ANN's redis key
(set.ann.redis_key) even when a newer, more specific profile was
already registered in the ZLIST. If that new profile had no ANN data
yet (e.g. controller created a placeholder after a symbols change),
load_new_ann silently left set.ann untouched, so vectors kept piling
up under the old key while the controller waited for the new key's
spam/ham sets to fill — a deadlock visible as repeated "more specific
ann is available" log lines without any retrain ever happening.
Track the best-known profile in set.training_profile, updated on every
process_existing_ann tick whenever sel_elt is picked. ann_push_task_result
now routes target_key and the vectors_len script call through
(set.training_profile or set.ann).redis_key, so training data lands on
the newest profile immediately. set.ann is left alone so inference
keeps using the previously loaded ANN until a fresh one is trained.
- lua_redis: args[0] is allocated as g_malloc(arglens[0]) + memcpy and
is NOT NUL-terminated, so reusing it as a diagnostic label (read
later via %s) could read past the buffer. Store a dedicated
g_strdup'd copy of the command in sp_ud->cmd and use that for the
event label; free it alongside args in the dtor.
- async_session: the overflow counter increments once per event that
could not be placed into the first 32 dump groups, not once per
lost group. Rename overflow_groups -> overflow_events and reword
the suffix to "(+N events not shown)" so the message matches
reality (the lost events may belong to any number of groups).
[Feature] async_session: add labels for redis/fuzzy/lua_http events
Propagate per-subsystem identifying info as event labels so timeout
logs spell out what each stalled event is waiting on:
- lua_redis.c: use the redis command (args[0], e.g. "HGET" / "SET").
- fuzzy_check.c: use rule->name in all four add_event sites so the
stalled backend rule is visible (the plugin may be running many
named fuzzy rules in parallel, and the symcache item is just
FUZZY_CHECK for all of them).
- lua_http.c: use the resolved hostname (cbd->host).
Example timeout line for a task stuck on two Redis calls and a
hostname that won't connect:
[Feature] async_session: annotate DNS and TCP events with operation labels
Add rspamd_session_event_update_label() for mutating a live event's
label field, and use it to give operators a better idea of what a
long-lived event is waiting on at timeout time:
- dns.c: pass rdns_strtype(type) ("A" / "AAAA" / "TXT" / ...) as the
label at add time, so timeout logs show which record type stalled.
- lua_tcp.c: update the label as the per-connection event transitions
between operations ("tcp connect" / "tcp write" / "tcp read").
Timeout summary now reads
"rspamd dns[RBL_FOO/A]=3, rspamd lua tcp[RULE/tcp read]=1"
instead of just "rspamd dns=3, rspamd lua tcp=1".
[Feature] task: route all task sessions through single constructor
Only worker.c and 1 of 8 controller.c task sessions had the item-name
resolver wired, so timeout logs for scans from other entry points
(check, learn, scan-lua, worker_util startup, proxy, lua.task_from_mime)
lost symbol context entirely: the summary said only "rspamd lua http=1"
with no hint of which rule stalled.
Introduce rspamd_task_create_session() that wraps session_create and
set_item_name_resolver, and switch all 11 task-scoped session creation
sites to use it. The two controller sites with cbdata user_data (stats,
metrics) stay on rspamd_session_create — they do not execute symcache.
While here, collapse rspamd_session_describe_pending into a single
line that folds symbol/label info into each group key, e.g.
"total=5; rspamd dns[RBL_FOO]=3, rspamd lua http[X]=1". Previously the
symbol detail was a separate line that was silently omitted whenever
the resolver was not wired — exactly the breakage this fixes.
[Feature] async_session: replace G_STRLOC event_source with item_name + label
event_source was always G_STRLOC, which carries no useful information
for operators: "src/libserver/dns.c:274" tells nothing about which rule
stalled the task. Drop it entirely. The event now carries:
- item_name: auto-snapshot of the currently-running symcache symbol,
resolved through a new session-level callback (wired into task
sessions in worker.c / controller.c to return the current symbol);
- label: optional human annotation (e.g. "tcp write"), NULL by default.
rspamd_session_describe_pending groups by (item_name, label), so the
timeout detail line now reads
"[rspamd dns: RBL_DNSBL_FOO x5, SURBL_CHECK x2]" instead of file:line.
Events that carry neither field are still counted in the subsystem
summary but omitted from the detail line.
As a knock-on simplification, rspamd_session_remove_event_full loses
its G_STRLOC parameter (now just rspamd_session_remove_event), and the
four direct add_event_full callers in lua_udp / lua_http / lua_task /
lua_tcp no longer need to pass rspamd_symcache_dyn_item_name manually —
the auto-resolver handles it uniformly.
When a task hits the timeout the log now includes a grouped summary of
the still-pending async events (DNS / Redis / Lua HTTP / fuzzy check /
...) and the list of symbols that started but have not finished. This
gives operators an immediate picture of which subsystem or rule stalled
the scan, instead of only seeing "forced processing".
The per-event "forced removed event on destroy" lines that previously
printed at info level during rspamd_session_cleanup are demoted to
debug — the new summary replaces them and keeps the log compact.
Patrick C. [Sat, 18 Apr 2026 08:09:43 +0000 (10:09 +0200)]
[Feature] Add --batch-wait option to dmarc_report (#5985)
* [Feature] Add --batch-wait option to dmarc_report
- Add -w/--batch-wait option to rspamadm dmarc_report to wait n
seconds between sending batches. This can be used to avoid
overloading SMTP server with too many messages in a short timeframe.
* replaced os.execute(sleep) with rspamadm_ev_base:sleep
* also delay generation of reports if batch-wait is given
this reduces the peak of DNS requests to avoid overwelming a weak resolver
* fixed indentation
* fix: 0 is true in case of lua, so the correct thing is to compare with 0
* skip last wait when preparing reports
---------
Co-authored-by: Patrick Cernko <pcernko@mpi-klsb.mpg.de>
[Fix] html: preserve verbatim href as url->raw (issue #5986)
When the same URL appears in both the HTML and plain-text parts of a
multipart/alternative message, the HTML variant wins deduplication (HTML
parts are processed first). html_process_url builds a partially
percent-decoded scratch buffer — decoding %40, %3A, %7C, %2F, %3F, %5C
to defeat URL obfuscation — and rspamd_url_parse sets url->raw to that
buffer. Consumers of url:get_raw() then saw a partially decoded form
instead of the bytes that actually appeared in the message.
Re-point url->raw to a mempool-owned copy of the verbatim (trimmed)
href after a successful parse. Length is clamped to G_MAXUINT16 / 2
with RSPAMD_URL_FLAG_TRUNCATED on overflow, matching rspamd_url_parse's
own truncation policy.
Adds test/lua/unit/url_raw_preserve.lua covering plain-text-only,
HTML-only, same-URL-in-both-parts (the issue reproduction), and a
GC-stability check for the mempool-backed raw buffer.
Add 32 HTML5 tags used in modern email (sectioning, media, text-level,
interactive, forms, web components). Notably adds video/audio/source/
track/picture/svg recognition so their URLs and structure are visible
to the parser. Also fixes latent bug where Tag_KEYGEN existed in the
enum but was missing from the defs array.
New tag IDs are appended after Tag_NEXTID so existing tag IDs remain
stable.
[Minor] Preserve real_charset in set_part_binary()
Early exit via set_part_binary() skipped the text_part->real_charset
assignment, leaving it NULL and causing downstream consumers to report
#cs:unk instead of #cs:x-binaryenc for binary parts.
[Fix] Skip ICU conversion for x-binaryenc charset in all detection paths
x-binaryenc is a synthetic name returned by CED (Google Compact
Encoding Detection) to signal binary content, not a real text
encoding. ICU has no converter for it and always fails with
U_FILE_ACCESS_ERROR, producing a misleading warning in the logs.
Add an early exit in rspamd_mime_text_part_maybe_convert() for all
three detection paths: announced-charset missing, announced-charset
unknown, and rspamd_mime_charset_utf_check content-heuristic. When
the detected charset is x-binaryenc, mark the part as raw binary
immediately, consistent with what the existing fallback already does.
Extract set_part_binary() helper and RSPAMD_BINARYENC_CHARSET constant
to eliminate the resulting code duplication.
Fixes the spurious "cannot open converter for x-binaryenc" warning
seen when processing messages with binary MIME parts.
* [Minor] selectors: deduplicate helpers and reuse cheaper task APIs
Extract the "find largest text part" loop into common.largest_text_part
shared by fuzzy_digest and fuzzy_shingles. Drop the pcall wrappers
around get_fuzzy_hashes since the C API does not throw. Use
get_header_count('Received') instead of allocating the full received
headers table just to take its length. Add selector smoke tests for
authenticated and received_count.
* [Minor] selectors: use lua_mime.get_displayed_text_part
Switch fuzzy_digest and fuzzy_shingles to lua_mime.get_displayed_text_part,
which already handles the right MUA-display semantics: prefer non-attachment
HTML, fall back to plain text, then to attachment HTML/text within size
limits, with a minimum-words threshold. Drops the largest_text_part helper
added in the previous commit since it picked the wrong part (e.g. a verbose
text/plain alternative or a .txt attachment).
[Test] lua_feedback_parsers: add unit tests for DSN and ARF
Cover the pure helpers (strip_angles, parse_field_blocks) directly via
internal exports and exercise parse_dsn / parse_arf end-to-end on
synthetic tasks built with rspamd_task.load_from_string. Tests cover
header folding, repeated fields, CRLF normalisation, original-message
extraction, and detection of non-report messages.
Replace local trim/split_lines/normalize_eol with lua_util.str_trim and
lua_util.rspamd_str_split. Drop the safe_get_type_full/safe_get_content
pcall wrappers since the mime_part C API does not throw, and remove the
internal _-prefixed exports that nothing consumes.
[Fix] regexp: do not discard all capture groups that follow an empty one (#5974)
* [Test] Test empty capture groups in the middle of the match
Tests #5973
* [Fix] Empty capture groups in the middle of a match don't discard all remaining groups (PCRE2)
Do not stop on first empty capture group, rather lookup from the end for the last non-empty capture group, and truncate there.
By the way the results array thus gets correctly dimensioned from the start, instead of being truncated afterwards.
Fixes #5973
* [Fix] Empty capture groups in the middle of a match don't discard all remaining groups (PCRE1)
Do not stop on first empty capture group, rather lookup from the end for the last non-empty capture group, and truncate there.
By the way the results array thus gets correctly dimensioned from the start, instead of being truncated afterwards.
Fixes #5973
[Fix] Propagate redis_timeout into nested greylist.redis{} block
When Redis is configured via greylist { redis { ... } }, parse_redis_server
reads opts.redis and ignores opts.timeout, so redis_timeout had no effect.
Shallow-copy the nested redis subtable and set its timeout when not
explicitly configured by the operator.
[Fix] Avoid mutating cached redis_params in greylist
Pass a shallow copy of opts with redis_timeout set before calling
parse_redis_server, so the corrected timeout is baked into the cached
table. The previous post-assignment mutated the shared cached object,
potentially affecting other modules reusing the same Redis config.
[Fix] Check per-worker task_timeout overrides in configtest
The previous fix only validated cfg->task_timeout, missing cases where
individual workers (normal, controller) override task_timeout in their
own config section. Iterate cfg->workers and check each worker's
effective timeout via its UCL options object.
[Fix] Warn on task_timeout less than symcache symbol timeout
Call rspamd_worker_check_and_adjust_timeout during configtest so
misconfigured plugin timeouts are reported at configuration validation
time. Elevate the diagnostic from info to warning level.
Fix rspamd_symcache_add_symbol_augmentation to parse "key=value" format
in augmentation strings, allowing numeric timeout augmentations from
Lua plugins to be stored and compared correctly.
Clarify mx_check timeout comment to explain that the effective symbol
timeout includes dns.timeout in addition to the configured value.
[Fix] Separate greylisting period from Redis connection timeout in greylist
settings.timeout (greylisting period, 5 min) was being picked up by
lua_redis as the Redis connection timeout, inflating the symbol's
augmentation timeout to 300s. Add redis_timeout (default 1.0s) and
explicitly set redis_params.timeout after parse_redis_server.
[Fix] Fix infinite loop and OOB read in archive processing
- RAR v5 extra area parsing: fix wrong loop boundary (used advanced `p`
instead of `ex + extra_sz`), wrong advancement (`cur_sz` alone didn't
account for the size vint bytes), and wrong remain calculation for the
second vint read. Add bounds validation for `cur_sz`.
- 7zip codec ID: validate `p + sz <= end` before reading codec ID bytes
to prevent out-of-bounds read with malformed archives.
[Feature] Allow custom response codes for fuzzy dynamic blocks
Dynamic blocks always returned 503, making them indistinguishable
from static blacklist blocks on the client side. Ratelimit scripts
need to return 403 so that fuzzy_check clients can map the response
to FUZZY_RATELIMITED instead of FUZZY_FORBIDDEN.
Add response_code field to rspamd_fuzzy_dynamic_ban and propagate
it through rspamd_fuzzy_check_client (now returns int: 0=allowed,
>0=block code) and rspamd_fuzzy_block_addr. The Lua API
block_fuzzy_client accepts an optional 6th argument for the code.
[Test] Add functional test for rspamadm autolearnstats command
Scans sample messages through rspamd to generate autolearn logs dynamically,
then validates autolearnstats output parsing, column headers, and filtering.
Replace the naive split-by-semicolon + regex approach in
parse_arc_header with a proper LPeg-based parser that respects
RFC 8601 comments (...) and quoted strings "..." when splitting
AAR elements. The old unanchored pattern matched "i=N" inside
comments like arc=pass ("domain:s=sel:i=2"), misclassifying
the element as an ARC instance index. This left cur_aar.ar nil,
crashing gen_arc_seal_cb and producing cv=fail on valid chains.
Add parse_aar_header() to lua_auth_results.lua with shared LPeg
atoms for comment/quote-aware semicolon splitting, and unit tests
covering the #5963 scenario and edge cases.
[Fix] Keep original fd in proxy session after milter dup
Revert the client_sock = -1 added in 2c56437e7 which incorrectly
assumed the milter library owned the accepted fd. The milter library
calls dup() internally (since 4a75d59cc) and works with the copy,
so the original fd must stay in session->client_sock for
proxy_session_dtor to close it. Without this, every milter connection
leaked one fd permanently, causing unbounded CLOSE_WAIT accumulation.
This restores the pre-2c56437e7 behaviour where proxy_session_dtor
closed the original fd and the milter dtor closed the dup'd copy.
[Fix] Use task_timeout instead of upstream timeout for proxy self-scan
The proxy self-scan path was using default_upstream->timeout (the
milter/HTTP wire timeout, typically 120s) as the task processing
timeout. This caused tasks to linger for up to 2 minutes before
being reaped, leading to massive CLOSE_WAIT fd accumulation under
load.
Use ctx->task_timeout (derived from cfg->task_timeout, default 8s)
instead, matching how the normal worker handles task timeouts. The
previous else-if branch also had a bug where the repeat interval
used upstream timeout while the initial used cfg->task_timeout.
[Feature] Expose ratelimit_whitelist check to Lua via worker method
Add worker:is_ratelimit_whitelisted(addr) Lua method that checks the
fuzzy worker's existing ratelimit_whitelist radix map, so Lua scripts
can skip provisional blocks for whitelisted IPs (e.g. spamtraps).
[Fix] Use ev_now() instead of rspamd_get_ticks() for dynamic ban expiry
rspamd_get_ticks(FALSE) returns CLOCK_MONOTONIC (seconds since boot),
while rspamd_util.get_time() in Lua returns ev_time() (wall-clock unix
epoch). Using mismatched clocks made expire_ts comparisons nonsensical.
Switch to ev_now(ctx->event_loop) which matches the ev_time() epoch used
by Lua callers of block_fuzzy_client.
[Feature] Add dynamic block API to fuzzy storage worker
Adds worker:block_fuzzy_client(addr, prefix_len[, expire_ts[, reason]])
Lua method that inserts a CIDR network block into a per-worker btrie
(dynamic_blocked_nets). Checked in rspamd_fuzzy_check_client() after
the static blocked_ips map.
expire_ts is an absolute monotonic timestamp (0 = permanent). Callers
compute get_time() once and pass it to a batch of block calls — C never
calls get_time() itself. Re-inserting an existing prefix updates the
expire_ts and reason in-place (idempotent).
Each fuzzy worker independently applies bans via Redis polling; no new
IPC commands or control socket extensions are needed.
Vsevolod Stakhov [Tue, 31 Mar 2026 15:57:12 +0000 (16:57 +0100)]
[Rework] Make hard deps opt-in, weak deps default
Invert the dependency semantics for backward compatibility:
- Default (no flag) = weak/soft dep, no cascade-disable (old behavior)
- hard=true = cascade-disable when dep is disabled by settings
This preserves backward compat: all existing register_dependency
calls without flags behave exactly as before. Hard cascade-disable
is an explicit opt-in feature.
Lua API: register_dependency('A', 'B', true) now means hard dep.
Remove redundant weak annotations from plugins (default is weak).
Vsevolod Stakhov [Tue, 31 Mar 2026 11:08:09 +0000 (12:08 +0100)]
[Rework] Replace GHashTable with id_list for force-enabled IDs
Move force_enabled_ids from GHashTable mempool variable to a
native id_list* field in symcache_runtime. Uses lazy allocation
and cleanup in savepoint_dtor. Avoids GLib in C++ internals
and mempool variable indirection.
Vsevolod Stakhov [Tue, 31 Mar 2026 10:45:33 +0000 (11:45 +0100)]
[Fix] Use finished status in disable_all_symbols
disable_all_symbols implements "disable all, then enable some"
for symbols_enabled/groups_enabled. Using `disabled` status here
caused cascade-disable of hard dependents: if an enabled symbol
depended on a non-enabled one, the enabled symbol was wrongly
cascade-disabled.
Revert to `finished` for disable_all_symbols. Reserve `disabled`
status for explicit symbol/group disable via settings rules
(disable_symbol), where cascade-disable IS the intended behavior.
Vsevolod Stakhov [Tue, 31 Mar 2026 09:03:11 +0000 (10:03 +0100)]
[Fix] Create rdeps for same-stage non-filter dependencies
When a prefilter depends on another prefilter (or postfilter on
postfilter, etc.), create reverse dependencies so that
process_item_rdeps can eagerly start dependents when async deps
complete. Previously rdeps were only created for filter-type deps,
causing same-stage async deps to wait for the next event loop pass.
Vsevolod Stakhov [Tue, 31 Mar 2026 08:55:34 +0000 (09:55 +0100)]
[Fix] Enable deps in pre/postfilters and fix settings flow
Add dependency checking to process_pre_postfilters, allowing
prefilters, postfilters and idempotent filters to declare deps
on symbols in the same stage. Previously deps only worked for
the FILTER stage.
Use this to make SETTINGS_APPLY depend on SETTINGS_CHECK (and
REDIS_SETTINGS* when configured), ensuring proper ordering via
the dep system instead of fragile priority hacks.
Restore immediate settings application in collectors for backward
compatibility - SETTINGS_APPLY now only triggers when multiple
layers need merging.
Also fix missing #include <algorithm> for std::sort on GCC.
Hiroshi Murakami [Tue, 31 Mar 2026 04:55:33 +0000 (13:55 +0900)]
[Fix] Add missing #include <memory> in redis_backend.cxx
std::make_unique is used (line 584) but <memory> is not explicitly
included. This causes compilation failures with some compiler/platform
combinations (e.g. gcc-toolset-12 on Rocky Linux 8) where <memory>
is not transitively included by other headers.
Vsevolod Stakhov [Mon, 30 Mar 2026 15:30:59 +0000 (16:30 +0100)]
[Feature] Add force-enable override for settings conflicts
When merged settings explicitly enable a symbol that settings_elt
has in forbidden_ids, the merged result now wins. Uses a per-task
GHashTable (mempool variable "force_enabled_ids") populated during
process_settings() and checked in is_allowed() before consulting
the precomputed bitsets. This implements the "enable > disable,
explicit > implicit" conflict resolution rule.
Vsevolod Stakhov [Mon, 30 Mar 2026 15:09:40 +0000 (16:09 +0100)]
[Feature] Add settings merge point and collect-then-merge flow
Refactor settings.lua from first-match-wins to collect-then-merge:
- Settings sources (rules, HTTP, redis, external maps) now collect
layers via collect_settings_layer() instead of applying directly
- New SETTINGS_APPLY prefilter (priority high=9, below top=10)
merges collected layers and applies the result
- Prefilter priority ordering ensures SETTINGS_APPLY runs after
all collectors (including async redis callbacks) complete
- Single-layer case applies directly without merge overhead
- New task:merge_and_apply_settings() Lua method calls C merge
Vsevolod Stakhov [Mon, 30 Mar 2026 15:02:38 +0000 (16:02 +0100)]
[Feature] Allow settings_elt and settings to coexist
Remove mutual exclusion between Settings-ID and Settings header.
Both mechanisms now live on the same task: settings_elt provides
precomputed bitset gating, settings UCL provides runtime overrides
for scores, actions, and symbol enable/disable.
Also allow lua_task_set_settings to replace existing settings
instead of erroring, preparing for future merge infrastructure.
Vsevolod Stakhov [Mon, 30 Mar 2026 15:00:43 +0000 (16:00 +0100)]
[Feature] Add weak dependencies and disabled status to symcache
Introduce weak/hard dependency distinction in the symbol cache.
When a hard dependency is disabled by settings, the dependent symbol
is cascade-disabled. Weak dependencies allow the dependent to proceed
without the dependency's output.
Add `disabled` status to cache_item_status (distinct from `finished`)
so cascade-disable logic can distinguish "genuinely completed" from
"disabled by settings".
Annotate known weak deps in plugins: DKIM signing, DMARC, ARC, RBL,
bounce detection, and header checks.
Vsevolod Stakhov [Mon, 30 Mar 2026 11:56:55 +0000 (12:56 +0100)]
Release 4.0.0
** Incompatible changes **
* Replace Jump Hash with Ring Hash (Ketama) for consistent upstream hashing;
per-user Bayes with Redis sharding requires migration via rspamadm statistics_dump migrate
* Include content URLs by default in URL API calls (include_content_urls = true)
* Auto-detect SSL from bind sockets, remove ssl = true worker option
* Replace libfasttext with built-in mmap-based shim; ENABLE_FASTTEXT cmake option removed
* Replace builtin_suspicious TLDs with map-based configuration
* Rename neural autolearn options to match RBL module naming
* proxy: Enable token bucket load balancing by default
* Disable Validity SenderScore RBLs by default (requires MyValidity account)
* Make unknown and broken DKIM keys behaviour conforming to RFC
** Major features **
* Add /checkv3 multipart scan endpoint with per-part zstd compression
* Pluggable async hyperscan cache backend with Redis storage support
* Add multi-flag fuzzy hash support with Lua-based Redis update path (epoch 12)
* Add dual-mode HTML fuzzy: template matching + phishing detection
* Implement HTTPS server support for workers
* proxy: Implement token bucket load balancing for upstreams
* Expose milter headers and extended symbols in legacy RSPAMC/SPAMC protocol
* Add native UUID v7 per task with ClickHouse support
* Add structured formatter to metadata_exporter with zstd compression
* arc: Add trusted_authserv_id option for reuse_auth_results
* Add external pretrained neural model support
* Fasttext embed: multi-model, mean+max pooling, SIF word weighting
* Multi-layer funnel architecture for LLM embeddings
* Add expression-based autolearn for neural LLM providers
* Add rspamadm autolearnstats, logstats, mapstats subcommands
* Add shard migration and multi-class support to statistics_dump
* headers_checks: Add Reply-To address validity checks
* Store matched fuzzy hashes in Redis history
* Add HTTP content negotiation framework with zstd compression
* Extend /stat and /bayes/classifiers endpoints with classifier metadata
* WebUI: Add multi-class classifier support to learning UI
* Split attachment filenames into sub-tokens for Bayes classifier
* Make GPT consensus thresholds configurable with context_augment hook
** Bug fixes **
* [CritFix] Stop ev_io watcher in fuzzy UDP session destroy
* Fix CPU busy-loop in fuzzy TCP client
* Fix SPF address family flag inheritance
* Fix RHEL/CentOS 10+ crypto-policies for SHA-1 DKIM verification
* Fix EVP_PKEY_CTX memory leak in DKIM RSA signing
* Fix ratelimit compatibility with old records
* Fix use-after-free in pending regexp map and multipattern queues
* Self-healing hyperscan cache: delete stale blobs and trigger recompile
* Move binary data from KEYS to ARGV in Redis scripts
* Rework alternative parts detection
* Add PCRE2 complexity checks before JIT compilation
* Default map URL path to "/" when no path component is present
[Fix] Skip HELO IP address literals for SPF domain resolution
Per RFC 7208 §2.3, IP address literals (e.g. [192.0.2.1]) cannot be
used as SPF domains. When MAIL FROM is empty, rspamd falls back to
HELO as the SPF domain.
With an IP literal in HELO, rspamd_spf_resolve() failed (DNS name
rejected by the resolver), but spf_cred was non-NULL, so the result
was incorrectly set to R_SPF_DNSFAIL instead of R_SPF_NA.
Skip building spf_cred from HELO when it starts with '[', so
rspamd_spf_get_cred() returns NULL and the result is R_SPF_NA.
[Feature] Split attachment filenames into sub-tokens for Bayes classifier
Instead of a single whole-filename token, generate a structured set:
- #f: original filename (case-preserved, backward-compatible)
- #fe: last extension + double extension when present (e.g. #fe:gz, #fe:tar.gz)
- #fp: meaningful parts split by delimiters, CamelCase and letter/digit
boundaries; purely numeric and single-character parts discarded
Improves generalization for emails with similar attachment names differing
only in dates, IDs or other numeric suffixes, and provides a strong signal
for malware disguise patterns (e.g. pdf.exe, doc.vbs).
Vsevolod Stakhov [Sat, 28 Mar 2026 09:14:15 +0000 (09:14 +0000)]
[Fix] Default map URL path to "/" when no path component is present
When an HTTP(S) map URL has no path (e.g. "https://maps.rspamd.com"),
the URL parser leaves hdata->path as NULL. This causes a segfault in
write_http_request() which calls strlen() on the NULL path pointer.
Default path to "/" and rest to "" when UF_PATH is not set, matching
standard HTTP semantics.
Vsevolod Stakhov [Sat, 28 Mar 2026 09:07:19 +0000 (09:07 +0000)]
[Fix] Exclude injected parts from SA body/rawbody regexp scanning
PDF text extraction injects synthetic text parts that were being scanned
by sa_body and sa_raw_body rules, causing 30x false positive increase on
rules matching null bytes (e.g. /\x00/{sa_raw_body}). PDF hex strings
produce raw bytes including \x00 which are not meaningful in extracted text.
Two fixes:
- Strip null bytes and control characters from extracted PDF text in
sanitize_pdf_text() for the non-UTF-16 code path
- Skip RSPAMD_MIME_PART_COMPUTED parts in SA body and rawbody scanning
to follow original SA semantics where only real MIME text parts are matched
Vsevolod Stakhov [Fri, 27 Mar 2026 09:10:56 +0000 (09:10 +0000)]
[CritFix] Stop ev_io watcher in fuzzy UDP session destroy
fuzzy_session_destroy() freed the session without stopping a
potentially active ev_io watcher (pending UDP write retry on EAGAIN).
The watcher remained in libev's anfds[fd].head linked list pointing
to freed memory, corrupting the list and causing fd_reify() to loop
infinitely on the next fd_change — making the worker consume 100% CPU
and stop processing queries entirely.
The TCP counterpart (fuzzy_tcp_session_destroy) already had the
correct ev_io_stop guard. Apply the same pattern for UDP sessions.
Vsevolod Stakhov [Thu, 26 Mar 2026 12:03:57 +0000 (12:03 +0000)]
[Fix] Handle OpenAI-compatible response format in Ollama GPT module
Ollama's /v1/chat/completions endpoint returns responses in OpenAI
format ({choices: [{message: {content: ...}}]}) rather than native
Ollama format ({message: {content: ...}}). The ollama conversion
functions only handled the native format, failing with "bad message
in reply" for users connecting via the OpenAI-compatible endpoint.
Add ollama_extract_content helper that handles both response formats.
Vsevolod Stakhov [Thu, 26 Mar 2026 11:51:04 +0000 (11:51 +0000)]
[Fix] Add text attachment fallback in get_displayed_text_part
When all text parts are classified as attachments (e.g. due to
Content-Type name= parameter), get_displayed_text_part returned nil
causing GPT module to skip with "no text part found". Add text/plain
attachment fallback (< 100KB) similar to existing HTML attachment
fallback.
Also propagate min_words setting consistently through the extraction
pipeline: gpt.lua -> llm_common.build_llm_input -> extract_text_limited
-> get_displayed_text_part, so all three call sites use the same
threshold.
Vsevolod Stakhov [Thu, 26 Mar 2026 10:23:06 +0000 (10:23 +0000)]
[Fix] Clear inherited address family flags when creating new SPF addr nodes
When spf_record_process_addr creates a new address node for subsequent
DNS replies (MX A/AAAA resolution), it copies the original addr via
memcpy, inheriting all flags including the address family flag from the
first reply. The new reply's flag was OR'd in without clearing the
inherited one, resulting in nodes with both IPV4 and IPV6 flags set.
Since spf_addr_mask_to_string checks IPV4 before IPV6, such nodes would
always display as IPv4, causing duplicate IPv4 addresses in output
instead of the correct IPv4 + IPv6 pair.
Vsevolod Stakhov [Wed, 25 Mar 2026 21:28:41 +0000 (21:28 +0000)]
[Fix] Load only the notified scope on per-scope hyperscan notifications
When hs_helper compiles scopes sequentially, it sends a per-scope
HS_LOADED notification after each scope finishes. Workers previously
ignored the scope field and attempted to reload ALL scopes on every
notification, causing spurious "no valid expressions" errors for scopes
that hadn't been compiled yet.
Now workers dispatch per-scope notifications to a new single-scope
loader (rspamd_re_cache_load_hyperscan_single_scope_async), and only
the final empty-scope notification triggers a full reload of all scopes.
Vsevolod Stakhov [Wed, 25 Mar 2026 19:48:35 +0000 (19:48 +0000)]
[Fix] Defer settings application for symbols registered after settings init
When regexp_rules maps (or other dynamic sources) register symbols after
settings have already been processed, the symbols_enabled/symbols_disabled
lists were silently ignored. Store pending settings operations and apply
them when the symbol is later registered via add_symbol_with_callback or
add_virtual_symbol.
Vsevolod Stakhov [Wed, 25 Mar 2026 16:18:41 +0000 (16:18 +0000)]
[Fix] Only request recompile on deserialize failure, not cache miss
Avoid infinite recompile loop: only send RSPAMD_SRV_RECOMPILE_REQUEST
when a cached blob exists but fails to deserialize (stale/corrupt),
not on cache misses where hs_helper cannot compile certain expressions.
Vsevolod Stakhov [Wed, 25 Mar 2026 11:35:58 +0000 (11:35 +0000)]
[Fix] Self-healing hyperscan cache: delete stale blobs and trigger recompile
When workers fail to deserialize a cached hyperscan blob (version
mismatch, corruption, etc.), the system previously got stuck in PCRE
fallback mode permanently because hs_helper's exists check would pass
for the stale entry and skip recompilation.
Now workers delete bad cache entries immediately on deserialize failure
and send RSPAMD_SRV_RECOMPILE_REQUEST to trigger hs_helper to recompile.
A 5-second debounce in hs_helper prevents thrashing from multiple workers
detecting the same bad blobs simultaneously.
Also removes the dead needs_recompile flag which was set on the worker's
re_class but checked during compilation in hs_helper (different process).
Vsevolod Stakhov [Wed, 25 Mar 2026 08:32:38 +0000 (08:32 +0000)]
[Fix] Backport fixes from libucl
- ucl_msgpack: add bounds check after switch in ucl_msgpack_parse_ignore
to prevent heap-buffer-overread when len is modified by fixext types
- ucl_sexp: fix broken length parsing, correct bounds check direction,
add null pointer checks, fix memory leaks in error paths
projects / thirdparty / rspamd.git / log
