[Feature] fuzzy: track unkeyed clients in fuzzystat
Traffic from clients without a key (e.g. permitted by allow_update)
was invisible in the key statistics, which makes storage writers
untraceable. Track such clients under a dedicated 'unkeyed'
pseudo-key with the same aggregate and per-IP counters as regular
keys.
[Feature] fuzzy: per-hash introspection via rspamadm control fuzzyhash
rspamadm control fuzzyhash <128 hex digest> asks all fuzzy workers for
storage-side diagnostics of a specific hash: flag slots and values,
creation time, remaining ttl and, using the persisted shingle set,
slot ownership (owned/foreign/vacant out of total). Ownership directly
answers the key false-positive triage question: can this hash still
produce fuzzy matches, or has its shingle anchor decayed?
Implemented as a new inspect backend API: the redis backend gathers
everything in a single server-side script round trip; sqlite queries
the digests and shingles tables synchronously. The worker control
handler replies asynchronously from the backend callback (the control
pipe is persistent) using the same fd-attachment mechanism as
fuzzy_stat.
[Feature] fuzzy: persist shingle sets with digests in redis
Store the shingle key suffixes as the 'S' field of the digest hash on
every ADD, making each digest self-describing. This fixes two
long-standing blind spots:
- DEL: digest-only deletions (e.g. periodic deletions by a hash list)
carried no shingle information, so the digest's shingle slots were
left orphaned; now they are reconstructed from the persisted set and
removed when still owned by the deleted digest.
- REFRESH: the refresh used to touch the shingle slots of the message
that happened to trigger it (a different message by definition);
now it refreshes the slots the digest still owns and reclaims the
vacant ones, so a hash that keeps matching keeps its full shingle
anchor instead of decaying.
The sqlite backend needs no changes: its shingles table references
digests with ON DELETE CASCADE and foreign keys are enforced.
rspamadm fuzzy_hash computes the fuzzy hashes of a message per
configured rule (matching the scanner tokenization bit for bit),
optionally checks them against the storage with shingles included
(-C), and queries storages for explicit hex digests (-H), printing
found/queried digests, probability, flag, timestamp and whether the
storage confirmed the matched digest.
Add Fuzzy.Hash/Fuzzy.Queried/Fuzzy.Rule/Fuzzy.Prob/Fuzzy.Flag nested
columns (schema version 11) filled from structured fuzzy results, so
false positive investigations can start from the scanner-side journal
instead of storage forensics.
X-Rspamd-Fuzzy now uses structured fuzzy results: every matched hash
is annotated with the rule, flag, probability, the queried hash for
non-exact matches and the storage timestamp. Falls back to the legacy
fuzzy_hashes pool variable when no structured results exist.
[Feature] fuzzy_check: structured match results and diagnostics API
Every fuzzy match now produces a structured record (rule, symbol,
upstream, stored and queried digests, type, probability, score, flag,
value, hash timestamp) kept in the fuzzy_matches mempool variable and
exposed to Lua via task:get_fuzzy_results(). All queried hashes,
misses included, are recorded in the fuzzy_checked variable. The
symbol option for non-exact matches now carries both hash prefixes
(flag:found:prob:type:queried), so a match is diagnosable from a scan
report alone.
The storage sets a flag bit in the first reserved byte of the reply
when the digest field contains the digest actually resolved by the
backend; the client propagates it as the 'confirmed' field, removing
the guesswork about echoed queries when talking to legacy storages.
Add fuzzy_check.check(task, cb, rule, timeout[, hashes][, server]) Lua
API: queries a storage either with the hashes generated from the task
message (shingles included) or with an explicit list of hex digests,
reporting per-reply results including misses and error codes.
[Feature] fuzzy_check: anchor probability weight curve at the match threshold
The score multiplier for non-exact matches was sqrt(prob): a concave
curve anchored at zero. Since the storage never returns prob below the
shingle match threshold (0.5), the whole reachable range collapsed into
(~0.73 .. 1.0], so a marginal 17/32 shingle overlap scored almost as
high as an exact match; combined with high-weight deny lists this
turned weak matches into instant rejects.
Replace it with ((prob - prob_bias) / (1 - prob_bias)) ^ prob_power,
anchored at the match threshold (prob_bias 0.5). prob_power defaults to
1.0: the multiplier is the fraction of the way from the threshold to an
exact match (19/32 -> 0.19, 24/32 -> 0.50, 28/32 -> 0.75); raise it per
rule for harsher discounting of weak matches. The old zero-anchored
sqrt curve is not preserved: it was simply broken. Applies to text and
HTML hashes; small images keep their existing normalized curve.
[Fix] fuzzy_check: log rule and server for error replies
403 (rate limit), 503 (access denied) and 415 (encryption required)
replies used to insert their symbols silently; with several rules
configured there was no way to tell from the logs which rule and which
server produced the error. Log an info message with the rule name and
the upstream in both the UDP and TCP reply paths.
[Fix] cfg: warn when a module section is defined multiple times
rspamd_config_get_module_opt() returns the 'rule'/option object from the
first section only when a module section is duplicated at the top level
(e.g. a stray file in modules.d), so the remaining sections are silently
ignored and their rules are never loaded. Lua modules are not affected
as get_all_opt() flattens the whole duplicate chain.
Emit an explicit warning per configured C module so configtest and the
startup log surface the problem instead of silently dropping rules.
[Fix] lua_tcp: make connection ref release one-shot to avoid double free
A Lua callback calling close() sets the sticky LUA_TCP_FLAG_FINISHED
flag, which is then observed by more than one release site (the checks
in lua_tcp_push_data/lua_tcp_push_error, the post-helper check in
lua_tcp_handler, the fatal-error caller drops and the handlers queue
drain in lua_tcp_plan_handler_event), dropping the connection ownership
reference twice. That freed the cbd while lua_tcp_handler was still
running and crashed on the subsequent TCP_RELEASE (heap-use-after-free
on ref.refcount).
Guard the ownership drop with LUA_TCP_FLAG_FIN_RELEASED via the
lua_tcp_release_conn() helper so the first site to observe termination
wins and the rest become no-ops.
[Fix] lua_content/pdf: emit newlines for Td/TD line breaks
The Td/TD text positioning operators never produced a newline: the
grammar did not capture the operator name, so the handler compared a
numeric operand against the string 'Td' (never true) and additionally
read the wrong operand. As a result consecutive text lines were
concatenated, e.g. "...June 1, 2026." + "You may obtain..." became
"2026.You", which the URL parser mis-detected as http://2026.you and
raised false positives.
Capture the Td/TD operator, split the graphics ternary operators (d/m/l)
into their own dropped branch, and render a newline when the vertical
displacement (ty) is non-zero. Adds a regression unit test.
Microsoft's JMRP feedback loop ships a well-formed
multipart/report;report-type=feedback-report whose
message/feedback-report block only carries
Feedback-Type/User-Agent/Version. Source-IP, Arrival-Date,
Original-Mail-From and Original-Rcpt-To are omitted; that data lives
only in the embedded original-message headers.
parse_arf now recovers those fields from the original headers, filling
only what the report itself left empty and recording provenance in
result.derived so callers can tell reported data from inferred data:
* source_ip: X-Originating-IP, then client-ip= of Received-SPF /
Authentication-Results, then the first public IP while walking the
Received chain (is_nonpublic_ip skips RFC1918/CGNAT/ULA hops, which
ip:is_local() does not).
* arrival_date: topmost Received timestamp, else Date.
* original_mail_from: Return-Path, else smtp.mailfrom.
* original_rcpt_to: Delivered-To / X-Delivered-To / envelope-to.
* reported_domain: falls back to the envelope-from domain.
Also accept Received-Date as an Arrival-Date alias in the report block.
extract_original_message now returns the raw header maps alongside the
parsed subset; DSN parsing is unchanged. Adds unit tests for the
Received-SPF path and the public-IP-in-Received-chain fallback.
The style dimension parser found a digit but called rspamd_strtoul from
the start of the substring, ignored its return value, and assigned the
result. For substrings longer than 22 chars rspamd_strtoul returns
before writing *value, leaving img->height/width set from an
uninitialized variable; shorter CSS parsed to zero because the substring
began with ':' or whitespace.
Parse the digit run starting at the located digit, bound the length to
that run, and only assign on a successful parse.
[Feature] lua_http: forbid_local option to block requests to local networks
rspamd_http.request() connects to whatever the URL or its DNS resolution
yields, including loopback, link-local and RFC1918 destinations. Add a
forbid_local option enforced at the single connection chokepoint (after
keepalive, upstream, numeric and DNS address selection) using
rspamd_ip_is_local_cfg, i.e. the address class check plus the
configurable local_addrs radix.
Enable it by default in url_redirector: message URLs and their redirect
targets are attacker controlled, and hop limits alone do not stop a
crafted redirect from probing cloud metadata endpoints or internal
services. Operators resolving internal redirectors can set
forbid_local = false.
[Fix] lua_http: deliver errors to coroutine callers
DNS-stage failures (resolve error, no records, connection refused) went
through lua_http_push_error, which unconditionally invoked the callback
reference; coroutine-style requests have no callback (cbref == -1), so
the error was swallowed and the yielded thread was never resumed. Resume
the thread with (err, nil) instead, mirroring the connection error path.
Synchronous failures (unparseable URL, blocked session, immediate
connection or DNS-send failure) returned a bare false, which coroutine
callers read as err = false, response = nil - a crash on response.code
for every caller checking 'if not err'. Return (err, nil) there as well.
[Fix] http: enforce read deadline when data is pending at timer expiry
The read timer is a one-shot ev_timer. When it expired while data was
already waiting in the socket, the handler drained and parsed that data;
if the message remained incomplete, the connection continued with only
the I/O watcher active and no deadline at all. Report a 408 instead,
unless the drained bytes completed the message (finished or reset into
the keepalive pool).
Add a deterministic lifecycle test driving the expiry window via the
fake clock plus a max-priority ev_check writer, covering the incomplete,
salvaged and no-data cases.
Lua HTTP client responses were unlimited unless a plugin explicitly
passed max_size, and a negative max_size was converted straight to an
unsigned gsize, silently producing an effectively unlimited cap.
Add a max_lua_http_response option (256Mb by default, 0 disables)
applied as the reply body limit whenever a request does not specify
max_size; an explicit max_size of 0 still disables the limit per
request. Reject negative max_size values with a logged error instead
of letting them wrap.
[Fix] zstd: share one bounded decompression helper across HTTP, proxy and maps
The streaming zstd decompression loop was copy-pasted in seven places
with inconsistent bounding. The proxy had no output ceiling at all: on
request/response bodies (proxy_request_decompress) and on the v3
multipart result and body parts, a zstd bomb from a client or backend
could balloon memory without limit. The task and v3 protocol paths were
bounded but enforced the limit only when the buffer filled, letting the
output overshoot max_message up to twofold, and never validated the
final length. The map shm-cache, file-cache and static paths grew their
buffers with no ceiling.
Add rspamd_zstd_decompress_bounded that validates the frame-advertised
size before any allocation, caps buffer growth strictly at the limit,
drains pending decoder output after the input is consumed and frees the
buffer on every error path. Convert all seven sites to it: max_message
bounds the task, protocol v3 and proxy paths; max_map_size bounds the
map cache and static paths (the remote HTTP map path keeps its per-map
override). Cover the helper with unit tests including advertised-size
and unknown-size zstd bombs.
[Fix] map: bound remote HTTP map sizes, compressed and decompressed
Remote HTTP maps had no size ceiling: the map client connections never
called rspamd_http_connection_set_max_size, so a map server could feed
an arbitrarily large body that is fully retained in shared memory, and
the zstd path allocated the output buffer straight from the
frame-advertised decompressed size, then doubled it without limit, so a
small zstd bomb could balloon memory indefinitely.
Add a max_map_size option (256Mb by default, 0 disables) applied as the
HTTP body limit on both map client connections, reject frames
advertising a larger decompressed size upfront and enforce the same
ceiling in the streaming growth loop. The limit can be overridden per
map with max_size in the maps { } block, like the existing timeout and
keepalive knobs.
[Fix] http: bound request bodies on controller, proxy and control sockets
The normal worker caps HTTP bodies at cfg->max_message, but the
controller router, the proxy client connection and the main control
socket never called rspamd_http_connection_set_max_size. A declared
Content-Length caused an unbounded preallocation and chunked input
could grow until the connection ended, all before routing or
authentication.
Add a max_size knob to the HTTP router applied to every accepted
connection, and set cfg->max_message as the body cap on the
controller, proxy and control sockets. Oversized requests are now
rejected with 413 (controller) or a closed connection (proxy).
[Fix] task: bound zstd decompression by max_message and plug error-path leak
The legacy compressed-message path in rspamd_task_load_message allocated
the output buffer straight from the frame-advertised decompressed size
and doubled it without any ceiling, so a zstd bomb within the HTTP
max_size limit could balloon memory far beyond max_message. The
decompression-error return also leaked the output buffer, which was
registered with the task pool only on success.
Reject frames advertising more than max_message upfront, enforce
max_message in the buffer growth loop, and free the buffer on all error
returns. Apply the same upfront clamp to the v3 multipart path and honor
max_message == 0 as unlimited there, matching the HTTP layer convention.
[Fix] url: enforce max_urls at the central insertion boundary
All URL sources (HTML links, query URLs, images, displayed URLs,
subject URLs, Lua-injected URLs) now respect the configured max_urls
limit via rspamd_url_set_add_or_increase/add_or_return, not just
plain-text callbacks. Text pre-checks fixed from > to >= so the cap
is exact. Also remove a duplicate part-url add for newly-seen text
URLs introduced in 80f7567fb.
[Fix] 7zip: bound folder count and guarantee parser progress
The folder-count limit was applied only in the internal-folders branch,
so the external branch preserved an unchecked 64-bit count. The later
kCodersUnPackSize loop's error branch then logged once per folder
without consuming input, allowing a non-progressing CPU/log loop that
the synchronous parser cannot be interrupted from by the task timeout.
- Apply the > 8192 limit immediately after reading NumFolders, before
the external/internal split.
- Reject external-folder metadata instead of accepting it with
inconsistent downstream state.
- Replace the non-progressing error branch with an early return so every
parsing-loop iteration advances input or bails out.
- Bounds-check *p (the External flag) after the varint read.
- Make the uint64 -> unsigned int folder-count narrowing explicit.
- Move both folder_nstreams tables from g_alloca/g_malloc to the task
mempool to reduce stack risk and drop manual frees.
A single Content-Type or Content-Disposition field can carry an
unbounded number of parameters. Each one materialises a separate pool
object plus a hash/list entry, so a message bounded at max_message (50
MiB by default) could amplify into millions of allocations from one
folded header (~20x for Content-Type, more for Content-Disposition
which copies name and value per param). The existing header-count cap
limits header *fields*, not parameters within a field, so it does not
help here.
Cap the number of parameters materialised per header at 1024 (real
messages use a handful). Content-Type flags the truncation as BROKEN;
Content-Disposition has no flags field and simply stops. The counter
counts parameters, not distinct names, so a same-name flood is bounded
too.
Also fix rspamd_cmp_pieces: it subtracted two unsigned RFC 2231
continuation ids and truncated to int32_t, which overflows when the
ids differ by more than INT32_MAX and reverses the reconstruction
order. Use an explicit comparison instead.
Adds a C++ unit suite covering the parameter caps and the ordering fix.
Newline normalization recorded every newline as a pointer array entry
plus a mempool exception object and a GList node, then sorted the
whole exceptions list: a newline-dense body turned a few MiB of input
into hundreds of MiB of heap and millions of allocations. Cap the
recorded newline positions at 100k per part (text normalization and
line counters are unaffected), set a part flag on truncation and
expose it as newlines_truncated in textpart:get_stats().
[Fix] Bound alt-part linking and fasttext langdet cost
Alternative text part linking used one subtree scan per text part, so
a crafted multipart/alternative with thousands of sibling text parts
made linking quadratic (~100M sibling visits at the MIME part cap).
All searches of a task now share a single visit budget; when it is
exhausted the remaining parts are left unlinked with a warning.
Fasttext language detection fed up to 1M words into the model with no
bound on word length or on the resulting token count. As the UTF
tokenizer intentionally does not enforce max_word_len, a crafted body
made of huge unspaced words expanded into 3-4 subword ngram ids per
character: hundreds of megabytes of heap and matching vector-add work
in predict() from one message. Skip overlong words, cap the total
tokens fed to the model, and refuse OOV subword expansion in the shim
for words longer than any real dictionary entry.
[Fix] message: link plain text parts to their HTML alternative
The alternative part search compared the raw IS_TEXT_PART_HTML flag
value (0 or 4) with the boolean want_html argument (0 or 1), so a
text/plain part never matched its text/html sibling: only the
html-to-plain direction of alt_text_part was ever populated.
[Fix] html: eliminate DOM recursion to survive deeply nested messages
Convert html_append_tag_content, traverse_block_tags and
html_debug_structure from native-stack recursion to explicit heap
stacks, and maintain html_tag::depth incrementally instead of walking
the parent chain per tag. With max_tags = 8192 a message can legally
nest ~8190 elements, which overflowed 512KB worker-thread stacks.
A chain of bare "Content-Type: message/rfc822" wrappers recursed through
rspamd_mime_parse_message's own MESSAGE branch without ever tripping the
max_nested (64) limit. When descending into an embedded message the
parser allocates a fresh runtime (nst) and seeds nst->nesting from the
parent's value *before* incrementing, then increments the parent runtime
st which is not the one carried into the recursion. The multipart branch
and rspamd_mime_process_multipart_node bump the descended-through runtime
(and push it onto the stack so the recursive call's cleanup pops and
decrements it), but the message branch did neither, so nst->nesting
stayed at its initial value at every level and the entry guard never
fired.
The result was recursion bounded only by message size (~34 bytes per
level): deeply nested messages exhaust the worker stack (remote,
unauthenticated DoS via HTTP submission or SMTP/milter delivery), and the
per-level boundary pre-scan made even non-crashing depths quadratic in
CPU. The earlier S/MIME fix (f6536945) only covered the pkcs7-mime
re-entry path and did not address this.
Push npart onto nst->stack and bump nst->nesting in the message branch,
mirroring the multipart branch; the recursive call's existing cleanup
balances both. Bounding the depth to max_nested also caps the number of
preprocess passes, eliminating the quadratic-CPU vector.
Add a Lua unit regression test that feeds a 500-level message/rfc822
chain and asserts the parsed part count stays capped near max_nested.
[Feature] Configdump: show symcache flags in --symbol-details
Add a `flags` array (fine, empty, explicit_disable, explicit_enable,
ignore_passthrough, nostat, idempotent, mime, trivial, skip, composite,
ghost, coro) to the per-symbol details, so it is possible to check
whether a symbol requires explicit enabling; structural types are
already covered by the `type` field.
[Test] Settings: cover explicit_enable gating and implicit_allow policy
New functional test cases:
* explicit_enable symbols do not run without settings, under an
implicit-allow settings id, or with raw settings that do not list
them
* whitelist settings and task:enable_symbol() unlock them
* policy = implicit_allow keeps all symbols enabled while unlocking
listed explicit_enable symbols and disabling listed symbols, both
for a registered settings id and for raw settings
[Feature] Settings: add policy option and strict explicit_enable gating
Make symbols with the explicit_enable flag actually require explicit
enabling: previously they were executed whenever any settings were
applied to a task, as the implicit-allow settings branch in
cache_item::is_allowed never consulted the flag, and any non-null
task->settings unlocked such symbols regardless of content.
Now explicit_enable symbols run only when:
* listed in symbols_enabled of the applied settings (any policy), or
* force-enabled via task:enable_symbol()
To make it possible to combine the default-allow behaviour with
selectively enabled explicit_enable symbols and disabled symbols,
introduce a settings policy option in the apply block:
With policy = implicit_allow, symbols_enabled no longer switches the
settings to whitelist mode: all symbols stay enabled, the listed ones
are merely unlocked. The policy is also honoured for raw settings
passed via the Settings header.
Additionally, symbols listed in symbols_enabled are now always tracked
as force-enabled (previously only when the settings element was already
attached to the task, which missed the map-driven settings path where
the id is set after settings are applied).
[Fix] Split MIME address lists on semicolons as well as commas
A semicolon is not an RFC 5322 address-list separator (it only
terminates a group construct), but Outlook-style ;-separated recipient
lists are pervasive in real mail and most MTAs (e.g. Postfix) split
them. Previously such a list was parsed as a single malformed address
whose local part swallowed the semicolon and the next address, breaking
recipient counts and producing false invalid-address signals.
Treat ';' exactly like ',' in the list parser. This mirrors the
existing comma behaviour in every edge case: semicolons inside quoted
strings are preserved, an unquoted semicolon in a display name drops
the preceding phrase just like a comma does, an empty group
(undisclosed-recipients:;) now yields no addresses like the comma
variant, and the trailing ';' of a group construct no longer leaks
into the last member's domain.
Fix synchronous context_augment firing two GPT requests for one model
When your context_augment function calls the callback synchronously
(e.g., return function(task, content, cb) cb('some context') end),
the pending_fetches counter drops to 0 inside the callback,
triggering maybe_proceed() which calls proceed().
Then control returns to the main flow where the now-redundant guard
also sees pending_fetches == 0 and calls proceed(nil) a second time
— this time without the context snippet.
The else branch (line 1324) already handles the case where none
of the context features are enabled, so the removed fallback
was purely a duplicate path for synchronous callbacks.
[Test] Use unique envelope senders in aliases From tests
The three From-identity tests shared one envelope From/Rcpt pair, so the
greylist module (meta key = envelope from + rcpts + ip) soft-rejected
the third scan of the pair within its retry window, skipping the filters
stage and losing the GET_FROM symbol. This only manifested with tight
test pacing (pabot shard in CI); give each test a unique envelope
sender, as the rest of the suite does.
[Minor] Selectors: document the 'orig' address flavour
The from/rcpts extractors pass their arguments through to task:get_from
and task:get_recipients, so from('mime', 'orig') already selects the
address as it was seen in the message; document it and add a functional
test exercising the flavour through the selector pipeline.
[Fix] Forged recipients: compare addresses as they were transmitted
The rule compared the rewritten envelope view against the wire MIME
headers, so a recipient alias mapping to another domain (legal for
recipients) could make a perfectly matching To header look forged.
Compare the envelope and the headers at the same altitude: both as they
were seen in the message, which is also what this rule checked before
alias rewriting was applied to tasks.
[Test] Add functional tests for From identity preservation
- aliases: googlemail.com From keeps its domain while the user part is
canonicalized; a cross-domain virtual alias on the From is discarded;
a same-domain alias on the From local part still applies
- get_from: the 'orig' flavour returns the wire From (and only it) after
a task:set_from rewrite, and is a no-op without one
- dmarc: alignment and policy lookup use the wire From domain even when
a prefilter rewrote the MIME From
The REWRITE_MIME_FROM test prefilter is gated on a request header so it
stays inert in merged-suite tests that do not opt in.
SPF evaluates the RFC5321.MailFrom as it was transmitted (including
local-part macros), so resolve the SPF record and credentials from the
original envelope sender even if the envelope from has been rewritten,
e.g. by the aliases module.
[Fix] DMARC: evaluate the From and envelope domains as seen in the message
DMARC must align identifiers and look up the policy for the RFC5322.From
domain as it appears in the message, and check SPF alignment against the
RFC5321.MailFrom as transmitted. Use the 'orig' address flavour so that
rewrites done via task:set_from (e.g. by the aliases module) do not
affect alignment, policy lookup or rua report attribution.
[Rework] Lua API: make the 'orig' address flavour return wire addresses
Previously task:get_from({'mime', 'orig'}) returned both the preserved
original addresses and the rewritten ones appended by task:set_from,
making the flavour unusable for consumers that need the message as it
arrived (a duplicate-From check would misfire on the combined list, and
Lua could not tell the entries apart). Now, when the 'orig' flavour is
requested and originals were preserved by a rewrite, only the original
addresses are returned; without a rewrite the behaviour is unchanged.
Also expose the 'aliased' and 'original' email address flags to Lua and
document the flavour.
The From address is the sender's identity: SPF, DKIM and DMARC all key
on its domain. Service rules and alias resolution (e.g. a virtual alias
mapping to another domain) may only touch the local part of the From
address; any rewrite that would change the domain is now discarded.
Recipient rewriting is unaffected: recipients are interpreted by the
receiving host, and no authentication protocol keys on them.
Also fix a spurious task:set_from call: the set_from decision used the
callback-wide alias_resolved flag, so a rewrite of any earlier address
(e.g. an envelope recipient) caused the unmodified MIME From to be
marked as aliased and duplicated.
[Fix] Aliases: never rewrite googlemail.com domain to gmail.com
googlemail.com and gmail.com are DNS-distinct domains with independent
SPF and DMARC records (gmail.com publishes p=none while googlemail.com
publishes p=quarantine), so substituting one for the other breaks DKIM
alignment, changes the effective DMARC policy and misattributes DMARC
reports. Keep the gmail.com user-part canonicalization (dots and plus
tags) for googlemail.com addresses, but preserve the domain.
The domain equivalence remains available in lua_util.remove_email_aliases
for dedup-only consumers such as email hash lookups in RBL checks.
[Fix] Lua 5.5 compatibility: do not assign to for-loop variables
Lua 5.5 makes for-loop control variables read-only, so assigning to
them is now a compile-time error. Rebind the value to a local (or pass
it as a closure parameter) in all remaining places; found by
parse-checking the whole tree with luac 5.5.
Register waitForResponse listeners before clicking the reset button:
on localhost the historyreset response arrives before a listener
registered after the click can see it, which made the legacy
(Playwright 1.45.3) CI leg fail deterministically.
Lua 5.5 makes `for` expression variables constant. The recent upgrade to
lua 5.5 broke lua-lupa, which previously attempted to modify the loop
variable. This caused an unrecoverable error on startup:
rspamd_lua_require_function: require of lua_util.jinja_template failed: error loading module 'lupa' from file '/usr/share/rspamd/lualib/lupa.lua':\x0A\x09/usr/share/rspamd/lualib/lupa.lua:1858: attempt to assign to const variable 'word'
[Minor] WebUI: drop unreachable 304 success branch
The Rspamd controller sends "Cache-Control: no-store" on every WebUI
response (controller.c, issue #3330) and never emits Last-Modified/ETag
for these endpoints, so a browser never revalidates and a 304 response
is impossible. The `|| xhr.status === 304` in queryServer was dead code
copied from jQuery's default success range.
Remove it so queryServer, the /stat connect probe and the legacy /auth
probe all treat 200-299 as success. Treating 304 as an error is also
safer: a 304 body is empty, and success would have fed it to the JSON
parser.
[Fix] WebUI: route a malformed /stat response to the login dialog
The connect probe swallowed JSON parse errors (parseJsonSafe returned {}),
so a 2xx /stat with a non-JSON body (truncated by a proxy, a captive-portal
page, corruption) loaded an empty, half-loaded main UI with a "Cannot get
server status" alert and no login path. The former jQuery content-type
parse raised parsererror and fell through to the login form.
Parse the body explicitly and, on any failure, show the connect dialog —
matching the pre-migration behaviour and covering the broad case (any
non-JSON 2xx, not only application/json). Removes the now-unused
parseJsonSafe helper.
[Fix] WebUI: restore smooth scroll on the Scan tab
The jQuery-removal replaced .animate({scrollTop}) with
window.scrollTo({behavior:"smooth"}), which honours prefers-reduced-motion
and is interruptible by Tabulator's reflows. The former jQuery tween
ignored the preference and re-set scrollTop every frame, so it was always
smooth.
Add a requestAnimationFrame-based smoothScrollTo (400ms easeInOut, matching
jQuery's default) and use it for both Scan scrolls (scan result and the
"Clean" reset). Like the rest of the UI's animations it does not honour
prefers-reduced-motion, preserving the previous behaviour.
fireAjaxComplete (the active-request counter decrement) ran at the top of
finish(), before the success callback. During an "All SERVERS" query the
count therefore dipped to 0 between the neighbours probe and the
per-neighbour requests: ajaxStart re-fired and re-added fa-spin,
restarting the spinner's CSS animation. jQuery decremented after success,
keeping the count above zero for the whole fan-out.
Move the decrement to the end of finish(), after the success/complete
callbacks. This also removes the rapid fa-spin class churn that was
interrupting the Scan tab's smooth scroll.
- stats.js: guard the optional fuzzy_hashes key in addFuzzyStorage
(Object.entries/keys of undefined threw on every Status refresh when
fuzzy_check was disabled or a neighbour was down), matching the guard
addStatfiles already had.
- common.js: cancel the hide/show slide animations once the final state
is committed, so the fill:"forwards" no longer pins height:0 — a later
re-show was measuring the pinned height and animating 0→0.
- common.js: treat a single Element (incl. <select>/<form>, which expose
a numeric .length) as one node in toElements, not a collection.
- common.js: tolerate a null neighbours body before Object.keys.
- rspamd.js: null-guard the active-tab lookup in two spots, where a
missing active tab used to throw.
- upload.js: drop the dead params:{processData:false} (queryServer no
longer form-encodes).
- common.js: document the statusCode handler signature.
[Test] selectors: cases for partially-parsed selectors
Negative cases: a second ':' method, a stray closing paren and junk
after a ';' list element must be rejected. Control cases: valid
method/transform chains, selector lists and trailing whitespace must
keep parsing.
[Fix] lua_selectors: reject selectors that parse only as a prefix
The selector grammar was not anchored to the end of input, so lpeg
matched the longest valid prefix and silently dropped the rest of the
string. E.g. 'from("smtp"):domain:lower' (a second ':' cannot parse)
was accepted and evaluated as 'from("smtp"):domain', and any trailing
garbage after a selector or a ';' list element was ignored. All
consumers (multimap, rbl, ratelimit, settings, reputation, the
controller selector check used by the WebUI) reported such selectors
as valid while evaluating only the prefix.
Append an lpeg.Cp() capture after the grammar (plus optional trailing
whitespace) and make parse_selector require the whole input to be
consumed, logging the position and the unparsed tail otherwise.
User-visible change: selectors that previously loaded thanks to the
silent truncation now fail configuration load with an error pointing
at the offending token. Such selectors were never evaluating as
written, so failing loudly is the correct behaviour.
[Fix] rspamd_symcache.h: keep C linkage of lua.h for C++ includers
The raw lua.h include sat outside extern "C", so any C++ unit that
pulled rspamd_symcache.h before lua/lua_common.h got C++-mangled
declarations of the Lua C API and failed to link. Including
lua_common.h itself is not an option due to the include cycle via
rspamd.h -> cfg_file.h -> rspamd_symcache.h, so wrap lua.h the same
way lua_common.h does. This also lets composites sources include
lua_common.h in the natural position instead of first.
[Test] composites: functional cases for Lua conditions
Cover: option-inspecting condition, cross-symbol join on a shared
option value, numeric return as atom weight combined with an
expression limit, false condition, and depends_on deferring a
composite whose condition consults a postfilter symbol.
A condition is called as f(task, symbol) where symbol is a table in the
task:get_symbol() layout; it may return true/false or a number used as
the atom weight. The condition is ANDed with option filters of the atom
and a failed condition is treated exactly as a missing symbol, so
removal policies are not applied. Since a condition can consult symbols
invisible to the expression, the optional depends_on list feeds the
first/second pass placement as if those symbols were expression atoms.
rspamd_config:add_composite() now also accepts a full definition table,
and rspamd_lua_push_symbol_result() is exported for reuse.
[Fix] phishing: don't penalise same label under another TLD
Three fixes for cross-TLD brand false positives
(e.g. brand.co.uk displayed over a brand.com href):
- Normalise the DMARC_POLICY_ALLOW domain to eSLD before comparing it
with the link target tld: DMARC reports the raw From domain (possibly
a subdomain), so the existing exclusion for authenticated mail never
matched
- Compare registrable labels instead of stripping the last dot
component: the old code only handled single-label suffixes, so
brand.co.uk vs brand.com was levenshtein-compared as full strings and
scored full weight; now the identical label yields weight 0
- Check strict_domains regardless of the computed weight (gated on
tld ~= ptld, equivalent to the old reachable behaviour), so displaying
a strictly protected domain over a same-label different-suffix target
still fires at full weight
[Test] lua_cryptobox: unit tests for secretbox nonce padding
Guard against the regression fixed in #6122: a short nonce must produce
the same ciphertext as the explicitly zero-padded 24-byte nonce, and
short/padded forms must decrypt each other. Both assertions fail on the
unpatched code as they read past the nonce buffer.
[Fix] lua_cryptobox: pass padded nonce to secretbox encrypt/decrypt
Both secretbox encrypt and decrypt built a zero-padded 24-byte real_nonce
but passed the original short nonce pointer to libsodium, reading up to
23 bytes out of bounds and producing ciphertexts that depend on adjacent
memory contents.
[Minor] maps: move glob pattern anchoring into rspamd_str_regexp_escape
Add RSPAMD_REGEXP_ESCAPE_ANCHOR flag that wraps the escaped pattern
into ^(?:...)$ within the same allocation instead of a second
g_strdup_printf in the glob map insertion path.
[Fix] maps: anchor glob map patterns to match the whole subject
Glob map entries were compiled into unanchored regexps and matched
with substring search semantics, so a `t.co` entry matched
`walmart.com` and `*.bit.ly` matched `foo.bit.ly.evil.com`.
Wrap the translated pattern into `^(?:...)$` at map load time so
glob entries match the subject as a whole: bare names match exactly,
wildcards match only what they say.
This affects all glob maps: multimap glob/glob_multi, url_redirector
redirector_hosts_map, dkim_signing/arc signing_table and key_table,
mx_check exclusions and the rbl glob returncodes matcher. Maps that
relied on the accidental substring behaviour must now use explicit
wildcards.
Register the historyreset/history response listeners before the click
via Promise.all. waitForResponse only catches responses arriving after
it is set up, and the fast local historyreset round-trip could beat a
listener attached post-click, causing a flaky 10s timeout under CI load.
Remove jQuery from the RequireJS config (paths, app/bootstrap shims) and
delete the vendored jquery-3.7.1.min.js. All app modules now load without
jQuery; the bootstrap shim's stale "Popovers require jQuery" comment and
deps entry are removed.
Drive-by: rename the modalDialog data-backdrop attribute to its Bootstrap
5 name data-bs-backdrop, so the static-backdrop behaviour is actually
applied (the BS4 attribute has been silently inert).
Migrate libft.js off jQuery. $("<tag>", {hash}) element construction
becomes common.el, $.extend becomes spread, $.each becomes forEach.
The column-options dropdown's namespaced hidden.bs.dropdown.rspamd
rebind (jQuery namespaces) becomes a common.data-tracked
addEventListener that removes the previous handler before re-binding;
the delegated fuzzy-hash buttons (re-bound every render) use a cleanup
map so common.delegate doesn't stack. $(arr).not(arr) set-diffs become
Array.some/ includes, and .siblings()/.attr()/.html()/.val() move to
native DOM.
Migrate rspamd.js off jQuery. The /stat connect probe moves to a raw
XHR (avoiding common.query's error logging on the normal "not logged
in" path); $.ajaxSetup and $(document).ajaxStart/Complete are dropped
since no $.ajax calls remain. tabClick disables/enables the nav bar
controls via attribute+class helpers (li/a need both for Bootstrap);
:radio + .val([v]) and .data("value") become checkRadio / activeMenuValue
helpers; delegated handlers move to common.delegate; BS5 events move to
addEventListener (with {once:true} replacing self-removing .on/.off);
the popover content uses cloneNode and the outside-click dismiss uses
.contains(). Form/input re-binds become onsubmit/oninput properties.
Migrate stats.js off jQuery. The delegated mouseenter/mouseleave hover
highlighting (which doesn't bubble natively) is reimplemented on
mouseover/mouseout with a relatedTarget-containment check; the jQuery
sibling-traversal chains (prevAll/nextAll/slice/find) move to native
sibling walks. $.each becomes Object.entries/forEach, $(html).appendTo
becomes insertAdjacentHTML, and $.wrapAll becomes common.el DOM
construction. The legacy /auth fan-out keeps a raw XHR to preserve its
custom error message and alerted-session dedup; $.when().always becomes
Promise.all().finally. Adds common.getAjaxTimeout so direct XHRs honour
the configured timeout.
Migrate upload.js off jQuery: $.Deferred/$.when to Promise/Promise.all,
$.trim to trim, :selected to selectedIndex, multi-arg :not() to CSS,
.val/.prop/.attr/.find/.append/.empty/.children/.closest/.parent to
native DOM, .animate({scrollTop}) to scrollTo, and $("<option>") to
common.el. Option metadata stored via .data() (objects/arrays) moves
to common.data; the fuzzy-picker change handler is bound once instead
of re-bound on every refresh.
Migrate config.js off jQuery: HTML-string/$("<tag>") element building
to common.el, $.each to forEach, the :input/:checked pseudos to CSS
selectors, .data() round-trips (object values) to common.data, the
delegated .map-link handler to common.delegate, and .attr()/.val()/
.on()/.html()/.remove() to native DOM and addEventListener.
Map cells and the modal title now escape server data via textContent
(consistent with the leaf-modules hardening).
Migrate graph.js, selectors.js, history.js, and symbols.js off
jQuery: $.extend/$.each/$.trim and $().val/prop/attr/hasClass/is/
addClass/removeClass/change/click/on to native DOM and the shared
common.el / common.delegate helpers added in the common.js stage.
Delegated handlers on Tabulator-rendered DOM (.scorebar) move to
common.delegate; the repeated button disable/enable patterns
collapse into small per-module helpers.
jQuery remains loaded for the remaining modules (config, libft,
stats, upload, rspamd) — removed in later stages.
projects / thirdparty / rspamd.git / log
