[Fix] Split MIME address lists on semicolons as well as commas
A semicolon is not an RFC 5322 address-list separator (it only
terminates a group construct), but Outlook-style ;-separated recipient
lists are pervasive in real mail and most MTAs (e.g. Postfix) split
them. Previously such a list was parsed as a single malformed address
whose local part swallowed the semicolon and the next address, breaking
recipient counts and producing false invalid-address signals.
Treat ';' exactly like ',' in the list parser. This mirrors the
existing comma behaviour in every edge case: semicolons inside quoted
strings are preserved, an unquoted semicolon in a display name drops
the preceding phrase just like a comma does, an empty group
(undisclosed-recipients:;) now yields no addresses like the comma
variant, and the trailing ';' of a group construct no longer leaks
into the last member's domain.
Fix synchronous context_augment firing two GPT requests for one model
When your context_augment function calls the callback synchronously
(e.g., return function(task, content, cb) cb('some context') end),
the pending_fetches counter drops to 0 inside the callback,
triggering maybe_proceed() which calls proceed().
Then control returns to the main flow where the now-redundant guard
also sees pending_fetches == 0 and calls proceed(nil) a second time
— this time without the context snippet.
The else branch (line 1324) already handles the case where none
of the context features are enabled, so the removed fallback
was purely a duplicate path for synchronous callbacks.
[Test] Use unique envelope senders in aliases From tests
The three From-identity tests shared one envelope From/Rcpt pair, so the
greylist module (meta key = envelope from + rcpts + ip) soft-rejected
the third scan of the pair within its retry window, skipping the filters
stage and losing the GET_FROM symbol. This only manifested with tight
test pacing (pabot shard in CI); give each test a unique envelope
sender, as the rest of the suite does.
[Minor] Selectors: document the 'orig' address flavour
The from/rcpts extractors pass their arguments through to task:get_from
and task:get_recipients, so from('mime', 'orig') already selects the
address as it was seen in the message; document it and add a functional
test exercising the flavour through the selector pipeline.
[Fix] Forged recipients: compare addresses as they were transmitted
The rule compared the rewritten envelope view against the wire MIME
headers, so a recipient alias mapping to another domain (legal for
recipients) could make a perfectly matching To header look forged.
Compare the envelope and the headers at the same altitude: both as they
were seen in the message, which is also what this rule checked before
alias rewriting was applied to tasks.
[Test] Add functional tests for From identity preservation
- aliases: googlemail.com From keeps its domain while the user part is
canonicalized; a cross-domain virtual alias on the From is discarded;
a same-domain alias on the From local part still applies
- get_from: the 'orig' flavour returns the wire From (and only it) after
a task:set_from rewrite, and is a no-op without one
- dmarc: alignment and policy lookup use the wire From domain even when
a prefilter rewrote the MIME From
The REWRITE_MIME_FROM test prefilter is gated on a request header so it
stays inert in merged-suite tests that do not opt in.
SPF evaluates the RFC5321.MailFrom as it was transmitted (including
local-part macros), so resolve the SPF record and credentials from the
original envelope sender even if the envelope from has been rewritten,
e.g. by the aliases module.
[Fix] DMARC: evaluate the From and envelope domains as seen in the message
DMARC must align identifiers and look up the policy for the RFC5322.From
domain as it appears in the message, and check SPF alignment against the
RFC5321.MailFrom as transmitted. Use the 'orig' address flavour so that
rewrites done via task:set_from (e.g. by the aliases module) do not
affect alignment, policy lookup or rua report attribution.
[Rework] Lua API: make the 'orig' address flavour return wire addresses
Previously task:get_from({'mime', 'orig'}) returned both the preserved
original addresses and the rewritten ones appended by task:set_from,
making the flavour unusable for consumers that need the message as it
arrived (a duplicate-From check would misfire on the combined list, and
Lua could not tell the entries apart). Now, when the 'orig' flavour is
requested and originals were preserved by a rewrite, only the original
addresses are returned; without a rewrite the behaviour is unchanged.
Also expose the 'aliased' and 'original' email address flags to Lua and
document the flavour.
The From address is the sender's identity: SPF, DKIM and DMARC all key
on its domain. Service rules and alias resolution (e.g. a virtual alias
mapping to another domain) may only touch the local part of the From
address; any rewrite that would change the domain is now discarded.
Recipient rewriting is unaffected: recipients are interpreted by the
receiving host, and no authentication protocol keys on them.
Also fix a spurious task:set_from call: the set_from decision used the
callback-wide alias_resolved flag, so a rewrite of any earlier address
(e.g. an envelope recipient) caused the unmodified MIME From to be
marked as aliased and duplicated.
[Fix] Aliases: never rewrite googlemail.com domain to gmail.com
googlemail.com and gmail.com are DNS-distinct domains with independent
SPF and DMARC records (gmail.com publishes p=none while googlemail.com
publishes p=quarantine), so substituting one for the other breaks DKIM
alignment, changes the effective DMARC policy and misattributes DMARC
reports. Keep the gmail.com user-part canonicalization (dots and plus
tags) for googlemail.com addresses, but preserve the domain.
The domain equivalence remains available in lua_util.remove_email_aliases
for dedup-only consumers such as email hash lookups in RBL checks.
[Fix] Lua 5.5 compatibility: do not assign to for-loop variables
Lua 5.5 makes for-loop control variables read-only, so assigning to
them is now a compile-time error. Rebind the value to a local (or pass
it as a closure parameter) in all remaining places; found by
parse-checking the whole tree with luac 5.5.
Register waitForResponse listeners before clicking the reset button:
on localhost the historyreset response arrives before a listener
registered after the click can see it, which made the legacy
(Playwright 1.45.3) CI leg fail deterministically.
Lua 5.5 makes `for` expression variables constant. The recent upgrade to
lua 5.5 broke lua-lupa, which previously attempted to modify the loop
variable. This caused an unrecoverable error on startup:
rspamd_lua_require_function: require of lua_util.jinja_template failed: error loading module 'lupa' from file '/usr/share/rspamd/lualib/lupa.lua':\x0A\x09/usr/share/rspamd/lualib/lupa.lua:1858: attempt to assign to const variable 'word'
[Minor] WebUI: drop unreachable 304 success branch
The Rspamd controller sends "Cache-Control: no-store" on every WebUI
response (controller.c, issue #3330) and never emits Last-Modified/ETag
for these endpoints, so a browser never revalidates and a 304 response
is impossible. The `|| xhr.status === 304` in queryServer was dead code
copied from jQuery's default success range.
Remove it so queryServer, the /stat connect probe and the legacy /auth
probe all treat 200-299 as success. Treating 304 as an error is also
safer: a 304 body is empty, and success would have fed it to the JSON
parser.
[Fix] WebUI: route a malformed /stat response to the login dialog
The connect probe swallowed JSON parse errors (parseJsonSafe returned {}),
so a 2xx /stat with a non-JSON body (truncated by a proxy, a captive-portal
page, corruption) loaded an empty, half-loaded main UI with a "Cannot get
server status" alert and no login path. The former jQuery content-type
parse raised parsererror and fell through to the login form.
Parse the body explicitly and, on any failure, show the connect dialog —
matching the pre-migration behaviour and covering the broad case (any
non-JSON 2xx, not only application/json). Removes the now-unused
parseJsonSafe helper.
[Fix] WebUI: restore smooth scroll on the Scan tab
The jQuery-removal replaced .animate({scrollTop}) with
window.scrollTo({behavior:"smooth"}), which honours prefers-reduced-motion
and is interruptible by Tabulator's reflows. The former jQuery tween
ignored the preference and re-set scrollTop every frame, so it was always
smooth.
Add a requestAnimationFrame-based smoothScrollTo (400ms easeInOut, matching
jQuery's default) and use it for both Scan scrolls (scan result and the
"Clean" reset). Like the rest of the UI's animations it does not honour
prefers-reduced-motion, preserving the previous behaviour.
fireAjaxComplete (the active-request counter decrement) ran at the top of
finish(), before the success callback. During an "All SERVERS" query the
count therefore dipped to 0 between the neighbours probe and the
per-neighbour requests: ajaxStart re-fired and re-added fa-spin,
restarting the spinner's CSS animation. jQuery decremented after success,
keeping the count above zero for the whole fan-out.
Move the decrement to the end of finish(), after the success/complete
callbacks. This also removes the rapid fa-spin class churn that was
interrupting the Scan tab's smooth scroll.
- stats.js: guard the optional fuzzy_hashes key in addFuzzyStorage
(Object.entries/keys of undefined threw on every Status refresh when
fuzzy_check was disabled or a neighbour was down), matching the guard
addStatfiles already had.
- common.js: cancel the hide/show slide animations once the final state
is committed, so the fill:"forwards" no longer pins height:0 — a later
re-show was measuring the pinned height and animating 0→0.
- common.js: treat a single Element (incl. <select>/<form>, which expose
a numeric .length) as one node in toElements, not a collection.
- common.js: tolerate a null neighbours body before Object.keys.
- rspamd.js: null-guard the active-tab lookup in two spots, where a
missing active tab used to throw.
- upload.js: drop the dead params:{processData:false} (queryServer no
longer form-encodes).
- common.js: document the statusCode handler signature.
[Test] selectors: cases for partially-parsed selectors
Negative cases: a second ':' method, a stray closing paren and junk
after a ';' list element must be rejected. Control cases: valid
method/transform chains, selector lists and trailing whitespace must
keep parsing.
[Fix] lua_selectors: reject selectors that parse only as a prefix
The selector grammar was not anchored to the end of input, so lpeg
matched the longest valid prefix and silently dropped the rest of the
string. E.g. 'from("smtp"):domain:lower' (a second ':' cannot parse)
was accepted and evaluated as 'from("smtp"):domain', and any trailing
garbage after a selector or a ';' list element was ignored. All
consumers (multimap, rbl, ratelimit, settings, reputation, the
controller selector check used by the WebUI) reported such selectors
as valid while evaluating only the prefix.
Append an lpeg.Cp() capture after the grammar (plus optional trailing
whitespace) and make parse_selector require the whole input to be
consumed, logging the position and the unparsed tail otherwise.
User-visible change: selectors that previously loaded thanks to the
silent truncation now fail configuration load with an error pointing
at the offending token. Such selectors were never evaluating as
written, so failing loudly is the correct behaviour.
[Fix] rspamd_symcache.h: keep C linkage of lua.h for C++ includers
The raw lua.h include sat outside extern "C", so any C++ unit that
pulled rspamd_symcache.h before lua/lua_common.h got C++-mangled
declarations of the Lua C API and failed to link. Including
lua_common.h itself is not an option due to the include cycle via
rspamd.h -> cfg_file.h -> rspamd_symcache.h, so wrap lua.h the same
way lua_common.h does. This also lets composites sources include
lua_common.h in the natural position instead of first.
[Test] composites: functional cases for Lua conditions
Cover: option-inspecting condition, cross-symbol join on a shared
option value, numeric return as atom weight combined with an
expression limit, false condition, and depends_on deferring a
composite whose condition consults a postfilter symbol.
A condition is called as f(task, symbol) where symbol is a table in the
task:get_symbol() layout; it may return true/false or a number used as
the atom weight. The condition is ANDed with option filters of the atom
and a failed condition is treated exactly as a missing symbol, so
removal policies are not applied. Since a condition can consult symbols
invisible to the expression, the optional depends_on list feeds the
first/second pass placement as if those symbols were expression atoms.
rspamd_config:add_composite() now also accepts a full definition table,
and rspamd_lua_push_symbol_result() is exported for reuse.
[Fix] phishing: don't penalise same label under another TLD
Three fixes for cross-TLD brand false positives
(e.g. brand.co.uk displayed over a brand.com href):
- Normalise the DMARC_POLICY_ALLOW domain to eSLD before comparing it
with the link target tld: DMARC reports the raw From domain (possibly
a subdomain), so the existing exclusion for authenticated mail never
matched
- Compare registrable labels instead of stripping the last dot
component: the old code only handled single-label suffixes, so
brand.co.uk vs brand.com was levenshtein-compared as full strings and
scored full weight; now the identical label yields weight 0
- Check strict_domains regardless of the computed weight (gated on
tld ~= ptld, equivalent to the old reachable behaviour), so displaying
a strictly protected domain over a same-label different-suffix target
still fires at full weight
[Test] lua_cryptobox: unit tests for secretbox nonce padding
Guard against the regression fixed in #6122: a short nonce must produce
the same ciphertext as the explicitly zero-padded 24-byte nonce, and
short/padded forms must decrypt each other. Both assertions fail on the
unpatched code as they read past the nonce buffer.
[Fix] lua_cryptobox: pass padded nonce to secretbox encrypt/decrypt
Both secretbox encrypt and decrypt built a zero-padded 24-byte real_nonce
but passed the original short nonce pointer to libsodium, reading up to
23 bytes out of bounds and producing ciphertexts that depend on adjacent
memory contents.
[Minor] maps: move glob pattern anchoring into rspamd_str_regexp_escape
Add RSPAMD_REGEXP_ESCAPE_ANCHOR flag that wraps the escaped pattern
into ^(?:...)$ within the same allocation instead of a second
g_strdup_printf in the glob map insertion path.
[Fix] maps: anchor glob map patterns to match the whole subject
Glob map entries were compiled into unanchored regexps and matched
with substring search semantics, so a `t.co` entry matched
`walmart.com` and `*.bit.ly` matched `foo.bit.ly.evil.com`.
Wrap the translated pattern into `^(?:...)$` at map load time so
glob entries match the subject as a whole: bare names match exactly,
wildcards match only what they say.
This affects all glob maps: multimap glob/glob_multi, url_redirector
redirector_hosts_map, dkim_signing/arc signing_table and key_table,
mx_check exclusions and the rbl glob returncodes matcher. Maps that
relied on the accidental substring behaviour must now use explicit
wildcards.
Register the historyreset/history response listeners before the click
via Promise.all. waitForResponse only catches responses arriving after
it is set up, and the fast local historyreset round-trip could beat a
listener attached post-click, causing a flaky 10s timeout under CI load.
Remove jQuery from the RequireJS config (paths, app/bootstrap shims) and
delete the vendored jquery-3.7.1.min.js. All app modules now load without
jQuery; the bootstrap shim's stale "Popovers require jQuery" comment and
deps entry are removed.
Drive-by: rename the modalDialog data-backdrop attribute to its Bootstrap
5 name data-bs-backdrop, so the static-backdrop behaviour is actually
applied (the BS4 attribute has been silently inert).
Migrate libft.js off jQuery. $("<tag>", {hash}) element construction
becomes common.el, $.extend becomes spread, $.each becomes forEach.
The column-options dropdown's namespaced hidden.bs.dropdown.rspamd
rebind (jQuery namespaces) becomes a common.data-tracked
addEventListener that removes the previous handler before re-binding;
the delegated fuzzy-hash buttons (re-bound every render) use a cleanup
map so common.delegate doesn't stack. $(arr).not(arr) set-diffs become
Array.some/ includes, and .siblings()/.attr()/.html()/.val() move to
native DOM.
Migrate rspamd.js off jQuery. The /stat connect probe moves to a raw
XHR (avoiding common.query's error logging on the normal "not logged
in" path); $.ajaxSetup and $(document).ajaxStart/Complete are dropped
since no $.ajax calls remain. tabClick disables/enables the nav bar
controls via attribute+class helpers (li/a need both for Bootstrap);
:radio + .val([v]) and .data("value") become checkRadio / activeMenuValue
helpers; delegated handlers move to common.delegate; BS5 events move to
addEventListener (with {once:true} replacing self-removing .on/.off);
the popover content uses cloneNode and the outside-click dismiss uses
.contains(). Form/input re-binds become onsubmit/oninput properties.
Migrate stats.js off jQuery. The delegated mouseenter/mouseleave hover
highlighting (which doesn't bubble natively) is reimplemented on
mouseover/mouseout with a relatedTarget-containment check; the jQuery
sibling-traversal chains (prevAll/nextAll/slice/find) move to native
sibling walks. $.each becomes Object.entries/forEach, $(html).appendTo
becomes insertAdjacentHTML, and $.wrapAll becomes common.el DOM
construction. The legacy /auth fan-out keeps a raw XHR to preserve its
custom error message and alerted-session dedup; $.when().always becomes
Promise.all().finally. Adds common.getAjaxTimeout so direct XHRs honour
the configured timeout.
Migrate upload.js off jQuery: $.Deferred/$.when to Promise/Promise.all,
$.trim to trim, :selected to selectedIndex, multi-arg :not() to CSS,
.val/.prop/.attr/.find/.append/.empty/.children/.closest/.parent to
native DOM, .animate({scrollTop}) to scrollTo, and $("<option>") to
common.el. Option metadata stored via .data() (objects/arrays) moves
to common.data; the fuzzy-picker change handler is bound once instead
of re-bound on every refresh.
Migrate config.js off jQuery: HTML-string/$("<tag>") element building
to common.el, $.each to forEach, the :input/:checked pseudos to CSS
selectors, .data() round-trips (object values) to common.data, the
delegated .map-link handler to common.delegate, and .attr()/.val()/
.on()/.html()/.remove() to native DOM and addEventListener.
Map cells and the modal title now escape server data via textContent
(consistent with the leaf-modules hardening).
Migrate graph.js, selectors.js, history.js, and symbols.js off
jQuery: $.extend/$.each/$.trim and $().val/prop/attr/hasClass/is/
addClass/removeClass/change/click/on to native DOM and the shared
common.el / common.delegate helpers added in the common.js stage.
Delegated handlers on Tabulator-rendered DOM (.scorebar) move to
common.delegate; the repeated button disable/enable patterns
collapse into small per-module helpers.
jQuery remains loaded for the remaining modules (config, libft,
stats, upload, rspamd) — removed in later stages.
Rewrite the AJAX layer (query/queryServer) on XMLHttpRequest,
preserving jQuery's semantics: neighbours fan-out and aggregation,
per-server error dispatch, statusCode, timeout, GET data appended
to the URL as a query string, default Content-Type for POST bodies,
NProgress download progress, and jqXHR-compatible callback arguments.
Add a spinner pub/sub (onAjaxStart/onAjaxComplete with an in-flight
counter honoring global:false) and setAjaxTimeout, plus shared
vanilla helpers: el, delegate, data. Migrate alertMessage, hide,
show, toggle (Web Animations API), the error-log UI and fileUtils
to plain DOM.
Bridge rspamd.js to the new APIs (setAjaxTimeout, onAjaxStart/
onAjaxComplete) alongside the jQuery hooks, which stay for the
remaining direct $.ajax calls until a later stage. jQuery is still
loaded; this is stage 1 of the staged removal.
Replace the Bootstrap 5 jQuery bridge (.modal/.popover/.trigger
"click.bs.dropdown") with the vanilla constructor API
(bootstrap.{Modal,Popover,Dropdown}.getOrCreateInstance) in
rspamd.js, config.js and libft.js, capturing the bootstrap module
where needed.
Prerequisite for removing jQuery: the bridge only activates when
jQuery is loaded, while the vanilla API works regardless. With jQuery
still present the existing $(el).on("*.bs.*") subscriptions keep
firing, so those stay jQuery for now and move to addEventListener
together with the rest of the jQuery removal. The main.js shim
deps:["jquery"] becomes redundant and is dropped in the final stage.
Replace the jquery.stickytabs plugin with an inline vanilla
implementation using the BS5 Tab API, removing the last jQuery
bridge dependency of that plugin. Behavior (URL hash persistence
and initial-tab fallback) is preserved.
D3Evolution 2.0.4 -> 2.1.0, D3Pie 1.1.2 -> 1.2.0. Both libraries
dropped the jQuery dependency (replaced $.extend with an internal
deep-merge), so drop the stale "jquery" shim deps in main.js.
Removing footable.standalone.min.css dropped its .form-group
{margin-bottom:15px} rule, collapsing the gap above the buttons on
the selectors tab (and the action-score rows on the configuration
tab). .form-group is a Bootstrap 3 class with no meaning in BS5;
replace it with the mb-3 spacing utility that provides the same gap.
[Fix] lua_selectors: don't crash on a missing method call
Selectors like `time:digest` (method syntax used where a transform
was meant) aborted the whole selector at scan time with an uncaught
Lua error:
Cannot run callback: .../lua_selectors/init.lua: attempt to call
a nil value
The `:` separator always compiles to a method call on the extracted
value; when the value has no such method, the lookup yields nil and
the unguarded call crashed the callback. Guard the lookup (via pcall,
since indexing some types raises) and yield no value instead, logging
an error that hints at the `.name` transform syntax when the name
matches a known transform. Valid method calls (e.g. `:gsub`,
`:lower`) and table field access are unaffected.
installScrollPreservation bound listeners on the mount element and
window without removing the previous set. Each column-options rebuild
(destroy + re-init on the persistent mount element) stacked a new set
that was never released — a listener/memory leak on the history and
scan tables. Mirror bindRowClickToggle's WeakMap guard: key the set by
mount element and tear it down before re-binding.
FooTable was superseded by Tabulator across all tables (Phases 0-3).
Remove the vendored library/CSS, the footable-fontawesome icon module,
and every remaining reference:
- index.html: drop the two <link> tags
- main.js: drop the requirejs path and shim
- rspamd.js: drop the app/footable-fontawesome require; stickyTabs
runs directly (declared module dependency)
- common.js: delete dead appendButtonsToFtFilterDropdown (no callers)
- rspamd.css: drop all .footable rules and the dead .footable-details /
.footable-filtering-search selectors
Rename column-options dropdown hooks ft-columns-* -> tab-columns-*
(Tabulator; consistent with tab-utils.js). Legacy history path kept.
[Test] static_embed: pack fixture floats via rspamd_util.pack
math.frexp was removed in Lua 5.4, so the hand-rolled float32 packer
broke the test run on non-LuaJIT builds; rspamd_util.pack provides
string.pack semantics on every supported Lua version.
Add model:get_token_vectors(input[, opts]) for offline consumers
(external trainers exporting order-aware text features): the token
embedding sequence in token order instead of only the pooled mean.
- Accepts exactly what get_sentence_vector accepts (word list or whole
text) and tokenizes through the same shared code path; unk rows are
included the same way the pooled path includes them.
- opts.max_tokens truncates after tokenization to the first N tokens
(the returned count is post-truncation); opts.raw returns an
rspamd_text with ntokens*dim little-endian float32s packed row-major
instead of a table of tables. Invalid opts raise errors, no silent
coercions. Empty input yields an empty table/text and 0, never nil.
- The provider path is unchanged: fusion vectors stay fixed-dim, so the
neural provider keeps using only the pooled get_sentence_vector.
- Tests: id/row alignment, pooled-mean consistency (incl. unk-heavy
input), word-list/text equivalence, max_tokens, raw packing and
strict opts validation.
Add a static token-embedding provider (Model2Vec style), the cheap
multilingual successor to fasttext_embed: words from rspamd's regular
tokenization pipeline are re-tokenized into WordPiece subword tokens and
embedded by mean-pooling rows of a precomputed float32 matrix, with no
neural forward pass and no new dependencies.
- rspamd_static_embed: a Lua-C module combining a WordPiece tokenizer
(BertNormalizer via ICU + Bert pre-tokenizer + greedy WordPiece) with
an mmap-ed embedding matrix shared between workers. The model spec is
read from the model directory (config.json + vocab.txt + matrix +
optional HF tokenizer.json) and validated strictly, fail-fast: any
unsupported normalizer/pre-tokenizer/model type, pooling other than
mean, non-float32 matrix or size mismatch fails the load instead of
degrading silently. get_sentence_vector() accepts a word list (the
provider path) or a whole text; both produce identical vectors.
- The WordPiece tokenizer is internal to the vectorizer: the global
word-breaking / statistics tokenization path is untouched, so Bayes
tokens and fuzzy hashes are unaffected.
- static_embed provider: extracts words like fasttext_embed and feeds
them to the model; the Lua side holds no matrix data and uses no FFI.
- unit tests with a generated fixture covering normalization, subword
splitting, greedy matching, CJK padding, unk handling, mean pooling,
word-list/text equivalence and strict rejection of unsupported
configs (BPE model type, vocab/matrix size mismatches, pooling).
Verified against the reference tokenizer oracle: token ids match
exactly, pooled vectors match within 2.4e-05 max abs diff, and the
word-list path is bit-identical to raw-text tokenization on the corpus.
Add a per-table boolean search box (history/scan) with a query
language ported from FooTable's built-in filtering: whitespace =
AND, OR, - (exclude), "exact phrase". The predicate ANDs with the
per-column header filters.
Also add headerFilter:"input" to the text columns (matching the
errors-table convention) and align all header-filter inputs on a
shared bottom baseline (title-holder flex-grow) regardless of how
many lines a wrapped title occupies.
Migrate the last two FooTable tables, #historyTable_history and
#historyTable_scan, to Tabulator. Both share rendering code in libft.js
(process_history_v2/columns_v2, initHistoryTable, the column-options
dropdown and the symbol-order toggle).
- Action column uses a custom headerFilter (select + "not" checkbox)
with headerFilterFunc for exact matching.
- Column-options dropdown keeps Visible/Hidden plus "Row", emulated
via responsive:100 and persisted to localStorage.
- Symbol-order "Sort by:" buttons are embedded in the symbols column
title, so they are recreated on every render; the active state is
reapplied on tableBuilt/renderComplete.
- columnDefaults uses the "html" formatter so upstream-escaped values
decode instead of being re-escaped by "plaintext".
- Shared helpers (scroll preservation, etc.) live in tab-utils.js.
Update the scan/symbols Playwright specs for the Tabulator DOM and
replace the racy "disabled during reload" check with response waits.
- Extract reusable Tabulator helpers (scroll-prevention, tabindex
removal, footer-hide, row-click-toggle, scrollIntoView guard) from
history.js into a new tab-utils.js module — no footable dependency,
so graph.js no longer loads footable.min.js for the throughput tab
- Refactor history.js initErrorsTable to use tab-utils helpers
- Migrate the rrd summary table (graph.js) from FooTable to Tabulator:
row coloring via CSS variable, in-place updates via updateData,
dynamic column titles via updateColumnDefinition, header sorting
- Update rspamd.css: rrd-table selectors for Tabulator DOM, row color
via --rrd-row-color (visible in dark mode), row hover, layout tweaks
Replace FooTable with Tabulator (v6.4.0) for the errors log table — the
first step of the FooTable→Tabulator migration. Tabulator is vanilla-JS and
jQuery-free, so it survives the planned jQuery removal; FooTable is
unmaintained (GPLv3, last release 2017).
- Vendored tabulator.min.js + tabulator_bs5.min.css; wired via RequireJS
- Rewrote initErrorsTable on the Tabulator API: columns, local pagination,
responsive collapse, per-column header filters
- rspamd.css: map the bs5 theme to Bootstrap CSS variables so light/dark
flip automatically with data-bs-theme
- Row click toggles responsive collapse (only when collapse is active);
pagination footer hidden on a single page (FooTable parity)
- Scroll prevention: Tabulator scrolls the page internally on interactions;
the call is elusive, so window.scrollY is preserved/restored around
clicks/Update. Row hover backgrounds disabled to avoid the resulting
expand-scroll flicker (documented inline for future maintainers).
Stefan Benten [Tue, 30 Jun 2026 20:02:26 +0000 (22:02 +0200)]
[Feature]: src/rspamadm: allow reading password from env (#6114)
* src/rspamadm: allow reading password from stdin
Currently the cli only allows reading the password from tty.
This makes it challenging to check the password via automation,
ie. used by continous deployment to ensure a clean state.
This change adds the functionality to read from stdin, so that
passwords can be piped in.
* test/functional/cases: test for stdin password checks
Vsevolod Stakhov [Tue, 23 Jun 2026 07:58:48 +0000 (08:58 +0100)]
[Feature] rspamadm: discover command modules from external directories
rspamadm discovered Lua sub-commands by globbing only the built-in
LUALIBDIR/rspamadm directory, so a third-party or premium package could
not ship a rspamadm command without writing into the OSS-owned lualib
tree.
Scan two additional sources, in order, after the built-in directory:
- $CONFDIR/rspamadm.d/*.lua, a drop-in dir consistent with local.d
- every directory in the colon-separated RSPAMADM_COMMAND_PATH env var
(mirrors how PATH works)
Duplicate command names are skipped (first wins), so built-in commands
can never be shadowed. Externally loaded modules run with the same
globals and lua_path as built-in ones, so they can require premium
lualibs and use lua_redis. No behavior change when the env var and the
.d directory are absent.
Factor the per-file loader and per-directory scan into helpers, and fix
a latent out-of-bounds in the basename fallback (it searched .lua in the
full path but indexed the basename buffer). Document discovery order and
the new env var in the man page.
Vsevolod Stakhov [Tue, 23 Jun 2026 07:57:22 +0000 (08:57 +0100)]
[Fix] lua_kann: correct load arg index and save return value
rspamd_kann.load is a module function, not a method, so the options
table is at arg 1. The lua_istable(L, 1) branch incorrectly read the
filename field from arg 2 (a copy-paste from lua_kann_save where arg 1
is self), so the documented load({filename = ...}) form always failed
with 'missing filename'.
Also fix lua_kann_save: the trailing lua_pop popped the boolean result
it had just pushed, so save({filename = ...}) returned the filename
string instead of true. Pop the filename field before pushing the
result.
Add a regression test covering the file round-trip via the single-table
form.
[Test] Run push-triggered CI on all branches in forks
rspamd/rspamd restricts push to master (71605c25) to avoid the duplicate
push+pull_request run on feature branches that leaves a spurious *cancelled*
status. This trade-off is acceptable for the main repo where contributors
open PRs, but inconvenient in forks where CI is needed on every push without
opening a PR.
Vsevolod Stakhov [Sat, 20 Jun 2026 19:20:19 +0000 (20:20 +0100)]
[Feature] neural: forced-learn fast path and first-class freeze
Two training controls plus a supporting task primitive:
* train.forced_learn_minimal_scan (default on when disable_symbols_input):
a high-priority neural prefilter disables every non-neural symbol on an
ANN-Train scan, so a symbols-independent training vector is built without
issuing RBL/DNS, fuzzy, bayes, ClickHouse or capture/cluster work. The
stored vector and the profile key are byte-for-byte identical to the live
full-scan path (asserted in tests). For symbol-dependent rules it stays off,
and if any applicable neural rule needs symbols the whole task falls back to
a full scan.
* train.frozen: stops automatic training and auto-storing of live vectors so a
frozen model's pools cannot drift into an imbalanced live set, while
inference keeps serving the current ANN unchanged. An explicit ANN-Train
still stores and retrains on demand (gated by a per-profile retrain marker).
Supersedes the auto-learn side of store_set_only/store_pool_only; both keep
working when frozen is unset.
* task:disable_all_symbols([skip_mask]): Lua binding over the existing
rspamd_symcache_disable_all_symbols "process only these" primitive (defaults
to keeping explicit_disable symbols), used by the prefilter.
Functional coverage in test/functional/cases/330_neural/006_forced_learn_minimal
and 007_frozen.
Vsevolod Stakhov [Fri, 19 Jun 2026 11:31:37 +0000 (12:31 +0100)]
[Minor] lua_text stats: use 1-based byte indexing
The byte-distribution statistics methods (entropy, byte_mean,
byte_deviation, serial_correlation, monte_carlo_pi) took a 0-based
offset, which was inconsistent with the rest of the rspamd_text API
(span/sub/at) and with text:crc32. Switch the optional range argument
to a 1-based start index so all rspamd_text slicing uses the same
convention; len semantics and the lenient out-of-range-yields-0
behaviour are unchanged. Docs and tests updated accordingly.
Vsevolod Stakhov [Fri, 19 Jun 2026 11:28:15 +0000 (12:28 +0100)]
[Feature] lua: zlib/YARA-compatible crc32
Expose standard CRC-32 (poly 0xEDB88320, init/final 0xFFFFFFFF XOR),
computed via zlib crc32() so it is bit-exact with YARA hash.crc32:
- rspamd_cryptobox_hash.create_specific("crc32"): streaming
update + final, big-endian out so :hex() yields e.g. cbf43926
- rspamd_text:crc32([start[, len]]) and
rspamd_util.crc32(input[, start, len]): return the checksum as a
Lua integer over a zero-copy 1-based slice (no buffer copy)
Tests cover golden values ("" -> 0, "123456789" -> 0xCBF43926),
1-based slicing, streaming-vs-one-shot, reset, and a real
attachment-like buffer cross-checked against zlib/YARA crc32.
Vsevolod Stakhov [Fri, 19 Jun 2026 09:37:44 +0000 (10:37 +0100)]
[Feature] lua_archive: extraction limits vs zip bombs
archive.unpack/unzip/untar read whole members into memory via
libarchive, so a tiny hostile archive could expand to gigabytes
(OOM). Add an optional opts table that caps extraction, enforced
while reading so memory stays bounded:
- max_output total uncompressed bytes across all members
- max_file_size per-member uncompressed cap (truncated at cap)
- max_files member count cap
- max_ratio per-member uncompressed/compressed ratio cap
Ratio uses real compressed bytes consumed via
archive_filter_bytes(a, -1), checked per chunk above a 64 KiB
floor, so a bomb is stopped early even with no size cap set. All
limits are opt-in (0/absent = unlimited), preserving the previous
behaviour. A second boolean return flags truncation so a capped
extraction is never mistaken for a complete one.
Vsevolod Stakhov [Thu, 18 Jun 2026 18:16:54 +0000 (19:16 +0100)]
[Minor] CI: trigger push only on master to dedup PR runs
A commit on a branch with an open PR fired both a push and a
pull_request run; concurrency cancel-in-progress reaped one, leaving a
spurious *cancelled* run that reads like a CI failure. Restrict push to
master so feature branches run a single pull_request workflow; key the
concurrency group on PR number / ref to still cancel superseded runs.
Add byte-distribution statistics as methods on the rspamd_text class,
implemented in C++20 under src/lua (lua_text_stats.{hxx,cxx}); lua_text.c
is left untouched and the rspamd{text} metatable is augmented at load.
Methods (each takes an optional 0-based (off, len) range, defaulting to
the whole buffer):
- text:entropy([off[, len]]) Shannon entropy, bits/byte
- text:byte_mean([off[, len]]) mean of unsigned byte values
- text:byte_deviation(mean[, off[, len]]) mean abs deviation from mean
- text:serial_correlation([off[, len]]) ENT serial correlation
- text:monte_carlo_pi([off[, len]]) ENT Monte-Carlo Pi deviation
The core is header-only, allocation-free and O(n) (a single histogram
pass shared by entropy/mean/deviation) and produces deterministic,
bit-reproducible results. Offsets are byte offsets, 0-based; the range is
clamped to the buffer and an out-of-range or empty range yields 0.
Add C++ doctest golden-vector tests (analytically-derived exact values)
and Lua unit tests covering empty/single-byte/uniform/two-symbol buffers,
overlapping groups, slicing and edge cases.
Vsevolod Stakhov [Thu, 18 Jun 2026 13:22:25 +0000 (14:22 +0100)]
[Feature] multipattern: explicit SOM flag and offset docs
SOM (start-of-match) reporting already exists on master as the default
(hyperscan compiles every pattern with HS_FLAG_SOM_LEFTMOST), but there
was no explicit way to request it and the offset convention was
undocumented.
- Add RSPAMD_MULTIPATTERN_SOM (rspamd_trie.flags.som): an explicit
opt-in for start offsets that also overrides no_start/single_match
(forces SOM and drops the incompatible SINGLEMATCH).
- Document the offset convention: pattern id is 1-based; match start
and end are byte offsets, 0-based, start inclusive and end exclusive
(one past the last matched byte), so end - start is the match length.
- Fix the regex (flags.re) fallback used when hyperscan is unavailable:
it discarded the real PCRE start and reported end - strlen(pattern),
which is bogus for variable-length matches. It now reports the true
start/end from rspamd_regexp_search.
Add C++ (rspamd_cxx_unit_multipattern.hxx) and Lua (trie.lua) unit
tests asserting (id, start, end) against hand-computed positions:
multiple/overlapping occurrences, icase, literal vs regex, no-match,
SOM-overrides-single_match and a large buffer. Existing rspamd_trie
behaviour and its callers (url.c, lang_detection, lua plugins) are
unchanged.
Vsevolod Stakhov [Wed, 17 Jun 2026 17:52:16 +0000 (18:52 +0100)]
[Fix] monitored: alphanumeric-only random DNS prefixes
random_monitored RBL checks built random labels from an alphabet that
included '-' and '_'. That produced names like '_Q8...0-' (leading
underscore, trailing hyphen) or '-7d0...' (leading hyphen) which are
not valid DNS labels (RFC 952/1123: no leading/trailing hyphen, no
underscore in hostnames). Authoritative DNSBL servers such as
spfbl.net reject these with SERVFAIL.
Restrict the alphabet to alphanumerics, which always forms a valid
label regardless of position while keeping ample entropy.
Vsevolod Stakhov [Tue, 16 Jun 2026 19:05:26 +0000 (20:05 +0100)]
[Fix] lua: refcount coroutine thread entries
The state/generation guards from the previous commit read the entry on
resume, so they only help while the entry is still allocated. If the
owning task is torn down while an async request is in flight, the entry
could be freed before the late completion fires, turning the guard into
a use-after-free.
Make thread_entry refcounted (ref.h, non-atomic - workers are single
threaded). The pool holds the initial reference; terminate_thread() and
the pool-full path now drop it via REF_RELEASE instead of freeing
directly, so the struct is destroyed only once the last reference goes
away. Every async library that stashes an entry for a later completion
now takes its own reference and drops it when done:
- dns/util: retain at the request, release in the one-shot callback.
- http/redis: retain at yield, release in the cbdata/ctx destructor
(and, for redis, at each point that consumes ctx->thread).
- tcp: retain at each yield, release at the matching resume, since the
tcp cbdata has several direct-free error paths that bypass its
destructor; pairing with yield/resume keeps the balance exact and
leaves bad-argument paths (which return before yielding) untouched.
Combined with the generation guard, a completion that races task
teardown now finds the entry alive but DEAD/recycled and refuses the
resume, instead of dereferencing freed memory.
Vsevolod Stakhov [Tue, 16 Jun 2026 15:35:34 +0000 (16:35 +0100)]
[Fix] lua: state management for coroutine thread pool
Async libraries (dns/redis/tcp/http/util) capture the "currently
running" coroutine at a yield point and resume it later from a C
completion callback. Nothing previously guaranteed the entry resumed
was still the one captured: a double-fired event, a completion racing
task teardown, or an entry recycled into another task would resume the
wrong (or freed) coroutine and corrupt memory. These failures are
interleaving-dependent and invisible in isolation.
Give each pooled thread an explicit lifecycle (FREE/RUNNING/YIELDED/
DEAD) plus a generation counter, both carried in the existing
thread_entry and per-request cbdata structs - no new allocations on any
hot path:
- get/return/terminate/yield/resume enforce legal state transitions,
so returning a suspended thread or resuming a non-suspended one now
aborts at the exact violation instead of corrupting a core later.
- lua_thread_resume_checked() refuses to resume unless the thread is
still YIELDED and its generation matches the value snapshotted at
the yield point; a stale/duplicate completion becomes a logged
no-op rather than a wrong-coroutine resume. All five async libs are
migrated to it.
- lua_tcp keeps cbd->thread pointing at the coroutine actually
yielded by sync read/write, so the resume always targets it.
generation is bumped on every acquire and release, so an entry that
goes back to the pool and is handed out again no longer matches a
completion that was already in flight.
Vsevolod Stakhov [Tue, 16 Jun 2026 07:54:45 +0000 (08:54 +0100)]
[Fix] dns: do not defer resolver nameservers (fixes #6096)
A nameserver that failed to resolve at config time was turned into a
zero-address PENDING_RESOLVE upstream by 904fd6218, then dereferenced as
NULL in rspamd_dns_server_init -> SIGSEGV at worker startup (regression
in 4.1.0). DNS resolver nameservers are consumed synchronously by
rdns_resolver_add_server and can never be promoted async, so never defer
them; also NULL-guard rspamd_dns_server_init as defense in depth.
Vsevolod Stakhov [Mon, 15 Jun 2026 08:59:47 +0000 (09:59 +0100)]
[Fix] mx_check: loopback-only MX is LOCAL, not bogon
A domain whose MX resolves only to loopback is hosted on the scanning
host itself -- a self-MX, typically the host's own FQDN mapped to
127.0.0.1 in /etc/hosts, which rspamd's resolver honours as a fake reply
that shadows public DNS. That made fully DMARC-aligned self-hosted mail
score MX_BOGON_ONLY (+8.0): the strongest "not spam infrastructure"
signal treated as the strongest spam signal.
Move 127.0.0.0/8 and ::1/128 from BOGON_CIDRS to LOCAL_CIDRS so a
loopback-only MX emits MX_LOCAL_ONLY (3.0) instead. test_mode now lifts
loopback out of the LOCAL set (was: bogon) so the probe path stays
exercisable against local listeners.
Add a regression test (170_mx_check_selfmx.robot, test_mode = false):
the existing suites run test_mode = true and cannot cover the production
loopback-classification path.
Vsevolod Stakhov [Sat, 13 Jun 2026 12:18:18 +0000 (13:18 +0100)]
[Feature] neural: sequence output mode and SIF word selection in fasttext_embed
Add a generic word-vector sequence output to the fasttext_embed provider
so that custom ANN architectures (e.g. attention pooling) can learn their
own pooling instead of receiving a pre-pooled vector:
* output_mode = "sequence": emits the first max_words word vectors
flattened word-major and zero-padded to max_words * channels.
* word_selection = "sif": since order-invariant poolers do not need a
prefix, optionally fill the sequence with the max_words most
distinctive (highest SIF weight) words from anywhere in the message
instead of the leading ones. Default stays "prefix".
This is the data half only; how the sequence is consumed is left to the
ANN architecture. Both modes are bounded by max_words.
projects / thirdparty / rspamd.git / log
