Newline normalization recorded every newline as a pointer array entry
plus a mempool exception object and a GList node, then sorted the
whole exceptions list: a newline-dense body turned a few MiB of input
into hundreds of MiB of heap and millions of allocations. Cap the
recorded newline positions at 100k per part (text normalization and
line counters are unaffected), set a part flag on truncation and
expose it as newlines_truncated in textpart:get_stats().
[Fix] Bound alt-part linking and fasttext langdet cost
Alternative text part linking used one subtree scan per text part, so
a crafted multipart/alternative with thousands of sibling text parts
made linking quadratic (~100M sibling visits at the MIME part cap).
All searches of a task now share a single visit budget; when it is
exhausted the remaining parts are left unlinked with a warning.
Fasttext language detection fed up to 1M words into the model with no
bound on word length or on the resulting token count. As the UTF
tokenizer intentionally does not enforce max_word_len, a crafted body
made of huge unspaced words expanded into 3-4 subword ngram ids per
character: hundreds of megabytes of heap and matching vector-add work
in predict() from one message. Skip overlong words, cap the total
tokens fed to the model, and refuse OOV subword expansion in the shim
for words longer than any real dictionary entry.
[Fix] message: link plain text parts to their HTML alternative
The alternative part search compared the raw IS_TEXT_PART_HTML flag
value (0 or 4) with the boolean want_html argument (0 or 1), so a
text/plain part never matched its text/html sibling: only the
html-to-plain direction of alt_text_part was ever populated.
[Fix] html: eliminate DOM recursion to survive deeply nested messages
Convert html_append_tag_content, traverse_block_tags and
html_debug_structure from native-stack recursion to explicit heap
stacks, and maintain html_tag::depth incrementally instead of walking
the parent chain per tag. With max_tags = 8192 a message can legally
nest ~8190 elements, which overflowed 512KB worker-thread stacks.
A chain of bare "Content-Type: message/rfc822" wrappers recursed through
rspamd_mime_parse_message's own MESSAGE branch without ever tripping the
max_nested (64) limit. When descending into an embedded message the
parser allocates a fresh runtime (nst) and seeds nst->nesting from the
parent's value *before* incrementing, then increments the parent runtime
st which is not the one carried into the recursion. The multipart branch
and rspamd_mime_process_multipart_node bump the descended-through runtime
(and push it onto the stack so the recursive call's cleanup pops and
decrements it), but the message branch did neither, so nst->nesting
stayed at its initial value at every level and the entry guard never
fired.
The result was recursion bounded only by message size (~34 bytes per
level): deeply nested messages exhaust the worker stack (remote,
unauthenticated DoS via HTTP submission or SMTP/milter delivery), and the
per-level boundary pre-scan made even non-crashing depths quadratic in
CPU. The earlier S/MIME fix (f6536945) only covered the pkcs7-mime
re-entry path and did not address this.
Push npart onto nst->stack and bump nst->nesting in the message branch,
mirroring the multipart branch; the recursive call's existing cleanup
balances both. Bounding the depth to max_nested also caps the number of
preprocess passes, eliminating the quadratic-CPU vector.
Add a Lua unit regression test that feeds a 500-level message/rfc822
chain and asserts the parsed part count stays capped near max_nested.
[Feature] Configdump: show symcache flags in --symbol-details
Add a `flags` array (fine, empty, explicit_disable, explicit_enable,
ignore_passthrough, nostat, idempotent, mime, trivial, skip, composite,
ghost, coro) to the per-symbol details, so it is possible to check
whether a symbol requires explicit enabling; structural types are
already covered by the `type` field.
[Test] Settings: cover explicit_enable gating and implicit_allow policy
New functional test cases:
* explicit_enable symbols do not run without settings, under an
implicit-allow settings id, or with raw settings that do not list
them
* whitelist settings and task:enable_symbol() unlock them
* policy = implicit_allow keeps all symbols enabled while unlocking
listed explicit_enable symbols and disabling listed symbols, both
for a registered settings id and for raw settings
[Feature] Settings: add policy option and strict explicit_enable gating
Make symbols with the explicit_enable flag actually require explicit
enabling: previously they were executed whenever any settings were
applied to a task, as the implicit-allow settings branch in
cache_item::is_allowed never consulted the flag, and any non-null
task->settings unlocked such symbols regardless of content.
Now explicit_enable symbols run only when:
* listed in symbols_enabled of the applied settings (any policy), or
* force-enabled via task:enable_symbol()
To make it possible to combine the default-allow behaviour with
selectively enabled explicit_enable symbols and disabled symbols,
introduce a settings policy option in the apply block:
With policy = implicit_allow, symbols_enabled no longer switches the
settings to whitelist mode: all symbols stay enabled, the listed ones
are merely unlocked. The policy is also honoured for raw settings
passed via the Settings header.
Additionally, symbols listed in symbols_enabled are now always tracked
as force-enabled (previously only when the settings element was already
attached to the task, which missed the map-driven settings path where
the id is set after settings are applied).
[Fix] Split MIME address lists on semicolons as well as commas
A semicolon is not an RFC 5322 address-list separator (it only
terminates a group construct), but Outlook-style ;-separated recipient
lists are pervasive in real mail and most MTAs (e.g. Postfix) split
them. Previously such a list was parsed as a single malformed address
whose local part swallowed the semicolon and the next address, breaking
recipient counts and producing false invalid-address signals.
Treat ';' exactly like ',' in the list parser. This mirrors the
existing comma behaviour in every edge case: semicolons inside quoted
strings are preserved, an unquoted semicolon in a display name drops
the preceding phrase just like a comma does, an empty group
(undisclosed-recipients:;) now yields no addresses like the comma
variant, and the trailing ';' of a group construct no longer leaks
into the last member's domain.
Fix synchronous context_augment firing two GPT requests for one model
When your context_augment function calls the callback synchronously
(e.g., return function(task, content, cb) cb('some context') end),
the pending_fetches counter drops to 0 inside the callback,
triggering maybe_proceed() which calls proceed().
Then control returns to the main flow where the now-redundant guard
also sees pending_fetches == 0 and calls proceed(nil) a second time
— this time without the context snippet.
The else branch (line 1324) already handles the case where none
of the context features are enabled, so the removed fallback
was purely a duplicate path for synchronous callbacks.
[Test] Use unique envelope senders in aliases From tests
The three From-identity tests shared one envelope From/Rcpt pair, so the
greylist module (meta key = envelope from + rcpts + ip) soft-rejected
the third scan of the pair within its retry window, skipping the filters
stage and losing the GET_FROM symbol. This only manifested with tight
test pacing (pabot shard in CI); give each test a unique envelope
sender, as the rest of the suite does.
[Minor] Selectors: document the 'orig' address flavour
The from/rcpts extractors pass their arguments through to task:get_from
and task:get_recipients, so from('mime', 'orig') already selects the
address as it was seen in the message; document it and add a functional
test exercising the flavour through the selector pipeline.
[Fix] Forged recipients: compare addresses as they were transmitted
The rule compared the rewritten envelope view against the wire MIME
headers, so a recipient alias mapping to another domain (legal for
recipients) could make a perfectly matching To header look forged.
Compare the envelope and the headers at the same altitude: both as they
were seen in the message, which is also what this rule checked before
alias rewriting was applied to tasks.
[Test] Add functional tests for From identity preservation
- aliases: googlemail.com From keeps its domain while the user part is
canonicalized; a cross-domain virtual alias on the From is discarded;
a same-domain alias on the From local part still applies
- get_from: the 'orig' flavour returns the wire From (and only it) after
a task:set_from rewrite, and is a no-op without one
- dmarc: alignment and policy lookup use the wire From domain even when
a prefilter rewrote the MIME From
The REWRITE_MIME_FROM test prefilter is gated on a request header so it
stays inert in merged-suite tests that do not opt in.
SPF evaluates the RFC5321.MailFrom as it was transmitted (including
local-part macros), so resolve the SPF record and credentials from the
original envelope sender even if the envelope from has been rewritten,
e.g. by the aliases module.
[Fix] DMARC: evaluate the From and envelope domains as seen in the message
DMARC must align identifiers and look up the policy for the RFC5322.From
domain as it appears in the message, and check SPF alignment against the
RFC5321.MailFrom as transmitted. Use the 'orig' address flavour so that
rewrites done via task:set_from (e.g. by the aliases module) do not
affect alignment, policy lookup or rua report attribution.
[Rework] Lua API: make the 'orig' address flavour return wire addresses
Previously task:get_from({'mime', 'orig'}) returned both the preserved
original addresses and the rewritten ones appended by task:set_from,
making the flavour unusable for consumers that need the message as it
arrived (a duplicate-From check would misfire on the combined list, and
Lua could not tell the entries apart). Now, when the 'orig' flavour is
requested and originals were preserved by a rewrite, only the original
addresses are returned; without a rewrite the behaviour is unchanged.
Also expose the 'aliased' and 'original' email address flags to Lua and
document the flavour.
The From address is the sender's identity: SPF, DKIM and DMARC all key
on its domain. Service rules and alias resolution (e.g. a virtual alias
mapping to another domain) may only touch the local part of the From
address; any rewrite that would change the domain is now discarded.
Recipient rewriting is unaffected: recipients are interpreted by the
receiving host, and no authentication protocol keys on them.
Also fix a spurious task:set_from call: the set_from decision used the
callback-wide alias_resolved flag, so a rewrite of any earlier address
(e.g. an envelope recipient) caused the unmodified MIME From to be
marked as aliased and duplicated.
[Fix] Aliases: never rewrite googlemail.com domain to gmail.com
googlemail.com and gmail.com are DNS-distinct domains with independent
SPF and DMARC records (gmail.com publishes p=none while googlemail.com
publishes p=quarantine), so substituting one for the other breaks DKIM
alignment, changes the effective DMARC policy and misattributes DMARC
reports. Keep the gmail.com user-part canonicalization (dots and plus
tags) for googlemail.com addresses, but preserve the domain.
The domain equivalence remains available in lua_util.remove_email_aliases
for dedup-only consumers such as email hash lookups in RBL checks.
[Fix] Lua 5.5 compatibility: do not assign to for-loop variables
Lua 5.5 makes for-loop control variables read-only, so assigning to
them is now a compile-time error. Rebind the value to a local (or pass
it as a closure parameter) in all remaining places; found by
parse-checking the whole tree with luac 5.5.
Register waitForResponse listeners before clicking the reset button:
on localhost the historyreset response arrives before a listener
registered after the click can see it, which made the legacy
(Playwright 1.45.3) CI leg fail deterministically.
Lua 5.5 makes `for` expression variables constant. The recent upgrade to
lua 5.5 broke lua-lupa, which previously attempted to modify the loop
variable. This caused an unrecoverable error on startup:
rspamd_lua_require_function: require of lua_util.jinja_template failed: error loading module 'lupa' from file '/usr/share/rspamd/lualib/lupa.lua':\x0A\x09/usr/share/rspamd/lualib/lupa.lua:1858: attempt to assign to const variable 'word'
[Minor] WebUI: drop unreachable 304 success branch
The Rspamd controller sends "Cache-Control: no-store" on every WebUI
response (controller.c, issue #3330) and never emits Last-Modified/ETag
for these endpoints, so a browser never revalidates and a 304 response
is impossible. The `|| xhr.status === 304` in queryServer was dead code
copied from jQuery's default success range.
Remove it so queryServer, the /stat connect probe and the legacy /auth
probe all treat 200-299 as success. Treating 304 as an error is also
safer: a 304 body is empty, and success would have fed it to the JSON
parser.
[Fix] WebUI: route a malformed /stat response to the login dialog
The connect probe swallowed JSON parse errors (parseJsonSafe returned {}),
so a 2xx /stat with a non-JSON body (truncated by a proxy, a captive-portal
page, corruption) loaded an empty, half-loaded main UI with a "Cannot get
server status" alert and no login path. The former jQuery content-type
parse raised parsererror and fell through to the login form.
Parse the body explicitly and, on any failure, show the connect dialog —
matching the pre-migration behaviour and covering the broad case (any
non-JSON 2xx, not only application/json). Removes the now-unused
parseJsonSafe helper.
[Fix] WebUI: restore smooth scroll on the Scan tab
The jQuery-removal replaced .animate({scrollTop}) with
window.scrollTo({behavior:"smooth"}), which honours prefers-reduced-motion
and is interruptible by Tabulator's reflows. The former jQuery tween
ignored the preference and re-set scrollTop every frame, so it was always
smooth.
Add a requestAnimationFrame-based smoothScrollTo (400ms easeInOut, matching
jQuery's default) and use it for both Scan scrolls (scan result and the
"Clean" reset). Like the rest of the UI's animations it does not honour
prefers-reduced-motion, preserving the previous behaviour.
fireAjaxComplete (the active-request counter decrement) ran at the top of
finish(), before the success callback. During an "All SERVERS" query the
count therefore dipped to 0 between the neighbours probe and the
per-neighbour requests: ajaxStart re-fired and re-added fa-spin,
restarting the spinner's CSS animation. jQuery decremented after success,
keeping the count above zero for the whole fan-out.
Move the decrement to the end of finish(), after the success/complete
callbacks. This also removes the rapid fa-spin class churn that was
interrupting the Scan tab's smooth scroll.
- stats.js: guard the optional fuzzy_hashes key in addFuzzyStorage
(Object.entries/keys of undefined threw on every Status refresh when
fuzzy_check was disabled or a neighbour was down), matching the guard
addStatfiles already had.
- common.js: cancel the hide/show slide animations once the final state
is committed, so the fill:"forwards" no longer pins height:0 — a later
re-show was measuring the pinned height and animating 0→0.
- common.js: treat a single Element (incl. <select>/<form>, which expose
a numeric .length) as one node in toElements, not a collection.
- common.js: tolerate a null neighbours body before Object.keys.
- rspamd.js: null-guard the active-tab lookup in two spots, where a
missing active tab used to throw.
- upload.js: drop the dead params:{processData:false} (queryServer no
longer form-encodes).
- common.js: document the statusCode handler signature.
[Test] selectors: cases for partially-parsed selectors
Negative cases: a second ':' method, a stray closing paren and junk
after a ';' list element must be rejected. Control cases: valid
method/transform chains, selector lists and trailing whitespace must
keep parsing.
[Fix] lua_selectors: reject selectors that parse only as a prefix
The selector grammar was not anchored to the end of input, so lpeg
matched the longest valid prefix and silently dropped the rest of the
string. E.g. 'from("smtp"):domain:lower' (a second ':' cannot parse)
was accepted and evaluated as 'from("smtp"):domain', and any trailing
garbage after a selector or a ';' list element was ignored. All
consumers (multimap, rbl, ratelimit, settings, reputation, the
controller selector check used by the WebUI) reported such selectors
as valid while evaluating only the prefix.
Append an lpeg.Cp() capture after the grammar (plus optional trailing
whitespace) and make parse_selector require the whole input to be
consumed, logging the position and the unparsed tail otherwise.
User-visible change: selectors that previously loaded thanks to the
silent truncation now fail configuration load with an error pointing
at the offending token. Such selectors were never evaluating as
written, so failing loudly is the correct behaviour.
[Fix] rspamd_symcache.h: keep C linkage of lua.h for C++ includers
The raw lua.h include sat outside extern "C", so any C++ unit that
pulled rspamd_symcache.h before lua/lua_common.h got C++-mangled
declarations of the Lua C API and failed to link. Including
lua_common.h itself is not an option due to the include cycle via
rspamd.h -> cfg_file.h -> rspamd_symcache.h, so wrap lua.h the same
way lua_common.h does. This also lets composites sources include
lua_common.h in the natural position instead of first.
[Test] composites: functional cases for Lua conditions
Cover: option-inspecting condition, cross-symbol join on a shared
option value, numeric return as atom weight combined with an
expression limit, false condition, and depends_on deferring a
composite whose condition consults a postfilter symbol.
A condition is called as f(task, symbol) where symbol is a table in the
task:get_symbol() layout; it may return true/false or a number used as
the atom weight. The condition is ANDed with option filters of the atom
and a failed condition is treated exactly as a missing symbol, so
removal policies are not applied. Since a condition can consult symbols
invisible to the expression, the optional depends_on list feeds the
first/second pass placement as if those symbols were expression atoms.
rspamd_config:add_composite() now also accepts a full definition table,
and rspamd_lua_push_symbol_result() is exported for reuse.
[Fix] phishing: don't penalise same label under another TLD
Three fixes for cross-TLD brand false positives
(e.g. brand.co.uk displayed over a brand.com href):
- Normalise the DMARC_POLICY_ALLOW domain to eSLD before comparing it
with the link target tld: DMARC reports the raw From domain (possibly
a subdomain), so the existing exclusion for authenticated mail never
matched
- Compare registrable labels instead of stripping the last dot
component: the old code only handled single-label suffixes, so
brand.co.uk vs brand.com was levenshtein-compared as full strings and
scored full weight; now the identical label yields weight 0
- Check strict_domains regardless of the computed weight (gated on
tld ~= ptld, equivalent to the old reachable behaviour), so displaying
a strictly protected domain over a same-label different-suffix target
still fires at full weight
[Test] lua_cryptobox: unit tests for secretbox nonce padding
Guard against the regression fixed in #6122: a short nonce must produce
the same ciphertext as the explicitly zero-padded 24-byte nonce, and
short/padded forms must decrypt each other. Both assertions fail on the
unpatched code as they read past the nonce buffer.
[Fix] lua_cryptobox: pass padded nonce to secretbox encrypt/decrypt
Both secretbox encrypt and decrypt built a zero-padded 24-byte real_nonce
but passed the original short nonce pointer to libsodium, reading up to
23 bytes out of bounds and producing ciphertexts that depend on adjacent
memory contents.
[Minor] maps: move glob pattern anchoring into rspamd_str_regexp_escape
Add RSPAMD_REGEXP_ESCAPE_ANCHOR flag that wraps the escaped pattern
into ^(?:...)$ within the same allocation instead of a second
g_strdup_printf in the glob map insertion path.
[Fix] maps: anchor glob map patterns to match the whole subject
Glob map entries were compiled into unanchored regexps and matched
with substring search semantics, so a `t.co` entry matched
`walmart.com` and `*.bit.ly` matched `foo.bit.ly.evil.com`.
Wrap the translated pattern into `^(?:...)$` at map load time so
glob entries match the subject as a whole: bare names match exactly,
wildcards match only what they say.
This affects all glob maps: multimap glob/glob_multi, url_redirector
redirector_hosts_map, dkim_signing/arc signing_table and key_table,
mx_check exclusions and the rbl glob returncodes matcher. Maps that
relied on the accidental substring behaviour must now use explicit
wildcards.
Register the historyreset/history response listeners before the click
via Promise.all. waitForResponse only catches responses arriving after
it is set up, and the fast local historyreset round-trip could beat a
listener attached post-click, causing a flaky 10s timeout under CI load.
Remove jQuery from the RequireJS config (paths, app/bootstrap shims) and
delete the vendored jquery-3.7.1.min.js. All app modules now load without
jQuery; the bootstrap shim's stale "Popovers require jQuery" comment and
deps entry are removed.
Drive-by: rename the modalDialog data-backdrop attribute to its Bootstrap
5 name data-bs-backdrop, so the static-backdrop behaviour is actually
applied (the BS4 attribute has been silently inert).
Migrate libft.js off jQuery. $("<tag>", {hash}) element construction
becomes common.el, $.extend becomes spread, $.each becomes forEach.
The column-options dropdown's namespaced hidden.bs.dropdown.rspamd
rebind (jQuery namespaces) becomes a common.data-tracked
addEventListener that removes the previous handler before re-binding;
the delegated fuzzy-hash buttons (re-bound every render) use a cleanup
map so common.delegate doesn't stack. $(arr).not(arr) set-diffs become
Array.some/ includes, and .siblings()/.attr()/.html()/.val() move to
native DOM.
Migrate rspamd.js off jQuery. The /stat connect probe moves to a raw
XHR (avoiding common.query's error logging on the normal "not logged
in" path); $.ajaxSetup and $(document).ajaxStart/Complete are dropped
since no $.ajax calls remain. tabClick disables/enables the nav bar
controls via attribute+class helpers (li/a need both for Bootstrap);
:radio + .val([v]) and .data("value") become checkRadio / activeMenuValue
helpers; delegated handlers move to common.delegate; BS5 events move to
addEventListener (with {once:true} replacing self-removing .on/.off);
the popover content uses cloneNode and the outside-click dismiss uses
.contains(). Form/input re-binds become onsubmit/oninput properties.
Migrate stats.js off jQuery. The delegated mouseenter/mouseleave hover
highlighting (which doesn't bubble natively) is reimplemented on
mouseover/mouseout with a relatedTarget-containment check; the jQuery
sibling-traversal chains (prevAll/nextAll/slice/find) move to native
sibling walks. $.each becomes Object.entries/forEach, $(html).appendTo
becomes insertAdjacentHTML, and $.wrapAll becomes common.el DOM
construction. The legacy /auth fan-out keeps a raw XHR to preserve its
custom error message and alerted-session dedup; $.when().always becomes
Promise.all().finally. Adds common.getAjaxTimeout so direct XHRs honour
the configured timeout.
Migrate upload.js off jQuery: $.Deferred/$.when to Promise/Promise.all,
$.trim to trim, :selected to selectedIndex, multi-arg :not() to CSS,
.val/.prop/.attr/.find/.append/.empty/.children/.closest/.parent to
native DOM, .animate({scrollTop}) to scrollTo, and $("<option>") to
common.el. Option metadata stored via .data() (objects/arrays) moves
to common.data; the fuzzy-picker change handler is bound once instead
of re-bound on every refresh.
Migrate config.js off jQuery: HTML-string/$("<tag>") element building
to common.el, $.each to forEach, the :input/:checked pseudos to CSS
selectors, .data() round-trips (object values) to common.data, the
delegated .map-link handler to common.delegate, and .attr()/.val()/
.on()/.html()/.remove() to native DOM and addEventListener.
Map cells and the modal title now escape server data via textContent
(consistent with the leaf-modules hardening).
Migrate graph.js, selectors.js, history.js, and symbols.js off
jQuery: $.extend/$.each/$.trim and $().val/prop/attr/hasClass/is/
addClass/removeClass/change/click/on to native DOM and the shared
common.el / common.delegate helpers added in the common.js stage.
Delegated handlers on Tabulator-rendered DOM (.scorebar) move to
common.delegate; the repeated button disable/enable patterns
collapse into small per-module helpers.
jQuery remains loaded for the remaining modules (config, libft,
stats, upload, rspamd) — removed in later stages.
Rewrite the AJAX layer (query/queryServer) on XMLHttpRequest,
preserving jQuery's semantics: neighbours fan-out and aggregation,
per-server error dispatch, statusCode, timeout, GET data appended
to the URL as a query string, default Content-Type for POST bodies,
NProgress download progress, and jqXHR-compatible callback arguments.
Add a spinner pub/sub (onAjaxStart/onAjaxComplete with an in-flight
counter honoring global:false) and setAjaxTimeout, plus shared
vanilla helpers: el, delegate, data. Migrate alertMessage, hide,
show, toggle (Web Animations API), the error-log UI and fileUtils
to plain DOM.
Bridge rspamd.js to the new APIs (setAjaxTimeout, onAjaxStart/
onAjaxComplete) alongside the jQuery hooks, which stay for the
remaining direct $.ajax calls until a later stage. jQuery is still
loaded; this is stage 1 of the staged removal.
Replace the Bootstrap 5 jQuery bridge (.modal/.popover/.trigger
"click.bs.dropdown") with the vanilla constructor API
(bootstrap.{Modal,Popover,Dropdown}.getOrCreateInstance) in
rspamd.js, config.js and libft.js, capturing the bootstrap module
where needed.
Prerequisite for removing jQuery: the bridge only activates when
jQuery is loaded, while the vanilla API works regardless. With jQuery
still present the existing $(el).on("*.bs.*") subscriptions keep
firing, so those stay jQuery for now and move to addEventListener
together with the rest of the jQuery removal. The main.js shim
deps:["jquery"] becomes redundant and is dropped in the final stage.
Replace the jquery.stickytabs plugin with an inline vanilla
implementation using the BS5 Tab API, removing the last jQuery
bridge dependency of that plugin. Behavior (URL hash persistence
and initial-tab fallback) is preserved.
D3Evolution 2.0.4 -> 2.1.0, D3Pie 1.1.2 -> 1.2.0. Both libraries
dropped the jQuery dependency (replaced $.extend with an internal
deep-merge), so drop the stale "jquery" shim deps in main.js.
Removing footable.standalone.min.css dropped its .form-group
{margin-bottom:15px} rule, collapsing the gap above the buttons on
the selectors tab (and the action-score rows on the configuration
tab). .form-group is a Bootstrap 3 class with no meaning in BS5;
replace it with the mb-3 spacing utility that provides the same gap.
[Fix] lua_selectors: don't crash on a missing method call
Selectors like `time:digest` (method syntax used where a transform
was meant) aborted the whole selector at scan time with an uncaught
Lua error:
Cannot run callback: .../lua_selectors/init.lua: attempt to call
a nil value
The `:` separator always compiles to a method call on the extracted
value; when the value has no such method, the lookup yields nil and
the unguarded call crashed the callback. Guard the lookup (via pcall,
since indexing some types raises) and yield no value instead, logging
an error that hints at the `.name` transform syntax when the name
matches a known transform. Valid method calls (e.g. `:gsub`,
`:lower`) and table field access are unaffected.
installScrollPreservation bound listeners on the mount element and
window without removing the previous set. Each column-options rebuild
(destroy + re-init on the persistent mount element) stacked a new set
that was never released — a listener/memory leak on the history and
scan tables. Mirror bindRowClickToggle's WeakMap guard: key the set by
mount element and tear it down before re-binding.
FooTable was superseded by Tabulator across all tables (Phases 0-3).
Remove the vendored library/CSS, the footable-fontawesome icon module,
and every remaining reference:
- index.html: drop the two <link> tags
- main.js: drop the requirejs path and shim
- rspamd.js: drop the app/footable-fontawesome require; stickyTabs
runs directly (declared module dependency)
- common.js: delete dead appendButtonsToFtFilterDropdown (no callers)
- rspamd.css: drop all .footable rules and the dead .footable-details /
.footable-filtering-search selectors
Rename column-options dropdown hooks ft-columns-* -> tab-columns-*
(Tabulator; consistent with tab-utils.js). Legacy history path kept.
[Test] static_embed: pack fixture floats via rspamd_util.pack
math.frexp was removed in Lua 5.4, so the hand-rolled float32 packer
broke the test run on non-LuaJIT builds; rspamd_util.pack provides
string.pack semantics on every supported Lua version.
Add model:get_token_vectors(input[, opts]) for offline consumers
(external trainers exporting order-aware text features): the token
embedding sequence in token order instead of only the pooled mean.
- Accepts exactly what get_sentence_vector accepts (word list or whole
text) and tokenizes through the same shared code path; unk rows are
included the same way the pooled path includes them.
- opts.max_tokens truncates after tokenization to the first N tokens
(the returned count is post-truncation); opts.raw returns an
rspamd_text with ntokens*dim little-endian float32s packed row-major
instead of a table of tables. Invalid opts raise errors, no silent
coercions. Empty input yields an empty table/text and 0, never nil.
- The provider path is unchanged: fusion vectors stay fixed-dim, so the
neural provider keeps using only the pooled get_sentence_vector.
- Tests: id/row alignment, pooled-mean consistency (incl. unk-heavy
input), word-list/text equivalence, max_tokens, raw packing and
strict opts validation.
Add a static token-embedding provider (Model2Vec style), the cheap
multilingual successor to fasttext_embed: words from rspamd's regular
tokenization pipeline are re-tokenized into WordPiece subword tokens and
embedded by mean-pooling rows of a precomputed float32 matrix, with no
neural forward pass and no new dependencies.
- rspamd_static_embed: a Lua-C module combining a WordPiece tokenizer
(BertNormalizer via ICU + Bert pre-tokenizer + greedy WordPiece) with
an mmap-ed embedding matrix shared between workers. The model spec is
read from the model directory (config.json + vocab.txt + matrix +
optional HF tokenizer.json) and validated strictly, fail-fast: any
unsupported normalizer/pre-tokenizer/model type, pooling other than
mean, non-float32 matrix or size mismatch fails the load instead of
degrading silently. get_sentence_vector() accepts a word list (the
provider path) or a whole text; both produce identical vectors.
- The WordPiece tokenizer is internal to the vectorizer: the global
word-breaking / statistics tokenization path is untouched, so Bayes
tokens and fuzzy hashes are unaffected.
- static_embed provider: extracts words like fasttext_embed and feeds
them to the model; the Lua side holds no matrix data and uses no FFI.
- unit tests with a generated fixture covering normalization, subword
splitting, greedy matching, CJK padding, unk handling, mean pooling,
word-list/text equivalence and strict rejection of unsupported
configs (BPE model type, vocab/matrix size mismatches, pooling).
Verified against the reference tokenizer oracle: token ids match
exactly, pooled vectors match within 2.4e-05 max abs diff, and the
word-list path is bit-identical to raw-text tokenization on the corpus.
Add a per-table boolean search box (history/scan) with a query
language ported from FooTable's built-in filtering: whitespace =
AND, OR, - (exclude), "exact phrase". The predicate ANDs with the
per-column header filters.
Also add headerFilter:"input" to the text columns (matching the
errors-table convention) and align all header-filter inputs on a
shared bottom baseline (title-holder flex-grow) regardless of how
many lines a wrapped title occupies.
Migrate the last two FooTable tables, #historyTable_history and
#historyTable_scan, to Tabulator. Both share rendering code in libft.js
(process_history_v2/columns_v2, initHistoryTable, the column-options
dropdown and the symbol-order toggle).
- Action column uses a custom headerFilter (select + "not" checkbox)
with headerFilterFunc for exact matching.
- Column-options dropdown keeps Visible/Hidden plus "Row", emulated
via responsive:100 and persisted to localStorage.
- Symbol-order "Sort by:" buttons are embedded in the symbols column
title, so they are recreated on every render; the active state is
reapplied on tableBuilt/renderComplete.
- columnDefaults uses the "html" formatter so upstream-escaped values
decode instead of being re-escaped by "plaintext".
- Shared helpers (scroll preservation, etc.) live in tab-utils.js.
Update the scan/symbols Playwright specs for the Tabulator DOM and
replace the racy "disabled during reload" check with response waits.
- Extract reusable Tabulator helpers (scroll-prevention, tabindex
removal, footer-hide, row-click-toggle, scrollIntoView guard) from
history.js into a new tab-utils.js module — no footable dependency,
so graph.js no longer loads footable.min.js for the throughput tab
- Refactor history.js initErrorsTable to use tab-utils helpers
- Migrate the rrd summary table (graph.js) from FooTable to Tabulator:
row coloring via CSS variable, in-place updates via updateData,
dynamic column titles via updateColumnDefinition, header sorting
- Update rspamd.css: rrd-table selectors for Tabulator DOM, row color
via --rrd-row-color (visible in dark mode), row hover, layout tweaks
Replace FooTable with Tabulator (v6.4.0) for the errors log table — the
first step of the FooTable→Tabulator migration. Tabulator is vanilla-JS and
jQuery-free, so it survives the planned jQuery removal; FooTable is
unmaintained (GPLv3, last release 2017).
- Vendored tabulator.min.js + tabulator_bs5.min.css; wired via RequireJS
- Rewrote initErrorsTable on the Tabulator API: columns, local pagination,
responsive collapse, per-column header filters
- rspamd.css: map the bs5 theme to Bootstrap CSS variables so light/dark
flip automatically with data-bs-theme
- Row click toggles responsive collapse (only when collapse is active);
pagination footer hidden on a single page (FooTable parity)
- Scroll prevention: Tabulator scrolls the page internally on interactions;
the call is elusive, so window.scrollY is preserved/restored around
clicks/Update. Row hover backgrounds disabled to avoid the resulting
expand-scroll flicker (documented inline for future maintainers).
Stefan Benten [Tue, 30 Jun 2026 20:02:26 +0000 (22:02 +0200)]
[Feature]: src/rspamadm: allow reading password from env (#6114)
* src/rspamadm: allow reading password from stdin
Currently the cli only allows reading the password from tty.
This makes it challenging to check the password via automation,
ie. used by continous deployment to ensure a clean state.
This change adds the functionality to read from stdin, so that
passwords can be piped in.
* test/functional/cases: test for stdin password checks
Vsevolod Stakhov [Tue, 23 Jun 2026 07:58:48 +0000 (08:58 +0100)]
[Feature] rspamadm: discover command modules from external directories
rspamadm discovered Lua sub-commands by globbing only the built-in
LUALIBDIR/rspamadm directory, so a third-party or premium package could
not ship a rspamadm command without writing into the OSS-owned lualib
tree.
Scan two additional sources, in order, after the built-in directory:
- $CONFDIR/rspamadm.d/*.lua, a drop-in dir consistent with local.d
- every directory in the colon-separated RSPAMADM_COMMAND_PATH env var
(mirrors how PATH works)
Duplicate command names are skipped (first wins), so built-in commands
can never be shadowed. Externally loaded modules run with the same
globals and lua_path as built-in ones, so they can require premium
lualibs and use lua_redis. No behavior change when the env var and the
.d directory are absent.
Factor the per-file loader and per-directory scan into helpers, and fix
a latent out-of-bounds in the basename fallback (it searched .lua in the
full path but indexed the basename buffer). Document discovery order and
the new env var in the man page.
Vsevolod Stakhov [Tue, 23 Jun 2026 07:57:22 +0000 (08:57 +0100)]
[Fix] lua_kann: correct load arg index and save return value
rspamd_kann.load is a module function, not a method, so the options
table is at arg 1. The lua_istable(L, 1) branch incorrectly read the
filename field from arg 2 (a copy-paste from lua_kann_save where arg 1
is self), so the documented load({filename = ...}) form always failed
with 'missing filename'.
Also fix lua_kann_save: the trailing lua_pop popped the boolean result
it had just pushed, so save({filename = ...}) returned the filename
string instead of true. Pop the filename field before pushing the
result.
Add a regression test covering the file round-trip via the single-table
form.
[Test] Run push-triggered CI on all branches in forks
rspamd/rspamd restricts push to master (71605c25) to avoid the duplicate
push+pull_request run on feature branches that leaves a spurious *cancelled*
status. This trade-off is acceptable for the main repo where contributors
open PRs, but inconvenient in forks where CI is needed on every push without
opening a PR.
Vsevolod Stakhov [Sat, 20 Jun 2026 19:20:19 +0000 (20:20 +0100)]
[Feature] neural: forced-learn fast path and first-class freeze
Two training controls plus a supporting task primitive:
* train.forced_learn_minimal_scan (default on when disable_symbols_input):
a high-priority neural prefilter disables every non-neural symbol on an
ANN-Train scan, so a symbols-independent training vector is built without
issuing RBL/DNS, fuzzy, bayes, ClickHouse or capture/cluster work. The
stored vector and the profile key are byte-for-byte identical to the live
full-scan path (asserted in tests). For symbol-dependent rules it stays off,
and if any applicable neural rule needs symbols the whole task falls back to
a full scan.
* train.frozen: stops automatic training and auto-storing of live vectors so a
frozen model's pools cannot drift into an imbalanced live set, while
inference keeps serving the current ANN unchanged. An explicit ANN-Train
still stores and retrains on demand (gated by a per-profile retrain marker).
Supersedes the auto-learn side of store_set_only/store_pool_only; both keep
working when frozen is unset.
* task:disable_all_symbols([skip_mask]): Lua binding over the existing
rspamd_symcache_disable_all_symbols "process only these" primitive (defaults
to keeping explicit_disable symbols), used by the prefilter.
Functional coverage in test/functional/cases/330_neural/006_forced_learn_minimal
and 007_frozen.
projects / thirdparty / rspamd.git / log
