[Fix] rspamd_symcache.h: keep C linkage of lua.h for C++ includers
The raw lua.h include sat outside extern "C", so any C++ unit that
pulled rspamd_symcache.h before lua/lua_common.h got C++-mangled
declarations of the Lua C API and failed to link. Including
lua_common.h itself is not an option due to the include cycle via
rspamd.h -> cfg_file.h -> rspamd_symcache.h, so wrap lua.h the same
way lua_common.h does. This also lets composites sources include
lua_common.h in the natural position instead of first.
[Test] composites: functional cases for Lua conditions
Cover: option-inspecting condition, cross-symbol join on a shared
option value, numeric return as atom weight combined with an
expression limit, false condition, and depends_on deferring a
composite whose condition consults a postfilter symbol.
A condition is called as f(task, symbol) where symbol is a table in the
task:get_symbol() layout; it may return true/false or a number used as
the atom weight. The condition is ANDed with option filters of the atom
and a failed condition is treated exactly as a missing symbol, so
removal policies are not applied. Since a condition can consult symbols
invisible to the expression, the optional depends_on list feeds the
first/second pass placement as if those symbols were expression atoms.
rspamd_config:add_composite() now also accepts a full definition table,
and rspamd_lua_push_symbol_result() is exported for reuse.
[Fix] phishing: don't penalise same label under another TLD
Three fixes for cross-TLD brand false positives
(e.g. brand.co.uk displayed over a brand.com href):
- Normalise the DMARC_POLICY_ALLOW domain to eSLD before comparing it
with the link target tld: DMARC reports the raw From domain (possibly
a subdomain), so the existing exclusion for authenticated mail never
matched
- Compare registrable labels instead of stripping the last dot
component: the old code only handled single-label suffixes, so
brand.co.uk vs brand.com was levenshtein-compared as full strings and
scored full weight; now the identical label yields weight 0
- Check strict_domains regardless of the computed weight (gated on
tld ~= ptld, equivalent to the old reachable behaviour), so displaying
a strictly protected domain over a same-label different-suffix target
still fires at full weight
[Test] lua_cryptobox: unit tests for secretbox nonce padding
Guard against the regression fixed in #6122: a short nonce must produce
the same ciphertext as the explicitly zero-padded 24-byte nonce, and
short/padded forms must decrypt each other. Both assertions fail on the
unpatched code as they read past the nonce buffer.
[Fix] lua_cryptobox: pass padded nonce to secretbox encrypt/decrypt
Both secretbox encrypt and decrypt built a zero-padded 24-byte real_nonce
but passed the original short nonce pointer to libsodium, reading up to
23 bytes out of bounds and producing ciphertexts that depend on adjacent
memory contents.
[Minor] maps: move glob pattern anchoring into rspamd_str_regexp_escape
Add RSPAMD_REGEXP_ESCAPE_ANCHOR flag that wraps the escaped pattern
into ^(?:...)$ within the same allocation instead of a second
g_strdup_printf in the glob map insertion path.
[Fix] maps: anchor glob map patterns to match the whole subject
Glob map entries were compiled into unanchored regexps and matched
with substring search semantics, so a `t.co` entry matched
`walmart.com` and `*.bit.ly` matched `foo.bit.ly.evil.com`.
Wrap the translated pattern into `^(?:...)$` at map load time so
glob entries match the subject as a whole: bare names match exactly,
wildcards match only what they say.
This affects all glob maps: multimap glob/glob_multi, url_redirector
redirector_hosts_map, dkim_signing/arc signing_table and key_table,
mx_check exclusions and the rbl glob returncodes matcher. Maps that
relied on the accidental substring behaviour must now use explicit
wildcards.
Removing footable.standalone.min.css dropped its .form-group
{margin-bottom:15px} rule, collapsing the gap above the buttons on
the selectors tab (and the action-score rows on the configuration
tab). .form-group is a Bootstrap 3 class with no meaning in BS5;
replace it with the mb-3 spacing utility that provides the same gap.
[Fix] lua_selectors: don't crash on a missing method call
Selectors like `time:digest` (method syntax used where a transform
was meant) aborted the whole selector at scan time with an uncaught
Lua error:
Cannot run callback: .../lua_selectors/init.lua: attempt to call
a nil value
The `:` separator always compiles to a method call on the extracted
value; when the value has no such method, the lookup yields nil and
the unguarded call crashed the callback. Guard the lookup (via pcall,
since indexing some types raises) and yield no value instead, logging
an error that hints at the `.name` transform syntax when the name
matches a known transform. Valid method calls (e.g. `:gsub`,
`:lower`) and table field access are unaffected.
installScrollPreservation bound listeners on the mount element and
window without removing the previous set. Each column-options rebuild
(destroy + re-init on the persistent mount element) stacked a new set
that was never released — a listener/memory leak on the history and
scan tables. Mirror bindRowClickToggle's WeakMap guard: key the set by
mount element and tear it down before re-binding.
FooTable was superseded by Tabulator across all tables (Phases 0-3).
Remove the vendored library/CSS, the footable-fontawesome icon module,
and every remaining reference:
- index.html: drop the two <link> tags
- main.js: drop the requirejs path and shim
- rspamd.js: drop the app/footable-fontawesome require; stickyTabs
runs directly (declared module dependency)
- common.js: delete dead appendButtonsToFtFilterDropdown (no callers)
- rspamd.css: drop all .footable rules and the dead .footable-details /
.footable-filtering-search selectors
Rename column-options dropdown hooks ft-columns-* -> tab-columns-*
(Tabulator; consistent with tab-utils.js). Legacy history path kept.
[Test] static_embed: pack fixture floats via rspamd_util.pack
math.frexp was removed in Lua 5.4, so the hand-rolled float32 packer
broke the test run on non-LuaJIT builds; rspamd_util.pack provides
string.pack semantics on every supported Lua version.
Add model:get_token_vectors(input[, opts]) for offline consumers
(external trainers exporting order-aware text features): the token
embedding sequence in token order instead of only the pooled mean.
- Accepts exactly what get_sentence_vector accepts (word list or whole
text) and tokenizes through the same shared code path; unk rows are
included the same way the pooled path includes them.
- opts.max_tokens truncates after tokenization to the first N tokens
(the returned count is post-truncation); opts.raw returns an
rspamd_text with ntokens*dim little-endian float32s packed row-major
instead of a table of tables. Invalid opts raise errors, no silent
coercions. Empty input yields an empty table/text and 0, never nil.
- The provider path is unchanged: fusion vectors stay fixed-dim, so the
neural provider keeps using only the pooled get_sentence_vector.
- Tests: id/row alignment, pooled-mean consistency (incl. unk-heavy
input), word-list/text equivalence, max_tokens, raw packing and
strict opts validation.
Add a static token-embedding provider (Model2Vec style), the cheap
multilingual successor to fasttext_embed: words from rspamd's regular
tokenization pipeline are re-tokenized into WordPiece subword tokens and
embedded by mean-pooling rows of a precomputed float32 matrix, with no
neural forward pass and no new dependencies.
- rspamd_static_embed: a Lua-C module combining a WordPiece tokenizer
(BertNormalizer via ICU + Bert pre-tokenizer + greedy WordPiece) with
an mmap-ed embedding matrix shared between workers. The model spec is
read from the model directory (config.json + vocab.txt + matrix +
optional HF tokenizer.json) and validated strictly, fail-fast: any
unsupported normalizer/pre-tokenizer/model type, pooling other than
mean, non-float32 matrix or size mismatch fails the load instead of
degrading silently. get_sentence_vector() accepts a word list (the
provider path) or a whole text; both produce identical vectors.
- The WordPiece tokenizer is internal to the vectorizer: the global
word-breaking / statistics tokenization path is untouched, so Bayes
tokens and fuzzy hashes are unaffected.
- static_embed provider: extracts words like fasttext_embed and feeds
them to the model; the Lua side holds no matrix data and uses no FFI.
- unit tests with a generated fixture covering normalization, subword
splitting, greedy matching, CJK padding, unk handling, mean pooling,
word-list/text equivalence and strict rejection of unsupported
configs (BPE model type, vocab/matrix size mismatches, pooling).
Verified against the reference tokenizer oracle: token ids match
exactly, pooled vectors match within 2.4e-05 max abs diff, and the
word-list path is bit-identical to raw-text tokenization on the corpus.
Add a per-table boolean search box (history/scan) with a query
language ported from FooTable's built-in filtering: whitespace =
AND, OR, - (exclude), "exact phrase". The predicate ANDs with the
per-column header filters.
Also add headerFilter:"input" to the text columns (matching the
errors-table convention) and align all header-filter inputs on a
shared bottom baseline (title-holder flex-grow) regardless of how
many lines a wrapped title occupies.
Migrate the last two FooTable tables, #historyTable_history and
#historyTable_scan, to Tabulator. Both share rendering code in libft.js
(process_history_v2/columns_v2, initHistoryTable, the column-options
dropdown and the symbol-order toggle).
- Action column uses a custom headerFilter (select + "not" checkbox)
with headerFilterFunc for exact matching.
- Column-options dropdown keeps Visible/Hidden plus "Row", emulated
via responsive:100 and persisted to localStorage.
- Symbol-order "Sort by:" buttons are embedded in the symbols column
title, so they are recreated on every render; the active state is
reapplied on tableBuilt/renderComplete.
- columnDefaults uses the "html" formatter so upstream-escaped values
decode instead of being re-escaped by "plaintext".
- Shared helpers (scroll preservation, etc.) live in tab-utils.js.
Update the scan/symbols Playwright specs for the Tabulator DOM and
replace the racy "disabled during reload" check with response waits.
- Extract reusable Tabulator helpers (scroll-prevention, tabindex
removal, footer-hide, row-click-toggle, scrollIntoView guard) from
history.js into a new tab-utils.js module — no footable dependency,
so graph.js no longer loads footable.min.js for the throughput tab
- Refactor history.js initErrorsTable to use tab-utils helpers
- Migrate the rrd summary table (graph.js) from FooTable to Tabulator:
row coloring via CSS variable, in-place updates via updateData,
dynamic column titles via updateColumnDefinition, header sorting
- Update rspamd.css: rrd-table selectors for Tabulator DOM, row color
via --rrd-row-color (visible in dark mode), row hover, layout tweaks
Replace FooTable with Tabulator (v6.4.0) for the errors log table — the
first step of the FooTable→Tabulator migration. Tabulator is vanilla-JS and
jQuery-free, so it survives the planned jQuery removal; FooTable is
unmaintained (GPLv3, last release 2017).
- Vendored tabulator.min.js + tabulator_bs5.min.css; wired via RequireJS
- Rewrote initErrorsTable on the Tabulator API: columns, local pagination,
responsive collapse, per-column header filters
- rspamd.css: map the bs5 theme to Bootstrap CSS variables so light/dark
flip automatically with data-bs-theme
- Row click toggles responsive collapse (only when collapse is active);
pagination footer hidden on a single page (FooTable parity)
- Scroll prevention: Tabulator scrolls the page internally on interactions;
the call is elusive, so window.scrollY is preserved/restored around
clicks/Update. Row hover backgrounds disabled to avoid the resulting
expand-scroll flicker (documented inline for future maintainers).
Stefan Benten [Tue, 30 Jun 2026 20:02:26 +0000 (22:02 +0200)]
[Feature]: src/rspamadm: allow reading password from env (#6114)
* src/rspamadm: allow reading password from stdin
Currently the cli only allows reading the password from tty.
This makes it challenging to check the password via automation,
ie. used by continous deployment to ensure a clean state.
This change adds the functionality to read from stdin, so that
passwords can be piped in.
* test/functional/cases: test for stdin password checks
Vsevolod Stakhov [Tue, 23 Jun 2026 07:58:48 +0000 (08:58 +0100)]
[Feature] rspamadm: discover command modules from external directories
rspamadm discovered Lua sub-commands by globbing only the built-in
LUALIBDIR/rspamadm directory, so a third-party or premium package could
not ship a rspamadm command without writing into the OSS-owned lualib
tree.
Scan two additional sources, in order, after the built-in directory:
- $CONFDIR/rspamadm.d/*.lua, a drop-in dir consistent with local.d
- every directory in the colon-separated RSPAMADM_COMMAND_PATH env var
(mirrors how PATH works)
Duplicate command names are skipped (first wins), so built-in commands
can never be shadowed. Externally loaded modules run with the same
globals and lua_path as built-in ones, so they can require premium
lualibs and use lua_redis. No behavior change when the env var and the
.d directory are absent.
Factor the per-file loader and per-directory scan into helpers, and fix
a latent out-of-bounds in the basename fallback (it searched .lua in the
full path but indexed the basename buffer). Document discovery order and
the new env var in the man page.
Vsevolod Stakhov [Tue, 23 Jun 2026 07:57:22 +0000 (08:57 +0100)]
[Fix] lua_kann: correct load arg index and save return value
rspamd_kann.load is a module function, not a method, so the options
table is at arg 1. The lua_istable(L, 1) branch incorrectly read the
filename field from arg 2 (a copy-paste from lua_kann_save where arg 1
is self), so the documented load({filename = ...}) form always failed
with 'missing filename'.
Also fix lua_kann_save: the trailing lua_pop popped the boolean result
it had just pushed, so save({filename = ...}) returned the filename
string instead of true. Pop the filename field before pushing the
result.
Add a regression test covering the file round-trip via the single-table
form.
[Test] Run push-triggered CI on all branches in forks
rspamd/rspamd restricts push to master (71605c25) to avoid the duplicate
push+pull_request run on feature branches that leaves a spurious *cancelled*
status. This trade-off is acceptable for the main repo where contributors
open PRs, but inconvenient in forks where CI is needed on every push without
opening a PR.
Vsevolod Stakhov [Sat, 20 Jun 2026 19:20:19 +0000 (20:20 +0100)]
[Feature] neural: forced-learn fast path and first-class freeze
Two training controls plus a supporting task primitive:
* train.forced_learn_minimal_scan (default on when disable_symbols_input):
a high-priority neural prefilter disables every non-neural symbol on an
ANN-Train scan, so a symbols-independent training vector is built without
issuing RBL/DNS, fuzzy, bayes, ClickHouse or capture/cluster work. The
stored vector and the profile key are byte-for-byte identical to the live
full-scan path (asserted in tests). For symbol-dependent rules it stays off,
and if any applicable neural rule needs symbols the whole task falls back to
a full scan.
* train.frozen: stops automatic training and auto-storing of live vectors so a
frozen model's pools cannot drift into an imbalanced live set, while
inference keeps serving the current ANN unchanged. An explicit ANN-Train
still stores and retrains on demand (gated by a per-profile retrain marker).
Supersedes the auto-learn side of store_set_only/store_pool_only; both keep
working when frozen is unset.
* task:disable_all_symbols([skip_mask]): Lua binding over the existing
rspamd_symcache_disable_all_symbols "process only these" primitive (defaults
to keeping explicit_disable symbols), used by the prefilter.
Functional coverage in test/functional/cases/330_neural/006_forced_learn_minimal
and 007_frozen.
Vsevolod Stakhov [Fri, 19 Jun 2026 11:31:37 +0000 (12:31 +0100)]
[Minor] lua_text stats: use 1-based byte indexing
The byte-distribution statistics methods (entropy, byte_mean,
byte_deviation, serial_correlation, monte_carlo_pi) took a 0-based
offset, which was inconsistent with the rest of the rspamd_text API
(span/sub/at) and with text:crc32. Switch the optional range argument
to a 1-based start index so all rspamd_text slicing uses the same
convention; len semantics and the lenient out-of-range-yields-0
behaviour are unchanged. Docs and tests updated accordingly.
Vsevolod Stakhov [Fri, 19 Jun 2026 11:28:15 +0000 (12:28 +0100)]
[Feature] lua: zlib/YARA-compatible crc32
Expose standard CRC-32 (poly 0xEDB88320, init/final 0xFFFFFFFF XOR),
computed via zlib crc32() so it is bit-exact with YARA hash.crc32:
- rspamd_cryptobox_hash.create_specific("crc32"): streaming
update + final, big-endian out so :hex() yields e.g. cbf43926
- rspamd_text:crc32([start[, len]]) and
rspamd_util.crc32(input[, start, len]): return the checksum as a
Lua integer over a zero-copy 1-based slice (no buffer copy)
Tests cover golden values ("" -> 0, "123456789" -> 0xCBF43926),
1-based slicing, streaming-vs-one-shot, reset, and a real
attachment-like buffer cross-checked against zlib/YARA crc32.
Vsevolod Stakhov [Fri, 19 Jun 2026 09:37:44 +0000 (10:37 +0100)]
[Feature] lua_archive: extraction limits vs zip bombs
archive.unpack/unzip/untar read whole members into memory via
libarchive, so a tiny hostile archive could expand to gigabytes
(OOM). Add an optional opts table that caps extraction, enforced
while reading so memory stays bounded:
- max_output total uncompressed bytes across all members
- max_file_size per-member uncompressed cap (truncated at cap)
- max_files member count cap
- max_ratio per-member uncompressed/compressed ratio cap
Ratio uses real compressed bytes consumed via
archive_filter_bytes(a, -1), checked per chunk above a 64 KiB
floor, so a bomb is stopped early even with no size cap set. All
limits are opt-in (0/absent = unlimited), preserving the previous
behaviour. A second boolean return flags truncation so a capped
extraction is never mistaken for a complete one.
Vsevolod Stakhov [Thu, 18 Jun 2026 18:16:54 +0000 (19:16 +0100)]
[Minor] CI: trigger push only on master to dedup PR runs
A commit on a branch with an open PR fired both a push and a
pull_request run; concurrency cancel-in-progress reaped one, leaving a
spurious *cancelled* run that reads like a CI failure. Restrict push to
master so feature branches run a single pull_request workflow; key the
concurrency group on PR number / ref to still cancel superseded runs.
Add byte-distribution statistics as methods on the rspamd_text class,
implemented in C++20 under src/lua (lua_text_stats.{hxx,cxx}); lua_text.c
is left untouched and the rspamd{text} metatable is augmented at load.
Methods (each takes an optional 0-based (off, len) range, defaulting to
the whole buffer):
- text:entropy([off[, len]]) Shannon entropy, bits/byte
- text:byte_mean([off[, len]]) mean of unsigned byte values
- text:byte_deviation(mean[, off[, len]]) mean abs deviation from mean
- text:serial_correlation([off[, len]]) ENT serial correlation
- text:monte_carlo_pi([off[, len]]) ENT Monte-Carlo Pi deviation
The core is header-only, allocation-free and O(n) (a single histogram
pass shared by entropy/mean/deviation) and produces deterministic,
bit-reproducible results. Offsets are byte offsets, 0-based; the range is
clamped to the buffer and an out-of-range or empty range yields 0.
Add C++ doctest golden-vector tests (analytically-derived exact values)
and Lua unit tests covering empty/single-byte/uniform/two-symbol buffers,
overlapping groups, slicing and edge cases.
Vsevolod Stakhov [Thu, 18 Jun 2026 13:22:25 +0000 (14:22 +0100)]
[Feature] multipattern: explicit SOM flag and offset docs
SOM (start-of-match) reporting already exists on master as the default
(hyperscan compiles every pattern with HS_FLAG_SOM_LEFTMOST), but there
was no explicit way to request it and the offset convention was
undocumented.
- Add RSPAMD_MULTIPATTERN_SOM (rspamd_trie.flags.som): an explicit
opt-in for start offsets that also overrides no_start/single_match
(forces SOM and drops the incompatible SINGLEMATCH).
- Document the offset convention: pattern id is 1-based; match start
and end are byte offsets, 0-based, start inclusive and end exclusive
(one past the last matched byte), so end - start is the match length.
- Fix the regex (flags.re) fallback used when hyperscan is unavailable:
it discarded the real PCRE start and reported end - strlen(pattern),
which is bogus for variable-length matches. It now reports the true
start/end from rspamd_regexp_search.
Add C++ (rspamd_cxx_unit_multipattern.hxx) and Lua (trie.lua) unit
tests asserting (id, start, end) against hand-computed positions:
multiple/overlapping occurrences, icase, literal vs regex, no-match,
SOM-overrides-single_match and a large buffer. Existing rspamd_trie
behaviour and its callers (url.c, lang_detection, lua plugins) are
unchanged.
Vsevolod Stakhov [Wed, 17 Jun 2026 17:52:16 +0000 (18:52 +0100)]
[Fix] monitored: alphanumeric-only random DNS prefixes
random_monitored RBL checks built random labels from an alphabet that
included '-' and '_'. That produced names like '_Q8...0-' (leading
underscore, trailing hyphen) or '-7d0...' (leading hyphen) which are
not valid DNS labels (RFC 952/1123: no leading/trailing hyphen, no
underscore in hostnames). Authoritative DNSBL servers such as
spfbl.net reject these with SERVFAIL.
Restrict the alphabet to alphanumerics, which always forms a valid
label regardless of position while keeping ample entropy.
Vsevolod Stakhov [Tue, 16 Jun 2026 19:05:26 +0000 (20:05 +0100)]
[Fix] lua: refcount coroutine thread entries
The state/generation guards from the previous commit read the entry on
resume, so they only help while the entry is still allocated. If the
owning task is torn down while an async request is in flight, the entry
could be freed before the late completion fires, turning the guard into
a use-after-free.
Make thread_entry refcounted (ref.h, non-atomic - workers are single
threaded). The pool holds the initial reference; terminate_thread() and
the pool-full path now drop it via REF_RELEASE instead of freeing
directly, so the struct is destroyed only once the last reference goes
away. Every async library that stashes an entry for a later completion
now takes its own reference and drops it when done:
- dns/util: retain at the request, release in the one-shot callback.
- http/redis: retain at yield, release in the cbdata/ctx destructor
(and, for redis, at each point that consumes ctx->thread).
- tcp: retain at each yield, release at the matching resume, since the
tcp cbdata has several direct-free error paths that bypass its
destructor; pairing with yield/resume keeps the balance exact and
leaves bad-argument paths (which return before yielding) untouched.
Combined with the generation guard, a completion that races task
teardown now finds the entry alive but DEAD/recycled and refuses the
resume, instead of dereferencing freed memory.
Vsevolod Stakhov [Tue, 16 Jun 2026 15:35:34 +0000 (16:35 +0100)]
[Fix] lua: state management for coroutine thread pool
Async libraries (dns/redis/tcp/http/util) capture the "currently
running" coroutine at a yield point and resume it later from a C
completion callback. Nothing previously guaranteed the entry resumed
was still the one captured: a double-fired event, a completion racing
task teardown, or an entry recycled into another task would resume the
wrong (or freed) coroutine and corrupt memory. These failures are
interleaving-dependent and invisible in isolation.
Give each pooled thread an explicit lifecycle (FREE/RUNNING/YIELDED/
DEAD) plus a generation counter, both carried in the existing
thread_entry and per-request cbdata structs - no new allocations on any
hot path:
- get/return/terminate/yield/resume enforce legal state transitions,
so returning a suspended thread or resuming a non-suspended one now
aborts at the exact violation instead of corrupting a core later.
- lua_thread_resume_checked() refuses to resume unless the thread is
still YIELDED and its generation matches the value snapshotted at
the yield point; a stale/duplicate completion becomes a logged
no-op rather than a wrong-coroutine resume. All five async libs are
migrated to it.
- lua_tcp keeps cbd->thread pointing at the coroutine actually
yielded by sync read/write, so the resume always targets it.
generation is bumped on every acquire and release, so an entry that
goes back to the pool and is handed out again no longer matches a
completion that was already in flight.
Vsevolod Stakhov [Tue, 16 Jun 2026 07:54:45 +0000 (08:54 +0100)]
[Fix] dns: do not defer resolver nameservers (fixes #6096)
A nameserver that failed to resolve at config time was turned into a
zero-address PENDING_RESOLVE upstream by 904fd6218, then dereferenced as
NULL in rspamd_dns_server_init -> SIGSEGV at worker startup (regression
in 4.1.0). DNS resolver nameservers are consumed synchronously by
rdns_resolver_add_server and can never be promoted async, so never defer
them; also NULL-guard rspamd_dns_server_init as defense in depth.
Vsevolod Stakhov [Mon, 15 Jun 2026 08:59:47 +0000 (09:59 +0100)]
[Fix] mx_check: loopback-only MX is LOCAL, not bogon
A domain whose MX resolves only to loopback is hosted on the scanning
host itself -- a self-MX, typically the host's own FQDN mapped to
127.0.0.1 in /etc/hosts, which rspamd's resolver honours as a fake reply
that shadows public DNS. That made fully DMARC-aligned self-hosted mail
score MX_BOGON_ONLY (+8.0): the strongest "not spam infrastructure"
signal treated as the strongest spam signal.
Move 127.0.0.0/8 and ::1/128 from BOGON_CIDRS to LOCAL_CIDRS so a
loopback-only MX emits MX_LOCAL_ONLY (3.0) instead. test_mode now lifts
loopback out of the LOCAL set (was: bogon) so the probe path stays
exercisable against local listeners.
Add a regression test (170_mx_check_selfmx.robot, test_mode = false):
the existing suites run test_mode = true and cannot cover the production
loopback-classification path.
Vsevolod Stakhov [Sat, 13 Jun 2026 12:18:18 +0000 (13:18 +0100)]
[Feature] neural: sequence output mode and SIF word selection in fasttext_embed
Add a generic word-vector sequence output to the fasttext_embed provider
so that custom ANN architectures (e.g. attention pooling) can learn their
own pooling instead of receiving a pre-pooled vector:
* output_mode = "sequence": emits the first max_words word vectors
flattened word-major and zero-padded to max_words * channels.
* word_selection = "sif": since order-invariant poolers do not need a
prefix, optionally fill the sequence with the max_words most
distinctive (highest SIF weight) words from anywhere in the message
instead of the leading ones. Default stays "prefix".
This is the data half only; how the sequence is consumed is left to the
ANN architecture. Both modes are bounded by max_words.
Vsevolod Stakhov [Sat, 13 Jun 2026 12:18:01 +0000 (13:18 +0100)]
[Feature] neural: pluggable feature-provider and ANN-architecture registries
Turn the neural plugin into an extension point so third-party (including
closed-source) modules can add feature providers and network topologies
without patching the core.
* register_architecture(name, builder) / get_architecture(name): a
registry of ANN builders, function(n_inputs, rule) -> kann network.
The built-in 'symbol', 'embedding' and 'conv1d' architectures are now
registered through it; create_ann() dispatches on rule.architecture
and falls back to the historical auto-selection when it is unset, so
existing configs are unaffected.
* register_provider (already present) and register_architecture are
exported from the neural module, so a module that does
require 'plugins/neural' can register a custom provider or
architecture and select it with provider type / rule.architecture.
An unknown rule.architecture now fails loudly with a hint that the
providing module may not be loaded, instead of silently falling back.
Vsevolod Stakhov [Sat, 13 Jun 2026 12:17:42 +0000 (13:17 +0100)]
[Feature] lua_kann: expose slice and concat transforms
kad_slice and kad_concat_array were already implemented and serialized
in kautodiff but not reachable from Lua. Exposed as
rspamd_kann.transform.slice(node, axis, start, end) (0-based, end
exclusive, batch is axis 0) and rspamd_kann.transform.concat(axis,
node1, node2, ...). These make split/bypass/merge graph topologies
buildable from Lua, e.g. routing different parts of a fused input
vector through different sub-networks (needed by custom ANN
architectures such as attention pooling with late fusion).
New kad operator attn_pool (op 38, appended to preserve model
serialization compatibility): multi-head dot-product attention pooling
over a zero-padded sequence of word vectors with learned query vectors.
All-zero positions are treated as padding and masked out of the
softmax; attention weights are stashed in gtmp between the forward and
backward passes. Exposed as kann_layer_attn_pool() and
rspamd_kann.layer.attn_pool(node, n_words[, n_heads]).
Verified: converges on a needle-in-haystack task unsolvable by a flat
dense net (0.985 vs 0.715 accuracy), exact word-order invariance of the
pooled output, padding determinism and save/load roundtrip.
Vsevolod Stakhov [Fri, 12 Jun 2026 14:35:59 +0000 (15:35 +0100)]
[Fix] neural: stabilize training on dense embedding inputs
Training an ANN on dense provider features (fasttext_embed, text_hash)
could silently produce a degenerate model: with the historical
learning_rate=0.01 default, RMSprop drives the net into tanh saturation
depending on weight init luck - the loss freezes, yet the constant
all-one-class model is saved and classifies every message as spam (or
ham) until the next retrain. On a real corpus this happened in roughly
one of three weight inits.
Fixes:
* use the embedding (funnel) architecture for any rule with dense
feature providers, not only LLM ones: the simple symbol architecture
applies ReLU directly to the input, clipping the negative half of the
embedding space, and is the least stable option on such vectors
(it is also less accurate; layernorm in the funnel fixes the
conditioning)
* resolve the learning_rate default by input type: 0.01 for symbol
vectors as before, 0.001 for dense embeddings, which converges
reliably with equal accuracy; an explicit config value still wins
* add a quality gate to the training child: a model with constant or
single-class output on its own training set is rejected instead of
saved; the lock is released and training retries on the next cycle
with a different weight init, which converges in practice
* return an explicit msgpack rejection marker from the training child
instead of nil on the gate/NaN paths: a nil return used to deadlock
the controller against the training subprocess (see the lua_worker
fix) and stalled training forever
Vsevolod Stakhov [Fri, 12 Jun 2026 14:35:41 +0000 (15:35 +0100)]
[Fix] lua_worker: do not deadlock when subprocess returns an invalid value
When a function run via worker:spawn_process returned nil (or any
non-string value), the child logged an error but wrote nothing to the
result pipe. The parent then kept waiting for a reply while the child
blocked forever on the post-reply ack read, deadlocking both processes
and anything serialised behind them (e.g. the neural training lock,
which got extended indefinitely so training never retried until a full
restart).
Report invalid return values to the parent as a regular error reply so
on_complete fires and the caller can recover.
Vsevolod Stakhov [Fri, 12 Jun 2026 10:46:34 +0000 (11:46 +0100)]
[Fix] protocol: use case preserving boundary for HTTP multipart parsing
RFC 2046 boundaries are case sensitive, but the v3 HTTP multipart
callers used ct->boundary, which is lowercased for MIME clients quirks,
so requests with uppercase characters in the boundary failed to parse.
Use ct->orig_boundary in protocol.c, rspamd_proxy.c and rspamdclient.c.
Vsevolod Stakhov [Fri, 12 Jun 2026 00:15:50 +0000 (01:15 +0100)]
[Feature] css: detect more text hiding tricks
Extend the invisible-text detection with several common CSS hiding
techniques used to dilute visible content with hidden ham text:
- off-screen positioning: position:absolute|fixed with a large negative
left/top
- image-replacement text-indent: a large negative text-indent
- clip / clip-path collapsing the element to a zero area, e.g.
rect(0,0,0,0), inset(100%), circle(0)
- visibility:collapse (treated as hidden)
- tiny font sizes (<= 3px), not only font-size:0
These are modelled as a hidden display in compile_to_block so the
hiding correctly propagates to descendants. The previous off-screen
heuristic was a fragile substring match on the raw style that only
incremented a feature counter and never actually hid the text; it is
replaced by structured parsing of the position/left/top/text-indent/
clip properties, with the offscreen feature counter now driven by a
flag set on the compiled block.
Vsevolod Stakhov [Thu, 11 Jun 2026 22:02:56 +0000 (23:02 +0100)]
[Fix] css: detect text hidden via overflow clipping, opacity and max-* sizes
Phishing messages dilute the visible content with hidden ham text using
CSS hiding techniques that the parser did not understand:
- 'max-width:0; max-height:0; overflow:hidden' was fully ignored as
max-width/max-height/overflow were not parsed at all
- 'opacity:0' was parsed but the value was silently discarded in
compile_to_block
- 'height:0' was applied to the block width due to a copy-paste bug,
and zero dimensions were never considered by compute_visibility
Fixes:
- parse max-width/max-height (clamping width/height) and overflow
- treat a block with zero height or width and overflow:hidden as
invisible, propagating it to descendants via the display value
- treat opacity < 0.1 as a hidden display, as descendants cannot reset
the ancestor opacity
- do not allow a child display value to resurrect content of a hidden
ancestor in propagate_block (display:none is not resettable in CSS)
- fix the height->width copy-paste bug and a missing break that made
the font-size case fall through into the opacity case
Vsevolod Stakhov [Thu, 11 Jun 2026 17:20:02 +0000 (18:20 +0100)]
[Fix] mime: fix build with OpenSSL 4.0 opaque ASN1_STRING
OpenSSL 4.0 made ASN1_STRING (and thus ASN1_OCTET_STRING) opaque, so
direct access to its length/data fields no longer compiles. Use
ASN1_STRING_length()/ASN1_STRING_get0_data() which are available since
OpenSSL 1.1.0 and LibreSSL 2.7.
Also move the legacy OpenSSL init calls (ERR_load_crypto_strings,
SSL_load_error_strings, OpenSSL_add_all_*) under the pre-1.1.0 guard:
they are redundant on modern OpenSSL and break no-deprecated builds.
Vsevolod Stakhov [Thu, 11 Jun 2026 14:05:13 +0000 (15:05 +0100)]
[Fix] milter: send QUARANTINE even with a custom reply
The QUARANTINE command was sent inside the `if (!reply)` guard that
synthesises a default quarantine reason, so a caller-supplied SMTP
message (e.g. task:set_pre_result('quarantine', 'reason')) suppressed
the command entirely and the message was accepted instead of
quarantined. Affects both METRIC_ACTION_QUARANTINE and reject converted
via quarantine_on_reject. Regression from fbc6e35db (3.10.0).
The reply now becomes the quarantine reason, matching the reject and
tempfail branches where a caller-supplied message takes precedence over
the configured default.
Reported by @johnmosli, who also attached the fix.
Vsevolod Stakhov [Wed, 10 Jun 2026 19:02:34 +0000 (20:02 +0100)]
[Fix] url: scan bare query parameters containing '=' as a whole
A bare embedded URL is its own query parameter and can contain '='
itself (e.g. base64 padding in the path). The query-embedded scan
treated everything before the first '=' as a parameter key, so such
URLs were discarded. Treat the prefix as a key only when it has no
URL structure characters (':' or '/'); otherwise scan the whole
parameter.
[Fix] symcache: fix timeout inflation in pre_postfilter_iter grouping
The `saved_priority` initialization to -1 caused the first item in each
phase vector (prefilters/postfilters/idempotent) to be split off from
its priority group and counted individually, inflating the computed
maximum symbols cache timeout. Initialize to the first item's priority
instead so items at the same priority are correctly grouped.
[Test] checkv3: drop requests_toolbelt from form-data parse
The /checkv3 content negotiation tests parsed the multipart/form-data
reply with requests_toolbelt, a third-party module not present in every
test pipeline (e.g. the rspamd-docker functional run installs
python3-msgpack but not requests-toolbelt), so they failed with
ModuleNotFoundError.
Replace it with a self-contained stdlib HTTP-multipart splitter: split
on the boundary delimiter (HTTP-multipart style, deliberately not the
email/MIME parser used for the message/rfc822 case) and trim only the
single CRLF framing each part so binary (zstd) payloads stay byte-exact.
Drop the now-unneeded requests-toolbelt from the CI pip install.
A trained ANN could become unreachable to workers even though
training succeeded: NEURAL_SPAM/NEURAL_HAM stopped firing while the
controller logged "ann ... is changed, our version = N, remote
version = M" forever.
Root cause is a version regression, not a missing zset registration.
The new version was seeded from the in-memory set.ann, and
fill_set_ann resets set.ann.version to 0 whenever a worker never
loaded an ANN (restart, or the selected profile's blob was missing).
A worker that trained from the _4 profile then saved version 1.
process_existing_ann selects the highest version among compatible
profiles, so the live version-1 blob was shadowed by the stale
version-4 zset entry whose key was empty. The profile zset has no
TTL, so the dead high-version tombstone was immortal and the
condition self-perpetuated (the _4 blob was never rewritten).
Three fixes:
1. Version monotonicity (lualib/plugins/neural.lua): seed the new
version from the profile actually trained from (the trained-from
key encodes it as the trailing _<n>), max'd with
training_profile/set.ann, so the new entry always outranks the
profile it supersedes.
2. Liveness-aware selection (src/plugins/lua/neural.lua,
neural_maybe_invalidate.lua): when the selected profile's blob is
missing, fall back to the next compatible profile with a live blob
instead of going dark, and emit a throttled warning (was a silent
debug line). The invalidate script also GCs profile entries that
have no blob and no training data and are older than a grace
window.
3. Lifetime coupling (neural_save_unlock.lua,
src/plugins/lua/neural.lua): give the profile zset a TTL refreshed
each check_anns cycle, and refresh the blob TTL on every reload,
so an actively used ANN never expires out from under its entry.
Adds 330_neural/005_stale_version.robot, which injects a
higher-version tombstone and asserts inference recovers.
[Minor] url_redirector: distinguish direct URL errors from redirect errors
When http_callback reports an error on the first hop (orig_url == url),
no redirect has occurred yet, but the old message "found redirect error
from X to X" implied one. Split the message: "error checking URL" for
direct failures and "redirect error: A -> B" for mid-chain failures.
[Minor] checkv3: trim verbose comments to house style
Condense the multi-line explanatory blocks added with the negotiation
reply to single-line notes, and drop a dangling 'see contract above'
reference that pointed to nothing in the source.
The /checkv3 reply ignored Accept beyond a json-vs-msgpack toggle and
always emitted a hard-coded multipart/mixed body (with form-data part
headers). There was no way to ask for a plain v2-style json/msgpack
body, no true multipart/form-data reply for HTTP multipart parsers,
and Accept-Encoding had no defined default.
Negotiate the representation solely from Accept and compression solely
from Accept-Encoding on the single chokepoint all three workers (normal
scan worker, rspamd_proxy, controller) share, the reply_v3 helper:
Inside the multipart envelopes the result-part serialization mirrors
the input metadata serialization (json or msgpack); the two envelopes
differ only in the top-level Content-Type. Compression honours
Accept-Encoding: zstd and defaults to identity. Vary: Accept,
Accept-Encoding is always advertised. Negotiation reuses the existing
http_content_negotiation parser (q-values + wildcards), extended with
two media types; the input metadata serialization is recorded on the
task via a new protocol flag.
rspamc previously sent Accept: application/json|msgpack for v3, which
now selects a single-body reply it does not expect; it now requests
multipart/form-data and accepts any multipart/ subtype, with the result
serialization carried by the metadata Content-Type.
Tested by a new C++ content_negotiation suite, multipart envelope-mode
unit tests, and a functional negotiation suite run against both the
normal worker and the controller (json/msgpack/email-MIME/HTTP-multipart
parsers). Adds msgpack/requests/requests-toolbelt to functional CI deps.
The scan worker and rspamd_proxy both handle /checkv3 (multipart
metadata + message in, multipart/mixed results out), but the
controller's HTTP path router never registered it. Since rspamc
defaults to the controller port (11334) for localhost, a plain
`rspamc --protocol-v3` returned 404 with
"rspamd_http_router_finish_handler: path: /checkv3 not found".
Register /checkv3 on the controller (routed to the existing scan
handler) and branch on CMD_CHECK_V3 in both directions:
parse the body via rspamd_protocol_handle_v3_request() on input and
emit the multipart reply via rspamd_protocol_http_reply_v3() on
output, mirroring the proxy. Auth posture matches /check and
/checkv2 (read command, no enable password).
[Minor] url_redirector: clarify log messages for successful HTTP responses
The phrases "err code 200" and "err code <N>" are misleading since
they refer to HTTP status codes, not errors. Successful resolutions
(HTTP 200) and intermediate redirects (30x) now use unambiguous
wording that clearly separates the action from the status code.
[Fix] url: canonicalise bare emails to slash-less mailto:
A bare email in text/HTML and the same address inside an explicit
mailto: URL were extracted as two separate emails. The '@' matcher (and
the HTML bare-email path) injected a literal "mailto://" prefix, while a
parsed mailto: URL is non-hierarchical and drops the // (RFC 6068, a4ae51536). The URL/email khash dedup keys on the full url->string
(hash + a urllen guard ahead of rspamd_emails_cmp), so the two string
forms landed in different buckets and never collapsed -> the address was
duplicated.
Canonicalise both bare-email injection sites to the slash-less "mailto:"
form so every path yields the identical mailto:user@host string and the
existing dedup works:
- src/libserver/url.c: '@' matcher prefix "mailto://" -> "mailto:"
- src/libserver/html/html_url.cxx: same for bare emails in HTML
Adjust the extract_specific_urls scheme-strip helper to tolerate the
slash-less form, and add a regression test in test/lua/unit/url.lua.
[Fix] url: keep query nesting cap a fixed functional limit
f068a1156 derived RSPAMD_URL_QUERY_MAX_NESTING from the multipattern
scratch budget (MAX_REENTRANCY - 2), which silently bumped the
redirect/wrapper unwrap depth from 5 to 8 and broke the get_html_urls
unit test that pins the cap at 5.
The nesting depth is a functional/product decision, not a function of
the scratch pool size. Restore the fixed cap of 5 and instead assert at
compile time that it stays within the scratch budget (plus the
enclosing scan and the leaf TLD lookup); rspamd_multipattern_lookup()
still degrades gracefully if the bound is ever exceeded.
Schedule every external service under a stable <RULE>_CHECK callback
symbol so it is a predictable dependency target, regardless of how the
scan result symbols are named. This generalises the pattern vadesecure
and cloudmark already follow (VADE_CHECK, CLOUDMARK_CHECK).
A scanner whose main symbol is already a *_CHECK keeps it as the
callback. Otherwise the callback is named <KEY>_CHECK (from the config
block key -- unique per rule, so instances never collide) and the
scanner's result symbol (e.g. DCC_REJECT, RAZOR) becomes a virtual
child of it: its score and emitted results are unchanged, and an
existing dependency on the old name still resolves (virtual -> parent).
This lets register_dependency('<SERVICE>_CHECK', X) order any external
check after another symbol uniformly, e.g. after a sender-bypass rule.
[Fix] external_services: honour scanner default symbol name
Commit f654ec35d resolved the rule symbol names up-front as
`opts.symbol or sym:upper()` so the configure()-failed stub could still
register a fail symbol, then assigned them unconditionally - which
discarded the symbol the scanner resolves in configure(). A scanner
such as vadesecure defines `symbol = 'VADE_CHECK'` as its default main
symbol, but a rule keyed `vadesecure { ... }` without an explicit
`symbol =` was then registered as VADESECURE instead of VADE_CHECK.
Besides the surprising rename, this broke dependency registration
against the documented symbol:
Restore the pre-regression precedence: prefer the symbol the scanner
resolved (its own default, or the user's `symbol =` applied via
override_defaults) and fall back to the key-derived names only when the
scanner left them unset. The up-front names are still used for the
configure()-failed stub. Applies to every scanner with a default
symbol (vadesecure, dcc, razor, cloudmark, ...).
A URL whose query embeds a percent-escaped URL is unwrapped
recursively (PR #6066): rspamd_url_find_in_query re-enters
rspamd_multipattern_lookup on the URL trie while the enclosing scan
is still on the stack. Each scan borrows one of MAX_SCRATCH hyperscan
scratch contexts; once the recursion nests deeper than the pool, the
slot loop leaves scr == NULL and g_assert(scr != NULL) aborts the
worker. A crafted message with a few levels of nested query URLs thus
crashes a normal worker (DoS).
The peak number of simultaneously-held scratch contexts on the
deepest chain is RSPAMD_URL_QUERY_MAX_NESTING + 2: one for the
enclosing text/subject scan and one for the per-URL TLD lookup that
rspamd_url_parse runs on each freshly extracted leaf. The old pool of
4 with a nesting cap of 5 needed 7 -> assertion.
- Introduce RSPAMD_MULTIPATTERN_MAX_REENTRANCY (10) and size the
scratch stack from it; a scratch context is ~2.5-4 KiB, so the
deeper stack costs only tens of KiB per multipattern.
- Tie RSPAMD_URL_QUERY_MAX_NESTING to that budget (minus the two
implicit levels) so normal nesting stays on the fast path.
- Make scratch exhaustion non-fatal: allocate a one-off scratch for
the scan instead of aborting the worker on attacker input.
- Guard the unsigned-int scratch bitmask with a static assert.
Add functional regression test 170_url_query_nesting.
The two-phase composite evaluation classified a composite as second-pass
if any referenced symbol carried SYMBOL_TYPE_NOSTAT. NOSTAT is auto-set
on nearly every virtual/callback symbol (regexp rules, multimap, rbl,
...), so most composites that depend only on ordinary filter rules were
wrongly deferred to the COMPOSITES_POST stage, which runs after
post-filters. As a result task:get_groups()/get_symbols() called from a
postfilter no longer saw those composite symbols and their groups
(regression vs 3.5); they only became visible from idempotent rules.
Classifiers run before composites, so keying off SYMBOL_TYPE_CLASSIFIER
was also unnecessary.
Only postfilter-stage symbols are genuinely unavailable during the first
composites pass. Add rspamd_symcache_get_symbol_stage(), which resolves
virtual symbols to their parent and returns the processing stage as a
SYMBOL_TYPE_* bit, and defer a composite only when a dependency resolves
to the postfilter stage. NEURAL_SPAM (a virtual child of the
NEURAL_CHECK postfilter) still resolves to POSTFILTER, so the original
#5674 fix keeps working.
Add a functional regression test covering a composite that depends only
on a filter-stage NOSTAT symbol and must be visible from a postfilter.
[Fix] ratelimit: Track all buckets in selector rules
When a rule defines several buckets (e.g. "200 / 1h" plus "30 / 1m")
and uses a selector, limit_to_prefixes keyed the prefixes table by the
selector value alone. Every bucket therefore mapped to the same Redis
key and only the last bucket in the array was ever tracked, so the
other limits were silently ignored.
Give each bucket a distinct key by prefixing the selector value with a
per-bucket id (burst + rate), mirroring the burst component that
gen_rate_key already prepends for the non-selector path. The
non-selector path is unchanged, so its existing Redis keys are kept.
Adds a functional regression test (two buckets, burst 2 + burst 20)
that fails before the fix because the restrictive bucket is ignored.
Alexandra Parker [Sun, 31 May 2026 20:52:35 +0000 (13:52 -0700)]
[Fix] mime_string const iterator
mime_string's iterator is just a value iterator anyway. it's
intrinsically const. drop reference to const_iterator, let iterator_base
take a const pointer, and mark begin() and end() as const.
doctest 2.5.0 receives a const reference and can't use a mutable iterator,
which leads to compile error.
[Feature] protocol: Expose custom metadata for /checkv3
Add two complementary ways to read custom fields sent with a /checkv3
multipart scan request, both free of the 80KB HTTP header limit that v2
hits, since the metadata travels in the multipart body:
* A "headers" sub-object in the metadata part is injected into the
task request headers, so task:get_request_header() works for custom
fields exactly like v2 HTTP request headers. Reserved control-header
names (shm/file/path/dictionary/Content-Encoding...) are skipped so
client metadata cannot collide with the message-loading channel, and
a repeated name (collapsed by UCL into an array) expands to a
multi-valued request header.
* The parsed metadata object is kept on task->meta and exposed to Lua
via task:get_metadata() and task:get_metadata_field(key), mirroring
get_settings()/lookup_settings(). The task now owns the object and
frees it once in rspamd_task_free instead of via a pool destructor.
rspamc gains a repeatable --metadata-header KEY=VALUE option that builds
the metadata "headers" sub-object for v3 requests. Also drop a dead
is_msgpack variable in the v3 request handler.
Tests: functional cases in 430_checkv3.robot plus a checkv3_meta.lua
plugin exercising both options via raw multipart and rspamc.
projects / thirdparty / rspamd.git / log
