Hiroshi Murakami [Tue, 31 Mar 2026 04:55:33 +0000 (13:55 +0900)]
[Fix] Add missing #include <memory> in redis_backend.cxx
std::make_unique is used (line 584) but <memory> is not explicitly
included. This causes compilation failures with some compiler/platform
combinations (e.g. gcc-toolset-12 on Rocky Linux 8) where <memory>
is not transitively included by other headers.
Vsevolod Stakhov [Mon, 30 Mar 2026 11:56:55 +0000 (12:56 +0100)]
Release 4.0.0
** Incompatible changes **
* Replace Jump Hash with Ring Hash (Ketama) for consistent upstream hashing;
per-user Bayes with Redis sharding requires migration via rspamadm statistics_dump migrate
* Include content URLs by default in URL API calls (include_content_urls = true)
* Auto-detect SSL from bind sockets, remove ssl = true worker option
* Replace libfasttext with built-in mmap-based shim; ENABLE_FASTTEXT cmake option removed
* Replace builtin_suspicious TLDs with map-based configuration
* Rename neural autolearn options to match RBL module naming
* proxy: Enable token bucket load balancing by default
* Disable Validity SenderScore RBLs by default (requires MyValidity account)
* Make unknown and broken DKIM keys behaviour conforming to RFC
** Major features **
* Add /checkv3 multipart scan endpoint with per-part zstd compression
* Pluggable async hyperscan cache backend with Redis storage support
* Add multi-flag fuzzy hash support with Lua-based Redis update path (epoch 12)
* Add dual-mode HTML fuzzy: template matching + phishing detection
* Implement HTTPS server support for workers
* proxy: Implement token bucket load balancing for upstreams
* Expose milter headers and extended symbols in legacy RSPAMC/SPAMC protocol
* Add native UUID v7 per task with ClickHouse support
* Add structured formatter to metadata_exporter with zstd compression
* arc: Add trusted_authserv_id option for reuse_auth_results
* Add external pretrained neural model support
* Fasttext embed: multi-model, mean+max pooling, SIF word weighting
* Multi-layer funnel architecture for LLM embeddings
* Add expression-based autolearn for neural LLM providers
* Add rspamadm autolearnstats, logstats, mapstats subcommands
* Add shard migration and multi-class support to statistics_dump
* headers_checks: Add Reply-To address validity checks
* Store matched fuzzy hashes in Redis history
* Add HTTP content negotiation framework with zstd compression
* Extend /stat and /bayes/classifiers endpoints with classifier metadata
* WebUI: Add multi-class classifier support to learning UI
* Split attachment filenames into sub-tokens for Bayes classifier
* Make GPT consensus thresholds configurable with context_augment hook
** Bug fixes **
* [CritFix] Stop ev_io watcher in fuzzy UDP session destroy
* Fix CPU busy-loop in fuzzy TCP client
* Fix SPF address family flag inheritance
* Fix RHEL/CentOS 10+ crypto-policies for SHA-1 DKIM verification
* Fix EVP_PKEY_CTX memory leak in DKIM RSA signing
* Fix ratelimit compatibility with old records
* Fix use-after-free in pending regexp map and multipattern queues
* Self-healing hyperscan cache: delete stale blobs and trigger recompile
* Move binary data from KEYS to ARGV in Redis scripts
* Rework alternative parts detection
* Add PCRE2 complexity checks before JIT compilation
* Default map URL path to "/" when no path component is present
[Fix] Skip HELO IP address literals for SPF domain resolution
Per RFC 7208 ยง2.3, IP address literals (e.g. [192.0.2.1]) cannot be
used as SPF domains. When MAIL FROM is empty, rspamd falls back to
HELO as the SPF domain.
With an IP literal in HELO, rspamd_spf_resolve() failed (DNS name
rejected by the resolver), but spf_cred was non-NULL, so the result
was incorrectly set to R_SPF_DNSFAIL instead of R_SPF_NA.
Skip building spf_cred from HELO when it starts with '[', so
rspamd_spf_get_cred() returns NULL and the result is R_SPF_NA.
[Feature] Split attachment filenames into sub-tokens for Bayes classifier
Instead of a single whole-filename token, generate a structured set:
- #f: original filename (case-preserved, backward-compatible)
- #fe: last extension + double extension when present (e.g. #fe:gz, #fe:tar.gz)
- #fp: meaningful parts split by delimiters, CamelCase and letter/digit
boundaries; purely numeric and single-character parts discarded
Improves generalization for emails with similar attachment names differing
only in dates, IDs or other numeric suffixes, and provides a strong signal
for malware disguise patterns (e.g. pdf.exe, doc.vbs).
Vsevolod Stakhov [Sat, 28 Mar 2026 09:14:15 +0000 (09:14 +0000)]
[Fix] Default map URL path to "/" when no path component is present
When an HTTP(S) map URL has no path (e.g. "https://maps.rspamd.com"),
the URL parser leaves hdata->path as NULL. This causes a segfault in
write_http_request() which calls strlen() on the NULL path pointer.
Default path to "/" and rest to "" when UF_PATH is not set, matching
standard HTTP semantics.
Vsevolod Stakhov [Sat, 28 Mar 2026 09:07:19 +0000 (09:07 +0000)]
[Fix] Exclude injected parts from SA body/rawbody regexp scanning
PDF text extraction injects synthetic text parts that were being scanned
by sa_body and sa_raw_body rules, causing 30x false positive increase on
rules matching null bytes (e.g. /\x00/{sa_raw_body}). PDF hex strings
produce raw bytes including \x00 which are not meaningful in extracted text.
Two fixes:
- Strip null bytes and control characters from extracted PDF text in
sanitize_pdf_text() for the non-UTF-16 code path
- Skip RSPAMD_MIME_PART_COMPUTED parts in SA body and rawbody scanning
to follow original SA semantics where only real MIME text parts are matched
Vsevolod Stakhov [Fri, 27 Mar 2026 09:10:56 +0000 (09:10 +0000)]
[CritFix] Stop ev_io watcher in fuzzy UDP session destroy
fuzzy_session_destroy() freed the session without stopping a
potentially active ev_io watcher (pending UDP write retry on EAGAIN).
The watcher remained in libev's anfds[fd].head linked list pointing
to freed memory, corrupting the list and causing fd_reify() to loop
infinitely on the next fd_change โ making the worker consume 100% CPU
and stop processing queries entirely.
The TCP counterpart (fuzzy_tcp_session_destroy) already had the
correct ev_io_stop guard. Apply the same pattern for UDP sessions.
Vsevolod Stakhov [Thu, 26 Mar 2026 12:03:57 +0000 (12:03 +0000)]
[Fix] Handle OpenAI-compatible response format in Ollama GPT module
Ollama's /v1/chat/completions endpoint returns responses in OpenAI
format ({choices: [{message: {content: ...}}]}) rather than native
Ollama format ({message: {content: ...}}). The ollama conversion
functions only handled the native format, failing with "bad message
in reply" for users connecting via the OpenAI-compatible endpoint.
Add ollama_extract_content helper that handles both response formats.
Vsevolod Stakhov [Thu, 26 Mar 2026 11:51:04 +0000 (11:51 +0000)]
[Fix] Add text attachment fallback in get_displayed_text_part
When all text parts are classified as attachments (e.g. due to
Content-Type name= parameter), get_displayed_text_part returned nil
causing GPT module to skip with "no text part found". Add text/plain
attachment fallback (< 100KB) similar to existing HTML attachment
fallback.
Also propagate min_words setting consistently through the extraction
pipeline: gpt.lua -> llm_common.build_llm_input -> extract_text_limited
-> get_displayed_text_part, so all three call sites use the same
threshold.
Vsevolod Stakhov [Thu, 26 Mar 2026 10:23:06 +0000 (10:23 +0000)]
[Fix] Clear inherited address family flags when creating new SPF addr nodes
When spf_record_process_addr creates a new address node for subsequent
DNS replies (MX A/AAAA resolution), it copies the original addr via
memcpy, inheriting all flags including the address family flag from the
first reply. The new reply's flag was OR'd in without clearing the
inherited one, resulting in nodes with both IPV4 and IPV6 flags set.
Since spf_addr_mask_to_string checks IPV4 before IPV6, such nodes would
always display as IPv4, causing duplicate IPv4 addresses in output
instead of the correct IPv4 + IPv6 pair.
Vsevolod Stakhov [Wed, 25 Mar 2026 21:28:41 +0000 (21:28 +0000)]
[Fix] Load only the notified scope on per-scope hyperscan notifications
When hs_helper compiles scopes sequentially, it sends a per-scope
HS_LOADED notification after each scope finishes. Workers previously
ignored the scope field and attempted to reload ALL scopes on every
notification, causing spurious "no valid expressions" errors for scopes
that hadn't been compiled yet.
Now workers dispatch per-scope notifications to a new single-scope
loader (rspamd_re_cache_load_hyperscan_single_scope_async), and only
the final empty-scope notification triggers a full reload of all scopes.
Vsevolod Stakhov [Wed, 25 Mar 2026 19:48:35 +0000 (19:48 +0000)]
[Fix] Defer settings application for symbols registered after settings init
When regexp_rules maps (or other dynamic sources) register symbols after
settings have already been processed, the symbols_enabled/symbols_disabled
lists were silently ignored. Store pending settings operations and apply
them when the symbol is later registered via add_symbol_with_callback or
add_virtual_symbol.
Vsevolod Stakhov [Wed, 25 Mar 2026 16:18:41 +0000 (16:18 +0000)]
[Fix] Only request recompile on deserialize failure, not cache miss
Avoid infinite recompile loop: only send RSPAMD_SRV_RECOMPILE_REQUEST
when a cached blob exists but fails to deserialize (stale/corrupt),
not on cache misses where hs_helper cannot compile certain expressions.
Vsevolod Stakhov [Wed, 25 Mar 2026 11:35:58 +0000 (11:35 +0000)]
[Fix] Self-healing hyperscan cache: delete stale blobs and trigger recompile
When workers fail to deserialize a cached hyperscan blob (version
mismatch, corruption, etc.), the system previously got stuck in PCRE
fallback mode permanently because hs_helper's exists check would pass
for the stale entry and skip recompilation.
Now workers delete bad cache entries immediately on deserialize failure
and send RSPAMD_SRV_RECOMPILE_REQUEST to trigger hs_helper to recompile.
A 5-second debounce in hs_helper prevents thrashing from multiple workers
detecting the same bad blobs simultaneously.
Also removes the dead needs_recompile flag which was set on the worker's
re_class but checked during compilation in hs_helper (different process).
Vsevolod Stakhov [Wed, 25 Mar 2026 08:32:38 +0000 (08:32 +0000)]
[Fix] Backport fixes from libucl
- ucl_msgpack: add bounds check after switch in ucl_msgpack_parse_ignore
to prevent heap-buffer-overread when len is modified by fixext types
- ucl_sexp: fix broken length parsing, correct bounds check direction,
add null pointer checks, fix memory leaks in error paths
Vsevolod Stakhov [Tue, 24 Mar 2026 08:44:35 +0000 (08:44 +0000)]
[Feature] Add worker:get_mem_config() Lua method for jemalloc stats
Exposes structured jemalloc information via mallctl() as a nested table:
- stats: allocated, active, metadata, resident, mapped
- config: narenas, dirty_decay_ms, muzzy_decay_ms, tcache,
background_thread, malloc_conf
- version: jemalloc version string
Returns nil when jemalloc is not compiled in.
Vsevolod Stakhov [Sat, 21 Mar 2026 14:37:53 +0000 (14:37 +0000)]
[Minor] Tune jemalloc for single-threaded multi-process architecture
Set narenas:1 (one arena sufficient without threads), dirty_decay_ms:5000
(return dirty pages faster than default 10s), and muzzy_decay_ms:30000
(hold lazy-release pages for 30s before returning to OS).
Vsevolod Stakhov [Sat, 21 Mar 2026 13:53:12 +0000 (13:53 +0000)]
[Fix] Fix use-after-free in pending regexp map and multipattern queues
When a map reloads while awaiting async hyperscan compilation, the old
re_map/mp is destroyed but rspamd_regexp_map_add_pending (and its
multipattern counterpart) blindly appended a new entry without removing
the stale one. The linear search in find_pending then returned the first
(dangling) pointer, causing a heap-buffer-overflow in rspamd_snprintf
when accessing re_map->re_digest.
Fix by replacing existing entries with the same name in-place instead of
appending duplicates.
Vsevolod Stakhov [Sat, 21 Mar 2026 13:40:21 +0000 (13:40 +0000)]
[Minor] Optimize fuzzy_tcp_refresh_timeout
Use ev_timer_again instead of stop/set/start triple, reducing three
libev calls to one. Move timeout refresh out of the write loop so it
is called once after draining the queue rather than on every write().
Vsevolod Stakhov [Sat, 21 Mar 2026 13:36:18 +0000 (13:36 +0000)]
[Fix] Fix CPU busy-loop in fuzzy TCP client due to EV_WRITE not being cleared
When the TCP write queue drained, EV_WRITE was left armed on the socket
watcher. Since a connected TCP socket is always writable, libev fired
EV_WRITE continuously causing 100% CPU usage. The UDP path correctly
switched to EV_READ only after writing.
- Drop EV_WRITE when write queue empties in fuzzy_tcp_write_handler
- Only arm EV_WRITE on connection established if data is actually queued
- Remove redundant per-IO fuzzy_tcp_check_pending_timeouts call (timer
callback already handles this), which amplified the spin with O(N)
hash scans and GHashTable allocations on every iteration
Vsevolod Stakhov [Sat, 21 Mar 2026 11:15:22 +0000 (11:15 +0000)]
[Feature] Expose milter headers and extended symbols in legacy RSPAMC/SPAMC protocol
Previously, milter add/remove headers and symbol options were only
available via the JSON (HTTP) protocol. Legacy RSPAMC clients like
Exim had no way to access them through $spam_report.
This adds three new line types to the legacy text protocol output:
- X-Milter-Add: Header: value โ milter headers to add (with optional
[N] position bracket for insert-at-position)
- X-Milter-Del: Header โ milter headers to remove (with optional [N]
for specific instance removal)
- X-Symbol: Name(score); description [opt1, opt2] โ extended symbol
info with descriptions and options
Existing Symbol: lines are preserved for backward compatibility.
rspamd_stat_check_autolearn emits error log lines sharing the
same function prefix as its success messages. Narrow re_confirmed
to match only success lines, which start with <MSG-ID>: autolearn.
Vsevolod Stakhov [Wed, 18 Mar 2026 15:00:54 +0000 (15:00 +0000)]
[Fix] Weighted round-robin not respecting upstream weights across cycles
When all cur_weights reached zero after one complete weighted cycle,
the code fell through to the min_checked path which selects the
least-used upstream regardless of configured weights. This caused
effectively equal distribution (1:1:1) instead of the configured
ratio (e.g. 100:100:1).
Fix: when all cur_weights are exhausted and upstreams have configured
weights, reset all cur_weights simultaneously to restart the weighted
cycle. The min_checked fallback is now only used when all original
weights are truly zero.
Dmitriy Alekseev [Tue, 17 Mar 2026 10:43:51 +0000 (11:43 +0100)]
[Fix] Register redis and http as known hs_helper worker options
The redis and http configuration blocks in the hs_helper worker section
were not registered via rspamd_rcl_register_worker_option, causing
rspamadm configdump to emit "unknown worker attribute: redis" warnings.
The Lua backend reads these blocks at runtime through the full UCL
options object, so they worked correctly despite not being registered.
Add proper RCL registration for both redis and http as ucl_object_t
fields so the config schema recognizes them as valid worker attributes.
Dmitriy Alekseev [Mon, 16 Mar 2026 12:17:07 +0000 (13:17 +0100)]
feat(lupa): add validation filters, UCL-aware tests, and comprehensive unit tests
New filters:
- mandatory(msg): error if nil or empty
- require_int(msg): error if not a valid integer
- require_number(msg): error if not a valid number
- require_bool(msg): error if not a UCL boolean (true/false/yes/no/on/off/1/0)
- require_duration(msg): parse duration string to seconds, error if invalid
(supports s, ms, min, m, h, d, w, y)
- require_json(msg): error if not valid JSON/UCL
- require_size(msg): error if not a valid size (number with optional b/Kb/Mb/Gb)
- fromjson: parse JSON/UCL string into Lua table
- tobytes: convert size string to bytes (1Kb=1024, 1Mb=1048576, 1Gb=1073741824)
Modified tests to handle string inputs (env vars are always strings):
- is_number: now returns true for numeric strings like "42" or "3.14"
- is_integer: now returns true for integer strings like "42"
- is_float: now returns true for float strings like "3.14"
- is_true: now checks UCL truthy values (true/yes/on/1, case-insensitive)
- is_false: now checks UCL falsy values (false/no/off/0, case-insensitive)
New tests:
- is_json: check if value is valid JSON/UCL
- is_size: check if value is a valid size string
Added test/lua/unit/lupa.lua with 90+ test cases covering all filters,
tests, env var patterns, and real-world configuration scenarios.
Vsevolod Stakhov [Mon, 16 Mar 2026 12:19:23 +0000 (12:19 +0000)]
[Feature] Store GPT result in mempool variable
Store full GPT classification result (probability, reason,
categories, model) as JSON in task mempool variable gpt_result
for use by downstream postfilter plugins.
Vsevolod Stakhov [Sun, 15 Mar 2026 12:26:17 +0000 (12:26 +0000)]
[Fix] Add PCRE2 complexity checks before JIT compilation
Check compiled pattern size, frame size, and capture count
before calling pcre2_jit_compile to avoid crashes on
pathological patterns. Also set map->map pointer consistently
in lua_config_add_map for all map types.
Vsevolod Stakhov [Sat, 14 Mar 2026 22:22:28 +0000 (22:22 +0000)]
[Fix] Fix Clickhouse column name mismatch: UUID -> TaskUUID
The insert field list used 'UUID' but the actual column is named
'TaskUUID' (as defined in schema and migration 9), causing
NO_SUCH_COLUMN_IN_TABLE errors on insert.
Vsevolod Stakhov [Sat, 14 Mar 2026 22:13:42 +0000 (22:13 +0000)]
[Feature] Add context_augment hook to GPT module
Add a new `context_augment` configuration option that accepts Lua code
returning an async function(task, content, callback). The callback
receives a string that gets injected as additional context into the
LLM prompt alongside existing user/domain and search contexts.
This enables external Lua code to enrich LLM requests with arbitrary
context โ e.g., Telegram channel topic and recent messages for
community spam detection.
The augment function runs in parallel with other context fetchers
and supports async operations (Redis, HTTP).
Vsevolod Stakhov [Sat, 14 Mar 2026 20:31:15 +0000 (20:31 +0000)]
[Fix] Fix rspamc neural_learn config fetch and output
- Fix config fetch port: when user specifies the default scan port
(11333), redirect the /plugins/neural/config preflight request to
the controller port (11334) where the endpoint actually lives.
Previously, the config fetch used the user-specified port verbatim,
causing "invalid command" errors on the normal worker.
- Fix misleading output: rspamc_neural_learn_output no longer defaults
to "success = true" when the response lacks an explicit success field.
For scan-based learning (checkv2 path), it now detects the scan
response and shows method = "scan". For missing/error responses,
it correctly reports success = false.
- Add user-visible note when the config fetch fails, explaining the
fallback to scan-based learning.
Vsevolod Stakhov [Fri, 13 Mar 2026 13:58:33 +0000 (13:58 +0000)]
[Test] Add unit tests for caseless table
25 tests covering creation, case-insensitive lookup, key case
preservation, assignment, deletion, has_key, iteration via each(),
to_table conversion, multi-value get_all, and edge cases including
long keys, empty keys, and metamethod isolation.
Vsevolod Stakhov [Fri, 13 Mar 2026 13:54:35 +0000 (13:54 +0000)]
[Feature] Add case-insensitive table type for HTTP headers
Introduce rspamd{caseless_table} userdata type that provides
case-insensitive key lookup while preserving original key case.
HTTP response headers now use this type instead of plain Lua tables,
fixing two issues: headers were forcibly lowercased (mutating original
data) and duplicate headers were silently lost.
Multi-value headers are stored as arrays and can be retrieved via
get_all(key). The __index metamethod returns the first value for
convenience. The type is generic and reusable beyond HTTP.
Vsevolod Stakhov [Fri, 13 Mar 2026 12:20:09 +0000 (12:20 +0000)]
[Minor] Improve restore logging and add incremental GC
Add progress reporting every 10 seconds with throughput rate and
Lua memory usage. Run incremental GC after each batch flush and
full GC between files to avoid memory spikes on large restores.
Vsevolod Stakhov [Thu, 12 Mar 2026 12:20:31 +0000 (12:20 +0000)]
[Fix] Add reconnection with retry logic for statistics restore
When restoring large datasets (24M+ lines), Redis connections can get
terminated mid-restore. Add retry logic with automatic reconnection
that resumes from the failed pipeline chunk, avoiding HINCRBYFLOAT
double-counting by skipping past ambiguously-applied chunks.
Vsevolod Stakhov [Tue, 10 Mar 2026 17:35:51 +0000 (17:35 +0000)]
[Fix] Fix external neural model merge defects
- merge_weights returns boolean, not ANN object: use ext_ann directly
- Add missing digest/symbols/distance fields for external-only set.ann
- Fix inverted alpha in merge call (alpha meant external weight, not local)
- Add missing newline at EOF in lua_kann.c
The rspamd_fuzzy_tcp_frame payload was sized for v1 encrypted replies
(136 bytes) but v2 encrypted replies are 184 bytes, causing a buffer
overflow when sending multi-flag responses over TCP with encryption.
Use a union to accommodate both v1 and v2 reply sizes.
Add multi-flag delete tests for all TCP transport modes.
Rob4226 [Tue, 10 Mar 2026 09:04:36 +0000 (05:04 -0400)]
[Minor] Fix default date description in rspamadm dmarc_report help
Change the help text for the `date` argument of `rspamadm dmarc_report`
from "today" to "yesterday". When the command is run without specifying
a date, it actually processes reports for yesterday, so this update
makes the help message match the command's behavior.
Jess Sullivan [Thu, 5 Mar 2026 17:42:18 +0000 (12:42 -0500)]
[Fix] Prevent SIGSEGV when sign_headers is not a string
Add NULL guard in rspamd_create_dkim_sign_context() before strlen(headers)
to return a proper GError instead of crashing when headers is NULL.
Add type validation at config load time for sign_headers in dkim_signing,
dkim, and arc modules โ rejects non-string types with a clear error message.
Fixes: SIGSEGV in dkim.c when sign_headers is configured as a UCL array
(ucl_object_tostring returns NULL for arrays, which hits strlen).
Limit scheduled integration-test runs to rspamd/rspamd while keeping
manual start available in forks. This avoids unnecessary fork cron
runs and reduces noisy CI failures unrelated to upstream.
[Fix] Skip recipient check when no hash found in Redis
When a key is not found in Redis, lua_redis returns a redis.null
userdata (not nil), which is truthy and caused check_recipient()
to be called unconditionally, logging a misleading "no recipients
are matching hash" message despite no hash being stored.
[Feature] Add external pretrained neural model support
This commit adds the ability to load pretrained neural network models
from external sources (HTTP/HTTPS) and merge them with locally trained
weights. Users can receive a pretrained model and fine-tune it with
their own data.
Model format (msgpack with magic "RNM1"):
- magic: format identifier
- version: format version (currently 1)
- model_version: model training version
- providers_digest: must match local providers config
- ann_data: serialized KANN (zstd compressed)
- pca_data: optional PCA matrix
- norm_stats, roc_thresholds: optional metadata
Key changes:
- lualib/lua_neural_external.lua: new module for external model handling
- Model parsing, KANN loading, weight merging via interpolation
- Map-based loading with signature verification support
- Base model storage in Redis for future re-merge
- src/lua/lua_kann.c: Lua bindings for merge_weights and is_compatible
- Neural plugin integration:
- Register external model as callback map at config time
- Apply loaded model to all settings elements
- Automatic update checking via map infrastructure
[Fix] Preserve content flags for injected query URLs
Propagate the parent URL flags when task:inject_url() extracts nested query URLs. This keeps the content flag on URLs injected from computed parts such as PDF text, so follow-up query URLs are classified the same way as the outer injected URL.
Do not suppress URLs from mime_part:get_urls() when the same URL was already seen in another MIME part. This restores per-part URL visibility for multipart/alternative messages and keeps text/plain URLs available even when text/html contains the same links.