Vsevolod Stakhov [Thu, 29 Jan 2026 13:26:27 +0000 (13:26 +0000)]
[Feature] fuzzy_storage: enhance blacklist handler with richer context
Pass extended information to Lua blacklist handlers:
- event_type: distinguish "new", "existing", or "blacklist" events
- ratelimit_info: bucket state (level, burst, rate, exceeded_by)
- digest: hash when session context is available
- extensions: domain and source IP from fuzzy extensions
Backwards compatible - existing handlers still receive ip and reason
as first two arguments. New arguments are optional for Lua handlers.
Optimized with early-exit checks when no handlers are registered.
Vsevolod Stakhov [Thu, 29 Jan 2026 10:41:15 +0000 (10:41 +0000)]
[Fix] cryptobox: bypass RHEL/CentOS 10 crypto-policies for SHA-1 DKIM verification
RHEL/CentOS 10+ crypto-policies disable SHA-1 for signatures by default,
causing rsa-sha1 DKIM verification to fail. This is problematic as many
legitimate emails still use rsa-sha1 DKIM signatures.
For OpenSSL 3.0+, create a dedicated OSSL_LIB_CTX that bypasses system
crypto-policies specifically for SHA-1 DKIM signature verification.
SHA-256/SHA-512 verifications continue using the normal system context.
The legacy context is lazily initialized only when SHA-1 verification
is needed, avoiding overhead for modern rsa-sha256 signatures.
Vsevolod Stakhov [Wed, 28 Jan 2026 18:44:18 +0000 (18:44 +0000)]
[Fix] re_cache: detect and handle stale hyperscan files with mismatched regexp IDs
When the re_class hash doesn't include global regexp indices, regexps can
be reordered while the class hash stays the same. This causes old hyperscan
files to be loaded with stale position IDs that point to wrong regexps
(possibly in different re_classes with no hyperscan loaded), leading to
assertion failures when hs_scratch is NULL.
Fix:
- Reset match_type to PCRE during cleanup before freeing hs_ids, preventing
stale HYPERSCAN flags if reload fails
- Validate that each stored ID points to a regexp belonging to the current
re_class before setting match_type
- Delete stale hyperscan files to trigger recompilation by hs_helper
- For Redis cache, stale entries will expire or be overwritten
Both file-based and Redis-based loading paths are fixed.
Vsevolod Stakhov [Wed, 28 Jan 2026 12:51:15 +0000 (12:51 +0000)]
[Fix] Fix EVP_PKEY_CTX memory leak in DKIM RSA signing
The EVP_PKEY_CTX allocated in rspamd_dkim_sign() for RSA key signing
was never freed, causing continuous memory growth when using DKIM/ARC
signing with RSA keys.
Add EVP_PKEY_CTX_free() calls in all error paths and after successful
signing to properly release the OpenSSL context.
Vsevolod Stakhov [Wed, 28 Jan 2026 10:49:10 +0000 (10:49 +0000)]
[Fix] hs_helper: defer next multipattern compilation to prevent crash after Redis timeout
When a Redis timeout occurs during HS cache save operation, the error callback
immediately starts processing the next multipattern while still inside the
timeout handler. This leaves the connection in an inconsistent state, causing
a crash (SIGSEGV in hs_compile_multi) when compiling the next multipattern.
Fix by deferring the next operation to the next event loop iteration using
ev_timer with 0 timeout. This ensures the error handling completes fully
before starting the next operation.
Vsevolod Stakhov [Wed, 28 Jan 2026 09:05:25 +0000 (09:05 +0000)]
[Fix] Redis hyperscan cache: use write_servers for store/delete operations
The redis_backend was missing is_write=true in attrs for store and delete
operations, causing lua_redis.request to use read_servers instead of
write_servers. This resulted in READONLY errors when read and write servers
are configured separately.
Vsevolod Stakhov [Wed, 28 Jan 2026 08:56:28 +0000 (08:56 +0000)]
[Feature] Replace builtin_suspicious TLDs with map-based configuration
Convert hardcoded suspicious TLDs list to a proper map file following
rspamd's standard map loading pattern with fallback support.
Changes:
- Add conf/maps.d/suspicious_tlds.inc with default TLDs (.tk, .ml, .ga, .cf, .gq)
- Update url_suspect.conf to use fallback+file:// pattern for user overrides
- Update url_suspect.lua to load TLDs via rspamd_map_add_from_ucl()
Users can now:
- Override entirely: create local.d/maps.d/suspicious_tlds.inc
- Extend defaults: create local.d/maps.d/suspicious_tlds.inc.local
- Disable: set suspicious_tlds_map = null in local.d/url_suspect.conf
Supersedes #5864 - the map-based approach inherently handles nil/missing
config gracefully, making the type check unnecessary.
Vsevolod Stakhov [Mon, 26 Jan 2026 17:48:55 +0000 (17:48 +0000)]
[Fix] Simplify parse_sa_regexp by delegating to rspamd_regexp.create
Remove unnecessary flag parsing and inline modifier transform that
caused issues with invalid PCRE flags (g, u). rspamd_regexp.create
already handles /pattern/flags and m{pattern}flags formats natively.
Vsevolod Stakhov [Mon, 26 Jan 2026 16:40:19 +0000 (16:40 +0000)]
[Fix] GPT plugin: explicitly set POST method for API requests
Fixes #5859
Some API providers (like Ollama) strictly require POST method on their
endpoints and return 405 Method Not Allowed for GET requests. While
rspamd_http auto-detects POST when a body is present, explicitly setting
the method ensures correct behavior in all cases.
Vsevolod Stakhov [Mon, 26 Jan 2026 16:30:18 +0000 (16:30 +0000)]
[Fix] Clear pending multipatterns on config reload to prevent use-after-free
After SIGHUP reload, the global pending_compilations queue retained
stale multipattern pointers from the freed old config. When hs_helper
processed the queue, it accessed freed memory causing heap-buffer-overflow
in rspamd_multipattern_get_npatterns().
Add rspamd_multipattern_clear_pending() alongside the existing
rspamd_regexp_map_clear_pending() call before releasing old config.
Vsevolod Stakhov [Thu, 22 Jan 2026 19:48:30 +0000 (19:48 +0000)]
[Fix] Use versioned key for hybrid LLM+symbols manual training
Pending key is now only used for LLM-only mode where embedding
dimensions may vary. Hybrid (LLM+symbols) and symbols-only modes
use versioned key directly since dimension includes stable symbols.
Vsevolod Stakhov [Thu, 22 Jan 2026 18:36:26 +0000 (18:36 +0000)]
[Fix] Use versioned key for manual training in symbols-only mode
Manual training via ANN-Train header now writes to versioned key when
no LLM provider is configured. The pending key is only used with LLM
providers where embedding dimensions may vary between versions.
Vsevolod Stakhov [Thu, 22 Jan 2026 15:35:09 +0000 (15:35 +0000)]
[Fix] Match fuzzy_check.c hash generation in text_part:get_fuzzy_hashes
Fix text_part:get_fuzzy_hashes() to produce identical hashes as the
fuzzy_check plugin's fuzzy_cmd_from_text_part():
- For short text (<32 words): hash utf_stripped_content directly instead
of individual words, and optionally include subject
- For normal text: skip words with RSPAMD_WORD_FLAG_SKIPPED flag or
empty stems
Add optional subject parameter to include in short text hash calculation
(matches fuzzy_check.c behavior with no_subject=false).
Update rspamadm mime stat to pass subject to get_fuzzy_hashes().
Vsevolod Stakhov [Thu, 22 Jan 2026 11:51:53 +0000 (11:51 +0000)]
[Feature] Put subject first in LLM embedding input
Subject is highly valuable for spam detection and placing it first
ensures it's always included even if text content gets truncated
by model token limits.
Vsevolod Stakhov [Thu, 22 Jan 2026 11:21:52 +0000 (11:21 +0000)]
[Feature] Rename neural autolearn options to match RBL module naming
Rename check_local/check_authed to exclude_local/exclude_users for
consistency with RBL module conventions. Change exclude_users default
to true (authenticated users excluded by default).
Vsevolod Stakhov [Wed, 21 Jan 2026 09:57:54 +0000 (09:57 +0000)]
[Feature] Include content URLs by default in URL API calls
- Add `include_content_urls` global option (default: true) to control
whether URLs extracted from content (PDF, etc.) are included in API calls
- Update task:get_urls(), task:get_emails() to include content URLs by default
- Update lua_util.extract_specific_urls() to use config default when
need_content is not explicitly specified
- Mark URLs extracted from computed/virtual parts (PDF text) with CONTENT
flag instead of FROM_TEXT flag, since they may be clickable links
- Add commented documentation in conf/options.inc
Users who want the old behavior can set `include_content_urls = false`
in their options configuration.
Vsevolod Stakhov [Wed, 21 Jan 2026 08:57:13 +0000 (08:57 +0000)]
[Feature] Add order-independent table digest using XXH3 XOR accumulation
Add rspamd_cryptobox.fast_hash64() C function that returns XXH3 hash as
two 32-bit integers, enabling XOR accumulation for order-independent
hashing in Lua.
Add lua_util.unordered_table_digest() that produces consistent digests
regardless of table iteration order. This fixes issues where different
Rspamd instances produced different ANN digests for identical configs
due to non-deterministic key ordering in pairs().
The original table_digest had two bugs:
- Used pairs() which iterates in undefined order across Lua VMs
- Ignored numeric and boolean values in the hash
Update neural plugin's providers_config_digest to use the new function,
fixing the "providers config changed" warnings on identical configs.
Also update lua_maps and lua_urls_compose cache key generation to use
unordered_table_digest for more reliable cache hits.
Vsevolod Stakhov [Tue, 20 Jan 2026 21:41:15 +0000 (21:41 +0000)]
[Fix] Clear pending regexp maps on config reload to prevent use-after-free
During HUP-triggered config reload, the pending_regexp_maps array retained
pointers to re_map objects from the old config after they were freed. When
workers received "regexp map loaded" notifications, they accessed freed memory
(visible as 0x5A poison pattern in re_digest), causing SIGSEGV.
Fix by calling rspamd_regexp_map_clear_pending() before releasing the old
config in reread_config().
Vsevolod Stakhov [Tue, 20 Jan 2026 16:54:43 +0000 (16:54 +0000)]
[Fix] Fix race condition between I/O handler and SIGCHLD in subprocess
The subprocess callback could crash when SIGCHLD handler ran concurrently
with the I/O handler processing large training results. The race:
1. I/O handler receives full data, calls callback
2. SIGCHLD fires during callback execution
3. SIGCHLD handler frees cbdata while callback still uses it
4. Callback returns, I/O handler accesses freed memory -> crash
Fix:
- Add 'dead' flag to track when child has exited
- Set 'replied' BEFORE calling callback (not after)
- SIGCHLD handler skips cleanup if replied=TRUE (I/O handler owns it)
- I/O handler does cleanup after callback if dead=TRUE
- Extract cleanup into rspamd_lua_cbdata_free() helper
Vsevolod Stakhov [Tue, 20 Jan 2026 16:13:21 +0000 (16:13 +0000)]
[Fix] Use rspamd_text for subprocess callback data to avoid large allocations
Replace lua_pushlstring with lua_new_text(FALSE) when passing subprocess
result data to Lua callbacks. This avoids copying potentially large buffers
(e.g., 2.7MB neural network training results) into Lua's heap, which could
cause crashes under memory pressure.
Vsevolod Stakhov [Tue, 20 Jan 2026 16:12:41 +0000 (16:12 +0000)]
[Fix] Fix ROC threshold calculation for ham/spam labels
The ROC calculation was checking outputs[i][1] == 0 for ham samples,
but the ceb_neg cost function uses -1.0 for ham and 1.0 for spam.
Changed to check outputs[i][1] < 0 to correctly identify ham samples.
Vsevolod Stakhov [Tue, 20 Jan 2026 14:25:29 +0000 (14:25 +0000)]
[Feature] Multi-layer funnel architecture for LLM embeddings
Add improved neural network architecture specifically for LLM embedding
inputs, while preserving backward compatibility for symbol-based rules.
Key changes:
- New create_embedding_ann() with multi-layer funnel architecture
- Auto-detection of LLM providers via uses_llm_embeddings()
- Support for configurable layers, dropout, layer normalization
- GELU activation by default when available (falls back to ReLU)
- Layer size auto-scaling based on input dimension:
- >512 dims: 3 layers (0.5, 0.25, 0.125)
- 256-512 dims: 2 layers (0.5, 0.25)
- <256 dims: 1 layer (0.5)
Bug fixes:
- Wrap create_ann in pcall to handle errors gracefully
- Reset learning_spawned flag on ANN creation failure
- Replace assert(false) with proper error logging that resets state
- Prevents training from getting stuck after errors
New configuration options:
- layers: explicit layer size multipliers
- dropout: dropout rate (default 0.2 for embeddings)
- use_layernorm: enable layer normalization (default true)
- activation: 'gelu' or 'relu' (default 'gelu' if available)
Vsevolod Stakhov [Tue, 20 Jan 2026 12:16:36 +0000 (12:16 +0000)]
Add GPU and vast.ai support for neural embedding service
- Add Dockerfile.gpu for GPU-accelerated inference with PyTorch CUDA
- Add requirements-gpu.txt with pinned versions for CUDA compatibility
- Add vastai-launch.sh script for deploying on vast.ai cloud GPUs
- Update README with GPU deployment instructions and model recommendations
Default GPU model: intfloat/multilingual-e5-large (100+ languages including Russian)
Tested on RTX 4090 with ~20-50ms latency per embedding.
Vsevolod Stakhov [Tue, 20 Jan 2026 11:09:47 +0000 (11:09 +0000)]
[Fix] Prefer higher version ANN profiles when symbol distances are equal
When multiple ANN profiles have the same symbol distance, the profile
selection would pick the first one encountered rather than the newest.
This caused issues when a newly trained ANN (version 1) existed alongside
the initial profile (version 0) - the scanner would select version 0
which had no actual ANN data.
Fix by adding a secondary selection criterion: when distances are equal,
prefer the profile with the higher version number.
Vsevolod Stakhov [Mon, 19 Jan 2026 19:14:11 +0000 (19:14 +0000)]
[Fix] Fix messagepack cache decoding format string
The UCL parser's parse_text() only accepts 'msgpack' as the format
string for messagepack parsing, while to_format() accepts both
'msgpack' and 'messagepack'. This mismatch caused cached data to
fail decoding and appear as cache misses.
Fixes LLM embedding cache never being read back despite being stored.
Vsevolod Stakhov [Mon, 19 Jan 2026 18:41:02 +0000 (18:41 +0000)]
[Fix] Prevent concurrent neural network training races
- Add learning_spawned check at start of do_train_ann to prevent
concurrent async Redis operations
- Move learning_spawned flag to start of spawn_train for earlier
lock acquisition
- Remove redundant flag assignments later in the training flow
Vsevolod Stakhov [Mon, 19 Jan 2026 18:08:58 +0000 (18:08 +0000)]
[Feature] Add pending training keys and fix neural network training issues
- Add pending_train_key() for version-independent training vector storage
- Fix variable shadowing bug where ann_trained callback was overwritten
- Add concurrent training prevention via learning_spawned check
- Replace assert with proper error handling for msgpack parsing
- Clean up pending keys after successful training
- Update controller endpoint to use pending keys for manual training
- Fix ev_base:sleep() to register with session events properly
- Update classifier_test.lua to support llm_embeddings classifier testing
Vsevolod Stakhov [Mon, 19 Jan 2026 15:32:43 +0000 (15:32 +0000)]
[Feature] Add ev_base:sleep() method for Lua
Add sleep method to ev_base that supports both sync and async modes:
- ev_base:sleep(time) - sync mode using coroutines
- ev_base:sleep(time, callback) - async mode with callback
Sync mode yields the current coroutine and resumes after timeout.
Async mode schedules the callback to run after the timeout.
Vsevolod Stakhov [Mon, 19 Jan 2026 14:41:30 +0000 (14:41 +0000)]
[Fix] Skip external map queries when Settings header is provided
When settings are specified manually via the Settings HTTP header,
external map queries should not be executed as they may override
the manually provided settings asynchronously.
This prevents connection errors to external maps from affecting
requests that explicitly provide their own settings.
[Minor] WebUI: Add frequency stddev column and units to symbols table
- Add frequency standard deviation column with the same exponential
scaling as frequency for consistent notation. Hidden on smaller
screens (lg breakpoint)
- Display units (hits/s for frequencies, s for time) in table headers
- Remove "s" suffix from time cells (unit now in header)
[Fix] Calculate frequency exponent from non-zero values only
Fix suboptimal exponential notation selection in WebUI symbols
frequency display. Previously, the exponent was calculated from
the average of all frequency values including zeros, resulting
in unnecessarily small exponents (e.g., 2300.00e-8 instead of
2.30e-5). Now only non-zero values are used for calculation,
producing more readable notation.
Vsevolod Stakhov [Sat, 17 Jan 2026 15:58:14 +0000 (15:58 +0000)]
[Feature] Add extract_text_limited for email text extraction with limits
Add lua_mime.extract_text_limited() function to extract meaningful text from
emails with long reply chains while respecting size limits.
Features:
- max_bytes: Hard limit on output size (default: 32KB)
- max_words: Alternative limit by word count
- strip_quotes: Remove quoted replies (lines starting with >)
- strip_reply_headers: Remove reply headers (On X wrote:, From: Sent:)
- strip_signatures: Remove signature blocks (-- separator, mobile signatures)
- smart_trim: Enable all heuristics
Implementation:
- Uses rspamd_text:lines() iterator for memory-efficient line processing
- No full string interning of email content (better for large emails)
- rspamd_trie for multi-pattern matching (67 signature, 44 reply patterns)
- rspamd_regexp for regex patterns (wrote:, schrieb:, etc.)
- Single-pass O(n) algorithm with early termination
Multilingual support for 10+ languages:
- English, German, French, Spanish, Russian, Portuguese, Italian
- Chinese, Japanese, Polish
Configuration API:
- lua_mime.configure_text_extraction(cfg) for custom patterns
- Supports extend_defaults to add patterns without replacing defaults
[Minor] Unify sortValue functions to arrow functions
Convert all sortValue functions in FooTable column definitions to
arrow functions with consistent parameter naming for consistency
across the codebase.
Vsevolod Stakhov [Wed, 14 Jan 2026 22:48:06 +0000 (22:48 +0000)]
[Fix] Propagate control request ids in replies
Ensure workers include cmd->id in control replies to avoid 'unknown request id 0' warnings. Update functional control tests and make RSPAMD_TMPDIR visible to child suites.
[Fix] Use proper rounding for symbol frequency statistics
- Replace incorrect floor() with round() in rounding functions to avoid
losing small values
- Increase counters API frequency precision from 3 to 6 decimal places
(need 5 to avoid rspamc displaying values as multiples of 0.06, need 6
for /counters endpoint itself - no additional overhead as JSON stores
double anyway)
- Add frequency_stddev field to counters API output (fixes zero stdev in
`rspamc counters` output)
- Clarify `rspamc counters` table header with "avg (stddev)" subheading
- Fix WebUI to preserve frequency precision before scaling
Example for symbol with frequency 0.004772 hits/sec:
- Before: /symbols returns 0.004772, /counters returns 0.004000,
`rspamc counters` shows 0.240
- After: /symbols returns 0.004772, /counters returns 0.004772,
`rspamc counters` shows 0.286
Vsevolod Stakhov [Wed, 14 Jan 2026 14:29:32 +0000 (14:29 +0000)]
[Feature] Route all hyperscan cache operations through Lua backend
- Route file backend through Lua for consistency with redis/http
- Add zstd compression support with magic byte detection for backward
compatibility (reads both .hs and .hs.zst files)
- Fix rspamd_util.stat() return value handling (returns err, stat tuple)
- Fix timer management for synchronous Lua callbacks to prevent early
termination of re_cache compilation
- Fix use-after-free in load path by pre-counting pending items
- Add priority queue for re_cache compilation (short lists first)
- Add ev_run() flush before blocking hyperscan compilations to ensure
busy notifications are sent
- Add hyperscan_notice_known() and hyperscan_get_platform_id() Lua APIs
Vsevolod Stakhov [Tue, 13 Jan 2026 21:52:13 +0000 (21:52 +0000)]
[Fix] Refactor control socket to use ID-based request/reply matching
Replace the serialization-based control command handling with an ID-based
approach using khash, mirroring the existing rspamd_srv_requests pattern.
Key changes:
- Add uint64_t id field to control command/reply structs
- Use khash for O(1) request lookup by ID instead of GHashTable
- Add rspamd_control_reply_handler() for centralized reply processing
- Add rspamd_control_pending_new/destroy/remove_all() API functions
- Add control_ev watcher to worker struct for reply monitoring
- Call rspamd_srv_pipe_cleanup() on worker shutdown to prevent leaks
- Handle ID collisions gracefully (warn and free old entry)
This fixes hash table iterator corruption crashes that occurred when
modifying the hash during iteration, and provides more robust concurrent
command handling.
Brett Neumeier [Tue, 13 Jan 2026 15:03:47 +0000 (09:03 -0600)]
Allow for use of Lua 5.5
In Lua 5.5, the signature for lua_newstate changes
(there is now an additional "seed" argument),
and the mechanism for adjusting garbage collection also changes.
I don't know whether the seed of 0 is ideal when using lua_newstate;
probably, using a random seed would be better. This is a minimal patch
that gets back to a working build.
Vsevolod Stakhov [Tue, 13 Jan 2026 14:19:53 +0000 (14:19 +0000)]
[Feature] Disable periodic recompile timer for file cache backend
The periodic recompile timer (default 60s) is only useful for shared
backends (Redis, HTTP, Lua) where another rspamd instance might have
compiled new hyperscan databases.
For file backend, recompilation is already triggered by:
- Config reload (forks new hs_helper process)
- Explicit RECOMPILE command (sent on map updates)
This eliminates unnecessary periodic checks for file-based deployments.
Vsevolod Stakhov [Tue, 13 Jan 2026 13:56:27 +0000 (13:56 +0000)]
[Fix] Fix re_cache hyperscan file tracking and buffer size
Two fixes for hyperscan cache file handling:
1. Increase hyperscan_cache_file.filename buffer from 64 to 80 bytes
to accommodate full filenames (64 hex hash + ".hs.unser" = 73 chars)
2. Add rspamd_hyperscan_notice_known() call in re_cache.c after loading
hyperscan databases. Without this, re_cache files weren't registered
as "known" and would be deleted by cleanup_maybe() on restart,
causing unnecessary recompilation.
Vsevolod Stakhov [Tue, 13 Jan 2026 11:02:09 +0000 (11:02 +0000)]
[Fix] Fix fd leaks and double-free in srv_pipe error handling
- Close attached_fd before freeing request data when sendmsg fails
- Fix double-free in rspamd_srv_pipe_ctx_destroy: items in send_queue
are also in the hash table, so only iterate hash to free
- Close attached_fd for unsent requests during shutdown
Vsevolod Stakhov [Tue, 13 Jan 2026 10:44:42 +0000 (10:44 +0000)]
[Fix] Reduce hyperscan_cache_file command from CONTROL_PATHLEN to 64 bytes
Send only the filename (hash.hs) instead of the full path in the
hyperscan cache notification. Main process reconstructs the full
path using cfg->hs_cache_dir.
This is the last CONTROL_PATHLEN field in rspamd_srv_command.
Vsevolod Stakhov [Tue, 13 Jan 2026 10:20:58 +0000 (10:20 +0000)]
[Fix] Refactor srv_pipe to use queue-based architecture with ID dispatch
Replace per-request ev_io watchers with a single watcher using khash
for ID-based reply matching. This fixes potential deadlocks when multiple
commands are queued rapidly (e.g., during hyperscan compilation).
Changes:
- Add rspamd_srv_pipe_ctx with single watcher, send queue, and ID hash
- Make srv_pipe non-blocking on both ends with proper EAGAIN handling
- Add EAGAIN handling to main process write path
- Remove cache_dir from hs_loaded commands (available from config)