Vsevolod Stakhov [Wed, 12 Nov 2025 10:25:05 +0000 (10:25 +0000)]
[Fix] Keep groups_*/symbols_* fields in settings for runtime processing
The Lua settings preprocessing was removing groups_disabled, groups_enabled,
symbols_disabled, and symbols_enabled fields from settings objects after
converting them to individual symbol lists. However, the C++ runtime code
(rspamd_symcache_process_settings) also needs these fields to mark symbols
as enabled/disabled in the runtime state.
Background:
- June 2019: Lua preprocessing added, removed fields to 'avoid clash'
- April 2022: C++ runtime process_settings() added, which reads these fields
- The two mechanisms were never properly synchronized
How settings work:
1. Lua preprocessing (at config load): Converts groups_* to symbol lists,
registers via rspamd_config:register_settings_id(), creates settings_elt
with allowed/forbidden symbol IDs for enforcement
2. C++ runtime (when settings applied): Reads task->settings UCL object,
processes groups_*/symbols_* fields to mark symbols as finished/not_started
in the runtime state for optimization
The issue:
When settings are applied via task:set_settings(cached.settings.apply),
the Lua table is converted to UCL and stored in task->settings. If the
groups_* fields were removed by preprocessing, process_settings() cannot
find them, so symbols don't get marked as disabled in runtime state.
The fix:
Don't remove these fields. Both mechanisms work complementarily:
- Settings element: Enforcement (checks allowed/forbidden IDs per symbol)
- Runtime processing: Optimization (marks symbols as finished to skip them)
This is safe because:
- The preprocessing still builds correct allowed/forbidden symbol lists
- The runtime processing reads the same fields and applies them correctly
- No conflict occurs - they work on different aspects (enforcement vs state)
Note: This affects settings IDs (static settings), not dynamic settings
applied directly via Settings={} header, which never went through the
removal step.
Vsevolod Stakhov [Mon, 10 Nov 2025 13:58:25 +0000 (13:58 +0000)]
[Fix] Allow custom actions with flags in config validation
The config validation was warning about unknown actions even when they
were valid custom actions defined with flags like 'no_threshold'.
Custom actions (e.g., malware, virus, hard-reject) are properly supported
by the C code but the Lua validation didn't recognize them, causing
spurious warnings.
Fix by checking if an unknown action is defined as an object with flags
before emitting the warning.
Vsevolod Stakhov [Mon, 10 Nov 2025 09:26:25 +0000 (09:26 +0000)]
[Fix] Regenerate DNS transaction ID before copying packet to TCP buffer
When switching from UDP to TCP (e.g., on truncated response), the code
was copying the packet to the TCP output buffer before regenerating the
transaction ID to avoid collisions. This resulted in both UDP and TCP
packets having the same transaction ID.
Fix by moving the ID regeneration and IO channel switch logic before the
memcpy to the TCP output buffer, ensuring the TCP packet contains the
updated transaction ID.
Vsevolod Stakhov [Mon, 10 Nov 2025 09:21:36 +0000 (09:21 +0000)]
[Fix] Preserve req->pos during reply validation to prevent packet truncation on retransmit
The rdns_request_reply_cmp function modifies req->pos as a side effect
during reply validation. This caused packet truncation on retransmits
because req->pos (which tracks the full packet length) was overwritten
with the end position of the question section.
On timeout/retry, rdns_send_request would use the corrupted req->pos
value, resulting in truncated packets missing the OPT additional section.
This made resolvers like Knot and PowerDNS unable to parse retransmitted
packets.
Fix by saving and restoring req->pos around the reply comparison logic.
[Minor] Replace GHashTable with khash in lua_textpart_get_cta_urls
Replace GHashTable with embedded khash for tracking seen URLs in
lua_textpart_get_cta_urls. khash provides better performance for
simple pointer sets and is already embedded in the project, reducing
external dependencies on GLib for this functionality.
[Fix] Force GC after final batch to release all Redis connections
Previously GC was skipped for the final batch (batch_end >= #results),
leaving those connections checked out. The subsequent DEL of idx_key
could hit the pool limit if batch_size was near the limit, leaving the
_processing key behind and exhausting the pool for the next date.
Now collectgarbage() runs after every batch including the last one,
ensuring all connections are released before the DEL operation.
[Fix] Floor batch_size to integer to prevent fractional indexing
Use math.floor() to convert batch_size to integer before using in loops.
Fractional values like 1.5 would cause batch_start (e.g. 2.5) to index
results[] with non-integer, returning nil and crashing prepare_report()
with 'string expected, got nil'.
[Fix] Normalize batch_size once in handler to fix both code paths
Move batch_size validation to handler() right after argument parsing,
ensuring both process_report_date() and send_reports_by_smtp() use the
normalized value. Previously the validation was only in the Redis loop,
so send_reports_by_smtp() would still see invalid values (0 or negative),
causing nreports <= 0 which breaks sendmail callback scheduling and
prevents finish_cb from ever firing.
[Fix] Validate batch_size to prevent loop error with invalid input
Clamp batch_size to minimum of 1 to prevent Lua error 'bad for step'
when user provides --batch-size 0 or negative values. Previously this
would crash the tool; now it processes with batch_size=1.
[Fix] Add batching and forced GC for Redis connections in dmarc_report
The coroutine version of lua_redis.request() creates connections that
are only released when Lua garbage collector runs. In tight loops
processing hundreds of domains, GC doesn't run frequently enough,
causing connection exhaustion.
Root cause: lua_redis.connect_sync() + conn:exec() returns connections
to Lua but relies on __gc metamethod for pool release. The connections
pile up until GC runs.
Solution:
- Process reports in batches (reuse existing --batch-size option)
- Force collectgarbage("collect") between batches to trigger __gc
- This ensures connections are properly released back to the pool
Each prepare_report() makes 3-4 Redis requests (EXISTS, RENAME,
ZRANGE, DEL), so batching + forced GC prevents parallel connection
buildup.
[Rework] Refactor extract_specific_urls to prevent DoS and use hash-based deduplication
Replace tostring() with url:get_hash() throughout URL extraction to avoid
filling the Lua string interning table. Critical for handling malicious
messages with 100k+ URLs where each tostring() would create an interned
string causing memory exhaustion.
Key changes:
- Use dual data structure: array for results + hash set for O(1) dedup
- Add max_urls_to_process=50000 limit with warning for DoS protection
- Track url_index for stable sorting when priorities are equal
- Fix CTA priority preservation: prevent generic phished handling from
overwriting CTA priorities which include phished/subject bonuses
- Add verbose flag to test suite for debugging
This ensures memory usage is strictly bounded regardless of malicious
input while maintaining correct URL prioritization for spam detection.
HTML_DISPLAYED URLs are phishing bait text (display-only) and should
not be considered for CTA (call-to-action) detection. Only real link
targets should be analyzed for CTA purposes.
[Feature] Add url:get_hash() method for efficient URL deduplication
Expose rspamd_cryptobox_fast_hash via Lua API to allow hash-based
URL deduplication without string conversion overhead. This is critical
for handling messages with large numbers of URLs (100k+) where tostring()
would fill the Lua string interning table and cause memory issues.
Returns 64-bit hash as Lua number, using the same hash function as
internal URL storage for consistency.
[Feature] Add task:get_cta_urls() API for proper CTA domain extraction
- C code (message.c): collect top CTA URLs per HTML part by button weight,
store in task mempool variable "html_cta_urls"
- Lua API (lua_task.c): add task:get_cta_urls([max_urls]) method
- llm_search_context: use new API instead of reimplementing CTA logic in Lua
- Benefits: single source of truth for CTA logic, uses C knowledge of HTML
structure and button weights, cleaner Lua code
This provides proper architecture where C code handles HTML structure analysis
and Lua adds domain filtering (blacklists, infrastructure domains, etc.)
[Fix] Refactor llm_search_context to use lua_cache module
- Replace manual Redis operations with lua_cache API for better consistency
- Use messagepack serialization and automatic key hashing
- Fix Leta Mullvad API URL to /search/__data.json endpoint
- Add search_engine parameter support
- Remove redundant 'or DEFAULTS.xxx' patterns (opts already has defaults merged)
- Add proper debug_module propagation throughout call chain
- Improve JSON parsing to handle Leta Mullvad's nested pointer structure
[Fix] Fix llm_search_context to follow Rspamd idioms
- Use local N = 'llm_search_context' idiom instead of constant string reuse
- Replace rspamd_logger.debugm with lua_util.debugm (rspamd_logger has no debugm method)
- Use extract_specific_urls instead of task:get_urls()
- Add task parameter to query_search_api for proper logging and HTTP requests
- Remove retry logic using non-existent rspamd_config:add_delayed_callback
- Simplify to single HTTP attempt with graceful failure
- Remove retry_count and retry_delay options from config
[Feature] Add web search context support to GPT plugin
- New module llm_search_context.lua: extracts domains from email URLs and queries search API
- Integrated into gpt.lua with parallel context fetching (user + search)
- Redis caching with configurable TTL (default 1 hour)
- Retry logic with exponential backoff for search API failures
- Disabled by default for backward compatibility
- Configuration options in gpt.conf for customization
[Fix] Fix OpenBSD kinfo_proc structure member names
OpenBSD uses p_vm_* member names similar to NetBSD, not FreeBSD's ki_* names.
Separate OpenBSD implementation from FreeBSD to use correct structure members.
[Feature] Fix BSD workflow package installation and update OS versions
- FreeBSD: Add IGNORE_OSVERSION=yes to handle package version mismatches, update to 14.3/13.5
- OpenBSD: Set PKG_PATH for package repository, fix perl package name, update to 7.8/7.7/7.6
- Use latest stable versions of both operating systems
[Fix] NetBSD workflow: setup pkgin and PKG_PATH before installing packages
The pkg_add command requires PKG_PATH to be set in NetBSD 10.0.
Install and use pkgin for easier binary package management.
Changes:
- Set PKG_PATH to NetBSD CDN repository
- Install pkgin using /usr/sbin/pkg_add
- Use pkgin for all package installations
- Change perl5 to perl (correct package name)
- Add || true for non-critical package installation failures
[Feature] Add/improve BSD build workflows with Lua version selection
Add comprehensive GitHub Actions workflows for BSD systems with the ability
to select Lua version (LuaJIT, Lua 5.1, 5.3, 5.4).
NetBSD workflow improvements:
- Add Lua version selection (luajit, lua51, lua53, lua54)
- Fix missing dependencies (libarchive, zstd, xxhash, file)
- Remove || true from pkg_add to catch installation failures
- Add -DENABLE_HYPERSCAN=OFF (not available on NetBSD)
- Add -DSYSTEM_ZSTD=ON and -DSYSTEM_XXHASH=ON flags
- Add conditional ENABLE_LUAJIT based on Lua version selection
- Add NetBSD 9.4 to supported versions
- Fix test executable paths (add ./ prefix)
New FreeBSD workflow:
- Support FreeBSD 14.2, 13.4, 13.3
- Lua version selection (luajit, lua51, lua53, lua54)
- Full dependency list including hyperscan
- Enable hyperscan support (-DENABLE_HYPERSCAN=ON)
- Use system zstd and xxhash libraries
- Proper pkg update before installation
New OpenBSD workflow:
- Support OpenBSD 7.6, 7.5, 7.4
- Lua version selection (luajit, lua51, lua53)
- Full dependency list including hyperscan
- Enable hyperscan support
- Disable jemalloc (-DENABLE_JEMALLOC=OFF)
- Use OpenBSD-specific package names (icu4c, perl-5, lua%5.x)
- Use system zstd and xxhash libraries
All workflows:
- Use workflow_dispatch trigger for manual execution
- Allow selection of OS version and Lua version
- Use vmactions VMs for BSD testing
- Run both C++ and Lua unit tests
- Use Ninja build system for faster compilation
This provides comprehensive testing across different BSD platforms and
Lua versions, ensuring Rspamd builds correctly on various configurations.
[Fix] Keep srv events active during shutdown to track auxiliary processes
When Rspamd shuts down with auxiliary processes running (e.g., neural network
training spawned by workers), the main process was stopping srv_pipe event
handlers immediately after sending SIGTERM to workers. This prevented workers
from sending RSPAMD_SRV_ON_FORK notifications when their auxiliary child
processes terminated, causing these children to remain tracked indefinitely.
The main process would then hang for 90 seconds waiting for already-dead
processes that it couldn't properly clean up from the workers hash table.
Root cause analysis:
- Direct workers have ev_child watchers and are removed via SIGCHLD handler
- Auxiliary processes (fork from workers) have NO ev_child watchers
- They are removed ONLY via srv_pipe notifications (RSPAMD_SRV_ON_FORK)
- Stopping srv events during shutdown breaks this notification channel
The original stop_srv_ev code was added in 2019 (commit eafdd221) to avoid
"false notifications" during a major refactoring. However, this is no longer
an issue because:
1. srv_ev handlers automatically stop on EOF when worker pipes close
2. There is no risk of duplicate notifications
3. Auxiliary processes critically need these events to report termination
Solution: Remove the stop_srv_ev call from rspamd_term_handler. This allows
workers to continue sending process termination notifications during shutdown.
The srv_ev handlers will stop naturally when workers close their pipes.
[Fix] Recreate invalid unserialized Hyperscan cache files on version mismatch
When Hyperscan library is updated, previously cached .unser files become
invalid due to version mismatch. Previously, these files would remain
unusable, forcing fallback to slower deserialization on every load.
This commit adds automatic detection and recreation of invalid unserialized
files:
- Extract file creation logic into create_unserialized_file() helper function
- Add error handler that deletes and recreates invalid .unser files
- Maintain file locking protection against concurrent process access
- Fall back to serialized version if recreation fails
This ensures cache files are automatically updated after Hyperscan upgrades
while protecting against race conditions in multi-process environments.
[Fix] Use runtime Hyperscan version instead of compile-time version for database validation
The issue was that the database version check used HS_DB_VERSION macro defined
in headers at compile time, while Hyperscan .so library writes the version from
its own headers. When the system updates the Hyperscan package but Rspamd isn't
recompiled, this causes a version mismatch and database validation fails.
The fix calls hs_version() at runtime to get the actual library version and uses
that for validation instead. This ensures compatibility when the Hyperscan library
is updated independently.
Resolve crashes caused by mixing system malloc and jemalloc allocators.
The issue occurred when getline() and hiredis used system malloc, but
rspamd's free() used jemalloc, causing segmentation faults on macOS and
potentially other platforms.
Changes:
- Add rspamd_getline() wrapper using g_malloc/g_realloc/g_free to avoid
system malloc in getline()
- Replace getline() with rspamd_getline() in url.c, dns.c, lua_repl.c
- Fix memory leak in lua_repl.c by freeing input buffer on exit
- Configure hiredis allocators to use glib functions (jemalloc) in
rspamd_init_libs()
This ensures all memory operations use the same allocator (jemalloc)
throughout rspamd, preventing allocator mismatch crashes.
Resolve crashes caused by mixing jemalloc and system malloc allocators
in libucl. The issue occurred when memory allocated with one allocator
(e.g., strdup using system malloc) was freed with another (e.g., jemalloc's
free), causing segmentation faults.
Changes:
- Add UCL_REALLOC and UCL_STRDUP macros to ucl.h for consistent allocation
- Replace all strdup/malloc/realloc/free calls with UCL_* macros in:
- Variable and macro registration (ucl_parser.c)
- Parser state management (ucl_util.c)
- Object copying and trash stack operations (ucl_util.c)
- URL fetching - fix critical bug where malloc'd buffers were freed
with ucl_munmap (munmap) instead of free (ucl_util.c)
This ensures all memory operations use the same allocator throughout libucl,
preventing allocator mismatch crashes on systems using jemalloc.
Cursor Agent [Sat, 1 Nov 2025 12:56:18 +0000 (12:56 +0000)]
[Fix] Fix rspamd nameserver round-robin when using /etc/resolv.conf
When nameservers are parsed from /etc/resolv.conf, rspamd was setting
the upstream rotation strategy to RSPAMD_UPSTREAM_MASTER_SLAVE, which
caused it to only use the first nameserver unless it failed.
This behavior was inconsistent with the documented round-robin strategy
and with the behavior when nameservers are explicitly configured via
the configuration file.
Fixed by changing the rotation strategy to RSPAMD_UPSTREAM_ROUND_ROBIN
when parsing /etc/resolv.conf, matching the expected behavior.
Vsevolod Stakhov [Thu, 30 Oct 2025 10:17:31 +0000 (10:17 +0000)]
[Minor] Update libucl with automatic stack management
Merge changes from upstream libucl commit 26bec99:
- Add UCL_STACK_AUTOMATIC flag for automatic stack preservation
- Modify stack cleanup to conditionally preserve automatic stacks
- Enable parsing of included files without outer braces
This allows parsing of UCL configurations that include files with
content like 'key = value;' without requiring explicit braces,
improving compatibility with various UCL file formats.
- **D3Evolution** 2.0.2 → 2.0.3
Improves flexibility for different themes.
- **rspamd-D3Pie** 1.1.0 → 1.1.1
Fixes a bug where the color was not applied to the label of the placeholder slice.
Vsevolod Stakhov [Wed, 29 Oct 2025 13:45:38 +0000 (13:45 +0000)]
[Fix] Fix TCP DNS uninitialized memory leak
When rescheduling a DNS request from UDP to TCP, the code was using
req->packet_len (allocated buffer size) instead of req->pos (actual
packet size) to copy and send the DNS packet. This caused random
garbage from uninitialized memory to be appended to TCP DNS queries.
The bug was particularly noticeable with short queries like TXT records,
where the allocated buffer could be 2-3x larger than the actual packet.
Vsevolod Stakhov [Tue, 28 Oct 2025 15:51:51 +0000 (15:51 +0000)]
[Fix] Fix memory leak in fuzzy storage khash tables
In init_fuzzy(), two khash tables were created but their destructors
were not added to the config mempool:
- ctx->default_forbidden_ids
- ctx->weak_ids
While these tables were destroyed in the worker cleanup code (before
exit), this cleanup doesn't run during configtest, causing a memory leak.
Fix: Add mempool destructors for both hash tables, similar to how
ctx->keys and ctx->errors_ips are handled. This ensures proper cleanup
in all scenarios including configtest.
Vsevolod Stakhov [Tue, 28 Oct 2025 15:01:21 +0000 (15:01 +0000)]
[Fix] Fix memory leak in address parsing for *-any addresses
In rspamd_parse_host_port_priority(), when handling '*' (any address),
the GPtrArray was created with a conditional destructor:
pool == NULL ? NULL : (GDestroyNotify) rspamd_inet_address_free
This meant that when pool == NULL, the array had NO destructor for
elements. Later, when rspamd_upstreams_add_upstream() copied addresses
and called g_ptr_array_free(addrs, TRUE), the original address objects
were not freed, causing a memory leak.
Fix: Always set the destructor to rspamd_inet_address_free, regardless
of pool presence. The destructor will properly free address elements
when the array is freed, while mempool destructor (if pool exists)
will handle freeing the array itself.
Vsevolod Stakhov [Tue, 28 Oct 2025 14:37:36 +0000 (14:37 +0000)]
[Fix] Fix memory leak in upstream address parsing
When parsing upstream addresses in rspamd_upstreams_add_upstream(),
the GPtrArray 'addrs' was not properly freed in several cases:
1. For service= variant: destructor was not added to mempool
2. After copying addresses to upstream: array was not freed when
ups->ctx is NULL
3. On parse failure: addrs was not freed
This commit adds proper cleanup:
- Add mempool destructor for service= variant when ctx exists
- Free addrs after copying addresses if no ctx (not managed by mempool)
- Free addrs on parse failure if not managed by mempool
- Set addrs = NULL after freeing to avoid dangling pointer
The fix is careful about mempool-managed vs manual cleanup to avoid
double-free issues, especially important since upstreams can be called
from destructor functions.
projects / thirdparty / rspamd.git / log
