[Feature] Fuzzy check: implement TCP error handling and command sending
Add comprehensive error handling for TCP connections:
- Cleanup pending requests when connections fail
- Handle timeout, write, read, and protocol errors
- Track connection per pending command for cleanup
Implement TCP command sending:
- Add TCP framing to encrypted commands
- Queue commands for asynchronous sending
- Register in pending pool for reply matching
- Integrate with main check flow with UDP fallback
Add async TCP connection establishment and I/O framework. This
implements Phase 2 of the TCP support - connection management with
event-driven architecture.
Changes:
- Add fuzzy_tcp_connection structure for per-rule TCP state
- Add fuzzy_tcp_pending_command for request/reply matching
- Implement fuzzy_tcp_connect_async() with non-blocking connect
- Implement fuzzy_tcp_io_handler() for connection/read/write events
- Add connection lifecycle management with reference counting
- Handle connection establishment with getsockopt SO_ERROR check
- Add timeout handling and upstream failure reporting
- Add placeholder write and read handlers for next phase
TCP connection is established lazily when rate threshold is exceeded.
Event handler manages connection state machine: connecting -> connected.
Write/read handlers will be implemented in Phase 3.
[Feature] Fuzzy check: add TCP support with auto-switch
Add TCP protocol support to fuzzy check client with rate-based
automatic switching between UDP and TCP transports. This enables
efficient bulk checking while maintaining UDP fallback.
Changes:
- Add TCP configuration parameters (enabled, auto, threshold, window, timeout)
- Implement sliding window rate tracker for request frequency monitoring
- Add TCP connection state tracking (connected, connecting)
- Implement fuzzy_should_use_tcp() decision logic
- Add fuzzy_update_rate_tracker() for rate tracking
- Add fuzzy_tcp_connect_async() placeholder for lazy TCP connection
- Integrate TCP/UDP selection in register_fuzzy_client_call()
[Feature] Fuzzy storage: implement TCP protocol support
Implement TCP transport for fuzzy storage protocol to enable efficient
bulk request handling. This adds TCP accept handlers, frame-based I/O
processing, and proper session management.
Changes:
- Add TCP session structure with framing state machine
- Implement TCP accept handler with rate limiting and access control
- Add TCP I/O handler supporting frame-based protocol (size header + payload)
- Implement TCP write reply with queuing support
- Add TCP timeout configuration parameter (default: 5.0 seconds)
- Refactor rate limit checks to accept parameters instead of session objects
- Update worker socket type to support both UDP and TCP
- Add debug logging infrastructure for fuzzy storage
[Feature] Add type specifiers support to lua_logger
Add support for format type specifiers in lua_logger:
- %d - signed integer (int64)
- %ud - unsigned integer (uint64)
- %f - floating point with smart formatting (no trailing zeros)
- %.Nf - floating point with N decimal places precision
- %% - escape literal percent sign
Type specifiers can be combined with positional (%1d) and
sequential (%d) argument references. String to number conversion
is supported. Added comprehensive unit tests.
Ensure classifiers are fetched when the dropdown is empty even if cache suggests skipping,
preventing an empty selector on Scan tab after RO → Disconnect → Enable.
* [Feature] Fuzzy check: Add separate encryption keys for read and write operations
* [Feature] DKIM: Add ED25519 support for DKIM signing and verification with OpenSSL version checks
* [Feature] Vault: Add HashiCorp Vault KV version 2 support for DKIM key management
* [Feature] MetaDefender: Add MetaDefender Cloud Lua module for SHA256 hash lookups
* [Feature] LLM: Add user/domain context support for LLM-based classification with Redis-based conversation context
* [Feature] DMARC: Add RUA address exclusion configuration option
* [Fix] DKIM: Fix relaxed bodyhash calculation for lines with only spaces to comply with RFC 6376
* [Fix] DKIM: Fix ED25519 key loading to prevent memory corruption in union handling
* [Fix] HTTP maps: Enforce server-controlled refresh intervals and prevent aggressive polling
* [Fix] HTTP maps: Prevent time_t overflow in expires header processing
* [Fix] Once received plugin: Fix duplicate symbol addition by changing break to return
* [Fix] Redis: Propagate unused Sentinel options properly
* [Fix] Fuzzy check: Fix reply decryption when using separate read/write keys
* [Fix] Fuzzy check: Add fallback when only one specific encryption key is set
* [Fix] Fuzzy check: Fix duplicate key filtering in reply decryption
* [Fix] Fuzzy ping: Allow read/write servers configuration
* [Minor] Fuzzy check: Refactor encryption key selection into helper functions
* [Minor] Fuzzy check: Stop early when found a correct key
* [Minor] Add cursor rules for development
[Fix] Add fallback when only one specific encryption key is set
When only read_encryption_key or write_encryption_key is configured without
a general encryption_key, the unspecified operation type was left with NULL
keys. Now if only one specific key is set, it's used for both read and write
operations as a fallback, ensuring encryption works in all configurations.
[Fix] Fix duplicate key filtering in reply decryption
When read/write encryption keys fall back to common encryption_key,
rspamd_pubkey_ref() returns pointer to the same object. Previous duplicate
checks using pointer comparison incorrectly filtered out these keys,
causing decryption failures. Now properly checks if key was already added
to the decryption attempt list before adding it.
[Minor] Refactor encryption key selection into helper functions
Extract repeated key selection logic into fuzzy_select_encryption_keys()
and fuzzy_rule_has_encryption() helper functions. This reduces code
duplication and improves readability across fuzzy_cmd_stat(),
fuzzy_cmd_ping(), fuzzy_cmd_hash(), fuzzy_cmd_from_text_part(),
fuzzy_cmd_from_data_part(), and fuzzy_process_reply() functions.
[Fix] Fix reply decryption when using only separate read/write keys
In fuzzy_process_reply(), the tag was accessed from encrypted data before
decryption, leading to incorrect key selection. When only separate
read_encryption_key and write_encryption_key were configured (without common
encryption_key), the fallback to NULL keys caused crashes.
Now the function tries decryption with all available key pairs (read, write,
and common) until MAC verification succeeds, properly handling all key
configuration scenarios.
[Fix] Ensure encryption works with separate read/write keys in fuzzy_check
Fix condition checks that determine whether to use encryption. Previously,
functions checked only rule->peer_key, causing encryption to be disabled
when using only read_encryption_key and write_encryption_key without a
common encryption_key. Now checks for any encryption keys (peer_key,
read_peer_key, or write_peer_key) to properly enable encryption.
[Feature] Add separate encryption keys for read and write operations in fuzzy_check
Allow using different encryption keys for read (CHECK, STAT, PING) and write
(WRITE, DEL) operations by introducing read_encryption_key and write_encryption_key
configuration parameters. Falls back to encryption_key if separate keys are not
specified for backward compatibility.
[Fix] Fix union handling in ED25519 key loading to prevent memory corruption
When loading ED25519 keys from PEM, the code was writing to key_eddsa in the
union and then attempting to free key_ssl pointers, which corrupted the
key_eddsa pointer and caused use-after-free/double-free during cleanup.
The fix saves the EVP_PKEY and BIO pointers to temporary variables, extracts
the raw key, frees the OpenSSL objects, and only then assigns to the union.
This prevents memory corruption and resource leaks.
[Feature] Add ED25519 support for DKIM signing with OpenSSL version checks
This commit adds support for ED25519 DKIM signatures when OpenSSL 1.1.1+ is available.
Key changes:
- Added HAVE_ED25519 detection in CMake to check for EVP_PKEY_ED25519 support
- All ED25519-specific code is conditionally compiled based on HAVE_ED25519
- When ED25519 is not supported, informative error messages are returned
- ED25519 keys loaded from PEM files are extracted and converted to libsodium format
- Fixed union handling to prevent double-free issues
- Updated tests to dynamically select key type based on request header
- Removed unused dkim-ed25519-pem.conf (cannot be passed via rspamc)
The implementation gracefully degrades on older OpenSSL versions while maintaining
full functionality when ED25519 support is available.
feat: Add ED25519 support for DKIM signing and verification
This commit introduces support for ED25519 keys in DKIM signing and verification. It includes changes to the DKIM library to handle ED25519 keys, along with new test cases and configuration files to demonstrate and test this functionality.
[Fix] Improve HTTP map interval logic for cache validation
Properly differentiate between maps with and without cache validation:
- With ETag/Last-Modified: use 4x multiplier (cheap conditional requests)
- Without cache validation: enforce strict 10 minute minimum
- Add overflow protection for interval multiplication
- Actually use has_etag/has_last_modified parameters
This avoids overly aggressive slowdown (120x -> 4x) for maps with cache
validation while still preventing abuse of maps without validation.
[CritFix] Prevent time_t overflow in HTTP map expires header processing
Add validation to detect and reject absurdly invalid or overflow-inducing
expires headers (>1 year in future). When expires header is invalid or
causes overflow, properly call rspamd_http_map_process_next_check with
expires=0 instead of setting map->next_check=0 which left stale overflow
values.
This prevents crashes and invalid scheduling like 'next check at Thu,
09 Nov 438498967' when servers send malformed Expires headers.
Prevent aggressive HTTP map polling by implementing proper interval bounds:
- Cap absurdly high Expires headers (>8h) to min(map_interval * 10, 8h)
- Enforce configured map_interval as minimum when server requests faster refresh
- Apply 10 minute minimum interval when no Expires header and low map_interval
- Simplify logic by consolidating interval calculation in single function
This change ensures servers can control refresh rates and prevents clients
from causing issues with overly aggressive polling behavior.
[Feature] Add symbol categories for MetaDefender and VirusTotal
Implemented a category-based symbol system for hash lookup antivirus
scanners (MetaDefender and VirusTotal) to replace dynamic scoring:
- Added 4 symbol categories: CLEAN (-0.5), LOW (2.0), MEDIUM (5.0), HIGH (8.0)
- Replaced full_score_engines with threshold-based categorization (low_category, medium_category)
- Fixed symbol registration in antivirus.lua to use rule instead of config
- Updated cache format to preserve symbol category across requests
- Added backward compatibility for old cache format
- Added symbols registration and metric score assignment
- Updated configuration documentation with examples
The new system provides:
- Clear threat categorization instead of linear interpolation
- Proper symbol weights applied automatically
- Consistent behavior between MetaDefender and VirusTotal
- Cache that preserves symbol categories
[Fix] Add nil check for vault_data in show_handler
Prevent runtime errors when parsing Vault KV v2 responses if obj.data.data is nil.
This adds a safety check before accessing vault_data.selectors, consistent with
other handlers in the file (newkey_handler and roll_handler).
[Feature] Improve LLM prompt and add sender frequency tracking
* Update default prompt to reduce false positives on legitimate emails
- Explicitly recognize verification emails as legitimate
- Require MULTIPLE red flags for phishing classification
- Add guidance on known/frequent senders
* Add sender frequency detection in context
- Classify senders as: new, occasional, known, frequent
- Based on sender_counts from user context
- Passed to LLM via context snippet
* Prompt instructs LLM to reduce phishing score for known senders
* Helps avoid false positives on transactional/verification emails
[Feature] Improve GPT module with uncertain caching and server timeout
* Add GPT_UNCERTAIN symbol for caching uncertain classifications
- Cache results even when no consensus is reached
- Avoid repeated expensive LLM queries for borderline cases
- Set X-GPT-Reason header with detailed vote statistics
* Add server-side timeout support for OpenAI API requests
- New request_timeout parameter (optional, multiplied by 0.95)
- Only sent if explicitly configured (not all APIs support this)
- Accounts for connection setup and data transfer overhead
* Fix max_ham_prob initialization (was 0, now correctly 1.0)
* Add pcall protection for fold_header_with_encoding with raw fallback
* Improve error messages for token limit exceeded
* Add detailed logging for context snippets and consensus decisions
* Pass debug_module parameter to llm_context functions
[Feature] Add cache expiration timestamps to debug logs
* Show when cached data will expire in human-readable format
* Log expiration time both when caching and after successful write
* Helps with debugging cache TTL issues
[Feature] Add bidirectional context support for LLM
* Unify context for incoming and outgoing mail
* Same identity used for authenticated/local sender and recipient
* Follows replies module pattern for direction detection
* Make llm_context.lua module-agnostic with debug_module parameter
* Improve userdata handling (use :sub instead of string.sub)
* Add nil-safety to all debug logging calls
* Add cache expiration timestamps to context logs
[Fix] Add full Lua traceback to HTTP callback errors
Improved error diagnostics in lua_http_finish_handler by adding
rspamd_lua_traceback handler. Now shows complete call stack with
file names and line numbers when Lua HTTP callbacks fail, making
debugging much easier.
[Feature] Add user/domain context support for LLM-based classification
* Add llm_context.lua module for Redis-based conversation context
* Context features: sliding window, top senders, keywords, flagged phrases
* Use low-level word API (get_words('full')) with stop_word flags
* Flexible gating via maps/selectors (enable_map/enable_expression)
* Update context even when GPT condition not met (BAYES_SPAM/HAM)
* Add min_messages warm-up threshold to prevent weak context injection
* Configurable scope: user/domain/esld with TTL and sliding window
* [Feature] Archive module: Full support for encrypted ZIP archives with ZipCrypto and AES encryption
* [Feature] Archive module: Both reading and writing of AES-encrypted ZIP archives is supported
* [Feature] Archive module: Updated Lua bindings for libarchive
* [Feature] Encrypted maps: Support for encrypted maps to enable new distribution scenarios
* [Feature] Redis TLS: Configurable TLS connections in Redis backend
* [Feature] Map helpers alignment: Enforce 64-byte alignment to prevent unaligned memory access
* [Feature] Enhanced CLI for secretbox with additional security test coverage
* [Fix] MIME encoding: Major overhauls and multiple fixes for MIME encoding logic
* [Fix] MIME encoding: Improved handling and decoding of UTF-8 in MIME headers
* [Fix] Learning system: Numerous fixes to learn checks and autolearn flag handling
* [Fix] Learning system: Prevention of duplicate message learning
* [Fix] Learning system: Extended multiclass learning test coverage
* [Fix] Critical: Fixed bug when converting zero-length strings to numbers
* [Fix] Critical: Fixed XML prolog detection in lua_magic module
* [Fix] Build: Fixed build issues on 32-bit platforms
* [Fix] Compatibility: Improved compatibility with Lua versions above 5.1
* [Fix] Empty input: Addressed issues with empty input handling in lua_magic
* [Fix] Testing: Improved stability of automated testing with multiple test fixes
* [Fix] Minor compatibility improvements (buffer allocation, missing cmath include)
* Refactored rspamd_control_fill_msghdr to accept
a caller-provided control buffer, fixing the
lifetime bug where a pointer to a local array
was stored in msg_control.
* Replaced static buffers with automatic (stack)
buffers at the exact call sites of sendmsg/recvmsg,
so PowerPC and similar platforms won’t choke on
non-constant expressions.
- Removed g_strdup/g_free of TLS paths in src/lua/lua_redis.c.
- Now we:
- Keep TLS values (booleans + strings) on the Lua stack temporarily.
- Use an absolute table index (so gettable calls aren’t confused by
the growing stack).
- Call rspamd_redis_pool_connect_ext while those values are on the
stack.
- Pop all postponed values and then the table in one go immediately
after the connect call.
- The C++ pool still copies into std::string on element creation; we
only ensure Lua strings live through the call without extra
allocations.
- remove redundant `ensure_ssl_inited` function and calls. Core SSL init
should suffice.
- Refactor TLS initiation into `redis_pool_elt::initiate_tls(...)` to
eliminate duplication
- Switch TLS flags to `bool` in the public struct
- Fix ephemeral string usage in lua by duplicating the values into
locals and freeing after connect. Flags are boolean. (it's not super
likely that Lua will GC the strings before we connect to Redis, but
this ensures that it won't be a problem)
- Remove the redis TLS options propagation unit test
projects / thirdparty / rspamd.git / log
