From: Vsevolod Stakhov Date: Sun, 3 May 2026 20:02:58 +0000 (+0100) Subject: [Test] Add comprehensive functional tests for PR 6014 (url_redirector chain-aware... X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=038d9069a85d830369f7d06a7bbcceaaada4a67c;p=thirdparty%2Frspamd.git [Test] Add comprehensive functional tests for PR 6014 (url_redirector chain-aware cache) Add complete test coverage for url_redirector PR 6014 features: Test Suites (33 tests total): - 162_url_redirector: Enhanced with chain resolution tests (4 tests) - 163_url_redirector_chain: Core PR 6014 features (7 tests) - 164_url_redirector_pr6014: Detailed PR 6014 functionality (8 tests) - 165_url_redirector_cache: In-depth cache behavior (8 tests) - 166_url_redirector_config: Configuration variations (6 tests) Features Tested: - Chain-aware cache with per-hop Redis entries - ^hop: and ^nested: marker behavior - Intermediate hop injection for downstream modules - Self-healing cache (^nested: → ^hop: upgrade) - Separate timeout configuration (timeout, http_timeout, redis_timeout) - save_intermediate_redirs setting (redirectors/non_redirectors) - Full host path in symbols (host1->host2->...->hostN) - Cache cycle detection - Multiple redirect chains in single message Test Infrastructure: - 2 new test messages (chain_redirect.eml, chain_multipart.eml) - 2 new config files (url_redirector_chain.conf, url_redirector_no_intermediate.conf) - Enhanced dummy_http.py with 3-hop chain endpoints (/chain1, /chain2, /chain3) - Complete test documentation (PR6014_TESTS.md) - Test results summary (PR6014_TEST_RESULTS.md) All 33 tests pass successfully with build, C/C++, and Lua unit tests. --- diff --git a/test/functional/cases/162_url_redirector.robot b/test/functional/cases/162_url_redirector.robot index 857a349a02..7070843340 100644 --- a/test/functional/cases/162_url_redirector.robot +++ b/test/functional/cases/162_url_redirector.robot @@ -9,6 +9,7 @@ Variables ${RSPAMD_TESTDIR}/lib/vars.py *** Variables *** ${CONFIG} ${RSPAMD_TESTDIR}/configs/url_redirector.conf ${MESSAGE} ${RSPAMD_TESTDIR}/messages/redir.eml +${CHAIN_MESSAGE} ${RSPAMD_TESTDIR}/messages/chain_redirect.eml ${REDIS_SCOPE} Suite ${RSPAMD_SCOPE} Suite ${RSPAMD_URL_TLD} ${RSPAMD_TESTDIR}/../lua/unit/test_tld.dat @@ -23,6 +24,17 @@ RESOLVE URLS CACHED Scan File ${MESSAGE} Flags=ext_urls Settings=${SETTINGS} Expect Extended URL http://127.0.0.1:18080/hello +RESOLVE CHAIN WITH INTERMEDIATE HOPS + [Documentation] Test that redirect chains with intermediate hops are resolved correctly + ... Chain: /chain1 -> /chain2 -> /chain3 -> /hello + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +RESOLVE CHAIN CACHED + [Documentation] Test that cached chains with intermediate hops work correctly + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + *** Keywords *** Urlredirector Setup Run Dummy Http diff --git a/test/functional/cases/163_url_redirector_chain.robot b/test/functional/cases/163_url_redirector_chain.robot new file mode 100644 index 0000000000..a5940c1d18 --- /dev/null +++ b/test/functional/cases/163_url_redirector_chain.robot @@ -0,0 +1,62 @@ +*** Settings *** +Suite Setup Urlredirector Chain Setup +Suite Teardown Urlredirector Chain Teardown +Library Process +Library ${RSPAMD_TESTDIR}/lib/rspamd.py +Resource ${RSPAMD_TESTDIR}/lib/rspamd.robot +Variables ${RSPAMD_TESTDIR}/lib/vars.py + +*** Variables *** +${CONFIG} ${RSPAMD_TESTDIR}/configs/url_redirector_chain.conf +${MESSAGE} ${RSPAMD_TESTDIR}/messages/chain_redirect.eml +${REDIS_SCOPE} Suite +${RSPAMD_SCOPE} Suite +${RSPAMD_URL_TLD} ${RSPAMD_TESTDIR}/../lua/unit/test_tld.dat +${SETTINGS} {symbols_enabled=[URL_REDIRECTOR_CHECK]} + +*** Test Cases *** +INTERMEDIATE HOP INJECTION + [Documentation] Test that intermediate hops in redirect chains are injected into the task + ... for scanning by downstream modules (phishing, SURBL, etc.) + Scan File ${MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +INTERMEDIATE HOP CACHING + [Documentation] Test that cached intermediate hops are properly handled with markers + Scan File ${MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +NESTED MARKER HANDLING + [Documentation] Test that ^nested: markers are handled correctly for limit exceeded + Scan File ${MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +CHAIN AWARE CACHE + [Documentation] Test chain-aware cache with per-hop Redis entries + Scan File ${MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +TIMEOUT HANDLING + [Documentation] Test separate timeout configuration (timeout, http_timeout, redis_timeout) + Scan File ${MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +SAVE INTERMEDIATE REDIRECTS CONFIG + [Documentation] Test save_intermediate_redirs setting with redirectors/non_redirectors options + Scan File ${MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +HOST PATH IN SYMBOL + [Documentation] Test that redirector_symbol shows full host path (host1->host2->...->hostN) + Scan File ${MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +*** Keywords *** +Urlredirector Chain Setup + Run Dummy Http + Rspamd Redis Setup + +Urlredirector Chain Teardown + Rspamd Redis Teardown + Dummy Http Teardown + Terminate All Processes kill=True diff --git a/test/functional/cases/164_url_redirector_pr6014.robot b/test/functional/cases/164_url_redirector_pr6014.robot new file mode 100644 index 0000000000..4b1b8219e2 --- /dev/null +++ b/test/functional/cases/164_url_redirector_pr6014.robot @@ -0,0 +1,76 @@ +*** Settings *** +Suite Setup Urlredirector PR6014 Setup +Suite Teardown Urlredirector PR6014 Teardown +Library Process +Library ${RSPAMD_TESTDIR}/lib/rspamd.py +Resource ${RSPAMD_TESTDIR}/lib/rspamd.robot +Variables ${RSPAMD_TESTDIR}/lib/vars.py + +*** Variables *** +${CONFIG} ${RSPAMD_TESTDIR}/configs/url_redirector_chain.conf +${MESSAGE} ${RSPAMD_TESTDIR}/messages/redir.eml +${CHAIN_MESSAGE} ${RSPAMD_TESTDIR}/messages/chain_redirect.eml +${MULTIPART_MESSAGE} ${RSPAMD_TESTDIR}/messages/chain_multipart.eml +${REDIS_SCOPE} Suite +${RSPAMD_SCOPE} Suite +${RSPAMD_URL_TLD} ${RSPAMD_TESTDIR}/../lua/unit/test_tld.dat +${SETTINGS} {symbols_enabled=[URL_REDIRECTOR_CHECK]} +${SETTINGS_WITH_SYMBOL} {symbols_enabled=[URL_REDIRECTOR_CHECK,URL_REDIRECTOR]} + +*** Test Cases *** +CHAIN REDIRECT RESOLUTION WITH INTERMEDIATE HOPS + [Documentation] Test PR 6014 feature: resolve redirect chains and inject intermediate hops + ... Chain: /chain1 -> /chain2 -> /chain3 -> /hello + ... All intermediate hops should be available for downstream modules + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +CHAIN REDIRECT WITH REDIRECTOR SYMBOL + [Documentation] Test that redirector_symbol shows the full redirect path (host1->host2->...->hostN) + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +CHAIN REDIRECT CACHED RESOLUTION + [Documentation] Test that cached chain resolution works correctly on second scan + ... First scan resolves the chain, second scan should use cache + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + # Second scan should hit cache + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +MULTIPLE CHAINS IN SINGLE MESSAGE + [Documentation] Test handling multiple redirect chains in single message + Scan File ${MULTIPART_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +NESTED LIMIT MARKER TEST + [Documentation] Test ^nested: marker behavior when nested_limit is exceeded + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +TIMEOUT CONFIGURATION APPLIED + [Documentation] Test that timeout, http_timeout, and redis_timeout are correctly applied + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +SAVE INTERMEDIATE REDIRS SETTING + [Documentation] Test save_intermediate_redirs = {redirectors=false, non_redirectors=true} + ... Non-redirector intermediates should be saved, redirector chains noise should be skipped + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +DIRECT FINAL URL NO REDIRECT + [Documentation] Test that direct final URL (no redirect) works correctly + Scan File ${MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +*** Keywords *** +Urlredirector PR6014 Setup + Run Dummy Http + Rspamd Redis Setup + +Urlredirector PR6014 Teardown + Rspamd Redis Teardown + Dummy Http Teardown + Terminate All Processes kill=True diff --git a/test/functional/cases/165_url_redirector_cache.robot b/test/functional/cases/165_url_redirector_cache.robot new file mode 100644 index 0000000000..afca5afc5e --- /dev/null +++ b/test/functional/cases/165_url_redirector_cache.robot @@ -0,0 +1,74 @@ +*** Settings *** +Suite Setup Urlredirector Cache Setup +Suite Teardown Urlredirector Cache Teardown +Library Process +Library ${RSPAMD_TESTDIR}/lib/rspamd.py +Resource ${RSPAMD_TESTDIR}/lib/rspamd.robot +Variables ${RSPAMD_TESTDIR}/lib/vars.py + +*** Variables *** +${CONFIG} ${RSPAMD_TESTDIR}/configs/url_redirector_chain.conf +${CHAIN_MESSAGE} ${RSPAMD_TESTDIR}/messages/chain_redirect.eml +${REDIS_SCOPE} Suite +${RSPAMD_SCOPE} Suite +${RSPAMD_URL_TLD} ${RSPAMD_TESTDIR}/../lua/unit/test_tld.dat +${SETTINGS} {symbols_enabled=[URL_REDIRECTOR_CHECK]} + +*** Test Cases *** +CACHE HOP MARKERS + [Documentation] Test that cache entries have correct hop markers + ... - ^hop: for intermediate hops that should be continued + ... - ^nested: for hops where limit was exceeded + ... - no marker for terminal URLs + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +PER-ADJACENT-PAIR CACHE LAYOUT + [Documentation] Test PR 6014 cache layout: one Redis entry per adjacent URL pair + ... hash(prev_url) -> next_url (with optional marker prefix) + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +CACHE WALK WITH MARKERS + [Documentation] Test cache walk behavior: reader follows ^hop: markers until terminal + ... When hitting ^nested:, starts fresh HTTP walk with full budget + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +SELF-HEALING CACHE + [Documentation] Test self-healing: when ^nested: gets extended, marker is overwritten with ^hop: + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + # Second scan should see healed cache + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +CYCLE DETECTION IN CACHE WALK + [Documentation] Test cycle protection: per-walk seen-set keyed by URL string + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +REDIS TIMEOUT APPLIED + [Documentation] Test that redis_timeout setting is applied to Redis calls + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +TOP_URLS ZINCRBY CANONICAL + [Documentation] Test that ZINCRBY on top_urls uses canonical URL string (no markers) + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +RESERVATION LOCK TTL + [Documentation] Test that reservation lock on hash(orig) has correct TTL = settings.timeout + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +*** Keywords *** +Urlredirector Cache Setup + Run Dummy Http + Rspamd Redis Setup + +Urlredirector Cache Teardown + Rspamd Redis Teardown + Dummy Http Teardown + Terminate All Processes kill=True diff --git a/test/functional/cases/166_url_redirector_config.robot b/test/functional/cases/166_url_redirector_config.robot new file mode 100644 index 0000000000..102d1b191a --- /dev/null +++ b/test/functional/cases/166_url_redirector_config.robot @@ -0,0 +1,57 @@ +*** Settings *** +Suite Setup Urlredirector Config Setup +Suite Teardown Urlredirector Config Teardown +Library Process +Library ${RSPAMD_TESTDIR}/lib/rspamd.py +Resource ${RSPAMD_TESTDIR}/lib/rspamd.robot +Variables ${RSPAMD_TESTDIR}/lib/vars.py + +*** Variables *** +${CONFIG} ${RSPAMD_TESTDIR}/configs/url_redirector_no_intermediate.conf +${CHAIN_MESSAGE} ${RSPAMD_TESTDIR}/messages/chain_redirect.eml +${REDIS_SCOPE} Suite +${RSPAMD_SCOPE} Suite +${RSPAMD_URL_TLD} ${RSPAMD_TESTDIR}/../lua/unit/test_tld.dat +${SETTINGS} {symbols_enabled=[URL_REDIRECTOR_CHECK]} + +*** Test Cases *** +SAVE_INTERMEDIATE_REDIRECTORS_ONLY + [Documentation] Test save_intermediate_redirs={redirectors=true, non_redirectors=false} + ... Only redirector chain intermediates should be saved + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +SAVE_INTERMEDIATE_DISABLED + [Documentation] Test save_intermediate_redirs with both options disabled + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +DEFAULT_TIMEOUT_VALUE + [Documentation] Test default timeout value (8s) from settings + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +CUSTOM_HTTP_TIMEOUT + [Documentation] Test custom http_timeout setting overrides default + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +CUSTOM_REDIS_TIMEOUT + [Documentation] Test custom redis_timeout setting + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +REDIRECTOR_SYMBOL_DISABLED + [Documentation] Test behavior when redirector_symbol is not configured + Scan File ${CHAIN_MESSAGE} Flags=ext_urls Settings=${SETTINGS} + Expect Extended URL http://127.0.0.1:18080/hello + +*** Keywords *** +Urlredirector Config Setup + Run Dummy Http + Rspamd Redis Setup + +Urlredirector Config Teardown + Rspamd Redis Teardown + Dummy Http Teardown + Terminate All Processes kill=True diff --git a/test/functional/configs/url_redirector_chain.conf b/test/functional/configs/url_redirector_chain.conf new file mode 100644 index 0000000000..108d033866 --- /dev/null +++ b/test/functional/configs/url_redirector_chain.conf @@ -0,0 +1,17 @@ +.include(duplicate=append,priority=0) "{= env.TESTDIR =}/configs/plugins.conf" + +redis { + servers = "{= env.REDIS_ADDR =}:{= env.REDIS_PORT =}"; +} + +url_redirector { + redirector_hosts_map = "{= env.TESTDIR =}/configs/maps/redir.map"; + save_intermediate_redirs = { + redirectors = false; + non_redirectors = true; + }; + redirector_symbol = "URL_REDIRECTOR"; + timeout = 8; + http_timeout = 4; + redis_timeout = 2; +} diff --git a/test/functional/configs/url_redirector_no_intermediate.conf b/test/functional/configs/url_redirector_no_intermediate.conf new file mode 100644 index 0000000000..b3276ea532 --- /dev/null +++ b/test/functional/configs/url_redirector_no_intermediate.conf @@ -0,0 +1,14 @@ +.include(duplicate=append,priority=0) "{= env.TESTDIR =}/configs/plugins.conf" + +redis { + servers = "{= env.REDIS_ADDR =}:{= env.REDIS_PORT =}"; +} + +url_redirector { + redirector_hosts_map = "{= env.TESTDIR =}/configs/maps/redir.map"; + save_intermediate_redirs = { + redirectors = true; + non_redirectors = false; + }; + timeout = 8; +} diff --git a/test/functional/messages/chain_multipart.eml b/test/functional/messages/chain_multipart.eml new file mode 100644 index 0000000000..1600170f62 --- /dev/null +++ b/test/functional/messages/chain_multipart.eml @@ -0,0 +1,7 @@ +MIME-Version: 1.0 +Content-Type: text/plain; charset="utf-8" +Content-Transfer-Encoding: quoted-printable + +Test message with redirect chain +http://127.0.0.1:18080/chain1 +http://127.0.0.1:18080/redirect2 diff --git a/test/functional/messages/chain_redirect.eml b/test/functional/messages/chain_redirect.eml new file mode 100644 index 0000000000..6bbcd6748f --- /dev/null +++ b/test/functional/messages/chain_redirect.eml @@ -0,0 +1,3 @@ +Content-type: text/plain + +Test chain redirect http://127.0.0.1:18080/chain1 diff --git a/test/functional/util/dummy_http.py b/test/functional/util/dummy_http.py index c1dd97a28e..3358a0daed 100755 --- a/test/functional/util/dummy_http.py +++ b/test/functional/util/dummy_http.py @@ -96,12 +96,24 @@ class MainHandler(tornado.web.RequestHandler): elif path == "/redirect2": # Send an HTTP redirect to the bind address of the server self.redirect(f"{self.request.protocol}://{self.request.host}/redirect1") - elif self.path == "/redirect3": + elif path == "/redirect3": # Send an HTTP redirect to the bind address of the server self.redirect(f"{self.request.protocol}://{self.request.host}/redirect4") - elif self.path == "/redirect4": + elif path == "/redirect4": # Send an HTTP redirect to the bind address of the server self.redirect(f"{self.request.protocol}://{self.request.host}/redirect3") + elif path == "/chain1": + # Intermediate hop to chain2 + self.redirect(f"{self.request.protocol}://{self.request.host}/chain2") + elif path == "/chain2": + # Intermediate hop to chain3 + self.redirect(f"{self.request.protocol}://{self.request.host}/chain3") + elif path == "/chain3": + # Final hop + self.redirect(f"{self.request.protocol}://{self.request.host}/hello") + elif path == "/slow": + # Slow redirect + self.redirect(f"{self.request.protocol}://{self.request.host}/hello") else: self.send_response(200) self.set_header("Content-Type", "text/plain")