From: Otto Moerbeek Date: Thu, 23 Oct 2025 09:30:22 +0000 (+0200) Subject: Docs and bit of a dirty hack: flush protobuf queue immediately in test mode X-Git-Tag: rec-5.4.0-alpha1~103^2~5 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=ff0c6582f0b8486ed4b2a9958227ebd72f72c42e;p=thirdparty%2Fpdns.git Docs and bit of a dirty hack: flush protobuf queue immediately in test mode Saves 50s on a Protobuf test run (180s before) Signed-off-by: Otto Moerbeek --- diff --git a/pdns/recursordist/docs/performance.rst b/pdns/recursordist/docs/performance.rst index 3a403cb67a..e92d2e0cf8 100644 --- a/pdns/recursordist/docs/performance.rst +++ b/pdns/recursordist/docs/performance.rst @@ -408,14 +408,90 @@ An example where various Lua related events can be seen: There is no packet cache hit, so SyncRes is called which does a couple of outgoing queries. +.. _opentelemetry_tracing: OpenTelemetry Traces ^^^^^^^^^^^^^^^^^^^^ -OpenTelemetry Traces are generated by setting :ref:`setting-yaml-recursor.event_trace_enabled` or by using the :doc:`rec_control ` subcommand ``set-event-trace-enabled``. +OpenTelemetry Traces are generated by setting :ref:`setting-yaml-recursor.event_trace_enabled` or by using the :doc:`rec_control ` subcommand ``set-event-trace-enabled`` to a value that includes 4. + :program:`Recursor` will set the ``openTelemetryData`` field of ``dnsmessage.proto`` messages generated to contain OpenTelemetry Traces, encoded as Protobuf data. The encoding used is defined in https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto. +OpenTelemetry Trace Conditions +"""""""""""""""""""""""""""""" + +Starting with version 5.4.0, the OpenTelemetry Trace data is only generated if :ref:`setting-yaml-recursor.event_trace_enabled` includes 4 *and* a matching condition from :ref:`setting-yaml-logging.opentelemetry_trace_conditions` evaluates to ``true``. + +For a match to occur, the origin address of the query should match a subnet in `acls`. +If the proxy protocol is used, the source specified by the proxy protocol header is used to match. +If Table Based Proxy Mapping is active, the mapped address is used. + +If a matching condition is found, the corresponding subconditions are evaluated. +If all mentioned subconditions are satisfied, the condittion evalutes to ``true`` and OpenTelemetry Trace data is generated. +If a subcondition is absent, it is not relevant. +The following subcondtions can be specified: + +- ``qnames``: a suffixmatch with the incoming qname is done against the suffixes specified in ``qnames``. +- ``qtypes``: the qtype of the incoming query must be listed in ``qtypes`` +- ``qid``: the query id of the incoming query must match ``qid``. +- ``edns_option_required``: the incoing query must have an EDSN option specifying the TraceID. +- ``traceid_only`` if ``true`` only the TraceID is picked up from EDNS data (if present) and placed into the protobuf `openTelemetryTraceID` field; no detailed `openTelemetryData` is produced. + +In the following example two conditions are specified: + +.. code-block:: yaml + + logging: + opentelemetry_trace_conditions: + - acls: [127.0.0.1, '::1'] + - acls: [192.168.178.0/24] + qnames: [a.very.specific.suffix] + qtypes: ['A', 'AAAA'] + qid: 1234 + edns_option_required: true + +The first condition specifies that all queries coming from ``127.0.0.1`` or ``::1`` should generate trace information. +No subconditionss are relevant. + +The second condition specifies that queries coming from the ``192.167.178.0/24`` subnet should generate trace info only if the query name is ``a.very.specific.suffix`` (or has that suffix), the query type is ``A`` or ``AAAA``, the query id is ``1234`` and the EDNS option containing a TraceID is present. + +Queries coming from an IP not matching any of the mentioned subnets will not generate OpenTelemetry Trace information. + +Note that only the source IP is used to select a condition to evaluate. +It is undefined what happens if a subnet occurs multiple times in all :ref:`setting-yaml-logging.opentelemetry_trace_conditions`. + +To generate OpenTelemetry Trace information for all queries (the 5.3.x behaviour), the follwing condition can be used: + +.. code-block:: yaml + + logging: + opentelemetry_trace_conditions: + - acls: ['0.0.0.0/0', '::/0'] + +To generate OpenTelemetry Trace information for all queries coming from ``127.0.0.1`` but only pick up the EDNS specified TraceID from all other queries: + +.. code-block:: yaml + + logging: + opentelemetry_trace_conditions: + - acls: ['0.0.0.0/0', '::/0'] + traceid_only: true + - acls: [127.0.0.1] + +To include a TraceID in a query ``sdig`` can be used, or a modern ``dig``, specifying the EDNS option code in decimal and the payload in hex: + +.. code-block:: sh + + dig +ednsopt=65500:00000102030405060708090a0b0c0d0e0fff ... + +The first 4 zeroes after the colon specify the version byte and a reserved byte, followed by a 16 byte (32 hex characters) TraceID. +Note that the EDNS option number 65500 is subject to change in the future. +To also add a parent SpanID to the EDNS value, append 8 bytes (16 hex characters) more. + +Example OpenTelemetry Trace in in JSON representation +""""""""""""""""""""""""""""""""""""""""""""""""""""" + The example decoder in https://github.com/PowerDNS/pdns/blob/master/contrib/ProtobufLogger.py displays a JSON representation of the data and optionally submits the data (in Protobuf format) to an OpenTelemetry trace collector using a POST to a URL given on the command line. An example, encoded in JSON: diff --git a/pdns/recursordist/pdns_recursor.cc b/pdns/recursordist/pdns_recursor.cc index d8a4587329..d13d6b0e5a 100644 --- a/pdns/recursordist/pdns_recursor.cc +++ b/pdns/recursordist/pdns_recursor.cc @@ -2183,7 +2183,7 @@ bool matchOTConditions(RecEventTrace& eventTrace, const std::unique_ptrd_writer.write(data); +#ifdef RECURSOR + extern bool g_regressionTestMode; + if (g_regressionTestMode) { + runtime->d_writer.flush(runtime->d_socket->getHandle()); + } +#endif ++runtime->d_stats.d_queued; return Result::Queued; } -void RemoteLogger::maintenanceThread() +void RemoteLogger::maintenanceThread() { try { #ifdef RECURSOR diff --git a/regression-tests.recursor-dnssec/test_Protobuf.py b/regression-tests.recursor-dnssec/test_Protobuf.py index 169e763b95..6397ff1c8a 100644 --- a/regression-tests.recursor-dnssec/test_Protobuf.py +++ b/regression-tests.recursor-dnssec/test_Protobuf.py @@ -371,6 +371,7 @@ recursor: - zone: example file: configs/%s/example.zone event_trace_enabled: 4 + devonly_regression_test_mode: true logging: protobuf_servers: - servers: [127.0.0.1:%s, 127.0.0.1:%s] @@ -812,6 +813,7 @@ class ProtobufProxyMappingTest(TestRecursorProtobuf): _config_template = """ auth-zones=example=configs/%s/example.zone allow-from=3.4.5.0/24 + devonly-regression-test-mode """ % _confdir _lua_config_file = """ @@ -849,7 +851,8 @@ class ProtobufProxyMappingLogMappedTest(TestRecursorProtobuf): _confdir = 'ProtobufProxyMappingLogMapped' _config_template = """ auth-zones=example=configs/%s/example.zone - allow-from=3.4.5.0/0" + devonly-regression-test-mode + allow-from=3.4.5.0/0v """ % _confdir _lua_config_file = """ @@ -889,6 +892,7 @@ class ProtobufProxyTest(TestRecursorProtobuf): auth-zones=example=configs/%s/example.zone proxy-protocol-from=127.0.0.1/32 allow-from=127.0.0.1,6.6.6.6 +devonly-regression-test-mode """ % _confdir def testA(self): @@ -923,6 +927,7 @@ class ProtobufProxyWithProxyByTableTest(TestRecursorProtobuf): auth-zones=example=configs/%s/example.zone proxy-protocol-from=127.0.0.1/32 allow-from=3.4.5.6 +devonly-regression-test-mode """ % _confdir _lua_config_file = """ @@ -962,6 +967,7 @@ class ProtobufProxyWithProxyByTableLogMappedTest(TestRecursorProtobuf): auth-zones=example=configs/%s/example.zone proxy-protocol-from=127.0.0.1/32 allow-from=3.4.5.6 +devonly-regression-test-mode """ % _confdir _lua_config_file = """ @@ -1006,6 +1012,7 @@ class OutgoingProtobufDefaultTest(TestRecursorProtobuf): qname-minimization=no max-cache-ttl=600 loglevel=9 + devonly-regression-test-mode """ _lua_config_file = """ outgoingProtobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}) @@ -1075,6 +1082,7 @@ class OutgoingProtobufWithECSMappingTest(TestRecursorProtobuf): # this is to not let . queries interfere max-cache-ttl=600 loglevel=9 + devonly-regression-test-mode """ _lua_config_file = """ outgoingProtobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}) @@ -1177,6 +1185,7 @@ class OutgoingProtobufNoQueriesTest(TestRecursorProtobuf): qname-minimization=no max-cache-ttl=600 loglevel=9 + devonly-regression-test-mode """ _lua_config_file = """ outgoingProtobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}, { logQueries=false, logResponses=true }) @@ -1233,6 +1242,7 @@ class ProtobufMasksTest(TestRecursorProtobuf): _confdir = 'ProtobufMasks' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.zone""" % _confdir _protobufMaskV4 = 4 _protobufMaskV6 = 128 @@ -1270,6 +1280,7 @@ class ProtobufQueriesOnlyTest(TestRecursorProtobuf): _confdir = 'ProtobufQueriesOnly' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.zone""" % _confdir _lua_config_file = """ protobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}, { logQueries=true, logResponses=false } ) @@ -1327,6 +1338,7 @@ class ProtobufTaggedOnlyTest(TestRecursorProtobuf): _confdir = 'ProtobufTaggedOnly' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.zone""" % _confdir _lua_config_file = """ protobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}, { logQueries=true, logResponses=true, taggedOnly=true } ) @@ -1522,6 +1534,7 @@ class ProtobufTagCacheFFITest(ProtobufTagCacheBase): __test__ = True _confdir = 'ProtobufTagCacheFFI' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.zone""" % _confdir _lua_config_file = """ protobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}, { logQueries=false, logResponses=true } ) @@ -1633,6 +1646,7 @@ class ProtobufExportTypesTest(TestRecursorProtobuf): _confdir = 'ProtobufExportTypes' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.zone""" % _confdir _lua_config_file = """ protobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}, { exportTypes={"AAAA", "MX", "SPF", "SRV", "TXT"} } ) @@ -1690,6 +1704,7 @@ class ProtobufTaggedExtraFieldsTest(TestRecursorProtobuf): _confdir = 'ProtobufTaggedExtraFields' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.zone""" % _confdir _lua_config_file = """ protobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}, { logQueries=true, logResponses=true } ) @@ -1786,6 +1801,7 @@ class ProtobufTaggedExtraFieldsFFITest(ProtobufTaggedExtraFieldsTest): """ _confdir = 'ProtobufTaggedExtraFieldsFFI' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.zone""" % _confdir _lua_config_file = """ protobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}, { logQueries=true, logResponses=true } ) @@ -1824,6 +1840,7 @@ class ProtobufRPZTest(TestRecursorProtobuf): _confdir = 'ProtobufRPZ' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.rpz.zone""" % _confdir _lua_config_file = """ protobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}, { logQueries=true, logResponses=true } ) @@ -1903,6 +1920,7 @@ class ProtobufRPZTagsTest(TestRecursorProtobuf): _confdir = 'ProtobufRPZTags' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.rpz.zone""" % _confdir _tags = ['tag1', 'tag2'] _tags_from_gettag = ['tag1-from-gettag', 'tag2-from-gettag'] @@ -1971,6 +1989,7 @@ class ProtobufMetaFFITest(TestRecursorProtobuf): """ _confdir = 'ProtobufMetaFFI' _config_template = """ + devonly-regression-test-mode auth-zones=example=configs/%s/example.zone""" % _confdir _lua_config_file = """ protobufServer({"127.0.0.1:%d", "127.0.0.1:%d"}, { logQueries=true, logResponses=true } )