From 4976f6ade2a8fef7ba7a003bf2b25f8c6a1c49c2 Mon Sep 17 00:00:00 2001 From: Otto Moerbeek Date: Mon, 3 Apr 2023 14:57:47 +0200 Subject: [PATCH] Prep for rec-4.9.0-alpha1 --- .github/actions/spell-check/expect.txt | 1 + docs/secpoll.zone | 3 +- pdns/recursordist/docs/changelog/4.9.rst | 199 +++++++++++++++++++++ pdns/recursordist/docs/changelog/index.rst | 1 + pdns/recursordist/docs/performance.rst | 20 ++- pdns/recursordist/docs/settings.rst | 6 +- pdns/recursordist/docs/upgrade.rst | 20 ++- 7 files changed, 241 insertions(+), 9 deletions(-) create mode 100644 pdns/recursordist/docs/changelog/4.9.rst diff --git a/.github/actions/spell-check/expect.txt b/.github/actions/spell-check/expect.txt index a9008ccf60..43b7f2bacd 100644 --- a/.github/actions/spell-check/expect.txt +++ b/.github/actions/spell-check/expect.txt @@ -1368,6 +1368,7 @@ Tarnell taskqueue tbhandler tcely +TCounters tcp tcpconnecttimeouts tcpdump diff --git a/docs/secpoll.zone b/docs/secpoll.zone index 9c2b988f2a..abd2209fe2 100644 --- a/docs/secpoll.zone +++ b/docs/secpoll.zone @@ -1,4 +1,4 @@ -@ 86400 IN SOA pdns-public-ns1.powerdns.com. peter\.van\.dijk.powerdns.com. 2023033001 10800 3600 604800 10800 +@ 86400 IN SOA pdns-public-ns1.powerdns.com. peter\.van\.dijk.powerdns.com. 2023041201 10800 3600 604800 10800 @ 3600 IN NS pdns-public-ns1.powerdns.com. @ 3600 IN NS pdns-public-ns2.powerdns.com. @@ -346,6 +346,7 @@ recursor-4.8.1.security-status 60 IN TXT "3 Upgrade now recursor-4.8.2.security-status 60 IN TXT "3 Upgrade now, see https://doc.powerdns.com/recursor/security-advisories/powerdns-advisory-2023-02.html" recursor-4.8.3.security-status 60 IN TXT "3 Upgrade now, see https://doc.powerdns.com/recursor/security-advisories/powerdns-advisory-2023-02.html" recursor-4.8.4.security-status 60 IN TXT "1 OK" +recursor-4.9.0-alpha1.security-status 60 IN TXT "1 Unsupported pre-release" ; Recursor Debian recursor-3.6.2-2.debian.security-status 60 IN TXT "3 Upgrade now, see https://doc.powerdns.com/3/security/powerdns-advisory-2015-01/ and https://doc.powerdns.com/3/security/powerdns-advisory-2016-02/" diff --git a/pdns/recursordist/docs/changelog/4.9.rst b/pdns/recursordist/docs/changelog/4.9.rst new file mode 100644 index 0000000000..b0976cb126 --- /dev/null +++ b/pdns/recursordist/docs/changelog/4.9.rst @@ -0,0 +1,199 @@ +Changelogs for 4.9.X +==================== + +.. changelog:: + :version: 4.9.0-alpha1 + :released: 14th of April 2023 + + .. change:: + :tags: Improvements + :pullreq: 12710 + + Cleanup rcode enums: base one is 8 bit unsigned, extended one 16 bit unsigned + + .. change:: + :tags: Improvements + :pullreq: 12594 + + Sharded and shared packet cache. + + .. change:: + :tags: Improvements + :pullreq: 12709 + + More fine grained capping of packet cache TTL. + + .. change:: + :tags: Bug Fixes + :pullreq: 12655 + :tickets: 12486 + + Rework root priming code to allow multiple addresses per NS. + + .. change:: + :tags: Improvements + :pullreq: 10072,12716 + + Update Debian packaging for Recursor (Chris Hofstaedtler). + + .. change:: + :tags: Improvements + :pullreq: 12497 + + Unify shorthands for seconds in log messages (Josh Soref). + + .. change:: + :tags: Improvements + :pullreq: 12674 + + Validate: Stop passing shared pointers all the way down. + + .. change:: + :tags: Improvements + :pullreq: 12688 + + Re-establish "recursion depth is always increasing" invariant. + + .. change:: + :tags: Bug Fixes + :pullreq: 12672 + + Fix a dnsheader unaligned case. + + .. change:: + :tags: Improvements + :pullreq: 12550,12540,12524,12516,12515,12513,12502,12501,12462,12412,12401 + + OpenSSL 3.0 compatibility. + + .. change:: + :tags: Bug Fixes + :pullreq: 12554 + + Serve-stale-extensions works on 30s so an hour should be 120. (Andreas Jakum) + + .. change:: + :tags: Bug Fixes + :pullreq: 12539 + + Fix doc typo (Matt Nordhoff). + + .. change:: + :tags: Improvements + :pullreq: 12493 + + Only store NSEC3 records in aggressive cache if we expect them to be effective. + + .. change:: + :tags: Improvements + :pullreq: 11777 + + rec_control trace-regex: trace to a file or stdout instead of the general log. + + .. change:: + :tags: Bug Fixes + :pullreq: 12495 + + Logging tweaks (Josh Soref). + + .. change:: + :tags: Improvements + :pullreq: 12434 + + Unify trace logging code in syncres and validator. + + .. change:: + :tags: Improvements + :pullreq: 12446,12695 + + Stack protector for mthread stacks. + + .. change:: + :tags: Improvements + :pullreq: 12425 + + Change the way RD=0 forwarded queries are handled. + + .. change:: + :tags: Improvements + :pullreq: 12381 + + Enable FORTIFY_SOURCE=3 when supported by the compiler. + + .. change:: + :tags: Bug Fixes + :pullreq: 12419 + :tickets: 12374 + + Negcache dump code: close fd on fdopen fail. + + .. change:: + :tags: Improvements + :pullreq: 12396 + + Introduce a thread-safe version of stringerror(). + + .. change:: + :tags: Improvements + :pullreq: 12399 + :tickets: 11138 + + Name recursor threads consistently with a "rec/" prefix. + + .. change:: + :tags: Bug Fixes + :pullreq: 12392 + + Be more careful saving errno in makeClientSocket() and closesocket() + + .. change:: + :tags: Improvements + :pullreq: 12373 + + Rec: Warn on high (90%) mthread stack usage. + + .. change:: + :tags: Improvements + :pullreq: 12334,12691,12698 + + Rec: Generate EDE in more cases, specifically on unreachable auths or synthesized results. + + .. change:: + :tags: Bug Fixes + :pullreq: 12368 + + Add the 'parse packet from auth' error message to structured logging. + + .. change:: + :tags: Improvements + :pullreq: 12292 + + Wrap the CURL raw pointers in smart pointers. + + .. change:: + :tags: Improvements + :pullreq: 12318 + :tickets: 12241 + + Reorganization: move recursor specific files to recursordist. + + .. change:: + :tags: Improvements + :pullreq: 12193,12348,12323 + + Introducing TCounters. + + .. change:: + :tags: Improvements + :pullreq: 12120 + :tickets: 12090 + + If we encounter a loop in QM, continue with the next iteration. + + .. change:: + :tags: Improvements + :pullreq: 12121 + :tickets: 12080 + + More clear trace message for cache-only lookups. + diff --git a/pdns/recursordist/docs/changelog/index.rst b/pdns/recursordist/docs/changelog/index.rst index 2db2f634bd..56b0bd0e99 100644 --- a/pdns/recursordist/docs/changelog/index.rst +++ b/pdns/recursordist/docs/changelog/index.rst @@ -6,6 +6,7 @@ The changelogs for the recursor are split between release trains. .. toctree:: :maxdepth: 2 + 4.9 4.8 4.7 4.6 diff --git a/pdns/recursordist/docs/performance.rst b/pdns/recursordist/docs/performance.rst index ffa66b9f7b..3834a68eaf 100644 --- a/pdns/recursordist/docs/performance.rst +++ b/pdns/recursordist/docs/performance.rst @@ -15,21 +15,26 @@ See below for more information about the various caches. When deploying (large scale) IPv6, please be aware some Linux distributions leave IPv6 routing cache tables at very small default values. Please check and if necessary raise ``sysctl net.ipv6.route.max_size``. -Set :ref:`setting-threads` to your number of CPU cores minus the number of distributor threads (but values above 8 rarely improve performance). +Set :ref:`setting-threads` to your number of CPU cores minus the number of distributor threads. Threading and distribution of queries ------------------------------------- -When running with several threads, you can either ask PowerDNS to start one or more special threads to dispatch the incoming queries to the workers by setting :ref:`setting-pdns-distributes-queries` to true, or let the worker threads handle the incoming queries themselves. +When running with several threads, you can either ask PowerDNS to start one or more special threads to dispatch the incoming queries to the workers by setting :ref:`setting-pdns-distributes-queries` to ``yes``, or let the worker threads handle the incoming queries themselves. +The latter is the default since version 4.9.0. The dispatch thread enabled by :ref:`setting-pdns-distributes-queries` tries to send the same queries to the same thread to maximize the cache-hit ratio. If the incoming query rate is so high that the dispatch thread becomes a bottleneck, you can increase :ref:`setting-distributor-threads` to use more than one. -If :ref:`setting-pdns-distributes-queries` is set to false and either ``SO_REUSEPORT`` support is not available or the :ref:`setting-reuseport` directive is set to false, all worker threads share the same listening sockets. +If :ref:`setting-pdns-distributes-queries` is set to ``no`` and either ``SO_REUSEPORT`` support is not available or the :ref:`setting-reuseport` directive is set to ``no``, all worker threads share the same listening sockets. This prevents a single thread from having to handle every incoming queries, but can lead to thundering herd issues where all threads are awoken at once when a query arrives. -If ``SO_REUSEPORT`` support is available and :ref:`setting-reuseport` is set to true, separate listening sockets are opened for each worker thread and the query distributions is handled by the kernel, avoiding any thundering herd issue as well as preventing the distributor thread from becoming the bottleneck. +If ``SO_REUSEPORT`` support is available and :ref:`setting-reuseport` is set to ``yes``, which is the +default since version 4.9.0, separate listening sockets are opened for each worker thread and the query distributions is handled by the kernel, avoiding any thundering herd issue as well as preventing the distributor thread from becoming the bottleneck. + +On some systems setting :ref:`setting-reuseport` to ``yes`` does not have the desired effect. +If your systems shows imbalance in the number of queries processed per thread (as reported by the periodic statistics report), try switching :ref:`setting-reuseport` to ``no`` and/or setting :ref:`setting-pdns-distributes-queries` to ``yes``. .. versionadded:: 4.1.0 The :ref:`setting-cpu-map` parameter can be used to pin worker threads to specific CPUs, in order to keep caches as warm as possible and optimize memory access on NUMA systems. @@ -37,6 +42,13 @@ If ``SO_REUSEPORT`` support is available and :ref:`setting-reuseport` is set to .. versionadded:: 4.2.0 The :ref:`setting-distributor-threads` parameter can be used to run more than one distributor thread. +.. versionchanged:: 4.9.0 + The :ref:`setting-reuseport` parameter now defaults to ``yes``. + +.. versionchanged:: 4.9.0 + The :ref:`setting-pdns-distributes-queries` parameter now defaults to ``no``. + + MTasker and MThreads -------------------- diff --git a/pdns/recursordist/docs/settings.rst b/pdns/recursordist/docs/settings.rst index ba22422f98..427b39904f 100644 --- a/pdns/recursordist/docs/settings.rst +++ b/pdns/recursordist/docs/settings.rst @@ -42,12 +42,11 @@ In this case the address ``128.66.1.2`` is excluded from the addresses allowed a The number of records to cache in the aggressive cache. If set to a value greater than 0, the recursor will cache NSEC and NSEC3 records to generate negative answers, as defined in :rfc:`8198`. To use this, DNSSEC processing or validation must be enabled by setting `dnssec`_ to ``process``, ``log-fail`` or ``validate``. -.. _setting-aggressive-cache-max-nsec3-hit-ratio: +.. _setting-aggressive-cache-min-nsec3-hit-ratio: ``aggressive-cache-min-nsec3-hit-ratio`` ---------------------------------------- - -.. versionadded: 4.9.0 +.. versionadded:: 4.9.0 - Integer - Default: 2000 @@ -1611,6 +1610,7 @@ i.e. setting ``packetcache-ttl=15`` and keeping ``packetcache-negative-ttl`` at Maximum number of seconds to cache an answer indicating a failure to resolve in the packet cache. Before version 4.6.0 only ``ServFail`` answers were considered as such. Starting with 4.6.0, all responses with a code other than ``NoError`` and ``NXDomain``, or without records in the answer and authority sections, are considered as a failure to resolve. +Since 4.9.0, negative answers are handled separately from resolving failures. .. versionchanged:: 4.0.0 diff --git a/pdns/recursordist/docs/upgrade.rst b/pdns/recursordist/docs/upgrade.rst index 5f5e2b6c07..871186e6d3 100644 --- a/pdns/recursordist/docs/upgrade.rst +++ b/pdns/recursordist/docs/upgrade.rst @@ -15,6 +15,25 @@ This is now resolved, but has the consequence that some metrics (in particular r This affects the results shown by ``rec_control get-qtypelist`` and the ``response-by-qtype``, ``response-sizes`` and ``response-by-rcode`` items returned by the ``/api/v1/servers/localhost/statistics`` API endpoint. Additionally, most ``RCodes`` and ``QTypes`` that are marked ``Unassigned``, ``Reserved`` or ``Obsolete`` by IANA are not accounted, to reduce the memory consumed by these metrics. +New settings +~~~~~~~~~~~~ +- The :ref:`setting-packetcache-negative-ttl` settings to control the TTL of negative (NxDomain or NoData) answers in the packet cache has been introduced. +- The :ref:`setting-stack-cache-size` setting to control the number of allocated mthread stacks has been introduced. +- The :ref:`setting-packetcache-shards` settings to control the number of shards in the packet cache has been introduced. +- The :ref:`setting-aggressive-cache-min-nsec3-hit-ratio` setting to control which NSEC3 records are stored in the aggressive NSEC cache has been introduced. + +Changed settings +~~~~~~~~~~~~~~~~ +The first two settings below have effect on the way the recursor distributes queries over threads. +In some rare cases, this can have negative performance impact. +In those cases it might be needed to change these settings. +See :doc:`performance`. + +- The :ref:`setting-pdns-distributes-queries` default has been changed to ``no``. +- The :ref:`setting-reuseport` default has been changed to ``yes``. + +- The :ref:`setting-packetcache-ttl` default has been changed to 24 hours. + :program:`rec_control` ^^^^^^^^^^^^^^^^^^^^^^ The ``trace_regex`` subcommand has been changed to take a file argument. @@ -61,7 +80,6 @@ Additionally, the ``dump-edns`` command now only lists IPs that have a not OK s The ``dump-nsspeeds`` command has changed format to make it more readable and lists the last round trip time recorded for each address. The ``get-proxymapping-stats`` and ``get-remotelogger-stats`` subcommands have been added. - 4.7.2 to 4.7.3 -------------- -- 2.47.2