From: Remi Gacogne Date: Mon, 29 Mar 2021 15:14:55 +0000 (+0200) Subject: dnsdist: Document internal design, add tables and pictures X-Git-Tag: dnsdist-1.6.0-rc1~18^2 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=refs%2Fpull%2F10244%2Fhead;p=thirdparty%2Fpdns.git dnsdist: Document internal design, add tables and pictures --- diff --git a/.github/actions/spell-check/expect.txt b/.github/actions/spell-check/expect.txt index 320499b889..4130fe17b0 100644 --- a/.github/actions/spell-check/expect.txt +++ b/.github/actions/spell-check/expect.txt @@ -1093,6 +1093,7 @@ Neuf newcontent nextval nf +nginx nic nimber Nixu diff --git a/pdns/dnsdistdist/docs/advanced/ebpf.rst b/pdns/dnsdistdist/docs/advanced/ebpf.rst index 75d59cf023..ba3b50f2cd 100644 --- a/pdns/dnsdistdist/docs/advanced/ebpf.rst +++ b/pdns/dnsdistdist/docs/advanced/ebpf.rst @@ -2,11 +2,14 @@ eBPF Socket Filtering ===================== :program:`dnsdist` can use `eBPF `_ socket filtering on recent Linux kernels (4.1+) built with eBPF support (``CONFIG_BPF``, ``CONFIG_BPF_SYSCALL``, ideally ``CONFIG_BPF_JIT``). -This feature might require an increase of the memory limit associated to a socket, via the sysctl setting ``net.core.optmem_max``. -When attaching an eBPF program to a socket, the size of the program is checked against this limit, and the default value might not be enough. -Large map sizes might also require an increase of ``RLIMIT_MEMLOCK``. -This feature allows dnsdist to ask the kernel to discard incoming packets in kernel-space instead of them being copied to userspace just to be dropped, thus being a lot of faster. +This feature allows dnsdist to ask the kernel to discard incoming packets in kernel-space instead of them being copied to userspace just to be dropped, thus being a lot of faster. The current implementation supports dropping UDP and TCP queries based on the source IP and UDP datagrams on exact DNS names. We have not been able to implement suffix matching yet, due to a limit on the maximum number of EBPF instructions. + +The following figure show the CPU usage of dropping around 20k qps of traffic, first in userspace (34 to 36) then in kernel space with eBPF (37 to 39). The spikes are caused because the drops are triggered by dynamic rules, so the first spike is the abuse traffic before a rule is automatically inserted, and the second spike is because the rule expires automatically after 60s before being inserted again. + +.. figure:: ../imgs/ebpf_drops.png + :align: center + :alt: eBPF in action The BPF filter can be used to block incoming queries manually:: @@ -62,4 +65,6 @@ They can be unregistered at a later point using the :func:`unregisterDynBPFFilte Since 1.6.0, the default BPF filter set via :func:`setDefaultBPFFilter` will automatically get used when a dynamic block is inserted via a :ref:`DynBlockRulesGroup`. -This feature has been successfully tested on Arch Linux, Arch Linux ARM, Fedora Core 23 and Ubuntu Xenial +That feature might require an increase of the memory limit associated to a socket, via the sysctl setting ``net.core.optmem_max``. +When attaching an eBPF program to a socket, the size of the program is checked against this limit, and the default value might not be enough. +Large map sizes might also require an increase of ``RLIMIT_MEMLOCK``. diff --git a/pdns/dnsdistdist/docs/advanced/ecs.rst b/pdns/dnsdistdist/docs/advanced/ecs.rst deleted file mode 100644 index c6aee687aa..0000000000 --- a/pdns/dnsdistdist/docs/advanced/ecs.rst +++ /dev/null @@ -1,18 +0,0 @@ -Using EDNS Client Subnet ------------------------- - -In order to provide the downstream server with the address of the real client, or at least the one talking to dnsdist, the ``useClientSubnet`` parameter can be used when creating a :func:`new server `. -This parameter indicates whether an EDNS Client Subnet option should be added to the request. -If the incoming request already contains an EDNS Client Subnet value, it will not be overridden unless :func:`setECSOverride` is set to ``true``. -The default source prefix-length is 24 for IPv4 and 56 for IPv6, meaning that for a query received from 192.0.2.42, the EDNS Client Subnet value sent to the backend will be 192.0.2.0. -This can be changed with :func:`setECSSourcePrefixV4` and :func:`setECSSourcePrefixV6`. - -In addition to the global settings, rules and Lua bindings can alter this behavior per query: - - * calling :func:`SetDisableECSAction` or setting ``dq.useECS`` to ``false`` prevents the sending of the ECS option. - * calling :func:`SetECSOverrideAction` or setting ``dq.ecsOverride`` will override the global :func:`setECSOverride` value. - * calling :func:`SetECSPrefixLengthAction(v4, v6)` or setting ``dq.ecsPrefixLength`` will override the global :func:`setECSSourcePrefixV4()` and :func:`setECSSourcePrefixV6()` values. - -In effect this means that for the EDNS Client Subnet option to be added to the request, ``useClientSubnet`` should be set to ``true`` for the backend used (default to ``false``) and ECS should not have been disabled by calling :func:`SetDisableECSAction` or setting ``dq.useECS`` to ``false`` (default to true). - -Note that any trailing data present in the incoming query is removed when an OPT (or XPF) record has to be inserted. diff --git a/pdns/dnsdistdist/docs/advanced/index.rst b/pdns/dnsdistdist/docs/advanced/index.rst index 4698980c19..a028a9292f 100644 --- a/pdns/dnsdistdist/docs/advanced/index.rst +++ b/pdns/dnsdistdist/docs/advanced/index.rst @@ -7,12 +7,10 @@ These chapters contain information on the advanced features of dnsdist :maxdepth: 2 acl + passing-source-address teeaction luaaction timedipsetrule - ecs - xpf - proxyprotocol qpslimits ebpf tuning @@ -22,3 +20,4 @@ These chapters contain information on the advanced features of dnsdist out-of-order ocsp-stapling tls-sessions-management + internal-design diff --git a/pdns/dnsdistdist/docs/advanced/internal-design.rst b/pdns/dnsdistdist/docs/advanced/internal-design.rst new file mode 100644 index 0000000000..3c3f8a8ac7 --- /dev/null +++ b/pdns/dnsdistdist/docs/advanced/internal-design.rst @@ -0,0 +1,38 @@ +Internal Design +=============== + +This part of the documentation is intended for developers interested in understanding how the actual code works, and might not be of much interest to regular users. + +UDP design +---------- + +.. figure:: ../imgs/DNSDistUDP.png + :align: center + :alt: DNSDist UDP design + +For UDP queries, dnsdist stores the initial ID in a per-backend table called *IDState*. That ID then replaced by one derived from a counter before forwarding the query to the backend, to prevent duplicated IDs sent clients from making it to the backend. +When the response is received, dnsdist uses the ID sent by the backend to find the corresponding *IDState* and restores the initial ID, as well as some flags if needed, before sending the response to the client. + +That design means that there is a maximum of 65535 in-flight UDP queries per backend. It can actually be even less than that if :func:`setMaxUDPOutstanding` is set to a lower value, for example to reduce the overall memory usage. + +Note that the source address and port used to contact a given backend is set at startup, for performance reasons, and then only changes on reconnect. There might be more than one socket, and thus several ports, if the ``sockets`` parameter was set to a higher value than 1 on the :func:`newServer` directive. + +TCP / DoT design +---------------- + +.. figure:: ../imgs/DNSDistTCP.png + :align: center + :alt: DNSDist TCP and DoT design + +For TCP and DoT, a single thread is created for each :func:`addLocal` and :func:`addTLSLocal` directive, listening to the incoming TCP sockets, accepting new connections and distributing them over a pipe to the TCP worker threads. These threads handle both the TCP connection with the client and the one with the backend. + +DoH design +---------- + +.. figure:: ../imgs/DNSDistDoH.png + :align: center + :alt: DNSDist DoH design + +For DoH, two threads are created for each :func:`addDOHLocal` directive, one handling the TLS and HTTP layers, then passing the queries to the second one over a pipe. The second thread does DNS processing, applying rules and forwarding the query to the backend if needed, over UDP. +Note that even if the query does not need to be passed to a backend (cache-hit, self-generated answer), the response will be passed back to the first thread via a pipe, since only that thread deals with the client. +If the response comes from a backend, it will be picked up by the regular UDP listener for that backend, the corresponding *IDState* object located, and the response sent to the first thread over a pipe. diff --git a/pdns/dnsdistdist/docs/advanced/passing-source-address.rst b/pdns/dnsdistdist/docs/advanced/passing-source-address.rst new file mode 100644 index 0000000000..ea006fb58d --- /dev/null +++ b/pdns/dnsdistdist/docs/advanced/passing-source-address.rst @@ -0,0 +1,104 @@ +Passing the source address to the backend +========================================= + +dnsdist, as a load-balancer, receives the UDP datagrams and terminates the TCP connections with the client. It therefore knows the source IP address and port of that client, as well as the original destination address, port, and protocol. +Very often the backend needs to know that information as well, to pass EDNS Client Subnet to an authoritative server, to do GeoIP-based processing or even custom filtering. + +There are several ways to pass that information using dnsdist: EDNS Client Subnet, X-Proxied-For and the Proxy Protocol. + +Using EDNS Client Subnet +------------------------ + +EDNS Client Subnet (ECS) is a standardized EDNS option designed to pass a bit of information about the client from a resolver to authoritative servers. While it was not designed with our use-case in mind, it can be used by dnsdist to send the source IP, but only the source IP, to its backend. + +In order to provide the downstream server with the address of the real client, or at least the one talking to dnsdist, the ``useClientSubnet`` parameter can be used when creating a :func:`new server `. This parameter indicates whether an EDNS Client Subnet option should be added to the request. + +The default source prefix-length is 24 for IPv4 and 56 for IPv6, meaning that for a query received from 192.0.2.42, the EDNS Client Subnet value sent to the backend will be 192.0.2.0. +This can be changed with :func:`setECSSourcePrefixV4` and :func:`setECSSourcePrefixV6`. + +If the incoming request already contains an EDNS Client Subnet value, it will not be overridden unless :func:`setECSOverride` is set to ``true``. + +In addition to the global settings, rules and Lua bindings can alter this behavior per query: + + * calling :func:`SetDisableECSAction` or setting ``dq.useECS`` to ``false`` prevents the sending of the ECS option. + * calling :func:`SetECSOverrideAction` or setting ``dq.ecsOverride`` will override the global :func:`setECSOverride` value. + * calling :func:`SetECSPrefixLengthAction(v4, v6)` or setting ``dq.ecsPrefixLength`` will override the global :func:`setECSSourcePrefixV4()` and :func:`setECSSourcePrefixV6()` values. + +In effect this means that for the EDNS Client Subnet option to be added to the request, ``useClientSubnet`` should be set to ``true`` for the backend used (default to ``false``) and ECS should not have been disabled by calling :func:`SetDisableECSAction` or setting ``dq.useECS`` to ``false`` (default to true). + +Note that any trailing data present in the incoming query is removed when an OPT (or XPF) record has to be inserted. + +In addition to the drawback that it can only pass the source IP address, and the fact that it needs to override any existing ECS option, adding that option requires parsing and editing the query, as well as parsing and editing the response in most cases. + ++----------------------------+-------------------------------------------------+ +| Payload | Required processing | ++============================+=================================================+ +| Query, no EDNS | add an OPT record | ++----------------------------+-------------------------------------------------+ +| Query, EDNS without ECS | edit the OPT record to add an ECS option | ++----------------------------+-------------------------------------------------+ +| Query, ECS | edit the OPT record to overwrite the ECS option | ++----------------------------+-------------------------------------------------+ +| Response, no EDNS | none | ++----------------------------+-------------------------------------------------+ +| Response, EDNS without ECS | remove the OPT record if needed | ++----------------------------+-------------------------------------------------+ +| Response, EDNS with ECS | remove or edit the ECS option if needed | ++----------------------------+-------------------------------------------------+ + +X-Proxied-For +------------- + +The experimental XPF record (from `draft-bellis-dnsop-xpf `_) is an alternative to the use of EDNS Client Subnet which has the advantages of preserving any existing EDNS Client Subnet value sent by the client, and of passing along the original destination address, as well as the initial source and destination ports. + +In order to provide the downstream server with the address of the real client, or at least the one talking to dnsdist, the ``addXPF`` parameter can be used when creating a :func:`new server `. +This parameter indicates whether an XPF record shall be added to the query. Since that record is experimental, there is currently no option code assigned to it, and therefore one needs to be specified as an argument to the ``addXPF`` parameter. + +If the incoming request already contains a XPF record, it will not be overwritten. Instead a new one will be added to the query and the existing one will be preserved. +That might be an issue by allowing clients to spoof their source address by adding a forged XPF record to their query. That can be prevented by using a rule to drop incoming queries containing a XPF record (in that example the 65280 option code has been assigned to XPF): + + addAction(RecordsTypeCountRule(DNSSection.Additional, 65280, 1, 65535), DropAction()) + +Proxy Protocol +-------------- + +The Proxy Protocol has been designed by the HAProxy folks for HTTP over TCP, but is generic enough to be used in other places, and is a de-facto standard with implementations in nginx and postfix, for example. +It works by pre-pending a small header at the very beginning of a UDP datagram or TCP connection, which holds the initial source and destination addresses and ports, and can also contain several custom values in a Type-Length-Value format. More information about the Proxy Protocol can be found at https://www.haproxy.org/download/2.2/doc/proxy-protocol.txt + +In order to use it in dnsdist, the ``useProxyProtocol`` parameter can be used when creating a :func:`new server `. +This parameter indicates whether a Proxy Protocol version 2 (binary) header should be prepended to the query before forwarding it to the backend, over UDP or TCP. +Such a Proxy Protocol header can also be passed from the client to dnsdist, using :func:`setProxyProtocolACL` to specify which clients to accept it from. +If :func:`setProxyProtocolApplyACLToProxiedClients` is set (default is false), the general ACL will be applied to the source IP address as seen by dnsdist first, but also to the source IP address provided in the Proxy Protocol header. + +Custom values can be added to the header via :meth:`DNSQuestion:addProxyProtocolValue`, :meth:`DNSQuestion:setProxyProtocolValues`, :func:`SetAdditionalProxyProtocolValueAction` and :func:`SetProxyProtocolValuesAction`. +Be careful that Proxy Protocol values are sent once at the beginning of the TCP connection for TCP and DoT queries. +That means that values received on an incoming TCP connection will be inherited by subsequent queries received over the same incoming TCP connection, if any, but values set to a query will not be inherited by subsequent queries. +Please also note that the maximum size of a Proxy Protocol header dnsdist is willing to accept is 512 bytes by default, although it can be set via :func:`setProxyProtocolMaximumPayloadSize`. + +dnsdist 1.5.0 only supports outgoing Proxy Protocol. Support for parsing incoming Proxy Protocol headers has been implemented in 1.6.0, except for DoH where it does not make sense anyway, since HTTP headers already provide a mechanism for that. + +Influence on caching +-------------------- + +When dnsdist's packet cache is in use, it is important to note that the cache lookup is done **after** adding ECS, because it prevents serving the same response to clients from different subnets when ECS is passed to an authoritative server doing GeoIP, or to a backend doing custom filtering. +However that means that passing a narrow ECS source will effectively kill dnsdist's cache ratio, since a given answer will only be a cache hit for clients in the same ECS subnet. Therefore, unless a broad ECS source (greater than 24, for example) is used, it's better to disable caching. + +One exception to that rule is the zero-scope feature, which allows dnsdist to detect that a response sent by the backend has a 0-scope ECS value, indicating that the answer is not ECS-specific and can be used for all clients. dnsdist will then store the answer in its packet cache using the initial query, before ECS has been added. +For that feature to work, dnsdist will look up twice into the packet cache when a query arrives, first without and then with ECS. That way, when most of the responses sent by a backend are not ECS-specific and can be served to all clients, dnsdist will still be able to have a great cache-hit ratio for non ECS-specific entries. + +That feature is enabled by setting ``disableZeroScope=false`` on :func:`newServer` (default) and ``parseECS=true`` on :func:`newPacketCache` (not the default). + + +Things are different for XPF and the proxy protocol, because dnsdist then does the cache lookup **before** adding the payload. It means that caching can still be enabled as long as the response is not source-dependant, but should be disabled otherwise. + ++------------------+----------+---------------------+----------------+------------------------+ +| Protocol | Standard | Require DNS parsing | Contains ports | Caching | ++==================+==========+=====================+================+========================+ +| ECS | Yes | Query and response | No | Only with broad source | ++------------------+----------+---------------------+----------------+------------------------+ +| ECS (zero-scope) | Yes | Query and response | No | Yes | ++------------------+----------+---------------------+----------------+------------------------+ +| XPF | No | Query | Yes | Depends on the backend | ++------------------+----------+---------------------+----------------+------------------------+ +| Proxy Protocol | No | No | Yes | Depends on the backend | ++------------------+----------+---------------------+----------------+------------------------+ diff --git a/pdns/dnsdistdist/docs/advanced/proxyprotocol.rst b/pdns/dnsdistdist/docs/advanced/proxyprotocol.rst deleted file mode 100644 index bb24b3fe9b..0000000000 --- a/pdns/dnsdistdist/docs/advanced/proxyprotocol.rst +++ /dev/null @@ -1,14 +0,0 @@ -Using the Proxy Protocol ------------------------- - -In order to provide the downstream server with the address of the real client, or at least the one talking to dnsdist, the ``useProxyProtocol`` parameter can be used when creating a :func:`new server `. -This parameter indicates whether a Proxy Protocol version 2 (binary) header should be prepended to the query before forwarding it to the backend, over UDP or TCP. This header contains the initial source and destination addresses and ports, and can also contain several custom values in a Type-Length-Value format. More information about the Proxy Protocol can be found at https://www.haproxy.org/download/2.2/doc/proxy-protocol.txt -Such a Proxy Protocol header can also be passed from the client to dnsdist, using :func:`setProxyProtocolACL` to specify which clients to accept it from. -If :func:`setProxyProtocolApplyACLToProxiedClients` is set (default is false), the general ACL will be applied to the source IP address as seen by dnsdist first, but also to the source IP address provided in the Proxy Protocol header. - -Custom values can be added to the header via :meth:`DNSQuestion:addProxyProtocolValue`, :meth:`DNSQuestion:setProxyProtocolValues`, :func:`SetAdditionalProxyProtocolValueAction` and :func:`SetProxyProtocolValuesAction`. -Be careful that Proxy Protocol values are sent once at the beginning of the TCP connection for TCP and DoT queries. -That means that values received on an incoming TCP connection will be inherited by subsequent queries received over the same incoming TCP connection, if any, but values set to a query will not be inherited by subsequent queries. -Please also note that the maximum size of a Proxy Protocol header dnsdist is willing to accept is 512 bytes by default, although it can be set via :func:`setProxyProtocolMaximumPayloadSize`. - -dnsdist 1.5.0 only supports outgoing Proxy Protocol. Support for parsing incoming Proxy Protocol headers has been implemented in 1.6.0, except for DoH where it does not make sense anyway, since HTTP headers already provide a mechanism for that. diff --git a/pdns/dnsdistdist/docs/advanced/tls-sessions-management.rst b/pdns/dnsdistdist/docs/advanced/tls-sessions-management.rst index 89cae86fab..0479f90964 100644 --- a/pdns/dnsdistdist/docs/advanced/tls-sessions-management.rst +++ b/pdns/dnsdistdist/docs/advanced/tls-sessions-management.rst @@ -6,6 +6,12 @@ TLS sessions One of the most costly TLS operation is the negotiation of a new session, since both the client and the server need to generate and agree on cryptographic materials. In order to reduce that cost, TLS implements what is called session resumption, where a client opening a new connection to a server can reuse the cryptographic materials negotiated for a previous TLS session. +The following figures show that, with the same number of established connections and queries per second, the ratio of new TLS sessions and resumed sessions has a huge impact on CPU usage: + +.. figure:: ../imgs/tls_resumptions.png + :align: center + :alt: Resumed and new TLS sessions + The necessary information to resume a session can either be kept on the server's side (sessions) or on the client's one (tickets). Initially only the server-side approach existed, with two drawbacks: - the server needs to keep that information at hand, for a client that might never come back; diff --git a/pdns/dnsdistdist/docs/advanced/tuning.rst b/pdns/dnsdistdist/docs/advanced/tuning.rst index fd3b19c741..8732fce6c2 100644 --- a/pdns/dnsdistdist/docs/advanced/tuning.rst +++ b/pdns/dnsdistdist/docs/advanced/tuning.rst @@ -4,45 +4,24 @@ Performance Tuning First, a few words about :program:`dnsdist` architecture: * Each local bind has its own thread listening for incoming UDP queries - * and its own thread listening for incoming TCP connections, dispatching them right away to a pool of threads + * and its own thread listening for incoming TCP connections, dispatching them right away to a pool of TCP worker threads * Each backend has its own thread listening for UDP responses, including the ones triggered by DoH queries, if any * A maintenance thread calls the maintenance() Lua function every second if any, and is responsible for cleaning the cache * A health check thread checks the backends availability - * A control thread handles console connections + * A control thread handles console connections, plus one thread per connection * A carbon thread exports statistics to carbon servers if needed - * One or more webserver threads handle queries to the internal webserver - -TCP and DNS over TLS --------------------- - -The maximum number of threads in the TCP / DNS over TLS pool is controlled by the :func:`setMaxTCPClientThreads` directive, and defaults to 10. -This number can be increased to handle a large number of simultaneous TCP / DNS over TLS connections. -If all the TCP threads are busy, new TCP connections are queued while they wait to be picked up. -Before 1.4.0, a TCP thread could only handle a single incoming connection at a time. Starting with 1.4.0 the handling of TCP connections is now event-based, so a single TCP worker can handle a large number of TCP incoming connections simultaneously. -Note that before 1.6.0 the TCP worker threads were created at runtime, adding a new thread when the existing ones seemed to struggle with the load, until the maximum number of threads had been reached. Starting with 1.6.0 the configured number of worker threads are immediately created at startup. - -The maximum number of queued connections can be configured with :func:`setMaxTCPQueuedConnections` and defaults to 1000 (10000 on Linux since 1.6.0). Note that the size of the internal pipe used to distribute queries might need to be increased as well, using :func:`setTCPInternalPipeBufferSize`. -Any value larger than 0 will cause new connections to be dropped if there are already too many queued. -By default, every TCP worker thread has its own queue, and the incoming TCP connections are dispatched to TCP workers on a round-robin basis. -This might cause issues if some connections are taking a very long time, since incoming ones will be waiting until the TCP worker they have been assigned to has finished handling its current query, while other TCP workers might be available. - -The experimental :func:`setTCPUseSinglePipe` directive can be used so that all the incoming TCP connections are put into a single queue and handled by the first TCP worker available. This used to be useful before 1.4.0 because a single connection could block a TCP worker, but the "one pipe per TCP worker" is preferable now that workers can handle multiple connections to prevent waking up all idle workers when a new connection arrives. - -Rules and Lua -------------- - -Most of the query processing is done in C++ for maximum performance, but some operations are executed in Lua for maximum flexibility: - - * Rules added by :func:`LuaAction`, :func:`LuaResponseAction`, :func:`LuaFFIAction` or :func:`LuaFFIResponseAction` - * Server selection policies defined via :func:`setServerPolicyLua`, :func:`setServerPolicyLuaFFI`, :func:`setServerPolicyLuaFFIPerThread` or :func:`newServerPolicy` - -While Lua is fast, its use should be restricted to the strict necessary in order to achieve maximum performance, it might be worth considering using LuaJIT instead of Lua. -When Lua inspection is needed, the best course of action is to restrict the queries sent to Lua inspection by using :func:`addLuaAction` with a selector. + * One or more webserver threads handle queries to the internal webserver, plus one thread per HTTP connection + * A SNMP thread handles SNMP operations, when enabled. UDP and DNS over HTTPS ----------------------- -:program:`dnsdist` design choices mean that the processing of UDP and DNS over HTTPS queries is done by only one thread per local bind. +.. figure:: ../imgs/DNSDistUDP.png + :align: center + :alt: DNSDist UDP design + +:program:`dnsdist` design choices mean that the processing of UDP and DNS over HTTPS queries is done by only one thread per local bind (per :func:`addLocal`, :func:`addDNSCryptLocal` and :func:`addDOHLocal` directive). + This is great to keep lock contention to a low level, but might not be optimal for setups using a lot of processing power, caused for example by a large number of complicated rules. To be able to use more CPU cores for UDP queries processing, it is possible to use the ``reusePort`` parameter of the :func:`addLocal` and :func:`setLocal` directives to be able to add several identical local binds to dnsdist:: @@ -64,10 +43,6 @@ The UDP threads handling the responses from the backends do not use a lot of CPU newServer({address="192.0.2.127:53", name="Backend3"}) newServer({address="192.0.2.127:53", name="Backend4"}) -For DNS over HTTPS, every :func:`addDOHLocal` directive adds a new thread dealing with incoming connections, so it might be useful to add more than one directive, as indicated above. - -When dealing with a large traffic load, it might happen that the internal pipe used to pass queries between the threads handling the incoming connections and the one getting a response from the backend become full too quickly, degrading performance and causing timeouts. This can be prevented by increasing the size of the internal pipe buffer, via the `internalPipeBufferSize` option of :func:`addDOHLocal`. Setting a value of `1048576` is known to yield good results on Linux. - When dispatching UDP queries to backend servers, dnsdist keeps track of at most **n** outstanding queries for each backend. This number **n** can be tuned by the :func:`setMaxUDPOutstanding` directive, defaulting to 65535 which is the maximum value. @@ -76,9 +51,102 @@ This number **n** can be tuned by the :func:`setMaxUDPOutstanding` directive, de Large installations running dnsdist before 1.4.0 are advised to increase the default value at the cost of a slightly increased memory usage. +Looking at ``udp-in-errors`` in :func:`dumpStats` will reveal whether the system is dropping UDP datagrams because dnsdist does not pick them up fast enough. In that case it might be good to add more :func:`addLocal` directives. +In the same way, if the number of ``Drops`` in :func:`showServers` increase fast enough, it might mean that the backend is overloaded but also that the UDP received thread is. In that case adding more :func:`newServer` + +Using a single connected UDP socket to contact a backend, and thus a single (source address, source port, destination address, destination port) tuple, might not play well with some load-balancing mechanisms present in front of the backend. Linux's ``reuseport``, for example, does not balance the incoming datagrams to several threads in that case. That can be worked around by using the ``sockets`` option of the :func:`newServer` directive to open several sockets instead of one. dnsdist will then select a socket using round-robin to forward a query to the backend, and use event multiplexing on the receiving side. + +.. figure:: ../imgs/DNSDistDoH.png + :align: center + :alt: DNSDist DoH design + +For DNS over HTTPS, every :func:`addDOHLocal` directive adds a new thread dealing with incoming connections, so it might be useful to add more than one directive, as indicated above. + +When dealing with a large traffic load, it might happen that the internal pipe used to pass queries between the threads handling the incoming connections and the one getting a response from the backend become full too quickly, degrading performance and causing timeouts. This can be prevented by increasing the size of the internal pipe buffer, via the `internalPipeBufferSize` option of :func:`addDOHLocal`. Setting a value of `1048576` is known to yield good results on Linux. + +TCP and DNS over TLS +-------------------- + +.. figure:: ../imgs/DNSDistTCP.png + :align: center + :alt: DNSDist TCP and DoT design + +Before 1.4.0, a TCP thread could only handle a single incoming connection at a time. Starting with 1.4.0 the handling of TCP connections is now event-based, so a single TCP worker can handle a large number of TCP incoming connections simultaneously. +Note that before 1.6.0 the TCP worker threads were created at runtime, adding a new thread when the existing ones seemed to struggle with the load, until the maximum number of threads had been reached. Starting with 1.6.0 the configured number of worker threads are immediately created at startup. + +The maximum number of threads in the TCP / DNS over TLS pool is controlled by the :func:`setMaxTCPClientThreads` directive, and defaults to 10. +This number can be increased to handle a large number of simultaneous TCP / DNS over TLS connections. + +If all the TCP threads are busy, new TCP connections are queued while they wait to be picked up. The maximum number of queued connections can be configured with :func:`setMaxTCPQueuedConnections` and defaults to 1000 (10000 on Linux since 1.6.0). Note that the size of the internal pipe used to distribute queries might need to be increased as well, using :func:`setTCPInternalPipeBufferSize`. +Any value larger than 0 will cause new connections to be dropped if there are already too many queued. + +By default, every TCP worker thread has its own queue, and the incoming TCP connections are dispatched to TCP workers on a round-robin basis. +This might cause issues if some connections are taking a very long time, since incoming ones will be waiting until the TCP worker they have been assigned to has finished handling its current query, while other TCP workers might be available. + +The experimental :func:`setTCPUseSinglePipe` directive can be used so that all the incoming TCP connections are put into a single queue and handled by the first TCP worker available. This used to be useful before 1.4.0 because a single connection could block a TCP worker, but the "one pipe per TCP worker" is preferable now that workers can handle multiple connections to prevent waking up all idle workers when a new connection arrives. + +One of the first starting point when investigating TCP or DNS over TLS issues is to look at the :func:`showTCPStats` command. It provides a lot of metrics about the current and passed connections, and why they were closed. + +If the number of queued connections ("Queued" in :func:`showTCPStats`) reaches the maximum number of queued connections ("Max Queued" in :func:`showTCPStats`) then there is clearly a problem with TCP workers not picking up new connections quickly enough. It might be a good idea to increase the number of TCP workers. + +A different possibility is that there is not enough threads accepting new connections and distributing them to worker threads. Looking at whether the ``listenOverflows`` metric in :func:`dumpStats` increase over time will tell if we are losing TCP connections because the queue is full. In that case, since a single :func:`addLocal` or :func:`addTLSLocal` directive results in only one acceptor thread, it might useful to add more of these. + +Rules and Lua +------------- + +Most of the query processing is done in C++ for maximum performance, but some operations are executed in Lua for maximum flexibility: + + * Rules added by :func:`LuaAction`, :func:`LuaResponseAction`, :func:`LuaFFIAction` or :func:`LuaFFIResponseAction` + * Server selection policies defined via :func:`setServerPolicyLua`, :func:`setServerPolicyLuaFFI`, :func:`setServerPolicyLuaFFIPerThread` or :func:`newServerPolicy` + +While Lua is fast, its use should be restricted to the strict necessary in order to achieve maximum performance, it might be worth considering using LuaJIT instead of Lua. +When Lua inspection is needed, the best course of action is to restrict the queries sent to Lua inspection by using :func:`addLuaAction` with a selector. + ++------------------------------+-------------+-----------------+ +| Type | Performance | Locking | ++==============================+=============+=================+ +| C++ rule | fast | none | ++------------------------------+-------------+-----------------+ +| Lua rue | slow | global Lua lock | ++------------------------------+-------------+-----------------+ +| Lua FFI rule | fast | global Lua lock | ++------------------------------+-------------+-----------------+ +| C++ LB policy | fast | none | ++------------------------------+-------------+-----------------+ +| Lua LB policy | slow | global Lua lock | ++------------------------------+-------------+-----------------+ +| Lua FFI LB policy | fast | global Lua lock | ++------------------------------+-------------+-----------------+ +| Lua per-thread FFI LB policy | fast | none | ++------------------------------+-------------+-----------------+ + + Lock contention and sharding ---------------------------- -Adding more threads makes it possible to use more CPU cores to deal with the load, but at the cost of possibly increasing lock contention between threads. This is especially true for Lua-intensive setups, because Lua processing in dnsdist is serialized by a unique lock for all threads. +Adding more threads makes it possible to use more CPU cores to deal with the load, but at the cost of possibly increasing lock contention between threads. This is especially true for Lua-intensive setups, because Lua processing in dnsdist is serialized by a unique lock for all threads, as seen above. + For other components, like the packet cache and the in-memory ring buffers, it is possible to reduce the amount of contention by using sharding. Sharding divides the memory into several pieces, every one of these having its own separate lock, reducing the amount of times two threads or more will need to access the same data. + Sharding was disabled by default before 1.6.0 and could be enabled via the `numberOfShards` option to :func:`newPacketCache` and :func:`setRingBuffersSize`. It might still make sense to increment the number of shards when dealing with a lot of threads. + +Memory usage for TCP, DoH and DoT +--------------------------------- + ++--------------------------------+-----------------------------+ +| Protocol | Memory usage per connection | ++================================+=============================+ +| TCP | 6 kB | ++--------------------------------+-----------------------------+ +| DoT (GnuTLS) | 16 kB | ++--------------------------------+-----------------------------+ +| DoT (OpenSSL) | 52 kB | ++--------------------------------+-----------------------------+ +| DoT (OpenSSL w/ releaseBuffers | 19 kB | ++--------------------------------+-----------------------------+ +| DoH (http) | 2 kB | ++--------------------------------+-----------------------------+ +| DoH | 48 kB | ++--------------------------------+-----------------------------+ +| DoH (w/ releaseBuffers) | 15 kB | ++--------------------------------+-----------------------------+ diff --git a/pdns/dnsdistdist/docs/advanced/xpf.rst b/pdns/dnsdistdist/docs/advanced/xpf.rst deleted file mode 100644 index abe30ce588..0000000000 --- a/pdns/dnsdistdist/docs/advanced/xpf.rst +++ /dev/null @@ -1,13 +0,0 @@ -Using XPF ---------- - -In order to provide the downstream server with the address of the real client, or at least the one talking to dnsdist, the ``addXPF`` parameter can be used when creating a :func:`new server `. -This parameter indicates whether an experimental XPF record (from `draft-bellis-dnsop-xpf `_) shall be added to the query. Since that record is experimental, there is currently no option code assigned to it, and therefore one needs to be specified as an argument to the ``addXPF`` parameter. - -The XPF record is an alternative to the use of EDNS Client Subnet which has the advantages of preserving any existing EDNS Client Subnet value sent by the client, and of passing along the original destination address, as well as the initial source and destination ports. - -If the incoming request already contains a XPF record, it will not be overwritten. Instead a new one will be added to the query and the existing one will be preserved. -That might be an issue by allowing clients to spoof their source address by adding a forged XPF record to their query. That can be prevented by using a rule to drop incoming queries containing a XPF record (in that example the 65280 option code has been assigned to XPF): - - addAction(RecordsTypeCountRule(DNSSection.Additional, 65280, 1, 65535), DropAction()) - diff --git a/pdns/dnsdistdist/docs/guides/dns-over-https.rst b/pdns/dnsdistdist/docs/guides/dns-over-https.rst index be9663e657..dfa250d1dd 100644 --- a/pdns/dnsdistdist/docs/guides/dns-over-https.rst +++ b/pdns/dnsdistdist/docs/guides/dns-over-https.rst @@ -28,6 +28,13 @@ A more complicated (and more realistic) example is when you want to indicate met addDOHLocal('2001:db8:1:f00::1', '/etc/ssl/certs/example.com.pem', '/etc/ssl/private/example.com.key', "/", {customResponseHeaders={["link"]=" rel=\\"service-meta\\"; type=\\"text/html\\""}}) +A particular attention should be taken to the permissions of the certificate and key files. Many ACME clients used to get and renew certificates, like CertBot, set permissions assuming that services are started as root, which is no longer true for dnsdist as of 1.5.0. For that particular case, making a copy of the necessary files in the /etc/dnsdist directory is advised, using for example CertBot's ``--deploy-hook`` feature to copy the files with the right permissions after a renewal. + +More information about sessions management can also be found in :doc:`../advanced/tls-sessions-management`. + +Custom responses +---------------- + It is also possible to set HTTP response rules to intercept HTTP queries early, before the DNS payload, if any, has been processed, to send custom responses including error pages, redirects or even serve static content. First a rule needs to be defined using :func:`newDOHResponseMapEntry`, then a set of rules can be applied to a DoH frontend via :meth:`DOHFrontend.setResponsesMap`. For example, to send an HTTP redirect to queries asking for ``/rfc``, the following configuration can be used:: @@ -35,12 +42,31 @@ For example, to send an HTTP redirect to queries asking for ``/rfc``, the follow dohFE = getDOHFrontend(0) dohFE:setResponsesMap(map) +DNS over HTTP +------------- + In case you want to run DNS-over-HTTPS behind a reverse proxy you probably don't want to encrypt your traffic between reverse proxy and dnsdist. To let dnsdist listen for DoH queries over HTTP on localhost at port 8053 add one of the following to your config:: addDOHLocal("127.0.0.1:8053") addDOHLocal("127.0.0.1:8053", nil, nil, "/", { reusePort=true }) -A particular attention should be taken to the permissions of the certificate and key files. Many ACME clients used to get and renew certificates, like CertBot, set permissions assuming that services are started as root, which is no longer true for dnsdist as of 1.5.0. For that particular case, making a copy of the necessary files in the /etc/dnsdist directory is advised, using for example CertBot's ``--deploy-hook`` feature to copy the files with the right permissions after a renewal. +Internal design +--------------- + +The internal design used for DoH handling uses two threads per :func:`addDOHLocal` directive. The first thread will handle the HTTP/2 communication with the client and pass the received DNS queries to a second thread which will apply the rules and pass the query to a backend, over **UDP**. The response will be received by the regular UDP response handler for that backend and passed back to the first thread. That allows the first thread to be low-latency dealing with TLS and HTTP/2 only and never blocking. + +.. figure:: ../imgs/DNSDistDoH.png + :align: center + :alt: DNSDist DoH design + +The fact that the queries are forwarded over UDP means that a large UDP payload size should be configured between dnsdist and the backend to avoid most truncation issues, and dnsdist will advise a 4096-byte UDP Payload Buffer size. UDP datagrams can still be larger than the MTU as long as fragmented datagrams are not dropped on the path between dnsdist and the backend. + +Investigating issues +-------------------- + +dnsdist provides a lot of counters to investigate issues: -More information about sessions management can also be found in :doc:`guides/tls-sessions-management`. + * :func:`showTCPStats` will display a lot of information about current and passed connections + * :func:`showTLSErrorCounters` some metrics about why TLS sessions failed to establish + * :func:`showDOHResponseCodes` returns metrics about HTTP response codes sent by dnsdist diff --git a/pdns/dnsdistdist/docs/guides/dns-over-tls.rst b/pdns/dnsdistdist/docs/guides/dns-over-tls.rst index c409b94c62..da716bb517 100644 --- a/pdns/dnsdistdist/docs/guides/dns-over-tls.rst +++ b/pdns/dnsdistdist/docs/guides/dns-over-tls.rst @@ -19,4 +19,13 @@ The certificate chain presented by the server to an incoming client will then be A particular attention should be taken to the permissions of the certificate and key files. Many ACME clients used to get and renew certificates, like CertBot, set permissions assuming that services are started as root, which is no longer true for dnsdist as of 1.5.0. For that particular case, making a copy of the necessary files in the /etc/dnsdist directory is advised, using for example CertBot's ``--deploy-hook`` feature to copy the files with the right permissions after a renewal. -More information about sessions management can also be found in :doc:`guides/tls-sessions-management`. +More information about sessions management can also be found in :doc:`../advanced/tls-sessions-management`. + +Investigating issues +-------------------- + +dnsdist provides a lot of counters to investigate issues: + + * :func:`showTCPStats` will display a lot of information about current and passed connections + * :func:`showTLSErrorCounters` some metrics about why TLS sessions failed to establish + diff --git a/pdns/dnsdistdist/docs/imgs/DNSDistDoH.png b/pdns/dnsdistdist/docs/imgs/DNSDistDoH.png new file mode 100644 index 0000000000..01f1568e97 Binary files /dev/null and b/pdns/dnsdistdist/docs/imgs/DNSDistDoH.png differ diff --git a/pdns/dnsdistdist/docs/imgs/DNSDistTCP.png b/pdns/dnsdistdist/docs/imgs/DNSDistTCP.png new file mode 100644 index 0000000000..ab8f94e370 Binary files /dev/null and b/pdns/dnsdistdist/docs/imgs/DNSDistTCP.png differ diff --git a/pdns/dnsdistdist/docs/imgs/DNSDistUDP.png b/pdns/dnsdistdist/docs/imgs/DNSDistUDP.png new file mode 100644 index 0000000000..3c6b927eb5 Binary files /dev/null and b/pdns/dnsdistdist/docs/imgs/DNSDistUDP.png differ diff --git a/pdns/dnsdistdist/docs/imgs/ebpf_drops.png b/pdns/dnsdistdist/docs/imgs/ebpf_drops.png new file mode 100644 index 0000000000..25ab92ef23 Binary files /dev/null and b/pdns/dnsdistdist/docs/imgs/ebpf_drops.png differ diff --git a/pdns/dnsdistdist/docs/imgs/tls_resumptions.png b/pdns/dnsdistdist/docs/imgs/tls_resumptions.png new file mode 100644 index 0000000000..807ff6f47a Binary files /dev/null and b/pdns/dnsdistdist/docs/imgs/tls_resumptions.png differ diff --git a/pdns/dnsdistdist/docs/index.rst b/pdns/dnsdistdist/docs/index.rst index c176c07f95..3b8ede8174 100644 --- a/pdns/dnsdistdist/docs/index.rst +++ b/pdns/dnsdistdist/docs/index.rst @@ -10,11 +10,11 @@ A configuration to balance DNS queries to several backend servers: .. code-block:: lua - newServer({address="2001:4860:4860::8888", qps=1}) - newServer({address="2001:4860:4860::8844", qps=1}) - newServer({address="2620:0:ccc::2", qps=10}) - newServer({address="2620:0:ccd::2", name="dns1", qps=10}) - newServer("192.168.1.2") + newServer({address="2620:fe::fe2620:fe::9", qps=1}) + newServer({address="9.9.9.9", qps=1}) + newServer({address="2001:db8::1", qps=10}) + newServer({address="[2001:db8::2]:5300", name="dns1", qps=10}) + newServer("192.0.2.1") setServerPolicy(firstAvailable) -- first server within its QPS limit Running dnsdist diff --git a/pdns/dnsdistdist/docs/quickstart.rst b/pdns/dnsdistdist/docs/quickstart.rst index 386ee71f35..f0246e3ede 100644 --- a/pdns/dnsdistdist/docs/quickstart.rst +++ b/pdns/dnsdistdist/docs/quickstart.rst @@ -8,7 +8,7 @@ Running in the Foreground After :doc:`installing ` dnsdist, the quickest way to start experimenting is launching it on the foreground with:: - dnsdist -l 127.0.0.1:5300 8.8.8.8 2001:4860:4860::8888 + dnsdist -l 127.0.0.1:5300 9.9.9.9 2620:fe::fe2620:fe::9 This will make dnsdist listen on IP address 127.0.0.1, port 5300 and forward all queries to the two listed IP addresses, with a sensible balancing policy. @@ -17,11 +17,11 @@ This will make dnsdist listen on IP address 127.0.0.1, port 5300 and forward all Here is more complete configuration, save it to ``dnsdist.conf``:: - newServer({address="2001:4860:4860::8888", qps=1}) - newServer({address="2001:4860:4860::8844", qps=1}) - newServer({address="2620:0:ccc::2", qps=10}) - newServer({address="2620:0:ccd::2", name="dns1", qps=10}) - newServer("192.168.1.2") + newServer({address="2001:db8::1", qps=1}) + newServer({address="2001:db8::2", qps=1}) + newServer({address="[2001:db8::3]:5300", qps=10}) + newServer({address="2001:db8::4", name="dns1", qps=10}) + newServer("192.0.2.1") setServerPolicy(firstAvailable) -- first server within its QPS limit The :func:`newServer` function is used to add a backend server to the configuration. @@ -29,11 +29,11 @@ The :func:`newServer` function is used to add a backend server to the configurat Now run dnsdist again, reading this configuration:: $ dnsdist -C dnsdist.conf --local=0.0.0.0:5300 - Marking downstream [2001:4860:4860::8888]:53 as 'up' - Marking downstream [2001:4860:4860::8844]:53 as 'up' - Marking downstream [2620:0:ccc::2]:53 as 'up' - Marking downstream [2620:0:ccd::2]:53 as 'up' - Marking downstream 192.168.1.2:53 as 'up' + Marking downstream [2001:db8::1]:53 as 'up' + Marking downstream [2001:db8::2]:53 as 'up' + Marking downstream [2001:db8::3]:5300 as 'up' + Marking downstream [2001:db8::4]:53 as 'up' + Marking downstream 192.0.2.1.:53 as 'up' Listening on 0.0.0.0:5300 > @@ -46,11 +46,11 @@ Note that dnsdist dropped us in a prompt above, where we can get some statistics > showServers() # Address State Qps Qlim Ord Wt Queries Drops Drate Lat Pools - 0 [2001:4860:4860::8888]:53 up 0.0 1 1 1 1 0 0.0 0.0 - 1 [2001:4860:4860::8844]:53 up 0.0 1 1 1 0 0 0.0 0.0 - 2 [2620:0:ccc::2]:53 up 0.0 10 1 1 0 0 0.0 0.0 - 3 [2620:0:ccd::2]:53 up 0.0 10 1 1 0 0 0.0 0.0 - 4 192.168.1.2:53 up 0.0 0 1 1 0 0 0.0 0.0 + 0 [2001:db8::1]:53 up 0.0 1 1 1 1 0 0.0 0.0 + 1 [2001:db8::2]:53 up 0.0 1 1 1 0 0 0.0 0.0 + 2 [2001:db8::3]:5300 up 0.0 10 1 1 0 0 0.0 0.0 + 3 [2001:db8::4]:53 up 0.0 10 1 1 0 0 0.0 0.0 + 4 192.0.2.1:53 up 0.0 0 1 1 0 0 0.0 0.0 All 0.0 1 0 :func:`showServers()` is usually one of the first commands you will use when logging into the console. More advanced topics are covered in :doc:`guides/console`. @@ -65,11 +65,11 @@ The final server has no limit, which we can easily test:: > showServers() # Address State Qps Qlim Ord Wt Queries Drops Drate Lat Pools - 0 [2001:4860:4860::8888]:53 up 1.0 1 1 1 7 0 0.0 1.6 - 1 [2001:4860:4860::8844]:53 up 1.0 1 1 1 6 0 0.0 0.6 - 2 [2620:0:ccc::2]:53 up 10.3 10 1 1 64 0 0.0 2.4 - 3 [2620:0:ccd::2]:53 up 10.3 10 1 1 63 0 0.0 2.4 - 4 192.168.1.2:53 up 125.8 0 1 1 671 0 0.0 0.4 + 0 [2001:db8::1]:53 up 1.0 1 1 1 7 0 0.0 1.6 + 1 [2001:db8::2]:53 up 1.0 1 1 1 6 0 0.0 0.6 + 2 [2001:db8::3]:5300 up 10.3 10 1 1 64 0 0.0 2.4 + 3 [2001:db8::4]:53 up 10.3 10 1 1 63 0 0.0 2.4 + 4 192.0.2.1:53 up 125.8 0 1 1 671 0 0.0 0.4 All 145.0 811 0 Note that the first 4 servers were all limited to near their configured QPS, and that our final server was taking up most of the traffic. @@ -85,7 +85,7 @@ To force a server down, try :attr:`Server:setDown()`:: > getServer(0):setDown() > showServers() # Address State Qps Qlim Ord Wt Queries Drops Drate Lat Pools - 0 [2001:4860:4860::8888]:53 DOWN 0.0 1 1 1 8 0 0.0 0.0 + 0 [2001:db8::1]:53 DOWN 0.0 1 1 1 8 0 0.0 0.0 ... The ``DOWN`` in all caps means it was forced down. @@ -115,8 +115,8 @@ Adding network ranges to the :term:`ACL` is done with the :func:`setACL` and :fu .. code-block:: lua - setACL({'192.0.2.0/28', '2001:DB8:1::/56'}) -- Set the ACL to only allow these subnets - addACL('2001:DB8:2::/56') -- Add this subnet to the existing ACL + setACL({'192.0.2.0/28', '2001:db8:1::/56'}) -- Set the ACL to only allow these subnets + addACL('2001:db8:2::/56') -- Add this subnet to the existing ACL More Information ---------------- diff --git a/pdns/dnsdistdist/docs/reference/config.rst b/pdns/dnsdistdist/docs/reference/config.rst index a4d09c36d5..77c93bcc96 100644 --- a/pdns/dnsdistdist/docs/reference/config.rst +++ b/pdns/dnsdistdist/docs/reference/config.rst @@ -133,7 +133,7 @@ Listen Sockets * ``ocspResponses``: list - List of files containing OCSP responses, in the same order than the certificates and keys, that will be used to provide OCSP stapling responses. * ``minTLSVersion``: str - Minimum version of the TLS protocol to support. Possible values are 'tls1.0', 'tls1.1', 'tls1.2' and 'tls1.3'. Default is to require at least TLS 1.0. * ``numberOfTicketsKeys``: int - The maximum number of tickets keys to keep in memory at the same time. Only one key is marked as active and used to encrypt new tickets while the remaining ones can still be used to decrypt existing tickets after a rotation. Default to 5. - * ``ticketKeyFile``: str - The path to a file from where TLS tickets keys should be loaded, to support :rfc:`5077`. These keys should be rotated often and never written to persistent storage to preserve forward secrecy. The default is to generate a random key. dnsdist supports several tickets keys to be able to decrypt existing sessions after the rotation. See :doc:`guides/tls-sessions-management` for more information. + * ``ticketKeyFile``: str - The path to a file from where TLS tickets keys should be loaded, to support :rfc:`5077`. These keys should be rotated often and never written to persistent storage to preserve forward secrecy. The default is to generate a random key. dnsdist supports several tickets keys to be able to decrypt existing sessions after the rotation. See :doc:`../advanced/tls-sessions-management` for more information. * ``ticketsKeysRotationDelay``: int - Set the delay before the TLS tickets key is rotated, in seconds. Default is 43200 (12h). * ``sessionTimeout``: int - Set the TLS session lifetime in seconds, this is used both for TLS ticket lifetime and for sessions kept in memory. * ``sessionTickets``: bool - Whether session resumption via session tickets is enabled. Default is true, meaning tickets are enabled. @@ -176,7 +176,7 @@ Listen Sockets * ``ciphers``: str - The TLS ciphers to use. The exact format depends on the provider used. When the OpenSSL provider is used, ciphers for TLS 1.3 must be specified via ``ciphersTLS13``. * ``ciphersTLS13``: str - The ciphers to use for TLS 1.3, when the OpenSSL provider is used. When the GnuTLS provider is used, ``ciphers`` applies regardless of the TLS protocol and this setting is not used. * ``numberOfTicketsKeys``: int - The maximum number of tickets keys to keep in memory at the same time, if the provider supports it (GnuTLS doesn't, OpenSSL does). Only one key is marked as active and used to encrypt new tickets while the remaining ones can still be used to decrypt existing tickets after a rotation. Default to 5. - * ``ticketKeyFile``: str - The path to a file from where TLS tickets keys should be loaded, to support :rfc:`5077`. These keys should be rotated often and never written to persistent storage to preserve forward secrecy. The default is to generate a random key. The OpenSSL provider supports several tickets keys to be able to decrypt existing sessions after the rotation, while the GnuTLS provider only supports one key. See :doc:`guides/tls-sessions-management` for more information. + * ``ticketKeyFile``: str - The path to a file from where TLS tickets keys should be loaded, to support :rfc:`5077`. These keys should be rotated often and never written to persistent storage to preserve forward secrecy. The default is to generate a random key. The OpenSSL provider supports several tickets keys to be able to decrypt existing sessions after the rotation, while the GnuTLS provider only supports one key. See :doc:`../advanced/tls-sessions-management` for more information. * ``ticketsKeysRotationDelay``: int - Set the delay before the TLS tickets key is rotated, in seconds. Default is 43200 (12h). * ``sessionTimeout``: int - Set the TLS session lifetime in seconds, this is used both for TLS ticket lifetime and for sessions kept in memory. * ``sessionTickets``: bool - Whether session resumption via session tickets is enabled. Default is true, meaning tickets are enabled. @@ -467,6 +467,7 @@ Ringbuffers ``numberOfShards`` defaults to 10. Set the capacity of the ringbuffers used for live traffic inspection to ``num``, and the number of shards to ``numberOfShards`` if specified. + Increasing the number of entries comes at both a memory cost (around 250 MB for 1 million entries) and a CPU processing cost, so we strongly advise not going over 1 million entries. :param int num: The maximum amount of queries to keep in the ringbuffer. Defaults to 10000 :param int numberOfShards: the number of shards to use to limit lock contention. Default is 10, used to be 1 before 1.6.0 @@ -1503,7 +1504,7 @@ DOHFrontend .. method:: DOHFrontend:loadTicketsKeys(ticketsKeysFile) Load new tickets keys from the selected file, replacing the existing ones. These keys should be rotated often and never written to persistent storage to preserve forward secrecy. The default is to generate a random key. dnsdist supports several tickets keys to be able to decrypt existing sessions after the rotation. - See :doc:`guides/tls-sessions-management` for more information. + See :doc:`../advanced/tls-sessions-management` for more information. :param str ticketsKeysFile: The path to a file from where TLS tickets keys should be loaded. @@ -1544,7 +1545,7 @@ TLSContext .. method:: TLSContext:loadTicketsKeys(ticketsKeysFile) Load new tickets keys from the selected file, replacing the existing ones. These keys should be rotated often and never written to persistent storage to preserve forward secrecy. The default is to generate a random key. The OpenSSL provider supports several tickets keys to be able to decrypt existing sessions after the rotation, while the GnuTLS provider only supports one key. - See :doc:`guides/tls-sessions-management` for more information. + See :doc:`../advanced/tls-sessions-management` for more information. :param str ticketsKeysFile: The path to a file from where TLS tickets keys should be loaded. @@ -1571,7 +1572,7 @@ TLSFrontend .. versionadded:: 1.6.0 Load new tickets keys from the selected file, replacing the existing ones. These keys should be rotated often and never written to persistent storage to preserve forward secrecy. The default is to generate a random key. The OpenSSL provider supports several tickets keys to be able to decrypt existing sessions after the rotation, while the GnuTLS provider only supports one key. - See :doc:`guides/tls-sessions-management` for more information. + See :doc:`../advanced/tls-sessions-management` for more information. :param str ticketsKeysFile: The path to a file from where TLS tickets keys should be loaded. diff --git a/pdns/dnsdistdist/docs/running.rst b/pdns/dnsdistdist/docs/running.rst index ada3d95d6f..4158c12938 100644 --- a/pdns/dnsdistdist/docs/running.rst +++ b/pdns/dnsdistdist/docs/running.rst @@ -17,11 +17,7 @@ Issuing :func:`delta` on the console will print the changes to the configuration -- Wed Feb 22 2017 11:31:44 CET addLocal('127.0.0.1:5301', false) -- Wed Feb 22 2017 12:03:48 CET - addACL('2.0.0.0/8') - -- Wed Feb 22 2017 12:03:50 CET - addACL('2.0.0.0/8') - -- Wed Feb 22 2017 12:03:51 CET - addACL('2.0.0.0/8') + addACL('192.0.2.1/8') -- Wed Feb 22 2017 12:05:51 CET addACL('2001:db8::1') @@ -34,3 +30,26 @@ Running as unprivileged user Note that :program:`dnsdist` drops its privileges **after** parsing its startup configuration and binding its listening and initial :func:`newServer` sockets as user `root`. It is highly recommended to create a system user and group for :program:`dnsdist`. Note that most packaged versions of :program:`dnsdist` already create this user. + +Understanding how queries are forwarded to backends +--------------------------------------------------- + +Initially dnsdist tried to forward a query to the backend using the same protocol than the client used to contact dnsdist: queries received over UDP were forwarded over UDP, and the same for TCP. When incoming DNSCrypt and DNS over TLS support were added, the same logic was applied, so DoT queries are forwarded over TCP. For DNS over HTTPS, UDP was selected instead for performance reason, breaking with the existing logic: + ++--------------+----------+ +| Incoming | Outgoing | ++==============+==========+ +| UDP | UDP | ++--------------+----------+ +| TCP | TCP | ++--------------+----------+ +| DNSCrypt UDP | UDP | ++--------------+----------+ +| DNSCrypt TCP | TCP | ++--------------+----------+ +| DoT | TCP | ++--------------+----------+ +| DoH | **UDP** | ++--------------+----------+ + +That means that there is a potential issue with very large answers and DNS over HTTPS, requiring careful configuration of the path between dnsdist and the backend. More information about that is available in the :doc:`DNS over HTTPS section `.