From: Vladimír Čunát Date: Mon, 26 Oct 2020 19:41:13 +0000 (+0100) Subject: XDP: add documentation X-Git-Tag: v5.2.0~1^2~7 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=4e0fb7cb98291f57a180c6a45f91bf082987c479;p=thirdparty%2Fknot-resolver.git XDP: add documentation --- diff --git a/daemon/bindings/net_server.rst b/daemon/bindings/net_server.rst index a785d5167..53e0efee9 100644 --- a/daemon/bindings/net_server.rst +++ b/daemon/bindings/net_server.rst @@ -13,6 +13,7 @@ First you need to decide what service should be available on given IP address :header: "Protocol/service", "net.listen *kind*" "DNS (unencrypted UDP+TCP, :rfc:`1034`)","``dns``" + "DNS (unencrypted UDP, :ref:`using XDP Linux API `)","``xdp``" ":ref:`dns-over-tls`","``tls``" ":ref:`dns-over-https`","``doh2``" ":ref:`Web management `","``webmgmt``" @@ -38,6 +39,7 @@ First you need to decide what service should be available on given IP address :header: "**Network protocol**", "**Configuration command**" "DNS (UDP+TCP, :rfc:`1034`)","``net.listen('192.0.2.123', 53)``" + "DNS (UDP, :ref:`using XDP `)","``net.listen('192.0.2.123', 53, { kind = 'xdp' })``" ":ref:`dns-over-tls`","``net.listen('192.0.2.123', 853, { kind = 'tls' })``" ":ref:`dns-over-https`","``net.listen('192.0.2.123', 443, { kind = 'doh2' })``" ":ref:`Web management `","``net.listen('192.0.2.123', 8453, { kind = 'webmgmt' })``" @@ -56,6 +58,8 @@ Examples: net.listen('::', 443, { kind = 'doh2' }) net.listen('::', 8453, { kind = 'webmgmt' }) -- see http module net.listen('/tmp/kresd-socket', nil, { kind = 'webmgmt' }) -- http module supports AF_UNIX + net.listen('eth0', 53, { kind = 'xdp' }) + net.listen('192.0.2.123', 53, { kind = 'xdp', nic_queue = 0 }) .. warning:: On machines with multiple IP addresses avoid listening on wildcards ``0.0.0.0`` or ``::``. Knot Resolver could answer from different IP @@ -159,3 +163,4 @@ Following configuration functions are useful mainly for scripting or :ref:`runti .. _`dnsproxy module`: https://www.knot-dns.cz/docs/2.7/html/modules.html#dnsproxy-tiny-dns-proxy + diff --git a/daemon/bindings/net_xdpsrv.rst b/daemon/bindings/net_xdpsrv.rst new file mode 100644 index 000000000..f3dbcbabf --- /dev/null +++ b/daemon/bindings/net_xdpsrv.rst @@ -0,0 +1,133 @@ +.. SPDX-License-Identifier: GPL-3.0-or-later + +.. _dns-over-xdp: + +XDP for higher UDP performance +------------------------------ + +Using XDP allows significant speedup of UDP packet processing in recent Linux kernels, +especially with some network drivers that implement good support. +The basic idea is that for selected packets the Linux networking stack is bypassed, +and some drivers can even directly use the user-space buffers for reading and writing. + +.. warning:: + Bypassing the network stack has significant implications, such as bypassing the firewall + and monitoring solutions. + Make sure you're familiar with the trade-offs before using this feature. + Read more in :ref:`dns-over-xdp_limitations`. + +.. TODO perhaps some hint/link about how significant speedup one might get? (link to some talk video?) + + +Prerequisites +^^^^^^^^^^^^^ +.. this is mostly copied from knot-dns doc/operations.rst + +* Linux kernel 4.18+ (5.x+ is recommended for optimal performance) compiled with + the `CONFIG_XDP_SOCKETS=y` option. XDP isn't supported in other operating systems. +* libknot compiled with XDP support +* A multiqueue network card with native XDP support is highly recommended, + otherwise the performance gains will be much lower. + Successfully tested cards: + + * Intel series 700 (driver `i40e`), maximum number of queues per interface is 64. + * Intel series 500 (driver `ixgbe`), maximum number of queues per interface is 64. + The number of CPUs available has to be at most 64! + + +Set up +^^^^^^ +.. first parts are mostly copied from knot-dns doc/operations.rst + +The server instances need additional Linux **capabilities** during startup. +(Or you could start them as `root`.) +Execute command + +.. code-block:: bash + + systemctl edit kresd@.service + +And insert these lines: + +.. code-block:: ini + + [Service] + CapabilityBoundingSet=CAP_NET_RAW CAP_NET_ADMIN CAP_SYS_ADMIN CAP_SYS_RESOURCE + AmbientCapabilities=CAP_NET_RAW CAP_NET_ADMIN CAP_SYS_ADMIN CAP_SYS_RESOURCE + +.. TODO suggest some way for ethtool -L? Perhaps via systemd units? + +You want the same number of kresd instances and network **queues** on your card; +you can use ``ethtool -L`` before the services start. +With XDP this is more important than with vanilla UDP, as we only support one instance +per queue and unclaimed queues will fall back to vanilla UDP. +Ideally you can set these numbers as high as the number of CPUs that you want kresd to use. + +Modification of ``/etc/knot-resolver/kresd.conf`` may often be quite simple, for example: + +.. code-block:: lua + + net.listen('eth2', 53, { kind = 'xdp' }) + net.listen('203.0.113.53', 53, { kind = 'dns' }) + +Note that you want to also keep the vanilla DNS line to service TCP +and possibly any fallback UDP (e.g. from unclaimed queues). +XDP listening is in principle done on queues of whole network interfaces +and the target addresses of incoming packets aren't checked in any way, +but you are still allowed to specify interface by an address +(if it's unambiguous at that moment): + +.. code-block:: lua + + net.listen('203.0.113.53', 53, { kind = 'xdp' }) + net.listen('203.0.113.53', 53, { kind = 'dns' }) + +The default selection of queues is tailored for the usual naming convention: +``kresd@1.service``, ``kresd@2.service``, ... +but you can still specify them explicitly, e.g. the default is effectively the same as: + +.. code-block:: lua + + net.listen('eth2', 53, { kind = 'xdp', nic_queue = env.SYSTEMD_INSTANCE - 1 }) + + +Optimizations +^^^^^^^^^^^^^ +.. this is basically copied from knot-dns doc/operations.rst + +Some helpful commands: + +.. code-block:: text + + ethtool -N rx-flow-hash udp4 sdfn + ethtool -N rx-flow-hash udp6 sdfn + ethtool -L combined + ethtool -G rx tx + renice -n 19 -p $(pgrep '^ksoftirqd/[0-9]*$') + +.. TODO CPU affinities? `CPUAffinity=%i` in systemd unit sounds good. + + +.. _dns-over-xdp_limitations: + +Limitations +^^^^^^^^^^^ +.. this is basically copied from knot-dns doc/operations.rst + +* VLAN segmentation is not supported. +* MTU higher than 1792 bytes is not supported. +* Multiple BPF filters per one network device are not supported. +* Symmetrical routing is required (query source MAC/IP addresses and + reply destination MAC/IP addresses are the same). +* Systems with big-endian byte ordering require special recompilation of libknot. +* IPv4 header and UDP checksums are not verified on received DNS messages. +* DNS over XDP traffic is not visible to common system tools (e.g. firewall, tcpdump etc.). +* BPF filter is not automatically unloaded from the network device. Manual filter unload:: + + ip link set dev xdp off + +* Knot Resolver only supports using XDP towards clients currently (not towards upstreams). +* When starting up an XDP socket you may get a harmless warning:: + + libbpf: Kernel error message: XDP program already attached + diff --git a/doc/config-performance.rst b/doc/config-performance.rst index 36e73b3cb..9df0f93e3 100644 --- a/doc/config-performance.rst +++ b/doc/config-performance.rst @@ -32,4 +32,5 @@ impact than cache settings and number of instances. modules-rfc7706 modules-priming modules-edns_keepalive + daemon-bindings-net_xdpsrv diff --git a/doc/daemon-bindings-net_xdpsrv.rst b/doc/daemon-bindings-net_xdpsrv.rst new file mode 120000 index 000000000..da7870b25 --- /dev/null +++ b/doc/daemon-bindings-net_xdpsrv.rst @@ -0,0 +1 @@ +../daemon/bindings/net_xdpsrv.rst \ No newline at end of file