pdns/recursordist/docs/lua-config/protobuf.rst

   1 Logging DNS messages with Protocol Buffers
   2 ==========================================
   3 The PowerDNS Recursor has the ability to emit a stream of protocol buffers messages over TCP, containing information about queries, answers and policy decisions.
   4
   5 Messages contain the IP address of the client initiating the query, the one on which the message was received, whether it was received over UDP or TCP, a timestamp and the qname, qtype and qclass of the question.
   6 In addition, messages related to responses contain the name, type, class and rdata of A, AAAA and CNAME records present in the response, as well as the response code.
   7
   8 Finally, if a RPZ or custom Lua policy has been applied, response messages also contain the applied policy name and some tags.
   9 This is particularly useful to detect and act on infected hosts.
  10
  11 Configuring Protocol Buffer logs
  12 --------------------------------
  13 Protobuf export to a server is enabled using the ``protobufServer()`` directive:
  14
  15 .. function:: protobufServer(servers [, options]))
  16
  17   .. versionadded:: 4.2.0
  18
  19   Send protocol buffer messages to one or more servers for incoming queries and/or outgoing responses. The client address may be masked using :func:`setProtobufMasks`, for anonymization purposes.
  20
  21   :param servers: The IP and port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
  22   :type servers: string or list of strings
  23   :param table options: A table with ``key=value`` pairs with options.
  24
  25   Options:
  26
  27   * ``timeout=2``: int - Time in seconds to wait when sending a message
  28   * ``maxQueuedEntries=100``: int - How many entries will be kept in memory if the server becomes unreachable
  29   * ``reconnectWaitTime=1``: int - How long to wait, in seconds, between two reconnection attempts
  30   * ``taggedOnly=false``: bool - Only entries with a policy or a policy tag set will be sent
  31   * ``asyncConnect``: bool - When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued
  32   * ``logQueries=true``: bool - Whether to export queries
  33   * ``logResponses=true``: bool - Whether to export responses
  34   * ``exportTypes={'A', 'AAAA', 'CNAME'}``: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported
  35
  36 .. function:: protobufServer(server [[[[[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], maskV4=32], maskV6=128], asyncConnect=false], taggedOnly=false])
  37
  38   .. deprecated:: 4.2.0
  39
  40   :param string server: The IP and port to connect to
  41   :param int timeout: Time in seconds to wait when sending a message
  42   :param int maxQueuedEntries: How many entries will be kept in memory if the server becomes unreachable
  43   :param int reconnectWaitTime: How long to wait, in seconds, between two reconnection attempts
  44   :param int maskV4: network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
  45   :param int maskV6: Same as maskV4, but for IPv6. Defaults to 128.
  46   :param bool taggedOnly: Only entries with a policy or a policy tag set will be sent.
  47   :param bool asyncConnect: When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued..
  48
  49 .. function:: setProtobufMasks(maskv4, maskV6)
  50
  51   .. versionadded:: 4.2.0
  52
  53   :param int maskV4: network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
  54   :param int maskV6: Same as maskV4, but for IPv6. Defaults to 128.
  55
  56 Logging outgoing queries and responses
  57 --------------------------------------
  58
  59 While :func:`protobufServer` only exports the queries sent to the recursor from clients, with the corresponding responses, ``outgoingProtobufServer()`` can be used to export outgoing queries sent by the recursor to authoritative servers, along with the corresponding responses.
  60
  61 .. function:: outgoingProtobufServer(servers [, options])
  62
  63   .. versionadded:: 4.2.0
  64
  65   Send protocol buffer messages to one or more servers for outgoing queries and/or incoming responses.
  66
  67   :param servers: The IP and port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
  68   :type servers: string or list of strings
  69   :param table options: A table with ``key=value`` pairs with options.
  70
  71   Options:
  72
  73   * ``timeout=2``: int - Time in seconds to wait when sending a message
  74   * ``maxQueuedEntries=100``: int - How many entries will be kept in memory if the server becomes unreachable
  75   * ``reconnectWaitTime=1``: int - How long to wait, in seconds, between two reconnection attempts
  76   * ``taggedOnly=false``: bool - Only entries with a policy or a policy tag set will be sent
  77   * ``asyncConnect``: bool - When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued
  78   * ``logQueries=true``: bool - Whether to export queries
  79   * ``logResponses=true``: bool - Whether to export responses
  80   * ``exportTypes={'A', 'AAAA', 'CNAME'}``: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported
  81
  82 .. function:: outgoingProtobufServer(server [[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], asyncConnect=false])
  83
  84   .. deprecated:: 4.2.0
  85
  86   :param string server: The IP and port to connect to
  87   :param int timeout: Time in seconds to wait when sending a message
  88   :param int maxQueuedEntries: How many entries will be kept in memory if the server becomes unreachable
  89   :param int reconnectWaitTime: How long to wait, in seconds, between two reconnection attempts
  90   :param bool asyncConnect: When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued..
  91
  92 Protobol Buffers Definition
  93 ---------------------------
  94
  95 The protocol buffers message types can be found in the `dnsmessage.proto <https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto>`_ file and is included here:
  96
  97 .. literalinclude:: ../../../dnsmessage.proto
  98
  99 Logging in ``dnstap`` format using framestreams
 100 -----------------------------------------------
 101 Define the following function to enable logging of outgoing queries and/or responses in ``dnstap`` format.
 102 The recursor must have been built with configure ``--enable-dnstap`` to make this feature available.
 103
 104 .. function:: dnstapFrameStreamServer(servers, [, options])
 105
 106   .. versionadded:: 4.X.0
 107
 108   Send dnstap formatted message to one or more framestream servers for outgoing queries and/or incoming responses.
 109
 110   :param servers: The IP and port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
 111   :type servers: string or list of strings
 112   :param table options: A table with ``key=value`` pairs with options.
 113
 114   Options:
 115
 116   * ``logQueries=true``: bool - log oputgoing queries
 117   * ``logResponses=true``: bool - log incoming responses
 118
 119   The follwing options apply to the settings of the framestream library. Refer to the documentation of that
 120   library for the default values, exact description and allowable values for these options.
 121   For all these options, absence or a zero value has the effect of using the library-provided default value.
 122
 123   * ``bufferHint=0``: unsigned
 124   * ``flushTimeout=0``: unsigned
 125   * ``inputQueueSize=0``: unsigned
 126   * ``outputQueueSize=0``: unsigned
 127   * ``queueNotifyThreshold=0``: unsigned
 128   * ``reopenInterval=0``: unsigned
 129