[thirdparty/pdns.git] / pdns / recursordist / docs / lua-config / protobuf.rst

Logging DNS messages with Protocol Buffers
==========================================
The PowerDNS Recursor has the ability to emit a stream of protocol buffers messages over TCP, containing information about queries, answers and policy decisions.

Messages contain the IP address of the client initiating the query, the one on which the message was received, whether it was received over UDP or TCP, a timestamp and the qname, qtype and qclass of the question.
In addition, messages related to responses contain the name, type, class and rdata of A, AAAA and CNAME records present in the response, as well as the response code.

Finally, if a RPZ or custom Lua policy has been applied, response messages also contain the applied policy name and some tags.
This is particularly useful to detect and act on infected hosts.

Configuring Protocol Buffer logs
--------------------------------
Protobuf export to a server is enabled using the ``protobufServer()`` directive:

.. function:: protobufServer(servers [, options]))

  .. versionadded:: 4.2.0

  Send protocol buffer messages to one or more servers for incoming queries and/or outgoing responses. The client address may be masked using :func:`setProtobufMasks`, for anonymization purposes.

  :param string or list of strings servers: The IP and port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
  :param table options: A table with key: value pairs with options.

  Options:

  * ``timeout=2``: int - Time in seconds to wait when sending a message
  * ``maxQueuedEntries=100``: int - How many entries will be kept in memory if the server becomes unreachable
  * ``reconnectWaitTime=1``: int - How long to wait, in seconds, between two reconnection attempts
  * ``taggedOnly=false``: bool - Only entries with a policy or a policy tag set will be sent
  * ``asyncConnect``: bool - When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued
  * ``logQueries=true``: bool - Whether to export queries
  * ``logResponses=true``: bool - Whether to export responses
  * ``exportTypes={'A', 'AAAA', 'CNAME'}``: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported

.. function:: protobufServer(server [[[[[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], maskV4=32], maskV6=128], asyncConnect=false], taggedOnly=false])

  .. deprecated:: 4.2.0

  :param string server: The IP and port to connect to
  :param int timeout: Time in seconds to wait when sending a message
  :param int maxQueuedEntries: How many entries will be kept in memory if the server becomes unreachable
  :param int reconnectWaitTime: How long to wait, in seconds, between two reconnection attempts
  :param int maskV4: network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
  :param int maskV6: Same as maskV4, but for IPv6. Defaults to 128.
  :param bool taggedOnly: Only entries with a policy or a policy tag set will be sent.
  :param bool asyncConnect: When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued..

.. function:: setProtobufMasks(maskv4, maskV6)

  .. versionadded:: 4.2.0

  :param int maskV4: network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
  :param int maskV6: Same as maskV4, but for IPv6. Defaults to 128.

Logging outgoing queries and responses
--------------------------------------

While :func:`protobufServer` only exports the queries sent to the recursor from clients, with the corresponding responses, ``outgoingProtobufServer()`` can be used to export outgoing queries sent by the recursor to authoritative servers, along with the corresponding responses.

.. function:: outgoingProtobufServer(servers [, options])

  .. versionadded:: 4.2.0

  Send protocol buffer messages to one or more servers for outgoing queries and/or incoming responses.

  :param string or list of strings servers: The IP and port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
  :param table options: A table with key: value pairs with options.

  Options:

  * ``timeout=2``: int - Time in seconds to wait when sending a message
  * ``maxQueuedEntries=100``: int - How many entries will be kept in memory if the server becomes unreachable
  * ``reconnectWaitTime=1``: int - How long to wait, in seconds, between two reconnection attempts
  * ``taggedOnly=false``: bool - Only entries with a policy or a policy tag set will be sent
  * ``asyncConnect``: bool - When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued
  * ``logQueries=true``: bool - Whether to export queries
  * ``logResponses=true``: bool - Whether to export responses
  * ``exportTypes={'A', 'AAAA', 'CNAME'}``: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported

.. function:: outgoingProtobufServer(server [[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], asyncConnect=false])

  .. deprecated:: 4.2.0

  :param string server: The IP and port to connect to
  :param int timeout: Time in seconds to wait when sending a message
  :param int maxQueuedEntries: How many entries will be kept in memory if the server becomes unreachable
  :param int reconnectWaitTime: How long to wait, in seconds, between two reconnection attempts
  :param bool asyncConnect: When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued..

Protobol Buffers Definition
---------------------------

The protocol buffers message types can be found in the `dnsmessage.proto <https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto>`_ file and is included here:

.. literalinclude:: ../../../dnsmessage.proto
Commit	Line	Data
223bb49e PL	1	Logging DNS messages with Protocol Buffers
	2	==========================================
	3	The PowerDNS Recursor has the ability to emit a stream of protocol buffers messages over TCP, containing information about queries, answers and policy decisions.
	4
	5	Messages contain the IP address of the client initiating the query, the one on which the message was received, whether it was received over UDP or TCP, a timestamp and the qname, qtype and qclass of the question.
	6	In addition, messages related to responses contain the name, type, class and rdata of A, AAAA and CNAME records present in the response, as well as the response code.
	7
	8	Finally, if a RPZ or custom Lua policy has been applied, response messages also contain the applied policy name and some tags.
	9	This is particularly useful to detect and act on infected hosts.
	10
223bb49e PL	11	Configuring Protocol Buffer logs
	12	--------------------------------
	13	Protobuf export to a server is enabled using the ``protobufServer()`` directive:
	14
b773359c	15	.. function:: protobufServer(servers [, options]))
f1c7929a RG	16
	17	.. versionadded:: 4.2.0
	18
b773359c	19	Send protocol buffer messages to one or more servers for incoming queries and/or outgoing responses. The client address may be masked using :func:`setProtobufMasks`, for anonymization purposes.
8fcd94b3	20
b773359c	21	:param string or list of strings servers: The IP and port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
f1c7929a RG	22	:param table options: A table with key: value pairs with options.
	23
	24	Options:
	25
	26	* ``timeout=2``: int - Time in seconds to wait when sending a message
	27	* ``maxQueuedEntries=100``: int - How many entries will be kept in memory if the server becomes unreachable
	28	* ``reconnectWaitTime=1``: int - How long to wait, in seconds, between two reconnection attempts
	29	* ``taggedOnly=false``: bool - Only entries with a policy or a policy tag set will be sent
	30	* ``asyncConnect``: bool - When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued
	31	* ``logQueries=true``: bool - Whether to export queries
	32	* ``logResponses=true``: bool - Whether to export responses
0bd2e252	33	* ``exportTypes={'A', 'AAAA', 'CNAME'}``: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported
f1c7929a	34
4368d62f	35	.. function:: protobufServer(server [[[[[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], maskV4=32], maskV6=128], asyncConnect=false], taggedOnly=false])
223bb49e	36
f1c7929a RG	37	.. deprecated:: 4.2.0
f1c7929a RG	38
88fe29ba PL	39	:param string server: The IP and port to connect to
	40	:param int timeout: Time in seconds to wait when sending a message
	41	:param int maxQueuedEntries: How many entries will be kept in memory if the server becomes unreachable
	42	:param int reconnectWaitTime: How long to wait, in seconds, between two reconnection attempts
	43	:param int maskV4: network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
	44	:param int maskV6: Same as maskV4, but for IPv6. Defaults to 128.
	45	:param bool taggedOnly: Only entries with a policy or a policy tag set will be sent.
d0dbda45	46	:param bool asyncConnect: When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued..
223bb49e	47
f1c7929a RG	48	.. function:: setProtobufMasks(maskv4, maskV6)
	49
	50	.. versionadded:: 4.2.0
	51
	52	:param int maskV4: network mask to apply to the client IPv4 addresses, for anonymization purposes. The default of 32 means no anonymization.
	53	:param int maskV6: Same as maskV4, but for IPv6. Defaults to 128.
	54
223bb49e PL	55	Logging outgoing queries and responses
	56	--------------------------------------
	57
4368d62f	58	While :func:`protobufServer` only exports the queries sent to the recursor from clients, with the corresponding responses, ``outgoingProtobufServer()`` can be used to export outgoing queries sent by the recursor to authoritative servers, along with the corresponding responses.
223bb49e	59
b773359c	60	.. function:: outgoingProtobufServer(servers [, options])
f1c7929a RG	61
	62	.. versionadded:: 4.2.0
	63
b773359c	64	Send protocol buffer messages to one or more servers for outgoing queries and/or incoming responses.
8fcd94b3	65
b773359c	66	:param string or list of strings servers: The IP and port to connect to, or a list of those. If more than one server is configured, all messages are sent to every server.
f1c7929a RG	67	:param table options: A table with key: value pairs with options.
	68
	69	Options:
	70
	71	* ``timeout=2``: int - Time in seconds to wait when sending a message
	72	* ``maxQueuedEntries=100``: int - How many entries will be kept in memory if the server becomes unreachable
	73	* ``reconnectWaitTime=1``: int - How long to wait, in seconds, between two reconnection attempts
	74	* ``taggedOnly=false``: bool - Only entries with a policy or a policy tag set will be sent
	75	* ``asyncConnect``: bool - When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued
	76	* ``logQueries=true``: bool - Whether to export queries
	77	* ``logResponses=true``: bool - Whether to export responses
0bd2e252	78	* ``exportTypes={'A', 'AAAA', 'CNAME'}``: list of strings - The list of record types found in the answer section to export. Only A, AAAA, CNAME, MX, NS, PTR, SPF, SRV and TXT are currently supported
f1c7929a	79
4368d62f	80	.. function:: outgoingProtobufServer(server [[[[, timeout=2], maxQueuedEntries=100], reconnectWaitTime=1], asyncConnect=false])
223bb49e	81
f1c7929a RG	82	.. deprecated:: 4.2.0
f1c7929a RG	83
88fe29ba PL	84	:param string server: The IP and port to connect to
	85	:param int timeout: Time in seconds to wait when sending a message
	86	:param int maxQueuedEntries: How many entries will be kept in memory if the server becomes unreachable
	87	:param int reconnectWaitTime: How long to wait, in seconds, between two reconnection attempts
d0dbda45	88	:param bool asyncConnect: When set to false (default) the first connection to the server during startup will block up to ``timeout`` seconds, otherwise the connection is done in a separate thread, after the first message has been queued..
223bb49e	89
4368d62f PL	90	Protobol Buffers Definition
4368d62f PL	91	---------------------------
223bb49e	92
4368d62f	93	The protocol buffers message types can be found in the `dnsmessage.proto <https://github.com/PowerDNS/pdns/blob/master/pdns/dnsmessage.proto>`_ file and is included here:
223bb49e	94
4368d62f	95	.. literalinclude:: ../../../dnsmessage.proto