Artem Boldariev [Tue, 14 Nov 2023 16:19:12 +0000 (18:19 +0200)]
System test for PROXYv2
This commit adds a system test suite for PROXYv2. The idea on which it
is based is simple:
1. Firstly we check that 'allow-proxy' and 'allow-proxy-on' (whatever
is using the new 'isc_nmhandle_real_localaddr/peeraddr()') do what
they intended to do.
2. Anything else that needs an interface or peer address (ACL
functionality, for example) is using the old
'isc_nmhandle_localaddr/peeraddr()' - which are now returning
addresses received via PROXY (if any) instead of the real connection
addresses. The beauty of it that we DO NOT need to verify every bit of
the code relying on these functions: whatever works in one place will
work everywhere else, as these were the only functions that allowed
any higher level code to get peer and interface addresses.
This way it is relatively easy to see if PROXYv2 works as intended.
Artem Boldariev [Tue, 31 Oct 2023 10:39:04 +0000 (12:39 +0200)]
BIND: Add 'allow-proxy' and 'allow-proxy-on' options
The main intention of PROXY protocol is to pass endpoints information
to a back-end server (in our case - BIND). That means that it is a
valid way to spoof endpoints information, as the addresses and ports
extracted from PROXYv2 headers, from the point of view of BIND, are
used instead of the real connection addresses.
Of course, an ability to easily spoof endpoints information can be
considered a security issue when used uncontrollably. To resolve that,
we introduce 'allow-proxy' and 'allow-proxy-on' ACL options. These are
the only ACL options in BIND that work with real PROXY connections
addresses, allowing a DNS server operator to specify from what clients
and on which interfaces he or she is willing to accept PROXY
headers. By default, for security reasons we do not allow to accept
them.
Artem Boldariev [Mon, 30 Oct 2023 17:37:34 +0000 (19:37 +0200)]
Avoid using sock->iface and sock->peer from the lower transport
This commit modifies TLS Stream and DNS-over-HTTPS transports so that
they do not use the "sock->iface" and "sock->peer" of the lower level
transport directly.
That did not cause any problems before, as things worked as expected,
but with the introduction of PROXYv2 support we use handles to store
the information in both PROXY Stream and UDP Proxy
transports. Therefore, in order to propagate the information (like
addresses), extracted from PROXYv2 headers, from the lower level
transports to the higher-level ones, we need to get that information
from the lower-level handles rather than sockets. That means that we
should get the peer and interface addresses using the intended
APIs ("isc_nmhandle_peeraddr()" and "isc_nmhandle_localaddr()").
Artem Boldariev [Mon, 30 Oct 2023 15:03:30 +0000 (17:03 +0200)]
Add 'proxy' option to 'listen-on' statement
This commit extends "listen-on" statement with "proxy" options that
allows one to enable PROXYv2 support on a dedicated listener. It can
have the following values:
- "plain" to send PROXYv2 headers without encryption, even in the case
of encrypted transports.
- "encrypted" to send PROXYv2 headers encrypted right after the TLS
handshake.
Artem Boldariev [Thu, 26 Oct 2023 22:43:00 +0000 (01:43 +0300)]
Increase conn. timeout in the timeout recovery tests
Due to increased number of the NM unit tests and, thus, increased load
on the system timeout recovery tests can sometimes fail, in particular
on FreeBSD. This commit fixes that. Besides, it seems that use of
T_SOFT here was unintentional to begin with.
Artem Boldariev [Mon, 16 Oct 2023 19:54:13 +0000 (22:54 +0300)]
Integrate PROXYv2 support into dig
This commit adds PROXYv2 support into dig by the means of adding
+[no]proxy and +[no]proxy-plain options. Since this commit dig
supports sending PROXYv2 headers on all supported DNS-transports.
The support for PROXYv2 is modelled after that one in kdig.
Artem Boldariev [Fri, 13 Oct 2023 12:31:32 +0000 (15:31 +0300)]
PROXY over UDP unit tests
This commit adds a unit test suite for the new PROXY over UDP
transport. Most of the code is reused from the UDP unit test suite, as
the new transport aims to be fully compatible with UDP on the API
level.
Artem Boldariev [Wed, 11 Oct 2023 19:18:54 +0000 (22:18 +0300)]
Refactor UDP tests
This commit mostly moves the code around to make the parts of the UDP
unit test suite reusable. That changes the unit test suite structure
to resemble that of stream based unit tests.
The motivation behind this is to reuse most of the code for the new
PROXY over UDP uni tests suite.
Add a utility function to dump all active sockets on a NM instance
Add the new isc__nm_dump_active_manager() function that can be used
for debugging purposes: it dumps all active sockets withing the
network manager instance.
This commit adds a new transport that supports PROXYv2 over UDP. It is
built on top of PROXYv2 handling code (just like PROXY Stream). It
works by processing and stripping the PROXYv2 headers at the beginning
of a datagram (when accepting a datagram) or by placing a PROXYv2
header to the beginning of an outgoing datagram.
The transport is built in such a way that incoming datagrams are being
handled with minimal memory allocations and copying.
Artem Boldariev [Thu, 25 May 2023 15:47:07 +0000 (18:47 +0300)]
TLS: detect ISC_R_SHUTTINGDOWN and ISC_R_CANCELED cases properly
In the previous versions of the NM, detecting the case when worker is
shutting down was not that important and actual status code did not
matter much. However, that might be not the case all the time.
This commit makes necessary modifications to the code.
Artem Boldariev [Tue, 17 Oct 2023 17:36:58 +0000 (20:36 +0300)]
Add PROXY over TLS support to PROXY Stream
This commit makes it possible to use PROXY Stream not only over TCP,
but also over TLS. That is, now PROXY Stream can work in two modes as
far as TLS is involved:
1. PROXY over (plain) TCP - PROXYv2 headers are sent unencrypted before
TLS handshake messages. That is the main mode as described in the
PROXY protocol specification (as it is clearly stated there), and most
of the software expects PROXYv2 support to be implemented that
way (e.g. HAProxy);
2. PROXY over (encrypted) TLS - PROXYv2 headers are sent after the TLS
handshake has happened. For example, this mode is being used (only ?)
by "dnsdist". As far as I can see, that is, in fact, a deviation from
the spec, but I can certainly see how PROXYv2 could end up being
implemented this way elsewhere.
Artem Boldariev [Thu, 4 May 2023 20:06:23 +0000 (23:06 +0300)]
Make it possible to use TLS Stream on top of PROXY Stream
This commit modifies TLS Stream to make it possible to use over PROXY
Stream. That is required to add PROVYv2 support into TLS-based
transports (DNS over HTTP, DNS over TLS).
This commit adds a specialised test suite for the PROXY Stream
transport by reusing most of the testing code from other unit tests
for other stream-based transports.
Artem Boldariev [Thu, 16 Mar 2023 10:50:04 +0000 (12:50 +0200)]
PROXY Stream transport
This commit adds a new stream-based transport with an interface
compatible with TCP. The transport is built on top of TCP transport
and the new PROXYv2 handling code. Despite being built on top of TCP,
it can be easily extended to work on top of any TCP-like stream-based
transport. The intention of having this transport is to add PROXYv2
support into all existing stream-based DNS transport (DNS over TCP,
DNS over TLS, DNS over HTTP) by making the work on top of this new
transport.
The idea behind the transport is simple after accepting the connection
or connecting to a remote server it enters PROXYv2 handling mode: that
is, it either attempts to read (when accepting the connection) or send
(when establishing a connection) a PROXYv2 header. After that it works
like a mere wrapper on top of the underlying stream-based
transport (TCP).
Artem Boldariev [Fri, 3 Mar 2023 14:43:15 +0000 (16:43 +0200)]
Add PROXY header handling unit test suite
The commit adds a fairly comprehensive unit test suite for our new
PROXYv2 handling code. The unit tests suite ensures both the
correctness of the code and ensures that the part responsible for
handling incoming headers is very strict regarding what to accept as
valid.
Artem Boldariev [Fri, 3 Mar 2023 14:42:47 +0000 (16:42 +0200)]
Add PROXYv2 header utilities
This commit adds a set of utilities for dealing with PROXYv2 headers,
both parsing and generating them. The code has no dependencies from
the networking code and is (for the most part) a "separate library".
The part responsible for handling incoming PROXYv2 headers is
structured as a state machine which accepts data as input and calls a
callback to notify the upper-level code about the data processing
status.
Such a design, among other things, makes it easy to write a thorough
unit test suite for that, as there are fewer dependencies as well as
will not stand in the way of any changes in the networking code.
Michal Nowak [Tue, 5 Dec 2023 08:44:50 +0000 (09:44 +0100)]
Run "make check" in system:gcc:sid:amd64
Following the discontinuation of the OpenBSD system test in CI, transfer
the execution of the system test using the "make check" method to the
Debian "sid" system test CI job.
Michal Nowak [Mon, 4 Dec 2023 16:13:55 +0000 (17:13 +0100)]
Drop OpenBSD system test CI job
The system tests on OpenBSD consistently exhibit lower stability
compared to our other CI platforms. Some of these challenges are
intrinsic to the system test itself and require attention. However,
there are OpenBSD issues, which seem to be more widespread on this
platform than others. In our daily CI pipelines, OpenBSD system tests
often bear the brunt of all failed CI jobs.
It's possible that our OpenBSD CI image could be optimized, but we
currently lack the domain-specific knowledge needed to make
improvements.
Matthijs Mekking [Wed, 22 Nov 2023 15:12:51 +0000 (16:12 +0100)]
Lower the maximum allowed NSEC3 iterations to 50
BIND 9 will now treat the response as insecure when processing NSEC3
records with iterations larger than 50.
Earlier, we limited the number of iterations to 150 (in #2445).
RFC 9276 says: Because there has been a large growth of open (public)
DNSSEC validating resolvers that are subject to compute resource
constraints when handling requests from anonymous clients, this
document recommends that validating resolvers reduce their iteration
count limits over time. Specifically, validating resolver operators and
validating resolver software implementers are encouraged to continue
evaluating NSEC3 iteration count deployment trends and lower their
acceptable iteration limits over time.
After evaluation, we decided that the next major BIND release should
lower the maximum allowed NSEC3 iterations to 50, which should be
fine for 99,87% of the domain names.
Matthijs Mekking [Wed, 22 Nov 2023 11:32:55 +0000 (12:32 +0100)]
Change NSEC3 iterations to 0 in system tests
The system tests need to be updated because non-zero iterations are no
longer accepted.
The autosign system test changes its iterations from 1 to 0 in one
test case. This requires the hash to be updated.
The checkconf system test needs to change the iterations in the good
configuration files to 0, and in the bad ones to 1 (any non-zero value
would suffice, but we test the corner case here). Also, the expected
failure message is change, so needs to be adjusted.
The nsec3 system test also needs iteration configuration adjustments.
In addition, the test script no longer needs the ITERATIONS environment
variable.
In the process of updating the system tests, I noticed an error
in the dnssec-policy "nsec3-other", where the salt length in one
configuration file is different than in the other (they need to be
the same). Furthermore, the 'rndc signing -nsec3param' test case
is operated on the zone 'nsec-change.kasp', so is moved so that the
tests on the same zone are grouped together.
Tom Krizek [Mon, 4 Dec 2023 17:05:50 +0000 (18:05 +0100)]
Add system test utility package isctest
Create a utility package for code shared by the python tests. The
utility functions should use reasonable defaults and be split up into
modules according to their functionality.
Ensure assert rewriting is enabled for the modules to get the most
useful output from pytest.
Tom Krizek [Fri, 24 Nov 2023 14:50:08 +0000 (15:50 +0100)]
Allow assertion message rewrite in statschannel test
By default, the useful assertion message rewrite is used by pytest for
test modules only. Since another module is imported with shared
functionality, ensure it has pytest's assertion message rewriting
enabled to obtain more debug information in case it fails.
Tom Krizek [Fri, 24 Nov 2023 14:32:09 +0000 (15:32 +0100)]
Add assert message to test doth/stress_http_quota.py
This file is executed outside of pytest with pure python, which doesn't
do any AssertionError message rewriting like pytest. Ensure the assert
messages in this file provide a useful debug message.
Mark Andrews [Mon, 4 Dec 2023 01:23:18 +0000 (12:23 +1100)]
Destroy the message before detaching the view
With shared name memory pools (f5af981831ea8a707090c1b09a47c25b75d86b5a)
the message needs to be destroyed before the view is detached which
in turn detaches the resolver which checks that all resources have
been returned.
Aram Sargsyan [Thu, 16 Nov 2023 13:30:32 +0000 (13:30 +0000)]
Emit "no servers could be reached" also for UDP setup failure
When all the servers are exhausted for UDP setup emit "no servers
could be reached" in udp_ready(). This message can also be emitted
for a recv_done() error and for TCP connection failure similarly.
Tom Krizek [Mon, 6 Nov 2023 13:45:07 +0000 (14:45 +0100)]
Update system test documentation
Rewrite and reorganize the test documentation to focus on the pytest
runner, omit any mentions of the legacy runner which are no longer
relevant, and mention a few pytest tricks.
Ondřej Surý [Wed, 29 Nov 2023 08:01:56 +0000 (09:01 +0100)]
Refactor the handling of isc_mem overmem condition
Previously, there were two methods of working with the overmem
condition:
1. hi/lo water callback - when the overmem condition was reached
for the first time, the water callback was called with HIWATER
mark and .is_overmem boolean was set internally. Similarly,
when the used memory went below the lo water mark, the water
callback would be called with LOWATER mark and .is_overmem
was reset. This check would be called **every** time memory
was allocated or freed.
2. isc_mem_isovermem() - a simple getter for the internal
.is_overmem flag
This commit refactors removes the first method and move the hi/lo water
checks to the isc_mem_isovermem() function, thus we now have only a
single method of checking overmem condition and the check for hi/lo
water is removed from the hot path for memory contexts that doesn't use
overmem checks.
Michał Kępień [Tue, 21 Nov 2023 09:18:52 +0000 (10:18 +0100)]
Move job definitions to the proper place
The definitions of the "ci-variables" and "cross-version-config-tests"
GitLab CI jobs were accidentally added in the .gitlab-ci.yml section
that claims to only contain job templates. Move the definitions of
these two jobs to a more appropriate location in .gitlab-ci.yml, without
changing the job definitions themselves.
Michał Kępień [Tue, 21 Nov 2023 09:18:52 +0000 (10:18 +0100)]
Drop the TARBALL_EXTENSION variable
All currently supported BIND 9 branches use xz-packed tarballs for
source code distribution. Having a variable with a lengthy name that
only holds two characters does not improve readability - it was only
useful for maintaining .gitlab-ci.yml consistency between BIND 9.11 and
all the newer branches, but that era has come to an end a while ago.
Replace all occurrences of the TARBALL_EXTENSION variable in
.gitlab-ci.yml with a fixed string ("xz") to simplify the contents of
that file.
projects / thirdparty / bind9.git / log
