Mark Andrews [Tue, 19 Sep 2023 04:06:15 +0000 (14:06 +1000)]
Wait for the test zone to finish re-loading
'rndc thaw' initiates asynchrous loading of all the zones
similar to 'rndc load'. Wait for the test zone's load to
complete before testing that it is updatable again.
Change dns_message_create() function to accept memory pools
Instead of creating new memory pools for each new dns_message, change
dns_message_create() method to optionally accept externally created
dns_fixedname_t and dns_rdataset_t memory pools. This allows us to
preallocate the memory pools in ns_client and dns_resolver units for the
lifetime of dns_resolver_t and ns_clientmgr_t.
Fix the incoming transfers' "Needs Refresh" state in stats channel
The "Needs Refresh" flag is exposed in two places in the statistics
channel: first - there is a state called "Needs Refresh", when the
process hasn't started yet, but the zone needs a refresh, and second
- there there is a field called "Additional Refresh Queued", when the
process is ongoing, but another refresh is queued for the same zone.
The DNS_ZONEFLG_NEEDREFRESH flag, however, is set only when there is
an ongoing zone transfer and a new notify is received. That is, the
flag is not set for the first case above.
In order to fix the issue, use the DNS_ZONEFLG_NEEDREFRESH flag only
when the zone transfer is running, otherwise, decide whether a zone
needs a refresh using its refresh and expire times.
xfrin: rename XFRST_INITIALSOA to XFRST_ZONEXFRREQUEST
The XFRST_INITIALSOA state in the xfrin module is named like that,
because the first RR in a zone transfer must be SOA. However, the
name of the state is a bit confusing (especially when exposed to
the users with statistics channel), because it can be mistaken with
the refresh SOA request step, which takes place before the zone
transfer starts.
Rename the state to XFRST_ZONEXFRREQUEST (i.e. Zone Transfer Request).
During that step the state machine performs several operations -
establishing a connection, sending a request, and receiving/parsing
the first RR in the answer.
Show the local and remote addresses for the "Refresh SOA" query
Currently in the statsistics channel's incoming zone transfers list
the local and remote addresses are shown only when the zone transfer
is already running. Since we have now introduced the "Refresh SOA"
state, which shows the state of the SOA query before the zone transfer
is started, this commit implements a feature to show the local and
remote addresses for the SOA query, when the state is "Refresh SOA".
Improve the "Duration (s)" field of the incoming xfers in stats channel
Improve the "Duration (s)" field, so that it can show the duration of
all the major states of an incoming zone transfer process, while they
are taking place. In particular, it will now show the duration of the
"Pending", "Refresh SOA" and "Deferred" states too, before the actual
zone transfer starts.
Add the "Refresh SOA" state for the incoming zone transfers
With adding this state to the statistics channel, it can now show
the zone transfer in this state instead of as "Pending" when the
zone.c module is performing a refresh SOA request, before actually
starting the transfer process. This will help to understand
whether the process is waiting because of the rate limiter (i.e.
"Pending"), or the rate limiter is passed and it is now waiting for
the refresh SOA query to complete or time out.
Check zone transfer transports in the statistics channel
Add two more secondary zones to ns3 to be transferred from ns1,
using its IPv6 address for which the 'tcp-only' is set to 'yes'.
Check the statistics channel's incoming zone transfers information
to confirm that the expected transports were used for each of the
SOA query cases (UDP, TCP, TLS), and also for zone transfers (TCP,
TLS).
Aram Sargsyan [Wed, 23 Aug 2023 10:46:44 +0000 (10:46 +0000)]
Expose the SOA query transport type used before/during XFR
Add a new field in the incoming zone transfers section of the
statistics channel to show the transport used for the SOA request.
When the transfer is started beginning from the XFRST_SOAQUERY state,
it means that the SOA query will be performed by xfrin itself, using
the same transport. Otherwise, it means that the SOA query was already
performed by other means (e.g. by zone.c:soa_query()), and, in that
case, we use the SOA query transport type information passed by the
'soa_transport_type' argument, when the xfrin object was created.
Mark Andrews [Fri, 11 Aug 2023 03:28:05 +0000 (13:28 +1000)]
Wait for slow zone transfer to complete before ending test
This allows the statistics channel to be viewed in a browser while
the transfer is in progress. Also set the transfer format to
one-answer to extend the amount of time the re-transfer takes.
When running the statschannel test on its own, use
<http://10.53.0.3:5304/xml/v3/xfrins> to see the output.
Mark Andrews [Thu, 6 Jul 2023 04:00:48 +0000 (14:00 +1000)]
Provide thread safe access to dns_xfrin_t state
dns_xfrin_t state may be accessed from different threads when
when reporting transfer state. Ensure access is thread safe by
using atomics and locks where appropriate.
Aram Sargsyan [Tue, 30 May 2023 15:00:33 +0000 (15:00 +0000)]
Add a test case for checking zone transfers in statschannel
Use the named -T transferslowly test options to slow down a zone
transfer from the primary server, and test that it's correctly
exposed in the statistics channel of the secondary server, while
it's in-progress.
Aram Sargsyan [Tue, 30 May 2023 14:32:02 +0000 (14:32 +0000)]
dns_transport: use const arguments in getters when possible
In some dns_transport getter functions it's possible to use a
const dns_transport_t as the first argument instead of just
dns_transport_t. Convert the function prototypes to use const.
Explicitly cast chars to unsigned chars for <ctype.h> functions
Apply the semantic patch to catch all the places where we pass 'char' to
the <ctype.h> family of functions (isalpha() and friends, toupper(),
tolower()).
Add semantic patch to explicitly cast chars to unsigned for ctype.h
Add a semantic patch to catch all the places where we pass 'char' to the
<ctype.h> family of functions (isalpha() and friends, toupper(),
tolower()). While it generally works because the way how these
functions are constructed in the libc, it's safer to do the explicit
cast.
Michal Nowak [Thu, 31 Aug 2023 16:55:36 +0000 (18:55 +0200)]
Add a Sphinx role for linking CVEs to the ISC Knowledgebase
The new :cve: Sphinx role takes a CVE number as an argument and creates
a hyperlink to the relevant ISC Knowledgebase document that might have
more up-to-date or verbose information than the relevant release note.
This makes reaching ISC Knowledgebase pages directly from the release
notes easier.
Make all CVE references in the release notes use the new Sphinx role.
Use the new isc_sockaddr_hash_ex() to fix QID table hashing
The QID table hashing used a custom merging of the sockaddr, port and id
into a single hashvalue. Normalize the QID table hashing function to
use isc_hash32 API for all the values.
Use incremental hashing in the isc_sockaddr_hash() function
Instead of copying address back and forth when hashing addr+port, we can
use incremental hashing. Additionally, switch from 64-bit
isc_hash_function to 32-bit isc_hash32() as the resulting value is
32-bit.
Completely remove the Unix Domain Socket support from BIND 9
The Unix Domain Sockets support in BIND 9 has been completely disabled
since BIND 9.18 and it has been a fatal error since then. Cleanup the
code and the documentation that suggest that Unix Domain Sockets are
supported.
Tom Krizek [Tue, 19 Sep 2023 15:20:32 +0000 (17:20 +0200)]
Rename convenience symlink to pytest artifacts
The previous symlink name convention was prone to name collisions If a
system test contained both a shell test and a pytest module of the same
name (e.g. dnstap test has both tests.sh and tests_dnstap.py), then
these would have the same convenience symlink, which could cause test
setup issues as well as confusion when examining test artifacts.
Update the naming convention to include the full pytest module name.
This results in a slightly more verbose names for shell tests (e.g.
dnstap_sh_dnstap instead of the previous dnstap_dnstap), but it removes
the chance of a collision.
Tom Krizek [Tue, 15 Aug 2023 11:55:56 +0000 (13:55 +0200)]
Use integers for ports fixtures in pytest
Reorganize individual port fixtures and re-use the ports fixture to
obtain their number. Store it as integer and only cast it to string when
setting it as environment variable.
Tom Krizek [Thu, 7 Sep 2023 13:21:54 +0000 (15:21 +0200)]
Remove legacy runner support from conftest.py
Remove code fork for legacy runner, reorganize imports and move a
pylint-silencing snippet to the top of the file. The rest of the code
was just unindented.
Tom Krizek [Tue, 15 Aug 2023 11:40:13 +0000 (13:40 +0200)]
Remove pytest invocation from legacy runner
In order to python system tests, pytest (runner) has to be used
directly. This makes it possible to simplify the pytest runner and make
its behavior simpler and easier to extend.
The legacy runner can still be used to run shell system tests.
Tom Krizek [Fri, 8 Sep 2023 10:44:08 +0000 (12:44 +0200)]
Remove make check invocation from legacy.run.sh
The legacy runner no longer uses make check. Ensure the legacy runner
script doesn't interact with that automake target in any way. The legacy
runner script remains available to execute the legacy runner, but there
is no out-of-the box support for running tests in parallel. Other tools
such as xargs can be utilized for that.
Tom Krizek [Wed, 6 Sep 2023 11:43:18 +0000 (13:43 +0200)]
ci: switch OpenBSD job to use make check
Invoking pytest directly provides a better formatted output and more
flexibility. However, it's prudent to verify that `make check` keeps
working as expected. Use it in the OpenBSD job which isn't executed as
frequently and its output is of least concern.
Tom Krizek [Tue, 5 Sep 2023 08:29:13 +0000 (10:29 +0200)]
Modify custom-test-driver to interpret JUnit results
Pytest provides JUnit output and uses different exit codes from
Automake. Use the conversion script to interpret the JUnit test results
from python rather than relying on the status code.
Tom Krizek [Tue, 5 Sep 2023 14:16:20 +0000 (16:16 +0200)]
Convert JUnit XML from pytest into Automake .trs files
It's important to parse the JUnit result file rather than relying on the
exit code from pytest, which has a different meaning. Include a .trs test
result for each test case and set an exit code which is most appropriate
as the aggregate result (e.g. it will be set to 77 (SKIP) if there's at
least one test case that was skipped).
Tom Krizek [Mon, 4 Sep 2023 11:41:09 +0000 (13:41 +0200)]
Remove redundant dependency checks for system tests
Dependencies for these tests are already checked in prereq.sh - if the
dependencies are missing, these tests will be skipped. The extra
dependency check in Makefile.am is extraneous and only applied for the
legacy test runner.
Tom Krizek [Thu, 31 Aug 2023 11:18:17 +0000 (13:18 +0200)]
Fix pytest module detection for run.sh
To allow concurrent invocations of pytest, it is necessary to assign
ports properly to avoid conflicts. In order to do that, pytest needs to
know a complete list of all test modules.
When pytest is invoked from run.sh, the current working directory is the
system test directory. To properly detect other tests, the conftest.py
has to look in the bin/tests/system directory, rather than the current
working directory.
Tom Krizek [Wed, 30 Aug 2023 11:51:05 +0000 (13:51 +0200)]
danger: check system test convetions for pytest runner
When adding a new system test, it might easy to forget to add the
required files for the pytest runner or break a naming convention. Add
danger checks to cover these cases.
Tom Krizek [Wed, 30 Aug 2023 11:37:09 +0000 (13:37 +0200)]
Rename allow-query pytest glue file
To conform with the expected naming convention, the pytest glue file for
the `allow-query` test should use underscore as the word separator in
the python file name: allow-query/tests_sh_allow_query.py
Tom Krizek [Mon, 18 Sep 2023 15:20:01 +0000 (17:20 +0200)]
Treat bin/tests/system/_common as non-temp directory
The _common directory is a special case directory which contains shared
files for other system test directories. Make sure it's tracked in git
and not deleted during temporary directory cleanup.
Tom Krizek [Mon, 18 Sep 2023 15:25:17 +0000 (17:25 +0200)]
Rename system test directory with common files to _common
The old name "common" clashes with the convention of system test
directory naming. It appears as a system test directory, but it only
contains helper files.
To reduce confusion and to allow automatic detection of issues with
possibly missing test files, rename the helper directory to "_common".
The leading underscore indicates the directory is different and the its
name can no longer be confused with regular system test directories.
Add assertion failure when adding to hashmap when iterating
When iterating the table, we can't add new nodes to the hashmap because
we can't assure that we are not adding the new node before the iterator.
This also applies to rehashing - which might be triggered by both
isc_hashmap_add() and isc_hashmap_delete(), but not
isc_hashmap_iter_delcurrent_next().
Mark Andrews [Tue, 19 Sep 2023 01:42:03 +0000 (11:42 +1000)]
Fix hashmap iteration
When isc_hashmap_iter_delcurrent_next calls hashmap_delete_node
nodes from the front of the table could be added to the end of
the table resulting in them being returned twice. Detect when
this is happening and prevent those nodes being returned twice
buy reducing the effective size of the table by one each time
it happens.
Replace the linked list of TCP dispatches with hash table
Reusing TCP connections with dns_dispatch_gettcp() used linear linked
list to lookup existing outgoing TCP connections that could be reused.
Replace the linked list with per-loop cds_lfht hashtable to speedup the
lookups. We use cds_lfht because it allows non-unique node insertion
that we need to check for dispatches in different connection states.
Instead of high number of dispatches (4 * named_g_udpdisp)[1], make the
dispatches bound to threads and make dns_dispatchset_t create a dispatch
for each thread (event loop).
This required couple of other changes:
1. The dns_dispatch_createudp() must be called on loop, so the isc_tid()
is already initialized - changes to nsupdate and mdig were required.
2. The dns_requestmgr had only a single dispatch per v4 and v6. Instead
of using single dispatch, use dns_dispatchset_t for each protocol -
this is same as dns_resolver.
Looking up unique message ID in the dns_dispatch has been using custom
hash tables. Rewrite the custom hashtable to use cds_lfht API, removing
one extra lock in the cold-cache resolver hot path.
Ondřej Surý [Mon, 28 Aug 2023 08:08:59 +0000 (10:08 +0200)]
Refactor isc_hashmap to accept custom match function
Refactor isc_hashmap to allow custom matching functions. This allows us
to have better tailored keys that don't require fixed uint8_t arrays,
but can be composed of more fields from the stored data structure.
projects / thirdparty / bind9.git / log
