Alessio Podda [Wed, 7 May 2025 12:52:11 +0000 (14:52 +0200)]
Replace per-zone lock buckets with global buckets
Qpzone employs a locking strategy where rwlocks are grouped into
buckets, and each zone gets 17 buckets.
This strategy is suboptimal in two ways:
- If named is serving a single zone or a zone is the majority of the
traffic, this strategy pretty much guarantees contention when using
more than a dozen threads.
- If named is serving many small zones, it causes substantial memory
usage.
This commit switches the locking to a global table initialized at start
time. This should have three effects:
- Performance should improve in the single zone case, since now we are
selecting from a bigger pool of locks.
- Memory consumption should go down significantly in the many zone
cases.
- Performance should not degrade substantially in the many zone cases.
The reason for this is that, while we could have substantially more
zones than locks, we can query/edit only O(num threads) at the same
time. So by making the global table much bigger than the expected
number of threads, we can limit contention.
chg: dev: Extract the resigning heap into a separate struct
In the current implementation, the resigning heap is part of the zone
database. This leads to a cycle, as the database has a reference to its
nodes, but each node needs a reference to the database.
This MR splits the resigning heap into its own separate struct, in order
to help breaking the cycle.
Merge branch 'alessio/split-qpzone-heap-from-qpdb' into 'main'
In the current implementation, the resigning heap is part of the zone
database. This leads to a cycle, as the database has a reference to its
nodes, but each node needs a reference to the database.
This MR splits the resigning heap into its own separate struct, in order
to help breaking the cycle.
Recovering the node lock from a pointer to the header and a pointer to
the db is a common operation. This commit abstracts it away into a
function, so that the node lock selection logic may be modified more
easily.
Mark Andrews [Wed, 9 Jul 2025 00:55:28 +0000 (10:55 +1000)]
fix: dev: Fix a possible crash when adding a zone while recursing
A query for a zone that was not yet loaded may yield an unexpected result such as a CNAME or DNAME, triggering an assertion failure. This has been fixed.
Petr Menšík [Tue, 10 Jun 2025 12:35:03 +0000 (14:35 +0200)]
Add few extra WANT_QUERYTRACE logs into resume_qmin
Print optionally a bit more details not passed to event in case
dns_view_findzonecut returns unexpected result. Result would be
visible later in foundevent, but found fname would be lost. Print it
into the log.
Petr Mensik [Tue, 3 Jun 2025 19:00:58 +0000 (21:00 +0200)]
Handle CNAME and DNAME in resume_min in a special way
When authoritative zone is loaded when query minimization query for the
same zone is already pending, it might receive unexpected result codes.
Normally DNS_R_CNAME would follow to query_cname after processing sent
events, but dns_view_findzonecut does not fill CNAME target into
event->foundevent. Usual lookup via query_lookup would always have that
filled.
Ideally we would restart the query with unmodified search name, if
unexpected change from recursing to local zone cut were detected. Until
dns_view_findzonecut is modified to export zone/cache source of the cut,
at least fail queries which went into unexpected state.
Michał Kępień [Tue, 8 Jul 2025 08:27:22 +0000 (10:27 +0200)]
Fix named-makejournal man page installation
The man page for named-makejournal is erroneously not installed when
building from a source tarball. Add that man page to the appropriate
lists in the build system so that it is installed both when building
from a Git repository and from a source tarball.
fix: usr: Clean enough memory when adding new ADB names/entries under memory pressure
The ADB memory cleaning is opportunistic even when we are under
memory pressure (in the overmem condition). Split the opportunistic
LRU cleaning and overmem cleaning and make the overmem cleaning
always cleanup double of the newly allocated adbname/adbentry to
ensure we never allocate more memory than the assigned limit.
Merge branch 'ondrej/enforce-memory-cleanup-in-ADB-when-overmem' into 'main'
Ondřej Surý [Wed, 25 Jun 2025 10:05:14 +0000 (12:05 +0200)]
When overmem, clean enough memory when adding new ADB names/entries
The purge_stale_names()/purge_stale_entries() is opportunistic even when
we are under memory pressure (overmem). Split the opportunistic LRU
cleaning and overmem cleaning. This makes the stale purging much
simpler as we don't have to try that hard and makes the overmem cleaning
always cleanup double the amount of the newly allocated ADB name/entry.
It's possible to use pytest.mark.flaky, which achieves the exact same
thing as our custom-defined isctest.mark.flaky -- attempts to rerun the
test on failure, but only is flaky package is available.
Mark secondary.kasp test case as flaky on freebsd13
The test_kasp_case[secondary.kasp] can sometimes fail on freebsd13. It
appears the test gets stuck on some operation which should be very
quick, but for some reason takes at least a few seconds, causing the
cb_ixfr_is_signed() function to time out.
In one of the cases I investigated, it wasn't a query/response that
caused a timeout, but rather some operation in between. The test
attempts to read from a keyfile/statefile, but I see no reason why that
should block.
In any case, try to increase the timeout for the verification, as that
shouldn't hurt. Also allow the test to be re-run on freebsd13, as it's
likely to be caused by some odd behaviour on that platform -- the issue
doesn't appear anywhere else.
The check "unix socket message counts" sometimes fails with "dnstap
output file smaller than expected". This only happens on freebsd13 and
can't be reproduced easily. There was an attempt to decrease the
required file size in the past, but apparently, the issue can still
occur.
The serve_stale test has some inherent instabilities affecting many
different checks. While the failure rate isn't too high (about four
failures in past three weeks of nightlies), it gets ignored, because the
test has been unstable for a very long time.
This removes a leftover check which should've been removed in a prior
change (see #5244). The softhsm2 failures when attempting to delete the
token should be ignored.
Previously, the one-second sleep was unreliable, as it didn't properly
indicate that the rndc reconfig has been processed. The "test 'rndc
reconfig' with a broken config" check would sometimes fail under TSAN
in CI, because the previous rndc reconfig was still ongoing, and the
subsequent rndc reconfig was ignored.
These tests have been unstable under TSAN in the past, but it appears
that the same failure mode can happen outside of TSAN tests as well.
These tests have produced 12 failures combined in the past three weeks
in nightlies.
The fetchlimit test has failed 8 times in the nightly CI over the past
three weeks. That makes the overall failure rate somewhere around 1 %,
which isn't a lot, but is still annoying when lots of testing is going
on.
Mark Andrews [Mon, 7 Jul 2025 02:21:58 +0000 (12:21 +1000)]
fix: test: rndc test: second 'rndc reconfig' happens too soon
Rndc test "test 'rndc reconfig' with a broken config" was failing
intermittently.
Wait for 'running' to be logged rather than just using 'sleep 1' before
calling 'rndc reconfig' a second time to get the expected error message
rather than 'reconfig request ignored: already running'.
Closes #5408
Merge branch '5408-rndc-test-second-rndc-reconfig-happens-too-soon' into 'main'
Mark Andrews [Wed, 2 Jul 2025 23:13:11 +0000 (09:13 +1000)]
rndc test: second 'rndc reconfig' happens too soon
Rndc test "test 'rndc reconfig' with a broken config" was failing
intermittently.
Wait for 'running' to be logged rather than just using 'sleep 1' before
calling 'rndc reconfig' a second time to get the expected error message
rather than 'reconfig request ignored: already running'.
new: ci: Run an additional respdiff job for merge requests and schedules
On MRs it uses the merge target as the reference.
In schedules it uses the latest released version for this branch as the reference.
This MR lays the ground work for using respdiff on non-standard configurations (like ECS) in the public repo, see https://gitlab.isc.org/isc-private/bind9/-/merge_requests/807#note_573140.
To reduce the future hassle when maintaining the -S version, most of the work (including an added job, so we know that it actually works) is done here.
Merge branch 'stepan/respdiff-against-merge-target-or-last-release' into 'main'
Mark Andrews [Sun, 6 Jul 2025 13:09:13 +0000 (23:09 +1000)]
fix: dev: Separate out adbname type flags
There are three adbname flags that are used to identify different
types of adbname lookups when hashing rather than using multiple
hash tables. Separate these to their own structure element as these
need to be able to be read without locking the adbname structure.
Closes #5404
Merge branch '5404-seperate-out-adbname-type-flags' into 'main'
Mark Andrews [Tue, 1 Jul 2025 06:45:39 +0000 (16:45 +1000)]
Separate out adbname flags that are hashed
There are three adbname flags that are used to identify different
types of adbname lookups when hashing rather than using multiple
hash tables. Separate these to their own structure element as these
need to be able to be read without locking the adbname structure.
[CVE-2025-40777] sec: usr: Fix a possible assertion failure when using the 'stale-answer-client-timeout 0' option
In specific circumstances the :iscman:`named` resolver process could
terminate unexpectedly when stale answers were enabled and the
``stale-answer-client-timeout 0`` configuration option was used.
This has been fixed.
See isc-projects/bind9#5372
Merge branch '5372-security-serve-stale-crash-on-insist-unreachable' into 'v9.21.10-release'
Aram Sargsyan [Wed, 18 Jun 2025 13:32:03 +0000 (13:32 +0000)]
Reset DNS_DBFIND_STALETIMEOUT in query_lookup()
If ns__query_start() is called because of a chained query (e.g.
after encountering a CNAME), a previously set DNS_DBFIND_STALETIMEOUT
flag on the query's 'dboptions' field can cause an assertion
failure if the new query's 'stalefirst' value is not true (e.g. if the
target qname is an authoritative zone for the server). Reset the
DNS_DBFIND_STALETIMEOUT flag in the query_lookup() function before
evaluating the 'stalefirst' value, and make sure to assign a fresh
value to the `stalefirst' flag instead of conditionally assigning it
only if the value is 'true'.
dangerfile.py checked for new configure switches in `configure.ac`,
these were annotated with "# [pairwise:..." in a leading line. Meson
reads those from `meson_options.txt` instead.
fix: usr: Fix the default interface-interval from 60s to 60m
When the interface-interval parser was changed from uint32 parser to
duration parser, the default value stayed at plain number `60` which
now means 60 seconds instead of 60 minutes. The documentation also
incorrectly states that the value is in minutes. That has been fixed.
Closes #5246
Merge branch '5246-fix-default-interface-interval' into 'main'
Ondřej Surý [Tue, 18 Mar 2025 13:05:39 +0000 (14:05 +0100)]
Fix the default interface-interval docs and default value
When the interface-interval parser was changed from uint32 parser to
duration parser, the default value stayed at plain 60 which now means 60
seconds instead of 60 minutes. Fix the default value and the
documentation to match the reality.
Colin Vidal [Mon, 30 Jun 2025 12:51:20 +0000 (14:51 +0200)]
new: test: add startup root DNSKEY refresh system test
Root trust anchors are automatically updated as described in RFC5011.
Add a system test which ensures the root DNSKEYs are always queried by
named during startup.
Because this test uses real internet DNS root servers, it is enabled
only when `CI_ENABLE_LIVE_INTERNET_TESTS` is set.
Colin Vidal [Tue, 24 Jun 2025 09:55:42 +0000 (11:55 +0200)]
add startup root DNSKEY refresh system test
Root trust anchors are automatically updated as described in RFC5011.
Add a system test which ensures the root DNSKEYs are always queried by
named during startup.
Because this test uses real internet DNS root servers, it is enabled
only when `CI_ENABLE_LIVE_INTERNET_TESTS` is set.
Ondřej Surý [Mon, 30 Jun 2025 11:23:38 +0000 (13:23 +0200)]
fix: dev: Prevent false sharing for the .inuse member of isc_mem_t
Change the .inuse member of memory context to have a loop-local
variable, so there's no contention even when the same memory
context is shared among multiple threads.
Closes #5354
Merge branch '5354-prevent-false-sharing-in-isc_mem' into 'main'
Ondřej Surý [Wed, 4 Jun 2025 16:14:23 +0000 (18:14 +0200)]
Change the .inuse member of isc_mem to be per-thread/per-loop
The .inuse member was causing a lot of contention between threads using
the same memory context. Scather the .inuse and .overmem members of
isc_mem_t structure to be an per-tid array of variables to reduce the
contention as the writes are now independent of each other.
The array uses one tad bit nasty trick, as ISC_TID_UNKNOWN is now -1,
the array has been sized to fit the unknown tid with [-1] index into the
array accomplished with `ctx->stat = &ctx->stat_s[1];`. It will not win
a beauty contest, but it works seamlessly by just passing `isc_tid()` as
an index into the array.
The caveat here is that gathering the real inuse value requires walking
the whole array for all registered tid values (isc_tid_count()). The
gather part happens only when statistics are being gathered or when
isc_mem_isovermem() is called. As the isc_mem_isovermem() call happens
only when new data is being added to cache or ADB, it doesn't happen on
the hottest (read-only) path and according to the measurements, it
doesn't slow down neither the cold cache nor the hot cache latency.
Ondřej Surý [Thu, 5 Jun 2025 10:19:43 +0000 (12:19 +0200)]
Don't use ssize_t for storing difference between sizes
As POSIX guarantees only that the type ssize_t shall be capable of
storing values at least in the range [-1, {SSIZE_MAX}], it can't be used
to calculate the difference between two memory sizes. Change the logic
for junk filling to test whether the new size is larger than old size
and then use size_t as the result will be always positive.
Ondřej Surý [Wed, 4 Jun 2025 15:43:34 +0000 (17:43 +0200)]
Remove .hi_called member of isc_mem_t structure
The .hi_called member was dead structure member and it hasn't been used
since the overmem callback has been removed in commit 14bdd21e0a7ad5f115bb2427d4f88fe7a84e9324.
Ondřej Surý [Wed, 4 Jun 2025 08:35:57 +0000 (10:35 +0200)]
Delete jemalloc arena support from isc_mem
The jemalloc arena in isc_mem was added to solve runaway memory problem
for outgoing TCP connections. In the end, this was a red herring and
the jemalloc arena code is now unused (via e28266bf). Remove the
support for jemalloc memory arenas as we can restore this at any time if
we need it ever again, but right now it's just a dead code.
Ondřej Surý [Wed, 25 Jun 2025 06:25:41 +0000 (08:25 +0200)]
Fix implicit headers when using isc/overflow.h header
In jemalloc_shim.h, we relied on including <isc/overflow.h> implicitly
instead of explicitly and same was happening inside isc/overflow.h - the
stdbool.h (for bool type) was being included implicitly instead of
explicitly.
Aydın Mercan [Tue, 24 Jun 2025 13:30:15 +0000 (16:30 +0300)]
do not install manpages for unbuilt binaries
Building and installing from a git release installed all manpages
unconditionally even if binaries like dnstap-read were disabled and not
built.
Now the manpage configuration checks for such cases and also cleans up
remaining artifacts and unnecessary pages if the build directory is
reconfigured.
Ondřej Surý [Sat, 28 Jun 2025 12:06:05 +0000 (14:06 +0200)]
chg: dev: Change isc_tid to be isc_tid_t type (a signed integer type)
Change the internal type used for isc_tid unit to isc_tid_t to hide the
specific integer type being used for the 'tid'. Internally, the isc_tid
unit is now using signed integer type. This allows us to have negatively
indexed arrays that works both for threads with assigned tid and the
threads with unassigned tid. Additionally, limit the number of threads
(loops) to 512 (compile time default).
Ondřej Surý [Wed, 4 Jun 2025 15:54:20 +0000 (17:54 +0200)]
Convert the isc/tid.h to use own signed integer isc_tid_t type
Change the internal type used for isc_tid unit to isc_tid_t to hide the
specific integer type being used for the 'tid'. Internally, the signed
integer type is being used. This allows us to have negatively indexed
arrays that works both for threads with assigned tid and the threads
with unassigned tid. This should be used only in specific situations.
Štěpán Balážik [Sat, 28 Jun 2025 10:51:59 +0000 (10:51 +0000)]
fix: nil: Only run ci-orphaned-anchors on MR events
Now, it is also run in schedules and most annoyingly on push which means
that it is run twice on a push to a branch where a MR exists and `.gitlab-ci.yml` is changed.
This was an oversight in https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/10654
Merge branch 'stepan/remove-additional-pipeline' into 'main'
Štěpán Balážik [Fri, 27 Jun 2025 15:20:29 +0000 (15:20 +0000)]
fix: nil: Move root zone mirror system test to a separate directory
This test doesn't require artifact checking but when bundled in the same
directory with the shell based tests, the `system:clang:tsan` job was
failing non-deterministically.
An example of the job failing and succeeding on the same commit:
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/5809299
- https://gitlab.isc.org/isc-projects/bind9/-/jobs/5809447
Merge branch 'stepan/move-root-zone-mirror-test-to-a-separate-directory' into 'main'
Štěpán Balážik [Fri, 27 Jun 2025 13:51:05 +0000 (15:51 +0200)]
Move root zone mirror system test to a separate directory
This test doesn't require artifact checking but when bundled in the same
directory with the shell based tests, the `system:clang:tsan` job was
failing non-deterministically.
Nicki Křížek [Fri, 27 Jun 2025 15:03:54 +0000 (17:03 +0200)]
chg: test: Improve pytest log output
- increase clarity of multiline messages
- support `isc.query.*()` query&response logging
- replace use of `print()` statement with proper logging
- omit empty lines from test result output
Merge branch 'nicki/improve-pytest-logging' into 'main'
Nicki Křížek [Thu, 26 Jun 2025 16:20:06 +0000 (18:20 +0200)]
Log assertion failures right after test result
The extra messages are typically traceback from assertion failures.
Previously, they'd be printed only after all individual test case
results have been printed. That made it difficult to pair the traceback
to the failing test in some cases, as the node information (aka test
name) might not always be present.
Instead, log any extra messages related to a particular test failure
directly after reporting its result, making the failure details more
readily available and easy to connect with a particular test case.
Nicki Křížek [Thu, 26 Jun 2025 14:14:50 +0000 (16:14 +0200)]
Log query and response when using isctest.query.*
Make sure the queries and responses are logged at the DEBUG level, which
may provide useful information in case of failing tests.
This doesn't seem to significantly increase the overall artifacts size.
Previously, pytest.log.txt files from all system tests would take around
3 MB, with this change, it's around 8 MB).
Nicki Křížek [Tue, 17 Jun 2025 15:40:07 +0000 (17:40 +0200)]
Add options for query&response logging to pytest
In some cases, it's useful to log the sent and received DNS messages.
Add options to enable this on demand. Query is only logged the first
time it's sent, since it doesn't change. If response logging is turned
on, then each response is logged, since it might be different every
time.
Nicki Křížek [Tue, 17 Jun 2025 15:33:22 +0000 (17:33 +0200)]
Indent multiline output in pytest logging
When multiline message is logged, indent all but the first line (which
will be preceeded by the LOG_FORMAT). This improves the clarity of logs,
as it's immediately clear which lines are regular log output, and which
ones are multiline debug output.
Adjust the isctest.run.cmd() stdout/stderr logging to this new format.
Nicki Křížek [Tue, 17 Jun 2025 15:21:33 +0000 (17:21 +0200)]
Don't log empty test result messages
The messages obtained from test results may contain stuff like detailed
failure/error information, tracebacks etc. In many cases, the message
will be empty, in which case it doesn't need to be logged.
For an example, run test with many test cases, e.g.
verify/test_verify.py, and inspect the tail of the pytest.log.txt before
and after this commit.
Nicki Křížek [Tue, 17 Jun 2025 13:47:48 +0000 (15:47 +0200)]
Replace print statements in checkds test
Use isctest.log logging facility for consistent and predictable logging
output rather than using print(). Remove writes of stderr, as that
output will be logged in the debug log in case the commands called with
isctest.run.cmd() fails.
Petr Špaček [Wed, 25 Jun 2025 15:53:25 +0000 (17:53 +0200)]
Simplify maintenance of NO_BUILD_TEST_PREREQ CI hack
Our split between build and test phases in CI triggers odd corner case
in Meson:
- Newer Meson versions (1.7.0+) do not build test targets as part of
"all" target.
- We copy build artifacts from build phase into test container.
- meson test --no-rebuild does not build test artifacts even if they are
missing.
- To build these test binaries Meson has special target
"meson-test-prereq". This target exists only in Meson >= 0.63.
- Ubuntu 22.04 has only Meson 0.61.2 so this target does not exist.
To counter this problem, we introduced BUILD_TEST_PREREQ variable in CI
to explicitly build "meson-test-prereq" target in the "build" phase only
inside images with new-enough Meson versions. This worked, but it forced
us to keep track of Meson versions on various
distros and update the variable accordingly.
This commit inverts the logic so we build the special target by default
(in the build phase) and skip building it only if Meson version is too
old. So once we drop the old image, the variable (or rather it's usage)
will be gone and we don't need to touch it for newer images.
We have also considered installing newer Meson into the test image, but
decided to keep the old version around so we can test minimal Meson
version specified in meson.build file.
projects / thirdparty / bind9.git / log
