From: Tom Krizek <tkrizek@isc.org>
Date: Mon, 6 Nov 2023 13:45:07 +0000 (+0100)
Subject: Update system test documentation
X-Git-Tag: v9.19.19~21^2~2
X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=fba295600b80fc034600e9c2962b93ed3397c7b2;p=thirdparty%2Fbind9.git

Update system test documentation

Rewrite and reorganize the test documentation to focus on the pytest
runner, omit any mentions of the legacy runner which are no longer
relevant, and mention a few pytest tricks.
---

diff --git a/bin/tests/system/README b/bin/tests/system/README
deleted file mode 100644
index 36b01ea47bd..00000000000
--- a/bin/tests/system/README
+++ /dev/null
@@ -1,793 +0,0 @@
-Copyright (C) Internet Systems Consortium, Inc. ("ISC")
-
-SPDX-License-Identifier: MPL-2.0
-
-This Source Code Form is subject to the terms of the Mozilla Public
-License, v. 2.0.  If a copy of the MPL was not distributed with this
-file, you can obtain one at https://mozilla.org/MPL/2.0/.
-
-See the COPYRIGHT file distributed with this work for additional
-information regarding copyright ownership.
-
-Introduction
-===
-This directory holds a simple test environment for running bind9 system tests
-involving multiple name servers.
-
-Each system test directory holds a set of scripts and configuration files to
-test different parts of BIND.  The directories are named for the aspect of BIND
-they test, for example:
-
-  dnssec/       DNSSEC tests
-  forward/      Forwarding tests
-  glue/         Glue handling tests
-
-etc.
-
-A system test directory must start with an alphabetic character and may not
-contain any special characters. Only hyphen may be used as a word separator.
-
-Typically each set of tests sets up 2-5 name servers and then performs one or
-more tests against them.  Within the test subdirectory, each name server has a
-separate subdirectory containing its configuration data.  These subdirectories
-are named "nsN" or "ansN" (where N is a number between 1 and 8, e.g. ns1, ans2
-etc.)
-
-The tests are completely self-contained and do not require access to the real
-DNS.  Generally, one of the test servers (usually ns1) is set up as a root
-nameserver and is listed in the hints file of the others.
-
-
-Preparing to Run the Tests
-===
-To enable all servers to run on the same machine, they bind to separate virtual
-IP addresses on the loopback interface.  ns1 runs on 10.53.0.1, ns2 on
-10.53.0.2, etc.  Before running any tests, you must set up these addresses by
-running the command
-
-    sh ifconfig.sh up
-
-as root.  The interfaces can be removed by executing the command:
-
-    sh ifconfig.sh down
-
-... also as root.
-
-The servers use unprivileged ports (above 1024) instead of the usual port 53,
-so they can be run without root privileges once the interfaces have been set
-up.
-
-
-Note for MacOS Users
----
-If you wish to make the interfaces survive across reboots, copy
-org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons
-then run
-
-    launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
-
-... as root.
-
-
-Running the System Tests with pytest
-===
-
-The pytest system test runner is currently in development, but it is the
-recommended way to run tests. Please report issues to QA.
-
-Running an Individual Test
----
-
-pytest -k <test-name>
-
-Note that in comparison to the legacy test runner, some additional tests might
-be picked up when specifying just the system test directory name. To check
-which tests will be executed, you can use the `--collect-only` option. You
-might also be able to find a more specific test name to provide to ensure only
-your desired test is executed. See help for `-k` option in `pytest --help` for
-more info.
-
-It is also possible to run a single individual pytest test case. For example,
-you can use the name test_sslyze_dot to execute just the test_sslyze_dot()
-function from doth/tests_sslyze.py. The entire needed setup and teardown will
-be handled by the framework.
-
-Running All the System Tests
----
-
-Issuing plain `pytest` command without any argument will execute all tests
-sequenatially. To execute them in parallel, ensure you have pytest-xdist
-installed and run:
-
-pytest -n <number-of-workers>
-
-
-Running the System Tests Using the Legacy Runner
-===
-
-!!! WARNING !!!
----
-The legacy way to run system tests is currently being reworked into a pytest
-system test runner described in the previous section. The contents of this
-section might be out of date and no longer applicable. Please try and use the
-pytest runner if possible and report issues and missing features.
-
-Running an Individual Test
----
-The tests can be run individually using the following command:
-
-    sh legacy.run.sh [flags] <test-name> [<test-arguments>]
-
-e.g.
-
-    sh legacy.run.sh [flags] notify
-
-Optional flags are:
-
-    -k              Keep servers running after the test completes.  Each test
-                    usually starts a number of nameservers, either instances
-                    of the "named" being tested, or custom servers (written in
-                    Python or Perl) that feature test-specific behavior.  The
-                    servers are automatically started before the test is run
-                    and stopped after it ends.  This flag leaves them running
-                    at the end of the test, so that additional queries can be
-                    sent by hand.  To stop the servers afterwards, use the
-                    command "sh stop.sh <test-name>".
-
-    -n              Noclean - do not remove the output files if the test
-                    completes successfully.  By default, files created by the
-                    test are deleted if it passes;  they are not deleted if the
-                    test fails.
-
-    -p <number>     Sets the range of ports used by the test.  A block of 100
-                    ports is available for each test, the number given to the
-                    "-p" switch being the number of the start of that block
-                    (e.g. "-p 7900" will mean that the test is able to use
-                    ports 7900 through 7999).  If not specified, the test will
-                    have ports 5000 to 5099 available to it.
-
-Arguments are:
-
-    test-name       Mandatory.  The name of the test, which is the name of the
-                    subdirectory in bin/tests/system holding the test files.
-
-    test-arguments  Optional arguments that are passed to each of the test's
-                    scripts.
-
-
-Running All The System Tests
----
-To run all the system tests, enter the command:
-
-    make [-j numproc] test
-
-The optional "numproc" argument specifies the maximum number of tests that can
-run in parallel.  The default is 1, which means that all of the tests run
-sequentially.  If greater than 1, up to "numproc" tests will run simultaneously,
-new tests being started as tests finish.  Each test will get a unique set of
-ports, so there is no danger of tests interfering with one another.  Parallel
-running will reduce the total time taken to run the BIND system tests, but will
-mean that the output from all the tests sent to the screen will be mixed up
-with one another.
-
-In this case, retention of the output files after a test completes successfully
-is specified by setting the environment variable SYSTEMTEST_NO_CLEAN to 1 prior
-to running make, e.g.
-
-    SYSTEMTEST_NO_CLEAN=1 make [-j numproc] test
-
-while setting environment variable SYSTEMTEST_FORCE_COLOR to 1 forces system
-test output to be printed in color.
-
-
-Format of Test Output
----
-All output from the system tests is in the form of lines with the following
-structure:
-
-    <letter>:<test-name>:<message> [(<number>)]
-
-e.g.
-
-    I:catz:checking that dom1.example is not served by primary (1)
-
-The meanings of the fields are as follows:
-
-<letter>
-This indicates the type of message.  This is one of:
-
-    S   Start of the test
-    A   Start of test (retained for backwards compatibility)
-    T   Start of test (retained for backwards compatibility)
-    E   End of the test
-    I   Information.  A test will typically output many of these messages
-        during its run, indicating test progress.  Note that such a message may
-        be of the form "I:testname:failed", indicating that a sub-test has
-        failed.
-    R   Result.  Each test will result in one such message, which is of the
-        form:
-
-                R:<test-name>:<result>
-
-        where <result> is one of:
-
-            PASS        The test passed
-            FAIL        The test failed
-            SKIPPED     The test was not run, usually because some
-                        prerequisites required to run the test are missing.
-
-<test-name>
-This is the name of the test from which the message emanated, which is also the
-name of the subdirectory holding the test files.
-
-<message>
-This is text output by the test during its execution.
-
-(<number>)
-If present, this will correlate with a file created by the test.  The tests
-execute commands and route the output of each command to a file.  The name of
-this file depends on the command and the test, but will usually be of the form:
-
-    <command>.out.<suffix><number>
-
-e.g. nsupdate.out.test28, dig.out.q3.  This aids diagnosis of problems by
-allowing the output that caused the problem message to be identified.
-
-
-Re-Running the Tests
----
-If there is a requirement to re-run a test (or the entire test suite), the
-files produced by the tests should be deleted first.  Normally, these files are
-deleted if the test succeeds but are retained on error.  The legacy.run.sh
-script automatically calls a given test's clean.sh script before invoking its
-setup.sh script.
-
-Deletion of the files produced by the set of tests (e.g. after the execution of
-make) can be carried out using the command:
-
-    sh cleanall.sh
-
-or
-
-    make testclean
-
-(Note that the Makefile has two other targets for cleaning up files: "clean"
-will delete all the files produced by the tests, as well as the object and
-executable files used by the tests.  "distclean" does all the work of "clean"
-as well as deleting configuration files produced by "configure".)
-
-
-Developer Notes
-===
-This section is intended for developers writing new tests.
-
-
-Overview
----
-As noted above, each test is in a separate directory.  To interact with the
-test framework, the directories contain the following standard files:
-
-prereq.sh   Run at the beginning to determine whether the test can be run at
-            all; if not, we see a R:SKIPPED result.  This file is optional:
-            if not present, the test is assumed to have all its prerequisites
-            met.
-
-setup.sh    Run after prereq.sh, this sets up the preconditions for the tests.
-            Although optional, virtually all tests will require such a file to
-            set up the ports they should use for the test.
-
-tests.sh    Runs the actual tests.  This file is mandatory.
-
-tests_sh_xyz.py  A glue file for the pytest runner for executing shell tests.
-
-clean.sh    Run at the end to clean up temporary files, but only if the test
-            was completed successfully and its running was not inhibited by the
-            "-n" switch being passed to "legacy.run.sh".  Otherwise the
-            temporary files are left in place for inspection.
-
-ns<N>       These subdirectories contain test name servers that can be queried
-	    or can interact with each other.  The value of N indicates the
-	    address the server listens on: for example, ns2 listens on
-	    10.53.0.2, and ns4 on 10.53.0.4.  All test servers use an
-	    unprivileged port, so they don't need to run as root.  These
-	    servers log at the highest debug level and the log is captured in
-	    the file "named.run".
-
-ans<N>      Like ns[X], but these are simple mock name servers implemented in
-            Perl or Python.  They are generally programmed to misbehave in ways
-            named would not so as to exercise named's ability to interoperate
-            with badly behaved name servers.
-
-
-Port Usage
----
-In order for the tests to run in parallel, each test requires a unique set of
-ports.  These are specified by the "-p" option passed to "legacy.run.sh", which
-sets environment variables that the scripts listed above can reference.
-
-The convention used in the system tests is that the number passed is the start
-of a range of 100 ports.  The test is free to use the ports as required,
-although the first ten ports in the block are named and generally tests use the
-named ports for their intended purpose.  The names of the environment variables
-are:
-
-    PORT                     Number to be used for the query port.
-    CONTROLPORT              Number to be used as the RNDC control port.
-    EXTRAPORT1 - EXTRAPORT8  Eight port numbers that can be used as needed.
-
-Two other environment variables are defined:
-
-    LOWPORT                  The lowest port number in the range.
-    HIGHPORT                 The highest port number in the range.
-
-Since port ranges usually start on a boundary of 10, the variables are set such
-that the last digit of the port number corresponds to the number of the
-EXTRAPORTn variable.  For example, if the port range were to start at 5200, the
-port assignments would be:
-
-    PORT = 5200
-    EXTRAPORT1 = 5201
-        :
-    EXTRAPORT8 = 5208
-    CONTROLPORT = 5209
-    LOWPORT = 5200
-    HIGHPORT = 5299
-
-When running tests in parallel (i.e. giving a value of "numproc" greater than 1
-in the "make" command listed above), it is guaranteed that each
-test will get a set of unique port numbers.
-
-
-Writing a Test
----
-The test framework requires up to four shell scripts (listed above) as well as
-a number of nameserver instances to run.  Certain expectations are put on each
-script:
-
-
-General
----
-1. Each of the four scripts will be invoked with the command
-
-    (cd <test-directory> ; sh <script> [<arguments>] )
-
-... so that working directory when the script starts executing is the test
-directory.
-
-2. Arguments can be only passed to the script if the test is being run as a
-one-off with "legacy.run.sh".  In this case, everything on the command line
-after the name of the test is passed to each script.  For example, the command:
-
-    sh legacy.run.sh -p 12300 mytest -D xyz
-
-... will run "mytest" with a port range of 12300 to 12399.  Each of the
-framework scripts provided by the test will be invoked using the remaining
-arguments, e.g.:
-
-   (cd mytest ; sh prereq.sh -D xyz)
-   (cd mytest ; sh setup.sh -D xyz)
-   (cd mytest ; sh tests.sh -D xyz)
-   (cd mytest ; sh clean.sh -D xyz)
-
-No arguments will be passed to the test scripts if the test is run as part of
-a run of the full test suite (e.g. the tests are started with make).
-
-3.  Each script should start with the following lines:
-
-    . ../conf.sh
-
-"conf.sh" defines a series of environment variables together with functions
-useful for the test scripts.
-
-
-prereq.sh
----
-As noted above, this is optional.  If present, it should check whether specific
-software needed to run the test is available and/or whether BIND has been
-configured with the appropriate options required.
-
-    * If the software required to run the test is present and the BIND
-      configure options are correct, prereq.sh should return with a status code
-      of 0.
-
-    * If the software required to run the test is not available and/or BIND
-      has not been configured with the appropriate options, prereq.sh should
-      return with a status code of 1.
-
-    * If there is some other problem (e.g. prerequisite software is available
-      but is not properly configured), a status code of 255 should be returned.
-
-
-setup.sh
----
-This is responsible for setting up the configuration files used in the test.
-
-To cope with the varying port number, ports are not hard-coded into
-configuration files (or, for that matter, scripts that emulate nameservers).
-Instead, setup.sh is responsible for editing the configuration files to set the
-port numbers.
-
-To do this, configuration files should be supplied in the form of templates
-containing tokens identifying ports.  The tokens have the same name as the
-environment variables listed above, but are prefixed and suffixed by the "@"
-symbol.  For example, a fragment of a configuration file template might look
-like:
-
-    controls {
-        inet 10.53.0.1 port @CONTROLPORT@ allow { any; } keys { rndc_key; };
-    };
-
-    options {
-        query-source address 10.53.0.1;
-        notify-source 10.53.0.1;
-        transfer-source 10.53.0.1;
-        port @PORT@;
-        allow-new-zones yes;
-    };
-
-setup.sh should copy the template to the desired filename using the
-"copy_setports" shell function defined in "conf.sh", i.e.
-
-    copy_setports ns1/named.conf.in ns1/named.conf
-
-This replaces the tokens @PORT@, @CONTROLPORT@, @EXTRAPORT1@ through
-@EXTRAPORT8@ with the contents of the environment variables listed above.
-setup.sh should do this for all configuration files required when the test
-starts.
-
-("setup.sh" should also use this method for replacing the tokens in any Perl or
-Python name servers used in the test.)
-
-
-tests.sh
----
-This is the main test file and the contents depend on the test.  The contents
-are completely up to the developer, although most test scripts have a form
-similar to the following for each sub-test:
-
-    1. n=$((n + 1))
-    2. echo_i "prime cache nodata.example ($n)"
-    3. ret=0
-    4. $DIG -p ${PORT} @10.53.0.1 nodata.example TXT > dig.out.test$n
-    5. grep "status: NOERROR" dig.out.test$n > /dev/null || ret=1
-    6. grep "ANSWER: 0," dig.out.test$n > /dev/null || ret=1
-    7. if [ $ret != 0 ]; then echo_i "failed"; fi
-    8. status=$((status + ret))
-
-1.  Increment the test number "n" (initialized to zero at the start of the
-    script).
-
-2.  Indicate that the sub-test is about to begin.  Note that "echo_i" instead
-    of "echo" is used.  echo_i is a function defined in "conf.sh" which will
-    prefix the message with "I:<testname>:", so allowing the output from each
-    test to be identified within the output.  The test number is included in
-    the message in order to tie the sub-test with its output.
-
-3. Initialize return status.
-
-4 - 6. Carry out the sub-test.  In this case, a nameserver is queried (note
-    that the port used is given by the PORT environment variable, which was set
-    by the inclusion of the file "conf.sh" at the start of the script).  The
-    output is routed to a file whose suffix includes the test number.  The
-    response from the server is examined and, in this case, if the required
-    string is not found, an error is indicated by setting "ret" to 1.
-
-7.  If the sub-test failed, a message is printed. "echo_i" is used to print
-    the message to add the prefix "I:<test-name>:" before it is output.
-
-8.  "status", used to track how many of the sub-tests have failed, is
-    incremented accordingly.  The value of "status" determines the status
-    returned by "tests.sh", which in turn determines whether the framework
-    prints the PASS or FAIL message.
-
-Regardless of this, rules that should be followed are:
-
-a.  Use the environment variables set by conf.sh to determine the ports to use
-    for sending and receiving queries.
-
-b.  Use a counter to tag messages and to associate the messages with the output
-    files.
-
-c.  Store all output produced by queries/commands into files.  These files
-    should be named according to the command that produced them, e.g. "dig"
-    output should be stored in a file "dig.out.<suffix>", the suffix being
-    related to the value of the counter.
-
-d.  Use "echo_i" to output informational messages.
-
-e.  Retain a count of test failures and return this as the exit status from
-    the script.
-
-
-tests_sh_xyz.py
----------------
-This glue file is required by the pytest runner in order to find and execute
-the shell tests in tests.sh.
-
-Replace the "xyz" with the system test name and create the file with the
-following contents.
-
-    def test_xyz(run_tests_sh):
-        run_tests_sh()
-
-clean.sh
----
-The inverse of "setup.sh", this is invoked by the framework to clean up the
-test directory.  It should delete all files that have been created by the test
-during its run.
-
-
-Starting Nameservers
----
-As noted earlier, a system test will involve a number of nameservers.  These
-will be either instances of named, or special servers written in a language
-such as Perl or Python.
-
-For the former, the version of "named" being run is that in the "bin/named"
-directory in the tree holding the tests (i.e. if "make test" is being run
-immediately after "make", the version of "named" used is that just built).  The
-configuration files, zone files etc. for these servers are located in
-subdirectories of the test directory named "nsN", where N is a small integer.
-The latter are special nameservers, mostly used for generating deliberately bad
-responses, located in subdirectories named "ansN" (again, N is an integer).
-In addition to configuration files, these directories should hold the
-appropriate script files as well.
-
-Note that the "N" for a particular test forms a single number space, e.g. if
-there is an "ns2" directory, there cannot be an "ans2" directory as well.
-Ideally, the directory numbers should start at 1 and work upwards.
-
-When running a test, the servers are started using "start.sh" (which is nothing
-more than a wrapper for start.pl).  The options for "start.pl" are documented
-in the header for that file, so will not be repeated here.  In summary, when
-invoked by "legacy.run.sh", start.pl looks for directories named "nsN" or
-"ansN" in the test directory and starts the servers it finds there.
-
-
-"named" Command-Line Options
----
-By default, start.pl starts a "named" server with the following options:
-
-    -c named.conf   Specifies the configuration file to use (so by implication,
-                    each "nsN" nameserver's configuration file must be called
-                    named.conf).
-
-    -d 99           Sets the maximum debugging level.
-
-    -D <name>       The "-D" option sets a string used to identify the
-                    nameserver in a process listing.  In this case, the string
-                    is the name of the subdirectory.
-
-    -g              Runs the server in the foreground and logs everything to
-                    stderr.
-
-    -m record
-                    Turns on these memory usage debugging flags.
-
-    -U 4            Uses four listeners.
-
-      Acquires a lock on this file in the "nsN" directory, so
-                    preventing multiple instances of this named running in this
-                    directory (which could possibly interfere with the test).
-
-All output is sent to a file called "named.run" in the nameserver directory.
-
-The options used to start named can be altered.  There are three ways of doing
-this.  "start.pl" checks the methods in a specific order: if a check succeeds,
-the options are set and any other specification is ignored.  In order, these
-are:
-
-1. Specifying options to "start.sh"/"start.pl" after the name of the test
-directory, e.g.
-
-    sh start.sh reclimit ns1 -- "-c n.conf -d 43"
-
-(This is only really useful when running tests interactively.)
-
-2. Including a file called "named.args" in the "nsN" directory.  If present,
-the contents of the first non-commented, non-blank line of the file are used as
-the named command-line arguments.  The rest of the file is ignored.
-
-3. Tweaking the default command line arguments with "-T" options.  This flag is
-used to alter the behavior of BIND for testing and is not documented in the
-ARM.  The presence of certain files in the "nsN" directory adds flags to
-the default command line (the content of the files is irrelevant - it
-is only the presence that counts):
-
-    named.noaa       Appends "-T noaa" to the command line, which causes
-                     "named" to never set the AA bit in an answer.
-
-    named.dropedns   Adds "-T dropedns" to the command line, which causes
-                     "named" to recognise EDNS options in messages, but drop
-                     messages containing them.
-
-    named.maxudp1460 Adds "-T maxudp1460" to the command line, setting the
-                     maximum UDP size handled by named to 1460.
-
-    named.maxudp512  Adds "-T maxudp512" to the command line, setting the
-                     maximum UDP size handled by named to 512.
-
-    named.noedns     Appends "-T noedns" to the command line, which disables
-                     recognition of EDNS options in messages.
-
-    named.notcp      Adds "-T notcp", which disables TCP in "named".
-
-    named.soa        Appends "-T nosoa" to the command line, which disables
-                     the addition of SOA records to negative responses (or to
-                     the additional section if the response is triggered by RPZ
-                     rewriting).
-
-Starting Other Nameservers
----
-In contrast to "named", nameservers written in Perl or Python (whose script
-file should have the name "ans.pl" or "ans.py" respectively) are started with a
-fixed command line.  In essence, the server is given the address and nothing
-else.
-
-(This is not strictly true: Python servers are provided with the number of the
-query port to use.  Altering the port used by Perl servers currently requires
-creating a template file containing the "@PORT@" token, and having "setup.sh"
-substitute the actual port being used before the test starts.)
-
-
-Stopping Nameservers
----
-As might be expected, the test system stops nameservers with the script
-"stop.sh", which is little more than a wrapper for "stop.pl".  Like "start.pl",
-the options available are listed in the file's header and will not be repeated
-here.
-
-In summary though, the nameservers for a given test, if left running by
-specifying the "-k" flag to "legacy.run.sh" when the test is started, can be
-stopped by the command:
-
-    sh stop.sh <test-name> [server]
-
-... where if the server (e.g. "ns1", "ans3") is not specified, all servers
-associated with the test are stopped.
-
-
-Adding a Test to the System Test Suite
----
-Once a test has been created, the following files should be edited:
-
-* conf.sh.common The name of the test should be added to the PARALLEL_COMMON
-variable.
-
-* Makefile.am The name of the test should be added to the TESTS variable.
-
-(It is likely that a future iteration of the system test suite will remove the
-need to edit multiple files to add a test.)
-
-
-Valgrind
----
-When running system tests, named can be run under Valgrind.  The output from
-Valgrind are sent to per-process files that can be reviewed after the test has
-completed.  To enable this, set the USE_VALGRIND environment variable to
-"helgrind" to run the Helgrind tool, or any other value to run the Memcheck
-tool.  To use "helgrind" effectively, build BIND with --disable-atomic.
-
-Developer Notes for pytest runner
-===
-
-Test discovery and collection
----
-There are two distinct types of system tests. The first is a shell script
-tests.sh containing individual test cases executed sequentially and the
-success/failure is determined by return code. The second type is a regular
-pytest file which contains test functions.
-
-Dealing with the regular pytest files doesn't require any special consideration
-as long as the naming conventions are met. Discovering the tests.sh tests is
-more complicated.
-
-The chosen solution is to add a bit of glue for each system test. For every
-tests.sh, there is an accompanying tests_sh_*.py file that contains a test
-function which utilizes a custom run_tests_sh fixture to call the tests.sh
-script. Other solutions were tried and eventually rejected. While this
-introduces a bit of extra glue, it is the most portable, compatible and least
-complex solution.
-
-Module scope
----
-Pytest fixtures can have a scope. The "module" scope is the most important for
-our use. A module is a python file which contains test functions. Every system
-test directory may contain multiple modules (i.e. tests_*.py files)!
-
-The server setup/teardown is done for a module. Bundling test cases together
-inside a single module may save some resources. However, test cases inside a
-single module can't be executed in parallel.
-
-It is possible to execute different modules defined within a single system test
-directory in parallel. This is possible thanks to executing the tests inside a
-temporary directory and proper port assignment to ensure there won't be any
-conflicts.
-
-Test logging
----
-Each module has a separate log which will be saved as pytest.log.txt in the
-temporary directory in which the test is executed. This log includes messages
-for this module setup/teardown as well as any logging from the tests using the
-`logger` fixture. Logging level DEBUG and above will be present in this log.
-
-In general, any log messages using INFO or above will also be printed out
-during pytest execution. In CI, the pytest output is also saved to
-pytest.out.txt in the bin/tests/system directory.
-
-Parallel execution
----
-As mentioned in the previous section, test cases inside a single module can't
-be executed in parallel. To put it differently, all tests cases inside the same
-module must be performed by the same worker/thread. Otherwise, server
-setup/teardown fixtures won't be shared and runtime issues due to port
-collisions are likely to occur.
-
-Pytest-xdist is used for executing pytest test cases in parallel using the `-n
-N_WORKERS` option. By default, xdist will distribute any test case to any
-worker, which would lead to the issue described above. Therefore, conftest.py
-enforces equivalent of `--dist loadscope` option which ensures that test cases
-within the same (module) scope will be handled by the same worker. Parallelism
-is automatically disabled when xdist.scheduler.loadscope library is not
-available.
-
-$ pytest -n auto
-
-Test selection
----
-It is possible to run just a single pytest test case from any module. Use
-standard pytest facility to select the desired test case(s), i.e. pass a
-sufficiently unique identifier for `-k` parameter. You can also check which
-tests will be executed by using the `--collect-only` flag to debug your `-k`
-expression.
-
-Compatibility with older pytest version
----
-Keep in mind that the pytest runner must work with ancient versions of pytest.
-When implementing new features, it is advisable to check feature support in
-pytest and pytest-xdist in older distributions first.
-
-As a general rule, any changes to the pytest runner need to keep working on all
-platforms in CI that use the pytest runner. As of 2023-01-13, the oldest
-supported version is whatever is available in EL8.
-
-We may need to add more compat code eventually to handle breaking upstream
-changes. For example, using request.fspath attribute is already deprecatred in
-latest pytest.
-
-Maintenance Notes for legacy runner
-===
-This section is aimed at developers maintaining BIND's system test framework.
-
-Notes on Parallel Execution
----
-Although execution of an individual test is controlled by "legacy.run.sh",
-which executes the above shell scripts (and starts the relevant servers) for
-each test, the running of all tests in the test suite is controlled by the
-Makefile.
-
-All system tests are capable of being run in parallel.  For this to work, each
-test needs to use a unique set of ports.  To avoid the need to define which
-tests use which ports (and so risk port clashes as further tests are added),
-the ports are determined by "get_ports.sh", a port broker script which keeps
-track of ports given to each individual system test.
-
-
-Cleaning Up From Tests
----
-When a test is run, up to three different types of files are created:
-
-1. Files generated by the test itself, e.g. output from "dig" and "rndc", are
-stored in the test directory.
-
-2. Files produced by named which may not be cleaned up if named exits
-abnormally, e.g. core files, PID files etc., are stored in the test directory.
-
-If the test fails, all these files are retained.  But if the test succeeds,
-they are cleaned up at different times:
-
-1. Files generated by the test itself are cleaned up by the test's own
-"clean.sh", which is called from "legacy.run.sh".
-
-2. Files that may not be cleaned up if named exits abnormally can be removed
-using the "cleanall.sh" script.
diff --git a/bin/tests/system/README.md b/bin/tests/system/README.md
new file mode 100644
index 00000000000..7bbf339a4f7
--- /dev/null
+++ b/bin/tests/system/README.md
@@ -0,0 +1,470 @@
+<!--
+Copyright (C) Internet Systems Consortium, Inc. ("ISC")
+
+SPDX-License-Identifier: MPL-2.0
+
+This Source Code Form is subject to the terms of the Mozilla Public
+License, v. 2.0.  If a copy of the MPL was not distributed with this
+file, you can obtain one at https://mozilla.org/MPL/2.0/.
+
+See the COPYRIGHT file distributed with this work for additional
+information regarding copyright ownership.
+-->
+
+# BIND9 System Test Framework
+
+This directory holds test environments for running bind9 system tests involving
+multiple name servers.
+
+Each system test directory holds a set of scripts and configuration files to
+test different parts of BIND.  The directories are named for the aspect of BIND
+they test, for example:
+
+    dnssec/       DNSSEC tests
+    forward/      Forwarding tests
+    glue/         Glue handling tests
+
+etc.
+
+A system test directory must start with an alphabetic character and may not
+contain any special characters. Only hyphen may be used as a word separator.
+
+Typically each set of tests sets up 2-5 name servers and then performs one or
+more tests against them.  Within the test subdirectory, each name server has a
+separate subdirectory containing its configuration data.  These subdirectories
+are named "nsN" or "ansN" (where N is a number between 1 and 8, e.g. ns1, ans2
+etc.)
+
+The tests are completely self-contained and do not require access to the real
+DNS.  Generally, one of the test servers (usually ns1) is set up as a root
+nameserver and is listed in the hints file of the others.
+
+
+## Running the Tests
+
+### Prerequisites
+
+To run system tests, make sure you have the following dependencies installed:
+
+- python3
+- pytest
+- perl
+- dnspython
+- pytest-xdist (for parallel execution)
+
+Individual system tests might also require additional dependencies. If those
+are missing, the affected tests will be skipped and should produce a message
+specifying what additional prerequisites they expect.
+
+### Network Setup
+
+To enable all servers to run on the same machine, they bind to separate virtual
+IP addresses on the loopback interface.  ns1 runs on 10.53.0.1, ns2 on
+10.53.0.2, etc.  Before running any tests, you must set up these addresses by
+running the command
+
+    sh ifconfig.sh up
+
+as root.  The interfaces can be removed by executing the command:
+
+    sh ifconfig.sh down
+
+... also as root.
+
+The servers use unprivileged ports (above 1024) instead of the usual port 53,
+so they can be run without root privileges once the interfaces have been set
+up.
+
+**Note for MacOS Users**
+
+If you wish to make the interfaces survive across reboots, copy
+org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons
+then run
+
+    launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
+
+... as root.
+
+### Running a Single Test
+
+The recommended way is to use pytest and its test selection facilities:
+
+    pytest -k <test-name-or-pattern>
+
+Using `-k` to specify a pattern allows to run a single pytest test case within
+a system test. E.g. you can use `-k test_sslyze_dot` to execute just the
+`test_sslyze_dot()` function from `doth/tests_sslyze.py`.
+
+However, using the `-k` pattern might pick up more tests than intended. You can
+use the `--collect-only` option to check the list of tests which match you `-k`
+pattern. If you just want to execute all system tests within a single test
+directory, you can also use the utility script:
+
+    ./run.sh system_test_dir_name
+
+### Running All the System Tests
+
+Issuing plain `pytest` command without any argument will execute all tests
+sequentially. To execute them in parallel, ensure you have pytest-xdist
+installed and run:
+
+    pytest [-n <number-of-workers>]
+
+Alternately, using the make command is also supported:
+
+    make [-j numproc] test
+
+### Test Artifacts
+
+Each test module is executed inside a unique temporary directory which contains
+all the artifacts from the test run. If the tests succeed, they are deleted by
+default. To override this behaviour, pass `--noclean` to pytest.
+
+The directory name starts with the system test name, followed by `_tmp_XXXXXX`,
+i.e. `dns64_tmp_r07vei9s` for `dns64` test run. Since this name changes each
+run, a convenience symlink that has a stable name is also created. It points to
+the latest test artifacts directory and has a form of `dns64_sh_dns64`
+(depending on the particular test module).
+
+To clean up the temporary directories and symlinks, run `make clean-local` in
+the system test directory.
+
+The following test artifacts are typically available:
+
+- pytest.log.txt: main log file with test output
+- files generated by the test itself, e.g. output from "dig" and "rndc"
+- files produced by named, other tools or helper scripts
+
+
+## Writing System Tests
+
+### File Overview
+
+Tests are organized into system test directories which may hold one or more
+test modules (python files). Each module may have multiple test cases. The
+system test directories may contain the following standard files:
+
+- `tests_*.py`: These python files are picked up by pytest as modules. If they
+  contain any test functions, they're added to the test suite.
+
+- `setup.sh`: This sets up the preconditions for the tests. Although optional,
+  virtually all tests will require such a file to set up the ports they should
+  use for the test.
+
+- `tests.sh`: Any shell-based tests are located within this file. Runs the
+  actual tests.
+
+- `tests_sh_*.py`: A glue file for the pytest runner for executing shell tests.
+
+- `ns<N>`: These subdirectories contain test name servers that can be queried
+  or can interact with each other. The value of N indicates the address the
+  server listens on: for example, ns2 listens on 10.53.0.2, and ns4 on
+  10.53.0.4. All test servers use an unprivileged port, so they don't need to
+  run as root. These servers log at the highest debug level and the log is
+  captured in the file "named.run".
+
+- `ans<N>`: Like ns[X], but these are simple mock name servers implemented in
+  Perl or Python.  They are generally programmed to misbehave in ways named
+  would not so as to exercise named's ability to interoperate with badly
+  behaved name servers.
+
+### Module Scope
+
+A module is a python file which contains test functions. Every system
+test directory may contain multiple modules (i.e. tests_*.py files).
+
+The server setup/teardown is performed for each module. Bundling test cases
+together inside a single module may save some resources. However, test cases
+inside a single module can't be executed in parallel.
+
+It is possible to execute different modules defined within a single system test
+directory in parallel. This is possible thanks to executing the tests inside a
+temporary directory and proper port assignment to ensure there won't be any
+conflicts.
+
+### Port Usage
+
+In order for the tests to run in parallel, each test requires a unique set of
+ports. This is ensured by the pytest runner, which assigns a unique set of
+ports to each test module.
+
+Inside the python tests, it is possible to use the `ports` fixture to get the
+assigned port numbers. They're also set as environment variables. These include:
+
+- `PORT`: used as the basic dns port
+- `TLSPORT`: used as the port for DNS-over-TLS
+- `HTTPPORT`, `HTTPSPORT`: used as the ports for DNS-over-HTTP
+- `CONTROLPORT`: used as the RNDC control port
+- `EXTRAPORT1` through `EXTRAPORT8`: additional ports that can be used as needed
+
+### Logging
+
+Each module has a separate log which will be saved as pytest.log.txt in the
+temporary directory in which the test is executed. This log includes messages
+for this module setup/teardown as well as any logging from the tests using the
+`logger` fixture. Logging level DEBUG and above will be present in this log.
+
+In general, any log messages using INFO or above will also be printed out
+during pytest execution. In CI, the pytest output is also saved to
+pytest.out.txt in the bin/tests/system directory.
+
+### Adding a Test to the System Test Suite
+
+Once a test has been created it will be automatically picked up by the pytest
+runner if it upholds the convention expected by pytest (especially when it comes
+to naming files and test functions).
+
+However, if a new system test directory is created, it also needs to be added to
+`TESTS` in `Makefile.am`, in order to work with `make check`.
+
+
+## Test Files
+
+### setup.sh
+
+This script is responsible for setting up the configuration files used in the
+test. It is used by both the python and shell tests. It is interpreted just
+before the servers are started up for each test module.
+
+To cope with the varying port number, ports are not hard-coded into
+configuration files (or, for that matter, scripts that emulate nameservers).
+Instead, setup.sh is responsible for editing the configuration files to set the
+port numbers.
+
+To do this, configuration files should be supplied in the form of templates
+containing tokens identifying ports.  The tokens have the same name as the
+environment variables listed above, but are prefixed and suffixed by the "@"
+symbol.  For example, a fragment of a configuration file template might look
+like:
+
+    controls {
+        inet 10.53.0.1 port @CONTROLPORT@ allow { any; } keys { rndc_key; };
+    };
+
+    options {
+        query-source address 10.53.0.1;
+        notify-source 10.53.0.1;
+        transfer-source 10.53.0.1;
+        port @PORT@;
+        allow-new-zones yes;
+    };
+
+setup.sh should copy the template to the desired filename using the
+"copy_setports" shell function defined in "conf.sh", i.e.
+
+    copy_setports ns1/named.conf.in ns1/named.conf
+
+This replaces tokens like @PORT@ with the contents of the environment variables
+listed above. setup.sh should do this for all configuration files required when
+the test starts.
+
+("setup.sh" should also use this method for replacing the tokens in any Perl or
+Python name servers used in the test.)
+
+### tests_*.py
+
+These are test modules containing tests written in python. Every test is a
+function which begins with the name `test_` (according to pytest convention). It
+is possible to pass fixtures to the test function by specifying their name as
+function arguments. Fixtures are used to provide context to the tests, e.g.:
+
+- `ports` is a dictionary with assigned port numbers
+- `logger` is a test-specific logging object
+
+### tests_sh_*.py
+
+These are glue files that are required to execute shell based tests (see below).
+These modules shouldn't contain any python tests (use a separate file instead).
+
+### tests.sh
+
+This is the test file for shell based tests.
+
+
+## Nameservers
+
+As noted earlier, a system test will involve a number of nameservers.  These
+will be either instances of named, or special servers written in a language
+such as Perl or Python.
+
+For the former, the version of "named" being run is that in the "bin/named"
+directory in the tree holding the tests (i.e. if "make test" is being run
+immediately after "make", the version of "named" used is that just built).  The
+configuration files, zone files etc. for these servers are located in
+subdirectories of the test directory named "nsN", where N is a small integer.
+The latter are special nameservers, mostly used for generating deliberately bad
+responses, located in subdirectories named "ansN" (again, N is an integer).
+In addition to configuration files, these directories should hold the
+appropriate script files as well.
+
+Note that the "N" for a particular test forms a single number space, e.g. if
+there is an "ns2" directory, there cannot be an "ans2" directory as well.
+Ideally, the directory numbers should start at 1 and work upwards.
+
+When tests are executed, pytest takes care of the test setup and teardown. It
+looks for any `nsN` and `ansN` directories in the system test directory and
+starts those servers.
+
+### `named` Command-Line Options
+
+By default, `named` server is started with the following options:
+
+    -c named.conf   Specifies the configuration file to use (so by implication,
+                    each "nsN" nameserver's configuration file must be called
+                    named.conf).
+
+    -d 99           Sets the maximum debugging level.
+
+    -D <name>       The "-D" option sets a string used to identify the
+                    nameserver in a process listing.  In this case, the string
+                    is the name of the subdirectory.
+
+    -g              Runs the server in the foreground and logs everything to
+                    stderr.
+
+    -m record
+                    Turns on these memory usage debugging flags.
+
+    -U 4            Uses four listeners.
+
+All output is sent to a file called `named.run` in the nameserver directory.
+
+The options used to start named can be altered. There are a couple ways of
+doing this. `start.pl` checks the methods in a specific order: if a check
+succeeds, the options are set and any other specification is ignored.  In order,
+these are:
+
+1. Specifying options to `start.pl` or `start_server` shell utility function
+   after the name of the test directory, e.g.
+
+   start_server --noclean --restart --port ${PORT} ns1 -- "-D xfer-ns1 -T transferinsecs -T transferslowly"
+
+2. Including a file called "named.args" in the "nsN" directory.  If present,
+the contents of the first non-commented, non-blank line of the file are used as
+the named command-line arguments.  The rest of the file is ignored.
+
+3. Tweaking the default command line arguments with "-T" options.  This flag is
+used to alter the behavior of BIND for testing and is not documented in the
+ARM.  The presence of certain files in the "nsN" directory adds flags to
+the default command line (the content of the files is irrelevant - it
+is only the presence that counts):
+
+    named.noaa       Appends "-T noaa" to the command line, which causes
+                     "named" to never set the AA bit in an answer.
+
+    named.dropedns   Adds "-T dropedns" to the command line, which causes
+                     "named" to recognise EDNS options in messages, but drop
+                     messages containing them.
+
+    named.maxudp1460 Adds "-T maxudp1460" to the command line, setting the
+                     maximum UDP size handled by named to 1460.
+
+    named.maxudp512  Adds "-T maxudp512" to the command line, setting the
+                     maximum UDP size handled by named to 512.
+
+    named.noedns     Appends "-T noedns" to the command line, which disables
+                     recognition of EDNS options in messages.
+
+    named.notcp      Adds "-T notcp", which disables TCP in "named".
+
+    named.soa        Appends "-T nosoa" to the command line, which disables
+                     the addition of SOA records to negative responses (or to
+                     the additional section if the response is triggered by RPZ
+                     rewriting).
+
+### Running Nameservers Interactively
+
+In order to debug the nameservers, you can let pytest perform the nameserver
+setup and interact with the servers before the test starts, or even at specific
+points during the test, using the `--trace` option to drop you into pdb debugger
+which pauses the execution of the tests, while keeping the server state intact:
+
+    pytest -k dns64 --trace
+
+
+## Developer Notes
+
+### Test discovery and collection
+
+There are two distinct types of system tests. The first is a shell script
+tests.sh containing individual test cases executed sequentially and the
+success/failure is determined by return code. The second type is a regular
+pytest file which contains test functions.
+
+Dealing with the regular pytest files doesn't require any special consideration
+as long as the naming conventions are met. Discovering the tests.sh tests is
+more complicated.
+
+The chosen solution is to add a bit of glue for each system test. For every
+tests.sh, there is an accompanying tests_sh_*.py file that contains a test
+function which utilizes a custom run_tests_sh fixture to call the tests.sh
+script. Other solutions were tried and eventually rejected. While this
+introduces a bit of extra glue, it is the most portable, compatible and least
+complex solution.
+
+### Compatibility with older pytest version
+
+Keep in mind that the pytest runner must work with ancient versions of pytest.
+When implementing new features, it is advisable to check feature support in
+pytest and pytest-xdist in older distributions first.
+
+As a general rule, any changes to the pytest runner need to keep working on all
+platforms in CI that use the pytest runner. As of 2023-11-14, the oldest
+supported version is whatever is available in EL8.
+
+We may need to add more compat code eventually to handle breaking upstream
+changes. For example, using request.fspath attribute is already deprecated in
+latest pytest.
+
+### Format of Shell Test Output
+
+Shell-based tests have the following format of output:
+
+    <letter>:<test-name>:<message> [(<number>)]
+
+e.g.
+
+    I:catz:checking that dom1.example is not served by primary (1)
+
+The meanings of the fields are as follows:
+
+<letter>
+This indicates the type of message.  This is one of:
+
+    S   Start of the test
+    A   Start of test (retained for backwards compatibility)
+    T   Start of test (retained for backwards compatibility)
+    E   End of the test
+    I   Information.  A test will typically output many of these messages
+        during its run, indicating test progress.  Note that such a message may
+        be of the form "I:testname:failed", indicating that a sub-test has
+        failed.
+    R   Result.  Each test will result in one such message, which is of the
+        form:
+
+                R:<test-tmpdir>:<result>
+
+        where <result> is one of:
+
+            PASS        The test passed
+            FAIL        The test failed
+            SKIPPED     The test was not run, usually because some
+                        prerequisites required to run the test are missing.
+
+<test-tmpdir>
+This is the name of the temporary test directory from which the message
+emanated, which is also the name of the subdirectory holding the test files.
+
+<message>
+This is text output by the test during its execution.
+
+(<number>)
+If present, this will correlate with a file created by the test.  The tests
+execute commands and route the output of each command to a file.  The name of
+this file depends on the command and the test, but will usually be of the form:
+
+    <command>.out.<suffix><number>
+
+e.g. nsupdate.out.test28, dig.out.q3.  This aids diagnosis of problems by
+allowing the output that caused the problem message to be identified.
+