This directory holds test environments for running bind9 system tests involving
multiple name servers.
-Each system test directory holds a set of scripts and configuration files to
-test different parts of BIND. The directories are named for the aspect of BIND
-they test, for example:
+Each system test directory holds a set of test scripts and configuration files
+to test different parts of BIND. The directories are named for the aspect of
+BIND they test, for example:
- dnssec/ DNSSEC tests
- forward/ Forwarding tests
- glue/ Glue handling tests
+```
+dnssec/ DNSSEC tests
+forward/ Forwarding tests
+glue/ Glue handling tests
+```
etc.
-A system test directory must start with an alphabetic character and may not
-contain any special characters. Only hyphen may be used as a word separator.
+A system test directory name must start with an alphabetic character and may
+only contain alphanumeric characters and underscores. Use underscore as the
+word separator; hyphens are not allowed — they are reserved for the temporary
+directories and symlinks the test runner creates.
Typically each set of tests sets up 2-5 name servers and then performs one or
more tests against them. Within the test subdirectory, each name server has a
separate subdirectory containing its configuration data. These subdirectories
-are named "nsN" or "ansN" (where N is a number between 1 and 8, e.g. ns1, ans2
-etc.)
+are named "nsN" or "ansN" (where N is a number between 1 and 11, e.g. ns1,
+ans2 etc.)
The tests are completely self-contained and do not require access to the real
DNS. Generally, one of the test servers (usually ns1) is set up as a root
## Running the Tests
+### Building BIND
+
+The system tests run the binaries from the build directory, so BIND must be
+built first, along with a few test-only binaries and plugins:
+
+```sh
+meson setup build
+meson compile -C build
+meson compile -C build system-test-dependencies
+```
+
+When BIND is configured with `-Ddeveloper=enabled`, the test dependencies are
+built by default and the last step is not needed.
+
+Every `meson compile` invocation re-registers its build directory with the
+source tree, so the tests always use the binaries from the most recently
+compiled build directory.
+
### Prerequisites
To run system tests, make sure you have the following dependencies installed:
- python3 (3.10 and newer)
-- perl
+- pytest (7.0 and newer)
+- pytest-xdist
+- perl (still needed by the test runner internals; some legacy tests
+ additionally need the Net::DNS module and are skipped when it is missing)
-List of required python packages and their versions can be found in
-requirements.txt (can be installed with `pip3 install -r requirements.txt`).
+The full list of required and optional python packages can be found in
+[requirements.txt](requirements.txt) (it can be installed with
+`pip3 install -r requirements.txt`).
### Network Setup
10.53.0.2, etc. Before running any tests, you must set up these addresses by
running the command
- sh ifconfig.sh up
+```sh
+sh ifconfig.sh up
+```
as root. The interfaces can be removed by executing the command:
- sh ifconfig.sh down
+```sh
+sh ifconfig.sh down
+```
... also as root.
org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons
then run
- launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
+```sh
+launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
+```
... as root.
+### Running All the System Tests
+
+Issue a plain `pytest` command in this directory to execute all tests
+sequentially. To execute them in parallel instead, run:
+
+```sh
+pytest -n <number-of-workers>
+```
+
+Parallel execution requires pytest-xdist; `-n auto` uses one worker per CPU.
+
### Running a Single Test
-The recommended way is to use pytest and its test selection facilities:
+To run all test modules in a single system test directory, pass the directory
+name to pytest:
- pytest -k <test-name-or-pattern>
+```sh
+pytest dns64
+```
-Using `-k` to specify a pattern allows to run a single pytest test case within
-a system test. E.g. you can use `-k test_sslyze_dot` to execute just the
-`test_sslyze_dot()` function from `doth/tests_sslyze.py`.
+The utility script `./run.sh dns64` does the same thing.
-However, using the `-k` pattern might pick up more tests than intended. You can
-use the `--collect-only` option to check the list of tests which match you `-k`
-pattern. If you just want to execute all system tests within a single test
-directory, you can also use the utility script:
+To narrow the run down further, prefer pytest node IDs over `-k` matching —
+they are exact:
- ./run.sh system_test_dir_name
+```sh
+pytest dnssec_py/tests_mixed_ds.py
+pytest doth/tests_sslyze.py::test_sslyze_dot
+```
-### Running All the System Tests
+Parametrized tests have the parameter ID in brackets, so a single case of a
+parametrized test can be selected as:
-Issuing plain `pytest` command without any argument will execute all tests
-sequentially. To execute them in parallel, ensure you have pytest-xdist
-installed and run:
+```sh
+pytest "dnssec_py/tests_nsec3_answer.py::test_nodata[ns2]"
+```
- pytest [-n <number-of-workers>]
+The `-k` option selects tests by pattern matching:
-Alternately, using the make command is also supported:
+```sh
+pytest -k <test-name-or-pattern>
+```
- make [-j numproc] test
+Beware that a `-k` pattern might pick up more tests than intended. Use the
+`--collect-only` option to check the list of tests which match your `-k`
+pattern.
### rr
When running system tests, named can be run under the rr tool. rr records a
trace to the $system_test/nsX/named-Y/ directory, which can be later used to
-replay named. To enable this, execute start.pl with the USE_RR environment
-variable set.
+replay named. To enable this, run pytest with the USE_RR environment variable
+set.
### Test Artifacts
- `tests_*.py`: These python files are picked up by pytest as modules. If they
contain any test functions, they're added to the test suite.
-- `*.j2`: These jinja2 templates can be used for configuration files or any
- other files which require certain variables filled in, e.g. ports from the
- environment variables. During test setup, the pytest runner will automatically
- fill those in and strip the filename extension .j2, e.g. `ns1/named.conf.j2`
- becomes `ns1/named.conf`. When using advanced templating to conditionally
- include/omit entire sections or when filling in custom variables used for the
- test, ensure the templates always include the defaults. If you don't need the
- file to be auto-templated during test setup, use `.j2.manual` instead and then
- no defaults are needed.
-
-- `setup.sh`: This sets up the preconditions for the tests.
-
-- `tests.sh`: Any shell-based tests are located within this file. Runs the
- actual tests.
-
-- `tests_sh_*.py`: A glue file for the pytest runner for executing shell tests.
+- `*.j2`: Jinja2 templates, rendered automatically during test setup (see
+ [Templates](#templates) below).
- `ns<N>`: These subdirectories contain test name servers that can be queried
or can interact with each other. The value of N indicates the address the
run as root. These servers log at the highest debug level and the log is
captured in the file "named.run".
-- `ans<N>`: Like ns[X], but these are simple mock name servers implemented in
- Perl or Python. They are generally programmed to misbehave in ways named
- would not so as to exercise named's ability to interoperate with badly
- behaved name servers.
+- `ans<N>`: Like ns<N>, but these are mock name servers implemented in python
+ (`ans.py`), usually with the `isctest.asyncserver` module. They are
+ generally programmed to misbehave in ways named would not, so as to exercise
+ named's ability to interoperate with badly behaved name servers. A few
+ legacy mock servers are still implemented in perl (`ans.pl`); don't write
+ new ones.
+
+The following files appear in test directories that have not yet been fully
+ported to python; do not add them to new tests:
+
+- `tests.sh`: Legacy shell-based tests, run via a `tests_sh_*.py` glue module.
+
+- `setup.sh`: Legacy shell test setup. New tests use templates and a
+ `bootstrap()` function instead.
+
+- `prereq.sh`: Legacy prerequisite check; when it exits non-zero, the test is
+ skipped. New tests use pytest marks (see `isctest/mark.py`).
### Module Scope
ports. This is ensured by the pytest runner, which assigns a unique set of
ports to each test module.
-Inside the python tests, it is possible to use the `ports` fixture to get the
-assigned port numbers. They're also set as environment variables. These include:
+Inside the python tests, it is possible to use fixtures like `named_port` to
+get the assigned port numbers. They're also set as environment variables.
+These include:
- `PORT`: used as the basic dns port
- `TLSPORT`: used as the port for DNS-over-TLS
-- `HTTPPORT`, `HTTPSPORT`: used as the ports for DNS-over-HTTP
+- `HTTPPORT`, `HTTPSPORT`: used as the ports for DNS-over-HTTP(S)
- `CONTROLPORT`: used as the RNDC control port
- `EXTRAPORT1` through `EXTRAPORT8`: additional ports that can be used as needed
+### Templates
+
+Configuration files which need values that are only known at test run time —
+ports, default crypto algorithms, conditional sections — are written as jinja2
+templates with a `.j2` extension. During test setup, the pytest runner
+renders every `*.j2` file in the test directory and strips the extension:
+`ns1/named.conf.j2` becomes `ns1/named.conf`.
+
+Inside a template, all the runner's environment variables are available with
+`@...@` delimiters, e.g.:
+
+```jinja
+options {
+ port @PORT@;
+ listen-on { 10.53.0.1; };
+};
+key rndc_key {
+ secret "1234abcd8765";
+ algorithm @DEFAULT_HMAC@;
+};
+controls {
+ inet 10.53.0.1 port @CONTROLPORT@ allow { any; } keys { rndc_key; };
+};
+```
+
+Standard jinja2 block syntax (`{% if %}`, `{% for %}`, …) can be used for
+conditional or repeated sections.
+
+Custom template variables come from an optional module-level `bootstrap()`
+function. When a test module defines one, the runner calls it before
+rendering the templates and passes the returned dict to the template engine:
+
+```python
+def bootstrap():
+ return {"valid": True}
+```
+
+Templates using custom variables must always provide defaults, so that the
+file also renders when no value is supplied (e.g. when another module in the
+same directory has no `bootstrap()`):
+
+```jinja
+{% set valid = valid | default(False) %}
+```
+
+`bootstrap()` is also the place where a module generates test data that has
+to exist before the servers start — typically zone files and DNSSEC keys.
+
+Templates can also be re-rendered while the test is running, using the
+`templates` fixture, e.g. to change a server's config before reloading it:
+
+```python
+def test_reload(ns1, templates):
+ templates.render("ns1/named.conf", {"valid": True})
+ ns1.reconfigure()
+```
+
+If you don't need the file to be auto-templated during test setup, use the
+extension `.j2.manual` instead; such templates are only rendered when the test
+calls `templates.render()` explicitly, and no defaults are needed.
+
+### Fixtures and Helpers
+
+Fixtures defined in `conftest.py` provide the test context:
+
+- `servers` is a dictionary of all started `isctest.instance.NamedInstance`
+ servers, keyed by directory name; the shortcut fixtures `ns1` through `ns11`
+ return the corresponding instance directly. A `NamedInstance` is the
+ interface for driving a server: `ns1.rndc("...")`, `ns1.reconfigure()`,
+ `ns1.nsupdate(...)`, `ns1.watch_log_from_here()`, `ns1.ip`, ...
+- `templates` renders jinja2 templates at runtime (see above).
+- `named_port`, `named_tlsport`, `control_port`, ... return the assigned port
+ numbers.
+- `system_test_dir` is the temporary directory the module runs in.
+
+The `isctest` package provides the helper library; the modules most tests
+need are `isctest.query` (send DNS queries), `isctest.check` (assert on
+responses), `isctest.zone` (zone and key setup), `isctest.kasp` (DNSSEC
+key state checks), `isctest.asyncserver` (mock servers) and `isctest.log`
+(logging and log watchers).
+
+Pytest marks control test collection and setup:
+
+- `@pytest.mark.extra_artifacts([...])` declares the files (globs) the test is
+ expected to leave behind in addition to the common ones; undeclared
+ leftovers fail the run.
+- `@pytest.mark.requires_zones_loaded("ns1", ...)` delays the test until the
+ listed servers have loaded all zones.
+- `isctest.mark` has skip-unless conditions for environment prerequisites,
+ e.g. `isctest.mark.with_dnstap`, `isctest.mark.softhsm2_environment`,
+ `isctest.mark.live_internet_test`.
+
+### Parametrization
+
+Use `pytest.mark.parametrize` to run one test function over several inputs
+instead of copy-pasting test cases or looping inside one test function — each
+case is reported (and can be re-run) individually:
+
+```python
+@pytest.mark.parametrize(
+ "qname,rdtype",
+ [
+ ("exists.example.", "A"),
+ ("exists.example.", "TXT"),
+ ("other.example.", "A"),
+ ],
+)
+def test_answers(qname, rdtype, ns1):
+ msg = isctest.query.create(qname, rdtype)
+ response = isctest.query.udp(msg, ns1.ip)
+ isctest.check.noerror(response)
+```
+
+This creates the test cases `test_answers[exists.example.-A]` etc., which can
+be passed to pytest as node IDs.
+
### Logging
Each module has a separate log which will be saved as pytest.log.txt in the
for this module setup/teardown as well as any logging from the tests. Logging
level DEBUG and above will be present in this log.
-In general, any log messages using INFO or above will also be printed out
-during pytest execution. In CI, the pytest output is also saved to
-pytest.out.txt in the bin/tests/system directory.
+Use `isctest.log` for test output (`isctest.log.info("...")` etc.); in
+general, any log messages using INFO or above will also be printed out during
+pytest execution. In CI, the pytest output is also saved to pytest.out.txt in
+the bin/tests/system directory.
### Adding a Test to the System Test Suite
Once a test has been created it will be automatically picked up by the pytest
-runner if it upholds the convention expected by pytest (especially when it comes
-to naming files and test functions).
-
-However, if a new system test directory is created, it also needs to be added to
-`TESTS` in `Makefile.am`, in order to work with `make check`.
-
-
-## Test Files
-
-### setup.sh
-
-This script is responsible for setting up the configuration files used in the
-test. It is used by both the python and shell tests. It is interpreted just
-before the servers are started up for each test module.
-
-### tests_*.py
-
-These are test modules containing tests written in python. Every test is a
-function which begins with the name `test_` (according to pytest convention). It
-is possible to pass fixtures to the test function by specifying their name as
-function arguments. Fixtures are used to provide context to the tests, e.g.:
-
-- `ports` is a dictionary with assigned port numbers
-
-### tests_sh_*.py
-
-These are glue files that are required to execute shell based tests (see below).
-These modules shouldn't contain any python tests (use a separate file instead).
-
-### tests.sh
-
-This is the test file for shell based tests.
+runner if it upholds the convention expected by pytest (especially when it
+comes to naming files and test functions). New system test directories are
+discovered automatically; no registration in the build system is needed.
## Nameservers
As noted earlier, a system test will involve a number of nameservers. These
-will be either instances of named, or special servers written in a language
-such as Perl or Python.
+will be either instances of named, or mock servers, typically written in
+Python.
-For the former, the version of "named" being run is that in the "bin/named"
-directory in the tree holding the tests (i.e. if "make test" is being run
-immediately after "make", the version of "named" used is that just built). The
+For the former, the version of "named" being run is the one from the build
+directory registered by the most recent `meson compile` invocation. The
configuration files, zone files etc. for these servers are located in
subdirectories of the test directory named "nsN", where N is a small integer.
The latter are special nameservers, mostly used for generating deliberately bad
By default, `named` server is started with the following options:
- -c named.conf Specifies the configuration file to use (so by implication,
- each "nsN" nameserver's configuration file must be called
- named.conf).
+```
+-c named.conf Specifies the configuration file to use (so by implication,
+ each "nsN" nameserver's configuration file must be called
+ named.conf).
- -d 99 Sets the maximum debugging level.
+-d 99 Sets the maximum debugging level.
- -D <name> The "-D" option sets a string used to identify the
- nameserver in a process listing. In this case, the string
- is the name of the subdirectory.
+-D <name> The "-D" option sets a string used to identify the
+ nameserver in a process listing. In this case, the string
+ is the name of the subdirectory.
- -g Runs the server in the foreground and logs everything to
- stderr.
+-g Runs the server in the foreground and logs everything to
+ stderr.
- -m record
- Turns on these memory usage debugging flags.
+-m record
+ Turns on these memory usage debugging flags.
+```
All output is sent to a file called `named.run` in the nameserver directory.
The options used to start named can be altered. There are a couple ways of
-doing this. `start.pl` checks the methods in a specific order: if a check
-succeeds, the options are set and any other specification is ignored. In order,
-these are:
-
-1. Specifying options to `start.pl` or `start_server` shell utility function
- after the name of the test directory, e.g.
-
- start_server --noclean --restart --port ${PORT} ns1 -- "-D xfer-ns1 -T transferinsecs -T transferslowly"
+doing this. The runner checks the methods in a specific order: if a check
+succeeds, the options are set and any other specification is ignored. In
+order, these are:
-2. Including a file called "named.args" in the "nsN" directory. If present,
+1. Including a file called "named.args" in the "nsN" directory. If present,
the contents of the first non-commented, non-blank line of the file are used as
the named command-line arguments. The rest of the file is ignored.
-3. Tweaking the default command line arguments with "-T" options. This flag is
+2. Tweaking the default command line arguments with "-T" options. This flag is
used to alter the behavior of BIND for testing and is not documented in the
-ARM. The presence of certain files in the "nsN" directory adds flags to
-the default command line (the content of the files is irrelevant - it
-is only the presence that counts):
+ARM. The presence of a file called `named.<flag>` in the "nsN" directory adds
+`-T <flag>` to the default command line (the content of the file is irrelevant
+- it is only the presence that counts). The recognized flags are:
+
+```
+dropedns Recognise EDNS options in messages, but drop messages
+ containing them.
+
+ednsformerr, ednsnotimp, ednsrefused
+ Answer EDNS queries with the given rcode, pretending to
+ be an old server that doesn't understand EDNS.
- named.noaa Appends "-T noaa" to the command line, which causes
- "named" to never set the AA bit in an answer.
+cookiealwaysvalid
+ Accept any DNS cookie presented by a client.
- named.dropedns Adds "-T dropedns" to the command line, which causes
- "named" to recognise EDNS options in messages, but drop
- messages containing them.
+noaa Never set the AA bit in an answer.
- named.maxudp1460 Adds "-T maxudp1460" to the command line, setting the
- maximum UDP size handled by named to 1460.
+noedns Disable recognition of EDNS options in messages.
- named.maxudp512 Adds "-T maxudp512" to the command line, setting the
- maximum UDP size handled by named to 512.
+nonearest Omit the closest-encloser NSEC3 proof from negative
+ responses (except for DS queries).
- named.noedns Appends "-T noedns" to the command line, which disables
- recognition of EDNS options in messages.
+nosoa Disable the addition of SOA records to negative
+ responses (or to the additional section if the response
+ is triggered by RPZ rewriting).
- named.notcp Adds "-T notcp", which disables TCP in "named".
+maxudp512, maxudp1460
+ Set the maximum UDP size handled by named to 512/1460.
- named.soa Appends "-T nosoa" to the command line, which disables
- the addition of SOA records to negative responses (or to
- the additional section if the response is triggered by RPZ
- rewriting).
+tat=1, tat=3 Send trust-anchor-telemetry queries every N seconds.
+
+notcp Disable TCP in "named". Unlike the other flags, this
+ one is also applied when "named.args" is used.
+```
### Running Nameservers Interactively
points during the test, using the `--trace` option to drop you into pdb debugger
which pauses the execution of the tests, while keeping the server state intact:
- pytest -k dns64 --trace
+```sh
+pytest -k dns64 --trace
+```
## Developer Notes
### Test discovery and collection
-There are two distinct types of system tests. The first is a shell script
-tests.sh containing individual test cases executed sequentially and the
+There are two distinct types of system tests. The first is a legacy shell
+script tests.sh containing individual test cases executed sequentially and the
success/failure is determined by return code. The second type is a regular
pytest file which contains test functions.
### Compatibility with older pytest version
-Keep in mind that the pytest runner must work with ancient versions of pytest.
-When implementing new features, it is advisable to check feature support in
-pytest and pytest-xdist in older distributions first.
-
-As a general rule, any changes to the pytest runner need to keep working on all
-platforms in CI that use the pytest runner. As of 2023-11-14, the oldest
-supported version is whatever is available in EL8.
-
-We may need to add more compat code eventually to handle breaking upstream
-changes. For example, using request.fspath attribute is already deprecated in
-latest pytest.
+The minimum supported versions of python and the required python packages are
+declared in [requirements.txt](requirements.txt) and in the
+`pytest_configure()` check in `conftest.py`. When implementing new runner
+features, check feature support in the pytest and pytest-xdist versions
+available in the oldest distributions covered by CI first; we may need to add
+compat code to handle breaking upstream changes in either direction.
### Format of Shell Test Output
-Shell-based tests have the following format of output:
+Legacy shell-based tests have the following format of output:
- <letter>:<test-name>:<message> [(<number>)]
+```
+<letter>:<test-name>:<message> [(<number>)]
+```
e.g.
- I:catz:checking that dom1.example is not served by primary (1)
+```
+I:catz:checking that dom1.example is not served by primary (1)
+```
The meanings of the fields are as follows:
<letter>
This indicates the type of message. This is one of:
- S Start of the test
- A Start of test (retained for backwards compatibility)
- T Start of test (retained for backwards compatibility)
- E End of the test
- I Information. A test will typically output many of these messages
- during its run, indicating test progress. Note that such a message may
- be of the form "I:testname:failed", indicating that a sub-test has
- failed.
- R Result. Each test will result in one such message, which is of the
- form:
+```
+S Start of the test
+A Start of test (retained for backwards compatibility)
+T Start of test (retained for backwards compatibility)
+E End of the test
+I Information. A test will typically output many of these messages
+ during its run, indicating test progress. Note that such a message may
+ be of the form "I:testname:failed", indicating that a sub-test has
+ failed.
+R Result. Each test will result in one such message, which is of the
+ form:
- R:<test-tmpdir>:<result>
+ R:<test-tmpdir>:<result>
- where <result> is one of:
+ where <result> is one of:
- PASS The test passed
- FAIL The test failed
- SKIPPED The test was not run, usually because some
- prerequisites required to run the test are missing.
+ PASS The test passed
+ FAIL The test failed
+ SKIPPED The test was not run, usually because some
+ prerequisites required to run the test are missing.
+```
<test-tmpdir>
This is the name of the temporary test directory from which the message
execute commands and route the output of each command to a file. The name of
this file depends on the command and the test, but will usually be of the form:
- <command>.out.<suffix><number>
+```
+<command>.out.<suffix><number>
+```
e.g. nsupdate.out.test28, dig.out.q3. This aids diagnosis of problems by
allowing the output that caused the problem message to be identified.
-
projects / thirdparty / bind9.git / commitdiff
