Petr Menšík [Thu, 22 Feb 2018 14:32:16 +0000 (15:32 +0100)]
unit/unittest.sh is generated by configure. It will always be
generated into builddir. If out-of-tree build is used, make unit
will always fail. Kyuafiles and testdata still have to be copied
manually into the builddir.
Michał Kępień [Wed, 21 Feb 2018 13:59:38 +0000 (14:59 +0100)]
Preserve values of environment variables which are already set
This enables the environment variables controlling run.sh behavior to be
permanently set in a working environment (e.g. to automatically force
colored output without using "-c" in each runall.sh invocation).
Relevant runall.sh command line arguments still have a higher priority.
Michał Kępień [Wed, 21 Feb 2018 13:59:35 +0000 (14:59 +0100)]
Allow forcing colored system test output
As parallel.mk and runsequential.sh both pipe system test output through
"tee" (for the purpose of creating test.output), run.sh invoked from
these two files detects it is not writing to a terminal, which prevents
colored output from being generated. Allow forcing colored output using
a new command line argument for runall.sh, "-c", which sets an
environment variable (SYSTEMTEST_FORCE_COLOR) causing conf.sh to
unconditionally enable colored output.
The same environment variable can also be used directly to force colored
output when using "make test" instead of runall.sh.
Michał Kępień [Wed, 21 Feb 2018 13:59:33 +0000 (14:59 +0100)]
Allow retaining system test output using an environment variable
Instead of exporting an environment variable containing a command line
argument (NOCLEAN="-n"), extend run.sh to handle a "boolean" environment
variable (SYSTEMTEST_NO_CLEAN) itself. The former method is buggy
because the value of NOCLEAN is set in parallel.mk when that file is
first created, but it is not subsequently updated upon each test run
(because make considers parallel.mk to be up to date).
To retain backward compatibility, the "-n" command line argument for
run.sh is still supported (and has a higher priority than the relevant
environment variable).
The SYSTEMTEST_NO_CLEAN environment variable can also be used directly
to prevent cleanup when using "make test" instead of runall.sh.
Apart from fixing a bug, this simplifies the way runall.sh controls
run.sh behavior due to the Makefile being bypassed. Direct processing
of environment variables in run.sh is more scalable in the long run,
given that the previously utilized technique, even with its
implementation fixed, would still require Makefile.in to be modified in
two places each time a new flag needed to be passed from runall.sh to
run.sh.
Michał Kępień [Tue, 20 Feb 2018 12:59:29 +0000 (13:59 +0100)]
Improve the way cache contents are searched for "ns.flushtest.example"
During the "check flushtree clears adb correctly" check, expecting
"ns.flushtest.example" to always be the first name in the ADB dump is
fragile, because in a certain corner case "a.root-servers.nil" will be
the first name instead.
As the purpose of the relevant check is to ensure "ns.flushtest.example"
is removed from ADB by "rndc flushtree flushtest.example", search the
entire list of names present in ADB instead of just the first entry when
looking for "ns.flushtest.example".
Michał Kępień [Tue, 20 Feb 2018 12:59:28 +0000 (13:59 +0100)]
Wait until a cache dump completes instead of waiting for a fixed amount of time
Dumping the cache is an asynchronous operation, so sleeping for a fixed
amount of time after running "rndc dumpdb" is imperfect as dumping cache
contents may take longer than expected on slower machines. Instead of
always sleeping for 1 second, wait until the "; Dump complete" line
appears in the dump or 10 seconds pass, whichever comes first.
Michał Kępień [Tue, 20 Feb 2018 12:59:27 +0000 (13:59 +0100)]
Do not overwrite cache dumps
Unless configured otherwise in named.conf, "rndc dumpdb" causes a cache
dump to be written to a file called "named_dump.db" in the working
directory of the given named instance. Repeatedly using this command
throughout different checks in the cacheclean system test causes cache
dumps for older checks to be overwritten, which hinders failure
diagnosis. Prevent this by moving each cache dump to a check-specific
location after running "rndc dumpdb".
Furthermore, during the "check flushtree clears adb correctly" check,
dump_cache() is called twice without renaming the resulting files.
Prevent the first cache dump from being overwritten by moving it to a
different file before calling "rndc dumpdb" for the second time.
Stephen Morris [Fri, 26 Jan 2018 19:42:57 +0000 (19:42 +0000)]
[rt46602] Changes to get tests running on Windows
1. Track changes to conf.sh.in in conf.sh.win32
2. Modifications to prevent Windows "Configure" script replacing
the sed "@PORT@" substitution tokens in conf.sh.win32.
3. runall.sh now runs Windows tests sequentially
Stephen Morris [Thu, 15 Feb 2018 15:55:57 +0000 (15:55 +0000)]
[rt46602] Change name of shell variable holding current test directory
It was TESTNAME, but this is an obvious name and was used in one of
the system tests, something that interfered with the content of
progress messages. It is now SYSTESTDIR.
Michał Kępień [Fri, 26 Jan 2018 12:46:46 +0000 (13:46 +0100)]
[rt46602] Re-add clean.sh invocation to catz/setup.sh
Commit 57aa7b60fd caused catz/setup.sh to no longer call clean.sh, which
results in the catz system test failing on subsequent runs if the first
run is interrupted or fails.
Stephen Morris [Thu, 25 Jan 2018 14:45:49 +0000 (14:45 +0000)]
[rt46602] Escape line ends in shell script variable assignment
Escape the line ends in a multi-line variable assignment. Under some
circumstances, substituting the variable caused syntax errors when
used as the list of values in a shell script "for" statement.
Stephen Morris [Fri, 15 Dec 2017 15:56:33 +0000 (15:56 +0000)]
[rt46602] Ensure test output in systests.output is not mixed up
When running all the system tests, output from a test is sent to a
test.output file in the test directory. These are combined in to
systests.output when the run finishes.
Stephen Morris [Thu, 23 Nov 2017 10:02:15 +0000 (10:02 +0000)]
[rt46602] Tidy up run management
Miscellaneous tidying up of run management. The most significant
change is that "runall.sh" now runs _all_ the tests, even the
ones that can run in parallel. runsequential.sh is the script
to run tests that have not been converted to parallel running.
Stephen Morris [Thu, 23 Nov 2017 09:58:57 +0000 (09:58 +0000)]
[rt46602] Assign block of ports for each test
Some tests use more ports than just the query and control ports.
Each test that can run in parallel with other tests is now assigned
a unique block of 10 ports.
Stephen Morris [Fri, 17 Nov 2017 17:29:21 +0000 (17:29 +0000)]
[rt46602] Miscellaneous changes to existing parallelised tests
Currently these tests are allow_query, rpzrecurse and serve-stale
1. Function to copy files and set port numbers renamed from copy_config
to copy_setports, as this is used to change the ports in Perl and Python
test scripts as well.
2. Changes to rpzrecurse/tests.sh to handle two calls to getopts (one to
parse port numbers, the other to parse rpzrecurse-specific options). Also
fixed various commands to use correct ports.
3. Updates to "clean.sh" scripts to ensure that all files created in the
test are removed.
Stephen Morris [Fri, 17 Nov 2017 13:21:05 +0000 (13:21 +0000)]
[rt46602] Ensure that tests running in parallel use unique ports
Via an intermediate make file, tests that have been modified to be able
to run in parallel are assigned unique query and control port numbers
(other than 5300 and 9953 respectively). Tests that have not yet been
modified all use ports 5300 and 9953, so must be run sequentially.