regression-tests/README.md

   1 These tests can be used to verify standards compliance of PowerDNS and to
   2 spot regressions.
   3
   4 Dependencies
   5 ------------
   6 We need very recent versions of:
   7
   8  * validns (http://www.validns.net/)
   9  * ldns-verify-zone (part of ldns)
  10  * jdnssec-verifyzone (https://github.com/dblacka/jdnssec-tools)
  11  * named-checkzone (part of BIND9)
  12  * unbound-host (part of unbound)
  13  * drill (part of ldns)
  14
  15 Nice to have:
  16  * godns q
  17
  18 If you omit the tools above, some tests fail, but you can still run the
  19 tests.
  20
  21
  22
  23 Automated tests
  24 ---------------
  25
  26 Make sure PowerDNS is built with suitable modules, and use:
  27
  28 ```sh
  29 ./start-test-stop 5300 gmysql
  30 ```
  31
  32 To start PowerDNS in gmysql mode (including DNSSEC), run all tests, and
  33 write reports, using udp port 5300 in the process. Use:
  34
  35 ```sh
  36 ./start-test-stop help
  37 ```
  38
  39 to see all available suites.
  40
  41 In addition to diff-files in all test dirs, start-test-stop generates a jUnit-
  42 compatible XML report.
  43
  44 Manual testing
  45 --------------
  46
  47 Start PowerDNS with `--no-shuffle` for best results - the 'too-big-udp' test
  48 fails otherwise, for cosmetical reasons.
  49
  50 Method of operation
  51 -------------------
  52
  53 Each directory contains a command that, when executed, process the 'cleandig'
  54 output for the answer the nameserver sent. This output is stored in the file
  55 `real_result`, and compared to `expected_result` or one of its replacement
  56 files.
  57
  58 Then, `diff -u` is run on those two files, and the output sent to the file
  59 `diff`. If everything is as it should be, this diff file is empty, and
  60 PowerDNS passed the test.
  61
  62 Caveats
  63 -------
  64
  65 Optional features of the DNS standard can cause a difference to arise. For
  66 example, "Bind 8" appears to fail on nearly all our tests, whereas it is in
  67 compliance (except for one case).
  68
  69 Study the diff output to be sure if there is a problem.
  70
  71
  72 Analysing results
  73 -----------------
  74
  75 Failed tests appear in the file `failed_tests`, passed tests in
  76 `passed_tests`. A complete log is in the file `log`. Examine the `diff`
  77 files in all subdirectories to see what happened.
  78
  79
  80 Debian Jessie notes
  81 -------------------
  82 On debian-jessie, most of these tools can be retrieved with:
  83 ```sh
  84 sudo apt-get install validns ldnsutils bind9utils libnet-dns-perl
  85 sudo apt-get -t jessie-backports install unbound-host libunbound2
  86 ```
  87
  88 libnet-dns-perl is needed for one dyndns test.
  89
  90 This does not install the jdnssec-verifyzone tools. The test that will break without that can be disabled with:
  91 ```sh
  92 touch tests/verify-dnssec-zone/allow-missing
  93 ```
  94
  95 Getting required daemons from Docker
  96 ------------------------------------
  97
  98 Please keep in mind that databases may need a few seconds to start up.
  99
 100 'MySQL':
 101 ```sh
 102 docker run -p 3306:3306 --rm -d -e MYSQL_ALLOW_EMPTY_PASSWORD=1 mariadb
 103 GMYSQLHOST=127.0.0.1 ./start-test-stop 5300 gmysql
 104 ```
 105
 106 (`mysql:5` and `mysql:8` work too. Version 8 may need `--default-authentication-plugin=mysql_native_password` at the end if your client is older.)
 107
 108 Postgres:
 109 ```sh
 110 docker run -p 5432:5432 --rm -e POSTGRES_HOST_AUTH_METHOD=trust -d postgres
 111 GPGSQLUSER=postgres PGHOST=127.0.0.1  ./start-test-stop 5300 gpgsql
 112 ```