regression-tests/README.md

   1 These tests can be used to verify standards compliance of PowerDNS and to
   2 spot regressions.
   3
   4 Dependencies
   5 ------------
   6 We need very recent versions of:
   7
   8  * validns (http://www.validns.net/)
   9  * ldns-verify-zone (part of ldns)
  10  * jdnssec-verifyzone (http://www.verisignlabs.com/dnssec-tools/)
  11  * named-checkzone (part of BIND9)
  12  * unbound-host (part of unbound)
  13  * drill (part of ldns)
  14
  15 Nice to have:
  16  * godns q
  17
  18 If you omit the tools above, some tests fail, but you can still run the
  19 tests.
  20
  21
  22
  23 Automated tests
  24 ---------------
  25
  26 For a more hands-off approach, make sure PowerDNS is built with suitable
  27 modules, and use:
  28
  29 ```sh
  30 ./start-test-stop 5300 gmysql
  31 ```
  32
  33 To start PowerDNS in gmysql mode (including DNSSEC), run all tests, and
  34 write reports, using udp port 5300 in the process. Use:
  35
  36 ```sh
  37 ./start-test-stop help
  38 ```
  39
  40 to see all available suites.
  41
  42 In addition to diff-files in all test dirs, start-test-stop generates a jUnit-
  43 compatible XML report.
  44
  45 Manual testing
  46 --------------
  47
  48 Start PowerDNS with `--no-shuffle` for best results - the 'too-big-udp' test
  49 fails otherwise, for cosmetical reasons.
  50
  51 Method of operation
  52 -------------------
  53
  54 Each directory contains a command that, when executed, process the 'cleandig'
  55 output for the answer the nameserver sent. This output is stored in the file
  56 `real_result`, and compared to `expected_result` or one of its replacement
  57 files.
  58
  59 Then, `diff -u` is run on those two files, and the output sent to the file
  60 `diff`. If everything is as it should be, this diff file is empty, and
  61 PowerDNS passed the test.
  62
  63 Caveats
  64 -------
  65
  66 Optional features of the DNS standard can cause a difference to arise. For
  67 example, "Bind 8" appears to fail on nearly all our tests, whereas it is in
  68 compliance (except for one case).
  69
  70 Study the diff output to be sure if there is a problem.
  71
  72 How to run
  73 ----------
  74
  75 See beyond this section for information on how to run automatically.
  76
  77 First make sure PowerDNS has access to the testing zones, which are all
  78 referred in the `named.conf` file. Use the `zone2sql` tool to create SQL and
  79 feed it to your database if you want to test one of the sql backends.
  80
  81 Run PowerDNS as (to test gmysql):
  82
  83 ```sh
  84 ../pdns/pdns_server --daemon=no --local-port=5300 --socket-dir=./  \
  85 --no-shuffle --launch=gmysql --gmysql-dbname=pdnstest --gmysql-user=root \
  86 --fancy-records --query-logging --loglevel=9 \
  87 --cache-ttl=0 --no-config
  88 ```
  89
  90 or (to test bind, without DNSSEC):
  91 ```sh
  92 ../pdns/pdns_server --daemon=no --local-port=5300 --socket-dir=./  \
  93 --no-shuffle --launch=bind --bind-config=./named.conf                \
  94 --fancy-records --query-logging --loglevel=9    \
  95 --cache-ttl=0 --no-config
  96 ```
  97
  98 or (to test bind with DNSSEC):
  99
 100 ```sh
 101 ./bind-dnssec-setup
 102 ../pdns/pdns_server --daemon=no --local-port=5300 --socket-dir=./  \
 103 --no-shuffle --launch=bind --bind-config=./named.conf                \
 104 --query-logging --loglevel=9                    \
 105 --cache-ttl=0 --no-config
 106 ```
 107
 108 Or only sqlite3:
 109 ```sh
 110 rm powerdns.sqlite3
 111 sqlite3 powerdns.sqlite3 < ../pdns/no-dnssec.schema.sqlite3.sql
 112 sqlite3 powerdns.sqlite3 < ../pdns/dnssec.schema.sqlite3.sql
 113 ../pdns/backends/bind/zone2sql --named-conf=./named.conf --gsqlite \
 114 --transactions --dnssec | sqlite3 powerdns.sqlite3
 115 echo 'analyze;' | sqlite3 powerdns.sqlite3
 116
 117 ../pdns/pdns_server --daemon=no --local-port=5300 --socket-dir=./  \
 118 --no-shuffle --launch=gsqlite3 \
 119 --gsqlite3-database=./powerdns.sqlite3 --gsqlite3-dnssec             \
 120 --query-logging --loglevel=9                    \
 121 --cache-ttl=0 --no-config
 122 ```
 123
 124 Set the `nameserver` and `port` variables to point to your pdns\_server
 125 instance:
 126
 127 ```sh
 128 nameserver=127.0.0.1 port=5300 ./runtests
 129 ```
 130
 131
 132 Analysing results
 133 -----------------
 134
 135 Failed tests appear in the file `failed_tests`, passed tests in
 136 `passed_tests`. A complete log is in the file `log`. Examine the `diff`
 137 files in all subdirectories to see what happened.
 138
 139
 140 Debian Jessie notes
 141 -------------------
 142 On debian-jessie, most of these tools can be retrieved with:
 143 ```sh
 144 sudo apt-get install validns ldnsutils bind9utils libnet-dns-perl
 145 sudo apt-get -t jessie-backports install unbound-host libunbound2
 146 ```
 147
 148 libnet-dns-perl is needed for one dyndns test.
 149
 150 This does not install the jdnssec-verifyzone tools. The test that will break without that can be disabled with:
 151 ```sh
 152 touch tests/verify-dnssec-zone/allow-missing
 153 ```
 154