test/README.md

   1 Test OpenSSL
   2 ============
   3
   4 After a successful build, and before installing, the libraries should be tested.
   5 Run:
   6
   7     $ make test                                      # Unix
   8     $ mms test                                       ! OpenVMS
   9     $ nmake test                                     # Windows
  10
  11 **Warning:** you MUST run the tests from an unprivileged account
  12 (or disable your privileges temporarily if your platform allows it).
  13
  14 If some tests fail, take a look at the section Test Failures below.
  15
  16 Test Failures
  17 -------------
  18
  19 If some tests fail, look at the output.  There may be reasons for the failure
  20 that isn't a problem in OpenSSL itself (like an OS malfunction or a Perl issue).
  21 You may want increased verbosity, that can be accomplished like this:
  22
  23 Full verbosity, showing full output of all successful and failed test cases
  24 (`make` macro `VERBOSE` or `V`):
  25
  26     $ make V=1 test                                  # Unix
  27     $ mms /macro=(V=1) test                          ! OpenVMS
  28     $ nmake V=1 test                                 # Windows
  29
  30 Verbosity on test failure (`VERBOSE_FAILURE` or `VF`, Unix example shown):
  31
  32     $ make test VF=1
  33
  34 Verbosity on failed (sub-)tests only (`VERBOSE_FAILURES_ONLY` or `VFO`):
  35
  36     $ make test VFO=1
  37
  38 Verbosity on failed (sub-)tests, in addition progress on succeeded (sub-)tests
  39 (`VERBOSE_FAILURES_PROGRESS` or `VFP`):
  40
  41     $ make test VFP=1
  42
  43 If you want to run just one or a few specific tests, you can use
  44 the `make` variable `TESTS` to specify them, like this:
  45
  46     $ make TESTS='test_rsa test_dsa' test            # Unix
  47     $ mms/macro="TESTS=test_rsa test_dsa" test       ! OpenVMS
  48     $ nmake TESTS='test_rsa test_dsa' test           # Windows
  49
  50 And of course, you can combine (Unix examples shown):
  51
  52     $ make test TESTS='test_rsa test_dsa' VF=1
  53     $ make test TESTS="test_cmp_*" VFO=1
  54
  55 You can find the list of available tests like this:
  56
  57     $ make list-tests                                # Unix
  58     $ mms list-tests                                 ! OpenVMS
  59     $ nmake list-tests                               # Windows
  60
  61 Have a look at the manual for the perl module Test::Harness to
  62 see what other HARNESS_* variables there are.
  63
  64 To report a bug please open an issue on GitHub, at
  65 <https://github.com/openssl/openssl/issues>.
  66
  67 For more details on how the `make` variables `TESTS` can be used,
  68 see section Running Selected Tests below.
  69
  70 Running Selected Tests
  71 ----------------------
  72
  73 The `make` variable `TESTS` supports a versatile set of space separated tokens
  74 with which you can specify a set of tests to be performed.  With a "current
  75 set of tests" in mind, initially being empty, here are the possible tokens:
  76
  77      alltests      The current set of tests becomes the whole set of available
  78                    tests (as listed when you do 'make list-tests' or similar).
  79
  80      xxx           Adds the test 'xxx' to the current set of tests.
  81
  82     -xxx           Removes 'xxx' from the current set of tests.  If this is the
  83                    first token in the list, the current set of tests is first
  84                    assigned the whole set of available tests, effectively making
  85                    this token equivalent to TESTS="alltests -xxx".
  86
  87      nn            Adds the test group 'nn' (which is a number) to the current
  88                    set of tests.
  89
  90     -nn            Removes the test group 'nn' from the current set of tests.
  91                    If this is the first token in the list, the current set of
  92                    tests is first assigned the whole set of available tests,
  93                    effectively making this token equivalent to
  94                    TESTS="alltests -xxx".
  95
  96 Also, all tokens except for "alltests" may have wildcards, such as *.
  97 (on Unix and Windows, BSD style wildcards are supported, while on VMS,
  98 it's VMS style wildcards)
  99
 100 ### Examples
 101
 102 Run all tests except for the fuzz tests:
 103
 104     $ make TESTS=-test_fuzz test
 105
 106 or, if you want to be explicit:
 107
 108     $ make TESTS='alltests -test_fuzz' test
 109
 110 Run all tests that have a name starting with "test_ssl" but not those
 111 starting with "test_ssl_":
 112
 113     $ make TESTS='test_ssl* -test_ssl_*' test
 114
 115 Run only test group 10:
 116
 117     $ make TESTS='10'
 118
 119 Run all tests except the slow group (group 99):
 120
 121     $ make TESTS='-99'
 122
 123 Run all tests in test groups 80 to 99 except for tests in group 90:
 124
 125     $ make TESTS='[89]? -90'
 126
 127 To stochastically verify that the algorithm that produces uniformly distributed
 128 random numbers is operating correctly (with a false positive rate of 0.01%):
 129
 130     $ ./util/wrap.sh test/bntest -stochastic