test/README-dev.md

   1 Guidelines for test developers
   2 ==============================
   3
   4 How to add recipes
   5 ------------------
   6
   7 For any test that you want to perform, you write a script located in
   8 `test/recipes/`, named `{nn}-test_{name}.t`,
   9 where `{nn}` is a two digit number and
  10 `{name}` is a unique name of your choice.
  11
  12 Please note that if a test involves a new testing executable, you will need to
  13 do some additions in test/build.info. Please refer to the section
  14 ["Changes to test/build.info"](README.md#changes-to-testbuildinfo) below.
  15
  16 Naming conventions
  17 ------------------
  18
  19 A test executable is named `test/{name}test.c`
  20
  21 A test recipe is named `test/recipes/{nn}-test_{name}.t`, where `{nn}` is a two
  22 digit number and `{name}` is a unique name of your choice.
  23
  24 The number `{nn}` is (somewhat loosely) grouped as follows:
  25
  26     00-04  sanity, internal and essential API tests
  27     05-09  individual symmetric cipher algorithms
  28     10-14  math (bignum)
  29     15-19  individual asymmetric cipher algorithms
  30     20-24  openssl commands (some otherwise not tested)
  31     25-29  certificate forms, generation and verification
  32     30-35  engine and evp
  33     60-79  APIs:
  34        60  X509 subsystem
  35        61  BIO subsystem
  36        65  CMP subsystem
  37        70  PACKET layer
  38     80-89  "larger" protocols (CA, CMS, OCSP, SSL, TSA)
  39     90-98  misc
  40     99     most time consuming tests [such as test_fuzz]
  41
  42 A recipe that just runs a test executable
  43 -----------------------------------------
  44
  45 A script that just runs a program looks like this:
  46
  47     #! /usr/bin/env perl
  48
  49     use OpenSSL::Test::Simple;
  50
  51     simple_test("test_{name}", "{name}test", "{name}");
  52
  53 `{name}` is the unique name you have chosen for your test.
  54
  55 The second argument to `simple_test` is the test executable, and `simple_test`
  56 expects it to be located in `test/`
  57
  58 For documentation on `OpenSSL::Test::Simple`,
  59 do `perldoc util/perl/OpenSSL/Test/Simple.pm`.
  60
  61 A recipe that runs a more complex test
  62 --------------------------------------
  63
  64 For more complex tests, you will need to read up on Test::More and
  65 OpenSSL::Test.  Test::More is normally preinstalled, do `man Test::More` for
  66 documentation.  For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm`.
  67
  68 A script to start from could be this:
  69
  70     #! /usr/bin/env perl
  71
  72     use strict;
  73     use warnings;
  74     use OpenSSL::Test;
  75
  76     setup("test_{name}");
  77
  78     plan tests => 2;                # The number of tests being performed
  79
  80     ok(test1, "test1");
  81     ok(test2, "test1");
  82
  83     sub test1
  84     {
  85         # test feature 1
  86     }
  87
  88     sub test2
  89     {
  90         # test feature 2
  91     }
  92
  93 Changes to test/build.info
  94 --------------------------
  95
  96 Whenever a new test involves a new test executable you need to do the
  97 following (at all times, replace {NAME} and {name} with the name of your
  98 test):
  99
 100  * add `{name}` to the list of programs under `PROGRAMS_NO_INST`
 101
 102  * create a three line description of how to build the test, you will have
 103    to modify the include paths and source files if you don't want to use the
 104    basic test framework:
 105
 106        SOURCE[{name}]={name}.c
 107        INCLUDE[{name}]=.. ../include ../apps/include
 108        DEPEND[{name}]=../libcrypto libtestutil.a
 109
 110 Generic form of C test executables
 111 ----------------------------------
 112
 113     #include "testutil.h"
 114
 115     static int my_test(void)
 116     {
 117         int testresult = 0;                 /* Assume the test will fail    */
 118         int observed;
 119
 120         observed = function();              /* Call the code under test     */
 121         if (!TEST_int_eq(observed, 2))      /* Check the result is correct  */
 122             goto end;                       /* Exit on failure - optional   */
 123
 124         testresult = 1;                     /* Mark the test case a success */
 125     end:
 126         cleanup();                          /* Any cleanup you require      */
 127         return testresult;
 128     }
 129
 130     int setup_tests(void)
 131     {
 132         ADD_TEST(my_test);                  /* Add each test separately     */
 133         return 1;                           /* Indicates success.  Return 0 */
 134                                             /* to produce an error with a   */
 135                                             /* usage message and -1 for     */
 136                                             /* failure to set up with no    */
 137                                             /* usage message.               */
 138     }
 139
 140 You should use the `TEST_xxx` macros provided by `testutil.h` to test all failure
 141 conditions.  These macros produce an error message in a standard format if the
 142 condition is not met (and nothing if the condition is met).  Additional
 143 information can be presented with the `TEST_info` macro that takes a `printf`
 144 format string and arguments.  `TEST_error` is useful for complicated conditions,
 145 it also takes a `printf` format string and argument.  In all cases the `TEST_xxx`
 146 macros are guaranteed to evaluate their arguments exactly once.  This means
 147 that expressions with side effects are allowed as parameters.  Thus,
 148
 149     if (!TEST_ptr(ptr = OPENSSL_malloc(..)))
 150
 151 works fine and can be used in place of:
 152
 153     ptr = OPENSSL_malloc(..);
 154     if (!TEST_ptr(ptr))
 155
 156 The former produces a more meaningful message on failure than the latter.
 157
 158 Note that the test infrastructure automatically sets up all required environment
 159 variables (such as `OPENSSL_MODULES`, `OPENSSL_CONF`, etc.) for the tests.
 160 Individual tests may choose to override the default settings as required.