1 Guidelines for test developers
2 ==============================
7 For any test that you want to perform, you write a script located in
8 `test/recipes/`, named `{nn}-test_{name}.t`,
9 where `{nn}` is a two digit number and
10 `{name}` is a unique name of your choice.
12 Please note that if a test involves a new testing executable, you will need to
13 do some additions in test/build.info. Please refer to the section
14 ["Changes to test/build.info"](README.md#changes-to-testbuildinfo) below.
19 A test executable is named `test/{name}test.c`
21 A test recipe is named `test/recipes/{nn}-test_{name}.t`, where `{nn}` is a two
22 digit number and `{name}` is a unique name of your choice.
24 The number `{nn}` is (somewhat loosely) grouped as follows:
26 00-04 sanity, internal and essential API tests
27 05-09 individual symmetric cipher algorithms
29 15-19 individual asymmetric cipher algorithms
30 20-24 openssl commands (some otherwise not tested)
31 25-29 certificate forms, generation and verification
38 80-89 "larger" protocols (CA, CMS, OCSP, SSL, TSA)
40 99 most time consuming tests [such as test_fuzz]
42 A recipe that just runs a test executable
43 -----------------------------------------
45 A script that just runs a program looks like this:
49 use OpenSSL::Test::Simple;
51 simple_test("test_{name}", "{name}test", "{name}");
53 `{name}` is the unique name you have chosen for your test.
55 The second argument to `simple_test` is the test executable, and `simple_test`
56 expects it to be located in `test/`
58 For documentation on `OpenSSL::Test::Simple`,
59 do `perldoc util/perl/OpenSSL/Test/Simple.pm`.
61 A recipe that runs a more complex test
62 --------------------------------------
64 For more complex tests, you will need to read up on Test::More and
65 OpenSSL::Test. Test::More is normally preinstalled, do `man Test::More` for
66 documentation. For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm`.
68 A script to start from could be this:
78 plan tests => 2; # The number of tests being performed
93 Changes to test/build.info
94 --------------------------
96 Whenever a new test involves a new test executable you need to do the
97 following (at all times, replace {NAME} and {name} with the name of your
100 * add `{name}` to the list of programs under `PROGRAMS_NO_INST`
102 * create a three line description of how to build the test, you will have
103 to modify the include paths and source files if you don't want to use the
104 basic test framework:
106 SOURCE[{name}]={name}.c
107 INCLUDE[{name}]=.. ../include ../apps/include
108 DEPEND[{name}]=../libcrypto libtestutil.a
110 Generic form of C test executables
111 ----------------------------------
113 #include "testutil.h"
115 static int my_test(void)
117 int testresult = 0; /* Assume the test will fail */
120 observed = function(); /* Call the code under test */
121 if (!TEST_int_eq(observed, 2)) /* Check the result is correct */
122 goto end; /* Exit on failure - optional */
124 testresult = 1; /* Mark the test case a success */
126 cleanup(); /* Any cleanup you require */
130 int setup_tests(void)
132 ADD_TEST(my_test); /* Add each test separately */
133 return 1; /* Indicates success. Return 0 */
134 /* to produce an error with a */
135 /* usage message and -1 for */
136 /* failure to set up with no */
140 You should use the `TEST_xxx` macros provided by `testutil.h` to test all failure
141 conditions. These macros produce an error message in a standard format if the
142 condition is not met (and nothing if the condition is met). Additional
143 information can be presented with the `TEST_info` macro that takes a `printf`
144 format string and arguments. `TEST_error` is useful for complicated conditions,
145 it also takes a `printf` format string and argument. In all cases the `TEST_xxx`
146 macros are guaranteed to evaluate their arguments exactly once. This means
147 that expressions with side effects are allowed as parameters. Thus,
149 if (!TEST_ptr(ptr = OPENSSL_malloc(..)))
151 works fine and can be used in place of:
153 ptr = OPENSSL_malloc(..);
156 The former produces a more meaningful message on failure than the latter.
158 Note that the test infrastructure automatically sets up all required environment
159 variables (such as `OPENSSL_MODULES`, `OPENSSL_CONF`, etc.) for the tests.
160 Individual tests may choose to override the default settings as required.