]>
Commit | Line | Data |
---|---|---|
036cbb6b DDO |
1 | Guidelines for test developers |
2 | ============================== | |
3 | ||
5ab4f893 | 4 | How to add recipes |
036cbb6b | 5 | ------------------ |
5ab4f893 RL |
6 | |
7 | For any test that you want to perform, you write a script located in | |
036cbb6b DDO |
8 | `test/recipes/`, named `{nn}-test_{name}.t`, |
9 | where `{nn}` is a two digit number and | |
10 | `{name}` is a unique name of your choice. | |
5ab4f893 RL |
11 | |
12 | Please note that if a test involves a new testing executable, you will need to | |
036cbb6b DDO |
13 | do some additions in test/build.info. Please refer to the section |
14 | ["Changes to test/build.info"](README.md#changes-to-testbuildinfo) below. | |
5ab4f893 | 15 | |
69b86d4b | 16 | Naming conventions |
036cbb6b DDO |
17 | ------------------ |
18 | ||
19 | A test executable is named `test/{name}test.c` | |
20 | ||
21 | A test recipe is named `test/recipes/{nn}-test_{name}.t`, where `{nn}` is a two | |
22 | digit number and `{name}` is a unique name of your choice. | |
23 | ||
24 | The number `{nn}` is (somewhat loosely) grouped as follows: | |
25 | ||
26 | 00-04 sanity, internal and essential API tests | |
27 | 05-09 individual symmetric cipher algorithms | |
28 | 10-14 math (bignum) | |
29 | 15-19 individual asymmetric cipher algorithms | |
30 | 20-24 openssl commands (some otherwise not tested) | |
31 | 25-29 certificate forms, generation and verification | |
32 | 30-35 engine and evp | |
33 | 60-79 APIs: | |
34 | 60 X509 subsystem | |
35 | 61 BIO subsystem | |
36 | 65 CMP subsystem | |
37 | 70 PACKET layer | |
38 | 80-89 "larger" protocols (CA, CMS, OCSP, SSL, TSA) | |
39 | 90-98 misc | |
40 | 99 most time consuming tests [such as test_fuzz] | |
5ab4f893 RL |
41 | |
42 | A recipe that just runs a test executable | |
036cbb6b | 43 | ----------------------------------------- |
5ab4f893 RL |
44 | |
45 | A script that just runs a program looks like this: | |
46 | ||
473664aa | 47 | #! /usr/bin/env perl |
c2500f65 | 48 | |
5ab4f893 | 49 | use OpenSSL::Test::Simple; |
c2500f65 | 50 | |
5ab4f893 RL |
51 | simple_test("test_{name}", "{name}test", "{name}"); |
52 | ||
036cbb6b | 53 | `{name}` is the unique name you have chosen for your test. |
5ab4f893 | 54 | |
036cbb6b DDO |
55 | The second argument to `simple_test` is the test executable, and `simple_test` |
56 | expects it to be located in `test/` | |
5ab4f893 | 57 | |
036cbb6b DDO |
58 | For documentation on `OpenSSL::Test::Simple`, |
59 | do `perldoc util/perl/OpenSSL/Test/Simple.pm`. | |
5ab4f893 RL |
60 | |
61 | A recipe that runs a more complex test | |
036cbb6b | 62 | -------------------------------------- |
5ab4f893 RL |
63 | |
64 | For more complex tests, you will need to read up on Test::More and | |
036cbb6b DDO |
65 | OpenSSL::Test. Test::More is normally preinstalled, do `man Test::More` for |
66 | documentation. For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm`. | |
5ab4f893 RL |
67 | |
68 | A script to start from could be this: | |
69 | ||
473664aa | 70 | #! /usr/bin/env perl |
c2500f65 | 71 | |
5ab4f893 RL |
72 | use strict; |
73 | use warnings; | |
74 | use OpenSSL::Test; | |
c2500f65 | 75 | |
5ab4f893 | 76 | setup("test_{name}"); |
c2500f65 | 77 | |
5ab4f893 | 78 | plan tests => 2; # The number of tests being performed |
c2500f65 | 79 | |
5ab4f893 RL |
80 | ok(test1, "test1"); |
81 | ok(test2, "test1"); | |
c2500f65 | 82 | |
5ab4f893 RL |
83 | sub test1 |
84 | { | |
85 | # test feature 1 | |
86 | } | |
c2500f65 | 87 | |
5ab4f893 RL |
88 | sub test2 |
89 | { | |
90 | # test feature 2 | |
91 | } | |
c2500f65 | 92 | |
2fae041d | 93 | Changes to test/build.info |
036cbb6b | 94 | -------------------------- |
5ab4f893 RL |
95 | |
96 | Whenever a new test involves a new test executable you need to do the | |
97 | following (at all times, replace {NAME} and {name} with the name of your | |
98 | test): | |
99 | ||
036cbb6b | 100 | * add `{name}` to the list of programs under `PROGRAMS_NO_INST` |
5ab4f893 | 101 | |
036cbb6b DDO |
102 | * create a three line description of how to build the test, you will have |
103 | to modify the include paths and source files if you don't want to use the | |
104 | basic test framework: | |
5ab4f893 | 105 | |
036cbb6b DDO |
106 | SOURCE[{name}]={name}.c |
107 | INCLUDE[{name}]=.. ../include ../apps/include | |
108 | DEPEND[{name}]=../libcrypto libtestutil.a | |
5ab4f893 | 109 | |
2fae041d | 110 | Generic form of C test executables |
036cbb6b | 111 | ---------------------------------- |
5ab4f893 | 112 | |
2fae041d | 113 | #include "testutil.h" |
5ab4f893 | 114 | |
2fae041d P |
115 | static int my_test(void) |
116 | { | |
117 | int testresult = 0; /* Assume the test will fail */ | |
118 | int observed; | |
119 | ||
120 | observed = function(); /* Call the code under test */ | |
f3448f54 | 121 | if (!TEST_int_eq(observed, 2)) /* Check the result is correct */ |
2fae041d P |
122 | goto end; /* Exit on failure - optional */ |
123 | ||
124 | testresult = 1; /* Mark the test case a success */ | |
125 | end: | |
126 | cleanup(); /* Any cleanup you require */ | |
127 | return testresult; | |
128 | } | |
129 | ||
ad887416 | 130 | int setup_tests(void) |
2fae041d P |
131 | { |
132 | ADD_TEST(my_test); /* Add each test separately */ | |
badf3c16 P |
133 | return 1; /* Indicates success. Return 0 */ |
134 | /* to produce an error with a */ | |
135 | /* usage message and -1 for */ | |
136 | /* failure to set up with no */ | |
137 | /* usage message. */ | |
2fae041d | 138 | } |
5ab4f893 | 139 | |
036cbb6b | 140 | You should use the `TEST_xxx` macros provided by `testutil.h` to test all failure |
2fae041d P |
141 | conditions. These macros produce an error message in a standard format if the |
142 | condition is not met (and nothing if the condition is met). Additional | |
036cbb6b DDO |
143 | information can be presented with the `TEST_info` macro that takes a `printf` |
144 | format string and arguments. `TEST_error` is useful for complicated conditions, | |
145 | it also takes a `printf` format string and argument. In all cases the `TEST_xxx` | |
d063add7 P |
146 | macros are guaranteed to evaluate their arguments exactly once. This means |
147 | that expressions with side effects are allowed as parameters. Thus, | |
148 | ||
149 | if (!TEST_ptr(ptr = OPENSSL_malloc(..))) | |
150 | ||
151 | works fine and can be used in place of: | |
152 | ||
153 | ptr = OPENSSL_malloc(..); | |
154 | if (!TEST_ptr(ptr)) | |
155 | ||
156 | The former produces a more meaningful message on failure than the latter. | |
157 | ||
6ed34b3e | 158 | Note that the test infrastructure automatically sets up all required environment |
036cbb6b DDO |
159 | variables (such as `OPENSSL_MODULES`, `OPENSSL_CONF`, etc.) for the tests. |
160 | Individual tests may choose to override the default settings as required. |