]>
Commit | Line | Data |
---|---|---|
036cbb6b DDO |
1 | TEST DATA2 |
2 | ||
fb68fba0 RL |
3 | Running external test suites with OpenSSL |
4 | ========================================= | |
5227337a | 5 | |
fb68fba0 | 6 | It is possible to integrate external test suites into OpenSSL's "make test". |
5c9e3447 | 7 | This capability is considered a developer option and does not work on all |
5227337a MC |
8 | platforms. |
9 | ||
fb68fba0 RL |
10 | |
11 | ||
12 | The BoringSSL test suite | |
13 | ======================== | |
14 | ||
5227337a | 15 | In order to run the BoringSSL tests with OpenSSL, first checkout the BoringSSL |
5c9e3447 | 16 | source code into an appropriate directory. This can be done in two ways: |
5227337a | 17 | |
5c9e3447 | 18 | 1) Separately from the OpenSSL checkout using: |
5227337a | 19 | |
5c9e3447 | 20 | $ git clone https://boringssl.googlesource.com/boringssl boringssl |
5227337a | 21 | |
5c9e3447 JS |
22 | The BoringSSL tests are only confirmed to work at a specific commit in the |
23 | BoringSSL repository. Later commits may or may not pass the test suite: | |
5227337a | 24 | |
5c9e3447 JS |
25 | $ cd boringssl |
26 | $ git checkout 490469f850e | |
27 | ||
28 | 2) Using the already configured submodule settings in OpenSSL: | |
29 | ||
30 | $ git submodule update --init | |
31 | ||
32 | Configure the OpenSSL source code to enable the external tests: | |
5227337a MC |
33 | |
34 | $ cd ../openssl | |
35 | $ ./config enable-ssl3 enable-ssl3-method enable-weak-ssl-ciphers \ | |
36 | enable-external-tests | |
37 | ||
5c9e3447 | 38 | Note that using other config options than those given above may cause the tests |
5227337a MC |
39 | to fail. |
40 | ||
41 | Run the OpenSSL tests by providing the path to the BoringSSL test runner in the | |
42 | BORING_RUNNER_DIR environment variable: | |
43 | ||
44 | $ BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make test | |
45 | ||
46 | Note that the test suite may change directory while running so the path provided | |
47 | should be absolute and not relative to the current working directory. | |
48 | ||
49 | To see more detailed output you can run just the BoringSSL tests with the | |
50 | verbose option: | |
51 | ||
52 | $ VERBOSE=1 BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make \ | |
fb68fba0 | 53 | TESTS="test_external_boringssl" test |
5227337a MC |
54 | |
55 | ||
56 | Test failures and suppressions | |
fb68fba0 | 57 | ------------------------------ |
5227337a MC |
58 | |
59 | A large number of the BoringSSL tests are known to fail. A test could fail | |
60 | because of many possible reasons. For example: | |
61 | ||
62 | - A bug in OpenSSL | |
63 | - Different interpretations of standards | |
64 | - Assumptions about the way BoringSSL works that do not apply to OpenSSL | |
65 | - The test uses APIs added to BoringSSL that are not present in OpenSSL | |
66 | - etc | |
67 | ||
68 | In order to provide a "clean" baseline run with all the tests passing a config | |
69 | file has been provided that suppresses the running of tests that are known to | |
70 | fail. These suppressions are held in the file "test/ossl_shim/ossl_config.json" | |
71 | within the OpenSSL source code. | |
72 | ||
73 | The community is encouraged to contribute patches which reduce the number of | |
74 | suppressions that are currently present. | |
75 | ||
5c9e3447 JS |
76 | |
77 | Python PYCA/Cryptography test suite | |
78 | =================================== | |
79 | ||
80 | This python test suite runs cryptographic tests with a local OpenSSL build as | |
81 | the implementation. | |
82 | ||
83 | First checkout the PYCA/Cryptography module into ./pyca-cryptography using: | |
84 | ||
85 | $ git submodule update --init | |
86 | ||
87 | Then configure/build OpenSSL compatible with the python module: | |
88 | ||
89 | $ ./config shared enable-external-tests | |
90 | $ make | |
91 | ||
92 | The tests will run in a python virtual environment which requires virtualenv | |
93 | to be installed. | |
94 | ||
95 | $ make test VERBOSE=1 TESTS=test_external_pyca | |
96 | ||
97 | Test failures and suppressions | |
483bc2dc | 98 | ------------------------------ |
5c9e3447 JS |
99 | |
100 | Some tests target older (<=1.0.2) versions so will not run. Other tests target | |
101 | other crypto implementations so are not relevant. Currently no tests fail. | |
102 | ||
77edd02f RH |
103 | |
104 | krb5 test suite | |
105 | =============== | |
106 | ||
107 | Much like the PYCA/Cryptography test suite, this builds and runs the krb5 | |
108 | tests against the local OpenSSL build. | |
109 | ||
110 | You will need a git checkout of krb5 at the top level: | |
111 | ||
112 | $ git clone https://github.com/krb5/krb5 | |
113 | ||
114 | krb5's master has to pass this same CI, but a known-good version is | |
115 | krb5-1.15.1-final if you want to be sure. | |
116 | ||
117 | $ cd krb5 | |
118 | $ git checkout krb5-1.15.1-final | |
119 | $ cd .. | |
120 | ||
121 | OpenSSL must be built with external tests enabled: | |
122 | ||
123 | $ ./config enable-external-tests | |
124 | $ make | |
125 | ||
126 | krb5's tests will then be run as part of the rest of the suite, or can be | |
127 | explicitly run (with more debugging): | |
128 | ||
129 | $ VERBOSE=1 make TESTS=test_external_krb5 test | |
130 | ||
44e69951 | 131 | Test-failures suppressions |
46f4e1be | 132 | -------------------------- |
77edd02f RH |
133 | |
134 | krb5 will automatically adapt its test suite to account for the configuration | |
135 | of your system. Certain tests may require more installed packages to run. No | |
136 | tests are expected to fail. | |
5ff5f745 JS |
137 | |
138 | ||
aa2cb51d DB |
139 | GOST engine test suite |
140 | =============== | |
141 | ||
142 | Much like the PYCA/Cryptography test suite, this builds and runs the GOST engine | |
143 | tests against the local OpenSSL build. | |
144 | ||
145 | You will need a git checkout of gost-engine at the top level: | |
146 | ||
147 | $ git submodule update --init | |
148 | ||
149 | Then configure/build OpenSSL enabling external tests: | |
150 | ||
151 | $ ./config shared enable-external-tests | |
152 | $ make | |
153 | ||
154 | GOST engine requires CMake for the build process. | |
155 | ||
156 | GOST engine tests will then be run as part of the rest of the suite, or can be | |
157 | explicitly run (with more debugging): | |
158 | ||
159 | $ make test VERBOSE=1 TESTS=test_external_gost_engine | |
160 | ||
5ff5f745 JS |
161 | Updating test suites |
162 | ==================== | |
163 | ||
164 | To update the commit for any of the above test suites: | |
165 | ||
166 | - Make sure the submodules are cloned locally: | |
167 | ||
168 | $ git submodule update --init --recursive | |
169 | ||
170 | - Enter subdirectory and pull from the repository (use a specific branch/tag if required): | |
171 | ||
df443918 | 172 | $ cd <submodule-dir> |
5ff5f745 JS |
173 | $ git pull origin master |
174 | ||
175 | - Go to root directory, there should be a new git status: | |
176 | ||
177 | $ cd ../ | |
178 | $ git status | |
179 | ... | |
180 | # modified: <submodule-dir> (new commits) | |
181 | ... | |
182 | ||
183 | - Add/commit/push the update | |
184 | ||
185 | git add <submodule-dir> | |
186 | git commit -m "Updated <submodule> to latest commit" | |
187 | git push | |
188 |