]>
Commit | Line | Data |
---|---|---|
fb68fba0 RL |
1 | Running external test suites with OpenSSL |
2 | ========================================= | |
5227337a | 3 | |
1dc1ea18 | 4 | It is possible to integrate external test suites into OpenSSL's `make test`. |
5c9e3447 | 5 | This capability is considered a developer option and does not work on all |
5227337a MC |
6 | platforms. |
7 | ||
fb68fba0 RL |
8 | The BoringSSL test suite |
9 | ======================== | |
10 | ||
5227337a | 11 | In order to run the BoringSSL tests with OpenSSL, first checkout the BoringSSL |
5c9e3447 | 12 | source code into an appropriate directory. This can be done in two ways: |
5227337a | 13 | |
5c9e3447 | 14 | 1) Separately from the OpenSSL checkout using: |
5227337a | 15 | |
1dc1ea18 | 16 | $ git clone https://boringssl.googlesource.com/boringssl boringssl |
5227337a | 17 | |
5c9e3447 JS |
18 | The BoringSSL tests are only confirmed to work at a specific commit in the |
19 | BoringSSL repository. Later commits may or may not pass the test suite: | |
5227337a | 20 | |
1dc1ea18 DDO |
21 | $ cd boringssl |
22 | $ git checkout 490469f850e | |
5c9e3447 JS |
23 | |
24 | 2) Using the already configured submodule settings in OpenSSL: | |
25 | ||
1dc1ea18 | 26 | $ git submodule update --init |
5c9e3447 JS |
27 | |
28 | Configure the OpenSSL source code to enable the external tests: | |
5227337a | 29 | |
1dc1ea18 DDO |
30 | $ cd ../openssl |
31 | $ ./config enable-ssl3 enable-ssl3-method enable-weak-ssl-ciphers \ | |
32 | enable-external-tests | |
5227337a | 33 | |
5c9e3447 | 34 | Note that using other config options than those given above may cause the tests |
5227337a MC |
35 | to fail. |
36 | ||
37 | Run the OpenSSL tests by providing the path to the BoringSSL test runner in the | |
1dc1ea18 | 38 | `BORING_RUNNER_DIR` environment variable: |
5227337a | 39 | |
1dc1ea18 | 40 | $ BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make test |
5227337a MC |
41 | |
42 | Note that the test suite may change directory while running so the path provided | |
43 | should be absolute and not relative to the current working directory. | |
44 | ||
45 | To see more detailed output you can run just the BoringSSL tests with the | |
46 | verbose option: | |
47 | ||
1dc1ea18 DDO |
48 | $ VERBOSE=1 BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make \ |
49 | TESTS="test_external_boringssl" test | |
5227337a MC |
50 | |
51 | Test failures and suppressions | |
fb68fba0 | 52 | ------------------------------ |
5227337a MC |
53 | |
54 | A large number of the BoringSSL tests are known to fail. A test could fail | |
55 | because of many possible reasons. For example: | |
56 | ||
57 | - A bug in OpenSSL | |
58 | - Different interpretations of standards | |
59 | - Assumptions about the way BoringSSL works that do not apply to OpenSSL | |
60 | - The test uses APIs added to BoringSSL that are not present in OpenSSL | |
61 | - etc | |
62 | ||
63 | In order to provide a "clean" baseline run with all the tests passing a config | |
64 | file has been provided that suppresses the running of tests that are known to | |
65 | fail. These suppressions are held in the file "test/ossl_shim/ossl_config.json" | |
66 | within the OpenSSL source code. | |
67 | ||
68 | The community is encouraged to contribute patches which reduce the number of | |
69 | suppressions that are currently present. | |
70 | ||
5c9e3447 JS |
71 | Python PYCA/Cryptography test suite |
72 | =================================== | |
73 | ||
74 | This python test suite runs cryptographic tests with a local OpenSSL build as | |
75 | the implementation. | |
76 | ||
1dc1ea18 | 77 | First checkout the `PYCA/Cryptography` module into `./pyca-cryptography` using: |
5c9e3447 | 78 | |
1dc1ea18 | 79 | $ git submodule update --init |
5c9e3447 JS |
80 | |
81 | Then configure/build OpenSSL compatible with the python module: | |
82 | ||
1dc1ea18 DDO |
83 | $ ./config shared enable-external-tests |
84 | $ make | |
5c9e3447 JS |
85 | |
86 | The tests will run in a python virtual environment which requires virtualenv | |
87 | to be installed. | |
88 | ||
1dc1ea18 | 89 | $ make test VERBOSE=1 TESTS=test_external_pyca |
5c9e3447 JS |
90 | |
91 | Test failures and suppressions | |
483bc2dc | 92 | ------------------------------ |
5c9e3447 JS |
93 | |
94 | Some tests target older (<=1.0.2) versions so will not run. Other tests target | |
95 | other crypto implementations so are not relevant. Currently no tests fail. | |
96 | ||
77edd02f RH |
97 | krb5 test suite |
98 | =============== | |
99 | ||
100 | Much like the PYCA/Cryptography test suite, this builds and runs the krb5 | |
101 | tests against the local OpenSSL build. | |
102 | ||
103 | You will need a git checkout of krb5 at the top level: | |
104 | ||
1dc1ea18 | 105 | $ git clone https://github.com/krb5/krb5 |
77edd02f RH |
106 | |
107 | krb5's master has to pass this same CI, but a known-good version is | |
108 | krb5-1.15.1-final if you want to be sure. | |
109 | ||
1dc1ea18 DDO |
110 | $ cd krb5 |
111 | $ git checkout krb5-1.15.1-final | |
112 | $ cd .. | |
77edd02f RH |
113 | |
114 | OpenSSL must be built with external tests enabled: | |
115 | ||
1dc1ea18 DDO |
116 | $ ./config enable-external-tests |
117 | $ make | |
77edd02f RH |
118 | |
119 | krb5's tests will then be run as part of the rest of the suite, or can be | |
120 | explicitly run (with more debugging): | |
121 | ||
1dc1ea18 | 122 | $ VERBOSE=1 make TESTS=test_external_krb5 test |
77edd02f | 123 | |
44e69951 | 124 | Test-failures suppressions |
46f4e1be | 125 | -------------------------- |
77edd02f RH |
126 | |
127 | krb5 will automatically adapt its test suite to account for the configuration | |
128 | of your system. Certain tests may require more installed packages to run. No | |
129 | tests are expected to fail. | |
5ff5f745 | 130 | |
aa2cb51d DB |
131 | GOST engine test suite |
132 | =============== | |
133 | ||
134 | Much like the PYCA/Cryptography test suite, this builds and runs the GOST engine | |
135 | tests against the local OpenSSL build. | |
136 | ||
137 | You will need a git checkout of gost-engine at the top level: | |
138 | ||
1dc1ea18 | 139 | $ git submodule update --init |
aa2cb51d DB |
140 | |
141 | Then configure/build OpenSSL enabling external tests: | |
142 | ||
1dc1ea18 DDO |
143 | $ ./config shared enable-external-tests |
144 | $ make | |
aa2cb51d DB |
145 | |
146 | GOST engine requires CMake for the build process. | |
147 | ||
148 | GOST engine tests will then be run as part of the rest of the suite, or can be | |
149 | explicitly run (with more debugging): | |
150 | ||
1dc1ea18 | 151 | $ make test VERBOSE=1 TESTS=test_external_gost_engine |
aa2cb51d | 152 | |
5ff5f745 JS |
153 | Updating test suites |
154 | ==================== | |
155 | ||
156 | To update the commit for any of the above test suites: | |
157 | ||
158 | - Make sure the submodules are cloned locally: | |
159 | ||
1dc1ea18 | 160 | $ git submodule update --init --recursive |
5ff5f745 JS |
161 | |
162 | - Enter subdirectory and pull from the repository (use a specific branch/tag if required): | |
163 | ||
1dc1ea18 DDO |
164 | $ cd `<submodule-dir>` |
165 | $ git pull origin master | |
5ff5f745 JS |
166 | |
167 | - Go to root directory, there should be a new git status: | |
168 | ||
1dc1ea18 DDO |
169 | $ cd ../ |
170 | $ git status | |
171 | ... | |
172 | # modified: `<submodule-dir>` (new commits) | |
173 | ... | |
5ff5f745 JS |
174 | |
175 | - Add/commit/push the update | |
176 | ||
1dc1ea18 DDO |
177 | $ git add `<submodule-dir>` |
178 | $ git commit -m `"Updated <submodule> to latest commit"` | |
179 | $ git push |