]>
Commit | Line | Data |
---|---|---|
fb68fba0 RL |
1 | Running external test suites with OpenSSL |
2 | ========================================= | |
5227337a | 3 | |
fb68fba0 | 4 | It is possible to integrate external test suites into OpenSSL's "make test". |
5c9e3447 | 5 | This capability is considered a developer option and does not work on all |
5227337a MC |
6 | platforms. |
7 | ||
fb68fba0 RL |
8 | |
9 | ||
10 | The BoringSSL test suite | |
11 | ======================== | |
12 | ||
5227337a | 13 | In order to run the BoringSSL tests with OpenSSL, first checkout the BoringSSL |
5c9e3447 | 14 | source code into an appropriate directory. This can be done in two ways: |
5227337a | 15 | |
5c9e3447 | 16 | 1) Separately from the OpenSSL checkout using: |
5227337a | 17 | |
5c9e3447 | 18 | $ git clone https://boringssl.googlesource.com/boringssl boringssl |
5227337a | 19 | |
5c9e3447 JS |
20 | The BoringSSL tests are only confirmed to work at a specific commit in the |
21 | BoringSSL repository. Later commits may or may not pass the test suite: | |
5227337a | 22 | |
5c9e3447 JS |
23 | $ cd boringssl |
24 | $ git checkout 490469f850e | |
25 | ||
26 | 2) Using the already configured submodule settings in OpenSSL: | |
27 | ||
28 | $ git submodule update --init | |
29 | ||
30 | Configure the OpenSSL source code to enable the external tests: | |
5227337a MC |
31 | |
32 | $ cd ../openssl | |
33 | $ ./config enable-ssl3 enable-ssl3-method enable-weak-ssl-ciphers \ | |
34 | enable-external-tests | |
35 | ||
5c9e3447 | 36 | Note that using other config options than those given above may cause the tests |
5227337a MC |
37 | to fail. |
38 | ||
39 | Run the OpenSSL tests by providing the path to the BoringSSL test runner in the | |
40 | BORING_RUNNER_DIR environment variable: | |
41 | ||
42 | $ BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make test | |
43 | ||
44 | Note that the test suite may change directory while running so the path provided | |
45 | should be absolute and not relative to the current working directory. | |
46 | ||
47 | To see more detailed output you can run just the BoringSSL tests with the | |
48 | verbose option: | |
49 | ||
50 | $ VERBOSE=1 BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make \ | |
fb68fba0 | 51 | TESTS="test_external_boringssl" test |
5227337a MC |
52 | |
53 | ||
54 | Test failures and suppressions | |
fb68fba0 | 55 | ------------------------------ |
5227337a MC |
56 | |
57 | A large number of the BoringSSL tests are known to fail. A test could fail | |
58 | because of many possible reasons. For example: | |
59 | ||
60 | - A bug in OpenSSL | |
61 | - Different interpretations of standards | |
62 | - Assumptions about the way BoringSSL works that do not apply to OpenSSL | |
63 | - The test uses APIs added to BoringSSL that are not present in OpenSSL | |
64 | - etc | |
65 | ||
66 | In order to provide a "clean" baseline run with all the tests passing a config | |
67 | file has been provided that suppresses the running of tests that are known to | |
68 | fail. These suppressions are held in the file "test/ossl_shim/ossl_config.json" | |
69 | within the OpenSSL source code. | |
70 | ||
71 | The community is encouraged to contribute patches which reduce the number of | |
72 | suppressions that are currently present. | |
73 | ||
5c9e3447 JS |
74 | |
75 | Python PYCA/Cryptography test suite | |
76 | =================================== | |
77 | ||
78 | This python test suite runs cryptographic tests with a local OpenSSL build as | |
79 | the implementation. | |
80 | ||
81 | First checkout the PYCA/Cryptography module into ./pyca-cryptography using: | |
82 | ||
83 | $ git submodule update --init | |
84 | ||
85 | Then configure/build OpenSSL compatible with the python module: | |
86 | ||
87 | $ ./config shared enable-external-tests | |
88 | $ make | |
89 | ||
90 | The tests will run in a python virtual environment which requires virtualenv | |
91 | to be installed. | |
92 | ||
93 | $ make test VERBOSE=1 TESTS=test_external_pyca | |
94 | ||
95 | Test failures and suppressions | |
483bc2dc | 96 | ------------------------------ |
5c9e3447 JS |
97 | |
98 | Some tests target older (<=1.0.2) versions so will not run. Other tests target | |
99 | other crypto implementations so are not relevant. Currently no tests fail. | |
100 | ||
77edd02f RH |
101 | |
102 | krb5 test suite | |
103 | =============== | |
104 | ||
105 | Much like the PYCA/Cryptography test suite, this builds and runs the krb5 | |
106 | tests against the local OpenSSL build. | |
107 | ||
108 | You will need a git checkout of krb5 at the top level: | |
109 | ||
110 | $ git clone https://github.com/krb5/krb5 | |
111 | ||
112 | krb5's master has to pass this same CI, but a known-good version is | |
113 | krb5-1.15.1-final if you want to be sure. | |
114 | ||
115 | $ cd krb5 | |
116 | $ git checkout krb5-1.15.1-final | |
117 | $ cd .. | |
118 | ||
119 | OpenSSL must be built with external tests enabled: | |
120 | ||
121 | $ ./config enable-external-tests | |
122 | $ make | |
123 | ||
124 | krb5's tests will then be run as part of the rest of the suite, or can be | |
125 | explicitly run (with more debugging): | |
126 | ||
127 | $ VERBOSE=1 make TESTS=test_external_krb5 test | |
128 | ||
44e69951 | 129 | Test-failures suppressions |
46f4e1be | 130 | -------------------------- |
77edd02f RH |
131 | |
132 | krb5 will automatically adapt its test suite to account for the configuration | |
133 | of your system. Certain tests may require more installed packages to run. No | |
134 | tests are expected to fail. | |
5ff5f745 JS |
135 | |
136 | ||
137 | Updating test suites | |
138 | ==================== | |
139 | ||
140 | To update the commit for any of the above test suites: | |
141 | ||
142 | - Make sure the submodules are cloned locally: | |
143 | ||
144 | $ git submodule update --init --recursive | |
145 | ||
146 | - Enter subdirectory and pull from the repository (use a specific branch/tag if required): | |
147 | ||
3b09585b | 148 | $ cd <submodule-dir> |
5ff5f745 JS |
149 | $ git pull origin master |
150 | ||
151 | - Go to root directory, there should be a new git status: | |
152 | ||
153 | $ cd ../ | |
154 | $ git status | |
155 | ... | |
156 | # modified: <submodule-dir> (new commits) | |
157 | ... | |
158 | ||
159 | - Add/commit/push the update | |
160 | ||
161 | git add <submodule-dir> | |
162 | git commit -m "Updated <submodule> to latest commit" | |
163 | git push | |
164 |