]>
Commit | Line | Data |
---|---|---|
d201506c SW |
1 | # U-Boot pytest suite |
2 | ||
3 | ## Introduction | |
4 | ||
5 | This tool aims to test U-Boot by executing U-Boot shell commands using the | |
6 | console interface. A single top-level script exists to execute or attach to the | |
7 | U-Boot console, run the entire script of tests against it, and summarize the | |
8 | results. Advantages of this approach are: | |
9 | ||
10 | - Testing is performed in the same way a user or script would interact with | |
11 | U-Boot; there can be no disconnect. | |
12 | - There is no need to write or embed test-related code into U-Boot itself. | |
13 | It is asserted that writing test-related code in Python is simpler and more | |
3aca4a44 | 14 | flexible than writing it all in C. |
d201506c SW |
15 | - It is reasonably simple to interact with U-Boot in this way. |
16 | ||
17 | ## Requirements | |
18 | ||
19 | The test suite is implemented using pytest. Interaction with the U-Boot console | |
20 | involves executing some binary and interacting with its stdin/stdout. You will | |
21 | need to implement various "hook" scripts that are called by the test suite at | |
22 | the appropriate time. | |
23 | ||
24 | On Debian or Debian-like distributions, the following packages are required. | |
af3fae0b SW |
25 | Some packages are required to execute any test, and others only for specific |
26 | tests. Similar package names should exist in other distributions. | |
d201506c SW |
27 | |
28 | | Package | Version tested (Ubuntu 14.04) | | |
29 | | -------------- | ----------------------------- | | |
30 | | python | 2.7.5-5ubuntu3 | | |
31 | | python-pytest | 2.5.1-1 | | |
af3fae0b SW |
32 | | gdisk | 0.8.8-1ubuntu0.1 | |
33 | | dfu-util | 0.5-1 | | |
34 | | dtc | 1.4.0+dfsg-1 | | |
35 | | openssl | 1.0.1f-1ubuntu2.22 | | |
d201506c SW |
36 | |
37 | The test script supports either: | |
38 | ||
39 | - Executing a sandbox port of U-Boot on the local machine as a sub-process, | |
40 | and interacting with it over stdin/stdout. | |
41 | - Executing an external "hook" scripts to flash a U-Boot binary onto a | |
42 | physical board, attach to the board's console stream, and reset the board. | |
43 | Further details are described later. | |
44 | ||
45 | ### Using `virtualenv` to provide requirements | |
46 | ||
47 | Older distributions (e.g. Ubuntu 10.04) may not provide all the required | |
48 | packages, or may provide versions that are too old to run the test suite. One | |
49 | can use the Python `virtualenv` script to locally install more up-to-date | |
50 | versions of the required packages without interfering with the OS installation. | |
51 | For example: | |
52 | ||
53 | ```bash | |
54 | $ cd /path/to/u-boot | |
55 | $ sudo apt-get install python python-virtualenv | |
56 | $ virtualenv venv | |
57 | $ . ./venv/bin/activate | |
58 | $ pip install pytest | |
59 | ``` | |
60 | ||
61 | ## Testing sandbox | |
62 | ||
63 | To run the testsuite on the sandbox port (U-Boot built as a native user-space | |
64 | application), simply execute: | |
65 | ||
66 | ``` | |
67 | ./test/py/test.py --bd sandbox --build | |
68 | ``` | |
69 | ||
70 | The `--bd` option tells the test suite which board type is being tested. This | |
71 | lets the test suite know which features the board has, and hence exactly what | |
72 | can be tested. | |
73 | ||
74 | The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may | |
75 | omit this option and build U-Boot yourself, in whatever way you choose, before | |
76 | running the test script. | |
77 | ||
78 | The test script will attach to U-Boot, execute all valid tests for the board, | |
79 | then print a summary of the test process. A complete log of the test session | |
80 | will be written to `${build_dir}/test-log.html`. This is best viewed in a web | |
81 | browser, but may be read directly as plain text, perhaps with the aid of the | |
82 | `html2text` utility. | |
83 | ||
d70facf8 SW |
84 | ### Testing under a debugger |
85 | ||
86 | If you need to run sandbox under a debugger, you may pass the command-line | |
87 | option `--gdbserver COMM`. This causes two things to happens: | |
88 | ||
89 | - Instead of running U-Boot directly, it will be run under gdbserver, with | |
90 | debug communication via the channel `COMM`. You can attach a debugger to the | |
91 | sandbox process in order to debug it. See `man gdbserver` and the example | |
92 | below for details of valid values for `COMM`. | |
93 | - All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of | |
94 | time to execute commands. This is useful if U-Boot is stopped at a breakpoint | |
95 | during debugging. | |
96 | ||
97 | A usage example is: | |
98 | ||
99 | Window 1: | |
100 | ```shell | |
101 | ./test/py/test.py --bd sandbox --gdbserver localhost:1234 | |
102 | ``` | |
103 | ||
104 | Window 2: | |
105 | ```shell | |
106 | gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' | |
107 | ``` | |
108 | ||
109 | Alternatively, you could leave off the `-ex` option and type the command | |
110 | manually into gdb once it starts. | |
111 | ||
112 | You can use any debugger you wish, so long as it speaks the gdb remote | |
113 | protocol, or any graphical wrapper around gdb. | |
114 | ||
115 | Some tests deliberately cause the sandbox process to exit, e.g. to test the | |
116 | reset command, or sandbox's CTRL-C handling. When this happens, you will need | |
117 | to attach the debugger to the new sandbox instance. If these tests are not | |
118 | relevant to your debugging session, you can skip them using pytest's -k | |
119 | command-line option; see the next section. | |
120 | ||
d201506c SW |
121 | ## Command-line options |
122 | ||
123 | - `--board-type`, `--bd`, `-B` set the type of the board to be tested. For | |
124 | example, `sandbox` or `seaboard`. | |
125 | - `--board-identity`, `--id` set the identity of the board to be tested. | |
126 | This allows differentiation between multiple instances of the same type of | |
127 | physical board that are attached to the same host machine. This parameter is | |
128 | not interpreted by the test script in any way, but rather is simply passed | |
129 | to the hook scripts described below, and may be used in any site-specific | |
130 | way deemed necessary. | |
131 | - `--build` indicates that the test script should compile U-Boot itself | |
132 | before running the tests. If using this option, make sure that any | |
133 | environment variables required by the build process are already set, such as | |
134 | `$CROSS_COMPILE`. | |
135 | - `--build-dir` sets the directory containing the compiled U-Boot binaries. | |
136 | If omitted, this is `${source_dir}/build-${board_type}`. | |
137 | - `--result-dir` sets the directory to write results, such as log files, | |
138 | into. If omitted, the build directory is used. | |
139 | - `--persistent-data-dir` sets the directory used to store persistent test | |
140 | data. This is test data that may be re-used across test runs, such as file- | |
141 | system images. | |
142 | ||
d70facf8 SW |
143 | `pytest` also implements a number of its own command-line options. Commonly used |
144 | options are mentioned below. Please see `pytest` documentation for complete | |
145 | details. Execute `py.test --version` for a brief summary. Note that U-Boot's | |
146 | test.py script passes all command-line arguments directly to `pytest` for | |
147 | processing. | |
148 | ||
149 | - `-k` selects which tests to run. The default is to run all known tests. This | |
150 | option takes a single argument which is used to filter test names. Simple | |
151 | logical operators are supported. For example: | |
152 | - `'ums'` runs only tests with "ums" in their name. | |
153 | - ``ut_dm'` runs only tests with "ut_dm" in their name. Note that in this | |
154 | case, "ut_dm" is a parameter to a test rather than the test name. The full | |
155 | test name is e.g. "test_ut[ut_dm_leak]". | |
156 | - `'not reset'` runs everything except tests with "reset" in their name. | |
157 | - `'ut or hush'` runs only tests with "ut" or "hush" in their name. | |
158 | - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in | |
159 | their name. | |
160 | - `-s` prevents pytest from hiding a test's stdout. This allows you to see | |
161 | U-Boot's console log in real time on pytest's stdout. | |
d201506c SW |
162 | |
163 | ## Testing real hardware | |
164 | ||
165 | The tools and techniques used to interact with real hardware will vary | |
166 | radically between different host and target systems, and the whims of the user. | |
167 | For this reason, the test suite does not attempt to directly interact with real | |
168 | hardware in any way. Rather, it executes a standardized set of "hook" scripts | |
169 | via `$PATH`. These scripts implement certain actions on behalf of the test | |
170 | suite. This keeps the test suite simple and isolated from system variances | |
171 | unrelated to U-Boot features. | |
172 | ||
173 | ### Hook scripts | |
174 | ||
175 | #### Environment variables | |
176 | ||
177 | The following environment variables are set when running hook scripts: | |
178 | ||
179 | - `UBOOT_BOARD_TYPE` the board type being tested. | |
180 | - `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was | |
181 | specified. | |
182 | - `UBOOT_SOURCE_DIR` the U-Boot source directory. | |
183 | - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. | |
184 | - `UBOOT_BUILD_DIR` the U-Boot build directory. | |
185 | - `UBOOT_RESULT_DIR` the test result directory. | |
3aca4a44 | 186 | - `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory. |
d201506c SW |
187 | |
188 | #### `u-boot-test-console` | |
189 | ||
190 | This script provides access to the U-Boot console. The script's stdin/stdout | |
191 | should be connected to the board's console. This process should continue to run | |
192 | indefinitely, until killed. The test suite will run this script in parallel | |
193 | with all other hooks. | |
194 | ||
195 | This script may be implemented e.g. by exec()ing `cu`, `kermit`, `conmux`, etc. | |
196 | ||
197 | If you are able to run U-Boot under a hardware simulator such as qemu, then | |
198 | you would likely spawn that simulator from this script. However, note that | |
199 | `u-boot-test-reset` may be called multiple times per test script run, and must | |
200 | cause U-Boot to start execution from scratch each time. Hopefully your | |
201 | simulator includes a virtual reset button! If not, you can launch the | |
202 | simulator from `u-boot-test-reset` instead, while arranging for this console | |
203 | process to always communicate with the current simulator instance. | |
204 | ||
205 | #### `u-boot-test-flash` | |
206 | ||
207 | Prior to running the test suite against a board, some arrangement must be made | |
208 | so that the board executes the particular U-Boot binary to be tested. Often, | |
209 | this involves writing the U-Boot binary to the board's flash ROM. The test | |
210 | suite calls this hook script for that purpose. | |
211 | ||
212 | This script should perform the entire flashing process synchronously; the | |
213 | script should only exit once flashing is complete, and a board reset will | |
214 | cause the newly flashed U-Boot binary to be executed. | |
215 | ||
216 | It is conceivable that this script will do nothing. This might be useful in | |
217 | the following cases: | |
218 | ||
219 | - Some other process has already written the desired U-Boot binary into the | |
220 | board's flash prior to running the test suite. | |
221 | - The board allows U-Boot to be downloaded directly into RAM, and executed | |
222 | from there. Use of this feature will reduce wear on the board's flash, so | |
223 | may be preferable if available, and if cold boot testing of U-Boot is not | |
224 | required. If this feature is used, the `u-boot-test-reset` script should | |
3aca4a44 | 225 | perform this download, since the board could conceivably be reset multiple |
d201506c SW |
226 | times in a single test run. |
227 | ||
228 | It is up to the user to determine if those situations exist, and to code this | |
229 | hook script appropriately. | |
230 | ||
231 | This script will typically be implemented by calling out to some SoC- or | |
232 | board-specific vendor flashing utility. | |
233 | ||
234 | #### `u-boot-test-reset` | |
235 | ||
236 | Whenever the test suite needs to reset the target board, this script is | |
237 | executed. This is guaranteed to happen at least once, prior to executing the | |
238 | first test function. If any test fails, the test infra-structure will execute | |
239 | this script again to restore U-Boot to an operational state before running the | |
240 | next test function. | |
241 | ||
242 | This script will likely be implemented by communicating with some form of | |
243 | relay or electronic switch attached to the board's reset signal. | |
244 | ||
245 | The semantics of this script require that when it is executed, U-Boot will | |
246 | start running from scratch. If the U-Boot binary to be tested has been written | |
247 | to flash, pulsing the board's reset signal is likely all this script need do. | |
248 | However, in some scenarios, this script may perform other actions. For | |
249 | example, it may call out to some SoC- or board-specific vendor utility in order | |
250 | to download the U-Boot binary directly into RAM and execute it. This would | |
251 | avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus | |
252 | saving wear on the flash chip(s). | |
253 | ||
5b2beab5 SW |
254 | #### Examples |
255 | ||
256 | https://github.com/swarren/uboot-test-hooks contains some working example hook | |
257 | scripts, and may be useful as a reference when implementing hook scripts for | |
258 | your platform. These scripts are not considered part of U-Boot itself. | |
259 | ||
d201506c SW |
260 | ### Board-type-specific configuration |
261 | ||
262 | Each board has a different configuration and behaviour. Many of these | |
263 | differences can be automatically detected by parsing the `.config` file in the | |
264 | build directory. However, some differences can't yet be handled automatically. | |
265 | ||
266 | For each board, an optional Python module `u_boot_board_${board_type}` may exist | |
267 | to provide board-specific information to the test script. Any global value | |
268 | defined in these modules is available for use by any test function. The data | |
269 | contained in these scripts must be purely derived from U-Boot source code. | |
270 | Hence, these configuration files are part of the U-Boot source tree too. | |
271 | ||
272 | ### Execution environment configuration | |
273 | ||
274 | Each user's hardware setup may enable testing different subsets of the features | |
275 | implemented by a particular board's configuration of U-Boot. For example, a | |
276 | U-Boot configuration may support USB device mode and USB Mass Storage, but this | |
277 | can only be tested if a USB cable is connected between the board and the host | |
278 | machine running the test script. | |
279 | ||
280 | For each board, optional Python modules `u_boot_boardenv_${board_type}` and | |
281 | `u_boot_boardenv_${board_type}_${board_identity}` may exist to provide | |
282 | board-specific and board-identity-specific information to the test script. Any | |
283 | global value defined in these modules is available for use by any test | |
284 | function. The data contained in these is specific to a particular user's | |
285 | hardware configuration. Hence, these configuration files are not part of the | |
286 | U-Boot source tree, and should be installed outside of the source tree. Users | |
287 | should set `$PYTHONPATH` prior to running the test script to allow these | |
288 | modules to be loaded. | |
289 | ||
290 | ### Board module parameter usage | |
291 | ||
292 | The test scripts rely on the following variables being defined by the board | |
293 | module: | |
294 | ||
295 | - None at present. | |
296 | ||
297 | ### U-Boot `.config` feature usage | |
298 | ||
299 | The test scripts rely on various U-Boot `.config` features, either directly in | |
300 | order to test those features, or indirectly in order to query information from | |
301 | the running U-Boot instance in order to test other features. | |
302 | ||
303 | One example is that testing of the `md` command requires knowledge of a RAM | |
304 | address to use for the test. This data is parsed from the output of the | |
305 | `bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. | |
306 | ||
307 | For a complete list of dependencies, please search the test scripts for | |
308 | instances of: | |
309 | ||
310 | - `buildconfig.get(...` | |
311 | - `@pytest.mark.buildconfigspec(...` | |
312 | ||
313 | ### Complete invocation example | |
314 | ||
315 | Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and | |
316 | any required environment configuration Python modules into $HOME/ubtest/py, | |
317 | then you would likely invoke the test script as follows: | |
318 | ||
319 | If U-Boot has already been built: | |
320 | ||
321 | ```bash | |
322 | PATH=$HOME/ubtest/bin:$PATH \ | |
323 | PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \ | |
324 | ./test/py/test.py --bd seaboard | |
325 | ``` | |
326 | ||
327 | If you want the test script to compile U-Boot for you too, then you likely | |
328 | need to set `$CROSS_COMPILE` to allow this, and invoke the test script as | |
329 | follow: | |
330 | ||
331 | ```bash | |
332 | CROSS_COMPILE=arm-none-eabi- \ | |
333 | PATH=$HOME/ubtest/bin:$PATH \ | |
334 | PYTHONPATH=${HOME}/ubtest/py:${PYTHONPATH} \ | |
335 | ./test/py/test.py --bd seaboard --build | |
336 | ``` | |
337 | ||
338 | ## Writing tests | |
339 | ||
340 | Please refer to the pytest documentation for details of writing pytest tests. | |
341 | Details specific to the U-Boot test suite are described below. | |
342 | ||
343 | A test fixture named `u_boot_console` should be used by each test function. This | |
344 | provides the means to interact with the U-Boot console, and retrieve board and | |
345 | environment configuration information. | |
346 | ||
347 | The function `u_boot_console.run_command()` executes a shell command on the | |
348 | U-Boot console, and returns all output from that command. This allows | |
349 | validation or interpretation of the command output. This function validates | |
350 | that certain strings are not seen on the U-Boot console. These include shell | |
351 | error messages and the U-Boot sign-on message (in order to detect unexpected | |
352 | board resets). See the source of `u_boot_console_base.py` for a complete list of | |
353 | "bad" strings. Some test scenarios are expected to trigger these strings. Use | |
354 | `u_boot_console.disable_check()` to temporarily disable checking for specific | |
355 | strings. See `test_unknown_cmd.py` for an example. | |
356 | ||
357 | Board- and board-environment configuration values may be accessed as sub-fields | |
358 | of the `u_boot_console.config` object, for example | |
359 | `u_boot_console.config.ram_base`. | |
360 | ||
361 | Build configuration values (from `.config`) may be accessed via the dictionary | |
362 | `u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable | |
363 | names. |