]> git.ipfire.org Git - thirdparty/gcc.git/blob - libstdc++-v3/docs/html/test.html
[multiple changes]
[thirdparty/gcc.git] / libstdc++-v3 / docs / html / test.html
1 <?xml version="1.0" encoding="ISO-8859-1"?>
2 <!DOCTYPE html
3 PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
4 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
5
6 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
7 <head>
8 <meta name="AUTHOR" content="bkoz@gcc.gnu.org (Benjamin Kosnik)" />
9 <meta name="KEYWORDS" content="c++, libstdc++, test, regression, g++" />
10 <meta name="DESCRIPTION" content="README for the GNU libstdc++ effort." />
11 <meta name="GENERATOR" content="vi and eight fingers" />
12 <title>libstdc++-v3 Testing Instructions</title>
13 <link rel="StyleSheet" href="lib3styles.css" />
14 </head>
15 <body>
16
17 <h1 class="centered"><a name="top">Testing Details</a></h1>
18
19 <p class="fineprint"><em>
20 The latest version of this document is always available at
21 <a href="http://gcc.gnu.org/onlinedocs/libstdc++/test.html">
22 http://gcc.gnu.org/onlinedocs/libstdc++/test.html</a>.
23 </em></p>
24
25 <p><em>
26 To the <a href="http://gcc.gnu.org/libstdc++/">libstdc++-v3 homepage</a>.
27 </em></p>
28
29 <!-- ####################################################### -->
30 <hr />
31 <h2>Contents</h2>
32 <ul>
33 <li><a href="#org">Testsuite organization and naming conventions</a></li>
34 <li><a href="#util">Utilities: abicheck and libv3test</a></li>
35 <li><a href="#new">How to write a new test case</a></li>
36 <li><a href="#check">Options for running the tests</a></li>
37 <li><a href="#debug">Running debug-mode tests</a></li>
38 <li><a href="#future">Future</a></li>
39 <li><a href="#internals">DejaGNU internals</a></li>
40 </ul>
41
42 <hr />
43
44 <!-- ####################################################### -->
45
46 <h2><a name="org">Testsuite organization and naming conventions</a></h2>
47 <p>
48 The directory <em>libsrcdir/testsuite</em> contains the test
49 files, test harness, and utility information for verifying the
50 correctness of C++ library on a given host. It includes the
51 following directories, each named after a specific chapter of
52 the C++ standard, and each containing test files or
53 subdirectories of test files that test for that particular part
54 of the standard.
55 </p>
56
57 <pre>
58 17_intro
59 18_support
60 19_diagnostics
61 20_util
62 21_strings
63 22_locale
64 23_containers
65 25_algorithms
66 26_numerics
67 27_io
68 </pre>
69
70 <p>
71 In addition, the following directories include test files:
72 </p>
73
74 <pre>
75 backward Tests for backwards compatibility and deprecated features.
76 demangle Tests for __cxa_demangle, the IA 64 C++ ABI demangler
77 ext Tests for extensions.
78 performance Tests for performance analysis, and performance regressions.
79 thread Tests for threads.
80 </pre>
81
82 <p>
83 Some directories don't have test files, but instead contain
84 auxiliary information (<a href="#internals">more information</a>):
85 </p>
86
87 <pre>
88 config Files for the dejagnu test harness.
89 lib Files for the dejagnu test harness.
90 libstdc++* Files for the dejagnu test harness.
91 data Sample text files for testing input and output.
92 </pre>
93
94 <p>
95 Within a directory that includes test files, there may be
96 additional subdirectories, or files: this particular point is in
97 flux. Originally, test cases were appended to one file that
98 represented a particular section of the chapter under test, and
99 was named accordingly. For instance, to test items related to
100 <code> 21.3.6.1 - basic_string::find [lib.string::find]</code>
101 in the standard, the following was used:
102 </p>
103 <pre>
104 21_strings/find.cc
105 </pre>
106 <p>
107 However, that practice soon became a liability as the test cases
108 became huge and unwieldy, and testing new or extended
109 functionality (like wide characters or named locales) became
110 frustrating, leading to aggressive pruning of test cases on some
111 platforms that covered up implementation errors. Now, the test
112 suite is converging on a policy of one file, one test case,
113 which solves the above issues and gives finer grained results
114 and more manageable error debugging. As an example, the test case
115 quoted above becomes:
116 </p>
117 <pre>
118 21_strings/basic_string/find/char/1.cc
119 21_strings/basic_string/find/char/2.cc
120 21_strings/basic_string/find/char/3.cc
121 21_strings/basic_string/find/wchar_t/1.cc
122 21_strings/basic_string/find/wchar_t/2.cc
123 21_strings/basic_string/find/wchar_t/3.cc
124 </pre>
125
126 <p>
127 All new tests should be written with the policy of one test
128 case, one file in mind. At some point the entire testsuite will
129 be converted: the current status is that the 21_string,
130 22_locale, 23_containers, 27_io, and demangle directories have all been
131 transitioned.
132 </p>
133
134 <p>
135 In addition, there are some special names and suffixes that are
136 used within the testsuite to designate particular kinds of
137 tests.
138 </p>
139
140 <ul>
141 <li>
142 <em>_xin.cc</em>
143 <p>
144 This test case expects some kind of interactive input in order
145 to finish or pass. At the moment, the interactive tests are not
146 run by default. Instead, they are run by hand, like:
147 </p>
148 <pre>
149 g++ 27_io/objects/char/3_xin.cc
150 cat 27_io/objects/char/3_xin.in | a.out
151 </pre>
152 </li>
153 <li>
154 <em>.in</em>
155 <p>
156 This file contains the expected input for the corresponding <em>
157 _xin.cc</em> test case.
158 </p>
159 </li>
160 <li>
161 <em>_neg.cc</em>
162 <p>
163 This test case is expected to fail: it's a negative test. At the
164 moment, these are almost always compile time errors.
165 </p>
166 </li>
167 <li>
168 <em>char</em>
169 <p>
170 This can either be a directory name or part of a longer file
171 name, and indicates that this file, or the files within this
172 directory are testing the <code>char</code> instantiation of a
173 template.
174 </p>
175 </li>
176 <li>
177 <em>wchar_t</em>
178 <p>
179 This can either be a directory name or part of a longer file
180 name, and indicates that this file, or the files within this
181 directory are testing the <code>wchar_t</code> instantiation of
182 a template. Some hosts do not support <code>wchar_t</code>
183 functionality, so for these targets, all of these tests will not
184 be run.
185 </p>
186 </li>
187 <li>
188 <em>performance</em>
189 <p>
190 This can either be an enclosing directory name or part of a
191 specific file name. This indicates a test that is used to
192 analyze runtime performance, for performance regression testing,
193 or for other optimization related analysis. At the moment, these
194 test cases are not run by default.
195 </p>
196 </li>
197 </ul>
198
199 <hr />
200 <h2><a name="util">Utilities: abicheck and libv3test</a></h2>
201 <p>
202 The testsuite directory also contains some files that implement
203 functionality that is intended to make writing test cases easier,
204 or to avoid duplication, or to provide error checking in a way that
205 is consistent across platforms and test harnesses. A stand-alone
206 executable, called <em>abi_check</em>, and a static library called
207 <em>libv3test</em> are constructed during the build. Both of these
208 items are not installed, and only used during testing.
209 </p>
210
211 <p>
212 These files include the following functionality:
213 </p>
214
215 <ul>
216 <li>
217 <em>abi_check.cc</em>
218 <p>
219 Creates the executable <em>abi_check</em>.
220 Used to check correctness of symbol versioning, visibility of
221 exported symbols, and compatibility on symbols in the shared
222 library, for hosts that support this feature. More information
223 can be found in the ABI documentation <a href="abi.txt"> here</a>
224 </p>
225 </li>
226 <li>
227 <em>testsuite_allocator.h and </em>
228 <em>testsuite_allocator.cc</em>
229 <p>
230 Specialized allocators that keep track of construction and destruction
231 </p>
232 </li>
233 <li>
234 <em>testsuite_hooks.h and </em>
235 <em>testsuite_hooks.cc</em>
236 <p>
237 A large number of utilities, including:
238 </p>
239 <ul>
240 <li>VERIFY</li>
241 <li>set_memory_limits</li>
242 <li>verify_demangle</li>
243 <li>run_tests_wrapped_locale</li>
244 <li>run_tests_wrapped_env</li>
245 <li>try_named_locale</li>
246 <li>counter</li>
247 <li>copy_constructor</li>
248 <li>assignment_operator</li>
249 <li>destructor</li>
250 <li>copy_tracker</li>
251 <li>pod_char, pod_int and associated char_traits specializations</li>
252 </ul>
253 <p></p>
254 </li>
255 <li>
256 <em>testsuite_performance.h</em>
257 <p>
258 A number of class abstractions for performance counters, and
259 reporting functions including:
260 </p>
261 <ul>
262 <li>time_counter</li>
263 <li>resource_counter</li>
264 <li>report_performance</li>
265 </ul>
266 <p></p>
267 </li>
268 <li>
269 <em>printnow.c</em>
270 <p>
271 A cross-platform timer for use in one of the older harnesses
272 to determine compilation and link time.
273 </p>
274 </li>
275 </ul>
276
277 <hr />
278 <h2><a name="new">How to write a new test case</a></h2>
279
280 <p>
281 The first step in making a new test case is to choose the correct
282 directory and file name, given the organization as previously
283 described.
284 </p>
285
286 <p>
287 All files are copyright the FSF, and GPL'd: this is very
288 important. The first copyright year should correspond to the date
289 the file was checked in to CVS.
290 </p>
291
292 <p>
293 As per the dejagnu instructions, always return 0 from main to
294 indicate success.
295 </p>
296
297 <p>
298 A bunch of utility functions and classes have already been
299 abstracted out into the testsuite utility library, <code>
300 libv3test</code>. To use this functionality, just include the
301 appropriate header file: the library will automatically be linked
302 in as part of the testsuite run.
303 </p>
304
305 <p>
306 For a test that needs to take advantage of the dejagnu test
307 harness, what follows below is a list of special keyword that
308 harness uses. Basically, a test case contains dg-keywords (see
309 dg.exp) indicating what to do and what kinds of behavior are to be
310 expected. New test cases should be written with the new style
311 DejaGnu framework in mind.
312 </p>
313
314 <p>
315 To ease transition, here is the list of dg-keyword documentation
316 lifted from dg.exp.
317 </p>
318
319 <pre>
320 # The currently supported options are:
321 #
322 # dg-prms-id N
323 # set prms_id to N
324 #
325 # dg-options "options ..." [{ target selector }]
326 # specify special options to pass to the tool (eg: compiler)
327 #
328 # dg-do do-what-keyword [{ target/xfail selector }]
329 # `do-what-keyword' is tool specific and is passed unchanged to
330 # ${tool}-dg-test. An example is gcc where `keyword' can be any of:
331 # preprocess|compile|assemble|link|run
332 # and will do one of: produce a .i, produce a .s, produce a .o,
333 # produce an a.out, or produce an a.out and run it (the default is
334 # compile).
335 #
336 # dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]
337 # indicate an error message &lt;regexp&gt; is expected on this line
338 # (the test fails if it doesn't occur)
339 # Linenum=0 for general tool messages (eg: -V arg missing).
340 # "." means the current line.
341 #
342 # dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]
343 # indicate a warning message &lt;regexp&gt; is expected on this line
344 # (the test fails if it doesn't occur)
345 #
346 # dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]
347 # indicate a bogus error message &lt;regexp&gt; use to occur here
348 # (the test fails if it does occur)
349 #
350 # dg-build regexp comment [{ target/xfail selector }]
351 # indicate the build use to fail for some reason
352 # (errors covered here include bad assembler generated, tool crashes,
353 # and link failures)
354 # (the test fails if it does occur)
355 #
356 # dg-excess-errors comment [{ target/xfail selector }]
357 # indicate excess errors are expected (any line)
358 # (this should only be used sparingly and temporarily)
359 #
360 # dg-output regexp [{ target selector }]
361 # indicate the expected output of the program is &lt;regexp&gt;
362 # (there may be multiple occurrences of this, they are concatenated)
363 #
364 # dg-final { tcl code }
365 # add some tcl code to be run at the end
366 # (there may be multiple occurrences of this, they are concatenated)
367 # (unbalanced braces must be \-escaped)
368 #
369 # "{ target selector }" is a list of expressions that determine whether the
370 # test succeeds or fails for a particular target, or in some cases whether the
371 # option applies for a particular target. If the case of `dg-do' it specifies
372 # whether the test case is even attempted on the specified target.
373 #
374 # The target selector is always optional. The format is one of:
375 #
376 # { xfail *-*-* ... } - the test is expected to fail for the given targets
377 # { target *-*-* ... } - the option only applies to the given targets
378 #
379 # At least one target must be specified, use *-*-* for "all targets".
380 # At present it is not possible to specify both `xfail' and `target'.
381 # "native" may be used in place of "*-*-*".
382
383 Example 1: Testing compilation only
384 // { dg-do compile }
385
386 Example 2: Testing for expected warnings on line 36, which all targets fail
387 // { dg-warning "string literals" "" { xfail *-*-* } 36
388
389 Example 3: Testing for expected warnings on line 36
390 // { dg-warning "string literals" "" { target *-*-* } 36
391
392 Example 4: Testing for compilation errors on line 41
393 // { dg-do compile }
394 // { dg-error "no match for" "" { target *-*-* } 41 }
395 </pre>
396
397 <p>
398 More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.
399 </p>
400
401 <hr />
402 <h2><a name="check">Options for running the tests</a></h2>
403
404 <p> There are several ways to run the testsuite. There are two
405 harnesses, one using dejagnu and one using bash. In addition, there
406 is a special rule for checking the ABI of the shared library.
407 </p>
408
409 <p>You can check the status of the build without installing it
410 using the dejagnu harness, much like the rest of the gcc tools.</p>
411 <pre> make check</pre>
412 <p>in the <em>libbuilddir</em> directory.</p>
413 <p>or</p>
414 <pre> make check-target-libstdc++-v3</pre>
415 <p>in the <em>gccbuilddir</em> directory.</p>
416
417 <p>
418 These commands are equivalent and will create a 'testsuite'
419 directory underneath <em>libbuilddir</em> containing the results
420 of the tests. Two results files will be generated: <em>
421 libstdc++-v3.sum</em>, which is a PASS/FAIL summary for each
422 test, and <em>libstdc++.log</em> which is a log of the exact
423 command line passed to the compiler, the compiler output, and
424 the executable output (if any). In addition, four files are
425 generated that determine what test files are run. These files
426 are:
427 </p>
428
429 <ul>
430 <li>
431 <em>testsuite_files </em>
432 <p> This is a list of all the test cases that will be run. Each
433 test case is on a separate line, given with an absolute path
434 from the <em>libsrcdir/testsuite</em> directory.
435 </p>
436 </li>
437
438 <li>
439 <em>testsuite_files_interactive </em>
440 <p> This is a list of all the interactive test cases, using the
441 same format as the file list above. These tests are not run by default.
442 </p>
443 </li>
444
445 <li>
446 <em>testsuite_files_performance</em>
447 <p> This is a list of all the performance test cases, using the
448 same format as the file list above. These tests are not run by default.
449 </p>
450 </li>
451
452 <li>
453 <em>testsuite_wchar_t </em>
454 <p> This file indicates that the host system can run the wchar_t
455 tests, and corresponds to the macro definition <code>
456 _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h.
457 </p>
458 </li>
459 </ul>
460
461 <p>
462 To debug the dejagnu test harness during runs, try invoking with a
463 specific argument to the variable RUNTESTFLAGS, as below.
464 </p>
465
466 <pre>
467 make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
468 </pre>
469 or
470 <pre>
471 make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
472 </pre>
473
474 There are two ways to run on a simulator: set up DEJAGNU to point to a
475 specially crafted site.exp, or pass down --target_board flags.
476
477 Example flags to pass down for various embedded builds are as follows:
478
479 <pre>
480 --target=powerpc-eabism (libgloss/sim)
481 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"
482
483 --target=calmrisc32 (libgloss/sid)
484 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"
485
486 --target=xscale-elf (newlib/sim)
487 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
488 </pre>
489
490 <p> To run a subset of the library tests, simply edit the generated
491 file, <em>testsuite_files </em>, to include only the files that are
492 desired instead of all available test cases.
493 </p>
494
495 <p> In addition, there are some testing options that are mostly of
496 interest to library maintainers and system integrators. As such,
497 these tests may not work on all cpu and host combinations, and must
498 be executed in the <em>libbuilddir/testsuite</em> directory. These options
499 include, but are not necessarily limited to, the following:
500 </p>
501
502 <p>
503 The library can also be tested using a bash script, instead of
504 the default dejagnu test harness.
505 </p>
506 <pre>
507 make check-script</pre>
508 <p>
509 These commands use the generated test_file lists as above, but
510 run all the tests using both shared and static linking, and in
511 addition provide some additional diffing of expected output
512 files for the input/output tests. (This added diff may or may
513 not be useful or necessary at the moment.) In addition, these
514 tests provide size information for all the generated test cases,
515 so that size data for new compiler or linker features can be
516 collected. At one time timing information was attempted, so that
517 compile speeds, link speeds, etc. could be measured, however at
518 the moment all timing information is currently disabled.
519 </p>
520
521 <pre>
522 make check-script-install</pre>
523 <p> As directly above, but tests an installed library, not the
524 library and compiler in the build tree.
525 </p>
526
527 <pre>
528 make check-abi</pre>
529 <p>The library ABI can be tested. This involves testing the shared
530 library against an ABI-defining previous version. </p>
531
532 <pre>
533 make check-performance</pre>
534 <p>This rule runs through the <em>testsuite_files_performance</em>
535 test cases and collects information for performance analysis and
536 can be used to spot performance regressions. Various timing
537 information is collected, as well as number of hard page faults,
538 and memory used. This is not run by default, and the implementation
539 is in flux.
540 </p>
541
542 <p>
543 We are interested in any strange failures of the
544 testsuite; please see <a href="faq/index.html#2_4">FAQ 2.4</a>
545 for which files to examine.
546 </p>
547
548 <hr/>
549 <h2><a name="debug">Running debug-mode tests</a></h2>
550 <p>To run the libstdc++ test suite under the <a
551 href="debug.html#safe">debug mode</a>,
552 edit <code>libstdc++/scripts/testsuite_flags</code> to add the
553 compile-time flag <code>-D_GLIBCXX_DEBUG</code> to the result
554 printed by the <code>--build-cxx</code> option. Additionally, add
555 the <code>-D_GLIBCXX_DEBUG_PEDANTIC</code> flag to turn on pedantic
556 checking. The libstdc++ test suite should produce precisely the same
557 results under debug mode that it does under release mode: any
558 deviation indicates an error in either the library or the test
559 suite.</p>
560
561 <hr />
562 <h2><a name="future">Future</a></h2>
563
564 <p>
565 Shared runs need to be implemented, for targets that support shared libraries.
566 </p>
567
568 <p>
569 Diffing of expected output to standard streams needs to be finished off.
570 </p>
571
572 <p>
573 The V3 testing framework supports, or will eventually support,
574 additional keywords for the purpose of easing the job of writing
575 test cases. All V3-keywords are of the form <code>@xxx@</code>.
576 Currently plans for supported keywords include:
577 </p>
578
579 <dl>
580 <dt> <code> @require@ &lt;files&gt; </code> </dt>
581 <dd>
582 <p>
583 The existence of &lt;files&gt; is essential for the test to complete
584 successfully. For example, a test case foo.C using bar.baz as
585 input file could say
586 </p>
587 <pre>
588 // @require@ bar.baz</pre>
589 <p>
590 The special variable % stands for the rootname, e.g. the
591 file-name without its `.C' extension. Example of use (taken
592 verbatim from 27_io/filebuf.cc)
593 </p>
594 <pre>
595 // @require@ %-*.tst %-*.txt</pre>
596 </dd>
597 <dt> <code> @diff@ &lt;first-list&gt; &lt;second-list&gt; </code> </dt>
598 <dd>
599 <p>
600 After the test case compiles and ran successfully, diff
601 &lt;first-list&gt; against &lt;second-list&gt;, these lists should
602 have the same length. The test fails if diff returns non-zero a
603 pair of files.
604 </p>
605 </dd>
606 </dl>
607
608 <hr />
609 <h2><a name="internals">DejaGNU internals</a></h2>
610
611 <p>This is information for those looking at making changes to the testsuite
612 structure, and/or needing to trace dejagnu's actions with --verbose. This
613 will not be useful to people who are "merely" adding new tests to the existing
614 structure.
615 </p>
616
617 <p>The first key point when working with dejagnu is the idea of a "tool".
618 Files, directories, and functions are all implicitly used when they are
619 named after the tool in use. Here, the tool will always be "libstdc++".
620 </p>
621
622 <p>The <code>lib</code> subdir contains support routines. The
623 <code>lib/libstdc++.exp</code> file ("support library") is loaded
624 automagically, and must explicitly load the others. For example, files can
625 be copied from the core compiler's support directory into <code>lib</code>.
626 </p>
627
628 <p>Some routines in <code>lib/libstdc++.exp</code> are callbacks, some are
629 our own. Callbacks must be prefixed with the name of the tool. To easily
630 distinguish the others, by convention our own routines are named "v3-*".
631 </p>
632
633 <p>The next key point when working with dejagnu is "test files". Any
634 directory whose name starts with the tool name will be searched for test files.
635 (We have only one.) In those directories, any <code>.exp</code> file is
636 considered a test file, and will be run in turn. Our main test file is called
637 <code>normal.exp</code>; it runs all the tests in testsuite_files using the
638 callbacks loaded from the support library.
639 </p>
640
641 <p>The <code>config</code> directory is searched for any particular "target
642 board" information unique to this library. This is currently unused and sets
643 only default variables.
644 </p>
645
646
647 <!-- ####################################################### -->
648
649 <hr />
650 <p class="fineprint"><em>
651 See <a href="17_intro/license.html">license.html</a> for copying conditions.
652 Comments and suggestions are welcome, and may be sent to
653 <a href="mailto:libstdc++@gcc.gnu.org">the libstdc++ mailing list</a>.
654 </em></p>
655
656
657 </body>
658 </html>