]>
Commit | Line | Data |
---|---|---|
7f52823c | 1 | Testing with QMTest |
2 | =================== | |
3 | ||
4 | You can use QMTest to test G++. (In the future, it may be possible to | |
5 | test other parts of GCC with QMTest as well, but it is not possible | |
6 | yet.) | |
7 | ||
8 | The use of QMTest to run the G++ tests has not been approved as an | |
9 | officially supported testing procedure. Therefore, you must run the | |
10 | tests using DejaGNU (with "make check-g++") before committing changes | |
11 | that affect G++. | |
12 | ||
cb770a56 | 13 | QMTest emulates DejaGNU behavior very closely when running the tests. |
14 | ||
15 | QMTest has two output modes: a DejaGNU emulation mode and a native | |
16 | QMTest mode. | |
17 | ||
18 | In the DejaGNU mode, you should receive output that is almost exactly | |
19 | the same as the DejaGNU output; in particular, you should see the same | |
20 | number of passes, failures, etc. When using the DejaGNU-style output, | |
21 | QMTest uses the "xfail" indications in the test cases to determine | |
22 | which tests are expected to pass and which are expected to fail, and | |
23 | presents that information in the same way as DejaGNU. | |
24 | ||
25 | In the QMTest mode, the number of passes and failures will be | |
26 | different from that obtained when using DejaGNU. The reason is that a | |
27 | single source file may contain multiple DejaGNU tests. In DejaGNU, | |
28 | each line where a diagnostic is expected is considered a separate | |
29 | test. Testing for successful compilation and testing for successful | |
30 | execution of the generated program are considered separate tests. So, | |
31 | a single source file "test.C" could contain, say, seven tests; some of | |
32 | which might pass and some of which might fail. | |
33 | ||
34 | In the QMTest mode, each source file is considered a single test. If | |
35 | any of the seven sub-tests fail, the entire test is considered to | |
36 | fail. However, QMTest does present information about *why* the test | |
37 | failed, so the same information is effectively available. | |
38 | ||
39 | In the QMTest mode, whether or not a test is expected to fail is | |
40 | determined not by an indication in the test, but rather by comparing | |
41 | the new results to the results of a previous run. Testing for whether | |
42 | a change caused a regression is very simple: run the tests before | |
43 | making the change, run them again after making the change, and let | |
44 | QMTest compare the results. | |
45 | ||
46 | The mode chosen only affects the output from QMTest, not how it runs | |
47 | the tests or how it stores the data. Therefore, if you choose to run | |
48 | in the QMTest mode and later want to get the DejaGNU style output, or | |
49 | vice versa, you can do that as described below. | |
7f52823c | 50 | |
51 | Setting Up | |
52 | ========== | |
53 | ||
54 | You must download and install the following software: | |
55 | ||
cb770a56 | 56 | - Python 2.2 (or greater) |
7f52823c | 57 | |
58 | See http://www.python.org. | |
59 | ||
60 | You may already have Python on your system; in particular, many | |
61 | GNU/Linux systems ship with Python installed. | |
62 | ||
63 | Installation instructions are available on the web-site. | |
64 | ||
cb770a56 | 65 | - A current version of QMTest. No released version provides all of |
66 | the functionality required, so you must obtain QMTest from CVS. | |
7f52823c | 67 | |
cb770a56 | 68 | To do that, follow the instructions at: |
7f52823c | 69 | |
ee852bd0 | 70 | http://www.codesourcery.com/qmtest |
7f52823c | 71 | |
cb770a56 | 72 | Installation instructions are available in the file called README |
73 | after you check out QMTest. | |
7f52823c | 74 | |
cb770a56 | 75 | - The "qmtc" and "qmtest_gcc" QMTest support packages. These are |
76 | available from the same CVS repository as QMTest. For example, to | |
77 | check out "qmtc", do: | |
7f52823c | 78 | |
cb770a56 | 79 | cvs -d :pserver:anoncvs@cvs.codesourcery.com:/home/qm/Repository \ |
80 | co qmtc | |
7f52823c | 81 | |
cb770a56 | 82 | You do not have to install these packages; you need only check them |
83 | out. | |
84 | ||
7f52823c | 85 | Running the Tests |
86 | ================= | |
87 | ||
cb770a56 | 88 | First, you must set QMTEST_CLASS_PATH so that it can find the qmtc and |
89 | qmtest_gcc support packages: | |
90 | ||
91 | export QMTEST_CLASS_PATH=/path/to/qmtc:/path/to/qmtest_gcc | |
7f52823c | 92 | |
cb770a56 | 93 | The, run "make qmtest-g++" in the gcc directory of your build tree. |
7f52823c | 94 | |
95 | Here are some more advanced usage instructions: | |
96 | ||
97 | 1. To run a particular set of tests (rather than all of the tests), | |
98 | use the make variable "QMTEST_GPP_TESTS". For example, | |
99 | ||
f70cb9e6 | 100 | make QMTEST_GPP_TESTS="g++.dg" qmtest-g++ |
7f52823c | 101 | |
102 | will run only the tests in the g++.dg subdirectory, and: | |
103 | ||
cb770a56 | 104 | make QMTEST_GPP_TESTS="g++.dg/special/conpr-1.C \ |
f70cb9e6 | 105 | g++.old-deja/g++.other/access2.C" |
7f52823c | 106 | qmtest-g++ |
107 | ||
108 | will run only the two tests indicated. | |
109 | ||
110 | 2. To run qmtest with particular flags, use the make variables | |
111 | "QMTESTFLAGS" and "QMTESTRUNFLAGS". For example: | |
112 | ||
113 | make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++ | |
114 | ||
115 | will run qmtest like this: | |
116 | ||
117 | qmtest -v run -f full ... | |
118 | ||
cb770a56 | 119 | (The "-f full" mode will provide detailed information about each |
120 | test as it runs.) | |
121 | ||
7f52823c | 122 | 3. To run the compiler with particular flags, use QMTESTRUNFLAGS to |
cb770a56 | 123 | set the QMTest context variable "CompilerTable.cplusplus_options", |
124 | like this: | |
7f52823c | 125 | |
cb770a56 | 126 | make \ |
127 | QMTESTRUNFLAGS='-c CompilerTable.cplusplus_options="-funroll-loops"' \ | |
128 | qmtest-g++ | |
7f52823c | 129 | |
130 | The compiler will then use the "-funroll-loops" switch when | |
131 | compiling. | |
132 | ||
133 | 4. If qmtest is not in your path, you can indicate the full path to | |
134 | QMTest by using the make variable "QMTEST_PATH", like this: | |
135 | ||
136 | make QMTEST_PATH=/path/to/qmtest qmtest-g++ | |
137 | ||
138 | 5. To start the QMTest GUI, use: | |
139 | ||
140 | make qmtest-gui | |
141 | ||
cb770a56 | 142 | (Note that this will run the program called "mozilla" in your path. |
7f52823c | 143 | If you want to use another browser, you must configure qmtest as |
144 | described in its manual.) | |
145 | ||
146 | Bear in mind that the QMTest GUI is insecure; malicious users with | |
147 | access to your machine may be able to run commands as if they were | |
148 | you. The QMTest GUI only binds to the loopback IP addresss, which | |
149 | provides a measure of security, but not enough for use in untrusted | |
150 | environments. | |
151 | ||
152 | 6. If you have a multiprocessor, you can run the tests in parallel by | |
153 | passing the "-j" option to qmtest: | |
154 | ||
155 | make QMTESTRUNFLAGS="-j 4" qmtest-g++ | |
156 | ||
157 | will run tests in four threads. (It is also possible to run tests | |
158 | across multiple machines; for more information see the QMTest | |
159 | manual.) | |
160 | ||
cb770a56 | 161 | 7. If a test (say "g++.dg/abi/bitfield1.C") fails, and you want to get |
162 | more detailed information, you can do: | |
7f52823c | 163 | |
cb770a56 | 164 | cd qmtestsuite |
165 | qmtest summarize g++.qmr g++.dg/abi/bitfield1.C | |
7f52823c | 166 | |
cb770a56 | 167 | to get more information about the commands that were run and the |
168 | output produced. | |
7dfbd804 | 169 | |
170 | \f | |
171 | Copyright (C) 2002, 2003, 2004 Free Software Foundation, Inc. | |
172 | ||
173 | Copying and distribution of this file, with or without modification, | |
174 | are permitted in any medium without royalty provided the copyright | |
175 | notice and this notice are preserved. |