From: Paul Floyd Date: Wed, 15 May 2024 19:40:48 +0000 (+0200) Subject: README_DEVELOPERS: describe how to write regression tests. X-Git-Tag: VALGRIND_3_24_0~135 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=a94cee615a1cf98ad91ccb092a24ead60ba90d29;p=thirdparty%2Fvalgrind.git README_DEVELOPERS: describe how to write regression tests. --- diff --git a/README_DEVELOPERS b/README_DEVELOPERS index eeb594508..a13b35574 100644 --- a/README_DEVELOPERS +++ b/README_DEVELOPERS @@ -87,6 +87,72 @@ compare them on all the performance tests: perl perf/vg_perf --vg=../trunk1 --vg=../trunk2 perf/ +Writing regression tests +~~~~~~~~~~~~~~~~~~~~~~~~ + +Each tool has a tests directory containing regression tests. There is also +the gdbserver_tests directory at the top level. Test directories may have +architecture, OS and and architecture-OS sub-directories for tests that are +specific to one architecture, OS or both. + +Once you have written a C or C++ executable that performs the required +tests you will have to modify and create several files. The new files that +will need adding are +(a) a .vgtest file, which controls the test +(b) a .stderr.exp file, which is the golden reference for any Valgrind output + Note that even if Valgrind doesn't produce any output you will need to + create an empty .stderr.exp file. +(c) [optional] a .stdout.exp file, the golden reference for any output from the + test executable +(d) [optional] filter files (see the test directories for examples). + +Quite often the output will depend on the platform the test is run on. +Getting the tests to be 'portable' can require filtering or adding multiple +.stderr.exp reference files. + +If the test only runs under certain conditions like the availability of functions +in libc or C++ standard versions you will need to modify configure.ac in the +top level directory. See AC_CHECK_FUNCS and the various blocks starting with +AC_MSG_CHECKING. + +In the test directory, modify Makefile.am. Add to EXTRA_DIST the .vgtest, +A .stderr.exp and .stderr.out files. Add any filters to dist_noinst_SCRIPTS. +Add the test executable name to check_PROGRAMS. Try to respect the +formatting and alphabetical ordering of the Makefile.am, For simple C files that is +sufficient. If you needed to add a feature test to configure.ac then you should +use the same condition to add the executable name to check_PROGRAMS. If the +executable uses C++ you need to add exename_SOURCES. If the executable needs +special compilation or link options, use exename_CFLAGS, exename_CXXFLAGS, +exename_LDFLAGS or exename_LDADD. Finally in Makefile.am it's nice not to +have any warnings, even if they were done on purpose. See configure.ac +and various Makefile.am files for examples of using FLAG_W_*. + +The vgtest file contains the instructions for running the test. Typically +it will contain (with examples in quotes) +(a) the name of the test executable: "prog: exename" +(b) arguments for the test executable: "args: hello world" +(c) arguments for the Valgrind tool: "vgopts: -q" +(d) [optional] a check for prerequisites: "prereq: ! ../../tests/os_test darwin" +(e) [optional] a filter: "stderr_filter: filter_fdleak" + +See tests/vg_regtest for a full description of all possible vgtest directives. + +The easiest way to generate the expected files is to run the test. Create empty +files with touch (otherwise the test won't run) then run the test from the +top directory using perl and vg_regtest script (as in the "Running the +regression tests" section. Then copy "tool/tests/newtest.stderr.out" to +"tool/tests/newtest.stderr.exp". Do the same for the .stderr.exp file. + +The last file to change is .gitignore in the top directory. Add a new entry, +for example "/tool/tests/newtest". + +Check for mistakes in Makefile.am. In the top directory run + make post-regtest-checks + +You should only see + ...checking makefile consistency + ...checking header files and include directives +and no messages related to EXTRA_DIST. Commit access and try branches ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~