README_DEVELOPERS: describe how to write regression tests.

diff --git a/README_DEVELOPERS b/README_DEVELOPERS
index eeb594508ae440266913b58c353deef756f452bf..a13b35574f57635c1313a45195d7075700c3ca67 100644 (file)
--- a/README_DEVELOPERS
+++ b/README_DEVELOPERS
@@ -87,6 +87,72 @@ compare them on all the performance tests:
 
   perl perf/vg_perf --vg=../trunk1 --vg=../trunk2 perf/
 
+Writing regression tests
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each tool has a tests directory containing regression tests. There is also
+the gdbserver_tests directory at the top level. Test directories may have
+architecture, OS and and architecture-OS sub-directories for tests that are
+specific to one architecture, OS or both.
+
+Once you have written a C or C++ executable that performs the required
+tests you will have to modify and create several files. The new files that
+will need adding are
+(a) a .vgtest file, which controls the test
+(b) a .stderr.exp file, which is the golden reference for any Valgrind output
+    Note that even if Valgrind doesn't produce any output you will need to
+    create an empty .stderr.exp file.
+(c) [optional] a .stdout.exp file, the golden reference for any output from the
+    test executable
+(d) [optional] filter files (see the test directories for examples).
+
+Quite often the output will depend on the platform the test is run on.
+Getting the tests to be 'portable' can require filtering or adding multiple
+.stderr.exp reference files.
+
+If the test only runs under certain conditions like the availability of functions
+in libc or C++ standard versions you will need to modify configure.ac in the
+top level directory. See AC_CHECK_FUNCS and the various blocks starting with
+AC_MSG_CHECKING.
+
+In the test directory, modify Makefile.am. Add to EXTRA_DIST the .vgtest,
+A .stderr.exp and .stderr.out files. Add any filters to dist_noinst_SCRIPTS.
+Add the test executable name to check_PROGRAMS. Try to respect the
+formatting and alphabetical ordering of the Makefile.am, For simple C files that is
+sufficient. If you needed to add a feature test to configure.ac then you should
+use the same condition to add the executable name to check_PROGRAMS. If the
+executable uses C++ you need to add exename_SOURCES. If the executable needs
+special compilation or link options, use exename_CFLAGS, exename_CXXFLAGS,
+exename_LDFLAGS or exename_LDADD. Finally in Makefile.am it's nice not to
+have any warnings, even if they were done on purpose. See configure.ac
+and various Makefile.am files for examples of using FLAG_W_*.
+
+The vgtest file contains the instructions for running the test. Typically
+it will contain (with examples in quotes)
+(a) the name of the test executable:      "prog: exename"
+(b) arguments for the test executable:    "args: hello world"
+(c) arguments for the Valgrind tool:      "vgopts: -q"
+(d) [optional] a check for prerequisites: "prereq: ! ../../tests/os_test darwin"
+(e) [optional] a filter:                  "stderr_filter: filter_fdleak"
+
+See tests/vg_regtest for a full description of all possible vgtest directives.
+
+The easiest way to generate the expected files is to run the test. Create empty
+files with touch (otherwise the test won't run) then run the test from the
+top directory using perl and vg_regtest script (as in the "Running the
+regression tests" section. Then copy "tool/tests/newtest.stderr.out" to
+"tool/tests/newtest.stderr.exp". Do the same for the .stderr.exp file.
+
+The last file to change is .gitignore in the top directory. Add a new entry,
+for example "/tool/tests/newtest".
+
+Check for mistakes in Makefile.am. In the top directory run
+    make post-regtest-checks
+
+You should only see
+    ...checking makefile consistency
+    ...checking header files and include directives
+and no messages related to EXTRA_DIST.
 
 Commit access and try branches
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~