tets: make test documentation more user-friendly

Put the instructions to run tests right at the top of tests/README.md.

Give instructions to read the runtests.1 man page for information
about flags. Delete redundant copy of the flags documentation in the
README.

Add a mention in README.md of the important parallelism flag, to make
test runs go much faster.

Move documentation of output line format into the runtests.1 man page,
and update it with missing flags.

Fix the order of two flags in the man page.

Closes #12193

diff --git a/tests/README.md b/tests/README.md
index 65af2a0bd9ba32f3cbfbb772e5372143a135c077..6f027bddebb52534e1928d511feb921231ad83be 100644 (file)
--- a/tests/README.md
+++ b/tests/README.md
@@ -8,6 +8,41 @@ SPDX-License-Identifier: curl
 
 # Running
 
+  See the "Requires to run" section for prerequisites.
+
+  In the root of the curl repository:
+
+    ./configure && make && make test
+
+  To run a specific set of tests (e.g. 303 and 410):
+
+    make test TFLAGS="303 410"
+
+  To run the tests faster, pass the -j (parallelism) flag:
+
+    make test TFLAGS="-j10"
+
+  "make test" builds the test suite support code and invokes the 'runtests.pl'
+  perl script to run all the tests. The value of `TFLAGS` is passed
+  directly to 'runtests.pl'.
+
+  When you run tests via make, the flags `-a` and `-s` are passed, meaning
+  to continue running tests even after one fails, and to emit short output.
+
+  If you'd like to not use those flags, you can run 'runtests.pl' directly.
+  You must `chdir` into the tests directory, then you can run it like so:
+
+    ./runtests.pl 303 410
+
+  You must have run `make test` at least once first to build the support code.
+
+  To see what flags are available for runtests.pl, and what output it emits, run:
+
+    man ./tests/runtests.1
+
+  After a test fails, examine the tests/log directory for stdout, stderr, and
+  output from the servers used in the test.
+
 ## Requires to run
 
   - perl (and a unix-style shell)
@@ -75,44 +110,6 @@ SPDX-License-Identifier: curl
   used, set the environment variable `NGHTTPX`. The default can also be
   changed by specifying `--with-test-nghttpx=<path>` as argument to `configure`.
 
-### Run
-
-  `./configure && make && make test`. This builds the test suite support code
-  and invokes the 'runtests.pl' perl script to run all the tests. Edit the top
-  variables of that script in case you have some specific needs, or run the
-  script manually (after the support code has been built).
-
-  The script breaks on the first test that doesn't do OK. Use `-a` to prevent
-  the script from aborting on the first error. Run the script with `-v` for
-  more verbose output. Use `-d` to run the test servers with debug output
-  enabled as well. Specifying `-k` keeps all the log files generated by the
-  test intact.
-
-  Use `-s` for shorter output, or pass test numbers to run specific tests only
-  (like `./runtests.pl 3 4` to test 3 and 4 only). It also supports test case
-  ranges with 'to', as in `./runtests.pl 3 to 9` which runs the seven tests
-  from 3 to 9. Any test numbers starting with ! are disabled, as are any test
-  numbers found in the files `data/DISABLED` or `data/DISABLED.local` (one per
-  line). The latter is meant for local temporary disables and will be ignored
-  by git.
-
-  Test cases mentioned in `DISABLED` can still be run if `-f` is provided.
-
-  When `-s` is not present, each successful test will display on one line the
-  test number and description and on the next line a set of flags, the test
-  result, current test sequence, total number of tests to be run and an
-  estimated amount of time to complete the test run. The flags consist of
-  these letters describing what is checked in this test:
-
-    s stdout
-    d data
-    u upload
-    p protocol
-    o output
-    e exit code
-    m memory
-    v valgrind
-
 ### Shell startup scripts
 
   Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly

diff --git a/tests/runtests.1 b/tests/runtests.1
index 571e5ce0622de12ca57f4a4d34aa2dd3421c9263..6d895d2b9b34a46eef3047880e1c25586a98f5aa 100644 (file)
--- a/tests/runtests.1
+++ b/tests/runtests.1
@@ -53,6 +53,35 @@ this keyword. Remember that the exclamation marks and spaces will need to be
 quoted somehow when entered at many command shells.
 
 Prefix a keyword with a tilde (~) to still run it, but ignore the results.
+
+.SH "OUTPUT"
+
+When running without -s (short output), for instance when running runtests.pl
+directly rather than via make, each test will emits a pair of lines like this:
+
+Test 0045...[simple HTTP Location: without protocol in initial URL]
+--pd---e-v- OK (45  out of 1427, remaining: 16:08, took 6.188s, duration: 00:31)
+
+the first line contains the test number and a description. On the second line,
+the characters at the beginning are flags indicating which aspects of curl's
+behavior were checked by the test:
+
+    s stdout
+    r stderr
+    p protocol
+    d data
+    u upload
+    P proxy
+    o output
+    e exit code
+    m memory
+    v valgrind
+    E the test was run event-based
+
+The remainder of the second line contains the test result, current test sequence,
+total number of tests to be run and an estimated amount of time to complete the
+test run.
+
 .SH OPTIONS
 .IP "-a"
 Continue running the rest of the test cases even if one test fails. By
@@ -91,9 +120,6 @@ start.
 Run the given test(s) with gdb as a windowed application.
 .IP "-h, --help"
 Displays a help text about this program's command line options.
-.IP "-k"
-Keep output and log files in log/ after a test run, even if no error was
-detected. Useful for debugging.
 .IP "-j[num]"
 Spawn num processes to run tests. This defaults to 0 to run tests serially
 within a single process. Using a number greater than one allows multiple tests
@@ -101,6 +127,9 @@ to run in parallel, speeding up a test run. The optimum number is dependent on
 the system and set of tests to run, but 7*number of CPU cores is a good figure
 to start with, or 1.3*number of CPU cores if Valgrind is in use. Enabling
 parallel tests is not recommended in conjunction with the \-g option.
+.IP "-k"
+Keep output and log files in log/ after a test run, even if no error was
+detected. Useful for debugging.
 .IP "-L <file>"
 Load and execute the specified file which should contain perl code.
 This option allows one to change \fIruntests.pl\fP behaviour by overwriting

diff --git a/tests/runtests.pl b/tests/runtests.pl
index 32e0680378242d2d7d2382c0d74767534522f7fc..e7b9f01f6a6e7dd84d91fdae3da3133eb34a0ce8 100755 (executable)
--- a/tests/runtests.pl
+++ b/tests/runtests.pl
@@ -23,6 +23,8 @@
 #
 ###########################################################################
 
+# For documentation, run `man ./runtests.1` and see README.md.
+
 # Experimental hooks are available to run tests remotely on machines that
 # are able to run curl but are unable to run the test harness.
 # The following sections need to be modified:
@@ -1503,7 +1505,7 @@ sub singletest_check {
 
     }
     else {
-        $ok .= "-"; # protocol not checked
+        $ok .= "-"; # proxy not checked
     }
 
     my $outputok;