Update testrunner.tcl documentation. Fix typo in testrunner.tcl output.

FossilOrigin-Name: ba8800b4ba0a690c89d9fb96bb9a0d8789d04160ff9c47ea9afb2b403bec7f0b

diff --git a/doc/testrunner.md b/doc/testrunner.md
index d0248573ee05cdfdcb2a202d78e415078ad32afd..c2808a1e4810b6ae5dad4bbbc60ee5bae8417a3d 100644 (file)
--- a/doc/testrunner.md
+++ b/doc/testrunner.md
@@ -15,6 +15,7 @@
   <li> 3.1. <a href=#commands_to_run_tests>Commands to Run SQLite Tests</a>
   <li> 3.2. <a href=#zipvfs_tests>Running ZipVFS Tests</a>
   <li> 3.3. <a href=#source_code_test_failures>Investigating Source Code Test Failures</a>
+  <li> 3.4. <a href=#fuzzdb>External Fuzzcheck Databases</a>
 </ul>
   <li> 4. <a href=#testrunner_options>Extra testrunner.tcl Options</a>
   <li> 5. <a href=#cpu_cores>Controlling CPU Core Utilization</a>
@@ -23,27 +24,30 @@
 <a name=overview></a>
 # 1. Overview
 
-testrunner.tcl is a Tcl script used to run multiple SQLite tests using 
-multiple jobs. It supports the following types of tests:
+The testrunner.tcl program is a Tcl script used to run multiple SQLite
+tests in parallel, thus reducing testing time on multi-core machines.
+It supports the following types of tests:
 
   *  Tcl test scripts.
 
+  *  Fuzzcheck tests, including using an external fuzzcheck database.
+
   *  Tests run with `make` commands.  Examples:
-      -  `make mdevtest`
+      -  `make devtest`
       -  `make releasetest`
       -  `make sdevtest`
       -  `make testrunner`
 
-testrunner.tcl pipes the output of all tests and builds run into log file
-**testrunner.log**, created in the current working directory. Search this
-file to find details of errors.  Suggested search commands:
+The testrunner.tcl program stores output of all tests and builds run in
+log file **testrunner.log**, created in the current working directory.
+Search this file to find details of errors.  Suggested search commands:
 
    *  `grep "^!" testrunner.log`
    *  `grep failed testrunner.log`
 
-testrunner.tcl also populates SQLite database **testrunner.db**. This database
-contains details of all tests run, running and to be run. A useful query
-might be:
+The testrunner.tcl program also populates SQLite database **testrunner.db**.
+This database contains details of all tests run, running and to be run.
+A useful query might be:
 
 ```
   SELECT * FROM script WHERE state='failed'
@@ -52,7 +56,7 @@ might be:
 Running the command:
 
 ```
-  ./testfixture $(TESTDIR)/testrunner.tcl status
+  tclsh $(TESTDIR)/testrunner.tcl status
 ```
 
 in the directory containing the testrunner.db database runs various queries
@@ -60,7 +64,7 @@ to produce a succinct report on the state of a running testrunner.tcl script.
 Running:
 
 ```
-  watch ./testfixture $(TESTDIR)/testrunner.tcl status
+  watch tclsh $(TESTDIR)/testrunner.tcl status
 ```
 
 in another terminal is a good way to keep an eye on a long running test.
@@ -239,9 +243,7 @@ of the specific tests run.
 
 As with <a href=#source code tests>source code tests</a>, one or more patterns
 may be appended to any of the above commands (mdevtest, sdevtest or release).
-In that case only Tcl tests (no fuzz or other tests) that match the specified
-pattern are run. For example, to run the just the Tcl rtree tests in all 
-builds and configurations supported by "release":
+Pattern matching is used for both Tcl tests and fuzz tests.
 
 ```
   tclsh $TESTDIR/testrunner.tcl release rtree%
@@ -292,6 +294,33 @@ target to build. This may be used either to run a `make` command test directly,
 or else to build a testfixture (or testfixture.exe) binary with which to
 run a Tcl test script, as <a href=#binary_test_failures>described above</a>.
 
+<a name=fuzzdb></a>
+## 3.4 External Fuzzcheck Databases
+
+Testrunner.tcl will also run fuzzcheck against external (out of tree)
+databases, for example fuzzcheck databases generated by dbsqlfuzz.  To do
+this, simply set the FUZZDB environment variable to the name of the external
+database.  For large external databases, testrunner.tcl will automatically use
+the "`--slice`" command-line option of fuzzcheck to divide the work up between
+multiple jobs, thus increasing parallelism.
+
+Thus, for example, to run a full releasetest including an external
+dbsqlfuzz database, run a command like one of these:
+
+```
+  FUZZDB=../fuzz/20250415.db test/testrunner.tcl releasetest
+  FUZZDB=../fuzz/20250415.db make releasetest
+  nmake /f Makefile.msc FUZZDB=../fuzz/20250415.db releasetest
+```
+
+The patternlist option to testrunner.tcl will match against fuzzcheck
+databases.  So if you want to run *only* tests involving the external
+database, you can use a command something like this:
+
+```
+  FUZZDB=../fuzz/20250415.db test/testrunner.tcl releasetest 20250415
+```
+
 <a name=testrunner_options></a>
 # 4. Extra testrunner.tcl Options
 
@@ -315,16 +344,22 @@ would normally execute into the testrunner.log file. Example:
   tclsh $TESTDIR/testrunner.tcl --dryrun mdevtest"
 ```
 
-The **--explain** option is similar to --dryrun in that it prevents testrunner.tcl
-from building any binaries or running any tests.  The difference is that --explain
-prints on standard output a human-readable summary of all the builds and tests that
-would have been run.
+The **--explain** option is similar to --dryrun in that it prevents
+testrunner.tcl from building any binaries or running any tests.  The
+difference is that --explain prints on standard output a human-readable
+summary of all the builds and tests that would have been run.
 
 ```
   # Show what builds and tests would have been run
   tclsh $TESTDIR/testrunner.tcl --explain mdevtest
 ```
 
+The **--status** option uses VT100 escape sequences to display the test
+status full-screen.  This is similar to running 
+"`watch test/testrunner status`" in a separate window, just more convenient.
+Unfortunately, this option does not work correctly on Windows, due to the
+sketchy implementation of VT100 escapes on the Windows console.
+
 <a name=cpu_cores></a>
 # 5. Controlling CPU Core Utilization
 

diff --git a/manifest b/manifest
index db0f69d9c33c9967522f854f3ac5c13ad7437067..8616d04e15e84f771bd9579b5796d63a04903374 100644 (file)
--- a/manifest
+++ b/manifest
@@ -1,5 +1,5 @@
-C Improved\s--explain\sdisplay\sof\sFUZZDB\stest\splans\sin\stestrunner.tcl.
-D 2025-04-29T10:41:16.849
+C Update\stestrunner.tcl\sdocumentation.\s\sFix\stypo\sin\stestrunner.tcl\soutput.
+D 2025-04-29T11:35:51.925
 F .fossil-settings/binary-glob 61195414528fb3ea9693577e1980230d78a1f8b0a54c78cf1b9b24d0a409ed6a x
 F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1
 F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea
@@ -64,7 +64,7 @@ F doc/jsonb.md ede3238186e3a90bb79d20b2a6a06d0f9429a38e069e9da0efbad0f2ed48b924
 F doc/lemon.html 89ea833a6f71773ab1a9063fbb7fb9b32147bc0b1057b53ecab94a3b30c0aef5
 F doc/pager-invariants.txt 83aa3a4724b2d7970cc3f3461f0295c46d4fc19a835a5781cbb35cb52feb0577
 F doc/tcl-extension-testing.md b88861804fc1eaf83249f8e206334189b61e150c360e1b80d0dcf91af82354f5
-F doc/testrunner.md 15583cf8c7d8a1c3378fd5d4319ca769a14c4d950a5df9b015d01d5be290dc69
+F doc/testrunner.md 41f3ab025a4e6f7505f476c9b3c01a65d109015dc4f4b23797d9c18bed4749be
 F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a
 F doc/vdbesort-memory.md 4da2639c14cd24a31e0af694b1a8dd37eaf277aff3867e9a8cc14046bc49df56
 F doc/vfs-shm.txt 1a55f3f0e7b6745931b117ba5c9df3640d7a0536f532ef0052563100f4416f86
@@ -1734,7 +1734,7 @@ F test/temptable2.test 76821347810ecc88203e6ef0dd6897b6036ac788e9dd3e6b04fd4d163
 F test/temptable3.test d11a0974e52b347e45ee54ef1923c91ed91e4637
 F test/temptrigger.test 38f0ca479b1822d3117069e014daabcaacefffcc
 F test/tester.tcl 463ae33b8bf75ac77451df19bd65e7c415c2e9891227c7c9e657d0a2d8e1074a
-F test/testrunner.tcl 038b538f04c857066b7057661878f4cc1517860f6c3f040087455780fc9f20dd x
+F test/testrunner.tcl dc0230b45cae84607dfcd5b8713966b47c0820fa4718a34d705e1423dedc0b0f x
 F test/testrunner_data.tcl 8d5fa3851c48bc94e26db0be325202e44f6ca4ed838272b8d5b10c23817621e6
 F test/thread001.test a0985c117eab62c0c65526e9fa5d1360dd1cac5b03bde223902763274ce21899
 F test/thread002.test c24c83408e35ba5a952a3638b7ac03ccdf1ce4409289c54a050ac4c5f1de7502
@@ -2207,8 +2207,8 @@ F tool/version-info.c 3b36468a90faf1bbd59c65fd0eb66522d9f941eedd364fabccd7227350
 F tool/warnings-clang.sh bbf6a1e685e534c92ec2bfba5b1745f34fb6f0bc2a362850723a9ee87c1b31a7
 F tool/warnings.sh 49a486c5069de041aedcbde4de178293e0463ae9918ecad7539eedf0ec77a139
 F tool/win/sqlite.vsix deb315d026cc8400325c5863eef847784a219a2f
-P 10d8401d59136dda7051af0da398b06a18031560f2734f4fa347d339765d2089
-R f8d86c9e45d3a96c13f67cb70749dc31
+P 42bdd7262ec4f8163263e0535c2cc35b526754d77e1f3ccf5b73b0f44d470d9c
+R 79541406257d79ca4afc3c688f589b5b
 U drh
-Z 1e9ce377b8e70ac62de3d9fa2977f8de
+Z 6a36b9638ff8ab1c4c150879edd48798
 # Remove this line to create a well-formed Fossil manifest.

diff --git a/manifest.uuid b/manifest.uuid
index 9ef39976ab9647aae432215940841cf965bf5de4..1f12a6917ffa84402ac8bc88014978c6e299a248 100644 (file)
--- a/manifest.uuid
+++ b/manifest.uuid
@@ -1 +1 @@
-42bdd7262ec4f8163263e0535c2cc35b526754d77e1f3ccf5b73b0f44d470d9c
+ba8800b4ba0a690c89d9fb96bb9a0d8789d04160ff9c47ea9afb2b403bec7f0b

diff --git a/test/testrunner.tcl b/test/testrunner.tcl
index 1248048103d52cd11894f606b9fdaf1d16d3fd76..a2896fd08f1f8889e57d40d91a322162c8ccbf98 100755 (executable)
--- a/test/testrunner.tcl
+++ b/test/testrunner.tcl
@@ -1647,7 +1647,7 @@ proc run_testset {} {
   puts "\nTest database is $TRG(dbname)"
   puts "Test log is $TRG(logname)"
   if {[info exists TRG(FUZZDB)]} {
-    puts "Extra fuzztest data taken from $TRG(FUZZDB)"
+    puts "Extra fuzzcheck data taken from $TRG(FUZZDB)"
   }
   trdb eval {
      SELECT sum(ntest) AS totaltest,