From: drh <>
Date: Thu, 1 Jan 2026 21:35:25 +0000 (+0000)
Subject: Further improvements to testrunner.tcl documentation.
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=b0a1ed2980e6af9173157d7f6e9d102c26f9c234;p=thirdparty%2Fsqlite.git

Further improvements to testrunner.tcl documentation.

FossilOrigin-Name: c71402031ce81b4415a793afa3152f5691b197dce634c1322948f39086b8c819
---

diff --git a/doc/testrunner.md b/doc/testrunner.md
index a79e517547..90ef4b71f2 100644
--- a/doc/testrunner.md
+++ b/doc/testrunner.md
@@ -4,6 +4,12 @@
 
 <ul type=none>
   <li> 1. <a href=#overview>Overview</a>
+<ul type=none>
+  <li> 1.1. <a href=#runtr>Running testrunner.tcl</a>
+  <li> 1.2. <a href=#runviamake>Run using "make"</a>
+  <li> 1.3. <a href=#outputs>Outputs from testrunner.tcl</a>
+  <li> 1.4. <a href=#help>Built-in help</a>
+</ul>
   <li> 2. <a href=#binary_tests>Binary Tests</a>
 <ul type=none>
   <li> 2.1. <a href=#organization_tests>Organization of Tcl Tests</a>
@@ -26,17 +32,40 @@
 
 The testrunner.tcl program is a Tcl script used to run multiple SQLite
 tests in parallel, thus reducing testing time on multi-core machines.
-It supports the following types of tests:
+The testrunner.tcl supports running tests that based on `testfixture`,
+`sqlite3`, and `fuzzcheck`.
+
+<a name="runtr"></a>
+## 1.1 Running testrunner.tcl
+
+The testrunner.tcl script is located in the "test" subdirectory of the
+SQLite source tree.  So if your shell is current positioned at the top
+of the source tree, you would normally run the script using the command:
+"<tt>test/testrunner.tcl</tt>".  On Windows, you have to specify the
+`tclsh` interpreter command first, like this:
+"<tt>tclsh&nbsp;test/testrunner.tcl</tt>".
 
-  *  Tcl test scripts.
+In this document, we will assume that you are on a unix-like OS
+(not on Windows) and that your current directory is the root
+of the SQLite source tree, and so all invocations of the testrunner.tcl
+script will be of the form "<tt>test/testrunner.tcl</tt>".  If you
+are in a different directory, then make appropriate adjustments to
+the path.  On Windows, add the "<tt>tclsh</tt>" interpreter command
+up front.
 
-  *  Fuzzcheck tests, including using an external fuzzcheck database.
+<a name="runviamake"></a>
+## 1.2 Run using make
 
-  *  Tests run with `make` commands.  Examples:
-      -  `make devtest`
-      -  `make releasetest`
-      -  `make sdevtest`
-      -  `make testrunner`
+The standard Makefiles for SQLite include targets that invoke
+testrunner.tcl.  So the following commands also run testrunner.tcl:
+
+   *  `make devtest`
+   *  `make releasetest`
+   *  `make sdevtest`
+   *  `make testrunner`
+
+<a name="outputs"></a>
+## 1.3 Outputs from testrunner.tcl
 
 The testrunner.tcl program stores output of all tests and builds run in
 log file **testrunner.log**, created in the current working directory.
@@ -54,17 +83,19 @@ A useful query might be:
 ```
 
 You can get a summary of errors in a prior run by invoking commands like
-these:
+those shown below.  Note that the testrunner.tcl script can be run directly
+on unix systems (including Macs) but you will need to add <tt>tclsh</tt>
+to the front on Windows.
 
 ```
-  tclsh $(TESTDIR)/testrunner.tcl errors
-  tclsh $(TESTDIR)/testrunner.tcl errors -v
+  test/testrunner.tcl errors
+  test/testrunner.tcl errors -v
 ```
 
 Running the command:
 
 ```
-  tclsh $(TESTDIR)/testrunner.tcl status
+  test/testrunner.tcl status
 ```
 
 in the directory containing the testrunner.db database runs various queries
@@ -73,8 +104,8 @@ A good way to keep and eye on test progress is to run either of the two
 following commands:
 
 ```
-  watch tclsh $(TESTDIR)/testrunner.tcl status
-  tclsh $(TESTDIR)/testrunner.tcl status -d 2
+  watch test/testrunner.tcl status
+  test/testrunner.tcl status -d 2
 ```
 
 Both of the commands above accomplish about the same thing, but the second
@@ -86,6 +117,19 @@ are in the directory from which testrunner.tcl is run.
 (see "Binary Tests" below). Sometimes it builds testfixture and
 other binaries in specific configurations to test (see "Source Tests").
 
+<a name=help></a>
+## 1.4 Built-in help
+
+Run this command:
+
+```
+  test/testrunner.tcl help
+```
+
+To get a summary of all of the various command-line options available
+with testrunner.tcl
+
+
 <a name=binary_tests></a>
 # 2. Binary Tests
 
@@ -94,8 +138,9 @@ test scripts using whatever `testfixture` binary (and maybe also the `sqlite3`
 binary, depending on the test) that is found in the directory from which
 testrunner.tcl is launched.  So typically, one must first run something
 like "`make testfixture sqlite3`" before launching binary tests.  In other
-words, testrunner.tcl does not build the binaries under test itself; it
-expects them to be available already.
+words, testrunner.tcl does not automatically build the binaries under test
+for binary tests.  The testrunner.tcl expects the binaries to be available
+already.
 
 The following sub-sections describe the various options that can be
 passed to testrunner.tcl to test binary testfixture builds.
@@ -137,22 +182,22 @@ are defined in file *testrunner_data.tcl*.
 To run the "veryquick" test set, use either of the following:
 
 ```
-  tclsh $TESTDIR/testrunner.tcl
-  tclsh $TESTDIR/testrunner.tcl veryquick
+  test/testrunner.tcl
+  test/testrunner.tcl veryquick
 ```
 
 To run the "full" test suite:
 
 ```
-  tclsh $TESTDIR/testrunner.tcl full
+  test/testrunner.tcl full
 ```
 
 To run the subset of the "full" test suite for which the test file name matches
 a specified pattern (e.g. all tests that start with "fts5"), either of:
 
 ```
-  tclsh $TESTDIR/testrunner.tcl fts5%
-  tclsh $TESTDIR/testrunner.tcl 'fts5*'
+  test/testrunner.tcl fts5%
+  test/testrunner.tcl 'fts5*'
 ```
 
 Strictly speaking, for a test to be run the pattern must match the script
@@ -164,14 +209,9 @@ characters specified as part of the pattern are transformed to "\*".
 To run "all" tests (full + permutations):
 
 ```
-  tclsh $TESTDIR/testrunner.tcl all
+  test/testrunner.tcl all
 ```
 
-Note that in all of the previous examples, when running on Unix, the
-initial "<tt>tclsh</tt>" on the command-line may be omitted.  The
-testrunner.tcl is marked as executable and begins with an appropriate
-"<tt>#!</tt>" line to invoke tclsh automatically.
-
 <a name=binary_test_failures></a>
 ## 2.3. Investigating Binary Test Failures
 
@@ -182,20 +222,20 @@ be retrieved from either *testrunner.log* or *testrunner.db*.
 If there is no permutation, the individual test script may be run with:
 
 ```
-  tclsh $PATH_TO_SCRIPT
+  ./testfixture $PATH_TO_SCRIPT
 ```
 
 Or, if the failure occured as part of a permutation:
 
 ```
-  tclsh $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
+  test/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
 ```
 
-One can also run all tests that failed in the previous invocation by
-typing:
+One can also rerun all tests that failed or did not complete
+in the previous invocation by typing:
 
 ```
-  tclsh $TESTDIR/testrunner.tcl retest
+  test/testrunner.tcl retest
 ```
 
 <a name=source_code_tests></a>
@@ -223,7 +263,7 @@ the `make fuzztest` target once for each of two --enable-all builds - one
 with debugging enabled and one without:
 
 ```
-  tclsh $TESTDIR/testrunner.tcl mdevtest
+  test/testrunner.tcl mdevtest
 ```
 
 In other words, it is equivalent to running:
@@ -232,13 +272,13 @@ In other words, it is equivalent to running:
   $TOP/configure --enable-all --enable-debug
   make fuzztest
   make testfixture
-  tclsh $TOP/test/testrunner.tcl veryquick
+  $TOP/test/testrunner.tcl veryquick
 
   # Then, after removing files created by the tests above:
   $TOP/configure --enable-all OPTS="-O0"
   make fuzztest
   make testfixture
-  tclsh $TOP/test/testrunner.tcl veryquick
+  $TOP/test/testrunner.tcl veryquick
 ```
 
 The **sdevtest** command is identical to the mdevtest command, except that the
@@ -246,7 +286,7 @@ second of the two builds is a sanitizer build. Specifically, this means that
 OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":
 
 ```
-  tclsh $TESTDIR/testrunner.tcl sdevtest
+  test/testrunner.tcl sdevtest
 ```
 
 The **release** command runs lots of tests under lots of builds. It runs
@@ -255,7 +295,7 @@ on Linux, Windows or OSX. Refer to *testrunner\_data.tcl* for the details
 of the specific tests run.
 
 ```
-  tclsh $TESTDIR/testrunner.tcl release
+  test/testrunner.tcl release
 ```
 
 As with <a href=#source code tests>source code tests</a>, one or more patterns
@@ -263,7 +303,7 @@ may be appended to any of the above commands (mdevtest, sdevtest or release).
 Pattern matching is used for both Tcl tests and fuzz tests.
 
 ```
-  tclsh $TESTDIR/testrunner.tcl release rtree%
+  test/testrunner.tcl release rtree%
 ```
 
 <a name=zipvfs_tests></a>
@@ -273,14 +313,14 @@ testrunner.tcl can build a zipvfs-enabled testfixture and use it to run
 tests from the Zipvfs project with the following command:
 
 ```
-  tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
+  test/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
 ```
 
 This can be combined with any of "mdevtest", "sdevtest" or "release" to
 test both SQLite and Zipvfs with a single command:
 
 ```
-  tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
+  test/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
 ```
 
 <a name=source_code_test_failures></a>
@@ -300,10 +340,10 @@ a dos \*.bat file on windows. For example:
 ```
   # Create a script that recreates build configuration "Device-One" on 
   # Linux or OSX:
-  tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh 
+  test/testrunner.tcl script Device-One > make.sh 
 
   # Create a script that recreates build configuration "Have-Not" on Windows:
-  tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat 
+  test/testrunner.tcl script Have-Not > make.bat 
 ```
 
 The generated bash or \*.bat file script accepts a single argument - a makefile
@@ -326,7 +366,7 @@ Thus, for example, to run a full releasetest including an external
 dbsqlfuzz database, run a command like one of these:
 
 ```
-  tclsh test/testrunner.tcl releasetest --fuzzdb ../fuzz/20250415.db
+  test/testrunner.tcl releasetest --fuzzdb ../fuzz/20250415.db
   FUZZDB=../fuzz/20250415.db make releasetest
   nmake /f Makefile.msc FUZZDB=../fuzz/20250415.db releasetest
 ```
@@ -336,7 +376,7 @@ databases.  So if you want to run *only* tests involving the external
 database, you can use a command something like this:
 
 ```
-  tclsh test/testrunner.tcl releasetest 20250415 --fuzzdb ../fuzz/20250415.db
+  test/testrunner.tcl releasetest 20250415 --fuzzdb ../fuzz/20250415.db
 ```
 
 <a name=testrunner_options></a>
@@ -350,7 +390,7 @@ required by a test, not to run any actual tests. For example:
 
 ```
   # Build binaries required by release test.
-  tclsh $TESTDIR/testrunner.tcl --buildonly release"
+  test/testrunner.tcl --buildonly release"
 ```
 
 The **--dryrun** option prevents testrunner.tcl from building any binaries
@@ -359,7 +399,7 @@ would normally execute into the testrunner.log file. Example:
 
 ```
   # Log the shell commmands that make up the mdevtest test.
-  tclsh $TESTDIR/testrunner.tcl --dryrun mdevtest"
+  test/testrunner.tcl --dryrun mdevtest"
 ```
 
 The **--explain** option is similar to --dryrun in that it prevents
@@ -369,7 +409,7 @@ summary of all the builds and tests that would have been run.
 
 ```
   # Show what builds and tests would have been run
-  tclsh $TESTDIR/testrunner.tcl --explain mdevtest
+  test/testrunner.tcl --explain mdevtest
 ```
 
 The **--status** option uses VT100 escape sequences to display the test
@@ -385,7 +425,7 @@ When running either binary or source code tests, testrunner.tcl reports the
 number of jobs it intends to use to stdout. e.g.
 
 ```
-  $ tclsh $TESTDIR/testrunner.tcl
+  $ test/testrunner.tcl
   splitting work across 16 jobs
   ... more output ...
 ```
@@ -395,7 +435,7 @@ of real cores on the machine. This can be overridden using the "--jobs" (or -j)
 switch:
 
 ```
-  $ tclsh $TESTDIR/testrunner.tcl --jobs 8
+  $ test/testrunner.tcl --jobs 8
   splitting work across 8 jobs
   ... more output ...
 ```
@@ -405,5 +445,5 @@ running by exucuting the following command from the directory containing the
 testrunner.log and testrunner.db files:
 
 ```
-  $ tclsh $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
+  $ test/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
 ```
diff --git a/manifest b/manifest
index 88e8b0bcae..5b1f6a2577 100644
--- a/manifest
+++ b/manifest
@@ -1,5 +1,5 @@
-C Updates\sto\sthe\stestrunner\sdocumentation.
-D 2026-01-01T15:41:50.965
+C Further\simprovements\sto\stestrunner.tcl\sdocumentation.
+D 2026-01-01T21:35:25.298
 F .fossil-settings/binary-glob 61195414528fb3ea9693577e1980230d78a1f8b0a54c78cf1b9b24d0a409ed6a x
 F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1
 F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea
@@ -63,7 +63,7 @@ F doc/jsonb.md acd77fc3a709f51242655ad7803510c886aa8304202fa9cf2abc5f5c4e9d7ae5
 F doc/lemon.html 2085fda0a90a94fe92159a79dccc5c30d5a2313524887a31659cd66162f17233
 F doc/pager-invariants.txt 83aa3a4724b2d7970cc3f3461f0295c46d4fc19a835a5781cbb35cb52feb0577
 F doc/tcl-extension-testing.md b88861804fc1eaf83249f8e206334189b61e150c360e1b80d0dcf91af82354f5
-F doc/testrunner.md 1232d8095a15a6aa6fbb6685313558038dde49e6b0b7b1daebd5cbeec5d0abf9
+F doc/testrunner.md daffa0ebbbe397a73537ae1b19b3124d489ce5f89dfe570781d1f1ef1809597c
 F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a
 F doc/vdbesort-memory.md 4da2639c14cd24a31e0af694b1a8dd37eaf277aff3867e9a8cc14046bc49df56
 F doc/vfs-shm.txt 1a55f3f0e7b6745931b117ba5c9df3640d7a0536f532ef0052563100f4416f86
@@ -2189,8 +2189,8 @@ F tool/warnings-clang.sh bbf6a1e685e534c92ec2bfba5b1745f34fb6f0bc2a362850723a9ee
 F tool/warnings.sh d924598cf2f55a4ecbc2aeb055c10bd5f48114793e7ba25f9585435da29e7e98
 F tool/win/sqlite.vsix deb315d026cc8400325c5863eef847784a219a2f
 F tool/winmain.c 00c8fb88e365c9017db14c73d3c78af62194d9644feaf60e220ab0f411f3604c
-P a5f922b0832d9544b083f3d55d4029aa80e320bf5505547c5a1ddb21f4685e3e
-R 9df0730403c6e3e10d8224307f93708f
+P bcac99f962c6403fddfcaeef309ea9f825fa661fee0b3e3fd24f04c861be1d1b
+R 9387e28ba84b235b58989d6495dd42c1
 U drh
-Z 4765444ce4544ead9159cfe583142a35
+Z a047d21b7e9c1686bf63375354e2eb38
 # Remove this line to create a well-formed Fossil manifest.
diff --git a/manifest.uuid b/manifest.uuid
index 13a3fa8d35..9fa37b4ad6 100644
--- a/manifest.uuid
+++ b/manifest.uuid
@@ -1 +1 @@
-bcac99f962c6403fddfcaeef309ea9f825fa661fee0b3e3fd24f04c861be1d1b
+c71402031ce81b4415a793afa3152f5691b197dce634c1322948f39086b8c819