<ul type=none>
<li> 1. <a href=#overview>Overview</a>
+<ul type=none>
+ <li> 1.1. <a href=#runtr>Running testrunner.tcl</a>
+ <li> 1.2. <a href=#runviamake>Run using "make"</a>
+ <li> 1.3. <a href=#outputs>Outputs from testrunner.tcl</a>
+ <li> 1.4. <a href=#help>Built-in help</a>
+</ul>
<li> 2. <a href=#binary_tests>Binary Tests</a>
<ul type=none>
<li> 2.1. <a href=#organization_tests>Organization of Tcl Tests</a>
The testrunner.tcl program is a Tcl script used to run multiple SQLite
tests in parallel, thus reducing testing time on multi-core machines.
-It supports the following types of tests:
+The testrunner.tcl supports running tests that based on `testfixture`,
+`sqlite3`, and `fuzzcheck`.
+
+<a name="runtr"></a>
+## 1.1 Running testrunner.tcl
+
+The testrunner.tcl script is located in the "test" subdirectory of the
+SQLite source tree. So if your shell is current positioned at the top
+of the source tree, you would normally run the script using the command:
+"<tt>test/testrunner.tcl</tt>". On Windows, you have to specify the
+`tclsh` interpreter command first, like this:
+"<tt>tclsh test/testrunner.tcl</tt>".
- * Tcl test scripts.
+In this document, we will assume that you are on a unix-like OS
+(not on Windows) and that your current directory is the root
+of the SQLite source tree, and so all invocations of the testrunner.tcl
+script will be of the form "<tt>test/testrunner.tcl</tt>". If you
+are in a different directory, then make appropriate adjustments to
+the path. On Windows, add the "<tt>tclsh</tt>" interpreter command
+up front.
- * Fuzzcheck tests, including using an external fuzzcheck database.
+<a name="runviamake"></a>
+## 1.2 Run using make
- * Tests run with `make` commands. Examples:
- - `make devtest`
- - `make releasetest`
- - `make sdevtest`
- - `make testrunner`
+The standard Makefiles for SQLite include targets that invoke
+testrunner.tcl. So the following commands also run testrunner.tcl:
+
+ * `make devtest`
+ * `make releasetest`
+ * `make sdevtest`
+ * `make testrunner`
+
+<a name="outputs"></a>
+## 1.3 Outputs from testrunner.tcl
The testrunner.tcl program stores output of all tests and builds run in
log file **testrunner.log**, created in the current working directory.
```
You can get a summary of errors in a prior run by invoking commands like
-these:
+those shown below. Note that the testrunner.tcl script can be run directly
+on unix systems (including Macs) but you will need to add <tt>tclsh</tt>
+to the front on Windows.
```
- tclsh $(TESTDIR)/testrunner.tcl errors
- tclsh $(TESTDIR)/testrunner.tcl errors -v
+ test/testrunner.tcl errors
+ test/testrunner.tcl errors -v
```
Running the command:
```
- tclsh $(TESTDIR)/testrunner.tcl status
+ test/testrunner.tcl status
```
in the directory containing the testrunner.db database runs various queries
following commands:
```
- watch tclsh $(TESTDIR)/testrunner.tcl status
- tclsh $(TESTDIR)/testrunner.tcl status -d 2
+ watch test/testrunner.tcl status
+ test/testrunner.tcl status -d 2
```
Both of the commands above accomplish about the same thing, but the second
(see "Binary Tests" below). Sometimes it builds testfixture and
other binaries in specific configurations to test (see "Source Tests").
+<a name=help></a>
+## 1.4 Built-in help
+
+Run this command:
+
+```
+ test/testrunner.tcl help
+```
+
+To get a summary of all of the various command-line options available
+with testrunner.tcl
+
+
<a name=binary_tests></a>
# 2. Binary Tests
binary, depending on the test) that is found in the directory from which
testrunner.tcl is launched. So typically, one must first run something
like "`make testfixture sqlite3`" before launching binary tests. In other
-words, testrunner.tcl does not build the binaries under test itself; it
-expects them to be available already.
+words, testrunner.tcl does not automatically build the binaries under test
+for binary tests. The testrunner.tcl expects the binaries to be available
+already.
The following sub-sections describe the various options that can be
passed to testrunner.tcl to test binary testfixture builds.
To run the "veryquick" test set, use either of the following:
```
- tclsh $TESTDIR/testrunner.tcl
- tclsh $TESTDIR/testrunner.tcl veryquick
+ test/testrunner.tcl
+ test/testrunner.tcl veryquick
```
To run the "full" test suite:
```
- tclsh $TESTDIR/testrunner.tcl full
+ test/testrunner.tcl full
```
To run the subset of the "full" test suite for which the test file name matches
a specified pattern (e.g. all tests that start with "fts5"), either of:
```
- tclsh $TESTDIR/testrunner.tcl fts5%
- tclsh $TESTDIR/testrunner.tcl 'fts5*'
+ test/testrunner.tcl fts5%
+ test/testrunner.tcl 'fts5*'
```
Strictly speaking, for a test to be run the pattern must match the script
To run "all" tests (full + permutations):
```
- tclsh $TESTDIR/testrunner.tcl all
+ test/testrunner.tcl all
```
-Note that in all of the previous examples, when running on Unix, the
-initial "<tt>tclsh</tt>" on the command-line may be omitted. The
-testrunner.tcl is marked as executable and begins with an appropriate
-"<tt>#!</tt>" line to invoke tclsh automatically.
-
<a name=binary_test_failures></a>
## 2.3. Investigating Binary Test Failures
If there is no permutation, the individual test script may be run with:
```
- tclsh $PATH_TO_SCRIPT
+ ./testfixture $PATH_TO_SCRIPT
```
Or, if the failure occured as part of a permutation:
```
- tclsh $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
+ test/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
```
-One can also run all tests that failed in the previous invocation by
-typing:
+One can also rerun all tests that failed or did not complete
+in the previous invocation by typing:
```
- tclsh $TESTDIR/testrunner.tcl retest
+ test/testrunner.tcl retest
```
<a name=source_code_tests></a>
with debugging enabled and one without:
```
- tclsh $TESTDIR/testrunner.tcl mdevtest
+ test/testrunner.tcl mdevtest
```
In other words, it is equivalent to running:
$TOP/configure --enable-all --enable-debug
make fuzztest
make testfixture
- tclsh $TOP/test/testrunner.tcl veryquick
+ $TOP/test/testrunner.tcl veryquick
# Then, after removing files created by the tests above:
$TOP/configure --enable-all OPTS="-O0"
make fuzztest
make testfixture
- tclsh $TOP/test/testrunner.tcl veryquick
+ $TOP/test/testrunner.tcl veryquick
```
The **sdevtest** command is identical to the mdevtest command, except that the
OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":
```
- tclsh $TESTDIR/testrunner.tcl sdevtest
+ test/testrunner.tcl sdevtest
```
The **release** command runs lots of tests under lots of builds. It runs
of the specific tests run.
```
- tclsh $TESTDIR/testrunner.tcl release
+ test/testrunner.tcl release
```
As with <a href=#source code tests>source code tests</a>, one or more patterns
Pattern matching is used for both Tcl tests and fuzz tests.
```
- tclsh $TESTDIR/testrunner.tcl release rtree%
+ test/testrunner.tcl release rtree%
```
<a name=zipvfs_tests></a>
tests from the Zipvfs project with the following command:
```
- tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
+ test/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
```
This can be combined with any of "mdevtest", "sdevtest" or "release" to
test both SQLite and Zipvfs with a single command:
```
- tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
+ test/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
```
<a name=source_code_test_failures></a>
```
# Create a script that recreates build configuration "Device-One" on
# Linux or OSX:
- tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh
+ test/testrunner.tcl script Device-One > make.sh
# Create a script that recreates build configuration "Have-Not" on Windows:
- tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat
+ test/testrunner.tcl script Have-Not > make.bat
```
The generated bash or \*.bat file script accepts a single argument - a makefile
dbsqlfuzz database, run a command like one of these:
```
- tclsh test/testrunner.tcl releasetest --fuzzdb ../fuzz/20250415.db
+ test/testrunner.tcl releasetest --fuzzdb ../fuzz/20250415.db
FUZZDB=../fuzz/20250415.db make releasetest
nmake /f Makefile.msc FUZZDB=../fuzz/20250415.db releasetest
```
database, you can use a command something like this:
```
- tclsh test/testrunner.tcl releasetest 20250415 --fuzzdb ../fuzz/20250415.db
+ test/testrunner.tcl releasetest 20250415 --fuzzdb ../fuzz/20250415.db
```
<a name=testrunner_options></a>
```
# Build binaries required by release test.
- tclsh $TESTDIR/testrunner.tcl --buildonly release"
+ test/testrunner.tcl --buildonly release"
```
The **--dryrun** option prevents testrunner.tcl from building any binaries
```
# Log the shell commmands that make up the mdevtest test.
- tclsh $TESTDIR/testrunner.tcl --dryrun mdevtest"
+ test/testrunner.tcl --dryrun mdevtest"
```
The **--explain** option is similar to --dryrun in that it prevents
```
# Show what builds and tests would have been run
- tclsh $TESTDIR/testrunner.tcl --explain mdevtest
+ test/testrunner.tcl --explain mdevtest
```
The **--status** option uses VT100 escape sequences to display the test
number of jobs it intends to use to stdout. e.g.
```
- $ tclsh $TESTDIR/testrunner.tcl
+ $ test/testrunner.tcl
splitting work across 16 jobs
... more output ...
```
switch:
```
- $ tclsh $TESTDIR/testrunner.tcl --jobs 8
+ $ test/testrunner.tcl --jobs 8
splitting work across 8 jobs
... more output ...
```
testrunner.log and testrunner.db files:
```
- $ tclsh $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
+ $ test/testrunner.tcl njob $NEW_NUMBER_OF_JOBS
```
projects / thirdparty / sqlite.git / commitdiff
