Small. Fast. Reliable.
Choose any three.
About Documentation Download License Support Purchase

TH3: SQLite Test Harness #3

1.0 Overview

SQLite Test Harness #3 (hereafter "TH3") is one of three test harnesses used for testing SQLite. TH3 is designed to meet the following objectives:

TH3 was originally written for validation testing only, but has subsequently been used for development testing and debugging as well, and has proven very helpful in those roles. A full-coverage test takes less than five minutes on a workstation and hence serves as a fast regression test during day-to-day maintenance of the SQLite code base.

2.0 Operation

TH3 is a test program generator. The output of TH3 is a program implemented in C-code and intended to be linked against the SQLite library under test. The generated test program is compiled and run on the target platform in order to verify correct operation of SQLite on that platform.

The inputs to TH3 are test modules written in C or SQL and small configuration files that determine how to initialize SQLite. The TH3 package includes over eleven hundred test modules and more than three dozen configuration files. New modules and configurations can be added to customize TH3 for specialized applications. Each time TH3 is run, it reads a subset of the available test modules and configuration files to generate a custom C program that performs all of the specified tests under all specified configurations. A complete test of SQLite normally involves running TH3 multiple times to generate multiple test programs covering different aspects of SQLite's operation, then linking all test programs against a common SQLite library and running them separately on the target platform. SQLite will be found to work if all test programs pass.

There are no arbitrary limits in TH3. One could generate a single test program that contained all test modules and all configuration files. However, such a test program might be too large to deploy on embedded platforms. Hence, TH3 provides the ability to break the library of test modules up into smaller, more easily digested pieces.

Each individual test module might contain dozens, hundreds, or thousands of separate tests. The test modules can be written in C or as scripts of SQL or a mixture of the two. About two-thirds of the existing test modules are written SQL with the remainder either in pure C or a combination of C and SQL.

Each test module file contains a header which describes the circumstances under which the test is valid. For a particular configuration, only those modules that are compatible with the configuration are run.

3.0 Generating A Test Program

The TH3 program generator is a TCL script named "mkth3.tcl". To generate a test program, one has merely to run this script and supply the names of files containing test modules and configurations on the command line. Test modules are files that use the ".test" suffix and configurations are files that use the ".cfg" suffix. A typical invocation of mkth3.tcl might look something like the following:

tclsh mkth3.tcl *.test *.cfg >testprog1.c

The output from the mkth3.tcl script is a C program that contains everything needed to run the tests - everything that is except for the SQLite library itself. The generated test program contains implementations for all of the support interfaces used by the test modules and it contains the main() routine that drives the tests. To convert the test program into a working executable, simply compile it against SQLite:

cc -o testprog1 testprog1.c sqlite3.c

The compilation step shown immediately above is merely representative. In a working installation, one would normally want to specify optimization parameters and compile-time switches on the compiler command line.

For testing on embedded systems, the mkth3.tcl script and the compiler steps shown above are performed on an ordinary workstation using a cross-compiler, then the resulting test program is transfer onto the device to be run.

Once the test program is generated, it is run with no arguments to perform the tests. Progress information as well as error diagnostics appear on standard output. (Alternative output arrangements can be made using a compile-time option for embedded devices that lack a standard output channel.) The program returns zero if there are no errors and non-zero if any problems were detected.

Typical output from a single TH3 test program run looks like this: 

With SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e
-DSQLITE_COVERAGE_TEST
-DSQLITE_NO_SYNC
-DSQLITE_SYSTEM_MALLOC
-DSQLITE_THREADSAFE=1
Config-begin c1.
Begin c1.pager08
End c1.pager08
Begin c1.build33
End c1.build33
Begin c1.orderby01
End c1.orderby01
... 15014 lines of output omitted ....
Begin 64k.syscall01
End 64k.syscall01
Begin 64k.build01
End 64k.build01
Begin 64k.auth01
End 64k.auth01
Config-end 64k. TH3 memory used: 6373738
Config-begin wal1.
Begin wal1.wal37
End wal1.wal37
Config-end wal1. TH3 memory used: 100961
All 226 VDBE coverage points reached
th3: 0 errors out of 1442264 tests in 213.741 seconds. 64-bit little-endian
th3: SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e

The output begins with a report of the SQLITE_SOURCE_ID (cross-checked again sqlite3_sourceid()) for the SQLite under test and the compile-time options used as reported by sqlite3_compileoption_get(). The output concludes with a summary of the test results and a repeat of the SQLITE_SOURCE_ID. If any errors are detected, additional lines detail the problem. The error reporting lines always begin with a single space character so that they can be quickly extracted from large output files using: 

grep "^ "

The default output shows the beginning and end of each configuration and test module combination. In the example above "c1" and "64k" are configurations and "pager08", "build33", "orderby01", etc. are test modules. Compile-time and run-time options are available to increase or decrease the amount of output. The output can be increased by showing each test case within each test module. The output can be decreased by degrees: omitting test modules starts and stops, omitting configuration starts and stops, and finally by omitting all output.

3.1 Test Automation Scripts

TH3 comes with additional TCL scripts that help automate the testing process on workstations. The "th3make" script automatically runs "mkth3.tcl" and "gcc" and then runs the resulting test program and checks the results. Arguments to th3make include all of the "*.test" test modules and "*.cfg" configurations that are to be included in the test. Additional options to th3make can cause the test program to be compiled using different compilers (GCC, Clang, MSVC), to use different output verbosity levels, to run the test program under valgrind, to check the output for coverage using gcov, and so forth. The th3make script also accepts "*.rc" filenames as arguments. These *.rc files are just collections of other arguments that are commonly used together for a single purpose. For example, the "quick.rc" file contains a set of eight arguments to th3make that run a fast (3-minute) full-coverage test. This allows the operator to type "./th3make quick.rc" as a short-cut to typing out all of the required command-line options. The following are a few of the more than 40 available *.rc files:

The TH3 repository also includes the "multitest.tcl" script, another TCL script used to automate TH3 testing on workstations. Multitest.tcl automatically compiles SQLite, then runs ./th3make repeatedly with a variety of alignments, and captures the output in a succinct summary screen. A typical multitest.tcl run generates output that looks like this: 

file mkdir sqlite3bld
cd sqlite3bld
exec sh /home/drh/sqlite/sqlite/configure
file copy -force config.h ../config.h
exec make clean sqlite3.c
file rename sqlite3.c ../sqlite3.c
aa4f0f90c9c77424943e026a2ecee4a6c7f9e0d3  ../sqlite3.c
file rename sqlite3.h ../sqlite3.h
exec make clean sqlite3.c OPTS=-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1
file rename sqlite3.c ../sqlite3udl.c
0d3bbc92c433f940253bb2c7c19de7783133929d  ../sqlite3udl.c
exec make clean sqlite3.c OPTS=-DSQLITE_SMALL_STACK=1
file rename sqlite3.c ../sqlite3ss.c
fcf6963e94096324461076d3b9e9dc1888e066e1  ../sqlite3ss.c
cd ..
*******************************************************************************
t01: quick.rc.................................................. Ok   (00:04:00)
t02: cov.rc.................................................... Ok   (00:04:40)
t03: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok   (00:05:22)
t04: cov.rc -DSQLITE_ENABLE_STAT4 -D_HAVE_SQLITE_CONFIG_H...... Ok   (00:05:20)
t05: test.rc ../th3private/*.test.............................. Ok   (00:00:17)
t06: test.rc ../th3private/*.test -DSQLITE_ENABLE_STAT4........ Ok   (00:00:43)
t07: quick.rc -DSQLITE_TEST_REALLOC_STRESS -funsigned-char..... Ok   (00:04:56)
t08: quick.rc -DSQLITE_THREADSAFE=0 -fsigned-char.............. Ok   (00:03:12)
t09: quick.rc sqlite3ss.c -DSQLITE_MAX_ATTACHED=125............ Ok   (00:04:04)
t10: quick.rc -DSQLITE_RUNTIME_BYTEORDER....................... Ok   (00:03:58)
t11: quick.rc -DSQLITE_DIRECT_OVERFLOW_READ.................... Ok   (00:04:01)
t12: fast.rc................................................... Ok   (00:14:19)
t13: fast.rc alignment1.rc -m32................................ Ok   (00:20:51)
t14: fast.rc alignment2.rc sqlite3udl.c........................ Ok   (00:16:06)
t15: fast.rc alignment4.rc..................................... Ok   (00:12:55)
t16: fast.rc alignment5.rc..................................... Ok   (00:14:58)
t17: fast.rc alignment6.rc..................................... Ok   (00:14:31)
t18: fast.rc alignment7.rc..................................... Ok   (00:16:06)
t19: fast.rc alignment8.rc sqlite3udl.c........................ Ok   (00:24:09)
t20: test.rc alignment1.rc..................................... Ok   (00:49:27)
t21: test.rc alignment2.rc sqlite3udl.c........................ Ok   (00:38:43)
t22: test.rc alignment4.rc -m32 CC=clang....................... Ok   (00:39:49)
t23: test.rc alignment5.rc..................................... Ok   (00:36:33)
t24: test.rc alignment6.rc..................................... Ok   (00:33:53)
t25: test.rc alignment7.rc..................................... Ok   (00:42:16)
t26: test.rc alignment8.rc sqlite3udl.c........................ Ok   (01:05:22)
t27: memdebug.rc extensions.rc................................. Ok   (01:35:56)
t28: fast.rc -fsanitize=undefined.............................. Ok   (00:15:09)
t29: min.rc -O3 -valgrind...................................... Ok   (01:26:10)
t30: min.rc -O3 -valgrind extensions.rc........................ Ok   (01:47:12)
t31: test-ex.rc................................................ Ok   (03:20:18)
*******************************************************************************
0 failures on 31 th3make runs and 166721387 tests in (16:25:29)
SQLite 3.8.10 2015-05-05 18:52:54 04afa3febee32854fbb09ef8d4ffffd432119716

As can be seen above, a single run of multitest.tcl invokes th3make dozens times and takes between 12 and 24 hours. The middle section of the output shows the arguments to each individual th3make run and the result and elapse time for that th3make. All build products and output for the separate th3make runs are captures in subdirectories for post-test analysis. The two-line summary at the bottom shows the total number of errors and tests over all th3make runs and the total elapse time, together with the SQLITE_SOURCE_ID information for the version of SQLite that was tested. This summary information is recorded in the release checklist during testing.

4.0 Test Coverage

Using one particular subset of the available TH3 test modules (the "cov1" tests) SQLite obtained 100% branch test coverage and 100% MC/DC as measured by gcov on Linux x86 and x86_64 hardware. All releases of SQLite since version 3.6.17 (circa 2009-08-10) have been tested to this standard. The SQLite developers are committed to maintaining 100% branch coverage and MC/DC for all future releases of SQLite.

The cov1 test set used to obtain 100% branch test coverage are only a subset of the tests currently implemented using TH3. New test modules are added on a regular basis.

5.0 TH3 License

SQLite itself is in the public domain and can be used for any purpose. But TH3 is proprietary and requires a license.

Even though open-source users do not have direct access to TH3, all users of SQLite benefit from TH3 indirectly since each version of SQLite is validated running TH3 on multiple platforms (Linux, Windows, WinRT, Mac, OpenBSD, Solaris Sparc) prior to release. So anyone using an official release of SQLite can deploy their application with the confidence of knowing that it has been tested using TH3. They simply cannot rerun those tests themselves without purchasing a TH3 license.