Quickcheck

This manual provides documentation for a Racket implementation of Quickcheck, a library that tests the specification of a program with randomly generated test inputs.

The original Quickcheck is a library for Haskell.

1 Quickstart

1.1 Installation

If you have not installed the package yet, run the following command from the command-line:

  raco pkg install quickcheck

Note that this package will require Racket v6.0 or newer to install.

Alternatively, you may use the GUI package manager from DrRacket to install the package.

1.2 Running tests

To use Quickcheck, you first write a specification for a program as a property in ordinary Racket code. For example, the following is a property that expresses the English specification “the string->number function produces numbers when given strings”:

> (define string->number-returns-number     (property ([str arbitrary-string])       (number? (string->number str))))

Given such a property, we can run random tests using quickcheck:

> (quickcheck string->number-returns-number)

Falsifiable, after 0 tests: str = "\u0002\u0003\u0002"

You may have already guessed that this property will be easily falsified, since most strings do not actually parse as numbers.

Next, let’s write a property that we expect will actually succeed. For example, the following encodes the English specification “given two lists of integers, the result of appending them will have the same length as the sum of the original list lengths”:

> (define append-and-length-agree     (property ([lst-1 (arbitrary-list arbitrary-integer)]                [lst-2 (arbitrary-list arbitrary-integer)])       (= (+ (length lst-1) (length lst-2))          (length (append lst-1 lst-2)))))

Testing the property reveals that it holds up:

> (quickcheck append-and-length-agree)

OK, passed 100 tests.

1.3 Creating new generators

Oops, there is a bug in the distance function. If you fix it, all the checks should pass.

2 Running checks

Prints the result of the check, including a counterexample for the property if one was found.

• an integer representing the number of tests run,

• a list of ...,

• either #t if the tests were successful, #f if the arguments were exhausted, or an instance of the result structure type on a test failure.

Prints the result of the check, including a counterexample for the property if one was found.

3 Configuration, results, and utilities

The max-test field represents the maximum number of succeeding tests that will be run. The max-fail field represents the maximum number of tests that can be run with no result before the checker terminates.

The size field should be a function of one argument that produces the test size given the current test number. The print-every field should be a function that takes the test number and the generated arguments and is called for its side effect.

The quickcheck and quickcheck-results functions use a default config where the test count is 100, the test size for test n is (+ 3 (quotient n 2)), the max fail count is ten times the test count, and nothing is printed. The default config can be adjusted to run different numbers of tests with with-small-test-count, with-medium-test-count, and with-large-test-count.

The stamp field represents the labels that were relevant to this test execution. The arguments-list is a list of the values generated for checking this test case.

The ids are bound to the result of the given gen/arb-exprs inside the body expressions. The body expressions are used as the bodies of a predicate function that will be run with newly generated values from the specified generators or arbitraries.

Values that can be tested are the following:

• boolean values (#t and #f),

• instances of the result structure type,

• instances of the property structure type,

• or instances of the generator structure type.

4 Integration with RackUnit

By default quickcheck simply displays whether the property check results in success or, in case of failure, it prints the arguments that falsify the property along with some other metadata. This is not helpful for (semi-) automatic testing as it is done using RackUnit. Fortunately it is possible to combine the best of both worlds.

For example, let us check the previous string->number-returns-number property:

> (check-property string->number-returns-number)

-------------------- FAILURE name:       check-property location:   eval:20:0 params:     '(#<property>) ntest:      0 stamps:     () arguments:  "str = \"\\u0002\\u0003\\u0002\"" Falsifiable --------------------

The purpose of this form is to improve error messages when a property fails. For instance, to compare whether two functions are observationally equivalent for arbitrary arguments, we can define the following form using add-property-check-info to state the expected and actual values inside the declaration of a property:

(define-syntax-rule (conforms-to f g ([name gen] ...))

(property ([name gen] ...)

(let ([expected (f name ...)]

[actual (g name ...)])

(add-property-check-info (['expected expected]

['actual actual]))

(equal? expected actual))))

(define (f n) (+ n 1))

(define (g n) (- n 1))

> (check-property (conforms-to f g ([n arbitrary-integer])))

-------------------- FAILURE name:       check-property location:   eval:24:0 expected:   1 actual:     -1 ntest:      0 stamps:     () arguments:  "n = 0" Falsifiable --------------------

Crucially, every time the property is tested with a different set of arguments add-property-check-info overrides the previous data.

5 Generators

The proc value should be a function that accepts an integer representing the size of the value, a random number generator, and returns a value for testing.

If val-expr produces a generator?, the remaining code is threaded through generator-bind, and id will be bound to the value produced by the generator. Note that this delays evaluation until the generator is asked to produce a value.

(struct foo (a b) #:transparent)

; choose-foo creates a generator that produces a foo such that

;  foo-a is an integer?

;  foo-b is (or/c foo? #f)

(define (choose-foo [recurse-limit 3])

(bind-generators

([rand (choose-integer 0 1)]

[recurse? (and (positive? recurse-limit)

(= 0 rand))]

[a (choose-integer 0 9)]

[b (if recurse?

(choose-foo (sub1 recurse-limit))

#f)])

(foo a b)))

6 Operations on properties

If bool-expr is #t, equivalent to prop. Otherwise, produces a property that returns no result.

7 Random number generation