rackcheck: property testing
Rackcheck is a property-based testing library for Racket with support for shrinking.
I am still in the process of experimenting with the implementation of this library so things may change without notice at this point.
1 What about quickcheck?
I initially started by forking the quickcheck library to add support for shrinking, but found that I would have to make many breaking changes to get shrinking to work the way I wanted so I decided to start from scratch instead.
2 Reference
(require rackcheck) | package: rackcheck |
2.1 Generators
Generators produce arbitrary values based upon certain constraints. The generators provided by this library can be mixed and matched in order to produce complex values suitable for any domain.
By convention, all generators and combinators are prefixed with gen:.
2.1.1 Debugging
The following functions come in handy when debugging generators. Don’t use them to produce values for your tests.
procedure
g : gen? n : exact-positive-integer?
rng : pseudo-random-generator? = (current-pseudo-random-generator)
procedure
(shrink g size [rng]) →
any/c (listof any/c) g : gen? size : exact-nonnegative-integer?
rng : pseudo-random-generator? = (current-pseudo-random-generator)
2.1.2 Core Combinators
In general, you won’t have to write generator functions yourself. Instead, you’ll use the generators and combinators provided by this library to generate values for your domain. That said, there may be cases where you want to tightly control how values are generated or shrunk and that’s when you might reach for a custom generator.
When shrinking, a new shrink sequence is produced for every application of f to all of g’s shrinks.
> (define gen:list-of-trues (gen:bind gen:natural (lambda (len) (make-gen (lambda (rng size) (stream (make-list len #t)))))))
> (sample gen:list-of-trues 5) '(() () (#t #t #t #t) (#t #t #t #t #t) (#t #t #t #t #t #t #t #t))
> (shrink gen:list-of-trues 5)
'(#t)
'(())
procedure
(gen:filter g p [max-attempts]) → gen?
g : gen? p : (-> any/c boolean?) max-attempts : (or/c exact-positive-integer? +inf.0) = 1000
An exception is raised when the generator runs out of attempts.
This is a very brute-force way of generating values and you should avoid using it as much as possible, especially if the range of outputs is very small compared to the domain. Take a generator for non-empty strings as an example. Instead of:
> (gen:filter (gen:string gen:char-alphanumeric) (lambda (s) (not (string=? s "")))) #<procedure:...eck/gen/core.rkt:78:3>
Write:
> (gen:let ([hd gen:char-alphanumeric] [tl (gen:string gen:char-alphanumeric)]) (string-append (string hd) tl)) #<procedure:...eck/gen/core.rkt:67:3>
The latter takes a little more effort to write, but it doesn’t depend on the whims of the random number generator and will always generate a non-empty string on the first try.
procedure
(gen:choice g ...+) → gen?
g : gen?
procedure
(gen:no-shrink g) → gen?
g : gen?
syntax
(gen:let ([id gen-expr] ...+) body ...+)
> (define gen:list-of-trues-2 (gen:let ([len gen:natural]) (make-list len #t)))
> (sample gen:list-of-trues-2 5) '(() () (#t #t #t #t) (#t #t #t #t #t) (#t #t #t #t #t #t #t #t))
> (shrink gen:list-of-trues-2 5)
'(#t)
'(())
syntax
(gen:delay gen-expr)
2.1.3 Basic Generators
value
> (sample gen:natural) '(0 0 4 5 8 20 22 39 43 20)
> (shrink gen:natural)
9
'(4 2 1 0)
procedure
(gen:integer-in lo hi) → gen?
lo : exact-integer? hi : exact-integer?
> (sample (gen:integer-in 1 255)) '(75 46 239 152 124 200 155 202 172 65)
> (shrink (gen:integer-in 1 255))
75
'(37 18 9 4 2 1)
> (sample gen:real)
'(0.2904158091187683
0.17902984405826025
0.9348212358175817
0.592848361775386
0.4846099332903666
0.7816821100632378
0.6078124617750272
0.788902313469835
0.6710271948421507
0.25158978983077135)
procedure
(gen:one-of xs) → gen?
xs : (non-empty-listof any/c)
> (define gen:letters (gen:one-of '(a b c)))
> (sample gen:letters) '(c a b a c a c c b b)
> (shrink gen:letters)
'c
'()
value
> (sample gen:boolean) '(#f #f #t #t #f #t #t #t #t #f)
> (shrink gen:boolean)
#f
'()
value
> (sample gen:char-letter) '(#\e #\P #\u #\U #\G #\O #\g #\E #\u #\o)
> (shrink gen:char-letter)
#\e
'()
value
> (sample gen:char-digit) '(#\2 #\1 #\9 #\5 #\4 #\7 #\6 #\7 #\6 #\2)
> (shrink gen:char-digit)
#\2
'()
value
> (sample gen:char-alphanumeric) '(#\1 #\M #\U #\q #\g #\d #\o #\H #\2 #\7)
> (shrink gen:char-alphanumeric)
#\1
'()
> (sample (gen:tuple gen:natural gen:boolean)) '((0 #f) (1 #t) (2 #t) (6 #t) (11 #f) (16 #t) (9 #f) (46 #f) (8 #t) (38 #t))
> (shrink (gen:tuple gen:natural gen:boolean))
'(9 #f)
'()
procedure
g : gen? max-len : exact-nonnegative-integer? = 128
> (sample (gen:list gen:natural) 5) '(() () (2 2 3 3) (6 2 6 5 2 2 9) (2 13))
> (shrink (gen:list gen:natural))
'(5 28 18 15 24 18 24 20 7)
'((5 28 18 15 24 18 24 20)
(5 28 18 15 24 18 24)
(5 28 18 15 24 18)
(5 28 18 15 24)
(5 28 18 15)
(5 28 18)
(5 28)
(5)
())
procedure
(gen:vector g [#:max-length max-len]) → gen?
g : gen? max-len : exact-nonnegative-integer? = 128
> (sample (gen:vector gen:natural) 5) '(#() #() #(2 2 3 3) #(6 2 6 5 2 2 9) #(2 13))
> (shrink (gen:vector gen:natural))
'#(5 28 18 15 24 18 24 20 7)
'(#(5 28 18 15 24 18 24 20)
#(5 28 18 15 24 18 24)
#(5 28 18 15 24 18)
#(5 28 18 15 24)
#(5 28 18 15)
#(5 28 18)
#(5 28)
#(5)
#())
procedure
g : gen? = (gen:integer-in 0 255) max-len : exact-nonnegative-integer? = 128
> (shrink (gen:bytes))
#"-\357\227|\310\233\311\253@"
'(#"-\357\227|\310\233\311\253"
#"-\357\227|\310\233\311"
#"-\357\227|\310\233"
#"-\357\227|\310"
#"-\357\227|"
#"-\357\227"
#"-\357"
#"-"
#"")
procedure
(gen:string [g #:max-length max-len]) → gen?
g : gen? = gen:char max-len : exact-nonnegative-integer? = 128
> (sample (gen:string gen:char-letter) 5) '("" "" "MPRq" "gEuoX" "fuee")
> (shrink (gen:string gen:char-letter))
"yMPRqGydM"
'("yMPRqGyd" "yMPRqGy" "yMPRqG" "yMPRq" "yMPR" "yMP" "yM" "y" "")
procedure
(gen:symbol [g #:max-length max-len]) → gen?
g : gen? = gen:char max-len : exact-nonnegative-integer? = 128
> (sample (gen:symbol gen:char-letter) 5) '(|| || MPRq gEuoX fuee)
> (shrink (gen:symbol gen:char-letter))
'yMPRqGydM
'(yMPRqGyd yMPRqGy yMPRqG yMPRq yMPR yMP yM y ||)
procedure
k : any/c g : gen?
procedure
(gen:hasheq k g ...+ ...+) → gen?
k : any/c g : gen?
procedure
(gen:hasheqv k g ...+ ...+) → gen?
k : any/c g : gen?
> (sample (gen:hasheq 'a gen:natural 'b (gen:string gen:char-letter)) 5)
'(#hasheq((a . 0) (b . ""))
#hasheq((a . 1) (b . "u"))
#hasheq((a . 3) (b . "GOg"))
#hasheq((a . 9) (b . "u"))
#hasheq((a . 7) (b . "XafFfaeLg")))
> (shrink (gen:hasheq 'a gen:natural 'b (gen:string gen:char-letter)))
'#hasheq((a . 9) (b . "PuUGO"))
'(#hasheq((a . 4) (b . "PuUGO"))
#hasheq((a . 2) (b . "PuUGO"))
#hasheq((a . 1) (b . "PuUGO"))
#hasheq((a . 0) (b . "PuUGO"))
#hasheq((a . 9) (b . "PuUG"))
#hasheq((a . 4) (b . "PuUG"))
#hasheq((a . 2) (b . "PuUG"))
#hasheq((a . 1) (b . "PuUG"))
#hasheq((a . 0) (b . "PuUG"))
#hasheq((a . 9) (b . "PuU"))
#hasheq((a . 4) (b . "PuU"))
#hasheq((a . 2) (b . "PuU"))
#hasheq((a . 1) (b . "PuU"))
#hasheq((a . 0) (b . "PuU"))
#hasheq((a . 9) (b . "Pu"))
#hasheq((a . 4) (b . "Pu"))
#hasheq((a . 2) (b . "Pu"))
#hasheq((a . 1) (b . "Pu"))
#hasheq((a . 0) (b . "Pu"))
#hasheq((a . 9) (b . "P"))
#hasheq((a . 4) (b . "P"))
#hasheq((a . 2) (b . "P"))
#hasheq((a . 1) (b . "P"))
#hasheq((a . 0) (b . "P"))
#hasheq((a . 9) (b . ""))
#hasheq((a . 4) (b . ""))
#hasheq((a . 2) (b . ""))
#hasheq((a . 1) (b . ""))
#hasheq((a . 0) (b . "")))
procedure
(gen:frequency frequencies) → gen?
frequencies : (non-empty-listof (cons/c exact-positive-integer? gen?))
> (sample (gen:frequency `((5 . ,gen:natural) (2 . ,gen:boolean)))) '(0 #t 3 7 4 14 9 #f 51 46)
> (shrink (gen:frequency `((5 . ,gen:natural) (2 . ,gen:boolean))))
5
'(2 1 0)
2.1.4 Unicode Generators
(require rackcheck/gen/unicode) | package: rackcheck |
value
value
value
value
value
value
value
> (sample gen:unicode)
'(#\U0003C314
#\耎
#\U000D7AFB
#\ꩧ
#\㙗
#\登
#\U00050685
#\⏲
#\U000DA289
#\U000A1C2C)
> (sample gen:unicode-letter) '(#\醃 #\𨲠 #\𐍙 #\ꡩ #\𐙕 #\啖 #\項 #\𥌽 #\𥐖 #\𧻹)
> (sample gen:unicode-mark) '(#\𑂀 #\᪶ #\︢ #\󠅼 #\󠅕 #\⃮ #\ौ #\͗ #\󠄕 #\َ)
> (sample gen:unicode-number) '(#\𐋤 #\㊼ #\፪ #\𐮬 #\𞣊 #\꠰ #\🄉 #\𐋧 #\𖭓 #\𑁝)
> (sample gen:unicode-punctuation) '(#\᪠ #\⸿ #\﹍ #\⸣ #\𐫶 #\⦆ #\⸓ #\* #\\ #\𑇍)
> (sample gen:unicode-symbol) '(#\⣢ #\🢃 #\𝄯 #\🝛 #\♹ #\🌸 #\🝖 #\🎀 #\㌿ #\🐎)
> (sample gen:unicode-separator)
'(#\u202F
#\u2001
#\u202F
#\u2000
#\u2002
#\u2008
#\u2028
#\u2007
#\u00A0
#\u2004)
2.2 Properties
syntax
(property ([id gen-expr] ...) body ...+)
> (property ([xs (gen:list gen:natural)]) (check-equal? (reverse (reverse xs)) xs)) #<prop>
syntax
(define-property name ([id gen-expr] ...) body ...+)
syntax
(check-property maybe-config prop-expr)
maybe-config =
| config-expr
> (check-property (property ([xs (gen:list gen:natural)]) (check-equal? (reverse (reverse xs)) xs))) ✓ property unnamed passed 100 tests.
> (check-property (property ([xs (gen:list gen:natural)]) (check-equal? (reverse xs) xs)))
--------------------
FAILURE
location: eval:54:0
name: unnamed
seed: 763024896
actual: '(2 5)
expected: '(5 2)
Failed after 5 tests:
xs = (5 2 1 6 9 1 6 4)
Shrunk:
xs = (5 2)
--------------------
Does nothing when s is #f.
> (check-property (property ([a gen:natural] [b gen:natural]) (label! (case a [(0) "zero"] [else "non-zero"])) (+ a b)))
✓ property unnamed passed 100 tests.
Labels:
├ 98.00% non-zero
└ 2.00% zero
procedure
(make-config [ #:seed seed #:tests tests #:size size #:deadline deadline]) → config? seed : (integer-in 0 (sub1 (expt 2 31))) = ... tests : exact-positive-integer? = 100
size : (-> exact-positive-integer? exact-nonnegative-integer?) = (lambda (n) (expt (sub1 n) 2))
deadline : (>=/c 0) = (+ (current-inexact-milliseconds) (* 60 1000))