GTP Checkup

Source: https://github.com/bennn/gtp-checkup

Contains a few configurations of the GTP benchmark programs, code for continuously testing their performance as Racket changes, and data from past runs.

1 Basic Usage

To run a performance test on your machine:

• Clone the repo

  • Either via git clone, or better yet via raco pkg install --clone gtp-checkup

• Run make

• See results on STDOUT, with a summary of errors at the bottom.

The Makefile compiles and runs all scripts that match the pattern benchmarks/*/main.rkt. Each compile job and each run job has a time limit. Run racket main.rkt --help for more information.

2 Version History

• Changed in version 0.1: Changed style of benchmarks to focus on one worst-case configuration instead of spot-checking N mixed ones.

• Changed in version 0.1: Renamed quadBG to quadU and replaced quadMB with quadT. The version notes in the GTP Benchmarks documentation explain why. Updated other benchmarks to match the GTP benchmarks versions.

3 Checkup API

The value of i determines the number of times to run each benchmark configuration.

The value of time-limit determines how long to wait for each configuration to compile and run. If time-limit is a cons-pair, then its car is the compile-time limit and its cdr is the run-time limit.

4 Checkup data for racket/racket

The plots in this section show the performance of different snapshots of the Racket language. A snapshot begins with one commit to the racket/racket repository and includes contemporaneous commits to other main distribution repositories.

Quick guide to plots:

• x-axis = time, commits occur from left to right

• y-axis = runtime (seconds), lower is better.

• Each plot shows all commits for one benchmark, each commit is represented by one point.

• If one commit is much worse than the previous one, then the line is labeled with a short hash of the new/bad commit.

The points labeled typed-worst-case are for a configuration where: (1) every module is typed and (2) every import is guarded with contracts via require/typed.

The benchmarks come from GTP benchmarks v1.0.

4.1 acquire

4.2 dungeon

4.3 forth

4.4 fsm

4.5 fsmoo

4.6 gregor

4.7 jpeg

4.8 kcfa

4.9 lnm

4.10 mbta

4.11 morsecode

4.12 quadT

4.13 quadU

4.14 sieve

4.15 snake

4.16 suffixtree

4.17 synth

4.18 take5

4.19 tetris

4.20 zombie

4.21 zordoz