brag: a better Racket AST generator

This is a fork of the ragg package. It has a variety of bugfixes and new features. Some of these features have required new notation that’s not necessarily compatible with all existing ragg files.

Originally brag was built to support the Beautiful Racket project. But it has no dependencies on any Beautiful Racket libraries (and will remain thus). So it can be used independently.

1 Quick start

How would we turn this string into a structured value? That is, how would we parse it? (Let’s also suppose we’ve never heard of read.)

First, we need to consider the structure of the things we’d like to parse. The string above looks like a nested list of words. Good start.

Second, how might we describe this formally — meaning, in a way that a computer could understand? A common notation to describe the structure of these things is Backus-Naur Form (BNF). So let’s try to notate the structure of nested word lists in BNF.

nested-word-list: WORD

| LEFT-PAREN nested-word-list* RIGHT-PAREN

What we intend by this notation is this: nested-word-list is either a WORD, or a parenthesized list of nested-word-lists. We use the character * to represent zero or more repetitions of the previous thing. We treat the uppercased LEFT-PAREN, RIGHT-PAREN, and WORD as placeholders for tokens (a token being the smallest meaningful item in the parsed string):

This BNF description is also known as a grammar. Just as it does in a natural language like English or French, a grammar describes something in terms of what elements can fit where.

Have we made progress? We have a valid grammar. But we’re still missing a parser: a function that can use that description to make structures out of a sequence of tokens.

Meanwhile, it’s clear that we don’t yet have a valid program because there’s no #lang line. Let’s add one: put #lang brag at the top of the grammar, and save it as a file called "nested-word-list.rkt".

"nested-word-list.rkt"

#lang brag

nested-word-list: WORD

| LEFT-PAREN nested-word-list* RIGHT-PAREN

Now it’s a proper program. But what does it do?

> (require "nested-word-list.rkt")

> parse

#<procedure:parse>

It gives us a parse function. Let’s investigate what parse does. What happens if we pass it a sequence of tokens?

> (define a-parsed-value     (parse (list (token 'LEFT-PAREN "(")                  (token 'WORD "some")                  (token 'LEFT-PAREN "[")                  (token 'WORD "pig")                  (token 'RIGHT-PAREN "]")                  (token 'RIGHT-PAREN ")"))))

> a-parsed-value

#<syntax:unknown::-1 (nested-word-list "(" (nested-word-list "some") (nested-word-list "[" (nested-word-list "pig") "]") ")")>

Those who have messed around with macros will recognize this as a syntax object.

> (syntax->datum a-parsed-value)

'(nested-word-list   "("   (nested-word-list "some")   (nested-word-list "[" (nested-word-list "pig") "]")   ")")

That’s (some [pig]), essentially.

What happens if we pass our parse function a bigger source of tokens?

; tokenize: string -> (sequenceof token-struct?)

; Generate tokens from a string:

> (define (tokenize s)     (for/list ([str (regexp-match* #px"\\(|\\)|\\w+" s)])               (match str                 ["("                  (token 'LEFT-PAREN str)]                 [")"                  (token 'RIGHT-PAREN str)]                 [else                  (token 'WORD str)])))

; For example:

> (define token-source (tokenize "(welcome (to (((brag)) ())))"))

> (define v (parse token-source))

> (syntax->datum v)

'(nested-word-list   "("   (nested-word-list "welcome")   (nested-word-list    "("    (nested-word-list "to")    (nested-word-list     "("     (nested-word-list      "("      (nested-word-list "(" (nested-word-list "brag") ")")      ")")     (nested-word-list "(" ")")     ")")    ")")   ")")

Welcome to brag.

2 Introduction

brag is a parser generator designed to be easy to use:

• It provides a #lang for writing BNF grammars. A module written in #lang brag automatically generates a parser. The grammar controls the structure of the syntax objects it generates.

• The language uses a few conventions to simplify the expression of grammars. The first rule in the grammar is assumed to be the starting production. Identifiers in UPPERCASE are treated as terminal tokens. All other identifiers are treated as nonterminals.

• Tokenizers can be developed independently of parsers. brag takes a liberal view on tokens: they can be strings, symbols, or instances constructed with token. Tokens can optionally provide source location, in which case a syntax object generated by the parser will too.

• The parser can usually handle ambiguous grammars.

• It integrates with the rest of the Racket language toolchain.

2.1 Example: a small DSL for ASCII diagrams

Suppose we’d like to define a language for drawing simple ASCII diagrams. So if we write something like this:

3 9 X;

6 3 b 3 X 3 b;

3 9 X;

It should generate the following picture:

XXXXXXXXX

XXXXXXXXX

XXXXXXXXX

XXX

XXX

XXX

XXX

XXX

XXX

XXXXXXXXX

XXXXXXXXX

XXXXXXXXX

This makes sense in a casual way. But let’s be more precise about how the language works.

Each line of the program has a semicolon at the end, and describes the output of several rows of the line drawing. Let’s look at two of the lines in the example:

• 3 9 X;: “Repeat the following 3 times: print "X" nine times, followed by a newline.”

• 6 3 b 3 X 3 b;: “Repeat the following 6 times: print " " three times, followed by "X" three times, followed by " " three times, followed by a newline.”

Then each line consists of a repeat number, followed by pairs of (number, character) chunks. We’ll assume here that the intent of the lowercased character b is to represent the printing of a 1-character whitespace " ", and for other uppercase letters to represent the printing of themselves.

By understanding the pieces of each line, we can more easily capture that meaning in a grammar. Once we have each instruction of our ASCII DSL in a structured format, we should be able to parse it.

Here’s a first pass at expressing the structure of these line-drawing programs.

2.2 Parsing the concrete syntax

"simple-line-drawing.rkt"

#lang brag

drawing: rows*

rows: repeat chunk+ ";"

repeat: INTEGER

chunk: INTEGER STRING

Let’s try this function:

> (require brag/support)

> (require "simple-line-drawing.rkt")

> (define stx     (parse (list (token 'INTEGER 6)                  (token 'INTEGER 2)                  (token 'STRING " ")                  (token 'INTEGER 3)                  (token 'STRING "X")                  ";")))

> (syntax->datum stx)

'(drawing (rows (repeat 6) (chunk 2 " ") (chunk 3 "X") ";"))

A token is the smallest meaningful element of a source program. Tokens can be strings, symbols, or instances of the token data structure. (Plus a few other special cases, which we’ll discuss later.) Usually, a token holds a single character from the source program. But sometimes it makes sense to package a sequence of characters into a single token, if the sequence has an indivisible meaning.

If possible, we also want to attach source location information to each token. Why? Because this information will be incorporated into the syntax objects produced by parse.

A parser often works in conjunction with a helper function called a lexer that converts the raw code of the source program into tokens. The br-parser-tools/lex library can help us write a position-sensitive tokenizer:

> (require br-parser-tools/lex)

> (define (tokenize ip)     (port-count-lines! ip)     (define my-lexer       (lexer-src-pos        [(repetition 1 +inf.0 numeric)         (token 'INTEGER (string->number lexeme))]        [upper-case         (token 'STRING lexeme)]        ["b"         (token 'STRING " ")]        [";"         (token ";" lexeme)]        [whitespace         (token 'WHITESPACE lexeme #:skip? #t)]        [(eof)         (void)]))     (define (next-token) (my-lexer ip))     next-token)

> (define a-sample-input-port (open-input-string "6 2 b 3 X;"))

> (define token-thunk (tokenize a-sample-input-port))

; Now we can pass token-thunk to the parser:

> (define another-stx (parse token-thunk))

> (syntax->datum another-stx)

'(drawing (rows (repeat 6) (chunk 2 " ") (chunk 3 "X") ";"))

; The syntax object has location information:

> (syntax-line another-stx)

1

> (syntax-column another-stx)

0

> (syntax-span another-stx)

10

Note also from this lexer example:

• parse accepts as input either a sequence of tokens, or a function that produces tokens (which parse will call repeatedly to get the next token).

• As an alternative to the basic token structure, a token can also be an instance of the position-token structure (also found in br-parser-tools/lex). In that case, the token will try to derive its position from that of the position-token.

• parse will stop if it gets void (or 'eof) as a token.

• parse will skip any token that has #:skip? attribute set to #t. For instance, tokens representing comments often use #:skip?.

2.3 From parsing to interpretation

We now have a parser for programs written in this simple-line-drawing language. Our parser will return syntax objects:

> (define parsed-program     (parse (tokenize (open-input-string "3 9 X; 6 3 b 3 X 3 b; 3 9 X;"))))

> (syntax->datum parsed-program)

'(drawing   (rows (repeat 3) (chunk 9 "X") ";")   (rows (repeat 6) (chunk 3 " ") (chunk 3 "X") (chunk 3 " ") ";")   (rows (repeat 3) (chunk 9 "X") ";"))

Better still, these syntax objects will have a predictable structure that follows the grammar:

(drawing (rows (repeat <number>)

(chunk <number> <string>) ... ";")

...)

where drawing, rows, repeat, and chunk should be treated literally, and everything else will be numbers or strings.

Still, these syntax-object values are just inert structures. How do we interpret them, and make them print? We claimed at the beginning of this section that these syntax objects should be easy to interpret. So let’s do it.

As a simple example, we can write a function that looks at a syntax object and says #t if it’s the literal yes, and #f otherwise:

> (require syntax/parse)

; yes-syntax-object?: syntax-object -> boolean

; Returns true if the syntax-object is yes.

> (define (yes-syntax-object? stx)     (syntax-parse stx       [(~literal yes)        #t]       [else        #f]))

> (yes-syntax-object? #'yes)

#t

> (yes-syntax-object? #'nooooooooooo)

#f

When we encounter a syntax object with (drawing rows-stx ...), then interpret-rows each rows-stx.

For a rows, we extract out the repeat-number out of the syntax object and use it as the range of the for loop. The inner loop walks across each chunk-stx and calls interpret-chunk on it.

Finally, we need to write a definition for interpret-chunk. We want it to extract out the chunk-size and chunk-string portions, and print to standard output:

> (define (interpret-chunk chunk-stx)     (syntax-parse chunk-stx       [({~literal chunk} chunk-size chunk-string)           (for ([k (syntax-e #'chunk-size)])             (display (syntax-e #'chunk-string)))]))

> (interpret-chunk #'(chunk 3 "X"))

XXX

> (interpret-drawing #'(drawing (rows (repeat 5) (chunk 3 "X") ";")))

XXX XXX XXX XXX XXX

And now we’ve got an interpreter!

2.4 From interpretation to compilation

Wouldn’t it be nice to be able to write something like:

3 9 X;

6 3 b 3 X 3 b;

3 9 X;

Well, of course it won’t work: we don’t have a #lang line.

Let’s add one.

"letter-i.rkt"

#lang brag/examples/simple-line-drawing

3 9 X;

6 3 b 3 X 3 b;

3 9 X;

Now "letter-i.rkt" is a program.

How does this work? From the previous sections, we’ve seen how to take the contents of a file and interpret it. What we want to do now is teach Racket how to compile programs labeled with this #lang line. We’ll do two things:

• Tell Racket to use the brag-generated parser and lexer we defined earlier whenever it sees a program written with #lang brag/examples/simple-line-drawing.

• Define transformation rules for drawing, rows, and chunk to rewrite these into standard Racket forms.

The second part, the writing of the transformation rules, will look very similar to the definitions we wrote for the interpreter, but the transformation will happen at compile-time. (We could just resort to simply calling into the interpreter we just wrote up, but this section is meant to show that compilation is also viable.)

We do the first part by defining a module reader: a module reader tells Racket how to parse and compile a file. Whenever Racket sees a #lang <name>, it looks for a corresponding module reader in "<name>/lang/reader".

Here’s the definition for "brag/examples/simple-line-drawing/lang/reader.rkt":

"brag/examples/simple-line-drawing/lang/reader.rkt"

#lang s-exp syntax/module-reader

brag/examples/simple-line-drawing/semantics

#:read my-read

#:read-syntax my-read-syntax

#:whole-body-readers? #t

(require brag/examples/simple-line-drawing/lexer

brag/examples/simple-line-drawing/grammar)

(define (my-read in)

(syntax->datum (my-read-syntax #f in)))

(define (my-read-syntax src ip)

(list (parse src (tokenize ip))))

We use a helper module syntax/module-reader, which provides utilities for creating a module reader. It uses the lexer and brag-generated parser we defined earlier, and also tells Racket that it should compile the forms in the syntax object using a module called "semantics.rkt".

The semantics hold definitions for compile-drawing, compile-rows, and compile-chunk, similar to what we had for interpretation with interpret-drawing, interpret-rows, and interpret-chunk. However, compilation is not the same as interpretation: each definition does not immediately execute the act of drawing, but rather returns a syntax object whose evaluation will do the actual work.

There are a few things to note:

By the way, we can just as easily rewrite the semantics so that compile-rows does explicitly call compile-chunk. Often, though, it’s easier to write the transformation functions in this piecemeal way and depend on the Racket macro expansion system to do the rewriting as it encounters each of the forms.

• brag’s native data structure is the syntax object because the majority of Racket’s language-processing infrastructure knows how to read and write this structured value.

• Unlike in interpretation, compile-rows doesn’t compile each chunk by directly calling compile-chunk. Rather, it depends on the Racket macro expander to call each compile-XXX function as it encounters a drawing, rows, or chunk in the parsed value. The three statements at the bottom of "semantics.rkt" inform the macro expansion system to do this:

  (define-syntax drawing compile-drawing)

  (define-syntax rows compile-rows)

  (define-syntax chunk compile-chunk)

Altogether, brag’s intent is to be a parser generator for Racket that’s easy and fun to use. It’s meant to fit naturally with the other tools in the Racket language toolchain. Hopefully, it will reduce the friction in making new languages with alternative concrete syntaxes.

The rest of this document describes the brag language and the parsers it generates.

3 The language

3.1 Syntax and terminology

A program in the brag language consists of the language line #lang brag, followed by a collection of rules and possibly line comments or multiline comments.

A rule is a sequence consisting of: a rule identifier, a separator (either ":" or "::="), and a pattern.

A rule identifier is an identifier that is not in upper case.

A symbolic token identifier is an identifier that is in upper case.

A line comment begins with either # or ; and continues till the end of the line.

A multiline comment begins with (* and ends with *).

An identifier is a sequence of letters, numbers, or characters in the set "-.!$%&/<=>?^_~@". It must not contain *, +, or { and }, as those characters are used to denote quantification.

the elements sentence, verb, greeting, and object are rule identifiers. The first rule, sentence: verb optional-adjective object, is a rule whose right side is an implicit pattern sequence of three sub-patterns. The uppercased WORLD is a symbolic token identifier. The fourth rule in the program associates greeting with a choice pattern.

3.2 Cuts & splices

By default, every matched token shows up in the parse tree. But sometimes that means that the parse tree ends up holding a bunch of tokens that were only needed to complete the parsing. Once they’ve served their purpose, it’s sometimes useful to filter them out (for instance, to simplify the implementation of a language expander). To help with this kind of housekeeping, brag supports cuts and splices.

A cut in a grammar will delete an item from the parse tree. A cut is notated by prefixing either the left-hand rule name or a right-hand pattern element with a slash /.

If the cut is applied to a left-hand rule name, the rule name is omitted from the parse tree, but its node and its matched elements remain.

If the cut is applied to a right-hand pattern element, then that element is omitted from every node matching that rule.

For instance, consider this simple grammar for arithmetic expressions:

If we use it to parse this string:

We get this parse tree:

'(expr (term (factor "1")) "+" (term (factor "2") "*" (factor "3")))

Suppose we felt the + and * characters were superfluous. We can add cuts to the grammar by prefixing these pattern elements with /:

Our parse tree changes accordingly:

'(expr (term (factor "1")) (term (factor "2") (factor "3")))

Now suppose we apply a cut on the rule name, factor:

This time, the rule name disappears from the parse tree, but its nodes and elements remain:

'(expr (term ("1")) (term ("2") ("3")))

A splice in a grammar will merge the elements of a node into the surrounding node. A splice is notated by prefixing either the left-hand rule name or a right-hand pattern element with an at sign @.

If the splice is applied to a left-hand rule name, then the splice is applied every time the rule is used in the parse tree.

If the splice is applied to a right-hand pattern element, that element is spliced only when it appears as part of the production for that rule.

Suppose we remove the cut from the factor rule name and instead splice the second appearance of factor in the pattern for the term rule:

The factor elements matching the first position of the term pattern remain as they were, but the factor element matching the second position is spliced into the surrounding node:

'(expr (term (factor "1")) (term (factor "2") "3"))

Finally, suppose we add a splice to the term rule name:

This time, all the appearances of term nodes in the parse tree will have their elements spliced into the surrounding nodes:

'(expr (factor "1") (factor "2") "3")

As a convenience, when a grammar element is spliced, or a rule name is cut, brag preserves the rule name by adding it as a syntax property to the residual elements, using the rule name as a key, and the original syntax object representing the rule name as the value.

Caveat for the top-level rule: though the rule name can have a cut, it cannot have a splice — once you’re at the top level, there’s nothing above to splice into.

3.3 Syntax errors

Besides the basic syntax errors that can occur with a malformed grammar, there are a few other classes of situations that #lang brag will consider as syntax errors.

Otherwise, brag should be fairly tolerant and permit even ambiguous grammars.

3.4 Semantics

A program written in #lang brag produces a module that provides a few bindings. The most important of these is parse:

The token source can either be a sequence, or a 0-arity function that produces tokens.

A token whose type is either void or 'EOF terminates the source.

If parse succeeds, it will return a structured syntax object. The structure of the syntax object follows the overall structure of the rules in the BNF grammar. For each rule r and its associated pattern p, parse generates a syntax object #'(r p-value) where p-value’s structure follows a case analysis on p:

• For implicit and explicit sequences of patterns p1, p2, ..., the corresponding values, spliced into the structure.

• For terminals, the value of the token.

• For rule identifiers: the associated parse value for the rule.

• For choice patterns: the associated parse value for one of the matching subpatterns.

• For quantified patterns and optional patterns: the corresponding values, spliced into the structure.

Thus, it’s only the presence of rule identifiers in a rule’s pattern that tells the parser to introduce nested structure into the syntax object.

If the grammar is ambiguous, brag will choose one of the possible parse results, though it doesn’t guarantee which.

If the parse cannot be performed successfully, or if a token in the token-source uses a type that isn’t mentioned in the grammar, then parse raises an instance of exn:fail:parsing.

4 Support API

The brag/support module provides functions to interact with brag programs. The most useful is the token function, which produces tokens to be parsed.

In addition to the exports shown below, the brag/support module also provides everything from brag/support, and everything from br-parser-tools/lex.

The syntax objects produced by a parse will inject the value val in place of the token name in the grammar.

If #:skip? is true, then the parser will skip over it during a parse.

Rather than directly using the token-struct constructor, please use the helper function token to construct instances.

exn:fail:parsing implements Racket’s prop:exn:srcloc property, so if this exception reaches DrRacket’s default error handler, DrRacket should highlight the offending locations in the source.

4.1 Differences with ragg

This package is a fork of ragg. The most salient additions:

• Cuts & splices.

• Improved syntax for grammars, including comments and and quantified patterns.

• Support for REPL interactions.

• Some new conveniences, like parse-to-datum, apply-lexer, trim-ends, from/to, and from/stop-before.