1 "pollen.rkt"

We can write a "pollen.rkt" file in any #lang. Most commonly, you’ll choose #lang racket/base. #lang racket is a tiny bit more convenient because it loads more libraries by default. But as a result, it can be slightly slower to load. So racket/base is the more virtuous habit.

This particular "pollen.rkt", however, is written in #lang scribble/lp2, which is the literate-programming variant of Racket’s Scribble documentation language (which is also the basis of Pollen). #lang scribble/lp2 is like a text-based version of #lang racket/base. The “literate programming” angle is that we can mix documentation and source code in one file. Probably not the option you’d choose for your own project. But in this teaching project, it’s the right tool for the job.

In this documentation, chunks of source code look like this:

There’s nothing special about the chunk name — it’s just a label that #lang scribble/lp2 will use to snap the code together at the end. The result is that if you require this file normally, you’ll get the usual functions and values; but if you run it with Scribble, it turns into the documentation you see here.

Aside from that wrinkle, all the code shown here is standard racket/base, and can be copied & adapted for your own "pollen.rkt" files written in #lang racket/base (or #lang racket).

1.1 What is "pollen.rkt" for?

The "pollen.rkt" source file is the main source of functions and values for the source files in a Pollen project. Everything provided from a "pollen.rkt" is automatically available to #lang pollen source files in the same directory or subdirectories (unless superseded by another "pollen.rkt" within a subdirectory).

For more, see Introducing "pollen.rkt" in the main Pollen docs.

1.2 Imports

Though require declarations can live anywhere at the top layer of a source file, it’s typical to consolidate them at the top.

If we were using #lang racket, these next libraries would already be imported. But we aren’t, so they’re not.

Other libraries we’ll be using. sugar and txexpr are utility libraries installed with Pollen. hyphenate is separate, but can be installed easily:

It’s up to you whether you store all your functions and definitions in "pollen.rkt", or spread them over different source files. Regardless of where you store them, "pollen.rkt" remains the central gathering point for everything you want to propagate to the #lang pollen source files in your project.

If you end up making reusable functions, you can share them between Pollen projects (and with other Racket users, if you want) by moving them into a package. For more, see Creating Packages.

1.3 Exports

Note that all-defined-out would only export the definitions that are created in this file. To make imported definitions available too, we need to re-export them explicitly with all-from-out.

1.4 Definitions

1.4.1 Values

Definitions in a "pollen.rkt" can be functions or values. Because they get propagated to other Pollen source files, they’re almost like global definitions. As usual with global definitions, you should use them when you need them, but it’s still wise to localize things within specific directories or source files when you can. Otherwise the main "pollen.rkt" can get to be unwieldy.

content-rule-color is a CSS value.

buy-url is the master URL for buying the TFL paperback. In general, it’s wise to store hard-coded values like these in a variable so that you can change the value from one location later if you need to.

no-hyphens-attr is an X-expression attribute we’ll use to signal that a certain X-expression should not be hyphenated. The need for this will be explained later.

1.4.2 Tag functions

1.4.2.1 Making tagged X-expressions (txexprs)

In a "pollen.rkt" you’ll be making a lot of tagged X-expressions (txexprs for short). A txexpr is just a Racket list, so you can make txexprs with any of Racket’s list-making functions — which are plentiful. Which one you use depends on what fits most naturally with the current task.

X-expressions are introduced in the Pollen docs.

Let’s run through a few of them, so they start to become familiar. Suppose we want to generate the txexpr ’(div ((class "big")) "text"). Here are some ways to do it.

1.4.3 txexpr

A utility function from the txexpr module. We used it in the link function above. The major advantage of txexpr is that it will raise an error if your arguments are invalid types for a tagged X-expression.

The second and third arguments to txexpr are lists, so you can use any list notation. If your txexpr doesn’t have attributes, you pass empty or null for the second argument.

Because txexpr is designed to check its arguments for correctness, it insists on getting an explicit argument for the attributes, even if empty. When you’re using generic list-making functions (see below) to create txexprs, you can omit the attribute list if it’s empty.

1.4.4 list and list*

list* is particularly useful for making txexprs, because it automatically splices the last argument.

Unlike txexpr, however, list and list* will happily let you create invalid txexprs.

This isn’t necessarily a bad thing. When a txexpr needs to pass through multiple layers of processing, it can be useful to create intermediate results that are txexpr-ish, and simplify them at the end.

1.4.5 cons

All lists are ultimately made of cons cells. So you can make txexprs with cons too, though it’s more cumbersome than the other methods. In most cases, list* is clearer & more flexible, because cons can only take two arguments, whereas list* can take any number.

1.4.6 quasiquote

As the name suggests, quasiquote works like quote, but lets you unquote variables within. Quasiquote notation is pleasingly compact for simple cases. But it can be unruly for complex ones. The unquote operator (,) puts a variable’s value into the list. The unquote splicing operator (,@) does the same thing, but if the variable holds a list of items, it merges those items into the list (i.e., does not leave them as a sublist).

Below, we unquote attrs because we want them to remain a sublist. But we splice elements because we want them to be merged with the main list.

1.4.7 Functions

In Pollen notation, we’ll invoke the tag function like so:

This will become, in Racket notation:

Getting a feel for the duality of Pollen & Racket notation is a necessary part of the learning curve. If it seems like an annoying complication, consider that the two styles are optimized for different contexts: Pollen notation is for embedding commands in text, and Racket notation is for writing code. The fact that the two are interchangeable is what guarantees that everything that can be done in Racket can also be done in Pollen.

The result of our tag function will be a tagged X-expression that looks like this:

The definition of link follows the arguments above.

url is a mandatory argument.

css-class is a keyword argument (= must be introduced with #:class) and also optional (if it’s not provided, it will default to #f).

tx-elements is an optional argument that represents the text (or other content) that gets linked. If we don’t have anything to link, use url as the link text too.

tx-elements is a rest argument, as in “put the rest of the arguments here.” Most definitions of tag functions should end with a rest argument. Why? Because in Pollen notation, the {text ...} in ◊func[arg]{text ...} can return any number of arguments. Maybe one (e.g., if {text ...} is a word) or maybe more (e.g, if {text ...} is a multiline block).

If you don’t use a rest argument, and pass multiple text arguments to your tag function, you’ll get an error (namely an “arity error,” which means the function got more arguments than it expected).

let* is the idiomatic Racket way to do what looks like mutation. Though you’re not really mutating the variable — you’re creating copies, all of which have the same name. For true mutation, you could also use set! — not wrong, but not idiomatic.

The next three tag functions are just convenience variations of link. But they involve some crafty (and necessary) uses of apply.

Notice that we have to use apply to correctly pass our tx-elements rest argument to link. Why? Because link expects its text arguments to look like this:

Not like this:

But that’s what will happen if we just do (link tx-elements), and link will complain. (Try it!)

The role of apply is to take a list of arguments and append them to the end of the function call, so

Is equivalent to:

The difference here is that we’re not providing a specific URL. Rather, we want to pass through whatever URL we get from the Pollen source. So we add a url argument.

“Right, but shouldn’t we use a rest argument just in case?” It depends on how you like errors to be handled. You could capture the text arguments with a rest argument and then just silently dispose of them. But this might be mysterious to the person editing the Pollen source (whether you or someone else). "Where did my text go?"

Whereas if we omit the rest argument, and try to pass text arguments anyhow, image will immediately raise an error, letting us know that we’re misusing it.

Here, we’ll use default-tag-function, which is an easy way to make a simple tag function. Any keywords passed in will be propagated to every use of the tag function.

The idea is to interpret a sequence of three (or more) linebreaks in the text as a list-item delimiter (i.e., drop in a <li> tag). Why three linebreaks? Because later on, we’ll use one linebreak to denote a new line, and two linebreaks to denote a new paragraph.

This function will be used within a decode function (more on that below) in a position where it will be passed a list of X-expresssion elements, so it also needs to return a list of X-expression elements.

The idiomatic Racket way to enforce requirements on input & output values is with a function contract. For simplicity, I’m not using them here, but they are another virtuous habit .

Our list of elements could contain sequences like '("\n" "\n" "\n"), which should mean the same thing as "\n\n\n". So first, we’ll combine adjacent newlines with merge-newlines.

filter-split will divide a list into sublists based on a test for the list-item delimiter. The result will be a list of lists, each representing the contents of an 'li tag. We’ll convert any paragraphs that are inside the list items. Finally we’ll wrap each of these lists of paragraphs in an 'li tag.

Explicit type checking — e.g., (string? elem) — is common in Racket. You can do “duck typing” (see with-handlers) but it’s not idiomatic. IMO this is wise — better to have an explicit, readable test (like string?) rather than an implicit, indirect one (“If the input isn’t a string?, then a certain error will arise.”)

Because of the expression-based structure of Racket, it’s often possible to write functions in an absurdly nested style. For instance, the last function could be written like so:

This is a good way to lose your friends, and then your mind. You may not care to spell everything out the way I’ve been doing in this sample project. But readability is a virtuous habit.

In Racket you’ll often see functions that make other functions. (These are sometimes called higher-order functions.) This is a good way to avoid making a bunch of functions that have small variations.

One way to write this function is like so:

(define (make-list-function tag [attrs empty])

(define (listifier . args)

(list* tag attrs (detect-list-items args)))

listifier)

That is, explicitly define a new function called listifier and then return that function. That’s the best way to do it in many programming languages.

In Racket, it’s wouldn’t be wrong. But you should feel comfortable with the idea that any function can be equivalently expressed in lambda notation, which is the more idiomatic form.

Another example of using a tag function to handle fiddly HTML markup. The btw tag expands to an HTML list. We will then crack this open and slip in a div for the headline.

For this tag function, we’ll assume that target is a single text argument, because that’s how it will be used. But to be safe, we’ll raise an error if we get too many arguments.

What makes this function a little tricky is that url is optional, but if it appears, it appears first. That makes this a good job for case-lambda, which lets you define separate branches for your function depending on the number of arguments provided.

In the one-argument case, rather than duplicate the line of code in the two-argument case, we call the function again with a second argument.

This function depends on a personal commitment to name source files in a logical, predictable way, e.g., “Why Does Typography Matter?” becomes why-does-typography-matter.html. This way, the name of the source file for a page can be derived from its title.

If you needed to associate targets with URLs arbitrarily, you could store the targets and URLs in an association list or hashtable. But I prefer this approach, because it’s easy to add new pages and cross-references, without the extra housekeeping step.

Well, almost. One wrinkle that arises is connecting singular and plural versions of the target text to the right URL. For instance, “typewriter habit” and “typewriter habits” should both link to typewriter-habits.html. But “point size” and “point sizes” should both link to point-size.html. Again, you could keep a list manually. But that’s a drag. Instead, let’s make the singular and plural versions of the target (called target-variants) and compare these against a list of all possible HTML files in the project directory (called actual-filenames). When we find a match, that will be the URL we’re looking for.

This could also be done with default-tag-function. And as a rule of thumb, it’s wise to reserve macros for the times you can’t avoid using them. Otherwise, use a function.

We’ll bend that rule here because this is a quick & easy example macro. What makes it suitable to be handled as a macro is that we want to use the name of the identifier (for instance topic) as an argument to the function. With a function, we can’t do that. But with a macro, we can.

define-syntax-rule is the easiest macro form: essentially you’re writing a code template with arguments that will be filled in when you invoke the macro. Notice how heading-name appears in two roles: first as an identifier name, and then as a literal symbol.

“Wait, why does 'heading-name not produce the literal symbol 'heading-name?” The 'heading-name syntax is just shorthand for (quote heading-name). Because this is a macro, the heading-name inside this expression gets replaced with the value of the macro argument heading-name before quote is evaluated.

This macro relies on syntax-case rather than define-syntax-rule. It’s a little more complicated, but also more flexible (and more idiomatic in Racket). define-syntax-rule is actually a special-case version of syntax-case. The best advice on learning macros is to start with syntax-case, because you can’t live without it.

Greg Hendershott’s Fear of Macros is a great place to start if you’re new to Racket macros.

Otherwise this macro is similar to define-heading, except that we want to introduce a new identifier with a different name, but based on the argument given to the macro. So if we pass topic to the macro, it will define an identifier called topic-from-metas. You can’t do that with define-syntax-rule.

procedure (topic-from-metas metas) → txexpr?   metas : hash?

procedure (section-from-metas metas) → txexpr?   metas : hash?

procedure (chapter-from-metas metas) → txexpr?   metas : hash?

This function assumes that each row has the same number of columns. You could improve it to fill in blank cells in rows that need them.

The idea is to break down the input into table headings and cells, and then work back up, wrapping each layer in the appropriate tags.

This function will only work properly if you have sips on your system (= command-line image-processing program, included with OS X).

This shows how you can fold other kinds of project housekeeping into Pollen commands. Here, the function generates the thumbnail it needs when the page is compiled.

One disadvantage of this approach is that the thumbnail will always be generated on recompile, though you could put in some logic to avoid this (e.g., check the modification date of the PDF). In this case, sips is fast enough that it’s not bothersome.

In a Pollen markup source, the output is a tagged X-expression that starts with root:

Recall that every Pollen tag calls a function with the same name (if it exists, otherwise it just becomes a tag). This is also true of root.

root has slightly special status inasmuch as it is the top tag of the X-expression, and thus the last tag function that will get called. Therefore, root is a good place to put any processing that should happen once all the page content has been filled in.

Often, you’ll want to use a decode or decode-elements function, which can recursively perform different kinds of processing on different types of page elements.

In this case, we’ll use decode-elements twice. First, we’ll use it just to detect paragraphs. We’ll do this so that they’re treated as blocks (see block-txexpr?) in the second phase, which does the rest of the processing.

The basic hyphenate function comes from the hyphenate module. We could attach hyphenate to our root decoder as a string processor rather than block processor. But we want to be able to handle our no-hyphens flag (aka no-hyphens-attr), which is stored in the attributes of the X-expression. Therefore, we have to look at blocks, not just strings.

Because I’m a typography snob I like to push quotation marks into the margin a little bit when they appear at the left edge of a line (aka hanging quotes). This function just wraps left-hand quote marks in two little tags (push and pull) that I can then manipulate in CSS to get the effect.

When I type an em dash in my sources, I will often leave a space around it, but I don’t want spaces in the output, so this function removes them.

1.4.7.1 Miscellaneous tag functions

Presented without docs or comment, as it should be obvious at this point what they do.

1.5 Utility functions

Rather than the ordinary:

This functions will be useful later when we want to change the behavior of certain functions when Pollen runs in dev mode. For instance, we might want to run certain functions in a higher speed / lower quality mode. Or output debug information about others.

Though the environment variable name is fixed as POLLEN, there’s no special magic to DEV. We could pick any value we wanted to denote development mode:

1.6 Finally

This last incantation is needed so this scribble/lp2 document knows how to put together all the code chunks we’ve introduced in this file.