» Running Terraform in Automation

For teams that use Terraform as a key part of a change management and deployment pipeline, it can be desirable to orchestrate Terraform runs in some sort of automation in order to ensure consistency between runs, and provide other interesting features such as integration with version control hooks.

Automation of Terraform can come in various forms, and to varying degrees. Some teams continue to run Terraform locally but use wrapper scripts to prepare a consistent working directory for Terraform to run in, while other teams run Terraform entirely within an orchestration tool such as Jenkins.

This guide covers some things that should be considered when implementing such automation, both to ensure safe operation of Terraform and to accommodate some current limitations in Terraform's workflow that require careful attention in automation.

The guide assumes that Terraform will be running in an non-interactive environment, where it is not possible to prompt for input at the terminal. This is not necessarily true for wrapper scripts, but is often true when running in orchestration tools.

This is a general guide, giving an overview of things to consider when implementing orchestration of Terraform. Due to its general nature, it is not possible to go into specifics about any particular tools, though other tool-specific guides may be produced later if best practices emerge around such a tool.

» Automated Workflow Overview

When running Terraform in automation, the focus is usually on the core plan/apply cycle. The main path, then, is broadly the same as for CLI usage:

1. Initialize the Terraform working directory.
2. Produce a plan for changing resources to match the current configuration.
3. Have a human operator review that plan, to ensure it is acceptable.
4. Apply the changes described by the plan.

Steps 1, 2 and 4 can be carried out using the familiar Terraform CLI commands, with some additional options:

• terraform init -input=false to initialize the working directory.
• terraform plan -out=tfplan -input=false to create a plan and save it to the local file tfplan.
• terraform apply -input=false tfplan to apply the plan stored in the file tfplan.

The -input=false option indicates that Terraform should not attempt to prompt for input, and instead expect all necessary values to be provided by either configuration files or the command line. It may therefore be necessary to use the -var and -var-file options on terraform plan to specify any variable values that would traditionally have been manually-entered under interactive usage.

It is strongly recommended to use a backend that supports remote state, since that allows Terraform to automatically save the state in a persistent location where it can be found and updated by subsequent runs. Selecting a backend that supports state locking will additionally provide safety against race conditions that can be caused by concurrent Terraform runs.

» Controlling Terraform Output in Automation

By default, some Terraform commands conclude by presenting a description of a possible next step to the user, often including a specific command to run next.

An automation tool will often abstract away the details of exactly which commands are being run, causing these messages to be confusing and un-actionable, and possibly harmful if they inadvertently encourage a user to bypass the automation tool entirely.

When the environment variable TF_IN_AUTOMATION is set to any non-empty value, Terraform makes some minor adjustments to its output to de-emphasize specific commands to run. The specific changes made will vary over time, but generally-speaking Terraform will consider this variable to indicate that there is some wrapping application that will help the user with the next step.

To reduce complexity, this feature is implemented primarily for the main workflow commands described above. Other ancillary commands may still produce command line suggestions, regardless of this setting.

» Plan and Apply on different machines

When running in an orchestration tool, it can be difficult or impossible to ensure that the plan and apply subcommands are run on the same machine, in the same directory, with all of the same files present.

Running plan and apply on different machines requires some additional steps to ensure correct behavior. A robust strategy is as follows:

• After plan completes, archive the entire working directory, including the .terraform subdirectory created during init, and save it somewhere where it will be available to the apply step. A common choice is as a "build artifact" within the chosen orchestration tool.
• Before running apply, obtain the archive created in the previous step and extract it at the same absolute path. This re-creates everything that was present after plan, avoiding strange issues where local files were created during the plan step.

Terraform currently makes some assumptions which must be accommodated by such an automation setup:

• The saved plan file can contain absolute paths to child modules and other data files referred to by configuration. Therefore it is necessary to ensure that the archived configuration is extracted at an identical absolute path. This is most commonly achieved by running Terraform in some sort of isolation, such as a Docker container, where the filesystem layout can be controlled.
• Terraform assumes that the plan will be applied on the same operating system and CPU architecture as where it was created. For example, this means that it is not possible to create a plan on a Windows computer and then apply it on a Linux server.
• Terraform expects the provider plugins that were used to produce a plan to be available and identical when the plan is applied, to ensure that the plan is interpreted correctly. An error will be produced if Terraform or any plugins are upgraded between creating and applying a plan.
• Terraform can't automatically detect if the credentials used to create a plan grant access to the same resources used to apply that plan. If using different credentials for each (e.g. to generate the plan using read-only credentials) it is important to ensure that the two are consistent in which account on the corresponding service they belong to.

» Interactive Approval of Plans

Another challenge with automating the Terraform workflow is the desire for an interactive approval step between plan and apply. To implement this robustly, it is important to ensure that either only one plan can be outstanding at a time or that the two steps are connected such that approving a plan passes along enough information to the apply step to ensure that the correct plan is applied, as opposed to some later plan that also exists.

Different orchestration tools address this in different ways, but generally this is implemented via a build pipeline feature, where different steps can be applied in sequence, with later steps having access to data produced by earlier steps.

The recommended approach is to allow only one plan to be outstanding at a time. When a plan is applied, any other existing plans that were produced against the same state are invalidated, since they must now be recomputed relative to the new state. By forcing plans to be approved (or dismissed) in sequence, this can be avoided.

» Auto-Approval of Plans

While manual review of plans is strongly recommended for production use-cases, it is sometimes desirable to take a more automatic approach when deploying in pre-production or development situations.

Where manual approval is not required, a simpler sequence of commands can be used:

• terraform init -input=false
• terraform apply -input=false -auto-approve

This variant of the apply command implicitly creates a new plan and then immediately applies it. The -auto-approve option tells Terraform not to require interactive approval of the plan before applying it.

» Testing Pull Requests with terraform plan

terraform plan can be used as a way to perform certain limited verification of the validity of a Terraform configuration, without affecting real infrastructure. Although the plan step updates the state to match real resources, thus ensuring an accurate plan, the updated state is not persisted, and so this command can safely be used to produce "throwaway" plans that are created only to aid in code review.

When implementing such a workflow, hooks can be used within the code review tool in question (for example, Github Pull Requests) to trigger an orchestration tool for each new commit under review. Terraform can be run in this case as follows:

• terraform plan -input=false

As in the "main" workflow, it may be necessary to provide -var or -var-file as appropriate. The -out option is not used in this scenario because a plan produced for code review purposes will never be applied. Instead, a new plan can be created and applied from the primary version control branch once the change is merged.

» Multi-environment Deployment

Automation of Terraform often goes hand-in-hand with creating the same configuration multiple times to produce parallel environments for use-cases such as pre-release testing or multi-tenant infrastructure. Automation in such a situation can help ensure that the correct settings are used for each environment, and that the working directory is properly configured before each operation.

The two most interesting commands for multi-environment orchestration are terraform init and terraform workspace. The former can be used with additional options to tailor the backend configuration for any differences between environments, while the latter can be used to safely switch between multiple states for the same config stored in a single backend.

Where possible, it's recommended to use a single backend configuration for all environments and use the terraform workspace command to switch between workspaces:

• terraform init -input=false
• terraform workspace select QA

In this usage model, a fixed naming scheme is used within the backend storage to allow multiple states to exist without any further configuration.

Alternatively, the automation tool can set the environment variable TF_WORKSPACE to an existing workspace name, which overrides any selection made with the terraform workspace select command. Using this environment variable is recommended only for non-interactive usage, since in a local shell environment it can be easy to forget the variable is set and apply changes to the wrong state.

In some more complex situations it is impossible to share the same backend configuration across environments. For example, the environments may exist in entirely separate accounts within the target service, and thus need to use different credentials or endpoints for the backend itself. In such situations, backend configuration settings can be overridden via the -backend-config option to terraform init.

» Pre-installed Plugins

In default usage, terraform init downloads and installs the plugins for any providers used in the configuration automatically, placing them in a subdirectory of the .terraform directory. This affords a simpler workflow for straightforward cases, and allows each configuration to potentially use different versions of plugins.

In automation environments, it can be desirable to disable this behavior and instead provide a fixed set of plugins already installed on the system where Terraform is running. This then avoids the overhead of re-downloading the plugins on each execution, and allows the system administrator to control which plugins are available.

To use this mechanism, create a directory somewhere on the system where Terraform will run and place into it the plugin executable files. The plugin release archives are available for download on releases.hashicorp.com. Be sure to download the appropriate archive for the target operating system and architecture.

After extracting the necessary plugins, the contents of the new plugin directory will look something like this:

$ ls -lah /usr/lib/custom-terraform-plugins
-rwxrwxr-x 1 user user  84M Jun 13 15:13 terraform-provider-aws-v1.0.0-x3
-rwxrwxr-x 1 user user  84M Jun 13 15:15 terraform-provider-rundeck-v2.3.0-x3
-rwxrwxr-x 1 user user  84M Jun 13 15:15 terraform-provider-mysql-v1.2.0-x3

The version information at the end of the filenames is important so that Terraform can infer the version number of each plugin. Multiple versions of the same provider plugin can be installed, and Terraform will use the newest one that matches the provider version constraints in the Terraform configuration.

With this directory populated, the usual auto-download and plugin discovery behavior can be bypassed using the -plugin-dir option to terraform init:

• terraform init -input=false -plugin-dir=/usr/lib/custom-terraform-plugins

When this option is used, only the plugins in the given directory are available for use. This gives the system administrator a high level of control over the execution environment, but on the other hand it prevents use of newer plugin versions that have not yet been installed into the local plugin directory. Which approach is more appropriate will depend on unique constraints within each organization.

Plugins can also be provided along with the configuration by creating a terraform.d/plugins/OS_ARCH directory, which will be searched before automatically downloading additional plugins. The -get-plugins=false flag can be used to prevent Terraform from automatically downloading additional plugins.

» Terraform Enterprise

As an alternative to home-grown automation solutions, Hashicorp offers Terraform Enterprise.

Internally, Terraform Enterprise runs the same Terraform CLI commands described above, using the same release binaries offered for download on this site.

Terraform Enterprise builds on the core Terraform CLI functionality to add additional features such as role-based access control, orchestration of the plan and apply lifecycle, a user interface for reviewing and approving plans, and much more.

It will always be possible to run Terraform via in-house automation, to allow for usage in situations where Terraform Enterprise is not appropriate. It is recommended to consider Terraform Enterprise as an alternative to in-house solutions, since it provides an out-of-the-box solution that already incorporates the best practices described in this guide and can thus reduce time spent developing and maintaining an in-house alternative.