» Testing Patterns

In Testing Terraform Plugins we introduce Terraform’s Testing Framework, providing reference for its functionality and introducing the basic parts of writing acceptance tests. In this section we’ll cover some test patterns that are common and considered a best practice to have when developing and verifying your Terraform plugins. At time of writing these guides are particular to Terraform Resources, but other testing best practices may be added later.

» Built-in Patterns

Acceptance tests use TestCases to construct scenarios that can be evaluated with Terraform’s lifecycle of plan, apply, refresh, and destroy. The test framework has some behaviors built in that provide very basic workflow assurance tests, such as verifying configurations apply with no diff generated by the next plan.

Each TestCase will run any PreCheck function provided before running the test, and then any CheckDestroy functions after the test concludes. These functions allow developers to verify the state of the resource and test before and after it runs.

When a test is ran, Terraform runs plan, apply, refresh, and then final plan for each TestStep in the TestCase. If the last plan results in a non-empty plan, Terraform will exit with an error. This enables developers to ensure that configurations apply cleanly. In the case of introducing regression tests or otherwise testing specific error behavior, TestStep offers a boolean field ExpectNonEmptyPlan as well ExpectError regex field to specify ways the test framework can handle expected failures. If these properties are omitted and either a non-empty plan occurs or an error encountered, Terraform will fail the test.

After all TestSteps have been ran, Terraform then runs destroy, and ends by running any CheckDestroy function provided.

» Basic test to verify attributes

The most basic resource acceptance test should use what is likely to be a common configuration for the resource under test, and verify that Terraform can correctly create the resource, and that resources attributes are what Terraform expects them to be. At a high level, the first basic test for a resource should establish the following:

Terraform can plan and apply a common resource configuration without error.
Verify the expected attributes are saved to state, and contain the values expected.
Verify the values in the remote API/Service for the resource match what is stored in state.
Verify that a subsequent terraform plan does not produce a diff/change.

The first and last item are provided by the test framework as described above in Built-in Patterns. The middle items are implemented by composing a series of Check Functions as described in Acceptance Tests: TestSteps.

To verify attributes are saved to the state file correctly, use a combination of the built-in check functions provided by the testing framework. See Built-in Check Functions to see available functions.

Checking the values in a remote API generally consists of two parts: a function to verify the corresponding object exists remotely, and a separate function to verify the values of the object. By separating the check used to verify the object exists into its own function, developers are free to re-use it for all TestCases as a means of retrieving it’s values, and can provide custom check functions per TestCase to verify different attributes or scenarios specific to that TestCase.

Here’s an example test, with in-line comments to demonstrate the key parts of a basic test.

package example

// example.Widget represents a concrete Go type that represents an API resource
func TestAccExampleWidget_basic(t *testing.T) {
    var widget example.Widget

    // generate a random name for each widget test run, to avoid
    // collisions from multiple concurrent tests.
    // the acctest package includes many helpers such as RandStringFromCharSet
    // See https://godoc.org/github.com/hashicorp/terraform/helper/acctest
    rName := acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum)

    resource.Test(t, resource.TestCase{
        PreCheck:     func() { testAccPreCheck(t) },
        Providers:    testAccProviders,
        CheckDestroy: testAccCheckExampleResourceDestroy,
        Steps: []resource.TestStep{
            {
                // use a dynamic configuration with the random name from above
                Config: testAccExampleResource(rName),
                // compose a basic test, checking both remote and local values
                Check: resource.ComposeTestCheckFunc(
                    // query the API to retrieve the widget object
                    testAccCheckExampleResourceExists("example_widget.foo", &widget),
                    // verify remote values
                    testAccCheckExampleWidgetValues(widget, rName),
                    // verify local values
                    resource.TestCheckResourceAttr("example_widget.foo", "active", "true"),
                    resource.TestCheckResourceAttr("example_widget.foo", "name", rName),
                ),
            },
        },
    })
}

func testAccCheckExampleWidgetValues(widget *example.Widget, name string) resource.TestCheckFunc {
    return func(s *terraform.State) error {
        if *widget.Active != true {
            return fmt.Errorf("bad active state, expected \"true\", got: %#v", *widget.Active)
        }
        if *widget.Name != name {
            return fmt.Errorf("bad name, expected \"%s\", got: %#v", name, *widget.Name)
        }
        return nil
    }
}

// testAccCheckExampleResourceExists queries the API and retrieves the matching Widget.
func testAccCheckExampleResourceExists(n string, widget *example.Widget) resource.TestCheckFunc {
    return func(s *terraform.State) error {
        // find the corresponding state object
        rs, ok := s.RootModule().Resources[n]
        if !ok {
            return fmt.Errorf("Not found: %s", n)
        }

        // retrieve the configured client from the test setup
        conn := testAccProvider.Meta().(*ExampleClient)
        resp, err := conn.DescribeWidget(&example.DescribeWidgetsInput{
            WidgetIdentifier: rs.Primary.ID,
        })

        if err != nil {
            return err
        }

        // If no error, assign the response Widget attribute to the widget pointer
        *widget = *resp.Widget

        return fmt.Errorf("Widget (%s) not found", rs.Primary.ID)
    }
}

// testAccExampleResource returns an configuration for an Example Widget with the provided name
func testAccExampleResource(name string) string {
    return fmt.Sprintf(`
resource "example_widget" "foo" {
  active = true
  name = "%s"
}`, name)
}

This example covers all the items needed for a basic test, and will be referenced or added to in the other test cases to come.

» Update test verify configuration changes

A basic test covers a simple configuration that should apply successfully and with no follow up differences in state. To verify a resource correctly applies updates, the second most common test found is an extension of the basic test, that simply applies another TestStep with a modified version of the original configuration.

Below is an example test, copied and modified from the basic test. Here we preserve the TestStep from the basic test, but we add an additional TestStep, changing the configuration and rechecking the values, with a different configuration function testAccExampleResourceUpdated and check function testAccCheckExampleWidgetValuesUpdated for verifying the values.

func TestAccExampleWidget_update(t *testing.T) {
    var widget example.Widget
    rName := acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum)

    resource.Test(t, resource.TestCase{
        PreCheck:     func() { testAccPreCheck(t) },
        Providers:    testAccProviders,
        CheckDestroy: testAccCheckExampleResourceDestroy,
        Steps: []resource.TestStep{
            {
                // use a dynamic configuration with the random name from above
                Config: testAccExampleResource(rName),
                Check: resource.ComposeTestCheckFunc(
                    testAccCheckExampleResourceExists("example_widget.foo", &widget),
                    testAccCheckExampleWidgetValues(widget, rName),
                    resource.TestCheckResourceAttr("example_widget.foo", "active", "true"),
                    resource.TestCheckResourceAttr("example_widget.foo", "name", rName),
                ),
            },
            {
                // use a dynamic configuration with the random name from above
                Config: testAccExampleResourceUpdated(rName),
                Check: resource.ComposeTestCheckFunc(
                    testAccCheckExampleResourceExists("example_widget.foo", &widget),
                    testAccCheckExampleWidgetValuesUpdated(widget, rName),
                    resource.TestCheckResourceAttr("example_widget.foo", "active", "false"),
                    resource.TestCheckResourceAttr("example_widget.foo", "name", rName),
                ),
            },
        },
    })
}

func testAccCheckExampleWidgetValuesUpdated(widget *example.Widget, name string) resource.TestCheckFunc {
    return func(s *terraform.State) error {
        if *widget.Active != false {
            return fmt.Errorf("bad active state, expected \"false\", got: %#v", *widget.Active)
        }
        if *widget.Name != name {
            return fmt.Errorf("bad name, expected \"%s\", got: %#v", name, *widget.Name)
        }
        return nil
    }
}

// testAccExampleResource returns an configuration for an Example Widget with the provided name
func testAccExampleResourceUpdated(name string) string {
    return fmt.Sprintf(`
resource "example_widget" "foo" {
  active = false
  name = "%s"
}`, name)
}

It’s common for resources to just have the above update test, as it is a superset of the basic test. So long as the basics are covered, combining the two tests is sufficient as opposed to having two separate tests.

» Expecting errors or non-empty plans

The number of acceptance tests for a given resource typically start small with the basic and update scenarios covered. Other tests should be added to demonstrate common expected configurations or behavior scenarios for a given resource, such as typical updates or changes to configuration, or exercising logic that uses polling for updates such as an autoscaling group adding or draining instances.

It is possible for scenarios to exist where a valid configuration (no errors during plan) would result in a non-empty plan after successfully running terraform apply. This is typically due to a valid but otherwise misconfiguration of the resource, and is generally undesirable. Occasionally it is useful to intentionally create this scenario in an early TestStep in order to demonstrate correcting the state with proper configuration in a follow-up TestStep. Normally a TestStep that results in a non-empty plan would fail the test after apply, however developers can use the ExpectNonEmptyPlan attribute to prevent failure and allow the TestCase to continue:

func TestAccExampleWidget_expectPlan(t *testing.T) {
    var widget example.Widget
    rName := acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum)

    resource.Test(t, resource.TestCase{
        PreCheck:     func() { testAccPreCheck(t) },
        Providers:    testAccProviders,
        CheckDestroy: testAccCheckExampleResourceDestroy,
        Steps: []resource.TestStep{
            {
                // use an incomplete configuration that we expect
                // to result in a non-empty plan after apply
                Config: testAccExampleResourceIncomplete(rName),
                Check: resource.ComposeTestCheckFunc(
                    resource.TestCheckResourceAttr("example_widget.foo", "name", rName),
                ),
                ExpectNonEmptyPlan: true,
            },
            {
                // apply the complete configuration
                Config: testAccExampleResourceComplete(rName),
                Check: resource.ComposeTestCheckFunc(
                    resource.TestCheckResourceAttr("example_widget.foo", "name", rName),
                ),
            },
        },
    })
}

In addition to ExpectNonEmptyPlan, TestStep also exposes an ExpectError hook, allowing developers to test configuration that they expect to produce an error, such as configuration that fails schema validators:

func TestAccExampleWidget_expectError(t *testing.T) {
    var widget example.Widget
    rName := acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum)

    resource.Test(t, resource.TestCase{
        PreCheck:     func() { testAccPreCheck(t) },
        Providers:    testAccProviders,
        CheckDestroy: testAccCheckExampleResourceDestroy,
        Steps: []resource.TestStep{
            {
                // use a configuration that we expect to fail a validator
                // on the resource Name attribute, which only allows alphanumeric
                // characters
                Config:      testAccExampleResourceError(rName + "*$%%^"),
                // No check function is given because we expect this configuration
                // to fail before any infrastructure is created
                ExpectError: regexp.MustCompile("Widget names may only contain alphanumeric characters"),
            },
        },
    })
}

ExpectError expects a valid regular expression, and the error message must match in order to consider the error as expected and allow the test to pass. If the regular expression does not match, the TestStep fails explaining that the configuration did not produce the error expected.

» Regression tests

As resources are put into use, issues can arise as bugs that need to be fixed and released in a new version. Developers are encouraged to introduce regression tests that demonstrate not only any bugs reported, but that code modified to address any bug is verified as fixing the issues. These regression tests should be named and documented appropriately to identify the issue(s) they demonstrate fixes for. When possible the documentation for a regression test should include a link to the original bug report.

An ideal bug fix would include at least 2 commits to source control:

A single commit introducing the regression test, verifying the issue(s) 1 or more commits that modify code to fix the issue(s)

This allows other developers to independently verify that a regression test indeed reproduces the issue by checking out the source at that commit first, and then advancing the revisions to evaluate the fix.

» Conclusion

Terraform’s Testing Framework allows for powerful, iterative acceptance tests that enable developers to fully test the behavior of Terraform plugins. By following the above best practices, developers can ensure their plugin behavies correctly across the most common use cases and everyday operations users will have using their plugins, and ensure that Terraform remains a world-class tool for safely managing infrastructure.