Big Data

How to Use Terraform with Rockset


The goal of this blog post is to provide best practices on how to use terraform to configure Rockset to ingest the data into two collections, and how to setup a view and query lambdas that are used in an application, plus to show the workflow of later updating the query lambdas. This mimics how we use terraform at Rockset to manage Rockset resources.

Terraform is the most popular used DevOps tool for infrastructure management, that lets you define your infrastructure as code, and then the tool will take the configuration and compute the steps needed to take it from the current state to the desired state.

Last we’ll look at how to use GitHub actions to automatically run terraform plan for pull requests, and once the pull request are approved and merged, it will run terraform apply to make the required changes.

The full terraform configuration used in this blog post is available here.

Terraform

To follow along on your own, you will need:

and you also need to install terraform on your computer, which is as simple as this on macOS.

$ brew tap hashicorp/tap
$ brew install hashicorp/tap/terraform

(instructions for other operating systems are available in the link above)

Provider setup

The first step to using terraform is to configure the providers we will be using, Rockset and AWS. Create a file called _provider.tf with the contents.

terraform {
  required_providers {
        aws = {
            source = "hashicorp/aws"
            version = "~> 4"
    }
    rockset = {
      source = "rockset/rockset"
            version = "0.6.2"
    }
  }
}

provider rockset {}
provider aws {
    region = "us-west-2"
}

Both providers use environment variables to read the credentials they require to access the respective services.

  • Rockset: ROCKSET_APIKEY and ROCKSET_APISERVER
  • AWS: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or AWS_PROFILE

Backend configuration

Terraform saves information about the managed infrastructure and configuration in a state file. To share this state between local runs on your computer, and automated runs from GitHub actions, we use a so called backend configuration, which stores the state in an AWS S3 bucket, so all invocations of terraform can use it.

backend "s3" {
    bucket = "rockset-community-terraform"
    key    = "blog/state"
    region = "us-west-2"
  }

⚠️ For a production deployment, make sure to configure state locking too.

AWS IAM Role

To allow Rockset to ingest the contents of an S3 bucket, we first have to create an AWS IAM role which Rockset will use to access the contents of the bucket. It uses a data source to read information about your Rockset organization, so it can configure AWS correctly.

data rockset_account current {}

resource "aws_iam_policy" "rockset-s3-integration" {
  name   = var.rockset_role_name
  policy = templatefile("${path.module}/data/policy.json", {
    bucket = var.bucket
    prefix = var.bucket_prefix
  })
}

resource "aws_iam_role" "rockset" {
  name               = var.rockset_role_name
  assume_role_policy = data.aws_iam_policy_document.rockset-trust-policy.json
}

data "aws_iam_policy_document" "rockset-trust-policy" {
  statement {
    sid     = ""
    effect  = "Allow"
    actions = [
      "sts:AssumeRole"
    ]
    principals {
      identifiers = [
        "arn:aws:iam::${data.rockset_account.current.account_id}:root"
      ]
      type = "AWS"
    }
    condition {
      test   = "StringEquals"
      values = [
        data.rockset_account.current.external_id
      ]
      variable = "sts:ExternalId"
    }
  }
}

resource "aws_iam_role_policy_attachment" "rockset_s3_integration" {
  role       = aws_iam_role.rockset.name
  policy_arn = aws_iam_policy.rockset-s3-integration.arn
}

This creates an AWS IAM cross-account role which Rockset is allowed to use to ingest data.

Rockset S3 integration

Now we can create the integration that enables Rockset to ingest data from S3, using the IAM role above.

resource "time_sleep" "wait_30s" {
  depends_on      = [aws_iam_role.rockset]
  create_duration = "15s"
}

resource "rockset_s3_integration" "integration" {
  name         = var.bucket
  aws_role_arn = aws_iam_role.rockset.arn
  depends_on   = [time_sleep.wait_30s]
}

⚠️ You can get an AWS cross-account role error if you skip the time_sleep resource, because it takes a few seconds for the newly created AWS role to propagate, so this saves you from having to rerun terraform apply again.

Rockset collection

With the integration we are now able to create a workspace to hold all resources we will add, and then setup a collection which ingest data using the above S3 integration.

resource rockset_workspace blog {
  name = "blog"
}

resource "rockset_s3_collection" "collection" {
  name           = var.collection
  workspace      = rockset_workspace.blog.name
  retention_secs = var.retention_secs
  source {
    format           = "json"
    integration_name = rockset_s3_integration.integration.name
    bucket           = var.bucket
    pattern          = "public/movies/*.json"
  }
}

Kafka Collection

Next we’ll setup a collection from a Confluent Cloud source, and add an ingest transformation that summarizes the data.

resource "rockset_kafka_integration" "confluent" {
  name         = var.bucket
  aws_role_arn = aws_iam_role.rockset.arn
  use_v3            = true
  bootstrap_servers = var.KAFKA_REST_ENDPOINT
  security_config = {
    api_key = var.KAFKA_API_KEY
    secret  = var.KAFKA_API_SECRET
  }
}

resource "rockset_kafka_collection" "orders" {
  name           = "orders"
  workspace      = rockset_workspace.blog.name
  retention_secs = var.retention_secs
  source {
    integration_name = rockset_kafka_integration.confluent.name
  }
  field_mapping_query = file("data/transformation.sql")
}

The SQL for the ingest transformation is stored in a separate file, which terraform injects into the configuration.

SELECT
    COUNT(i.orderid) AS orders,
    SUM(i.orderunits) AS units,
    i.address.zipcode,
    i.address.state,
    -- bucket data in five minute buckets
    TIME_BUCKET(MINUTES(5), TIMESTAMP_MILLIS(i.ordertime)) AS _event_time
FROM
    _input AS i
WHERE
    -- drop all records with an incorrect state
    i.address.state != 'State_'
GROUP BY
    _event_time,
    i.address.zipcode,
    i.address.state

View

With the data ingested into a collection we can create a view, which limits which documents in a collection can be accessed through that view.

resource rockset_view english-movies {
  name      = "english-movies"
  query     = file("data/view.sql")
  workspace = rockset_workspace.blog.name
  depends_on = [rockset_alias.movies]
}

The view needs an explicit depends_on meta-argument as terraform doesn’t interpret the SQL for the view which resides in a separate file.

Alias

An alias is a way to refer to an existing collection by a different name. This is a convenient way to be able to change the which collection a set of queries use, without having to update the SQL for all of them.

resource rockset_alias movies {
  collections = ["${rockset_workspace.blog.name}.${rockset_s3_collection.movies.name}"]
  name        = "movies"
  workspace   = rockset_workspace.blog.name
}

For instance, if we started to ingest movies from a Kafka stream, we can update the alias to reference the new collection and all queries start using it immediately.

Role

We create a role which is limited to only executing query lambdas only in the blog workspace, and then save the API key in the AWS Systems Manager Parameter Store for later retrieval by the code which will execute the lambda. This way the credentials will never have to be exposed to a human.

resource rockset_role read-only {
  name = "blog-read-only"
  privilege {
    action = "EXECUTE_QUERY_LAMBDA_WS"
    cluster = "*ALL*"
    resource_name = rockset_workspace.blog.name
  }
}

resource "rockset_api_key" "ql-only" {
  name = "blog-ql-only"
  role = rockset_role.read-only.name
}

resource "aws_ssm_parameter" "api-key" {
  name  = "/rockset/blog/apikey"
  type  = "SecureString"
  value = rockset_api_key.ql-only.key
}

Query Lambda

The query lambda stores the SQL in a separate file, and has a tag that uses the terraform variable stable_version which when set, is used to pin the stable tag to that version of the query lambda, and if not set it will point to the latest version.

Placing the SQL in a separate file isn’t a requirement, but it makes for easier reading and you can copy/paste the SQL into the Rockset console to manually try the changes. Another benefit is that reviewing changes to the SQL is easier when it isn’t intermingled with other changes, like it would if it was placed in-line with the terraform configuration.

SELECT
    title,
    TIME_BUCKET(
            YEARS(1),
            PARSE_TIMESTAMP('%Y-%m-%d', release_date)
        ) as year,
  popularity
FROM
    blog.movies AS m
where
    release_date != ''
  AND popularity > 10
GROUP BY
    year,
    title,
    popularity
order by
    popularity desc
resource "rockset_query_lambda" "top-rated" {
  name      = "top-rated-movies"
  workspace = rockset_workspace.blog.name
  sql {
    query = file("data/top-rated.sql")
  }
}

resource "rockset_query_lambda_tag" "stable" {
  name         = "stable"
  query_lambda = rockset_query_lambda.top-rated.name
  version      = var.stable_version == "" ? rockset_query_lambda.top-rated.version : var.stable_version
  workspace    = rockset_workspace.blog.name
}

Applying the configuration

With all configuration files in place, it is time to “apply” the changes, which means that terraform will read the configuration files, and interrogate Rockset and AWS for the current configuration, and then calculate what steps it needs to take to get to the end state.

The first step is to run terraform init, which will download all required terraform providers and configure the S3 backend.

$ terraform init

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 4.0"...
- Finding rockset/rockset versions matching "~> 0.6.2"...
- Installing hashicorp/aws v4.39.0...
- Installed hashicorp/aws v4.39.0 (signed by HashiCorp)
- Installing hashicorp/time v0.9.1...
- Installed hashicorp/time v0.9.1 (signed by HashiCorp)
- Installing rockset/rockset v0.6.2...
- Installed rockset/rockset v0.6.2 (signed by a HashiCorp partner, key ID DB47D0C3DF97C936)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Next we run terraform plan to get a list of which resources terraform is going to create, and to see the order in which it will create it.

$ terraform plan
data.rockset_account.current: Reading...
data.rockset_account.current: Read complete after 0s [id=318212636800]
data.aws_iam_policy_document.rockset-trust-policy: Reading...
data.aws_iam_policy_document.rockset-trust-policy: Read complete after 0s [id=2982727827]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_policy.rockset-s3-integration will be created
  + resource "aws_iam_policy" "rockset-s3-integration" {
      + arn       = (known after apply)
      + id        = (known after apply)
      + name      = "rockset-s3-integration"
      + path      = "/"
      + policy    = jsonencode(
            {
              + Id        = "RocksetS3IntegrationPolicy"
              + Statement = [
                  + {
                      + Action   = [
                          + "s3:ListBucket",
                        ]
                      + Effect   = "Allow"
                      + Resource = [
                          + "arn:aws:s3:::rockset-community-datasets",
                        ]
                      + Sid      = "BucketActions"
                    },
                  + {
                      + Action   = [
                          + "s3:GetObject",
                        ]
                      + Effect   = "Allow"
                      + Resource = [
                          + "arn:aws:s3:::rockset-community-datasets/*",
                        ]
                      + Sid      = "ObjectActions"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + policy_id = (known after apply)
      + tags_all  = (known after apply)
    }

...

# rockset_workspace.blog will be created
  + resource "rockset_workspace" "blog" {
      + created_by  = (known after apply)
      + description = "created by Rockset terraform provider"
      + id          = (known after apply)
      + name        = "blog"
    }

Plan: 15 to add, 0 to change, 0 to destroy.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.                                                                                         7s 665ms  13:53:32

Review the output and verify that it is doing what you expect, and then you are ready to apply the changes using terraform apply. This repeats the plan output, and asks you to verify that you are ready to apply the changes

$ terraform apply
data.rockset_account.current: Reading...
data.rockset_account.current: Read complete after 0s [id=318212636800]
data.aws_iam_policy_document.rockset-trust-policy: Reading...
data.aws_iam_policy_document.rockset-trust-policy: Read complete after 0s [id=2982727827]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

...

# time_sleep.wait_30s will be created
  + resource "time_sleep" "wait_30s" {
      + create_duration = "15s"
      + id              = (known after apply)
    }

Plan: 16 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

rockset_workspace.blog: Creating...
rockset_kafka_integration.confluent: Creating...
rockset_workspace.blog: Creation complete after 0s [id=blog]
rockset_role.read-only: Creating...
rockset_query_lambda.top-rated: Creating...
rockset_role.read-only: Creation complete after 1s [id=blog-read-only]
rockset_api_key.ql-only: Creating...
rockset_api_key.ql-only: Creation complete after 0s [id=blog-ql-only]
rockset_query_lambda.top-rated: Creation complete after 1s [id=blog.top-rated-movies]
rockset_query_lambda_tag.stable: Creating...
rockset_query_lambda_tag.stable: Creation complete after 0s [id=blog.top-rated-movies.stable]
rockset_kafka_integration.confluent: Creation complete after 1s [id=confluent-cloud-blog]
rockset_kafka_collection.orders: Creating...
aws_ssm_parameter.api-key: Creating...
aws_iam_role.rockset: Creating...
aws_iam_policy.rockset-s3-integration: Creating...
aws_ssm_parameter.api-key: Creation complete after 1s [id=/rockset/blog/apikey]
aws_iam_policy.rockset-s3-integration: Creation complete after 1s [id=arn:aws:iam::459021908517:policy/rockset-s3-integration]
aws_iam_role.rockset: Creation complete after 2s [id=rockset-s3-integration]
aws_iam_role_policy_attachment.rockset_s3_integration: Creating...
time_sleep.wait_30s: Creating...
aws_iam_role_policy_attachment.rockset_s3_integration: Creation complete after 0s [id=rockset-s3-integration-20221114233744029000000001]
rockset_kafka_collection.orders: Still creating... [10s elapsed]
time_sleep.wait_30s: Still creating... [10s elapsed]
time_sleep.wait_30s: Creation complete after 15s [id=2022-11-14T23:37:58Z]
rockset_s3_integration.integration: Creating...
rockset_s3_integration.integration: Creation complete after 0s [id=rockset-community-datasets]
rockset_s3_collection.movies: Creating...
rockset_kafka_collection.orders: Still creating... [20s elapsed]
rockset_s3_collection.movies: Still creating... [10s elapsed]
rockset_kafka_collection.orders: Still creating... [30s elapsed]
rockset_kafka_collection.orders: Creation complete after 34s [id=blog.orders]
rockset_s3_collection.movies: Still creating... [20s elapsed]
rockset_s3_collection.movies: Still creating... [30s elapsed]
rockset_s3_collection.movies: Still creating... [40s elapsed]
rockset_s3_collection.movies: Creation complete after 43s [id=blog.movies-s3]
rockset_alias.movies: Creating...
rockset_alias.movies: Creation complete after 1s [id=blog.movies]
rockset_view.english-movies: Creating...
rockset_view.english-movies: Creation complete after 1s [id=blog.english-movies]

Apply complete! Resources: 16 added, 0 changed, 0 destroyed.

Outputs:

latest-version = "0eb04bfed335946d"

So in about 1 minute it created all required resources (and 30 seconds were spent waiting for the AWS IAM role to propagate).

Updating resources

Once the initial configuration has been applied, we might have to make modifications to one or more resources, e.g. update the SQL for a query lambda. Terraform will help us plan these changes, and only apply what has changed.

SELECT
    title,
    TIME_BUCKET(
            YEARS(1),
            PARSE_TIMESTAMP('%Y-%m-%d', release_date)
        ) as year,
  popularity
FROM
    blog.movies AS m
where
    release_date != ''
  AND popularity > 11
GROUP BY
    year,
    title,
    popularity
order by
    popularity desc

We’ll also update the variables.tf file to pin the stable tag to the current version, so that the stable doesn’t change until we have properly tested it.

variable "stable_version" {
  type = string
  default = "0eb04bfed335946d"
  description = "Query Lambda version for the stable tag. If empty, the latest version is used."
}

Now we can go ahead and apply the changes.

$ terraform apply
data.rockset_account.current: Reading...
rockset_workspace.blog: Refreshing state... [id=blog]
rockset_kafka_integration.confluent: Refreshing state... [id=confluent-cloud-blog]
rockset_role.read-only: Refreshing state... [id=blog-read-only]
rockset_query_lambda.top-rated: Refreshing state... [id=blog.top-rated-movies]
rockset_kafka_collection.orders: Refreshing state... [id=blog.orders]
rockset_api_key.ql-only: Refreshing state... [id=blog-ql-only]
rockset_query_lambda_tag.stable: Refreshing state... [id=blog.top-rated-movies.stable]
data.rockset_account.current: Read complete after 1s [id=318212636800]
data.aws_iam_policy_document.rockset-trust-policy: Reading...
aws_iam_policy.rockset-s3-integration: Refreshing state... [id=arn:aws:iam::459021908517:policy/rockset-s3-integration]
aws_ssm_parameter.api-key: Refreshing state... [id=/rockset/blog/apikey]
data.aws_iam_policy_document.rockset-trust-policy: Read complete after 0s [id=2982727827]
aws_iam_role.rockset: Refreshing state... [id=rockset-s3-integration]
aws_iam_role_policy_attachment.rockset_s3_integration: Refreshing state... [id=rockset-s3-integration-20221114233744029000000001]
time_sleep.wait_30s: Refreshing state... [id=2022-11-14T23:37:58Z]
rockset_s3_integration.integration: Refreshing state... [id=rockset-community-datasets]
rockset_s3_collection.movies: Refreshing state... [id=blog.movies-s3]
rockset_alias.movies: Refreshing state... [id=blog.movies]
rockset_view.english-movies: Refreshing state... [id=blog.english-movies]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # rockset_query_lambda.top-rated will be updated in-place
  ~ resource "rockset_query_lambda" "top-rated" {
        id          = "blog.top-rated-movies"
        name        = "top-rated-movies"
      ~ version     = "0eb04bfed335946d" -> (known after apply)
        # (3 unchanged attributes hidden)

      - sql {
          - query = <<-EOT
                SELECT
                    title,
                    TIME_BUCKET(
                            YEARS(1),
                            PARSE_TIMESTAMP('%Y-%m-%d', release_date)
                        ) as year,
                  popularity
                FROM
                    blog.movies AS m
                where
                    release_date != ''
                  AND popularity > 10
                GROUP BY
                    year,
                    title,
                    popularity
                order by
                    popularity desc
            EOT -> null
        }
      + sql {
          + query = <<-EOT
                SELECT
                    title,
                    TIME_BUCKET(
                            YEARS(1),
                            PARSE_TIMESTAMP('%Y-%m-%d', release_date)
                        ) as year,
                  popularity
                FROM
                    blog.movies AS m
                where
                    release_date != ''
                  AND popularity > 11
                GROUP BY
                    year,
                    title,
                    popularity
                ORDER BY
                    popularity desc
            EOT
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

rockset_query_lambda.top-rated: Modifying... [id=blog.top-rated-movies]
rockset_query_lambda.top-rated: Modifications complete after 0s [id=blog.top-rated-movies]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

latest-version = "2e268a64224ce9b2"

As you can see it updated the query lambda version as the SQL changed.

Executing the Query Lambda

You can execute the query lambda from the command line using curl. This reads the apikey from the AWS SSM Parameter Store, and then executes the lambda using the latest tag.

$ curl --request POST \
    --url https://api.usw2a1.rockset.com/v1/orgs/self/ws/blog/lambdas/top-rated-movies/tags/latest \
  -H "Authorization: ApiKey $(aws ssm get-parameters --with-decryption --query 'Parameters[*].{Value:Value}'  --output=text --names /rockset/blog/apikey)" \
  -H 'Content-Type: application/json'

When we have verified that the query lambda returns the correct results, we can go ahead and update the stable tag to the output of the last terraform apply command.

variable "stable_version" {
  type = string
  default = "2e268a64224ce9b2"
  description = "Query Lambda version for the stable tag. If empty, the latest version is used."
}

Finally apply the changes again to update tag.

$ terraform apply
rockset_workspace.blog: Refreshing state... [id=blog]
data.rockset_account.current: Reading...
rockset_kafka_integration.confluent: Refreshing state... [id=confluent-cloud-blog]
rockset_query_lambda.top-rated: Refreshing state... [id=blog.top-rated-movies]
rockset_role.read-only: Refreshing state... [id=blog-read-only]
rockset_kafka_collection.orders: Refreshing state... [id=blog.orders]
rockset_api_key.ql-only: Refreshing state... [id=blog-ql-only]
rockset_query_lambda_tag.stable: Refreshing state... [id=blog.top-rated-movies.stable]
data.rockset_account.current: Read complete after 1s [id=318212636800]
aws_iam_policy.rockset-s3-integration: Refreshing state... [id=arn:aws:iam::459021908517:policy/rockset-s3-integration]
data.aws_iam_policy_document.rockset-trust-policy: Reading...
aws_ssm_parameter.api-key: Refreshing state... [id=/rockset/blog/apikey]
data.aws_iam_policy_document.rockset-trust-policy: Read complete after 0s [id=2982727827]
aws_iam_role.rockset: Refreshing state... [id=rockset-s3-integration]
aws_iam_role_policy_attachment.rockset_s3_integration: Refreshing state... [id=rockset-s3-integration-20221114233744029000000001]
time_sleep.wait_30s: Refreshing state... [id=2022-11-14T23:37:58Z]
rockset_s3_integration.integration: Refreshing state... [id=rockset-community-datasets]
rockset_s3_collection.movies: Refreshing state... [id=blog.movies-s3]
rockset_alias.movies: Refreshing state... [id=blog.movies]
rockset_view.english-movies: Refreshing state... [id=blog.english-movies]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # rockset_query_lambda_tag.stable will be updated in-place
  ~ resource "rockset_query_lambda_tag" "stable" {
        id           = "blog.top-rated-movies.stable"
        name         = "stable"
      ~ version      = "0eb04bfed335946d" -> "2af51ce4d09ec319"
        # (2 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

rockset_query_lambda_tag.stable: Modifying... [id=blog.top-rated-movies.stable]
rockset_query_lambda_tag.stable: Modifications complete after 1s [id=blog.top-rated-movies.stable]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

latest-version = "2e268a64224ce9b2"

Now the stable tag refers to the latest query lambda version.

GitHub Action

To make use of Infrastructure as Code, we’re going to put all terraform configurations in a git repository hosted by GitHub, and utilize the pull request workflow for terraform changes.

We will setup a GitHub action to automatically run terraform plan for each pull request, and post a comment on the PR showing the planned changes.

Once the pull request is approved and merged, it will run terraform apply to make the changes in your pull request to Rockset.

Setup

This section is a shortened version of Automate Terraform with GitHub Actions, which will talk you through all steps in much greater detail.

Save the below file as .github/workflows/terraform.yml

name: "Terraform"

on:
  push:
    branches:
      - master
  pull_request:

jobs:
  terraform:
    name: "Terraform"
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v1
        with:
          # terraform_version: 0.13.0:
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}

      - name: Terraform Format
        id: fmt
        run: terraform fmt -check
        working-directory: terraform/blog

      - name: Terraform Init
        id: init
        run: terraform init
        working-directory: terraform/blog

      - name: Terraform Validate
        id: validate
        run: terraform validate -no-color
        working-directory: terraform/blog

      - name: Terraform Plan
        id: plan
        if: github.event_name == 'pull_request'
        run: terraform plan -no-color -input=false
        working-directory: terraform/blog
        continue-on-error: true

      - uses: actions/github-script@v6
        if: github.event_name == 'pull_request'
        env:
          PLAN: "terraform\n${{ steps.plan.outputs.stdout }}"
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const output = `#### Terraform Format and Style ?\`${{ steps.fmt.outcome }}\`
            #### Terraform Initialization ⚙️\`${{ steps.init.outcome }}\`
            #### Terraform Validation ?\`${{ steps.validate.outcome }}\`
            #### Terraform Plan ?\`${{ steps.plan.outcome }}\`

            <details><summary>Show Plan</summary>

            \`\`\`\n
            ${process.env.PLAN}
            \`\`\`

            </details>

            *Pushed by: @${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: output
            })
          working-directory: terraform/blog

      - name: Terraform Plan Status
        if: steps.plan.outcome == 'failure'
        run: exit 1

      - name: Terraform Apply
        if: github.ref == 'refs/heads/master' && github.event_name == 'push'
        run: terraform apply -auto-approve -input=false
        working-directory: terraform/blog

⚠️ Note that this is a simplified setup, for a production grade configuration you should run terraform plan -out FILE and save the file, so it can be used as input to terraform apply FILE, so only the exact approved changes in the pull request are applied. More information can be found here.

Pull Request

When you create a pull request that changes the terraform config, the workflow will run terraform plan and comment on the PR, which contains the plan output.


rockset-terraform-1

This lets the reviewer see that the change can be applied, and by clicking on “Show Plan” they can see exactly what changes are going to be made.

When the PR is approved and merged into the main branch, it will trigger another GitHub action workflow run which applies the change.


rockset-terraform-2

Final words

Now we have a fully functional Infrastructure as Code setup that will deploy changes to your Rockset configuration automatically after peer review.