Input validation – claim back your time from Terraform!

Cloud Survey 2021

Benchmark your organization against competitors and other industries. Share your experiences and receive the free Cloud report.

Hiring

We are Binx. We make every organization cloud-native.

I’ve been using Terraform for over two years now, and from the start, I’ve found the need for proper input validation. Although Terraform has added variable types with HCL 2.0 and even input variable validation rules in Terraform CLI v0.13.0, it still appears challenging.

Why input validation?

There are several reasons why input validation is a good idea, but the most important one is saving time!
It has often happened that a plan output looked good to me, and Terraform itself found no errors, but when I ran apply, the code would still break. Not all providers seem to tell you in advance that a specific combination of settings is not correct. It can also happen that the resource’s name does not validate, but this won’t detect during the plan stage. Especially when deploying GKE clusters or services like Composer on Google Cloud, you could be waiting for more than 30 minutes before these kinds of errors pop up. And then you can wait another 30 minutes for the resources to be deleted again.
You see how being notified about these setting errors in advance can help you save a lot of time.

Using custom assertions

One of the first things we implemented in our custom modules is the use of assertions. Making sure Terraform would show a proper error message came with a challenge, as there is no predefined function for this. On top of that, adding custom checks to break out of the code is not supported by HCL.

example:

# variables.tf
variable "prefix" {
  description = "Company naming prefix, ensures uniqueness of bucket names"
  type        = string
  default     = "binx"
}
variable "project" {
  description = "Company project name."
  type        = string
  default     = "blog"
}
variable "environment" {
  description = "Company environment for which the resources are created (e.g. dev, tst, acc, prd, all)."
  type        = string
  default     = "dev"
}
variable "purpose" {
  description = "Bucket purpose, will be used as part of the bucket name"
  type        = string
}

# main.tf
locals {
  bucket_name = format("%s-%s-%s-%s", var.prefix, var.project, var.environment, var.purpose)
}

resource "google_storage_bucket" "demo" {
  provider = google-beta
  name     = local.bucket_name
}

When you run terraform plan on this example code, and enter a purpose with invalid chars, you will get the following output:

var.purpose
  Bucket purpose, will be used as part of the bucket name
  Enter a value: test^&asd

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_storage_bucket.demo will be created
  + resource "google_storage_bucket" "demo" {
      + bucket_policy_only          = (known after apply)
      + force_destroy               = false
      + id                          = (known after apply)
      + location                    = "US"
      + name                        = "binx-blog-dev-test^&asd"
      + project                     = (known after apply)
      + self_link                   = (known after apply)
      + storage_class               = "STANDARD"
      + uniform_bucket_level_access = (known after apply)
      + url                         = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

As you can see, Terraform does not check the characters in the bucket name and will let you think everything is fine. However, if you try to apply this code, you will be given an error during the bucket creation telling you the name is invalid:

google_storage_bucket.demo: Creating...
╷
│ Error: googleapi: Error 400: Invalid bucket name: 'binx-blog-dev-test^&asd', invalid
│
│   with google_storage_bucket.demo,
│   on test.tf line 30, in resource "google_storage_bucket" "demo":
│   30: resource "google_storage_bucket" "demo" {
│
╵

We really want to detect this during the plan phase to tell the developer to fix his code before merging it into master for rollout.
So how did we solve this problem?

It appears that we can abuse the file function to show a proper error message when our checks fail. The nice thing about this function is that it will show you an error message if the file you’re trying to load does not exist. We can use this to our advantage by providing an error description as the file name.

example:

# main.tf
locals {
  regex_bucket_name = "(([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])\\.)*([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])" # See https://cloud.google.com/storage/docs/naming-buckets

  bucket_name       = format("%s-%s-%s-%s", var.prefix, var.project, var.environment, var.purpose)
  bucket_name_check = length(regexall("^${local.regex_bucket_name}$", local.bucket_name)) == 0 ? file(format("Bucket [%s]'s generated name [%s] does not match regex ^%s$", var.purpose, local.bucket_name, local.regex_bucket_name)) : "ok"
}

resource "google_storage_bucket" "demo" {
  provider = google-beta
  name     = local.bucket_name
}

Using the example above, when we run Terraform plan with the same purpose input as before, we’ll get the following message:

│ Error: Invalid function argument
│
│   on main.tf line 5, in locals:
│    5:   bucket_name_check = length(regexall("^${local.regex_bucket_name}$", local.bucket_name)) == 0 ? file(format("Bucket [%s]'s generated name [%s] does not match regex ^%s$", var.purpose, local.bucket_name, local.regex_bucket_name)) : "ok"
│     ├────────────────
│     │ local.bucket_name is "binx-blog-dev-test^&asd"
│     │ local.regex_bucket_name is "(([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])\\.)*([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])"
│     │ var.purpose is "test^&asd"
│
│ Invalid value for "path" parameter: no file exists at Bucket [test^&asd]'s generated name [binx-blog-dev-test^&asd] does not match regex ^(([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])$; this function works only with files that are distributed as part of the configuration source code, so if this file will be created by a resource in this configuration you must instead obtain this result
│ from an attribute of that resource.

As you can see, Terraform will try to load a file named:

Bucket [test^&asdf]'s generated name [lot-blog-dev-test^&asdf] does not match regex:
^(([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])$

This filename is actually the error message we want to show. Obviously, this file does not exist, so it will fail and display this message on the console.

Nice! So we have a way to tell the developer in advance that the provided input is not valid. But how can we make the message even more apparent? We can use newlines!

Check out the following example:

# main.tf
locals {
  regex_bucket_name = "(([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])\\.)*([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])" # See https://cloud.google.com/storage/docs/naming-buckets

  assert_head = "\n\n-------------------------- /!\\ CUSTOM ASSERTION FAILED /!\\ --------------------------\n\n"
  assert_foot = "\n\n-------------------------- /!\\ ^^^^^^^^^^^^^^^^^^^^^^^ /!\\ --------------------------\n\n"

  bucket_name       = format("%s-%s-%s-%s", var.prefix, var.project, var.environment, var.purpose)
  bucket_name_check = length(regexall("^${local.regex_bucket_name}$", local.bucket_name)) == 0 ? file(format("%sBucket [%s]'s generated name [%s] does not match regex:\n^%s$%s", local.assert_head, var.purpose, local.bucket_name, local.regex_bucket_name, local.assert_foot)) : "ok"
}

resource "google_storage_bucket" "demo" {
  provider = google-beta
  name     = local.bucket_name
}

We will now get the following error output:

│ Error: Invalid function argument
│
│   on main.tf line 8, in locals:
│    8:   bucket_name_check = length(regexall("^${local.regex_bucket_name}$", local.bucket_name)) == 0 ? file(format("%sBucket [%s]'s generated name [%s] does not match regex:\n^%s$%s", local.assert_head, var.purpose, local.bucket_name, local.regex_bucket_name, local.assert_foot)) : "ok"
│     ├────────────────
│     │ local.assert_foot is "\n\n-------------------------- /!\\ ^^^^^^^^^^^^^^^^^^^^^^^ /!\\ --------------------------\n\n"
│     │ local.assert_head is "\n\n-------------------------- /!\\ CUSTOM ASSERTION FAILED /!\\ --------------------------\n\n"
│     │ local.bucket_name is "binx-blog-dev-test^&asd"
│     │ local.regex_bucket_name is "(([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])\\.)*([a-z0-9]|[a-z0-9][a-z0-9\\-]*[a-z0-9])"
│     │ var.purpose is "test^&asd"
│
│ Invalid value for "path" parameter: no file exists at
│
│ -------------------------- /!\ CUSTOM ASSERTION FAILED /!\ --------------------------
│
│ Bucket [test^&asd]'s generated name [binx-blog-dev-test^&asd] does not match regex:
│ ^(([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])$
│
│ -------------------------- /!\ ^^^^^^^^^^^^^^^^^^^^^^^ /!\ --------------------------
│
│ ; this function works only with files that are distributed as part of the configuration source code, so if this file will be created by a resource in this configuration you must instead obtain this result from an attribute of that resource.

Conlusion

As you can see, using the file function to generate assertions opens up many possibilities when it comes to input validation. You can see even more examples on how to use this in our own built Terraform modules found on the Terraform Registry. Part of each of these modules is an asserts.tf file, which defines assertions using the file method as described above.

I still think it would be nice if Terraform would create an actual function for us that we can use to generate custom errors. The workaround I showed in this blog post can easily confuse you, as Terraform tells you no file exists while you’re not trying to actually read any files.

Share this article: Tweet this post / Post on LinkedIn