Table for locking

Without a lock table, two engineers (or two CI/CD pipelines) can run terraform apply at the same time, both reading the same state, and then both writing conflicting results back. It can corrupt your state file silently, leaving your real infrastructure, and the recorded state, permanently out of sync and you rethinking your life choices.

Adding a DynamoDB lock table

resource "aws_dynamodb_table" "tf_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

terraform {
  backend "s3" {
    bucket         = "my-tf-state"
    key            = "terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-lock"
    encrypt        = true
  }
}

Versioning S3 bucket

Your state file is not code, it's a live record of everything Terraform believes about your infrastructure (that's a beautiful quote). Without versioning, a bad apply, a manual edit or an accidental terraform state rm overwrites the state with no way back. Enabling versioning keeps a full history of every state revision, turning recovery from a crisis into a five-minute rollback instead of a life of suffering.

Enabling versioning on the state bucket

resource "aws_s3_bucket" "tf_state" {
  bucket = "my-tf-state"
}

resource "aws_s3_bucket_versioning" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id

  rule {
    id     = "expire-old-versions"
    status = "Enabled"
    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }
}

It's worth having a lifecycle rule to keep storage costs in check, in cases where there are a lot of changes.

Enabling cross-region bucket replication

While it's rare for an AWS region to go down, when it does, your S3 bucket goes with it. If your state file is only in one region, you can't run Terraform at all during the outage, even to fix the very infrastructure that's affected. Cross-region replication keeps a live copy of every state version in a second region so your operations don't have a single geographic point of failure. Also, for the paranoid ones, if you want to add to the already humongous durability of S3 objects, having another region is a way to do that.

Replicating state to a second region

resource "aws_s3_bucket" "tf_state_replica" {
  bucket   = "my-tf-state-replica"
  provider = aws.us_west_2 # Secondary region
}

resource "aws_s3_bucket_versioning" "tf_state_replica" {
  bucket   = aws_s3_bucket.tf_state_replica.id
  provider = aws.us_west_2

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_replication_configuration" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  role   = aws_iam_role.replication.arn

  rule {
    id     = "replicate-state"
    status = "Enabled"
    destination {
      bucket             = aws_s3_bucket.tf_state_replica.arn
      storage_class      = "STANDARD"
      replica_kms_key_id = aws_kms_key.replica.arn
    }
  }
}

Replication requires versioning enabled on both source and destination buckets.

Wrap-up

These three settings are pretty quick to set up and they help mitigating a class of incidents that make software engineers wake up at 3 am. So, ensure you're locking, versioning and replicating your state - sleep tight.