How to Move a Terraform State File Between AWS Accounts

Key Takeaways

• Why Move State?: Common reasons include consolidating AWS accounts, separating environments for security or compliance, changing project ownership, or improving access control.
• Core Process: The migration is a three-step manual process: pull the current state to your local machine, push it to the new S3 bucket in the target account, and re-initialize Terraform to recognize the new remote backend.
• Prerequisites are Critical: Before starting, you must have correct IAM permissions for both the source and destination AWS accounts and back up your existing state file to prevent data loss.
• Update Your Configuration: You must update your Terraform backend configuration block (backend.tf) to point to the new S3 bucket and DynamoDB table before re-initializing.
• Verify After Moving: After the move, always run a terraform plan to confirm that Terraform sees no changes to your infrastructure, ensuring the migration was successful.

Terraform State is a way to track and manage your Terraform configurations and resources. It stores your Terraform configurations in a centralized location and makes it easy to query and visualize your state.

What is Terraform State?

A Terraform state file is a JSON file that acts as a blueprint of the infrastructure you manage. Terraform creates this file to map the real-world resources defined in your configuration files (like AWS EC2 instances or S3 buckets) to your code. It records metadata and dependencies, allowing Terraform to know what it is responsible for managing, what changes to make, and how to destroy resources correctly. For a deeper dive, explore our guide on Core Terraform Concepts.

Why Do you need to move Terraform State files?

Here are some key reasons why you might want to move the state files.

You would move a Terraform state file to a new AWS account to reflect changes in your organizational structure, security posture, or billing strategy. This operation is essential for maintaining control and accurate tracking of your infrastructure as your cloud environment evolves.

Key reasons include:

• Account Consolidation: Merging infrastructure from multiple AWS accounts into a single one to streamline management and reduce overhead.
• Change of Ownership: Transferring control of an application and its underlying infrastructure from one team or business unit to another that uses a different AWS account.
• Enhanced Security: Moving state to a more secure, isolated AWS account with stricter IAM policies and access controls, particularly for production environments.
• Compliance Requirements: Adhering to regulatory standards (like GDPR or HIPAA) that may mandate specific resources or data reside in a dedicated, compliant account.

In general, moving Terraform state between AWS accounts can be useful in situations where you need to transfer ownership, enforce compliance requirements, tighten security controls, or consolidate accounts. However, it is important to carefully plan and execute the migration to avoid any disruption to existing infrastructure.

How does Terraform state work?

When you run a Terraform command, the state file is updated to reflect your changes. The state is stored in a file on your local machine. You can use the state file to reconstruct the state of your configuration at any point in time.
Consequently, the remote state data source needs to have parameters added to its config block which specify a suitable IAM role that may be assumed in the account containing the state file; for example:

data "terraform_remote_state" "example" {
  backend = "s3"

  config = {
    bucket         = var.my_state_bucket1
    dynamodb_table = var.tf_state_dynamo
    region         = var.region
    key            = var.tfstate_key
    role_arn       = data.aws_ssm_parameter.cross_account_role.value
    external_id    = data.aws_ssm_parameter.external_id.value
  }
}

data "terraform_remote_state" "example" {
  backend = "s3"

  config = {
    bucket         = var.my_state_bucket1
    dynamodb_table = var.tf_state_dynamo
    region         = var.region
    key            = var.tfstate_key
    role_arn       = data.aws_ssm_parameter.cross_account_role.value
    external_id    = data.aws_ssm_parameter.external_id.value
  }
}

Note that where a project that acts as the provider of remote state data has its state file moved to a different account, it is important to identify all other projects that act as consumers, and if their state files are not being moved at the same time, the above modification will need to be made to the terraform_remote_state data sources.

Alternatively, it may be preferable to refactor the code in the impacted projects to use the SSM Parameter Store, which is now the preferred method of sharing Terraform outputs between projects.

What Are the Prerequisites and Risks?

Before you begin the migration, you must ensure you have the necessary permissions and understand the potential risks. Failure to prepare can lead to state corruption or accidental infrastructure changes.

Prerequisites:

1. IAM Permissions: You need programmatic access with sufficient IAM permissions for both the source and destination AWS accounts. This includes s3:GetObject for the original bucket and s3:PutObject for the new bucket.
2. AWS CLI & Terraform CLI: Ensure both tools are installed and configured on your local machine.
3. Backup: Always back up your current state file. You can do this by simply downloading it from the S3 bucket.

Risks:

• State Lock Interruption: If multiple team members are working, ensure you have a maintenance window. A state lock from the old location will not protect the new one during the move.
• State Drift: If any infrastructure changes occur while you are migrating the state, the final state file may not match reality. This is why it’s crucial to perform the move during a period of no planned changes. After moving, you can always run a Terraform Plan to verify that no infrastructure changes are proposed.

What Is the Manual Process to Move Terraform State?

The manual process involves using the AWS and Terraform command-line tools to pull the state file locally and then push it to the new remote location. This method gives you full control over the migration.

Note that I use AWS-VAULT to query my AWS infrastructure, you can find out how to set this up here

Step 1 – Pull the Terraform State

Pull the remote state file into your local state (Note, the following example is pulling the state file from the non-prod account)

# Ensure your AWS credentials point to the SOURCE account
terraform state pull > terraform.tfstate

# Ensure your AWS credentials point to the SOURCE account
terraform state pull > terraform.tfstate

Step 2 – Push to a new S3 bucket

Once the file is pulled down locally, it can be uploaded to the s3 path you desire (Make sure after the file has been pulled that your backend.conf within your tf code has been updated to where you will upload the state file. (Note, the following example is uploading the state file to an s3 path within the AWS account)

# Ensure your AWS credentials point to the DESTINATION account
aws s3 cp terraform.tfstate s3://my-new-state-bucket-in-target-account/global/s3/terraform.tfstate

# Ensure your AWS credentials point to the DESTINATION account
aws s3 cp terraform.tfstate s3://my-new-state-bucket-in-target-account/global/s3/terraform.tfstate

Step 3 – Run Terraform init

Lastly, run a terraform init to confirm the new state file can be remotely initialised (Terraform Init Example)

# Credentials can point to the DESTINATION account
terraform init -reconfigure

# Credentials can point to the DESTINATION account
terraform init -reconfigure

With Terraform, you can easily manage your state and keep your infrastructure up-to-date. Whether you need to push a new version of your software to all your servers or just update a single one, Terraform simplifies the process.

Want to know more about Terraform? Check out our other articles:

Learn Core Terraform Concepts

Terraform Core Commands

Learn out Terraform Plan -Out