Terraform — Cross Account Replication(CAR) of S3 buckets

In today’s cloud-centric world, businesses often operate across multiple AWS accounts for various reasons, such as security, compliance, and cost management. However, managing data replication between S3 buckets across different AWS accounts can be challenging. Amazon S3 provides a solution called Cross-Account Replication (CAR), allowing users to replicate objects between S3 buckets in different AWS accounts.

In this article, we’ll explore how to set up Cross-Account Replication for S3 buckets using Terraform, an infrastructure as a code tool.

Understanding Cross-Account Replication (CAR) in Amazon S3

Cross-Account Replication (CAR) in Amazon S3 enables automatic, asynchronous replication of objects from a source bucket in one AWS account to a destination bucket in another AWS account. This feature helps in scenarios where organizations need to share data securely across accounts while maintaining control and compliance.

Prerequisites

Before you begin, make sure you have the following prerequisites:

1. An AWS account with the necessary permissions to create an S3 bucket and configure the S3 replication rule.
2. Terraform should be installed on the machine. If Terraform does not exist you can download and install it from here.

So, let’s start!

1. Create a “provider.tf”

The provider file tells Terraform which provider you are using.

We are using provider “aws” because here we will deploy the script on AWS.

provider "aws" {
  alias   = "source"
  region  = "${var.source_region}"
  profile = "<source-profile-name>"
}

provider "aws" {
  alias   = "dest"
  region  = "${var.dest_region}"
  profile = "<destination-profile-name>"
}

2. Create “variables.tf”

All variables will be in this file.

variable "source_region" {
  description = "AWS Deployment region.."
  default = "us-east-1"
}

variable "dest_region" {
  description = "AWS Deployment region.."
  default = "us-east-1"
}

variable "source_bucket_name" {
  description = "Your Source Bucket Name"
  default = "source"
}

variable "dest_bucket_name" {
  description = "Your Destination Bucket Name"
  default = "destination"
}

If you are using terraform.tfvars you just need to add a description only.

3. Create a “terraform.tfvars”

To persist variable values, create a file, and assign variables within this file. Create a file named terraform.tfvars with the following contents:

source_region = "us-east-2"
dest_region   = "us-east-2"

dest_bucket_name   = "sync-source-bucket"
source_bucket_name = "sync-dest-bucket"

For all files that match terraform.tfvars or *.auto.tfvars present in the current directory, Terraform automatically loads them to populate variables. If the file is named something else, you can use the -var-file flag directly to specify a file.

I don’t recommend saving usernames and passwords to version control, but you can create a local secret variables file and use -var-file to load it.

4. Create a “modules > s3-cross-account-replicartion” Folder

A module is a container for multiple resources that are used together. Modules can be used to create lightweight abstractions, so that you can describe your infrastructure in terms of its architecture, rather than directly in terms of physical objects.

5. Create “Main.tf” in the s3-cross-account-replicartion folder.

This Terraform script does the following:

• Create an S3 bucket on both AWS Accounts.
• Enable S3 versioning on both S3 buckets.
• Create an IAM role/policy on both AWS Accounts.
• Create an S3 replication rule for both buckets so It will sync the both directions. (if you want to sync in one single direction you can remove aws_s3_bucket_replication_configuration it for the destination bucket.

main.tf file of the ca_replication module

6. Create “variables.tf” in the ca_replication folder.

This is the same as the above variable.tf file just declares all variables that we are using in main.tf a file so we can get all variable's values from production.tf file.

7. Create “output.tf” in the s3-cross-account-replicartion folder.

We can export any details from created resources and give that as input for another module.

output "source_bucket_arn" {
  value = aws_s3_bucket.source.arn
}

output "dest_bucket_arn" {
  value = aws_s3_bucket.destination.arn
}

We can access output value in another submodule like

source_bucket_arn   = "${module.ca_replication.source_bucket_arn}"

8. Create “production.tf”

production.tf files in your working directory when you run terraform plan or terraform apply together from the root module. That module may call other modules and connect them by passing output values from one to the input values of another. To learn how to use modules, see the Modules Configuration section.

module "s3-cross-account-replication" {
  source = "./modules/s3-cross-account-replicartion"

  source_bucket_name = "sync-source-bucket"
  source_region      = "us-west-1"
  dest_bucket_name   = "sync-dest-bucket"
  dest_region        = "us-east-1"

  providers = {
    aws.source = aws.source
    aws.dest   = aws.dest
  }
}

Now, We are ready to init!

Run terraform init that downloads all module information and downloads terraform in your project file.

terraform init

After that, you can see the .terraform folder in your project directory that contains terraform setup and modules information.

terraform plan

The terraform plan a command is used to create an execution plan. Terraform performs a refresh, unless explicitly disabled, and then determines what actions are necessary to achieve the desired state specified in the configuration files.

This command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or the state. For example, terraform plan might be run before committing a change to version control, to create confidence that it will behave as expected.

terraform plan or apply output

terraform apply

The terraform apply a command is used to apply the changes required to reach the desired state of the configuration or the pre-determined set of actions generated by an terraform plan execution plan.

🎊 🎉🤖🎊 🎉 Our Cross-Account Replication Setup is ready on AWS.

You just need to follow all the steps or clone this repository to start terraforming.

GitHub - LazziBear/tf-aws-s3-cross-account-replication: aws s3 cross-account replication terraform…

After cloning the repo, just run the following commands.

rename sample.terraform.tfvars to terraform.tfvars

change values of variables in terraform.tfvars.

terraform init

terraform plan

terraform apply

Test Cross-Account Replication

Upload a test object to the source bucket and verify that it gets replicated to the destination bucket in the other AWS account.

Thank you for reading, if you have anything to add please send a response or add a note!