Sometimes you need objects from one region into another region. A common use case is centralizing logs from multiple regions for data analytics. Amazon Simple Storage Storage (S3) is a service that stores objects by means of a key and a value. The data associated with a key can easily be accessed by means of a web service API. S3 is a core service of AWS and is perfectly suited for storing log data. S3 stores data locally in a region. Data is never copied to another region without explicit configuration. Lets take a look what is involved setting up a cross region copy between two S3 buckets.
Cross Region Replication (CRR)
Cross Region Replication (CRR) is a service that automatically copies objects from one region to another. The service is part of S3 and can be enabled by means of configuration. Data is then securely copied from one region to another and stored in another bucket.
The CRR example consists of two buckets, one in us-east-1 and one is eu-west-1. The bucket in us-east-1 contains content that must be replicated to eu-west-1.
The example project consists of two CloudFormation templates, that configures buckets in separate regions. For us-east-1 a bucket needs to be created that has
ReplicationConfiguration enabled. The bucket configures the Amazon Resource Name (ARN) of the eu-west-1 bucket to replicate as the
Destination. The bucket also defines a Role – a security configuration – that S3 assumes to get permissions to write to the eu-west-1 bucket.
ReplicatedBucket: Type: AWS::S3::Bucket Properties: AccessControl: BucketOwnerFullControl VersioningConfiguration: Status: Enabled ReplicationConfiguration: Role: !GetAtt ReplicationRole.Arn Rules: - Prefix: '' Status: Enabled Destination: Bucket: !Ref ReplicatedBucketArn ReplicationRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: s3.amazonaws.com Action: sts:AssumeRole Path: / Policies: - PolicyName: Allow PolicyDocument: Statement: - Effect: Allow Action: - s3:* Resource: - '*'
The eu-west-1 bucket only has an enabled
SimpleBucket: Type: AWS::S3::Bucket Properties: AccessControl: BucketOwnerFullControl VersioningConfiguration: Status: Enabled
To deploy the example type
make deploy. To get information about the bucket names type
To replicate data, we need to copy data to us-east-1. To get the bucket name type
make info that shows the
BucketName for both regions. To copy data to us-east-1 we need the name of that bucket and type:
$ aws s3 cp aws s3 cp LICENSE s3://cross-region-replication-example-eu-simplebucket-1adlmia5ye8ji
After a second the file is replicated in eu-west-1:
$ aws s3 ls s3://cross-region-replication-example-eu-simplebucket-8eeig5mggnah 2018-11-14 05:19:50 11357 LICENSE
It is easy to configure S3 Cross Region Replication (CRR). Versioning has to be enabled and the bucket needs permissions to access the bucket in another region. With CRR we can copy log like access logs, error logs and so on to a central location for data analytics. Next time we’ll look at AWS Athena to do simple log analytics on S3 buckets.