Skip to content

Commit 2f4a0e3

Browse files
julianwoodjulianwood
authored
Merge pull request #2062 from jorgetovar/jorgetovar-feature-s3-replication
S3 to S3 Replication in SAM
2 parents 2308353 + f19d209 commit 2f4a0e3

File tree

4 files changed

+306
-0
lines changed

4 files changed

+306
-0
lines changed

‎s3-s3-replication-sam/README.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Amazon S3 Bucket Replication
2+
3+
This pattern demonstrates how to replicate S3 bucket objects to multiple S3 buckets. Implemented with SAM.
4+
5+
**Key Benefits**:
6+
7+
- **Data Redundancy and Durability**:
8+
S3 replication ensures that your data is replicated across multiple S3 buckets, offering redundancy and safeguarding against data loss.
9+
- **Disaster Recovery**:
10+
By replicating your data to a different region or account, you establish a robust disaster recovery strategy. In the event of an outage or data corruption, you can quickly restore operations using the replicated data.
11+
- **Global Content Distribution**:
12+
Replicate frequently accessed data to different geographical locations, reducing latency and enhancing the user experience for global audiences.
13+
- **Compliance and Data Retention**:
14+
Address compliance requirements by replicating data to a separate account or region. This ensures data integrity and facilitates adherence to regulatory standards.
15+
- **Operational Efficiency**:
16+
S3 replication operates asynchronously, allowing you to focus on other tasks while data is efficiently copied in the background.
17+
18+
19+
Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/s3-s3-replication-sam](https://serverlessland.com/patterns/s3-s3-replication-sam)
20+
21+
**Important**: This application utilizes various AWS services, and there are associated costs after Free Tier usage. Please refer to the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any incurred AWS costs. No warranty is implied in this example.
22+
23+
## Requirements
24+
25+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. Ensure that the IAM user used has sufficient permissions to make necessary AWS service calls and manage resources.
26+
* [Install and configure AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
27+
* [Install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
28+
* [Install AWS Serverless Application Model (AWS SAM)](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
29+
30+
## Deployment Instructions
31+
32+
1. Create a new directory, navigate to that directory in a terminal, and clone the GitHub repository:
33+
```bash
34+
git clone https://github.com/aws-samples/serverless-patterns
35+
```
36+
1. Change the directory to the pattern directory:
37+
```bash
38+
cd s3-s3-replication-sam
39+
```
40+
1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file:
41+
```bash
42+
sam deploy --guided
43+
```
44+
1. During the prompts:
45+
* Enter a stack name.
46+
* Enter the desired AWS Region.
47+
* Allow SAM CLI to create IAM roles with the required permissions.
48+
49+
Once you have run `sam deploy --guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in the future to use these defaults.
50+
51+
1. Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs used for testing.
52+
53+
## How it Works
54+
55+
S3 replication in the same region involves copying objects from one S3 bucket to another within the same AWS region. When replication is configured, S3 automatically copies objects from the source bucket to the destination bucket, ensuring redundancy and data durability. This process helps in scenarios such as data backup, compliance requirements, and minimizing latency for accessing data.
56+
57+
58+
## Testing
59+
60+
Upload some images to the source bucket, and then check the replication bucket. You should be able to see the same data replicated there.
61+
62+
## Cleanup
63+
64+
1. Empty buckets
65+
```bash
66+
aws s3 rm s3://BUCKET_NAME_SOURCE --recursive
67+
aws s3 rm s3://BUCKET_NAME_REPLICA --recursive
68+
69+
```
70+
2. Confirm the stack has been deleted:
71+
```bash
72+
sam delete --stack-name STACK_NAME
73+
74+
```
75+
76+
----
77+
78+
Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
79+
80+
SPDX-License-Identifier: MIT-0

‎s3-s3-replication-sam/example-pattern.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"title": "Amazon S3 bucket replication",
3+
"description": "This pattern sets up replication between S3 buckets.",
4+
"language": "Python",
5+
"level": "200",
6+
"framework": "SAM",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This pattern sets up S3 replication between two buckets. The replication configuration is set up in the source bucket, and the destination bucket is set up to receive the replicated objects. The replication configuration is set up to replicate all objects in the source bucket to the destination bucket."
11+
]
12+
},
13+
"gitHub": {
14+
"template": {
15+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/s3-s3-replication-sam",
16+
"templateURL": "serverless-patterns/s3-s3-replication-sam",
17+
"projectFolder": "s3-s3-replication-sam",
18+
"templateFile": "template.yaml"
19+
}
20+
},
21+
"resources": {
22+
"bullets": [
23+
{
24+
"text": "Same Region Replication",
25+
"link": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html#srr-scenario"
26+
}
27+
]
28+
},
29+
"deploy": {
30+
"text": [
31+
"sam deploy"
32+
]
33+
},
34+
"testing": {
35+
"text": [
36+
"See the GitHub repo for detailed testing instructions."
37+
]
38+
},
39+
"cleanup": {
40+
"text": [
41+
"<code>sam delete</code>"
42+
]
43+
},
44+
"authors": [
45+
{
46+
"name": "Jorge Tovar",
47+
"image": "https://avatars.githubusercontent.com/u/21094604?v=4",
48+
"bio": "Software Architect, passionate about serverless technologies and cloud computing.",
49+
"linkedin": "jorgetovar-sa",
50+
"twitter": "https://twitter.com/jorgetovar621"
51+
}
52+
]
53+
}

‎s3-s3-replication-sam/s3-s3-replication-sam.json‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
{
2+
"title": "Amazon S3 bucket replication",
3+
"description": "This pattern sets up replication between S3 buckets.",
4+
"language": "Python",
5+
"level": "200",
6+
"framework": "SAM",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This pattern sets up S3 replication between two buckets. The replication configuration is set up in the source bucket, and the destination bucket is set up to receive the replicated objects. The replication configuration is set up to replicate all objects in the source bucket to the destination bucket."
11+
]
12+
},
13+
"gitHub": {
14+
"template": {
15+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/s3-s3-replication-sam",
16+
"templateURL": "serverless-patterns/s3-s3-replication-sam",
17+
"projectFolder": "s3-s3-replication-sam",
18+
"templateFile": "template.yaml"
19+
}
20+
},
21+
"resources": {
22+
"bullets": [
23+
{
24+
"text": "Same Region Replication",
25+
"link": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html#srr-scenario"
26+
}
27+
]
28+
},
29+
"deploy": {
30+
"text": [
31+
"sam deploy"
32+
]
33+
},
34+
"testing": {
35+
"text": [
36+
"See the GitHub repo for detailed testing instructions."
37+
]
38+
},
39+
"cleanup": {
40+
"text": [
41+
"<code>sam delete</code>"
42+
]
43+
},
44+
"authors": [
45+
{
46+
"name": "Jorge Tovar",
47+
"image": "https://avatars.githubusercontent.com/u/21094604?v=4",
48+
"bio": "Software Architect, passionate about serverless technologies and cloud computing.",
49+
"linkedin": "jorgetovar-sa",
50+
"twitter": "https://twitter.com/jorgetovar621"
51+
}
52+
],
53+
"patternArch": {
54+
"icon1": {
55+
"x": 20,
56+
"y": 50,
57+
"service": "s3",
58+
"label": "Source S3 bucket"
59+
},
60+
"icon2": {
61+
"x": 80,
62+
"y": 50,
63+
"service": "s3",
64+
"label": "Destination S3 bucket"
65+
},
66+
"line1": {
67+
"from": "icon1",
68+
"to": "icon2",
69+
"label": "Replication"
70+
}
71+
}
72+
}

‎s3-s3-replication-sam/template.yaml‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
AWSTemplateFormatVersion: '2010-09-09'
2+
Transform: 'AWS::Serverless-2016-10-31'
3+
Description: 'S3 bucket replication'
4+
5+
Parameters:
6+
7+
BucketSourceName:
8+
Description: "Bucket Source Name (must be unique)"
9+
Type: String
10+
11+
BucketReplicaName:
12+
Description: "Bucket Replica Name (must be unique)"
13+
Type: String
14+
15+
Resources:
16+
17+
ReplicationRole:
18+
Type: 'AWS::IAM::Role'
19+
Properties:
20+
AssumeRolePolicyDocument:
21+
Version: '2012-10-17'
22+
Statement:
23+
- Effect: 'Allow'
24+
Principal:
25+
Service: 's3.amazonaws.com'
26+
Action: 'sts:AssumeRole'
27+
Policies:
28+
- PolicyName: 'ReplicationPolicy'
29+
PolicyDocument:
30+
Version: '2012-10-17'
31+
Statement:
32+
- Effect: 'Allow'
33+
Action:
34+
- 's3:GetBucketVersioning'
35+
- 's3:ListBucket'
36+
- s3:GetReplicationConfiguration
37+
- s3:GetObjectVersionForReplication
38+
- s3:GetObjectVersionAcl
39+
- s3:GetObjectVersionTagging
40+
- s3:GetObjectRetention
41+
- s3:GetObjectLegalHold
42+
Resource:
43+
- !Sub "arn:aws:s3:::${BucketSource}"
44+
- !Sub "arn:aws:s3:::${BucketReplica}"
45+
- Effect: 'Allow'
46+
Action:
47+
- 's3:ReplicateObject'
48+
- 's3:ReplicateDelete'
49+
- 's3:ReplicateTags'
50+
- 's3:GetObjectVersionTagging'
51+
- 's3:ObjectOwnerOverrideToBucketOwner'
52+
Resource:
53+
- !Sub "arn:aws:s3:::${BucketSource}"
54+
- !Sub "arn:aws:s3:::${BucketReplica}"
55+
56+
57+
BucketSource:
58+
Type: 'AWS::S3::Bucket'
59+
Properties:
60+
VersioningConfiguration:
61+
Status: 'Enabled'
62+
ReplicationConfiguration:
63+
Role: !GetAtt ReplicationRole.Arn
64+
Rules:
65+
- Destination:
66+
Bucket: !GetAtt BucketReplica.Arn
67+
Prefix: ''
68+
Status: 'Enabled'
69+
BucketName: !Ref BucketSourceName
70+
71+
BucketReplica:
72+
Type: 'AWS::S3::Bucket'
73+
Properties:
74+
VersioningConfiguration:
75+
Status: 'Enabled'
76+
BucketName: !Ref BucketReplicaName
77+
78+
BucketReplicaPolicy:
79+
Type: AWS::S3::BucketPolicy
80+
Properties:
81+
Bucket: !Ref BucketReplica
82+
PolicyDocument:
83+
Version: 2012-10-17
84+
Statement:
85+
- Sid: "Object Level Permissions"
86+
Effect: "Allow"
87+
Principal:
88+
AWS: !GetAtt ReplicationRole.Arn
89+
Action:
90+
- "s3:ReplicateObject"
91+
- "s3:ReplicateDelete"
92+
Resource: !Sub "arn:aws:s3:::${BucketReplica}/*"
93+
- Sid: "Bucket Level Permissions"
94+
Effect: "Allow"
95+
Principal:
96+
AWS: !GetAtt ReplicationRole.Arn
97+
Action:
98+
- "s3:List*"
99+
- "s3:GetBucketVersioning"
100+
- "s3:PutBucketVersioning"
101+
Resource: !Sub "arn:aws:s3:::${BucketReplica}"

0 commit comments

Comments
 (0)