Skip to content

Commit dca61cd

Browse files
julianwoodjulianwood
authored
Merge pull request #2602 from anuprajpara/anuprajpara-feature-eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication
New serverless pattern - feature eventbridge lambda fsx openzfs cross account cross region periodic replication
2 parents dd7e665 + 33706ba commit dca61cd

File tree

8 files changed

+889
-0
lines changed

8 files changed

+889
-0
lines changed

‎eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication/ArchDiagram.png‎

275 KB
Loading

‎eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication/README.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Cross-account cross-region replication for FSx for OpenZFS volumes with AWS Lambda
2+
Amazon EventBridge triggers an AWS Lambda function to replicate FSx for OpenZFS volumes across file systems located in the same account and region, or across different accounts and regions.
3+
4+
![Architecture diagram](./ArchDiagram.png)
5+
6+
The [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) template deploys an Amazon EventBridge Scheduler to trigger an AWS Lambda function based on a user-defined schedule. This function copies the snapshot of a volume and transfers it to the target FSx system, which can be located in the same or a different AWS account and/or region.
7+
8+
For FSx for OpenZFS periodic volume replication in same account and same region, please refer to the Serverless Land Pattern <https://serverlessland.com/patterns/eventbridge-lambda-fsx-openzfs-periodic-replication>
9+
10+
The template contains a sample Lambda function that creates a snapshot of the source FSx Volume ID. Once the snapshot becomes available, it invokes another Lambda function in the destination AWS account and/or region, which initiates the replication by calling the copy_snapshot_and_update_volume API. This solution also notifies users via an Amazon SNS topic of any errors and snapshot creation details.
11+
12+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
13+
14+
## Requirements
15+
16+
* [Create two AWS accounts for cross account setup](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have, create them and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
17+
18+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configure two profiles with credentials for the individual accounts as below:
19+
20+
```
21+
[default]
22+
[crossaccount]
23+
```
24+
25+
26+
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
27+
- [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
28+
- [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed
29+
- Make sure that you have the ID of the source and destination volumes that you would like to initiate the replication between. For more information on these resources, see [Creating FSx for OpenZFS file systems](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/creating-file-systems.html), [Creating a volume](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/creating-volumes.html), [Creating a snapshot](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/snapshots-openzfs.html#creating-snapshots), and [Using on-demand data replication](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/on-demand-replication.html#how-to-use-data-replication).
30+
31+
## Deployment Instructions
32+
33+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
34+
```
35+
git clone https://github.com/aws-samples/serverless-patterns
36+
```
37+
2. Change directory to the pattern directory:
38+
```
39+
cd eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication
40+
```
41+
3. From the command line, use the AWS SAM command listed below to deploy the AWS resources in the destination AWS account as specified in the destination-template.yaml file. Note that an AWS CLI profile named crossaccount must be configured with AWS credentials for the destination/target AWS account.
42+
```
43+
sam deploy --guided --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM -t destination-template.yaml --profile crossaccount
44+
```
45+
4. During the prompts:
46+
- Enter a target stack name
47+
- Enter the desired AWS Region
48+
- Enter a TargetVolumeID
49+
- Enter a CopySnapshotAndUpdateVolume - "Options" parameter. Comma (,) separated values
50+
- Enter a CopySnapshotAndUpdateVolume - "CopyStrategy" parameter (Default = INCREMENTAL_COPY)
51+
- Enter source AWS account Id
52+
- Allow SAM CLI to create IAM roles with the required permissions.
53+
- Save arguments to configuration file [Y/n]: N
54+
5. Once the above stack is deployed in the target account/region, use the AWS SAM command listed below to deploy the resources in the source AWS account using the source-template.yaml file.
55+
```
56+
sam deploy --guided --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM -t source-template.yaml
57+
```
58+
6. During the prompts:
59+
- Enter a source stack name
60+
- Enter the desired AWS Region
61+
- Enter a SourceVolumeID
62+
- Enter a CRON schedule for snapshots (Default = [0 0/6 * * ? *] every six hours)
63+
- Enter a value of snapshot Name (Default = fsx_scheduled_snapshot)
64+
- Enter an Email for notifications
65+
- Allow Success Notification (Default = Yes)
66+
- Enter number of days to retain custom-scheduled snapshots (Default = 7 days)
67+
- Enter target AWS account Id
68+
- Enter target region
69+
- Enter target stack name used previously
70+
- Save arguments to configuration file [Y/n]: N
71+
7. Note the outputs from the previous SAM deploy commands. These contain the resource names and/or ARNs which will be used for later review.
72+
73+
## How it works
74+
75+
This pattern sets up the following resources:
76+
77+
- An Amazon EventBridge Scheduler that triggers a Lambda function based on the schedule defined by you to create snapshots of the provided FSx Source Volume ID.
78+
- A sample [Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) function that creates snapshots of the source FSx Volume ID and invokes another Lambda function in the destination AWS account and/or region, which will initiate the replication by calling the [CopySnapshotAndUpdateVolume](https://docs.aws.amazon.com/fsx/latest/APIReference/API_CopySnapshotAndUpdateVolume.html) API.
79+
- The function also deletes the older snapshots in the source and target AWS account/region based on the configured retention period.
80+
- An SNS topic that sends notifications for any success or failure events while creating or replicating snapshots.
81+
82+
## Testing
83+
1. Based on the provided schedule, monitor the CloudWatch logs and the FSx snapshots that are created.
84+
2. The Lambda function will send various success and failure notifications to the configured email address via an SNS topic.
85+
86+
## Cleanup
87+
1. Change directory to the pattern directory:
88+
```
89+
cd serverless-patterns/eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication
90+
```
91+
2. Delete resources from the source account.
92+
```
93+
sam delete --stack-name <source-stack-name>
94+
```
95+
96+
3. Delete resources from the destination/target account.
97+
```
98+
sam delete --stack-name <target-stack stackname> --profile crossaccount --config-env crossacct
99+
``````
100+
3. During the prompts:
101+
* Enter all details as per requirement.
102+
103+
---
104+
105+
Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved.
106+
107+
SPDX-License-Identifier: MIT-0
108+

‎eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication/destination-template.yaml‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
AWSTemplateFormatVersion: "2010-09-09"
2+
3+
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
4+
#
5+
# Permission is hereby granted, free of charge, to any person obtaining a copy of
6+
# this software and associated documentation files (the "Software"), to deal in
7+
# the Software without restriction, including without limitation the rights to
8+
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
9+
# the Software, and to permit persons to whom the Software is furnished to do so.
10+
#
11+
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
12+
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
13+
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
14+
# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
15+
# IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
16+
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
17+
18+
Transform: AWS::Serverless-2016-10-31
19+
Description: >
20+
Configure periodic replication schedule for your Amazon FSx for OpenZFS file system volumes.
21+
22+
Parameters:
23+
TargetVolumeID:
24+
Description: Amazon FSx for OpenZFS Target Volume ID
25+
Type: String
26+
AllowedPattern: "^fsvol-[A-Za-z0-9]+"
27+
28+
# CopySnapshotAndUpdateVolume - "Options" parameter.
29+
# Comma (,) separated values such as "DELETE_INTERMEDIATE_SNAPSHOTS,DELETE_INTERMEDIATE_DATA,DELETE_CLONED_VOLUMES"
30+
Options:
31+
Description: Options parameter value for the CopySnapshotAndUpdateVolume API
32+
Type: String
33+
34+
# CopySnapshotAndUpdateVolume - "CopyStrategy" parameter.
35+
# example = INCREMENTAL_COPY or FULL_COPY
36+
CopyStrategy:
37+
Description: CopyStrategy parameter value for the CopySnapshotAndUpdateVolume API
38+
Type: String
39+
Default: INCREMENTAL_COPY
40+
41+
SourceAWSAcctId:
42+
Description: FSx Source AWS Account ID
43+
Type: String
44+
AllowedPattern: ^\d{12}
45+
46+
Resources:
47+
#
48+
# IAM role going to be assumed by Source Lambda function to invoke target Lambda function.
49+
#
50+
CrossAcctIAMRole:
51+
Type: AWS::IAM::Role
52+
Properties:
53+
RoleName: !Sub CrossAcctIAMRole-${AWS::StackName}
54+
AssumeRolePolicyDocument:
55+
Version: "2012-10-17"
56+
Statement:
57+
- Effect: Allow
58+
Principal:
59+
AWS: !Sub "arn:aws:iam::${SourceAWSAcctId}:root"
60+
Action: sts:AssumeRole
61+
Path: /
62+
Policies:
63+
- PolicyName: !Sub CrossAcctIAMRole-Policy-${AWS::StackName}
64+
PolicyDocument:
65+
Version: "2012-10-17"
66+
Statement:
67+
- Effect: Allow
68+
Action:
69+
- lambda:InvokeFunction
70+
Resource: !GetAtt CopySnapshotAndUpdateVolumeLambda.Arn
71+
72+
#
73+
# Lambda Execution Role
74+
#
75+
CopySnapshotAndUpdateVolumeLambdaRole:
76+
Type: AWS::IAM::Role
77+
Properties:
78+
RoleName: !Sub CopySnapshotAndUpdateVolumeLambda-Role-${AWS::StackName}
79+
AssumeRolePolicyDocument:
80+
Version: "2012-10-17"
81+
Statement:
82+
- Effect: Allow
83+
Principal:
84+
Service: lambda.amazonaws.com
85+
Action: sts:AssumeRole
86+
Path: /
87+
Policies:
88+
- PolicyName: !Sub CopySnapshotAndUpdateVolumeLambda-Policy-${AWS::StackName}
89+
PolicyDocument:
90+
Version: "2012-10-17"
91+
Statement:
92+
- Effect: Allow
93+
Action:
94+
- logs:CreateLogGroup
95+
- logs:CreateLogStream
96+
- logs:PutLogEvents
97+
Resource: !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*"
98+
- Effect: Allow
99+
Action:
100+
- fsx:CreateSnapshot
101+
- fsx:DescribeSnapshots
102+
- fsx:DescribeVolumes
103+
- fsx:DeleteSnapshot
104+
- fsx:TagResource
105+
- fsx:ListTagsForResource
106+
- fsx:CopySnapshotAndUpdateVolume
107+
Resource:
108+
- !Sub "arn:aws:fsx:${AWS::Region}:${AWS::AccountId}:*"
109+
- !Sub "arn:aws:fsx:*:${SourceAWSAcctId}:*"
110+
#
111+
# Lambda function that will make CopySnapshotAndUpdateVolume API call.
112+
#
113+
CopySnapshotAndUpdateVolumeLambda:
114+
Type: AWS::Serverless::Function
115+
Properties:
116+
FunctionName: !Sub CopySnapshotAndUpdateVolumeLambda-${AWS::StackName}
117+
Description: Lambda function that will make CopySnapshotAndUpdateVolume API call.
118+
Environment:
119+
Variables:
120+
DEST_VOLUME_ID: !Ref TargetVolumeID
121+
OPTIONS: !Ref Options
122+
COPY_STRATEGY: !Ref CopyStrategy
123+
Handler: CopySnapshotAndUpdateVolume.lambda_handler
124+
Role: !GetAtt CopySnapshotAndUpdateVolumeLambdaRole.Arn
125+
CodeUri: src/CopySnapshotAndUpdateVolume.py
126+
Runtime: python3.13
127+
Timeout: 600
128+
129+
#
130+
# Stack output section
131+
#
132+
Outputs:
133+
CrossAcctIAMRole:
134+
Description: Cross account IAM role going to be assumed by Source Lambda function to invoke target Lambda function.
135+
Value: !GetAtt CrossAcctIAMRole.Arn
136+
LambdaFunction:
137+
Description: Target account Lambda Function that will make CopySnapshotAndUpdateVolume API call
138+
Value: !GetAtt CopySnapshotAndUpdateVolumeLambda.Arn
139+
LambdaExecutionRole:
140+
Description: IAM role used by target Lambda function
141+
Value: !GetAtt CopySnapshotAndUpdateVolumeLambdaRole.Arn
142+

‎eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication/eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication.json‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
{
2+
"title": "Replicate FSx-OpenZFS volumes across file systems",
3+
"description": "Periodic Amazon FSx for OpenZFS volume replication across AWS Regions and accounts using Amazon EventBridge Scheduler and AWS Lambda",
4+
"language": "Python",
5+
"level": "200",
6+
"framework": "SAM",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"The AWS SAM template deploys an Amazon EventBridge Scheduler to trigger an AWS Lambda function based on a user-defined schedule. This function copies snapshots of the volume and replicates them to the target FSx system available in a different AWS account and/or region. The template contains a sample Lambda function that creates a snapshot of the source FSx VolumeID. Once the snapshot becomes available, it invokes another Lambda function in the destination AWS account or region, which initiates the replication by calling the copy_snapshot_and_update_volume API. This solution also notifies users using an SNS topic for any errors and snapshot creation details.",
11+
"This pattern sets up the following resources:",
12+
"An Amazon EventBridge Scheduler that triggers a Lambda function based on the schedule defined by the customer to take snapshots of the provided FSx Source VolumeID.",
13+
"An SNS topic that sends notifications for any failures while creating snapshots.",
14+
"The function also deletes older snapshots.",
15+
"Sample Lambda functions that create snapshots of the source FSx VolumeID and replicate them by invoking another Lambda function, which calls the copy_snapshot_and_update_volume API for the target VolumeID in the destination AWS account or region."
16+
]
17+
},
18+
"gitHub": {
19+
"template": {
20+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication",
21+
"templateURL": "serverless-patterns/eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication",
22+
"projectFolder": "eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication",
23+
"templateFile": "source-template.yaml"
24+
}
25+
},
26+
"deploy": {
27+
"text": [
28+
"sam deploy --guided --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM -t source-template.yaml"
29+
]
30+
},
31+
"testing": {
32+
"text": [
33+
"See the GitHub repo for detailed testing instructions."
34+
]
35+
},
36+
"cleanup": {
37+
"text": [
38+
"Delete the stack: <code>sam delete --stack-name <stackname></code>."
39+
]
40+
},
41+
"authors": [
42+
{
43+
"name": "Anup Rajpara",
44+
"image": "https://drive.google.com/file/d/1MqpPNLCqbU4kvvtTspNXZBqD99aVIJI9/view?usp=sharing",
45+
"bio": "Anup is passionate about serverless & event-driven architectures.",
46+
"linkedin": "anup-rajpara-developer/"
47+
}
48+
],
49+
"patternArch": {
50+
"icon1": {
51+
"x": 20,
52+
"y": 50,
53+
"service": "eventbridge-scheduler",
54+
"label": "EventBridge Scheduler"
55+
},
56+
"icon2": {
57+
"x": 50,
58+
"y": 50,
59+
"service": "lambda",
60+
"label": "AWS Lambda"
61+
},
62+
"icon3": {
63+
"x": 80,
64+
"y": 50,
65+
"service": "fsx",
66+
"label": "FSx for OpenZFS"
67+
},
68+
"line1": {
69+
"from": "icon1",
70+
"to": "icon2",
71+
"label": ""
72+
},
73+
"line2": {
74+
"from": "icon2",
75+
"to": "icon3",
76+
"label": ""
77+
}
78+
}
79+
}

‎eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication/example-pattern.json‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
{
2+
"title": "Replicate FSx-OpenZFS volumes across file systems",
3+
"description": "Periodic Amazon FSx for OpenZFS volume replication across AWS Regions and accounts using Amazon EventBridge Scheduler and AWS Lambda",
4+
"language": "Python",
5+
"level": "200",
6+
"framework": "SAM",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"The AWS SAM template deploys an Amazon EventBridge Scheduler to trigger an AWS Lambda function based on a user-defined schedule. This function copies snapshots of the volume and replicates them to the target FSx system available in a different AWS account and/or region. The template contains a sample Lambda function that creates a snapshot of the source FSx VolumeID. Once the snapshot becomes available, it invokes another Lambda function in the destination AWS account or region, which initiates the replication by calling the copy_snapshot_and_update_volume API. This solution also notifies users using an SNS topic for any errors and snapshot creation details.",
11+
"This pattern sets up the following resources:",
12+
"An Amazon EventBridge Scheduler that triggers a Lambda function based on the schedule defined by the customer to take snapshots of the provided FSx Source VolumeID.",
13+
"An SNS topic that sends notifications for any failures while creating snapshots.",
14+
"The function also deletes older snapshots.",
15+
"Sample Lambda functions that create snapshots of the source FSx VolumeID and replicate them by invoking another Lambda function, which calls the copy_snapshot_and_update_volume API for the target VolumeID in the destination AWS account or region."
16+
]
17+
},
18+
"gitHub": {
19+
"template": {
20+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication",
21+
"templateURL": "serverless-patterns/eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication",
22+
"projectFolder": "eventbridge-lambda-fsx-openzfs-cross-account-region-periodic-replication",
23+
"templateFile": "source-template.yaml"
24+
}
25+
},
26+
"deploy": {
27+
"text": [
28+
"sam deploy --guided --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM -t source-template.yaml"
29+
]
30+
},
31+
"testing": {
32+
"text": [
33+
"See the GitHub repo for detailed testing instructions."
34+
]
35+
},
36+
"cleanup": {
37+
"text": [
38+
"Delete the stack: <code>sam delete --stack-name <stackname></code>."
39+
]
40+
},
41+
"authors": [
42+
{
43+
"name": "Anup Rajpara",
44+
"image": "https://drive.google.com/file/d/1MqpPNLCqbU4kvvtTspNXZBqD99aVIJI9/view?usp=sharing",
45+
"bio": "Anup is passionate about serverless & event-driven architectures.",
46+
"linkedin": "anup-rajpara-developer/"
47+
}
48+
]
49+
}

0 commit comments

Comments
 (0)