Skip to content

Commit f8d4f06

Browse files
julianwoodjulianwood
authored
Merge pull request #2360 from enmata/s3-sqs-lambda-terraform_enmata
Add s3-sqs-lambda-terraform pattern
2 parents 18cb48a + 8a32f8a commit f8d4f06

File tree

5 files changed

+405
-0
lines changed

5 files changed

+405
-0
lines changed

‎s3-sqs-lambda-terraform/README.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Amazon S3 to Amazon SQS queue to AWS Lambda
2+
3+
The Terraform code deploys an AWS Lambda function, an Amazon SQS queue, one AWS S3 buckets and the AWS IAM resources required to run the application. The created Lambda function is triggered on every new `.jpg` image file uploaded to the S3 bucket using an SQS queue as a notification target. The Lambda function code contains only contains minimal code for demo purposes.
4+
5+
Learn more about this pattern at Serverless Land Patterns: [serverlessland.com/patterns/s3-sqs-lambda-terraform](https://serverlessland.com/patterns/s3-sqs-lambda-terraform)
6+
7+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
8+
9+
10+
## Requirements
11+
12+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
13+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
14+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
15+
* [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli?in=terraform/aws-get-started) installed
16+
17+
18+
## Deployment Instructions
19+
20+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
21+
```
22+
git clone https://github.com/aws-samples/serverless-patterns
23+
```
24+
1. Change directory to the pattern directory:
25+
```
26+
cd s3-sqs-lambda-terraform
27+
```
28+
1. From the command line, initialize terraform to to downloads and installs the providers defined in the configuration:
29+
```
30+
terraform init
31+
```
32+
1. From the command line, apply the configuration in the main.tf file:
33+
```
34+
terraform apply
35+
```
36+
1. During the prompts:
37+
* Enter yes
38+
1. Note the outputs from the deployment process. These contain the resource names and/or ARNs which are used for testing.
39+
40+
41+
## How it works
42+
43+
* Use the AWS CLI or AWS console to upload an image to the source S3 Bucket
44+
* If the object is a .jpg file, the Lambda function is triggered using SQS as a notification target.
45+
46+
## Testing
47+
48+
Run the following AWS CLI command to upload an image to the S3 bucket. Note, you must edit the {SourceBucketName} placeholder with the name of the source S3 bucket. This is provided in the stack outputs.
49+
50+
```bash
51+
aws s3 cp './events/exampleImage.png' s3://{SourceBucketName}
52+
```
53+
54+
## Documentations and next step
55+
56+
To expand the Step Functions workflow that the pattern created, you can find out example workflows at Step Functions Workflow: [serverlessland.com/workflows](https://serverlessland.com/workflows)
57+
58+
59+
## Cleanup
60+
61+
1. Change directory to the pattern directory:
62+
```
63+
cd s3-sqs-lambda-terraform
64+
```
65+
1. Delete all files from the S3 bucket
66+
1. Delete all created resources by terraform
67+
```bash
68+
terraform destroy
69+
```
70+
1. During the prompts:
71+
* Enter yes
72+
1. Confirm all created resources has been deleted
73+
```bash
74+
terraform show
75+
```
76+
77+
78+
----
79+
Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
80+
81+
SPDX-License-Identifier: MIT-0

‎s3-sqs-lambda-terraform/example-pattern.json‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
{
2+
"title": "Amazon S3 to AWS Lambda with Amazon SQS queue",
3+
"description": "Create a Lambda function that is triggered for every jpg image file uploaded to S3 via an SQS queue.",
4+
"language": "Node.js",
5+
"level": "200",
6+
"framework": "Terraform",
7+
"introBox": {
8+
"headline": "Lambda function triggered for every image file stored to S3 via SQS",
9+
"text": [
10+
"The terraform manifest deploys a Lambda function, an SQS queue, one S3 bucket and the IAM resources required to run the application.",
11+
"An SQS queue consumes ObjectCreated events from an Amazon S3 bucket if the file has .jpg extension. The SQS triggers a Lambda function.",
12+
"The Lambda function serve as a pre-configured template, providing a starting point for developing your application."
13+
]
14+
},
15+
"gitHub": {
16+
"template": {
17+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/s3-sqs-lambda-terraform",
18+
"templateURL": "serverless-patterns/s3-sqs-lambda-terraform",
19+
"projectFolder": "s3-sqs-lambda-terraform",
20+
"templateFile": "main.tf"
21+
}
22+
},
23+
"resources": {
24+
"bullets": [
25+
{
26+
"text": "Configuring an Amazon SQS queue to trigger an AWS Lambda function",
27+
"link": "https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-lambda-function-trigger.html"
28+
}
29+
]
30+
},
31+
"deploy": {
32+
"text": [
33+
"terraform init",
34+
"terraform apply"
35+
]
36+
},
37+
"testing": {
38+
"text": [
39+
"See the GitHub repo for detailed testing instructions."
40+
]
41+
},
42+
"cleanup": {
43+
"text": [
44+
"<code>terraform destroy</code>"
45+
]
46+
},
47+
"authors": [
48+
{
49+
"name": "Oriol Matavacas",
50+
"image": "",
51+
"bio": "Oriol Matavacas is a Sr. Solutions Architect at AWS based in Barcelona. Oriol primarily supporting customers on the journey to the Cloud. He enjoys building new solutions with scalability, availability and easy to maintain by using serverless.",
52+
"linkedin": "oriol-matavacas-rodriguez-b165868a",
53+
"twitter": ""
54+
}
55+
]
56+
}
57+

‎s3-sqs-lambda-terraform/main.tf‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
terraform {
2+
required_providers {
3+
aws = {
4+
source = "hashicorp/aws"
5+
version = "~> 5.57.0"
6+
}
7+
}
8+
9+
required_version = ">= 0.14.9"
10+
}
11+
12+
data "aws_caller_identity" "current" {}
13+
data "aws_region" "current" {}
14+
15+
16+
#################################################################
17+
# S3 Buckets
18+
#################################################################
19+
# Create a new Source S3 bucket
20+
resource "aws_s3_bucket" "MySourceS3Bucket" {
21+
bucket_prefix = "s3-sqs-lambda-tf-sources3bucket-"
22+
}
23+
24+
# Send notifications to SQS for all events in the bucket
25+
resource "aws_s3_bucket_notification" "MySourceS3BucketNotification" {
26+
bucket = aws_s3_bucket.MySourceS3Bucket.id
27+
28+
queue {
29+
queue_arn = aws_sqs_queue.MyHandlerQueue.arn
30+
events = [
31+
"s3:ObjectCreated:*"
32+
]
33+
filter_suffix = ".jpg"
34+
}
35+
36+
}
37+
38+
#################################################################
39+
# SQS - Queue
40+
#################################################################
41+
# Create SQS - Queue
42+
resource "aws_sqs_queue" "MyHandlerQueue" {
43+
name = "s3-sqs-lambda-tf-SQSResizerQueue"
44+
}
45+
46+
# Create SQS - Policy
47+
resource "aws_sqs_queue_policy" "MyHandlerQueuePolicy" {
48+
queue_url = aws_sqs_queue.MyHandlerQueue.id
49+
50+
policy = <<POLICY
51+
{
52+
"Version": "2012-10-17",
53+
"Id": "QueuePolicy",
54+
"Statement": [
55+
{
56+
"Sid": "Allow-SendMessage-To-Queue-From-S3-Event-Notification",
57+
"Effect": "Allow",
58+
"Principal": {
59+
"Service": "s3.amazonaws.com"
60+
},
61+
"Action": "sqs:SendMessage",
62+
"Resource": "arn:aws:sqs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:${aws_sqs_queue.MyHandlerQueue.name}",
63+
"Condition": {
64+
"StringEquals": {
65+
"aws:SourceAccount": "${data.aws_caller_identity.current.account_id}"
66+
},
67+
"ArnLike": {
68+
"aws:SourceArn": "arn:aws:s3:::${aws_s3_bucket.MySourceS3Bucket.id}"
69+
}
70+
}
71+
}
72+
]
73+
}
74+
POLICY
75+
}
76+
77+
78+
#################################################################
79+
# Lambda Function
80+
#################################################################
81+
# Creating Lambda Function
82+
resource "aws_lambda_function" "MyHandlerFunction-Function" {
83+
filename = data.archive_file.LambdaZipFile.output_path
84+
function_name = "s3-sqs-lambda-tf-LambdaFunction"
85+
role = aws_iam_role.MyHandlerFunction-Role.arn
86+
handler = "app.handler"
87+
runtime = "nodejs20.x"
88+
}
89+
90+
# Create a zip file from the Lambda source code
91+
data "archive_file" "LambdaZipFile" {
92+
type = "zip"
93+
source_file = "${path.module}/src/app.mjs"
94+
output_path = "${path.module}/lambda-src.zip"
95+
}
96+
97+
# Creating SQS Queue Trigger for Lambda Function
98+
resource "aws_lambda_event_source_mapping" "MyHandlerFunction-Function-to-SQS" {
99+
event_source_arn = aws_sqs_queue.MyHandlerQueue.arn
100+
function_name = aws_lambda_function.MyHandlerFunction-Function.arn
101+
}
102+
103+
# Creating IAM Role for Lambda Function
104+
resource "aws_iam_role" "MyHandlerFunction-Role" {
105+
name = "s3-sqs-lambda-tf-MyHandlerFunction-Role"
106+
107+
assume_role_policy = jsonencode({
108+
Version = "2012-10-17"
109+
Statement = [
110+
{
111+
Action = "sts:AssumeRole"
112+
Effect = "Allow"
113+
Principal = {
114+
Service = "lambda.amazonaws.com"
115+
}
116+
}
117+
]
118+
})
119+
}
120+
121+
# Creating IAM Policies for Lambda
122+
resource "aws_iam_role_policy" "MyHandlerFunction-Policy-source" {
123+
name = "s3-sqs-lambda-tf-MyHandlerFunction-Role"
124+
policy = jsonencode(
125+
{
126+
"Statement": [
127+
{
128+
"Action": [
129+
"s3:GetObject",
130+
"s3:ListBucket",
131+
"s3:GetBucketLocation",
132+
"s3:GetObjectVersion",
133+
"s3:GetLifecycleConfiguration"
134+
],
135+
"Resource": [
136+
"arn:aws:s3:::${aws_s3_bucket.MySourceS3Bucket.id}",
137+
"arn:aws:s3:::${aws_s3_bucket.MySourceS3Bucket.id}/*"
138+
],
139+
"Effect": "Allow"
140+
}
141+
]
142+
}
143+
)
144+
role = aws_iam_role.MyHandlerFunction-Role.name
145+
}
146+
147+
resource "aws_iam_role_policy_attachment" "AWSLambdaBasicExecutionRole" {
148+
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
149+
role = "${aws_iam_role.MyHandlerFunction-Role.name}"
150+
}
151+
152+
resource "aws_iam_role_policy_attachment" "AWSLambdaSQSQueueExecutionRole" {
153+
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole"
154+
role = "${aws_iam_role.MyHandlerFunction-Role.name}"
155+
}
156+
157+
158+
#################################################################
159+
# Outputs
160+
#################################################################
161+
# Displaying the SQS Queue, SourceS3 buckets and Lambda Function
162+
output "SQSQueueName" {
163+
value = aws_sqs_queue.MyHandlerQueue.name
164+
description = "SQS Queue for queuing the s3 events"
165+
}
166+
output "SourceS3BucketName" {
167+
value = aws_s3_bucket.MySourceS3Bucket.id
168+
description = "S3 Bucket for object storage"
169+
}
170+
output "LambdaFunctionArn" {
171+
value = aws_lambda_function.MyHandlerFunction-Function.arn
172+
description = "HandlerFunction function Arn"
173+
}

0 commit comments

Comments
 (0)