Skip to content

Commit 11ab665

Browse files
julianwoodjulianwood
authored
Merge pull request #2328 from aayushwadhwa/aayushw-feature-sqs-timestream-pattern
New serverless pattern - eventbridge-pipes-sqs-to-timestream
2 parents b576279 + badc5a0 commit 11ab665

File tree

5 files changed

+328
-0
lines changed

5 files changed

+328
-0
lines changed

‎eventbridge-pipes-sqs-to-timestream/README.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# SQS Queue to Amazon Timestream with AWS EventBridge Pipes
2+
3+
This pattern demostrates the process of ingesting data into Amazon Timestream from SQS Queue via EventBridge Pipes.
4+
5+
Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/eventbridge-pipes-sqs-to-timestream
6+
7+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
8+
9+
## Requirements
10+
11+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
12+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
13+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
14+
* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed
15+
16+
## Deployment Instructions
17+
18+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
19+
```
20+
git clone https://github.com/aws-samples/serverless-patterns
21+
```
22+
1. Change directory to the pattern directory:
23+
```
24+
cd eventbridge-pipes-sqs-to-timestream
25+
```
26+
1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file:
27+
```
28+
sam deploy --guided
29+
```
30+
1. During the prompts:
31+
* Enter a stack name
32+
* Enter the desired AWS Region
33+
* Allow SAM CLI to create IAM roles with the required permissions.
34+
35+
Once you have run `sam deploy --guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in future to use these defaults.
36+
37+
1. Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs which are used for testing.
38+
39+
## How it works
40+
41+
The template will create Timestream table, SQS queue, Cloudwatch LogGroup and IAM Role, and connect it together with a new EventBridge Pipe.
42+
43+
When records are sent to the source queue, the Pipe will convert into a valid Timestream record and ingest it to the Timestream table.
44+
45+
Please follow [this documentation](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html) for more information.
46+
47+
## Testing
48+
49+
```
50+
sh command.sh <SQS Queue URL> <Region>
51+
```
52+
53+
## Cleanup
54+
55+
1. Delete the stack
56+
```bash
57+
aws cloudformation delete-stack --stack-name STACK_NAME
58+
```
59+
1. Confirm the stack has been deleted
60+
```bash
61+
aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus"
62+
```
63+
----
64+
Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
65+
66+
SPDX-License-Identifier: MIT-0

‎eventbridge-pipes-sqs-to-timestream/command.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
2+
DATE_TIME=$(date -u '+%Y-%m-%d %H:%M:%S.000')
3+
DATA="{\"dimension_1\": \"dimension1\", \"measure_1\": \"1.0\", \"time\": \"${DATE_TIME}\", \"version\": \"1\" }"
4+
aws sqs send-message --queue-url "$1" --message-body "$DATA" --region "$2"

‎eventbridge-pipes-sqs-to-timestream/eventbridge-pipes-sqs-to-timestream.json‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
{
2+
"title": "SQS to Timestream using EventBridge Pipes",
3+
"description": "This pattern demostrates the process of ingesting data into Amazon Timestream from SQS Queue via EventBridge Pipes.",
4+
"language": "YAML",
5+
"level": "200",
6+
"framework": "CDK",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This project demostrates how to send data to Amazon Timestream directly from SQS. This pattern uses EventBridge Pipes to connect SQS and Timestream together.",
11+
"The pattern creates an Amazon SQS queue, a Timestream table, Cloudwatch LogGroup, IAM Policy and an EventBridge Pipe."
12+
]
13+
},
14+
"gitHub": {
15+
"template": {
16+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-pipes-sqs-to-timestream",
17+
"templateURL": "serverless-patterns/eventbridge-pipes-sqs-to-timestream",
18+
"projectFolder": "eventbridge-pipes-sqs-to-timestream",
19+
"templateFile": "template.yaml"
20+
}
21+
},
22+
"resources": {
23+
"bullets": [
24+
{
25+
"text": "Send SQS data to Timestream",
26+
"link": "https://docs.aws.amazon.com/timestream/latest/developerguide/SQS.html"
27+
},
28+
{
29+
"text": "Amazon Timestream",
30+
"link": "https://aws.amazon.com/timestream/"
31+
},
32+
{
33+
"text": "AWS EventBridge Pipes Targets",
34+
"link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html"
35+
}
36+
]
37+
},
38+
"deploy": {
39+
"text": [
40+
"sam deploy"
41+
]
42+
},
43+
"testing": {
44+
"text": [
45+
"See the GitHub repo for detailed testing instructions."
46+
]
47+
},
48+
"cleanup": {
49+
"text": [
50+
"Delete the stack: <code>aws cloudformation delete-stack --stack-name STACK_NAME</code>."
51+
]
52+
},
53+
"authors": [
54+
{
55+
"name": "Aayush Wadhwa",
56+
"image": "https://avatars.githubusercontent.com/u/20082863?s=400&u=aaf983bea1df62c7f9c75869aa0a51a8877edfa8&v=4",
57+
"bio": "Software Development Engineer at AWS",
58+
"linkedin": "aayushwadhwa"
59+
}
60+
],
61+
"patternArch": {
62+
"icon1": {
63+
"x": 20,
64+
"y": 50,
65+
"service": "sqs",
66+
"label": "Amazon SQS"
67+
},
68+
"icon2": {
69+
"x": 50,
70+
"y": 50,
71+
"service": "eventbridge-pipes",
72+
"label": "Amazon EventBridge Pipes"
73+
},
74+
"icon3": {
75+
"x": 80,
76+
"y": 50,
77+
"service": "timestream",
78+
"label": "Amazon Timestream"
79+
},
80+
"line1": {
81+
"from": "icon1",
82+
"to": "icon2"
83+
},
84+
"line2": {
85+
"from": "icon2",
86+
"to": "icon3"
87+
}
88+
}
89+
}

‎eventbridge-pipes-sqs-to-timestream/example-pattern.json‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
{
2+
"title": "SQS to Timestream using EventBridge Pipes",
3+
"description": "This pattern demostrates the process of ingesting data into Amazon Timestream from SQS Queue via EventBridge Pipes.",
4+
"language": "YAML",
5+
"level": "200",
6+
"framework": "CDK",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This project demostrates how to send data to Amazon Timestream directly from SQS. This pattern uses EventBridge Pipes to connect SQS and Timestream together.",
11+
"The pattern creates an Amazon SQS queue, a Timestream table, Cloudwatch LogGroup, IAM Policy and an EventBridge Pipe."
12+
]
13+
},
14+
"gitHub": {
15+
"template": {
16+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-pipes-sqs-to-timestream",
17+
"templateURL": "serverless-patterns/eventbridge-pipes-sqs-to-timestream",
18+
"projectFolder": "eventbridge-pipes-sqs-to-timestream",
19+
"templateFile": "template.yaml"
20+
}
21+
},
22+
"resources": {
23+
"bullets": [
24+
{
25+
"text": "Send SQS data to Timestream",
26+
"link": "https://docs.aws.amazon.com/timestream/latest/developerguide/SQS.html"
27+
},
28+
{
29+
"text": "Amazon Timestream",
30+
"link": "https://aws.amazon.com/timestream/"
31+
},
32+
{
33+
"text": "AWS EventBridge Pipes Targets",
34+
"link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html"
35+
}
36+
]
37+
},
38+
"deploy": {
39+
"text": [
40+
"sam deploy"
41+
]
42+
},
43+
"testing": {
44+
"text": [
45+
"See the GitHub repo for detailed testing instructions."
46+
]
47+
},
48+
"cleanup": {
49+
"text": [
50+
"Delete the stack: <code>aws cloudformation delete-stack --stack-name STACK_NAME</code>."
51+
]
52+
},
53+
"authors": [
54+
{
55+
"name": "Aayush Wadhwa",
56+
"image": "https://avatars.githubusercontent.com/u/20082863?s=400&u=aaf983bea1df62c7f9c75869aa0a51a8877edfa8&v=4",
57+
"bio": "Software Development Engineer at AWS",
58+
"linkedin": "https://www.linkedin.com/in/aayushwadhwa/"
59+
}
60+
]
61+
}

‎eventbridge-pipes-sqs-to-timestream/template.yaml‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
AWSTemplateFormatVersion: '2010-09-09'
2+
Transform: AWS::Serverless-2016-10-31
3+
Description: Template to connect SQS Queue to Timestream
4+
5+
Resources:
6+
# Input SQS Queue
7+
Source:
8+
Type: AWS::SQS::Queue
9+
Properties:
10+
QueueName: !Sub ${AWS::StackName}-source
11+
# Target
12+
TimestreamTargetDatabase:
13+
Type: AWS::Timestream::Database
14+
Properties:
15+
DatabaseName: !Sub ${AWS::StackName}-db
16+
TimestreamTargetTable:
17+
Type: AWS::Timestream::Table
18+
Properties:
19+
DatabaseName: !Ref TimestreamTargetDatabase
20+
TableName: !Sub ${AWS::StackName}-tbl
21+
RetentionProperties:
22+
MemoryStoreRetentionPeriodInHours: "168"
23+
MagneticStoreRetentionPeriodInDays: "7"
24+
# Cloudwatch log group for debugging
25+
CloudwatchLogGroup:
26+
Type: AWS::Logs::LogGroup
27+
Properties:
28+
LogGroupName: !Sub ${AWS::StackName}-pipe-logs
29+
# IAM Role for Pipe
30+
PipeRole:
31+
Type: AWS::IAM::Role
32+
Properties:
33+
AssumeRolePolicyDocument:
34+
Version: 2012-10-17
35+
Statement:
36+
- Effect: Allow
37+
Principal:
38+
Service:
39+
- pipes.amazonaws.com
40+
Action:
41+
- sts:AssumeRole
42+
Policies:
43+
- PolicyName: SourcePolicy
44+
PolicyDocument:
45+
Version: 2012-10-17
46+
Statement:
47+
- Effect: Allow
48+
Action:
49+
- sqs:ReceiveMessage
50+
- sqs:DeleteMessage
51+
- sqs:GetQueueAttributes
52+
Resource: !GetAtt Source.Arn
53+
- PolicyName: TargetPolicy
54+
PolicyDocument:
55+
Version: 2012-10-17
56+
Statement:
57+
- Effect: Allow
58+
Action:
59+
- timestream:WriteRecords
60+
Resource: !GetAtt TimestreamTargetTable.Arn
61+
- PolicyName: DescribeEndpointsPolicy
62+
PolicyDocument:
63+
Version: 2012-10-17
64+
Statement:
65+
- Effect: Allow
66+
Action:
67+
- timestream:DescribeEndpoints
68+
Resource: '*'
69+
- PolicyName: CloudwatchLogPolicy
70+
PolicyDocument:
71+
Version: 2012-10-17
72+
Statement:
73+
- Effect: Allow
74+
Action:
75+
- logs:PutLogEvents
76+
Resource: !GetAtt CloudwatchLogGroup.Arn
77+
# Pipe connecting SQS to Timestream
78+
TimestreamPipe:
79+
Type: AWS::Pipes::Pipe
80+
Properties:
81+
Name: !Sub ${AWS::StackName}-pipe
82+
RoleArn: !GetAtt PipeRole.Arn
83+
Source: !GetAtt Source.Arn
84+
SourceParameters:
85+
SqsQueueParameters:
86+
BatchSize: 10
87+
MaximumBatchingWindowInSeconds: 5
88+
Target: !GetAtt TimestreamTargetTable.Arn
89+
TargetParameters:
90+
TimestreamParameters:
91+
DimensionMappings:
92+
- DimensionName: dimension_1
93+
DimensionValue: $.body.dimension_1
94+
DimensionValueType: VARCHAR
95+
SingleMeasureMappings:
96+
- MeasureName: measure_1
97+
MeasureValue: $.body.measure_1
98+
MeasureValueType: DOUBLE
99+
TimeFieldType: TIMESTAMP_FORMAT
100+
TimeValue: $.body.time
101+
TimestampFormat: yyyy-MM-dd HH:mm:ss.SSS
102+
VersionValue: $.body.version
103+
LogConfiguration:
104+
CloudwatchLogsLogDestination:
105+
LogGroupArn: !GetAtt CloudwatchLogGroup.Arn
106+
Level: TRACE
107+
IncludeExecutionData:
108+
- ALL

0 commit comments

Comments
 (0)