Skip to content

Commit 0141a65

Browse files
baermatHarshCasper
baermat
and
HarshCasper
committed
Add fargate sample to developer hub (#1)
* make the sample work with LS, potentially bugged * more instructions * cleanup, prettifying * remove unnecessary pictures, add architecture diagram, enhance README * fix diagram * add badge * work in review suggestions * improve the README * set a GHA workflow to build & test --------- Co-authored-by: HarshCasper <erbeusgriffincasper@gmail.com>
1 parent 6b1ca59 commit 0141a65

File tree

12 files changed

+4760
-84
lines changed

12 files changed

+4760
-84
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
name: Deploy on LocalStack
2+
3+
on:
4+
push:
5+
paths-ignore:
6+
- 'README.md'
7+
branches:
8+
- main
9+
pull_request:
10+
branches:
11+
- main
12+
13+
jobs:
14+
cdk:
15+
name: Setup infrastructure using CDK
16+
runs-on: ubuntu-latest
17+
steps:
18+
- name: Checkout
19+
uses: actions/checkout@v3
20+
21+
- name: Setup Node.js
22+
uses: actions/setup-node@v3
23+
with:
24+
node-version: 18
25+
26+
- name: Install CDK
27+
run: |
28+
npm install -g aws-cdk-local aws-cdk
29+
cdklocal --version
30+
31+
- name: Install dependencies
32+
run: |
33+
yarn
34+
35+
- name: Start LocalStack
36+
env:
37+
LOCALSTACK_API_KEY: ${{ secrets.LOCALSTACK_API_KEY }}
38+
DNS_ADDRESS: 0
39+
run: |
40+
export SQS_QUEUE="sqs-fargate-queue"
41+
export NETWORK_NAME="localstack-shared-net"
42+
docker network create $NETWORK_NAME
43+
pip install localstack awscli-local[ver1]
44+
pip install terraform-local
45+
docker pull localstack/localstack-pro:latest
46+
# Start LocalStack in the background
47+
LAMBDA_DOCKER_NETWORK=$NETWORK_NAME DOCKER_FLAGS="--network $NETWORK_NAME" DEBUG=1 localstack start -d
48+
# Wait 30 seconds for the LocalStack container to become ready before timing out
49+
echo "Waiting for LocalStack startup..."
50+
localstack wait -t 15
51+
echo "Startup complete"
52+
53+
- name: Deploy using CDK
54+
run: |
55+
docker build -t go-fargate .
56+
cd cdk
57+
npm install
58+
cdklocal bootstrap aws://000000000000/us-east-1
59+
cdklocal deploy --require-approval never
60+
61+
- name: List the resources
62+
run: |
63+
awslocal sqs list-queues
64+
awslocal dynamodb list-tables

‎README.md‎

Lines changed: 140 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -1,86 +1,166 @@
1-
# Build and deploy message processing service using Fargate
1+
# Messaging Processing application with SQS, DynamoDB, and Fargate
22

3-
This pattern demonstrates how to build and deploy Fargate service using Go which receives messages from SQS queue and stores it in DynamoDB table.
3+
| Key | Value |
4+
| ------------ | ------------------------------------------------------------------------------------- |
5+
| Environment | <img src="https://img.shields.io/badge/LocalStack-deploys-4D29B4.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAKgAAACoABZrFArwAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAAALbSURBVHic7ZpNaxNRFIafczNTGIq0G2M7pXWRlRv3Lusf8AMFEQT3guDWhX9BcC/uFAr1B4igLgSF4EYDtsuQ3M5GYrTaj3Tmui2SpMnM3PlK3m1uzjnPw8xw50MoaNrttl+r1e4CNRv1jTG/+v3+c8dG8TSilHoAPLZVX0RYWlraUbYaJI2IuLZ7KKUWCisgq8wF5D1A3rF+EQyCYPHo6Ghh3BrP8wb1en3f9izDYlVAp9O5EkXRB8dxxl7QBoNBpLW+7fv+a5vzDIvVU0BELhpjJrmaK2NMw+YsIxunUaTZbLrdbveZ1vpmGvWyTOJToNlsuqurq1vAdWPMeSDzwzhJEh0Bp+FTmifzxBZQBXiIKaAq8BBDQJXgYUoBVYOHKQRUER4mFFBVeJhAQJXh4QwBVYeHMQJmAR5GCJgVeBgiYJbg4T8BswYPp+4GW63WwvLy8hZwLcd5TudvBj3+OFBIeA4PD596nvc1iiIrD21qtdr+ysrKR8cY42itCwUP0Gg0+sC27T5qb2/vMunB/0ipTmZxfN//orW+BCwmrGV6vd63BP9P2j9WxGbxbrd7B3g14fLfwFsROUlzBmNM33XdR6Meuxfp5eg54IYxJvXCx8fHL4F3w36blTdDI4/0WREwMnMBeQ+Qd+YC8h4g78wF5D1A3rEqwBiT6q4ubpRSI+ewuhP0PO/NwcHBExHJZZ8PICI/e73ep7z6zzNPwWP1djhuOp3OfRG5kLROFEXv19fXP49bU6TbYQDa7XZDRF6kUUtEtoFb49YUbh/gOM7YbwqnyG4URQ/PWlQ4ASllNwzDzY2NDX3WwioKmBgeqidgKnioloCp4aE6AmLBQzUExIaH8gtIBA/lFrCTFB7KK2AnDMOrSeGhnAJSg4fyCUgVHsolIHV4KI8AK/BQDgHW4KH4AqzCQwEfiIRheKKUAvjuuu7m2tpakPdMmcYYI1rre0EQ1LPo9w82qyNziMdZ3AAAAABJRU5ErkJggg=="> <img src="https://img.shields.io/badge/AWS-deploys-F29100.svg?logo=amazon"> |
6+
| Services | Step Functions, SQS, DynamoDB, Fargate |
7+
| Integrations | CDK, AWS CLI |
8+
| Categories | Serverless; Event-Driven architecture |
9+
| Level | Beginner |
10+
| GitHub | [Repository link](https://github.com/baermat/sqs-fargate-ddb-cdk-go) |
411

5-
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
612

7-
## Requirements
13+
## Introduction
814

9-
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
10-
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured with named AWS profile
11-
* [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed
12-
* [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) installed
13-
* [Go](https://go.dev/doc/install) installed
14-
* [Docker](https://docs.docker.com/engine/install/) installed
15+
The Messaging Processing application demonstrates how to deploy and configure a Fargate container to interact with other services, specifically SQS and DynamoDB. The sample application implements the following integration among the various AWS services:
1516

16-
## Deployment Instructions
17+
- User submits a message to the specified SQS queue.
18+
- The Fargate container fetches any messages sent to the queue.
19+
- The Fargate container then writes any fetched messages into DynamoDB.
1720

18-
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
19-
```
20-
git clone https://github.com/aws-samples/sqs-fargate-ddb-cdk-go.git
21-
```
22-
2. Change directory to the pattern directory:
23-
```
24-
cd sqs-fargate-ddb-cdk-go
25-
```
21+
Users can deploy this application sample on AWS & LocalStack using Cloud Development Kit (CDK) with minimal changes. To test this application sample, we will demonstrate how you use LocalStack to deploy the infrastructure on your developer machine and your CI environment.
2622

27-
3. From the command line, use the following commands to deploy the stack using CDK:
28-
```
29-
docker build -t go-fargate .
30-
cd cdk
31-
npm i
32-
cdk deploy --profile ${AWS_PROFILE}
33-
```
23+
## Architecture diagram
3424

35-
## How it works
25+
The following diagram shows the architecture that this sample application builds and deploys:
3626

37-
In this pattern we created SQS queue, Fargate Service, and DynamoDB table.
38-
Fargate service is receiving messages from SQS queue using long polling (20 seconds)
39-
Once you send SQS message to the queue, Fargate service receives this message, processes it, and puts the message text into a new item of DynamoDB table.
27+
![LocalStack Fargate Messaging Processing application with AWS SQS, DynamoDB, and Fargate](./images/architecture-diagram.png)
4028

29+
- [Fargate ECS](https://docs.localstack.cloud/tutorials/ecs-ecr-container-app/) to spawn a container to act as a custom broker and relay any incoming messages.
30+
- [SQS](https://docs.localstack.cloud/user-guide/aws/sqs/) to send messages to the application using long polling (20 seconds).
31+
- [DynamoDB](https://docs.localstack.cloud/user-guide/aws/dynamodb/) to persist the messages received after being processed by the Fargate service.
4132

42-
## Testing
33+
## Prerequisites
4334

44-
1. Open SQS console
45-
2. Find SQS queue with name `sqs-fargate-queue`
46-
3. Press `Send and Receive message` button
47-
4. Enter the message into "Message Body" field, e.g.
35+
- LocalStack Pro with the [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli).
36+
- [Cloud Development Kit](https://docs.localstack.cloud/user-guide/integrations/aws-cdk/) with the [`cdklocal`](https://www.npmjs.com/package/aws-cdk-local) installed.
37+
- [AWS CLI](https://docs.localstack.cloud/user-guide/integrations/aws-cli/) with the [`awslocal` wrapper](https://docs.localstack.cloud/user-guide/integrations/aws-cli/#localstack-aws-cli-awslocal).
38+
- [Node.js](https://nodejs.org/en/download)
39+
40+
## Instructions
41+
42+
You can build and deploy the sample application on LocalStack by running the `run.sh` script.
43+
Here are instructions to deploy and test it manually step-by-step.
44+
45+
### Running the LocalStack container
46+
47+
Before starting the LocalStack container, configure the following environment variables which act as configurations for the LocalStack container:
48+
49+
```bash
50+
export LOCALSTACK_API_KEY=<your_api_key>
51+
export SQS_QUEUE="sqs-fargate-queue"
52+
export NETWORK_NAME="localstack-shared-net"
53+
```
54+
55+
The `SQS_QUEUE` and `NETWORK_NAME` can be any name that confirms to the naming conventions of SQS and Docker networks respectively.
56+
57+
To create a Docker network, run the following command:
58+
59+
```bash
60+
docker network create $NETWORK_NAME
61+
```
62+
63+
This network is required for the Fargate ECS container to be able to [use LocalStack services from the running Docker container](https://docs.localstack.cloud/references/network-troubleshooting/endpoint-url/#from-your-container).
64+
65+
Run the following command to start the LocalStack container:
66+
67+
```bash
68+
LAMBDA_DOCKER_NETWORK=$NETWORK_NAME DOCKER_FLAGS="--network $NETWORK_NAME" DEBUG=1 localstack start -d
69+
```
70+
71+
This starts LocalStack in detached mode with the necessary configurations.
72+
> If you prefer to be able to follow the debug output while deploying this sample, omit the `-d` flag, then open a new terminal window and navigate to the same location as before.
73+
74+
### Build the infrastructure
75+
76+
After running the LocalStack container, you can build the Fargate Docker image:
77+
78+
```bash
79+
docker build -t go-fargate .
80+
```
81+
82+
As specified in the [`Dockerfile](./Dockerfile), this builds the Docker image with the name `go-fargate`. You can now install the necessary dependencies for CDK and deploy the infrastructure. Run the following command:
83+
84+
```bash
85+
cd cdk
86+
npm i
87+
```
88+
89+
This will navigate into the correct folder and install all necessary packages
90+
91+
### Deploy the infrastructure
92+
93+
To deploy the sample application, we will use CDK to bootstrap the environment and deploy the infrastructure. Run the following commands:
94+
95+
```bash
96+
cdklocal bootstrap
97+
cdklocal deploy
98+
```
99+
100+
> While deploying the infrastructure, CDK will ask your permission — If you would rather skip the approval process, you can add the `--require-approval never` flag to the deploy command.
101+
102+
### Testing the application
103+
104+
To assert that the SQS queue and the DynamoDB have been created, run the following commands:
105+
106+
```bash
107+
awslocal sqs list-queues
108+
awslocal dynamodb list-tables
109+
```
110+
111+
You should see output similar to the following:
48112
```
49113
{
50-
"message": "Hello, Fargate"
114+
"QueueUrls": [
115+
"http://localhost:4566/000000000000/sqs-fargate-queue"
116+
]
117+
}
118+
{
119+
"TableNames": [
120+
"sqs-fargate-ddb-table"
121+
]
51122
}
52123
```
53124

54-
![image1](images/image1.png)
55-
56-
5. Press `Send message`
57-
6. Open ECS console
58-
7. Find cluster with name `go-service-cluster`
59-
8. Find service named `go-fargate-service`
60-
9. Open CloudWatch logs for an active task of this service and you'll see the messages from the service in the log.
125+
Next, send a message to the SQS queue. Run the following command:
61126

62-
![image2](images/image2.png)
127+
```bash
128+
awslocal sqs send-message --queue $SQS_QUEUE --message-body '{"message": "hello world"}'
129+
```
63130

64-
10. Open DynamoDB console and find `sqs-fargate-ddb-table` table
65-
11. Press `Explore table items` button and you'll see messages that Fargate service put into the table.
131+
This sends a `hello world` message to the SQS queue, which is then processed by the Fargate container. You can wait for a couple of seconds for the container to finish its task.
66132

67-
![image3](images/image3.png)
133+
To check if the message has been written to the DynamoDB table, run the following command:
68134

69-
12. Check CloudWatch dashboard `go-service-dashboard` to monitor number of messages received from SQS and added into DynamoDB by the service.
135+
```bash
136+
awslocal dynamodb scan --table-name sqs-fargate-ddb-table
137+
```
70138

71-
## Cleanup
139+
You should see an answer similar to the following when executing the given command:
72140

73-
1. Delete the stack:
74-
```
75-
cd cdk
76-
cdk destroy --profile ${AWS_PROFILE}
77-
```
78-
2. Confirm the stack has been deleted:
79-
```
80-
aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'SqsFargate')].StackStatus" --profile ${AWS_PROFILE}
81-
```
141+
```bash
142+
{
143+
"Items": [
144+
{
145+
"timestamp_utc": {
146+
"S": "2023-05-24T10:56:22.456Z"
147+
},
148+
"message": {
149+
"S": "hello world"
150+
},
151+
"id": {
152+
"S": "f00e46d3-414f-4170-b964-9a3d397111e4"
153+
}
154+
}
155+
],
156+
"Count": 1,
157+
"ScannedCount": 1,
158+
"ConsumedCapacity": null
159+
}
160+
```
82161

162+
To run this sample against AWS, check [the original repository](https://github.com/aws-samples/sqs-fargate-ddb-cdk-go).
83163

84-
## License
164+
## Learn more
85165

86-
This code is licensed under the MIT-0 License. See the LICENSE file.
166+
The sample application is based on a [public AWS sample app](https://github.com/aws-samples/sqs-fargate-ddb-cdk-go) that deploys a message processing service using Fargate. See this AWS patterns post for more details: [Run message-driven workloads at scale by using AWS Fargate](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/run-message-driven-workloads-at-scale-by-using-aws-fargate.html).

‎cdk/lib/FargateServiceStack.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ export class FargateServiceStack extends Stack {
5858
taskDefinition: taskDef,
5959
environment: {
6060
SQS_URL: queue.queueUrl,
61-
DDB_TABLE: ddbTable.tableName
61+
DDB_TABLE: ddbTable.tableName,
6262
},
6363
logging: new ecs.AwsLogDriver({
6464
logGroup: logGroup,

0 commit comments

Comments
 (0)