|
| 1 | +# Assigning a dynamic message group ID from the message body using EventBridge Pipes |
| 2 | + |
| 3 | +EventBridge, a serverless event bus service, and SQS, a managed message queuing service, work together in event-driven architectures to route and process messages between AWS services and applications. While EventBridge routes messages to SQS queues for processing by microservices, FIFO queues can be used for strict message ordering. Although EventBridge cannot directly set message group IDs for organizing related messages, EventBridge Pipes provides a solution by allowing dynamic message group ID assignment based on event properties, enabling ordered processing for specific users, applications, or locations without additional coding. |
| 4 | + |
| 5 | +Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/eventbridge-pipes-dynamic-message-group-id |
| 6 | + |
| 7 | +> [!Important] |
| 8 | +> This application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. |
| 9 | +
|
| 10 | +## Requirements |
| 11 | +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. |
| 12 | +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured |
| 13 | +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) |
| 14 | +* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed |
| 15 | + |
| 16 | + |
| 17 | +## Deployment |
| 18 | + |
| 19 | +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: |
| 20 | + ``` |
| 21 | + git clone https://github.com/aws-samples/serverless-patterns |
| 22 | + ``` |
| 23 | +2. Change directory to the pattern directory: |
| 24 | + ``` |
| 25 | + cd eventbridge-pipes-dynamic-message-group-id |
| 26 | + ``` |
| 27 | +
|
| 28 | +3. Build and deploy the SAM application |
| 29 | +```bash |
| 30 | +sam build && sam deploy --guided |
| 31 | +``` |
| 32 | + |
| 33 | +During the prompts: |
| 34 | +* Enter a stack name |
| 35 | +* Enter the desired AWS Region |
| 36 | +* Allow SAM CLI to create IAM roles with the required permissions. |
| 37 | + |
| 38 | +You can accept all other defaults. Copy down the SQS Output Queue URL. You'll use this later in testing. |
| 39 | + |
| 40 | +Once you have run `sam deploy --guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in future to use these defaults. |
| 41 | + |
| 42 | +## How it works |
| 43 | + |
| 44 | + |
| 45 | +This project implements an event processing pipeline using AWS EventBridge and SQS FIFO queues. The pipeline consists of: |
| 46 | + |
| 47 | +- An EventBridge rule that captures events from "my-custom-app" source |
| 48 | +- An input SQS FIFO queue that receives events from the EventBridge rule |
| 49 | +- An EventBridge pipe that processes messages from the input queue and dynamically sets the message group ID for the target SQS FIFO queue |
| 50 | +- An output SQS FIFO queue that receives processed messages |
| 51 | + |
| 52 | +A key component of this pattern is the syntax in the EventBrige Pipe Target to obtain the correct property value to set the message group id. The following shows an example in the AWS Console. You can also view this configuration in the SAM template under the CustomEventPipe resource. |
| 53 | + |
| 54 | + |
| 55 | +## Testing |
| 56 | + |
| 57 | +To test, you'll send an event from a custom source to EventBridge, which will trigger the EventBridge rule to send the event to an SQS FIFO Queue. You'll then wait for the EventBridge pipe to process and finally verify the message within the destination SQS FIFO queue with the message group ID set. |
| 58 | + |
| 59 | +1. Send a test event to EventBridge. Take a look at the event structure and attributes. The account attribute will be used as the message group ID. |
| 60 | +```bash |
| 61 | +aws events put-events --entries file://events/test-event.json |
| 62 | +``` |
| 63 | + |
| 64 | +2. Wait for a couple seconds for the EventBridge pipe to process the message, then check the output queue. Make sure you replace the queue-url value with the correct output queue URL from deployment. |
| 65 | +```bash |
| 66 | +aws sqs receive-message \ |
| 67 | + --queue-url <OUTPUT_QUEUE_URL> \ |
| 68 | + --attribute-names All \ |
| 69 | + --message-attribute-names All \ |
| 70 | + --max-number-of-messages 10 |
| 71 | +``` |
| 72 | + |
| 73 | +You should see a list of messages. View the event and take note of the MessageGroupID under the Attributes section. This originally came from your event input "accountId" property. The EventBridge Pipe is what allows us to perform this modification. Feel free to modify the event, send additional events to EventBridge and review the output queue messages. |
| 74 | + |
| 75 | +## Cleanup |
| 76 | + |
| 77 | +To remove all resources: |
| 78 | + |
| 79 | +```bash |
| 80 | +sam delete |
| 81 | +``` |
| 82 | + |
| 83 | +---- |
| 84 | +Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved. |
| 85 | + |
| 86 | +SPDX-License-Identifier: MIT-0 |
0 commit comments