Skip to content

Commit d9f2d94

Browse files
committed
instructions to verify filtering
1 parent a823e17 commit d9f2d94

File tree

1 file changed

+38
-0
lines changed

1 file changed

+38
-0
lines changed

msk-lambda-schema-avro-java-sam/README.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,44 @@ The Key and Value are base64 encoded and have to be decoded. A message can also
160160

161161
The code in this example prints out the fields in the Kafka message and also decrypts the key and the value and logs them in Cloudwatch logs.
162162

163+
### Message Filtering with Event Source Mapping
164+
165+
This sample application demonstrates how to use event source mapping filters with Amazon MSK and Lambda. The producer Lambda function generates contacts with zip codes that start with either "1000" or "2000" (with approximately 50% probability for each). However, the consumer Lambda function is configured to only process messages where the zip code starts with "1000".
166+
167+
#### Filter Configuration
168+
169+
The filter is configured in the SAM template using the `FilterCriteria` property of the MSK event source mapping:
170+
171+
```yaml
172+
FilterCriteria:
173+
Filters:
174+
- Pattern: '{ "value": { "zip": [ { "prefix": "1000" } ] }}'
175+
```
176+
177+
This filter pattern instructs the event source mapping to only send messages to the Lambda function if the message value contains a "zip" field that starts with "1000".
178+
179+
#### Verifying the Filter Behavior
180+
181+
To verify that the filter is working correctly, follow these steps:
182+
183+
1. **Invoke the producer Lambda function** using one of the methods described above.
184+
185+
2. **Check the producer function logs** in CloudWatch:
186+
- Navigate to the CloudWatch Logs console
187+
- Find the log group for the producer function (`/aws/lambda/msk-lambda-schema-avro-java-sam-LambdaMSKProducerJavaFunction-XXXXXXXXXXXX`)
188+
- Open the most recent log stream
189+
- Look for the "ZIP CODE DISTRIBUTION SUMMARY" section, which shows how many messages were generated with zip codes starting with "1000" and how many with "2000"
190+
- You should see that the producer generated a mix of both zip code types
191+
192+
3. **Check the consumer function logs** in CloudWatch:
193+
- Navigate to the CloudWatch Logs console
194+
- Find the log group for the consumer function (`/aws/lambda/msk-lambda-schema-avro-java-sam-LambdaMSKConsumerJavaFunction-XXXXXXXXXXXX`)
195+
- Open the most recent log stream
196+
- You should see that the consumer only processed messages with zip codes starting with "1000"
197+
- Messages with zip codes starting with "2000" were filtered out by the event source mapping and never reached the Lambda function
198+
199+
This demonstrates how event source mapping filters can be used to efficiently process only the messages that match specific criteria, reducing Lambda invocation costs and processing overhead.
200+
163201
## Cleanup
164202

165203
You can first clean-up the Lambda function by running the sam delete command

0 commit comments

Comments
 (0)