Commit 5dec21e
authored
feat(kinesisfirehose): add built-in data processors to decompress CloudWatch logs and extract messages (#33749)
### Issue # (if applicable)
Closes #33691.
Closes #20242.
### Reason for this change
Data Firehose supports following convenient conversions:
- [Decompress CloudWatch logs](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-cloudwatch-logs-decompression.html)
- [Extract message after decompression of CloudWatch Logs](https://docs.aws.amazon.com/firehose/latest/dev/Message_extraction.html)
- [Add a new line delimiter when delivering data to Amazon S3](https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning-s3bucketprefix.html#dynamic-partitioning-new-line-delimiter)
This PR adds above 3 processors because:
- Decompression (without CloudWatchLogProcessing) may need AppendDelimiterToRecord
- CloudWatchLogProcessing requires Decompression
### Description of changes
- introduce `processors` (plural) prop to specify multiple processors
- deprecate `processor` (singular) prop in favor of `processors`
- add built-in processor classes:
- `DecompressionProcessor` - Decompress CloudWatch logs
- `CloudWatchLogProcessor` - Extract message after decompression of CloudWatch Logs
- `AppendDelimiterToRecordProcessor` - Add a new line delimiter when delivering data to Amazon S3
### Describe any new or updated permissions being added
N/A - No permissions are required by these built-in processors.
### Description of how you validated changes
Added unit tests and integ tests
see also #33691 (comment)
### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)
----
*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*1 parent 551018b commit 5dec21e
File tree
22 files changed
+2443
-27
lines changed- packages
- @aws-cdk-testing/framework-integ/test/aws-kinesisfirehose/test
- integ.cloudwatch-logs-processors.js.snapshot
- asset.44e9c4d7a5d3fd2d677e1a7e416b2b56f6b0104bd5eff9cac5557b4c65a9dc61
- asset.52cf96d8e37139faa98832bfcf5d3af4afc4e7353b74595c3f179e45410e31cb
- aws-cdk-lib/aws-kinesisfirehose
- lib
- private
- processors
- test
22 files changed
+2443
-27
lines changedSome generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments