Skip to content

Commit 5dec21e

Browse files
authored
feat(kinesisfirehose): add built-in data processors to decompress CloudWatch logs and extract messages (#33749)
### Issue # (if applicable) Closes #33691. Closes #20242. ### Reason for this change Data Firehose supports following convenient conversions: - [Decompress CloudWatch logs](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-cloudwatch-logs-decompression.html) - [Extract message after decompression of CloudWatch Logs](https://docs.aws.amazon.com/firehose/latest/dev/Message_extraction.html) - [Add a new line delimiter when delivering data to Amazon S3](https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning-s3bucketprefix.html#dynamic-partitioning-new-line-delimiter) This PR adds above 3 processors because: - Decompression (without CloudWatchLogProcessing) may need AppendDelimiterToRecord - CloudWatchLogProcessing requires Decompression ### Description of changes - introduce `processors` (plural) prop to specify multiple processors - deprecate `processor` (singular) prop in favor of `processors` - add built-in processor classes: - `DecompressionProcessor` - Decompress CloudWatch logs - `CloudWatchLogProcessor` - Extract message after decompression of CloudWatch Logs - `AppendDelimiterToRecordProcessor` - Add a new line delimiter when delivering data to Amazon S3 ### Describe any new or updated permissions being added N/A - No permissions are required by these built-in processors. ### Description of how you validated changes Added unit tests and integ tests see also #33691 (comment) ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
1 parent 551018b commit 5dec21e

File tree

22 files changed

+2443
-27
lines changed

22 files changed

+2443
-27
lines changed

packages/@aws-cdk-testing/framework-integ/test/aws-kinesisfirehose/test/integ.cloudwatch-logs-processors.js.snapshot/asset.44e9c4d7a5d3fd2d677e1a7e416b2b56f6b0104bd5eff9cac5557b4c65a9dc61/index.js

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk-testing/framework-integ/test/aws-kinesisfirehose/test/integ.cloudwatch-logs-processors.js.snapshot/asset.52cf96d8e37139faa98832bfcf5d3af4afc4e7353b74595c3f179e45410e31cb/index.js

Lines changed: 39 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk-testing/framework-integ/test/aws-kinesisfirehose/test/integ.cloudwatch-logs-processors.js.snapshot/cdk.out

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/@aws-cdk-testing/framework-integ/test/aws-kinesisfirehose/test/integ.cloudwatch-logs-processors.js.snapshot/firehose-delivery-stream-cloudwatch-logs-processors.assets.json

Lines changed: 48 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)