Skip to content

Commit 23eac60

Browse files
julianwoodjulianwood
committed
2 parents 34b4ed3 + f8d4f06 commit 23eac60

File tree

24 files changed

+1655
-0
lines changed

24 files changed

+1655
-0
lines changed

‎apigw-websocket-api-bedrock-streaming-rust-cdk/.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
*.js
2+
!jest.config.js
3+
*.d.ts
4+
node_modules
5+
.DS_Store
6+
7+
# CDK asset staging directory
8+
.cdk.staging
9+
cdk.out
10+

‎apigw-websocket-api-bedrock-streaming-rust-cdk/README.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Streaming Amazon Bedrock response with Amazon API Gateway WebSocket API and AWS Lambda
2+
3+
This CDK application demonstrates a simple, serverless approach to integrating Amazon Bedrock with AWS Lambda and Amazon API Gateway. Written in Rust, it showcases how to efficiently stream responses from Amazon Bedrock to a client via WebSocket connections. The example serves as a practical illustration of implementing real-time, serverless communication between Bedrock's GenAI capabilities and a client application.
4+
5+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
6+
7+
## Requirements
8+
9+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
10+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
11+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
12+
* [Node and NPM](https://nodejs.org/en/download/) installed
13+
* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) (AWS CDK) installed
14+
* [Docker](https://docs.docker.com/engine/install/) installed and running locally (needed for Rust cross-platform Lambda build)
15+
* [Rust](https://www.rust-lang.org/) 🦀 installed with v1.79.0 or higher
16+
* [Cargo Lambda](https://www.cargo-lambda.info/) installed
17+
* [cross](https://github.com/cross-rs/cross) compilation installed for Cargo Lambda: `cargo install cross --git https://github.com/cross-rs/cross`
18+
* [wscat](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-how-to-call-websocket-api-wscat.html) installed for CLI WebSocket capabilities
19+
20+
21+
## Amazon Bedrock Setup Instructions
22+
23+
You must request access to the Bedrock LLM model before you can use it. This example uses `Claude 3 Sonnet`, so make sure you have `Access granted` to this model. For more information, see [Model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html).
24+
25+
## Deployment Instructions
26+
27+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
28+
```bash
29+
git clone https://github.com/aws-samples/serverless-patterns
30+
```
31+
2. Change directory to the pattern's CDK directory:
32+
```bash
33+
cd apigw-websocket-api-bedrock-streaming-rust-cdk/cdk
34+
```
35+
3. From the command line, use npm to install the development dependencies:
36+
```bash
37+
npm install
38+
```
39+
4. If you haven't done so previously for this account, run this command to bootstrap CDK:
40+
```bash
41+
cdk bootstrap
42+
```
43+
5. Review the CloudFormation template that CDK generates for your stack using the following AWS CDK CLI command:
44+
```bash
45+
cdk synth
46+
```
47+
6. Use AWS CDK to deploy your AWS resources
48+
```bash
49+
cdk deploy
50+
```
51+
52+
After the deployment completes, note the URL in the `Outputs` section at the end. The `BedrockStreamerStack.WebSocketURL` followed by the WebSocket URL will be used to connect to API Gateway. It should look something like `wss://{YOUR_API_ID_HERE}.execute-api.{YOUR_REGION_HERE}.amazonaws.com/prod`
53+
54+
## How it works
55+
56+
This pattern establishes a WebSocket connection to Amazon API Gateway. When requests are made to this API, API Gateway routes them to an AWS Lambda function. The Lambda function then initiates a streaming request to Amazon Bedrock using the [ConverseStream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html) API. This allows the response from the LLM in Bedrock to start streaming back to the Lambda function as soon as generation begins, without waiting for the entire response to be ready.
57+
58+
Generating Bedrock responses often takes a long time. By using a streaming approach, the Lambda function can immediately start processing the incoming response and writing the data chunks to the API Gateway WebSocket. The WebSocket then delivers these chunks in real-time to the connected client, providing an extremely interactive user experience.
59+
60+
## Testing
61+
62+
1. From your terminal, use `wscat` to connect to API Gateway using the WebSocket API and generate a short story about `CATS` by entering the `{"storyType": "CATS"}` line after `wscat` startup.
63+
```bash
64+
# Connect to the API Gateway via WebSocket
65+
wscat -c <API_GATEWAY_URL_FROM_CDK_OUTPUT>
66+
67+
Connected (press CTRL+C to quit)
68+
> {"storyType": "CATS"} <--- ENTER THIS...PRESS RETURN
69+
< {"type":"other","message":null}
70+
< {"type":"text","message":"Here"}
71+
< {"type":"text","message":" is"}
72+
< {"type":"text","message":" a"}
73+
< {"type":"text","message":" very"}
74+
< {"type":"text","message":" short"}
75+
< {"type":"text","message":" story"}
76+
< {"type":"text","message":" about"}
77+
< {"type":"text","message":" cats"}
78+
< {"type":"text","message":":"}
79+
< {"type":"text","message":"\n\nMitt"}
80+
< {"type":"text","message":"ens"}
81+
< {"type":"text","message":" cur"}
82+
< {"type":"text","message":"le"}
83+
< {"type":"text","message":"d up"}
84+
< {"type":"text","message":" on"}
85+
< {"type":"text","message":" the"}
86+
< {"type":"text","message":" window"}
87+
< {"type":"text","message":"s"}
88+
< {"type":"text","message":"ill"}
89+
< {"type":"text","message":","}
90+
.
91+
.
92+
< {"type":"other","message":null}
93+
< {"type":"other","message":null}
94+
< {"type":"other","message":null}
95+
> ⏎ <--- CTRL+C HERE>
96+
```
97+
2. As the `wscat` CLI says, press `CTRL+C` to disconnect
98+
99+
## Cleanup
100+
101+
You can use the following commands to destroy the AWS resources created during deployment. This assumes you're currently at the `apigw-websocket-api-bedrock-streaming-rust-cdk/cdk` directory in your terminal:
102+
103+
```bash
104+
cdk destroy
105+
```
106+
----
107+
Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
108+
109+
SPDX-License-Identifier: MIT-0

‎apigw-websocket-api-bedrock-streaming-rust-cdk/apigw-websocket-api-bedrock-streaming-rust-cdk.json‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
{
2+
"title": "Streaming Amazon Bedrock response with Amazon API Gateway",
3+
"description": "Stream an Amazon Bedrock LLM response with API Gateway WebSocket API and AWS Lambda function .",
4+
"language": "Rust",
5+
"level": "200",
6+
"framework": "CDK",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This pattern establishes a WebSocket connection to Amazon API Gateway. When requests are made to this API, API Gateway routes them to an AWS Lambda function.",
11+
"The Lambda function then initiates a streaming request to Amazon Bedrock using the ConverseStream API. This allows the response from the LLM in Bedrock to start streaming back to the Lambda function as soon as generation begins, without waiting for the entire response to be ready."
12+
]
13+
},
14+
"gitHub": {
15+
"template": {
16+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/apigw-websocket-api-bedrock-streaming-rust-cdk",
17+
"templateURL": "serverless-patterns/apigw-websocket-api-bedrock-streaming-rust-cdk",
18+
"projectFolder": "apigw-websocket-api-bedrock-streaming-rust-cdk",
19+
"templateFile": "cdk/lib/bedrock-streamer-stack.ts"
20+
}
21+
},
22+
"resources": {
23+
"bullets": [
24+
{
25+
"text": "Working with WebSocket APIs",
26+
"link": "https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api.html"
27+
},
28+
{
29+
"text": "Amazon Bedrock",
30+
"link": "https://aws.amazon.com/bedrock/"
31+
},
32+
{
33+
"text": "Cloud Development Kit",
34+
"link": "https://docs.aws.amazon.com/cdk/v2/guide/home.html"
35+
},
36+
{
37+
"text": "AWS SDK for Rust",
38+
"link": "https://aws.amazon.com/sdk-for-rust/"
39+
}
40+
]
41+
},
42+
"deploy": {
43+
"text": [
44+
"cdk deploy"
45+
]
46+
},
47+
"testing": {
48+
"text": [
49+
"See the GitHub repo for detailed testing instructions."
50+
]
51+
},
52+
"cleanup": {
53+
"text": [
54+
"Delete the stack: <code>cdk destroy</code>."
55+
]
56+
},
57+
"authors": [
58+
{
59+
"name": "Phil Callister",
60+
"image": "https://media.licdn.com/dms/image/v2/D5603AQHElNV-QFIGQw/profile-displayphoto-shrink_200_200/profile-displayphoto-shrink_200_200/0/1725218566874?e=1735171200&v=beta&t=gr82L1aMjvNX8CZn1Nitg3fViTUYoPK_xnXNc8CdohQ",
61+
"bio": "I'm an Enterprise Solutions Architect at AWS, with a focus on Financial Services. As a passionate builder, I enjoy helping customers create innovative solutions to achieve their business objectives.",
62+
"linkedin": "philcallister"
63+
}
64+
],
65+
"patternArch": {
66+
"icon1": {
67+
"x": 20,
68+
"y": 50,
69+
"service": "apigw",
70+
"label": "API Gateway Websocket API"
71+
},
72+
"icon2": {
73+
"x": 50,
74+
"y": 50,
75+
"service": "lambda",
76+
"label": "AWS Lambda"
77+
},
78+
"icon3": {
79+
"x": 80,
80+
"y": 50,
81+
"service": "bedrock",
82+
"label": "Amazon Bedrock"
83+
},
84+
"line1": {
85+
"from": "icon1",
86+
"to": "icon2"
87+
},
88+
"line3": {
89+
"from": "icon2",
90+
"to": "icon3"
91+
}
92+
}
93+
}

‎apigw-websocket-api-bedrock-streaming-rust-cdk/cdk/.npmignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out

‎apigw-websocket-api-bedrock-streaming-rust-cdk/cdk/bin/bedrock-streamer.ts‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/usr/bin/env node
2+
import 'source-map-support/register';
3+
import * as cdk from 'aws-cdk-lib';
4+
import { BedrockStreamerStack } from '../lib/bedrock-streamer-stack';
5+
6+
const app = new cdk.App();
7+
const description = "Serverlessland Bedrock streamining pattern. (uksb-1tthgi812) (tag:apigw-websocket-api-bedrock-streamining-rust-cdk)"
8+
new BedrockStreamerStack(app, 'BedrockStreamerStack', {description:description
9+
/* If you don't specify 'env', this stack will be environment-agnostic.
10+
* Account/Region-dependent features and context lookups will not work,
11+
* but a single synthesized template can be deployed anywhere. */
12+
13+
/* Uncomment the next line to specialize this stack for the AWS Account
14+
* and Region that are implied by the current CLI configuration. */
15+
// env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
16+
17+
/* Uncomment the next line if you know exactly what Account and Region you
18+
* want to deploy the stack to. */
19+
// env: { account: '123456789012', region: 'us-east-1' },
20+
21+
/* For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html */
22+
});

‎apigw-websocket-api-bedrock-streaming-rust-cdk/cdk/cdk.json‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/bedrock-streamer.ts",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"**/*.d.ts",
11+
"**/*.js",
12+
"tsconfig.json",
13+
"package*.json",
14+
"yarn.lock",
15+
"node_modules",
16+
"test"
17+
]
18+
},
19+
"context": {
20+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
21+
"@aws-cdk/core:checkSecretUsage": true,
22+
"@aws-cdk/core:target-partitions": [
23+
"aws",
24+
"aws-cn"
25+
],
26+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
27+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
28+
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
29+
"@aws-cdk/aws-iam:minimizePolicies": true,
30+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
31+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
32+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
33+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
34+
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
35+
"@aws-cdk/core:enablePartitionLiterals": true,
36+
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
37+
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
38+
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
39+
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
40+
"@aws-cdk/aws-route53-patters:useCertificate": true,
41+
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
42+
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
43+
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
44+
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
45+
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
46+
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
47+
"@aws-cdk/aws-redshift:columnId": true,
48+
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
49+
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
50+
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
51+
"@aws-cdk/aws-kms:aliasNameRef": true,
52+
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
53+
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
54+
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
55+
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
56+
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
57+
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
58+
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
59+
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
60+
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
61+
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
62+
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
63+
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
64+
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
65+
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
66+
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
67+
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
68+
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
69+
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false
70+
}
71+
}

‎apigw-websocket-api-bedrock-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/target

‎apigw-websocket-api-bedrock-streaming-rust-cdk/cdk/lib/bedrock-streamer-lambda/Cargo.toml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
[package]
2+
name = "bedrock-streamer"
3+
version = "0.1.0"
4+
edition = "2021"
5+
6+
[dependencies]
7+
aws_lambda_events = { version = "0.11.1", default-features = false, features = ["apigw"] }
8+
aws-config = "1.1.1"
9+
aws-sdk-bedrockruntime = "1.53.0"
10+
aws-sdk-apigatewaymanagement = "1.39.0"
11+
bytes = "1.5.0"
12+
http = "1.1.0"
13+
lambda_runtime = "0.9.1"
14+
serde = { version = "1.0.193", features = ["derive"] }
15+
serde_json = "1.0.108"
16+
tokio = { version = "1.34.0", features = ["full"] }
17+
tracing = { version = "0.1", features = ["log"] }
18+
tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt"] }
19+
20+
[profile.release]
21+
opt-level = "z"
22+
strip = true
23+
lto = true
24+
codegen-units = 1

0 commit comments

Comments
 (0)