Skip to content

Commit 3b27594

Browse files
zac-nixonzac-nixon
authored
l7 documentation, correct some warnings in other docs (#4352)
* l7 documentation, correct some warnings in other docs * update examples to L7 gateway * add invalid backend behavior * add missing fields in lb config documentation
1 parent 0a97512 commit 3b27594

File tree

7 files changed

+774
-30
lines changed

7 files changed

+774
-30
lines changed

‎docs/guide/gateway/customization.md‎

Lines changed: 137 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
## Customizing your ELB resources
22

3-
The AWS Load Balancer Controller (LBC) provides sensible defaults for provisioning and managing Elastic Load Balancing (ELB) resources in response to Kubernetes Gateway API objects. However, to accommodate diverse use cases and specific operational requirements, the LBC offers extensive, fine-grained customization capabilities through two Custom Resource Definitions (CRDs): [LoadBalancerConfiguration](../spec/#loadbalancerconfiguration) and [TargetGroupConfiguration](../spec/#targetgroupconfiguration).
3+
The AWS Load Balancer Controller (LBC) provides sensible defaults for provisioning and managing Elastic Load Balancing (ELB) resources in response to Kubernetes Gateway API objects. However, to accommodate diverse use cases and specific operational requirements, the LBC offers extensive, fine-grained customization capabilities through three Custom Resource Definitions (CRDs): [LoadBalancerConfiguration](./spec.md#loadbalancerconfiguration), [TargetGroupConfiguration](./spec.md/#targetgroupconfiguration), and [ListenerRuleConfiguration](./spec.md#listenerruleconfiguration).
44

55
![screen showing all LBC Gateway API components](assets/gateway-full.png)
66

@@ -28,7 +28,7 @@ This configuration can then be applied by attaching the `LoadBalancerConfigurati
2828
When attached directly to a `Gateway` resource, the specified configuration applies specifically to the Load Balancer provisioned for that individual Gateway.
2929

3030
!!! note
31-
Make sure that the `LoadBalancerConfiguration` must be in same namepace as the `Gateway`.
31+
Make sure that the `LoadBalancerConfiguration` is located in the same namespace as the `Gateway`.
3232

3333
```yaml
3434
apiVersion: gateway.networking.k8s.io/v1
@@ -66,7 +66,7 @@ spec:
6666

6767
#### Conflict Resolution for `LoadBalancerConfiguration`
6868

69-
It is possible for a `LoadBalancerConfiguration` to be attached to both a `Gateway` and its associated `GatewayClass`. In such scenarios, when identical fields are specified in both configurations, the LBC employs a merging algorithm to resolve conflicts. The precedence of values is determined by the `mergingMode` field, which is exclusively read from the `GatewayClass`'s `LoadBalancerConfiguration`. If `mergingMode` is not explicitly set, the `GatewayClass` configuration implicitly takes higher precedence. For more info on `mergingMode`, refer this [doc](../loadbalancerconfig/#mergingmode)
69+
It is possible for a `LoadBalancerConfiguration` to be attached to both a `Gateway` and its associated `GatewayClass`. In such scenarios, when identical fields are specified in both configurations, the LBC employs a merging algorithm to resolve conflicts. The precedence of values is determined by the `mergingMode` field, which is exclusively read from the `GatewayClass`'s `LoadBalancerConfiguration`. If `mergingMode` is not explicitly set, the `GatewayClass` configuration implicitly takes higher precedence. For more info on `mergingMode`, refer to [the merging mode documentation](./loadbalancerconfig.md#mergingmode).
7070

7171
The following fields exhibit specific merge behaviors:
7272

@@ -80,7 +80,7 @@ The following fields exhibit specific merge behaviors:
8080

8181
The `TargetGroupConfiguration` CRD enables granular customization of the AWS Target Groups created for Kubernetes Services.
8282

83-
For a comprehensive overview of configurable parameters, please refer the [TargetGroupConfiguration CRD documentation](./targetgroupconfig.md).
83+
For a comprehensive overview of configurable parameters, please refer to the [TargetGroupConfiguration CRD documentation](./targetgroupconfig.md).
8484

8585
**Example: Default Target Group Configuration for a Service**
8686

@@ -141,9 +141,140 @@ spec:
141141

142142
#### How Default and Route-Specific Configurations Merge
143143

144-
When both `defaultConfiguration` and `routeConfigurations` within a `TargetGroupConfiguration` specify the same field, route-specific configurations take precedence. The controller identifies the most relevant route specification from the list of `routeConfigurations` and merges its `targetGroupProps` with the `defaultConfiguration`'s settings. For detailed information on the route matching logic employed, refer to the [Route Matching section](../targetgroupconfig/#route-matching-logic).
144+
When both `defaultConfiguration` and `routeConfigurations` within a `TargetGroupConfiguration` specify the same field, route-specific configurations take precedence. The controller identifies the most relevant route specification from the list of `routeConfigurations` and merges its `targetGroupProps` with the `defaultConfiguration`'s settings. For detailed information on the route matching logic employed, refer to the [Route Matching section](./targetgroupconfig.md#route-matching-logic).
145145

146146
The following fields exhibit specific merge behaviors:
147147

148148
* **`tags`**: The two tag maps are combined. Any duplicate tag keys will result in the value from the higher-priority (route-specific) configuration being used.
149-
* **`targetGroupAttributes`**: The two attribute lists are combined. Any duplicate attribute keys will result in the attribute value from the higher-priority (route-specific) configuration being applied.
149+
* **`targetGroupAttributes`**: The two attribute lists are combined. Any duplicate attribute keys will result in the attribute value from the higher-priority (route-specific) configuration being applied.
150+
151+
#### Customizing L7 Routing Rules
152+
153+
The `ListenerRuleConfiguration` CRD allows representation of features present in AWS ALB,
154+
that are not represented in the standard Gateway API spec.
155+
156+
An exhaustive list is:
157+
158+
- [Cognito Authentication](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html#cognito-requirements)
159+
- [OIDC Authentication](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-authenticate-users.html#oidc-requirements)
160+
- [Fixed Response](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/rule-action-types.html#fixed-response-actions)
161+
- [Source IP Conditions](https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_SourceIpConditionConfig.html#API_SourceIpConditionConfig_Contents)
162+
163+
For a comprehensive overview of the CRD, please refer to the [ListenerRuleConfiguration CRD documentation](./listenerruleconfig.md).
164+
165+
**Example: Adding source IP routing conditions**
166+
167+
This example adds upon the example found [here](./l7gateway.md#step-by-step-l7-gateway-api-resource-implementation-with-an-example). It adds
168+
a routing rule that only allows requests originating from the range 10.0.0.0/5 to be routed to the backend.
169+
170+
```
171+
# source-ip-condition.yaml
172+
apiVersion: gateway.k8s.aws/v1beta1
173+
kind: ListenerRuleConfiguration
174+
metadata:
175+
name: custom-rule-config-source-ip
176+
namespace: example-ns
177+
spec:
178+
conditions:
179+
- field: source-ip
180+
sourceIPConfig:
181+
values:
182+
- 10.0.0.0/5
183+
---
184+
# updated-http-route.yaml
185+
apiVersion: gateway.networking.k8s.io/v1beta1
186+
kind: HTTPRoute
187+
metadata:
188+
name: my-http-app-route
189+
namespace: example-ns
190+
spec:
191+
parentRefs:
192+
- group: gateway.networking.k8s.io
193+
kind: Gateway
194+
name: my-alb-gateway
195+
sectionName: http
196+
- group: gateway.networking.k8s.io
197+
kind: Gateway
198+
name: my-alb-gateway
199+
sectionName: https
200+
rules:
201+
- backendRefs:
202+
- name: <your service>
203+
port: <your service port>
204+
filters:
205+
- type: ExtensionRef
206+
extensionRef:
207+
group: "gateway.k8s.aws"
208+
kind: "ListenerRuleConfiguration"
209+
name: "custom-rule-config-source-ip"
210+
```
211+
212+
To add granular rules, specify the index match:
213+
214+
```
215+
# source-ip-condition.yaml
216+
apiVersion: gateway.k8s.aws/v1beta1
217+
kind: ListenerRuleConfiguration
218+
metadata:
219+
name: custom-rule-config-source-ip
220+
namespace: example-ns
221+
spec:
222+
conditions:
223+
- field: source-ip
224+
matchIndexes: [0,1]
225+
sourceIPConfig:
226+
values:
227+
- 10.0.0.0/5
228+
---
229+
# updated-http-route.yaml
230+
apiVersion: gateway.networking.k8s.io/v1
231+
kind: HTTPRoute
232+
metadata:
233+
name: http-app-source-ip
234+
namespace: example-ns
235+
spec:
236+
parentRefs:
237+
- name: my-alb-gateway
238+
port: 90
239+
rules:
240+
- backendRefs:
241+
- name: echoserver
242+
port: 80
243+
filters:
244+
- type: ExtensionRef
245+
extensionRef:
246+
group: "gateway.k8s.aws"
247+
kind: "ListenerRuleConfiguration"
248+
name: "custom-rule-config-source-ip"
249+
matches:
250+
- path: # Path Pattern
251+
type: Exact
252+
value: /pathExactMatch
253+
queryParams: # Query String
254+
- name: "user"
255+
value: "john"
256+
method: GET # HTTP Request Method
257+
- path: # Regex path match
258+
type: RegularExpression
259+
value: "/firstRule/some?/users"
260+
- backendRefs:
261+
- name: echoserver
262+
port: 80
263+
filters:
264+
- type: ExtensionRef
265+
extensionRef:
266+
group: "gateway.k8s.aws"
267+
kind: "ListenerRuleConfiguration"
268+
name: "custom-rule-config-source-ip-2"
269+
matches:
270+
- path: # Path Pattern
271+
type: Exact
272+
value: /secondRulePath
273+
method: POST # HTTP Request Method
274+
- path: # Regex path match
275+
type: RegularExpression
276+
value: "/secondRule/some?/users"
277+
- path:
278+
type: "PathPrefix"
279+
value: "/secondRule"
280+
```

‎docs/guide/gateway/gateway.md‎

Lines changed: 72 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Gateway API
22

33
!!! warning
4-
- Only very basic (and not conforming) support of the Gateway API spec currently exists. The team is actively trying to close conformance and support gaps.
4+
- The team is actively trying to close conformance and support gaps.
55
- Using the LBC and Gateway API together is not suggested for production workloads (yet!)
66

77

@@ -11,13 +11,10 @@ Mixing protocol layers, e.g. TCPRoute and HTTPRoute on the same Gateway, is not
1111

1212
## Current Support
1313

14-
!!! warning
15-
- GRPCRoute and HTTPS Listeners for L7 gateways do not currently work. And only basic support is added for HTTPRoute.
16-
1714
The LBC Gateway API implementation supports the following Gateway API routes:
1815

1916
* L4 (NLBGatewayAPI): UDPRoute, TCPRoute, TLSRoute >=v2.13.3
20-
* L7 (ALBGatewayAPI): HTTPRoute (Still work in progress, support matching and filtering is not added yet!)
17+
* L7 (ALBGatewayAPI): HTTPRoute, GRPCRoute >= 2.14.0
2118

2219
## Prerequisites
2320
* LBC >= v2.13.0
@@ -48,6 +45,13 @@ See [Subnet Discovery](../../deploy/subnet_discovery.md) for details on configur
4845
!!! tip "disable worker node security group rule management"
4946
You can disable the worker node security group rule management using the [LoadBalancerConfiguration CRD](./loadbalancerconfig.md).
5047

48+
## Certificate Discovery for secure listeners
49+
50+
Both L4 and L7 Gateway implementations support static certificate configuration and certificate discovery using Listener hostname.
51+
The caveat is that configuration of TLS certificates can not be done via the `certificateRefs` field of a Gateway Listener,
52+
as the controller only supports certificate references via an ARN. In the future, we may support syncing Kubernetes secrets into ACM.
53+
54+
5155
### Worker node security groups selection
5256
The controller automatically selects the worker node security groups that it modifies to allow inbound traffic using the following rules:
5357

@@ -64,4 +68,66 @@ The controller automatically selects the worker node security groups that it mod
6468

6569
`${cluster-name}` is the name of the Kubernetes cluster.
6670

67-
If it is possible for multiple security groups with the tag `kubernetes.io/cluster/${cluster-name}` to be on a target ENI, you may use the `--service-target-eni-security-group-tags` flag to specify additional tags that must also match in order for a security group to be used.
71+
If it is possible for multiple security groups with the tag `kubernetes.io/cluster/${cluster-name}` to be on a target ENI, you may use the `--service-target-eni-security-group-tags` flag to specify additional tags that must also match in order for a security group to be used.
72+
73+
74+
## Misconfigured Services
75+
76+
The L4 and L7 gateways handle misconfigured services differently.
77+
78+
79+
### L4
80+
81+
```
82+
# my-tcproute.yaml
83+
apiVersion: gateway.networking.k8s.io/v1beta1
84+
kind: TCPRoute
85+
metadata:
86+
name: my-tcp-app-route
87+
namespace: example-ns
88+
spec:
89+
parentRefs:
90+
- group: gateway.networking.k8s.io
91+
kind: Gateway
92+
name: my-tcp-gateway
93+
sectionName: tcp-app # Refers to the specific listener on the Gateway
94+
rules:
95+
- backendRefs:
96+
- name: my-tcp-service # Kubernetes Service
97+
port: 9000
98+
```
99+
100+
When `my-tcp-service` or the configured service port can't be found,
101+
the target group will not be materialized on any NLBs that the route attaches to.
102+
103+
104+
### L7
105+
106+
```
107+
apiVersion: gateway.networking.k8s.io/v1beta1
108+
kind: HTTPRoute
109+
metadata:
110+
name: my-http-app-route
111+
namespace: example-ns
112+
spec:
113+
parentRefs:
114+
- group: gateway.networking.k8s.io
115+
kind: Gateway
116+
name: my-alb-gateway
117+
sectionName: http
118+
- group: gateway.networking.k8s.io
119+
kind: Gateway
120+
name: my-alb-gateway
121+
sectionName: https
122+
rules:
123+
- backendRefs:
124+
- name: my-http-service
125+
port: 9000
126+
```
127+
128+
When `my-http-service` or the configured service port can't be found,
129+
the target group will not be materialized on any ALBs that the route attaches to.
130+
An [503 Fixed Response](https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_FixedResponseActionConfig.html)
131+
will be added to any Listener Rules that would have referenced the invalid backend.
132+
133+

‎docs/guide/gateway/l4gateway.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Gateway API for Layer 4 (NLB) Implementation
22

3-
This section details the **AWS Load Balancer Controller's (LBC)** architecture and operational flow when processing Gateway API resources for Layer 4 traffic.
3+
This section details the **AWS Load Balancer Controller's (LBC)** architecture and operational flow when processing Gateway API resources for Layer 4 traffic utilizing AWS NLB.
44

55
### Gateway API Resources and Controller Architecture
66

@@ -13,8 +13,9 @@ The LBC instances dedicated to L4 routing monitor the following Gateway API reso
1313
* **`TLSRoute`**: Defines TLS-specific routing rules, enabling secure Layer 4 communication. These routes are satisfied by an **AWS NLB**.
1414
* **`TCPRoute`**: Defines TCP-specific routing rules, facilitating direct TCP traffic management. These routes are satisfied by an **AWS NLB**.
1515
* **`UDPRoute`**: Defines UDP-specific routing rules, facilitating UDP traffic management. These routes are satisfied by an **AWS NLB**.
16-
* **`LoadBalancerConfiguration` (LBC CRD)**: A Custom Resource Definition utilized for fine-grained customization of the provisioned NLB. This CRD can be attached to a `Gateway` or its `GatewayClass`. For more info, please refer [How customization works](../customization)
17-
* **`TargetGroupConfiguration` (LBC CRD)**: A Custom Resource Definition used for service-specific customizations of AWS Target Groups. This CRD is associated with a Kubernetes `Service`. For more info, please refer [How customization works](../customization)
16+
* **`ReferenceGrant`**: Defines cross-namespace access. For more information [see](https://gateway-api.sigs.k8s.io/api-types/referencegrant/)
17+
* **`LoadBalancerConfiguration` (LBC CRD)**: A Custom Resource Definition utilized for fine-grained customization of the provisioned NLB. This CRD can be attached to a `Gateway` or its `GatewayClass`. For more info, please refer [How customization works](customization.md#customizing-the-gateway-load-balancer-using-loadbalancerconfiguration-crd)
18+
* **`TargetGroupConfiguration` (LBC CRD)**: A Custom Resource Definition used for service-specific customizations of AWS Target Groups. This CRD is associated with a Kubernetes `Service`. For more info, please refer [How customization works](customization.md#customizing-services-target-groups-using-targetgroupconfiguration-crd)
1819

1920
### The Reconciliation Loop
2021

@@ -82,7 +83,7 @@ spec:
8283
* **API Event Detection:** The LBC's L4 controller continuously monitors the Kubernetes API server. Upon detecting the `aws-nlb-gateway-class` (with `controllerName: gateway.k8s.aws/nlb`), the `my-tcp-gateway` (referencing this `GatewayClass`), and `my-tcp-app-route` (referencing `my-tcp-gateway`'s `tcp-app` listener) resources, it recognizes its responsibility to manage these objects and initiates the provisioning of AWS resources.
8384
* **NLB Provisioning:** An **AWS Network Load Balancer (NLB)** is provisioned in AWS for the `my-tcp-gateway` resource with default settings. At this stage, the NLB is active but does not yet have any configured listeners. As soon as the NLB becomes active, the status of the gateway is updated.
8485
* **L4 Listener Materialization:** The controller processes the `my-tcp-app-route` resource. Given that the `TCPRoute` validly references the `my-tcp-gateway` and its `tcp-app` listener, an **NLB Listener** is materialized on the provisioned NLB. This listener will be configured for `TCP` protocol on `port 8080`, as specified in the `Gateway`'s listener definition. A default forward action is subsequently configured on the NLB Listener, directing all incoming traffic on `port 8080` to the newly created Target Group for service `my-tcp-service` in `backendRefs` section of `my-tcp-app-route`.
85-
* **Target Group Creation:** An **AWS Target Group** is created for the Kubernetes Service `my-tcp-service` with default configuration. The Pods associated with `my-tcp-service` are then registered as targets within this new Target Group.
86+
* **Target Group Creation:** An **AWS Target Group** is created for the Kubernetes Service `my-tcp-service` with default configuration. The cluster nodes are then registered as targets within this new Target Group.
8687

8788
### L4 Gateway API Limitations for NLBs
8889
The LBC implementation of the Gateway API for L4 routes, which provisions NLB, introduces specific constraints to align with NLB capabilities. These limitations are enforced during the reconciliation process and are critical for successful L4 traffic management.

0 commit comments

Comments
 (0)