Skip to content

Commit f5ab058

Browse files
wozniakjanjkremser
andauthored
support portName in HTTPScaledObject service scaleTargetRef (#1174)
* support portName in HTTPScaledObject service scaleTargetRef Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> * mutually exclusive port and portName Co-authored-by: Jirka Kremser <535866+jkremser@users.noreply.github.com> Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> * make manifests Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> * fix CEL syntax Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> * e2e test for portName Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> * use service lister instead of endpoints cache to get port from portName Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> * docs for v0.8.1 HTTPScaledObject Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> --------- Signed-off-by: Jan Wozniak <wozniak.jan@gmail.com> Co-authored-by: Jirka Kremser <535866+jkremser@users.noreply.github.com>
1 parent 5d2e0ad commit f5ab058

File tree

12 files changed

+592
-18
lines changed

12 files changed

+592
-18
lines changed

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ This changelog keeps track of work items that have been completed and are ready
2323

2424
### New
2525

26+
- **General**: Support portName in HTTPScaledObject service scaleTargetRef ([#1174](https://github.com/kedacore/http-add-on/issues/1174))
2627
- **General**: Support setting multiple TLS certs for different domains on the interceptor proxy ([#1116](https://github.com/kedacore/http-add-on/issues/1116))
2728
- **General**: TODO ([#TODO](https://github.com/kedacore/http-add-on/issues/TODO))
2829

config/crd/bases/http.keda.sh_httpscaledobjects.yaml

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,9 @@ spec:
9393
type: integer
9494
type: object
9595
scaleTargetRef:
96-
description: The name of the deployment to route HTTP requests to
97-
(and to autoscale).
96+
description: |-
97+
The name of the deployment to route HTTP requests to (and to autoscale).
98+
Including validation as a requirement to define either the PortName or the Port
9899
properties:
99100
apiVersion:
100101
type: string
@@ -106,13 +107,18 @@ spec:
106107
description: The port to route to
107108
format: int32
108109
type: integer
110+
portName:
111+
description: The port to route to referenced by name
112+
type: string
109113
service:
110114
description: The name of the service to route to
111115
type: string
112116
required:
113-
- port
114117
- service
115118
type: object
119+
x-kubernetes-validations:
120+
- message: must define either the 'portName' or the 'port'
121+
rule: has(self.portName) != has(self.port)
116122
scaledownPeriod:
117123
description: (optional) Cooldown period value
118124
format: int32

config/interceptor/role.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,14 @@ rules:
1212
- get
1313
- list
1414
- watch
15+
- apiGroups:
16+
- ""
17+
resources:
18+
- services
19+
verbs:
20+
- get
21+
- list
22+
- watch
1523
- apiGroups:
1624
- http.keda.sh
1725
resources:
Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
# The `HTTPScaledObject`
2+
3+
>This document reflects the specification of the `HTTPScaledObject` resource for the `v0.8.1` version.
4+
5+
Each `HTTPScaledObject` looks approximately like the below:
6+
7+
```yaml
8+
kind: HTTPScaledObject
9+
apiVersion: http.keda.sh/v1alpha1
10+
metadata:
11+
name: xkcd
12+
annotations:
13+
httpscaledobject.keda.sh/skip-scaledobject-creation: "false"
14+
spec:
15+
hosts:
16+
- myhost.com
17+
pathPrefixes:
18+
- /test
19+
scaleTargetRef:
20+
name: xkcd
21+
kind: Deployment
22+
apiVersion: apps/v1
23+
service: xkcd
24+
port: 8080
25+
replicas:
26+
min: 5
27+
max: 10
28+
scaledownPeriod: 300
29+
scalingMetric: # requestRate and concurrency are mutually exclusive
30+
requestRate:
31+
granularity: 1s
32+
targetValue: 100
33+
window: 1m
34+
concurrency:
35+
targetValue: 100
36+
```
37+
38+
This document is a narrated reference guide for the `HTTPScaledObject`.
39+
40+
## `httpscaledobject.keda.sh/skip-scaledobject-creation` annotation
41+
42+
This annotation will disable the ScaledObject generation and management but keeping the routing and metrics available. This is done removing the current ScaledObject if it has been already created, allowing to use user managed ScaledObjects pointing the add-on scaler directly (supporting all the ScaledObject configurations and multiple triggers). You can read more about this [here](./../../walkthrough.md#integrating-http-add-on-scaler-with-other-keda-scalers)
43+
44+
45+
## `hosts`
46+
47+
These are the hosts to apply this scaling rule to. All incoming requests with one of these values in their `Host` header will be forwarded to the `Service` and port specified in the below `scaleTargetRef`, and that same `scaleTargetRef`'s workload will be scaled accordingly.
48+
49+
## `pathPrefixes`
50+
51+
>Default: "/"
52+
53+
These are the paths to apply this scaling rule to. All incoming requests with one of these values as path prefix will be forwarded to the `Service` and port specified in the below `scaleTargetRef`, and that same `scaleTargetRef`'s workload will be scaled accordingly.
54+
55+
## `scaleTargetRef`
56+
57+
This is the primary and most important part of the `spec` because it describes:
58+
59+
1. The incoming host to apply this scaling rule to.
60+
2. What workload to scale.
61+
3. The service to which to route HTTP traffic.
62+
63+
### `deployment` (DEPRECTATED: removed as part of v0.9.0)
64+
65+
This is the name of the `Deployment` to scale. It must exist in the same namespace as this `HTTPScaledObject` and shouldn't be managed by any other autoscaling system. This means that there should not be any `ScaledObject` already created for this `Deployment`. The HTTP Add-on will manage a `ScaledObject` internally.
66+
67+
### `name`
68+
69+
This is the name of the workload to scale. It must exist in the same namespace as this `HTTPScaledObject` and shouldn't be managed by any other autoscaling system. This means that there should not be any `ScaledObject` already created for this workload. The HTTP Add-on will manage a `ScaledObject` internally.
70+
71+
### `kind`
72+
73+
This is the kind of the workload to scale.
74+
75+
### `apiVersion`
76+
77+
This is the apiVersion of the workload to scale.
78+
79+
### `service`
80+
81+
This is the name of the service to route traffic to. The add-on will create autoscaling and routing components that route to this `Service`. It must exist in the same namespace as this `HTTPScaledObject` and should route to the same `Deployment` as you entered in the `deployment` field.
82+
83+
### `port`
84+
85+
This is the port to route to on the service that you specified in the `service` field. It should be exposed on the service and should route to a valid `containerPort` on the `Deployment` you gave in the `deployment` field.
86+
87+
### `portName`
88+
89+
Alternatively, the port can be referenced using it's `name` as defined in the `Service`.
90+
91+
### `targetPendingRequests` (DEPRECTATED: removed as part of v0.9.0)
92+
93+
>Default: 100
94+
95+
This is the number of _pending_ (or in-progress) requests that your application needs to have before the HTTP Add-on will scale it. Conversely, if your application has below this number of pending requests, the HTTP add-on will scale it down.
96+
97+
For example, if you set this field to 100, the HTTP Add-on will scale your app up if it sees that there are 200 in-progress requests. On the other hand, it will scale down if it sees that there are only 20 in-progress requests. Note that it will _never_ scale your app to zero replicas unless there are _no_ requests in-progress. Even if you set this value to a very high number and only have a single in-progress request, your app will still have one replica.
98+
99+
### `scaledownPeriod`
100+
101+
>Default: 300
102+
103+
The period to wait after the last reported active before scaling the resource back to 0.
104+
105+
> Note: This time is measured on KEDA side based on in-flight requests, so workloads with few and random traffic could have unexpected scale to 0 cases. In those case we recommend to extend this period to ensure it doesn't happen.
106+
107+
108+
## `scalingMetric`
109+
110+
This is the second most important part of the `spec` because it describes how the workload has to scale. This section contains 2 nested sections (`requestRate` and `concurrency`) which are mutually exclusive between themselves.
111+
112+
### `requestRate`
113+
114+
This section enables scaling based on the request rate.
115+
116+
> **NOTE**: Requests information is stored in memory, aggragating long periods (longer than 5 minutes) or too fine granularity (less than 1 second) could produce perfomance issues or memory usage increase.
117+
118+
> **NOTE 2**: Although updating `window` and/or `granularity` is something doable, the process just replaces all the stored request count infomation. This can produce unexpected scaling behaviours until the window is populated again.
119+
120+
#### `targetValue`
121+
122+
>Default: 100
123+
124+
This is the target value for the scaling configuration.
125+
126+
#### `window`
127+
128+
>Default: "1m"
129+
130+
This value defines the aggregation window for the request rate calculation.
131+
132+
#### `granularity`
133+
134+
>Default: "1s"
135+
136+
This value defines the granualarity of the aggregated requests for the request rate calculation.
137+
138+
### `concurrency`
139+
140+
This section enables scaling based on the request concurrency.
141+
142+
> **NOTE**: This is the only scaling behaviour before v0.8.0
143+
144+
#### `targetValue`
145+
146+
>Default: 100
147+
148+
This is the target value for the scaling configuration.

interceptor/main.go

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ import (
1717
"github.com/prometheus/client_golang/prometheus/promhttp"
1818
"golang.org/x/exp/maps"
1919
"golang.org/x/sync/errgroup"
20+
k8sinformers "k8s.io/client-go/informers"
2021
"k8s.io/client-go/kubernetes"
2122
ctrl "sigs.k8s.io/controller-runtime"
2223
"sigs.k8s.io/controller-runtime/pkg/log/zap"
@@ -42,6 +43,7 @@ var (
4243

4344
// +kubebuilder:rbac:groups=http.keda.sh,resources=httpscaledobjects,verbs=get;list;watch
4445
// +kubebuilder:rbac:groups="",resources=endpoints,verbs=get;list;watch
46+
// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch
4547

4648
func main() {
4749
timeoutCfg := config.MustParseTimeouts()
@@ -85,11 +87,10 @@ func main() {
8587
setupLog.Error(err, "creating new Kubernetes ClientSet")
8688
os.Exit(1)
8789
}
88-
endpointsCache := k8s.NewInformerBackedEndpointsCache(
89-
ctrl.Log,
90-
cl,
91-
time.Millisecond*time.Duration(servingCfg.EndpointsCachePollIntervalMS),
92-
)
90+
91+
k8sSharedInformerFactory := k8sinformers.NewSharedInformerFactory(cl, time.Millisecond*time.Duration(servingCfg.EndpointsCachePollIntervalMS))
92+
svcCache := k8s.NewInformerBackedServiceCache(ctrl.Log, cl, k8sSharedInformerFactory)
93+
endpointsCache := k8s.NewInformerBackedEndpointsCache(ctrl.Log, cl, time.Millisecond*time.Duration(servingCfg.EndpointsCachePollIntervalMS))
9394
if err != nil {
9495
setupLog.Error(err, "creating new endpoints cache")
9596
os.Exit(1)
@@ -123,6 +124,7 @@ func main() {
123124
setupLog.Info("starting the endpoints cache")
124125

125126
endpointsCache.Start(ctx)
127+
k8sSharedInformerFactory.Start(ctx.Done())
126128
return nil
127129
})
128130

@@ -173,10 +175,11 @@ func main() {
173175
eg.Go(func() error {
174176
proxyTLSConfig := map[string]string{"certificatePath": servingCfg.TLSCertPath, "keyPath": servingCfg.TLSKeyPath, "certstorePaths": servingCfg.TLSCertStorePaths}
175177
proxyTLSPort := servingCfg.TLSPort
178+
k8sSharedInformerFactory.WaitForCacheSync(ctx.Done())
176179

177180
setupLog.Info("starting the proxy server with TLS enabled", "port", proxyTLSPort)
178181

179-
if err := runProxyServer(ctx, ctrl.Log, queues, waitFunc, routingTable, timeoutCfg, proxyTLSPort, proxyTLSEnabled, proxyTLSConfig); !util.IsIgnoredErr(err) {
182+
if err := runProxyServer(ctx, ctrl.Log, queues, waitFunc, routingTable, svcCache, timeoutCfg, proxyTLSPort, proxyTLSEnabled, proxyTLSConfig); !util.IsIgnoredErr(err) {
180183
setupLog.Error(err, "tls proxy server failed")
181184
return err
182185
}
@@ -186,9 +189,11 @@ func main() {
186189

187190
// start a proxy server without TLS.
188191
eg.Go(func() error {
192+
k8sSharedInformerFactory.WaitForCacheSync(ctx.Done())
189193
setupLog.Info("starting the proxy server with TLS disabled", "port", proxyPort)
190194

191-
if err := runProxyServer(ctx, ctrl.Log, queues, waitFunc, routingTable, timeoutCfg, proxyPort, false, nil); !util.IsIgnoredErr(err) {
195+
k8sSharedInformerFactory.WaitForCacheSync(ctx.Done())
196+
if err := runProxyServer(ctx, ctrl.Log, queues, waitFunc, routingTable, svcCache, timeoutCfg, proxyPort, false, nil); !util.IsIgnoredErr(err) {
192197
setupLog.Error(err, "proxy server failed")
193198
return err
194199
}
@@ -369,6 +374,7 @@ func runProxyServer(
369374
q queue.Counter,
370375
waitFunc forwardWaitFunc,
371376
routingTable routing.Table,
377+
svcCache k8s.ServiceCache,
372378
timeouts *config.Timeouts,
373379
port int,
374380
tlsEnabled bool,
@@ -416,6 +422,7 @@ func runProxyServer(
416422
routingTable,
417423
probeHandler,
418424
upstreamHandler,
425+
svcCache,
419426
tlsEnabled,
420427
)
421428
rootHandler = middleware.NewLogging(

interceptor/main_test.go

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ func TestRunProxyServerCountMiddleware(t *testing.T) {
6363
// server
6464
routingTable := routingtest.NewTable()
6565
routingTable.Memory[host] = httpso
66+
svcCache := k8s.NewFakeServiceCache()
6667

6768
timeouts := &config.Timeouts{}
6869
waiterCh := make(chan struct{})
@@ -77,6 +78,7 @@ func TestRunProxyServerCountMiddleware(t *testing.T) {
7778
q,
7879
waitFunc,
7980
routingTable,
81+
svcCache,
8082
timeouts,
8183
port,
8284
false,
@@ -194,6 +196,7 @@ func TestRunProxyServerWithTLSCountMiddleware(t *testing.T) {
194196
// server
195197
routingTable := routingtest.NewTable()
196198
routingTable.Memory[host] = httpso
199+
svcCache := k8s.NewFakeServiceCache()
197200

198201
timeouts := &config.Timeouts{}
199202
waiterCh := make(chan struct{})
@@ -209,6 +212,7 @@ func TestRunProxyServerWithTLSCountMiddleware(t *testing.T) {
209212
q,
210213
waitFunc,
211214
routingTable,
215+
svcCache,
212216
timeouts,
213217
port,
214218
true,
@@ -339,6 +343,7 @@ func TestRunProxyServerWithMultipleCertsTLSCountMiddleware(t *testing.T) {
339343
// server
340344
routingTable := routingtest.NewTable()
341345
routingTable.Memory[host] = httpso
346+
svcCache := k8s.NewFakeServiceCache()
342347

343348
timeouts := &config.Timeouts{}
344349
waiterCh := make(chan struct{})
@@ -354,6 +359,7 @@ func TestRunProxyServerWithMultipleCertsTLSCountMiddleware(t *testing.T) {
354359
q,
355360
waitFunc,
356361
routingTable,
362+
svcCache,
357363
timeouts,
358364
port,
359365
true,

0 commit comments

Comments
 (0)