Skip to content

Commit 1296155

Browse files
kmesh-botkmesh-bot
authored
Merge pull request kmesh-net#1489 from xiaojiangao123/main
add authz and ipsec E2E test
2 parents f4d1eea + 93b6b85 commit 1296155

File tree

7 files changed

+1580
-137
lines changed

7 files changed

+1580
-137
lines changed

‎deploy/charts/kmesh-helm/templates/kmesh-rbac.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ rules:
2626
- daemonsets
2727
verbs:
2828
- get
29+
- apiGroups: ["kmesh.net"]
30+
resources: ["kmeshnodeinfos"]
31+
verbs: ["get", "create", "update", "delete", "list", "watch"]
2932
---
3033
apiVersion: rbac.authorization.k8s.io/v1
3134
kind: ClusterRoleBinding

‎deploy/charts/kmesh-helm/templates/kmesh.net_kmeshnodeinfos.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
apiVersion: apiextensions.k8s.io/v1
3+
kind: CustomResourceDefinition
4+
metadata:
5+
annotations:
6+
controller-gen.kubebuilder.io/version: v0.16.4
7+
name: kmeshnodeinfos.kmesh.net
8+
spec:
9+
group: kmesh.net
10+
names:
11+
kind: KmeshNodeInfo
12+
listKind: KmeshNodeInfoList
13+
plural: kmeshnodeinfos
14+
singular: kmeshnodeinfo
15+
scope: Namespaced
16+
versions:
17+
- name: v1alpha1
18+
schema:
19+
openAPIV3Schema:
20+
description: KmeshNode is the Schema for the kmeshnodes API
21+
properties:
22+
apiVersion:
23+
description: |-
24+
APIVersion defines the versioned schema of this representation of an object.
25+
Servers should convert recognized schemas to the latest internal value, and
26+
may reject unrecognized values.
27+
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
28+
type: string
29+
kind:
30+
description: |-
31+
Kind is a string value representing the REST resource this object represents.
32+
Servers may infer this from the endpoint the client submits requests to.
33+
Cannot be updated.
34+
In CamelCase.
35+
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
36+
type: string
37+
metadata:
38+
type: object
39+
spec:
40+
properties:
41+
addresses:
42+
description: |-
43+
Addresses is used to store the internal ip address information on the
44+
host. The IP address information is used to generate the IPsec state
45+
information. IPsec uses this information to determine which network
46+
adapter is used to encrypt and send data.
47+
items:
48+
type: string
49+
type: array
50+
bootID:
51+
description: |-
52+
bootid is used to generate the ipsec key. After the node is restarted,
53+
the key needs to be updated.
54+
type: string
55+
podCIDRS:
56+
description: |-
57+
PodCIDRs used in IPsec checks the destination of the data to
58+
determine which IPsec state is used for encryption.
59+
items:
60+
type: string
61+
type: array
62+
spi:
63+
description: |-
64+
The SPI is used to identify the version number of the current key.
65+
The communication can be normal only when both communication parties
66+
have spis and the spi keys are the same.
67+
type: integer
68+
required:
69+
- addresses
70+
- bootID
71+
- podCIDRS
72+
- spi
73+
type: object
74+
status:
75+
type: object
76+
type: object
77+
served: true
78+
storage: true

‎docs/proposal/ipsec_authz_e2e_test.md‎

Lines changed: 302 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,302 @@
1+
---
2+
title: IPSec & Offload Authorization E2E Test
3+
authors:
4+
- "@xiaojiangao123" # Authors' GitHub accounts here.
5+
reviewers:
6+
- "@robot"
7+
- TBD
8+
approvers:
9+
- "@robot"
10+
- TBD
11+
12+
creation-date: 2025-5-12
13+
14+
---
15+
16+
## IPSec & Offload Authorization E2E Test Proposal
17+
18+
### Overview
19+
20+
This proposal aims to design end-to-end (E2E) test cases for the IPSec feature and Offload Authorization of Kmesh. The IPSec E2E tests focus on basic functionality, security, and reliability, verifying proper execution of IPSec tunnel setup, encryption/decryption, key management, and fault recovery. The Offload Authorization E2E tests include IP authorization, Port authorization, Header authorization, namespace authorization, and hosts authorization.
21+
22+
### Motivation
23+
24+
Kmesh’s IPSec functionality is a key component for secure communication within the service mesh. Its stability and reliability directly affect the security of the entire mesh. The Offload Authorization feature is a unique Kmesh capability that offloads IP and Port authorization to the Linux kernel via XDP. Without comprehensive E2E testing, version upgrades and releases may pose potential risks.
25+
26+
#### Goals
27+
28+
1. Develop complete E2E test cases
29+
2. Cover all functional scenarios of Kmesh IPSec
30+
3. Cover all types of Offload Authorization mechanisms
31+
32+
### Proposal
33+
34+
For the IPSec feature, we designed IPSec E2E test cases covering three core scenarios: basic connectivity, key rotation, and fault recovery.
35+
36+
1. The basic connectivity test verifies IPSec tunnel establishment and the correctness of encrypted communication using tcpdump to inspect ESP packets.
37+
38+
2. The key rotation test ensures the reliability of the PSK update mechanism and validates service continuity during key changes.
39+
40+
For the Offload Authorization feature, E2E test cases are designed for IP, Port, Header, Namespace, and Hosts authorization.
41+
42+
The test environment requires at least a 2-nodes Kubernetes cluster using httpbin and sleep or fortio as test applications.
43+
44+
### Design Details
45+
46+
#### 1. Test Environment Preparation
47+
48+
##### Requirements
49+
50+
- At least a 2-nodes Kubernetes cluster with Kmesh installed
51+
- Tools: tcpdump
52+
- Applications: httpbin, sleep, fortio
53+
54+
#### 2. IPSec Test Scenarios
55+
56+
##### 2.1 Basic Connectivity E2E Test
57+
58+
###### Test Steps
59+
60+
- Deploy httpbin and sleep applications on different nodes
61+
- Verify connectivity between applications
62+
- Check IPSec state and policy rules:
63+
64+
```bash
65+
ip xfrm state show
66+
```
67+
68+
**Expected Output:**
69+
70+
```plaintext
71+
src {{SRC_IP}} dst {{DST_IP}}
72+
proto esp spi 0x{{SPI}} reqid 1 mode tunnel
73+
replay-window 0
74+
output-mark 0xd0/0xffffffff
75+
aead rfc4106(gcm(aes)) {{KEY}} 128
76+
anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
77+
sel src ::/0 dst ::/0
78+
```
79+
80+
```bash
81+
ip xfrm policy show
82+
```
83+
84+
**Expected Output:**
85+
86+
```plaintext
87+
src ::/0 dst {{DST_SUBNET}}
88+
dir out priority 0
89+
mark 0xe0/0xffffffff
90+
tmpl src {{SRC_IP}} dst {{DST_IP}}
91+
proto esp spi 0x{{SPI}} reqid 1 mode tunnel
92+
```
93+
94+
- Verify encryption using tcpdump; analyze ESP headers and confirm payload is encrypted:
95+
96+
```bash
97+
tcpdump -i any esp
98+
```
99+
100+
**Expected Output:** ESP packets should be visible during communication.
101+
102+
##### 2.2 Key Rotation E2E Test
103+
104+
###### Test Steps
105+
106+
- Deploy httpbin and sleep applications on different nodes
107+
- Record initial SPI and pre-shared key:
108+
109+
```bash
110+
ip xfrm state show
111+
```
112+
113+
**Expected Output:**
114+
115+
```plaintext
116+
src {{SRC_IP}} dst {{DST_IP}}
117+
proto esp spi 0x{{INITIAL_SPI}} reqid 1 mode tunnel
118+
aead rfc4106(gcm(aes)) {{INITIAL_KEY}} 128
119+
```
120+
121+
- Send continuous traffic between applications
122+
- Update the pre-shared key
123+
124+
```bash
125+
kubectl create secret
126+
```
127+
128+
- Check if SPI and key are updated in xfrm rules:
129+
130+
```bash
131+
ip xfrm state show
132+
```
133+
134+
**Expected Output:**
135+
136+
```plaintext
137+
src {{SRC_IP}} dst {{DST_IP}}
138+
proto esp spi 0x{{NEW_SPI}} reqid 1 mode tunnel
139+
aead rfc4106(gcm(aes)) {{NEW_KEY}} 128
140+
```
141+
142+
- Verify communication continuity, encryption status
143+
144+
#### 3. Offload Authorization Test Scenarios
145+
146+
Apply the corresponding security policies (ALLOW/DENY) and routing strategies, and verify connectivity, including traffic that matches the rules and will be allowed and traffic that will be denied by the rules.
147+
148+
Here are some AuthorizationPolicy samples.
149+
150+
##### 3.1 IP Authorization Test
151+
152+
- Example security policy YAML:
153+
154+
```yaml
155+
apiVersion: security.istio.io/v1beta1
156+
kind: AuthorizationPolicy
157+
metadata:
158+
name: ip-allow-policy
159+
namespace: test-ns1
160+
spec:
161+
action: ALLOW
162+
rules:
163+
- from:
164+
- source:
165+
ipBlocks:
166+
- "{{.Ip}}"
167+
---
168+
apiVersion: security.istio.io/v1beta1
169+
kind: AuthorizationPolicy
170+
metadata:
171+
name: ip-deny-policy
172+
namespace: test-ns1
173+
spec:
174+
action: DENY
175+
rules:
176+
- from:
177+
- source:
178+
ipBlocks:
179+
- "{{.Ip}}"
180+
```
181+
182+
##### 3.2 Port Authorization Test
183+
184+
- Example security policy YAML:
185+
186+
```yaml
187+
apiVersion: security.istio.io/v1beta1
188+
kind: AuthorizationPolicy
189+
metadata:
190+
name: port-allow-policy
191+
namespace: test-ns1
192+
spec:
193+
action: ALLOW
194+
rules:
195+
- to:
196+
- operation:
197+
ports: {{.Ports}}
198+
---
199+
apiVersion: security.istio.io/v1beta1
200+
kind: AuthorizationPolicy
201+
metadata:
202+
name: port-deny-policy
203+
namespace: test-ns1
204+
spec:
205+
action: DENY
206+
rules:
207+
- to:
208+
- operation:
209+
ports: {{.Ports}}
210+
```
211+
212+
##### 3.3 Namespace Authorization Test
213+
214+
- Example security policy YAML:
215+
216+
```yaml
217+
apiVersion: security.istio.io/v1beta1
218+
kind: AuthorizationPolicy
219+
metadata:
220+
name: namespace-allow-policy
221+
namespace: test-ns1
222+
spec:
223+
action: ALLOW
224+
rules:
225+
- from:
226+
- source:
227+
namespaces:
228+
- "{{.SourceNamespace}}"
229+
---
230+
apiVersion: security.istio.io/v1beta1
231+
kind: AuthorizationPolicy
232+
metadata:
233+
name: namespace-deny-policy
234+
namespace: test-ns1
235+
spec:
236+
action: DENY
237+
rules:
238+
- from:
239+
- source:
240+
namespaces:
241+
- "{{.SourceNamespace}}"
242+
```
243+
244+
##### 3.4 Header Authorization Test
245+
246+
- Example security policy YAML:
247+
248+
```yaml
249+
apiVersion: security.istio.io/v1beta1
250+
kind: AuthorizationPolicy
251+
metadata:
252+
name: header-allow-policy
253+
namespace: test-ns1
254+
spec:
255+
action: ALLOW
256+
rules:
257+
- when:
258+
- key: request.headers[{{.HeaderName}}]
259+
values: ["{{.HeaderValue}}"]
260+
---
261+
apiVersion: security.istio.io/v1beta1
262+
kind: AuthorizationPolicy
263+
metadata:
264+
name: header-deny-policy
265+
namespace: test-ns1
266+
spec:
267+
action: DENY
268+
rules:
269+
- when:
270+
- key: request.headers[{{.HeaderName}}]
271+
values: ["{{.HeaderValue}}"]
272+
```
273+
274+
##### 3.5 Hosts Authorization Test
275+
276+
- Example security policy YAML:
277+
278+
```yaml
279+
apiVersion: security.istio.io/v1beta1
280+
kind: AuthorizationPolicy
281+
metadata:
282+
name: host-allow-policy
283+
namespace: test-ns1
284+
spec:
285+
action: ALLOW
286+
rules:
287+
- to:
288+
- operation:
289+
hosts: ["{{.TargetHost}}"]
290+
---
291+
apiVersion: security.istio.io/v1beta1
292+
kind: AuthorizationPolicy
293+
metadata:
294+
name: host-deny-policy
295+
namespace: test-ns1
296+
spec:
297+
action: DENY
298+
rules:
299+
- to:
300+
- operation:
301+
hosts: ["{{.TargetHost}}"]
302+
```

0 commit comments

Comments
 (0)