Skip to content

Commit cfd6678

Browse files
patrickdappolloniopatrickdappollonio
authored
Create testing-crd-versions.md
1 parent 81b07e8 commit cfd6678

File tree

1 file changed

+226
-0
lines changed

1 file changed

+226
-0
lines changed

‎docs/testing-crd-versions.md‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# Testing Kubernetes CRD versions
2+
3+
One interesting thing to see when working with CRDs `storage` and `served` flags is that you can actually query different versions of the same resource.
4+
5+
This example will work in Kubernetes above 1.15 and below 1.21. If you're using `kind`, you should get a cluster, at the time of writing this document, with Kubernetes 1.21.
6+
7+
Copy the ["Minimal example" from the Ingress documentation page of Kubernetes](https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource), copied below:
8+
9+
```yaml
10+
apiVersion: networking.k8s.io/v1
11+
kind: Ingress
12+
metadata:
13+
name: minimal-ingress
14+
annotations:
15+
nginx.ingress.kubernetes.io/rewrite-target: /
16+
spec:
17+
rules:
18+
- http:
19+
paths:
20+
- path: /testpath
21+
pathType: Prefix
22+
backend:
23+
service:
24+
name: test
25+
port:
26+
number: 80
27+
```
28+
29+
Now we know it won't work because we don't have a `Service` called `test`, but we will use it to show some interesting behaviour.
30+
31+
On Kubernetes 1.21, when you query an `Ingress`, you might get the following message:
32+
33+
```
34+
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
35+
```
36+
37+
We can use it to play with it and versions of the ingress above. Save the contents of the ingress into a file called `ingress.yaml`, then apply it:
38+
39+
```
40+
$ kubectl apply -f ingress.yaml
41+
ingress.networking.k8s.io/minimal-ingress created
42+
```
43+
44+
Now, let's query multiple versions of this object. Since the original one has the following version data:
45+
46+
```yaml
47+
apiVersion: networking.k8s.io/v1
48+
kind: Ingress
49+
```
50+
51+
We can use that, plus `kubectl`, to query it especifically by version. The `kubectl get` help content says that the usage can be either of the following (see `kubectl get -h`):
52+
53+
```bash
54+
kubectl get TYPE[.VERSION][.GROUP] [NAME | -l label]
55+
kubectl get TYPE[.VERSION][.GROUP]/NAME
56+
```
57+
58+
So let's use the first one, which is:
59+
60+
```bash
61+
$ kubectl get ingress.v1.networking.k8s.io minimal-ingress
62+
NAME CLASS HOSTS ADDRESS PORTS AGE
63+
minimal-ingress <none> * 80 11m
64+
```
65+
66+
This works. How about...
67+
68+
```bash
69+
$ kubectl get ingress.v1beta1.extensions minimal-ingress
70+
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
71+
NAME CLASS HOSTS ADDRESS PORTS AGE
72+
minimal-ingress <none> * 80 11m
73+
```
74+
75+
... This also works! Although we received a warning, we are still able to query the version of the Ingress using an old CRD version of it. Let's go one step further. Let's request the resources as YAML (and we will use a quick `grep` to only show the `spec` part, excluding the `metadata`). For the first one we have:
76+
77+
78+
```bash
79+
kubectl get ingress.v1.networking.k8s.io minimal-ingress -o yaml | grep -A 100 "spec:"
80+
```
81+
```yaml
82+
spec:
83+
rules:
84+
- http:
85+
paths:
86+
- backend:
87+
service:
88+
name: test
89+
port:
90+
number: 80
91+
path: /testpath
92+
pathType: Prefix
93+
status:
94+
loadBalancer: {}
95+
```
96+
97+
As expected, we get the response we were hoping for. I mean, we created an Ingress using this specific version, version `v1` of the group `networking.k8s.io`. But since we were able to retrieve it before using the old version, will we be able to retrieve the YAML for this old version too? Will it be like old ingresses or will it be like the new ingresses? Let's see:
98+
99+
```bash
100+
kubectl get ingress.v1beta1.extensions minimal-ingress -o yaml | grep -A 100 "spec:"
101+
```
102+
103+
```yaml
104+
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
105+
spec:
106+
rules:
107+
- http:
108+
paths:
109+
- backend:
110+
serviceName: test
111+
servicePort: 80
112+
path: /testpath
113+
pathType: Prefix
114+
status:
115+
loadBalancer: {}
116+
```
117+
118+
Wait... This doesn't look at all how we submitted it ourselves! We defined a `spec.rules[0].http.paths[0].backend.service.name` with the value of `test`, not a `spec.rules[0].http.paths[0].backend.serviceName`!
119+
120+
Was this done by the API or by `kubectl`? Let's see. Let's proxy the API locally:
121+
122+
```bash
123+
kubectl proxy
124+
```
125+
126+
And now, let's run the previous commandds with verbose mode enabled at level 8, and grab that log output sent to `stderr`, pipe it to `stdout` and find the `HTTP GET` call done by `kubectl`:
127+
128+
```bash
129+
kubectl get ingress.v1.networking.k8s.io minimal-ingress -v=8 2>&1 | grep "minimal-ingress"
130+
```
131+
132+
```
133+
I1214 16:19:47.039228 75843 round_trippers.go:432] GET https://127.0.0.1:37691/apis/networking.k8s.io/v1/namespaces/default/ingresses/minimal-ingress
134+
minimal-ingress <none> * 80 23m
135+
```
136+
137+
And now let's call that endpoint, but against our proxy:
138+
139+
```bash
140+
curl -s localhost:8001/apis/networking.k8s.io/v1/namespaces/default/ingresses/minimal-ingress | jq .spec
141+
```
142+
143+
```json
144+
{
145+
"rules": [
146+
{
147+
"http": {
148+
"paths": [
149+
{
150+
"path": "/testpath",
151+
"pathType": "Prefix",
152+
"backend": {
153+
"service": {
154+
"name": "test",
155+
"port": {
156+
"number": 80
157+
}
158+
}
159+
}
160+
}
161+
]
162+
}
163+
}
164+
]
165+
}
166+
```
167+
168+
Ok, looks good. Let's look at the new one, which we can guess the URL, but let's do it still nonetheless. First, get the API Endpoint call using `kubectl` verbose mode:
169+
170+
```bash
171+
kubectl get ingress.v1beta1.extensions minimal-ingress -v=8 2>&1 | grep "minimal-ingress"
172+
```
173+
174+
```
175+
I1214 16:21:07.989565 76368 round_trippers.go:432] GET https://127.0.0.1:37691/apis/extensions/v1beta1/namespaces/default/ingresses/minimal-ingress
176+
minimal-ingress <none> * 80 24m
177+
```
178+
179+
Then let's replace the host for our proxy, and make the call:
180+
181+
```bash
182+
curl -s localhost:8001/apis/extensions/v1beta1/namespaces/default/ingresses/minimal-ingress | jq .spec
183+
```
184+
185+
```json
186+
{
187+
"rules": [
188+
{
189+
"http": {
190+
"paths": [
191+
{
192+
"path": "/testpath",
193+
"pathType": "Prefix",
194+
"backend": {
195+
"serviceName": "test",
196+
"servicePort": 80
197+
}
198+
}
199+
]
200+
}
201+
}
202+
]
203+
}
204+
```
205+
206+
So `kubectl` didn't do the "conversion", the Kubernetes API did!
207+
208+
Keep in mind this is one edge case scenario [of which not many people are happy](https://github.com/kubernetes/kubernetes/issues/94761). The standard is that custom resources should create their own Conversion logic [using a Conversion Webhook](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#webhook-conversion). No conversion is done automagically for 3rd party resources.
209+
210+
Ingresses are handled differently, just because they are ingresses. In better words:
211+
212+
> When you create an ingress object, it can be read via any version (the server handles converting into the requested version). `kubectl get ingress` is an ambiguous request, since it does not indicate what version is desired to be read.
213+
>
214+
> When an ambiguous request is made, kubectl searches the discovery docs returned by the server to find the first group/version that contains the specified resource.
215+
>
216+
> For compatibility reasons, extensions/v1beta1 has historically been preferred over all other api versions. Now that ingress is the only resource remaining in that group, and is deprecated and has a GA replacement, 1.20 will drop it in priority so that `kubectl get ingress` would read from `networking.k8s.io/v1`, but a 1.19 server will still follow the historical priority.
217+
>
218+
> If you want to read a specific version, you can qualify the get request (like `kubectl get ingresses.v1.networking.k8s.io ...`) or can pass in a manifest file to request the same version specified in the file (`kubectl get -f ing.yaml -o yaml`)
219+
> [#](https://github.com/kubernetes/kubernetes/issues/94761#issuecomment-691982480).
220+
221+
It's also worth nothing that:
222+
223+
> The API version used to create an object is not intentionally exposed in the API... all ingress objects are available via all served apiVersions, regardless of the version they were created with (the API server converts between versions in order to return ingress objects in the requested version).
224+
> [#](https://github.com/kubernetes/kubernetes/issues/94761#issuecomment-880900951)
225+
226+
On all other third-party cases, the [standard Kubernetes CRD versioning rules](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#overview) remain in place.

0 commit comments

Comments
 (0)