Skip to content

Commit a9dee48

Browse files
informalictinformalict
authored
[Documentation] Topology Awareness (#1048)
1 parent 5304ef8 commit a9dee48

File tree

2 files changed

+200
-1
lines changed

2 files changed

+200
-1
lines changed

‎docs/design/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,5 @@
1010
- [Upgrading](./upgrading.md)
1111
- [Rotating Pods](./rotating.md)
1212
- [Maintenance](./maintenance.md)
13-
- [Additional configuration](./additional_configuration.md)
13+
- [Additional configuration](./additional_configuration.md)
14+
- [Topology awareness](./topology_awareness.md)

‎docs/design/topology_awareness.md‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
# Topology awareness
2+
3+
## Table of contents
4+
1. [Overview](#1)
5+
2. [Requirements](#2)
6+
3. [Enable/Disable topology](#3)
7+
4. [Check topology](#4)
8+
9+
## Overview <a name="1"></a>
10+
11+
Topology awareness is responsible for the even distribution of groups of pods across nodes in the cluster.
12+
A distribution should be done by the zone, so thanks to that if one of the zone fails there are other working pods
13+
in different zones. For the time being, there are 3 groups of pods that can be distributed evenly
14+
(coordinators, agents, DB servers). For each of these groups, the Kube-ArangoDB operator
15+
tries to distribute them in different zones in a cluster, so there can not
16+
be a situation where many pods of the same group exist in one zone and there are no
17+
pods in other zones. It would lead to many issues when a zone with many pods failed.
18+
When Kube-ArangoDB operator is going to add a new pod, but all zones already contain a pod of this group,
19+
it will choose the zone with the fewest number of pods of this group.
20+
21+
#### Example
22+
Let's say we have two zones (uswest-1, uswest-2) and we would like to distribute ArangoDB cluster
23+
with 3 coordinators, 3 agents, and 3 DB servers. First coordinator, agent, and DB server would go to random zone (e.g. uswest-1).
24+
Second coordinator must be assigned to the `uswest-2` zone, because the zone `uswest-1` already contains one coordinator.
25+
The same happens for the second agent and the second DB server. Third coordinator can be placed randomly
26+
because each of the zone contains exactly one coordinator, so after this operation one of the zone should have 2 coordinators
27+
and second zone should have 1 coordinator. The same applies to agents and DB servers.
28+
29+
According to the above example we can see that:
30+
- coordinators should not be assigned to the same zone with other coordinators, unless ALL zones contain coordinators.
31+
- agents should not be placed in the same zone with other agents, unless ALL zones contain agents.
32+
- DB servers should not be placed in the same zone with other DB servers, unless ALL zones contain DB servers.
33+
34+
## Requirements <a name="2"></a>
35+
36+
- It does not work in a `Single` mode of a deployment.
37+
The `spec.mode` of the Kubernetes resource ArangoDeployment can not be set to `Single`.
38+
- Kube-ArangoDB version should be at least 1.2.10 and enterprise version.
39+
40+
## How to enable/disable topology awareness for the ArangoDeployment <a name="3"></a>
41+
42+
Enable topology:
43+
```yaml
44+
spec:
45+
topology:
46+
enabled: true
47+
label: string # A node's label which will be considered as distribution affinity. By default: 'topology.kubernetes.io/zone'
48+
zones: int # How many zones will be used to assign pods there. It must be higher than 0.
49+
```
50+
51+
Disable topology:
52+
```yaml
53+
spec:
54+
topology:
55+
enable: false
56+
```
57+
or remove `spec.topology` object.
58+
59+
## How to check which ArangoDB members are assigned to the topology <a name="4"></a>
60+
61+
#### Topology aware
62+
63+
Each member should be topology aware, and it can be checked in list of conditions here `status.members.[agents|coordinators|dbservers].conditions`.
64+
Example:
65+
```yaml
66+
status:
67+
...
68+
members:
69+
agents:
70+
- conditions:
71+
reason: Topology awareness enabled
72+
status: True
73+
type: TopologyAware
74+
```
75+
76+
If `status` for the condition's type `TopologyAware` is set to `false` then it is required to replace ArangoDB member.
77+
To do so we need to set pod's annotation `deployment.arangodb.com/replace` to `true`, starting from all
78+
coordinators which are not assigned to any zone. This situation usually happens when
79+
topology was enabled on an existing ArangoDeployment resource.
80+
81+
#### Member topology
82+
Each member's status should have topology, and it can be checked here `status.members.[agents|coordinators|dbservers].topology` and here `status.topology`.
83+
Example:
84+
```yaml
85+
status:
86+
...
87+
members:
88+
agents:
89+
- id: AGNT-2shphs7a
90+
topology:
91+
id: 35a61527-9d2b-49df-8a31-e62417fcd7e6
92+
label: eu-central-1c
93+
rack: 0
94+
...
95+
topology:
96+
id: 35a61527-9d2b-49df-8a31-e62417fcd7e6
97+
label: topology.kubernetes.io/zone
98+
size: 3
99+
zones:
100+
- id: 0
101+
labels:
102+
- eu-central-1c
103+
members:
104+
agnt:
105+
- AGNT-2shphs7a
106+
...
107+
- ...
108+
...
109+
```
110+
which means that `AGNT-2shphs7a` is assigned to `eu-central-1c`.
111+
112+
#### Pod's labels
113+
114+
A pod which belongs to the member should have two new labels.
115+
Example:
116+
```yaml
117+
apiVersion: v1
118+
kind: Pod
119+
metadata:
120+
labels:
121+
deployment.arangodb.com/topology: 35a61527-9d2b-49df-8a31-e62417fcd7e6
122+
deployment.arangodb.com/zone: "0"
123+
```
124+
125+
#### Pod anti-affinity
126+
127+
A pod which belongs to the member should have a new pod anti affinity rules.
128+
Example:
129+
```yaml
130+
spec:
131+
affinity:
132+
podAntiAffinity:
133+
requiredDuringSchedulingIgnoredDuringExecution:
134+
- labelSelector:
135+
matchExpressions:
136+
- key: deployment.arangodb.com/topology
137+
operator: In
138+
values:
139+
- 35a61527-9d2b-49df-8a31-e62417fcd7e6
140+
- ...
141+
- key: deployment.arangodb.com/zone
142+
operator: In
143+
values:
144+
- "1"
145+
- "2"
146+
- ...
147+
topologyKey: topology.kubernetes.io/zone
148+
- ...
149+
```
150+
which means that pod can not be assigned to zone `1` and `2`.
151+
152+
#### Node affinity
153+
154+
A pod which belongs to the member can have a node affinity rules. If a pod does not have it then it will have pod affinities.
155+
Example:
156+
```yaml
157+
spec:
158+
affinity:
159+
nodeAffinity:
160+
requiredDuringSchedulingIgnoredDuringExecution:
161+
nodeSelectorTerms:
162+
- matchExpressions:
163+
- key: topology.kubernetes.io/zone
164+
operator: In
165+
values:
166+
- eu-central-1c
167+
- ...
168+
- matchExpressions:
169+
- key: topology.kubernetes.io/zone
170+
operator: NotIn
171+
values:
172+
- eu-central-1a
173+
- eu-central-1b
174+
- ...
175+
```
176+
177+
#### Pod affinity
178+
179+
A pod which belongs to the member can have a pod affinity rules. If a pod does not have it then it will have node affinity.
180+
Example:
181+
```yaml
182+
spec:
183+
affinity:
184+
podAffinity:
185+
requiredDuringSchedulingIgnoredDuringExecution:
186+
- labelSelector:
187+
matchExpressions:
188+
- key: deployment.arangodb.com/topology
189+
operator: In
190+
values:
191+
- 35a61527-9d2b-49df-8a31-e62417fcd7e6
192+
- key: deployment.arangodb.com/zone
193+
operator: In
194+
values:
195+
- "1"
196+
- ...
197+
topologyKey: topology.kubernetes.io/zone
198+
```

0 commit comments

Comments
 (0)