You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -135,15 +136,15 @@ Implementing this KEP will empower nodes to recognize and adapt to changes in th
135
136
136
137
### Goals
137
138
138
-
* Achieve seamless node capacity expansion through hot plugging resources.
139
+
* Achieve seamless node capacity expansion through resource hotplug.
139
140
* Enable the re-initialization of resource managers (CPU manager, memory manager) and kube runtime manager without reset to accommodate alterations in the node's resource allocation.
140
141
* Recalculating and updating the OOMScoreAdj and swap memory limit for existing pods.
141
142
142
143
### Non-Goals
143
144
144
145
* Dynamically adjust system reserved and kube reserved values.
145
146
* Hot unplug of node resources.
146
-
* Update the autoscaler to utilize resource hot plugging.
147
+
* Update the autoscaler to utilize resource hotplug.
147
148
* Re-balance workloads across the nodes.
148
149
* Update runtime/NRI plugins with host resource changes.
149
150
@@ -278,9 +279,9 @@ With increase in cluster resources the following components will be updated:
278
279
Once the capacity of the node is altered, the following are the sequence of events that occur in the kubelet. If any errors are
279
280
observed in any of the steps, operation is retried from step 1 along with a `FailedNodeResize` event under the node object.
280
281
1. Resizing existing containers:
281
-
a.With the increased memory capacity of the nodes, the kubelet proceeds to update fields that are directly related to
282
+
a.With the increased memory capacity of the nodes, the kubelet proceeds to update fields that are directly related to
282
283
the available memory on the host. This would lead to recalculation of oom_score_adj and swap_limits.
283
-
b.This is achieved by invoking the CRI API - UpdateContainerResources.
284
+
b.This is achieved by invoking the CRI API - UpdateContainerResources.
284
285
285
286
2. Reinitialise Resource Manager:
286
287
a. Resource managers such as CPU,Memory are updated with the latest available capacities on the host. This posts the latest
@@ -318,6 +319,21 @@ T=1: Resize Instance to Hotplug Memory:
318
319
319
320
Similar flow is applicable for updating oom_score_adj.
320
321
322
+
#### Compatability with Cluster Autoscaler
323
+
324
+
The Cluster Autoscaler (CA) presently anticipates uniform allocatable values among nodes within the same NodeGroup, using existing Nodes as templates for
325
+
newly provisioned Nodes from the same NodeGroup. However, with the introduction of NodeResourceHotplug, this assumption may no longer hold true.
326
+
If not appropriately addressed, this could cause the Cluster Autoscaler to randomly select a Node from the group and assume identical allocatable values for all upcoming Nodes.
327
+
This could lead to suboptimal decisions, such as repeatedly attempting to provision Nodes for pending Pods that are incompatible, or overlooking potential Nodes that could accommodate such Pods.
328
+
329
+
To ensure the Cluster Autoscaler acknowledges resource hotplug, the following approaches have been proposed by the Cluster Autoscaler team:
330
+
1. Capture Node's Initial Allocatable Values:
331
+
* Introduce a new field within the Node object to record initial node allocatable values, which remain unchanged during resource hotplug.
332
+
* The Cluster Autoscaler can leverage this field to anticipate potential hotplug of resources, using it as a template for configuring new Nodes.
333
+
334
+
2. Identify Nodes Affected by Hotplug:
335
+
* By flagging a Node as being impacted by hotplug, the Cluster Autoscaler could revert to a less reliable but more conservative "scale from 0 nodes" logic.
336
+
321
337
### Handling HotUnplug Events
322
338
323
339
Though this KEP focuses only on resource hotplug, It will enable the kubelet to capture the current available capacity of the node (Regardless of whether it was a hotplug or hotunplug of resources.)
0 commit comments