Skip to content

Commit e0f3064

Browse files
abrennan89abrennan89
committed
CNV-15425: Add CNV NUMA topology docs
1 parent 4135cf0 commit e0f3064

File tree

10 files changed

+414
-0
lines changed

10 files changed

+414
-0
lines changed

‎_topic_maps/_topic_map.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4921,6 +4921,8 @@ Topics:
49214921
File: virt-about-multi-queue
49224922
- Name: Managing virtual machines by using OpenShift GitOps
49234923
File: virt-managing-virtual-machines-by-using-openshift-gitops
4924+
- Name: Working with NUMA topology for virtual machines
4925+
File: virt-NUMA-topology
49244926
- Name: VM disks
49254927
Dir: virtual_disks
49264928
Topics:

‎modules/virt-NUMA-limitations.adoc‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * /virt/managing_vms/advanced_vm_management/virt-NUMA-topology.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="virt-NUMA-limitations_{context}"]
7+
= Limitations of NUMA for {VirtProductName}
8+
9+
[role="_abstract"]
10+
When you use NUMA topology with {VirtProductName} VMs, certain limitations can impact performance and VM management.
11+
12+
Asymmetrical topology:: The host scheduler cannot guarantee assigning specific NUMA nodes to a VM. For example, if a VM is rescheduled to a different host machine because of a restart or maintenance, the new host might have a different physical NUMA layout. This means that the VM could be presented with an asymmetrical NUMA topology that reflects the new host's configuration, rather than its original or desired layout. This change can have a negative impact on the VM's performance.
13+
14+
Live migration challenges:: Migrating a NUMA-enabled VM to a different host node can be challenging if the destination node's NUMA topology differs significantly from the source node's. A mismatch between the NUMA layouts of the source and destination can lead to a degradation of the VM's performance after the migration is complete.
15+
16+
No support for PCI NUMA nodes:: There is no explicit support for passing GPU NUMA zone information to the VM. This means that the VM's guest operating system is not aware of the NUMA locality of PCI devices such as GPUs. For workloads that heavily rely on these devices, this lack of awareness could potentially lead to reduced performance if the GPU's memory is not local to the accessing CPU within the NUMA architecture.

‎modules/virt-NUMA-live-migration.adoc‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/managing_vms/advanced_vm_management/virt-NUMA-topology.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="virt-NUMA-live-migration_{context}"]
7+
= Live migration outcomes using vNUMA
8+
9+
[role="_abstract"]
10+
Migration outcomes for VMs are dependent on the configured Topology Manager policies.
11+
These policies determine how CPU and memory resources are allocated with respect to the physical NUMA nodes of the host.
12+
There are four available policies: `None`, `single-numa-node`, `best-effort`, and `restricted`.
13+
14+
The following table outlines which policies are supported for different VM configurations, and their effect on live migration.
15+
16+
* A small VM is defined as a VM with less total cores than half of cores in NUMA node.
17+
* A large VM is defined as a VM with more total cores than half of cores in NUMA node.
18+
* An extra large VM is defined as a VM with more cores than 1 NUMA node.
19+
20+
[cols="3",options="header"]
21+
|===
22+
|VM size
23+
|Topology Manager policy
24+
|Tested support status
25+
26+
|Any
27+
|single-numa-node
28+
|The VM fails to start because the pod requests more cpus than a single NUMA node on the host can provide. This triggers a topology affinity error during scheduling, which is expected behavior given the node’s hardware limits.
29+
30+
|Any
31+
|None
32+
|Live migration does not work. This is a known issue. The process ends with an incorrect memnode allocation error, and libvirt rejects the XML manifest generated by KubeVirt. See release notes for additional information.
33+
// OCP 4.20 - this may need to be updated later if the issue is resolved in a future release
34+
35+
|Small
36+
|None
37+
|Live migration works, as expected.
38+
39+
|Small
40+
|single-numa-node
41+
|Live migration works, as expected.
42+
43+
|Small
44+
|best-effort
45+
|Live migration works, as expected.
46+
47+
|Small
48+
|restricted
49+
|Live migration works, as expected.
50+
51+
|Large
52+
|single-numa-node
53+
|Live migration works, as expected.
54+
55+
|Large
56+
|best-effort
57+
|Live migration works, as expected.
58+
59+
|Large
60+
|restricted
61+
|Live migration works, as expected.
62+
63+
|Extra large
64+
|None
65+
|Live migration works, as expected.
66+
67+
|Extra large
68+
|best-effort
69+
|Live migration works, as expected.
70+
71+
|Extra large
72+
|restricted
73+
|VMs do not work, as expected.
74+
|===

‎modules/virt-NUMA-prereqs.adoc‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * /virt/managing_vms/advanced_vm_management/virt-NUMA-topology.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="virt-NUMA-prereqs_{context}"]
7+
= Prerequisites
8+
9+
[role="_abstract"]
10+
Before you can enable NUMA functionality with {VirtProductName} VMs, you must ensure that your environment meets the following prerequisites.
11+
12+
* Worker nodes must have huge pages enabled.
13+
* The `KubeletConfig` object on worker nodes must be configured with the `cpuManagerPolicy: static` spec to guarantee dedicated CPU allocation, which is a prerequisite for NUMA pinning.
14+
+
15+
.Example `cpuManagerPolicy: static` spec
16+
[source,yaml]
17+
----
18+
apiVersion: machineconfiguration.openshift.io/v1
19+
kind: KubeletConfig
20+
metadata:
21+
name: cpu-numa-static-config
22+
spec:
23+
kubeletConfig:
24+
cpuManagerPolicy: static
25+
# ...
26+
----

‎modules/virt-disable-CPU-VM-hotplug-instancetype.adoc‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * /virt/managing_vms/advanced_vm_management/virt-NUMA-topology.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="virt-disable-CPU-VM-hotplug-instancetype_{context}"]
7+
= Disabling the CPU hot plug by instance type
8+
9+
[role="_abstract"]
10+
As a cluster administrator, you can disable the CPU hot plug by instance type.
11+
This is the recommended approach to standardize VM configurations and ensure NUMA-aware CPU allocation without hot plugs for specific instance types.
12+
13+
When a VM is created by using an instance type where the CPU hot plug is disabled, the VM inherits these settings and the CPU hot plug is disabled for that VM.
14+
15+
.Prerequisites
16+
17+
* You have installed the {oc-first}.
18+
19+
.Procedure
20+
21+
. Create a YAML file for a `VirtualMachineClusterInstancetype` custom resource (CR). Add a `maxSockets` spec to the instance type that you want to configure:
22+
+
23+
.Example `VirtualMachineClusterInstancetype` CR
24+
[source,yaml]
25+
----
26+
apiVersion: instancetype.kubevirt.io/v1beta1
27+
kind: VirtualMachineClusterInstancetype
28+
metadata:
29+
name: cx1.mycustom-numa-instance
30+
spec:
31+
cpu:
32+
dedicatedCPUPlacement: true
33+
isolateEmulatorThread: true
34+
numa:
35+
guestMappingPassthrough: {}
36+
guest: 8
37+
maxSockets: 8
38+
memory:
39+
guest: 16Gi
40+
hugepages:
41+
pageSize: 1Gi
42+
----
43+
+
44+
where:
45+
+
46+
spec.cpu.dedicatedCPUPlacement:: Specifies whether dedicated resources are allocated to the VM instance. If this is set to `true`, the VM's VCPUs are pinned to physical host CPUs. This is often used for high-performance workloads to minimize scheduling jitter.
47+
+
48+
spec.cpu.isolateEmulatorThread:: Specifies whether the QEMU emulator thread should be isolated and run on a dedicated physical CPU core. This is a performance optimization that is typically used alongside the `dedicatedCPUPlacement` spec.
49+
+
50+
spec.cpu.numa:: Specifies the NUMA topology configuration for the VM.
51+
+
52+
spec.cpu.numa.guestMappingPassthrough:: Specifies that the VM's NUMA topology should directly pass through the NUMA topology of the underlying host machine. This is critical for applications that are NUMA-aware and require optimal performance.
53+
+
54+
spec.cpu.guest:: Specifies the total number of vCPUs to be allocated to the VM.
55+
+
56+
spec.cpu.maxSockets:: Specifies the maximum number of CPU sockets the VM is allowed to have.
57+
+
58+
spec.memory:: Specifies the memory configuration for the VM.
59+
+
60+
spec.memory.guest:: Specifies the total amount of memory to be allocated to the VM.
61+
+
62+
spec.memory.hugepages:: Specifies configuration related to hugepages.
63+
+
64+
spec.memory.hugepages.pageSize:: Specifies the size of the hugepages to be used for the VM's memory.
65+
66+
. Create the `VirtualMachineClusterInstancetype` CR by running the following command:
67+
+
68+
[source,terminal]
69+
----
70+
$ oc create -f <filename>.yaml
71+
----
72+
73+
.Verification
74+
75+
. Create a VM that uses the updated `VirtualMachineClusterInstancetype` configuration.
76+
77+
. Inspect the configuration of the created VM by running the following command and inspecting the output:
78+
+
79+
[source,terminal]
80+
----
81+
$ oc get vmi <vm_name> -o yaml
82+
----
83+
+
84+
*Example output*
85+
+
86+
[source,yaml]
87+
----
88+
apiVersion: kubevirt.io/v1
89+
kind: VirtualMachineInstance
90+
metadata:
91+
name: example-vmi
92+
labels:
93+
instancetype.kubevirt.io/cluster-instancetype: cx1.example-numa-instance
94+
spec:
95+
domain:
96+
cpu:
97+
dedicatedCPUPlacement: true
98+
isolateEmulatorThread: true
99+
sockets: 8
100+
cores: 1
101+
threads: 1
102+
numa:
103+
guestMappingPassthrough: {}
104+
guest: 8
105+
maxSockets: 8
106+
# ...
107+
----
108+
+
109+
The update has applied successfully if in the `spec.template.spec.domain.cpu` section:
110+
+
111+
* The `sockets` value matches the `maxSockets` and `guest` values from the instance type, which ensures that no extra hot plug slots are configured.
112+
* The `dedicatedCPUPlacement` and `isolateEmulatorThread` fields are present and set to `true`.

‎modules/virt-disable-CPU-VM-hotplug.adoc‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * /virt/managing_vms/advanced_vm_management/virt-NUMA-topology.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="virt-disable-CPU-VM-hotplug_{context}"]
7+
= Adjusting or disabling the CPU hot plug by VM
8+
9+
[role="_abstract"]
10+
As a VM owner, you can adjust or disable the CPU hot plug for individual VMs.
11+
This is the simplest solution for large, performance-critical VMs where you want to ensure a fixed CPU allocation from the start.
12+
13+
.Prerequisites
14+
15+
* You have installed the {oc-first}.
16+
17+
.Procedure
18+
19+
. Modify the `VirtualMachine` custom resource (CR) for the VM that you want to configure to add a `maxSockets` and `sockets` spec:
20+
+
21+
[source,yaml]
22+
----
23+
apiVersion: kubevirt.io/v1
24+
kind: VirtualMachine
25+
metadata:
26+
name: large-numa-vm
27+
spec:
28+
template:
29+
spec:
30+
domain:
31+
cpu:
32+
maxSockets: 10
33+
sockets: 10
34+
cores: 1
35+
threads: 1
36+
----
37+
+
38+
By explicitly setting `maxSockets` and `sockets` to a value of 10 or higher, you are specifying that additional capacity is not reserved for hot plugging, which ensures that the entire requested cores are the actual cores allocated.
39+
40+
. Apply the changes to the `VirtualMachine` CR by running the following command:
41+
+
42+
[source,terminal]
43+
----
44+
$ oc apply -f <filename>.yaml
45+
----
46+
47+
.Verification
48+
49+
. Check that you have configured the `maxSockets` and `sockets` values correctly, by running the following commands:
50+
+
51+
[source,terminal]
52+
----
53+
$ oc get vmi -o jsonpath='{.spec.domain.cpu.maxSockets}'
54+
----
55+
+
56+
[source,terminal]
57+
----
58+
$ oc get vmi -o jsonpath='{.spec.domain.cpu.sockets}'
59+
----
60+
+
61+
If the configuration was successful, the outputs are the `maxSockets` and `sockets` values that you set in the previous procedure:
62+
+
63+
*Example output*
64+
+
65+
[source,terminal]
66+
----
67+
10
68+
----

‎modules/virt-disable-kubervirt-hotplug-ratio.adoc‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * /virt/managing_vms/advanced_vm_management/virt-NUMA-topology.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="virt-disable-kubervirt-hotplug-ratio_{context}"]
7+
= Disabling hot plugging for all VMs on a cluster
8+
9+
[role="_abstract"]
10+
If you are a cluster administrator and want to disable hot plugging for an entire cluster, you must modify the `spec.configuration.kubevirtConfiguration.developerConfiguration.maxHotplugRatio` setting in the `HyperConverged` custom resource (CR).
11+
12+
.Prerequisites
13+
14+
* You have installed the {oc-first}.
15+
* You have installed the {CNVOperatorDisplayName}.
16+
17+
.Procedure
18+
19+
. Modify the `HyperConverged` CR and set the `maxHotplugRatio` value to `1.0`:
20+
+
21+
[source,yaml]
22+
----
23+
apiVersion: hco.kubevirt.io/v1beta1
24+
kind: HyperConverged
25+
metadata:
26+
name: kubevirt-hyperconverged
27+
namespace: kubevirt-hyperconverged
28+
spec:
29+
# ...
30+
kubevirtConfiguration:
31+
developerConfiguration:
32+
maxHotplugRatio: 1.0
33+
# ...
34+
----
35+
36+
. Apply the changes to the `HyperConverged` CR by running the following command:
37+
+
38+
[source,terminal]
39+
----
40+
$ oc apply -f <filename>.yaml
41+
----
42+
43+
.Verification
44+
45+
. Check that you have configured the `maxHotplugRatio` value correctly, by running the following command:
46+
+
47+
[source,terminal]
48+
----
49+
$ oc get hyperconverged -n openshift-cnv -o jsonpath='{.spec.liveUpdateConfiguration.maxHotplugRatio}'
50+
----
51+
+
52+
If the configuration was successful, the output is the `maxHotplugRatio` value that you set in the previous procedure:
53+
+
54+
*Example output*
55+
+
56+
[source,terminal]
57+
----
58+
1.0
59+
----

‎modules/virt-using-NUMA.adoc‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * /virt/managing_vms/advanced_vm_management/virt-NUMA-topology.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="virt-using-NUMA_{context}"]
7+
= Using NUMA topology with {VirtProductName}
8+
9+
You must enable the NUMA functionality for {VirtProductName} VMs to prevent performance degradation on nodes with multiple NUMA zones. This feature is vital for high-performance and latency-sensitive workloads.
10+
11+
Without NUMA awareness, a VM's virtual CPUs might run on one physical NUMA zone, while its memory is allocated on another. This "cross-node" communication significantly increases latency and reduces memory bandwidth, and can cause the interconnect buses which link the NUMA zones to become a bottleneck.
12+
13+
When you enable the NUMA functionality for {VirtProductName} VMs, you allow the host to pass its physical topology directly to the VM's guest operating system (OS).
14+
The guest OS can then make intelligent, NUMA-aware decisions about scheduling and memory allocation. This ensures that process threads and memory are kept on the same physical NUMA node. By aligning the virtual topology with the physical one, you minimize latency and maximize performance.

‎snippets/about-numa.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// Snippets included in the following assemblies and modules:
22
//
33
// *scalability_and_performance/cnf-numa-aware-scheduling.adoc
4+
// * /virt/managing_vms/advanced_vm_management/virt-NUMA-topology.adoc
45

56
:_mod-docs-content-type: SNIPPET
67

0 commit comments

Comments
 (0)