Allow nvcdi feature flags to be set for the jit-cdi mode

[elezar]

This changs allows nvcdi feature flags to be set for the (now default) jit-cdi mode. This can be used as a workaround for issues such as #1398 where there are issues with a specific driver version.

With the change from NVIDIA/k8s-device-plugin#1495 it will also be possible to set this in the device plugin.

To opt-in to this feature for the jit-cdi mode, run:

sudo nvidia-ctk config --in-place --set nvidia-container-runtime.modes.jit-cdi.nvcdi-feature-flags=enable-nvsandboxutils 

To opt-in for nvidia-ctk cdi generate run:

nvidia-ctk cdi generate --feature-flag=enable-nvsandboxutils

To opt-in for the nvidia-cdi-refresh.service add:

NVIDIA_CTK_CDI_GENERATE_FEATURE_FLAGS=enable-nvsandboxutils

to /etc/nvidia-container-toolkit/nvidia-cdi-refresh.env.

[elezar]

/cherry-pick release-1.18

[ArangoGutierrez]

Do we have a policy on how long to keep a deprecated feature before total removal?

[elezar]

No we don't.

For this feature specifically, I know that @jgehrcke mentioned using this while working on the dynamic MIG for DRA.

[cdesiniotis]

Just a general question -- what is nvsandboxutils used for currently? As in, what parts of the CDI spec does it help us generate?

[elezar]

Just a general question -- what is nvsandboxutils used for currently? As in, what parts of the CDI spec does it help us generate?

It's functionality depends on driver version. Currently we use it here for device node symlinks

nvidia-container-toolkit/internal/platform-support/dgpu/nvsandboxutils.go

Lines 66 to 107 in 754cfa7

	func (d *nvsandboxutilsDGPU) Devices() ([]discover.Device, error) {
	gpuFileInfos, ret := d.lib.GetGpuResource(d.uuid)
	if ret != nvsandboxutils.SUCCESS {
	return nil, fmt.Errorf("failed to get GPU resource: %w", ret)
	}
	var devices []discover.Device
	for _, info := range gpuFileInfos {
	switch info.SubType {
	case nvsandboxutils.NV_DEV_DRI_CARD, nvsandboxutils.NV_DEV_DRI_RENDERD:
	if d.isMig {
	continue
	}
	fallthrough
	case nvsandboxutils.NV_DEV_NVIDIA, nvsandboxutils.NV_DEV_NVIDIA_CAPS_NVIDIA_CAP:
	containerPath := info.Path
	if d.devRoot != "/" {
	containerPath = strings.TrimPrefix(containerPath, d.devRoot)
	}
	// TODO: Extend discover.Device with additional information.
	device := discover.Device{
	HostPath: info.Path,
	Path: containerPath,
	}
	devices = append(devices, device)
	case nvsandboxutils.NV_DEV_DRI_CARD_SYMLINK, nvsandboxutils.NV_DEV_DRI_RENDERD_SYMLINK:
	if d.isMig {
	continue
	}
	if info.Flags == nvsandboxutils.NV_FILE_FLAG_CONTENT {
	targetPath, ret := d.lib.GetFileContent(info.Path)
	if ret != nvsandboxutils.SUCCESS {
	return nil, fmt.Errorf("failed to get symlink: %w", ret)
	}
	d.deviceLinks = append(d.deviceLinks, fmt.Sprintf("%v::%v", targetPath, info.Path))
	}
	}
	}
	return devices, nil
	}

#825 aimed to refactor the driver discovery so that we could add additional functionality, but it has not made it in.

@cdesiniotis thinking about this again, do we want to have opt-in or opt-out behaviour here. The issues that users are experiencing were limited to the 565 driver branch, and we may not want to roll back things now and lose the experience that we would gain by having this on by default.

[cdesiniotis]

The same question crossed my mind when initially reviewing this. Do we know what driver branches / versions besides 565 are affected? 565 is a short-lived branch and is no longer supported, see https://docs.nvidia.com/datacenter/tesla/drivers/supported-drivers-and-cuda-toolkit-versions.html. If the fix is present in all of the active branches (535 / 570 / 580), then I would definitely be in favor of continuing to use nvsandboxutils by default and providing opt-out behavior.

[elezar]

If the fix is present in all of the active branches (535 / 570 / 580), then I would definitely be in favor of continuing to use nvsandboxutils by default and providing opt-out behavior.

If I recall correctly, nvsandboxutils was only added with a driver version after 535 meaning that this bug is not applicable there. The bug that was triggered in 565 was already fixed in 570 at the point where we discovered it. The 580 branch also includes this fix.

[github-actions]

🤖 Backport PR created for release-1.18: #1443 ✅