Skip to content

Get Started with DRA Support for NVIDIA GPUs

Overview

Dynamic Resource Allocation (DRA) is a Kubernetes API for flexibly requesting, configuring, and sharing specialized devices like GPUs — a modern replacement for the extended-resource model (nvidia.com/gpu: 1) used by the NVIDIA Kubernetes Device Plugin. Starting with GPU Operator v26.7.0, the Operator can deploy and manage the DRA Driver for NVIDIA GPUs directly as a native operand, through a GPUCluster custom resource — no separate Helm chart required.

This guide walks through installing the GPU Operator with DRA support enabled, then allocating GPUs to workloads under two scenarios:

  • Full GPU — exclusive access to an entire physical GPU
  • MIG — a hardware-isolated Multi-Instance GPU slice of a supported GPU

Assumptions

This guide assumes the following are already in place:

  • A Kubernetes cluster on v1.34.2 or later, with the resource.k8s.io DeviceClass API served by the cluster (this is the DRA API; if it isn't served, the Helm install in Part 2 fails with a validation error)
  • One or more nodes with NVIDIA GPUs attached
  • For the MIG scenario, at least one MIG-capable data center GPU (Ampere architecture or newer — A100, A30, H100, and similar)
  • helm and kubectl configured against the target cluster

What You Will Do

Step Action
1 Install the GPU Operator with GPUCluster (DRA) support enabled
2 Validate the installation
3 Allocate a Full GPU to a workload
4 Allocate a MIG slice to a workload

Part 1 — Understand the DRA Model

The GPU Operator supports two GPU resource management models, and a cluster can only run one of them at a time:

Device-plugin model (ClusterPolicy) DRA model (GPUCluster)
GPU allocation NVIDIA Kubernetes Device Plugin (extended resources) DRA Driver for NVIDIA GPUs (ResourceClaims)
GPU driver Managed by ClusterPolicy or NVIDIADriver Pre-installed, or managed separately by NVIDIADriver — not managed by GPUCluster
NVIDIA Container Toolkit Deployed Not deployed (workloads use CDI through DRA)
MIG Manager Deployed Not deployed — MIG is handled by the DRA driver itself

Important: deploying and managing the DRA driver through GPUCluster is currently a Technology Preview, served under the nvidia.com/v1alpha1 API, and supports greenfield (new) installations only. There is no supported path to migrate an existing ClusterPolicy (device-plugin) installation to GPUCluster in place — don't run both on the same cluster.

Since GPUCluster doesn't manage the GPU driver, decide up front how the driver gets onto your GPU nodes:

  • Operator-managed — the GPU Operator installs and manages the driver through the NVIDIADriver custom resource
  • Pre-installed — the NVIDIA GPU driver is already installed on each GPU node, and the Operator leaves it alone

Both options are covered in Part 2.

Part 2 — Install the GPU Operator with DRA Support

Step 1 — Add the NVIDIA Helm Repository

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
  && helm repo update

Step 2 — Install with GPUCluster Enabled

Choose the command matching how the driver is managed on your cluster. Both set gpuCluster.deployCR=true and clusterPolicy.deployCR=false — the chart's own values file states plainly that having both CRs deployed at once is an invalid configuration, so clusterPolicy.deployCR=false isn't optional.

Operator-managed driver:

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --version=v26.7.0 \
  --create-namespace \
  --namespace gpu-operator-resources \
  --set gpuCluster.deployCR=true \
  --set clusterPolicy.deployCR=false \
  --set driver.nvidiaDriverCRD.enabled=true

Pre-installed driver:

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --version=v26.7.0 \
  --create-namespace \
  --namespace gpu-operator-resources \
  --set gpuCluster.deployCR=true \
  --set clusterPolicy.deployCR=false \
  --set driver.enabled=false

Note: if you're managing drivers through an NVIDIADriver resource on DRA nodes, disable automatic driver upgrades (spec.driver.upgradePolicy.autoUpgrade: false) — automatic driver upgrades aren't yet supported for DRA nodes.

If this node was already running device-plugin-model GPU workloads before this install: the driver-manager's uninstall step will fail if any process still holds the old NVIDIA kernel modules open (resource temporarily unavailable in the k8s-driver-manager init container logs, with ref counts on nvidia/nvidia_uvm greater than zero). Stop or delete whatever pod is still using the GPU on that node — the driver-manager can't unload a kernel module that's actively in use, and the DRA install won't proceed until it's free.

Part 3 — Validate the Installation

Step 1 — Confirm the GPUCluster Resource is Ready

kubectl get gpucluster

Expected output: 

NAME          STATUS   AGE
gpu-cluster   ready    3m12s

If this returns "No resources found" instead: Helm doesn't error on an unrecognized --set path — it silently accepts it and does nothing. Check what actually got deployed: 

kubectl get clusterpolicy
If you see a ClusterPolicy named cluster-policy instead of a GPUCluster, the values keys from Part 2 weren't applied. Double-check them against this chart version's actual values file (helm show values nvidia/gpu-operator --version v26.7.0) rather than trusting a flag name from memory or an older doc.

Step 2 — Confirm the DRA Operands are Running

kubectl get pods -n gpu-operator-resources

Expected output includes: 

NAME                                             READY   STATUS      RESTARTS   AGE
gpu-operator-...                                 1/1     Running     0          4m
nvidia-dra-driver-controller-...                 1/1     Running     0          3m
nvidia-dra-driver-kubelet-plugin-...              2/2     Running     0          3m
nvidia-dra-validator-...                         1/1     Running     0          2m
nvidia-dcgm-exporter-dra-...                      1/1     Running     0          2m

Note: nvidia-dra-validator becomes Running/Ready only after the DRA driver successfully allocates a GPU on the node — if it's stuck, the driver isn't allocating GPUs yet, and the rest of this guide won't work until that's resolved. The kubelet-plugin pod shows 2/2 because ComputeDomains are enabled by default alongside GPU allocation — if you disable ComputeDomains, expect 1/1 instead and no nvidia-dra-driver-controller Deployment.

Step 3 — Confirm the DeviceClasses are Available

kubectl get deviceclass

Expected output: 

NAME                                        AGE
compute-domain-daemon.nvidia.com            3m
compute-domain-default-channel.nvidia.com   3m
gpu.nvidia.com                              3m
mig.nvidia.com                              3m
vfio.gpu.nvidia.com                         3m

  • gpu.nvidia.com — Full GPU allocation (Part 4)
  • mig.nvidia.com — MIG slice allocation (Part 5)
  • The two compute-domain-* classes are for Multi-Node NVLink and aren't used in either scenario in this guide

Step 4 — Confirm GPUs are Published as ResourceSlices

kubectl get resourceslices

Expected output: 

NAME                                    NODE       DRIVER           POOL       AGE
00000-gpu.nvidia.com-<node>-...         <node>     gpu.nvidia.com   <node>     99s

Each GPU node should have one or more ResourceSlice objects describing its GPUs.

Part 4 — Scenario: Full GPU

A Full GPU gives a container exclusive access to one physical GPU — the default allocation mode, requiring no additional configuration.

Step 1 — Create a ResourceClaimTemplate

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: single-gpu
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
          allocationMode: ExactCount
          count: 1

Save as single-gpu.yaml and apply:

kubectl apply -f single-gpu.yaml

Step 2 — Create a Pod Referencing the Claim

apiVersion: v1
kind: Pod
metadata:
  name: gpu-workload
spec:
  restartPolicy: Never
  resourceClaims:
  - name: gpu
    resourceClaimTemplateName: single-gpu
  containers:
  - name: workload
    image: nvcr.io/nvidia/cuda:12.6.2-base-ubi9
    command: ["nvidia-smi", "-L"]
    resources:
      claims:
      - name: gpu

Save as gpu-pod.yaml and apply:

kubectl apply -f gpu-pod.yaml

Step 3 — Verify

kubectl logs gpu-workload

You should see the GPU listed (e.g. GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-...)), confirming the container received exclusive access to a physical GPU through the DRA claim rather than an extended resource request.

Part 5 — Scenario: MIG

MIG (Multi-Instance GPU) partitions a MIG-capable GPU into hardware-isolated slices, each with dedicated compute, memory, and L2 cache. Unlike time-slicing, MIG isolation is enforced by the GPU hardware itself.

Prerequisite: MIG mode must already be enabled on the target GPU, and (for static MIG, the default and simplest path) the MIG partitions must already exist on the node — created with nvidia-smi mig or mig-parted — before the DRA driver's kubelet plugin starts. On Ampere GPUs (e.g. A100), MIG mode can't be toggled without a GPU reset; if MIG mode is off, the driver falls back to full-GPU allocation and advertises no MIG partitions at all. Hopper and later architectures can enable MIG mode on demand.

Dynamic MIG — where the driver creates and destroys partitions automatically based on workload requests, with no pre-configuration needed — is available as an alpha feature gate (DynamicMIG, default false) if you want the driver to manage partitioning itself instead. The ResourceClaimTemplate and pod manifests below work identically in either mode.

Step 1 — Create the Example Namespace

kubectl create namespace mig-example

Step 2 — Request Any MIG Device

To request any available MIG slice without constraining the profile:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  namespace: mig-example
  name: any-mig
spec:
  spec:
    devices:
      requests:
      - name: mig
        exactly:
          deviceClassName: mig.nvidia.com

Save as any-mig.yaml and apply:

kubectl apply -f any-mig.yaml

Step 3 — Create a Pod Referencing the Claim

apiVersion: v1
kind: Pod
metadata:
  namespace: mig-example
  name: mig-pod
spec:
  containers:
  - name: workload
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["nvidia-smi -L; sleep 9999"]
    resources:
      claims:
      - name: mig
  resourceClaims:
  - name: mig
    resourceClaimTemplateName: any-mig
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"

Save as mig-pod.yaml and apply:

kubectl apply -f mig-pod.yaml

Note: ubuntu:22.04 doesn't include nvidia-smi — the DRA driver's CDI integration injects nvidia-smi and the host driver's libraries into the container at start, so a plain OS image is sufficient for verification.

Step 4 — Verify

kubectl get pod -n mig-example mig-pod
kubectl exec -n mig-example mig-pod -c workload -- nvidia-smi -L

Expected output: 

NAME      READY   STATUS    RESTARTS   AGE
mig-pod   1/1     Running   0          12s
GPU 0: NVIDIA A100-SXM4-40GB (UUID: GPU-ba3915a6-7bb2-0987-bcf1-6acec5d998b8)
  MIG 1g.10gb     Device  0: (UUID: MIG-76b4e048-83cf-5017-9d25-b1fac0396016)

The container sees only its allocated MIG slice (1g.10gb) as a distinct device with its own UUID — not the full physical GPU — confirming the hardware-level isolation.

Step 5 — Select a Specific MIG Profile (Optional)

To request a specific MIG profile (e.g. 1g.5gb) rather than any available slice, add a CEL selector matching the profile attribute the driver advertises:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  namespace: mig-example
  name: mig-profile
spec:
  spec:
    devices:
      requests:
      - name: mig
        exactly:
          deviceClassName: mig.nvidia.com
          selectors:
          - cel:
              expression: "device.attributes['gpu.nvidia.com'].profile == '1g.5gb'"

Note: available profile strings depend on the physical GPU model — check the ResourceSlice attributes on your cluster (kubectl get resourceslices -o yaml) to confirm what your hardware actually advertises before hardcoding a profile string.

Save as mig-profile.yaml, apply it the same way, and reference it from a pod with resourceClaimTemplateName: mig-profile, following the same pod pattern as Step 3.