Skip to content

Getting Started with HAMi on Rafay MKS

HAMi (Heterogeneous AI Computing Virtualization Middleware, formerly k8s-vGPU-scheduler) lets you split physical GPUs into fractional shares so multiple pods can run on the same card at once, without changing application code. This guide walks through running HAMi on top of a Rafay MKS cluster, installing the NVIDIA GPU Operator to handle the driver/toolkit layer, and HAMi to own GPU scheduling and sharing.

Prerequisites

  • A Rafay MKS cluster with a GPU node pool already provisioned
  • Console access to that cluster, and a working kubectl context
  • Helm >= 3.0 available wherever you're running kubectl/helm against the cluster
  • Kubernetes >= 1.23 (check with kubectl version)

1. Install the GPU Operator

From your kubectl/helm context against the MKS cluster:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install --wait --generate-name \
  -n gpu-operator-resources --create-namespace \
  nvidia/gpu-operator \
  --set devicePlugin.enabled=false \
  --set dcgmExporter.serviceMonitor.enabled=true

--set devicePlugin.enabled=false is the important part — it stops the Operator from registering its own nvidia.com/gpu device plugin, so HAMi's device plugin can take over that role without conflict. --wait blocks until all Operator pods are ready; the first install can take a few minutes while driver images download.

If the Operator's already installed on this cluster without that flag, helm upgrade it with --set devicePlugin.enabled=false before continuing.

Verify the Operator's resources

kubectl get pods -n gpu-operator-resources

You should see the driver daemonset, nvidia-container-toolkit-daemonset, feature discovery, and DCGM exporter pods Running — but no nvidia-device-plugin-daemonset, since it's disabled.

Check which runtime classes the toolkit registered

Modern GPU Operator versions default to CDI mode, but usually still register legacy runtime classes alongside it. Confirm what you have before moving on — it matters for step 7:

kubectl get runtimeclass

You're looking for nvidia, nvidia-cdi, and/or nvidia-legacy.

2. Label the GPU nodes

HAMi's scheduler only manages nodes labeled gpu=on:

kubectl label nodes <node-name> gpu=on

3. Install HAMi

From the same kubectl/helm context:

helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update

Rather than a plain helm install, use a values file — on a GPU-Operator-managed driver, HAMi's defaults don't reliably find the driver without a couple of explicit settings:

cat <<'EOF' > hami-driver-values.yaml
devicePlugin:
  nvidiaDriverRoot: /run/nvidia/driver
  extraArgs:
    - "-v=4"
    - "--device-discovery-strategy=nvml"
EOF

helm install hami hami-charts/hami -n kube-system -f hami-driver-values.yaml

What these two values are for:

  • nvidiaDriverRoot: /run/nvidia/driver — this is where the GPU Operator's containerized driver actually places driver files on the host (confirm with sudo find /run/nvidia/driver -iname 'libnvidia-ml.so*' on a GPU node). HAMi's default assumes a bare-metal driver at /, which doesn't exist on a GPU-Operator-managed node.
  • --device-discovery-strategy=nvml — HAMi's default auto strategy tries to read a marker file at /run/nvidia/validations/driver-ready to decide which driver root to use, but that path isn't part of the /driver-root mount the chart creates (that mount only covers /run/nvidia/driver). auto can never see the marker, so it falls through to an "incompatible strategy" error even though the driver is fine. Forcing nvml skips that broken check.

Verify both HAMi components come up:

kubectl get pods -n kube-system

Look for hami-scheduler and hami-device-plugin (as 2/2) in Running state. If either isn't coming up, double-check the GPU Operator's device plugin is actually disabled (step 3) — a lingering Operator device plugin is the most common cause of HAMi's device plugin failing to register.

hami-device-plugin runs two containers: device-plugin (the part that actually matters — registers and shares GPUs) and vgpu-monitor (a sidecar that only exports Prometheus metrics). On some GPU-Operator setups, vgpu-monitor can't initialize NVML in its own container namespace even once device-plugin is working fine — its logs end in failed to initialize NVML: ERROR_LIBRARY_NOT_FOUND, unrelated to whatever fixed device-plugin in step 5.

Since it's metrics-only, and not required for scheduling or sharing to work, the pragmatic fix is to drop it from the DaemonSet:

kubectl -n kube-system patch ds hami-device-plugin --type=json \
  -p '[{"op":"remove","path":"/spec/template/spec/containers/1"}]'

kubectl rollout status ds/hami-device-plugin -n kube-system --timeout=300s
kubectl get pods -n kube-system | grep hami-device-plugin

Once running, nvidia.com/gpu on each labeled node reports the vGPU count HAMi is offering rather than the raw physical GPU count — that's the resource type your workloads should request. Confirm with:

kubectl describe node <node-name> | grep -A15 "Allocatable:"

4. Run a GPU-sharing workload

Request a fraction of a GPU the same way you'd request any Kubernetes resource, using nvidia.com/gpu and, optionally, nvidia.com/gpumem to cap device memory.

Check your runtime class situation first (from step 3). If your GPU Operator's CDI specs don't cover plain workload GPU requests — check with sudo ls /etc/cdi/ /var/run/cdi/ on a GPU node and look for a generic per-GPU spec, not just management/compute-domain-specific files — add runtimeClassName: nvidia-legacy to force the older, hook-based driver injection instead of relying on CDI:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  runtimeClassName: nvidia-legacy   # only needed if plain nvidia.com/gpu CDI specs aren't present — see step 3
  containers:
    - name: ubuntu-container
      image: ubuntu:22.04
      command: ["bash", "-c", "sleep 86400"]
      resources:
        limits:
          nvidia.com/gpu: 1        # request 1 vGPU
          nvidia.com/gpumem: 10240 # cap this vGPU at 10240 MiB of device memory (optional)

kubectl apply -f gpu-pod.yaml

Because nvidia.com/gpu: 1 means "1 vGPU," not "1 whole physical card," you can schedule several pods like this one onto the same physical GPU, each with its own memory cap. To see two pods actually sharing one card, apply a second pod with a different cap:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod-2
spec:
  runtimeClassName: nvidia-legacy
  containers:
    - name: ubuntu-container
      image: ubuntu:22.04
      command: ["bash", "-c", "sleep 86400"]
      resources:
        limits:
          nvidia.com/gpu: 1
          nvidia.com/gpumem: 8192

kubectl apply -f gpu-pod-2.yaml
kubectl get pods -o wide

Both should end up Running on the same node, scheduled onto the same physical GPU. Confirm the scheduler's allocation:

kubectl describe node <node-name> | grep -A20 "Allocated resources:"

Alternatively, apply the repo's ready-made single-pod example instead of writing your own:

kubectl apply -f examples/nvidia/default_use.yaml

5. Verify the memory limit is enforced

kubectl exec -it gpu-pod -- nvidia-smi
kubectl exec -it gpu-pod-2 -- nvidia-smi

Each pod's nvidia-smi should report memory matching that pod's nvidia.com/gpumem cap, not the physical card's full memory, and not each other's:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.91.07              Driver Version: 595.91.07      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|=========================================+========================+======================|
|   0  NVIDIA A10                     On  |   00000000:00:04.0 Off |                    0 |
|  0%   28C    P8             21W /  150W |       0MiB /  10240MiB |      0%      Default |
+-----------------------------------------+------------------------+----------------------+

You can also compare both pods at once:

kubectl exec -it gpu-pod   -- nvidia-smi --query-gpu=memory.total,memory.used --format=csv
kubectl exec -it gpu-pod-2 -- nvidia-smi --query-gpu=memory.total,memory.used --format=csv

gpu-pod should report a 10240 MiB total and gpu-pod-2 an 8192 MiB total — two independent memory ceilings, both pods scheduled onto the same underlying card. That's HAMi's memory isolation working, not just device-count sharing.