Skip to content

2026

NUMA-Aware GPU VMs: How NVIDIA's Reference Architecture Actually Fixes It

In Part 1 of the blog series, we walked through the basics: NUMA nodes, remote-access penalties, and why a GPU VM that doesn't know its own topology can quietly sabotage itself by pinning a hot thread to one vCPU while its data lands on a memory bank a socket away.

We ended with a question we didn't answer: what does the fix actually look like in practice?

NVIDIA has published a prescriptive recommendation called Performance Reference Architecture. It's worth walking through this in some detail, because it's the closest thing to an industry-standard answer for how do I make a GPU VM behave like bare metal?.

Why is your GPU VM slower Than Its Twin?

Picture two GPU VMs on the same bare-metal host, provisioned identically: same GPU model, same vCPU count, same memory allocation, same container image. One of them runs your inference workload 20% slower than the other — every time, consistently, with no other tenant contention and nothing wrong in the logs.

The difference isn't the GPU. It's the distance between the CPU cores feeding that GPU and the memory those cores are reading from. That distance has a name Non-Uniform Memory Access (NUMA). On modern multi-socket, multi-GPU hosts, it's one of the most common, least visible sources of inconsistent performance in GPU cloud environments.

At Rafay, we spend a lot of time thinking about how to make GPU infrastructure predictable because "predictable" is the whole point of a platform. So, we thought it is worth taking a step back and explaining what NUMA actually is, why it matters esp. once GPUs enter the picture, and what a platform needs to do about it.

This is the first part of a blog series on NUMA, how it impacts VMs esp. GPU VMs and approaches that are required to deliver great performance to GPU VMs on moden platforms.

Serving LLMs on Arm: Running Rafay Token Factory on NVIDIA DGX Spark

NVIDIA DGX Spark put a Grace Blackwell-class machine on the desk. It is roughly the size of a hardback book, draws a fraction of the power of a rack server, and ships with 128 GB of unified memory that lets you load models far larger than a typical workstation GPU can hold. For developers, researchers, and platform teams, it is one of the most interesting pieces of AI hardware to appear in a long time.

It also comes with a characteristic that trips up a lot of inference tooling: the DGX Spark is Arm-only. The NVIDIA GB10 Grace Blackwell Superchip pairs a Blackwell GPU with a 20-core Grace CPU powered by Arm (10 Cortex-X925 performance cores and 10 Cortex-A725 efficiency cores) over an NVLink-C2C link. By default, it runs DGX OS, an Ubuntu-based, AArch64 operating system.

Inference stacks that assume amd64 containers, x86 wheels, or x86-only base images do not run here without work.

This is exactly the kind of heterogeneity Rafay's Token Factory is designed to absorb. In this post I will walk through how Token Factory turns a single Arm-based DGX Spark into a managed, multi-tenant LLM serving endpoint, using a real deployment of Qwen2-0.5B-Instruct as the example.

DGX Spark in Hand

Running LLM Inference on Arm-based systems

When most teams say "deploy an LLM," they mean a workflow that has been quietly assuming x86 for years. The inference server image is built for amd64. The CUDA wheels are compiled for x86. The orchestration layer schedules onto x86 worker nodes. Most of this stack will require some work to run on a Grace Blackwell box by default.

On the DGX Spark, every layer has to be AArch64-native:

  • The container images for the inference engine and its dependencies
  • The CUDA and driver stack matched to the Blackwell GPU on the GB10 (compute capability sm_121, which is distinct from data-center Blackwell parts like the B200)
  • The Kubernetes node components and the GPU operator that exposes the device to pods
  • Any sidecars, gateways, or metering agents that ride alongside the model

The promise of Token Factory is that Rafay invisibly handles this substrate for you. You register the machine as a compute cluster, and from that point on the experience is the same whether the underlying silicon is an x86 H100 server, a GB200 NVL72 rack, or a single Arm DGX Spark on a desk.

The Architecture

Token Factory sits on top of a Kubernetes substrate and exposes LLM inference serving as a set of higher-level objects: a compute cluster (where models run), an endpoint (the network front door), a provider and model (what you are serving), and a model deployment (the running, scalable instance with its inference engine, rate limits, and pricing).

End users never see Kubernetes. They get an OpenAI-compatible API, an API key, and a usage dashboard. The operator/service provider gets multi-tenancy, metering, and governance. The DGX Spark just happens to be the place where the tokens are generated.

Step 1 — Provision Kubernetes on DGX Spark

Token Factory runs on a Kubernetes substrate, so before any model can be served, the DGX Spark systems need a cluster on it. This is the first place an Arm-only machine needs a different configuration: many Kubernetes distributions and installers still assume x86 worker nodes, ship amd64-only system images, or pull control-plane components that have no aarch64 build. On a Grace Blackwell box, all of that has to be native Arm.

We provision the cluster using Rafay MKS, Rafay's upstream, CNCF-conformant Kubernetes distribution for bare metal and VM environments. MKS is built to run directly on the hardware you bring, and it supports a fully AArch64-native, Arm-only deployment, which is what makes it a fit for the DGX Spark. There is no x86 control-plane node hiding in the topology; the entire cluster runs on the Grace Blackwell silicon.

On a single DGX Spark the result is a compact, single-node cluster where the control plane and the worker role co-reside on the same machine:

  • Control plane and kubelet run as AArch64 components on the Spark's Arm cores. The 20-core Grace CPU has more than enough headroom to host the control plane and still leave the bulk of its cores, and the unified memory, for inference.
  • The container runtime and CNI are Arm-native. (A common pattern here is to pair a CNI such as Calico or Cilium for pod networking)
  • The NVIDIA GPU Operator installs the AArch64 driver, CUDA runtime, and Kubernetes device plugin that expose the GB10's Blackwell GPU to pods. This is the layer that turns "a GPU exists in the box" into "the scheduler can place a model on it," and it is the piece where architecture alignment matters on a non-x86 platform. MKS handles this as part of bringing up a GPU cluster.

Practically, the operator points Rafay at the DGX Spark, which bootstraps Kubernetes and the GPU software stack on the node, and a few minutes later there is a healthy, GPU-aware, Arm-native cluster ready to serve workloads. The DGX Spark is now a Kubernetes node like any other, except that every layer of that stack is AArch64.

Step 2 — Register the DGX Spark as a Compute Cluster

The next step is to bring the Kubernetes cluster on DGX Spark under management as a GPU compute cluster. Once the Kubernetes cluster is registered, Token Factory will automatically discover the hardware and surfaces it in the Rafay Console.

GB10 in Compute Cluster

A few things in this view are worth calling out:

  • The single node, spark-2ceb, reports its accelerator as NVIDIA-GB10, and the GPU Inventory by Type panel classifies it as Blackwell. The platform correctly identifies the GB10 silicon rather than treating it as a generic device.
  • The CPU panel reads 6,511 / 20,000 mCores. That 20,000 millicore ceiling is the 20 Arm cores of the Grace CPU, visible directly in the dashboard.
  • Memory reads 16.14 GB / 119.67 GB. That ~120 GB pool is the DGX Spark's 128 GB of coherent unified memory, which is the whole reason this box can hold sizable models in the first place.
  • GPUs show 1 allocated / 1 total at 100%, the single Blackwell GPU on the superchip.

From here, day-2 operations are managed: utilization telemetry, monitoring alerts etc. The DGX Spark is now a first-class citizen of the fleet, indistinguishable in workflow from any other GPU cluster.

Step 3 — Define the endpoint, provider, and model

With the cluster connected, the operator stitches together the serving objects. In Token Factory terms:

  • The endpoint is the addressable front door for inference traffic.
  • The provider and model describe what is being served. Here the model is Qwen2-0.5B-Instruct, a small instruct-tuned model that is a sensible first deployment on a single-GPU box: it loads comfortably into unified memory, serves quickly, and is a clean way to validate the full path before moving to larger models.
  • Model sharing controls which tenants and projects can see and consume the model, which is what makes a single DGX Spark useful to more than one team.

Info

Note that none of these objects care about the CPU architecture underneath. The endpoint and model abstractions are the same on Arm as on x86, which is the point.

Step 4 — Create the Model Deployment

The model deployment is where everything comes together: the model, the endpoint, the target GPU, the inference engine, rate limits, and pricing. This is the screen the operator fills in to actually bring the model online.

Model Deployment on GB10

Reading down the form:

  • Details name the deployment Qwen2-0.5B-Instruct-model
  • Model & Endpoint bind Qwen2-0.5B-Instruct to the token-factory-on-gb10-spark endpoint.
  • GPU Type is the important part. Token Factory presents NVIDIA-GB10 as a selectable target with 1 GPU available of 1 total, and the deployment is pinned to it. The platform has already done the hard scheduling work of matching the model to the Blackwell GPU on the Arm node. The operator simply selects it.
  • The remaining sections, Inference Engine, Rate Limiting, and Pricing, configure how the model is served, how aggressively clients can call it, and how usage is metered and charged back.
  • Click Create Model Deployment, and the model comes online on the DGX Spark.

Info

Behind that Inference Engine selection is where the Arm-native heavy lifting lives. Token Factory pulls AArch64 inference-engine images and the matching CUDA stack for the GB10, schedules the serving pod onto the Spark node via the GPU operator, and wires it to the endpoint. The operator never builds an Arm container, never recompiles a wheel, and never debugs an architecture mismatch. They pick an engine from a dropdown.

Step 5 — Operate & Use It

Once deployed, the DGX Spark behaves like a managed inference service, not a hobbyist's desktop experiment:

  • End users consume an OpenAI-compatible API. They generate API keys, send requests, and watch their own usage dashboards. They have no idea, and no need to know, that the tokens are coming off an Arm-based superchip under someone's desk.
  • Rate limiting protects the single GPU from being overwhelmed by any one tenant, which matters far more on a one-GPU box than on a large cluster.
  • Usage metering and pricing make the deployment chargeable, so even a desktop-class machine can participate in showback or a paid internal service.
  • Multi-tenancy and model sharing let several teams share the one DGX Spark with isolation and quotas, turning a single device into shared infrastructure.

Scaling is the natural next move. The same workflow that brought up one DGX Spark brings up a second, and NVIDIA's own design anticipates this: two Sparks can be linked over ConnectX networking into a 256 GB combined-memory pair for models in the 405B-parameter range. Token Factory treats those as additional capacity in the fleet, and the model-deployment abstraction is unchanged.

Why this Matters

The DGX Spark is a preview of where a lot of AI compute is heading: Arm-based, memory-rich, energy-efficient, and increasingly distributed out toward the edge rather than concentrated in a few data centers. The hardware is genuinely exciting. The operational reality, an AArch64 stack that is not aligned with x86 assumptions at every layer, is where most teams lose weeks.

Rafay Token Factory collapses that gap. The same five-object workflow, compute cluster, endpoint, provider and model, model deployment, and end-user consumption, applies whether the silicon is x86 or Arm, a single desktop or a rack-scale GB200 system. You register the machine, pick the GPU, pick the engine, and serve tokens. The architecture under the hood becomes an implementation detail, which is exactly what platform software is supposed to do.

A Grace Blackwell machine on your desk, serving a production-style LLM endpoint with metering, rate limits, and multi-tenancy, and no one had to think about Arm. That is the whole idea!

Bring Rafay Into Your AI Workflows with the Rafay MCP Server

AI assistants are now part of everyday work for platform, DevOps, and SRE teams. We use them to debug code, make sense of configuration, and understand how systems behave. But when it comes to managing Kubernetes clusters and platform infrastructure, these assistants hit a wall: they have no secure, real-time view of your environment.

Without a secure window into your actual operational state AI tools are forced to guess rely on stale data or require engineers to manually copy paste massive YAML files and CLI outputs into chat windows.

To bridge this gap without compromising on security, we are thrilled to introduce the Rafay MCP Server.

Kubernetes v1.36 for Rafay MKS

As part of our continuous effort to bring the latest Kubernetes versions to our users, support for Kubernetes v1.36 (codename ハル / Haru) is now available on the Rafay Operations Platform for MKS cluster types.

Both new cluster provisioning and in-place upgrades of existing clusters are supported. As with most Kubernetes releases, this version deprecates and removes a number of features. To ensure zero impact to our customers, we have validated every feature in the Rafay Kubernetes Operations Platform on this Kubernetes version.

Recommended: Platform Version 1.3.0

Rafay Platform Version 1.3.0 is the default selection in the UI when provisioning new Kubernetes clusters. It includes containerd 2.3.0 (CRI 2.3.0). Rafay recommends using Platform Version 1.3.0 with Kubernetes v1.36 to take advantage of all new stable features in the 1.36 release. For upgrading existing clusters, you can upgrade to Platform Version 1.3.0 separately first, or together with the Kubernetes v1.36 upgrade.

Kubernetes v1.36 Release

Automated GPU Health Monitoring with NVIDIA NVSentinel on the Rafay Platform

GPU clusters are expensive and GPU failures are costly. In modern AI infrastructure, organizations operate large fleets of NVIDIA GPUs that can cost tens of thousands of dollars each. When a GPU develops a hardware fault (e.g. a double-bit ECC error, a thermal throttle, or a silent data corruption event), the consequences ripple outward: training jobs fail hours into a run, inference latency spikes, and expensive hardware sits idle while engineers scramble to diagnose the root cause.

Traditional monitoring catches these problems eventually, but rarely fixes them. Diagnosing and remediating GPU faults still requires deep expertise, and remediation timelines are measured in hours or days. For organizations running AI workloads at scale — and especially for GPU cloud providers who must deliver uptime SLAs to their tenants — this gap between detection and resolution translates directly into SLA breaches, lost revenue, and eroded customer trust.

NVIDIA's answer to this challenge is NVSentinel — an open-source, Kubernetes-native system that continuously monitors GPU health and automatically remediates issues before they disrupt workloads.

In this blog, we describe how Rafay integrates with NVSentinel enabling GPU cloud operators and enterprises to deploy intelligent GPU fault detection and self-healing across their entire fleet — consistently, repeatably, and at scale.

Rafay and NVSentinel

  • 5 min read

As AI workloads continue to scale, the bottleneck is no longer just raw GPU availability. Increasingly, performance depends on how efficiently GPUs can communicate across nodes. That is especially true for large model training, distributed inference, and pipeline-parallel workloads that rely heavily on high-bandwidth, low-latency interconnects.

NVIDIA’s ComputeDomains introduce a Kubernetes-native abstraction for managing multi-node NVLink-connected GPU communication dynamically. Rather than forcing platform teams to preconfigure static communication groupings between nodes, ComputeDomains enable Kubernetes to allocate and tear down those groupings as part of workload scheduling.

This shift has important implications for performance, utilization, isolation, and the operational model of large-scale AI platforms.

Infra Gap

The Infrastructure Gap for AI Platforms

Kubernetes has become the default control plane for modern infrastructure, including GPU-backed environments. But traditional Kubernetes scheduling is still mostly centered on node-local resources such as CPU, memory, storage, and device counts.

That model starts to break down for modern AI systems. In rack-scale systems such as NVIDIA’s GB200 NVL72, GPUs are no longer best viewed as isolated devices attached to individual servers.

Through NVLink, they are part of a larger connected fabric that enables high-speed communication across GPUs and, increasingly, across node boundaries. In practice, that means a distributed workload may care less about “how many GPUs are free” and more about “which GPUs can communicate efficiently as a coordinated compute domain.”

That distinction matters because many AI jobs do not simply request GPU capacity. They implicitly require:

  • Fast cross-GPU communication,
  • Coordinated memory access,
  • Workload-scoped isolation, and
  • Topology-aware placement.

Without fabric awareness in the orchestration layer, platform teams are left bridging the gap manually.

Static Configuration in Kubernetes

Historically, enabling secure, high-bandwidth GPU communication across multiple nodes has required a fair amount of manual planning. Infrastructure operators had to predefine how workloads would map to node groups and how communication permissions would be set up between them.

That works very poorly in a Kubernetes world.

Kubernetes is designed for dynamic placement, rescheduling, elasticity, and failure recovery. Static node-to-node communication domains work against those strengths. They introduce operational rigidity, increase the burden on cluster admins, and make it harder to share expensive GPU infrastructure efficiently across multiple teams or tenants. For AI platform teams, the result is a familiar tradeoff:

  • Preserve flexibility and sacrifice performance-aware placement,
  • Optimize for interconnect performance and accept a more brittle operational model.

ComputeDomains are meant to remove that tradeoff.

What Compute Domains do

At a high level, ComputeDomains extend NVIDIA’s Dynamic Resource Allocation (DRA) driver to make Kubernetes aware of cross-node NVLink-enabled communication requirements. When a distributed job is scheduled, the platform can automatically create a communication domain around the nodes that host that workload. When the job finishes, the domain is removed.

Under the hood, this is tied to NVIDIA’s Internode Memory Exchange Service (IMEX), which manages GPU memory permissions across nodes. In earlier implementations, IMEX domains had to be configured manually. ComputeDomains effectively bring that lifecycle into the Kubernetes control plane.

The result is a simpler model:

  1. A workload requests distributed GPU resources.
  2. Kubernetes schedules the workload on eligible nodes.
  3. A matching compute domain is created automatically.
  4. The job gets the communication and memory-sharing capabilities it needs.
  5. The domain is torn down when the workload completes.

That is a meaningful step forward because the communication fabric becomes a dynamic, workload-scoped resource, rather than a static infrastructure construct.

Compute Domains Lifecycle

Why does this Matter?

ComputeDomains are interesting for three reasons.

1. Make GPU scheduling more Topology-aware

Traditional GPU scheduling mostly focuses on quantity. Distributed AI workloads, however, often care just as much about connectivity and communication bandwidth.

ComputeDomains push the platform toward a more accurate scheduling model, one that better reflects how large-scale GPU systems actually behave. Instead of merely assigning accelerators, the control plane begins to account for how those accelerators work together as a communication fabric.

2. Improve Shared-cluster Utilization

Static partitioning of high-performance GPU fabrics often leads to fragmentation and idle capacity. Clusters reserve topology-specific resources for certain workloads even when those resources are not being fully used.

By creating communication domains dynamically, the platform can align resource allocation more closely with actual demand. That makes it easier to run shared AI infrastructure efficiently without hard-coding cluster topology into every workload deployment.

3. Strengthen Isolation for Multi-tenancy

In large shared GPU environments, performance is only one concern. Isolation matters just as much.

ComputeDomains help create isolated communication zones around a workload so that neighboring jobs cannot access GPU memory spaces outside their assigned domain. For enterprises operating multi-tenant AI platforms, that is a significant capability. Strong isolation has to be part of the design, not an afterthought layered on later.

Multi Tenancy

How Kubernetes Needs to Evolve for AI

The larger takeaway is not just about one NVIDIA feature. It is about where AI infrastructure is headed.

The classic Kubernetes model assumes that compute is mostly node-local and that devices are consumable as independent resources. Modern AI hardware challenges that assumption. Increasingly, the meaningful resource is not a single GPU but a connected set of GPUs with specific bandwidth and latency characteristics.

As that becomes the norm, orchestration has to evolve as well. ComputeDomains are an example of that evolution.

This represents a move from "Device-centric" allocation to "Fabric-aware" allocation with workload-scoped communication and isolation built into scheduling.

That direction is likely to influence not just high-end training clusters, but eventually a broader set of AI platform architectures as distributed inference and larger model serving topologies become more common.

Operational Considerations

NVIDIA’s current implementation requires Kubernetes 1.32 or later and Container Device Interface (CDI) support. NVIDIA has also indicated that the feature is still evolving, with more work planned around elasticity and fault tolerance.

That means platform teams should view ComputeDomains as an important architectural direction rather than a finished end state.

Summary

AI infrastructure is becoming more fabric-centric, more distributed, and more dependent on interconnect performance. As that happens, the orchestration layer cannot remain blind to communication topology.

ComputeDomains are a notable step toward making Kubernetes more aware of how modern GPU systems actually operate. By dynamically creating and managing workload-scoped communication domains for NVLink-connected GPUs, NVIDIA is moving multi-node GPU fabrics closer to becoming a first-class platform resource.

NVIDIA Dynamo: Turning Disaggregated Inference Into a Production System

In Part 1, we covered the core idea behind disaggregated inference. That architectural split is no longer just a research pattern. Disaggregated inference changes inference from a simple “deploy a container on GPUs” exercise into a distributed system problem.

Once prefill and decode are separated, the platform has to coordinate routing, GPU-to-GPU KV cache transfer, placement, autoscaling, service discovery, and fault handling across multiple worker pools. NVIDIA Dynamo provides the distributed inference framework for this, and Kubernetes provides the control plane foundation to operate it at scale. 

In this blog post, we will review NVIDIA's Dynamo project with a focus on what it does and when it it makes sense to use it.

NVIDIA Dynamo Logo

OpenClaw and NemoClaw: A Better Way to Consume AI Services Through Token Factory

As AI adoption accelerates, most businesses do not actually want to manage GPU clusters, model serving stacks, or low-level infrastructure. What they want is simple, reliable access to powerful models through tools their teams can use immediately. That is exactly the value of combining OpenClaw and NVIDIA NemoClaw with a service provider’s deployment of Rafay Token Factory.

OpenClaw is the user-facing interface where people interact with models and AI assistants. NemoClaw extends that experience with additional security and control for long-running or always-on agents. In both cases, the user experience can remain simple: connect to the provider, use tokens, and start working.

The complexity of GPUs, inference infrastructure, scaling, and capacity planning stays behind the scenes. OpenClaw is the open-source AI agent platform, while NVIDIA describes NemoClaw as an open-source reference stack for running OpenClaw more safely with policy-based privacy and security guardrails.

OpenClaw with Token Factory