NVIDIA AI factory stack · Part 2 of 4

Every GPU Kubernetes team eventually hits the same wall: the cluster is up, but which GPU driver, which GPU Operator, which network operator, which kernel modules and which inference stack versions actually work together on this cloud, with this GPU and this OS? NVIDIA AI Cluster Runtime (AICR) turns that tribal knowledge into versioned, signed, testable recipes. This part explains how a recipe is built, how to generate and deploy one, what aicr validate really runs on your cluster, and how AICR puts NVIDIA Dynamo on Kubernetes.

The problemWhy do GPU clusters need recipes at all?

A GPU node is a tall stack: kernel and kernel modules, the NVIDIA driver, the container toolkit, the GPU Operator, the Network Operator for InfiniBand or EFA, Dynamic Resource Allocation (DRA) drivers, a scheduler such as KAI or Kueue, monitoring, and finally the platform you actually care about, such as Dynamo, Kubeflow or Slurm. Each piece has its own release cycle, and a small mismatch (a driver too old for a CUDA image, a missing kernel module for RDMA, a Kubernetes version without DRA) shows up as a failure that is slow to diagnose and expensive to reproduce.

The number of combinations is the real problem. AICR’s own support list today covers 15 Kubernetes services, 11 GPU types, 8 operating systems, 2 workload intents and 6 platforms:

AICR is NVIDIA’s way of publishing what used to live in its internal validation pipelines and runbooks. Every recipe is described as optimised for one combination, validated by automated checks before publishing, and reproducible: the same inputs produce the same deployment every time.

How recipes are builtWhat is inside a recipe?

Recipes are not written one per combination. They are assembled from overlays, small YAML files that each add what is specific to one layer, and mixins, shared fragments such as “Ubuntu OS constraints” that many overlays reuse. Each overlay names the overlay it builds on. Here is the real chain behind the H100 + EKS + Ubuntu + Dynamo recipe in the repository:

The leaf overlay is short and readable. This is an abbreviated copy of the one above from the repository. Notice the three parts: the criteria it matches, the constraints a cluster must meet (here, Kubernetes 1.34 or newer, because DRA must be GA), and the components it adds with their dependencies, followed by the validation checks it declares:

kind: RecipeMetadata
apiVersion: aicr.run/v1beta1
metadata:
  name: h100-eks-ubuntu-inference-dynamo
spec:
  base: h100-eks-ubuntu-inference
  criteria: {service: eks, accelerator: h100, os: ubuntu,
             intent: inference, platform: dynamo}
  constraints:
    - name: K8s.server.version
      value: ">= 1.34"            # DRA is GA from 1.34
  componentRefs:
    - name: grove
      type: Helm
      source: oci://ghcr.io/ai-dynamo/grove
    - name: dynamo-platform
      type: Helm
      source: https://helm.ngc.nvidia.com/nvidia/ai-dynamo
      dependencyRefs: [grove, cert-manager, prometheus-operator-crds,
                       gpu-operator, kai-scheduler]
  validation:
    deployment:
      checks: [operator-health, expected-resources]

The components a recipe can draw on fall into six groups: the GPU stack (GPU Operator, DRA GPU driver, Network Operator, Node Feature Discovery, NVSentinel), cloud integration (AWS EFA, EBS CSI, GKE TCPXO), node tuning (Nodewright, cert-manager), observability (kube-prometheus-stack and friends), training platforms (Kubeflow Trainer, Slinky Slurm, KAI Scheduler, Kueue) and inference platforms (Dynamo, Grove, NIM Operator, Agent Gateway).

Hands onHow do you go from recipe to running cluster?

# install
brew tap NVIDIA/aicr && brew install aicr
# or: curl -sfL https://get.aicr.run | bash -s --

# 1. capture what the cluster really runs (runs a short-lived Job on a GPU node)
aicr snapshot --output snapshot.yaml

# 2. resolve the recipe for your target
aicr recipe --service eks --accelerator h100 --os ubuntu 
  --intent inference --platform dynamo -o recipe.yaml

# ask for a single resolved value, such as the GPU driver version
aicr query --service eks --accelerator h100 --os ubuntu 
  --intent inference --platform dynamo 
  --selector components.gpu-operator.values.driver.version

# 3. render it for your deployment tool
aicr bundle --recipe recipe.yaml --deployer helmfile --output ./bundles

# 4. deploy with that tool, then 5. validate what the bundle delivered
aicr validate --recipe ./bundles/recipe.yaml --phase deployment
aicr validate --recipe ./bundles/recipe.yaml

Two details are easy to miss. First, every bundle writes the recipe it was built from to recipe.yaml at its root, minus anything it did not render, so validating against ./bundles/recipe.yaml checks what was actually deployed rather than what you originally asked for. Second, aicr snapshot can detect the service for you: on NVKE, for example, nodes carry a nico:// provider ID, so aicr recipe --snapshot picks the NVKE recipe without a --service flag.

Trust, but verifyWhat does aicr validate actually run?

This is the part that separates AICR from a folder of Helm values. Validation is not a YAML lint. It deploys real Kubernetes Jobs into your cluster and checks results in phases:

Two practical notes. The identity running aicr validate effectively needs cluster-admin, because each run creates a service account for its validator Jobs; run it from a CI identity, not a personal one. And the inference benchmark sends warm-up requests first, so the reported throughput and p99 time to first token reflect steady state rather than vLLM’s one-time compile on a cold worker. If you have read how many GPUs you need, these are the same two numbers that decide a sizing.

Support levelWhich recipes can you rely on?

A recipe file existing in the repository is not a promise that it works. AICR labels every coordinate with a maturity level, and publishes the evidence behind it on a public dashboard where each result can be re-verified with aicr evidence verify:

Supply chain is handled to the same standard. Recipes are signed by NVIDIA CI, and that provenance can be checked offline. Images carry SLSA Build Level 3 provenance, signed SBOMs and Cosign attestations, and aicr verify checks a bundle before it reaches a cluster. For recipes with published evidence, a second signature ties a named contributor to a recorded validation result.

InferenceHow does AICR put Dynamo on Kubernetes?

NVIDIA Dynamo is the inference framework that this stack builds toward. A Dynamo deployment on Kubernetes is described by a DynamoGraphDeployment, which says how vLLM, SGLang or TensorRT-LLM workers should run. But that graph is only the top layer; underneath it needs the GPU Operator, networking, DRA, a gang scheduler, Grove, certificates and monitoring. An AICR Dynamo recipe installs all of that, plus the Dynamo platform chart (version 1.4.2 in NVIDIA’s walkthrough), and validates it. It supports both routing paths:

NVIDIA’s walkthrough flags a couple of changes worth knowing if you used Dynamo earlier. The default AICR path no longer installs NATS (it remains an opt-in), using TCP for requests and ZMQ for KV-cache events instead, and the DynamoGraphDeployment API moved to v1beta1. The routing ideas will be familiar from the llm-d post; AICR’s contribution is getting everything under the router installed and checked in one step.

DecisionShould you use AICR?

Quick answers

Does AICR work on VMware VKS or my own bare metal?

There is no VKS-specific recipe in the list. There are generic, kind, Metal3, RKE2 and k0s services, and the recipe development guide explains how to add an overlay for your own environment; treat that as integration work, not a supported path.

Does it lock me into one GitOps tool?

No. The same recipe renders for Helm, Helmfile, Argo CD (two styles), Flux, Terraform or Rancher Fleet. That is the main reason AICR stops before the deploy step.

Can I change values in a recipe?

Yes, at bundle time with --set. Note that bundle-time overrides are not written back into the bundle’s recipe.yaml, so validation will not see them.

Can I run it without internet access?

Yes. The aicrd API server and the air-gap mirror guide cover disconnected clusters, and signature checks work offline.

How does it relate to NVKE?

NVKE (Part 1) creates the cluster and stops at networking. AICR installs everything the GPUs need inside it, and NVKE is one of AICR’s supported services.

Sources: AICR repository and documentation (github.com/NVIDIA/aicr), including the README, ROADMAP.md, CLI reference, Validating a Cluster, NVKE GB200 Setup and the recipe overlays, read at commit d2db931 (7 November 2026, latest release v1.2.0). NVIDIA Technical Blog, “Build a Validated Kubernetes Runtime for NVIDIA Dynamo”. The overlay listing is abbreviated from recipes/overlays/h100-eks-ubuntu-inference-dynamo.yaml. Supported services, GPUs and maturity levels change between releases; check the validation dashboard for current status.