NVIDIA AI factory stack · Part 1 of 4

If you run NVIDIA GPU servers and want to give customers their own Kubernetes clusters, you need more than a Kubernetes installer. You need an API that creates clusters on request, a separate control plane for every customer, upgrades, node replacement, a way to trust each node, and telemetry across the fleet. NVIDIA Kubernetes Engine (NVKE) is NVIDIA’s open-source answer to that problem. This part explains what NVKE is, how it builds a cluster from bare metal, how to try it on one Linux machine without GPUs, and what is still missing before you would run it for real.

The problemWhat does NVKE actually do?

NVKE is a management plane that an AI factory operator installs to run Kubernetes as a service on its own NVIDIA GPU machines. The operator asks for a cluster through the NVKE API or the nvkectl CLI. NVKE gets bare-metal machines, forms a Kubernetes cluster on them with its own control plane, etcd and node pools, and hands the customer a kubeconfig. After that NVKE carries out the day-2 work: scaling node pools, upgrading Kubernetes, replacing nodes and renewing certificates.

Three things it is not, because the name invites confusion:

If you know VMware VKS, the shape is familiar: a management cluster that creates tenant clusters. The difference is the target. NVKE runs on bare-metal NVIDIA hardware through NICo, and is aimed at GPU clouds and NVIDIA Cloud Partners that sell Kubernetes, inference or training services to their own customers.

Where it sitsHow does NVKE fit with NICo, AICR and the rest?

NVKE is one layer of DSX OS, NVIDIA’s software stack for AI factories. It sits on top of NICo and below everything that runs inside a cluster. Each layer is a separate open-source project.

A typical site comes up in this order: NICo is stood up and hosts are enrolled; NVKE bootstraps the management cluster; the NVKE services are installed on it; a workload cluster is created; an AICR recipe installs the GPU stack in that cluster; CRE runs real GPU workloads to certify it; NVSentinel watches node health from then on. NVKE’s own docs are clear about who owns what:

ArchitectureWhat are the parts of an NVKE installation?

Every installation has two kinds of cluster. The management cluster is a small cluster, an odd number of nodes such as three, where every node is a control-plane node that also runs the NVKE services. Operators manage it; customers never see it. Workload clusters (the docs also call them tenant clusters) are the GPU clusters customers use.

Clusters use OVN-Kubernetes for networking, in an NVIDIA build of that open-source project. The preview pins Kubernetes 1.36, Ubuntu 24.04 or 26.04 with NVIDIA-tuned kernels, and amd64 or arm64 nodes.

Under the hoodHow does an API call become a cluster?

The interesting design choice is that the API never starts the work itself. It writes a database row, and something else picks it up. Follow one request:

This is called the outbox pattern. If the controller restarted between saving the cluster and starting the workflow, a naive design would leave a cluster record with nothing building it. Here, the outbox row survives the restart and the processor starts the workflow later. Long operations such as rolling upgrades run as Temporal workflows for the same reason: a failed step is retried, not the whole operation.

Notice also step 9. The controller dials nvke-agent on each node over gRPC. NVKE needs no SSH access to nodes, and the agent never opens connections back to the controller.

Day 0How is the management cluster built?

There is a chicken-and-egg problem: NVKE builds clusters, but NVKE itself needs a cluster to run on. It solves this with a seed cluster, a temporary single-node kind cluster on the bootstrap host that runs a copy of the management plane just long enough to build the real management cluster, then deletes itself.

In practice, after you have booted the nodes from the node image and recorded each node’s TPM thumbprint, the bootstrap is three commands. The site spec is generated from a CSV of your nodes:

# hosts.csv: name,ip,tpm_ek_thumbprint (one row per management node)
nvkectl generate sitespec example-mgmt --nodes hosts.csv 
  --set cluster.k8sVersion=1.36.3 
  --set cluster.controlPlaneVIP=<API_VIP> 
  --set spire.trustDomain=mgmt.example.com 
  --set spire.upstreamCA=true > sitespec.yaml

# check everything without touching a node
nvkectl bootstrap site -f sitespec.yaml --dry-run

# form the management cluster (30-minute timeout by default)
nvkectl bootstrap site -f sitespec.yaml

Then you install the management-plane services (the controller, Postgres, Temporal, SPIRE, CAPNICo and telemetry) from the repository’s deploy path, connect NVKE to NICo, and turn on API authentication. Two outputs of the bootstrap need care: the break-glass admin kubeconfig, and the etcd encryption key file, which is the only copy of that key that is not on a node.

TrustHow does NVKE know a node is genuine?

On a shared GPU platform, a node that joins the wrong cluster, or a fake node that joins at all, is a serious incident. NVKE does not use shared join tokens copied around by scripts. Each node proves its identity with its TPM chip through SPIFFE/SPIRE before it gets anything.

The same identity carries through the node’s life. When the controller later dials nvke-agent to drain or upgrade a node, both sides have already verified each other’s hardware-rooted identity.

Day 1 and 2What does creating and running a cluster look like?

A workload cluster is one YAML request. This example is trimmed from the repository’s NICo example: a three-node control plane behind a VIP and a pool of two GPU workers. The IDs come from your NICo site, and the release names a node image release you have published.

# cluster.yaml (IDs shortened)
name: team-a
controlPlaneKubernetesVersion: 1.36.3
cniPlugin: ovn-kubernetes
etcdEncrypted: true
controlPlaneVIP: 10.0.0.100
nico:
  orgID: <NICO_ORG_ID>
  siteID: <NICO_SITE_ID>
  vpcID: <NICO_VPC_ID>
controlPlaneConfig:
  desiredCount: 3
  nico:
    instanceTypePreference: [<CP_INSTANCE_TYPE_ID>]
    vpcPrefixIDs: [<VPC_PREFIX_ID>]
    release: <CP_RELEASE>
nodePools:
  - name: worker
    desiredNodeCount: 2
    controlPlane: false
    nico:
      instanceTypeID: <WORKER_INSTANCE_TYPE_ID>
      vpcPrefixIDs: [<VPC_PREFIX_ID>]
      release: <WORKER_RELEASE>

nvkectl create cluster -f cluster.yaml          # returns at once; provisioning is async
nvkectl get clusters                            # wait for STATUS Provisioned
nvkectl get kubeconfig team-a --output-file team-a.kubeconfig
nvkectl patch nodepool worker --cluster team-a --patch 'desiredNodeCount: 8'
nvkectl patch cluster team-a ...                # control-plane Kubernetes upgrade

From there NVKE handles the lifecycle. Deleting a worker node returns the machine to NICo, and NVKE provisions a replacement on any free machine of the pool’s instance type. A control-plane upgrade can move one minor version at a time, and worker pools may trail by up to three minor versions. With NVSentinel installed, nodes it reports as faulty are rebooted or replaced through the same lifecycle.

Try itCan you run NVKE without GPUs?

Yes. The local quickstart runs the whole management plane on one Linux machine, with mock-NICo, a simulator that answers the NICo API and starts containers as “machines” with virtual TPMs. You need a Linux host or VM (amd64 or arm64) with 6 CPUs, 12 GiB free memory, 100 GiB free disk, Docker, and the tpm_vtpm_proxy kernel module. No NVIDIA account is required. macOS users run it inside a Colima VM.

git clone https://github.com/dsx-ai-factory/nvke.git && cd nvke
sudo sysctl -w fs.inotify.max_user_instances=1024
make -C controller host-up                 # loads the virtual TPM module
make -C cli build && export PATH="$PWD/cli/bin:$PATH"
# copy the pinned CAPNICo source and write controller/tilt_config.json
# and controller/.env.tilt as the quickstart guide shows, then:
make -C controller capd-up                 # kind management cluster + Cluster API
cd controller && tilt up                   # management plane, CAPNICo, mock-NICo

# second terminal, in the nvke directory
kubectl config view --raw --minify --context kind-controller-capi > local-mgmt.kubeconfig
nvkectl config set-site --name local --url http://localhost:8080 
  --kubeconfig "$PWD/local-mgmt.kubeconfig"
nvkectl get providers                      # nico row: CONFIGURED
nvkectl create cluster -f cli/examples/nico-mock-cluster.yaml
nvkectl get clusters                       # wait for Provisioned
nvkectl get kubeconfig my-cluster --output-file my-cluster.kubeconfig
kubectl --kubeconfig my-cluster.kubeconfig get nodes

You end with a two-node cluster (one control plane, one worker) built through the same API, workflows, CAPNICo and node agent as a real installation. It is the quickest way to see the outbox, Temporal and agent flow from Figure 2 working; the Temporal web UI shows each workflow step. Clean up with nvkectl delete cluster my-cluster before stopping Tilt.

Reality checkWhat is missing in the preview?

NVKE’s documentation is unusually honest about its limits. These are the ones that matter most before you plan anything serious:

The public roadmap shows what is next. Coming soon: per-customer API authorisation, managing the management cluster’s own nodes, cluster certificates from your own PKI, and Kubernetes 1.37. Working on it: GB300 and Vera Rubin worker nodes, separate control-plane and worker networks, deleting single nodes and automatic scale-down. Exploring: backup and restore, management-plane upgrades, and an NVKE MCP server so AI agents can operate clusters.

ComparedWhy not just use kubeadm or Cluster API?

NVKE uses kubeadm and Cluster API underneath. What it adds is the operational layer an AI factory would otherwise have to build itself: the service API, durable workflows, node identity, encryption key handling and NICo integration.

DecisionDo you need NVKE?

Quick answers

Can NVKE run without NICo?

For evaluation, yes, with the mock-NICo simulator. For real installations NICo is the default infrastructure provider; the docs also list LaunchLayer as supported once a deployment enables it.

Does NVKE install GPU drivers?

No. NVKE stops at a running cluster with OVN-Kubernetes networking. The GPU Operator, drivers and other runtime components come from an AICR recipe installed into each workload cluster.

Is it production ready?

No. It is a preview, and NVIDIA does not recommend it for production workloads. APIs and behaviour can change between releases.

How do customers get access?

The operator runs nvkectl get kubeconfig and hands over the admin kubeconfig. The customer’s administrators then create narrower credentials inside their cluster. Per-user credentials from NVKE are not available yet.

Which versions does the preview support?

Kubernetes 1.36 (pinned), Ubuntu 24.04 or 26.04 on amd64 or arm64, OVN-Kubernetes networking and Cluster API 1.13. GB300 and Vera Rubin worker support is on the roadmap.

Sources: NVKE repository and documentation (github.com/dsx-ai-factory/nvke), including What is NVKE, Architecture, NVKE in the DSX stack, Plan your installation, Bootstrap the management cluster, Quickstart: evaluate locally with mock-NICo, Manage node pools, Supported platforms and versions, Preview status and known limitations, and ROADMAP.md, read at commit 92c39b8 (6 November 2026). Version numbers and limitations reflect that preview and will change. The cluster YAML is shortened from the repository’s NICo example; IDs are placeholders.