You followed the deployment guide, the deep learning VM provisioned, and then… the GPU is invisible, the VM refuses to power on, or inference runs at a crawl. Nine times out of ten the model and the framework are fine — the break is in the GPU configuration on the ESX host underneath Private AI. This post walks through the five host-side failure modes I see most often on VMware Private AI Foundation (PAIF) in VCF 9.1, what each one looks like, and the exact commands to confirm and fix it.

If you are still deciding whether Private AI is the right platform, start with the Private AI Foundation FAQ and the broader picture in how PAIF with NVIDIA is changing enterprise AI.

1. Host vGPU driver does not match the guest driver branch

Symptom: Inside the deep learning VM, nvidia-smi returns Failed to initialize NVML: Driver/library version mismatch, or the GPU never appears at all. This is the number-one breakage in VCF 9.1, because the platform now ships NVIDIA GPU Operator 25.10.1, which installs the v580.x guest driver. If the ESX host is still running an older NVD-AIE host VIB, the host and guest branches no longer line up.

Fix: Check the installed host driver and align it with the guest branch before you provision anything.

# On the ESX host: list the installed NVIDIA vGPU host VIB
esxcli software vib list | grep -i NVD

# Confirm the GPU and the running host driver version
nvidia-smi

# Inside the deep learning VM, the guest driver branch must match the host
nvidia-smi --query-gpu=driver_version --format=csv

Treat the host VIB, the guest driver, and the GPU Operator version as one matched set. When you upgrade Private AI Services, re-check the host VIB — GPU Operator 25.10.1 expects the v580.x branch and will not tolerate a stale host driver.

2. GPU left on the default graphics type instead of Shared Direct

Symptom: The vGPU VM fails to boot, or Private AI Services and VKS GPU clusters simply will not schedule onto the host. The cause is almost always that the GPU device is still set to the default Shared (vSGA) graphics type rather than Shared Direct, which is what vGPU requires.

Fix: In vCenter, place the host in maintenance mode, open Configure → Hardware → Graphics, edit each GPU device, set Device Type to Shared Direct, then restart the host. For Ampere and newer GPUs, make sure SR-IOV is enabled in the server BIOS first, or the profiles will not appear. Verify after reboot:

# Graphics type should report sharedPassthru (Shared Direct)
esxcli graphics device list

# Confirm SR-IOV virtual functions are present for Ampere/Hopper GPUs
lspci | grep -i nvidia

3. Large-framebuffer VMs that will not power on

Symptom: A small vGPU profile boots fine, but the moment you assign a large profile — a 48 GB or 80 GB C-type for an LLM — the VM throws Module DevicePowerOn power on failed or an MMIO / insufficient memory error. Big GPUs need a large PCI memory-mapped I/O window that the VM does not get by default.

Fix: Add the 64-bit MMIO advanced settings to the VM and reserve all guest memory. Size 64bitMMIOSizeGB to the next power of two above the total GPU framebuffer assigned to the VM.

# VM Options -> Advanced -> Edit Configuration, add:
pciPassthru.use64bitMMIO    = TRUE
pciPassthru.64bitMMIOSizeGB = 128   # next power of two above total framebuffer

# Also set: Reserve all guest memory (All locked)

The full-memory reservation is mandatory for passthrough and large vGPU profiles — without it the hypervisor cannot pin the pages the GPU DMA engine needs. This is the same near-bare-metal data path that lets virtualized GPUs run at roughly 1% off bare-metal performance.

4. The driver is running unlicensed

Symptom: The GPU is visible and the VM boots, but after a grace period performance falls off a cliff. Running AI workloads on vGPU requires two licenses: the VMware Private AI Foundation with NVIDIA add-on, and an NVIDIA AI Enterprise (NVAIE) / vGPU license served from a License Server (NLS). Miss the client token and the guest driver runs unlicensed and gets throttled.

Fix: Place the NLS client configuration token in the guest and confirm the licensed state.

# Drop the token from your NLS instance into the guest, then restart the service
sudo cp client_configuration_token.tok /etc/nvidia/ClientConfigToken/
sudo systemctl restart nvidia-gridd

# Expected: License status: Licensed
nvidia-smi -q | grep -i license

5. Mixed MIG and time-sliced profiles on the same GPU

Symptom: A profile request fails, or you cannot place the workload you expected on a GPU that clearly has capacity. A single physical GPU cannot serve both MIG-backed and time-sliced vGPUs at once — you choose one mode per device. From vGPU 19.0 on vSphere 9.0, MIG-backed vGPU is supported, but you have to enable MIG explicitly and you cannot mix it with time-sliced C-type profiles on the same card.

Fix: Decide per GPU. For hard isolation between tenants, enable MIG; for flexible oversubscription, leave the GPU time-sliced. Switch modes from the host:

# List GPUs and current MIG state
nvidia-smi

# Enable MIG on a specific GPU (then reboot the host)
nvidia-smi -i 0 -mig 1

# Turn MIG off again to return to time-sliced vGPU
nvidia-smi -i 0 -mig 0
nvidia-smi -i 0

If you are switching a host all the way from MIG to plain GPU passthrough, turn MIG off for every GPU and uninstall the host VIB first, for example esxcli software vib remove -n NVD-AIE_ESXi_9.1.0_Driver, then reboot before reconfiguring.

Final Thoughts

None of these five failure modes are exotic. They are the boring, host-side details — driver branch, graphics type, MMIO window, licensing, and GPU sharing mode — that decide whether Private AI Foundation runs at all. Work them top to bottom: match the drivers, set Shared Direct, size the MMIO window and reserve memory, license the guest, then pick MIG or time-sliced deliberately. Get those right on one host, verify with nvidia-smi, and the rest of the cluster becomes a copy-paste.

References

• Broadcom TechDocs — Configure NVIDIA vGPU or GPU Passthrough on the ESX Hosts
• Broadcom TechDocs — Requirements for Deploying VMware Private AI Foundation with NVIDIA
• NVIDIA Docs — VMware vSphere Installation and vGPU Configuration

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts