Category: AI Stack

The infrastructure view of NVIDIA Auto Ontology: what the Helm chart deploys, which defaults to change before go-live, where the security boundary sits, hosted vs self-hosted NIM models and GPU sizing, protecting the Postgres that holds certified definitions, and how it fits into the AI factory stack.

Continue Reading

NVIDIA Auto Ontology gives AI agents a certified map of what enterprise data means, so text-to-SQL answers use the right columns and joins and come back with their SQL. What the ontology holds, how it is generated and certified, how a question is answered, and how to connect agents over MCP.

Continue Reading

NVIDIA OpenShell runs each AI agent in a sandbox and checks every file access, system call and network connection against a policy the agent cannot change. How the gateway, supervisor and sandbox work, how credentials stay hidden, how the policy advisor and prover work, and how it runs on Kubernetes.

Continue Reading

NVIDIA AI Cluster Runtime (AICR) turns known-good GPU driver, operator and kernel combinations into signed, version-locked recipes. How recipes are built from overlays, how to generate, deploy and validate one, what the validation phases run, and how AICR deploys NVIDIA Dynamo.

Continue Reading

NVIDIA Kubernetes Engine (NVKE) is an open-source management plane for running Kubernetes as a service on NVIDIA GPU bare metal. How it builds clusters through NICo, how nodes prove their identity with TPMs, how to try it without GPUs, and what the preview still lacks.

Continue Reading

NVIDIA has open-sourced four projects that together form an AI factory stack: NVKE for GPU Kubernetes clusters, AICR for validated runtime recipes, OpenShell for governing AI agents, and Auto Ontology for trusted data context. What each does, how mature it is, and where to start.

Continue Reading

A five-step method to size GPUs for serving an LLM: weights, KV cache per user, GPUs per copy, users per copy from memory and speed limits, and copies plus spares. With a worked Llama 3.1 70B example, a table of common setups and what the method leaves out.

Continue Reading

How LoRA customises a model by training a small adapter next to frozen weights: when fine-tuning helps, how low-rank adapters work, GPU memory for full fine-tuning vs LoRA vs QLoRA, how to train an adapter, and serving many adapters on one base model with vLLM.

Continue Reading

How quantization shrinks LLM weights from 16-bit to 8-bit and 4-bit: the formula, why it speeds up decode, FP8, INT8, INT4, NVFP4 and MXFP4 explained, how much quality you lose, how to quantize a model and how to choose a format for your GPUs.

Continue Reading

vLLM explained for infrastructure teams: continuous batching, PagedAttention and the KV cache, prefix caching and other speed-ups, the settings that matter, and how llm-d adds cache-aware routing and split prefill and decode on top of vLLM.

Continue Reading

What turns a folder of model weights into an API: what is in a model file, what a serving engine does (API, tokenizer, scheduler, GPU workers, streaming), the main engines, what happens when a server starts, and what else production serving needs.

Continue Reading

Training builds a model once; inference runs it every day. How the two differ in memory (16 bytes vs 2 bytes per parameter), compute, network and operations, when serving overtakes training, and what production inference needs.

Continue Reading