Author: Dr. Pranay Jha
-
Multi-Node LLM Training: Scheduling, Checkpointing and Fault Tolerance (NVIDIA AI Series, Part 25)
At thousands of GPUs, failures are routine. This part covers gang scheduling (Slurm vs Kubernetes vs NVIDIA Run:ai), async distributed checkpointing with NeMo, and the NVIDIA Resiliency Extension stack for fault tolerance, straggler detection, and elastic restart.
-
Inference Economics: Throughput, Latency, Batching and Cost Per Token (NVIDIA AI Series, Part 21)
TTFT, ITL, continuous batching, KV cache pressure, FP8 quantization — this is how you compute and actually drive down $/1M tokens on NVIDIA H100, H200, and B200 GPUs without breaking your latency SLOs.
-
NeMo Customization: LoRA, SFT, and RLHF on NVIDIA NeMo (NVIDIA AI Series, Part 23)
A practical decision guide for AI infrastructure architects on the full NeMo customization spectrum: when to use LoRA, full SFT, DPO, or GRPO, what data and GPU budget each method needs, and how the NeMo Customizer microservice ties it all together.
-
Data Preparation at Scale with NeMo Curator (NVIDIA AI Series, Part 24)
NeMo Curator is NVIDIA’s GPU-accelerated data curation toolkit that runs exact dedup, fuzzy dedup, semantic dedup, heuristic filtering, classifier-based quality filters, and PII redaction at trillion-token scale using RAPIDS cuDF and Dask. Learn why investing in data curation beats buying more GPUs.
-
Triton Inference Server vs NIM: When to Use Which (NVIDIA AI Series, Part 19)
Triton Inference Server and NVIDIA NIM solve different problems. This guide breaks down when to use each — and when to run both — covering backends, ensembles, dynamic batching, and the NIM packaged-microservice approach for LLM serving.
-
NVIDIA Dynamo Disaggregated Inference: Prefill, Decode, and KV-Aware Routing at Scale (NVIDIA AI Series, Part 20)
NVIDIA Dynamo separates prefill and decode onto independent GPU pools, routing requests via a KV-aware smart router and transferring KV cache blocks via NIXL. Here is when disaggregation wins, when it does not, and what to validate before committing to the architecture.
-
Deploying and Autoscaling NIM in Production on Kubernetes (NVIDIA AI Series, Part 17)
How to deploy NVIDIA NIM in production using the NIM Operator and Helm, wire autoscaling on the right GPU and KV-cache signals instead of CPU, handle cold-start model load, and run blue-green rollouts without dropping throughput.
-
TensorRT and TensorRT-LLM: Optimization, Quantization, and Engine Building (NVIDIA AI Series, Part 18)
What TensorRT does at build time versus what TensorRT-LLM adds at runtime — kernel fusion, paged KV cache, in-flight batching, and quantization choices from FP8 to NVFP4 — and when to hand-build engines instead of relying on a NIM.
-
NVIDIA NIM Inference Microservices: What a NIM Is and How It Serves a Model (NVIDIA AI Series, Part 16)
NVIDIA NIM packages a model, an optimized inference engine, and an OpenAI-compatible API into a single container. Pull it, pass your NGC API key, and you have a production inference endpoint on your own GPU infrastructure in minutes.
-
NVIDIA Network Operator on Kubernetes: RDMA, SR-IOV, and the Accelerated Fabric (NVIDIA AI Series, Part 13)
The NVIDIA Network Operator provisions MOFED drivers, RDMA shared device plugin, SR-IOV VFs, and Multus secondary networks to Kubernetes pods. This is how GPUDirect RDMA actually works at scale on ConnectX-7 and NDR InfiniBand clusters.
-
NVIDIA Drivers, CUDA, and the Container Toolkit: Building a Clean GPU Host Baseline (NVIDIA AI Series, Part 11)
The GPU host stack has three distinct layers: the data-center driver (open kernel module now required for Hopper and Blackwell), the CUDA Toolkit, and the NVIDIA Container Toolkit. Get the install order or versions wrong and containers fail silently. Here is the right sequence, the compatibility matrix, and the failure modes.
-
Air-Gapped Deployment, Lifecycle and CVE Patching for the NVIDIA Stack (NVIDIA AI Series, Part 15)
Running NVIDIA AI Enterprise in an air-gapped environment requires mirroring nvcr.io containers, Helm charts, and model weights before you cut the wire. Here is the branch selection, driver patch cadence, and CVE triage workflow that keeps regulated deployments defensible.
Architect’s Toolkit
PJ’s Tools
VMware Cloud Foundation
- VCF Documentation
- VCF 9 Planning & Preparation Workbook
- VCF Bill of Materials (BoM)
- VMware Compatibility Guide
- VMware Interoperability Matrix
- VMware Configuration Maximums
- VMware Ports & Protocols
- VMware Hands-on Labs
- RVTools Download
Nutanix
AI & Cloud-Native Platform
- NVIDIA Build (Model Catalog)
- NVIDIA AI Enterprise Reference Architecture
- NVIDIA NIM Performance Benchmarking
- NVIDIA NGC Catalog
- NeMo Microservices Helm Chart
- Helm Charts Repository
- Hugging Face Models
Architecture & Design
About the Author
Dr Pranay Jha
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
You May Have Missed
DrJha