Tag: vLLM
-
Cost and FinOps for Self Hosted GenAI on OpenShift AI (Red Hat Gen AI Series, Part 28)
Self hosting Granite rarely wins on unit cost until you reach billions of tokens a month. Here is how to price a self hosted token, measure real GPU utilisation on OpenShift AI, and find where a managed API stops being cheaper.
-
Benchmarking a Self Hosted Inference Deployment With GuideLLM (Red Hat Gen AI Series, Part 25)
A single hand run is not a benchmark. Here is how to measure a self hosted Granite deployment with GuideLLM and gate every model or config change on tail latency, not throughput, so no regression ships unmeasured.
-
Token Economics and Latency Tuning for Self Hosted Granite (Red Hat Gen AI Series, Part 24)
What one answer from a self hosted Granite model actually costs, and the three vLLM flags that decide it. A latency aware guide to throughput, TTFT and cost per token on the Red Hat AI Inference Server.
-
llm-d Distributed Inference on Kubernetes for Granite at Scale (Red Hat Gen AI Series, Part 23)
llm-d spreads vLLM inference across a Kubernetes cluster with prefill decode disaggregation and KV cache aware routing. When it pays off, and how to deploy it behind an inference gateway.
-
Model Compression and Quantization With LLM Compressor (Red Hat Gen AI Series, Part 22)
Quantizing Granite with LLM Compressor cuts the support assistant from 16 GB to under 5 GB of weights. A format by format comparison of FP8, INT8, INT4 and NVFP4, with the memory, accuracy recovery and hardware trade offs named.
-
vLLM Internals, PagedAttention and Continuous Batching Explained (Red Hat Gen AI Series, Part 21)
vLLM turns model serving from a compute problem into a paging problem. Here is how PagedAttention and continuous batching actually work inside the Red Hat AI Inference Server, and where they break under load.
-
Red Hat AI Inference Server, a Hardened vLLM Distribution (Red Hat Gen AI Series, Part 20)
Red Hat AI Inference Server is upstream vLLM packaged as a supported, hardened container. Here is what it adds, how to serve Granite behind it, and how to benchmark it honestly.
-
GPU Sharing on OpenShift AI With Time Slicing and MIG (Red Hat Gen AI Series, Part 18)
One 8B model on a whole A100 is a card billed at full price and used at a fraction. Here is how to split a GPU on OpenShift AI with time slicing and MIG, and which one to pick.
-
Serving Granite With KServe and Serving Runtimes on OpenShift AI (Red Hat Gen AI Series, Part 17)
Take the production Granite version from the registry and turn it into a live OpenAI compatible endpoint with KServe on OpenShift AI, then learn where serving runtimes, deployment modes and autoscaling actually break under real traffic.
-
Hardware and Accelerator Sizing for RHEL AI (Red Hat Gen AI Series, Part 12)
Sizing GPUs for RHEL AI is a KV cache problem, not a weights problem. A worked memory calculator, a per accelerator concurrency table for Granite 3.1 8B, and the one flag that wakes a stalled server.
-
Serving Granite Locally With ilab and vLLM (Red Hat Gen AI Series, Part 8)
Serving Granite on one RHEL AI box with ilab model serve and vLLM, from the first token to a locked down endpoint, including the tensor parallel error nearly everyone hits on day one.
-
Installing RHEL AI From the Bootable Image to First Boot (Red Hat Gen AI Series, Part 7)
RHEL AI ships as a whole bootable operating system, not a package you add to Linux. Here is how to stand up a first box: download the image, size the disks, write a safe Kickstart, log in to the registry and get Granite downloaded before you serve it.
Architect’s Toolkit
PJ’s Tools
- Infra 360 Hub – All Series
- VCF 9 Interactive Walkthroughs
- VCF Design Cheatsheet
- VCF Upgrade Planner
- VCF 9 Series Hub
- VCF Deployment Hub
- AI Stack Hub
- AI Infra Sizing & Cost Calculator
- LLM & RAG Cost Calculator
- DrJhaGPT – Ask Pranay
VMware Cloud Foundation
- VCF Documentation
- VCF 9 Planning & Preparation Workbook
- VCF Bill of Materials (BoM)
- VMware Compatibility Guide
- VMware Interoperability Matrix
- VMware Configuration Maximums
- VMware Ports & Protocols
- VMware Hands-on Labs
- RVTools Download
Nutanix
AI & Cloud-Native Platform
- NVIDIA Build (Model Catalog)
- NVIDIA AI Enterprise Reference Architecture
- NVIDIA NIM Performance Benchmarking
- NVIDIA NGC Catalog
- NeMo Microservices Helm Chart
- Helm Charts Repository
- Hugging Face Models
Architecture & Design
About the Author
Dr Pranay Jha
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
You May Have Missed
DrJha