Tag: GPU
-
GPU Cost, Scale and Sizing Decisions for Machine Learning Workloads (Data Science Series, Part 28)
Nine percent average GPU utilisation on a cluster I inherited turned out to be the whole story. Here is how I size, buy and scale machine learning compute, with current instance rates and the arithmetic that decides each call.
-
FinOps for AI and GPU Spend, Tokens and Utilisation (Cloud FinOps Series, Part 17)
A GPU sitting at 30 percent utilisation costs more per useful hour than the provider you rejected for being expensive. Here is how AI spend actually meters, and which levers move it.
-
watsonx.ai Training Infrastructure and Scaling on OpenShift (IBM Gen AI Series, Part 14)
Every watsonx tune is a Kubernetes job under the hood. Here is how GPU nodes, the NVIDIA operators, autoscaling to zero, MIG limits, and Kueue quotas fit together on Red Hat OpenShift.
-
GPUs and Running watsonx.ai on OpenShift (IBM Gen AI Series, Part 6)
How watsonx.ai actually uses GPUs on Red Hat OpenShift: the operators that expose a card, why memory sizing decides everything, the single node rule for multi GPU models, and when MIG or time slicing is worth it.
-
Cloud TPUs vs GPUs on Google Cloud, and When Each Wins (Google Cloud Gen AI Series, Part 7)
When you train or self-host on Vertex AI, the accelerator choice lands on you. Here is how Cloud TPUs and GPUs really differ, and a plain rule for picking one.
-
Azure ND-Series GPUs and the Maia Accelerator (Azure Gen AI Series, Part 7)
Azure’s ND-series gives you NVIDIA H200, Blackwell GB200, and AMD MI300X GPUs by the node, while Maia is Microsoft’s own inference silicon you cannot rent. What each one is, how to size a GPU to your model, and the short list of times self-hosting beats the managed endpoint.
-
Hugging Face Air-Gapped: Enterprise Hub, Offline Mirroring, and the On-Prem Build (Hugging Face Series, Part 16)
How to run Hugging Face models on a segmented or air-gapped network: mirror the artifacts to local storage, force offline mode at runtime, and use the Enterprise Hub for identity, governance, and the rate limits a proxy needs.
-
Text Generation Inference (TGI) in Production: A Real Serving Example (Hugging Face Series, Part 12)
TGI turns a Hugging Face model into an OpenAI-compatible endpoint with one docker run. Here are the flags that decide whether it fits your VRAM, how to consume it, and an honest verdict now that TGI is in maintenance mode and Hugging Face points new builds at vLLM.
-
How to Run a Hugging Face Model: Inference Providers vs Endpoints vs Self-Host (Hugging Face Series, Part 11)
Three ways to serve a Hugging Face model: the serverless Inference Providers proxy, dedicated Inference Endpoints, or self-hosting TGI on your own GPUs. A build-vs-buy verdict for the infra team that owns the bill.
-
Accelerate: Multi-GPU Training Without Rewriting Your Code (Hugging Face Series, Part 9)
How Hugging Face accelerate turns a single-GPU training script into a multi-GPU job, when to use DDP versus FSDP versus DeepSpeed, and what the choice means for interconnect, capacity, and cost on the boxes you run.
-
Fine-Tuning with Trainer and LoRA/PEFT: When You Cannot Afford a Full Fine-Tune (Hugging Face Series, Part 8)
Full fine-tuning a 7B model can need two 80GB GPUs you do not have. Here is how the Trainer, LoRA, and QLoRA change the capacity math, with runnable code and the failure modes.
-
Running Your First Model with Transformers Pipelines (Hugging Face Series, Part 4)
The transformers pipeline is the shortest path from a model repo to a running inference call. Here is what actually loads onto your box, and how device, device_map and dtype decide whether a model fits your GPU.
Architect’s Toolkit
PJ’s Tools
- Infra 360 Hub – All Series
- VCF 9 Interactive Walkthroughs
- VCF Design Cheatsheet
- VCF Upgrade Planner
- VCF 9 Series Hub
- VCF Deployment Hub
- AI Stack Hub
- AI Infra Sizing & Cost Calculator
- LLM & RAG Cost Calculator
- DrJhaGPT – Ask Pranay
VMware Cloud Foundation
- VCF Documentation
- VCF 9 Planning & Preparation Workbook
- VCF Bill of Materials (BoM)
- VMware Compatibility Guide
- VMware Interoperability Matrix
- VMware Configuration Maximums
- VMware Ports & Protocols
- VMware Hands-on Labs
- RVTools Download
Nutanix
AI & Cloud-Native Platform
- NVIDIA Build (Model Catalog)
- NVIDIA AI Enterprise Reference Architecture
- NVIDIA NIM Performance Benchmarking
- NVIDIA NGC Catalog
- NeMo Microservices Helm Chart
- Helm Charts Repository
- Hugging Face Models
Architecture & Design
About the Author
Dr Pranay Jha
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
You May Have Missed
DrJha