Tag: distributed training
-
watsonx.ai Training Infrastructure and Scaling on OpenShift (IBM Gen AI Series, Part 14)
Every watsonx tune is a Kubernetes job under the hood. Here is how GPU nodes, the NVIDIA operators, autoscaling to zero, MIG limits, and Kueue quotas fit together on Red Hat OpenShift.
-
TPU Pods and Multislice Distributed Training on GKE (Google Cloud Gen AI Series, Part 19)
Where a single TPU slice stops fitting your model, Multislice takes over. How v6e Pods, ICI and the DCN, and GKE JobSets scale training from 16 chips to thousands.
-
Distributed Training on Azure ML with ND GPU Clusters (Azure Gen AI Series, Part 19)
How to spread one training run across Azure ND H100 clusters with PyTorch, NCCL, and InfiniBand, when adding nodes pays off, and how to keep scaling efficiency from collapsing.
-
Amazon SageMaker HyperPod for Resilient Model Training (AWS Gen AI Series, Part 19)
How Amazon SageMaker HyperPod runs resilient GPU and Trainium clusters for multi-week training: Slurm versus EKS, automatic node recovery, checkpointless training, task governance, and what the cluster actually costs.
-
What It Takes to Train a Model Across Thousands of GPUs (GenAI Series, Part 28)
Training a frontier model coordinates thousands of GPUs for months. How data, tensor, pipeline and expert parallelism, the memory math, and checkpointing make it possible.
Architect’s Toolkit
PJ’s Tools
VMware Cloud Foundation
- VCF Documentation
- VCF 9 Planning & Preparation Workbook
- VCF Bill of Materials (BoM)
- VMware Compatibility Guide
- VMware Interoperability Matrix
- VMware Configuration Maximums
- VMware Ports & Protocols
- VMware Hands-on Labs
- RVTools Download
Nutanix
AI & Cloud-Native Platform
- NVIDIA Build (Model Catalog)
- NVIDIA AI Enterprise Reference Architecture
- NVIDIA NIM Performance Benchmarking
- NVIDIA NGC Catalog
- NeMo Microservices Helm Chart
- Helm Charts Repository
- Hugging Face Models
Architecture & Design
About the Author
Dr Pranay Jha
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
You May Have Missed
DrJha