Tag: Model Evaluation
-
Model Evaluation in Python: Metrics, Cross Validation and Data Leakage (Data Science Series, Part 11)
A single train test split moved our churn AUC by nine points depending on the seed. Here is how to pick a metric that matches the decision, read a cross validation spread honestly, and catch the leakage that manufactures scores you should not believe.
-
watsonx Model Evaluation and Benchmarking (IBM Gen AI Series, Part 17)
How to evaluate and benchmark models and prompts on IBM watsonx using watsonx.governance, Evaluation Studio, and the ibm-watsonx-gov SDK. Score faithfulness, answer relevance, and context relevance on your own golden set, not a public leaderboard.
-
Azure AI Foundry Evaluation and Observability, from CI Gate to Live Traffic (Azure Gen AI Series, Part 23)
Evaluation scores catch a bad agent; tracing tells you why it went bad. Here is how Microsoft Foundry runs the same evaluators at dev time, in your CI gate, and against live traffic, and what continuous evaluation actually costs.
-
Amazon Bedrock Model Evaluation, Automatic to LLM-as-a-Judge (AWS Gen AI Series, Part 23)
Amazon Bedrock has three ways to score a model: programmatic metrics, an LLM judge, and human review. Here is what each measures, what it costs, and how to run a judge job without fooling yourself.
Architect’s Toolkit
PJ’s Tools
VMware Cloud Foundation
- VCF Documentation
- VCF 9 Planning & Preparation Workbook
- VCF Bill of Materials (BoM)
- VMware Compatibility Guide
- VMware Interoperability Matrix
- VMware Configuration Maximums
- VMware Ports & Protocols
- VMware Hands-on Labs
- RVTools Download
Nutanix
AI & Cloud-Native Platform
- NVIDIA Build (Model Catalog)
- NVIDIA AI Enterprise Reference Architecture
- NVIDIA NIM Performance Benchmarking
- NVIDIA NGC Catalog
- NeMo Microservices Helm Chart
- Helm Charts Repository
- Hugging Face Models
Architecture & Design
About the Author
Dr Pranay Jha
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
You May Have Missed
DrJha