Category: AI/ML
-
Automated Evaluation With LLM as Judge, and Where It Lies to You (AI Engineering Series, Part 21)
An LLM judge that agrees with your humans 71 percent of the time can still be worthless, because a judge that passes everything scores 60. Here is how I calibrate one with Cohen kappa, and the four biases that make it lie.
-
Building an Eval Set Before You Build the Feature (AI Engineering Series, Part 20)
An eval set written before the feature is the only thing that tells you whether a prompt change helped. Here is the case schema, the coverage measurement, and a pytest runner, with the real failures they produce.
-
Human in the Loop and Approval Gates for LLM Agents (AI Engineering Series, Part 19)
Pausing an agent for approval is the easy half. This part builds an approval gate in the OpenAI Agents SDK and in LangGraph, then explains why the gate has to be enforced at the service that moves the money rather than inside the agent process.
-
Multi Agent Patterns and Their Coordination Cost (AI Engineering Series, Part 18)
Subagents, handoffs, router and skills compared on measured token and model call counts, plus the coordination ratio I use to decide whether a split is worth paying for. Includes the handoff bug that returns a 400 and how to avoid it.
-
Multi Step Workflows, State and Memory in LLM Applications (AI Engineering Series, Part 17)
A checkpointer is a crash recovery log, not memory, and confusing the two is how a support assistant ends up carrying 47,000 tokens of unrelated history into every request. Threads, checkpoints, stores and compaction, with runnable LangGraph code.
-
Model Context Protocol and Standardising Tool Access, With a Working Python Server (AI Engineering Series, Part 16)
Part 14 gave the support assistant a working tool loop. This part turns those hand wired functions into an MCP server any client can discover, with the transport choice, the failure modes and the context cost measured rather than assumed.
-
Agent Loops and Planning, and When an Agent Is the Wrong Answer (AI Engineering Series, Part 15)
An agent loop is a cost decision before it is an architecture decision. This part covers how the loop works, why per step accuracy compounds against you, the four workflow patterns that beat an agent in production, and the decision table I use to choose between them.
-
Tool Calling From First Principles, With Python and a Real Loop (AI Engineering Series, Part 14)
Tool calling is a protocol, not a capability the model gains. Here is the full loop in Python, the schema decisions that cut wrong calls, and the two errors every tool calling app hits in its first week.
-
Retrieval Evaluation for RAG, Separated From Answer Evaluation (AI Engineering Series, Part 13)
Most RAG teams report one number and tune the wrong component. Here is how I split retrieval evaluation from answer evaluation on a 120 query set, with ranx, real metric output and the failure attribution table that came out of it.
-
Hybrid Search and Reranking for RAG Retrieval (AI Engineering Series, Part 12)
Dense retrieval cannot find an error code. This part adds BM25, fuses the two ranked lists with reciprocal rank fusion, and reranks the survivors with a cross encoder, with measured nDCG and latency for each step.
-
Vector Stores in Practice: pgvector, Chroma and Qdrant Compared (AI Engineering Series, Part 11)
pgvector, Chroma and Qdrant measured on the same 50,000 chunk corpus: build time, p95 latency, resident memory and the filtered recall collapse that default settings will hand you. With the configuration fixes that actually work.
-
Choosing an Embedding Model: Benchmarking, Dimensions and Cost (AI Engineering Series, Part 10)
Embedding model choice sets your recall ceiling and, far more expensively, the memory your vector index needs forever. Here is the sizing arithmetic, a recall benchmark run on a real corpus, and why 3072 dimensions is rarely the right default.
Architect’s Toolkit
PJ’s Tools
VMware Cloud Foundation
- VCF Documentation
- VCF 9 Planning & Preparation Workbook
- VCF Bill of Materials (BoM)
- VMware Compatibility Guide
- VMware Interoperability Matrix
- VMware Configuration Maximums
- VMware Ports & Protocols
- VMware Hands-on Labs
- RVTools Download
Nutanix
AI & Cloud-Native Platform
- NVIDIA Build (Model Catalog)
- NVIDIA AI Enterprise Reference Architecture
- NVIDIA NIM Performance Benchmarking
- NVIDIA NGC Catalog
- NeMo Microservices Helm Chart
- Helm Charts Repository
- Hugging Face Models
Architecture & Design
About the Author
Dr Pranay Jha
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
You May Have Missed
