Here is a metric summary from a real evaluation run: Helpfulness 0.91, Correctness 0.88, Harmfulness 0.02. It looks like a pass. On its own it is close to meaningless. I cannot tell you if 0.91 is good until I know which model judged it, on how many prompts, and whether those prompts look anything like production traffic. A score with no method behind it is decoration you can put in a slide.

Amazon Bedrock gives you three ways to produce those numbers, and they cost wildly different amounts of money and time. This part walks through what Bedrock Evaluations measures, which metrics are worth trusting, how to grade a RAG pipeline instead of a bare model, and how to run a judge job from the command line without lying to yourself about the result.

What Bedrock Evaluations covers

Open the Bedrock console, find the Inference and Assessment section, and choose Evaluations. You get two tabs. Model evaluation grades a foundation model on a dataset of prompts. Knowledge Base evaluation grades a full retrieval augmented generation pipeline, which is a system that fetches your documents and then answers from them. They sound similar and they are not interchangeable. Model evaluation tells you whether the model can write a good answer when you hand it the context. RAG evaluation tells you whether your pipeline fetched the right context in the first place, which is where most production RAG systems actually fail.

Every evaluation is a job. You give it a dataset in JSONL, one prompt per line, optionally with a reference answer that acts as ground truth. You pick the model under test, the metrics you care about, and an S3 location for results. Bedrock runs the model over the prompts, scores the answers, and writes a report you can read in the console or pull from S3. Scores are normalised from 0 to 1 so a helpfulness of 0.9 and a correctness of 0.9 sit on the same scale.

One feature changes how far this reaches. You can bring your own inference responses. Instead of letting Bedrock call the model, you supply the answers already generated, from any model, on any platform, even one running outside AWS, and Bedrock only does the scoring. That means the judge and the metrics are usable as a grading service for a system Bedrock never touches. The map below is what I draw before a first evaluation to keep the two paths straight.

Three ways to score a model

Programmatic evaluation is the oldest and the cheapest. You give Bedrock a labelled dataset, and it computes exact metrics: accuracy against a known answer, robustness to small input changes, and toxicity from a classifier. These are deterministic and fast, and they are the right tool when your task has a correct answer you can check, like classification or extraction. They are close to useless for open-ended writing, where there is no single right string and an exact-match score punishes a good answer for using different words.

Model as a judge fills that gap. An evaluator model, Anthropic Claude 3.5 Sonnet is a common choice, reads each answer and scores it on quality metrics like correctness and helpfulness, returning a number from 0 to 1 with a written explanation for every score. This is the method that made evaluation practical for chat and summarisation, because it grades meaning rather than string overlap. The judge is not perfect, models tend to be strict and can carry their own bias, but it is consistent across thousands of prompts in a way a tired human reviewer is not.

Human review is the slow, expensive backstop. You define the metrics your reviewers score, friendliness, brand voice, whatever matters, and either your own team rates each answer in a Bedrock user interface, or you hire an AWS managed work team. Humans catch what a judge misses: subtle tone problems, domain errors a general model cannot see, edge cases that matter to your business. You do not run them at scale. You run them on the disputes and the samples that decide a launch. Here is how the three line up.

Which metrics actually mean something

Bedrock ships 11 built-in judge metrics, and the temptation is to switch them all on. Do not. Every metric you add is another judge inference per prompt, so more metrics means a bigger bill and a slower job, and half of them will not move for your task. Choose the three or four that map to how your application actually fails.

The distinction people trip on is correctness against faithfulness. Correctness asks whether the answer is right, judged against a reference answer if you supply one. Faithfulness asks whether the answer stuck to the information it was given and did not invent anything, regardless of whether the invented part happens to be true. For a RAG system, faithfulness is the one that catches hallucination, because a model can produce a factually correct sentence that appears nowhere in your retrieved documents, and for a grounded assistant that is still a failure. The responsible AI metrics, harmfulness, stereotyping, and refusal, are the ones I always keep on even when they read near zero, because the day one of them spikes is the day you needed it. These are the built-in metrics worth knowing by name.

What the numbers look like matters more than any single average. When I compare two candidate generators, I plot the per-metric scores side by side, because an average hides the trade-off that decides the call. A cheaper model that ties on correctness but drops on faithfulness is not cheaper, it is a hallucination risk you are choosing to buy. The chart below is an illustrative comparison of two Amazon Nova sizes on the same prompt set.

RAG evaluation on Knowledge Bases

When you grade a RAG pipeline, the model is only half of what you are testing. The other half is retrieval, and a bad answer is often a retrieval failure wearing a generation costume. Bedrock Knowledge Base evaluation lets you score two things separately: retrieval only, which asks whether the chunks pulled from your data were the right ones, and retrieve and generate, which grades the final answer built on those chunks. Testing retrieval alone is the faster loop when you are tuning chunk size, parsing, or the embedding model from Part 20, because you can compare configurations without paying for a generator pass each time.

The RAG-specific metrics are where this earns its place. Context relevance asks whether the retrieved passages were on topic. Context coverage asks whether they contained enough to answer. Citation precision and citation coverage, both added in 2025, check whether the answer cited the right sources and cited enough of them. Faithfulness ties it together by flagging any sentence the retrieval did not support. My habit: tune retrieval to a good context relevance and coverage first, and only then judge the generated answers. Chasing generation quality on top of weak retrieval is polishing the wrong surface.

Running a judge job from the CLI

The console is fine for a first run, but you want this in a pipeline so it runs on every model change. The API call is create-evaluation-job on the bedrock control plane client, and the cleanest way to drive it is a JSON job file passed with the CLI. Here is a trimmed model-as-a-judge job that evaluates Amazon Nova Lite with Claude 3.5 Sonnet as the judge, plus the output you should see and the two ways it usually breaks.

# Kick off an LLM-as-a-judge evaluation job
aws bedrock create-evaluation-job 
  --job-name 'nova-lite-quality-jul2026' 
  --role-arn 'arn:aws:iam::111122223333:role/BedrockEvalRole' 
  --cli-input-json file://judge-job.json

# judge-job.json (trimmed; exact field nesting varies by API version [VERIFY])
{
  "evaluationConfig": {
    "automated": {
      "datasetMetricConfigs": [
        {
          "taskType": "General",
          "dataset": { "name": "prompts", "datasetLocation": { "s3Uri": "s3://my-evals/prompts.jsonl" } },
          "metricNames": ["Builtin.Correctness", "Builtin.Faithfulness", "Builtin.Harmfulness"]
        }
      ],
      "evaluatorModelConfig": {
        "bedrockEvaluatorModels": [ { "modelIdentifier": "anthropic.claude-3-5-sonnet-20240620-v1:0" } ]
      }
    }
  },
  "inferenceConfig": {
    "models": [ { "bedrockModel": { "modelIdentifier": "amazon.nova-lite-v1:0" } } ]
  },
  "outputDataConfig": { "s3Uri": "s3://my-evals/results/" }
}

# Expected output
{
    "jobArn": "arn:aws:bedrock:us-east-1:111122223333:evaluation-job/abc123def456"
}

Expected result: the call returns a jobArn, the job moves from InProgress to Completed in the console, and a report lands in your S3 output prefix with per-metric averages and per-prompt scores. Failure mode one: an AccessDeniedException when the role cannot invoke the judge model or read the dataset bucket, so confirm the role can call both the generator and evaluator models and read and write those S3 paths. Failure mode two: a ValidationException when the evaluator model you named is not available for judge use in your Region, since evaluator availability differs by Region. Verify the exact JSON field names and the evaluator model id against the current bedrock API before you wire this into a pipeline. [VERIFY create-evaluation-job schema for judge metrics]

What evaluation costs, and where it hides

There is no line item for the evaluation feature. Bedrock does not charge you for running a job, for the scoring, or for the report. You pay for inference: the model under test generating an answer for every prompt, and the judge model reading and scoring every answer on every metric you selected. That second part is the one that surprises people. Three metrics on 500 prompts is 1,500 judge inferences on top of the 500 generation calls, and the judge prompt template, which Bedrock publishes in the docs, counts as input tokens on each of those.

Human evaluation is the expensive path and it is priced plainly. With your own team, each completed task, one reviewer scoring one prompt and its response, costs 0.21 dollars, on top of the inference to generate the answers. An AWS managed work team is priced by your dataset, task types, and metrics. The gap between a judge and a human panel is not a few percent. It is a factor, in both money and calendar time.

Fifty prompts, a judge model, humans on the tail

Here is the call I give teams. Build a versioned evaluation set from real user prompts, start at 50 and grow it toward a few hundred, and write reference answers once. Grade every model change with a model-as-a-judge job on three or four metrics that match how your application fails, and always keep faithfulness and the responsible AI metrics switched on. Use programmatic scoring only where the task has a genuinely correct answer, and route the small set of flagged or near-tie prompts to a short human pass instead of grading everything by hand. That gives you a number you can defend, cheaply, on every release.

For RAG, grade retrieval before generation, and treat context relevance and coverage as the first thing to fix. The same judge-first discipline shows up across clouds; if you run workloads on more than one, the Azure evaluation story in the Azure Gen AI series uses the same idea with different plumbing, and the vendor-neutral view of what guardrails catch is in the GenAI series guardrails part. Next in this series I move from grading answers to grading inputs and outputs at scale, with Bedrock Data Automation and multimodal content. This week, take 50 of your real production prompts, write reference answers, and run one judge job; the metric deltas will tell you more than any vendor benchmark. The regressions that hurt are the ones aggregate scores hide. Overall accuracy holds steady after a model swap while faithfulness on retrieval answers quietly drops, because the new model is more willing to fill gaps from its own weights. A per-metric judge catches that, a single average never will.

References

• Amazon Bedrock Evaluations product page
• New RAG evaluation and LLM-as-a-judge capabilities in Amazon Bedrock
• Built-in metrics for model evaluation, Amazon Bedrock User Guide
• Amazon Bedrock model evaluation is now generally available

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts