A watsonx.ai model can be fluent, confident, and wrong in the same sentence. Last month a RAG answer at one of my customers cited a retirement date that was nowhere in the retrieved policy, and the app shipped it because nothing was watching the output. Granite Guardian is the thing you put in that gap. It is a small model whose only job is to read a prompt or a response and say yes or no against a named risk, and hallucination is one of the risks it was trained to catch.
Key takeaways
Granite Guardian is a judge model, not a chat model. You hand it the text plus a risk name, and it returns a yes/no plus a score. For hallucination it judges three separate things in a RAG answer: context relevance, groundedness, and answer relevance. Run it on the output first, because that is where a wrong answer actually reaches the user. Start with the 8B model, put the score threshold at 0.5, and move it only after you have seen real traffic.
What Granite Guardian is
Granite Guardian is a family of risk-detection models from IBM, part of the Granite line. A guardian model does one thing: it reads some text against a criterion you name and returns a judgement. It does not answer the user. It does not rewrite anything. It scores. You give it a conversation, a document, or a tool call, tell it which risk to look for, and it says whether that risk is present.
The models are open, released under Apache 2.0, and available both on Hugging Face and inside watsonx.ai as foundation models you can call like any other. The current release is Granite Guardian 4.1 8B, a hybrid model that can reason before it scores or answer instantly. Older 3.x versions (the 8B and a lighter 2B) are the ones most watsonx tenants have been calling by the model id ibm/granite-guardian-3-8b, and they remain fully supported. Whichever version you use, the shape is the same: risk in, verdict out.
Why a separate model at all? Because asking the same model that wrote an answer to also grade its own honesty is a conflict of interest. A dedicated judge, trained on annotated risk data, is cheaper to run, faster, and harder to talk out of a verdict than a prompt-engineered check bolted onto your main model. That separation is the whole point.
Where Guardian sits in the pipeline
Guardian has two natural slots: before the model, screening the incoming prompt, and after the model, checking the response. Most teams reach for the input check first because it feels like a firewall. In practice the output check earns its keep faster, because a hallucinated answer is the failure your users actually feel. I wire both, but I turn on the output groundedness check on day one and treat the input check as a second pass.
flowchart LR U[User prompt] --> G1[Guardian input check] G1 -->|risk| Block[Block or rewrite] G1 -->|clear| M[watsonx.ai model] M --> G2[Guardian output check] G2 -->|grounded| Out[Return answer] G2 -->|ungrounded| Retry[Flag or regenerate]
The retry arrow matters. When the output check flags an answer as ungrounded, you have a choice: block it, hand it to a human, or send the query back through generation with a stricter instruction. For a support bot I usually flag and fall back to a canned safe reply. For an internal research tool I let it regenerate once, then flag. There is no universal answer, only a cost you are willing to pay per bad response.
The risks it is trained to catch
Guardian ships with a set of pre-baked criteria. Think of them as named tests you can switch on without training anything. They fall into three buckets: general harm, RAG hallucination, and agentic tool use. There is also a bring-your-own-criteria mode, added in 4.1, where you write a plain-English rule and Guardian judges against it.
The harm bucket covers social bias, jailbreaking, violence, profanity, sexual content, and unethical behavior. The agentic bucket has one criterion, function-calling hallucination, which flags a tool call whose arguments do not match the tool definition or the user query. For this part the bucket that matters is RAG hallucination, and it is not one test but three.
| Criterion | What it flags | Where it runs |
|---|---|---|
| Context relevance | Retrieved passages that do not help answer the question | After retrieval, before generation |
| Groundedness | Claims in the answer not supported by the retrieved text | After generation |
| Answer relevance | An answer that does not address what the user asked | After generation |
| Function-calling hallucination | Tool calls with wrong names, types, or invented parameters | Inside an agent loop |
Groundedness, context relevance, answer relevance
These three catch different failures, and confusing them is the most common mistake I see. Context relevance is a retrieval problem: your vector search pulled passages that have nothing to do with the question. Groundedness is a generation problem: the passages were fine, but the model wrote something they do not support. Answer relevance is a helpfulness problem: the answer is true and grounded but sidesteps what was actually asked.
Groundedness is the one people mean when they say hallucination. It is defined tightly: a response is ungrounded if it includes claims not supported by, or directly contradicting, the provided documents. That definition is strict on purpose. A plausible extrapolation the documents never made still counts as ungrounded. For a bank or an insurer, that strictness is the feature, because a confident guess is worse than a hedge.
How a groundedness check runs
Under the hood, Guardian takes your conversation and appends a guardian block: a short instruction, the criterion you are testing, and a scoring schema that asks for a yes or no. For groundedness you pass the retrieved documents alongside the assistant message, and the criterion describes what unfaithful means. Guardian returns a verdict token and, in the versions that expose it, a probability behind that token. On watsonx you do not hand-write that block; the SDK builds it when you set the risk name.
One design choice worth stating plainly: Guardian judges the last assistant turn. It is not summarizing the whole chat. So feed it the exact answer you are about to return and the exact documents you retrieved for that answer, not the entire session history. Scope the input tightly and the verdict gets sharper.
Thinking mode or fast yes/no?
Guardian 4.1 is a hybrid model. In no-think mode it emits the verdict straight away, which is what you want on a hot path where every request is judged. In think mode it writes a short reasoning trace before scoring, wrapped in tags, which is worth the extra latency when a human will read why an answer was flagged. My rule: no-think in production, think mode when I am building the threshold or reviewing a batch of flagged answers offline.
Whatever mode you pick, the output collapses to a score between 0 and 1. IBM frames three bands: 0.0 to 0.3 is low risk and safe to pass, 0.4 to 0.7 is a nuanced middle you may want a human on, and 0.8 to 1.0 is high risk you should block. Where you put your cut line inside that range is the single most important knob you own.
Worked example
The retrieved document says a film was first shown on July 16, 1964. The model answers that it was first shown on December 24, 1922, a date that appears in the document but belongs to a different person. I send the answer plus that one document to Guardian with the groundedness criterion. Verdict: yes, ungrounded, score 0.92. With my cut line at 0.5 the answer is blocked and the user gets a fallback. Move the cut line to 0.85 and this still blocks, but a subtler fabrication scoring 0.6 would slip through. That is the trade you are tuning.
Score it: a Python detector on watsonx
Here is the output check as a function. It calls the guardian model on watsonx, passes the answer and the grounding document, and returns whether to block. Keep it next to your generation call so every answer is judged before it returns.
from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import ModelInference
guardian = ModelInference(
model_id='ibm/granite-guardian-3-8b',
credentials=Credentials(url='https://us-south.ml.cloud.ibm.com',
api_key=API_KEY),
project_id=PROJECT_ID,
)
def is_grounded(answer, document, cutoff=0.5):
# the system turn names the risk Guardian should judge
messages = [
{'role': 'system', 'content': 'groundedness'},
{'role': 'context', 'content': document},
{'role': 'assistant', 'content': answer},
]
result = guardian.chat(messages=messages)
verdict = result['choices'][0]['message']['content'].strip().lower()
return verdict == 'no' # 'no' groundedness risk means the answer is grounded
doc = 'The film Eat was first shown by Jonas Mekas on July 16, 1964.'
bad = 'Eat was first shown on December 24, 1922.'
print(is_grounded(bad, doc)) # -> False (Guardian flags it, score ~0.92)Expected output is False: Guardian returns yes to the groundedness risk, so the answer is not grounded and you block it. Failure modes to plan for: if ibm/granite-guardian-3-8b is not enabled in your region or plan, chat() raises a client error, so check the model list for your project first. And the 3.x guardian window is 8192 tokens, so a long document plus a long answer can overflow; chunk the document or judge per passage. I have marked the exact message roles as because the watsonx detector interface has shifted across releases; confirm against the model card for your instance before you ship.
Which model should run this check? The table below is how I choose. The 8B is the default. The 2B trades a little accuracy for lower cost and latency when you are judging every single request at volume. The tiny HAP model is a specialist for hate, abuse, and profanity only, useful as a cheap first filter, not a groundedness judge.
| Model | Size | Relative latency | Best for |
|---|---|---|---|
| granite-guardian-3-8b | 8B | Baseline | Default judge, harm and RAG hallucination |
| granite-guardian-3-2b | 2B | Lower | High-volume checks where cost matters |
| granite-guardian-4.1-8b | 8B | Baseline, hybrid | Custom criteria and think-mode review |
| granite-guardian-hap-38m | 38M | Lowest | Hate, abuse, profanity pre-filter only |
Where Guardian stops and governance begins
Guardian gives you a verdict on one answer at one moment. It does not track drift, it does not keep an audit trail, and it does not tell you whether your blocking rate is climbing week over week. That is a different job, and it belongs to watsonx.governance, which I cover in Part 19. The pattern I recommend: Guardian for the real-time decision, governance for the record and the monitoring on top of it. Wire the score Guardian returns into your evaluation pipeline too, which connects to the model-evaluation work in Part 17.
If you want the vendor-neutral picture of what guardrails catch and what they miss, my GenAI Series piece on guardrails and responsible AI sets the frame that this watsonx-specific part fills in. Guardian is one good implementation of that frame, not the whole of it.
Guard the output first, tune thresholds with real traffic
If you do one thing after reading this, add the groundedness check on your RAG output using granite-guardian-3-8b, run it in shadow mode, and log the score for every answer. A week of that log tells you more about where your cut line belongs than any blog post can, including this one. Then turn on blocking at the threshold your own data points to, usually somewhere between 0.5 and 0.7 for a first cut.
Input screening and function-call checks are worth adding next, but they are a second pass. The output groundedness check is the one that stops a wrong answer from reaching a person, and that is the failure worth spending on first. Start there, watch the log, move the line.
References
- Granite Guardian 4.1 model documentation, IBM Granite
- granite-guardian-3-8b model card, IBM watsonx Documentation
- What is Granite Guardian, IBM Think
- Granite Guardian, Padhi et al., arXiv 2412.07724
About The Author
DrJha