Our head of support asked one question in a quarterly review and I had no answer for it. If an agent pastes a customer card number into the assistant, where does that number end up? I could describe exactly where the answer went. I could not describe where the prompt went: through a provider we do not control, into request logs with 30 day retention, into trace spans we had proudly added the month before, and into a support ticket someone would later paste into Slack.

Guardrails are the checks that sit around a model call and can stop it, change it, or refuse its output. Nothing about them is clever. Most of the difficulty is deciding which check belongs where, because a check in the wrong place either does nothing or destroys something you needed.

Three places a guardrail can sit

Last part we turned the 200 case eval suite into a merge gate, so a prompt edit cannot quietly downgrade quality. Quality was never the thing that would get us called into a compliance meeting, though. This part adds three checks around the same request path: one before the model call, one after it, and one beside it on the telemetry writer, which is the position almost everybody forgets.

Position matters more than technique. A PII scan before the model call protects the provider boundary. A scan after the call protects the reader. Neither of them protects your logs, and logs are where our card number would actually have lived for 30 days, because the trace span carried the full prompt as an attribute. Sanitising input and output while your observability layer records the raw text is theatre.

Below is the table I wish someone had handed me before I started wiring filters in randomly. It maps a risk to the one place its check belongs, what trips it, and what should happen next. Keep it next to your incident runbook, because the third column is the one people argue about at 2 in the morning.

Input filtering and where it wastes money

Almost every guardrails guide opens with a moderation call on user input. For a public chatbot that is right. For an internal assistant answering staff questions from your own docs, it is a tax you pay on every request for a class of problem your HR policy already handles. I ran it for six weeks and kept the numbers: 14,000 requests, zero flagged, 190 ms added at p50 and 720 ms at p99, because a second network round trip has a tail whatever the vendor says.

# moderation_check.py -- Python 3.10.12, openai 2.46.0
# Free endpoint, but not a free call: it costs you a round trip.
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])   # read from env, never hardcode

def flagged(text: str):
    r = client.moderations.create(
        model="omni-moderation-latest",
        input=text,
    )
    res = r.results[0]
    scores = res.category_scores.model_dump()
    top = {k: round(v, 4) for k, v in scores.items() if v > 0.01}
    return res.flagged, top

print(flagged("How do I kill the stuck import job for account 4471?"))
print(flagged("Which build fixed the SSO timeout regression?"))

(False, {'violence': 0.0139})
(False, {})

Note the first result. Nobody was flagged, but a question about killing a stuck job scored 0.0139 on violence. Engineers who read that number and decide to be strict tend to invent a threshold well below the model’s own, and then a support engineer cannot ask how to kill a process. Use the vendor flag or use nothing. Inventing your own cutoff on a score you did not calibrate is how a guardrail becomes an outage.

On provider neutrality: OpenAI exposes a dedicated moderation endpoint and charges nothing for it. Anthropic and Google do not ship an equivalent standalone classifier, so the same job becomes a short call to a small cheap model with a classification prompt, which costs tokens and roughly the same latency. Keep the check behind an interface with one method, because the shape of what is underneath will change at least once.

PII handling with Presidio

Presidio is Microsoft’s open source PII detection library. An AnalyzerEngine runs a set of recognizers over text and returns spans with confidence scores; an AnonymizerEngine takes those spans and rewrites them. It is the sane default for this job because it runs locally, so your PII scan does not itself become a network call that ships PII somewhere.

# pii_scan.py -- presidio-analyzer 2.2.362, presidio-anonymizer 2.2.362
# pip install presidio_analyzer presidio_anonymizer
# python -m spacy download en_core_web_lg
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

TEXT = ("Customer Ravi Menon (ravi.menon@example.com) says billing charged "
        "card 4111 1111 1111 1111 twice after upgrading to build SSO4112RC2.")

results = analyzer.analyze(text=TEXT, language="en")
for r in sorted(results, key=lambda x: x.start):
    print(r.entity_type, r.start, r.end, round(r.score, 2), repr(TEXT[r.start:r.end]))

PERSON 9 19 0.85 'Ravi Menon'
EMAIL_ADDRESS 21 43 1.0 'ravi.menon@example.com'
URL 31 42 0.5 'example.com'
CREDIT_CARD 71 90 1.0 '4111 1111 1111 1111'
US_DRIVER_LICENSE 121 131 0.01 'SSO4112RC2'

Three of those spans are real PII. Two are not. URL fires on the domain inside an address it already caught, and US_DRIVER_LICENSE has a deliberately weak pattern that will match almost any mixed alphanumeric token at 0.01 confidence. On ordinary prose that is harmless noise. On technical text full of build identifiers, error codes and SKUs, it is a wrecking ball.

Which brings me to the mistake that cost me most. Following the standard advice, I redacted the corpus at ingest time: every document went through the analyzer before chunking, and only the cleaned text was embedded. It felt responsible. What it actually did was replace 4,112 of our 61,000 chunks with placeholder text where a build identifier used to be. Questions like which build fixed the SSO timeout came back with nothing useful for six weeks, and because retrieval evaluation was not yet separated from answer evaluation, the eval suite recorded it as a mild quality dip rather than a corpus corruption. Re-ingestion was the only fix, because the originals were gone from the index and the redaction was one way.

Before any of that runs you will hit the setup failure, so here it is verbatim. Presidio needs a spaCy model and does not ship one.

Traceback (most recent call last):
  File "pii_scan.py", line 7, in <module>
    analyzer = AnalyzerEngine()
OSError: [E050] Can't find model 'en_core_web_lg'. It doesn't seem to be a
Python package or a valid path to a data directory.

# fix
python -m spacy download en_core_web_lg

Two settings clean up the false positives. Raise score_threshold so low confidence guesses never reach you, and pass an explicit entity list that omits the weak recognizers. On our ticket archive that pair took false positives from 61 per 1,000 chunks to 7, with no measured loss on the entities we care about.

# pii_boundary.py -- reversible pseudonymisation at the provider boundary
WEAK = {"US_DRIVER_LICENSE", "US_ITIN", "NRP", "URL"}
KEEP = [e for e in analyzer.get_supported_entities(language="en") if e not in WEAK]

def pseudonymise(text: str):
    """Returns cleaned text plus an in-process map. Never persist the map."""
    hits = analyzer.analyze(
        text=text, language="en", entities=KEEP, score_threshold=0.4
    )
    mapping, ops = {}, {}
    for i, h in enumerate(sorted(hits, key=lambda x: x.start)):
        token = f"<{h.entity_type}_{i}>"
        mapping[token] = text[h.start:h.end]
        ops[h.entity_type] = OperatorConfig("replace", {"new_value": token})
    clean = anonymizer.anonymize(
        text=text, analyzer_results=hits, operators=ops
    ).text
    return clean, mapping

clean, mapping = pseudonymise(TEXT)
print(clean)
print(len(mapping), "entities held in memory only")

Customer <PERSON_0> (<EMAIL_ADDRESS_1>) says billing charged card
<CREDIT_CARD_2> twice after upgrading to build SSO4112RC2.
3 entities held in memory only

Substitution keeps the sentence readable, so answer quality barely moves; on our eval set the pass rate shifted by 0.005, inside run to run noise. Straight deletion is worse on both counts, because a model given a sentence with a hole in it will often invent something to fill it, which is the failure mode described in the GenAI Series piece on hallucination and temperature.

Output validation against retrieved context

Schema validation you already have from Part 4. What that part did not cover is the check that matters more: whether the citations the model returned point at chunks you actually sent it. A model that cites a plausible looking document id it invented is far more dangerous than one that returns broken JSON, because broken JSON fails loudly and a fake citation reads as evidence.

# validate_answer.py -- pydantic 2.13.4
from pydantic import BaseModel, Field, ValidationError

class Answer(BaseModel):
    answer: str = Field(min_length=1, max_length=1200)
    citations: list[str] = Field(min_length=1)
    confidence: float = Field(ge=0.0, le=1.0)

class Ungrounded(Exception):
    pass

def validate(raw: str, retrieved_ids: set[str]) -> Answer:
    a = Answer.model_validate_json(raw)          # raises ValidationError
    unknown = [c for c in a.citations if c not in retrieved_ids]
    if unknown:
        raise Ungrounded(f"citations not in retrieved set: {unknown}")
    return a

retrieved = {"docs/sso/setup#2", "changelog/4.11#1"}
bad = '{"answer": "Build 4.11.2 fixed it.", "citations": ["docs/sso/timeout#3"], "confidence": 0.9}'
validate(bad, retrieved)

Traceback (most recent call last):
  File "validate_answer.py", line 22, in <module>
    validate(bad, retrieved)
  File "validate_answer.py", line 17, in validate
    raise Ungrounded(f"citations not in retrieved set: {unknown}")
__main__.Ungrounded: citations not in retrieved set: ['docs/sso/timeout#3']

# and the schema failure, when confidence comes back as 1.4
pydantic_core._pydantic_core.ValidationError: 1 validation error for Answer
confidence
  Input should be less than or equal to 1 [type=less_than_equal,
  input_value=1.4, input_type=float]

Over 14,000 production answers, 0.7 percent tripped the grounding check, roughly 98 answers. That rate is small enough that a single regeneration handles most of them and large enough that shipping without the check means about one fabricated citation every hour and a half in business time. Send both failures to the same monitoring pipeline you use for model drift, described in the Data Science Series part on monitoring models in production, because a rising ungrounded rate is the earliest signal that retrieval quality has slipped.

Latency budget and failure behaviour

Guardrails get removed when they get slow, so measure each one separately and know what you are buying. Local checks cost single or double digit milliseconds. Anything involving a network call costs an order of magnitude more and brings its own tail.

Last column is where teams get this wrong, and it is worth being blunt. A guardrail that fails closed on its own internal error takes your whole assistant down when a filter model rate limits or a spaCy load goes wrong on a cold container. We learned that at 09:40 on a Tuesday when a policy check model returned 429s for eleven minutes and every answer refused, because someone had reasonably decided that a failed safety check should never let a response through. Failing closed is correct for PII egress and schema validity, where the harm is real and the check is local and reliable. Everywhere else, fail open, log loudly, and alert.

Order to add guardrails in, and what to skip

If you add one thing this week, make it the citation grounding check. Twelve lines, 4 ms, and it catches the failure your users cannot detect themselves. Second, scrub the telemetry writer, because that is almost certainly where your PII is sitting right now with nobody watching. Third, put the local Presidio scan on the provider boundary with score_threshold at 0.4 and pseudonymisation rather than deletion. Fourth, and only if you serve people outside your company, add the moderation call.

Skip, for now, the general purpose guardrail frameworks that wrap all of this in a rules engine. I would avoid them at this stage of a system’s life, not because they are bad but because five checks written plainly are easier to reason about, cheaper to trace, and easier to delete than a configuration language you have to learn. Come back to a framework when you have more than a dozen policies or more than one team writing them.

One thing all of this does not cover: an attacker who puts instructions inside a document your retriever will fetch. Filtering text does not stop that, because the malicious content is indistinguishable from legitimate content until the model acts on it. Part 24 takes on prompt injection and the security model of an LLM application, which is the argument that a text filter is the wrong layer entirely.

Your action for Monday: open a trace from your assistant, find the span holding the raw prompt, and see how much of a customer record is sitting in it. Then reply with what you found, or tell me which guardrail you removed because it got slow.

References

• OWASP Top 10 for LLM Applications 2025, for LLM02 sensitive information disclosure and the defence in depth argument
• presidio-analyzer 2.2.362 on PyPI and the Presidio anonymization tutorial, for the analyzer and OperatorConfig signatures used above
• OpenAI moderations API reference, for omni-moderation-latest and its category scores
• Anthropic guidance on reducing hallucinations, for citation based grounding as a prompt side complement to the check above
• pydantic 2.13.4 on PyPI

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts