Responsible AI on Azure is two separate toolchains, not one. The classic Responsible AI dashboard in Azure Machine Learning debugs tabular regression and classification models with fairness, error analysis, interpretability, counterfactuals and causal inference, and prints a PDF scorecard for compliance. For generative apps you use a different stack: the risk and safety evaluators in Azure AI Foundry (now branded Microsoft Foundry), Azure AI Content Safety at runtime, and the AI Red Teaming Agent before launch. Pick the toolchain by model type first, then wire the results into a release gate and keep the scorecard for your EU AI Act file.
The first time a team asked me to "run the Responsible AI dashboard on our chatbot," the pipeline failed before it produced a single chart. The dashboard components only accept a registered scikit-learn model in MLflow format, and a chat app calling a hosted GPT-4o deployment is neither. The error was correct. The request was not. That confusion is the single most common thing I fix in Azure responsible AI reviews, so it is where this part starts.
Two toolchains, and which one your workload lands in
Microsoft ships two responsible AI toolchains under one banner, and they do not overlap. The Responsible AI dashboard lives in Azure Machine Learning. It is built for predictive models you train yourself: loan scoring, churn, fraud, anything that takes a table of features and returns a class or a number. Its whole job is to help you find where that model is wrong, where it is unfair, and why. It knows nothing about prompts or tokens.
The generative side is a different set of tools. A large language model does not have a fixed feature table or a ground-truth label per row, so fairness metrics like demographic parity do not apply the same way. Instead you measure the content it produces. That is what the Foundry risk and safety evaluators do, and what Azure AI Content Safety enforces at runtime. Same principles of fairness, safety, privacy, transparency and accountability, two engines.
So the first decision is boring and mechanical. Is the thing under review a tabular model you trained, or a generative model you are calling? Get that right and you never point the wrong tool at the wrong workload again.
What the Responsible AI dashboard actually contains
The dashboard is not one tool. It is six open-source components stitched into one interface inside Azure Machine Learning studio, each one a mature project on its own. You turn on only the ones you need. The value is that they share cohorts, so a subgroup you flag in error analysis carries straight into fairness and interpretability without re-wrangling data by hand.
Microsoft frames the workflow in three stages: identify where the model fails, diagnose why, then mitigate. Here is what sits in each stage.
| Component | Open-source engine | What it answers |
|---|---|---|
| Error analysis | Error Analysis | Which cohorts fail most, shown as a tree map |
| Model overview and fairness | Fairlearn | How performance splits across sensitive groups |
| Data analysis | Responsible AI Toolbox | Over and under-representation in the dataset |
| Model interpretability | InterpretML | Global and per-prediction feature importance |
| Counterfactual what-if | DiCE | Smallest change that flips a prediction |
| Causal analysis | EconML | Real-world effect of changing a feature |
Fairness uses Fairlearn to compare metrics across groups you define by sensitive attributes such as age, sex or region.
Two of these earn their keep more than the rest in my experience. Error analysis is where you stop trusting a single aggregate accuracy number, because a model at 92 percent overall can be at 61 percent for applicants over 60, and the tree map shows you that in one screen. Counterfactual what-if is the one that helps a real person: it tells an applicant the smallest change that would have flipped the decision, which is exactly the explanation a regulator asks for.
Causal analysis is the component people misuse. It reads observational data for treatment effects, and if your data has confounders you do not control for, it will hand you a confident wrong answer. Use it to form hypotheses, not to prove them.
Generate the dashboard and the PDF scorecard
You build the dashboard three ways: the studio UI wizard, the Azure Machine Learning CLI v2, or the Python SDK v2. For anything that has to survive an audit, script it. A pipeline job assembles the RAI components you chose, points them at your registered model and a test dataset, and produces both the interactive dashboard and the scorecard as pipeline outputs. Because it is a job, it is versioned, repeatable and traceable, which is the whole point when a reviewer asks how a number was produced six months later.
The scorecard is the deliverable non-engineers actually read. It is a PDF that summarizes performance, fairness disparities and the cohorts you flagged, meant to be attached to a compliance file or handed to a product owner for a deployment sign-off. It is the artifact that turns a dashboard nobody outside the team opens into a document a risk officer will.
How do you measure safety on a generative app?
You score the outputs. The Foundry evaluation service runs a set of hosted models that read each response and rate it against a specific risk. These are the built-in risk and safety evaluators, and each one is a name you reference in an evaluation run. The content ones, violence, sexual, self-harm and hate_unfairness, score on a 0 to 7 severity scale. Give a threshold, default 3, and each response passes if its score is at or below the threshold and fails otherwise. The rest, like indirect_attack and code_vulnerability, return a plain pass or fail.
Across a dataset the service reports a defect rate, the share of responses flagged. That single number is what you gate a release on. Here is the evaluator set worth knowing before you write a run.
| Evaluator | Measures | Scope |
|---|---|---|
| builtin.violence | Violent or threatening language | model, agents |
| builtin.hate_unfairness | Hateful or unfair representation of groups | model, agents |
| builtin.self_harm | Self-harm related content | model, agents |
| builtin.indirect_attack | Indirect jailbreak, prompt injection in a source | model |
| builtin.code_vulnerability | Insecure generated code, injection, weak crypto | model, agents |
| builtin.prohibited_actions | Agent breaking a stated policy (preview) | agents |
Two things to know before your first run cost surprises you. The safety evaluators do not take a deployment_name, because they run against Microsoft-hosted safety models, not your own deployment. And under the hood the service provisions helper models to generate and annotate adversarial cases, so a red-team scan is not free. Budget it as a real workload, not a unit test you run on every commit.
The run itself is short. You define testing criteria, map your dataset fields to each evaluator, and submit. This is the operational core of the part, so here it is as code you can adapt.
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
project = AIProjectClient(
endpoint='https://<your-resource>.services.ai.azure.com/api/projects/<proj>',
credential=DefaultAzureCredential(),
)
testing_criteria = [
{
'type': 'azure_ai_evaluator',
'name': 'Violence',
'evaluator_name': 'builtin.violence',
'data_mapping': {'query': '{{item.query}}', 'response': '{{item.response}}'},
},
{
'type': 'azure_ai_evaluator',
'name': 'HateUnfairness',
'evaluator_name': 'builtin.hate_unfairness',
'data_mapping': {'query': '{{item.query}}', 'response': '{{item.response}}'},
},
]
# submit against a JSONL dataset of query/response rows, then read the summary
# each content evaluator returns score (0 to 7), label, threshold, passed
Expected output per row: a record like {score: 0, label: pass, threshold: 3, passed: true} with a reason string. Aggregated, the run reports a defect rate per evaluator.
Failure mode: leave out the query field in data_mapping and the run fails validation, because every content evaluator needs both query and response. Add a deployment_name here and it is ignored, since these are hosted evaluators.
Worked example
Take a support assistant scored over a sample set of 500 prompts. The first pass returns these defect rates: hate_unfairness 1.2 percent, violence 0.8 percent, self_harm 0.6 percent, sexual 0.4 percent. That is six hateful, four violent, three self-harm and two sexual responses out of 500. You add an Azure AI Content Safety guardrail on the output path and tighten the system prompt, then re-run the identical dataset.
Second pass: hate_unfairness 0.2 percent, violence 0.2 percent, sexual 0.0 percent, self_harm 0.0 percent. Hate defects drop from six to one, violence from four to one, and the other two categories reach zero. The point is not the exact figures, which are illustrative, it is the method: same dataset, one variable changed, compare defect rates. That delta is your evidence the guardrail worked, and it is the chart a reviewer trusts.
Red teaming before you ship
Scoring a dataset you already have tells you how the app behaves on expected input. It says nothing about an attacker. The AI Red Teaming Agent closes that gap. It uses Microsoft PyRIT, an open framework for probing generative systems, to fire adversarial prompts at your endpoint and then scores the responses with the same safety evaluators. What comes back is an attack success rate per risk category, which is the number to watch trend down release over release.
Run it against a target that mirrors production, same system prompt, same grounding, same tools. A red-team pass on the bare model is theater, because the model is rarely what ships. The agent is where indirect_attack earns its place: injection hidden in a retrieved document is invisible to a dataset test and obvious to a scan that plants it on purpose.
Where governance and the EU AI Act fit
Measurement is only half of responsible AI. The other half is the paper trail. Microsoft anchors its approach in six principles, fairness, reliability and safety, privacy and security, inclusiveness, transparency and accountability, and every Azure model ships with a transparency note that documents intended uses and known limits. Those notes are not marketing. They are inputs to a conformity assessment.
For a workload that falls under the EU AI Act, you assemble a file: what the system does, its risk category, the evaluations you ran, the defect rates, the red-team results, and the mitigations. The Responsible AI scorecard PDF is the tabular side of that file. Purview Compliance Manager provides EU AI Act assessment templates to map your controls to obligations. The evaluators and the dashboard produce the evidence; these tools organize it.
The mistake I see is teams treating governance as a launch-day document. It is a living record. Every model swap, prompt change or new tool is a new version to re-evaluate, because your last scorecard describes a system you no longer run.
Safety evaluators in the gate, the dashboard only for tabular models
If you ship a generative app, put the Foundry safety evaluators in your release pipeline first and give each high-severity category a defect-rate threshold that blocks the deploy. Add the AI Red Teaming Agent as a scheduled gate before each major launch, targeting a production-shaped endpoint. Keep the Content Safety runtime filter from Part 14 on regardless, since evaluation tells you the rate and the filter is what stops the individual bad response.
Reach for the Responsible AI dashboard only when you have an actual tabular model to defend, and when you do, script it as a pipeline and keep the scorecard. Do not try to force a chat app through it. And do not let either toolchain run as a one-time exercise, because a responsible AI record that is not regenerated on change is out of date the day after you file it.
Your move this week: pick one live workload, decide which toolchain it belongs to, and run one baseline. A number you can improve beats a policy nobody measures. Part 28 takes these gates and wires them into the full LLMOps and CI/CD pipeline on Azure.
References
- Assess AI systems with the Responsible AI dashboard, Microsoft Learn
- Risk and safety evaluators for generative AI, Microsoft Foundry
- Responsible AI scorecard, Microsoft Learn
- Responsible AI for Microsoft Foundry, overview
- Related: Guardrails and Responsible AI, what they catch and miss (GenAI Series) and Responsible AI on Amazon Bedrock (AWS Gen AI Series)
About The Author
DrJha