A green build tells you the code compiled and the unit tests passed. It tells you nothing about whether your app still answers well. I once merged a one line prompt tweak that sailed through every test and quietly dropped answer quality on about a third of real questions, because no test in the suite ever read a model response. That gap is the whole reason LLMOps exists, and it is where a CI/CD pipeline for a generative app on Azure has to do more than a normal one.
An Azure LLMOps pipeline runs on the same spine as classic MLOps: commit, pull request, continuous integration, continuous deployment. What changes is the gate. You add an evaluation step that scores model output on a fixed dataset and blocks the merge when quality drops or a safety defect rate climbs. Register the flow or app as a versioned asset, deploy that exact version from the registry to a managed online endpoint, then shift traffic from blue to green with mirroring and small percentages before you commit to 100. One deadline reshapes new builds in 2026: Prompt Flow is on a retirement path, so start greenfield work on the Foundry SDK, not on Prompt Flow.
Why a passing test suite still ships a worse answer
LLM output is not deterministic and it has no single correct label per input, so the checks that guard normal software do not catch a regression in answer quality. A unit test asserts that a function returns 4. There is no equivalent assertion for "this answer is grounded and useful," because two different wordings can both be right, and a quiet drop in grounding passes every string check you write. That is the hole. LLMOps, sometimes called GenAIOps, is the practice of managing the build, test, deploy and monitor loop for LLM-infused apps, with evaluation of the model output as a first-class pipeline step. That last clause is the entire difference from classic machine learning operations.
So you move the judgment into the pipeline. Instead of asserting an exact string, you run the app over a curated dataset and score each response on metrics like groundedness, relevance and a safety defect rate, then gate the merge on those scores. The scoring is done by other models, which feels circular until you accept the trade: a measured, repeatable number you can trend beats a human skimming ten outputs the night before a release. The score can be wrong in a single case. It is consistent across releases, and consistency is what a gate needs.
What LLMOps adds on top of MLOps
Most of an Azure LLMOps pipeline is ordinary MLOps you already know. The endpoint, the registry, the traffic split, all of it is shared plumbing, which is why Microsoft ships one set of tools for both. The pieces that are new sit at the edges: what you version, what the CI check runs, and what the quality gate measures. Here is the delta, stage by stage.
| Stage | Classic MLOps | What LLMOps adds |
|---|---|---|
| Versioned asset | Model binary and weights | Flow or app code, prompts, connections, dataset |
| CI check | Unit tests and lint | Experiment run plus an evaluation run scored on data |
| Quality gate | Accuracy or F1 threshold | Groundedness, relevance, coherence, safety defect rate |
| Deploy target | Managed online endpoint | Same endpoint, flow packaged as a container |
| Rollout | Blue-green traffic split | Blue-green, plus prompt or model variant A/B |
| Monitor | Data drift, latency | Output quality drift, token cost, safety flags |
flowchart LR
C[Commit] --> PR{Pull request evaluation gate}
PR -->|Pass| CI[CI registers a flow version]
PR -->|Fail| C
CI --> CD[CD deploys the registered version]
CD --> E[Managed online endpoint]
Map the pipeline from commit to endpoint
Microsoft ships a reference implementation, the GenAIOps with Prompt Flow template, and it is worth reading even if you do not adopt it wholesale, because the branch model is the part most teams get wrong. The template uses two long-lived branches, main and development. You cut a feature branch, and when you open a pull request into development, a build validation pipeline runs your experimentation flows. Approve and merge, and the continuous integration pipeline for the dev environment runs the experimentation and evaluation flows in sequence, then registers the flows in the Azure Machine Learning registry. A continuous deployment trigger picks up from there, deploys the standard flow from the registry as a managed online endpoint, and runs integration and smoke tests against it. Promotion to production is a second pass through the same machinery from a release branch.
The template runs on GitHub Actions, Azure DevOps or Jenkins, so the shape holds whatever your org already uses. The orchestrator is not the interesting choice. Where you put the evaluation gate is.
Evaluation as the gate that matters
This is the step that earns the name LLMOps. Run the app over the dataset, score each response with evaluators, compare against a baseline, and block the merge if it regresses. The content safety evaluators from Part 27 return a defect rate, the share of responses flagged. The quality evaluators return numeric scores such as groundedness, how well the answer sticks to its source, and relevance, how well it answers the question. Pick a threshold for each and wire the job to exit non-zero when a candidate falls below it. A non-zero exit is what turns a report into a gate.
Here is that gate as a GitHub Actions job. It authenticates to Azure with a federated credential, runs the evaluation, and fails the check when the defect rate is too high.
name: pr-eval-gate
on:
pull_request:
branches: [ development ]
permissions:
id-token: write # needed for OIDC login, no stored secret
contents: read
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: azure/login@v2
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
- run: pip install -r requirements.txt
- name: run eval and gate
run: python eval/run_eval.py --data eval/pr_set.jsonl --gate defect_rate=0.02 groundedness=4.3Expected result: the job prints a per metric summary and exits 0 when every metric clears its gate, so the PR check goes green. Failure mode: your script must call sys.exit(1) when a gate is missed, or the workflow reports success on a regression and the whole exercise is decorative. Test the red path on purpose by lowering a gate below the current score once.
Worked example
A retrieval assistant is scored over 120 curated questions before merge. Groundedness comes back at 4.6 out of 5 against a 4.3 gate, relevance at 4.4 against 4.2, and a safety defect rate of 0.8 percent against a 2 percent ceiling. All three clear, the pull request check goes green, and the flow is registered as version 7.
A week later someone rewrites the system prompt to shorten answers. Same 120 questions, one variable changed. Groundedness drops to 4.0, under the 4.3 gate, because the shorter answers cite less of the source. The PR check goes red and the merge is blocked before anything reaches an endpoint. The figures are illustrative, but the method is the point: fixed dataset, one change, an automatic stop. The 4.6 and 4.3 groundedness bars above are exactly this comparison.
Register once, then deploy that version
The rule that keeps a pipeline honest is simple. CI registers a versioned asset, and CD deploys that exact version. Nothing rebuilds from source at deploy time. In Azure that asset is a registered model, environment or flow in the workspace or a shared registry, and a deployment references it by name and version, in the form azureml:my-model:7. For production, register the model and the environment as separate assets so a scoring image change and a model change get their own version numbers rather than hiding inside one deployment.
This looks like bureaucracy until the day a reviewer asks which version served a given week and why an answer changed. With registration, the reply is a version number and an eval report attached to it. Without it, the reply is an afternoon of reading git history and hoping the build was reproducible. I have lived both. The version number is cheaper.
How does safe rollout work on a managed endpoint?
A managed online endpoint is a stable HTTPS address that can hold more than one deployment behind it and route traffic between them by percentage. That routing is what makes blue-green safe. Blue is the version serving today. Green is the candidate. You bring green up with zero traffic, test it in isolation, mirror a slice of real traffic to it to gather metrics without touching client responses, then send a small live percentage, then all of it, then delete blue.
A few limits shape the plan. Mirrored traffic caps at 50 percent, targets only one deployment, and is not available on Kubernetes endpoints. Live traffic percentages across deployments must total either 0 or 100, so you cannot leave an endpoint at 90 by accident. And for real availability, run at least three instances per deployment, because Azure holds back about 20 percent of capacity during upgrades. Here is the full rollout as stages.
| Stage | Command essence | Live traffic to green | What you watch |
|---|---|---|---|
| Bring up green | deployment create, no traffic | 0 percent | cold start, smoke test |
| Mirror | mirror-traffic green=10 | 0 percent, 10 shadowed | latency, errors, no client impact |
| Canary | traffic blue=90 green=10 | 10 percent | live error rate, eval on real traffic |
| Full | traffic blue=0 green=100 | 100 percent | dashboards, cost, quality drift |
| Retire | deployment delete blue | n/a | endpoint stable on green |
The rollout itself is a short sequence of Azure CLI calls. This is the operational core of the part.
# blue already serves 100% of ENDPOINT_NAME. Roll out green safely.
# 1. create green with no traffic
az ml online-deployment create --name green
--endpoint-name $ENDPOINT_NAME -f green-deployment.yml
# 2. shadow 10% of live traffic to green, clients still get blue
az ml online-endpoint update --name $ENDPOINT_NAME
--mirror-traffic "green=10"
# 3. stop mirroring, send 10% of real traffic to green
az ml online-endpoint update --name $ENDPOINT_NAME --mirror-traffic "green=0"
az ml online-endpoint update --name $ENDPOINT_NAME --traffic "blue=90 green=10"
# 4. cut over fully, then retire blue
az ml online-endpoint update --name $ENDPOINT_NAME --traffic "blue=0 green=100"
az ml online-deployment delete --name blue --endpoint $ENDPOINT_NAME --yes --no-waitExpected result: step 2 shows green in the endpoint with 0 live traffic while its logs fill with mirrored requests; step 3 splits real traffic 90 to 10; step 4 leaves green on 100 percent and removes blue. Failure mode: skip the mirror-traffic green=0 reset in step 3 and the endpoint tries to both mirror and serve green, which is not allowed, so the update is rejected. Always zero the mirror before you assign live traffic.
Build new pipelines on the Foundry SDK, not Prompt Flow
Most of the LLMOps material online still centers on Prompt Flow, and it is worth knowing why I would not start a new pipeline there in 2026. Microsoft has put Prompt Flow on a retirement path. Feature development ended in April 2026, the feature enters read-only mode and is fully retired on April 20, 2027, and the guidance is to migrate to the Microsoft Agent Framework before then. Existing flows keep running until that date, so this is not a fire drill. It is a planning fact.
The practical read: if you have a Prompt Flow pipeline in production, it is fine, keep it, and schedule the migration well before the deadline. If you are starting fresh, build on the Foundry SDK path instead, since the CI/CD shape in this part, the eval gate, the registry step, the blue-green rollout, is the same either way. The evaluation gate and the safe rollout do not depend on Prompt Flow at all. Only the authoring layer does, and that is the piece being replaced.
The AWS half of this series walks the same discipline on Bedrock and SageMaker Pipelines, and the parallels are close enough to lift if you run both clouds. See LLMOps and CI/CD for Amazon Bedrock and SageMaker for the Amazon equivalent of the registry and safe rollout, and the guide for where each Azure piece fits.
Promote on evals, not on a green build
If you build one thing from this part, make it the evaluation gate. Wire an evaluation run into the pull request check and refuse to promote a flow that regresses on groundedness or trips the safety defect ceiling, exactly the way you refuse to merge failing unit tests. Register every candidate as a versioned asset and deploy that version to a managed online endpoint, never a fresh build at deploy time. Roll it out blue to green: zero traffic, mirror, ten percent, then all. And point new pipelines at the Foundry SDK rather than Prompt Flow, so you are not migrating a production pipeline against a deadline in 2027.
When this is the wrong amount of machinery: a throwaway prototype or a single internal demo does not need any of it. Ship it by hand and add the pipeline when more than one person depends on the output. What to validate first: run your eval dataset through the current production app once to set honest baselines before you turn any threshold into a blocking gate, or your first pipeline will block everything or nothing.
Your move this week: take one Azure GenAI app, write a 50 row eval set, and make a failing groundedness score block a pull request. One gate that stops a bad merge beats a diagram of a pipeline you have not wired. Part 29 assembles these gates and rollouts into full reference architectures on Azure.
References
- GenAIOps with prompt flow and GitHub, Microsoft Learn
- Safe rollout for online endpoints, Microsoft Learn
- GenAIOps with Prompt Flow template, GitHub
- Migrate Prompt Flow to the Microsoft Agent Framework, Microsoft Learn
- Related: Azure Prompt Flow authoring and evaluation (Part 15) and LLMOps and CI/CD on AWS (AWS Gen AI Series)
About The Author
DrJha