Most teams ship their first Bedrock feature with no pipeline behind it. A developer edits a prompt in the console, clicks save, and that is the deploy. It works right up to the morning someone rewrites the system prompt, the answers get worse, and nobody can say what the prompt used to be. LLMOps is the discipline that closes that gap.
On AWS it splits cleanly into two shapes, depending on whether you are calling a managed model or training your own. This part is about building the pipeline that turns a prompt edit or a model retrain into a reviewed, tested, reversible change. It is the operations backbone the earlier parts on evaluation and observability were quietly building toward.
Key takeaways
LLMOps on AWS is two pipelines, not one. The application path versions prompts, flow definitions, and guardrail configs and gates on evaluation. The model path versions training data and model artifacts and gates on the SageMaker Model Registry.
SageMaker Projects hands you a ready CI/CD template: a CodePipeline with source, build, deploy to staging, a manual approval, and deploy to production, wired to the Model Registry.
The gate that matters is evaluation, not accuracy alone. Wire an evaluation threshold into the pipeline so a regression flips the model to Rejected and the deploy stops before it reaches users.
What LLMOps actually means here
LLMOps, operations for large language model applications, is the set of practices that take a change to your generative system from a laptop to production without breaking it. It borrows from MLOps, the older discipline for machine learning models, and from plain DevOps, the practice of automating software delivery. The core idea is the same across all three. Every change is versioned, tested against a gate, deployed in a way you can undo, and watched after it ships.
What makes it its own thing is the artifact. In classic software the artifact is code. In MLOps it is a trained model. In an application built on a managed model like Amazon Bedrock, the thing you change most often is neither code nor a model, it is a prompt, a retrieval configuration, a guardrail policy, or a flow definition. Those are the artifacts you have to version and gate, and most teams do not, which is why a prompt edit can reach production with less review than a one line code fix would get.
AWS does not sell a single LLMOps product. It gives you parts: SageMaker Pipelines and Projects for the model path, the developer tools CodePipeline, CodeBuild, and CodeDeploy for orchestration, the Bedrock features for prompts, flows, and evaluation, and CloudWatch for what happens after. The work is deciding which parts you need, and the honest answer for a first feature is far fewer than a full platform.
Two pipelines, not one
The first design decision is which pipeline you are even building, because the two look different end to end. If you call a managed Bedrock model, you are on the application path. You are not training anything. Your changes are prompts, the retrieval sources behind a Knowledge Base, guardrail rules, and Flow or Agent definitions. You version those in Git, run an evaluation job against a held out set, and if it passes you publish a new prompt version and point the app at it. There is no GPU in this pipeline and no model artifact to store.
If you fine tune or train your own model, you are on the model path, and it looks like classic MLOps. Data preparation, training on a SageMaker job, evaluation, and registration of the resulting model package in the SageMaker Model Registry, the versioned catalog of trained models that carries an approval status on each one. Deployment means updating a SageMaker endpoint, and that is where the real infrastructure risk lives, because you are swapping a live model out from behind traffic. The line between these two worlds is the same one drawn in Part 5 on Bedrock versus SageMaker AI.
Plenty of real systems run both at once, a fine tuned model for one task and a managed model for another, so you may own two pipelines side by side. That is fine as long as you do not force one shape onto the other. The most common mistake I see is a team standing up a heavy SageMaker training pipeline for an application that only ever calls Claude through Bedrock, where there is nothing to train. Here is how the two paths line up.
| Aspect | Application path, managed model | Model path, custom model |
|---|---|---|
| What you change | Prompts, flows, guardrails, retrieval | Training data, model weights |
| Versioned artifact | Prompt version, IaC config | Model package in the Registry |
| Main gate | Evaluation score check | Registry approval after eval |
| Deploy mechanism | Publish version, update config | Update the SageMaker endpoint |
| Rollback | Point to the previous version | Blue green auto rollback |
The two LLMOps paths on AWS. Most teams start on the application path and only pick up the model path when they fine tune.
How a SageMaker Projects pipeline runs
On the model path, you do not have to wire CodePipeline by hand. Amazon SageMaker Projects is a feature that provisions a ready made CI/CD setup from a template, delivered through AWS Service Catalog, the service that hands out approved infrastructure blueprints. Pick the MLOps template and it creates two Git repositories, one for building the model and one for deploying it, plus the pipeline that connects them.
The build repository holds a SageMaker Pipeline, a directed graph of steps that runs data processing, training, evaluation, and a conditional step that only registers the model if it clears an accuracy bar. The deploy repository holds a CodePipeline with four stages: source, build, deploy to staging, and deploy to production. Between staging and production sits a manual approval, so an engineer has to click approve before a model reaches live traffic. CodeBuild runs the actual work in each stage, and CloudFormation stamps out the endpoint in each environment.
This is a lot of machinery, and it earns its keep once you are retraining on a schedule and pushing models to more than one account. For a single model you deploy by hand twice a year, the template is more than you need. Read what it generates before you adopt it, because the seed code encodes opinions, a multi account layout and a specific approval flow, that you will inherit whether or not they fit.
Versioning prompts like code
On the application path the artifact is the prompt, so version it the way you version code. Amazon Bedrock Prompt Management stores prompts as first class resources, and the create_prompt_version call, on the bedrock-agent client in boto3, takes a static snapshot of a prompt that you can deploy and roll back to. Versions are numbered from one and are immutable once created, which is exactly the property you want. The version running in production cannot be edited out from under you. The prompt and flow features themselves are covered in Part 15.
Keep the source of truth in Git anyway. The pattern I use is simple: the prompt text and its inference settings live in the repository, a CI job creates a new Bedrock prompt version on merge, and the application references that version by ARN rather than editing in the console. That gives you a diff on every prompt change, a reviewer on the pull request, and a version number to point back to when an answer regresses. Flow definitions and guardrail configs get the same treatment, stored as code and applied through CloudFormation or the CDK.
The console is still the right place to draft and experiment. Draft there, then move the final text into the repo so the change goes through review like anything else. A prompt edit that ships straight from the console is the LLM equivalent of editing code in production, and it fails the same way.
Worked example
A retrieval assistant on Bedrock gets a prompt rewrite meant to cut hallucination. The CI job creates prompt version 8 and runs the evaluation job from Part 23 against a 200 question held out set, scoring three metrics. Faithfulness comes back 0.91 and answer relevance 0.92, both above the 0.90 gate. Groundedness lands at 0.89, just under. The pipeline reads that one failing metric, sets the candidate model package to Rejected, and stops before staging. Nothing reaches users.
An engineer widens the retrieved context in the flow, reruns, and groundedness rises to 0.93. Now all three clear the gate, the version is approved, and the canary deploy begins. The point is the gate caught a change that looked like an improvement on two metrics and a regression on the third. Scores are illustrative, set your own thresholds against your baseline.
Where the evaluation gate goes
The gate is the whole point of the pipeline, so be deliberate about where it sits. Put it after the artifact is built, the prompt version created or the model trained, and before anything reaches staging traffic. The gate reads an evaluation result and makes a binary call: promote or stop. On AWS the cleanest place to record that call on the model path is the Model Registry approval status, because the deploy stage can be told to only ever deploy an Approved package.
That means your CI does not deploy directly. It flips an approval status, and the deploy pipeline reacts to it. The small script below runs after an evaluation job, compares each metric to the gate, and sets the model package to Approved or Rejected. Wire the deploy stage to the approval status, not just to this script exiting cleanly, so a rejected model cannot slip through even if someone reruns the job by hand.
import boto3
sm = boto3.client('sagemaker', region_name='us-east-1')
GATE = 0.90
scores = {'faithfulness': 0.91, 'relevance': 0.92, 'groundedness': 0.89}
package_arn = 'arn:aws:sagemaker:us-east-1:111122223333:model-package/rag-assistant/7'
passed = all(v >= GATE for v in scores.values())
status = 'Approved' if passed else 'Rejected'
failing = [k for k, v in scores.items() if v < GATE]
sm.update_model_package(
ModelPackageArn=package_arn,
ModelApprovalStatus=status,
ApprovalDescription=f'auto gate, min {min(scores.values())} vs {GATE}',
)
print(status, failing) # Rejected ['groundedness']
if not passed:
raise SystemExit(1) # fail the CI step so the deploy stopsExpected output: Rejected [‘groundedness’] and a non zero exit that fails the CI step. Failure mode: a wrong ModelPackageArn raises ValidationException, so the gate neither approves nor rejects and the pipeline stalls, which is safer than a false approve. The subtler trap is a deploy stage keyed only on the script exit rather than on the approval status, because a manual rerun can then push a rejected model. Make the approval status the condition.
Passing the gate is not the end of the risk, because the model still has to replace a live one. SageMaker deployment guardrails handle that swap for endpoints, shifting traffic from the old fleet to the new one in a controlled way with automatic rollback. In canary mode a small slice of traffic, say 10 percent, hits the new model for a baking period while CloudWatch alarms watch error rate and latency. If an alarm trips during the bake, all traffic rolls back to the old fleet on its own. On the application path the equivalent is cheaper, you point a percentage of calls at the new prompt version and revert by changing one reference. Here is how the swap unfolds in canary mode.
| Mode | How it shifts | When to use it |
|---|---|---|
| All at once | 100 percent, then bake | Low risk, dev and test |
| Canary | Small slice, then the rest after a bake | Most production model updates |
| Linear | Equal steps, n bakes | Large blast radius, slow ramp |
| Rolling updates | Batches of inference components | Big models, limited spare capacity |
Traffic shifting modes for SageMaker endpoint updates. Rolling updates for inference components arrived in 2025 for large models where spinning up a full second fleet is costly. Confirm current mode support for your instance and container.
Failure modes worth planning for
A few things break these pipelines in practice, and none are exotic. The first is a gate with no teeth, an evaluation set so small or so stale that it passes everything. A ten question set will not catch a regression, and a set that never changes drifts away from what users actually ask. Grow and refresh the eval set as part of the pipeline, not as a one time task.
The second is silent config drift. Prompts, guardrails, and flow definitions that live in the console instead of Git fall out of sync with what the repo claims is deployed, and the first sign is an incident nobody can reproduce because production does not match main. Keep everything as code. The third is a rollback you never tested. A rollback path that has never run is a hope, not a control, so exercise it on purpose. Revert a prompt version or trip a canary alarm on a quiet afternoon and confirm traffic actually moves back.
The last one is scope. A full SageMaker Projects setup around an application that only calls a managed model adds pipeline stages that guard a model you are not training, which is cost and complexity with no matching drop in risk. Match the pipeline to the path. If there is no training, there is no training pipeline to build.
My take
The evaluation gate is the single highest leverage thing in this whole stack, and it is the piece teams skip first because it is the least fun to build. A deploy pipeline with no gate just ships bad changes faster. I would rather have a plain GitHub Actions job that runs one honest evaluation and blocks the merge than a full SageMaker Projects deployment with a gate that waves everything through. Build the gate before you build the machinery around it.
Start with one gate, then grow the pipeline
Here is where I would start on a real system. First, decide which path you are on, and be honest that a managed model app has no training pipeline. Second, move your prompts, flows, and guardrail configs into Git and create Bedrock prompt versions from CI, so every change gets a diff and a reviewer. Third, stand up exactly one gate, an evaluation job that has to pass before a change reaches staging, recorded as a Model Registry approval on the model path or a required check on the application path. Only after those three would I reach for the full SageMaker Projects template, and only if I am retraining on a schedule across more than one account.
When is the heavy machinery wrong? A feature you change monthly, deployed to one account, does not need multi stage CodePipeline and manual approvals. It needs the gate and a rollback you have tested. Before you call any of this done, validate three things: that your eval set is large and current enough to fail a real regression, that a rejected model genuinely cannot deploy, and that a rollback moves traffic when you trigger it. The vendor neutral case for why evaluation and grounding matter more than model choice is in the GenAI Series, and the Azure equivalent of this pipeline story, Foundry, Prompt Flow, and PTU deploys, sits in the Azure Gen AI Series. The next part puts all of this together into reference architectures, chatbot, RAG, agentic, and batch, on AWS.
This week, add one evaluation gate to whatever you ship next, even a single check that blocks the merge. The first time it stops a change that looked fine, you will stop arguing about whether the gate is worth it.
References
- Use SageMaker AI provided project templates, Amazon SageMaker AI
- Deployment guardrails for updating models in production, Amazon SageMaker AI
- Deploy a prompt using versions in Prompt management, Amazon Bedrock
- Amazon SageMaker Pipelines, workflows for machine learning
- Implement GenAIOps to optimize the application lifecycle, Generative AI Lens
About The Author
DrJha