A tuned Gemini model that looked fine in a notebook shipped a worse answer three weeks later, and nobody noticed until a support ticket. The retrain had run, the data had shifted, and the new checkpoint went straight to the endpoint with no gate in front of it. That gap is what LLMOps closes, and on Google Cloud the piece that closes it is Vertex AI Pipelines.
What LLMOps means on Vertex AI
LLMOps is the operations layer around a language model: the plumbing that gets a model from an experiment into production and keeps it honest once it is there. It is MLOps with two extra worries. Prompts and eval sets become first-class artifacts you version like code, and quality is fuzzy, so a numeric gate matters more than it does for a classifier with a clean accuracy number.
Vertex AI Pipelines is Google Cloud managed workflow runner for that layer. A pipeline is a directed graph of steps, called components, where each component is a containerized unit of work with typed inputs and outputs. You define the graph in Python, Vertex compiles it to a template, and the service runs each component on its own compute, passing artifacts between them and recording every run in ML Metadata. That metadata is what gives you lineage: which data produced which model, evaluated by which set, promoted by whom.
The reason this belongs in the series here, near the end, is that it ties the earlier parts together. Tuning from Part 16, distillation from Part 17, the evaluation service from Part 23, and the tracing from Part 25 each become one component in a graph you can rerun on demand.
flowchart LR A[New logs in BigQuery] --> B[Prep and filter] B --> C[Supervised tune Gemini] C --> D[Eval component] D -->|score above gate| E[Register new version] D -->|score below gate| F[Stop and alert] E --> G[Deploy to endpoint] G --> H[Monitor drift] H --> B
Why not just a cron job and a bash script
You can retrain a model with a shell script on a VM and a crontab line. I have done it. It works until it does not, and when it breaks it breaks quietly. A script has no memory of what it ran last night, so it reprocesses the same data every time and pays for it. It has no typed hand-off between steps, so a corrupt intermediate file flows straight into training. And it has no record of which data produced which model, which is the first thing an auditor or an on-call engineer asks when a model starts misbehaving.
Pipelines fix three things a script cannot. Caching: if a step inputs are unchanged, Vertex reuses the prior output instead of recomputing it, which on a data-prep step that dominates runtime is real money. Lineage: every artifact is logged with its producer, so you can trace a bad answer back to the exact training run. Isolation: each component runs in its own container with its own machine type, so the tuning step can grab a GPU while the eval step runs on a cheap CPU box.
How a pipeline actually runs
There are three moving parts. The KFP SDK is the Kubeflow Pipelines Python library you author in; Vertex AI Pipelines supports KFP SDK v2 or later. The compiler turns your Python into a pipeline template, a JSON file that describes the graph. The PipelineJob is the running instance Vertex creates when you submit that template with a set of parameter values and a pipeline root, which is a GCS path where the run writes its artifacts.
The flow is compile once, run many. You compile the template in CI, store it, and submit it with different parameters for dev and prod. Because the template is a static file, the thing you tested is exactly the thing that runs. Nothing gets re-interpreted at submit time.
Building the pipeline in Python
A component is a Python function with a decorator that tells KFP how to containerize it. The pipeline function wires components together by passing one output into the next input. Here is a trimmed nightly tune-and-gate pipeline. It compiles to a template and submits as a PipelineJob with caching on.
from kfp import dsl, compiler
from google.cloud import aiplatform
@dsl.component(base_image='python:3.11', packages_to_install=['pandas'])
def prep_data(source: str, rows_out: dsl.OutputPath(int)):
# read yesterday logs, filter, write JSONL to GCS
with open(rows_out, 'w') as f:
f.write('4120')
@dsl.component(packages_to_install=['google-cloud-aiplatform'])
def eval_gate(model_name: str, threshold: float) -> bool:
from google.cloud import aiplatform as ai
score = 0.93 # replace with the Gen AI evaluation service call
return score >= threshold
@dsl.pipeline(name='nightly-gemini-tune')
def nightly(project: str, region: str = 'us-central1', gate: float = 0.90):
prep = prep_data(source='bq://logs.chat_turns')
# tuning component wired here, produces model_name
passed = eval_gate(model_name='projects/x/models/tuned', threshold=gate)
with dsl.If(passed.output == True, name='promote'):
pass # register + deploy components run only on pass
compiler.Compiler().compile(nightly, 'pipeline.json')
aiplatform.init(project='my-proj', location='us-central1')
job = aiplatform.PipelineJob(
display_name='nightly-gemini-tune',
template_path='pipeline.json',
pipeline_root='gs://my-bucket/pl-root',
parameter_values={'project': 'my-proj'},
enable_caching=True,
)
job.submit()
print(job.resource_name)Expected output: a resource name like projects/123/locations/us-central1/pipelineJobs/nightly-gemini-tune-20260711, and a run graph in the console. Failure mode: if pipeline_root is a bucket in a different region than the job location, the run fails at submit with a location mismatch; keep the bucket and the pipeline in the same region.
Worked example
A nightly pipeline runs 30 times a month. Vertex charges a small per-run execution fee of about 0.03 dollars, so orchestration overhead is 30 times 0.03, which is 0.90 dollars. The compute is the real cost: the tune and eval steps run about 2.10 dollars each night, so 30 times 2.10 is 63.00 dollars, and the month lands near 64 dollars. Turn caching on and the data-prep step is skipped on nights when yesterday logs already processed, cutting roughly a third of compute, so compute drops to about 1.37 dollars a night and the month lands near 42 dollars. Same pipeline, 22 dollars a month saved, from one flag.
Wiring evaluation as a promotion gate
The single most valuable component in the graph is the one that can say no. In the code above, eval_gate returns a boolean and the register and deploy steps sit inside a dsl.If block that only executes when the eval passes. A run that scores below the threshold stops with the new checkpoint sitting in a bucket, never registered, never deployed. That is the whole point: the retrain still runs, but it cannot ship a regression on its own.
Do not invent your own eval math inside that component. Call the Gen AI evaluation service from Part 23 and read back a pointwise score against a held-out set you version alongside the pipeline. The gate threshold is a parameter, so you can tighten it over time without recompiling. I keep the eval set in the same repo as the pipeline code and bump its version in the same commit, because a gate is only as trustworthy as the set it scores against.
Model Registry and lineage
The register step writes the tuned model into the Vertex AI Model Registry, the central store for model versions and their metadata. Registering a new checkpoint as a version of an existing model, rather than a brand new model, is what gives you a clean history: version 7 supersedes version 6, both point back to the pipeline run that produced them, and you can roll the endpoint back to version 6 in one call if version 7 turns out wrong.
Lineage is the quiet payoff. Because each artifact is logged in ML Metadata with its producer, you can start from a bad answer in production, find the endpoint version, trace it to the pipeline run, and see the exact data slice and eval score behind it. With a bash script that chain does not exist. This is also the hook into the governance work from Part 27: the registry version is what an audit points at.
| Setup | Overhead / mo | Compute / mo | Total / mo | Lineage |
|---|---|---|---|---|
| Manual VM + cron | idle VM | ~95 | ~95 | none |
| Pipeline, no cache | 0.90 | 63.00 | ~64 | full |
| Pipeline, cached | 0.90 | 41.10 | ~42 | full |
| Cached + Spot tune | 0.90 | 33.00 | ~34 | full |
Table 1. Illustrative monthly cost and lineage for the four setups in Figure 2. Overhead is the per-run execution fee; compute is the tune plus eval steps.
Scheduling and retriggering retrains
A pipeline you run by hand is a demo. To make it operational you attach a schedule, which submits the same template on a cron expression. Vertex runs it, applies the gate, and either promotes or alerts, all without a human in the loop on a normal night. You can also trigger a run from an event, such as a drift alarm from the monitoring in Part 25, so the model retrains when quality slips rather than only on the clock.
Here is the schedule call. It is one method on the same PipelineJob object.
job = aiplatform.PipelineJob(
display_name='nightly-gemini-tune',
template_path='pipeline.json',
pipeline_root='gs://my-bucket/pl-root',
enable_caching=True,
)
schedule = job.create_schedule(
display_name='nightly-2am',
cron='0 2 * * *',
max_concurrent_run_count=1,
)
print(schedule.resource_name)Expected output: a schedule resource name; the run appears each night at 02:00 in the pipeline location time zone. Failure mode: without max_concurrent_run_count set to 1, a slow run can overlap the next trigger and you get two tuning jobs racing for the same registry version. Cap concurrency.
Does the gate actually hold over time
A gate is only useful if it fires when it should. Here is eight weeks of a real-shaped nightly run scored against a fixed eval set with the promotion gate at 90 percent. Two nights fall below the line and stop, the model does not ship those nights, and the endpoint keeps the last good version. That is the system working, not failing.
What a run costs, and where it hides
Two lines make up the bill. The execution fee is a small per-run charge, around 0.03 dollars, for the orchestration itself. The compute is everything your components consume: Vertex custom training for the tune step, the machine you run eval on, and any managed service the components call. On any real pipeline the compute dwarfs the execution fee, so tune the compute, not the orchestration. Caching and Spot machines for the tuning step are the two levers that move the number, as Figure 2 shows.
The cost that hides is idle: a component that requests a large machine and then waits on an external call is paying for a GPU to do nothing. Split slow I/O into its own small component so the expensive machine only exists while it is computing. This mirrors the FinOps thinking from Part 26, applied per component instead of per endpoint.
Start with the gate, add the schedule last
Here is my recommendation, plainly. Do not build the whole loop on day one. Build the eval component and the gate first, run the pipeline by hand, and prove that a bad checkpoint stops before it registers. That single branch is where the value lives, and it is the piece a bash script can never give you. Once the gate holds, add the Model Registry step so you have version history, then wire caching to control cost, and only then attach the schedule so it runs itself.
When not to reach for this: a model you tune once and freeze does not need a pipeline, and a prototype you touch daily by hand is faster left as a notebook. Pipelines earn their keep the moment a retrain is recurring and a human is not going to eyeball every result. That is exactly when the quiet regression I opened with happens.
Next in the series pulls these pieces into full reference architectures on Google Cloud, so the pipeline here becomes one box in a larger picture. Build the gate this week, run it by hand five nights, and only schedule it once you have watched it refuse a bad model at least once. Your call to action is small: add one eval component in front of your next retrain.
References
- Introduction to Vertex AI Pipelines, Google Cloud documentation
- Build a pipeline with the KFP SDK, Google Cloud documentation
- Understand pipeline run costs, Google Cloud documentation
About The Author
DrJha