A hiring manager for a machine learning platform team once asked me one question that decided the interview: how would you roll back a model that started making worse predictions on a Friday afternoon. I answered it the way I would answer it for any service in trouble, and that answer is the argument of this part. Most of what the industry files under MLOps is operations you already run, wearing unfamiliar labels.

What MLOps actually covers

MLOps is the practice of getting a trained model into production and keeping it useful there. Its name suggests something new, but the work is mostly familiar. In a 2015 paper that every platform team should read, a group of Google engineers made the point with a single figure: the box holding the actual machine learning code is a tiny square inside a diagram crowded with data collection, data verification, feature extraction, configuration, serving infrastructure, resource management, analysis tooling and monitoring. Their conclusion was blunt, only a small fraction of a real world machine learning system is the model itself. Everything else is the system around it, and that system is operations.

Last part we engineered features from a month of one cluster CPU and memory metrics and trained a first classifier on them. This part writes no new code. It frames how you will operate that model, because Parts 17 to 20 serve it, monitor it and version it, and every one of those steps is a job you have done before for something that was not a model.

So the honest framing is not that you are learning a new discipline. You are extending one you already practise to a new kind of artifact. A model is a deployable thing with a version, a config, a set of dependencies and a failure mode. You know how to ship, watch and roll back deployable things. What follows is a map of what transfers cleanly, where the map has a gap, and the two or three items you genuinely have to add.

This is why the word trips people up. It looks like a new specialism with its own conferences and tools, and it is sold that way. Underneath, the day to day is provisioning, deploying, watching and responding, with a model in the slot where a service used to be. An operator who has carried a pager for a payments system has felt more production pressure than most people who can explain gradient boosting, and that pressure, not the algorithm, is what keeps a model useful after launch.

flowchart TB
  A[Data collection and validation] --> F[Feature pipelines]
  F --> M[Model code, the small box]
  M --> SV[Serving and rollback]
  SV --> MO[Monitoring and drift]
  MO --> A
  C[Config and versioning] --> M

Operations practices that map directly

Put your current toolkit next to its MLOps counterpart and most rows line up without much argument. Version control becomes versioning the model, the pipeline and the config together. Your CI suite becomes a pipeline that tests code and, this is the new clause, validates data. Blue green deployment and rollback become deploying a new model version and rolling back to the previous one held in a registry. Dashboards and alerting become model monitoring. On call and runbooks become model incident response. Capacity planning becomes serving capacity and retraining cost. Keep the reference table below from this part; it is the operations to MLOps map, and every later part in this phase fills in one of its rows.

Read the third column, because that is where the work actually is. Deployment does not change much; a model behind an API is still an artifact behind an API. Rollback does not change in shape, only in what you roll back to, a prior model version rather than a prior container tag. Two rows carry the real new content, data validation and monitoring, and both come from the same root cause, which is the next section.

Take deployment as the clearest case. A trained model is a file, or a few files, behind a prediction endpoint. You put it behind the same kind of gateway, health check and autoscaler you already run, you promote it through the same environments, and you keep the previous version warm so a rollback is a config change and not a rebuild. Nothing in that sentence is new to you. What is new is only that the artifact was produced by fitting data rather than by compiling code, and that difference matters for validation and monitoring, not for how you ship it.

Where the operations analogy breaks

A service depends on code and configuration. A model depends on code, configuration and data, and the data dependency is the one with no clean equivalent in the systems you run today. Those same Google engineers named the trap precisely: Changing Anything Changes Everything, which they shortened to CACE. Change one feature, retrain, and the whole model shifts in ways you cannot localise, because a model has no module boundaries to contain a change. A one line tweak to how you compute a rolling average of CPU can move predictions across every class at once.

That is why a model fails differently from a service, and it is the single most important thing to carry from operations without carrying the wrong instinct with it. A service tends to fail loudly. It throws, it times out, it pages you. A model usually fails silently. It keeps returning confident answers that are quietly worse, because live traffic has drifted away from the data it learned on while nothing in the code moved. Your uptime dashboard stays green through the entire failure. This is the place where the obvious operational instinct, if it is up and fast it is healthy, is simply wrong for a model, and it is wrong for a concrete reason: a model can be up, fast, and confidently incorrect at the same time.

Two genuinely new checks both fall out of this. First, data validation in the pipeline, so a change in the shape, range or distribution of an input fails the build the way a broken unit test would. Second, drift monitoring in production, so you are watching prediction quality and input distributions, not only latency and error rate. Neither is exotic. Both are the model specific versions of things you already believe in, tests before deploy and monitoring after.

MLOps maturity as an operations maturity model

Operations teams already think in maturity ladders, from a service someone deploys by hand to one that ships through a tested pipeline nobody has to babysit. Google published the same ladder for models, and it has three rungs. Level 0 is a fully manual process, a data scientist builds a model in a notebook and hands it to operations to deploy, with a gap between the two teams that should feel painfully familiar. Level 1 automates the training pipeline itself, so the model retrains and redeploys on new data without a human running each step. Level 2 applies CI and CD to the pipeline, so a change to the pipeline is built, tested and deployed automatically, the same way a mature platform team ships infrastructure.

Level 0 has a specific smell you will recognise from the worst deployments of your career. A model is built on someone laptop, works in a notebook, and is thrown over a wall to a team that never saw the training data and cannot reproduce the result. When it breaks, nobody can rebuild it, because the steps lived in one person head. If that sounds like a service deployed by hand with no pipeline and no runbook, that is exactly the parallel, and the cure is the same, put the steps in a pipeline and the artifact in a registry.

A useful move is to locate yourself honestly. Most teams shipping their first model sit at level 0 and imagine they need level 2 tooling to start. They do not. If you already have CI, artifact storage and automated deployment for your services, you are closer to level 1 than a data scientist with a strong notebook and no operations background, and you get there by pointing tools you already run at a new artifact, not by adopting a platform. Aim for level 1 first, and only add the pipeline CI of level 2 when the cost of a manual pipeline update starts to hurt.

Toil, error budgets and model drift

Three ideas from site reliability engineering transfer to models almost unchanged, and naming them makes the new job legible. Toil, the manual, repetitive, automatable work that scales with the service and creates no lasting value, has a direct model form: retraining by hand, copying model files, editing a threshold in a config every week. SRE guidance to keep toil under half of a team time is a good rule for a model too; if you are hand retraining more than that, automate the pipeline, which is exactly the level 0 to level 1 move.

Error budgets transfer as well. An error budget is one minus your service level objective, so a 99.9 percent objective spends a 0.1 percent budget of allowed failure before reliability work takes priority over new features. A model gets the same treatment, except the objective is about prediction quality rather than uptime. You set a floor on accuracy or on a business metric, you watch how much of that budget drift is spending, and when the budget runs out you retrain or roll back instead of shipping a new feature. Same governance, different signal.

That different signal is the last thing to internalise. A monitoring map below shows the swap you have to make. You keep every operational signal you already watch, and you add a layer that a service never needed.

Capacity planning carries over with one addition. You still size serving for peak request load, the same exercise as sizing any service, but a model adds a second cost centre, retraining, which can mean expensive GPU hours on a schedule. Treating retraining as a batch job with a budget, rather than a thing you run whenever accuracy feels low, is the same discipline you already apply to any scheduled job that costs money. GPU sizing has its own part in the Data Science Series, so weigh it there when the retraining bill starts to matter.

Deep mechanics of each of these live in the Data Science Series so this series does not repeat them: model monitoring and drift has its own part, serving models batch and real time has its own part, CI and CD for machine learning has its own part, and experiment tracking with a model registry through MLflow has its own part. Treat those as the reference manuals for the rows in the tables here. If you have run large service fleets, you may find the closest analogy is not in a data science text at all but in LLM observability, where tracing and debugging a probabilistic system faces the same problem, a component that does not crash but degrades.

What is genuinely new to learn

Subtract everything that transfers and a short list remains. First, data validation as a build gate, checking schema, ranges and distributions of inputs, which you can learn in an afternoon and wire into the CI you already run. Second, drift detection, the statistics of noticing that live data has moved, which leans on the distribution thinking from earlier in this series rather than on any new tool. Third, a model registry and experiment tracking, a versioned store of models and the runs that produced them, which is an artifact repository with model specific metadata and nothing conceptually strange. That is close to the entire net new surface for an operator moving into MLOps. Everything else on the job is your current job with a model in the artifact slot.

Start from your runbooks, not a new platform

Your strongest position going into a machine learning platform role is the one that answered the interview question at the top of this part. You already own the operational half of MLOps, and it is the half most data scientists reach production weak in. Lead with it. Do not let a job posting full of platform names convince you that you are starting over; those platforms are implementations of practices you have run for years.

Next part serves this model for real, batch and real time, and turns the deployment column of the map into working configuration.

References

• Sculley et al, Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015
• Google Cloud, MLOps continuous delivery and automation pipelines in machine learning
• Google SRE Book, Eliminating Toil
• Data Science Series, Monitoring Machine Learning Models and Drift
• AI Engineering Series, LLM Observability, Tracing and Debugging

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts