Five percent. That is the share of the code in a mature machine learning system that is actually machine learning, according to the Google team who wrote the paper on hidden technical debt in ML systems. The other ninety five percent is configuration, data plumbing, validation, serving infrastructure and glue.
That number reframes the whole job. If you picture a data science project as fitting a model, you have pictured five percent of it and you will be surprised by the rest. The lifecycle in this part is the other ninety five percent laid out in order, from the first conversation with a stakeholder through to the morning you switch a model off on purpose. Every later part of this series slots into one of these stages, so it is worth knowing the shape of the whole thing before you learn any single piece of it.
Key takeaways
The lifecycle has eight stages and none of them run once. The two loops that matter are back to the business question, and back to the data.
Modelling is the shortest stage in almost every project I have run. On the churn project used across this series it took nine of eighty one working days.
A model does not end when it ships. It decays, and the stage everyone forgets to plan for is retirement.
Write the retirement criteria in the project brief on day one, before anyone has fitted anything. It costs an hour, and it is the most valuable hour in the whole project.
Why the lifecycle is a loop
The reference process most teams borrow from is CRISP-DM, the Cross Industry Standard Process for Data Mining, which is a six phase description of how a data project runs. It was written in 1996 by a consortium including NCR, Daimler and an insurer called OHRA, and it has outlived every framework that tried to replace it. IBM still ships it as the documented methodology for SPSS Modeler, and the documentation is blunt about the thing beginners get wrong: the sequence of the phases is not strict, and most projects move back and forth between phases as necessary.
I use an eight stage version because CRISP-DM was written before models routinely lived in production for years. It splits deployment from monitoring, and it adds a retirement stage that CRISP-DM folds into its outer circle. The stages are: frame the question, assess the data, prepare the data, engineer features, model, evaluate against the decision, deploy, then monitor and retire.
Read that list once and it looks like a pipeline. It is not. In practice you will walk two loops repeatedly. The first runs from evaluation back to framing, because the model answered the question you asked and the question was wrong. The second runs from feature engineering back to data assessment, because the feature you wanted turns out not to exist in a usable form before the moment of prediction. Both loops are normal. A project that never loops is usually a project where nobody checked anything.
Business understanding, where projects are won or lost
The first stage produces no data and no code. It produces a sentence. Somebody senior says something like: too many subscribers are cancelling and we want to do something about it. Your job is to convert that into a statement precise enough to be right or wrong.
Four questions do most of the conversion work. What decision changes because of this, and who makes it. When does the decision get made, which fixes how far ahead you must predict. What is the current rule, because you have to beat something and it is never nothing. And what does a wrong answer cost in each direction, because the cost of chasing a customer who was never going to leave is not the same as the cost of missing one who does.
That last one is where beginners most often skip ahead. Churn is the running project for this whole series, using the public Telco Customer Churn dataset that IBM publishes: 7,043 fictional customers of a Californian telco, one row each, with a column recording whether they left in the last month. Suppose a retention call costs about 12 dollars in agent time and the discount offered costs another 40, while a lost customer costs roughly 780 dollars in remaining contract value. Those three numbers, not the algorithm, decide what a good model looks like. They tell you that missing a churner is roughly thirteen times more expensive than bothering a loyal one, which means you will want a model tuned toward catching churners even at the cost of a lot of false alarms. We will make that concrete later in this series when we get to metric choice. For now, notice that the tuning decision was already made in stage one, by an accountant, before anyone opened a notebook.
The output of this stage should be one written page. Not a deck. A page that states the decision, the prediction horizon, the current baseline rule, the cost of each error type, and the conditions under which you would switch the model off. If I had to pick one artefact in this list to defend, it is the last line, and I will come back to why. The thing I would avoid is the kickoff deck full of proposed techniques. A deck that names gradient boosting before it names the decision is a project already drifting.
| Stage | Who leads it | What it produces | Common failure |
|---|---|---|---|
| Frame the question | Data scientist with the business owner | A one page decision brief | No named decision maker |
| Assess the data | Data scientist and data engineer | A data inventory and gap list | Assuming a field exists at prediction time |
| Prepare the data | Data scientist | A reproducible cleaned table | Cleaning by hand in a notebook |
| Engineer features | Data scientist | A documented feature set | Leakage from the future |
| Model | Data scientist | A fitted, versioned candidate | Tuning before a baseline exists |
| Evaluate | Data scientist with the business owner | A decision, in money not accuracy | Reporting a metric nobody can price |
| Deploy | ML engineer | A scored population, on schedule | Scores land where nobody reads them |
| Monitor and retire | ML engineer with the data scientist | Drift alerts and a kill date | Nobody owns it after launch |
Getting the data ready, which is most of the calendar
Stages two, three and four are data assessment, preparation and feature engineering. Together they consume more calendar time than everything else combined, and they are the stages that tutorials compress into a single tidy import.
Assessment asks a narrower question than people expect. It is not what data do we have. It is what data will exist, in this form, at the moment the prediction has to be made. Those are wildly different questions. On a churn project you will find a field recording the reason a customer gave for leaving. It is beautifully predictive. It is also written down after they leave, which makes it useless and dangerous. That is data leakage, meaning information that would not be available at prediction time bleeding into your training set, and it is the single most common way a model that looks brilliant in a notebook does nothing in production.
Preparation is the cleaning work: missing values, duplicate rows, categories spelled three different ways, numbers stored as text. I am not going to re-teach it, because I covered it properly for analysts in cleaning messy data, and the exploration habits in exploratory data analysis carry over unchanged. The one difference for a modeller is that your cleaning must be reproducible as code, because every step you perform by hand is a step that will not happen when the model runs on Tuesday morning without you.
Feature engineering is where a raw column becomes something a model can use: turning a signup date into tenure in months, a set of monthly charges into a trend, a support ticket count into a rate per month. It is the stage with the highest ratio of accuracy gained to effort spent, which is why it gets its own part later. It is also the stage where leakage sneaks back in through the side door, because an aggregate feature quietly computed over the whole dataset has already seen the future.
Modelling and evaluation against a decision
A model, in the plainest terms, is a mathematical rule fitted to past examples that produces a number for a case it has not seen. Fitting one is the stage everybody imagines the job to be, and on the project above it took nine days out of eighty one.
Two habits keep this stage short and honest. Start with a baseline you can beat: the current business rule, or a model so simple it takes ten minutes. If your gradient boosted ensemble beats a plain logistic regression by two points, you have learned that the ensemble is not worth its operational cost, which is a genuinely useful result. Second, decide how you will measure success before you fit anything, and write it down. Otherwise you will fit five models, pick the metric each one happens to win on, and quietly fool yourself.
Evaluation is a separate stage from modelling for a reason. Modelling asks whether the model fits the data. Evaluation asks whether the model changes a decision profitably. Those come apart more often than you would think. A model can be measurably better than the baseline and still be worth nothing, because the retention team can only make 400 calls a week and the old rule already gave them 400 names worth calling. Improving the ranking below position 400 changes nobody Monday morning. This is also where the analyst discipline of separating correlation from causation earns its keep: a model that ranks who will churn does not tell you who can be persuaded to stay, and those are two different lists.
War story
On a subscription churn build a few years ago I shipped a model that scored customers as likely to leave within 30 days. It validated cleanly. It went live. Six weeks later the retention lead told me the calls were not converting.
The cause had nothing to do with the model. Scores were written to a table on the first of the month, pulled into the campaign tool on the eighth, and dialled from the fifteenth. By the time an agent rang, the 30 day window was half gone and a chunk of the list had already cancelled. The model was answering a question about day zero and the business was acting on day fifteen.
The fix took one afternoon: retrain on a 60 day horizon and move the scoring job to weekly. Conversion on the retention calls went from about 9 percent to 14 percent, with no change to the algorithm at all. I had spent five weeks on features and had never once asked how many days passed between the score being written and a human acting on it. It is now the fourth question in my project brief.
| Gate | Question it answers | Churn project threshold | If it fails |
|---|---|---|---|
| Feasibility | Does signal exist at all | Beat random ranking by any margin | Stop. Report honestly. |
| Baseline | Does it beat the current rule | Catch 15 percent more churners at the same call volume | Keep the rule. Cheaper. |
| Economics | Does it pay for itself | Net saving above 40,000 dollars a year | Shelve until costs change |
| Operability | Can the team act in time | Score to action gap under 7 days | Fix the process first |
| Retirement | When do we switch it off | Lift below 8 percent for two months | Retrain, then retire |
Deployment, monitoring and the silent decay
Deployment means getting a prediction to the place a decision is made, on the schedule that decision runs. That can be a nightly batch job writing scores to a table, an API answering in 50 milliseconds while a page loads, or a streaming job scoring events as they arrive. Google Cloud describes three levels of MLOps maturity for this, from level 0 where every step is manual and script driven, through level 1 where the training pipeline itself is automated so models retrain continuously, to level 2 where the pipeline has its own CI/CD. Most teams reading this are at level 0, and that is a reasonable place to ship a first model from. It stops being reasonable around the third model.
Then the model starts to get worse, and nothing tells you. This is the failure mode that makes machine learning different from ordinary software. A broken web service throws errors. A decayed model returns perfectly well formed numbers that are increasingly wrong. The world moves: a competitor launches a cheaper plan, the company changes its pricing tiers, an upstream team renames a column and fills it with nulls. Nothing crashes. The scores just quietly stop meaning what they meant.
Which brings me to the stage nobody plans. Retirement. In every organisation I have worked with that has run models for more than about three years, there are models still scoring that nobody has evaluated since launch and nobody will own up to. They cost compute, they carry risk, and they influence decisions on the strength of a relationship that stopped holding two years ago. The reason they survive is simple: no one wrote down what would make them wrong. Deleting a model feels like admitting failure unless the criteria for deleting it were agreed before anyone was proud of it.
Look at that chart month by month and nothing is wrong. Month four to month five is a drop of about one and a half points, well inside noise. The decay is only visible against a line drawn eleven months earlier. That is the whole argument for writing the retirement gate into the brief on day one: it is the only moment in the project when you are objective about it.
Map the whole loop before you open a notebook
My recommendation for this part is narrow and concrete: before your next project starts, write the one page decision brief, and make sure it contains the score to action gap and the retirement gate. Those two lines take fifteen minutes to write and they are the two that would have saved me five weeks on the project in the war story above.
The thing I would avoid is the pattern that feels most productive and is not: opening the data first. Loading a CSV and starting to explore is enormously satisfying, and it commits you to whatever question that particular table happens to be able to answer. I have watched capable people spend a month producing an excellent answer to a question nobody had asked, because the data was there and the brief was not.
Keep the eight stage loop somewhere you can see it. As this series progresses, every part lands on it: the Python environment work is what makes stages three and four reproducible, feature engineering is stage four, everything from linear regression through gradient boosting is stage five, model evaluation and leakage is stage six, serving and feature stores are stage seven, and drift monitoring is stage eight. Knowing which stage you are standing in is most of what separates a project that ships from one that quietly stops.
Part 3 gets practical. We set up a Python environment that will still work in six months, with virtual environments, notebooks used deliberately rather than as a scratchpad, and the reproducibility habits that make stage three of this lifecycle survivable. That is the last piece of scaffolding before we touch the churn data in Part 4.
References
- CRISP-DM Help Overview, IBM Documentation
- MLOps: Continuous delivery and automation pipelines in machine learning, Google Cloud Architecture Center
- Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015
- Telco Customer Churn dataset, IBM sample data on GitHub
About The Author
DrJha