Here is a directory listing from a project I inherited, reproduced exactly: model.pkl model_v2.pkl model_v2_final.pkl model_v2_final_REAL.pkl

One of those four files was serving live traffic. Nobody could say which, nobody could say what data trained it, and the AUC quoted in the business case belonged to none of them. That listing is not a joke about untidy engineers. It is what happens when a team ships models without a record, and it turns every later question into an archaeology project.

Cost of not knowing which model is live

Missing provenance costs money in three specific ways, and it helps to name them rather than appeal to good practice in the abstract. First, you cannot debug a bad prediction. A customer complains, you pull the score, and without a link from that score back to a model version and its inputs there is no investigation to run, only speculation. Second, you cannot roll back. Rolling back needs a previous artifact plus confidence about what it was, and a folder of pickles gives you neither. Third, and most expensively, you cannot defend a number. Somebody eventually asks why the model that tested at 0.87 is behaving like 0.79, and an answer of we think it might be a different one ends the conversation badly.

Two distinct tools address this and they get conflated constantly. Experiment tracking records what happened during development: hundreds of runs, most of them bad, each with its settings and scores. A model registry records what is deployable: a handful of versions that survived, with a marker saying which one production should load. Development needs the first, operations needs the second, and a team that installs one while assuming it covers the other ends up disappointed in a predictable direction.

Worth saying plainly at the start: this is bookkeeping, not modelling. No accuracy comes from it. What it buys is the ability to answer questions later, and the value shows up entirely in months two through twenty four of a model life, which is why it keeps losing budget arguments against another week of tuning.

Minimum record for every training run

Before reaching for a tool, decide what a complete record contains. My working definition is that a run record is complete when a colleague can reproduce the metric from it without asking you a single question. That standard is stricter than it sounds and most teams fail it on the data side rather than the model side.

Two of those five deserve emphasis because they are routinely skipped. Tags carry the git commit hash of the code that ran and a hash of the training data, and without both you have a record of a number rather than a recipe for it. Environment means pinned library versions, captured automatically by most tools, and it matters more than people expect because a scikit-learn minor release can shift a default and move a metric in the third decimal place. Setting up that environment properly was the subject of Part 3, and this is where the discipline pays off.

MLflow tracking on the churn project

Last part we had the Telco churn features consolidated into shared functions, so training, batch scoring and the live endpoint all compute inputs the same way. This part we wrap the training script in a tracking layer, so every experiment leaves a record and the winner can be promoted by name instead of by copying a file. Everything below runs locally with no server and no cloud account.

# train_churn.py
# Tested against MLflow 3.14.0, scikit-learn 1.7.0, pandas 2.3.0, Python 3.11.9
import subprocess
import hashlib
import pandas as pd
import mlflow
import mlflow.sklearn
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

CSV = 'data/telco_churn_features.csv'
df = pd.read_csv(CSV)
y = df.pop('churn')
X = df.drop(columns=['customer_id'])

data_hash = hashlib.md5(open(CSV, 'rb').read()).hexdigest()[:12]
git_sha = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode().strip()

mlflow.set_tracking_uri('sqlite:///mlflow.db')
mlflow.set_experiment('telco_churn')

params = {'learning_rate': 0.06, 'max_leaf_nodes': 31, 'max_iter': 300}

with mlflow.start_run(run_name='hgb_windowed_v2') as run:
    model = HistGradientBoostingClassifier(random_state=42, **params)
    auc = cross_val_score(model, X, y, cv=5, scoring='roc_auc')
    model.fit(X, y)

    mlflow.log_params(params)
    mlflow.set_tag('git_sha', git_sha)
    mlflow.set_tag('data_hash', data_hash)
    mlflow.set_tag('feature_set', 'windowed_24')
    mlflow.log_metric('cv_auc_mean', auc.mean())
    mlflow.log_metric('cv_auc_std', auc.std())
    mlflow.sklearn.log_model(model, name='churn_model', input_example=X.head(2))

    print(run.info.run_id, round(auc.mean(), 4), round(auc.std(), 4))

9f2c1a7b4e6d4c0a8b3f5e7d1c2a4b60 0.8688 0.0092

Note the name argument on log_model. MLflow 3 made models first class entities and deprecated the old artifact_path parameter in its favour, so tutorials written against version 2 will still work but will emit a deprecation warning. Passing input_example is optional and worth doing anyway, because MLflow infers a schema from it and later refuses predictions whose columns do not match.

Registry, aliases and version promotion

Registering a model takes one call, and your first attempt will very likely fail. Here is the error, because it catches almost everybody who started with the default local setup.

>>> mlflow.set_tracking_uri('file:./mlruns')
>>> mlflow.register_model(f'runs:/{run_id}/churn_model', 'telco_churn')
Traceback (most recent call last):
  File '<stdin>', line 1, in <module>
mlflow.exceptions.MlflowException: Model Registry features are not supported by
the store with URI: 'file:///home/pj/telco/mlruns'. Stores with the following
URI schemes are supported: ['databricks', 'databricks-uc', 'http', 'https',
'postgresql', 'mysql', 'sqlite', 'mssql'].

Cause is structural rather than a bug. Tracking only ever appends records, which a filesystem handles fine. Registry has to enforce uniqueness of version numbers and move a pointer atomically, and a directory cannot promise either under concurrent writers. Fix is one line: point the tracking URI at SQLite instead, exactly as the script above already does. Every reader who copies that first script never sees this error, which is why I put the SQLite line in from the start and am showing you the failure afterwards.

# promote.py, same environment as above
import mlflow
from mlflow import MlflowClient

mlflow.set_tracking_uri('sqlite:///mlflow.db')
client = MlflowClient()

run_id = '9f2c1a7b4e6d4c0a8b3f5e7d1c2a4b60'
mv = mlflow.register_model(f'runs:/{run_id}/churn_model', 'telco_churn')

client.set_model_version_tag('telco_churn', mv.version, 'cv_auc', '0.8688')
client.set_registered_model_alias('telco_churn', 'champion', mv.version)

loaded = mlflow.sklearn.load_model('models:/telco_churn@champion')
print('serving version', client.get_model_version_by_alias('telco_churn', 'champion').version)

Registered model 'telco_churn' already exists. Creating a new version of this model...
Created version '4' of model 'telco_churn'.
serving version 4

Look closely at that load URI, models:/telco_churn@champion, because it is the whole point of a registry. Serving code names a role, not a version. Promoting version 5 is then a single alias move that takes effect without redeploying anything, and rolling back is the same call pointing at 4. Compare that with the endpoint from Part 22, which loaded a file path and therefore needed a rebuild for every model change.

Older material will tell you to use stages instead, with names like Staging, Production and Archived. Ignore it. MLflow deprecated stages in version 2.9 and replaced them with aliases plus model version tags, for a reason that becomes obvious the first time you want a champion and a challenger live at once: a version could hold exactly one stage, whereas any number of aliases can point wherever you like. My convention on the churn project is three aliases, champion for the version taking traffic, challenger for the one being shadow scored, and last_good for the fallback, and that arrangement was simply impossible under stages.

Versioning data and code alongside a model

Here is the gap that catches experienced teams. MLflow versions models. Git versions code. Neither versions your training data, and training data changes constantly, because warehouse tables get backfilled, late arriving rows land, and somebody corrects a mapping. Re run last month script against today warehouse and you get a different model with no indication that anything moved.

Three approaches work, in ascending order of effort. Cheapest is the data_hash tag in the script above: hash the exact training file and log it, which does not let you recover the data but does let you prove two runs used the same input. Middle option is snapshotting, where each training run writes an immutable copy of its input to object storage under the run id, costing storage but making reproduction genuinely possible. Heaviest option is a data version control tool such as DVC or an open table format such as Delta Lake or Iceberg, which give you time travel over the source itself.

My recommendation for most teams is the middle one, and I hold it more firmly than most opinions in this series. Training sets for tabular problems are usually well under a gigabyte, storage is close to free, and a snapshot removes an entire category of argument for a few pence per run. Reach for DVC when your inputs are large binaries such as images or audio, where copying per run stops being free. Skip time travel over warehouse tables as your only mechanism, because retention policies quietly expire the exact history you were relying on, usually at ninety days and usually the week you need it.

Run comparison numbers from the churn project

Once tracking is running, comparison stops being a memory exercise. Below are six runs from the churn experiment, pulled with mlflow.search_runs(order_by=['metrics.cv_auc_mean DESC']) and trimmed to the interesting columns. Precision at the top 5 percent of scores is included because that is the metric the retention team actually acts on, a distinction worth keeping in view after exploratory data analysis has told you what the business cares about.

Read the shape of that curve rather than its endpoint. Moving from 18 baseline columns to 24 windowed ones bought 0.029 AUC. Every estimator change after that bought 0.038 between them, and tuning in Part 15 bought 0.008 for a day of compute. Without run records that ordering is a matter of opinion, and opinions about where accuracy came from tend to favour whoever is telling the story.

Tooling options and what I would pick

My pick is MLflow, and I would start on local SQLite and graduate to a small server backed by Postgres when a second person needs to see the runs. Reasoning is not that MLflow has the nicest interface, because it does not. Weights and Biases is better for watching a training curve and considerably better for large sweeps. MLflow wins on a duller property: tracking and registry live in one tool under Apache 2.0, and the same code runs against a local file, your own server, Databricks or a cloud managed backend by changing a URI. That portability is worth more over a five year horizon than a nicer chart.

Option to avoid is the last row, a bespoke tracking system built on a database table and an S3 bucket. I have seen this attempted twice. Both times it began as a fifty line helper, both times it grew a web interface nobody maintained, and both times it was abandoned inside two years leaving orphaned artifacts that nobody dared delete. Building it is not hard. Keeping it alive after the person who wrote it changes team is the part that never gets costed.

One honest caveat on MLflow. Autologging, enabled by mlflow.autolog(), is genuinely convenient and I still write explicit log calls for the metrics that matter. Autologging records what the library thinks is interesting, which changes between versions, and a metric that silently stops being captured is worse than one that was never captured, because dashboards keep showing the old values. Explicit beats automatic for anything a decision depends on.

Minimum tracking setup to put in place now

For the churn project as it stands, four things and roughly half a day. Install MLflow and point it at SQLite, exactly as in the first script. Add the six log calls to your training entry point and make git_sha and data_hash mandatory tags, failing the run if either is missing rather than logging an empty string. Register the current live model as version 1 and give it the champion alias, then change the batch job and the endpoint to load models:/telco_churn@champion instead of a path. Finally, snapshot each training input to object storage under the run id.

Resist two temptations while doing it. Do not retrofit historical runs, because reconstructing records for experiments nobody will revisit consumes days and produces a fiction. Do not stand up a shared server on day one either. Local SQLite handles thousands of runs comfortably, and a service with no users is a service that rots quietly until the day somebody needs it.

With a registry in place, a new question becomes answerable and therefore urgent. You can now say which version is live and what trained it, but nothing yet tells you whether that version is still any good. Part 25 covers monitoring in production: drift, decay and the silent failure where a model keeps returning confident scores about a world that has moved on. Before reading it, spend twenty minutes adding the tracking layer to your own training script and register whatever model you currently have deployed. Getting version 1 into a registry is the step people postpone, and it is the one that makes everything after it possible.

References

• Model Registry workflows, aliases and tags, MLflow documentation
• MLflow 3 migration guide, including the move from artifact_path to name
• MLflow release notes, version 3.14.0 and earlier
• RFC on deprecating model registry stages, MLflow issue 10336

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts