A line I still think about, printed at the bottom of a green continuous integration run on a machine learning repository: 1 passed in 0.41s. Four tenths of a second. Whatever that pipeline had just certified as safe to merge, it had not trained a model, had not scored a row, and had not looked at a single feature. It had imported a module and checked that a function existed.

That team had continuous integration in the sense that a badge on their README was green. What they did not have was any automated opinion about whether a change made the model better or worse. Every merge was a guess, and every guess was settled weeks later by a stakeholder noticing something odd in a report. Closing that gap is what this part is about.

Differences between software CI and machine learning CI

Continuous integration, CI for short, means every change is automatically built and tested before it lands on the main branch. Continuous delivery, the CD half, means the tested artefact is automatically packaged and made ready to release. Both ideas transfer to machine learning, but they transfer badly if you copy the web application version wholesale.

Three things break. First, a software test is deterministic: given this input, assert this exact output. A model test is statistical. Retrain the same code on the same data with a different random seed and your area under the curve moves by half a point, so an assertion on an exact score will fail constantly and get deleted within a fortnight. Second, correctness depends on an input your repository does not contain. Code can be perfect and the model still worthless because a feature table upstream started arriving half empty. Third, the expensive part of the build is not compilation, it is training, and training costs real money on real hardware, which means you cannot run the full pipeline on every commit the way you run a unit test suite.

Those three constraints produce three design rules that shape everything below. Assert on ranges and on relative comparisons, never on exact numbers. Test the data as seriously as you test the code. Tier the pipeline by cost so that cheap checks fail fast and expensive checks run rarely.

Stages of a machine learning pipeline

Where the churn project stands: Part 25 left us with a deployed model and five layers of monitoring watching it decay. What we never built was the road a replacement model travels to get there. Right now, if monitoring fires and says retrain, somebody opens a notebook. This part replaces that somebody with a pipeline.

Each stage earns its place by catching a class of failure that no earlier stage can see. Below is what each one caught on our churn repository across roughly nine months of pull requests, with the median wall clock time it added.

Read that last column carefully, because it argues against intuition. Lint caught five times as many failures as the full training comparison, and lint is trivial. Yet those six expensive catches included two data leakage bugs that would have shipped a model with a flattering offline score and no real predictive power. Frequency and severity are unrelated here. Both lanes have to exist.

Tests that catch model bugs before merge

Four families of test are worth writing, and they escalate in cost. Transform tests take a tiny hand built frame and assert what a single function does to it. Schema tests assert that the incoming data has the columns, types and ranges you expect, which is the automated version of the cleaning discipline covered in cleaning messy data. Behavioural tests assert that the model responds sensibly to a manipulated input: raise a customer tenure by two years and predicted churn probability should not go up. Quality gates compare the candidate against the incumbent.

Here is the gate itself, which is the test most teams get wrong. Note what it does not do: it does not compare against a stored number.

# tests/test_model_quality.py
# Tested against scikit-learn 1.9.0, pandas 2.3.3, pytest 8.4.2, Python 3.12.7
import pandas as pd
import pytest
from sklearn.metrics import roc_auc_score

from churn.pipeline import build_pipeline, load_incumbent

MIN_GAIN = -0.005   # candidate may lose at most half a point of AUC

@pytest.fixture(scope='module')
def split():
    df = pd.read_parquet('data/snapshot_2026_06.parquet')
    train = df[df['cohort_month'] < '2026-04']
    test = df[df['cohort_month'] >= '2026-04']
    return train, test

def test_candidate_not_worse_than_incumbent(split):
    train, test = split
    y = test['churned']

    candidate = build_pipeline().fit(train.drop(columns=['churned']),
                                     train['churned'])
    incumbent = load_incumbent().fit(train.drop(columns=['churned']),
                                     train['churned'])

    cand_auc = roc_auc_score(y, candidate.predict_proba(
        test.drop(columns=['churned']))[:, 1])
    inc_auc = roc_auc_score(y, incumbent.predict_proba(
        test.drop(columns=['churned']))[:, 1])

    print(f'candidate {cand_auc:.4f} incumbent {inc_auc:.4f}')
    assert cand_auc - inc_auc >= MIN_GAIN

Expected output on a passing run, with pytest called using the s flag so the print survives:

$ pytest tests/test_model_quality.py -s
================= test session starts =================
platform linux -- Python 3.12.7, pytest-8.4.2
collected 1 item

tests/test_model_quality.py candidate 0.8471 incumbent 0.8402
.
================== 1 passed in 214.86s ================

Two hundred and fifteen seconds, not four tenths. That number is the whole point. A model quality test that returns instantly is testing something other than model quality.

GitHub Actions workflow for the churn project

Below is the fast lane, which is the file you should write first. It runs on every push. Version pins matter here more than people expect, so I state them: actions/checkout at v5, actions/setup-python at v6, which upgraded its runtime to Node 24 and requires a runner on 2.327.1 or later.

# .github/workflows/ml-ci.yml
name: ml-ci
on:
  push:
    branches-ignore: [main]
  pull_request:
    branches: [main]

jobs:
  fast-lane:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v6
        with:
          python-version: '3.12'
          cache: pip
      - name: Install project
        run: pip install -e '.[dev]'
      - name: Lint
        run: ruff check src tests
      - name: Unit and schema tests
        run: pytest tests -m 'not slow' -q

  slow-lane:
    if: github.event_name == 'pull_request'
    needs: fast-lane
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v6
        with:
          python-version: '3.12'
          cache: pip
      - run: pip install -e '.[dev]'
      - name: Train and gate
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}
        run: pytest tests/test_model_quality.py -s

Two details carry most of the value. needs: fast-lane means the twenty minute job never starts if lint failed in eleven seconds, which saves both money and patience. And branches-ignore: [main] paired with a pull request trigger stops every merge commit from re-running work that already ran on the branch.

Now the failure you will hit, probably on your first run. Everything works locally, and the runner produces this:

ImportError while loading conftest '/home/runner/work/churn/churn/tests/conftest.py'.
tests/conftest.py:4: in <module>
    from churn.pipeline import build_pipeline
E   ModuleNotFoundError: No module named 'churn'
Error: Process completed with exit code 4.

Cause: your laptop resolves churn because you happen to run pytest from the repository root and Python puts the working directory on the path. A runner with a src layout does not. Fix: the pip install -e . step above, which is why it appears before any test step, plus a [tool.setuptools.packages.find] entry in pyproject.toml pointing at src. Exit code 4 specifically means pytest could not collect, not that a test failed, and confusing those two costs people an afternoon.

Continuous training and retrain triggers

Continuous training, sometimes written CT, is the third loop: a pipeline that produces a new model without anyone changing code, because the data changed. Three triggers are defensible. On a schedule, which is simplest and wasteful. On a monitoring signal, when the drift or performance layers from Part 25 cross a floor. On new labelled data volume, when enough fresh outcomes have accumulated to be worth learning from.

My preference, and this is a verdict rather than a survey: trigger on the monitoring signal, with a scheduled monthly run as a floor so the pipeline never goes stale from disuse. Avoid pure schedule triggers on a weekly cadence. We ran weekly retraining for a quarter on the churn model and produced thirteen models, of which one was meaningfully different from its predecessor. That is twelve rounds of compute, review and registry clutter bought for nothing, and it trains the team to stop reading retrain notifications.

Notice how narrow the useful band is. Across twelve pull requests the whole range of candidate scores spans about three points of area under the curve, and the gate sits five thousandths below the incumbent. Set that tolerance too tight and normal seed noise blocks good work; set it loose and slow erosion walks in one merge at a time. Half a point has held up for me across three different churn style models, but measure your own seed variance before copying the number.

Promotion gates and rollback

A passing gate does not mean a model belongs in production. It means a model belongs in the registry. Those are different claims, and collapsing them is how teams end up with an automated pipeline that quietly shipped a model nobody reviewed to a system that declines credit applications.

Shadow deployment is the step people skip and later regret. You route real production traffic to the candidate, record its predictions, act on none of them, and compare against the live model for a week. Training serving skew shows up here and almost nowhere else, which is exactly the failure mode covered in Part 23. A candidate that matched the incumbent offline and disagrees with it on nine percent of live customers is telling you something your held out test set could not.

Rollback deserves its own rehearsal. Restoring a model is not restoring a container image, because the model needs its feature transformations, its encoder vocabularies and its expected schema, all at the same version. If those live in three places, rollback is a thirty minute scramble at exactly the wrong moment. Store them as one versioned artefact and rehearse the restore quarterly. Ours takes four minutes because we practised it; the first attempt took fifty one and needed two people.

Data versioning and reproducible runs

Git tracks your code. It does not track the eight hundred megabyte Parquet file your model trained on, and it should not try. That gap is where most machine learning pipelines quietly lose reproducibility, because a run recorded as commit abc123 is only reproducible if you also know which version of the data commit abc123 saw.

Two approaches solve it and they suit different teams. Simplest is a content hash: write the snapshot to object storage under a path derived from its own checksum, commit only that short hash in a config file, and have your pipeline fetch by hash. No new tool, no new daemon, and any engineer can read what happened from a diff. Second is a dedicated data versioning layer such as DVC, which stores pointer files in Git alongside your code and reproduces a whole dependency graph of stages with a single reproduce command. That buys you cached intermediate stages, which matters once feature building takes longer than fitting.

For a single churn model with one feature build step, I would use the content hash and skip the tool. Adoption cost is close to zero and there is no second system to keep alive. Move to a versioning layer when you have several models sharing intermediate feature tables, because that is the point where recomputing everything on every run stops being affordable and cache invalidation stops being something a person can reason about correctly.

Whichever you pick, record the data version in the same run record as the model metrics. We had a fortnight of confusion on the churn project because two runs logged identical hyperparameters and different scores, and the answer, once somebody thought to check, was that a nightly job had rewritten the source table between them. Anything from the analyst toolkit applies here too: the same discipline behind a reliable aggregation query is what stops a training pull silently changing shape.

Runtime and cost of a CI pipeline

Pipeline economics decide whether the pipeline survives contact with a busy team. Anything over about ten minutes on the fast lane and people start pushing without waiting; anything over roughly forty on the slow lane and they merge on Friday afternoon to avoid it. Below is where our minutes actually went before and after a deliberate optimisation pass.

Both wins were unglamorous. Dependency caching through the cache: pip input on setup-python took one line and removed 144 seconds. Replacing a live SQL pull with a pinned Parquet snapshot removed another 144 and, more importantly, made the pipeline reproducible, because a query against a moving table gives a different training set every run and turns your gate into a coin flip. Training stayed exactly where it was, and that is correct: shortening training by sampling would make the gate less trustworthy, which is the one thing you cannot trade away.

Pipeline to build first for a churn model

If you build one thing this month, build the fast lane: lint, unit tests on your transforms, and schema checks against a pinned snapshot, wired to run on every push and finish inside three minutes. It is cheap, it catches most of what goes wrong, and it establishes the habit of a pipeline having an opinion. Add the quality gate second, and write it as a live comparison against a retrained incumbent from the very first version, because a hardcoded threshold rots quietly and you will not notice for months.

What I would avoid: fully automated promotion to production. Registry automation, yes, all of it. Deployment automation, yes, with shadow traffic in front. But keep a person on the transition into production for anything that touches revenue, pricing or a regulated decision, and make sure the pipeline hands that person a comparison they can read in a minute rather than a link to a dashboard. Part 27 steps back from the pipeline to the platform underneath it, where these choices turn into architecture, and where you can follow the whole route on the series guide.

One thing to do this week: open your model repository and time your test suite. If it finishes in under ten seconds, your pipeline is not testing the model, and you now know exactly which test to write first.

References

• actions/setup-python release history, current v6.2.0 and the Node 24 runner requirement in v6.0.0
• Workflow syntax reference for needs, if and timeout-minutes, GitHub Actions documentation
• scikit-learn release history, current stable 1.9.0 released June 2026
• MLflow release notes, including the pytest integration for gating quality in CI
• DVC pipelines and dvc repro, Data Version Control documentation
• IBM Telco Customer Churn dataset, used throughout this series

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts