On the holdout it scored 0.94. Six weeks after I put it in front of real clusters, measured against incidents I had finally labelled, it scored 0.71, and not one line of it had errored. My service was healthy, latency was flat, the pager was quiet, and the model was wrong more than a quarter of the time. That gap is the subject of this part. A served model is not a shipped model, it is a running one, and a running model decays quietly unless you point the same monitoring discipline at its predictions that you already point at your fleet.

Three kinds of drift, one monitoring job

Where the project stands: last part we served the classifier two ways, a batch scorer and an endpoint. This part watches the served model, because up and fast is not the same as correct. Monitoring a model splits into three questions, and only one of them needs labels, which matters because labels are the thing you almost never have in time.

Data drift is a shift in the inputs, the cpu and memory distributions moving away from the month the model learned on. Prediction drift is a shift in the outputs, the mix of risk scores changing even before you know if they are right. Concept drift is the hard one, the inputs look the same but the mapping from them to a real incident has changed, so yesterday healthy readings now precede failures. Each has an infrastructure analogue you already reason about, and each points at a different first response.

flowchart LR
  M[Served model] --> L[Log inputs and predictions]
  L --> F[Feature drift check]
  L --> P[Prediction drift check]
  G[Labels arrive later] --> Q[Quality check]
  F --> A[Alert only if quality at risk]
  P --> A
  Q --> A
  A --> R[Retrain or investigate]

Logging inputs and predictions before labels arrive

Every drift check compares two samples, so the first job is to keep the current one. Append each scored row to a log, the inputs, the prediction, a timestamp and the model version, and you have the raw material for all three checks. This is one function bolted onto the batch scorer from Part 17.

# log_predictions.py  tested with python 3.12, pandas 2.2.3
import os, datetime as dt, pandas as pd

def log_scored(frame, preds, model_version, path='pred_log.csv'):
    out = frame.copy()
    out['prediction'] = preds
    out['model_version'] = model_version
    out['scored_at'] = dt.datetime.now(dt.timezone.utc).isoformat()
    out.to_csv(path, mode='a', header=not os.path.exists(path), index=False)
    return len(out)

n = log_scored(new_windows, preds, 'incident_clf-v3')
print('logged', n, 'scored rows to pred_log.csv')

logged 100000 scored rows to pred_log.csv

Two small choices save pain later. That timestamp is timezone aware and in UTC, because a naive local one will bite you the moment you resample by day across a clock change, a bug the time series parts come back to. And the model version rides on every row, so when you retrain you can tell which model made which call without guessing. Store the log wherever your other operational data lands; a CSV is fine to start, a warehouse table is better once it grows.

Feature drift with a two sample test

To ask whether one feature has moved, compare its distribution now against its distribution in the reference window. The two sample Kolmogorov Smirnov test does exactly this, it measures the largest gap between the two cumulative distributions and returns a statistic and a p-value, and it assumes nothing about the shape of the data, which suits messy telemetry. scipy has it. Here is the first cut, and it hides a failure I have shipped.

# drift_ks.py  tested with python 3.12, scipy 1.14.1, pandas 2.2.3, numpy 2.1.3
import pandas as pd
from scipy.stats import ks_2samp

ref = pd.read_csv('metrics_train.csv')     # the month the model trained on
cur = pd.read_csv('metrics_2026_07.csv')   # last 30 days in production

for col in ['cpu_p95', 'mem_p95', 'load1_mean']:
    res = ks_2samp(ref[col], cur[col])
    flag = 'DRIFT' if res.pvalue < 0.05 else 'ok'
    print(f'{col:11s} ks={res.statistic:.3f} p={res.pvalue:.4f} {flag}')

cpu_p95     ks=0.147 p=0.0000 DRIFT
mem_p95     ks=0.041 p=0.2013 ok
load1_mean  ks=nan p=nan ok

load1_mean came back nan because the production export had gaps where a node exporter missed scrapes, and ks_2samp propagates NaN rather than inventing a value. Worse, the flag read ok, because nan < 0.05 evaluates to False, so a broken check passed silently, which is the exact failure mode this whole part exists to stop. Do not paper over it by filling zeros, that would fake a distribution. Drop the missing values per column, run the test on what is left, and surface the missing rate as its own signal.

for col in ['cpu_p95', 'mem_p95', 'load1_mean']:
    a, b = ref[col].dropna(), cur[col].dropna()
    missing = cur[col].isna().mean()
    res = ks_2samp(a, b)
    flag = 'DRIFT' if res.pvalue < 0.05 else 'ok'
    print(f'{col:11s} ks={res.statistic:.3f} p={res.pvalue:.4f} miss={missing:.0%} {flag}')

cpu_p95     ks=0.147 p=0.0000 miss=0% DRIFT
mem_p95     ks=0.041 p=0.2013 miss=0% ok
load1_mean  ks=0.203 p=0.0000 miss=12% DRIFT

Now cpu_p95 and load1_mean both flag, and the 12 percent missing on load1_mean is a second thing to chase, likely the same exporter gap. One caution before you wire a p-value into an alert. KS p-values are sensitive to sample size, and at 100,000 rows a shift far too small to matter still lands under 0.05, so the p-value tells you a shift exists, not that it is worth waking up for. That is why the next step turns the shift into a magnitude.

Population stability index in one number

The population stability index, PSI, bins both samples the same way and sums how far each bin has moved, weighted by the log of the ratio. It hands back a single comparable number per feature, which is exactly what a threshold and a trend line need. Credit risk teams have leaned on it for years, and the bands are well worn: under 0.1 is stable, 0.1 to 0.25 is a moderate shift worth watching, 0.25 and above is a real move. Here is a thirty line version, with the bug that greets almost everyone who writes it fresh.

# psi.py  tested with python 3.12, numpy 2.1.3
import numpy as np

def psi(expected, actual, bins=10):
    cuts = np.quantile(expected, np.linspace(0, 1, bins + 1))
    cuts[0], cuts[-1] = -np.inf, np.inf
    e = np.histogram(expected, cuts)[0] / len(expected)
    a = np.histogram(actual, cuts)[0] / len(actual)
    return float(np.sum((a - e) * np.log(a / e)))

print('cpu_p95 psi:', round(psi(ref['cpu_p95'], cur['cpu_p95']), 3))

psi.py:9: RuntimeWarning: divide by zero encountered in log
cpu_p95 psi: inf

A production bin went empty. One cpu band that held samples during training saw none this month, so its actual share was 0, log of 0 over the expected share is negative infinity, and the sum collapses to inf. An empty bin is genuine signal, the distribution really moved, but inf is useless as a threshold input. The standard fix is to floor both shares at a tiny epsilon before the log, so an empty bin contributes a large finite number instead of infinity.

def psi(expected, actual, bins=10, eps=1e-6):
    cuts = np.quantile(expected, np.linspace(0, 1, bins + 1))
    cuts[0], cuts[-1] = -np.inf, np.inf
    e = np.clip(np.histogram(expected, cuts)[0] / len(expected), eps, None)
    a = np.clip(np.histogram(actual, cuts)[0] / len(actual), eps, None)
    return float(np.sum((a - e) * np.log(a / e)))

for col in ['cpu_p95', 'mem_p95', 'load1_mean']:
    print(f'{col:11s} psi={psi(ref[col].dropna(), cur[col].dropna()):.3f}')

cpu_p95     psi=0.291
mem_p95     psi=0.043
load1_mean  psi=0.377

Read against the bands, cpu_p95 at 0.29 and load1_mean at 0.38 have both crossed into a real shift, while mem_p95 at 0.04 sits still. One number per feature, comparable from week to week, is what makes PSI the field standard for drift monitoring and a better alert input than a raw p-value. This psi function is the reference artifact of this part, no dependency past numpy, and it drops straight into the scheduled scorer so the number lands next to the predictions on every run.

Prediction drift and delayed ground truth

Inputs are only half the picture. Watch the output too, because the share of windows the model flags is visible immediately and moves fast when something breaks. If the flagged rate doubles overnight with no matching jump in real incidents, suspect a broken input or a shifted world before you believe failures actually tripled. Then, when labels finally arrive from the incident tracker, score the delayed slice for real and compare it to training.

# prediction_drift.py  tested with scikit-learn 1.6.1, pandas 2.2.3
import pandas as pd
from sklearn.metrics import precision_score, recall_score

log = pd.read_csv('pred_log.csv', parse_dates=['scored_at'])
daily = log.set_index('scored_at')['prediction'].resample('1D').mean()
print('share flagged, last 5 days:')
print(daily.tail())

# labels for a past month arrive from the incident tracker
labeled = pd.read_csv('confirmed_2026_06.csv')   # prediction and actual, one month old
print('precision', round(precision_score(labeled['actual'], labeled['prediction']), 3))
print('recall   ', round(recall_score(labeled['actual'], labeled['prediction']), 3))

share flagged, last 5 days:
scored_at
2026-07-27    0.066
2026-07-28    0.071
2026-07-29    0.180
2026-07-30    0.176
2026-07-31    0.172
Name: prediction, dtype: float64
precision 0.62
recall    0.55

Two readings, one story. The flagged share jumped from about 7 percent to 18 percent on the 29th with no matching rise in confirmed incidents, which is prediction drift and almost always means an input moved, not that the world got three times more dangerous. And the delayed June labels confirm the quiet decay, precision 0.62 and recall 0.55 against a training precision near 0.90, numbers that were never once visible on the serving dashboard. This is the whole point, drift monitoring is observability pointed at prediction quality, the same reflex as tracing a latency regression, aimed one layer up. For the full statistical treatment of drift and model decay, the Data Science Series goes deeper than this operator cut does.

A drift report with Evidently

Hand rolled KS and PSI are enough to start and carry no dependency to pin, but a library gives you a full report, per column visuals and a pass or fail test suite for stakeholders. Evidently is the common pick. Its API changed at the 0.6 to 0.7 boundary, and nearly every tutorial still online shows the old one, so the first thing most people hit is an import error.

# the version most blog posts still show, breaks on evidently 0.7.x
from evidently import Report
from evidently.metric_preset import DataDriftPreset
report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=ref, current_data=cur)

ModuleNotFoundError: No module named 'evidently.metric_preset'

The preset moved to evidently.presets, and Report now takes a positional list of metrics with run called as current then reference, not the old metrics and reference_data keywords. A working version on a current install is short.

# evidently_drift.py  tested with evidently 0.7.x, pandas 2.2.3
from evidently import Report
from evidently.presets import DataDriftPreset

report = Report([DataDriftPreset()])
my_eval = report.run(cur, ref)          # current first, then reference
my_eval.save_html('drift_report.html')
print('drift report written, open drift_report.html')

drift report written, open drift_report.html

Evidently picks a drift method per column by default, KS for numeric columns with enough samples, with PSI and others available, and marks the dataset as drifted when at least half the columns move. Reach for it when you want the report and the test suite; keep the hand rolled checks for the numbers you page on.

A retraining trigger you can defend

Drift is not a failure, it is the expected weather of a model in production, so the useful output of monitoring is not a graph, it is a decision that either fires or does not. Combine the signals into a small table you can point at when someone asks why you retrained, or why you did not.

Write your own numbers down and commit to them. Mine were PSI 0.25 on cpu_p95 or mem_p95, or a 10 point precision drop once labels landed, whichever came first, and retraining was a scheduled job with its own budget, not a scramble. That is the difference between a wall of graphs nobody reads and a monitor that makes one clear call.

Point your existing monitoring at prediction quality

None of this is a new discipline. You already log signals, set baselines, and page when a number crosses a line you chose in advance. Drift monitoring is that same habit aimed one layer up, at the quality of a prediction instead of the health of a process, and your instinct to alert on the thing that actually hurts, not on every wiggle, is the correct one to carry over. The only genuinely new idea is that the most important metric, accuracy, shows up late, so you lean on input and prediction drift to see trouble coming.

Next part turns these scored, monitored predictions into a pipeline, because a check you run by hand is a check you will forget, and continuous integration for models is the same automation you already write for everything else.

References

• scipy.stats.ks_2samp, two sample Kolmogorov Smirnov test
• numpy.histogram, binning for the PSI calculation
• Evidently, Data Drift preset and default methods
• Data Science Series, Monitoring Machine Learning Models, Drift and Decay
• AI Engineering Series, LLM Observability, Tracing and Debugging

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts