A retention lead asked me a question in a review that I could not answer on the spot: how would we know if the churn model stopped working? We had dashboards. We had a nightly scoring job with a Slack notification when it failed. What we did not have was any way to distinguish a model producing good predictions from a model producing confident nonsense, because both look identical from outside. Both return a probability between zero and one for every customer, on time, with no errors in the log.

That is the difference between software failure and model failure. Software fails loudly: a null pointer, a timeout, a 500. Machine learning fails silently, and it fails on a timescale of weeks, which is long enough that by the time somebody notices, the business has already spent real money acting on bad scores. Monitoring is the discipline of shortening that gap from a quarter to a week.

Cost of a model that fails quietly

Put a number on it before arguing about tooling. Our churn model scored about 7,000 active accounts a month and the retention team contacted the top decile, roughly 700 customers, with an offer costing about 40 dollars in margin each. When the model was working, that campaign converted well enough to pay for itself twice over. When the model was ranking badly, we were still spending 28,000 dollars a month, just aimed at people who were never going to leave.

Nobody in that chain had a reason to raise a flag. Marketing saw campaigns going out on schedule. Engineering saw a green pipeline. Finance saw the same line item as last quarter. Silent failure is not a technical curiosity, it is a governance hole, and it costs whatever the downstream action costs multiplied by however long it takes somebody to get suspicious.

Four things that break after deployment

Lumping every post deployment problem under the word drift is how teams end up with monitoring that catches the cheap failures and misses the expensive one. Separate them, because they have different detection signals and different response times.

Data quality failure means the inputs are broken rather than shifted: a column arriving as all nulls after an upstream schema change, a currency field switching from dollars to cents, a join silently going from inner to left and filling half your rows with missing values. This is the most common failure by a wide margin and the cheapest to detect, because you are checking schema, null rates and ranges, not distributions. It is also the one that produces the most spectacular damage, since a feature that is suddenly all zeros will move every prediction at once.

Covariate drift, also called data drift, means your input distribution has moved while the underlying relationship holds. Your customer base skews younger, or a marketing push brings in a segment with shorter tenure. Notation helps here: the distribution of inputs P(X) changes but the mapping P(y given X) is stable. A model can survive this reasonably well if the new region of feature space was represented in training, and it degrades badly if it was not.

Prediction drift means the distribution of your model outputs has moved. Average predicted churn probability climbing from 0.18 to 0.29 over six weeks is worth a look. This signal is cheap because it needs no labels and only one column, and it catches a lot of upstream breakage indirectly, since broken inputs usually push outputs somewhere strange.

Concept drift means P(y given X) itself has changed. Same inputs, different correct answer. This one is expensive, slow to detect, and the only reliable signal for it is realised performance measured against actual labels. No amount of input monitoring will find it. Anyone who tells you their drift dashboard covers concept drift has not thought about it carefully.

Ordering matters more than most teams realise. Layer one is nearly free and catches maybe 70 percent of real incidents, so it runs first and hard blocks the batch. Layer five is the only honest signal but arrives weeks late, so it can never be your first line of defence. Everything between is a proxy of decreasing cheapness and increasing usefulness.

Drift detection on churn features with Evidently

Last part we put every training run into MLflow and promoted a registered model version with an alias. This part takes that same deployed churn model, built on the IBM Telco Customer Churn dataset, and asks whether the data arriving today still looks like the data it was trained on. Concretely: we freeze a reference window, compare each new scoring window against it, and quantify the gap.

Evidently is the fastest way to get a defensible drift report without writing statistics yourself. It picks a detection method per column automatically based on column type and sample size, and reports overall dataset drift as the share of drifting columns, flagging the dataset when at least half of them move.

# monitor_drift.py
# Tested against evidently 0.7.21, pandas 2.3.3, numpy 2.3.2, Python 3.11.9
import pandas as pd
from evidently import Report
from evidently.presets import DataDriftPreset

ref = pd.read_csv('data/churn_reference_2025Q4.csv')
cur = pd.read_csv('data/churn_scored_2026Q2.csv')

COLS = ['tenure', 'monthly_charges', 'total_charges', 'contract_type',
        'payment_method', 'internet_service', 'tech_support',
        'support_tickets_90d']

report = Report([DataDriftPreset(columns=COLS)], include_tests=True)
result = report.run(cur[COLS], ref[COLS])
result.save_html('drift_2026Q2.html')
print(result.dict()['metrics'][0])

Import paths were reorganised in the Evidently 0.7 line. If from evidently.presets import DataDriftPreset raises an ImportError on your install, check what you actually have with pip show evidently before assuming the code is wrong. [VERIFY against your installed minor version]

Running that against our second quarter window returned 2 of 8 columns drifting, which is below the default dataset level threshold, so the preset reported no dataset drift. Read that carefully, because it is a trap. Dataset level drift is a blunt summary. Two drifting columns matter enormously if they are the two features your model leans on hardest, and matter not at all if they are features the model barely uses. Permutation importance from Part 11 is what tells you which case you are in.

For a number you can put a threshold on and explain to a risk committee, I compute the population stability index, or PSI, per feature. PSI compares binned proportions between reference and current and sums the divergence. Under 0.1 is stable, 0.1 to 0.2 is worth watching, above 0.2 warrants investigation. Twenty lines of numpy gets you there and you own every step.

import numpy as np

def psi(reference, current, bins=10, eps=1e-6):
    """Population stability index for a numeric column."""
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    ref_pct = np.histogram(reference, bins=edges)[0] / len(reference)
    cur_pct = np.histogram(current, bins=edges)[0] / len(current)
    ref_pct = np.clip(ref_pct, eps, None)
    cur_pct = np.clip(cur_pct, eps, None)
    return float(np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)))

for col in ['tenure', 'monthly_charges', 'support_tickets_90d']:
    print(f'{col:24s} {psi(ref[col], cur[col]):.3f}')

tenure                   0.041
monthly_charges          0.312
support_tickets_90d      0.409

Note the two clipping lines. Leave them out and the first low volume category that appears in current but not reference will give you a zero denominator inside the logarithm:

RuntimeWarning: divide by zero encountered in log
  return float(np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)))
contract_type            inf

An infinite PSI is not a catastrophic drift event, it is an empty bin. It is a warning rather than an exception, so it will not stop your pipeline, and if your alerting compares PSI against a numeric threshold then infinity fires an alarm every single run until someone stops trusting the alarm. That small epsilon is doing quiet, important work.

Estimating performance while labels are still in flight

Here is the constraint that makes model monitoring hard. Our churn label is defined as departure within the next 60 days, so a prediction made on 1 April cannot be scored until 31 May. Add a data warehouse lag and a reconciliation cycle and realised performance for April lands in mid June. Ten weeks of blindness on the only signal that catches concept drift.

NannyML narrows that window with confidence based performance estimation, CBPE. Idea is simple once you see it: if your model outputs well calibrated probabilities, meaning that among all customers scored at 0.30 roughly 30 percent actually churn, then those probabilities alone let you compute expected confusion matrix cells, and from those cells any metric you like. No labels required.

# estimate_performance.py
# Tested against nannyml 0.13.1, pandas 2.3.3, Python 3.11.9
import nannyml as nml
import pandas as pd

reference = pd.read_csv('data/churn_reference_scored.csv')  # has labels
analysis = pd.read_csv('data/churn_analysis_scored.csv')    # no labels yet

estimator = nml.CBPE(
    problem_type='classification_binary',
    y_pred='prediction',
    y_pred_proba='churn_proba',
    y_true='churn',
    timestamp_column_name='scored_at',
    metrics=['roc_auc'],
    chunk_period='M',
)
estimator.fit(reference)
results = estimator.estimate(analysis)
print(results.to_df()[['roc_auc']].round(3).head())

Two constraints matter more than the code. First, the reference set must not be the training set of the monitored model, because NannyML fits a probability calibrator on reference data and fitting it on training data introduces bias. Hold out a clean slice. Second, and this is the part people skip: CBPE handles covariate shift well and does not handle concept drift at all. If P(y given X) changes, the model keeps emitting confident probabilities, CBPE believes them, and your estimate stays flat while reality falls away underneath it.

Alert thresholds people will not mute

My first monitoring setup failed for a reason that had nothing to do with statistics. We ran a Kolmogorov Smirnov test per feature at p below 0.05 across 41 columns, on daily windows of about 7,000 rows. With 41 independent tests at a 5 percent false positive rate you expect two false alarms a day from pure chance, and large samples make the test sensitive to differences too small to matter. Reality delivered six to eight alerts every morning. Inside three weeks the channel was muted, and a muted channel is worse than no channel because it looks like coverage on an audit checklist.

Rebuilding it, I changed three things. Effect size replaced significance, so PSI above 0.2 rather than a p value, because PSI does not get more trigger happy as your sample grows. Scope narrowed from all 41 features to the 8 carrying the most permutation importance, since drift in a feature the model ignores is trivia. Windows widened from daily to weekly, which smoothed out day of week seasonality that was generating most of the noise anyway. Alert volume went from roughly 45 a week to roughly one a fortnight, and people started reading them.

Set thresholds against the cost of the action, not against a number you read in a blog post. Retraining a churn model costs about half a day and carries deployment risk, so our retrain floor sits at an AUC of 0.78, chosen because campaign economics stop working below roughly 0.76. A fraud model where a miss costs thousands would sit far tighter. Write the reasoning down next to the threshold, because in nine months somebody will ask why it is 0.78 and we picked it in a meeting is not an answer that survives a model risk review.

Twelve months of churn monitoring numbers

Numbers from the churn project make the argument better than any diagram. Below is a monthly monitoring record: highest feature PSI, mean predicted probability, CBPE estimate, and realised AUC once labels landed. Reference window is the fourth quarter of the prior year.

Read the columns against each other. Mean predicted probability climbed steadily from month 5, which was our earliest visible signal and needed no labels and no library at all. Max PSI lagged behind it. Gap between estimate and reality crossed a meaningful band at month 7, two full months before feature drift became loud enough to trip a 0.2 threshold. Cheapest signal in that whole table is the third column, and it is the one most teams never plot.

Second table is the one I actually put in front of stakeholders, because it converts signals into owned actions. A monitor with no named response is decoration.

Tooling choices and what I would avoid

Evidently is what I reach for first on drift and data quality. It covers a wide metric surface, produces a report a non specialist can read, and runs offline against two dataframes, so you can bolt it onto an existing batch job in an afternoon. Cost is that its report objects are not a time series store, so you still need somewhere to persist the numbers.

NannyML earns its place only when your label lag is genuinely painful, which for churn it is. If your labels arrive within a day, skip performance estimation entirely and measure the real thing. Adding a calibration layer and a second library to shorten a one day gap is complexity with no payoff.

What I would avoid is buying a full observability platform as your first move. Every team I have watched do that ended up with a beautiful dashboard nobody opened, because the hard part was never rendering the chart, it was deciding what a threshold breach obliges somebody to do. Get the five signals in the table above landing in a database and a weekly message, prove somebody acts on them, and only then pay for a platform. Also avoid monitoring every feature you have. Sensitivity you gain is real, and it is dwarfed by the alert fatigue you buy.

One judgement call worth naming: we chose not to auto retrain on threshold breach, and I still think that was right. Automatic retraining on a drift alert is how a data quality bug becomes a permanently poisoned model, because the retrain happily learns the broken feature. Retraining triggers a review, a human looks at whether the drift is real, and only then does a pipeline run. Two days slower, and it has saved us at least twice.

Monitoring layers to put in place this week

If you take one thing from this part, make it the ordering. Start with schema and null checks that hard block a bad batch, because they are an afternoon of work and catch most real incidents. Add a daily log of mean predicted probability and row count next, since it needs no library and gives you the earliest cheap warning. Layer weekly PSI on your top handful of features third, with an effect size threshold rather than a p value. Add performance estimation fourth, and only if your label lag runs to weeks. Track realised performance last, and track the gap between estimated and realised as its own series, because that gap is your only view of concept drift.

Next part takes this further into automation: continuous integration and continuous delivery for machine learning pipelines, where the tests that gate a deploy include model quality checks and not just unit tests. Monitoring tells you a model went bad. CI/CD is how you get the replacement out safely without repeating the incident that produced it.

Your move this week: open whatever model you have in production, and write down how you would find out it had stopped working. If the honest answer involves somebody eventually complaining, you have a monitoring gap and now you have a five step plan for closing it.

References

• Data Drift Preset, method selection and the 50 percent dataset drift default, Evidently documentation
• Confidence based performance estimation, assumptions and limitations, NannyML 0.13.1 documentation
• Evidently source and release history, version 0.7.21
• scikit-learn release history, current stable 1.9.0
• IBM Telco Customer Churn dataset, used throughout this series

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts