I built a dataset of 3,000 columns of pure random numbers, containing no signal of any kind, and cross validated a model on it. Mean AUC came back at 0.8110.

Nothing was wrong with the cross validation code. Five folds, stratified, a clean pipeline inside each fold. One step happened in the wrong place, and that single misplacement was enough to conjure an AUC of 0.81 out of a random number generator. Move that step inside the loop and the same experiment returns 0.5185, which is the honest answer for noise. That gap between 0.81 and 0.52 is the subject of this part.

Where the churn project stands after Part 10

Last part we fitted a logistic regression to the Telco churn data, read its coefficients as odds ratios, and picked a threshold by weighing a 15 dollar retention call against 500 dollars of saved margin. We finished with a scored customer list and one number describing model quality: an AUC of 0.853 on a held out split of about 120 customers. This part asks whether that number deserves the confidence we placed in it, and the answer turns out to be no.

Nothing changes about the model itself here. Same features from Part 6, same pipeline, same 486 row working extract of the public Telco Customer Churn file. What changes is how we measure it. By the end we will have replaced a single fragile number with a mean, a spread, a metric chosen deliberately, and a protocol that resists the three ways evaluation quietly lies to you.

Instability of a single train test split

Start with an experiment that takes four lines and changes how you read every score you have ever reported. Keep the model, the features and the split ratio identical. Change only random_state, which decides which customers land in the test set, and refit five times.

Nine points of AUC separate the luckiest split from the unluckiest, and neither is more correct than the other. Both used the same model on the same 486 customers. If you fit a model on Monday with seed 1, tried a competitor on Tuesday with seed 42, and concluded the second architecture was better, you would have been reading the seed rather than the architecture. I have watched that exact comparison drive a genuine build decision, and the giveaway was that the winning margin, around four points, was smaller than the split noise nobody had measured.

Small test sets drive this. Ours holds roughly 121 customers of whom about 33 churn, so moving a handful of hard cases across the boundary shifts the score noticeably. Scale is the cure that most teams do not have. Below roughly 10,000 rows, treat any single split score as provisional, and go straight to cross validation.

Choosing a metric that matches the decision

Before measuring anything more carefully, settle what you are measuring. Metrics are not interchangeable quality scores. Each answers one specific question, and a metric that answers a question nobody asked is worse than no metric at all, because it carries the authority of a number.

Our churn base rate is 27.16 percent. A model that predicts nobody ever leaves scores 72.84 percent accuracy. Our actual model scores 74.50 percent. Framed that way, months of feature engineering bought 1.7 accuracy points, which sounds like failure. Framed by AUC it scores 0.8306 against a coin flip baseline of 0.5, which sounds like success. Same model, same data, opposite stories, and only one of those framings is useful to a retention manager.

Average precision deserves more attention than it usually receives, because its baseline moves with your class balance. Random ranking scores an average precision equal to the positive rate, which here is 0.2716. Our 0.6121 therefore represents a genuine lift, but quote 0.6121 in a room without stating that baseline and half of them will privately compare it to AUC and conclude the model got worse. On a rare event problem with a 1 percent positive rate, an average precision of 0.15 can be excellent work. Context is doing all the interpretive labour.

Cross validation, and reading the fold spread

Cross validation replaces one arbitrary split with several. Divide the data into k folds, hold each fold out in turn, train on the remaining k minus 1, and collect k scores. Every row gets scored exactly once by a model that never saw it. Stratified folds preserve the class balance in each one, which matters at a 27 percent positive rate because an unlucky ordinary fold could arrive noticeably light on churners.

# pinned environment: Python 3.10, numpy 2.2.6, pandas 2.3.3, scikit-learn 1.7.2
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = pd.read_csv('Telco-Customer-Churn.csv')
df['TotalCharges'] = pd.to_numeric(df['TotalCharges'], errors='coerce')
df = df.dropna(subset=['TotalCharges'])

y = (df['Churn'] == 'Yes').astype(int)
num = ['tenure', 'MonthlyCharges', 'TotalCharges']
cat = ['Contract', 'InternetService', 'PaymentMethod',
       'PaperlessBilling', 'TechSupport', 'OnlineSecurity']
X = pd.get_dummies(df[num + cat], columns=cat, drop_first=True).astype(float)

# the scaler lives INSIDE the pipeline, so it refits on each training fold
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_validate(
    pipe, X, y, cv=cv,
    scoring=['roc_auc', 'average_precision', 'recall', 'neg_brier_score'])

for name in ['roc_auc', 'average_precision', 'recall']:
    v = scores['test_' + name]
    print('%-20s mean %.4f  std %.4f' % (name, v.mean(), v.std()))
print('auc per fold', ['%.4f' % s for s in scores['test_roc_auc']])

roc_auc              mean 0.8306  std 0.0253
average_precision    mean 0.6121  std 0.0796
recall               mean 0.4541  std 0.1088
auc per fold ['0.8023', '0.8748', '0.8104', '0.8359', '0.8297']

Compare those three standard deviations, because they are telling you something the means cannot. AUC is stable at 0.0253. Recall is wild at 0.1088, swinging from 0.3333 in one fold to 0.6296 in another. Ranking quality holds steady while the number of churners you would actually catch at a fixed threshold nearly doubles between folds. That happens because AUC integrates over every threshold, whereas recall is pinned to 0.5 and the probability distribution shifts slightly from fold to fold. Any metric evaluated at a fixed cut point inherits that instability, so a recall target written into a project plan needs a stated tolerance or it will be missed for reasons unrelated to the model.

Where leakage actually comes from

Data leakage means using information at training time that will not exist at prediction time. Scores go up, and they go up in a way no validation scheme can detect, because the validation data is contaminated by the same mechanism. Three varieties matter, and they are not equally dangerous, which is a point the standard advice tends to flatten.

First, preprocessing leakage. Fit StandardScaler on all 486 rows before splitting and the test rows contribute their mean and standard deviation to the transform. Every tutorial warns about this. On our data it moved AUC from 0.8127 to 0.8134, which is six ten thousandths, or roughly nothing. Say that plainly, because pretending otherwise trains people to hunt the harmless version and miss the lethal ones. Scaling leakage matters when the transform is aggressive, when the dataset is tiny, or when the transform is target aware, and a mean and variance over 486 rows is none of those. Keep the scaler in the pipeline anyway, since correctness is free here, but do not expect it to be the thing that saves you.

Second, target leakage, where a feature encodes the answer. I added a plausible looking exit_survey_sent flag to the churn features, the kind of column that appears in every real warehouse. AUC went from 0.8127 to 0.9879 and recall at the default threshold jumped to 0.9697. Exit surveys are sent after somebody cancels. That column is the label with a different name and a one day delay.

Third, and least discussed, selection leakage. This is the one that produced 0.8110 from random noise in the opening paragraph, and it is the variety a pipeline will not protect you from unless you build the pipeline correctly.

Leakage a pipeline cannot catch

Here is the experiment from the opening, written out. Generate 3,000 columns of pure Gaussian noise. Keep the real churn labels. Pick the 20 noise columns most correlated with churn, then cross validate a model on those 20. Selection happens once, on all the rows, before the folds are cut, which is exactly how it is usually written in a notebook.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
noise = rng.normal(size=(len(y), 3000))      # zero signal, by construction
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

# WRONG: selection sees every row, including the rows it will be scored on
picked = SelectKBest(f_classif, k=20).fit_transform(noise, y)
print('selection outside cv %.4f' % cross_val_score(
    model, picked, y, cv=cv, scoring='roc_auc').mean())

# RIGHT: selection is a pipeline step, refitted on each training fold
honest = make_pipeline(SelectKBest(f_classif, k=20), StandardScaler(),
                       LogisticRegression(max_iter=1000))
print('selection inside cv  %.4f' % cross_val_score(
    honest, noise, y, cv=cv, scoring='roc_auc').mean())

Mechanically it is simple. With 3,000 random columns and 486 rows, a handful will correlate with churn by chance alone. Selecting on all rows locks in those accidents, and because the held out fold was part of the selection, its accidents are locked in too. Cross validation then dutifully confirms a pattern that selection invented. Moving SelectKBest inside the pipeline forces a fresh choice on each training fold, the accidents no longer transfer, and 0.5185 is what honesty looks like on noise.

Anything that consults the target belongs inside. Feature selection, target encoding of high cardinality categoricals, oversampling, imputation fitted on the target, and threshold tuning all qualify. Threshold tuning is the one most often forgotten, because it happens after modelling and feels like reporting rather than fitting. Pick a threshold by scanning your test set for the best F1 and that threshold is fitted to the test set, so the F1 you quote is optimistic by construction. Sweep thresholds on out of fold predictions instead, which is what the table below does. Distinguishing a real effect from an artefact of how you looked is the same discipline described in A/B testing for analysts, and the failure mode is identical: decide the rule before you see the outcome.

Notice how much steadier this table is than the same sweep on 121 test rows in Part 10. Four times the customers, and each one scored out of fold, so a single borderline case no longer swings a whole column. Accuracy also does its usual disappearing act, sitting between 0.712 and 0.759 across a range where recall falls from 0.909 to 0.273. Anyone optimising accuracy here would land on 0.60 and miss three quarters of the leavers.

One error deserves showing because it catches people who have done everything else correctly. Cross validate on a small or heavily imbalanced dataset with too many folds and a fold eventually arrives with no positives in it. Scoring by AUC on that fold cannot work, and scikit-learn says so rather than guessing:

>>> from sklearn.metrics import roc_auc_score
>>> roc_auc_score([0, 0, 0, 0], [0.1, 0.4, 0.35, 0.8])
Traceback (most recent call last):
  File "sklearn/metrics/_ranking.py", line 640, in _binary_roc_auc_score
    raise ValueError(
ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.

Two structural cases break plain stratified folds entirely and no amount of care with pipelines will save them. Where rows are grouped, several transactions per customer for instance, splitting rows at random puts the same customer in train and test, so use GroupKFold and split on the customer identifier. Where data is ordered in time, random folds train on the future to predict the past, which is a guaranteed and completely unrealistic score. Part 19 handles the time series case properly, since it deserves a full treatment rather than a paragraph.

Evaluation protocol I would use before shipping a churn model

My recommendation is deliberately narrow. Hold out 20 percent as a final test set, touch it once, and do everything else with five fold stratified cross validation on the remaining 80 percent. Put every fitted step inside the pipeline, including selection and encoding, not just the scaler. Report AUC with its standard deviation as the model quality number and average precision beside the base rate as the campaign number. Sweep thresholds on out of fold predictions and never on the test set. Sort feature importances before celebrating any score that pleases you.

Between AUC and average precision, I would pick average precision as the number to optimise for a retention campaign and AUC as the number to report for model comparison, and I would avoid F1 as a headline metric entirely. F1 buries the precision and recall balance that somebody with a budget needs to see, and it treats a wasted 15 dollar call as equivalent to a lost 500 dollar customer, which is a claim nobody in the business would sign. Accuracy stays off slides altogether on any imbalanced problem. Where a stakeholder insists on a single number, give them expected value in currency, since that is the only single number that encodes the asymmetry honestly.

Our churn project now has a defensible number attached to it: AUC 0.8306 with a standard deviation of 0.0253, average precision 0.6121 against a 0.2716 base rate, and a threshold of 0.30 chosen on out of fold predictions that flags 205 customers to catch 107 of the 132 leavers. Part 10 quoted 0.853 from one split, which we can now see was a slightly lucky draw rather than a lie. Part 12 brings in decision trees, random forests and gradient boosting, which will comfortably beat this logistic baseline on ranking quality. Evaluate them with exactly the protocol above, because a boosted ensemble with hundreds of trees is far better at memorising a leaked column than logistic regression ever was, and a model that fits harder also fools you harder.

Run the noise experiment yourself before moving on. Ten lines, two minutes, and watching an AUC of 0.81 materialise out of a random number generator on your own screen is worth more than any warning I can write here.

References

• scikit-learn user guide, Common pitfalls and recommended practices, for the statement that feature selection on test data optimistically biases the model, and the list of transformers that leak.
• scikit-learn user guide, Cross validation, for StratifiedKFold being the default for classifiers and for GroupKFold on grouped data.
• scikit-learn user guide, Metrics and scoring, for the scorer names passed to cross_validate and the definition of average precision.
• scikit-learn, cross_validate API reference, for the returned dict keys and multi metric scoring.
• Telco Customer Churn dataset, IBM sample data on GitHub, 7043 rows, the file used throughout this series.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts