I added one column to a model that already scored a fair 0.8891 on a held out month, reran it, and watched the ROC AUC print 0.9745. For about ten seconds that felt like progress, until I remembered that operational data almost never hands you eight points of AUC for free, and the only thing that had changed was a rolling average I built without thinking about which minutes it could see.

Leakage that survives a correct split

Last part you turned a production alert rule into a classifier and chose an operating point against a false page budget, and that scoreboard is the first real piece of your portfolio. Before it goes anywhere near a resume it has to be a number you can defend, and the fastest way to lose that defence is a score that was never real. Part 13 fixed one leak, a random split that mixes future minutes into training. This part is about the leaks that survive even a perfectly time ordered split, because they enter through the features and the preprocessing rather than the split itself. Every mechanic here is treated in full in the Data Science Series part on model evaluation, cross validation and leakage, which this series builds on rather than repeats.

A blunt way to put it: a correct train test boundary is necessary and nowhere near sufficient. You can cut your month cleanly in time, hold out the last week, never let a test row touch fit, and still ship a model that scored 0.97 offline and folds in production, because one feature you engineered quietly encoded the answer. Leakage is not one mistake, it is a family, and the split is only its first member. Name the others and you can check for them on purpose rather than discover them three weeks after go live.

flowchart TD
  A[New feature or transform] --> B{Uses any minute at or after the one it scores}
  B -- yes --> C[Shift it back, recompute on past only]
  B -- no --> D{Fitted on rows outside the training fold}
  D -- yes --> E[Move it inside a Pipeline]
  D -- no --> F{Same incident on both sides of the split}
  F -- yes --> G[Group the folds by incident]
  F -- no --> H[Score you can trust]

A feature that encodes the future

Start where the eight points came from. Here is the month, the same generator as last part with one honest change: incidents now last several minutes instead of a single one, because real incidents do, and that autocorrelation is exactly what a careless feature exploits.

# tested on Python 3.10.12, scikit-learn 1.7.2, numpy 2.2.6, pandas 2.3.3
import numpy as np, pandas as pd
rng = np.random.default_rng(14)
N = 30000
frac = np.arange(N) / N
cpu  = np.clip(rng.normal(45, 12, N) + 6.0*frac, 1, 100)
mem  = np.clip(rng.normal(60, 10, N), 1, 100)
errs = rng.gamma(1.4, 0.6, N)
lat  = rng.lognormal(3.3, 0.5, N)
hidden = np.where(frac > 0.66, rng.normal(0, 1.8, N), 0.0)
sig = 0.05*(cpu-55) + 0.06*(mem-70) + 1.2*(errs-1.2) + 0.03*(lat-28) + hidden
start_p = 1 / (1 + np.exp(-(sig - 4.2)))          # hazard of an incident starting
label = np.zeros(N, int)
incident_id = np.full(N, -1)
i, cur = 0, -1
while i < N:                                       # incidents last minutes, like real ones
    if rng.random() < start_p[i]:
        cur += 1
        runlen = int(rng.integers(3, 16))
        for j in range(i, min(i + runlen, N)):
            label[j] = 1
            incident_id[j] = cur
            errs[j] += rng.gamma(2.0, 0.7)         # errors climb during an incident
        i += runlen
    else:
        i += 1
X = np.column_stack([cpu, mem, errs, lat])
print('incidents', int(incident_id.max()+1), 'incident minutes', int(label.sum()),
      'rate', round(label.mean()*100, 2), 'percent')

incidents 429 incident minutes 3824 rate 12.75 percent

With the month built, add the feature that caused the trouble and score three versions on the honest time split, the same cut in time you learned last part.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import roc_auc_score, average_precision_score

cut = int(N * 0.75)                                # train on the earlier weeks, test on the last
def score(Xf):
    m = make_pipeline(StandardScaler(),
                      LogisticRegression(class_weight='balanced', max_iter=1000))
    m.fit(Xf[:cut], label[:cut])
    p = m.predict_proba(Xf[cut:])[:, 1]            # column 1 is probability of an incident
    return roc_auc_score(label[cut:], p), average_precision_score(label[cut:], p)

err = pd.Series(errs)
centered = err.rolling(11, center=True, min_periods=1).mean().to_numpy()   # peeks ahead
causal   = err.shift(1).rolling(11, min_periods=1).mean().to_numpy()       # past only
causal   = np.nan_to_num(causal, nan=float(err.mean()))

print('base four columns    AUC %.4f  AP %.4f' % score(X))
print('+ centered window     AUC %.4f  AP %.4f' % score(np.column_stack([X, centered])))
print('+ causal past window  AUC %.4f  AP %.4f' % score(np.column_stack([X, causal])))

base four columns    AUC 0.8891  AP 0.7237
+ centered window     AUC 0.9745  AP 0.9174
+ causal past window  AUC 0.9147  AP 0.7756

Read the middle row and feel the pull. A centered eleven minute average of the error rate took AUC to 0.9745 and AP to 0.9174, and it looks like the best idea you have had all week. It is a lie. Passing center equal True labels each average at the middle of its window, so the value at a given minute is built partly from minutes that come after it, and during a multi minute incident those later minutes are themselves incident minutes with elevated errors. That feature quietly reads the outcome it is meant to predict.

Prove it on eleven numbers rather than thirty thousand, because a small example is easier to trust than a metric.

demo = pd.Series([0, 0, 0, 0, 0, 100, 0, 0, 0, 0, 0])   # one spike at index 5
print('centered', demo.rolling(3, center=True, min_periods=1).mean().round(1).tolist())
print('shift1  ', demo.shift(1).rolling(3, min_periods=1).mean().round(1).tolist())

centered [0.0, 0.0, 0.0, 0.0, 33.3, 33.3, 33.3, 0.0, 0.0, 0.0, 0.0]
shift1   [nan, 0.0, 0.0, 0.0, 0.0, 0.0, 33.3, 33.3, 33.3, 0.0, 0.0]

A spike sits at index five. With centering, index four already carries 33.3, a minute that has not happened at the moment you would score index four. Shift the series by one and use a trailing window and the feature only ever sees the past, which is the version that can run in production, where minute five does not exist yet when you score minute four. Rebuilt that way the same rolling average gave 0.9147, a real 2.6 point lift over the four raw columns, because a past only trend in the error rate genuinely does run ahead of incidents. That honest version is the one to keep, and it is the whole difference between a feature and a leak.

That honest gain came from a lag feature, a trend the system shows before it breaks, and building those safely is its own craft the Data Science Series covers in feature engineering. One rule keeps every one of them honest: a feature for minute t may use minute t minus one and earlier, never minute t itself and never later. Write that rule on a sticky note and most feature leaks never reach your evaluation.

Preprocessing that leaks across a split

Features are the loud leak. Preprocessing is the quiet one, and it hides inside steps you think of as harmless. Scaling, imputing a missing value, selecting the best few features, any step that learns something from the data can learn it from the test set if you fit it before you split.

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.pipeline import Pipeline
from sklearn.metrics import roc_auc_score
Xc = np.column_stack([X, causal])

# wrong: fit the scaler and the supervised selector on the WHOLE dataset first
scaler = StandardScaler().fit(Xc)
sel = SelectKBest(f_classif, k=3).fit(scaler.transform(Xc), label)      # sees every label
Xall = sel.transform(scaler.transform(Xc))
bad = LogisticRegression(class_weight='balanced', max_iter=1000).fit(Xall[:cut], label[:cut])
print('fit transforms on ALL data  AUC %.4f' % roc_auc_score(label[cut:], bad.predict_proba(Xall[cut:])[:, 1]))

# right: every transform fits inside the training slice only
clean = Pipeline([('sc', StandardScaler()),
                  ('sel', SelectKBest(f_classif, k=3)),
                  ('lr', LogisticRegression(class_weight='balanced', max_iter=1000))])
clean.fit(Xc[:cut], label[:cut])
print('fit inside the train slice  AUC %.4f' % roc_auc_score(label[cut:], clean.predict_proba(Xc[cut:])[:, 1]))

fit transforms on ALL data  AUC 0.9144
fit inside the train slice  AUC 0.9139

Here the wrong way scored 0.9144 and the right way 0.9139, a difference of five ten thousandths, and that tiny gap is itself the lesson. On thirty thousand rows a scaler barely notices the few hundred test rows it should not have seen, so the common warning that preprocessing leakage will wreck your score can feel overblown, and on data this size it is. Shrink the set to three hundred rows, or swap StandardScaler for a target mean encoder that averages the label per category, and the same mistake swings AUC by ten points or more. A rule survives its own weak example: fit every transform inside a Pipeline so it only ever learns from the training fold, and you never have to guess whether this was the dataset where it mattered. scikit-learn documents this as the single most common way leakage slips in.

Cross validation as a distribution, not a point

A single time ordered split gives one number, and one number hides how much it could have wobbled. Fold the split several times and you get a distribution, which is what you should quote to anyone who asks how good the model is.

from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score

def pipe():
    return make_pipeline(StandardScaler(),
                         LogisticRegression(class_weight='balanced', max_iter=1000))

rand = cross_val_score(pipe(), X, label, scoring='roc_auc',
                       cv=KFold(5, shuffle=True, random_state=14))
tss  = cross_val_score(pipe(), X, label, scoring='roc_auc', cv=TimeSeriesSplit(5))
print('random KFold   ', [round(float(x), 4) for x in rand], 'mean', round(float(rand.mean()), 4))
print('TimeSeriesSplit', [round(float(x), 4) for x in tss], 'mean', round(float(tss.mean()), 4),
      'std', round(float(tss.std()), 4))

random KFold    [0.8914, 0.8833, 0.8762, 0.8914, 0.885] mean 0.8855
TimeSeriesSplit [0.8733, 0.8799, 0.8865, 0.8834, 0.8951] mean 0.8836 std 0.0072

TimeSeriesSplit trained on an expanding past and tested on the next block five times, scoring 0.8733 up to 0.8951, mean 0.8836, standard deviation 0.0072. Quote the mean with its spread, not whichever fold flattered you, because a single split could have landed on the lucky 0.895 week or the plain 0.873 one. Now read the other number: random KFold on these four columns scored 0.8855, two thousandths above the honest scheme. After all of last part arguing for time ordered splits, that near tie deserves an honest word. That protocol gap stayed small here because these raw columns carry no strong minute to minute memory a shuffle could exploit; the day you add a lag feature or a slow moving counter, it opens wide and a random split flatters you again. Keep TimeSeriesSplit as your default not because it always changes the number, but because it is the only scheme that will not lie to you on the day it matters.

Group leakage and entity overlap

One family of leak earns a demonstration that partly refuses to cooperate, because an honest result is more useful than a tidy one. When rows come in groups, the same incident spread over twelve minutes, the same server sampled all month, the same customer across many sessions, a plain split can drop some of a group in training and the rest in test, and the model scores well by half remembering the group rather than learning the pattern. GroupKFold keeps every group wholly on one side.

from sklearn.model_selection import GroupKFold
try:
    cross_val_score(pipe(), X, label, scoring='roc_auc', cv=GroupKFold(5))
except ValueError as e:
    print('ValueError:', e)

groups = np.where(incident_id < 0, -(np.arange(N) + 1), incident_id)   # each healthy minute its own group
for name, data in (('base columns ', X), ('leaky feature', np.column_stack([X, centered]))):
    rk = cross_val_score(pipe(), data, label, scoring='roc_auc',
                         cv=KFold(5, shuffle=True, random_state=14)).mean()
    gk = cross_val_score(pipe(), data, label, scoring='roc_auc',
                         cv=GroupKFold(5), groups=groups).mean()
    print(name, ' randomKFold %.4f  GroupKFold %.4f' % (rk, gk))

ValueError: The 'groups' parameter should not be None.
base columns   randomKFold 0.8855  GroupKFold 0.8855
leaky feature  randomKFold 0.9829  GroupKFold 0.9830

Two things to read here. First, calling GroupKFold without groups raises a ValueError, the groups parameter should not be None, a real and common trip because cross_val_score does not carry groups for you unless you pass them. Second, and less expected, grouping by incident did not move the score at all on this linear model: 0.8855 either way on the base columns, and 0.9829 against 0.9830 on the leaky feature. Group leakage is real, but a linear model on generic metrics has little group identity to memorise, so it stays invisible here. Swap in a tree model that can carve out per incident quirks, or a feature keyed to a server name, and the same overlap can inflate AUC by ten points. Group your folds when rows have a natural entity and you cannot swear a random split keeps it intact, and do not assume the fix always moves the number, because sometimes it quietly reassures you and sometimes it saves you.

Leakage to cause, a lookup to keep

Keep one artifact from this part, a table that turns a symptom into a cause and a fix, so the next time a score looks too good you have somewhere to start rather than a shrug.

Keep the numbers beside it, the small table that shows what an honest evaluation of this month looks like once the leak is out and the feature is rebuilt to run in production.

Audit one feature for leakage this week

Do one concrete thing before the next part. Take the model you built last part, list every engineered feature, and for each one answer a single question out loud: does this use any measurement from the minute it scores, or later. Any yes is a leak, so shift it back and recompute on past minutes only, then rerun your TimeSeriesSplit and watch which points of your score were real. Getting an honest evaluation in place before you tune anything is the same discipline the AI Engineering Series argues for in building an eval set before the feature, where a number without a trustworthy floor is worse than no number at all. Your portfolio scoreboard is only worth showing once you can promise it was measured honestly, and this is the part that lets you promise it. Next part builds features on this same data on purpose, the causal, past only kind that lift a score without lying about it.

References

• scikit-learn documentation, common pitfalls and data leakage
• scikit-learn documentation, TimeSeriesSplit
• scikit-learn user guide, cross validation
• scikit-learn documentation, GroupKFold
• pandas documentation, Series.rolling
• Data Science Series, model evaluation, cross validation and leakage
• Infra to Data Science, the Complete Guide

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts