One extra column moved a fair 0.8785 to 0.9344 on a held out week, and it was a single lag: the error rate from one minute ago. Then five more engineered columns, together, added 0.029 on top of that. If you came expecting feature engineering to be a long list of clever transforms, the numbers on my own telemetry say the opposite, that most of the win hides in one or two features that carry a trend, and the rest is polish you can measure and mostly skip.

A causal rule for every feature

Last part we stopped a model from fooling us, cut the month in time, and learned that a centered rolling average had been reading minutes that had not happened yet. This part builds features on that same data on purpose, the past only kind that lift a score honestly. So the running project moves one step: last part we had four raw columns and a trustworthy evaluation, this part we turn those four columns into ten and watch which of the six new ones actually pay their way.

One rule governs every feature that touches time, and it is the same rule that caught the leak: a feature for minute t may use minute t minus one and earlier, never minute t itself and never later. In pandas that means a shift by one before any window, because a plain rolling average includes the current minute, and during a multi minute incident the current minute already carries the outcome. Write the rule on a sticky note and most feature leaks never reach your evaluation. Building these safely in depth is covered by the Data Science Series part on feature engineering, which this series leans on rather than repeats.

One lag that does most of the work

Start by rebuilding the month, the same simulated cluster as last part with one honest addition: a real per minute timestamp and a mild working day load pattern, so time of day features later have something real to try to capture. Incidents still last several minutes, because real ones do.

# tested on Python 3.10.12, scikit-learn 1.7.2, numpy 2.2.6, pandas 2.3.3
import numpy as np, pandas as pd
rng = np.random.default_rng(14)
N = 30000
idx = pd.date_range('2026-06-01', periods=N, freq='min')   # one row per minute, a real timestamp
frac = np.arange(N) / N
hour = idx.hour.to_numpy()
diurnal = 8.0 * np.sin((hour - 8) / 24 * 2 * np.pi)         # busier in the working day
cpu = np.clip(rng.normal(45, 12, N) + 6.0*frac + diurnal, 1, 100)
mem = np.clip(rng.normal(60, 10, N), 1, 100)
errs = rng.gamma(1.4, 0.6, N)
lat = rng.lognormal(3.3, 0.5, N)
hidden = np.where(frac > 0.66, rng.normal(0, 1.8, N), 0.0)
sig = 0.05*(cpu-55) + 0.06*(mem-70) + 1.2*(errs-1.2) + 0.03*(lat-28) + hidden
start_p = 1 / (1 + np.exp(-(sig - 4.2)))
label = np.zeros(N, int); incident_id = np.full(N, -1)
i, cur = 0, -1
while i < N:                                                # incidents last minutes
    if rng.random() < start_p[i]:
        cur += 1; runlen = int(rng.integers(3, 16))
        for j in range(i, min(i + runlen, N)):
            label[j] = 1; incident_id[j] = cur; errs[j] += rng.gamma(2.0, 0.7)
        i += runlen
    else:
        i += 1
df = pd.DataFrame({'cpu': cpu, 'mem': mem, 'errs': errs, 'lat': lat}, index=idx)
y = pd.Series(label, index=idx)
print('rows', len(df), 'incident minutes', int(y.sum()), 'rate', round(y.mean()*100, 2), 'percent')

rows 30000 incident minutes 4075 rate 13.58 percent

Now build the first real feature, the error rate one minute ago, and score it with TimeSeriesSplit, the fold scheme from last part that always trains on the past. A first attempt fails, and the failure is worth reading rather than stepping around.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.metrics import roc_auc_score

def cv_auc(X):
    m = make_pipeline(StandardScaler(), LogisticRegression(class_weight='balanced', max_iter=1000))
    s = cross_val_score(m, X, y, scoring='roc_auc', cv=TimeSeriesSplit(5))
    return s.mean(), s.std()

base = df[['cpu', 'mem', 'errs', 'lat']]
lag = pd.DataFrame(index=df.index)
lag['errs_lag1'] = df['errs'].shift(1)          # value one minute ago, past only
lag['cpu_lag1'] = df['cpu'].shift(1)
X_lag = pd.concat([base, lag], axis=1)
print('base four columns  AUC %.4f +/- %.4f' % cv_auc(base))
print('+ lag1             AUC %.4f +/- %.4f' % cv_auc(X_lag))

base four columns  AUC 0.8785 +/- 0.0025
ValueError: Input X contains NaN.
LogisticRegression does not accept missing values encoded as NaN natively.

The base scored fine, then the lag set threw. Shifting by one leaves the first minute with no prior value, so errs_lag1 holds exactly one NaN and LogisticRegression refuses it. This is the most common trip in feature work, a warm up gap at the edge of every window, and the reflex fix of fillna(0) is the wrong one: a zero error rate is a real and calm value, so filling a gap with it tells the model the system was healthy when you simply had no reading. Fill with the training median instead, or drop the warm up rows, and never with a value that means something.

X_lag = X_lag.fillna(X_lag.median())            # a neutral fill, not a meaningful zero
print('+ lag1 (filled)    AUC %.4f +/- %.4f' % cv_auc(X_lag))

+ lag1 (filled)    AUC 0.9344 +/- 0.0052

One lag feature took AUC from 0.8785 to 0.9344, a lift of 5.6 points, larger than anything the rest of this part will add combined. It works because an incident that lasts minutes leaves the previous minute already elevated, so last minute error rate is a genuine early signal that a raw snapshot of the current minute misses. If you take one habit from this part, make it this: before any exotic transform, try lagging the metric you already trust by one step.

Rolling windows and an exponential trend

A single lag sees one minute back. A rolling window sees a stretch and smooths the noise, and a rolling standard deviation catches volatility, the jitter that often precedes a fault. Both must be shifted by one to stay past only. An exponentially weighted average goes further, weighting recent minutes more heavily than old ones, which suits a system where the last few minutes matter most.

f = pd.DataFrame(index=df.index)
f['errs_lag1'] = df['errs'].shift(1)
f['cpu_lag1'] = df['cpu'].shift(1)
f['errs_roll11_mean'] = df['errs'].shift(1).rolling(11, min_periods=1).mean()  # trailing average
f['errs_roll11_std']  = df['errs'].shift(1).rolling(11, min_periods=1).std()   # trailing volatility
f['cpu_delta'] = df['cpu'].diff()                                              # change since last minute
f['errs_ewm']  = df['errs'].ewm(span=10).mean().shift(1)                       # exp weighted, then shifted

def build(cols):
    X = pd.concat([base] + [f[[c]] for c in cols], axis=1)
    return X.fillna(X.median())

steps = [
    ('base', []),
    ('+ lag1', ['errs_lag1', 'cpu_lag1']),
    ('+ rolling mean and std', ['errs_lag1', 'cpu_lag1', 'errs_roll11_mean', 'errs_roll11_std']),
    ('+ delta', ['errs_lag1', 'cpu_lag1', 'errs_roll11_mean', 'errs_roll11_std', 'cpu_delta']),
    ('+ ewm', ['errs_lag1', 'cpu_lag1', 'errs_roll11_mean', 'errs_roll11_std', 'cpu_delta', 'errs_ewm']),
]
for name, cols in steps:
    a, s = cv_auc(build(cols))
    print('%-24s AUC %.4f +/- %.4f' % (name, a, s))

base                     AUC 0.8785 +/- 0.0025
+ lag1                   AUC 0.9344 +/- 0.0052
+ rolling mean and std   AUC 0.9534 +/- 0.0091
+ delta                  AUC 0.9534 +/- 0.0091
+ ewm                    AUC 0.9631 +/- 0.0087

Read the ladder honestly. Rolling mean and standard deviation added 1.9 points, a real gain from smoothing and volatility. An exponentially weighted trend added another point, to 0.9631, the best honest score in this series so far. And the rate of change did nothing at all, 0.9534 before it and 0.9534 after, a feature that looked obvious and earned zero. Notice too that the spread widens as you add columns, from 0.0025 to 0.0087, because a richer model reacts more to which week it is tested on, so quote the mean with its spread rather than a single flattering fold.

One number in that code is a real decision, not a default: the window length. Too long and the mean smooths straight over the start of a short incident and reacts late, which on a metric that spikes for three to fifteen minutes is the gap between a warning and a shrug. I scored the same setup at three lengths, and the result is a small confession: five gave 0.9620, eleven gave 0.9524, and thirty one fell to 0.9409, a clean two points lost to a window that lagged the onset. So the eleven I used above is not the winner here, which is the honest lesson, tune the window against your own incident length rather than copy a constant, and set min_periods to one so the first minutes still return a value instead of a run of NaN.

Features that earned nothing

Two pieces of common advice deserve a hard look on real data. First, add time of day, because systems have daily rhythms. Second, always cyclically encode that hour so 23:00 sits beside 00:00 rather than 23 units away. Both are correct in the abstract and both did close to nothing here, and the reason is the point.

full = ['errs_lag1', 'cpu_lag1', 'errs_roll11_mean', 'errs_roll11_std', 'cpu_delta', 'errs_ewm']
h = df.index.hour.to_numpy()

X_int = build(full).assign(hour=h)                                    # hour as a plain integer
X_cyc = build(full).assign(hour_sin=np.sin(h/24*2*np.pi),
                           hour_cos=np.cos(h/24*2*np.pi))             # hour as sin and cos
print('hour as integer   AUC %.4f +/- %.4f' % cv_auc(X_int))
print('hour as sin, cos  AUC %.4f +/- %.4f' % cv_auc(X_cyc))

# which features earned their keep, standardized coefficients on the train slice
cut = int(len(X_cyc) * 0.75)
m = make_pipeline(StandardScaler(), LogisticRegression(class_weight='balanced', max_iter=1000))
m.fit(X_cyc.iloc[:cut], y.iloc[:cut])
imp = pd.Series(np.abs(m[-1].coef_[0]), index=X_cyc.columns).sort_values(ascending=False)
print(imp.round(3).head(6).to_string())

hour as integer   AUC 0.9632 +/- 0.0086
hour as sin, cos  AUC 0.9632 +/- 0.0088
errs_ewm            2.657
errs                1.659
errs_roll11_mean    1.242
errs_lag1           0.285
errs_roll11_std     0.279
hour_sin            0.183

Both hour encodings landed on 0.9632, a single ten thousandth above the run without any hour at all, and the coefficient table explains why: the exponentially weighted error trend dominates at 2.657, the current error rate and its rolling mean follow, and the hour terms sit near the bottom around 0.18 and below. Cyclical encoding is not wrong, the gap from 23:00 to 00:00 really is 23 for the integer and only 0.261 for the sine and cosine pair, but a correctly encoded feature that carries no signal about incidents is still worthless. What survives is blunt: engineer features that predict the target, not features that are merely textbook correct, and check the coefficient table before you keep a column that felt obligatory. Rate of change told the same story, a standardized weight of 0.023, the smallest in the model.

A feature pipeline that stays past only

Scattered feature lines rot. Two weeks later you cannot remember which columns were shifted and which were filled, and one un shifted window quietly rebuilds the leak from last part. Put the whole thing in one function that takes raw telemetry and returns a clean matrix, every window shifted, every fill neutral, so the causal rule lives in code rather than memory.

flowchart LR
  A[Raw telemetry per minute] --> B[Shift every metric by one]
  B --> C[Lag and rolling and ewm on shifted data]
  C --> D[Delta and time terms]
  D --> E[Fill warm up with training median]
  E --> F[Assemble matrix, keep column names]
  F --> G[Score with TimeSeriesSplit]

def make_features(raw):
    s = raw['errs'].shift(1)                     # shift once, up front, then build on the past
    out = pd.DataFrame({
        'cpu': raw['cpu'], 'mem': raw['mem'], 'errs': raw['errs'], 'lat': raw['lat'],
        'errs_lag1': s,
        'cpu_lag1': raw['cpu'].shift(1),
        'errs_roll11_mean': s.rolling(11, min_periods=1).mean(),
        'errs_roll11_std': s.rolling(11, min_periods=1).std(),
        'errs_ewm': raw['errs'].ewm(span=10).mean().shift(1),
    }, index=raw.index)
    return out.fillna(out.median())

X = make_features(df)
print('matrix', X.shape)
print('final past only set  AUC %.4f +/- %.4f' % cv_auc(X))

matrix (30000, 9)
final past only set  AUC 0.9631 +/- 0.0087

Nine columns, dropped of the two that earned nothing, scoring 0.9631, the same as the fuller set and easier to defend. One production caution the offline number hides: every one of these features has to be computable at serving time from data you actually have when the prediction fires. A rolling window over the last eleven minutes needs eleven minutes of history in memory at inference, and an exponentially weighted average needs its running state carried between calls, which is a real latency and state cost the same way an inference cache is in the AI Engineering Series on caching, batching and latency. A feature that is cheap in a notebook can be expensive in production, so cost the ones you keep before you promise them.

Keep one artifact from this part, a recipe card that turns a family of feature into a past only one liner and names the signal it captures and the trap it hides. It is the table I reach for at the start of any new dataset, before I have a single number to trust.

Beside it, keep the numbers that show what each family bought on this month, so the next reader who doubts that one lag beats a pile of columns has the evidence in one place.

Build three lag features on your own metrics

Do one concrete thing before the next part. Take the cleaned metrics from your own systems, pick the one signal you trust most, an error rate, a saturation number, a queue depth, and build three features on it and nothing else: its value one minute ago, a trailing eleven minute mean, and an exponentially weighted trend, every one shifted by one so it stays past only. Add them to the model you evaluated last part, rerun the same TimeSeriesSplit, and read the coefficient table to see which of the three pulled weight. If a single lag jumps your score the way it jumped mine, you have found the shape of the win before spending a weekend on the rest. Next part turns to MLOps, the operations you already know, where these features stop being notebook cells and become something you serve and monitor.

References

• pandas documentation, Series.shift
• pandas documentation, DataFrame.ewm
• scikit-learn documentation, TimeSeriesSplit
• scikit-learn documentation, common pitfalls and data leakage
• Numenta Anomaly Benchmark, real world time series to practice on
• Data Science Series, feature engineering in Python
• AI Engineering Series, caching, batching and latency
• Infra to Data Science, the Complete Guide

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts