A CPU alert wired to a fixed 85 percent line fired 140 times in one month on a single cluster and caught the incident that mattered at a precision of 0.05, so ninety five alerts in every hundred were the afternoon getting busy on schedule. Swap that line for a detector that first strips the daily and weekly shape out of the metric and the same month scores 0.63 precision at full recall on the same labels. Anomaly detection on infrastructure telemetry is mostly the work of deciding what normal looks like at this hour of this weekday, then measuring how far the latest point sits from it.
Static Thresholds Break on Seasonal Metrics
Where the project stands: last part we tracked and registered a trained incident classifier so a model could be found and rolled back by name. This part turns the same exported telemetry into an anomaly detector that needs no labels at all, the kind of build a hiring manager reads as real infrastructure data work rather than a tutorial exercise. Measuring distance from a normal baseline is the same reflex as pointing observability and tracing at a running system, watching the signal that says something changed.
A static threshold is where everyone starts and where the false pages begin. Infrastructure metrics are seasonal: CPU climbs every weekday afternoon and falls every night, memory sawtooths with a nightly batch, request rate follows business hours. A fixed line cannot tell a Tuesday afternoon peak that happens every week from a runaway process, because both sit at the same absolute value. On the cluster below the raw CPU series has a standard deviation of 14.8 percentage points, and almost all of that spread is the daily season swinging across a 43 point range, not noise and not trouble. Grade the 85 percent rule against confirmed incidents and it flags 140 hours to catch 17 real ones, a precision of 0.05 and a recall of only 0.41, because the one incident that pulled CPU down instead of up never crossed the line at all. Raising the threshold to 95 percent, the reflex fix, only trades those false pages for missing the next real spike. Level was never the problem. Shape was.
# anomaly.py tested with python 3.12, statsmodels 0.14.6, scikit-learn 1.7.2, pandas 2.3.3, numpy 2.2.6
import pandas as pd
from sklearn.metrics import precision_score, recall_score
# metrics.csv: hourly cpu, memory and load for one cluster, exported from monitoring
# the truth column is only for grading, the detector itself never sees a label
df = pd.read_csv('metrics.csv', index_col=0, parse_dates=True)
y = df['truth'].values
naive = (df['cpu'] > 85).astype(int).values
print('flagged', int(naive.sum()),
'precision', round(precision_score(y, naive, zero_division=0), 2),
'recall', round(recall_score(y, naive, zero_division=0), 2))
flagged 140 precision 0.05 recall 0.41
Deseasonalising a Metric With STL
STL, seasonal trend decomposition using LOESS, splits a series into three parts that add back together: a slow trend, a repeating season, and a residual that holds whatever the first two do not explain. That residual is the surprise, and it is what a detector should score. Because the Data Science Series already works through decomposition and backtesting in depth for a forecasting audience, I keep the mechanics short here and link the fuller treatment of time series forecasting and backtesting for the parts I skip. Every line below was run against python 3.12, statsmodels 0.14.6, scikit-learn 1.7.2, pandas 2.3.3 and numpy 2.2.6. Pass period as the number of points in one full seasonal cycle, here 24 for a day of hourly readings, and turn on the outlier resistant weighting so the injected spikes do not bend the fitted season around themselves.
from statsmodels.tsa.seasonal import STL
# period 24 = one day of hourly points; outlier resistant weighting keeps the two spikes
# from bending the season around themselves
res = STL(df['cpu'], period=24, robust=True).fit()
resid = res.resid # cpu minus its trend and daily season
print('resid std', round(resid.std(), 1),
'peak resid', round(resid.abs().max(), 1))
resid std 6.2 peak resid 42.3
Removing the season pulls the spread down from 14.8 on the raw metric to a residual standard deviation of 6.2, and the runaway spike now stands 42 points clear of a residual that normally sits within about 6. Against the raw series that same spike hid inside the afternoon envelope; against the residual it is the tallest thing on the chart. One failure waits for anyone who reaches for STL on a bare array.
STL(df['cpu'].values, robust=True).fit() # passing a raw numpy array, no period
ValueError: Unable to determine period from endog
Residual Detector Using Median Absolute Deviation
A residual still needs a line, and that line should not be a plain standard deviation, because one large anomaly inflates the very spread you measure it against and hides the next one. Median absolute deviation is the median of the absolute gaps from the median, and a handful of extreme points cannot drag it. Scale it by 1.4826 and it estimates the same spread a standard deviation would on clean normal data, so a MAD based z-score reads on the familiar scale where 3 is notable and 5 is rare. Compute the median and the MAD in a centred rolling window so the baseline tracks slow drift, then flag any point whose score clears a threshold. This function is the reference artifact of this part, a short recipe that turns an exported metric into a labelled anomaly stream; keep it in the repo next to the export script.
def residual_anomalies(series, period, z_thresh=4.0, win=48):
from statsmodels.tsa.seasonal import STL
resid = STL(series, period=period, robust=True).fit().resid
med = resid.rolling(win, center=True, min_periods=win // 4).median()
mad = (resid - med).abs().rolling(win, center=True, min_periods=win // 4).median()
score = (resid - med) / (1.4826 * mad) # MAD based z-score
return (score.abs() > z_thresh).astype(int)
flag = residual_anomalies(df['cpu'], period=24).values
print('flagged', int(flag.sum()),
'precision', round(precision_score(y, flag, zero_division=0), 2),
'recall', round(recall_score(y, flag, zero_division=0), 2))
flagged 27 precision 0.63 recall 1.0
On the same month this scores 0.63 precision at a recall of 1.0, so every real incident is caught for twenty seven flags against seventeen true ones. A threshold of 4 rather than 3 is a deliberate choice: 3 caught two more benign blips and cost precision, and on telemetry that pages a human, a quieter detector that still catches every real event is worth more than one that fires on every wobble. Move the threshold to fit what a false page costs your team, and hold recall at the level an incident review would accept. For a single seasonal metric this residual detector is the pick, and a raw isolation forest on the same one column, next, is the option to avoid.
One detail decides whether this detector works live or only in hindsight. A centred rolling window reads points on both sides of each timestamp, which is fine when you score an exported history but impossible in real time, where the future half of the window does not exist yet. For a live detector set center to false so the window trails the current point, accept that it reacts a little slower right after a shift in level, and warm it up on recent history before trusting its first scores. Scoring a backfill with a centred window and then shipping the same code untouched is a quiet way to report a recall you can never reproduce on live data.
Isolation Forest for Multivariate Telemetry
One metric rarely tells the whole story. A node draining shows up as CPU down, memory down and load down at the same time, a pattern no single line catches well. An isolation forest scores how few random splits it takes to isolate a point in the joint feature space, and points that separate in a handful of cuts are the odd ones. Feed it CPU, memory and load together and it reads the combination rather than each axis alone. Because a forest splits one feature at a time, it does not need the columns rescaled the way a distance based method would, one less preprocessing step than the residual path.
from sklearn.ensemble import IsolationForest
iso = IsolationForest(n_estimators=200, contamination=0.025, random_state=7)
multi = (iso.fit_predict(df[['cpu', 'mem', 'load']]) == -1).astype(int)
print('multi ', int(multi.sum()), round(precision_score(y, multi), 2), round(recall_score(y, multi), 2))
# the wrong way: the same forest on the single seasonal column
one = (IsolationForest(n_estimators=200, contamination=0.025, random_state=7)
.fit_predict(df[['cpu']]) == -1).astype(int)
print('cpu ', int(one.sum()), round(precision_score(y, one), 2), round(recall_score(y, one), 2))
multi 17 1.0 1.0 cpu 17 0.47 0.47
On the joint vector the forest flags seventeen points at a precision of 1.0 and a recall of 1.0. Read that perfect score with suspicion: this is clean generated data with two obvious injected events, and real telemetry never scores 1.0. What does hold on messier data is the gap between the two runs. Point the same forest at the raw CPU column alone and precision and recall both fall to 0.47, because on one seasonal axis the busiest legitimate afternoons look exactly as isolated as the incidents. An isolation forest earns its place across several metrics that move together, not as a drop in replacement for a threshold on one seasonal line, where deseasonalising first wins.
One argument does more damage than any other. contamination tells the forest what fraction of points to return as anomalies, and it acts as a hard prior, not a hint. Set it to 0.025 and almost exactly 2.5 percent of points come back flagged whether or not that many are truly odd, so a guess set too high manufactures false positives and one set too low buries real events. Rather than guess, score the raw decision function and pick the cut from labelled history, the same way the residual threshold was chosen. A second trap is quieter and costs nothing to avoid once you have seen it.
pred = iso.fit_predict(df[['cpu', 'mem', 'load']])
wrong = (pred == 1).astype(int) # treating 1 as the anomaly, the binary label reflex
print('label 1 as anomaly, recall', round(recall_score(y, wrong, zero_division=0), 2))
label 1 as anomaly, recall 0.0
predict and fit_predict return minus one for an anomaly and plus one for a normal point, the opposite of the 1 equals bad habit from a binary label. Treat plus one as the anomaly and the detector silently reports a recall of 0.0 while looking like it ran clean. Flag on minus one, and check that the flagged count is small before you trust it.
Scoring a Detector Against Labels, Not Accuracy
Every number above is a precision or a recall, never an accuracy, and that is on purpose. At an anomaly rate of 2.5 percent a detector that flags nothing is right 97.5 percent of the time and useless, so accuracy rewards exactly the silence you are trying to break. Precision answers how many of your alerts were real, recall answers how many real events you caught, and the two trade against each other at every threshold. An on call team feels low precision as pager fatigue and low recall as a missed outage, so name the balance you want before you tune, not after. Grade against a window of confirmed incidents, count a flag anywhere inside a real event as a catch, and treat a lone flag in calm as a false positive. If you want a public trace to try this on before your own, the Numenta Anomaly Benchmark ships a real AWS CPU utilisation series with a documented cause.
| Detector | Flagged | Precision | Recall |
|---|---|---|---|
| Static 85 percent line | 140 | 0.05 | 0.41 |
| Isolation forest, cpu only | 17 | 0.47 | 0.47 |
| STL residual, MAD z-score | 27 | 0.63 | 1.0 |
| Isolation forest, three metrics | 17 | 1.0 | 1.0 |
| Situation | Detector to use | Why |
|---|---|---|
| One metric, clear daily or weekly season | STL residual, MAD z-score | scores surprise, not time of day |
| Several metrics that move together | Isolation forest on the joint vector | reads the combination one line misses |
| Little history, no season yet | Median and MAD on a rolling window | outlier resistant with no fitted model |
| A flat metric with a hard limit | a plain threshold is fine | no season means no false peaks |
Start With Residuals, Add a Forest When Metrics Move Together
None of this needs a model you have to train and serve. For a single seasonal metric, decompose with STL, score the residual with a median and MAD based z, and set the threshold from what a false page costs, that is the pick and it runs in a short function. When a failure shows across several metrics at once, put an isolation forest on the joint vector and choose its cut from labelled history rather than from a contamination guess. Steer clear of the two defaults that look reasonable and are not: a static line on a seasonal metric, and accuracy as the score for a rare event.
Do this on Monday: export a month of one metric from your own monitoring, run STL with the period set to your daily cycle, and plot the residual next to the raw series. You will watch the season leave and the real surprises stand up, and you will have the first honest anomaly stream off your own systems. Next part keeps the same exported telemetry and asks a different question of it, capacity forecasting, turning a month of history into a defensible estimate of when a cluster runs out of headroom.
References
- statsmodels STL, seasonal trend decomposition using LOESS
- scikit-learn IsolationForest, contamination and decision_function
- scikit-learn worked example of an isolation forest
- Numenta Anomaly Benchmark, labelled real world time series
- Data Science Series, Time Series Forecasting and Backtesting
- AI Engineering Series, LLM Observability, Tracing and Debugging
About The Author
DrJha