From alert rule to trained model

Two parts back you exported and cleaned a month of one cluster of cpu, memory, error and latency metrics; last part you mapped machine learning vocabulary onto that export and fit a first logistic regression to see the terms land. This part builds the model end to end, starting from the literal rule your monitoring runs and finishing on a classifier whose operating point you pick on purpose rather than accept by default. Every number below comes from one export, so nothing here is a toy dataset standing in for real work.

Here is where most first models go wrong before a line of scikit-learn runs. A tutorial tells you to fit a DummyClassifier as your baseline, and that is fine practice, but it is not your real floor. Your real floor is the alert rule already paging your team, and it usually scores far worse than the dashboard next to it implies. Measure that rule as a model first, because a classifier that cannot beat the threshold you already run is not worth deploying, and one that beats it barely may still not be worth the false pages. Build the labelled month, then hold the production rule up to the same recall and precision you will judge the model on.

# tested on Python 3.10.12, scikit-learn 1.7.2, numpy 2.2.6
import numpy as np
rng = np.random.default_rng(13)
N = 30000
frac = np.arange(N) / N
cpu = np.clip(rng.normal(45, 12, N) + 6.0*frac, 1, 100)      # slow load creep over the month
mem = np.clip(rng.normal(60, 10, N), 1, 100)
errs = rng.gamma(1.4, 0.6, N)
lat_ms = rng.lognormal(3.3, 0.5, N)                          # p99 latency in milliseconds
# a noisy neighbour appears in the last third and drives incidents the four columns cannot see
hidden = np.where(frac > 0.66, rng.normal(0, 1.8, N), 0.0)
sig = 0.05*(cpu-55) + 0.06*(mem-70) + 1.2*(errs-1.2) + 0.03*(lat_ms-28) + hidden
prob = 1 / (1 + np.exp(-(sig - 2.6)))
label = (rng.random(N) < prob).astype(int)                  # 1 = incident minute
X = np.column_stack([cpu, mem, errs, lat_ms])
print('rows', N, 'incident rate', round(label.mean()*100, 2), 'percent, count', int(label.sum()))

rows 30000 incident rate 5.79 percent, count 1736

Swap in your own export with the same four columns and every step that follows runs unchanged. One detail carries weight later: keep the rows ordered by timestamp, minute 0 at the top and the end of the month at the bottom, because two sections below depend on that order being honest. A hidden driver enters in the final third on purpose, a stand in for the storage array or neighbour service that starts causing pages nobody instrumented, and it is what makes an honest evaluation earn its keep.

Going from an alert rule to a classifier changes less than it sounds. A rule of cpu over 90 is a boundary in one dimension, a single point on a number line. Four features make that boundary a weighted sum of four numbers, still a straight cut but now angled through a space you cannot picture, and the weights are fit from labelled minutes rather than typed by hand. That is the whole mechanical difference, and it is why logistic regression is the honest first model here: it stays a line, so you can read its coefficients and argue with them, while giving the boundary enough freedom to lean on errors and latency instead of cpu alone.

Your alert rule, measured honestly

Your monitoring already classifies every minute with one feature and a typed line, cpu over 90. Score that rule the way you will score the model, on how many real incidents it catches and how often it cries wolf, and the picture is bleak.

from sklearn.metrics import recall_score, precision_score
for line in (90, 75):
    rule = (cpu > line).astype(int)          # the alert already in production
    print('cpu over', line,
          'recall', round(recall_score(label, rule), 4),
          'precision', round(precision_score(label, rule, zero_division=0), 4),
          'fires', int(rule.sum()))

cpu over 90 recall 0.0012 precision 0.2 fires 10
cpu over 75 recall 0.0282 precision 0.1273 fires 385

cpu over 90 fired 10 times across 30,000 minutes and caught almost none of the 1,736 incidents, because most of them here are driven by errors and that unseen dependency, not by cpu at all. Drop the line to 75 and recall crawls to 2.8 percent while precision falls to 12.7 percent, more noise for barely more signal. That is the ceiling of one feature and a hand typed number, and it is the floor every model below has to clear. A classifier that reaches even 60 percent recall is not competing with a good baseline, it is competing with a rule that sleeps through nineteen incidents in twenty.

One feature fails here for a reason worth naming, because it is the reason multivariate models exist at all. cpu correlates weakly with these incidents; the real drivers are the error rate and a dependency the cpu column never sees. Any single metric threshold inherits that blind spot, so raising or lowering the cpu line only trades one kind of miss for another and never fixes the cause. A model that reads all four columns at once can weight the error rate up and cpu down on its own, and that reweighting, not any cleverness in the algorithm, is where the entire gain comes from. Hold that thought when a tutorial reaches for gradient boosting on the first page; most of the lift on operational data comes from using more of the right columns, not from a fancier estimator.

Random splits flatter a metric series

Reach for train_test_split with a random shuffle and you have quietly leaked the future into the past. Minutes seconds apart are nearly identical, so a random draw lands a minute in training and its near twin in the test set, and the model looks clairvoyant on data it has all but seen. Measure the same model both ways and read the gap.

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import roc_auc_score, average_precision_score

def fit_auc(Xtr, Xte, ytr, yte):
    m = make_pipeline(StandardScaler(), LogisticRegression(class_weight='balanced', max_iter=1000))
    m.fit(Xtr, ytr)
    p = m.predict_proba(Xte)[:, 1]                 # column 1 is probability of an incident
    return roc_auc_score(yte, p), average_precision_score(yte, p), m

# wrong: a random shuffle mixes future minutes into training
Xa, Xb, ya, yb = train_test_split(X, label, test_size=0.25, stratify=label, random_state=13)
auc_rand, _, _ = fit_auc(Xa, Xb, ya, yb)

# honest: cut in time, train on the earlier weeks, test on the later ones
cut = int(N * 0.75)
Xtr, Xte, ytr, yte = X[:cut], X[cut:], label[:cut], label[cut:]
auc_time, ap_time, model = fit_auc(Xtr, Xte, ytr, yte)
print('random split  ROC AUC', round(auc_rand, 4))
print('time split    ROC AUC', round(auc_time, 4), 'AP', round(ap_time, 4), 'test incidents', int(yte.sum()))

random split  ROC AUC 0.7977
time split    ROC AUC 0.7554 AP 0.2825 test incidents 691

Same rows, same model, two evaluation protocols, and the random split reads four points of AUC higher. That gap is not skill, it is the model half remembering minutes it will meet in the test set. Notice one more trap in the code: predict_proba returns two columns and you want index 1, the probability of an incident; grab column 0 by reflex and every score inverts silently, with no error raised. Notice too that the test window carries a 9.21 percent incident rate against the 5.79 percent month, because that hidden dependency pushed incidents up late, exactly the drift a time ordered split refuses to paper over. This failure mode, an evaluation that leaks and flatters, gets its full treatment in the Data Science Series part on model evaluation, cross validation and leakage, which this series relies on rather than repeats.

Read the AP figure beside the AUC, and lead with it on rare incidents. Average precision came out 0.2825 while ROC AUC read 0.7554 on the same honest split, and that gap is not a contradiction. ROC AUC can look respectable on a heavily imbalanced problem because it rewards ranking the many healthy minutes correctly, while average precision summarises the precision recall curve you actually operate on, where a wall of healthy minutes drags precision down. Quote both numbers, lead with AP when incidents are rare, and a healthy looking AUC will stop surprising you when the curve underneath it collapses toward the base rate.

A single time ordered cut is the smallest honest split; the fuller version folds it several times. scikit-learn ships TimeSeriesSplit, which trains on an expanding window of the past and tests on the next block, repeated so you get several honest scores rather than one that might have landed on a lucky week. Next part leans on it. For a first model a single cut is enough to expose the leakage, but reach for TimeSeriesSplit the moment one test window feels too small to trust, and never for a random KFold on a metric series, which leaks in every fold exactly the way the shuffle above did.

A rule of thumb saves this every time: if the rows carry a timestamp, the split respects it. Logs, metrics, traces, incident tickets, billing records, anything an operator touches sits on a clock, and for all of it a random split is a quiet lie about how the model will behave next week. That one habit, cutting in time, catches more first model mistakes than any hyperparameter you could tune, and it costs a slice of the array instead of a call to train_test_split.

Training the classifier and reading its curve

You already have the time split model. Now read it across every threshold rather than at the single line predict draws. predict decides incident at a probability of 0.5, but predict_proba hands back the raw score for every minute, and a precision recall curve turns those scores into the whole trade at once, one precision and recall pair for each place you could put the line. A first attempt to line up the three arrays fails, and the failure is worth seeing.

from sklearn.metrics import precision_recall_curve
proba = model.predict_proba(Xte)[:, 1]
precision, recall, thresholds = precision_recall_curve(yte, proba)
print('lengths  precision', len(precision), 'recall', len(recall), 'thresholds', len(thresholds))
grid = np.column_stack([thresholds, precision, recall])   # line the three up for a table

lengths  precision 7501 recall 7501 thresholds 7500
ValueError: all the input array dimensions except for the concatenation axis
must match exactly, but along dimension 0, the array at index 0 has size 7500
and the array at index 1 has size 7501

precision_recall_curve returns one more precision and recall value than thresholds by design, because the final point, recall 0 and precision 1, sits above the highest score and has no threshold beneath it. Drop that trailing point and the arrays align. With them aligned you can pick an operating point on a rule you state out loud: the highest precision available while holding recall at or above 0.80, a bar an on call team might set so four incidents in five are caught.

p_al, r_al = precision[:-1], recall[:-1]           # drop the point with no threshold
keep = r_al >= 0.80                                 # hold recall at 0.80 or better
op = np.argmax(np.where(keep, p_al, -1))           # best precision under that constraint
print('operating threshold', round(float(thresholds[op]), 3),
      'recall', round(float(r_al[op]), 4),
      'precision', round(float(p_al[op]), 4))

operating threshold 0.348 recall 0.8017 precision 0.1547

Before trusting any operating point, read what the model leaned on, because a linear model lets you. On the standardised features the fitted weights came out cpu 0.56, memory 0.58, errors 0.85 and latency 0.48, so the error rate carries the most weight and cpu the least of the four. That matches the story the single feature told: the rule failed because it watched the wrong column, and the classifier beat it mainly by shifting weight toward errors. Coefficients this readable are the reason to keep a linear model as your first real one; a gradient boosted forest might score a point higher and tell you nothing you could repeat to whoever owns the service.

Holding recall at 0.80 costs precision 0.155, roughly one real page in six or seven, because the last week hides a driver the four columns cannot see. Plotting the whole curve makes the shape of that cost visible: precision starts near 0.40 when the model only flags its most confident minutes and slides toward the 0.09 base rate as you push recall up. Wherever the curve sits above that flat base rate, the model is adding information a coin flip would not.

One caution about those scores before anyone reads them as probabilities. predict_proba returns a number between 0 and 1, but class_weight balanced pushes those numbers away from true frequencies, so a score of 0.35 does not mean a 35 percent chance of an incident here. That is fine for choosing an operating point, because the curve only needs the scores to rank minutes in the right order, not to be calibrated. It starts to matter the moment a stakeholder wants to treat the score as a probability, at which point you calibrate on held out data rather than trust the raw output, a step the evaluation part ahead covers rather than this one.

Choosing an operating point on a real budget

An operating point is a business decision wearing a number. Put the rule and the classifier at two thresholds side by side on the same held out week and read the two columns on call actually feels, incidents caught and false pages raised. Keep this scoreboard as the artifact you return to; it is the one table that answers whether a model earns its deploy.

Read the confusion counts behind the chosen row as four operational outcomes, and watch where the cost lands.

from sklearn.metrics import confusion_matrix
pred = (proba >= 0.348).astype(int)
tn, fp, fn, tp = confusion_matrix(yte, pred).ravel()
print('caught', tp, 'missed', fn, 'false pages', fp, 'quiet minutes', tn)

caught 554 missed 137 false pages 3027 quiet minutes 3782

Put currency on it and the pick makes itself. Say a missed incident costs an hour of degraded service and a false page costs five minutes of an engineer half asleep. Moving from the 0.5 threshold to 0.348 catches 119 more incidents and adds about 1,311 false pages, so you spend roughly 109 hours of interruptions to save 119 hours of degraded service, a trade that only clears if your service level matters more than your rota. Move it the other way when pages are cheap and misses are not, say a payments path where an hour of degradation makes a headline. No metric makes that call; it is the same judgement you already exercise every time you set an alert by hand, only now the line moves in one number and you can defend where you put it.

Two teams reading this scoreboard will land in different places, and both can be right. An internal batch platform where a missed incident means a late report will take the 0.5 threshold and its 1,716 false pages without complaint, because a page there costs almost nothing. A payments path will run the 0.348 line or lower and swallow the false pages, because one missed incident is a customer facing outage worth far more than an interrupted afternoon. Store the chosen threshold beside the model as a number you can change, alert when a minute scores above it, and you have turned a fitted classifier into the same tunable control you already trust in your alerting stack, which is exactly what the later parts on serving and monitoring pick up.

flowchart LR
  A[Alert rule baseline] --> B[Labelled month in time order]
  B --> C[Split by time not at random]
  C --> D[Fit scaled logistic regression]
  D --> E[Precision recall curve]
  E --> F[Pick threshold on cost]
  F --> G[Operating classifier]

Score your alert rule against a classifier this week

Do one thing before the next part. Take a real alert rule and a slice of labelled telemetry, minutes inside your last few incidents marked 1 and the rest 0, and score the rule itself with recall and precision so you know the floor. Then fit one scaled logistic regression on a time ordered split, read its precision recall curve, and pick a threshold against a recall you can name out loud. Measuring the baseline before trusting the model is the same discipline the AI Engineering Series argues for when evaluating generative AI output, where a score means nothing without a floor to read it against. Next part hardens this evaluation, cross validation and the quieter forms of leakage that inflate a model without ever raising an error.

This scoreboard is also the first real entry in the portfolio this series has been building toward. A hiring manager does not want to hear that you know logistic regression; they want to see that you took a production alert catching 1 incident in 691, replaced it with a model catching 554, and could say in one breath what it cost in false pages. Save the table, the curve and the threshold you chose, because that trio is a stronger portfolio piece than any tutorial notebook on a public dataset, and a later part turns exactly this kind of artifact into a resume line.

References

• scikit-learn documentation, precision_recall_curve
• scikit-learn documentation, average_precision_score
• scikit-learn documentation, roc_auc_score
• scikit-learn documentation, LogisticRegression
• scikit-learn documentation, train_test_split
• Data Science Series, model evaluation, cross validation and leakage
• Infra to Data Science, the Complete Guide

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts