An alert threshold is a model with one feature and no training. You picked a number, cpu over 90, wrote it into a config file, and every minute the system compares one value to that line and pages or stays quiet. Machine learning keeps the idea and drops the guessing: instead of typing the line, you show a program a few thousand labelled minutes and let it fit the boundary across many features at once. That is the whole leap from operations to modelling, and it is a leap in who chooses the number rather than in what the number does. Once you see a threshold as a hand tuned model of one variable, the rest of the vocabulary stops being foreign and starts being familiar work under new names.
Supervised learning in operations terms
Two parts back you exported a month of one cluster of cpu, memory, latency and error metrics and cleaned it; last part you named the distributions underneath. This part turns that same export into a labelled training set and fits a classifier, so every term lands on data you already read at 3 a.m. rather than on a toy dataset. Supervised learning needs two things you can already produce: features, the measured columns, and a label, the outcome you want to predict. For an operator the label is usually already written down. Every incident ticket, every minute inside a declared outage, every alert that turned out real is a labelled example sitting in a system you own, which means the hardest part of a first model, getting honest labels, is a query rather than a project.
Keep the table below as your reference artifact, an operations idea in the left column and the machine learning word for it on the right. Almost every term in a modelling tutorial maps onto something you already do by hand, and the map is short. Read a paper about classifiers as a paper about learned thresholds and you stop translating, which is most of the distance between watching a field feel alien and getting to work in it.
| Operations idea | Machine learning term | What actually changes |
|---|---|---|
| A metric column, cpu percent | Feature | nothing, it is already numeric input |
| An alert threshold | Decision boundary | the line is learned from data, not typed |
| Runbook answer, was this an incident | Label | you supply known answers to learn from |
| Tuning a threshold by hand | Training | an algorithm fits the line on examples |
| Running the check each minute | Inference | the fitted model scores new data |
| Held back recent data | Test set | data the model never saw, used to judge it |
| False page rate | Precision and recall | two numbers replace one accuracy figure |
Classification, regression and the unsupervised jobs
Supervised learning splits into two shapes by what the label looks like. When the answer is a category, incident or healthy, or which of several failure modes fired, the job is classification, and this part builds one of those. When the answer is a number, next hour cpu or days until a disk fills, the job is regression, which the capacity forecasting part later in this series leans on. Both need labelled history and both judge themselves on data held back from training, so the split discipline in this part carries straight over to a regression problem with no change.
Not every problem hands you labels, and that gap is where unsupervised learning earns its place. Anomaly detection asks which minutes look unlike the rest with nobody marking them first, and log clustering folds millions of lines into a handful of shapes you can name, both jobs you meet in later parts on infra data. A rule that saves projects: if you can write down the answer for past examples, reach for supervised learning, because a labelled problem is far easier to check than an unlabelled one. Save unsupervised methods for when labels are genuinely missing rather than merely tedious to gather, since an anomaly with no ground truth is much harder to argue about than a miss you can point to in a ticket.
Labelled training data from your own telemetry
A synthetic frame below stands in for that export: four features an operator already collects and one label, whether the minute became an incident, at a realistic 3.96 percent base rate. Splitting matters more than the model at this stage. You hold back a quarter of the rows the model never sees, so the number you report is earned on unseen minutes rather than memorised ones. Pass stratify so the rare incident class keeps its 4 percent share in both halves; skip it and a random draw can leave the test set with a handful of incidents and a recall figure that swings on noise.
# tested on Python 3.10.12, scikit-learn 1.7.2, numpy 2.2.6
import numpy as np
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(12)
N = 20000
cpu = np.clip(rng.normal(45, 12, N), 1, 100) # cpu percent
mem = np.clip(rng.normal(60, 10, N), 1, 100) # memory percent
errs = rng.gamma(1.4, 0.6, N) # error rate, per second
lat_us = rng.lognormal(3.4, 0.5, N) * 1000.0 # p99 latency in MICROSECONDS
score = 0.07*(cpu-55) + 0.08*(mem-70) + 1.4*(errs-1.2) + 0.02*(lat_us/1000-40)
prob = 1 / (1 + np.exp(-(score - 2.2)))
label = (rng.random(N) < prob).astype(int) # 1 = incident minute
X = np.column_stack([cpu, mem, errs, lat_us])
print('incident rate', round(label.mean()*100, 2), 'percent, count', int(label.sum()))
# stratify keeps the rare class share equal in both halves
Xtr, Xte, ytr, yte = train_test_split(X, label, test_size=0.25, stratify=label, random_state=12)
print('train', Xtr.shape[0], 'test', Xte.shape[0])
print('train rate', round(ytr.mean()*100,2), 'test rate', round(yte.mean()*100,2))
incident rate 3.96 percent, count 793 train 15000 test 5000 train rate 3.97 test rate 3.96
Both halves carry the same 3.96 percent of incidents, so the test set is a fair miniature of the whole. Four columns and a binary label are all a first supervised model needs, and you built them from signals a monitoring stack already emits. Nothing here is specific to synthetic data; swap in your own exported frame with the same four columns and the rest of this part runs unchanged. Richer features, a rolling average or the change since five minutes ago, come later in the feature engineering part; for a first model the raw columns are enough to see every idea land.
Baselines and the accuracy trap on rare incidents
Before fitting anything clever, fit the dumbest thing that could work and make every later number beat it. scikit-learn ships one for exactly this, DummyClassifier, which here always predicts the majority class, healthy, and never looks at a feature.
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score
dummy = DummyClassifier(strategy='most_frequent').fit(Xtr, ytr)
pred = dummy.predict(Xte)
print('accuracy', round(accuracy_score(yte, pred), 4))
print('recall ', round(recall_score(yte, pred, zero_division=0), 4))
accuracy 0.9604 recall 0.0
96.04 percent accuracy, and it never once said incident. Accuracy is correct predictions over all predictions, and when 96 percent of minutes are healthy, a model that answers healthy for everything is 96 percent right and completely useless. That is the most common way a first classifier fools its author, and it cuts against the reflex to lead with accuracy: on rare events, accuracy mostly measures the base rate. Recall is the honest counterpart, the fraction of real incidents caught, and for the dummy it is a flat zero. A chart makes the contrast visible. Watch how accuracy barely moves across three very different models while recall, the number that decides whether you get paged, travels from zero to eighty percent.
Feature scaling and a convergence failure
A reflex next step is to drop the four features into a logistic regression and fit. Run it on the raw columns and the solver chokes, because one feature, latency in microseconds, runs in the tens of thousands while cpu sits between 1 and 100, and gradient based solvers stall when inputs span such different ranges.
from sklearn.linear_model import LogisticRegression LogisticRegression(max_iter=100).fit(Xtr, ytr) # raw, unscaled features
ConvergenceWarning: lbfgs failed to converge after 100 iteration(s) (status=1): STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT.
Raising max_iter hides the warning without fixing the cause. Scaling fixes the cause: StandardScaler centres each feature and divides by its spread, so every column arrives on comparable footing and the solver converges quickly. Fit the scaler inside a pipeline and only on the training split, never on the full dataset, or you leak information from the test set into training through a shared mean and variance. That leakage is quiet because nothing errors; the score simply comes out flattering and then fails to hold in production. That exact failure mode gets a full treatment in the Data Science Series part on model evaluation and leakage, which this series relies on rather than repeats. A balanced class weight, added below, tells the model to treat a missed incident as heavily as a false alarm despite there being twenty five times fewer of them.
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import precision_score
# scaler is fit on the training split only, inside the pipeline, so no leakage
model = make_pipeline(StandardScaler(), LogisticRegression(class_weight='balanced', max_iter=1000))
model.fit(Xtr, ytr)
pred = model.predict(Xte)
print('accuracy ', round(accuracy_score(yte, pred), 4))
print('recall ', round(recall_score(yte, pred), 4))
print('precision', round(precision_score(yte, pred), 4))
accuracy 0.8082 recall 0.8081 precision 0.148
Scaling turned a non converging solver into one that fits in a blink, and the balanced weight pushed recall to 80.8 percent. Precision fell to 14.8 percent in the same move, which is the trade this part is really about, and the next section makes that cost concrete rather than abstract.
A word on model choice, since a first instinct is to reach for something fancier than a straight line. Logistic regression fits a linear boundary and trains in milliseconds, which makes it the right first model precisely because it is easy to reason about and hard to overfit on four features. Trees and gradient boosting bend to nonlinear structure and often win on tabular telemetry, but they also memorise noise when you let them run deep, so keep the linear model as a second baseline above the dummy and reach for a tree only once you can prove it beats the line on held out data. Overfitting has a plain signature you already understand: a model that scores far better on the training half than the test half has memorised rather than learned, the modelling version of a runbook that only works on the one outage it was written for.
Recall and precision as an operations trade off
With scaling sorted, the genuine decision appears, and it is not which algorithm but which mistake you would rather make. Put the three models side by side and read the two columns that matter to an on call engineer, incidents caught and false alarms raised.
| Model | Accuracy | Recall | Incidents caught | False alarms |
|---|---|---|---|---|
| Dummy, always healthy | 96.04 percent | 0 percent | 0 of 198 | 0 |
| Logistic, default weight | 96.6 percent | 18.2 percent | 36 of 198 | 8 |
| Logistic, balanced weight | 80.8 percent | 80.8 percent | 160 of 198 | 921 |
By every reflex the default model looks better, 96.6 percent accuracy and 81.8 percent precision, and it sleeps through 162 of 198 incidents. Its balanced counterpart catches 160 of them and pays with 921 false alarms across 5,000 test minutes. Neither is correct in the abstract; the pick depends on whether a missed incident or a false page costs your team more. A classification report lays the two error types out per class so the choice is made with eyes open.
from sklearn.metrics import classification_report, confusion_matrix
print(classification_report(yte, pred, digits=3, target_names=['healthy', 'incident']))
tn, fp, fn, tp = confusion_matrix(yte, pred).ravel()
print('confusion tn', tn, 'fp', fp, 'fn', fn, 'tp', tp)
precision recall f1-score support
healthy 0.990 0.808 0.890 4802
incident 0.148 0.808 0.250 198
accuracy 0.808 5000
macro avg 0.569 0.808 0.570 5000
weighted avg 0.957 0.808 0.865 5000
confusion tn 3881 fp 921 fn 38 tp 160
Read the confusion counts as four operational outcomes. True positives, 160, are incidents you caught in time; false negatives, 38, are the ones that slipped through; false positives, 921, are pages that woke someone for nothing; true negatives, 3,881, are quiet minutes left quiet. That framing is the same loop you already run when tuning an alert, only now the line is fitted and you can move it deliberately. That same loop, drawn as a machine learning pipeline, appears below, and it is the shape every later part in this series builds on.
Put currency on the trade and the choice makes itself. Say a missed incident costs an hour of degraded service and a false page costs five minutes of an engineer; then 921 false alarms burn about 77 hours of attention to save 124 incidents the default model would have missed, and whether that trades well depends on your service and your rota, not on a textbook. That arithmetic, cost of a miss against cost of a page, is the operator judgement no metric makes for you, and it is exactly the judgement you already exercise every time you set an alerting threshold by hand.
One more lever sits under the hood, and it is worth knowing before the next part leans on it. predict draws the line at a probability of 0.5, but predict_proba hands back the raw score for every minute, so you can slide the decision threshold to trade recall for precision with no retraining at all. Drop it and the balanced model catches even more incidents while raising more false alarms; lift it and the reverse holds. That one dial is how you land on the operating point the pipeline below ends on, and it means recall and precision are not fixed properties of a model but a curve you choose a point on.
Fit a baseline and one model this week
Do one thing before the next part. Take a slice of telemetry you can label, even crudely, every minute inside your last three incidents marked 1 and the rest 0, and run this exact sequence: a stratified split, a DummyClassifier for the floor, then one scaled logistic regression, and read recall and precision instead of accuracy. Fitting the baseline first is the same discipline the AI Engineering Series argues for in building an evaluation set before the feature, a floor you measure against before trusting anything above it. One caution specific to your data: telemetry is time ordered, so a random split leaks the future into the past and flatters every score, which is why a plain train_test_split is the wrong default on a metric series and a time ordered cut is the honest one. Next part builds the first model end to end, from a single metric threshold to a trained classifier, on this same export.
References
- scikit-learn documentation, DummyClassifier baseline
- scikit-learn documentation, train_test_split and stratify
- scikit-learn documentation, LogisticRegression and class_weight
- scikit-learn documentation, StandardScaler
- scikit-learn documentation, precision, recall and classification metrics
- Data Science Series, model evaluation, cross validation and leakage
- Infra to Data Science, the Complete Guide
About The Author
DrJha