Most incident prediction projects fail before the model is ever wrong. They fail on arithmetic. Incidents are rare, and rarity fixes a ceiling on precision that no algorithm clears, so a predictor that has the physics right still pages your on-call twenty times for every real event it catches. This part is the honest accounting the whole project has been building toward, what machine learning on infrastructure data can predict, what it only appears to predict, and where a vendor selling outage prophecy is selling you the base rate fallacy inside a dashboard.

Rare Events and the Base Rate Trap

Start with the number that ends most projects, because it costs nothing to compute and it settles the question before a line of model code exists. Pick a service that has a real incident once in a thousand one minute windows, a 0.1 percent base rate, which is already generous for a healthy system. Grant yourself a strong model, 80 percent recall, and a false positive rate of 2 percent, which sounds tight until you count what it does to the quiet windows. Out of a thousand windows one is a real incident and your model catches it four times in five; the other 999 are quiet and your model wrongly flags 2 percent of them, about twenty. Twenty false alarms next to one real catch is 3.8 percent precision, and precision is what your on-call feels at three in the morning.

# tested with python 3.10.12, scikit-learn 1.7.2, numpy 2.2.6, pandas 2.3.3
def precision_ceiling(base_rate, recall, fp_rate):
    tp = recall * base_rate                 # fraction of windows that are caught incidents
    fp = fp_rate * (1 - base_rate)          # fraction of windows that are false alarms
    return tp / (tp + fp)

for br in (0.10, 0.01, 0.001, 0.0001):
    p = precision_ceiling(br, recall=0.80, fp_rate=0.02)
    alarms = 0.02 * (1 - br) * 1440         # false alarms per day at 1440 windows
    print('base', br, 'precision', round(p * 100, 1), 'percent', 'false alarms/day', round(alarms, 1))

base 0.1 precision 81.6 percent false alarms/day 25.9
base 0.01 precision 28.8 percent false alarms/day 28.5
base 0.001 precision 3.8 percent false alarms/day 28.8
base 0.0001 precision 0.4 percent false alarms/day 28.8

Read the columns together and the trap is plain. False alarms per day barely move, from 25.9 to 28.8, because the quiet windows dominate at every base rate and 2 percent of a near constant number is a near constant number. Precision falls off a cliff, from 81.6 percent when incidents are common to 0.4 percent when they are genuinely rare, because the one quantity that shrinks is the count of real events in the numerator. That is the base rate fallacy in operational terms: your alert quality is set by how rare the event is, and no threshold, no fancier model and no deeper network changes the arithmetic. A predictor is worth building only when the base rate, the recall you can reach and the false alarms you can absorb line up, and precision_ceiling tells you that in one call before you spend a sprint.

Same assumptions as the code, 80 percent recall and a 2 percent false positive rate across 43,200 windows a month.

AIOps Claims Against the Evidence

Vendors have promised outage prediction for a decade under the banner AIOps, machine learning for IT operations. In 2025 the analyst firm that coined the market, Gartner, retired the label and renamed the category Event Intelligence, on the reasoning that the term had been stretched past meaning and that infrastructure teams had grown disillusioned with what it delivered. Read that move as a data point, not marketing. Correlation across noisy alert streams, deduplication, and routing an incident to the right team are the capabilities that survived the rename, because they work on data you already hold and their success is easy to check. Prediction of the outage before it starts is the capability that quietly fell out of the headline, because it runs into the base rate wall from the last section. My working boundary is blunt: buy event intelligence to correlate and route the alerts you already get, never to foretell the outage you have not had, and treat any vendor demo that predicts failures on their sandbox as a base rate that will not survive contact with your production. Deciding what counts as success before you build is the habit that keeps a shipping feature honest, a discipline the AI Engineering Series argues for in its note on building an eval set first.

Disk Failure Prediction, Where It Genuinely Works

Prediction is not hopeless, and the clearest place it works is disk failure. In the largest published study, researchers tracked 380,000 drives across 64 data center sites for roughly seventy days and reached a 0.95 F-measure at a ten day lead time, early enough to drain and replace a drive before it takes data with it. Two properties made that possible. Failure is repeated and labelled, since a fleet that size logs thousands of real failures a year, so the base rate over the fleet is workable even though any single drive almost never dies. Rich features carry the signal, because the model combined SMART self reporting with performance counters and physical location, and the same study showed SMART attributes alone were not enough. An honest caveat rides along with the headline: prediction quality drops as the lead time stretches, and independent work on the public Backblaze data lands nearer 95 percent precision at 67 percent recall, which means a third of failing drives still slip through in the best studied prediction problem in all of infrastructure.

flowchart TD
  S[Signal in your data] --> A{Lead time worth acting on}
  A -->|No| D[Detect faster instead]
  A -->|Yes| B{Hundreds of labelled failures}
  B -->|No| D
  B -->|Yes| C{Base rate above one percent}
  C -->|No| R[Rank and triage, no auto page]
  C -->|Yes| T[Train a predictor with an alarm budget]

Precision, Recall and Alert Fatigue

A model that predicts no incident, ever, scores 99.9 percent accuracy on a 0.1 percent base rate, and that single fact should retire accuracy as a metric for this work. ROC-AUC is only a little better, because it averages over thresholds that page the whole night to reach the last few true positives, and on rare events it can read a healthy 0.9 while precision at any usable threshold sits in the single digits. Precision recall on the actual operating point is the measurement that matches the pager, and PR-AUC summarises it across thresholds while ignoring the vast quiet majority that inflates every other score. Set an alarm budget before you train, a hard count of false pages per week your team will accept, and treat any model that cannot reach its recall inside that budget as a failed experiment rather than a tuning problem.

Leakage That Makes Prediction Look Solved

That war story has a lesson worth making concrete, because leakage is the single most common way a prediction project reports a success that evaporates in production. Leakage is training on information you would not possess at prediction time, and on operational data it hides in plain sight, since so many of your richest signals are written during or after the incident by the very systems reacting to it. Here is the whole failure in a dozen lines, a synthetic month of one minute windows with a leaked alert count sitting next to honest pre incident telemetry.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, recall_score

rng = np.random.default_rng(7)
n = 43200                                    # one month of one minute windows
y = (rng.random(n) < 0.001).astype(int)      # about 0.1 percent incident base rate

cpu = rng.normal(50, 12, n) + y * 6          # weak honest pre incident signals
lat = rng.normal(120, 30, n) + y * 20
alerts = y * rng.integers(1, 5, n)           # LEAK: only populated during the incident

cut = int(n * 0.8)                           # time ordered split, last 20 percent is the future
def score(X, tag):
    m = LogisticRegression(max_iter=1000, class_weight='balanced').fit(X[:cut], y[:cut])
    p = m.predict_proba(X[cut:])[:, 1]
    print(tag, 'PR-AUC', round(average_precision_score(y[cut:], p), 3),
          'recall', round(recall_score(y[cut:], m.predict(X[cut:]), zero_division=0), 3))

print('incident windows', int(y.sum()), 'base rate', round(y.mean() * 100, 3), 'percent')
score(np.c_[cpu, lat, alerts], 'with leaked column')
score(np.c_[cpu, lat],         'pre incident only ')

incident windows 37 base rate 0.086 percent
with leaked column PR-AUC 1.0 recall 1.0
pre incident only  PR-AUC 0.006 recall 0.545

Two honest features and one poison column tell the story. With the leaked alert count in the matrix, PR-AUC reads a perfect 1.0 and the model looks like a triumph, because the column is a copy of the label written by the monitoring pipeline. Drop it and keep only the CPU and latency you would actually have a minute before, and PR-AUC collapses to 0.006, a hair above the 0.00086 you would get by ranking at random for this base rate. Nothing about the honest model is broken; the pre incident telemetry simply does not carry a minute ahead signal for this failure, and the leaked run was measuring the monitoring system, not predicting anything. A time ordered split earns its place here too, since training on the past and scoring the future closes the other classic leak, a random split quietly borrowing future rows into training. For the full mechanics of both traps, the Data Science Series covers them in its part on evaluation, cross validation and leakage, so the rule to carry away is narrow: audit every feature for whether it exists before the event, and split on time.

Prediction Versus Detection, a Decision Table

Step back from prediction and the honest alternative comes into focus, detection that is fast rather than early. Most of what teams want from prediction, fewer surprises and less firefighting, they get more reliably by shrinking the gap between a failure starting and a human knowing, which is a detection problem the earlier parts already solved. Match the tool to the failure with the table below, and notice that supervised prediction earns a place in only one narrow row. The anomaly work from Part 21 is the detection side of this same choice.

Read the third row as the whole argument of this part. Supervised prediction is not useless, it is specialised, and it pays off in the same shape of problem every time, a failure that recurs often enough to label, that carries signal in features you hold before the event, and that gives you lead time long enough to act. Disk replacement fits. Predicting the next novel outage across your whole estate does not, and no model in the table changes that.

Detect Now, Predict Only Where Lead Time Pays

Spend your next week on detection, not prophecy. Take the anomaly detector and the log templates you already built, wire them to page faster and with less noise, and you will move the number your team actually cares about, time to know, further than any predictor would. Reserve supervised prediction for the one or two failures on your estate that are repeated, labelled and worth acting on ahead of time, and gate even those behind precision_ceiling and an alarm budget before you write a single feature. On Monday, list your five most frequent incident types, run precision_ceiling against each one honest base rate, and cross off every type where the ceiling sits below the precision your on-call will tolerate. What survives that arithmetic is your entire prediction backlog, and it is usually short. Next part turns from the data to the career, building a portfolio from these projects and translating an operations resume into the language a hiring manager reads.

References

• Lu et al, Making Disk Failure Predictions SMARTer!, USENIX FAST 2020
• Gartner, Market Guide for Event Intelligence Solutions, 2025
• scikit-learn average_precision_score
• scikit-learn, Precision Recall example
• Backblaze Drive Stats, public hard drive test data
• Data Science Series, Evaluation, Cross Validation and Leakage
• AI Engineering Series, Build an Eval Set First

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts