A director asked me one question in a review: what is our average response time. I said 40 milliseconds and the room relaxed. That number was true and the relief was misplaced, because at that same moment roughly one request in a hundred was taking a quarter of a second and the very slowest were past two seconds. An average is the number that hides the customers who are actually angry. Statistics for an operator starts right there, with the arithmetic that tells you what a single friendly summary is quietly leaving out, and none of it needs a maths degree, only the same suspicion you already bring to a green dashboard.
Averages hide the tail your users feel
Where the project stands: last part you reshaped the cluster frame with column operations, and you also have a latency log from the same systems. This part describes that data honestly before any model touches it, because a model trained on a lie about the data learns the lie. Start with the summary everyone reaches for, the mean, and put it next to the percentiles you already trust from your SLOs. You know p95 and p99 from latency dashboards, so this is your own vocabulary, made explicit.
# tested on Python 3.10.12, numpy 2.2.6, pandas 2.3.3, scipy 1.15.3
import numpy as np
# lat: 500,000 request latencies in milliseconds, from the monitoring export
print('mean', round(lat.mean(), 1))
for p in [50, 90, 95, 99, 99.9]:
print('p' + str(p), round(np.percentile(lat, p), 1)) # method is linear by default
print('above mean pct', round((lat > lat.mean()).mean() * 100, 1))
mean 39.6 p50 30.2 p90 63.4 p95 80.8 p99 228.1 p99.9 737.7 above mean pct 31.6
Read those seven lines slowly, because they overturn the instinct to report an average. A mean of 39.6 ms is not the middle of anything: only 31.6 percent of requests are slower than it, which places the average near the 68th percentile, dragged up by a tail it cannot see. A median of 30.2 ms is the request a typical user waits for. p99 at 228.1 ms is 7.6 times the median, and p99.9 at 737.7 ms is the slow morning nobody forgets. If you report one number to a stakeholder, report the median and one tail percentile, never the mean alone, because the mean is the average of people who are fine and people who are furious, and it flatters both away. Tail latency is exactly the signal the AI Engineering Series watches when it argues for caching and latency engineering, where p99 not the average decides whether a call budget holds.
Distributions before summaries
Every summary number assumes a shape, and the assumption almost every tutorial hands you is the bell curve, where mean plus or minus two standard deviations covers about 95 percent of values. Operational data rarely obeys it. Latency, request size, time between failures, queue depth, these pile up near a floor and trail off into a long right tail. Compute the three sigma bounds on the latency and watch the assumption fail out loud.
from scipy import stats
mu, sd = lat.mean(), lat.std()
print('mean', round(mu, 1), 'std', round(sd, 1))
print('mean+2std', round(mu + 2*sd, 1))
print('mean-2std', round(mu - 2*sd, 1)) # the tell
print('flagged by mean+2std pct', round((lat > mu + 2*sd).mean() * 100, 2))
print('skew', round(stats.skew(lat), 2)) # 0 would be symmetric
mean 39.6 std 51.9 mean+2std 143.4 mean-2std -64.3 flagged by mean+2std pct 1.52 skew 11.43
There is the failure, printed in one line: mean minus two standard deviations is negative 64.3 ms. No request finishes in negative time, so a rule that produces an impossible lower bound is telling you the shape is wrong, not that some latencies are below zero. Skew of 11.43, where a bell curve is 0, confirms a hard right lean. An upper bound of 143.4 ms looks plausible until you notice it flags only 1.52 percent of requests as outliers, sitting awkwardly between p95 at 80.8 ms and p99 at 228.1 ms, describing neither. That lesson contradicts the first thing most stats courses teach: do not summarise operational data with mean and standard deviation until you have looked at its shape, because the moment the data is skewed those two numbers describe a distribution you do not have. Plot a histogram, or at least read the percentiles, first. Distribution mechanics get built properly in the Data Science Series part on probability and distributions, which this series assumes rather than repeats.
One production gotcha worth carrying from the start: you cannot average percentiles. If three shards each report a p99 of 200 ms, the fleet p99 is not 200 ms, and averaging the three is meaningless, because a percentile is a position in a sorted list, not a quantity you add. To combine tail latency across shards you keep a histogram or a sketch such as t-digest per shard and merge those, then read the percentile off the merged shape. I have watched a dashboard proudly average per node p99 values and report a fleet number that was wrong by a factor that shifted with traffic, the kind of quiet error that survives for months because it always looks reasonable.
Spread that survives a long tail
If the standard deviation is misleading on skewed data, you still need a way to say how spread out the bulk of requests is. Two robust measures do the job without letting a handful of two second requests dominate. An interquartile range, IQR, is the width of the middle half, p75 minus p25. A median absolute deviation, MAD, is the median of how far each point sits from the median. Both ignore the extremes by construction, which is the point.
from scipy import stats
q1, q3 = np.percentile(lat, [25, 75])
print('std ', round(lat.std(), 1))
print('IQR ', round(q3 - q1, 1))
print('MAD ', round(stats.median_abs_deviation(lat), 1))
print('median', round(np.median(lat), 1))
std 51.9 IQR 23.4 MAD 10.9 median 30.2
Look at the gap. Standard deviation is 51.9 ms, larger than the median itself, because it squares distances and the tail carries the squares. IQR at 23.4 ms and MAD at 10.9 ms describe where the bulk of requests actually live, tight around 30 ms. When you build an anomaly rule later in this series, this is the difference between a threshold that fires on ordinary traffic and one that fires on genuine outliers. A common robust flag is a value more than three MADs from the median, which here is about 30.2 plus three times 10.9, near 63 ms, a threshold that tracks the real bulk instead of being inflated by the very outliers you want to catch.
Confidence in a single number
You quote a p99 in a review and it lands as a fact. It is really an estimate from a sample, and estimates wobble. A bootstrap makes the wobble visible: resample the data with replacement thousands of times, recompute the statistic each time, and read off the spread. scipy has this built in, and it has one sharp edge that catches everyone on the first try, so here is the wrong call and its real error before the fix.
from scipy import stats sample = np.random.default_rng(7).choice(lat, 5000, replace=False) ci = stats.bootstrap(sample, lambda x, axis=-1: np.percentile(x, 99, axis=axis))
ValueError: each sample in `data` must contain two or more observations along `axis`.
bootstrap expects data as a sequence of samples, not a single array, so it read each latency value as its own one element sample and refused. A fix is one character, wrap the array in a tuple, then ask for the interval.
ci = stats.bootstrap((sample,), lambda x, axis=-1: np.percentile(x, 99, axis=axis),
n_resamples=2000, confidence_level=0.95, random_state=7)
print('sample p99', round(np.percentile(sample, 99), 1))
print('95 pct CI ', round(ci.confidence_interval.low, 1), 'to',
round(ci.confidence_interval.high, 1))
sample p99 191.5 95 pct CI 152.6 to 296.3
That point estimate is 191.5 ms, and the interval runs from 152.6 to 296.3 ms, a width of nearly 144 ms. That is the honest reading: a p99 measured off 5,000 requests is not one number, it is a range, and the tail percentiles are the shakiest because so few points define them. So when a p99 jumps from 190 to 240 ms between two short windows, check whether that move even clears the interval before you page anyone, because often it does not. Report a tail percentile with its interval, or widen the window until the interval tightens.
Before and after a change, and the significance trap
You ship a config change and want to prove it helped. A textbook move is a significance test, and on operational data that test will mislead you in a specific, predictable way. Take 400,000 request latencies before a change and 400,000 after, where the after set is genuinely a hair faster, and run a Mann-Whitney U test, which compares distributions without assuming a bell curve.
from scipy import stats
print('before median', round(np.median(before), 2))
print('after median ', round(np.median(after), 2))
res = stats.mannwhitneyu(before, after, alternative='greater')
print('p-value', format(res.pvalue, '.2e'))
gain = np.median(before) - np.median(after)
print('median gain ms', round(gain, 2), 'which is', round(100*gain/np.median(before), 2), 'pct')
before median 29.95 after median 29.81 p-value 5.57e-05 median gain ms 0.13 which is 0.44 pct
A p-value of 5.57e-05 is significant by any bar you were taught, and the change it certifies is 0.13 ms, under half a percent of the median. Nothing is wrong with the test. A trap hides in what significance means: it answers whether a difference is real, not whether it is worth anything, and on 400,000 samples even a difference too small to feel clears the threshold easily, because the test grows more sensitive as the sample grows. Report the effect size first, the 0.13 ms, and let the p-value confirm it is real rather than lead the story. This is the same discipline the Data Science Series applies to model evaluation and leakage, where a number that looks decisive is worthless until you know what it is really measuring.
Statistics checklist and project status
Where the project stands now: the latency export and the cluster frame are described honestly, with the median, tail percentiles, a robust spread and an interval around the shaky numbers, which is the footing every model from Part 13 onward stands on. Keep the table below as the reference artifact, one row per question you actually ask about operational data and the statistic that answers it without lying.
| Question you are asking | Statistic to use | Not this |
|---|---|---|
| Typical experience | median, p50 | mean, the tail drags it up |
| Worst common case | p95 or p99 | max, one fluke sets it |
| Spread of the bulk | IQR or MAD | standard deviation on skewed data |
| How sure of a number | bootstrap interval | the point estimate alone |
| Did a change help | effect size, then a test | the p-value alone |
| Is this point unusual | three MADs from the median | mean plus two standard deviations |
| Metric | Value, ms | What it answers |
|---|---|---|
| mean | 39.6 | almost nothing useful here |
| p50, median | 30.2 | the request a typical user waits for |
| p95 | 80.8 | a slow but common request |
| p99 | 228.1 | the tail users complain about |
| p99.9 | 737.7 | the worst one in a thousand |
Describe before you model checklist
- Look at the shape, a histogram or the percentiles, before quoting any single summary.
- Lead with the median and one tail percentile, not the mean.
- On skewed data use IQR or MAD for spread, not standard deviation.
- Put a bootstrap interval around any tail percentile you plan to alert on.
- For a before and after, report the effect size first and let the p-value confirm it.
- Flag outliers by distance from the median in MADs, not by mean plus two standard deviations.
Compute five numbers on your own latency this week
Do one thing after this part. Pull a week of one real latency series you own, and compute five numbers on it: the median, p95, p99, the IQR, and a bootstrap interval around the p99. Compare the median to whatever average your dashboard reports today, and I expect the gap to surprise you the way 30.2 against 39.6 surprised the room in that review. Those five numbers are the honest description this series builds every later model on. Next part turns to probability and distributions for real systems, so you can name the shapes your data actually takes instead of assuming a bell curve that is not there.
References
- NumPy documentation, percentile and its linear default method
- SciPy documentation, stats.bootstrap for confidence intervals
- SciPy documentation, median absolute deviation
- SciPy documentation, Mann-Whitney U test
- pandas documentation, DataFrame describe
- Data Science Series, probability and distributions a modeller needs
- Infra to Data Science, the Complete Guide
About The Author
DrJha