One line of output ended an argument I had been losing for a week:

95 pct CI on AUC difference: [0.0032, 0.0938]

My model B beat model A by 4.8 points of area under the curve on a holdout of 1409 customers. Everybody in the room had already decided model B was the winner. That interval says something more uncomfortable: the honest claim is that model B is somewhere between a third of a point and nine points better. Both ends of that range lead to different decisions. Shipping on the point estimate alone means shipping on a number that the data does not support to three decimal places, or even to one.

Statistical inference is the discipline of putting that interval on every number you report. It is not an academic ritual bolted on at the end. It is how you avoid spending two sprints chasing an improvement that was noise all along.

Where the churn project stands after Part 7

Last part we had the vocabulary but no arithmetic. We knew that 1869 of the 7043 customers in the Telco Customer Churn table had churned, a base rate of 26.54 percent, and we knew that a predicted probability is a claim about a group rather than about one person. What we could not do was settle the question left hanging in Part 6: whether a 0.004 gain in area under the curve across five folds was a real improvement or fold noise.

This part adds the arithmetic. By the end you will have an interval around the base rate, an interval around a model metric, and a rule for when a difference between two models deserves a slide. No new features get engineered here. What changes is that every number the project produces from now on carries an error bar.

One practical note before we start. Where descriptive statistics describe the rows you have, inference makes a claim about the rows you do not have: next month’s customers, the ones you have not scored yet. That leap is the whole subject. Everything below is a way of quantifying how far you are allowed to leap.

Sampling variation, and why one number is never one number

Your 7043 rows are a sample. Not a sample in the survey sense where somebody drew names from a frame, but a sample in the sense that matters: had the export run a month later, or had the extract cut off at a different date, you would have a different table and every statistic you compute from it would come out slightly different.

Sampling distribution is the name for the spread of a statistic across all the samples you might have drawn. You only ever observe one draw from it. Inference is the set of techniques for reasoning about that unobserved spread using the single sample in front of you. Two routes exist. Where mathematics gives a formula for the spread, use the formula. Where it does not, simulate the spread by resampling. Both routes appear below and they agree closely when both apply, which is a good sanity check.

A quick intuition for scale. Standard error of a proportion is the square root of p times one minus p, divided by the square root of n. At a churn rate near 0.265 and n of 7043, that is about 0.0053. Multiply by 1.96 and you get roughly one percentage point on each side. So the base rate is 26.5 percent give or take a point, and any story that depends on whether it is 26.2 or 26.9 is a story your data cannot support.

Confidence intervals you can compute and defend

A 95 percent confidence interval is a recipe with a guarantee about the recipe, not about your particular interval. Repeat the whole exercise many times on fresh samples and 95 percent of the intervals the recipe produces will contain the true value. Your one interval either contains it or it does not. That distinction sounds pedantic until somebody asks whether there is a 95 percent chance the true churn rate lies between 25.5 and 27.6 percent, and the correct answer is no, that is not what the number claims.

For a proportion, four standard methods exist and they disagree in exactly the situations where you care. Here is the computation on the churn base rate, written out so you can see what each method actually does.

import math

n, k = 7043, 1869          # rows, churned
p = k / n
z = 1.959963984540054      # 97.5th percentile of the standard normal

# Wald, the textbook normal approximation
se = math.sqrt(p * (1 - p) / n)
wald = (p - z * se, p + z * se)

# Wilson, which inverts the test instead of approximating the interval
d = 1 + z * z / n
centre = (p + z * z / (2 * n)) / d
half = (z / d) * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
wilson = (centre - half, centre + half)

print('point   %.5f' % p)
print('wald    %.5f %.5f' % wald)
print('wilson  %.5f %.5f' % wilson)

Tested with Python 3.10.12 and NumPy 2.2.6. Only the standard library is needed for this block.

point   0.26537
wald    0.25506 0.27568
wilson  0.25519 0.27581

In production you would not hand roll this. A single call does all four methods, and it is worth reading the source once to see that the formulas match what is above.

from statsmodels.stats.proportion import proportion_confint

# method is one of normal, wilson, agresti_coull, beta, binom_test
proportion_confint(1869, 7043, alpha=0.05, method='wilson')

Signature verified against the statsmodels 0.14.6 API reference. Note that the argument is alpha and not confidence level, which catches people out: alpha of 0.05 gives you a 95 percent interval, and passing 0.95 by mistake gives you a 5 percent interval that looks absurdly tight and is easy to miss in a report.

Here are all four methods on the churn base rate, computed with the code above and with a Clopper-Pearson interval found by inverting the binomial tails.

At n of 7043 the four methods agree to within 1.4 hundredths of a percentage point, which is a useful thing to know: arguing about interval methods on a large table is wasted effort. Run the same comparison at n of 40 with two positives and the Wald interval will hand you a lower bound below zero, which is nonsense for a proportion. That is the case the choice is for.

Bootstrap intervals for metrics with no formula

Proportions have a formula. Area under the curve, precision at the top 5 percent, average revenue saved per intervention, and every other metric a stakeholder actually asks about either have no clean formula or have one that rests on assumptions your data breaks. Bootstrap solves all of them with one idea: treat your sample as a stand in for the population, draw new samples from it with replacement, and watch how much the metric moves.

Below is the bootstrap that produced the opening line. To keep it self contained and reproducible for you without a trained model, it uses simulated scores for two models over a holdout of 1409 rows, the size of a 20 percent split of the churn table, with a fixed seed. Point the same function at your real predictions and nothing else changes.

import numpy as np

rng = np.random.default_rng(42)
n = 1409
y = rng.binomial(1, 0.2654, n)          # labels at the churn base rate
signal = y * 0.55
a = signal + rng.normal(0, 1.000, n)    # model A scores
b = signal + rng.normal(0, 0.985, n)    # model B scores, slightly sharper

def auc(yv, sv):
    m = len(yv)
    order = np.argsort(sv, kind='mergesort')
    r = np.empty(m)
    r[order] = np.arange(1, m + 1, dtype=float)
    npos = yv.sum()
    nneg = m - npos
    return (r[yv == 1].sum() - npos * (npos + 1) / 2) / (npos * nneg)

print('AUC A %.4f  AUC B %.4f  diff %.4f'
      % (auc(y, a), auc(y, b), auc(y, b) - auc(y, a)))

B = 2000
diffs = np.empty(B)
for i in range(B):
    idx = rng.integers(0, n, n)         # resample rows with replacement
    diffs[i] = auc(y[idx], b[idx]) - auc(y[idx], a[idx])

lo, hi = np.percentile(diffs, [2.5, 97.5])
print('95 pct CI on diff: [%.4f, %.4f]' % (lo, hi))
print('resamples where B loses: %.3f' % (diffs < 0).mean())

AUC A 0.6253  AUC B 0.6736  diff 0.0483
95 pct CI on diff: [0.0032, 0.0938]
resamples where B loses: 0.019

Two details in that loop matter more than they look. Resampling row indices and applying them to both models keeps the comparison paired, so the resamples share the same customers and the difference is not inflated by two independent draws. And 2000 resamples is enough for a 95 percent interval; going to 10000 changed my bounds by less than 0.001 and cost five times the runtime.

Bootstrap has one failure mode that will bite you on a churn problem specifically, and it does not announce itself with an exception. It returns a number that is not a number.

# a small segment: 60 customers, roughly 5 percent churn
rng = np.random.default_rng(3)
y = rng.binomial(1, 0.05, 60)
s = rng.normal(0, 1, 60)

out = [auc(y[i], s[i]) for i in
       (rng.integers(0, 60, 60) for _ in range(500))]
print(np.percentile(out, [2.5, 97.5]))

e.py:10: RuntimeWarning: invalid value encountered in scalar divide
  return (r[yv == 1].sum() - npos * (npos + 1) / 2) / (npos * nneg)
[nan nan]

Why it happens: with three positives in sixty rows, some resamples draw zero positives, npos becomes 0, and the denominator collapses. NumPy warns rather than raising, so if warnings are suppressed in your notebook the nan propagates silently into a percentile and out into a slide. Fix it two ways. Stratify the resample so positives and negatives are drawn separately in their original proportions, and add an assertion that every resample has at least thirty positives. If it cannot, your segment is too small for a bootstrap interval and the honest output is a count, not a metric.

Misreadings of a p value that cost money

A p value answers one narrow question: assuming there is genuinely no effect, how often would data at least this extreme show up by chance. That is all. Everything people commonly infer from it beyond that is wrong, and three of those wrong inferences are expensive.

First, a p value of 0.03 does not mean a 97 percent chance the effect is real. It says nothing about the probability of any hypothesis. It is a statement about data given a hypothesis, and those two directions are not interchangeable. Second, a p value above 0.05 is not evidence of no effect. Absence of evidence and evidence of absence are different findings, and a wide confidence interval that includes zero tells you honestly that you do not know, which a bare p value hides. Third, and this is the one that quietly destroys model selection work, a p value is only valid for a test you decided on before looking at the data.

That third point deserves a demonstration, because it is the exact shape of a feature selection loop. Here are 24 comparisons between two groups where by construction there is no difference at all.

import numpy as np, math
from math import erfc

def welch_p(x, y):                 # two sided, large sample normal tail
    nx, ny = len(x), len(y)
    vx, vy = x.var(ddof=1), y.var(ddof=1)
    t = (x.mean() - y.mean()) / math.sqrt(vx / nx + vy / ny)
    return erfc(abs(t) / math.sqrt(2))

rng = np.random.default_rng(1)
ps = [welch_p(rng.normal(0, 1, 1409), rng.normal(0, 1, 1409))
      for _ in range(24)]

print('smallest p:', round(min(ps), 4))
print('how many under 0.05:', sum(p < 0.05 for p in ps))
print('chance of at least one by luck:', round(1 - 0.95 ** 24, 3))

smallest p: 0.0079
how many under 0.05: 4
chance of at least one by luck: 0.708

Four significant results out of twenty four, from data with no signal whatsoever. Repeating the whole exercise across 300 different seeds, 206 of them produced at least one result under 0.05, which is 68.7 percent against a theoretical 70.8 percent. That is not a quirk of seed 1. It is what happens whenever you test enough things.

Comparing two models without fooling yourself

Now we can answer the question left over from Part 6. Was a 0.004 gain in area under the curve real? Look at the bootstrap output above. A 4.8 point difference, roughly twelve times larger, produced a 95 percent interval running from 0.3 to 9.4 points. A 0.4 point difference on the same holdout would sit comfortably inside the noise. Verdict: that gain was fold noise, and the target encoded features it came from should be kept or dropped for reasons other than that number.

Three rules follow from this and they are worth writing on the wall. Always compare models on the same resampled rows, never on independently drawn ones, because paired comparison removes the variance you do not care about. Always report the interval on the difference rather than intervals on each model separately, since two overlapping intervals can still contain a difference that reliably favours one side. And always state the number of positives, because that, not the row count, drives how wide the interval is.

Choosing which tool to reach for is mechanical once you know what the metric is.

flowchart TD
  A[Interval needed for a metric] --> B{Metric is a proportion}
  B -->|yes| C[Wilson interval]
  B -->|no| D{Closed form standard error exists}
  D -->|yes| E[Normal or t interval]
  D -->|no| F[Bootstrap the rows]
  F --> G{At least 30 positives per resample}
  G -->|no| H[Report raw counts instead]
  G -->|yes| I[Percentile interval, 2000 draws]
  C --> J[Put the interval on the slide]
  E --> J
  I --> J

Sample size and what to promise a stakeholder

Precision costs data, and it costs it at an unforgiving rate. Halving an error bar needs four times the rows. That single fact should shape what you agree to before a project starts, because it converts a vague request into an arithmetic answer.

Read this table before you agree to detect a small effect on a small segment.

Notice how little the last two rows buy. Going from 20000 to 50000 rows, a 150 percent increase in data and cost, narrows the interval by 0.22 percentage points. Beyond a certain size the binding constraint stops being sample size and becomes whether your labels mean what you think they mean. Spending on label quality beats spending on volume at that point, and I have watched teams miss this and buy a bigger warehouse instead of fixing a churn definition.

Reporting rule I would adopt for model comparisons

Here is what I would put in a team standard, and it is three lines long. Every model metric on a slide carries a bootstrap 95 percent interval and a count of positives. Every comparison between two models is a paired bootstrap on the difference, not two separate intervals placed side by side. And any result that came from picking a winner out of more than about five candidates gets described as an estimate to be confirmed on a fresh holdout, never as a measurement.

What I would avoid is the reflex of reaching for a significance test on model comparisons. Corrected paired t tests over cross validation folds exist and they are defensible, but folds are not independent, the corrections are approximations, and the output is a single binary that stakeholders over read. A bootstrap interval on the difference gives the same information plus the magnitude, and nobody has ever misinterpreted an error bar as badly as they misinterpret a p value. Reach for a formal test when somebody external requires one; reach for the interval every other time.

Part 9 puts a linear regression on the churn table and starts producing coefficients, and coefficients come with standard errors, which is this part applied to a model’s internals rather than its scores. Everything about intervals and about what a p value on a coefficient does not tell you carries straight across. Before you go, do one thing on your own project: take whatever headline metric is currently on your team’s dashboard, bootstrap it, and put the interval next to it. If the interval is wider than the change everyone has been discussing, you have just saved yourself a quarter.

References

• statsmodels: proportion_confint API reference
• SciPy: stats.bootstrap, percentile, basic and BCa methods
• NumPy: random Generator and default_rng
• IBM: Telco customer churn sample dataset

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts