A retention manager once stopped a review of mine with a question I could not answer cleanly. She pointed at one row on the screen and asked what a churn probability of 0.30 meant for that specific customer, given that the customer would either leave or stay and there was no third option. I gave her the textbook answer about long run frequencies. She said that was fine for a spreadsheet, but she had to decide today whether to spend money on this one account, and could I tell her whether 0.30 was high.

She was right to push. That question is where probability stops being a maths topic and becomes the thing that decides whether your model earns its budget. Answering it properly needs four ideas: what a probability is claiming, what shape the underlying quantity has, what the base rate is, and whether your model’s numbers can be trusted as numbers rather than as a ranking. This part covers those four, with no code, because the concepts are the hard bit and the library calls come in Parts 8 through 11.

Where the churn project stands after Part 6

Last part we had a fitted feature pipeline over the Telco Customer Churn table: 7043 rows, 21 original columns, one hot encoding under fifteen categories, cross fitted target encoding above it, and a handful of ratio features that beat the raw columns they came from. This part adds no code and no new columns. What it adds is the vocabulary to say whether a number that pipeline produces means anything.

Two questions were left hanging in Part 6 and both are probability questions. First, whether a 0.004 gain in area under the curve across five folds is a real improvement or fold noise. Second, whether the target encoded columns were leaking, which we diagnosed by eye rather than by any principled test. Neither can be settled without knowing how much a metric bounces around when nothing has changed, and that is a question about sampling distributions. Part 8 answers it with confidence intervals. This part builds the ground under it.

One number to keep in your head throughout: of the 7043 customers in that table, 1869 churned. That is 26.54 percent. Every claim below gets tested against it, because a base rate you can recite from memory is the fastest bad model detector you will ever own.

Probability as a claim about long runs

Here is the answer I should have given the retention manager. A predicted probability of 0.30 is not a property of that customer. It is a claim about a group: among all customers your model scores near 0.30, roughly three in ten will churn within the prediction window. For the individual account it is a betting rate, not a forecast. Whether 0.30 is high depends entirely on what the rest of the book looks like, and on that table 0.30 is slightly above the base rate of 0.2654, so it is a mildly elevated risk rather than an alarm.

Two words carry most of the weight in that answer. A random variable is simply a quantity whose value you do not know yet, written as a rule that assigns numbers to outcomes: churn is a random variable taking the value 1 or 0. A distribution is the full list of values that variable can take together with how often each one shows up. Everything else in this part is a consequence of those two definitions.

Worth naming the split early, because it changes how you argue with stakeholders. A frequentist reading treats the probability as a long run rate over repeated draws. A Bayesian reading treats it as a degree of belief that gets updated as evidence arrives. Practically, supervised models are fitted the frequentist way and consumed the Bayesian way: the retention manager updated her belief about one account. You do not need to pick a camp, but you do need to notice when a stakeholder has quietly switched from one to the other, because that is usually the moment a conversation stops making sense.

Independence deserves a sentence too. Two events are independent when knowing one tells you nothing about the other. Almost nothing in a customer table is independent. Tenure, contract type and monthly charges move together, and treating them as independent is how naive Bayes classifiers end up producing confident probabilities near 0 and 1 that no sane person should believe. If a model outputs 0.998 on a customer table, suspect the independence assumption before you celebrate.

Distributions a modeller meets on real tables

Statistics courses teach roughly forty distributions. SciPy ships well over a hundred continuous families alone. In eleven years of modelling I have deliberately chosen a named distribution perhaps a dozen times, and it was always one of six. Learn these six properly and treat the rest as reference material you look up when a specific problem demands it.

Bernoulli is a single yes or no draw, and it is what churn is. Binomial counts successes across a fixed number of independent draws, which is what you are implicitly using when you ask how many of 200 flagged customers should have churned. Poisson counts events in a period when events are rare and roughly independent: support tickets per month, logins per week. Normal describes symmetric, thin tailed quantities and is far less common in business data than its reputation suggests. Lognormal describes anything positive and right skewed, which covers most spend, revenue and duration columns. Exponential describes waiting time until an event, and it is the natural starting point for time to churn rather than churn yes or no.

Table below maps each one to where it actually turns up in this project.

Six families cover the overwhelming majority of modelling decisions on tabular business data.

flowchart TD
  A[Quantity you are modelling] --> B{Count or continuous}
  B -->|Single yes or no| C[Bernoulli]
  B -->|Successes in n trials| D[Binomial]
  B -->|Events per period| E[Poisson]
  B -->|Continuous| F{Bounded below at zero}
  F -->|Yes and right skewed| G[Lognormal or gamma]
  F -->|Yes and time to event| H[Exponential or Weibull]
  F -->|No| I{Symmetric and thin tailed}
  I -->|Yes| J[Normal]
  I -->|No| K[Use the empirical distribution]

Shape, skew and what they do to a model

Skew is the asymmetry of a distribution. Positive or right skew means a long tail of large values pulling the mean above the median, which is the normal state of affairs for money, duration and count columns. Kurtosis describes how heavy the tails are, which matters because heavy tails mean extreme values arrive more often than a normal curve would predict, and those extremes are exactly the rows that dominate a squared error loss.

Shape has three practical consequences. Linear models assume the residuals are roughly symmetric, so a heavily skewed target breaks them until you take logs. Distance based methods such as k means and k nearest neighbours are dominated by whichever column has the widest raw spread, so skew plus no scaling means one column silently decides every neighbourhood. Tree based methods are indifferent to monotone transforms of the inputs, since a split at 100 and a split at log 100 partition the rows identically, which is a large part of why boosted trees win on tabular data with almost no preprocessing.

Worth looking at the actual shape of a column before assuming any of this. MonthlyCharges in the churn table is not skewed at all in the usual sense. It is bimodal, with a tall spike of low bill customers on basic phone service and a broad second hump of bundled fibre customers. A mean of roughly 65 sits in the valley between the two humps and describes almost nobody. That is the sort of thing a single summary statistic hides and a histogram exposes in one glance, which is the argument the exploratory data analysis post makes at length.

Conditional probability and base rates

Conditional probability is the probability of one thing given that you already know another. Written out, the probability of churn given a month to month contract. Every supervised model is an estimator of a conditional probability, which is a useful way to think about what you are actually building: not a churn detector, but a machine that reports how the churn rate changes once you condition on everything in the row.

Contract type on its own moves that conditional rate enormously. Month to month customers churn at roughly 43 percent, one year contracts at about 11 percent, and two year contracts at under 3 percent. One column, an eightfold spread. Any model that cannot beat that single column is not earning its infrastructure, and I have shipped review meetings where that comparison quietly killed a project. Always benchmark against the best single feature before you defend an ensemble.

Bayes theorem is the rule that lets you flip a conditional around, and its practical value is that it forces the base rate into view. Ask what fraction of the customers your model flags will actually churn and the answer depends far more on the 26.54 percent base rate than on how good the model feels. Run the arithmetic at 80 percent recall and 47 percent precision and you find that more than half of everyone you contact was never going to leave. Nothing is broken. That is simply what a minority class does to a screening problem, and it is the same trap that makes medical screening tests look worse than they are.

A related trap: a conditional relationship is not a causal one. Customers on month to month contracts churn more, but moving a customer onto an annual contract does not by itself make them loyal, it changes who they are in the data. Part 8 and the correlation and causation post both go after this, and the only clean answer is an experiment, which is why A/B testing belongs in a modeller’s toolkit and not just an analyst’s.

Calibration, or whether 0.7 means 0.7

Discrimination and calibration are separate properties and a model can have one without the other. Discrimination is the ability to put risky customers above safe ones, and area under the curve measures it. Calibration is whether the number attached to that ordering is accurate: among all customers scored 0.70, do about 70 percent churn. Sorting a list correctly while getting every number wrong is entirely possible, and it is the default behaviour of several popular algorithms.

Which algorithms misbehave is predictable enough to memorise. Logistic regression optimises log loss directly and comes out close to calibrated, which is one of the quiet reasons it survives in production. Boosted trees and random forests do not, and forests in particular pull probabilities toward the middle because averaging many trees rarely produces a unanimous vote. Support vector machines produce distances rather than probabilities and need conversion. Neural networks trained to convergence tend to be overconfident at both extremes.

Diagnosing it takes one chart. Bin your predictions, plot mean predicted probability on one axis and observed churn rate on the other, and compare against the diagonal. Anything below the diagonal is overconfident, anything above is underconfident. Scikit learn provides this as a calibration curve and pairs it with two summary numbers, log loss and Brier score, both of which reward being right and being honest at the same time. One caution from the scikit learn documentation that is worth repeating: a lower Brier score does not on its own prove better calibration, since it also rewards sharper discrimination. Read the curve, not just the scalar.

Fixing it is cheap. Platt scaling fits a small logistic regression on top of your scores and works well when the miscalibration is a smooth stretch. Isotonic regression fits a step function, handles odd shapes, and needs more data or it overfits: below roughly a thousand validation rows I stay with Platt. Both must be fitted on held out data, never on the rows the model trained on, which is the same rule that governed target encoding back in Part 6.

Expected value and the cost of a wrong prediction

Expected value is a probability weighted average of outcomes, and it is the bridge between a model output and a decision. Once probabilities are calibrated, you can multiply them by money, and the threshold stops being a modelling choice and becomes a finance one. This single move has done more for the credibility of my models in front of executives than any accuracy improvement.

Set up the economics first. Assume a retention offer costs 200 dollars per contacted customer, a saved customer is worth 900 dollars over the following year, and the offer succeeds on 40 percent of the customers who genuinely intended to leave. Contacting someone who was never going to churn costs the full 200 and returns nothing. Now the question is not which threshold gives the best F1, it is which threshold makes the most money.

Illustrative retention economics on 7043 customers with 1869 churners. Same model, same predictions, five different business outcomes.

Read the table twice. First, the highest recall setting loses 92,800 dollars, because contacting 3200 customers at 200 each costs more than the churners you rescue are worth. Second, the best threshold is 0.60 and not 0.50, which no default in any library would have found for you. Between the worst and best rows sits 162,000 dollars, and the model did not change at all. Only the cut point moved.

Note also how flat the curve is between 0.50 and 0.70. Anywhere in that band is fine, which is genuinely useful information: it means you do not need to defend 0.60 to three decimal places, and it means a modest drift in the score distribution will not immediately destroy the business case. Curvature around the optimum tells you how much monitoring the threshold will need once the model is live, which is a Part 25 problem.

Check calibration before you tune AUC

If you take one recommendation from this part, take this one. Before spending another week on hyperparameters, plot the reliability curve and compare your model’s mean predicted probability against the observed base rate of 0.2654. Those two numbers should match closely. When they do not, no amount of extra area under the curve will make the downstream decision correct, and you will be optimising a ranking that somebody is about to convert into money.

Practically, that means three habits. Write the base rate at the top of every model notebook so it is impossible to ignore. Check calibration before tuning, not after, because a badly calibrated model will send you chasing threshold problems that are really probability problems. And attach a cost to each error type early, even a rough one, because a threshold table like the one above changes the conversation with a stakeholder more than any metric ever will.

Two options I would avoid. Do not reach for naive Bayes when you need trustworthy probabilities on a customer table, since its independence assumption is badly violated there and it produces confident nonsense. And do not use isotonic regression on a small validation set, because it will happily memorise the noise you gave it. Platt scaling on a proper holdout is the safe default until you have several thousand validation rows.

Part 8 picks up the loose thread from Part 6 and answers whether a 0.004 gain is real, using sampling distributions, confidence intervals and a clear statement of what a p value does not tell you. That is the last conceptual stop before Part 9 puts a linear regression on the churn table and we start producing numbers to argue about. Before you move on, do one thing with your own copy: compute the base rate, compute your model’s mean predicted probability, and write both on a sticky note. If they differ by more than a couple of points, you have found your next week of work.

References

• SciPy: statistical functions and distribution families
• scikit-learn: probability calibration user guide
• scikit-learn: brier_score_loss API reference
• IBM: Telco customer churn sample dataset

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts