A product team changed the checkout button from grey to red on a Tuesday morning. By Wednesday lunchtime the dashboard showed a 9 percent lift in completed orders, the change was declared a win, and it shipped to everyone by Friday. Six weeks later, checkout completion sat slightly below where it had started. Nothing was faked and nobody lied. The team simply stopped counting at the moment the number looked best, which is the most common way a reasonable idea turns into a bad decision.

In Part 17 you learned to define a number so it means one thing. This part asks the harder question that follows: when that number moves, how do you know your change caused it? That is what an experiment is for, and the mechanics are far less intimidating than the vocabulary suggests. There is no code in this part, because the failures I see in real teams are never coding failures. They are planning failures, and they happen before anyone writes a line of anything.

What an experiment proves that a dashboard cannot

A dashboard tells you what happened. It cannot tell you what would have happened otherwise, and that missing half is where almost every wrong conclusion lives. If sales rose 4 percent in the week you launched a new homepage, the honest question is not whether sales rose, it is whether they would have risen anyway. Payday fell in that week. A competitor had an outage. The weather turned. A dashboard has no way to separate your change from any of that.

An A/B test solves the problem with one trick: it splits your users at random into two groups at the same moment in time. One group, the control, sees the current experience. The other, the treatment or variant, sees your change. Because the split is random and the two groups live through the same week, everything else that could affect the outcome, payday and weather and competitor outages included, lands on both groups roughly equally. Whatever difference remains between them is attributable to the one thing you deliberately changed.

That is the whole idea, and it is worth pausing on how strong it is. Randomisation is the only technique in the analyst toolkit that turns a correlation into a causal claim, and it does so without you needing to know or measure the confounding factors at all. It handles the ones you never thought of. Everything else in this part is bookkeeping around that central trick, making sure the split really was random and the difference you measured is bigger than the noise you would expect from splitting a crowd in two.

flowchart TD
  H[Write the hypothesis] --> M[Pick one primary metric]
  M --> S[Calculate sample size]
  S --> D[Set the stop date]
  D --> R[Randomise users into control and variant]
  R --> W[Wait, check health only]
  W --> Q{Sample size reached?}
  Q -->|No| W
  Q -->|Yes| A[Analyse once]
  A --> V[Ship, drop or iterate]

The words you need before the maths

Experiment vocabulary sounds harder than it is, and most of it is one plain idea wearing a formal coat. A hypothesis is a sentence stating what you will change, what you expect to happen to which number, and roughly why. Good ones are specific and falsifiable: moving the delivery cost above the fold will raise checkout completion, because cost surprises at the final step are the most common reason people abandon. Bad ones are vague enough to survive any result, which makes them useless.

The primary metric is the single number that decides the test. One number, chosen in advance, no exceptions. If you nominate five and then declare victory on whichever one moved, you have quietly made it very likely that something moved by chance alone, which is a problem statisticians call multiple comparisons and everyone else calls cheating. Everything else you track is a secondary metric for context, or a guardrail metric, which is a number you would not accept damaging even in exchange for a win on the primary. Page load time and refund rate are typical guardrails.

Three more terms carry the arithmetic. The minimum detectable effect, usually written MDE, is the smallest change you care enough about to act on. Statistical significance, conventionally set so there is a 5 percent chance of calling a difference real when it is not, controls how often you get a false alarm. Statistical power, conventionally 80 percent, is the chance your test actually spots an effect of your chosen MDE when that effect genuinely exists. Those three plus your current conversion rate are the only inputs a sample size calculator needs.

The one worth dwelling on is power, because it is the term beginners skip and the term that quietly wrecks the most experiments. A test with 30 percent power is not a cautious test, it is a test that will miss a real improvement roughly seven times out of ten and hand you a flat result you will wrongly read as proof the idea failed. Underpowered tests do not produce no answer, they produce confidently wrong answers, and they are far more expensive than the traffic they were meant to save.

How many people does the test need?

This is the question that kills more experiments than any other, and the answer surprises people every time. The required sample grows roughly with the square of how small an effect you want to catch, so halving your MDE quadruples the traffic you need. That relationship is unforgiving, and it is the reason large companies can test button colours while a startup with modest traffic cannot.

The table below uses a realistic starting point: a 5 percent conversion rate, the conventional 5 percent two sided significance level, and 80 percent power. The last column assumes 4,000 eligible visitors a day split evenly across the two groups. I calculated these with the standard two proportion formula rather than quoting a rule of thumb, because rules of thumb in this area are wrong by factors of two or three depending on which assumptions the author silently made.

Baseline 5 percent conversion, 5 percent two sided significance, 80 percent power. The day counts are arithmetic only; see the note below on why you still run at least two full weeks.

One correction to the day counts above. Even when the arithmetic says four days, run for at least two full weeks. User behaviour has a weekly rhythm, and a test that covers only Tuesday to Friday measures a population of weekday users rather than your actual customers. Two weeks gives you two complete cycles, which also lets you check whether the effect is stable rather than a burst on day one. If your calculated duration exceeds about six weeks, treat that as a signal to test a bolder change instead, because seasonality and product releases will contaminate anything running longer.

Read the result without fooling yourself

When the test ends you get a p value and, if your tool is any good, a confidence interval. The p value answers one narrow question: if my change actually did nothing at all, how likely is a difference at least this large purely from the luck of the split? A p value of 0.03 means such a difference would turn up 3 percent of the time by chance alone. It does not mean there is a 97 percent chance your change works, and it says nothing whatsoever about whether the effect is large enough to matter commercially.

The confidence interval is the more useful of the two and gets a fraction of the attention. It gives you a range of lifts consistent with what you observed. A result of plus 11.6 percent with an interval from plus 5.2 to plus 18.3 is a real win whose size you can plan around. A result of plus 2.1 percent with an interval from minus 3.4 to plus 7.8 is a shrug: the data is equally comfortable with a small gain and a small loss. Both of those may be reported as a lift by an over eager tool, and only one of them should change anything.

Now the failure from the opening story. Watching the p value daily and stopping the moment it dips below 0.05 is called peeking, and it is not a minor sin. Kohavi, Tang and Xu, who ran experimentation at Microsoft, Google and LinkedIn, document that continuous monitoring with early stopping inflates the false positive rate several times over. The mechanism is easy to picture: a p value wanders as data accumulates, so if you check it twenty times and stop at the first dip, you are not testing your hypothesis, you are fishing for the one moment noise flattered you. Look at the result once, when the planned sample lands.

Row three is the opening story. A 9 percent lift read at 18 percent of the planned sample is not a result, it is a weather report.

Checks that catch a broken test

A statistically significant result from a broken experiment is still garbage, and the significance makes it more dangerous rather than less, because it arrives wearing credentials. Three checks catch most of the damage, and none of them require advanced statistics.

The first is sample ratio mismatch, usually shortened to SRM. You asked for a 50 to 50 split; check whether you actually got one. At large volumes a split of 52 to 48 is not bad luck, it is evidence that randomisation, tracking or delivery is broken, and Microsoft Research treats a failed SRM check as grounds to discard the result entirely. The standard test is a chi squared goodness of fit against your intended ratio. This is the one number you are allowed to watch daily, because it is a data quality diagnostic rather than a hypothesis test, so checking it repeatedly costs you nothing.

The second is the novelty effect. Anything visually new attracts clicks simply for being new, and existing users may click it once out of curiosity and never again. The symptom is a strong effect on days one and two that decays steadily through the run. Plot your metric by day rather than looking only at the pooled total, and if the line trends downward across the test, either run longer or split the analysis between new and returning users. Its mirror image, the primacy effect, is a temporary dip while habitual users relearn an interface they had memorised.

The third is simply reading your guardrails before your primary. If checkout completion rose 12 percent while refund rate rose 30 percent, you have not found a win, you have found a way to push people into purchases they regret. I read guardrails first as a discipline, because once you have seen a good headline number your judgement about everything else quietly softens.

When you cannot run an experiment

Plenty of real questions cannot be answered by an A/B test, and pretending otherwise is its own kind of error. You cannot randomise a price increase across customers who talk to each other. You cannot split a brand campaign down the middle. You cannot experiment on a change that must launch on a fixed legal date. Traffic is often simply too thin: at 200 visitors a day, the 20 percent lift test in the table above would take forty days, and anything subtler is out of reach for good.

The usual fallbacks are weaker but honest if you label them. A before and after comparison is the weakest, because it inherits every confounder an experiment was designed to remove; use it only with a long stable history and no other changes in flight, and say out loud that it is suggestive rather than conclusive. Splitting by geography, where one region gets the change and a comparable region does not, is a real improvement, since the two regions absorb the same seasonality. A holdout group, where a small slice of users is deliberately kept on the old experience for months, is the best option for slow moving effects such as retention.

There is also a smaller decision hiding here: what to do with a flat result. A test that ends with an interval spanning zero has not proven the change is worthless, it has proven the change is not as large as your MDE. Those are different findings, and confusing them is how teams abandon good ideas. If the change is cheap to keep, costs nothing to maintain and did no harm, keeping it is defensible. What is not defensible is reporting it upward as a win.

Fix the sample size and the stop date before you launch

If you adopt one habit from this part, adopt this one: before a single user sees the variant, write down the primary metric, the MDE, the required sample per group and the date you will stop. Put it in the ticket where the engineers and the product manager can see it. Everything that goes wrong in A/B testing goes wrong because one of those four was decided after the data started arriving, and a decision made in sight of the data is no longer a decision, it is a rationalisation.

The second habit is to be honest about the traffic you have. If the arithmetic says you need sixty days to detect the effect you care about, you do not have an experiment, you have a wish. Test something bolder, pick a metric closer to the change so it moves further, or accept a regional split and label the weaker evidence honestly. Running an underpowered test and reporting the flat result as proof of failure wastes both the traffic and a perfectly good idea.

You can now design an experiment, size it, read the result properly and spot the three ways it breaks. Part 19 turns to the part nobody warns you about: standing in front of stakeholders and explaining what a wide confidence interval means without losing the room. Keep the Data Analyst guide as your map, and revisit Part 12 if the causal claims in this part felt like a leap.

This week, find one experiment your company has run and look for the planned sample size. If it was never written down, write the plan for the next one yourself. That single page is the most useful thing an analyst can hand a product team, and almost nobody offers it.

References

• Diagnosing Sample Ratio Mismatch in A/B Testing, Microsoft Research
• Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments, Cambridge University Press
• Statistical Challenges in Online Controlled Experiments, The American Statistician

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts