A service logged these 11 response times, in milliseconds:

Someone asks "how fast is the service?". You need one number, or a few, that summarise the list honestly. Which?

Sorted, the data are 95, 98, 102, 105, 108, 110, 115, 120, 125, 130, 940. The median is the 6th of 11 values: 110 ms. The mean is

Ten of the eleven requests took 95–130 ms, yet the mean is 186 ms, larger than every one of them. A single slow request (perhaps a cache miss or a garbage-collection pause) dragged it up. Remove the 940 and the mean drops to 110.8 while the median barely moves (110 to 109).

Two services both have a median latency of 110 ms. One is between 100 and 120 ms almost always; the other ranges from 20 ms to 3 s. You need measures of spread, and pictures that show shape.

From Chapter 30, the variance of a random variable is the expected squared deviation from its mean. For data, average the squared deviations from \(\bar x\), with one twist:

For the 11 response times, \(s \approx 250\) ms; without the 940, \(s \approx 11.6\) ms. Like the mean, the standard deviation is wrecked by one outlier, since it squares the deviations.

Robust measures of spread use positions in the sorted data. The quartiles split it into four equal parts.

Worked example. Sorted: 95, 98, 102, 105, 108, 110, 115, 120, 125, 130, 940. Try it: Quartiles and box plot.

The 940 is far beyond 159.5: flagged. Nothing else is.

Notice the last two lines: there are several conventions for quartiles, and software disagrees. For large data sets the differences vanish; for small ones, say which convention you used.

A box plot draws the five-number summary: a box from \(Q_1\) to \(Q_3\) with a line at the median, "whiskers" out to the most extreme non-outlier values, and outliers as individual points.

A histogram divides the range into bins and draws a bar for the count in each bin. It shows shape: one peak or two, symmetric or skewed, gaps, clusters. Box plots are compact and good for comparing many groups side by side; histograms show more detail about one group. Use both.

A distribution is right-skewed (positively skewed) if it has a long tail to the right: most values are moderate and a few are very large. Latencies, incomes, file sizes and city populations are right-skewed. Left-skewed is the mirror image (exam scores on an easy test). In a right-skewed distribution the mean is pulled into the tail, above the median; in a left-skewed one it’s below.

You want the average session length of all 40 million users, or the fraction of voters supporting a candidate, or the true conversion rate of a checkout page. You can’t measure everyone (or, for a conversion rate, "everyone" includes future users who don’t exist yet). You measure a sample and generalize. When is that legitimate, and how wrong might you be?

Random variation (noise) shrinks as samples grow. Bias doesn’t: it’s a systematic error that makes the sample unrepresentative no matter how large.

You sample 100 page loads and find a mean of 2.30 seconds. The true mean over all page loads is not exactly 2.30; a different sample of 100 would give a slightly different answer. How much would estimates bounce around from sample to sample?

Imagine repeating the whole study many times, each time drawing a fresh sample of 100 and computing \(\bar x\). The collection of all those \(\bar x\) values has a distribution, the sampling distribution of the mean. Chapter 30 already told you its shape and size: by the central limit theorem it’s approximately normal, centred on the true \(\mu\), with standard deviation \(\sigma/\sqrt n\).

If the 100 page loads have sample standard deviation \(s = 0.80\) s, then \(\text{SE} = 0.80/\sqrt{100} = 0.08\) s. Individual loads vary by about 0.8 s, but the average of 100 is uncertain by only about 0.08 s.

An estimator is unbiased if its sampling distribution is centred on the true parameter: \(E[\bar x] = \mu\), for instance. Good estimators are unbiased (or nearly) and have small standard errors.

"Average load time is 2.30 s" sounds more certain than it is. You want to report a range that honestly conveys the uncertainty, with a stated reliability.

By the CLT, \(\bar x\) is approximately normal around \(\mu\) with standard deviation \(\text{SE}\). The 68–95–99.7 rule (more precisely, 95% lies within 1.96 standard deviations) says:

"\(\bar x\) is within 1.96 SE of \(\mu\)" is the same statement as "\(\mu\) is within 1.96 SE of \(\bar x\)". So

Read this carefully. \(\mu\) is a fixed (unknown) number; the randomness is in \(\bar x\), so in the interval. Before you collect data, the procedure "take \(\bar x \pm 1.96\,\text{SE}\)" has a 95% chance of producing an interval that captures \(\mu\).

Try it: When z* is too narrow.

Worked example. 100 page loads: \(\bar x = 2.30\) s, \(s = 0.80\) s.

Worked example (proportion). A poll of 1,000 randomly chosen users finds 540 prefer the new design.

That "±3 percentage points" is the margin of error quoted under news polls of about 1,000 people. To halve it, poll 4,000.

Try it: Fifty intervals, live.

A coin is tossed 100 times and lands heads 61 times. Is it biased? A fair coin could produce 61 heads; it would just be unusual. How unusual is unusual enough? The same question, with higher stakes: the new checkout page converted 11.0% of users versus 10.0% for the old one. Is the new page better, or did it get lucky?

Hypothesis testing borrows the shape of proof by contradiction (Chapter 22). Assume the boring explanation, nothing is going on, and work out what data you’d expect. If the data you actually got would be very surprising under that assumption, you have evidence against it.

\(H_0\): \(p = 0.5\). \(H_1\): \(p \ne 0.5\). Significance level \(\alpha = 0.05\). Try it: Run the z-test.

Under \(H_0\), the number of heads \(X \sim \text{Bin}(100, 0.5)\), with mean 50 and standard deviation \(\sqrt{100 \times 0.5 \times 0.5} = 5\).

Normal approximation. The test statistic is \(z = \frac{61 - 50}{5} = 2.2\). "At least as extreme, in either direction" means \(|Z| \ge 2.2\):

Exact binomial. Add the binomial probabilities of 61 or more heads and 39 or fewer: \(p \approx 0.035\). (The normal approximation is a little optimistic here; a "continuity correction" using 60.5 instead of 61 would bring it closer.)

Either way \(p < 0.05\), so we reject \(H_0\) at the 5% level: 61 heads is evidence (moderate, not overwhelming) that the coin isn’t fair.

The p-value is \(P(\text{data this extreme} \mid H_0)\). Recall the warning from Chapter 29 about confusing \(P(A \mid B)\) with \(P(B \mid A)\).

A test makes a binary decision about an unknown truth, so it can be wrong in two ways:

The analogy is a smoke alarm. Make it more sensitive and it catches more fires (more power) but also goes off for toast (more type I errors). The only way to improve both is a better detector, which in statistics means more data or less noise.

The problem. A new build’s latencies (8 runs, ms): 190, 204, 185, 199, 210, 188, 195, 202. The old build: 212, 198, 225, 240, 205, 219, 230, 201. Means 196.6 and 216.3, a difference of 19.6 ms. Real, or noise? Try it: Run Welch’s t-test.

The idea. Every two-sample test asks the same thing: is the difference between groups large compared with how much it would vary by chance? The t statistic is

The denominator combines the standard errors of the two means (variances of independent quantities add, Chapter 30). Here \(t \approx 3.23\): the observed difference is more than three standard errors from zero.

Why "t" and not "z"? With small samples, \(s\) is itself a noisy estimate of \(\sigma\), which adds uncertainty. William Gosset worked out the exact null distribution of \(t\) for normal data in 1908: the t-distribution, a bell curve with heavier tails than the normal, controlled by the degrees of freedom (roughly \(n - 1\) per group). Some 95% two-sided critical values:

Welch’s formula for unequal variances gives about 11 degrees of freedom here, and \(|t| = 3.23 > 2.201\), so the difference is significant at the 5% level. As \(n\) grows, \(t^*\) approaches the normal’s 1.96.

An alternative that needs no tables: the permutation test. If the build made no difference (\(H_0\)), the labels "old" and "new" are arbitrary: any 8 of the 16 numbers could equally have been the "old" ones. So shuffle the labels thousands of times, recompute \(t\) each time, and see how often a shuffle gives a result as extreme as the real one. That fraction is the p-value, computed directly from its definition. Try it: Permutation test vs t-test.

Try it: Run this in your browser.

Choosing a method. Use a t-test for comparing means when data are roughly normal or samples are moderately large; it’s standard and fast. Use a permutation test when samples are small and skewed, or for any statistic (medians, ratios) without a textbook formula; it costs only computer time. For paired data (the same 8 benchmarks run on both builds) use a paired test on the differences, which removes benchmark-to-benchmark variation and is far more powerful.

For categorical data, compare observed counts with the counts expected under \(H_0\). A die is rolled 120 times; if fair, each face is expected 20 times. Observed: 15, 22, 18, 25, 17, 23. The chi-squared statistic sums the squared discrepancies, each scaled by its expected count: Try it: Run the χ² test.

(\(\chi\) is the Greek letter chi, pronounced "kai".) Under \(H_0\) this statistic follows a chi-squared distribution with \(6 - 1 = 5\) degrees of freedom, whose 95% critical value is 11.07. Since \(3.8 < 11.07\), there’s no evidence the die is unfair; deviations this size are routine. The same test checks whether two categorical variables are independent in a two-way table (does the browser affect whether checkout succeeds?), and whether a random number generator or hash function spreads values evenly across buckets.

Chapter 30 defined the correlation coefficient. For data pairs \((x_i, y_i)\), the sample correlation is

a number from \(-1\) to 1 measuring the strength of a linear association. Ice-cream sales correlate with drownings. Countries' chocolate consumption correlates with Nobel prizes per capita. Users who enable dark mode have higher retention. None of these shows that one thing causes the other. The usual explanations are:

The gold standard for causation is a randomized controlled experiment: assign the treatment at random, so that confounders are balanced between groups on average. That’s exactly what an A/B test does, and why it can answer causal questions that no amount of observational log analysis can.

A 1986 study compared two treatments for kidney stones: Try it: Slide through Simpson’s paradox.

Treatment A is better for small stones and better for large stones, yet B looks better overall. How?

Look at who got which treatment. Treatment A was given mostly to large stones (263 of 350), the hard cases; B mostly to small stones (270 of 350), the easy ones. The overall rate is a weighted average of the group rates (the law of total probability from Chapter 29), and the weights differ: A’s average is dominated by its 73% group, B’s by its 87% group. Stone size is a confounder: it affects both which treatment doctors chose and the chance of success. Try it: Same case mix, no paradox.

The right comparison here is within groups: A is better. But Simpson’s paradox doesn’t come with a rule that "always disaggregate"; which comparison is right depends on the causal story. If the grouping variable were itself caused by the treatment, aggregating would be correct.

A service’s median response time at six load levels (load in hundreds of concurrent users):

Response time clearly grows with load. You want a line through the data to summarize the trend and predict, say, the response at load 8. The points don’t lie exactly on any line, so which line is "best"?

For any candidate line \(\hat y = a + bx\), each point has a residual: the vertical miss \(e_i = y_i - \hat y_i\) (observed minus predicted). A good line makes residuals small. The least-squares line makes the sum of squared residuals as small as possible. Squaring treats overshoots and undershoots alike, punishes big misses heavily, and (the decisive practical reason) gives a clean formula.

Write the total squared error as \(\text{SSE}(a, b) = \sum (y_i - a - bx_i)^2\). Two observations give the formulas.

The intercept. For a fixed slope \(b\), SSE is the sum of squared distances from the numbers \(y_i - bx_i\) to a single number \(a\). The number that minimizes a sum of squared distances is the mean (you can verify this by completing the square, Chapter 8): \(a = \overline{y - bx} = \bar y - b\bar x\). So the best line passes through \((\bar x, \bar y)\).

The slope. Substitute \(a = \bar y - b\bar x\) and write \(X_i = x_i - \bar x\), \(Y_i = y_i - \bar y\) (centred data). Then \(\text{SSE}(b) = \sum (Y_i - bX_i)^2 = S_{yy} - 2bS_{xy} + b^2 S_{xx}\), a quadratic in \(b\) that opens upward. Its vertex (Chapter 8) is at \(b = S_{xy}/S_{xx}\).

(Optional, with calculus: setting \(\partial\,\text{SSE}/\partial a = 0\) and \(\partial\,\text{SSE}/\partial b = 0\) gives the same two equations, the normal equations; see Chapter 20. With many input variables they become the matrix equation of Section 33.2.)

\(\bar x = 21/6 = 3.5\) and \(\bar y = 360/6 = 60\). Try it: Fit the load data; Fit the regression line.

Each extra 100 concurrent users adds about 4 ms. The prediction at load 8 is \(46 + 32 = 78\) ms, an extrapolation beyond the data, to be trusted only if the trend stays linear (queueing theory says it won’t near saturation; see the Watch out below).

Compare the line’s errors with the errors of the dumbest prediction, "always predict \(\bar y\)":

Here the residuals are \(51 - 50 = 1\), \(52 - 54 = -2\), \(59 - 58 = 1\), \(63 - 62 = 1\), \(64 - 66 = -2\), \(71 - 70 = 1\), so \(\text{SS}_{\text{res}} = 1 + 4 + 1 + 1 + 4 + 1 = 12\). And \(\text{SS}_{\text{tot}} = 81 + 64 + 1 + 9 + 16 + 121 = 292\). So

The line explains about 96% of the variation in response time. For simple linear regression, \(R^2 = r^2\), so \(r = \sqrt{0.959} \approx 0.979\).

You run an A/B test and check 20 metrics: conversion, revenue, clicks, time on page, each on mobile and desktop, in five countries. One metric shows \(p = 0.03\). Discovery?

At \(\alpha = 0.05\), each test of a true null has a 5% chance of a false positive. With 20 independent tests of true nulls, Try it: Simulate twenty tests.

You’re more likely than not to find "something", even if the change did nothing. Testing many hypotheses and reporting the significant ones is called p-hacking or data dredging, and is a central cause of the replication crisis in psychology, medicine and other fields: many published "findings" were simply the lucky tail of many tries. The same thing happens if you test one hypothesis but try many analyses (different outlier rules, different subgroups, different stopping times) and report the one that worked.

Your checkout page converts 10% of visitors. A redesign might help. You’ll randomly show half of new visitors the old page (A, control) and half the new one (B, treatment). How many visitors do you need, how do you analyse the result, and what mistakes must you avoid?

The test statistic is the difference in sample proportions divided by its standard error. For the test to have power 80%, the true difference must be large enough that the test statistic’s distribution under \(H_1\) sits mostly beyond the cut-off 1.96 (the figure in the type I/II section). The standard approximation is Try it: Plan the sample size.

where \(z_{\alpha/2} = 1.96\) (two-sided 5%) and \(z_\beta = 0.84\) (\(\Phi(0.84) \approx 0.80\), for 80% power). With \(p_A = 0.10\), \(p_B = 0.11\):

About 30,000 visitors in total. Note the denominator: the required sample grows with the inverse square of the effect. Detecting a 0.5-point lift needs about four times as many users (roughly 58,000 per group); a 0.1-point lift, a hundred times as many. Try it: Half-point lift sample size.

Results: A converted 1,500 of 15,000 (10.0%); B converted 1,650 of 15,000 (11.0%). Try it: Run the two-proportion test.

Significant at the 5% level. For the confidence interval, use the unpooled standard error (we no longer assume the rates are equal):

Report: "B increased conversion by 1.0 percentage point (95% CI 0.3 to 1.7 points; \(p = 0.005\))." The interval matters as much as the p-value: the true lift could plausibly be as small as 0.3 points. Is that still worth shipping? That’s a business decision the statistics informs but doesn’t make.

Does the procedure really have a 5% false-positive rate and 80% power? And what does peeking do? Simulate thousands of experiments where you know the truth.

Try it: Watch peeking inflate false positives.

The planned test behaves as designed: about 5% false positives when nothing changed, about 80% detection of a real 1-point lift. But checking 15 times and stopping at the first significant result raises the false-positive rate from 5% to about 23%. Each look is another chance for noise to cross the line: a multiple-comparisons problem in time. If you need to monitor continuously, use methods designed for it (sequential tests, or "always-valid" p-values), which set stricter thresholds for early looks.

With a uniform prior on each conversion rate, the posterior for a rate after \(s\) successes and \(f\) failures is a Beta distribution, Beta(\(s + 1, f + 1\)). Sampling from both posteriors answers the question stakeholders actually ask:

"There’s a 99.8% probability B is better, and a 92% probability the lift exceeds half a point", given the model and the (flat) prior. With this much data the prior barely matters, and the conclusions agree with the frequentist analysis.

Try it: Watch the bootstrap resample. === The bootstrap

You have 200 response times and want a confidence interval for the median, or the p99, or the ratio of two medians. There’s no simple standard-error formula like \(s/\sqrt n\) for these. The sampling distribution you need comes from imagining many repeated samples from the population, but you only have one sample.

Your sample is your best picture of the population. So treat it as the population: draw a new sample of the same size from it, with replacement (some values appear twice, some not at all), and compute the statistic. Repeat thousands of times. The spread of these bootstrap statistics approximates the spread of the statistic across real repeated samples.

The name comes from the impossible feat of "pulling yourself up by your bootstraps". It does feel like getting something for nothing, but it works remarkably well when the sample is reasonably large and representative.

The p99 interval is wide and lopsided: with 200 data points, only the top two values inform the 99th percentile, so it is very uncertain. That’s useful knowledge before anyone alerts on a p99 from a small window of requests. Try it: Bootstrap the p99.

Choosing a method. Use formula-based intervals for means and proportions with decent sample sizes: they’re exact enough and instant. Use the bootstrap for medians, percentiles, ratios, correlations, model-evaluation metrics, or anything complicated. The bootstrap struggles for extreme statistics (maximum, very high percentiles with small samples), for very small samples, and for dependent data (time series need block resampling).

A fraud model flags transactions; a search engine returns documents; a spam filter sends email to junk. How good are they? "Accuracy" (fraction of correct decisions) is often useless: if 1% of transactions are fraudulent, a model that never flags anything is 99% accurate. You need measures that focus on the class you care about.

Every decision lands in one of four cells:

Worked example: search. A query has 30 relevant documents in the collection. The engine returns 20 results, of which 12 are relevant.

Worked example: a rare-class classifier. A fraud model on 10,000 transactions, 100 of them fraudulent, catches 90 frauds and wrongly flags 150 legitimate transactions. Recall \(= 90/100 = 0.90\); precision \(= 90/240 = 0.375\); F1 \(\approx 0.53\). Accuracy \(= (90 + 9\,750)/10\,000 = 98.4\%\), which sounds great and tells you almost nothing; it’s lower than the 99% of the do-nothing model. Try it: Precision, recall and F1.

Try it: Run this in your browser.

Most classifiers output a score, and you choose a threshold. Lower the threshold and you flag more: recall rises, precision usually falls. Raise it and the reverse happens. Which balance is right depends on the costs. Missing fraud is expensive, so favour recall; showing users junk results in the top 10 is expensive, so favour precision. Try it: Move the threshold.