Hypothesis Test Visualizer

A hypothesis test answers one narrow question. If nothing but chance were at work, how unusual would a statistic like this one be? Everything else people read into a test result is added afterwards, often by someone in a hurry.

The machinery has three parts. A null hypothesis names a value, such as a process mean of 50 particle counts per wafer, or no difference between two ways of assembling a device. An alternative hypothesis says which departures matter: any difference at all, or only a difference in one direction. A test statistic condenses the data into one number whose distribution is known when the null hypothesis holds. For a mean, that number is the distance between the sample mean and the null value, measured in standard errors. The NIST/SEMATECH e-Handbook writes the one-sample version as the sample mean minus the hypothesised mean, divided by the standard deviation over the square root of the sample size.

The curve behind every test

The known distribution is the whole trick. When the standard deviation is estimated from the same data, the statistic follows Student’s t distribution with N minus 1 degrees of freedom, whose tails are heavier than the normal curve’s for small samples. When the standard deviation is known in advance, the statistic is standard normal and the test is a z test. Counts compared with expected counts lead to a chi-squared statistic instead.

The rejection region is simply the part of that distribution least compatible with the null hypothesis. Choose a significance level alpha, often 0.05, and mark off that much area in the tail or tails the alternative points to. A two-sided test puts alpha over 2 in each tail. The edge of the region is the critical value, which the e-Handbook tabulates for common levels and whole numbers of degrees of freedom.

A p-value is the area beyond the observed statistic, on the side or sides the alternative names, computed as if the null hypothesis and the test’s assumptions were true. Drawn on the curve, it is a thin shaded sliver rather than a mysterious score. A statistic inside the rejection region has a p-value below alpha; one outside has a p-value above it. The two descriptions always agree, which is why the shaded picture and the number in a report should tell the same story.

The common misreadings come from dropping the condition in that definition. A p-value is not the probability that the null hypothesis is true, and it is not the probability that the result was produced by chance alone. It does not measure the size or importance of an effect either. A trivial difference measured on an enormous sample can produce a vanishingly small p-value.

The e-Handbook makes the parallel point about intervals. A 95% confidence interval does not contain the true mean with 95% probability; the level describes how often the method succeeds over many samples.

That is why a careful report pairs the p-value with an interval and an effect size. The interval shows which values of the parameter the data leave plausible. An effect size such as Cohen’s d expresses a difference in standard deviations, so results measured on different scales can be compared.

Pooled, Welch or paired

Two-sample tests are where degrees of freedom most often go wrong. The pooled test assumes both groups share one population standard deviation. It averages the two sample variances, weighted by their degrees of freedom, and uses N1 plus N2 minus 2 degrees of freedom. Welch’s test drops that assumption. It puts each group’s own variance into the standard error and estimates the degrees of freedom with the Welch-Satterthwaite formula, which usually gives a fractional number below the pooled count. With equal sample sizes the two standard errors coincide exactly, yet the degrees of freedom still differ unless the sample variances match.

The e-Handbook’s assembly example shows the difference with real numbers. Eleven devices built by an old process took a mean of 36.0909 minutes with a standard deviation of 4.9082; nine built by a new process took 32.2222 minutes with a standard deviation of 2.5386. The question is whether the new process is faster, so the test is one-sided. Welch’s statistic is 2.2694 on 15.5 degrees of freedom. The one-sided p-value is about 0.0189, and the critical value at alpha 0.05 is about 1.749, so the null hypothesis is rejected. The handbook rounds the degrees of freedom to 16 and reads 1.746 from its table, reaching the same decision.

The pooled test on the same 20 values uses 18 degrees of freedom and a different standard error. Its statistic is 2.135, with a one-sided p-value near 0.023. Both tests reject here. With smaller or more lopsided samples, the choice can flip the decision.

Which should be the default? The Wikipedia article on Welch’s test, citing the published comparisons, says it keeps its type I error rate close to nominal when variances and sample sizes are unequal, and that its power comes close to the pooled test’s even when the variances are equal. The same article advises against running a variance test first and choosing between the two on its result. The e-Handbook does describe an F test for equal standard deviations, and it is worth reading as a description of the data, not as a gate.

Paired data need neither formula. When each value in one sample belongs with exactly one value in the other, such as the same part measured by two methods, the test runs on the differences. That removes the variation between parts and leaves a one-sample t test with one degree of freedom fewer than the number of pairs.

Counts, tables and the chi-squared curve

When the data are counts in categories, the statistic is Pearson’s chi-squared. For each cell, take the squared difference between the observed and the expected count, divide by the expected count, and add the cells up. In a goodness-of-fit test the degrees of freedom are the number of categories minus 1, minus any parameters estimated from the data. In a contingency table the expected count of a cell is its row total times its column total divided by the grand total, and the degrees of freedom are the number of rows minus 1 times the number of columns minus 1.

The e-Handbook’s wafer example classifies 309 defects by type and by production shift. Its statistic is 19.18 on 6 degrees of freedom, above the critical value of 12.5916 at alpha 0.05, so type and shift are not independent in that data. The exact p-value is about 0.0039, and Cramer’s V, an effect size for tables, is about 0.18: a real but modest association.

The chi-squared curve is only an approximation for counts, and it degrades when expected counts are small. The e-Handbook asks for an expected frequency of at least 5 in each bin and suggests combining sparse bins in the tails. A looser rule quoted on Wikipedia allows up to a fifth of the cells of a larger table below 5, with none at zero.

Power, sample size and what arithmetic cannot do

Power is the probability that a test rejects the null hypothesis when a stated effect really exists. For a z test it follows in closed form from the normal curve shifted by the effect. For a t test the statistic under the alternative follows a noncentral t distribution, whose noncentrality grows with the effect size and the square root of the sample size, so exact power needs that distribution rather than a normal shortcut. The curves climb more slowly than intuition suggests: at alpha 0.05, two-sided, a two-sample t test needs about 64 per group to reach 80% power for an effect of half a standard deviation.

Plan the sample size before collecting data. Power computed afterwards from the observed effect is controversial, and many statisticians regard it as misleading.

The exact numbers behind all of this come from two special functions defined in the NIST Digital Library of Mathematical Functions. The t and F distributions are regularised incomplete beta functions in disguise; the chi-squared distribution and the normal tail are regularised incomplete gamma functions. Computing them directly means a critical value for 15.53 degrees of freedom needs no rounding to the nearest table row.

None of this arithmetic can rescue a poor design. A test cannot tell whether the samples were random, whether the measurements were independent, whether the right question was asked, or whether twenty tests were run until one crossed the line. It cannot say that a result is true, or that it matters. It can only report, exactly, how surprising the data would be if the null hypothesis held.