The logic of a hypothesis test

A hypothesis test uses evidence from a sample to assess a claim about a population. One claim acts as the working position. We then ask how surprising the observed result would be if that claim were correct.

The null hypothesis, written $H_0$, is the population claim treated as true while the test probability is calculated. Here, $H_0$ identifies the assumed claim and has no unit. The alternative hypothesis, written $H_1$, is the competing population claim supported when the evidence against $H_0$ is sufficiently strong. Again, $H_1$ identifies a claim and has no unit.

Hypotheses refer to population parameters, not sample statistics. For instance, a claim about a population mean might be written

Usually, the equality belongs in $H_0$. When the parameter isn’t represented algebraically, hypotheses can be stated clearly in words, as in a chi-squared test of independence.

The wording of $H_1$ sets the tail:

• $H_1:\mu>\mu_0$ gives an upper-tailed test.
• $H_1:\mu<\mu_0$ gives a lower-tailed test.
• $H_1:\mu\ne\mu_0$ gives a two-tailed test.

A one-tailed test is a hypothesis test whose alternative specifies one direction of departure. A two-tailed test allows departures in either direction. Choose the direction from the research claim before looking at the data; choosing it afterwards distorts the test.

Significance levels and p-values

The significance level, written $\alpha$, is a probability threshold fixed before the test. It limits the tolerated chance of rejecting a true null hypothesis, and $\alpha$ is dimensionless. Common values are $0.01$, $0.05$ and $0.10$.

The p-value is the probability, assuming $H_0$ is true, of obtaining a test statistic at least as extreme as the observed one in the direction or directions specified by $H_1$. It is written $p$, where $p$ is a dimensionless probability. It is not the probability that $H_0$ is true.

Use the decision rule

while $p>\alpha$ means there is insufficient evidence to reject $H_0$. Don’t write “accept $H_0$” for this second conclusion. An inconclusive sample does not prove the null claim.

A result is statistically significant when its p-value does not exceed the chosen significance level. Finish in context with a sentence about the population and the original claim: “There is sufficient evidence at the $5\%$ significance level to suggest that …” or “There is insufficient evidence …”. Statistical significance does not guarantee truth, practical importance or a large effect. Some researchers therefore object to treating a p-value as a yes-or-no certificate. Its value depends on the model, sample size, assumptions and selected threshold.

Simulation helps show the logic. Generate many samples under $H_0$ and calculate the statistic for each one. Then estimate the p-value using the proportion at least as extreme as the observed statistic. This simulation represents the repeated-sampling model; it does not simulate the probability that a hypothesis itself is true.

Observed and expected frequencies

An observed frequency is the count actually recorded in a category. Write it as $O$, where $O$ is a dimensionless count. An expected frequency is the count predicted for that category when $H_0$ is assumed true. Write it as $E$, where $E$ is also a dimensionless count.

The chi-squared statistic is

Every cell makes a non-negative contribution. Larger differences between $O$ and $E$ give larger values of $\chi^2$, so chi-squared tests use the upper tail.

Expected frequencies should generally exceed $5$ because smaller values make the chi-squared approximation unreliable. The examination setting uses expected frequencies greater than $5$, though this limitation still matters in an exploration or field study.

Chi-squared test for independence

A contingency table is a frequency table that classifies the same observations using two categorical variables. A chi-squared test for independence is an upper-tailed test used to assess whether those variables are associated in the population.

The hypotheses are typically

• $H_0$: the two categorical variables are independent;
• $H_1$: the two categorical variables are associated.

For one cell in a contingency table,

Technology can find the full set of expected frequencies and the test statistic, but the calculation is still worth understanding.

The degrees of freedom are

Degrees of freedom count how many cell values remain free to vary after the totals impose their restrictions.

The following layout connects the observed table, expected table and contributions to $\chi^2$.

Chi-squared table of observed, expected and contributions

In examinations, a contingency table has at most four rows or four columns, and $\nu>1$. Use technology to obtain $\chi^2$ and the p-value. When a critical value is supplied, either comparison leads to the same decision:

• reject $H_0$ when $p\leq\alpha$;
• reject $H_0$ when the calculated $\chi^2$ exceeds the supplied critical value.

Chi-squared goodness-of-fit test

A chi-squared goodness-of-fit test is an upper-tailed test comparing the observed frequencies of one categorical variable with frequencies predicted by a stated distribution or set of proportions.

The hypotheses are

• $H_0$: the population follows the stated distribution;
• $H_1$: the population does not follow the stated distribution.

If a category has hypothesized probability $q$, then

At this level, if there are $n_c$ categories, then

This notation separates the number of categories from the sample size $n$.

Use the same $\chi^2$ formula and the same p-value or critical-value decision rule. Only upper-tailed tests at the usual $1\%$, $5\%$ and $10\%$ levels are required. Keep the context clear: rejecting $H_0$ shows that the stated model is not a satisfactory fit. It does not identify the correct alternative model.

When expected frequencies are small, it may be better to combine defensibly related categories or use an exact or simulation-based method. Yates's continuity correction is sometimes used with small two-by-two tables because a continuous chi-squared distribution is approximating discrete counts. However, it can be conservative and is not universally preferred. The real issue is whether the approximation is adequate, not simply whether a correction button exists.

Comparing two population means with a t-test

A two-sample t-test compares two population means when the population variances are unknown. At this level, the two samples are unpaired, and the measured variable in each population is assumed to have a normal distribution.

Write $\mu_1$ and $\mu_2$ for the two population means, each with the units of the measured variable. Typical hypotheses are

and one of

Technology carries out the test. In the examination setting, assume that the two population variances are equal and use the pooled two-sample t-test. The samples must remain independent. A before-and-after study involving the same individuals would be paired, so it is not this SL procedure.

Calculator output includes a t-statistic and a p-value. Write $t$ for the dimensionless t-test statistic. Match the p-value to the correct one-tailed or two-tailed alternative, compare it with $\alpha$, then interpret the result using the two population means. Don’t replace “population means” with “sample means” in the conclusion. The sample means provide the evidence; the population means are the subject of the claim.

The normality condition matters because the reference t-distribution uses a model for normally distributed underlying variables. In fields such as psychology, strongly non-normal measurements or ordinal scores may lead to a rank-based procedure such as the Mann-Whitney U test. This test relies less directly on a normal population model, but choosing it changes what is being compared and does not automatically make a study reliable.

Hypothesis tests are widely used with field data in biology, environmental studies, psychology, health science and geography. The calculation may be routine, but the quality of the conclusion still depends on sampling, measurement, independence and whether the selected model suits the data.

Critical values and critical regions

A test statistic gives a numerical summary of the sample evidence. Its distribution is known or modelled under $H_0$. The critical region contains the test-statistic values that lead to rejection of $H_0$, while a critical value forms the boundary between this rejection region and the values that do not lead to rejection.

For a continuous distribution, the critical region has probability $\alpha$ under $H_0$. In a one-tailed test, all of this area lies in one tail. A two-tailed test normally splits it between the two tails. Use the alternative hypothesis to decide where the region should go.

A result is in the critical region exactly when its p-value satisfies $p\leq\alpha$. There is no need to calculate critical regions for t-tests here; use the technological output and interpret it.

Testing a population mean

When the population is normally distributed and its standard deviation is known, use the normal distribution to test the mean. The standardized statistic is

The denominator gives the standard deviation of the sampling distribution of $\bar{x}$.

Use $H_1$ to choose the correct lower-tail, upper-tail or two-tail probability. The test assumes a normal population model. A small p-value shows inconsistency with the specified null model; it does not prove that every other model is correct.

If $\sigma$ is unknown, use the t-distribution regardless of sample size. A one-sample t-test tests a population mean by estimating the unknown population standard deviation from one sample. Technology carries out the calculation, after which the p-value is interpreted in the usual way.

In a matched-pairs design, observations form meaningful pairs. These could be two measurements from the same individual or measurements from deliberately matched individuals. Convert each pair into a difference. If $d_i$ is the difference for pair $i$, where $d_i$ has the units of the measured variable, test the population mean difference with a one-sample t-test. Pairing is handled as a single-sample technique, not as an independent two-sample test.

Testing a population proportion with a binomial model

A binomial hypothesis test is an exact test of a population success probability. It requires a fixed number of independent trials, two outcomes and a constant success probability.

Let

If the claimed probability is $\pi_0$, where $\pi_0$ is a dimensionless hypothesized success probability, then $H_0:\pi=\pi_0$.

Only one-tailed binomial tests are required. For an observed number of successes $x$, where $x$ is a dimensionless observed count:

• if $H_1:\pi>\pi_0$, calculate $P(X\geq x)$ under $H_0$;
• if $H_1:\pi<\pi_0$, calculate $P(X\leq x)$ under $H_0$.

Since $X$ is discrete, the probability of a critical region usually cannot equal $\alpha$ exactly. Choose the largest tail region whose probability under $H_0$ stays below the stated significance level. Push the boundary inward as far as possible without letting the Type I error probability reach or exceed the permitted level.

Testing a Poisson population mean

A Poisson hypothesis test is an exact one-tailed test for the mean number of independent events in a fixed interval under a Poisson model. Write

In examinations, the data are supplied.

For $H_0:\lambda=\lambda_0$, where $\lambda_0$ is the hypothesized mean number of events per interval, use the relevant cumulative probability under $\operatorname{Po}(\lambda_0)$. For an upper-tailed claim, find the probability of the observed total or more. For a lower-tailed claim, use the probability of the observed total or fewer. As in a binomial test, choose a discrete critical region that maximizes its probability while keeping it below $\alpha$.

When several equal intervals are combined, their total count is also Poisson. Under $H_0$, the mean of this total equals the number of intervals multiplied by $\lambda_0$. Keep the interval attached to the parameter when interpreting it: a mean of events per hour is not interchangeable with a mean per day.

Testing a population correlation

The population product moment correlation coefficient, written $\rho$, is a dimensionless parameter measuring the direction and strength of linear association in a population. To test for the presence of a linear association, use

against the one-tailed or two-tailed alternative that matches the claim. The paired variables are assumed to follow a bivariate normal distribution, so their joint population distribution is modelled by a two-variable normal distribution.

Technology calculates the test result from the supplied bivariate data. The sample product moment correlation coefficient is $r$, where $r$ is a dimensionless measure of linear association in the sample. Its sign and size describe the sample; the hypothesis test concerns the population parameter $\rho$.

Rejecting $H_0$ provides evidence of non-zero linear correlation, not causation. The association may instead reflect a lurking variable, reverse direction or the study design. Failing to reject $H_0$ doesn’t prove that no relationship exists either. A relationship could be weak, non-linear or obscured by limited data.

Type I and Type II errors

A hypothesis-test decision can go wrong in two different ways.

A Type I error occurs when a true null hypothesis is rejected. Its probability is

A Type II error occurs when a false null hypothesis is not rejected. Its probability is written $\beta$, where $\beta$ is a dimensionless probability. Unlike $\alpha$, it must be calculated for a specified alternative parameter value. There is no single value of $\beta$ covering every possible alternative.

The power of a test is the probability that the test correctly rejects $H_0$ at a specified alternative parameter value:

For a normal mean with known variance, first calculate an observation or sample-mean critical boundary from the distribution under $H_0$. Then, using the distribution under the stated alternative, find the probability of falling outside the rejection region. The probability inside the non-rejection region under that alternative is $\beta$.

The same reasoning applies to binomial and Poisson models, using cumulative probabilities. First find the integer-valued critical region under $H_0$. Then use the alternative success probability or alternative Poisson mean to calculate the probability of landing outside that region. Only one-tailed discrete cases are required.

If the sample size and model stay unchanged, reducing $\alpha$ generally makes rejection harder and increases $\beta$. Increasing the sample size can reduce both error probabilities because the sampling distributions become more concentrated. The consequences determine which error needs greater protection. A false alarm may be especially serious when an intervention is costly or harmful, whereas a missed effect may matter more if failing to act creates substantial risk.

The two-distribution diagram shows how both the decision boundary and the true parameter value affect these errors.

Interpreting claims responsibly

A significant result indicates that the data would be unusually extreme under a particular null model. It does not show that the conclusion is certainly true. Tests focus on selected population parameters and may leave other scientifically important features unexamined. A test of a mean does not establish the shape of a distribution, while a test of correlation does not establish a causal mechanism.

This limitation matters particularly in field studies across the sciences and human sciences. Mathematical evidence contributes to a knowledge claim, but the confidence it deserves also depends on the study design, validity of measurements, plausibility of assumptions and costs of the two error types.