Bivariate data and scatter diagrams

Bivariate data is a set of paired observations that records two variables for each individual or item. We usually label the explanatory variable $x$ and the response variable $y$. Here, $x$ is the explanatory-variable value (in units appropriate to the context), while $y$ is the response-variable value (in units appropriate to the context). Describing a variable as explanatory doesn’t prove that it causes the other variable to change.

A scatter diagram is a coordinate plot in which each paired observation is represented by a point $(x,y)$. From the plot, we can judge the direction, strength and form of an association. We can also spot clusters or possible outliers.

• Positive correlation means that larger values of $x$ tend to occur with larger values of $y$.
• Negative correlation means that larger values of $x$ tend to occur with smaller values of $y$.
• Zero correlation means that there is no overall linear tendency.
• Linear correlation becomes stronger as the points lie closer to a line. A wide cloud suggests weak correlation, while a completely patternless cloud shows no evident correlation.

Zero correlation doesn’t always mean that there is no relationship. A clear curved pattern may have little or no linear correlation. That’s why the graph should be inspected before a numerical coefficient is trusted.

Common scatter-diagram patterns and their correlation.

Pearson’s product-moment correlation coefficient

Pearson’s product-moment correlation coefficient is a dimensionless statistic that measures the direction and strength of a linear relationship between two quantitative variables.

The sign of $r$ shows the direction of the linear association; its magnitude shows the strength:

• $r$ close to $1$: strong positive linear correlation;
• $r$ close to $-1$: strong negative linear correlation;
• $r$ close to $0$: weak or no linear correlation.

Don’t treat $r=0$ as proof that the variables are unrelated. Pearson’s coefficient describes linearity, so a strong non-linear relationship may be missed. Outliers can also affect it because the coefficient uses the original numerical values.

Calculate $r$ using technology. Although a hand calculation can show how standardized deviations combine, you don’t need to derive or prove the coefficient. When a critical value is provided, compare it with the magnitude $|r|$ to decide whether the evidence for linear correlation meets the stated criterion.

A line of best fit by eye

A line of best fit is a straight line that represents the central linear trend of a scatter diagram. A line drawn by eye should pass through the mean point $(\bar{x},\bar{y})$. In this point, $\bar{x}$ is the mean of the explanatory-variable values (same unit as $x$), and $\bar{y}$ is the mean of the response-variable values (same unit as $y$). Try to balance the spread of points above and below the line instead of joining particular data points.

Regression of $y$ on $x$

A regression line of $y$ on $x$ is a least-squares line used to estimate the response variable from a specified value of the explanatory variable. It has equation

For every one-unit increase in $x$, the model predicts a change of $a$ units in $y$. If $a$ is positive, the line increases; if $a$ is negative, it decreases. The intercept gives the predicted value of $y$ when $x=0$. It has useful contextual meaning only when zero is sensible and reasonably close to the observed domain.

Use technology to find the regression equation, and keep enough digits for any later substitution. To make a prediction, substitute the given $x$-value, then interpret the resulting $y$-value in context.

The order in “$y$ on $x$” is significant. The line minimizes vertical discrepancies and predicts $y$ from $x$. Simply rearranging it to predict $x$ from a known $y$ is not generally reliable. A separately calculated regression of $x$ on $y$ may be needed instead.

An interpolation is a prediction made within the observed range of the explanatory variable, whereas an extrapolation is a prediction made beyond that range. There is still uncertainty in interpolation. Extrapolation is more dangerous, however, because it assumes that the observed pattern continues. Forecasts in fields such as public health, environmental science or financial modelling rely on that unverified assumption.

Correlation is not causation

Correlation is a statistical association in which two variables tend to vary together. Causation is a relationship in which a change in one factor produces a change in another. Correlation alone cannot establish causation. The association could be caused by a lurking variable, reverse causation, selection bias, coincidence or poor measurement.

For example, a strong association in observational health data may prompt further scientific investigation. Before drawing a causal conclusion, though, we need controlled evidence and a plausible mechanism. This raises an important knowledge question: mathematics can measure an observed pattern with great precision without explaining why that pattern exists.

The reliability and validity of a regression model depend on more than $r$. Check how the data were collected and whether the variables were measured consistently. Also consider whether the sample represents the relevant population, whether influential outliers are present, whether a straight line is suitable and whether the prediction remains within the observed domain. Scatter diagrams and curves of best fit occur naturally in experimental science and geographical data analysis, but subject knowledge gives the model its meaning.

Ranking paired data

Spearman’s rank correlation coefficient is a dimensionless statistic that measures the direction and strength of a monotonic relationship using the ranks of paired observations. The symbol is $r_s$, where $r_s$ is Spearman’s rank correlation coefficient (no unit), and

A monotonic relationship is an association that moves consistently in one direction, although its rate of change need not be constant. As one variable increases, the other therefore tends always to increase or always to decrease. The pattern can be curved; it doesn’t have to be linear.

Spearman’s coefficient uses ordinal positions instead of the original measurements. Rank both sets of values in the same way—for example, give the smallest value rank $1$ in each column—and keep the paired observations together. When two or more values are tied, each receives the average of the ranks they would have occupied. Calculate $r_s$ using technology. You don’t need to derive or prove the formula.

Paired data ranked with average ranks for ties

Interpret it in much the same way as Pearson’s coefficient. A value near $1$ shows a strong increasing monotonic association, while a value near $-1$ shows a strong decreasing monotonic association. A value near $0$ suggests little monotonic association.

Choosing between Pearson and Spearman

Choose the coefficient based on the form and quality of the data, not simply on which value looks larger.

Spearman’s coefficient is less sensitive to outliers, though it isn’t immune to them. An extreme observation may still change rank positions and disrupt the monotonic pattern. Use Pearson’s coefficient when the relationship is genuinely linear and the numerical distances between values matter. Spearman’s is a better choice for ordinal data, a monotonic curve or quantitative data where extreme values would have a disproportionate influence on $r$.

Even a large coefficient doesn’t establish causation. Fieldwork in biology, psychology, environmental studies and exercise science often produces observational data where several factors vary together. International databases can help when exploring these associations. Before comparing country ranks, however, check for differences in definitions, collection methods and population structure.

A fitted pattern doesn’t reveal one uniquely “true” equation either. Several curves may describe the same finite dataset adequately. Choosing between them depends on residual behaviour, predictive performance, subject knowledge and plausible mechanisms. Predictions also assume that the observed pattern will persist. Extrapolation is especially vulnerable when social, environmental or economic conditions change.

Non-linear regression models

A regression curve is a function fitted to bivariate data in order to model the typical value of a response variable for each value of an explanatory variable. If a scatter diagram shows a systematic bend, a non-linear model may fit the data better than a straight line.

You should be able to use technology to fit and assess the following families:

• linear: $y=ax+b$;
• quadratic:

• cubic:

• exponential:

• power:

• sine:

These models use different letters to avoid confusing their coefficients with the slope and intercept already used in linear regression. Technology may show different parameter letters, so identify each one from its position in the equation.

An exponential model is closely linked to a geometric sequence: each equal increase in $x$ multiplies the predicted value by the same factor $q$. A quadratic model has one turning point, while a cubic model can change direction twice. Power models often describe scaling. Sine models represent repeated oscillation.

Comparison of the main non-linear regression model families and their typical curve shapes.

Start by choosing a plausible model from the scatter diagram and the context. Run the relevant regression, record the equation to suitable precision, then graph it over the data. Don’t assume that a more complicated curve gives a better model. Its extra flexibility may simply chase random variation.

Least squares and residuals

A residual is the signed vertical difference between an observed response and the response predicted by a model. For observation $i$,

A positive residual places the observed point above the curve; a negative residual places it below. Least-squares regression is a fitting method that selects model parameters to minimize the sum of the squared residuals. Squaring stops positive and negative residuals from cancelling. It also gives more weight to large discrepancies.

The sum of square residuals is a non-negative measure of the total squared vertical discrepancy between observations and a fitted model. It is calculated using

When comparing models for the same response data, a smaller $SS_{\mathrm{res}}$ shows a closer in-sample fit. Don’t use this value alone, though; inspect the residual plots as well. For a suitable model, the residuals should scatter around zero without a systematic curve, trend or changing spread.

The coefficient of determination

The coefficient of determination is a dimensionless statistic giving the proportion of variability in the response variable accounted for by a fitted model. It is written $R^2$, where $R^2$ is the coefficient of determination (no unit), and is evaluated using technology.

For instance, $R^2=0.82$ says that the model accounts for about $82\%$ of the observed variability in the response variable. It does not say that $82\%$ of the points lie on the curve. Nor does it mean that predictions are $82\%$ accurate or that the explanatory variable causes the response.

The underlying relationship is useful:

For a least-squares linear model with an intercept,

Squaring removes the sign. As a result, $R^2$ measures explained variation but cannot show whether the linear association is positive or negative. The sign of $r$ provides that information.

Evaluating validity

A high $R^2$ provides evidence that the model fits the observed responses closely. It does not prove that the model is valid. Before choosing one, ask:

• Does its shape make sense in the application?
• Are the residuals patternless and reasonably balanced around zero?
• Is an outlier distorting the fit?
• Could a simpler model explain nearly as much?
• Will the model be used for interpolation or risky extrapolation?
• Were the data measured reliably and sampled representatively?
• Do its predictions remain physically or practically possible?

For this reason, graphical analysis in the sciences combines $R^2$ with residuals and subject knowledge. Several models may fit the same dataset well, particularly across a short interval. The most defensible model isn’t necessarily the curve with the largest displayed $R^2$. Choose the one whose assumptions, residual behaviour and contextual interpretation best support its intended use.