Populations, samples and types of data

A population is the full set of individuals or items that a statistical investigation aims to find out about. A sample is a subset chosen from that population for observation. Unless the question states otherwise, treat an SL data set as the whole population when calculating its statistics.

A random sample is chosen through a chance process in which every member of the population has an equal opportunity of selection. This reduces deliberate or unconscious selection bias. Even so, chance can still produce a random sample that is unrepresentative.

Discrete data are numerical data with separate, countable possible values. For example, the number of faults in a device is discrete, even if many values are possible. Continuous data are numerical data that may take any value within an interval, limited by the precision of measurement. Time, mass and temperature are normally continuous.

On meeting a data set, first identify the population being studied and how the sample was chosen. Then consider what each variable measures, whether the data are discrete or continuous, and whether the observations actually answer the research question. Data become useful statistical information only when their context and meaning are known.

Reliability, validity and bias

A reliable data source uses collection and recording procedures that are consistent enough for the results to be reproduced. Reliability and validity aren’t the same. A procedure may repeatedly give similar values while still measuring the wrong thing.

Sampling bias is a systematic distortion that causes some members of a population to be represented more often than others. Possible causes include an incomplete sampling frame, voluntary participation, non-response, inaccessible groups, leading questions or collecting data at an unsuitable place or time. A larger sample won’t fix a biased selection method. It may simply give a more precise version of the same distortion.

Don’t replace missing values automatically with zero. Zero is a recorded value, while a missing entry says that no value is known. Check whether values are missing randomly or whether one group is less likely to respond. Recording errors may result from incorrect units, mistyped entries, duplicated records, impossible values or inconsistent measurement procedures. If a value is corrected or excluded, the decision should be justified and documented rather than made simply because the value looks inconvenient.

These concerns apply in experimental sciences, psychology, geography, economics and business research. Both historical surveys and modern digital data sets have led to misleading conclusions when respondents or users didn’t represent the intended population. International comparisons bring extra difficulties, as countries may define variables differently, use different collection methods or have unequal access to social groups.

There is an ethical issue too. A graph or average may be technically correct yet still mislead if it comes from a poor sample or is shown without important limitations. Statistics is often distinguished from pure mathematics because its conclusions depend on valid calculations and on the way real observations were produced. A correct displayed calculation does not justify deliberately hiding a known weakness in the sampling.

Sampling methods

How effective a method is depends on whether it is practical and likely to represent the target population.

For a systematic sample, an appropriate interval is approximately

where $k$ is the sampling interval (unitless), $N$ is the population size (a count, unitless) and $n$ is the required number of observations in the sample (a count, unitless). When $k$ is not an integer, clearly state the practical rule used for selection.

For proportional stratified sampling, allocate to group $h$ using

where $n_h$ is the number selected from group $h$ (a count, unitless), $h$ identifies the group (unitless) and $N_h$ is the population size of group $h$ (a count, unitless). Rounding the allocations may require a small adjustment to keep their total equal to $n$.

Outliers

The interquartile range measures the spread of the middle half of an ordered data set:

where $IQR$ is the interquartile range (the same unit as the data), $Q_1$ is the lower quartile separating the lowest quarter of the data (the data unit) and $Q_3$ is the upper quartile separating the highest quarter (the data unit).

In this course, an outlier is a data value more than $1.5$ interquartile ranges below $Q_1$ or above $Q_3$. Calculate the outlier fences from

and

where $L$ is the lower outlier fence (the data unit) and $U$ is the upper outlier fence (the data unit). Any value below $L$ or above $U$ is classified as an outlier. A value exactly on a fence is not more than $1.5IQR$ from its nearest quartile.

Treat an outlier as a reason to investigate, not an automatic instruction to delete it. It may be a genuine rare observation and therefore an important part of the population. Alternatively, it may come from a recording, unit or measurement error. Check the source and context before deciding what to do with it. Box-and-whisker diagrams show outliers explicitly, and these values can strongly affect some measures of central tendency and dispersion.

Frequency distributions

A frequency distribution is a table that arranges data values or class intervals alongside the number of observations in each category. This makes it easier to see the shape of a large data set and where its values are concentrated.

With discrete data, each row may show one exact value. Continuous data, by contrast, are grouped into class intervals written as inequalities. These intervals must cover the required range with no gaps or overlaps. For example, one class may include its lower boundary but exclude its upper boundary; the next class then starts at that same boundary. Each observation will therefore fall into exactly one class.

The table should clearly identify the measured variable and its unit. As a quick check, add the frequencies and compare the result with the total number of observations.

Grouped frequency table for height data.

Histograms

A histogram is a graph for numerical data. Adjacent bars represent continuous class intervals, while the size of each bar represents frequency. For the equal-width intervals required here, frequency is plotted on the vertical axis. Frequency-density histograms are not required.

Since adjacent intervals meet on a continuous scale, the bars touch. The horizontal scale gives class boundaries, not category names, so a histogram isn't just a bar chart with its gaps removed. Both axes should be labelled, including the unit of the measured variable.

Its overall shape may show concentration, spread or symmetry. It can also reveal skewness, gaps and possible unusual observations. Grouping does hide the exact values within each interval, though, so different class boundaries can alter the apparent shape.

Cumulative frequency

Cumulative frequency is the running total of observations at or below successive class boundaries. On a cumulative frequency graph, these totals are plotted against the upper class boundaries. The graph must never decrease because observations are added as the boundary moves to the right.

To read a required cumulative position, locate it on the vertical axis, move horizontally to the curve, then vertically to the data axis. Read the median at half the total frequency, $Q_1$ at one quarter and $Q_3$ at three quarters. Percentiles are found in the same way. The $p$th percentile is the value at or below which approximately $p$ percent of the observations lie, where $p$ is the percentile rank (percent, treated as unitless).

The range is the difference between the greatest and least observed values:

Use the previously defined quartiles to obtain the interquartile range from the cumulative graph. Since readings taken from a drawn curve are estimates, use sensible precision.

Box-and-whisker diagrams

A box-and-whisker diagram is a one-dimensional statistical display showing the quartiles, median, non-outlying extremes and any outliers. Its box extends from $Q_1$ to $Q_3$, with a line marking the median. The whiskers reach the smallest and largest observations that are not outliers. Any outliers are shown separately with crosses.

Box plots are especially useful when distributions are compared on the same scale:

• Compare the medians to discuss typical location.
• Compare the interquartile ranges to discuss the spread of the middle half.
• Compare the overall non-outlier ranges to discuss wider spread.
• Compare both sides of each box and the whisker lengths to discuss symmetry or skewness.
• Record the number and position of outliers instead of silently ignoring them.

A roughly symmetric box, with a central median and similarly sized whiskers, is consistent with a normal distribution. This doesn't prove normality. Many non-normal distributions can produce a fairly symmetric box plot, while a small sample may look irregular even when it comes from a normal population.

Different subjects produce data from raw observations in different ways. A laboratory measurement may be data; so may a coded survey response or an economic indicator. Their meanings and collection procedures still differ. A display becomes useful information only when its variables, units, population and method of collection are understood.

Measures of central tendency

A measure of central tendency is a statistic that represents the central or typical location of a distribution.

The arithmetic mean is found by adding all the observations and dividing by the number of observations:

Since the mean uses every value, extreme observations can affect it strongly.

The median is the middle value of an ordered data set. If there is an even number of observations, it is the mean of the two middle values. Because the median depends on position rather than distance from the centre, it is resistant to extreme values.

The mode is the most frequent observed value or category. A data set may have one mode, more than one mode or no mode at all. Unlike the mean and median, the mode also works with non-numerical categories.

Use technology efficiently to obtain these statistics, then interpret the result in context. The mean is often informative for reasonably balanced numerical data. For strongly skewed data or data containing outliers, the median is often more representative. The mode identifies the most common outcome.

Estimating the mean of grouped data

Grouping observations into intervals removes their exact values. To estimate the mean, represent each observation in a class by the midpoint of that interval:

The symbol $\approx$ shows that the result is an estimate, unless every observation actually equals its class midpoint.

A modal class is the class interval with the greatest frequency. In this course, identify it only for equal-width class intervals. It gives an interval rather than an exact modal value.

Grouped heights with midpoints, products and modal class.

Measures of dispersion

A measure of dispersion is a statistic that describes how widely the data values are spread. Two distributions may share the same mean or median yet have very different variability.

The interquartile range describes the spread of the middle half, so it resists extreme values. By contrast, the range uses only the two extremes and is highly sensitive to outliers.

The variance is the mean squared distance of the population values from their mean. For a population data set,

The standard deviation is the positive square root of the variance:

A small standard deviation shows that values tend to lie close to the mean; a large standard deviation shows that they are more widely spread. Calculate standard deviation and variance using technology, taking care to select the population statistic and interpret it. Hand calculation can show the structure, but it isn’t the required routine method.

Variance uses squared units, so it is less directly interpretable than standard deviation. Calculator outputs and references may follow different variance conventions: some describe a complete population, while others estimate population variation from a sample. They answer different statistical questions and aren’t contradictory mathematical truths. Under the SL convention, use population statistics unless the question explicitly states that the data form a sample.

Measures of spread can compare variation in scientific measurements and social indicators. They can also be used for agricultural output, prices and the reliability of manufactured components. Shared international data may reveal important patterns, but comparisons only make sense when definitions, units and collection methods are sufficiently compatible.

Constant changes to all observations

Suppose every data value undergoes the transformation

The resulting mean and standard deviation satisfy

and

where $\bar{y}$ is the mean of the transformed data (the transformed data unit) and $\sigma_y$ is its population standard deviation (the transformed data unit).

Adding the same constant changes location, but it does not change spread. For example, subtracting a fixed amount from every observation subtracts that amount from the mean and median. The range, interquartile range, variance and standard deviation stay unchanged.

Multiplying every observation by a positive constant multiplies the mean and median by that constant. It has the same effect on the range, interquartile range and standard deviation. In particular, doubling every value doubles both the mean and standard deviation. Variance changes according to

A negative multiplier reverses the order of the data as well. The lower and upper quartiles therefore switch roles after transformation; measures of spread remain non-negative and scale by $|a|$.

Effect of adding a constant and scaling data

Quartiles of discrete data

Use technology to obtain quartiles for discrete data. Calculators, spreadsheets and hand methods may follow different rules when a quartile position lies between observations or when the median is included in the two halves of the data. As a result, two accepted procedures can give slightly different quartiles and interquartile ranges for the same short data set.

Follow the method produced by the required technology consistently. Don’t combine quartiles from one convention with an outlier calculation based on another. This variation shows an important distinction: mathematical definitions can be precise within a stated convention, even when several conventions exist.

Descriptive statistics summarize measured attributes only. They may encourage too much attention to quantities that are easy to record while qualities that are harder to measure are neglected. Interpret a numerical summary alongside the investigation’s purpose, definitions and limitations.