Machine learning approaches

Why data cleaning matters

Selecting informative features

Reducing dimensionality

Data cleaning makes the data a model learns from more consistent and improves its quality. When explaining why it matters, link each cleaning action to the problem it prevents. Unreliable inputs can lead to unreliable patterns and predictions.

### Data quality and model performance

**Data cleaning** means identifying and dealing with errors, inconsistencies and unsuitable entries in a data set.

A model learns from the data it receives, not what that data was meant to represent. Incorrect measurements can distort a relationship. Duplicate records may give some observations too much influence, while inconsistent formats can stop equivalent values from being treated alike. Cleaning can improve model performance, but it doesn't guarantee accurate predictions.

The figure links common data-quality problems to suitable cleaning actions. Choose the action according to what caused the problem. Deleting every unusual record is not a sound general rule.
{{4f7bafe50dcc2603b0fb2d6dfe2235f72c6304cb}}

### Choosing a cleaning technique

| Data-quality problem | Suitable action | Why the choice matters |
|---|---|---|
| Outliers | Investigate values that are unusually distant from the others. Correct verified errors, but retain genuine unusual observations. Only remove an entry when there is a justified reason. | An extreme value can distort learning, yet a rare observation may still be important. |
| Duplicate data | Remove repeated copies of the same observation, or combine overlapping records into one consistent record. | Repetition can give an observation disproportionate influence. Separate observations are not automatically duplicates just because their measurements are identical. |
| Incorrect data | Check values against permitted ranges, consistency rules or the original source. Correct a value when its correct form can be established. | Even a value with the right data type can be wrong: a negative duration, for example. |
| Irrelevant data | Filter out records or fields that do not belong to the prediction task. | Unrelated information can add noise and increase processing work. |
| Improperly formatted data | Convert equivalent values to a consistent representation, using consistent units and data types. | A duration written as text must be interpreted consistently with numeric durations. |
| Missing data | Choose imputation, deletion or predictive modelling according to the amount and nature of the missing information. | Each treatment preserves or discards different amounts of information and can affect the patterns learned. |

An **outlier** is an observation that differs markedly from the other observations. Suppose a pump runtime is recorded as 600 minutes among much shorter runs. Check it rather than automatically replacing it: the value could be a recording error, or it could be a genuine long run.

### Worked cleaning example

In this invented example, a greenhouse records pump runtime to predict water use. The raw records below include a format inconsistency, a duplicate and a missing value.

| Record identifier | Runtime in minutes | Water use in litres |
|---|---:|---:|
| R1 | 2 | 8 |
| R2 | 2 min | 10 |
| R3 | 6 | 24 |
| R3 | 6 | 24 |
| R4 | 6 | 26 |
| R5 | Missing | 18 |

Start by converting the text entry for R2 to the numeric value 2, keeping minutes as the consistent unit. Then confirm that the repeated R3 entry represents the same pump run before removing the extra copy. Keep R1 and R2 separate. Equal runtimes do not make them duplicate observations.

There are several ways to handle the missing runtime for R5:

- **Imputation** replaces missing data with a chosen substitute value. The mean of the four observed runtimes gives (2 + 2 + 6 + 6) / 4 = 4 minutes.
- **Deletion** removes a record or field that contains missing data. Deleting R5 avoids estimating its runtime, but also loses its recorded water-use information. Extensive deletion can leave an unrepresentative data set.
- **Predictive modelling** estimates a missing value from relationships with other available attributes. This can use more information than a single replacement value, though the estimates can still be wrong.

Replacing every missing runtime with 4 creates repeated estimated values; it does not recover the actual measurements. Choose the treatment deliberately, because it changes the data.

### Normalization and standardization

**Normalization** rescales numeric values to a common range, commonly from 0 to 1.

For min–max normalization:

normalized value = (value − minimum) / (maximum − minimum)

**Standardization** rescales numeric values by subtracting their mean and dividing by their standard deviation. Provided the original standard deviation is not zero, the resulting feature has a mean of 0 and a standard deviation of 1.

standardized value = (value − mean) / standard deviation

These steps matter for models that are sensitive to numerical scale. Without scaling, a feature measured using large numbers can dominate one measured using small numbers, even if it is no more informative. Scaling makes the magnitudes more comparable. It does not correct inaccurate measurements or remove irrelevant features.

After duplicate removal, the four observed runtimes are 2, 2, 6 and 6. The minimum is 2 and the maximum is 6. Their mean is 4, with a population standard deviation of 2.

| Runtime | Normalization calculation | Normalized value | Standardization calculation | Standardized value |
|---:|---|---:|---|---:|
| 2 | (2 − 2) / (6 − 2) | 0 | (2 − 4) / 2 | −1 |
| 2 | (2 − 2) / (6 − 2) | 0 | (2 − 4) / 2 | −1 |
| 6 | (6 − 2) / (6 − 2) | 1 | (6 − 4) / 2 | 1 |
| 6 | (6 − 2) / (6 − 2) | 1 | (6 − 4) / 2 | 1 |

With those same scaling values, the imputed runtime of 4 becomes (4 − 2) / (6 − 2) = 0.5 after normalization, or (4 − 4) / 2 = 0 after standardization. Normalization gives a chosen range. Standardization instead expresses a value's position relative to the mean and spread, without fixing its range.

**Terms to use:** data cleaning, data quality, model performance, outliers, duplicate data, incorrect data, irrelevant data, improperly formatted data, imputation, deletion, predictive modelling, normalization, standardization.

A schematic linking data-quality problems to cleaning actions in paired rows: outliers to investigate; duplicate data to remove or consolidate; incorrect data to check and correct; irrelevant data to filter; improperly formatted data to transform; missing data to imputation, deletion or predictive modelling. Connect the cleaned data to model performance, without implying guaranteed accuracy.

Feature selection identifies which input attributes provide useful information for a prediction task, then retains an appropriate subset. It removes unhelpful or redundant inputs while keeping information that supports model performance.

### What is selected?

A **feature** is an input attribute used by a machine learning model.

**Feature selection** chooses a subset of the original features judged most informative for the task. It removes whole input attributes. It doesn't change their units or replace missing entries.

Useful features help distinguish outcomes. An irrelevant feature contributes no useful information for the task; a redundant feature repeats information already available from another feature. Removing either can reduce processing and storage requirements and make the model easier to interpret. Remove a genuinely useful feature, though, and predictive performance can fall.

### Worked example: selecting inputs

The greenhouse wants to predict water use. It has four complete observations with these candidate inputs:

| Record identifier | Runtime in minutes | Runtime in seconds | Recorded unit | Water use in litres: target |
|---|---:|---:|---|---:|
| R1 | 2 | 120 | minutes | 8 |
| R2 | 2 | 120 | minutes | 10 |
| R3 | 6 | 360 | minutes | 24 |
| R4 | 6 | 360 | minutes | 26 |

Runtime in seconds repeats exactly the information in runtime in minutes. Since the recorded unit is constant, it cannot distinguish these observations. The record identifier tells us which run it is but, in this example, has no meaningful relationship with water use.

A sensible candidate subset would keep runtime in minutes and remove runtime in seconds, recorded unit and record identifier. Water use is the target to predict, not an input feature. Although the table suggests a relationship between runtime and water use, four observations alone do not establish that the resulting model will perform well on other runs.

### Three feature selection strategies

The figure compares where selection happens. Filter methods assess features before model training. Wrapper methods compare subsets through model training and evaluation, whereas embedded methods select features during training.
{{4826fcf56a93bcd69316dc915b9723e1eef840c5}}

| Strategy | How it selects features | Strength | Limitation |
|---|---|---|---|
| **Filter methods** | Assess feature relevance or redundancy from properties of the data, independently of training a particular predictive model. | Usually comparatively quick, so they can screen many features. | A feature that looks weak on its own may be useful in combination with another feature. |
| **Wrapper methods** | Train and evaluate a model with different feature subsets, then choose a subset according to its performance. | Assess how useful features are for the chosen model, including combinations. | Repeated training and evaluation can require substantial computation. |
| **Embedded methods** | Select features as part of the model's own training process. | Combine model fitting with feature selection, without requiring a separate search around the model. | Selection depends on the training method and may not suit a different model. |

For the greenhouse, a filter method could identify the constant recorded-unit feature and the duplicated runtime information. To compare candidate subsets, a wrapper method could train and evaluate a model for each. With an embedded method, selection would happen within training itself.

All three strategies serve the same purpose. What differs is how they judge usefulness. The most informative subset depends on the task: keeping fewer features helps only if those retained preserve the information needed for prediction.

**Terms to use:** feature selection, informative attributes, filter methods, wrapper methods, embedded methods.

Three parallel process diagrams contrasting feature selection strategies. Filter methods: data, assess features, selected features, model training. Wrapper methods: candidate subsets, model training and evaluation, selected features, with a loop back to candidate subsets. Embedded methods: data, model training with feature selection, selected features. Emphasize before training versus repeated training and evaluation versus selection during training.

Dimensionality reduction cuts the number of input variables while keeping the information relevant to the task. It addresses difficulties that can arise when a data set has many features, especially if there are relatively few observations.

### Dimensions and the curse of dimensionality

A **dimension** is one feature used to describe an observation. When a data set has many input features, its observations occupy a high-dimensional space.

The **curse of dimensionality** refers to the difficulties that arise as the number of dimensions increases. More features don't automatically provide more useful information.

Two important consequences are **data sparsity** and **overfitting**:

- **Data sparsity:** adding dimensions gives observations a much larger space of possible feature combinations to occupy. If the number of observations stays the same, fewer nearby examples may be available to support a prediction.
- **Overfitting:** a model may learn accidental patterns in the available data instead of relationships that hold for new observations. Extra irrelevant features give it more opportunities to fit these patterns.

There are other considerations: computational complexity, the effectiveness of distance metrics, data visualization, increases in the required sample size and memory usage. Distances, for example, can become less useful for distinguishing nearby observations from distant ones. Storing and processing additional feature values also requires more resources.

### Preserving relevant information

**Dimensionality reduction** represents data with fewer variables while retaining the aspects relevant to its intended use.

One approach is feature selection, which keeps a subset of the original variables. A lower-dimensional representation may also combine information from existing variables. Fewer columns alone aren't enough: the representation must preserve useful information.

The figure contrasts retaining informative variables with discarding useful information. Reducing dimensions helps only if the simpler representation still supports the task.
{{41584059e21354de0dd6667d6ae884fc11c95fe8}}

### Worked example: fewer variables, same useful relationship

Consider two input variables from the greenhouse observations:

| Record identifier | Runtime in minutes | Runtime in seconds | Water use in litres: target |
|---|---:|---:|---:|
| R1 | 2 | 120 | 8 |
| R2 | 2 | 120 | 10 |
| R3 | 6 | 360 | 24 |
| R4 | 6 | 360 | 26 |

Runtime in seconds is always runtime in minutes multiplied by 60. Keep only runtime in minutes, and the input falls from two dimensions to one without losing the runtime information relevant to predicting water use. The target stays unchanged.

Removing both runtime variables, by contrast, would discard the only useful input relationship shown here. The aim is to remove unnecessary dimensions without removing information the model needs.

### Benefits and limitations

| Potential benefit | Why it can help | What must be preserved |
|---|---|---|
| Less opportunity for overfitting | Removing irrelevant inputs leaves fewer opportunities to learn accidental patterns. | Features that carry genuine predictive information. |
| Lower processing and memory requirements | Each observation has fewer input values to store and process. | Enough detail for the intended task. |
| Easier visualization | Fewer dimensions make a representation easier to display and inspect. | Relevant relationships that might otherwise disappear from view. |

Dimensionality reduction doesn't guarantee better accuracy. Reduce too far, and important distinctions between observations can disappear. Its value depends on keeping the information the model requires, not merely making the data set smaller.

**Terms to use:** dimensionality reduction, variables, relevant aspects of the data, curse of dimensionality, overfitting, computational complexity, data sparsity, distance metrics, data visualization, sample size, memory usage.

A branching schematic beginning with runtime in minutes and runtime in seconds as two input dimensions for predicting water use in litres. The useful branch retains runtime in minutes as one dimension and is labelled relevant information preserved. The contrasting branch removes both runtime variables and is labelled useful information lost. Keep the target water use in litres separate from the input dimensions.

Machine learning fundamentals

State one data-cleaning action that could be applied to the combined training data.

State one possible effect of poor-quality training data on the model.

A software fault has copied some maintenance records several times. Outline why the repeated copies should be removed before training.

Some records have a missing value for the age of an apartment’s heating system. Describe how predictive modelling could be used to handle these missing values.

A distance-based model uses repair cost in currency units and fault duration in days. Repair costs have a much larger numerical range than fault durations. Explain why normalization would be appropriate before training this model.

Records from older buildings are more likely to contain missing values. Marden Homes proposes deleting every incomplete record before training the model and then using its predictions to prioritize repairs. Discuss the ethical implications of this proposal.

State one potential benefit of feature selection for Orven’s model.

State one difficulty associated with the curse of dimensionality.

Orven has limited time available for preprocessing. Outline one reason for initially using a filter method rather than a wrapper method to select features.

Orven later has sufficient processing time to investigate feature combinations. Describe how a wrapper method could select a suitable subset of features for predicting journey duration.

Orven proposes keeping every available feature because each might contain useful information. Explain why reducing dimensionality could instead improve predictions for new journeys when only a small number of completed journeys is available.

Orven proposes excluding all passenger assistance requirements from the input features to protect privacy. Journey-duration predictions will be used to set pickup schedules. Discuss whether Orven should adopt this feature-selection policy.

Computer Science

IB Computer Science focuses on computational thinking and programming skills. Students learn about system fundamentals, network architecture, and software development, applying their knowledge to solve real-world problems through practical projects.

A4.2 Data preprocessing

Question 1

[1]

[1]

[2]

[3]

[3]

[5]

Question 2

[1]

[1]

[2]

[3]

[3]

[5]