Data cleaning makes the data a model learns from more consistent and improves its quality. When explaining why it matters, link each cleaning action to the problem it prevents. Unreliable inputs can lead to unreliable patterns and predictions.

Data quality and model performance

Data cleaning means identifying and dealing with errors, inconsistencies and unsuitable entries in a data set.

A model learns from the data it receives, not what that data was meant to represent. Incorrect measurements can distort a relationship. Duplicate records may give some observations too much influence, while inconsistent formats can stop equivalent values from being treated alike. Cleaning can improve model performance, but it doesn't guarantee accurate predictions.

Choosing a cleaning technique

An outlier is an observation that differs markedly from the other observations. Suppose a pump runtime is recorded as 600 minutes among much shorter runs. Check it rather than automatically replacing it: the value could be a recording error, or it could be a genuine long run.

Worked cleaning example

In this invented example, a greenhouse records pump runtime to predict water use. The raw records below include a format inconsistency, a duplicate and a missing value.

Start by converting the text entry for R2 to the numeric value 2, keeping minutes as the consistent unit. Then confirm that the repeated R3 entry represents the same pump run before removing the extra copy. Keep R1 and R2 separate. Equal runtimes do not make them duplicate observations.

There are several ways to handle the missing runtime for R5:

• Imputation replaces missing data with a chosen substitute value. The mean of the four observed runtimes gives (2 + 2 + 6 + 6) / 4 = 4 minutes.
• Deletion removes a record or field that contains missing data. Deleting R5 avoids estimating its runtime, but also loses its recorded water-use information. Extensive deletion can leave an unrepresentative data set.
• Predictive modelling estimates a missing value from relationships with other available attributes. This can use more information than a single replacement value, though the estimates can still be wrong.

Replacing every missing runtime with 4 creates repeated estimated values; it does not recover the actual measurements. Choose the treatment deliberately, because it changes the data.

Normalization and standardization

Normalization rescales numeric values to a common range, commonly from 0 to 1.

For min–max normalization:

normalized value = (value − minimum) / (maximum − minimum)

Standardization rescales numeric values by subtracting their mean and dividing by their standard deviation. Provided the original standard deviation is not zero, the resulting feature has a mean of 0 and a standard deviation of 1.

standardized value = (value − mean) / standard deviation

These steps matter for models that are sensitive to numerical scale. Without scaling, a feature measured using large numbers can dominate one measured using small numbers, even if it is no more informative. Scaling makes the magnitudes more comparable. It does not correct inaccurate measurements or remove irrelevant features.

After duplicate removal, the four observed runtimes are 2, 2, 6 and 6. The minimum is 2 and the maximum is 6. Their mean is 4, with a population standard deviation of 2.

With those same scaling values, the imputed runtime of 4 becomes (4 − 2) / (6 − 2) = 0.5 after normalization, or (4 − 4) / 2 = 0 after standardization. Normalization gives a chosen range. Standardization instead expresses a value's position relative to the mean and spread, without fixing its range.

Terms to use: data cleaning, data quality, model performance, outliers, duplicate data, incorrect data, irrelevant data, improperly formatted data, imputation, deletion, predictive modelling, normalization, standardization.

Feature selection identifies which input attributes provide useful information for a prediction task, then retains an appropriate subset. It removes unhelpful or redundant inputs while keeping information that supports model performance.

What is selected?

A feature is an input attribute used by a machine learning model.

Feature selection chooses a subset of the original features judged most informative for the task. It removes whole input attributes. It doesn't change their units or replace missing entries.

Useful features help distinguish outcomes. An irrelevant feature contributes no useful information for the task; a redundant feature repeats information already available from another feature. Removing either can reduce processing and storage requirements and make the model easier to interpret. Remove a genuinely useful feature, though, and predictive performance can fall.

Worked example: selecting inputs

The greenhouse wants to predict water use. It has four complete observations with these candidate inputs:

Runtime in seconds repeats exactly the information in runtime in minutes. Since the recorded unit is constant, it cannot distinguish these observations. The record identifier tells us which run it is but, in this example, has no meaningful relationship with water use.

A sensible candidate subset would keep runtime in minutes and remove runtime in seconds, recorded unit and record identifier. Water use is the target to predict, not an input feature. Although the table suggests a relationship between runtime and water use, four observations alone do not establish that the resulting model will perform well on other runs.

Three feature selection strategies

For the greenhouse, a filter method could identify the constant recorded-unit feature and the duplicated runtime information. To compare candidate subsets, a wrapper method could train and evaluate a model for each. With an embedded method, selection would happen within training itself.

All three strategies serve the same purpose. What differs is how they judge usefulness. The most informative subset depends on the task: keeping fewer features helps only if those retained preserve the information needed for prediction.

Terms to use: feature selection, informative attributes, filter methods, wrapper methods, embedded methods.

Dimensionality reduction cuts the number of input variables while keeping the information relevant to the task. It addresses difficulties that can arise when a data set has many features, especially if there are relatively few observations.

Dimensions and the curse of dimensionality

A dimension is one feature used to describe an observation. When a data set has many input features, its observations occupy a high-dimensional space.

The curse of dimensionality refers to the difficulties that arise as the number of dimensions increases. More features don't automatically provide more useful information.

Two important consequences are data sparsity and overfitting:

• Data sparsity: adding dimensions gives observations a much larger space of possible feature combinations to occupy. If the number of observations stays the same, fewer nearby examples may be available to support a prediction.
• Overfitting: a model may learn accidental patterns in the available data instead of relationships that hold for new observations. Extra irrelevant features give it more opportunities to fit these patterns.

There are other considerations: computational complexity, the effectiveness of distance metrics, data visualization, increases in the required sample size and memory usage. Distances, for example, can become less useful for distinguishing nearby observations from distant ones. Storing and processing additional feature values also requires more resources.

Preserving relevant information

Dimensionality reduction represents data with fewer variables while retaining the aspects relevant to its intended use.

One approach is feature selection, which keeps a subset of the original variables. A lower-dimensional representation may also combine information from existing variables. Fewer columns alone aren't enough: the representation must preserve useful information.

Worked example: fewer variables, same useful relationship

Consider two input variables from the greenhouse observations:

Runtime in seconds is always runtime in minutes multiplied by 60. Keep only runtime in minutes, and the input falls from two dimensions to one without losing the runtime information relevant to predicting water use. The target stays unchanged.

Removing both runtime variables, by contrast, would discard the only useful input relationship shown here. The aim is to remove unnecessary dimensions without removing information the model needs.

Benefits and limitations

Dimensionality reduction doesn't guarantee better accuracy. Reduce too far, and important distinctions between observations can disappear. Its value depends on keeping the information the model requires, not merely making the data set smaller.

Terms to use: dimensionality reduction, variables, relevant aspects of the data, curse of dimensionality, overfitting, computational complexity, data sparsity, distance metrics, data visualization, sample size, memory usage.