Linear regression uses a straight-line relationship with a predictor to model a continuous response. It predicts a numerical amount, not a category.

Predictor, response and regression equation

The input used to make a prediction is the independent variable, or predictor. The dependent variable, or response, is the outcome being predicted.

With one predictor, the regression equation is:

Predicted y = m × x + c

Here, x is the predictor, m is the slope and c is the intercept. During training, the model chooses suitable values for m and c to fit the line to the observed data.

• The slope gives the change in predicted y for a one-unit increase in x. A positive slope predicts an increase; a negative slope predicts a decrease.
• The intercept gives the predicted value of y when x is zero. It may serve as a mathematical starting point without representing a meaningful real-world situation.

Worked example

A workshop records how long a machine runs and how much electricity it consumes.

For each extra hour of running time, consumption rises by 2 kWh, so m = 2. Substitute the first observation: 3 = 2 × 1 + c, so c = 1. The fitted model is predicted y = 2 × x + 1.

How well does the line fit?

R-squared, written r2, measures the proportion of variation in the response that the fitted model accounts for, relative to predicting the mean response.

For a usual fitted regression model, a value near 1 indicates that the model accounts for much of the observed variation. A value near 0 indicates little improvement over predicting the mean. Here, the observations lie exactly on the line, giving r2 = 1.

Even a strong fit to the observed data doesn't guarantee accurate future predictions. The relationship may not continue beyond the range represented in the training data. Nor does a fitted relationship, by itself, establish causation.

Key terms: linear regression, independent variable, predictor, dependent variable, response, slope, intercept, r2.

Classification predicts a discrete category from input features. In supervised learning, examples come with known labels, which are used to classify new examples.

K-Nearest Neighbours

To classify a new example, K-Nearest Neighbours (K-NN) uses the labels of the k most similar training examples. Similarity is commonly measured by the distance between feature values.

Consider parcels described by their mass and longest side.

A new parcel has mass 2 kg and longest side 12 cm. With k = 3, the nearest neighbours are B, C and A. Each is labelled Small, so K-NN predicts Small. The new parcel wasn't given a label; its category comes from nearby labelled examples.

K-NN can also support recommendations. Users with similar past preferences can provide evidence about which workshop tutorials another user may like. This is one application of collaborative filtering, while classification has many other applications.

Decision trees

A decision tree predicts an outcome by following feature-based tests from a root to a leaf. At each internal decision, it chooses a branch. The leaf then supplies the predicted category.

To assign a handling category, a more complex tree can combine several tests, such as size, mass and fragility. K-NN consults nearby examples when making a prediction; a decision tree instead applies the sequence of tests it learned during training.

Key terms: classification, supervised learning, labelled data, K-Nearest Neighbours, K-NN, decision tree, discrete categorical outcomes.

Hyperparameter tuning involves comparing settings that control a learning algorithm, with the aim of improving its performance on unseen data. These settings sit outside the model's ordinary learning process. Examples include k in K-NN and maximum decision-tree depth.

Measuring classification performance

For a fault detector, treat Fault as the positive category. A correctly predicted fault is a true positive (TP); a correctly predicted non-fault is a true negative (TN). A false positive (FP) occurs when a non-fault is incorrectly flagged as a fault. A fault the model misses is a false negative (FN).

The detector makes 20 predictions altogether. Each metric answers a different question about the same results.

When one category is much more common than another, accuracy alone can mislead. Missed faults make recall a priority; costly unnecessary interventions make precision a priority. F1 combines precision and recall rather than rewarding correct negative predictions.

Tuning and generalization

Fit candidate models on training data and use validation data to compare their hyperparameter settings. Then assess the chosen model on separate test data. If you repeatedly choose settings from test results, the test no longer provides a reliable independent check.

A model shows Underfitting when it is too simple to capture important patterns. It typically performs poorly on both training and unseen data.

Overfitting occurs when a model learns incidental details or noise in the training data rather than patterns that generalize. It can perform strongly on training data but more weakly on unseen data.

Tuning isn't simply a search for the highest training accuracy. The aim is to find settings that perform well on independent examples, using metrics suited to the consequences of errors.

Key terms: accuracy, precision, recall, F1 score, hyperparameter tuning, model performance, overfitting, underfitting.

Clustering groups examples by similarities in their features, without preassigned category labels. As an unsupervised learning approach, it discovers groups in the data rather than learning them from labelled answers.

From features to groups

Measured features represent each example. A clustering technique compares them, looking for groups in which members are more similar to one another than to members of other groups.

Consider a tool-hire service that records two features for each account.

The algorithm received no labels such as Occasional or Frequent. Those descriptions are interpretations added after examining the groups it discovered.

Uses and interpretation

The service could offer different hire packages to these groups. Patterns in purchasing behaviour can similarly reveal customer segments. Other applications group documents or sensor readings.

What counts as similar depends on the chosen features. Treating account identifiers as meaningful measurements, for example, could produce misleading groups. Different feature choices or clustering settings may also produce different groupings. A cluster therefore needs to be interpreted in context, rather than assumed to represent a fixed natural category.

Key terms: clustering, unsupervised learning, features, similarities, groups.

An association rule describes a recurring relationship: when one attribute or combination of attributes is present, another tends to be present too. Written as A → B, the rule says that records containing A also tend to contain B. It does not say that A causes B.

Discovering and interpreting rules

Association-rule mining searches a large collection of records for combinations that repeatedly occur together. These combinations can then be expressed as rules to help describe the data or guide decisions.

Consider this small extract from a bicycle-repair booking system.

Four bookings include tyre replacement. Of those, three also include brake adjustment. The rule Tyre replacement → Brake adjustment therefore has a conditional frequency of 3 / 4, or 75%, in this extract.

Reversing the rule makes a separate claim. Brake adjustment also occurs four times here, and three of those bookings include tyre replacement, so the conditional frequency happens to be the same. This symmetry comes from these particular records; it is not a general property of association rules.

Using the result appropriately

The repair service might use the rule to make brake-adjustment appointment capacity available alongside tyre-replacement slots. But the rule does not establish that replacing a tyre creates a need for brake adjustment. Nor does it predict that every tyre booking will include both services.

A relationship found in just a few records may be unreliable. When interpreting a rule, consider how often the combination appears and whether the records represent the situation where the rule will be used. This small table illustrates the method; useful mining normally examines much larger data sets.

Key terms: association rule, association-rule mining, attributes, data sets, relationships.

Reinforcement learning involves learning how to act by interacting with an environment and using rewards as feedback. The goal is a high cumulative reward, rather than simply the best immediate reward.

The interaction cycle

The decision-making learner is called an agent. The system it interacts with is its environment.

A state describes the agent's current situation. The agent makes a choice, called an action, then receives numerical feedback: a reward.

A policy determines which action the agent chooses in a given state. Learning changes this policy as the agent gains experience, so its choices tend to lead to better outcomes.

Worked interaction

An automated greenhouse cart learns how to reach a charging point. Its state includes its location, and it can take actions such as Move to junction and Move to charger. Each movement in this simplified episode has a cost; reaching the charger gives a positive reward.

The first action earns a negative immediate reward. Yet it makes the later positive reward possible. The sequence earns -1 + 5 = 4, showing why decisions must be judged by their longer-term consequences.

Exploration and exploitation

Exploration means trying actions to discover their consequences. Choosing actions that current experience suggests are best is Exploitation.

The cart could exploit a known route to the charger or explore an unfamiliar route that might be shorter. If it only exploits, a better route may remain undiscovered. Too much exploration, though, wastes time on poor choices. Learning requires a balance between gathering information and using what has already been learned.

Key terms: reinforcement learning, agent, environment, state, action, reward, policy, cumulative reward, exploration, exploitation.

A genetic algorithm searches for a good solution by repeatedly evaluating and modifying a population of candidate solutions. It suits optimization problems with so many possible combinations that checking every candidate would be impractical.

Representing and evaluating candidates

The current collection of candidate solutions is the population. A fitness function scores each candidate according to how well it meets the objective.

Suppose four independent workshop jobs need to be allocated to two suitable workstations. Each candidate uses one digit per job: 0 means workstation A and 1 means workstation B. In this simplified example, 1010 is the preferred allocation. Fitness counts the jobs assigned to their preferred workstation.

For a practical scheduling problem, fitness could combine factors such as completion time and workstation workload. Both the representation and the fitness function must reflect the actual problem.

Producing new candidates

The genetic-algorithm cycle has these stages:

• Evaluation: calculate each candidate's fitness.
• Selection: choose candidates to reproduce, generally favouring higher fitness without necessarily excluding every weaker candidate.
• Crossover: combine parts of selected parents to produce offspring.
• Mutation: make occasional small changes that introduce variation.
• Termination: stop when a chosen condition is met, such as reaching a target fitness or completing a fixed number of generations.

Crossing 1010 and 1000 after the second digit, for example, produces 1000 and 1010. Mutating the third digit of 1000 produces 1010. These operations don't always improve fitness. Offspring must therefore be evaluated, rather than assumed to be better.

Real-world uses and limits

Genetic algorithms can search workshop schedules, communication-antenna arrangements or component designs. Each application needs its own candidate representation and fitness function. Any candidate that violates essential constraints must be excluded or penalized.

The process can find useful solutions without checking every possibility. It does not, however, guarantee the global best solution. If the fitness function is poorly chosen, it can reward an allocation that scores well mathematically yet fails the real objective.

Key terms: genetic algorithm, population, fitness function, selection, crossover, mutation, evaluation, termination, optimization.

An artificial neural network (ANN) consists of interconnected processing nodes. As it learns from data, it adjusts the weights of the connections between them. Depending on its outputs and training examples, it can perform classification, regression or pattern recognition.

A single perceptron

A perceptron combines weighted inputs with a bias, then applies an activation function to produce an output.

• Each input supplies a feature value.
• A weight controls how much its input influences the result.
• The bias shifts the weighted total so the decision threshold can move.
• The activation function transforms the weighted total into the node's output.
• The output passes onward or serves as the prediction.

Worked perceptron

A simple classifier uses two binary sensor inputs. Its activation function outputs 1 when the weighted total is greater than zero, and 0 otherwise.

With bias = -1, the weighted total is 2 + 0 - 1 = 1. Because this is greater than zero, the output is 1. A change to the inputs, weights or bias can change the result.

Multi-layer perceptrons

A multi-layer perceptron (MLP) is a neural network with an input layer, one or more hidden layers and an output layer.

Features enter through the input layer. Hidden layers combine and transform information from preceding layers, while the output layer produces the required prediction: a category score or a continuous numerical value, for example. During training, the network adjusts connection weights and biases to reduce prediction errors.

With nonlinear activation functions, hidden layers allow the network to represent more complex relationships than a single perceptron. Adding layers alone, without nonlinear transformations, would not give it the same ability to model complex patterns.

Key terms: artificial neural network, ANN, perceptron, input, weights, bias, activation function, output, multi-layer perceptron, MLP, input layer, hidden layers, output layer.

A convolutional neural network (CNN) learns spatial patterns by applying filters across an input, such as an image. Rather than treating pixel positions as irrelevant, it preserves and uses the local relationships between pixels.

Architecture and spatial hierarchy

In a typical image-classification CNN, information passes through these components.

A kernel is a small set of learned weights applied to local regions of an input. Convolution reuses the same kernel at different positions, so it can detect a feature in different parts of an image.

Early layers may respond to edges. Later layers may respond to shapes built from those edges, while deeper layers respond to combinations useful for recognizing objects. This progression forms a spatial hierarchy of features. During training, the kernels are adjusted so that the learned features help reduce classification errors. The programmer doesn't have to specify every useful edge or shape.

Worked classification example

Suppose the training data includes these labelled images of workshop fasteners.

A new image with a circular boundary and central opening may receive a high Washer output. These written descriptions are for the reader. The CNN receives pixels, learning useful visual features from those pixels and their labels. This tiny set illustrates the process; it isn't a sufficient practical training collection.

Effects of design choices

Stride is the distance a kernel moves between successive positions. Each choice below affects both processing and the resulting model.

The loss function measures the discrepancy between predictions and the required outputs. It has a different job from an activation function. An activation transforms signals within the network; the loss supplies the objective that training tries to reduce.

Key terms: convolutional neural network, CNN, convolutional layers, kernel size, stride, activation functions, pooling layers, fully connected layers, output layer, loss function, spatial hierarchies of features.

Model selection means choosing a machine learning approach and configuration to suit the problem, available data and required outcome. Models need comparing: an algorithm that works well on one data set may perform poorly on another.

Start with the required outcome

Start by asking what the model should output. Predicting a continuous amount calls for regression; predicting a known category calls for classification. To discover groups without labels, use clustering. Learning a sequence of decisions from rewards calls for reinforcement learning.

Models can tackle the same task while making different assumptions. Linear regression suits an approximately straight-line numerical relationship, whereas K-NN relies on meaningful similarity between examples. A decision tree learns feature-based splits. For more complex patterns, a multi-layer neural network can represent them but may need more data and computation.

Worked comparison

A workshop tests two fault classifiers on the same independent data. The data contain eight actual faults and twelve non-faults.

The accuracy is the same. The mistakes aren't. K-NN misses one fault rather than two, so its recall is higher. With two false alarms rather than three, the decision tree has higher precision. K-NN may be preferable here if missed faults are especially costly; the tree may be preferable if unnecessary interventions dominate the cost.

For the F1 calculations, the equivalent expression 2 × TP / (2 × TP + FP + FN) gives 12 / 16 for the tree and 14 / 18 for K-NN.

Match complexity to the data

How much training data is available? How noisy is it? Model performance depends on these characteristics, along with feature scales, the balance of categories and whether relationships are simple or complex.

A complex model may capture a relationship that a simple model misses. With limited data, though, it can also fit noise. Irrelevant features or unsuitable feature scales can weaken K-NN. If a model is evaluated mostly on a common category, it may appear successful while failing on the rare category that matters most.

Compare candidates on comparable independent data, using metrics that reflect the desired outcome. Check whether the model's complexity and computational demands are appropriate, and how easy it is to explain its decisions. No model is universally best: choose the one whose demonstrated behaviour best fits the actual task.

Key terms: model selection, model comparison, algorithms, data characteristics, problem complexity, desired outcomes, model performance.