Learning

Model types: Approximation (or function regression)

1.1. Approximation (or function regression)

In an approximation problem, a neural network learns how one or more continuous target variables depend on a set of input variables. No formula for that relationship is available beforehand; the model has to infer it from examples.

The examples come from a data set. Each sample pairs the values of the input variables with the values measured for the target variables. During training, the targets play the role of the correct answers against which the outputs of the neural network are compared.

Once trained, the model returns an estimate of the targets for any new combination of inputs. In the simplest case there is a single input, x, and a single target, y, and the model is a curve that runs through the cloud of points, as the following figure shows.

Approximation, also called regression, is one of the most widely used model types. Some examples are the following:

Measured targets are never exact. Each value combines a systematic part, which changes smoothly with the inputs, and a random part caused by sensor noise, unrecorded factors or rounding. In statistical terms, when errors are measured as squared differences, the best possible prediction for a given input is the mean of the target values that would be observed for it, and that mean is precisely the systematic part.

A good model therefore follows the trend and ignores the scatter around it. A neural network that bends to pass through every training point has also learned the noise, and its predictions for new samples get worse rather than better. The ability to predict well on samples that were not used for training is called generalization, and it is what ultimately decides how good an approximation model is. The Model selection chapter explains how to control it.

All model types