Model selection searches for the neural network architecture with the best generalization properties.
That is, the process that minimizes the error on the selected instances of the data set (the selection error).
There are two families of model selection algorithms in machine learning:
Neuron selection
The size of the hidden layers determines how complex a neural network is: the more neurons, the more flexible the family of functions it can represent. Choosing that size is a matter of balance: with too few neurons, the network cannot follow the relationship in the data; with too many, it starts to follow the noise as well.
Both failures show up in the same place: the error on samples that were not used to fit the parameters, which Neural Designer calls the selection error.
Underfitting, overfitting and the right model size
Consider a synthetic data set with one input, x, and one target, y. It contains 40 samples of a curve that rises steeply, stays almost flat and then has a narrow dip, with random noise added to every target value. We train networks of different sizes on 30 of the samples and keep the other 10 as selection samples.
A network with a single hidden neuron can only draw one S-shaped step. It captures the rise but misses the dip, so both errors stay high: the normalized squared error is 0.29 on the training samples and 0.20 on the selection samples. This is underfitting: the network lacks the flexibility that the data require.
With four hidden neurons, the network reproduces the rise, the flat stretch and the dip, and it passes between the noisy points rather than through them. Its complexity matches the problem: the error is 0.062 on the training samples and 0.089 on the selection samples.
With ten hidden neurons, the network has 31 parameters for 30 training samples. It bends to follow individual training points and oscillates between them. Its training error drops to 0.018, less than a third of the four-neuron value, while its selection error rises to 1.02, more than eleven times higher. This is overfitting: the model is complex enough to memorize the noise.
The error of a neural network on the training instances of the data set is called the training error. Similarly, the error on the selected instances is called the selection error.
The training error measures the ability of the neural network to fit the data it sees. However, the selection error measures the ability of the neural network to generalize to new data.
The following figure shows the training (blue) and selection (orange) errors as a function of the neural network complexity, represented by the number of hidden neurons.
As shown, increasing hidden neurons reduces training error, but both very small and very large networks lead to high selection error.
The first case indicates underfitting, while the second indicates overfitting.
Here, the best generalization occurs with four hidden neurons, where the selection error reaches its minimum.
Neuron selection algorithms find the neural network’s complexity that yields the best generalization properties.
Growing neurons is the most common neuron selection method.
Growing neurons
This algorithm begins with a small network and gradually adds neurons until a stopping criterion is reached.
The algorithm returns the neural network with the optimal number of neurons obtained.
Input selection
Which features should you use to create a predictive model? This is a difficult question that may require in-depth knowledge of the problem domain.
Input selection algorithms automatically extract those features in the data set that provide the best generalization capabilities.
They search for the subset of inputs that minimizes the selection error.
The most common input selection algorithms are:
Growing inputs
The growing inputs method calculates the correlation of every input with every output in the data set.
It starts with a neural network that only contains the most correlated input and calculates the selection error for that model.
It continues to add the most correlated variables until the selection error increases.
The algorithm returns the neural network with the optimal subset of inputs found.
Pruning inputs
The pruning inputs method starts with all the inputs in the data set.
It keeps removing those inputs with the smallest correlation with the outputs until the selection error increases.
Genetic algorithm
A different class of input selection method is the genetic algorithm.
This is a stochastic method based on natural genetics and biological evolution mechanics.
Genetic algorithms usually include fitness assignment, selection, crossover, and mutation operators.

You can find more information about this topic in the Genetic algorithms for feature selection post on our blog.
Related reading: Training, selection and testing samples in machine learning.