Feature selection algorithms in machine learning

By Roberto Lopez, Artelnics.

Many typical applications in machine learning, from customer segmentation to medical diagnosis, arise from complex interactions between all types of variables.

But analyzing multiple internal and external variables is very complicated. Indeed, data scientists might have huge data sets. Therefore, they need innovative methods that can select relevant variables.

In this post, we formulate the model selection problem and describe the algorithms most used in practice.


Neural Designer implements all the model selection algorithms described here. To use these algorithms in practice, you can download the Neural Designer.

Problem formulation

In this way, model selection searchs for the topology of a neural network that minimizes the error on new data.

Input selection algorithms search for the optimal subset of inputs.

We define the selection error as the error of a neural network for new data. It measures the ability of the model to predict the result in a new case.

Inputs selection is a method to improve the quality of the predictions. It consists in extracting the subset of inputs that have more influence on a particular physical, biological, social, etc. process.

Growing inputs

The growing inputs method starts by calculating every input's correlation with every output in the neural network.

The growing inputs method starts with the most correlated input. Then, it and keeps adding well-correlated variables. It stops when the selection error starts increasing.

Pruning inputs

The pruning input method also starts by calculating the correlations among every input and output in the neural network.

The pruning inputs algorithm starts with all the variables of the data set and removes the inputs with little correlation with the outputs.

Genetic algorithm

A different class of inputs selection method is the genetic algorithm.

This is a stochastic method based on the mechanics of natural genetics and biological evolution.

The genetic algorithm implemented includes several methods to perform fitness assignment, selection, crossover, and mutation operators.

The following figure shows a simplified flow diagram of the genetic algorithm.

Genetic algorithm

The genetic algorithm starts with a population of different subsets of variables.

In every generation, the fitness of every individual is computed as the selection error for that subset of inputs.

Then, the method evolves the population by selecting some individuals to generate the new population, performing a crossover with the selected population, and mutating the offspring generated during the crossover.


But model selection algorithms are very expensive in computational terms, so a big drawback here is the performance.

Neural Designer includes an advanced model selection framework capable of representing very complex data sets.

This system procures high added value to data scientists, providing them with results in a way previously unachievable.

Related posts: