{"id":3533,"date":"2025-11-25T11:12:58","date_gmt":"2025-11-25T10:12:58","guid":{"rendered":"https:\/\/neuraldesigner.com\/learning\/model-selection\/"},"modified":"2025-11-27T12:55:03","modified_gmt":"2025-11-27T11:55:03","slug":"model-selection","status":"publish","type":"learning","link":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/","title":{"rendered":"Machine learning tutorial: Model selection"},"content":{"rendered":"<p>Model selection searches for the neural network architecture with the best generalization properties.<\/p>\n<p>That is, the process that minimizes the error on the selected instances of the data set (the selection error).<\/p>\n<p>There are two families of model selection algorithms in machine learning:<\/p>\n<section>\n<ul>\n<li><b><a href=\"#NeuronsSelection\">5.1. Neuron selection<\/a><\/b><\/li>\n<li><b><a href=\"#InputsSelection\">5.2. Input selection<\/a><\/b><\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/model_selection.svg\" width=\"53\" height=\"53\" \/><\/p>\n<\/section>\n<section>\n<h2 id=\"NeuronsSelection\">1. Neuron selection<\/h2>\n<p>Two frequent problems in designing a neural network are called underfitting and overfitting.<\/p>\n<p>The best generalization is achieved using a model with the most appropriate complexity to produce a good data fit.<\/p>\n<h3>Underfitting and overfitting<\/h3>\n<p>To illustrate underfitting and overfitting, consider the following data set. It consists of data from a sine function to which white noise has been added.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/sinus.webp\" \/><\/p>\n<p>The best generalization is achieved using a model whose complexity is the most appropriate to produce an adequate data fit. In this case, we use a neural network with one input (x), three hidden neurons, and one output (y).<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/correct-fitting.webp\" \/><\/p>\n<p>In this way, underfitting is the effect of a selection error increasing due to a too-simple model. Here, we have used one hidden neuron.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/underfitting.webp\" \/><\/p>\n<p>On the contrary, overfitting is the effect of a selection error increasing due to a complex model. In this case, we have used 10 hidden neurons.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/overfitting.webp\" \/><\/p>\n<p>The error of a neural network on the training instances of the data set is called the training error. Similarly, the error on the selected instances is called the selection error.<\/p>\n<p>The training error measures the ability of the neural network to fit the data it sees. However, the selection error measures the ability of the neural network to generalize to new data.<\/p>\n<p>The following figure shows the training (blue) and selection (orange) errors as a function of the neural network complexity, represented by the number of hidden neurons.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/order-selection.webp\" \/><\/p>\n<p>As shown, increasing hidden neurons reduces training error, but both very small and very large networks lead to high selection error.<\/p>\n<p>The first case indicates underfitting, while the second indicates overfitting.<\/p>\n<p>Here, the best generalization occurs with four hidden neurons, where the selection error reaches its minimum.<\/p>\n<p>Neuron selection algorithms find the neural network&#8217;s complexity that yields the best generalization properties.<\/p>\n<p>Growing neurons is the most common neuron selection method.<\/p>\n<h3 id=\"GrowingNeurons\">Growing neurons<\/h3>\n<p>This algorithm begins with a small network and gradually adds neurons until a stopping criterion is reached.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/growing_neuron.svg\" alt=\"Growing neuron\" width=\"150\" \/><\/p>\n<p>The algorithm returns the neural network with the optimal number of neurons obtained.<\/p>\n<\/section>\n<section>\n<h2 id=\"InputsSelection\">2. Input selection<\/h2>\n<p>Which features should you use to create a predictive model? This is a difficult question that may require in-depth knowledge of the problem domain.<\/p>\n<p>Input selection algorithms automatically extract those features in the data set that provide the best generalization capabilities.<br \/>\nThey search for the subset of inputs that minimizes the selection error.<\/p>\n<p>The most common input selection algorithms are:<\/p>\n<ul>\n<li><a href=\"#GrowingInputs\">Growing inputs<\/a>.<\/li>\n<li><a href=\"#PrunningInputs\">Pruning inputs<\/a>.<\/li>\n<li><a href=\"#GeneticAlgorithm\">Genetic algorithm<\/a>.<\/li>\n<\/ul>\n<h3 id=\"GrowingInputs\">Growing inputs<\/h3>\n<p>The growing inputs method calculates the correlation of every input with every output in the data set.<\/p>\n<p>It starts with a neural network that only contains the most correlated input and calculates the selection error for that model.<\/p>\n<p>It continues to add the most correlated variables until the selection error increases.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/growing_input.svg\" alt=\"Growing input\" width=\"150\" \/><\/p>\n<p>The algorithm returns the neural network with the optimal subset of inputs found.<\/p>\n<h3 id=\"PruningInputs\">Pruning inputs<\/h3>\n<p>The pruning inputs method starts with all the inputs in the data set.<\/p>\n<p>It keeps removing those inputs with the smallest correlation with the outputs until the selection error increases.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/pruning_input.svg\" alt=\"Pruning input\" width=\"150\" \/><\/p>\n<h3 id=\"GeneticAlgorithm\">Genetic algorithm<\/h3>\n<p>A different class of input selection method is the genetic algorithm.<\/p>\n<p>This is a stochastic method based on natural genetics and biological evolution mechanics.<\/p>\n<p>Genetic algorithms usually include fitness assignment, selection, crossover, and mutation operators.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/genetic_algorithm.webp\" alt=\"genetic algorithm feature selection\" width=\"300\" \/><\/p>\n<p>You can find more information about this topic in the <a href=\"https:\/\/www.neuraldesigner.com\/blog\/genetic_algorithms_for_feature_selection\">Genetic algorithms for feature selection<\/a> post on our blog.<\/p>\n<\/section>\n<section><a style=\"float: left;\" href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\"> \u21d0 Training Strategy <\/a><br \/>\n<a style=\"float: right;\" href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/testing-analysis\"> Testing Analysis \u21d2<\/a><\/section>\n","protected":false},"author":122,"featured_media":1903,"template":"","categories":[30],"tags":[36],"class_list":["post-3533","learning","type-learning","status-publish","has-post-thumbnail","hentry","category-tutorials","tag-tutorials"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Machine learning tutorial: Model selection<\/title>\n<meta name=\"description\" content=\"This tutorial describes different model selection techniques in machine learning: neurons selection and inputs selection.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Neural networks tutorial: Model selection\" \/>\n<meta property=\"og:description\" content=\"This tutorial describes different techniques for improving the generalization performance of a neural network.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/\" \/>\n<meta property=\"og:site_name\" content=\"Neural Designer\" \/>\n<meta property=\"article:modified_time\" content=\"2025-11-27T11:55:03+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/model-graph-hcv.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"607\" \/>\n\t<meta property=\"og:image:height\" content=\"524\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Neural networks tutorial: Model selection\" \/>\n<meta name=\"twitter:description\" content=\"This tutorial describes different techniques for improving the generalization performance of a neural network.\" \/>\n<meta name=\"twitter:site\" content=\"@NeuralDesigner\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/\",\"url\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/\",\"name\":\"Machine learning tutorial: Model selection\",\"isPartOf\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/model-graph-hcv.webp\",\"datePublished\":\"2025-11-25T10:12:58+00:00\",\"dateModified\":\"2025-11-27T11:55:03+00:00\",\"description\":\"This tutorial describes different model selection techniques in machine learning: neurons selection and inputs selection.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#primaryimage\",\"url\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/model-graph-hcv.webp\",\"contentUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/model-graph-hcv.webp\",\"width\":607,\"height\":524},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.neuraldesigner.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Learning\",\"item\":\"https:\/\/www.neuraldesigner.com\/learning\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Machine learning tutorial: Model selection\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#website\",\"url\":\"https:\/\/www.neuraldesigner.com\/\",\"name\":\"Neural Designer\",\"description\":\"Explanable AI Platform\",\"publisher\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.neuraldesigner.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#organization\",\"name\":\"Neural Designer\",\"url\":\"https:\/\/www.neuraldesigner.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png\",\"contentUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png\",\"width\":1024,\"height\":223,\"caption\":\"Neural Designer\"},\"image\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/NeuralDesigner\",\"https:\/\/es.linkedin.com\/showcase\/neuraldesigner\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Machine learning tutorial: Model selection","description":"This tutorial describes different model selection techniques in machine learning: neurons selection and inputs selection.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/","og_locale":"en_US","og_type":"article","og_title":"Neural networks tutorial: Model selection","og_description":"This tutorial describes different techniques for improving the generalization performance of a neural network.","og_url":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/","og_site_name":"Neural Designer","article_modified_time":"2025-11-27T11:55:03+00:00","og_image":[{"width":607,"height":524,"url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/model-graph-hcv.webp","type":"image\/webp"}],"twitter_card":"summary_large_image","twitter_title":"Neural networks tutorial: Model selection","twitter_description":"This tutorial describes different techniques for improving the generalization performance of a neural network.","twitter_site":"@NeuralDesigner","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/","url":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/","name":"Machine learning tutorial: Model selection","isPartOf":{"@id":"https:\/\/www.neuraldesigner.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#primaryimage"},"image":{"@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#primaryimage"},"thumbnailUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/model-graph-hcv.webp","datePublished":"2025-11-25T10:12:58+00:00","dateModified":"2025-11-27T11:55:03+00:00","description":"This tutorial describes different model selection techniques in machine learning: neurons selection and inputs selection.","breadcrumb":{"@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#primaryimage","url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/model-graph-hcv.webp","contentUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/model-graph-hcv.webp","width":607,"height":524},{"@type":"BreadcrumbList","@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/model-selection\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.neuraldesigner.com\/"},{"@type":"ListItem","position":2,"name":"Learning","item":"https:\/\/www.neuraldesigner.com\/learning\/"},{"@type":"ListItem","position":3,"name":"Machine learning tutorial: Model selection"}]},{"@type":"WebSite","@id":"https:\/\/www.neuraldesigner.com\/#website","url":"https:\/\/www.neuraldesigner.com\/","name":"Neural Designer","description":"Explanable AI Platform","publisher":{"@id":"https:\/\/www.neuraldesigner.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.neuraldesigner.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.neuraldesigner.com\/#organization","name":"Neural Designer","url":"https:\/\/www.neuraldesigner.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png","contentUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png","width":1024,"height":223,"caption":"Neural Designer"},"image":{"@id":"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/NeuralDesigner","https:\/\/es.linkedin.com\/showcase\/neuraldesigner\/"]}]}},"_links":{"self":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning\/3533","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning"}],"about":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/types\/learning"}],"author":[{"embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/users\/122"}],"version-history":[{"count":1,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning\/3533\/revisions"}],"predecessor-version":[{"id":21349,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning\/3533\/revisions\/21349"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/media\/1903"}],"wp:attachment":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/media?parent=3533"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/categories?post=3533"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/tags?post=3533"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}