- \n
- 1. Loss index<\/a><\/b><\/li>\n
- 2. Optimization algorithm<\/a><\/b><\/li>\n<\/ul>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/training_strategy.svg\")
<\/p>\n<\/section>\n\n 1. Loss index<\/h2>\n
The loss index defines the task for the neural network and measures how well it learns.<\/p>\n
The choice of a suitable loss index depends on the application.<\/p>\n
When setting a loss index, we must choose two different terms: an error term<\/a> and a regularization term<\/a>.<\/p>\n
$$\\text{loss index} = \\text{error term} + \\text{regularization term}$$<\/p>\n
Error term<\/h3>\n
The error is the most important term in the loss index.<\/p>\n
It measures how well the neural network<\/a>\u00a0fits the\u00a0data set<\/a>.<\/p>\n
We can calculate errors on the training<\/a>,\u00a0 selection<\/a>, and testing<\/a>\u00a0samples<\/p>\n
Next, we describe the most important errors used in machine learning:<\/p>\n
- \n
- Mean squared error<\/a>.<\/li>\n
- Normalized squared error<\/a>.<\/li>\n
- Weighted squared error<\/a>.<\/li>\n
- Cross-entropy error<\/a>.<\/li>\n
- Minkowski error<\/a>.<\/li>\n<\/ul>\n
We calculate the gradient of all these error methods using the backpropagation algorithm.<\/p>\n
Mean squared error (MSE)<\/h4>\n
The mean squared error is the average of the squared differences between the neural network outputs and the dataset’s targets.<\/p>\n
$$\\text{mean squared error} = \\frac{\\sum (outputs – targets)^2}{\\text{samples number}}$$<\/p>\n
Normalized squared error (NSE)<\/h4>\n<\/section>\n
The normalized squared error is the squared difference between outputs and targets divided by a normalization factor.<\/p>\n
A value of zero means the network predicts the data perfectly.<\/p>\n
On the other side, a value of one means the network only predicts the average of the data.<\/p>\n
$$\\text{normalized squared error} = \\frac{\\sum (outputs – targets)^2}{\\text{normalization coefficient}}$$<\/section>\n <\/section>\n The normalized squared error is the default error term for approximation problems.<\/p>\n Weighted squared error (WSE)<\/h4>\n<\/section>\n
The weighted squared error is used in binary classification with unbalanced data.<\/p>\n
This happens when the number of positive and negative samples is very different.<\/p>\n
It assigns weights so that positives and negatives contribute equally to the error.<\/p>\n
<\/section>\n $$\\text{weighted squared error} =
\n\\text{positives weight} \\cdot \\sum (outputs – \\text{positive targets})^2
\n+ \\text{negatives weight} \\cdot \\sum (outputs – \\text{negative targets})^2$$<\/section>\n<\/section>\n \n Cross-entropy error<\/h4>\n
The cross-entropy error is used in both binary and multi-class classification problems.<\/p>\n
It penalizes heavily when the network assigns a high probability to the wrong class.<\/p>\n
For binary classification, the cross-entropy error is<\/p>\n<\/section>\n
$$\\text{Minkowski error} =
\n\\frac{\\sum \\Big( \\text{outputs} – \\text{targets} \\Big)^{\\text{Minkowski parameter}}}{\\text{samples number}}$$<\/p>\nA perfect model has a cross-entropy error of zero.<\/p>\n Minkowski error (ME)<\/h4>\n<\/section>\n
The previous errors can be very sensitive to outliers.<\/p>\n
In such cases, the Minkowski error provides better generalization.<\/p>\n
It raises the differences between outputs and targets to a power called the Minkowski parameter.<\/p>\n
This parameter ranges from 1 to 2, with a default value of 1.5.<\/p>\n
$$\\text{Minkowski error} =
\n\\frac{\\sum \\Big( \\text{outputs} – \\text{targets} \\Big)^{\\text{Minkowski parameter}}}{\\text{samples number}}$$<\/p>\nRegularization term<\/h3>\n
A model is regular when small changes in the inputs only cause small changes in the outputs.<\/p>\n
If the model is not regular, it may overfit the training data and fail to generalize.<\/p>\n
To avoid this, we add a regularization term to the loss index.<\/p>\n
The term keeps the network\u2019s weights and biases small, making the model simpler and smoother.<\/p>\n
The main types of regularization are:<\/p>\n
- \n
- L1 regularization<\/a>.<\/li>\n
- L2 regularization<\/a>.<\/li>\n<\/ul>\n
We can easily compute the gradient of these regularization terms.<\/p>\n
L1 regularization<\/h3>\n
L1 regularization adds up the absolute values of all the network\u2019s parameters.<\/p>\n
It drives the parameters to small values, which makes the model simpler and less likely to overfit.<\/p>\n
$$\\text{L1 regularization} =
\n\\text{regularization weight} \\cdot \\sum \\lvert \\text{parameters} \\rvert$$<\/p>\nL2 regularization<\/h3>\n
L2 regularization is the sum of the squared values of all the network\u2019s parameters.<\/p>\n
It also drives the parameters to small values, making the model more regular.<\/p>\n
$$\\text{L2 regularization} =
\n\\text{regularization weight} \\cdot \\sum \\text{parameters}^2$$<\/p>\nAs we can see, a parameter controls the weight of the regularization term.<\/p>\n
We decrease the weight if the model is too smooth and increase it if the model oscillates too much.<\/p>\n
Loss function<\/h3>\n
The loss index depends on the neural network function and the data set.<\/p>\n
We can imagine it as a surface in many dimensions, with the parameters as coordinates.<\/p>\n
The following figure shows this idea.<\/p>\n
![\"Loss](\"https:\/\/www.neuraldesigner.com\/images\/loss_function.svg\")
<\/p>\n<\/p>\n<\/section>\n
Training a neural network consists of finding the parameter values that minimize the loss index.<\/p>\n
\n 2. Optimization algorithm<\/h2>\n
An optimization algorithm is the method used to adjust the parameters of a neural network to minimize the loss index.<\/p>\n
Training starts with random parameters and iteratively improves them until reaching a minimum of the loss index.<\/p>\n<\/section>\n
The following figure shows this process.<\/p>\n
![\"Training](\"https:\/\/www.neuraldesigner.com\/images\/training_process.svg\")
<\/p>\nThe optimization algorithm stops when a chosen condition is met.<\/p>\n
Standard stopping criteria include:<\/p>\n
- \n
- The maximum number of epochs is reached.<\/li>\n
- The maximum computing time is exceeded.<\/li>\n
- The selection error increases for several epochs.<\/li>\n<\/ul>\n
The optimization algorithm decides how to adjust the neural network parameters.<\/p>\n
Different algorithms have different computational and memory requirements, and no one is best for all problems.<\/p>\n<\/section>\n
\n Next, we describe the most common optimization algorithms.<\/p>\n<\/section>\n
\n - \n
- Gradient descent<\/a>.<\/li>\n
- Newton’s method<\/a>.<\/li>\n
- Quasi-Newton method<\/a>.<\/li>\n
- Levenberg-Marquardt algorithm<\/a>.<\/li>\n
- Stochastic gradient descent<\/a>.<\/li>\n
- Adaptive linear momentum<\/a>.<\/li>\n<\/ul>\n
Gradient descent (GD)<\/h3>\n
The simplest optimization algorithm is gradient descent.<\/p>\n
It updates the parameters each epoch in the direction of the negative gradient of the loss index.<\/p>\n
A factor called the learning rate controls the change of parameters.<\/p>\n
$$\\text{New parameters} =
\n\\text{parameters} – \\text{loss gradient} \\cdot \\text{learning rate}$$<\/p>\nThe main drawback of gradient descent is that it can converge very slowly.<\/p>\n
Newton method (NM)<\/h3>\n
Newton\u2019s method uses the Hessian matrix, which contains all the second derivatives of the\u00a0loss function<\/a>.<\/p>\n
Unlike gradient descent, which only relies on first derivatives, Newton\u2019s method uses curvature information to find better training directions.<\/p>\n
$$\\text{New parameters} =
\n\\text{parameters} – \\text{loss Hessian}^{-1} \\cdot \\text{loss gradient} \\cdot \\text{learning rate}$$<\/p>\nThe main drawback of Newton\u2019s method is the high computational cost of calculating the Hessian matrix.<\/p>\n
Quasi-Newton method (QNM)<\/h3>\n<\/section>\n
Newton\u2019s method uses the Hessian matrix of second derivatives to set the learning direction.<\/p>\n
This gives high accuracy but is very expensive to compute.<\/p>\n
Quasi-Newton methods avoid this by approximating the inverse Hessian with only gradient information.<\/p>\n
Line minimization algorithms adjust the learning rate at each epoch.<\/p>\n
$$\\text{New parameters} =
\n\\text{parameters} – \\text{inverse Hessian approximation} \\cdot \\text{gradient} \\cdot \\text{learning rate}$$<\/section>\nThe Quasi-Newton method makes training faster than gradient descent and less costly than Newton\u2019s method.<\/p>\n
Therefore, it is the default algorithm for small and medium neural networks.<\/p>\n Levenberg-Marquardt algorithm (LM)<\/h3>\n
The Levenberg\u2013Marquardt algorithm is another optimizer that achieves near second-order speed without computing the Hessian matrix.<\/p>\n
This method applies only when the loss index is a sum of squares, such as the mean squared error or the normalized squared error.<\/p>\n
It requires the gradient and the Jacobian matrix of the loss index.<\/p>\n
$$\\text{New parameters} =
\n\\text{parameters} – \\text{damping parameter} \\cdot \\text{Jacobian} \\cdot \\text{gradient}$$<\/p>\nThe Levenberg-Marquardt algorithm is very fast but requires a lot of memory, so it is recommended only for small networks.<\/p>\n
Stochastic gradient descent (SGD)<\/h3>\n<\/section>\n
Stochastic gradient descent (SGD) works differently from the previous algorithms.<\/p>\n
It updates the parameters several times in each epoch using small batches of data.<\/p>\n
$$\\text{New parameters} =
\n\\text{parameters} – \\text{batch gradient} \\cdot \\text{learning rate} + \\text{momentum}$$<\/section>\n<\/section>\n Stochastic gradient descent makes training faster on large data sets.Its drawback is that the updates are noisy and may lead to slower convergence.<\/p>\n Adaptive linear momentum (ADAM)<\/h3>\n<\/section>\n
The Adam algorithm is similar to stochastic gradient descent but uses a more advanced way to calculate the training direction.<\/p>\n
It also adapts the learning rate for each parameter, which usually makes convergence faster.<\/p>\n
$$\\text{New parameters} =
\n\\text{parameters} – \\frac{\\text{gradient exponential decay}}{\\sqrt{\\text{square gradient exponential decay}}} \\cdot \\text{learning rate}$$<\/section>\n<\/section>\n Adam is the default optimization algorithm for large neural networks.<\/p>\n Performance considerations<\/h3>\n
For small datasets (about 10 variables and 10,000 samples), the Levenberg-Marquardt algorithm<\/a> is recommended for its speed and precision.<\/p>\n
With medium-sized problems, the quasi-Newton method<\/a> works\u00a0well.<\/p>\n
For\u00a0large data sets (about 1,000 variables and 1,000,000 samples), <\/span>adaptive linear momentum<\/a>\u00a0is the best choice.<\/p>\n
The article “5 Algorithms to Train a Neural Network”<\/a><\/span>\u00a0in the Neural Designer blog<\/a> contains more information about this subject.<\/p>\n<\/section>\n
\u21d0 Neural Network<\/a>
\nModel Selection \u21d2<\/a><\/section>\n","protected":false},"author":122,"featured_media":1428,"template":"","categories":[30],"tags":[36],"class_list":["post-3539","learning","type-learning","status-publish","has-post-thumbnail","hentry","category-tutorials","tag-tutorials"],"acf":[],"yoast_head":"\nMachine learning: Training strategy - tutorial<\/title>\n<meta name=\"description\" content=\"This tutorial shows the main training strategies used by neural networks to learn.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Neural networks tutorial: Training strategy\" \/>\n<meta property=\"og:description\" content=\"This tutorial shows the main training strategies used by neural networks to learn.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/\" \/>\n<meta property=\"og:site_name\" content=\"Neural Designer\" \/>\n<meta property=\"article:modified_time\" content=\"2026-02-11T08:01:00+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Neural networks tutorial: Training strategy\" \/>\n<meta name=\"twitter:description\" content=\"This tutorial shows the main training strategies used by neural networks to learn.\" \/>\n<meta name=\"twitter:site\" content=\"@NeuralDesigner\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/\",\"url\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/\",\"name\":\"Machine learning: Training strategy - tutorial\",\"isPartOf\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/training_process.svg\",\"datePublished\":\"2025-11-25T10:12:57+00:00\",\"dateModified\":\"2026-02-11T08:01:00+00:00\",\"description\":\"This tutorial shows the main training strategies used by neural networks to learn.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#primaryimage\",\"url\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/training_process.svg\",\"contentUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/training_process.svg\",\"width\":13341,\"height\":8113},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.neuraldesigner.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Learning\",\"item\":\"https:\/\/www.neuraldesigner.com\/learning\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Machine learning: Training strategy – tutorial\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#website\",\"url\":\"https:\/\/www.neuraldesigner.com\/\",\"name\":\"Neural Designer\",\"description\":\"Explanable AI Platform\",\"publisher\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.neuraldesigner.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#organization\",\"name\":\"Neural Designer\",\"url\":\"https:\/\/www.neuraldesigner.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png\",\"contentUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png\",\"width\":1024,\"height\":223,\"caption\":\"Neural Designer\"},\"image\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/NeuralDesigner\",\"https:\/\/es.linkedin.com\/showcase\/neuraldesigner\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Machine learning: Training strategy - tutorial","description":"This tutorial shows the main training strategies used by neural networks to learn.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/","og_locale":"en_US","og_type":"article","og_title":"Neural networks tutorial: Training strategy","og_description":"This tutorial shows the main training strategies used by neural networks to learn.","og_url":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/","og_site_name":"Neural Designer","article_modified_time":"2026-02-11T08:01:00+00:00","twitter_card":"summary_large_image","twitter_title":"Neural networks tutorial: Training strategy","twitter_description":"This tutorial shows the main training strategies used by neural networks to learn.","twitter_site":"@NeuralDesigner","twitter_misc":{"Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/","url":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/","name":"Machine learning: Training strategy - tutorial","isPartOf":{"@id":"https:\/\/www.neuraldesigner.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#primaryimage"},"image":{"@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#primaryimage"},"thumbnailUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/training_process.svg","datePublished":"2025-11-25T10:12:57+00:00","dateModified":"2026-02-11T08:01:00+00:00","description":"This tutorial shows the main training strategies used by neural networks to learn.","breadcrumb":{"@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#primaryimage","url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/training_process.svg","contentUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/training_process.svg","width":13341,"height":8113},{"@type":"BreadcrumbList","@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.neuraldesigner.com\/"},{"@type":"ListItem","position":2,"name":"Learning","item":"https:\/\/www.neuraldesigner.com\/learning\/"},{"@type":"ListItem","position":3,"name":"Machine learning: Training strategy – tutorial"}]},{"@type":"WebSite","@id":"https:\/\/www.neuraldesigner.com\/#website","url":"https:\/\/www.neuraldesigner.com\/","name":"Neural Designer","description":"Explanable AI Platform","publisher":{"@id":"https:\/\/www.neuraldesigner.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.neuraldesigner.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.neuraldesigner.com\/#organization","name":"Neural Designer","url":"https:\/\/www.neuraldesigner.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png","contentUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png","width":1024,"height":223,"caption":"Neural Designer"},"image":{"@id":"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/NeuralDesigner","https:\/\/es.linkedin.com\/showcase\/neuraldesigner\/"]}]}},"_links":{"self":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning\/3539","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning"}],"about":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/types\/learning"}],"author":[{"embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/users\/122"}],"version-history":[{"count":3,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning\/3539\/revisions"}],"predecessor-version":[{"id":21818,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning\/3539\/revisions\/21818"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/media\/1428"}],"wp:attachment":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/media?parent=3539"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/categories?post=3539"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/tags?post=3539"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}} - Newton’s method<\/a>.<\/li>\n
- Gradient descent<\/a>.<\/li>\n
- L2 regularization<\/a>.<\/li>\n<\/ul>\n
- Normalized squared error<\/a>.<\/li>\n
- 2. Optimization algorithm<\/a><\/b><\/li>\n<\/ul>\n