{"id":3534,"date":"2025-11-24T11:12:58","date_gmt":"2025-11-24T10:12:58","guid":{"rendered":"https:\/\/neuraldesigner.com\/learning\/neural-network\/"},"modified":"2026-08-26T10:34:23","modified_gmt":"2026-08-26T08:34:23","slug":"neural-network","status":"publish","type":"learning","link":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/","title":{"rendered":"Machine learning tutorial: Neural networks"},"content":{"rendered":"<style>\n.nd-math-block {\n  display: block;\n  max-width: 100%;\n  overflow-x: auto;\n  margin: 1rem 0;\n  padding: 0.45rem 0;\n  text-align: center;\n}\n.nd-math-inline {\n  display: inline-block;\n  max-width: 100%;\n  overflow-x: auto;\n  vertical-align: middle;\n}\n.nd-math-block math,\n.nd-math-inline math {\n  font-size: 1.04em;\n}\ntable .nd-math-block {\n  margin: 0.25rem 0;\n  padding: 0.2rem 0;\n}\n<\/style>\n<style>.ndg{width:100vw;margin-left:calc(50% - 50vw);background:#eeeeee;padding:22px 24px 12px;font-family:\"Outfit\",\"Roboto\",Arial,sans-serif;color:#1b2635}.ndg *{box-sizing:border-box}.ndg a{text-decoration:none}.ndg-wrap{width:min(100%,1000px);margin:0 auto}.ndg-lead{font-size:19px;line-height:1.6;color:#3a4a5a;font-weight:300;margin:0 0 24px}.ndg-lead a{color:#2d799f;font-weight:600}.ndg-eyebrow{margin:0 0 14px;text-align:center;color:#2d799f;font-size:13px;font-weight:800;letter-spacing:.14em;text-transform:uppercase}.ndg-toc{list-style:none;counter-reset:s;display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:12px;margin:0 0 48px;padding:0}.ndg-toc li{counter-increment:s}.ndg-toc a{display:flex;align-items:center;gap:12px;height:100%;padding:13px 16px;background:#f2f2f2;border-radius:12px;color:#12354b!important;font-size:14.5px;font-weight:600;line-height:1.25;box-shadow:-8px -8px 16px rgba(255,255,255,.9),8px 8px 16px rgba(30,83,116,.10)}.ndg-toc a:hover{color:#2d799f!important}.ndg-toc a:before{content:counter(s);display:flex;align-items:center;justify-content:center;width:26px;height:26px;flex:0 0 26px;border-radius:50%;background:#56a1c8;color:#fff;font-size:13px;font-weight:800}.ndg-step{display:grid;grid-template-columns:64px 1fr;gap:26px;margin:0 0 34px;padding:34px 38px;background:#f2f2f2;border-radius:22px;box-shadow:-14px -14px 28px rgba(255,255,255,.92),14px 14px 28px rgba(30,83,116,.12);scroll-margin-top:90px}.ndg-step__no{display:flex;align-items:center;justify-content:center;width:64px;height:64px;border-radius:50%;background:linear-gradient(135deg,#56a1c8 0%,#245e80 100%);color:#fff;font-size:25px;font-weight:800;flex:0 0 64px}.ndg-step__body{min-width:0}.ndg-step__body .mjx-chtml.MJXc-display{overflow-x:auto;overflow-y:hidden;max-width:100%;padding:2px 0 8px}.ndg-step__body .mjx-chtml.MathJax_CHTML{font-size:18px!important}.ndg-step__body h2{margin:8px 0 16px;color:#001233;font-size:25px;font-weight:700;line-height:1.2}.ndg-step__body h3{margin:28px 0 12px;color:#12354b;font-size:19px;font-weight:600;scroll-margin-top:90px}.ndg-anchor-alias{display:block;height:0;position:relative;top:-90px;visibility:hidden}.ndg-step__body p{margin:0 0 16px;font-size:17px;line-height:1.62;color:#33424f}.ndg-step__body a{color:#2d799f;font-weight:500}.ndg-step__body ul{margin:0 0 16px;padding-left:22px}.ndg-step__body li{margin:7px 0;font-size:16px;line-height:1.55;color:#33424f}.ndg-step__body img:not([src$=\".svg\"]):not([data-src$=\".svg\"]){display:block!important;width:auto!important;max-width:min(580px,100%)!important;height:auto!important;margin:22px auto!important;border-radius:12px!important;box-shadow:0 12px 28px rgba(0,18,51,.12)!important}.ndg-step__body img[src$=\".svg\"],.ndg-step__body img[data-src$=\".svg\"]{display:block;margin:18px auto;max-width:min(520px,100%);height:auto}.ndg-nav{display:flex;justify-content:space-between;gap:16px;margin:12px 0 8px;flex-wrap:wrap}.ndg-nav a{display:inline-flex;align-items:center;padding:13px 24px;border-radius:10px;background:#f2f2f2;color:#2d799f!important;font-weight:800;font-size:15px;box-shadow:-8px -8px 16px rgba(255,255,255,.92),8px 8px 16px rgba(30,83,116,.10)}.ndg-nav a:hover{color:#1f5f80!important}@media(max-width:820px){.ndg-toc{grid-template-columns:1fr}.ndg-step{grid-template-columns:1fr;gap:16px;padding:26px 22px}.ndg-step__no{width:52px;height:52px;flex:0 0 52px;font-size:22px}.ndg-step__body h2{font-size:22px}}@media(max-width:640px){.ndg{padding:12px 14px}}<\/style><div class=\"ndg\"><div class=\"ndg-wrap\"><div class=\"ndg-lead\"><p data-start=\"79\" data-end=\"325\">In machine learning, neural networks are biologically inspired computational models composed of interconnected artificial neurons. In essence, this network architecture includes a set of parameters that are adjusted to perform specific tasks.<\/p>\n<p data-start=\"327\" data-end=\"590\">Importantly, neural networks have universal approximation properties, meaning they can approximate any function in any dimension up to a desired degree of accuracy. Because of this capability, they are widely used in many types of predictive applications.<\/p>\n<p data-start=\"592\" data-end=\"966\">Neural Designer provides dense, convolutional, pooling, embedding, multi-head attention, long short-term memory (LSTM), scaling, unscaling, and clamping layers. These components can be combined into architectures for approximation, classification, forecasting, auto-association, image classification, and text classification.<\/p><\/p>\n<\/div><div class=\"ndg-herofig\"><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/neural_network.svg\" width=\"51\" height=\"51\"  alt=\"Neural Network\"\/><\/div><p class=\"ndg-eyebrow\">Contents<\/p><ul class=\"ndg-toc\"><li><a href=\"#DenseLayer\">Dense layer<\/a><\/li><li><a href=\"#ConvolutionalLayer\">Convolutional layer<\/a><\/li><li><a href=\"#PoolingLayer\">Pooling layer<\/a><\/li><li><a href=\"#EmbeddingLayer\">Embedding layer<\/a><\/li><li><a href=\"#MultiHeadAttentionLayer\">Multi-head attention layer<\/a><\/li><li><a href=\"#LongShortTermMemoryLayer\">Long short-term memory (LSTM) layer<\/a><\/li><li><a href=\"#ScalingLayer\">Scaling layer<\/a><\/li><li><a href=\"#UnscalingLayer\">Unscaling layer<\/a><\/li><li><a href=\"#ClampingLayer\">Clamping layer<\/a><\/li><li><a href=\"#NetworkArchitecture\">Network architecture<\/a><\/li><li><a href=\"#ModelParameters\">Model parameters<\/a><\/li><li><a href=\"#ApproximationNeuralNetworks\">Approximation neural networks<\/a><\/li><li><a href=\"#ClassificationNeuralNetworks\">Classification neural networks<\/a><\/li><li><a href=\"#ForecastingNeuralNetworks\">Forecasting neural networks<\/a><\/li><li><a href=\"#AutoAssociativeNeuralNetworks\">Auto-associative neural networks<\/a><\/li><li><a href=\"#ConvolutionalNeuralNetworks\">Convolutional neural networks<\/a><\/li><li><a href=\"#AttentionNeuralNetworks\">Attention neural networks<\/a><\/li><\/ul><div class=\"ndg-step\" id=\"DenseLayer\"><div class=\"ndg-step__no\">1<\/div><div class=\"ndg-step__body\"><span id=\"PerceptronLayer\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><span id=\"PerceptronsLayers\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><span id=\"ProbabilisticLayer\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><h2>Dense layer<\/h2>\n<p>A dense layer connects every input to every output neuron. Each neuron combines the inputs with trainable weights and a bias, and then applies an activation function. Dense layers learn relationships in tabular, image, and text representations and also produce the final model outputs.<\/p>\n<p>For neuron <em>j<\/em>, the combination and activation can be written as:<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><mrow><msub><mi>z<\/mi><mi>j<\/mi><\/msub><mo>=<\/mo><msub><mi>b<\/mi><mi>j<\/mi><\/msub><mo>+<\/mo><munderover><mo>\u2211<\/mo><mrow><mi>i<\/mi><mo>=<\/mo><mn>1<\/mn><\/mrow><mi>n<\/mi><\/munderover><msub><mi>w<\/mi><mrow><mi>j<\/mi><mi>i<\/mi><\/mrow><\/msub><msub><mi>x<\/mi><mi>i<\/mi><\/msub><mo>,<\/mo><mspace width=\"1em\"\/><msub><mi>y<\/mi><mi>j<\/mi><\/msub><mo>=<\/mo><mi>f<\/mi><mo>(<\/mo><msub><mi>z<\/mi><mi>j<\/mi><\/msub><mo>)<\/mo><\/mrow><\/math><\/span><\/p>\n<p>Neural Designer allows the number of neurons, activation function, and dropout rate to be configured. The main dense activation functions are:<\/p>\n<ul>\n<li><a href=\"#LinearActivationFunction\">Linear or identity activation<\/a>.<\/li>\n<li><a href=\"#HyperbolicTangentActivationFunction\">Hyperbolic tangent activation<\/a>.<\/li>\n<li><a href=\"#RectifiedLinearActivationFunction\">Rectified linear (ReLU) activation<\/a>.<\/li>\n<li><a href=\"#SigmoidActivationFunction\">Sigmoid activation<\/a>.<\/li>\n<li><a href=\"#SoftmaxActivationFunction\">Softmax activation<\/a>.<\/li>\n<\/ul>\n<h3 id=\"LinearActivationFunction\">Linear activation function<\/h3>\n<p>The linear or identity activation returns the neuron combination unchanged. It is commonly used by output neurons in approximation and forecasting models.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><mrow><mi>f<\/mi><mo>(<\/mo><mi>z<\/mi><mo>)<\/mo><mo>=<\/mo><mi>z<\/mi><\/mrow><\/math><\/span><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/linear.webp\" alt=\"linear activation function\" \/><\/p>\n<h3 id=\"HyperbolicTangentActivationFunction\">Hyperbolic tangent activation function<\/h3>\n<p>The hyperbolic tangent is a sigmoid-shaped function with outputs between -1 and +1. It allows hidden layers to represent positive and negative activations.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><mrow><mi>f<\/mi><mo>(<\/mo><mi>z<\/mi><mo>)<\/mo><mo>=<\/mo><mi>tanh<\/mi><mo>(<\/mo><mi>z<\/mi><mo>)<\/mo><\/mrow><\/math><\/span><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/hyperbolic_tangent.webp\" alt=\"hyperbolic tangent activation function\" \/><\/p>\n<h3 id=\"RectifiedLinearActivationFunction\">Rectified linear (ReLU) activation function<\/h3>\n<p>The rectified linear activation returns zero for negative combinations and the combination itself for positive values. It is widely used in hidden dense and convolutional layers.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><mrow><mi>f<\/mi><mo>(<\/mo><mi>z<\/mi><mo>)<\/mo><mo>=<\/mo><mi>max<\/mi><mo>(<\/mo><mn>0<\/mn><mo>,<\/mo><mi>z<\/mi><mo>)<\/mo><\/mrow><\/math><\/span><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/rectified-linear-activation.webp\" alt=\"rectified linear activation function\" \/><\/p>\n<span id=\"LogisticActivationFunction\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><span id=\"CompetitiveProbabilisticActivation\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><span id=\"BinaryProbabilisticMethod\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><span id=\"continuousProbabilisticMethod\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><span id=\"CompetitiveProbabilisticMethod\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><h3 id=\"SigmoidActivationFunction\">Sigmoid activation function<\/h3>\n<p>The sigmoid, also called logistic, activation produces values between 0 and 1. A dense output with one sigmoid neuron represents the positive-class probability in binary classification.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><mrow><mi>f<\/mi><mo>(<\/mo><mi>z<\/mi><mo>)<\/mo><mo>=<\/mo><mfrac><mn>1<\/mn><mrow><mn>1<\/mn><mo>+<\/mo><msup><mi>e<\/mi><mrow><mo>\u2212<\/mo><mi>z<\/mi><\/mrow><\/msup><\/mrow><\/mfrac><\/mrow><\/math><\/span><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/logistic.webp\" alt=\"sigmoid activation function\" \/><\/p>\n<span id=\"SoftmaxProbabilisticActivation\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><span id=\"SoftmaxProbabilisticMethod\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><h3 id=\"SoftmaxActivationFunction\">Softmax activation function<\/h3>\n<p>Softmax transforms several dense outputs into class probabilities between 0 and 1 whose sum is 1. It is used for multiple classification.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><mrow><msub><mi>y<\/mi><mi>j<\/mi><\/msub><mo>=<\/mo><mfrac><msup><mi>e<\/mi><msub><mi>z<\/mi><mi>j<\/mi><\/msub><\/msup><mrow><munderover><mo>\u2211<\/mo><mrow><mi>i<\/mi><mo>=<\/mo><mn>1<\/mn><\/mrow><mi>n<\/mi><\/munderover><msup><mi>e<\/mi><msub><mi>z<\/mi><mi>i<\/mi><\/msub><\/msup><\/mrow><\/mfrac><\/mrow><\/math><\/span><\/p>\n<\/div><\/div>\n<div class=\"ndg-step\" id=\"ConvolutionalLayer\"><div class=\"ndg-step__no\">2<\/div><div class=\"ndg-step__body\"><h2>Convolutional layer<\/h2>\n<p>A convolutional layer applies trainable filters, or kernels, across an input image. Each filter detects local spatial patterns such as edges, textures, and shapes and produces an output feature map.<\/p>\n<p>The layer setup defines the number of filters, kernel size, stride, convolution type, and activation function. Neural Designer provides ReLU, linear, hyperbolic tangent, and sigmoid activations for convolutional layers.<\/p>\n<p>For each spatial dimension, the output size is:<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><mrow><mtext>output size<\/mtext><mo>=<\/mo><mo>\u230a<\/mo><mfrac><mrow><mtext>input size<\/mtext><mo>+<\/mo><mn>2<\/mn><mo>\u00b7<\/mo><mtext>padding<\/mtext><mo>\u2212<\/mo><mtext>kernel size<\/mtext><\/mrow><mtext>stride<\/mtext><\/mfrac><mo>\u230b<\/mo><mo>+<\/mo><mn>1<\/mn><\/mrow><\/math><\/span><\/p>\n<p>The two available convolution types are:<\/p>\n<ul><li><a href=\"#SameConvolution\">Same convolution<\/a>.<\/li><li><a href=\"#ValidConvolution\">Valid convolution<\/a>.<\/li><\/ul>\n<h3 id=\"SameConvolution\">Same convolution<\/h3>\n<p>Same convolution adds symmetric padding around the input. With stride 1, it preserves the spatial height and width. Odd kernel sizes such as 3, 5, or 7 keep the padding symmetric.<\/p>\n<h3 id=\"ValidConvolution\">Valid convolution<\/h3>\n<p>Valid convolution does not add padding. The filter is applied only where it completely overlaps the input, so the output feature map is smaller when the kernel is larger than 1.<\/p>\n<p>Convolutional layers are commonly followed by pooling layers. After the spatial features have been extracted, a flatten operation converts the feature maps into a vector for the dense layers.<\/p>\n<\/div><\/div>\n<div class=\"ndg-step\" id=\"PoolingLayer\"><div class=\"ndg-step__no\">3<\/div><div class=\"ndg-step__body\"><h2>Pooling layer<\/h2>\n<p>A pooling layer summarizes local regions without trainable parameters. For images, it reduces the height and width of feature maps while preserving their channels. The pool size and stride determine the amount of spatial reduction.<\/p>\n<p>The available pooling methods are:<\/p>\n<ul><li><a href=\"#MaxPooling\">Maximum pooling<\/a>.<\/li><li><a href=\"#AveragePooling\">Average pooling<\/a>.<\/li><\/ul>\n<h3 id=\"MaxPooling\">Maximum pooling<\/h3>\n<p>Maximum pooling returns the largest value in every pooling region. It preserves the strongest activation and is commonly used after convolutional layers.<\/p>\n<h3 id=\"AveragePooling\">Average pooling<\/h3>\n<p>Average pooling returns the mean value in every pooling region. It produces a smoother summary of the feature map.<\/p>\n<p>For text models, sequence pooling applies maximum or average pooling across the token dimension. This transforms a sequence of contextual embeddings into one fixed-size vector that can be passed to dense classification layers.<\/p>\n<\/div><\/div>\n<div class=\"ndg-step\" id=\"EmbeddingLayer\"><div class=\"ndg-step__no\">4<\/div><div class=\"ndg-step__body\"><h2>Embedding layer<\/h2>\n<p>An embedding layer transforms every token index into a trainable dense vector. Tokens that appear in similar contexts can therefore acquire nearby representations in the embedding space.<\/p>\n<p>The vocabulary size and sequence length come from the text data set, while the embedding dimension determines the size of each token vector. An input sequence with shape <em>sequence length<\/em> produces an output with shape <em>sequence length \u00d7 embedding dimension<\/em>.<\/p>\n<p>Neural Designer scales the token embeddings and adds positional information so that subsequent attention layers can distinguish the order of the tokens. The embedding vectors are learned together with the rest of the network.<\/p>\n<\/div><\/div>\n<div class=\"ndg-step\" id=\"MultiHeadAttentionLayer\"><div class=\"ndg-step__no\">5<\/div><div class=\"ndg-step__body\"><h2>Multi-head attention layer<\/h2>\n<p>A multi-head attention layer builds contextual token representations by allowing every sequence position to weigh information from the other positions. It projects the embeddings into query, key, and value vectors.<\/p>\n<p>Scaled dot-product attention is calculated as:<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><mrow><mtext>Attention<\/mtext><mo>(<\/mo><mi>Q<\/mi><mo>,<\/mo><mi>K<\/mi><mo>,<\/mo><mi>V<\/mi><mo>)<\/mo><mo>=<\/mo><mtext>softmax<\/mtext><mo>(<\/mo><mfrac><mrow><mi>Q<\/mi><msup><mi>K<\/mi><mi>T<\/mi><\/msup><\/mrow><msqrt><msub><mi>d<\/mi><mi>k<\/mi><\/msub><\/msqrt><\/mfrac><mo>)<\/mo><mi>V<\/mi><\/mrow><\/math><\/span><\/p>\n<p>Several attention heads perform this operation in parallel and learn different relationships between tokens. Their outputs are concatenated and projected back to the embedding dimension. Consequently, the layer preserves the sequence length and embedding dimension.<\/p>\n<p>The number of heads is configurable and must divide the embedding dimension exactly. Multiple multi-head attention layers can be stacked before sequence pooling and dense classification layers.<\/p>\n<\/div><\/div>\n<div class=\"ndg-step\" id=\"LongShortTermMemoryLayer\"><div class=\"ndg-step__no\">6<\/div><div class=\"ndg-step__body\"><h2>Long-short-term memory (LSTM) layer<\/h2><p>Long-short-term memory (LSTM) layers are a particular recurrent layer widely used in forecasting applications.<\/p>\n<p>The following figure shows an LSTM layer. It receives information as a set of numerical inputs. This information is processed through forget, input, state, and output gates and stored in hidden and cell states. Finally, the layer produces the final outputs.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/lstm_layer_activity_diagram.webp\" alt=\"lstm layer\" \/><\/p>\n<p>As we can see, long-short-term memory (LSTM) layers are complex and contain many parameters. That structure makes them suitable for learning dependencies from time-series data.<\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"ScalingLayer\"><div class=\"ndg-step__no\">7<\/div><div class=\"ndg-step__body\"><h2>Scaling layer<\/h2><p>In practice, scaling the inputs to give them a proper range is always convenient. In the context of neural networks, the scaling layer performs this process.<\/p>\n<p>The scaling layer contains some basic statistics on the inputs. They include the mean, standard deviation, minimum, and maximum values.<\/p>\n<p>Some scaling methods used in practice are the following:<\/p>\n<ul>\n<li><a href=\"#MinimumMaximumScalingMethod\">Minimum and maximum scaling method<\/a>.<\/li>\n<li><a href=\"#MeanStandardDeviationScalingMethod\">Mean and standard deviation scaling method<\/a>.<\/li>\n<li><a href=\"#StandardDeviationScalingMethod\">Standard deviation scaling method<\/a>.<\/li>\n<\/ul>\n<h3 id=\"MinimumMaximumScalingMethod\">Minimum and maximum scaling method<\/h3>\n<p>The minimum and maximum methods produce a data set scaled between \u22121 and 1. This method is usually applied to variables with a uniform distribution.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">i<\/mi><\/msub><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x0003D;<\/mo><mo>&#x02212;<\/mo><mn>1<\/mn><mo>&#x0002B;<\/mo><mfrac><mrow><mo stretchy=\"false\">&#x00028;<\/mo><mrow><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x02212;<\/mo><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">m<\/mi><\/mrow><mo stretchy=\"false\">&#x00029;<\/mo><mo>&#x000B7;<\/mo><mo stretchy=\"false\">&#x00028;<\/mo><mn>1<\/mn><mo>&#x02212;<\/mo><mo stretchy=\"false\">&#x00028;<\/mo><mo>&#x02212;<\/mo><mn>1<\/mn><mo stretchy=\"false\">&#x00029;<\/mo><mo stretchy=\"false\">&#x00029;<\/mo><\/mrow><mrow><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">x<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">m<\/mi><\/mrow><mo>&#x02212;<\/mo><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">m<\/mi><\/mrow><\/mrow><\/mfrac><\/mrow><annotation encoding=\"application\/x-tex\">\\mathrm{scaled_input} = -1 + \\frac{(\\mathrm{input} &#8211; \\mathrm{minimum}) \\cdot (1 &#8211; (-1))}{\\mathrm{maximum} &#8211; \\mathrm{minimum}}<\/annotation><\/semantics><\/math><\/span><\/p>\n<div>\u00a0<\/div>\n<h3 id=\"MeanStandardDeviationScalingMethod\">Mean and standard deviation scaling method<\/h3>\n<p>The mean and standard deviation method scales the inputs to have a mean of 0 and a standard deviation of 1. This method usually applies to normal (or Gaussian) distribution variables.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">i<\/mi><\/msub><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x0003D;<\/mo><mfrac><mrow><mrow><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><\/mrow><mrow><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">r<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">d<\/mi><\/msub><mi mathvariant=\"normal\">e<\/mi><mi mathvariant=\"normal\">v<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">o<\/mi><mi mathvariant=\"normal\">n<\/mi><\/mrow><\/mrow><\/mfrac><\/mrow><annotation encoding=\"application\/x-tex\">\\mathrm{scaled_input} = \\frac{\\mathrm{input}}{\\mathrm{standard_deviation}}<\/annotation><\/semantics><\/math><\/span><\/p>\n<div>\u00a0<\/div>\n<h3 id=\"StandardDeviationScalingMethod\">Standard deviation scaling method<\/h3>\n<p>The standard deviation scaling method produces inputs with standard deviation 1. This is typically applied to half-normal distributions, variables centered at zero with only positive values.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">i<\/mi><\/msub><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x0003D;<\/mo><mfrac><mrow><mrow><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><\/mrow><mrow><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">r<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">d<\/mi><\/msub><mi mathvariant=\"normal\">e<\/mi><mi mathvariant=\"normal\">v<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">o<\/mi><mi mathvariant=\"normal\">n<\/mi><\/mrow><\/mrow><\/mfrac><\/mrow><annotation encoding=\"application\/x-tex\">\\mathrm{scaled_input} = \\frac{\\mathrm{input}}{\\mathrm{standard_deviation}}<\/annotation><\/semantics><\/math><\/span><\/p>\n<div>\u00a0<\/div>\n<p>All scaling methods are linear and, in general, produce similar results. In all cases, synchronizing the scaling of the inputs in the dataset with the scaling of the inputs in the neural network is necessary. Neural Designer does that without any intervention by the user.<\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"UnscalingLayer\"><div class=\"ndg-step__no\">8<\/div><div class=\"ndg-step__body\"><h2>Unscaling layer<\/h2><p>The scaled outputs from a neural network are unscaled to produce the original units. In the context of neural networks, the unscaling layer does this.<\/p>\n<p>An unscaling layer contains some basic statistics on the outputs. They include the mean, standard deviation, minimum, and maximum values.<\/p>\n<p>Four unscaling methods are utilized in practice:<\/p>\n<ul>\n<li><a href=\"#MinimumMaximumUnscalingMethod\">Minimum and maximum unscaling method<\/a>.<\/li>\n<li><a href=\"#MeanStandardDeviationUnscalingMethod\">Mean and standard deviation unscaling method<\/a>.<\/li>\n<li><a href=\"#StandardDeviationUnscalingMethod\">Standard deviation unscaling method<\/a>.<\/li>\n<li><a href=\"#LogarithmicUnscalingMethod\">Logarithmic unscaling method<\/a>.<\/li>\n<\/ul>\n<h3 id=\"MinimumMaximumUnscalingMethod\">Minimum and maximum unscaling method<\/h3>\n<p>The minimum and maximum method unscales variables that have been previously scaled to have minimum -1 and maximum +1, to produce outputs in the original range,<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mrow><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">o<\/mi><\/msub><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x0003D;<\/mo><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">m<\/mi><\/mrow><mo>&#x0002B;<\/mo><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">o<\/mi><\/msub><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x000B7;<\/mo><mo stretchy=\"false\">&#x00028;<\/mo><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">x<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">m<\/mi><\/mrow><mo>&#x02212;<\/mo><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">m<\/mi><\/mrow><mo stretchy=\"false\">&#x00029;<\/mo><\/mrow><annotation encoding=\"application\/x-tex\">\\mathrm{unscaled_output} = \\mathrm{minimum} + \\mathrm{scaled_output} \\cdot (\\mathrm{maximum}-\\mathrm{minimum})<\/annotation><\/semantics><\/math><\/span><\/p>\n<div>\u00a0<\/div>\n<h3 id=\"MeanStandardDeviationUnscalingMethod\">Mean and standard deviation unscaling method<\/h3>\n<p>The mean and standard deviation method unscales variables that have been previously scaled to have mean 0 and standard deviation 1,<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mrow><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">o<\/mi><\/msub><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x0003D;<\/mo><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">e<\/mi><mi mathvariant=\"normal\">a<\/mi><msub><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">s<\/mi><\/msub><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">o<\/mi><\/msub><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x000B7;<\/mo><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">r<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">d<\/mi><\/msub><mi mathvariant=\"normal\">e<\/mi><mi mathvariant=\"normal\">v<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">o<\/mi><mi mathvariant=\"normal\">n<\/mi><\/mrow><\/mrow><annotation encoding=\"application\/x-tex\">\\mathrm{unscaled_output} = \\mathrm{mean_scaled_output} \\cdot \\mathrm{standard_deviation}<\/annotation><\/semantics><\/math><\/span><\/p>\n<h3 id=\"StandardDeviationUnscalingMethod\">Standard deviation unscaling method<\/h3>\n<p>The standard deviation method unscales variables that have been previously scaled to have a standard deviation of 1, to produce outputs in the original range,<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mrow><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">o<\/mi><\/msub><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x0003D;<\/mo><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">o<\/mi><\/msub><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x000B7;<\/mo><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">r<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">d<\/mi><\/msub><mi mathvariant=\"normal\">e<\/mi><mi mathvariant=\"normal\">v<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">o<\/mi><mi mathvariant=\"normal\">n<\/mi><\/mrow><\/mrow><annotation encoding=\"application\/x-tex\">\\mathrm{unscaled_output} = \\mathrm{scaled_output} \\cdot \\mathrm{standard_deviation}<\/annotation><\/semantics><\/math><\/span><\/p>\n<div>\u00a0<\/div>\n<h3 id=\"LogarithmicUnscalingMethod\">Logarithmic unscaling method<\/h3>\n<p>The logarithmic method unscales variables that have undergone a logarithmic transformation previously.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mrow><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">o<\/mi><\/msub><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><mo>&#x0003D;<\/mo><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">m<\/mi><\/mrow><mo>&#x0002B;<\/mo><mn>0.5<\/mn><mo>&#x000B7;<\/mo><mrow><mo stretchy=\"true\" fence=\"true\" form=\"prefix\">&#x00028;<\/mo><msup><mi>e<\/mi><mrow><mrow><mi mathvariant=\"normal\">s<\/mi><mi mathvariant=\"normal\">c<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">l<\/mi><mi mathvariant=\"normal\">e<\/mi><msub><mi mathvariant=\"normal\">d<\/mi><mi mathvariant=\"normal\">o<\/mi><\/msub><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><mi mathvariant=\"normal\">p<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">t<\/mi><\/mrow><\/mrow><\/msup><mo>&#x0002B;<\/mo><mn>1<\/mn><mo stretchy=\"true\" fence=\"true\" form=\"postfix\">&#x00029;<\/mo><\/mrow><mo>&#x000B7;<\/mo><mo stretchy=\"false\">&#x00028;<\/mo><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">a<\/mi><mi mathvariant=\"normal\">x<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">m<\/mi><\/mrow><mo>&#x02212;<\/mo><mrow><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">n<\/mi><mi mathvariant=\"normal\">i<\/mi><mi mathvariant=\"normal\">m<\/mi><mi mathvariant=\"normal\">u<\/mi><mi mathvariant=\"normal\">m<\/mi><\/mrow><mo stretchy=\"false\">&#x00029;<\/mo><\/mrow><annotation encoding=\"application\/x-tex\">\\mathrm{unscaled_output} = \\mathrm{minimum} + 0.5 \\cdot \\left( e^{\\mathrm{scaled_output}} + 1 \\right) \\cdot (\\mathrm{maximum} &#8211; \\mathrm{minimum})<\/annotation><\/semantics><\/math><\/span><\/p>\n<div>\u00a0<\/div>\n<p>In all cases, synchronizing the scaling of the targets in the dataset with unscaling the outputs in the neural network is necessary. Neural Designer does that without any intervention by the user.<\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"ClampingLayer\"><div class=\"ndg-step__no\">9<\/div><div class=\"ndg-step__body\"><span id=\"BoundingLayer\" class=\"ndg-anchor-alias\" aria-hidden=\"true\"><\/span><h2>Clamping layer<\/h2>\n<p>Some model outputs must remain within a predefined range. For example, a product rating might range from 1 to 5 stars.<\/p>\n<p>The clamping layer applies an individual lower and upper limit to every output. Values inside the interval remain unchanged, while values outside it are set to the closest limit. This final transformation contains no trainable parameters.<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><mrow><mtext>clamped output<\/mtext><mo>=<\/mo><mi>min<\/mi><mo>(<\/mo><mi>max<\/mi><mo>(<\/mo><mtext>output<\/mtext><mo>,<\/mo><mtext>lower limit<\/mtext><mo>)<\/mo><mo>,<\/mo><mtext>upper limit<\/mtext><mo>)<\/mo><\/mrow><\/math><\/span><\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"NetworkArchitecture\"><div class=\"ndg-step__no\">10<\/div><div class=\"ndg-step__body\"><h2>Network architecture<\/h2><p>A neural network can be symbolized as a graph, where nodes represent neurons, and edges represent connectivities among neurons. An edge label represents the parameter of the neuron for which the flow goes in.<\/p>\n<p>Most neural networks, even biological neural networks, exhibit a layered structure. Therefore, layers are the basis for determining the architecture of a neural network.<\/p>\n<p>We build a neural network by organizing layers of neurons in a network architecture. The characteristic network architecture in this case is known as the feed-forward architecture. In a feed-forward neural network, we group layers into a sequence so that neurons in any layer connect only to neurons in the next layer.<\/p>\n<p>The following figure represents a neural network with four inputs, several layers of different types, and three outputs.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/deep_neural_network_big.webp\" alt=\"Neural network graph\" width=\"450\" \/><\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"ModelParameters\"><div class=\"ndg-step__no\">11<\/div><div class=\"ndg-step__body\"><h2>Model parameters<\/h2><p>The model parameters involve the parameters of each layer in the network architecture.<\/p>\n<p>You can group all these parameters into a vector (theta), which you can write as:<\/p>\n<p><span class=\"nd-math-block\"><math xmlns=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"block\"><semantics><mrow><mi>&#x003B8;<\/mi><mo>&#x0003D;<\/mo><mo stretchy=\"false\">&#x00028;<\/mo><msub><mi>&#x003B8;<\/mi><mn>1<\/mn><\/msub><mo>&#x0002C;<\/mo><mi>&#x02026;<\/mi><mo>&#x0002C;<\/mo><msub><mi>&#x003B8;<\/mi><mi>d<\/mi><\/msub><mo stretchy=\"false\">&#x00029;<\/mo><\/mrow><annotation encoding=\"application\/x-tex\">\\theta = (\\theta_1, \\ldots, \\theta_d)<\/annotation><\/semantics><\/math><\/span><\/p>\n<div>\u00a0<\/div>\n<p>The number of adaptable parameters, (d), is the sum of parameters in each layer.<\/p>\n<p>As we have seen, a neural network may consist of various types of layers, depending on the requirements of the predictive model.<\/p>\n<p>Next, we describe each application type&#8217;s most common neural network configurations.<\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"ApproximationNeuralNetworks\"><div class=\"ndg-step__no\">12<\/div><div class=\"ndg-step__body\"><h2>Approximation neural networks<\/h2>\n<p>An <a href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-networks-applications\/#Approximation\">approximation<\/a> model usually contains a <a href=\"#ScalingLayer\">scaling layer<\/a>, one or more <a href=\"#DenseLayer\">dense layers<\/a>, an <a href=\"#UnscalingLayer\">unscaling layer<\/a>, and an optional <a href=\"#ClampingLayer\">clamping layer<\/a>.<\/p>\n<p>Two dense layers are sufficient for many data sets. More complex relationships can require deeper architectures with additional hidden dense layers.<\/p>\n<p>The following figure represents a neural network that <a href=\"https:\/\/www.neuraldesigner.com\/learning\/examples\/combined-cycle-power-plant\/\">estimates the power generated by a combined cycle power plant<\/a> from meteorological and plant variables.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/combined-cycle-power-plant-initial-neural-network.webp\" alt=\"Combined Cycle Power Plant Initial Neural Network\" \/><\/p>\n<p>The network has four inputs and one output. It contains a scaling layer, a hidden dense layer with four neurons, an output dense layer with one neuron, and an unscaling layer.<\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"ClassificationNeuralNetworks\"><div class=\"ndg-step__no\">13<\/div><div class=\"ndg-step__body\"><h2>Classification neural networks<\/h2>\n<p>A <a href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-networks-applications\/#Classification\">classification<\/a> model usually contains a <a href=\"#ScalingLayer\">scaling layer<\/a> and one or more <a href=\"#DenseLayer\">dense layers<\/a>. The final dense layer uses sigmoid activation for binary classification or softmax activation for multiple classification.<\/p>\n<p>Two dense layers are sufficient for many classification data sets, although deeper architectures can learn more complex decision boundaries.<\/p>\n<p>The following figure represents a binary classification model for <a href=\"https:\/\/www.neuraldesigner.com\/learning\/examples\/breast-cancer-diagnosis\/\">diagnosing breast cancer from fine-needle aspirates<\/a>.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/breast-cancer-neural-network.webp\" alt=\"Breast Cancer Neural Network\" \/><\/p>\n<p>The network has nine inputs and one output. It contains a scaling layer, a hidden dense layer with three neurons, and an output dense layer with one sigmoid neuron.<\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"ForecastingNeuralNetworks\"><div class=\"ndg-step__no\">14<\/div><div class=\"ndg-step__body\"><h2>Forecasting neural networks<\/h2>\n<p>A <a href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-networks-applications\/#Forecasting\">forecasting<\/a> model can contain a <a href=\"#ScalingLayer\">scaling layer<\/a>, a <a href=\"#LongShortTermMemoryLayer\">long short-term memory layer<\/a>, one or more <a href=\"#DenseLayer\">dense layers<\/a>, an <a href=\"#UnscalingLayer\">unscaling layer<\/a>, and an optional <a href=\"#ClampingLayer\">clamping layer<\/a>.<\/p>\n<p>The following figure represents a one-day-ahead forecasting model for urban NO2 levels.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/www.neuraldesigner.com\/images\/air-quality-neural-network.webp\" alt=\"Air Quality Neural Network\" \/><\/p>\n<p>The network has 14 inputs and one output. It contains a scaling layer, an LSTM layer, an output dense layer, an unscaling layer, and a clamping layer.<\/p>\n<p>The inputs contain lagged variables, and the outputs contain the variables for the required forecasting horizon.<\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"AutoAssociativeNeuralNetworks\"><div class=\"ndg-step__no\">15<\/div><div class=\"ndg-step__body\"><h2>Auto-associative neural networks<\/h2>\n<p>An <a href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-networks-applications\/#AutoAssociation\">auto-associative<\/a> neural network learns to reproduce its inputs at the outputs. It normally contains a scaling layer, an encoder made of dense layers, a low-dimensional bottleneck, a symmetric decoder, and an unscaling layer.<\/p>\n<p>The encoder compresses the most relevant information into the bottleneck representation. The decoder reconstructs the original variables from that representation.<\/p>\n<p>Reconstruction errors and distances in the bottleneck space can be used to identify unusual samples and detect anomalies.<\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"ConvolutionalNeuralNetworks\"><div class=\"ndg-step__no\">16<\/div><div class=\"ndg-step__body\"><h2>Convolutional neural networks<\/h2>\n<p>A convolutional neural network for <a href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-networks-applications\/#ImageClassification\">image classification<\/a> starts with image scaling and then stacks <a href=\"#ConvolutionalLayer\">convolutional layers<\/a> and <a href=\"#PoolingLayer\">pooling layers<\/a>.<\/p>\n<p>The convolutional blocks learn increasingly complex spatial features. A flatten operation then converts the final feature maps into a vector, and one or more <a href=\"#DenseLayer\">dense layers<\/a> produce the class probabilities.<\/p>\n<p>The output dense layer uses sigmoid activation for binary image classification and softmax activation for multiple image classification.<\/p>\n<\/div><\/div><div class=\"ndg-step\" id=\"AttentionNeuralNetworks\"><div class=\"ndg-step__no\">17<\/div><div class=\"ndg-step__body\"><h2>Attention neural networks<\/h2>\n<p>An attention neural network for <a href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-networks-applications\/#TextClassification\">text classification<\/a> begins with an <a href=\"#EmbeddingLayer\">embedding layer<\/a> that transforms token indices into dense vectors.<\/p>\n<p>One or more <a href=\"#MultiHeadAttentionLayer\">multi-head attention layers<\/a> add context to every token representation. A <a href=\"#PoolingLayer\">sequence pooling operation<\/a> summarizes the sequence, and one or more <a href=\"#DenseLayer\">dense layers<\/a> produce the class probabilities.<\/p>\n<p>This architecture learns which words and relationships are most relevant to each text class while supporting variable linguistic contexts.<\/p>\n<p><a style=\"float: left;\" href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/data-set\/\">\u21d0 Data Set<\/a><br \/><a style=\"float: right;\" href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/training-strategy\/\">Training Strategy \u21d2<\/a><\/p>\n<\/div><\/div><\/div><\/div>","protected":false},"author":122,"featured_media":2701,"template":"","categories":[30],"tags":[],"class_list":["post-3534","learning","type-learning","status-publish","has-post-thumbnail","hentry","category-tutorials"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Machine learning tutorial: Neural networks<\/title>\n<meta name=\"description\" content=\"This tutorial shows the most used types of layers within neural networks and how they are assembled to perform Machine Learning tasks.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Neural networks tutorial: Neural network\" \/>\n<meta property=\"og:description\" content=\"In this tutorial we describe the most used types of layers within neural networks and how they are assembled to perform Machine Learning tasks.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/\" \/>\n<meta property=\"og:site_name\" content=\"Neural Designer\" \/>\n<meta property=\"article:modified_time\" content=\"2026-08-26T08:34:23+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/cropped-favicon.png\" \/>\n\t<meta property=\"og:image:width\" content=\"512\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Neural networks tutorial: Neural network\" \/>\n<meta name=\"twitter:description\" content=\"In this tutorial we describe the most used types of layers within neural networks and how they are assembled to perform Machine Learning tasks.\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/cropped-favicon.png\" \/>\n<meta name=\"twitter:site\" content=\"@NeuralDesigner\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/\",\"url\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/\",\"name\":\"Machine learning tutorial: Neural networks\",\"isPartOf\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/activity-diagram-neural-network.svg\",\"datePublished\":\"2025-11-24T10:12:58+00:00\",\"dateModified\":\"2026-08-26T08:34:23+00:00\",\"description\":\"This tutorial shows the most used types of layers within neural networks and how they are assembled to perform Machine Learning tasks.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#primaryimage\",\"url\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/activity-diagram-neural-network.svg\",\"contentUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/activity-diagram-neural-network.svg\",\"width\":1200,\"height\":627,\"caption\":\"Neural network activity diagram\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.neuraldesigner.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Learning\",\"item\":\"https:\/\/www.neuraldesigner.com\/learning\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Machine learning tutorial: Neural networks\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#website\",\"url\":\"https:\/\/www.neuraldesigner.com\/\",\"name\":\"Neural Designer\",\"description\":\"Explainable AI Platform\",\"publisher\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.neuraldesigner.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#organization\",\"name\":\"Neural Designer\",\"url\":\"https:\/\/www.neuraldesigner.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png\",\"contentUrl\":\"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png\",\"width\":1024,\"height\":223,\"caption\":\"Neural Designer\"},\"image\":{\"@id\":\"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/NeuralDesigner\",\"https:\/\/es.linkedin.com\/showcase\/neuraldesigner\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Machine learning tutorial: Neural networks","description":"This tutorial shows the most used types of layers within neural networks and how they are assembled to perform Machine Learning tasks.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/","og_locale":"en_US","og_type":"article","og_title":"Neural networks tutorial: Neural network","og_description":"In this tutorial we describe the most used types of layers within neural networks and how they are assembled to perform Machine Learning tasks.","og_url":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/","og_site_name":"Neural Designer","article_modified_time":"2026-08-26T08:34:23+00:00","og_image":[{"width":512,"height":512,"url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/cropped-favicon.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_title":"Neural networks tutorial: Neural network","twitter_description":"In this tutorial we describe the most used types of layers within neural networks and how they are assembled to perform Machine Learning tasks.","twitter_image":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/cropped-favicon.png","twitter_site":"@NeuralDesigner","twitter_misc":{"Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/","url":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/","name":"Machine learning tutorial: Neural networks","isPartOf":{"@id":"https:\/\/www.neuraldesigner.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#primaryimage"},"image":{"@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#primaryimage"},"thumbnailUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/activity-diagram-neural-network.svg","datePublished":"2025-11-24T10:12:58+00:00","dateModified":"2026-08-26T08:34:23+00:00","description":"This tutorial shows the most used types of layers within neural networks and how they are assembled to perform Machine Learning tasks.","breadcrumb":{"@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#primaryimage","url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/activity-diagram-neural-network.svg","contentUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/06\/activity-diagram-neural-network.svg","width":1200,"height":627,"caption":"Neural network activity diagram"},{"@type":"BreadcrumbList","@id":"https:\/\/www.neuraldesigner.com\/learning\/tutorials\/neural-network\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.neuraldesigner.com\/"},{"@type":"ListItem","position":2,"name":"Learning","item":"https:\/\/www.neuraldesigner.com\/learning\/"},{"@type":"ListItem","position":3,"name":"Machine learning tutorial: Neural networks"}]},{"@type":"WebSite","@id":"https:\/\/www.neuraldesigner.com\/#website","url":"https:\/\/www.neuraldesigner.com\/","name":"Neural Designer","description":"Explainable AI Platform","publisher":{"@id":"https:\/\/www.neuraldesigner.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.neuraldesigner.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.neuraldesigner.com\/#organization","name":"Neural Designer","url":"https:\/\/www.neuraldesigner.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png","contentUrl":"https:\/\/www.neuraldesigner.com\/wp-content\/uploads\/2023\/05\/logo-neural-1.png","width":1024,"height":223,"caption":"Neural Designer"},"image":{"@id":"https:\/\/www.neuraldesigner.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/NeuralDesigner","https:\/\/es.linkedin.com\/showcase\/neuraldesigner\/"]}]}},"_links":{"self":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning\/3534","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning"}],"about":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/types\/learning"}],"author":[{"embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/users\/122"}],"version-history":[{"count":14,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning\/3534\/revisions"}],"predecessor-version":[{"id":23721,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/learning\/3534\/revisions\/23721"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/media\/2701"}],"wp:attachment":[{"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/media?parent=3534"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/categories?post=3534"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.neuraldesigner.com\/api\/wp\/v2\/tags?post=3534"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}