Contents<\/span><\/h3>\n<\/section>\n\n- Application type<\/a>.<\/li>\n
- Data set<\/a>.<\/li>\n
- Neural network<\/a>.<\/li>\n
- Training strategy<\/a>.<\/li>\n
- Model selection<\/a>.<\/li>\n
- Testing analysis<\/a>.<\/li>\n
- Model deployment<\/a>.<\/li>\n<\/ol>\n<\/section>\n
\n1. Application type<\/h2>\n
This is a classification<\/a> project. Indeed, the variable to be predicted is categorical (low, medium, and high).<\/p>\nThe objective is to model the quality of the milk by knowing its characteristics and thus be able to make future predictions.<\/p>\n<\/section>\n\n2. Data set<\/h2>\n
The first step is to prepare the dataset,<\/a> which is the source of information for the classification model.<\/p>\nFor that, we need to configure the following concepts:<\/p>\n
\n- Data source.<\/li>\n
- Variables.<\/li>\n
- Instances.<\/li>\n<\/ul>\n
Data source<\/h3>\n
The data source<\/a> for this example is the CSV file milk.csv<\/a>.<\/p>\nThe number of columns is 8, and the number of rows is 1060.<\/p>\n
Variables<\/h3>\n
The variables<\/a> are:<\/p>\nNumerical variables<\/h4>\n\n- pH<\/strong> \u2013 Milk pH, ranging from 3 to 9.5.<\/li>\n
- temperature<\/strong> \u2013 Milk temperature, ranging from 34 \u00b0C to 90 \u00b0C.<\/li>\n
- colour<\/strong> \u2013 Milk color, with values between 240 and 255.<\/li>\n<\/ul>\n
Binary variables<\/h4>\n\n- taste<\/strong> \u2013 1 = good, 0 = bad.<\/li>\n
- odor<\/strong> \u2013 1 = good, 0 = bad.<\/li>\n
- fat<\/strong> \u2013 1 = good, 0 = bad.<\/li>\n
- turbidity<\/strong> \u2013 1 = good, 0 = bad.<\/li>\n<\/ul>\n
Target variable<\/h4>\n<\/section>\n\n- \n
grade<\/strong> \u2013 Milk quality class: low_quality<\/em>, medium_quality<\/em>, or high_quality<\/em>.<\/p>\n<\/li>\n<\/ul>\nAll variables in the study are inputs, except “grade”, which is the output that we want to extract from this machine learning study.<\/p>\n
Note that “grade” is categorical and can take the values low_quality, medium_quality, and high_quality.<\/p>\n\nInstances<\/h2>\n
The instances<\/a> are randomly split into 60.2% training (637 samples), 19.9% validation (211 samples), and 19.9% testing (211 samples).<\/p>\nDistributions<\/h3>\n
The following figure is the pie chart for the variable milk quality class, and it shows its distribution<\/a>.<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/milk_pie_chart.webp\")
<\/p>\n
The milk dataset has 429 low-quality, 374 medium-quality, and 256 high-quality instances.<\/p>\n
This shows an imbalanced target, with low quality making up 40.5% of the samples and high quality only 24.2%.<\/p>\n<\/section>\n\n3. Neural network<\/h2>\n
The second step is to choose a neural network<\/a>.<\/p>\nThe neural network must have seven inputs since the data set has seven input variables.<\/p>\n
The neural network has three outputs because we have three different “grades”: low_quality, medium_quality, and high_quality.<\/p>\n
In classification problems, it typically consists of:<\/p>\n
\n- A scaling layer.<\/li>\n
- Two dense layers.<\/li>\n<\/ul>\n
Scaling layer<\/h3>\n
The scaling layer<\/a> normalizes the input values. As our inputs have different distributions, they are scaled with different methods:<\/p>\n\n- The variables pH, temperature, and colour are scaled with the mean and standard deviation scaling method<\/a>.<\/li>\n
- The variables taste, odor, fat, and turbidity are scaled with the minimum and maximum scaling method<\/a>.<\/li>\n<\/ul>\n
Dense layers<\/h3>\n
In this case, as the first guest, we only use one\u00a0perceptron layer<\/a>.\u00a0This layer contains seven inputs, three neurons, and three outputs. For this example, the perceptron layer<\/a> is a hyperbolic tangent activation function<\/a>.<\/p>\nThe probabilistic layer<\/a> allows for interpreting the outputs as probabilities. In this regard, all outputs are between 0 and 1, and their sum is 1. The softmax probabilistic activation<\/a> is used here.<\/p>\nThe following figure is a graphical representation of this classification neural network<\/a>:<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/milk_network_no_training.webp\")
<\/p>\n<\/section>\n\n4. Training strategy<\/h2>\n
The next step is to set the training strategy<\/a>, which comprises:<\/p>\n\n- Loss index.<\/li>\n
- Optimization algorithm.<\/li>\n<\/ul>\n
Loss index<\/h3>\n
The loss index<\/a> chosen for this application is the\u00a0normalized squared error<\/a>\u00a0with L2 regularization<\/a>.<\/p>\nThe error term<\/a> fits the neural network to the training instances<\/a> of the data set.<\/p>\nThe regularization term<\/a> makes the model more stable and improves generalization.<\/p>\nOptimization algorithm<\/h3>\n
The optimization algorithm<\/a> searches for the neural network parameters that minimize the loss index.<\/p>\nThe quasi-Newton method<\/a>\u00a0is chosen here.<\/p>\nThe following chart shows how the training (blue) and selection (orange) errors decrease with the epochs during training.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/milk_error.webp\")
<\/p>\n
The final values are training error = 0.252 NSE<\/b> (blue) and selection error = 0.277 NSE<\/b> (orange).<\/p>\n<\/section>\n\n5. Model selection<\/h2>\n
It is essential to have low selection error in our model, allowing us to generalize well to the new data rather than simply memorizing the training data.<\/p>\n
The objective of model selection<\/a> is to find the network architecture with the best generalization properties. That is, the method that minimizes the error on the selected instances<\/a> of the data set.<\/p>\nWe\u00a0aim to develop a neural network with a selection error of less than\u00a00.277 NSE<\/strong>, the current best value we have achieved<\/span>.<\/p>\nNeuron selection<\/h3>\n
Order selection<\/a> algorithms train several network architectures with a different number of neurons and select the one with the smallest selection error.<\/p>\nThe incremental order<\/a> method starts with a small number of neurons and increases the complexity at each iteration. The following chart shows the training error (blue) and the selection error (yellow) as a function of the number of neurons.<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/milk_selection_neurons.webp\")
<\/p>\n
As we can see, the number of neurons that yield the minimum error is four. Therefore, we select the neural network with four neurons in the hidden layer.<\/p>\n
The following chart shows the new neural network architecture.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/milk_network.webp\")
<\/p>\n
The following chart shows how the training and selection errors decrease with the epochs during training in the new neural network. The final values are training error = 0.0877 NSE<\/b> (blue) and selection error = 0.107 NSE<\/b> (orange).<\/p>\n
With the new architecture of the neural network, we achieve around 50% less selection error.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/milk_errors.webp\")
<\/p>\n<\/section>\n\n6. Testing analysis<\/h2>\n
The purpose of the testing analysis<\/a> is to validate the generalization performance of the model.<\/p>\nHere, we compare the neural network outputs to the corresponding targets in the testing instances<\/a> of the data set.<\/p>\nConfusion matrix<\/h3>\n
\n\n\n<\/th>\n Predicted low_quality<\/th>\n Predicted medium_quality<\/th>\n Predicted high_quality<\/th>\n Total<\/th>\n<\/tr>\n \nReal low_quality<\/th>\n 49 (23.2%)<\/td>\n 0<\/td>\n 4 (1.9%)<\/td>\n 53 (23.7%)<\/td>\n<\/tr>\n \nReal medium_quality<\/th>\n 1 (0.5%)<\/td>\n 92 (43.6%)<\/td>\n 0<\/td>\n 93 (43.6%)<\/td>\n<\/tr>\n \nReal high_quality<\/th>\n 0<\/td>\n 0<\/td>\n 65 (30.8%)<\/td>\n 65 (32.7%)<\/td>\n<\/tr>\n \nTotal<\/th>\n 50 (23.7%)<\/td>\n 92 (43.6%)<\/td>\n 69 (32.7%)<\/td>\n 211 (100%)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nThe number of correctly classified samples<\/b> is 206<\/b>, and the number of misclassified samples<\/b> is 5<\/b>.<\/p>\nMultiple classification metrics<\/h3>\n
The confusion matrix allows us to calculate the model’s accuracy and error:<\/p>\n
\n- Classification accuracy: 97.6%<\/b>.<\/li>\n
- Error rate: 2.4%<\/b>.<\/li>\n<\/ul>\n<\/section>\n
\n7. Model deployment<\/h2>\n
The neural network is now ready to predict outputs for inputs that it has never seen. This process is called model deployment<\/a>.<\/p>\nOutputs<\/h3>\n
To classify a sample of milk, we calculate the neural network outputs<\/a>.<\/p>\nFor instance, consider a sample with the following features:<\/p>\n
1. Application type<\/h2>\n
This is a classification<\/a> project. Indeed, the variable to be predicted is categorical (low, medium, and high).<\/p>\n The objective is to model the quality of the milk by knowing its characteristics and thus be able to make future predictions.<\/p>\n<\/section>\n The first step is to prepare the dataset,<\/a> which is the source of information for the classification model.<\/p>\n For that, we need to configure the following concepts:<\/p>\n The data source<\/a> for this example is the CSV file milk.csv<\/a>.<\/p>\n The number of columns is 8, and the number of rows is 1060.<\/p>\n The variables<\/a> are:<\/p>\n grade<\/strong> \u2013 Milk quality class: low_quality<\/em>, medium_quality<\/em>, or high_quality<\/em>.<\/p>\n<\/li>\n<\/ul>\n All variables in the study are inputs, except “grade”, which is the output that we want to extract from this machine learning study.<\/p>\n Note that “grade” is categorical and can take the values low_quality, medium_quality, and high_quality.<\/p>\n The instances<\/a> are randomly split into 60.2% training (637 samples), 19.9% validation (211 samples), and 19.9% testing (211 samples).<\/p>\n The following figure is the pie chart for the variable milk quality class, and it shows its distribution<\/a>.<\/p>\n The milk dataset has 429 low-quality, 374 medium-quality, and 256 high-quality instances.<\/p>\n This shows an imbalanced target, with low quality making up 40.5% of the samples and high quality only 24.2%.<\/p>\n<\/section>\n The second step is to choose a neural network<\/a>.<\/p>\n The neural network must have seven inputs since the data set has seven input variables.<\/p>\n The neural network has three outputs because we have three different “grades”: low_quality, medium_quality, and high_quality.<\/p>\n In classification problems, it typically consists of:<\/p>\n The scaling layer<\/a> normalizes the input values. As our inputs have different distributions, they are scaled with different methods:<\/p>\n In this case, as the first guest, we only use one\u00a0perceptron layer<\/a>.\u00a0This layer contains seven inputs, three neurons, and three outputs. For this example, the perceptron layer<\/a> is a hyperbolic tangent activation function<\/a>.<\/p>\n The probabilistic layer<\/a> allows for interpreting the outputs as probabilities. In this regard, all outputs are between 0 and 1, and their sum is 1. The softmax probabilistic activation<\/a> is used here.<\/p>\n The following figure is a graphical representation of this classification neural network<\/a>:<\/p>\n The next step is to set the training strategy<\/a>, which comprises:<\/p>\n The loss index<\/a> chosen for this application is the\u00a0normalized squared error<\/a>\u00a0with L2 regularization<\/a>.<\/p>\n The error term<\/a> fits the neural network to the training instances<\/a> of the data set.<\/p>\n The regularization term<\/a> makes the model more stable and improves generalization.<\/p>\n The optimization algorithm<\/a> searches for the neural network parameters that minimize the loss index.<\/p>\n The quasi-Newton method<\/a>\u00a0is chosen here.<\/p>\n The following chart shows how the training (blue) and selection (orange) errors decrease with the epochs during training.<\/p>\n The final values are training error = 0.252 NSE<\/b> (blue) and selection error = 0.277 NSE<\/b> (orange).<\/p>\n<\/section>\n It is essential to have low selection error in our model, allowing us to generalize well to the new data rather than simply memorizing the training data.<\/p>\n The objective of model selection<\/a> is to find the network architecture with the best generalization properties. That is, the method that minimizes the error on the selected instances<\/a> of the data set.<\/p>\n We\u00a0aim to develop a neural network with a selection error of less than\u00a00.277 NSE<\/strong>, the current best value we have achieved<\/span>.<\/p>\n Order selection<\/a> algorithms train several network architectures with a different number of neurons and select the one with the smallest selection error.<\/p>\n The incremental order<\/a> method starts with a small number of neurons and increases the complexity at each iteration. The following chart shows the training error (blue) and the selection error (yellow) as a function of the number of neurons.<\/p>\n As we can see, the number of neurons that yield the minimum error is four. Therefore, we select the neural network with four neurons in the hidden layer.<\/p>\n The following chart shows the new neural network architecture.<\/p>\n The following chart shows how the training and selection errors decrease with the epochs during training in the new neural network. The final values are training error = 0.0877 NSE<\/b> (blue) and selection error = 0.107 NSE<\/b> (orange).<\/p>\n With the new architecture of the neural network, we achieve around 50% less selection error.<\/p>\n The purpose of the testing analysis<\/a> is to validate the generalization performance of the model.<\/p>\n Here, we compare the neural network outputs to the corresponding targets in the testing instances<\/a> of the data set.<\/p>\n The number of correctly classified samples<\/b> is 206<\/b>, and the number of misclassified samples<\/b> is 5<\/b>.<\/p>\n The confusion matrix allows us to calculate the model’s accuracy and error:<\/p>\n The neural network is now ready to predict outputs for inputs that it has never seen. This process is called model deployment<\/a>.<\/p>\n To classify a sample of milk, we calculate the neural network outputs<\/a>.<\/p>\n For instance, consider a sample with the following features:<\/p>\n2. Data set<\/h2>\n
\n
Data source<\/h3>\n
Variables<\/h3>\n
Numerical variables<\/h4>\n
\n
Binary variables<\/h4>\n
\n
Target variable<\/h4>\n<\/section>\n
\n
Instances<\/h2>\n
Distributions<\/h3>\n
<\/p>\n3. Neural network<\/h2>\n
\n
Scaling layer<\/h3>\n
\n
Dense layers<\/h3>\n
<\/p>\n<\/section>\n4. Training strategy<\/h2>\n
\n
Loss index<\/h3>\n
Optimization algorithm<\/h3>\n
<\/p>\n5. Model selection<\/h2>\n
Neuron selection<\/h3>\n
<\/p>\n<\/p>\n<\/p>\n<\/section>\n6. Testing analysis<\/h2>\n
Confusion matrix<\/h3>\n
\n\n
\n <\/th>\n Predicted low_quality<\/th>\n Predicted medium_quality<\/th>\n Predicted high_quality<\/th>\n Total<\/th>\n<\/tr>\n \n Real low_quality<\/th>\n 49 (23.2%)<\/td>\n 0<\/td>\n 4 (1.9%)<\/td>\n 53 (23.7%)<\/td>\n<\/tr>\n \n Real medium_quality<\/th>\n 1 (0.5%)<\/td>\n 92 (43.6%)<\/td>\n 0<\/td>\n 93 (43.6%)<\/td>\n<\/tr>\n \n Real high_quality<\/th>\n 0<\/td>\n 0<\/td>\n 65 (30.8%)<\/td>\n 65 (32.7%)<\/td>\n<\/tr>\n \n Total<\/th>\n 50 (23.7%)<\/td>\n 92 (43.6%)<\/td>\n 69 (32.7%)<\/td>\n 211 (100%)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Multiple classification metrics<\/h3>\n
\n
7. Model deployment<\/h2>\n
Outputs<\/h3>\n