Contents<\/span><\/h2>\n<\/section>\n<\/section>\n\n\n- Application type<\/a>.<\/li>\n
- Data set<\/a>.<\/li>\n
- Neural network<\/a>.<\/li>\n
- Training strategy<\/a>.<\/li>\n
- Model selection<\/a>.<\/li>\n
- Testing analysis<\/a>.<\/li>\n
- Model deployment<\/a>.<\/li>\n<\/ol>\n<\/section>\n
\n1. Application type<\/h2>\n
The variable we are predicting is binary (default or not). Therefore, this is a classification<\/a> project.<\/p>\nThe goal here is to model the probability of default as a function of the customer features.<\/p>\n<\/section>\n\n2. Data set<\/h2>\n
The data set<\/a> consists of four concepts:<\/p>\n\n- Data source.<\/li>\n
- Variables.<\/li>\n
- Instances.<\/li>\n
- Missing values.<\/li>\n<\/ul>\n
Data source<\/h3>\n
The data file credit_risk.csv<\/a> contains the information used to create the model. It consists of 30,000 rows and 25 columns. The columns represent the variables, while the rows represent the instances.<\/p>\nVariables<\/h3>\n
This data set uses the following 23 variables<\/a>:<\/p>\n<\/section>\nDemographics<\/strong><\/h4>\n\n- Sex<\/strong>: Gender (1 = male, 2 = female).<\/li>\n
- Education level<\/strong>: Highest education attained (1 = graduate school, 2 = university, 3 = high school, 4 = other).<\/li>\n
- Marital status<\/strong>: 1 = married, 2 = single, 3 = other.<\/li>\n
- Age<\/strong>: Age of the client (in years).<\/li>\n<\/ul>\n
Credit Information<\/strong><\/h4>\n\n- \n
Limit balance<\/strong>: Amount of credit granted in NT dollars (includes personal and family\/supplementary credit).<\/p>\n<\/li>\n<\/ul>\nRepayment History<\/strong><\/h4>\n\n- \n
Repayment status (lag 1\u20136)<\/strong>: Repayment status for the last 6 months (-1 = paid duly, 1 = one month delay, \u2026, 9 = nine months delay or more).<\/p>\n<\/li>\n<\/ul>\nBilling History<\/strong><\/h4>\n\n- \n
Bill statement amount (lag 1\u20136)<\/strong>: Bill amount for each of the past 6 months (NT dollars).<\/p>\n<\/li>\n<\/ul>\nPayment History<\/strong><\/h4>\n\n- \n
Payment amount (lag 1\u20136)<\/strong>: Amount paid for each of the past 6 months (NT dollars).<\/p>\n<\/li>\n<\/ul>\nTarget Variable<\/strong><\/h4>\n\n- \n
Default<\/strong>: Indicates loan default (failure to repay).<\/p>\n<\/li>\n<\/ul>\n\nInstances<\/h3>\n
Finally, the use of all instances<\/a> is selected.<\/p>\nEach customer represents an instance that contains the input and target variables.<\/p>\n
Neural Designer automatically splits the data into 60% training, 20% validation, and 20% testing.<\/p>\n
In this case, that means 18,000 samples for training, 6,000 for validation, and 6,000 for testing.<\/p>\n
Variables distribution<\/h3>\n
We can also calculate the data distributions<\/a> for each variable.<\/p>\nThe following figure depicts the number of customers who repay the loan and those who do not.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/credit-risk-data-distribution.webp\")
<\/p>\n
The data is unbalanced, as we can observe; this information will be used to configure the neural network later.<\/p>\n
Input-target correlations<\/h3>\n
The following figure depicts the input-target correlations<\/a> of all the inputs with the target.<\/p>\nThis helps us understand the impact of various inputs on the default.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/credit-risk-inputs-targets-correlations.webp\")
<\/p>\n<\/section>\n\n3. Neural network<\/h2>\n
The second step is to\u00a0select a\u00a0neural network<\/a>\u00a0that represents\u00a0the classification function.<\/p>\nFor classification problems, it is composed of:<\/p>\n
\n- A scaling layer<\/a>.<\/li>\n
- A hidden dense layer<\/a>.<\/li>\n
- An output dense layer<\/a>.<\/li>\n<\/ul>\n
Scaling layer<\/h3>\n
The mean and standard deviation scaling method<\/a> is set for the scaling layer<\/a>.<\/p>\nDense layers<\/h3>\n
We set up two dense layers: one hidden layer with 3 neurons as a first guess and one output layer with 1 neuron, both layers having the logistic activation function<\/a>.<\/p>\nNeural network graph<\/h3>\n
The following figure shows the neural network used in this example.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/credit-risk-initial-neural-network.webp\")
<\/p>\n<\/section>\n\n4. Training strategy<\/h2>\n
The fourth step is to configure the training strategy<\/a>, which is composed of two concepts:<\/p>\n\n- A loss index.<\/li>\n
- An optimization algorithm.<\/li>\n<\/ul>\n
Loss index<\/h3>\n
The error term is the weighted squared error. It weights the squared error of negative and positive values. If the weighted squared error has a value of unity, then the neural network predicts the data ‘in the mean’, while a value of zero means a perfect prediction of the data.<\/p>\n
In this case, the neural parameters norm weight term is 0.01. This parameter makes the model stable, avoiding oscillations.<\/p>\n
Optimization algorithm<\/h3>\n
The optimization algorithm<\/a> is applied to the neural network to achieve optimal performance.<\/p>\nWe chose the quasi-Newton method and left the default parameters.<\/p>\n
The following chart illustrates how training and selection errors decrease over the course of training epochs.
\n![](\"https:\/\/www.neuraldesigner.com\/images\/credit-risk-training-history.webp\")
<\/p>\n
The final results are training error = 0.755 WSE<\/b> and selection error = 0.802 WSE<\/b>, respectively.<\/p>\n<\/section>\n\n5. Model selection<\/h2>\n
A model selection<\/a> aims to find the network architecture with the best generalization properties, i.e., the one that minimizes the error on the selected instances<\/a> of the data set.<\/p>\nMore specifically, we\u00a0aim to develop a neural network with a selection error of less than\u00a00.802 WSE<\/strong>, the current best value we have achieved<\/span>.<\/p>\nNeuron selection<\/h3>\n
Order selection<\/a> algorithms train several network architectures with different numbers of neurons and select the one with the smallest selection error.<\/p>\nThe incremental order<\/a> method starts with a few neurons and increases the complexity at each iteration.
\nThe following chart shows the training error (blue) and the selection error (orange) as a function of the number of neurons.<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/credit-risk-model-selection.webp\")
<\/p>\n
The final selection error achieved is 0.801<\/b> for an optimal number of neurons of 3.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/credit-risk-final-neural-network.webp\")
<\/p>\n
The graph above represents the architecture of the final neural network.<\/p>\n<\/section>\n<\/section>\n\n6. Testing analysis<\/h2>\n
The next step is to evaluate the trained neural network’s performance through exhaustive testing analysis<\/a>.<\/p>\nThe standard way to do this is to compare the neural network’s outputs against previously unseen data, the training instances.<\/p>\n
ROC curve<\/h3>\n
A common way to measure generalization is with the ROC curve<\/a>, which shows how well the classifier separates the classes.<\/p>\nIts primary metric is the area under the curve (AUC), where values closer to 1 indicate better performance.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/credit-risk-roc-curve.webp\")
<\/p>\n
In this case, the AUC takes a high value: AUC = 0.772<\/b>.<\/p>\nConfusion matrix<\/h3>\n
This matrix contains the variable class\u2019s true positives, false positives, false negatives, and true negatives.<\/p>\n
The following table contains the elements of the confusion matrix<\/a>.<\/p>\n
- \n
- Application type<\/a>.<\/li>\n
- Data set<\/a>.<\/li>\n
- Neural network<\/a>.<\/li>\n
- Training strategy<\/a>.<\/li>\n
- Model selection<\/a>.<\/li>\n
- Testing analysis<\/a>.<\/li>\n
- Model deployment<\/a>.<\/li>\n<\/ol>\n<\/section>\n
\n 1. Application type<\/h2>\n
The variable we are predicting is binary (default or not). Therefore, this is a classification<\/a> project.<\/p>\n
The goal here is to model the probability of default as a function of the customer features.<\/p>\n<\/section>\n
\n 2. Data set<\/h2>\n
The data set<\/a> consists of four concepts:<\/p>\n
- \n
- Data source.<\/li>\n
- Variables.<\/li>\n
- Instances.<\/li>\n
- Missing values.<\/li>\n<\/ul>\n
Data source<\/h3>\n
The data file credit_risk.csv<\/a> contains the information used to create the model. It consists of 30,000 rows and 25 columns. The columns represent the variables, while the rows represent the instances.<\/p>\n
Variables<\/h3>\n
This data set uses the following 23 variables<\/a>:<\/p>\n<\/section>\n
Demographics<\/strong><\/h4>\n
- \n
- Sex<\/strong>: Gender (1 = male, 2 = female).<\/li>\n
- Education level<\/strong>: Highest education attained (1 = graduate school, 2 = university, 3 = high school, 4 = other).<\/li>\n
- Marital status<\/strong>: 1 = married, 2 = single, 3 = other.<\/li>\n
- Age<\/strong>: Age of the client (in years).<\/li>\n<\/ul>\n
Credit Information<\/strong><\/h4>\n
- \n
- \n
Limit balance<\/strong>: Amount of credit granted in NT dollars (includes personal and family\/supplementary credit).<\/p>\n<\/li>\n<\/ul>\n
Repayment History<\/strong><\/h4>\n
- \n
- \n
Repayment status (lag 1\u20136)<\/strong>: Repayment status for the last 6 months (-1 = paid duly, 1 = one month delay, \u2026, 9 = nine months delay or more).<\/p>\n<\/li>\n<\/ul>\n
Billing History<\/strong><\/h4>\n
- \n
- \n
Bill statement amount (lag 1\u20136)<\/strong>: Bill amount for each of the past 6 months (NT dollars).<\/p>\n<\/li>\n<\/ul>\n
Payment History<\/strong><\/h4>\n
- \n
- \n
Payment amount (lag 1\u20136)<\/strong>: Amount paid for each of the past 6 months (NT dollars).<\/p>\n<\/li>\n<\/ul>\n
Target Variable<\/strong><\/h4>\n
- \n
- \n
Default<\/strong>: Indicates loan default (failure to repay).<\/p>\n<\/li>\n<\/ul>\n
\n Instances<\/h3>\n
Finally, the use of all instances<\/a> is selected.<\/p>\n
Each customer represents an instance that contains the input and target variables.<\/p>\n
Neural Designer automatically splits the data into 60% training, 20% validation, and 20% testing.<\/p>\n
In this case, that means 18,000 samples for training, 6,000 for validation, and 6,000 for testing.<\/p>\n
Variables distribution<\/h3>\n
We can also calculate the data distributions<\/a> for each variable.<\/p>\n
The following figure depicts the number of customers who repay the loan and those who do not.<\/p>\n
<\/p>\nThe data is unbalanced, as we can observe; this information will be used to configure the neural network later.<\/p>\n
Input-target correlations<\/h3>\n
The following figure depicts the input-target correlations<\/a> of all the inputs with the target.<\/p>\n
This helps us understand the impact of various inputs on the default.<\/p>\n
<\/p>\n<\/section>\n\n 3. Neural network<\/h2>\n
The second step is to\u00a0select a\u00a0neural network<\/a>\u00a0that represents\u00a0the classification function.<\/p>\n
For classification problems, it is composed of:<\/p>\n
- \n
- A scaling layer<\/a>.<\/li>\n
- A hidden dense layer<\/a>.<\/li>\n
- An output dense layer<\/a>.<\/li>\n<\/ul>\n
Scaling layer<\/h3>\n
The mean and standard deviation scaling method<\/a> is set for the scaling layer<\/a>.<\/p>\n
Dense layers<\/h3>\n
We set up two dense layers: one hidden layer with 3 neurons as a first guess and one output layer with 1 neuron, both layers having the logistic activation function<\/a>.<\/p>\n
Neural network graph<\/h3>\n
The following figure shows the neural network used in this example.<\/p>\n
<\/p>\n<\/section>\n\n 4. Training strategy<\/h2>\n
The fourth step is to configure the training strategy<\/a>, which is composed of two concepts:<\/p>\n
- \n
- A loss index.<\/li>\n
- An optimization algorithm.<\/li>\n<\/ul>\n
Loss index<\/h3>\n
The error term is the weighted squared error. It weights the squared error of negative and positive values. If the weighted squared error has a value of unity, then the neural network predicts the data ‘in the mean’, while a value of zero means a perfect prediction of the data.<\/p>\n
In this case, the neural parameters norm weight term is 0.01. This parameter makes the model stable, avoiding oscillations.<\/p>\n
Optimization algorithm<\/h3>\n
The optimization algorithm<\/a> is applied to the neural network to achieve optimal performance.<\/p>\n
We chose the quasi-Newton method and left the default parameters.<\/p>\n
The following chart illustrates how training and selection errors decrease over the course of training epochs.
\n<\/p>\nThe final results are training error = 0.755 WSE<\/b> and selection error = 0.802 WSE<\/b>, respectively.<\/p>\n<\/section>\n
\n 5. Model selection<\/h2>\n
A model selection<\/a> aims to find the network architecture with the best generalization properties, i.e., the one that minimizes the error on the selected instances<\/a> of the data set.<\/p>\n
More specifically, we\u00a0aim to develop a neural network with a selection error of less than\u00a00.802 WSE<\/strong>, the current best value we have achieved<\/span>.<\/p>\n
Neuron selection<\/h3>\n
Order selection<\/a> algorithms train several network architectures with different numbers of neurons and select the one with the smallest selection error.<\/p>\n
The incremental order<\/a> method starts with a few neurons and increases the complexity at each iteration.
\nThe following chart shows the training error (blue) and the selection error (orange) as a function of the number of neurons.<\/p>\n<\/p>\nThe final selection error achieved is 0.801<\/b> for an optimal number of neurons of 3.<\/p>\n
<\/p>\nThe graph above represents the architecture of the final neural network.<\/p>\n<\/section>\n
<\/section>\n \n 6. Testing analysis<\/h2>\n
The next step is to evaluate the trained neural network’s performance through exhaustive testing analysis<\/a>.<\/p>\n
The standard way to do this is to compare the neural network’s outputs against previously unseen data, the training instances.<\/p>\n
ROC curve<\/h3>\n
A common way to measure generalization is with the ROC curve<\/a>, which shows how well the classifier separates the classes.<\/p>\n
Its primary metric is the area under the curve (AUC), where values closer to 1 indicate better performance.<\/p>\n
<\/p>\nIn this case, the AUC takes a high value: AUC = 0.772<\/b>.<\/p>\n
Confusion matrix<\/h3>\n
This matrix contains the variable class\u2019s true positives, false positives, false negatives, and true negatives.<\/p>\n
The following table contains the elements of the confusion matrix<\/a>.<\/p>\n
- A hidden dense layer<\/a>.<\/li>\n
- A scaling layer<\/a>.<\/li>\n
- \n
- \n
- \n
- \n
- Education level<\/strong>: Highest education attained (1 = graduate school, 2 = university, 3 = high school, 4 = other).<\/li>\n
- Sex<\/strong>: Gender (1 = male, 2 = female).<\/li>\n
- Data set<\/a>.<\/li>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/risk_assesment_cumulative_gain.webp\")
<\/p>\n