Contents<\/span><\/h2>\n<\/section>\n\n\n- Application type<\/a>.<\/li>\n
- Data set<\/a>.<\/li>\n
- Neural network<\/a>.<\/li>\n
- Training strategy<\/a>.<\/li>\n
- Model selection<\/a>.<\/li>\n
- Testing analysis<\/a>.<\/li>\n
- Model deployment<\/a>.<\/li>\n<\/ol>\n<\/section>\n
\n1. Application type<\/h2>\n
This is a classification<\/a> project since the variable to be predicted is binary (attrition or not).<\/p>\nThe goal here is to model the probability of attrition, conditioned on the employee features.<\/p>\n<\/section>\n\n2. Data set<\/h2>\nData source<\/h3>\n
The data file employee_attrition.csv<\/a> contains quantitative and qualitative information about a sample of employees at the company.<\/p>\nThe data set contains about 1,500 employees (or\u00a0instances<\/a>).<\/p>\nFor each, around 35 personal, professional, and socio-economic attributes (or variables<\/a>) are selected.<\/p>\nVariables<\/h3>\n
More specifically, the variables<\/a> of this example are the following.<\/p>\n<\/section>\nDemographics<\/strong><\/h4>\n\n- Age<\/li>\n
- Gender: Male, Female<\/li>\n
- Marital status: Single, Divorced, Married<\/li>\n
- Over 18: True or False<\/li>\n<\/ul>\n
Job Information<\/strong><\/h4>\n\n- Department: Sales, Research & Development, Human Resources<\/li>\n
- Job role: Sales Executive, Research Scientist, Laboratory Technician, Manufacturing Director, Healthcare Representative, Manager, Sales Representative, Research Director, Human Resources<\/li>\n
- Job level: 1\u20135<\/li>\n
- Job involvement: 1\u20134<\/li>\n
- Years at company<\/li>\n
- Years in current role<\/li>\n
- Years since last promotion<\/li>\n
- Years with current manager<\/li>\n<\/ul>\n
Compensation<\/strong><\/h4>\n\n- Daily rate<\/li>\n
- Hourly rate<\/li>\n
- Monthly income<\/li>\n
- Monthly rate<\/li>\n
- Percent salary hike<\/li>\n
- Stock option level: 0\u20133<\/li>\n<\/ul>\n
Performance & Engagement<\/strong><\/h4>\n\n- Performance rating<\/li>\n
- Job satisfaction: 1\u20134<\/li>\n
- Environment satisfaction: 1\u20134<\/li>\n
- Relationship satisfaction: 1\u20134<\/li>\n
- Work-life balance: 1\u20134<\/li>\n
- Training times last year<\/li>\n<\/ul>\n
Career History<\/strong><\/h4>\n\n- Number of companies worked<\/li>\n
- Total working years<\/li>\n
- Business travel: Non-travel (0), Rarely (1), Frequently (2)<\/li>\n
- Distance from home<\/li>\n<\/ul>\n
Education<\/strong><\/h4>\n\n- Education: 1\u20135<\/li>\n
- Education field: Life Sciences, Human Resources, Medical, Marketing, Technical Degree, Other<\/li>\n<\/ul>\n
Administrative<\/strong><\/h4>\n\n- Employee count<\/li>\n
- Employee number<\/li>\n
- Standard hours<\/li>\n
- Over time: True or False<\/li>\n<\/ul>\n
Target Variable<\/strong><\/h4>\n\n- \n
Attrition: Loyal or Attrition<\/p>\n<\/li>\n<\/ul>\nWe have 48 input variables<\/a>, which contain the characteristics of every employee.On the other hand, we have 1 target variable<\/a>, the variable “Attrition” mentioned before.There are 3 constant variables (“EmployeeCount”, “Over18”, and “StandardHours”).They will be set as unused variables<\/a> for the analysis since they do not provide any valuable information.<\/p>\nVariables distribution<\/h3>\n
Before starting the predictive analysis, it is important to know the distributions<\/a> of the variables.<\/p>\nThe following pie chart shows the ratio of negative and positive instances.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/employee-attrition-data-distribution.webp\")
<\/p>\n
The chart above shows that the data is unbalanced, i.e., the number of negative instances (1233) is much larger than that of positive instances (237).<\/p>\n
We use this information later to design the predictive model properly.<\/p>\n
Input-target correlations<\/h3>\n
The input-target correlations<\/a> analyze the dependencies between each input variable and the target.<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/employee-attrition-inputs-targets-correlations.webp\")
<\/p>\n
As we can see, the input variables that have more importance in the attrition are “over time” (0.246), “total working years” (0.223), and “years at company” (0.196).<\/p>\n<\/section>\n\n3. Neural network<\/h2>\n
The neural network<\/a> takes all the employees’ attributes and transforms them into a probability of attrition.<\/p>\nFor that purpose, we use a neural network composed of a scaling layer<\/a> with 48 neurons, a perceptron layer<\/a> with 3 neurons, and a\u00a0probabilistic layer<\/a> with 1 neuron.<\/p>\n<\/section>\n\n4. Training strategy<\/h2>\n
The next step is to select an appropriate training strategy that defines what the neural network will learn.<\/p>\n
A general training strategy is composed of two concepts:<\/p>\n
\n- A loss index.<\/li>\n
- An optimization algorithm.<\/li>\n<\/ul>\n
Loss index<\/h3>\n
As mentioned earlier, the dataset is unbalanced.<\/p>\n
To address this, we use the weighted squared error<\/strong> as the error method.<\/p>\nThis assigns a weight of 5.20 to positive instances and 1 to negative ones.<\/p>\n
With this adjustment, the total weight of positives equals that of negatives.<\/p>\n
Optimization algorithm<\/h3>\n
We use the quasi-Newton method<\/a> as the optimization algorithm<\/a>.<\/p>\nNow, the model is ready to be trained. The following chart shows how the training and selection errors decrease with the epochs of the optimization algorithm.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/employee-attrition-training-history.webp\")
<\/p>\n
The final errors are 0.285 WSE for training and 0.931 WSE for validation.<\/p>\n<\/section>\n\n5. Model selection<\/h2>\n
The objective of\u00a0model selection<\/a>\u00a0is to find the network architecture with the best generalization properties, that is, the one that minimizes the error on the\u00a0selected instances<\/a> of the data set.<\/span><\/p>\nMore specifically, we\u00a0aim to develop a neural network with a selection error of less than\u00a00.931 WSE<\/strong>, the current best value we have achieved<\/span>.<\/p>\nOrder selection<\/a> algorithms train several network architectures with a different number of neurons and select the one with the smallest selection error.<\/p>\nThe incremental order<\/a> method starts with a small number of neurons and increases the complexity at each iteration. The following chart shows the training error (blue) and the selection error (orange) as a function of the number of neurons.<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/employee-attrition-order-selection.webp\")
<\/p>\n
As we can see, the optimal number of neurons in the hidden layer is 1, resulting in an order selection error of 0.614 WSE<\/b>, which is far better than the previous one.<\/p>\n<\/section>\n\n6. Testing analysis<\/h2>\n
The testing analysis<\/a> assesses the model’s quality to determine its readiness for use in the production phase, i.e., in a real-world situation.<\/p>\nThe way to test the model is to compare the trained neural network’s outputs against the real targets for a data set that has been used neither for training nor selection, the testing subset.<\/p>\n
For that purpose, we use some testing methods commonly used in binary classification problems.<\/p>\n
ROC curve<\/h3>\n
The\u00a0ROC curve<\/a> measures the discrimination capacity of the classifier between positive and negative instances.<\/p>\nThe ROC curve should pass through the upper left corner for a perfect classifier.<\/p>\n
The following chart shows the ROC curve of our problem.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/employee-attrition-roc-curve.webp\")
<\/p>\n
In this case, the area takes the value of 0.804<\/b>, which confirms what we saw in the ROC chart, that the model predicts attrition with high accuracy.<\/p>\nConfusion matrix<\/h3>\n
For classification models with a binary target variable, constructing the confusion matrix<\/a> is also a good task to test the model. Below, this table is displayed.<\/p>\n
\n\n\n<\/th>\n Predicted positive<\/th>\n Predicted negative<\/th>\n<\/tr>\n \nReal positive<\/th>\n 35 (11.9%)<\/td>\n 16 (5.44%)<\/td>\n<\/tr>\n \nReal negative<\/th>\n 47 (16%)<\/td>\n 196 (66.7%)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nBinary classification metrics<\/h3>\n
The following list depicts the binary classification tests<\/a>. They are calculated from the values of the confusion matrix.<\/p>\n
- \n
- Application type<\/a>.<\/li>\n
- Data set<\/a>.<\/li>\n
- Neural network<\/a>.<\/li>\n
- Training strategy<\/a>.<\/li>\n
- Model selection<\/a>.<\/li>\n
- Testing analysis<\/a>.<\/li>\n
- Model deployment<\/a>.<\/li>\n<\/ol>\n<\/section>\n
\n 1. Application type<\/h2>\n
This is a classification<\/a> project since the variable to be predicted is binary (attrition or not).<\/p>\n
The goal here is to model the probability of attrition, conditioned on the employee features.<\/p>\n<\/section>\n
\n 2. Data set<\/h2>\n
Data source<\/h3>\n
The data file employee_attrition.csv<\/a> contains quantitative and qualitative information about a sample of employees at the company.<\/p>\n
The data set contains about 1,500 employees (or\u00a0instances<\/a>).<\/p>\n
For each, around 35 personal, professional, and socio-economic attributes (or variables<\/a>) are selected.<\/p>\n
Variables<\/h3>\n
More specifically, the variables<\/a> of this example are the following.<\/p>\n<\/section>\n
Demographics<\/strong><\/h4>\n
- \n
- Age<\/li>\n
- Gender: Male, Female<\/li>\n
- Marital status: Single, Divorced, Married<\/li>\n
- Over 18: True or False<\/li>\n<\/ul>\n
Job Information<\/strong><\/h4>\n
- \n
- Department: Sales, Research & Development, Human Resources<\/li>\n
- Job role: Sales Executive, Research Scientist, Laboratory Technician, Manufacturing Director, Healthcare Representative, Manager, Sales Representative, Research Director, Human Resources<\/li>\n
- Job level: 1\u20135<\/li>\n
- Job involvement: 1\u20134<\/li>\n
- Years at company<\/li>\n
- Years in current role<\/li>\n
- Years since last promotion<\/li>\n
- Years with current manager<\/li>\n<\/ul>\n
Compensation<\/strong><\/h4>\n
- \n
- Daily rate<\/li>\n
- Hourly rate<\/li>\n
- Monthly income<\/li>\n
- Monthly rate<\/li>\n
- Percent salary hike<\/li>\n
- Stock option level: 0\u20133<\/li>\n<\/ul>\n
Performance & Engagement<\/strong><\/h4>\n
- \n
- Performance rating<\/li>\n
- Job satisfaction: 1\u20134<\/li>\n
- Environment satisfaction: 1\u20134<\/li>\n
- Relationship satisfaction: 1\u20134<\/li>\n
- Work-life balance: 1\u20134<\/li>\n
- Training times last year<\/li>\n<\/ul>\n
Career History<\/strong><\/h4>\n
- \n
- Number of companies worked<\/li>\n
- Total working years<\/li>\n
- Business travel: Non-travel (0), Rarely (1), Frequently (2)<\/li>\n
- Distance from home<\/li>\n<\/ul>\n
Education<\/strong><\/h4>\n
- \n
- Education: 1\u20135<\/li>\n
- Education field: Life Sciences, Human Resources, Medical, Marketing, Technical Degree, Other<\/li>\n<\/ul>\n
Administrative<\/strong><\/h4>\n
- \n
- Employee count<\/li>\n
- Employee number<\/li>\n
- Standard hours<\/li>\n
- Over time: True or False<\/li>\n<\/ul>\n
Target Variable<\/strong><\/h4>\n
- \n
- \n
Attrition: Loyal or Attrition<\/p>\n<\/li>\n<\/ul>\n
We have 48 input variables<\/a>, which contain the characteristics of every employee.On the other hand, we have 1 target variable<\/a>, the variable “Attrition” mentioned before.There are 3 constant variables (“EmployeeCount”, “Over18”, and “StandardHours”).They will be set as unused variables<\/a> for the analysis since they do not provide any valuable information.<\/p>\n Variables distribution<\/h3>\n
Before starting the predictive analysis, it is important to know the distributions<\/a> of the variables.<\/p>\n
The following pie chart shows the ratio of negative and positive instances.<\/p>\n
<\/p>\nThe chart above shows that the data is unbalanced, i.e., the number of negative instances (1233) is much larger than that of positive instances (237).<\/p>\n
We use this information later to design the predictive model properly.<\/p>\n
Input-target correlations<\/h3>\n
The input-target correlations<\/a> analyze the dependencies between each input variable and the target.<\/p>\n
<\/p>\nAs we can see, the input variables that have more importance in the attrition are “over time” (0.246), “total working years” (0.223), and “years at company” (0.196).<\/p>\n<\/section>\n
\n 3. Neural network<\/h2>\n
The neural network<\/a> takes all the employees’ attributes and transforms them into a probability of attrition.<\/p>\n
For that purpose, we use a neural network composed of a scaling layer<\/a> with 48 neurons, a perceptron layer<\/a> with 3 neurons, and a\u00a0probabilistic layer<\/a> with 1 neuron.<\/p>\n<\/section>\n
\n 4. Training strategy<\/h2>\n
The next step is to select an appropriate training strategy that defines what the neural network will learn.<\/p>\n
A general training strategy is composed of two concepts:<\/p>\n
- \n
- A loss index.<\/li>\n
- An optimization algorithm.<\/li>\n<\/ul>\n
Loss index<\/h3>\n
As mentioned earlier, the dataset is unbalanced.<\/p>\n
To address this, we use the weighted squared error<\/strong> as the error method.<\/p>\n
This assigns a weight of 5.20 to positive instances and 1 to negative ones.<\/p>\n
With this adjustment, the total weight of positives equals that of negatives.<\/p>\n
Optimization algorithm<\/h3>\n
We use the quasi-Newton method<\/a> as the optimization algorithm<\/a>.<\/p>\n
Now, the model is ready to be trained. The following chart shows how the training and selection errors decrease with the epochs of the optimization algorithm.<\/p>\n
<\/p>\nThe final errors are 0.285 WSE for training and 0.931 WSE for validation.<\/p>\n<\/section>\n
\n 5. Model selection<\/h2>\n
The objective of\u00a0model selection<\/a>\u00a0is to find the network architecture with the best generalization properties, that is, the one that minimizes the error on the\u00a0selected instances<\/a> of the data set.<\/span><\/p>\n
More specifically, we\u00a0aim to develop a neural network with a selection error of less than\u00a00.931 WSE<\/strong>, the current best value we have achieved<\/span>.<\/p>\n
Order selection<\/a> algorithms train several network architectures with a different number of neurons and select the one with the smallest selection error.<\/p>\n
The incremental order<\/a> method starts with a small number of neurons and increases the complexity at each iteration. The following chart shows the training error (blue) and the selection error (orange) as a function of the number of neurons.<\/p>\n
<\/p>\nAs we can see, the optimal number of neurons in the hidden layer is 1, resulting in an order selection error of 0.614 WSE<\/b>, which is far better than the previous one.<\/p>\n<\/section>\n
\n 6. Testing analysis<\/h2>\n
The testing analysis<\/a> assesses the model’s quality to determine its readiness for use in the production phase, i.e., in a real-world situation.<\/p>\n
The way to test the model is to compare the trained neural network’s outputs against the real targets for a data set that has been used neither for training nor selection, the testing subset.<\/p>\n
For that purpose, we use some testing methods commonly used in binary classification problems.<\/p>\n
ROC curve<\/h3>\n
The\u00a0ROC curve<\/a> measures the discrimination capacity of the classifier between positive and negative instances.<\/p>\n
The ROC curve should pass through the upper left corner for a perfect classifier.<\/p>\n
The following chart shows the ROC curve of our problem.<\/p>\n
<\/p>\nIn this case, the area takes the value of 0.804<\/b>, which confirms what we saw in the ROC chart, that the model predicts attrition with high accuracy.<\/p>\n
Confusion matrix<\/h3>\n
For classification models with a binary target variable, constructing the confusion matrix<\/a> is also a good task to test the model. Below, this table is displayed.<\/p>\n
\n\n
\n <\/th>\n Predicted positive<\/th>\n Predicted negative<\/th>\n<\/tr>\n \n Real positive<\/th>\n 35 (11.9%)<\/td>\n 16 (5.44%)<\/td>\n<\/tr>\n \n Real negative<\/th>\n 47 (16%)<\/td>\n 196 (66.7%)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Binary classification metrics<\/h3>\n
The following list depicts the binary classification tests<\/a>. They are calculated from the values of the confusion matrix.<\/p>\n
- \n
- Data set<\/a>.<\/li>\n