Contents<\/h3>\n\n- Application type<\/a>.<\/li>\n
- Data set<\/a>.<\/li>\n
- Neural network<\/a>.<\/li>\n
- Training strategy<\/a>.<\/li>\n
- Model selection<\/a>.<\/li>\n
- Testing analysis<\/a>.<\/li>\n
- Model deployment<\/a>.<\/li>\n<\/ol>\n<\/section>\n
\n1. Application type<\/h2>\n
This is a classification<\/a> project since the variable to be predicted is binary (churn or loyal customer).<\/p>\nThe goal is to model churn probability conditioned on the customer features.<\/p>\n<\/section>\n\n2. Data set<\/h2>\n
The data file telecommunications_churn.csv<\/a> contains a total of 19 features for 3333 customers.<\/p>\nEach row corresponds to a client of a telecommunications company for whom information has been collected about the type of plan they have contracted, the minutes they have talked, or the charge they pay every month.<\/p>\n
Variables<\/h3>\n
The data set includes the following variables<\/a>:<\/p>\n<\/section>\nCustomer Profile<\/h4>\n\n- account_length<\/strong>: Number of months the customer has been with the company.<\/li>\n
- customer_service_calls<\/strong>: Number of calls made by the customer to the service center.<\/li>\n<\/ul>\n
Service Plans<\/h4>\n\n- voice_mail_plan<\/strong>: Whether the customer has subscribed to a voicemail plan (yes\/no).<\/li>\n
- voice_mail_messages<\/strong>: Number of voicemail messages recorded.<\/li>\n
- international_plan<\/strong>: Whether the customer has subscribed to an international plan (yes\/no).<\/li>\n<\/ul>\n
Usage (Minutes and Calls)<\/h4>\n\n- day_mins<\/strong>: Total minutes of calls made during the day.<\/li>\n
- day_calls<\/strong>: Total number of calls made during the day.<\/li>\n
- evening_mins<\/strong>: Total minutes of calls made in the evening.<\/li>\n
- evening_calls<\/strong>: Total number of calls made in the evening.<\/li>\n
- night_mins<\/strong>: Total minutes of calls made at night.<\/li>\n
- night_calls<\/strong>: Total number of calls made at night.<\/li>\n
- international_mins<\/strong>: Total minutes of international calls.<\/li>\n
- international_calls<\/strong>: Total number of international calls.<\/li>\n<\/ul>\n
Billing (Charges)<\/h4>\n\n- day_charge<\/strong>: Total charges for calls made during the day.<\/li>\n
- evening_charge<\/strong>: Total charges for calls made in the evening.<\/li>\n
- night_charge<\/strong>: Total charges for calls made at night.<\/li>\n
- international_charge<\/strong>: Total charges for international calls.<\/li>\n
- total_charge<\/strong>: Overall charges (sum of day, evening, night, and international charges).<\/li>\n<\/ul>\n
Target Variable<\/h4>\n\n- \n
churn<\/strong>: Indicates whether the customer has left the company (1) or stayed (0).<\/p>\n<\/li>\n<\/ul>\n\nVariables distribution<\/h3>\n
The first step of this analysis is to check the distributions<\/a> of the variables. The following figure shows a pie chart of churn and loyal customers.<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/telecommunications-churn-data-distribution.webp\")
<\/p>\n
As we can see, the annual churn rate in this company is almost 15%. Also, we observe that the dataset is unbalanced.<\/p>\n
Inputs-targets correlations<\/h3>\n
The inputs-targets correlations<\/a> might indicate to us what factors are most influential for the churn of customers.<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/telecommunications-churn-inputs-targets-correlations.webp\")
<\/p>\n
Here, the most correlated variable with churn is international_plan<\/b>. A positive correlation here means that a high ratio of customers with an international plan leaves the company.<\/p>\n<\/section>\n\n2. Neural network<\/h2>\n
The second step is to choose a neural network<\/a> to represent the classification function. For classification problems, it is composed of:<\/p>\n\n- A scaling layer<\/a>.<\/li>\n
- Two perceptron layers<\/a>.<\/li>\n
- A probabilistic layer<\/a>.<\/li>\n<\/ul>\n
Scaling layer<\/h3>\n
For the scaling layer<\/a>, the mean and standard deviation scaling method<\/a> is set.<\/p>\nPerceptron layers<\/h3>\n
We set 2 perceptron layers, one hidden layer with 3 neurons as a first guess and one output layer with 1 neuron, both layers having the logistic activation function<\/a>.<\/p>\nProbabilistic layer<\/h3>\n
Finally, we will implement the continuous probabilistic method<\/a> for the probabilistic layer<\/a>.<\/p>\nNetwork architecture<\/h3>\n
The architecture shown below consists of 18 scaling neurons (yellow), five neurons in the first layer (blue), and one probabilistic neuron (red).<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/telecommunications-churn-initial-neural-network.webp\")
<\/p>\n<\/section>\n\n4. Training strategy<\/h2>\n
The next step is\u00a0to select an appropriate\u00a0training strategy<\/a> that defines<\/span>\u00a0what the neural network will learn. A general training strategy is composed of two concepts:<\/p>\n\n- A loss index.<\/li>\n
- An optimization algorithm.<\/li>\n<\/ul>\n
Loss index<\/h3>\n
As we saw before, the data set is unbalanced. In that way, we set the weighted squared error<\/a>.<\/p>\nTraining process<\/h3>\n
The following chart shows how the loss decreases during the training process with the iterations of the Quasi-Newton method.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/telecommunications-churn-training-history.webp\")
<\/p>\n
As we can see, the final training and selection errors are\u00a0training error = 0.384 WSE<\/b> and selection error = 0.455 WSE<\/b>, respectively.<\/p>\n<\/section>\n\n5. Model selection<\/h2>\n
The objective of model selection<\/a> is to find the network architecture with the best generalization properties, which minimizes the error on the selected instances<\/a> of the data set.<\/p>\nWe\u00a0aim to develop a neural network with a selection error of less than\u00a00.455 WSE<\/strong>, which is <\/span>the value we have achieved so far.<\/p>\nOrder selection<\/a> algorithms train several network architectures with a different number of neurons and select that with the most minor selection error.<\/p>\nThe incremental order<\/a> method starts with a few neurons and increases the complexity at each iteration.
\nThe following chart shows the training error (blue) and the selection error (orange) as a function of the number of neurons.<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/telecommunications-churn-order-selection.webp\")
<\/p>\n
As we can see, the optimal number of perceptrons in the first layer is\u00a02<\/strong>, and the optimum error on the selected instances is <\/span>0.455 WSE<\/b>.<\/p>\n![](\"https:\/\/www.neuraldesigner.com\/images\/telecommunications-churn-final-neural-network.webp\")
<\/p>\n<\/section>\n\n6. Testing analysis<\/h2>\n
The testing analysis aims to evaluate the performance of the trained model on new data that have not been used for training or selection.<\/p>\n
For that, we use the testing instances<\/a>.<\/p>\nROC curve<\/h3>\n
The ROC curve<\/a> measures the discrimination capacity of the classifier between positive and negative instances.<\/p>\nThe following chart shows the ROC curve of our problem.<\/p>\n
![](\"https:\/\/www.neuraldesigner.com\/images\/telecommunications-churn-roc-curve.webp\")
<\/p>\n
The proximity of the curve to the upper left corner means that the model has an excellent capacity to discriminate between the two classes.<\/p>\n
The most important parameter from the ROC curve is the area under the curve (AUC). This value is 0.5 for a random classifier and 1 for a perfect classifier.<\/p>\n
For this example, we have AUC = 0.896<\/b>, indicating that the model performs well in predicting our customers’ churn.<\/p>\nConfusion matrix<\/h3>\n
The following figure shows the confusion matrix<\/a>.<\/p>\n
\n\n\n<\/th>\n Predicted positive<\/th>\n Predicted negative<\/th>\n<\/tr>\n \nReal positive<\/th>\n 316 (15.8%)<\/td>\n 96 (4.8%)<\/td>\n<\/tr>\n \nReal negative<\/th>\n 325 (16.3%)<\/td>\n 1263 (63.1%)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nThe binary classification tests<\/a> are calculated from the values of the confusion matrix.<\/p>\n\n- Classification accuracy: 91.2% <\/b> (ratio of correctly classified samples).<\/li>\n
- Error rate: 8.8%<\/b> (ratio of misclassified samples).<\/li>\n
- Sensitivity: 76.9%<\/b> (percentage of actual positives classified as positive).<\/li>\n
- Specificity: 93.9%<\/b> (percentage of actual negatives classified as negative).<\/li>\n<\/ul>\n
These binary classification tests show that the model can predict most instances correctly.<\/p>\n
7. Model deployment<\/h3>\n
1. Application type<\/h2>\n
This is a classification<\/a> project since the variable to be predicted is binary (churn or loyal customer).<\/p>\n The goal is to model churn probability conditioned on the customer features.<\/p>\n<\/section>\n The data file telecommunications_churn.csv<\/a> contains a total of 19 features for 3333 customers.<\/p>\n Each row corresponds to a client of a telecommunications company for whom information has been collected about the type of plan they have contracted, the minutes they have talked, or the charge they pay every month.<\/p>\n The data set includes the following variables<\/a>:<\/p>\n<\/section>\n churn<\/strong>: Indicates whether the customer has left the company (1) or stayed (0).<\/p>\n<\/li>\n<\/ul>\n The first step of this analysis is to check the distributions<\/a> of the variables. The following figure shows a pie chart of churn and loyal customers.<\/p>\n As we can see, the annual churn rate in this company is almost 15%. Also, we observe that the dataset is unbalanced.<\/p>\n The inputs-targets correlations<\/a> might indicate to us what factors are most influential for the churn of customers.<\/p>\n Here, the most correlated variable with churn is international_plan<\/b>. A positive correlation here means that a high ratio of customers with an international plan leaves the company.<\/p>\n<\/section>\n The second step is to choose a neural network<\/a> to represent the classification function. For classification problems, it is composed of:<\/p>\n For the scaling layer<\/a>, the mean and standard deviation scaling method<\/a> is set.<\/p>\n We set 2 perceptron layers, one hidden layer with 3 neurons as a first guess and one output layer with 1 neuron, both layers having the logistic activation function<\/a>.<\/p>\n Finally, we will implement the continuous probabilistic method<\/a> for the probabilistic layer<\/a>.<\/p>\n The architecture shown below consists of 18 scaling neurons (yellow), five neurons in the first layer (blue), and one probabilistic neuron (red).<\/p>\n The next step is\u00a0to select an appropriate\u00a0training strategy<\/a> that defines<\/span>\u00a0what the neural network will learn. A general training strategy is composed of two concepts:<\/p>\n As we saw before, the data set is unbalanced. In that way, we set the weighted squared error<\/a>.<\/p>\n The following chart shows how the loss decreases during the training process with the iterations of the Quasi-Newton method.<\/p>\n As we can see, the final training and selection errors are\u00a0training error = 0.384 WSE<\/b> and selection error = 0.455 WSE<\/b>, respectively.<\/p>\n<\/section>\n The objective of model selection<\/a> is to find the network architecture with the best generalization properties, which minimizes the error on the selected instances<\/a> of the data set.<\/p>\n We\u00a0aim to develop a neural network with a selection error of less than\u00a00.455 WSE<\/strong>, which is <\/span>the value we have achieved so far.<\/p>\n Order selection<\/a> algorithms train several network architectures with a different number of neurons and select that with the most minor selection error.<\/p>\n The incremental order<\/a> method starts with a few neurons and increases the complexity at each iteration. As we can see, the optimal number of perceptrons in the first layer is\u00a02<\/strong>, and the optimum error on the selected instances is <\/span>0.455 WSE<\/b>.<\/p>\n The testing analysis aims to evaluate the performance of the trained model on new data that have not been used for training or selection.<\/p>\n For that, we use the testing instances<\/a>.<\/p>\n The ROC curve<\/a> measures the discrimination capacity of the classifier between positive and negative instances.<\/p>\n The following chart shows the ROC curve of our problem.<\/p>\n The proximity of the curve to the upper left corner means that the model has an excellent capacity to discriminate between the two classes.<\/p>\n The most important parameter from the ROC curve is the area under the curve (AUC). This value is 0.5 for a random classifier and 1 for a perfect classifier.<\/p>\n For this example, we have AUC = 0.896<\/b>, indicating that the model performs well in predicting our customers’ churn.<\/p>\n The following figure shows the confusion matrix<\/a>.<\/p>\n The binary classification tests<\/a> are calculated from the values of the confusion matrix.<\/p>\n These binary classification tests show that the model can predict most instances correctly.<\/p>\n2. Data set<\/h2>\n
Variables<\/h3>\n
Customer Profile<\/h4>\n
\n
Service Plans<\/h4>\n
\n
Usage (Minutes and Calls)<\/h4>\n
\n
Billing (Charges)<\/h4>\n
\n
Target Variable<\/h4>\n
\n
Variables distribution<\/h3>\n
<\/p>\nInputs-targets correlations<\/h3>\n
<\/p>\n2. Neural network<\/h2>\n
\n
Scaling layer<\/h3>\n
Perceptron layers<\/h3>\n
Probabilistic layer<\/h3>\n
Network architecture<\/h3>\n
<\/p>\n<\/section>\n4. Training strategy<\/h2>\n
\n
Loss index<\/h3>\n
Training process<\/h3>\n
<\/p>\n5. Model selection<\/h2>\n
\nThe following chart shows the training error (blue) and the selection error (orange) as a function of the number of neurons.<\/p>\n<\/p>\n<\/p>\n<\/section>\n6. Testing analysis<\/h2>\n
ROC curve<\/h3>\n
<\/p>\nConfusion matrix<\/h3>\n
\n\n
\n <\/th>\n Predicted positive<\/th>\n Predicted negative<\/th>\n<\/tr>\n \n Real positive<\/th>\n 316 (15.8%)<\/td>\n 96 (4.8%)<\/td>\n<\/tr>\n \n Real negative<\/th>\n 325 (16.3%)<\/td>\n 1263 (63.1%)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \n
7. Model deployment<\/h3>\n