In the last part of the tutorial we introduced the basic qualitative model quality indicators. Let us recall them now:

- TP – True Positive – the number of observations correctly assigned to the positive class

Example: the model’s predictions are correct and resigning customers have been assigned to the class of „disloyal” customers - TN – True Negative – the number of observations correctly assigned to the negative class

Example: the model’s predictions are correct and customers who continue using the service have been assigned to the class of „loyal” customers. - FP – False Positive – the number of observations assigned by the model to the positive class, which in reality belong to the negative class.

Example: unfortunately the model is not perfect and made a mistake: some customers, who continue using the service have been assigned to the class of „disloyal” customers. - FN – False Negative – the number of observations assigned by the model to the negative class, which in reality belong to the positive class.

Example: unfortunately the model is not perfect and made a mistake: some churning customers have been assigned to the class of „loyal” customers.

We will now discuss derived variants of these indicators.

- TPR (True Positive Rate) – reflects the classifier’s ability to detect members of the positive class (pathological state)

TPR = TP / (TP + FN)

- TNR (True Negative Rate) – reflects the classifier’s ability to detect members of the negative class (normal state)

TNR = TN / (TN + FP)

- FPR (False Positive Rate) – reflects the frequency with which the classifier makes a mistake by classifying normal state as pathological

FPR = FP / (FP + TN)

- FNR (False Negative Rate) – reflects the frequency with which the classifier makes a mistake by classifying pathological state as normal

FNR = FN / (FN + TP)

- SE (sensitivity) – reflects the classifier’s ability to detect members of the positive class (pathological state)

SE = TP / (TP + FN)

- SP (specificity) – reflects the classifier’s ability to detect members of the negative class (normal state)

SP = TN / (TN + FP)

Note that the following equalities hold:

SE = TPR 1 – SE = FNR

SP = TNR 1-SP = FPR

- ACC (Total Accuracy) – reflects the classifier’s overall prediction correctness, i.e. the probability of making the correct prediction, equal to the ration of the number of correct decision to the total number of decisions

ACC = (TP + TN) / (TP + TN + FP + FN)

There are numerous measures of classifier quality and it is necessary to select the one which is appropriate for the situation at hand.

**Example**

During churn analysis we have two classes in the data: churn and no churn. There are 100 observations in the data and

- 90 observation belong to the class labeled as „0” (the positive class), which consists of customers who decided to stop using our services or products,
- 10 observation belong to the class labeled as „1” (the negative class), which consists of customers who continue to use our services or products.

For a certain classifier we obtain:

TP = 85 (85 churning customers have been classified as)

TN = 1 (1 not churning customer has been classified as such)

FP = 9 (9 not churning customers have been classified as churning)

FN = 5 (5 churning customers have been classified as not churning)

The sum of the above indicators equals the number of observations (customers):

TP + TN + FP + FN = 100

We now calculate the total accuracy of the classifier:

ACC = (85+1)/100 = 0,86

86 cases out of 100 has been correctly classified, which is not a bad result at all.

However, let us take a look at another indicator, the TNR, which reflects the classifier’s ability to detect the negative class, or, in our case, not churning customers:

TNR = 1/10 =0,1

Only a single observation from the negative class has been correctly classified as such (i.e. 1 customer who did not churn has been classified as not churning, and the remaining not churning customers has been classified as churning!).

**Conclusion**: ACC cannot be used to determine the extent to which the classes can be identified by the classifier.

In the subsequent parts of the tutorial we will introduce graphical indicators of classification model quality. We will start with the Confusion Matrix, which is a simple way of representing classification results.

Interested in similar content? Sign up for Newsletter

You can follow us at @Algolytics

© 2019 Data Science Central ® Powered by

Badges | Report an Issue | Privacy Policy | Terms of Service

**Most Popular Content on DSC**

To not miss this type of content in the future, subscribe to our newsletter.

- Book: Classification and Regression In a Weekend - With Python
- Book: Applied Stochastic Processes
- Long-range Correlations in Time Series: Modeling, Testing, Case Study
- How to Automatically Determine the Number of Clusters in your Data
- New Machine Learning Cheat Sheet | Old one
- Confidence Intervals Without Pain - With Resampling
- Advanced Machine Learning with Basic Excel
- New Perspectives on Statistical Distributions and Deep Learning
- Fascinating New Results in the Theory of Randomness
- Fast Combinatorial Feature Selection

**Other popular resources**

- Comprehensive Repository of Data Science and ML Resources
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- 100 Data Science Interview Questions and Answers
- Cheat Sheets | Curated Articles | Search | Jobs | Courses
- Post a Blog | Forum Questions | Books | Salaries | News

**Archives:** 2008-2014 |
2015-2016 |
2017-2019 |
Book 1 |
Book 2 |
More

**Most popular articles**

- Free Book and Resources for DSC Members
- New Perspectives on Statistical Distributions and Deep Learning
- Time series, Growth Modeling and Data Science Wizardy
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- Comprehensive Repository of Data Science and ML Resources
- Advanced Machine Learning with Basic Excel
- Difference between ML, Data Science, AI, Deep Learning, and Statistics
- Selected Business Analytics, Data Science and ML articles
- How to Automatically Determine the Number of Clusters in your Data
- Fascinating New Results in the Theory of Randomness
- Hire a Data Scientist | Search DSC | Find a Job
- Post a Blog | Forum Questions

## You need to be a member of Data Science Central to add comments!

Join Data Science Central