I am a firm believer that visual interpretation of concepts has a long-lasting retention period than the conventional approach. I assure you when you finish up reading this blog, you will establish a crystal clear understanding of Type-I & Type-II error. Let’s begin with some preamble about Hypothesis Testing:
What is Hypothesis testing and why do we need it?
Any existing belief/default position/status quo/general statement relating to a population parameter which someone is trying to reconfirm/disprove based on the principles of inferential Statistics is called Hypothesis testing.
It has two components:
· Null Hypothesis(H0) – Existing belief/default position/status quo/general statement about a population parameter
· Alternate Hypothesis(Ha) – Something contradictory to the Null Hypothesis
Since companies like Pfizer & Moderna have already disclosed the results of their Phase 3 trials of COVID Vaccine, we will take this example only to understand the concept of Type-I & Type-II error. Both companies claim above 90% effectiveness of their vaccines which is incredible, let’s see how can we use this example to visualize various terms associated with the confusion matrix.
We shall start by defining our Null & Alternate Hypothesis for COVID Vaccine Effectiveness, assuming measurement of an arbitrary metric that captures the effectiveness of the vaccine(Higher the value of this metric, more the effectiveness of the vaccine):
· Null Hypothesis(H0) – No difference(𝛿=0) in the metric value before & after the vaccination
· Alternate Hypothesis(Ha) – Difference 𝛿(>0) in the metric value before & after the vaccination
Here onwards, visualizations will be more dominant:
The above plot is the distribution of 𝛿 that assumes the Null Hypothesis is true and there is no difference reported in the metric before & after vaccination. The value of 𝛿 away from the mean value of 0 reflect the variation in sampling(due to chance/random causes).
The general practice of statisticians is to keep this significance level(α) at 5% or 0.05(in probability terms), this is nothing but the ‘Type I Error’ (Incorrectly rejecting a true null hypothesis).
The region to the left of the critical value(Red line) is denoted by (1-α), also known as the ‘Confidence level‘(Correctly rejecting a false alternate hypothesis). In the current case, it would be 95% or 0.95(in probability terms).
The above plot(Green) is the distribution of 𝛿 that assumes the Alternate Hypothesis is true and there is a difference 𝛿(>0) reported in the metric before & after vaccination. Again the values away from the mean value of 𝛿 reflect the variation in sampling(due to chance/random causes).
Focus on the green shaded area, it depicts that actually, the alternate hypothesis is true but again due to the extremity of sample representation to the left of the critical value(pre-decided red line) it is being rejected by mistake. This is nothing but the ‘Type II Error’ (Incorrectly rejecting a true alternate hypothesis).
The value of the green shaded area(Type II Error) is denoted by β.
The region to the right of the critical value(Red line) is denoted by (1-β), also known as the ‘Power‘(Correctly rejecting a false null hypothesis).
By now we have already described all that is needed for building a confusion matrix, it’s time to provide a structure to the same:
Major confusion takes place when these cells are referred to with different terms i.e. True Negative, False Positive, False Negative & True Positive. Let me share an easy to remember method with you for the same:
Referring to the above picture, ‘0’ in the column is assigned text – ‘Negative’ and ‘1’ in the column is assigned text – ‘Positive’. Now if the number in the column matches the number in the row we use the prefix ‘True’ else ‘False’.
Looking at the top-left cell of the matrix, since the number ‘0’ is matching with the row’s number, we will use the prefix ‘True’ followed by the already allocated suffix ‘Negative’. So this becomes ‘True Negative’. In a similar manner, we can derive these terms for all the leftover cells of the matrix.
The very next step is to derive performance metrics out of the confusion matrix namely Accuracy, Sensitivity(Recall), Specificity, Precision & F1-score. More discussion on the same will be carried forward in the upcoming blogs.
I will be concluding this blog here, I hope by now you have established a decent understanding of Type I & Type II Error alongside basic terminologies associated with the confusion matrix. Also, the above example represented a One-Tailed test whereas there exists a Two-Tailed test as well, an explanation of which is also on similar lines. I have/will be posting more blogs of the same domain in the future in a simplified & easy to interpret form. Keep a watch!
Thanks!!!
