The world’s leading publication for data science, AI, and ML professionals.

McNemar’s Test, with Python

The Complete Beginner's Guide to perform McNemar's Test (with code!)

Photo by Sergio Rota on Unsplash
Photo by Sergio Rota on Unsplash

In this article, I will introduce the fundamental of McNemar’s test, a statistical method for comparing proportions from two dependent populations. [1]

  • It is a type of paired chi-square test (χ2), but this time, both populations are dependent.
  • Only used for paired nominal data.

    1. Before-after comparison data, to uncover any changes in perception, attitude, behavior in marketing campaigns, drug testing, etc.
    2. Matched pair case-control: i. Each case has a matching control, i.e., matched on age, gender, race, etc. ii. Twins studies, i.e., the matched pairs are twins. [2]

Similar to the χ2 test, data need to be arranged in a 2×2 contingency table before calculating McNemar’s statistic. [1]

Table 1: 2x2 contingency table for the outcomes of two tests (before and after) on the same individual sample. Image by Author.
Table 1: 2×2 contingency table for the outcomes of two tests (before and after) on the same individual sample. Image by Author.

The null hypothesis (H₀) in McNemar’s test is marginal homogeneity, i.e., two marginal probabilities for each outcome are the same (R₁ Total = C₁ Total and similarly R₂ Total = C₂ Total), if we expand the equation → A + B = A + C → B = C [1]

The test statistic has an approximately χ2 distribution with 1 degree of freedom:

McNemar's test statistics. Image by Author.
McNemar’s test statistics. Image by Author.

A. McNemar’s Test on before-after data

Japanese automobile manufacturers Toyota and Mitsubishi both aim for increasing their market share in the Middle East. In a total of 100 customers, 60 preferred Toyota, and the rest preferred Mitsubishi before they see anything. However, after sitting through a premiere screening of Toyota’s new release of TV advertisements, the same respondents were asked to reconsider the two brands again and choose which brand they now preferred and the result are as shown in table below. At 1% significance level, is there evidence that customers has changed their mind after advertisement?

Table 2: Survey summary 2x2 contingency table. Image by Author.
Table 2: Survey summary 2×2 contingency table. Image by Author.

According to five steps process of hypothesis testing: Let π₁ = the true proportion of customers who prefer Toyota before the ad screening π₂ = the true proportion of customers who prefer Toyota after the ad screening H₀: π₁ = π₂ H₁: π₁ ≠ π₂ α = 0.01 Following McNemar’s test statistics:

Conclusion: We do not have enough evidence that the true proportion of customers who prefer Toyota before and after the ad screening is not the same, at 1% significant level.


B. McNemar’s Test on matched pair case-control data

An experiement is conducted to compare the effectiveness of 2 asthma treatments in eliminating emergency room visits. 200 pairs of asthma patients are paired by severity of disease. One patient in each pair is assigned treatment A and the other treatment B. The experiement will record whether or not the patients had at least one emergency room visit for breathing difficulites. At 1% significance level, is there evidence that the 2 treatments are not equally effective?

Table 3: Experiment summary 2x2 contingency table [3]
Table 3: Experiment summary 2×2 contingency table [3]

According to five steps process of hypothesis testing: Let π₁ = the true proportion of patients with treatment A who visit the emergency room π₂ = the true proportion of patients with treatment B who visit the emergency room H₀: π₁ = π₂ H₁: π₁ ≠ π₂ α = 0.01 Following McNemar’s test statistics:

Conclusion: We have enough evidence that the 2 treatments are not equally effective, at 1% significance level

Conclusion

McNemar’s test is a type of chi-square test. However, unlike the Chi-square test which is used to test the relationship between 2 variables (χ2 test of Independence), McNemar’s test is used to check if there are any changes in perception, attitude, behavior on 2 dependent populations. The Mcnemar Test is used whenever the same individuals are measured twice (before and after survey), matched pairs (twins or married couples), matching control (i.e. matched on some variable (pair of asthma patients by severity of disease in above example).


Recommended Reading

Chi-Square Test, with Python

ANOVA Test, with Python

Two-Way ANOVA Test, with Python

One-Sample Hypothesis Tests, with Python

Two-Sample Hypothesis Tests, with Python


References

[1] The McNemar’s Test – Statistics LibreTexts

[2] McNemar’s test – Wikipedia

[3] University of Wisconsin-Madison Lecture in BMI541/99:Biostatistics and Medical Informatics, 2022


Related Articles