Experimentation and Causal Inference
An A/B Test Loses Its Luster If A/A Tests Fail
A statistical approach to A/A tests
Introduction
A rigorous process of experimentation, aka., A/B tests, has become trendy and widely adopted in the tech sector. As the early adopters, FAANG companies have incorporated experimentation into their decision-making process.
For example, Microsoft Bing conducts A/B tests on 80% of its product changes. Google resorts to experimentation to identify top-performing candidates in the interview process. Netflix improves personalization algorithms using interleaving, a pairwise experimental design.
The increased adoption of experimentation originates from its high level of internal validity, which is further determined by two factors. First, data scientists are selective with the overall research design and model selection at the macro level, ensuring the selected design is appropriate for the question and its key assumptions intact. Second, data scientists are careful with…