Experimentation and Causal Inference

An A/B Test Loses Its Luster If A/A Tests Fail

A statistical approach to A/A tests

Leihua Ye, PhD
Towards Data Science
7 min readJul 9, 2021

Introduction

A rigorous process of experimentation, aka., A/B tests, has become trendy and widely adopted in the tech sector. As the early adopters, FAANG companies have incorporated experimentation into their decision-making process.

For example, Microsoft Bing conducts A/B tests on 80% of its product changes. Google resorts to experimentation to identify top-performing candidates in the interview process. Netflix improves personalization algorithms using interleaving, a pairwise experimental design.

The increased adoption of experimentation originates from its high level of internal validity, which is further determined by two factors. First, data scientists are selective with the overall research design and model selection at the macro level, ensuring the selected design is appropriate for the question and its key assumptions intact. Second, data scientists are careful with…

Create an account to read the full story.

The author made this story available to Medium members only.
If you’re new to Medium, create a new account to read this story on us.

Or, continue in mobile web

Already have an account? Sign in

Published in Towards Data Science

Your home for data science and AI. The world’s leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.

Written by Leihua Ye, PhD

Senior Data Scientist @ Walmart; PhD @ University of California. AI | Data Science | A/B Testing. 📧 Subscribe to my newsletter: https://techvalley.substack.com

Responses (2)

by the following formula:

--

--