Why we moved away from Conversion Rate as a primary metric

The impact of violating assumptions of independence on AB Tests and changing the primary metric for your company

Bhavik Patel
Towards Data Science
7 min readJun 24, 2021

--

Photo by Jonathan Petersson on Unsplash

Changing the primary metric for any company is not a straightforward thing to do. Why you may ask? There are dozens of reasons but here are some of the ones that come to my mind:

  • The company has been using that metric for a long time
  • Forecasts have been made from historical values
  • Team OKRs include targets against it
  • People understand the metric and the levers that drive it
  • Deviations from the expected values can easily be spotted
  • Causes for change to the metric can be quickly identified

This shortlist alone would cause anyone to break out into a sweat! So imagine my surprise when in my first month at Gousto, the Head of Product for our Menu Tribe suggested that our primary metric, Menu Conversion Rate (MCVR) was likely to be the wrong metric for us to be using for our experiments as it violated assumptions of statistical independence.

Violated the who with the what now???

--

--

Towards Data Science
Towards Data Science

Published in Towards Data Science

Your home for data science and AI. The world’s leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.

Bhavik Patel
Bhavik Patel

Written by Bhavik Patel

Director of Experimentation and Analytics. Ex-Director/Head of Analytics at Hopin/Gousto/MOO/PhotoBox. Twitter: @dodonerd

Responses (2)

What are your thoughts?