Why we moved away from Conversion Rate as a primary metric
The impact of violating assumptions of independence on AB Tests and changing the primary metric for your company
Changing the primary metric for any company is not a straightforward thing to do. Why you may ask? There are dozens of reasons but here are some of the ones that come to my mind:
- The company has been using that metric for a long time
- Forecasts have been made from historical values
- Team OKRs include targets against it
- People understand the metric and the levers that drive it
- Deviations from the expected values can easily be spotted
- Causes for change to the metric can be quickly identified
This shortlist alone would cause anyone to break out into a sweat! So imagine my surprise when in my first month at Gousto, the Head of Product for our Menu Tribe suggested that our primary metric, Menu Conversion Rate (MCVR) was likely to be the wrong metric for us to be using for our experiments as it violated assumptions of statistical independence.