Is My Model Really Better?

Why ML models that look good on paper are not guaranteed to work well in production

Photo by Florian Schmetz on Unsplash

These days, a typical ML research paper reads something like this:

We propose a new model architecture X. We show that X outperforms SOTA by Y%. We conclude X is better than the current SOTA. Our code is available online.