Machine Learning Rules in a Nutshell

A few days ago google engineers posted a huge manual on how to build great ML products(and, by the way, a new TensorFlow on GCP specialization on coursera). These guys have a lot of experience in developing large-scale projects in production and they have written down their best practices. The only problem — this doc is really huge, so I’ve decided to summarize the key points here. There are 43 rules in total, they are going roughly in the order of development stages and divided into 3 phases. Let’s begin.

0. Do not talk about Artificial General Intelligence. ML engineers may get you wrong and decide that you’re schizophrenic. Just kidding.

Phase 1: First Pipeline

1. Do not hesitate to launch a product without ML. If a simple heuristic or manual work can get the job done — put ML aside until you’ll get enough data.
2. Add metrics. Then add more metrics. Track everything to see how each part of your app does it’s job. When you have a problem or simply not sure — add a metric.
3. Do not invest too much time into a heuristic. Once it becomes complex — switch to a ML model. It would be easier to develop and maintain later.
4. Focus on infrastructure first. Build a pipeline with a simple model and make sure the whole thing works well.
5. Add tests to every piece of your pipeline. Test your infrastructure, test your data, test your models.
6. When you’re copying an existing pipeline for a new project, make sure that you are not throwing away potentially useful data.
7. Do not simply throw your heuristics away. They might generate useful features for your ML model.
8. Keep the model fresh. Track how much your model’s performance degrades with time and make sure it won’t become rotten in production.
9. Track your model’s performance before rolling out in production. Especially when it’s a user-facing model.
10. Test your data, again. Make sure that all features are gathered and data is fine.
11. Create documentation for features. Where they come from and why they are useful.
12. Choose the right objective. It must reflect the performance of the main goal of your project.
13. Picking the right objective might be hard, so choose a metric that is easy to measure and is a proxy for the “true” objective.
14. Simple model is easier to debug. Linear or Logistic regression might be a good choice.
15. Separate spam filtering and quality ranking. Such models are often used in different ways.

Phase 2: Feature Engineering

1. Plan to iterate. Your features and your model will evolve with time, so make sure you’re ready to update it.
2. Start with directly observed features. And leave learned features for future updates.
3. Try out features that generalize across contexts. Metrics like # of likes may be useful in recommendations, quality ranking and other models.
4. Don’t forget to add very specific features. Even when they don’t generalize well, they still have value. If they apply only to a small # of examples, regularization will filter them out.
5. Hand-craft new features from existing ones in human­-understandable ways. And try to not overthink them.
6. The number of feature weights you can learn in a linear model is roughly proportional to the amount of data you have. With only about 1000 examples — use a few dozens of features. With more than a billion — 10m features is enough.
7. Remove unused features from your pipeline. If you found that some feature does not improve your model — get rid of it.
8. Don’t let engineers do the UX testing. Do it through a crowdsourcing platform, or through a live experiment on real users.
9. Track how much models differ. Before testing a new model further, calculate how much it’s outputs differ from the previous one.
10. Focus on utility of your model, not predictive power. A new model may have higher accuracy, but how useful it is in practice is more important.
11. Look at your model’s errors and create new features to fix them. Try to find patterns in errors and add respective features.
12. Create metrics to track these error. Optimization becomes easier when you have metrics to optimize.
13. Model may have very different behavior in the long-term. Even when in short-term it looks exactly how you expected.
14. Gather data live and use it to test model during training. This way you’ll be sure that model will perform well after deployment.
15. Importance-weight sampled data, don’t arbitrarily drop it.
16. Don’t forget that data itself may change between training and serving. Sometimes, however, you can deal with it using hourly/daily/weekly snapshots.
17. Share the code between training and production pipelines. As much as possible.
18. Use earlier data for training, later data for testing. This will help you roughly estimate how your model will perform in the future.
19. Hold out a small % of examples in binary classification tasks to get clean training data. Show 1% or 0.1% of these to users to get new training data.
20. Regularize general features harder and allow only positive weights in ranking tasks. Don’t let your ranking system become too biased.
21. Avoid feedback loops with positional features. Separate them during training and avoid them during serving.
22. Measure how your model performs on training, validation, testing and live data. Track how much they differ and try regularize model more to reduce the difference.

Phase 3: Refinement and Complex Models

1. Revisit your objective. Make sure that it is aligned with your product goals.
2. Launch decisions are a proxy for long-term product goals. Usually a single decision affects multiple metrics, so don’t rush into quick decisions.
3. Keep your base and ensemble models separate. Base models only take raw inputs, ensemble only take outputs from base models.
4. When your performance stopped growing, look for new sources of features. Don’t focus on existing ones for too long.
5. Diversity, personalization, and relevance are important. But not as much as popularity.
6. Your friends tend to be the same across different products. Your interests tend not to be. We tend to have the same connections, but our goals change.

I think it’s nice to have these guidelines while working on ML project. Because if you won’t, you’ll have a good chance to get into technical debt. So, don’t forget to bookmark them and revisit from time to time during work.