The rise of the machine learning engineer
The machine learning engineer is the profession missing to take us to the AI future. Over the years with every technological shift, new opportunities emerged and a new type of profession was needed to fulfill the new needs.
These changes had a significant impact on how companies used to work, and right now we are in the middle of another change.
Here I take you on a journey through time, showing how some of these major shifts changed companies over time.
The Ford model
The truth is that Ford never actually said this quote. But it helps me make a point here. This model is all about the big idea and solving real problems. When Ford released the Model T, he was addressing traffic jams (horses!), traffic accidents, and making commuting affordable.
This model works well when you have great insight and people want your solution for the problem. But it is risky because you could say it is like throwing darts in the dark and expecting to win.
Ford did not have any data or any group of people demanding better cars. During his time, cars were an expensive/unreliable gadget. His success can be attributed to his vision and execution. He was right about cars and most of the world was wrong.
The main limitation of this model is that it is not often you can be so right about something and the rest of the world be so wrong. A modern example is Steve Jobs. Just google to see how many people wrote that it was a terrible idea for Apple to release a phone and that they should fire Steve Jobs for dragging Apple down.
To increase your chances of success, something else needs to be added to the mix, which leads us to the next model
Marketing research
While Ford was still alive, the world went through a transformation. One of the most relevant technological shifts for business was telecommunications. Telephones, radios and, telegraphs were all invented or available by the end of the 1800s or early 1900s. But like cars, they were mostly expensive gadgets and not accessible to the average person. As this started to change and communication started to become affordable, the sharing of information started to explode.
One of the consequences was that now you could hear about how a competitor is doing in another state or how your product is being received in another country. Data can now be centralized, and once it was available people started to see patterns and develop the first few theories in marketing. That is the beginning of focus groups and consumer theories.
This creates a need for a new type of profession. During those days, the way people made sense of patterns seem in the data was by bringing professionals with a background in math but also psychology. Don’t forget that this time was when Freud was around, and the idea of understanding and later manipulate people’s subconscious was appealing. This new type of profession will be the beginning of what would be today’s marketing departments. Learning about persuasion tricks to target groups of people is a routine in marketing classes now.
You can now learn from other people’s mistake, but more importantly, you can now learn about specific demographics and user segments before releasing your product.
If you were a company in the 30’s you would be getting a report from a focus group saying that customer would welcome your product in different colors. And after that, you would make a small rollout of a ‘limited edition’ of your product to gauge interest. But if you want to improve your sales after release, there isn’t much left besides invest in radio advertising. This is still a “one to many” communication type, most of the data available from costumers are high-level aggregations and from months if not years ago. It is just unthinkable to have real-time data or have more granular data about customer behavior. But this will change with the next technological shift: computers
Forecasting era
This is the era that started after World War II and we are currently living the end of it. Computers started to become a thing in companies and governments after the war. The immediate benefit was the ability to do the work of a floor full of mathematicians in a matter of seconds. Before computers, you would usually have teams of people dedicated to crunching numbers for hours if not days just to give a yes or no answer (check ‘The imitation game’ and ‘Hidden figures’). Thanks to computers you can now afford to run multiple scenarios on your data, and be able to handle as much data as you can afford to gather.
Later when the Internet became mainstream, real-time data became available. You can now for example track how many people are accessing your website as they browse. But with every technological shift, a new type of profession need to emerge to take advantage of all the new possibilities. This is the data scientist, also known in wall street as the ‘quant’.
This new profession has a strong grasp of the product with a strong background in math. The data scientist can take advantage of huge amounts of data by slicing it any way it is needed to fit complicated models and also forecasting. This new type of profession makes it finally possible to learn anything about the customer that is there to learn, it is only a matter of ‘do we have data for that?’. This is a big departure from qualitative methods (psychology to understand costumers) to quantitative methods ( ‘this is what the data shows’)
Compared to the previous model, the biggest difference is the live-ops cycle (steps 3,4,5). Because there is more data available, and faster, you can now react faster too. Think of a website that is released on a Monday and thanks to live-ops, you can adjust your content through the week. All based on user behavior/feedback. Broken link? no problem, this is captured and fixed in a couple of hours.
But you do not need to be a digital company to take advantage of this model. Think of Hollywood, have you noticed how some actors tend to be together in the same movie? How some actors tend to be cast to similar roles over and over again? There is a huge industry in Hollywood to forecast what do people would like to see. If you run a TV show, you can track how each episode is performing as they get released and also make a decision about each character’s fate based on social media buzz.
But this model still has limitations. There is now abundant data arriving every second, and the world is changing very fast around us. Computers can crunch all this data, but people still need to go home and sleep. To get an edge ahead of the competition, you need to be faster. And we can already have a glimpse of how the future model will look like
Age of AI
Compared to the rest of the world, some companies are already living in the future. Your Netflix recommendations get updated as you watch more shows, your google results are always current with world events and within your geographical context, Amazon recommends new products based on your history and other users behavior. You could still power all those businesses using the previous model. But that would require an insane number of people aggregating data, getting insights, and creating a curated list for every product mentioned. And by the time you are done, you need to start all over again. Because the last recommendation is already stale.
All those products have one thing in common behind the scenes: machine learning. And here how it adds to the previous model:
The difference is that now an AI is running the product based on user feedback, there is no time to wait for people to notice a new trend in the data. Speed and scale matter a lot. But the data scientist still has a role in this model, someone still needs to bring business insights about what the AI is doing. In Netflix, people could be having the following conversation:
Person 1: We noticed that people like Christmas movies way more than we anticipated.
Person 2: Yes, the AI keeps pushing more and more of those movies. But we eventually run out of them
Person 1: We should try to license as many of those movies as we can. Why don’t you crunch the numbers to check if we should also invest in making some of that in-house? This would save us a lot of money
Notice that all the business gains for recommending the right content never had to come from a curated list. The AI managed to distribute the current catalog the best way possible. The data scientist comes in later only to bring deeper business insights, that will later help the AI by making available an even better catalog.
AI is the current technological shift, and not all companies are ready for the transition. The new profession that will help in this transition need to be fluent in machine learning but also cloud computing. This is a machine learning engineer.
The future
If you make a quick search online for machine learning engineer on Linkedin, you will notice that the majority of those positions are posted by FAANG companies. This is an interesting correlation.
What makes the machine learning engineer so important is that machine learning models in production need to scale. One thing is to train a model overnight with a specific batch of data. Another thing is to deploy a model that feeds on streaming data and can transform data by reading from multiple data sources at scale for millions of users. That is no easy tasks, that is why most of today’s models are stuck in some Jupyter notebook somewhere, which is seen as a barrier by the backend team. Because they do not know how to take that and integrate into their services. The machine learning engineer can break this barrier by being a hybrid of both worlds. And that is why machine learning engineer is the profession that will take us to the AI future.
