Opinion
Stop Hiring “Data Scientists”
Getting AI models into production requires a range of skill sets
Want to implement an AI project? It’s more than just building a great model. This post describes the reasons why productionizing data science takes more than one person in a “data science” role.
“What does it take to get AI models to production?”
There are countless examples of companies that’ve developed models — only to have them sit in purgatory and never make it out to production. There’s been a lot of discussion on this topic in the AI community.
And it isn’t that people don’t know the answer of how to ship AI models to production. It just sucks.
Why?
It’s hard.
It takes as much work to put (and keep) a model in production as to build it.
Deploying a model often requires somebody to
— monitor availability of the pipeline
— monitor model accuracy over time
— continuously retrain the model with new data points
— version code, models, and maybe even datasets
— etc.
It’s complex.
If you’re a data scientist who knows how to tune learning rate and dropout values, do you also feel confident using DevOps tools like Terraform, Kubernetes, Slurm, Prometheus, Nomad, or Celery?
Probably not.
It’s probably not interesting to data scientists.
Some people are certainly highly interested and highly skilled in DevOps work. You can generally find those people in roles like “DevOps Engineer.”
These types of people are great at building stable, scalable, easily-maintainable infrastructure. They’re great at automating processes. They have an attention to detail that plans for their end users (developers) to perform weird corner cases. They write tests.
It’s definitely fascinating to some people, but it’s on the other side of the world from the traditional “data scientist” activities like model development. It’s a maintenance mentality versus the creative mentality of iterating towards proof of concept.
Let specialists be specialists.
Both the development and production sides of AI are complicated. And for AI (even more so than for traditional software dev), best practices are still being developed. It’s an investment to dive in and figure out the current best way of doing things.
Data scientists who’ve worked to hone their model-building skills shouldn’t have to dilute their expertise by learning their way around 100 DevOps technologies as well. A data scientist doing platform work isn’t going to be as productive as a data scientist doing data science.
Similarly, your DevOps engineers probably don’t have time to become experts in what your data scientists do. Somebody has to have deep knowledge on the infrastructure / maintenance / monitoring tasks.
Stop trying to make one person perform all the roles. We need specialists in all parts of the puzzle.
Call to Action
Companies, stop trying to hire data scientists to fill all roles.
If you haven’t seen it yet, here’s a great diagram of Hidden Technical Debt in Machine Learning from Google (from 2015, but the message is timeless).
That tiny black box in the middle is ML Code. How many job reqs do you have open for that item? How many do you have open for the other boxes?
“Data Scientist” shouldn’t be a catch-all role that implements the diverse components of an AI platform. There is no such thing as a “full-stack data scientist.”
Don’t have the budget to hire more than one person? Not an excuse. Augment with one of the many AI services companies.
Developers, stop trying to be a data scientist.
There’s such a range of interesting AI work beyond just “how to make a better XYZ model.” And those areas need help! By specializing in something slightly adjacent to model dev, you have an opportunity to shape best practices.
Education may be harder because training resources are scarcer, but it’ll be worth it because you’ll be in high demand.
(And a shoutout to educational groups: We need more material on MLOps, especially comprehensive real-world training.)
AI Startups, help make the DevOps side of AI easier.
There are a bunch of recent articles proposing AI platform architectures. They’re trying to collect the many components into a single view. It’s a good starting point, but the resulting plans are much easier written than implemented.
(A few overwhelming examples: A16Z’s “Emerging Architectures for Modern Data Infrastructure” & “Modern Data Engineer Roadmap 2021”)
It’s too complicated and too hard for the average organization to implement these 20-component flowcharts. Especially when we don’t yet have the discipline of hiring a range of specialists to build the platform.
We need to simplify the tools that manage our AI pipelines.
Conclusion
Organizations are often surprised by the effort it takes to deploy their AI models. But we’re still in the early days of deploying AI into real-world scenarios, so companies should plan for it to be an arduous task that can’t be bundled into existing roles. Create a separate work stream to engineer the DevOps side of AI projects, and let data scientists keep building models.
