The world’s leading publication for data science, AI, and ML professionals.

The Incentive Problem in Data Science

Understand why you love or hate your job as a data scientist

Photo by Carl Heyerdahl on Unsplash
Photo by Carl Heyerdahl on Unsplash

Over the last few weeks, I’ve been mulling over a decision of whether to take a new Data Science position or leave the field altogether. In trying to choose the best option, I didn’t find myself thinking about "where I wanted to be in 10 years" but rather was looking back 10 years, to the time I was introduced to Greg Mankiw’s 10 principles of economics. Specifically, his fourth was top of mind.

Principal #4: People respond to incentives

In mulling over this tenet and how it related to my decision of taking a new job in data science, I was left with the following question:

What are my incentives as a data scientist, and for the data scientist role more broadly?

Disclaimer: For the purposes of this article, what I mean by Data Scientist is someone who has in their job title "… works with stakeholders to influence our most impactful business decisions." I’m talking about the roles (Analyst/Expert Analyst/Statistician/Data Scientist) outlined by Cassie Kozyrkov.

In order to answer this question, I mapped an incentive framework for a data scientist (from my perspective; your mileage may vary).

Each of the five incentives outlined in this framework will vary in two dimensions: prominence based on where you work and by importance based on personal preference. As part of this exercise, I ranked the relevance and importance of each incentive to me and my current workplace from 1 (low) to 5 (high). Readers considering a job change might find this exercise useful as you go through the proposed framework.

An Incentive Framework for Data Scientists

Answering the Question (prominence: 3; importance 3)

Data scientists who once upon a time were first to shoot their hand in the air during class will understand this incentive (I admittedly, was one of these students). If you’re driven by this incentive, you’ll be more inclined to enjoy the ad-hoc analysis or dashboard creation workflow. The problem with this incentive is that it’s often short lived. Once you answer the question, it’s gone. So if you’re incentivized by this feeling, and the rate at which you get ad-hoc questions is low, your current role might not be a great fit.

Getting Buy-in For Your Argument (prominence: 1; importance 4)

The data scientist who gets a kick out of the story-telling aspect of their job is most likely to be driven by this incentive. I get motivated by the feeling of getting buy-in, however my work doesn’t that often focus on communicating a data-driven argument with the purpose of influencing a specific decision. Another nuance here is that sometimes a data scientist will be motivated by this feeling independent of the context surrounding the argument, while others might need to be truly interested or invested in the context of the argument.

For example, if I don’t feel motivated by the idea of increasing the click-through-rate of an email, I might not be incentivized enough to develop a strong argument for making aesthetic changes to an email to increase its click-through-rate.

Raise/Promotion/Not Getting Fired (prominence: 4; importance 4)

These extrinsic motivations are not unique to a data scientist, but I argue that their power to influence a data scientist is diminished relative to other types of roles. There are a couple of reasons for this.

First, data science is in a nascent stage, and as such, the career trajectory isn’t very clear. The two common tracks out there are "people managers" and "individual contributors" (ICs). There are certainly more changes in job function associated with moving from an IC to a people manager, but once you’re managing a team of data scientists there isn’t a clear next wrung up in the career ladder.

And this is related to my second reason for the muted effects of extrinsic motivators for data scientists – these teams are usually, (but not always) less directly influential on a company’s revenue, so they tend to be smaller and have fewer seats in leadership.

Building stuff… Mostly For Yourself (prominence: 5; importance 2)

This incentive will be most strongly felt by the data scientist who is driven by an innate academic curiosity. It can be fun to solve the small problems that come with building a predictive model or a big dashboard. Think of the inherent joy that comes from chopping up ingredients for a home cooked meal. Now imagine this scenario: after prepping a delicious meal, forty-nine out of fifty times the entire thing goes into the trash, and only one in fifty times will someone enjoy the fruits of your labor. How excited would you be to chop up those veggies?

Photo by Max Delsid on Unsplash
Photo by Max Delsid on Unsplash

Data science work is hardly ever as simple as chopping up some vegetables, it can require finding solutions to complex technical, practical and philosophical questions. Consequently, this incentive’s power doesn’t just depend on the individual, but also the types of problems they’re solving.

Doing Things for Others (prominence: 4; importance 4)

This motivation can be both intrinsic and extrinsic. It feels good to know you’re helping someone else out. Putting together that visualization they needed, or building a lead scoring model to help the Sales team win more deals (and boost their commission!) – it just feels good when its delivered. Similar to helping your friend move out of their apartment, it can be a drag in the midst of it, but it just feels good to know you helped. Again, this isn’t unique to the data science role, in fact, I think this incentive is inherent to working on any team. However, at this moment in time data scientists are more often than other job functions in a position to provide services to others given the dearth of data skills coupled with the ubiquity of data in every part of a business.

The Principal-Agent Problem

With this incentive framework in mind, I better understand why articles pop up like Here’s why so many data scientists are leaving their jobs (52K claps on Medium!). The field isn’t attracting the right people, to solve the right problems.

It’s a classic principal-agent problem that’s exacerbated by asymmetric information. Okay that was a jargony – let’s break that sentence down starting with the asymmetric information component.

Asymmetric Information

Data scientists enter into a job with an (almost certainly wrong) expectation of what their work will look like. The folks working alongside a data science team will have another (also likely wrong) expectation of what the data scientist will be doing.

The Really Bad Principal-Agent Problem

The data scientist with their set of expectations will act according to their own incentive structure (e.g some distribution of the above five incentives). And the team’s that data scientists work with will also have their own set of incentives, which can be varied (think incentives of product managers vs. a sales rep.) When the incentives of the two parties aren’t aligned toward the same goals, we get missed opportunities to create business value, frustrated teams and unhappy employees all around.

If you’re a data scientist, this might help you better understand why you love or are frustrated with your current role. I urge you to go through a similar exercise if you’re unhappy at work. You might find that your incentives are grossly misaligned, and perhaps, it’s time for a change, whether that be a new company or even a new job function all together.


Related Articles