Launching and Scaling Data Science Teams: Business Stakeholder Playbook

Best practice playbook interacting with your data science team

This post is intended to be a best practices guide to interacting with a data science team. This section picks up from my summary on Launching and Scaling Data Science Teams, and contains some overlap with the Data Science IC and Data Science Manager playbook. I am targeting a business stakeholder (PM, ops director, business analyst, CMO) that depends heavily on their data science team heavily, but is having trouble communicating with individuals and integrating with the team’s workflow. This post is for people asking the question: “I know I need better results out of collaborations with my data science team at ___________, but I don’t know how to get them, or what I can do to improve.”

The four sentence summary: Know what’s possible and what you’re asking for. Force alignment on “business impact” across the company. Develop business intelligence skills. Promote data culture.

Know What’s Possible and What You’re Asking For

(This section has a lot of overlap with the “Source your Own Problems” section of the data science IC playbook)

I asked my interviewees: “What frustrates you the most about the teams you collaborate with?” Business stakeholders and IC data scientists frequently gave answers that were two sides of the same coin. From stakeholders:

• “It’s hard to understand what they’re doing or how it works, hard for them to communicate what they work on.”
• “There’s a lack of consensus on what’s a two week vs. two month project. Everything is overpromised and underdelivered.”
• “They give an academic answer instead of a pragmatic.”
• “We talk over each other a lot. Five weeks can go by and we’re still grappling with data pipeline issues.”

From data scientists:

• “The stakeholders have no idea what data sets do and don’t exist.”
• “They don’t understand the complexity of the work and why things take as long as they do. When we set up a test they don’t realize that has to get on three different teams’ roadmaps, and they don’t understand the test metrics we give them.”
• “We consistently get non-scoped problems”
• “They don’t know what tech debt is or why we sometimes need to clean it up. Everything is ad hoc for them. They can’t interpret data science results.”
• “Nothing is organized in a way that gives us visibility”

To grossly oversimplify: stakeholders are frustrated that they can’t understand their data scientists, and data scientists are frustrated because they feel misunderstood. Again, the XKCD reference: “In CS [data science], it can be hard to explain the difference between the easy and the virtually impossible”. As a non-technical stakeholder, it’s critical for you to be aware: unless you put the work in, you will have no idea what your data science team is capable of. An example: I spent about six weeks writing a recommendation algorithm for Wayfair US that moved from an “actual” to Bayesian “actual vs. expectation” performance. The Wayfair Germany team didn’t think it was worth asking for the same algorithm — assuming it would be overkill for a much smaller part of the business. Building this algorithm for wayfair.de, in fact, required changing WHERE storeid=”wayfair.com” to WHERE storeid=”wayfair.de” in four lines of code (or something like that). If you’re a business stakeholder, you have to know enough to know what’s trivial to ask for.

The flip side of this: data scientists and stakeholders agree that there’s not visibility into what will kick off a “chain” of asks. Your team should be able to come to a conclusion quickly if it’s based on existing data. Other asks are much more difficult — if the data doesn’t exist, that might require work from the front-end team to collect, the DBAs to write, the the ETL team to add to a pipeline/clean/standardize. An A/B test will required even more collaboration with storefront and all other teams slotting their own tests. Understand what data sets exist and what dependencies your data science teams face in response to your requests.

The pragmatic/academic tension comes up a lot. To be sure, it is the responsibility of the data scientist to produce work that is understood and valuable to you. That being said, your technical bound limits the work your data scientists can do. If you depend on data scientists, you need to develop a minimum competency for understanding what they build and how they evaluate success. Regression, classification, clustering, mean squared error, f1 score, cross-validation and p-values don’t need to be second nature to you, but you should have enough familiarity to understand why your data scientists built what they did. One stakeholder to their DS team: “Don’t come up with a recommendation that I don’t understand and don’t agree with.” Develop your background to understand your team’s recommendations.

Force Alignment on “Business Impact” Across the Company

(This section has a lot of overlap with the “Be Intentional with Your Prioritization Process” section of the data science manager playbook)

“Know What’s Possible and What You’re Asking For” fills in your blind spots about what your data science team is capable of, what’s easy, what’s almost impossible. Recognize that you possess asymmetric information as well, you know the business context. To align your data science team with the company, you must fill in the blind spots they have about the value of their work. This must be a stakeholder led conversation; if your teams can’t align on the “business impact” of data science projects, you can’t even attempt to allocate data science resources efficiently. From an SVP of Analytics: “You need to go through the exercise to translate anything into a common currency. You can turn anything into return on investment, the challenge is to weed out bullshit business cases and assigning projects because it’s someone’s turn for data science resources.” Repeating from the data science manager playbook, prioritization can be somewhat political (“It’s a small enough company to know if you’re overusing or underusing data science resources”) to highly political (“Every team we support gets equal access to our time”). Your company needs to prioritize based on return on investment (in this case, data scientist time). You must know how to estimate the required data science resources, and provide a true apples-to-apples business impact of the outcome in a way that can be compared to others’ potential projects.

Include your data scientists in this conversation. By bringing the DS team into the room during prioritization talks, you can fill in the gaps in their business context, allowing them to source their own projects in the future. The best-functioning teams that I interviewed had this in common: projects could originate in a meritocratic/bubble-up manner, and it was expected of data scientists to suggest things to work on. People in great organizations figure out how to remove each others’ blind spots.

Additionally, share the results of past projects with your team. Data scientists can feel like they work hard on a model, implement it successfully, and then: radio silence. Give your data science team a great understanding of what was happening without the solution. They create a piece of something important and often don’t realize the success that they enabled. In fact, send the results as far upstream as possible. When a business stakeholder spreads credit for a dollar denominated win to the data science team that built the model, the ETL engineers that built the pipeline, the DBAs that stored the code, and the front end engineers that set it live on site, the reaction is contagious. This is the type of communication that gets everyone excited and ready to enable the next win.

Develop Business Intelligence Skills

If the “Know what’s Possible” section is about understanding what your data scientists are capable of, this is about helping them (or at the very least, not bothering them) with everything else. Business Intelligence means having the technical firepower to pull data in SQL and perform basic analysis yourself, and being able to self-service factual questions around reporting. A consistent frustration of data scientists is lack of analytical ability from stakeholders: the lower the bar is, the higher the hurdle that needs to be cleared to get buy-in on a solution.

This article is written for the “non-technical” business leader. I’ve seen too many stakeholders lean into that phrase like it’s a badge of pride. Don’t take “non-technical” to be set in stone, there are different gradations of non-technical manager, you and your team should strive to improve your technical acumen every day. Your team must help iterate on solutions with your data science support, and they can’t build complex solutions to a problem if you can’t pull the data from SQL to confirm it’s working.

A data scientist I interviewed has worked at a few places, and loves his current role because no one is asking, “Can you help us pull this?” He can ask the business analysts to get the data themselves, and everyone agrees on the input — a specific SQL query from a database as the ground truth. Another DS manager feels pulled in the opposite direction, she constantly feels a part of her job is protecting her team from ad hoc reporting asks that should fall to another team.

Perhaps the easiest way to justify developing these skills is by looking at the project backlog and the salaries in your data science department. High caliber data scientists are expensive. Don’t make them do your ad hoc reporting — free them up do the work you hired them for.

Promote Data Culture

Many of the people I spoke with for these articles are former Wayfair employees who have joined other tech companies. One of the surprising observations they share is how much you can tell the DNA of a company by the founders. Most of us felt like Wayfair is very much run by engineers that are trying to figure out how to sell furniture. Airbnb is famously product driven. Other firms are more high level business/marketing oriented. One firm I spoke has a deep legacy of investigative journalism, another ends meetings by asking how this will make the customer happier. A massive positive predictor of how successful a data science team will be is the “data culture” of a company: to what extent conversations are based on, and decisions are made with analytic backing.

I have sat in meetings at companies on both sides of this spectrum. At companies with good data culture, any business suggestion required either data to back it up, or a plan to generate that data. At another, someone would code a phenomenal, evidence backed solution to a major problem, but the pushback would be, “I’m not sure what the CRO is going to think about that — we need to make something we can explain to him in 30 seconds, otherwise it’s too complex.” You must build an ecosystem where decisions are factually driven, as opposed to personality driven, and information is constantly sought out and leveraged.

Many of the companies I spoke with were born with good data culture. However, some of the the most illuminating conversations I had were with a company that lacked data culture: the COO came in and found an old school company with the mindset: “We’ve been here fifty years and never needed data scientists.” Creating data culture was especially hard because the company had been doing well, and huge confirmation bias existed on past successes. Their DS team was hired more in response to actions of competitors than out of an internal desire, and they have subsequently been pinballed around the organization. The COO shared two lessons: the data science team had to be invited into the meetings, and those that were the most excited to receive data science insights were already those most capable of self-serving. What he needed most in his stakeholders: “Someone who can help me make business cases for things, and be willing to take on and support a data science team. Help build the coalition of the interested, assembled across the company.”

At a company without data culture, you need a first believer from the technical side. From that COO’s perspective, a first stakeholder willing to include and interact with data science is incredibly necessary. “You spent the time to be technically savvy on your computer, this is a new thing that you need to add to your skill set. There’s no forcing anyone, but it will eventually happen via adoption or replacement.” Do your part to enable data culture at your company, because it’s happening one way or another.