Data Science Interview
The Ultimate Guide to Cracking Product Case Interviews for Data Scientists (Part 1)
Metrics frameworks for data scientists to ace product sense interviews
Written by Emma Ding and Rob Wang
As Peter Drucker once said: “If you cannot measure it, you cannot improve it.” Indeed, the most critical skill of a data scientist (DS) is the ability to define the right metrics for measuring the success of a product. Because of this, there is significant emphasis on Business Case interviews (also known as Product Sense or Metrics interviews) for data scientist roles, as companies wish to identify candidates who have strong product sense and are able to help stakeholders make data-informed decisions.
Many people find it difficult to prepare for and crack Business Case interviews. There are several reasons:
- They are fresh out of school or were previously employed in non-tech industries, and do not have experience working with customer-facing products.
- They feel uneasy navigating through ambiguous problems.
- Unlike for other technical subjects such as coding, statistics, machine learning, etc., there are fewer resources on which to rely for preparation.
- They lack detailed knowledge of the company’s business model.
Indeed, even for relatively seasoned data scientists, it is easy to get caught off-guard when walking through challenging business cases. In this post, we summarize the most commonly encountered categories of business cases for DS interviews (with emphasis on metrics frameworks), and provide tips for effective preparation. Please feel free to reach out to us here if you think we might be able to make your journey easier in any way!
Table of Contents
3. Conclusion
Business Case Interview Basics
When is It Asked?
The degree to which business cases are asked in Data Science interviews varies from role to role. Naturally, Product Analytics Data Scientist positions have a particularly high affinity for such sessions. Roles with an emphasis on modeling / machine learning also include business cases, but may combine them with technical deep dives.
Regardless of the role’s emphasis / track, business case questions are nearly always asked during both the technical phone screen (TPS) and the onsite interview.
What are Interviewers Looking for?
Because business cases are typically open-ended, it is not essential (and not possible) to address every single aspect during interviews (indeed, in most situations, there isn’t a single correct answer). However, interviewers do expect several qualities that are representative of good answers:
- Structure: A good answer should have a systematic approach to the problem. Answers that lack structure or sound random will be viewed as red flags.
- Comprehensiveness: The solution covers all important aspects of that problem. There is no need (and it is not possible) to cover all aspects, but the most pertinent ones should be addressed.
- Feasibility: The solution should be practical enough so that it could be implemented realistically. Creativity is good, but not all novel ideas will be appreciated.
Some larger tech companies (such as Facebook, LinkedIn, and Twitter) may also evaluate whether a candidate exhibits basic familiarity with their products, but most companies do not have such a requirement.
In addition, evaluation criteria for business case interviews can vary from interviewer to interviewer. Specifically, depending on whether the interviewer is an individual contributor (IC), manager, or cross-functional partner, there will be different expectations.
- Data Scientists (ICs): Will typically put more emphasis on technical expertise, such as how to select the right metrics, technical details on hypothesis testing, and dealing with pitfalls in experimentation.
- Data Science Managers: Will typically evaluate candidates’ ability to drive decision-making processes and deliver effective verbal and written communication.
- Product Managers: Will typically value the ability to think practically and strategically about the bigger picture, including how to improve open-ended product and business problems.
Product Development Fundamentals
Before diving more deeply into business case interview specifics, we make a few quick remarks about the product development process. During such a process, data scientists play a critical role in decision making, alongside stakeholders such as engineers, product managers, designers, user experience researchers, etc. Because of this, all business case interviews will revolve around one or more of these product development stages. End-to-end discussions covering multiple stages are also quite common. The product development lifecycle can be summarized as follows:
Stage 1: Coming up with initial product ideas.
Stage 2: Selecting ideas:
- Quantitative analysis to select a subset of ideas to which to devote resources, often referred to as opportunity sizing.
Stage 3: Experiment design:
- Involved with selecting success and guardrail metrics, running sanity checks, choosing randomization units, etc.
- Candidates will need to consider alternatives when it is not possible to run A/B tests.
Stage 4: Making a launch decision:
- Making scientific decisions based on experimentation results.
- Diagnosing problems and evaluating tradeoffs.
Throughout the product development lifecycle, working with the appropriate metrics is of paramount importance, and metrics are indeed the most important component of business case interviews for data scientists. We elaborate on metrics in greater detail in the next section.
Metrics Frameworks
At most companies, business case interview questions for data scientists are adapted from real-world problems that the company faces. In this section, we provide an overview of the taxonomy of metrics as well as several different metric frameworks.
Taxonomy of Metrics
Success metrics (sometimes called a “north star metric” or “primary metric” or “OKR metric”) are used to measure the success or health of a product. Commonly used success metrics include: Daily/monthly active users, bookings / purchases, revenue, click through rate, conversion rate, etc.
Guardrail metrics measure core quantities that should not degrade in pursuit of a new product or feature. Examples can include: bounce rate, cancellation / unsubscription rate, latency, etc. Success metrics of other core product teams can sometimes also serve as guardrail metrics.
Good metrics are key to measuring business success. We summarize the qualities of good metrics through the following 3 characteristics:
- Simple: Easy to understand and calculate, and people should be able to remember and discuss it easily.
- Clear: The definition is explicit and there is no ambiguity in interpretation.
- Actionable: The metric can be moved by improving products, and not easily gamed (making you feel like you are getting results even though it offers no insights into actual business health or growth).
Although companies and teams vary from place to place, there are several “metrics frameworks” that cover the vast majority of business cases. We provide some discussion on two general frameworks followed by domain-specific frameworks.
General Funnel Metrics
At a high level, a funnel metric framework refers to any family of metrics that tracks the “user journey” through various parts of a product. As suggested by the term “funnel”, each metric describing a particular step of the journey is a refinement of the previous steps (and the number of customers becomes fewer and fewer through the funnel), examples of which include:
AARRR growth metrics framework (see details in Growth Metrics section below): A user who is activated must have been acquired (through some channel or organically); a user who generates revenue must have been previously activated.
“Conversion funnels”: For tech companies, there is usually a standard “conversion path” through the product, captured through basic metrics such as:
- Number of visitors to webpage;
- Number of logged in users;
- Number of users who click particular parts of the logged in pages;
- Number of users who visit the checkout page;
- Number of users who purchase.
In a B2B context, other relevant metrics may include:
- Number of visitors to webpage (leads);
- Number of leads who request free trials;
- Number of leads to which the Sales team proactively reaches out;
- Number of paid customers (each of which is a company / business) who made recurrent purchases.
Funnel metric frameworks are particularly favored by business case interviewers because they allow interviewers to probe deeply into a particular applied setting, and each candidate answer can easily generate numerous follow-up questions.
Input-output Metrics Frameworks
An input-output metrics framework revolves around two key concepts:
- Input / driver metrics: Metrics that track the activities / resources used to work towards an outcome;
- Output metrics: Metrics that demonstrate the outcome of an initiative.
While input-output metric frameworks are quite commonly adopted by companies, it is unlikely to encounter interview questions that explicitly call out this framework by name. However, formulating answers to business cases using this framework can be quite advantageous, as it offers better structure to an otherwise potentially unorganized thought process. For example, in the setting of search and recommendation:
- Input metrics may include various measures of user engagement, such as click-through-rate (CTR), as well as average time spent on a particular types of content;
- Output metrics may include average time spent on the platform per user, sessions-per-user, successful sessions per user, and advertisement revenue.
In the context of fraud detection:
- Input metrics may include the true / false positive rates of fraud rules and models, as well as operations manual review volumes and customer transaction amounts;
- Output metrics may include fraud loss volumes, losses prevented, revenue, etc.
Domain Specific Metrics
In this section, we go through a few domain-specific metrics to give you a better understanding on real word applications and several tradeoffs.
Growth Metrics (AARRR)
Growth teams are responsible for enlarging a product’s user base and keeping current users engaged. A well-known metric framework is AARRR. It consists of 5 parts:
- Acquisition: Getting customers to sign up for a website or product, which requires driving product awareness (often known as “top of funnel”).
- Activation: Getting customers to gain basic familiarity with the product, and appreciating the basic product value.
- Retention: Getting customers to come back to the product on a regular basis; a customer that exhibits no (or minimal) activity over some predetermined time period is known as churned (the precise definition depends on the business context and varies greatly across teams / companies).
- Referral: Getting customers to share the product with others, in turn further growing the user base.
- Revenue: Getting customers to adopt one or more paid features that generate revenue for the company; also known as Monetization.
If faced with a business case pertaining to this framework, there are several things to keep in mind:
- Tradeoffs between acquisition and revenue: How should a company strike the optimal balance between revenue maximization and acquiring new customers (the latter can always be done via expensive campaigns, such as providing large discounts / gifts for new users, etc.)?
- Different levels of engagement: Typically, a company will have multiple tiers of user engagement metrics, which reflect different levels of product engagement (with “activated user” being the lowest tier).
- Retention and churn cannot be easily estimated over the short term: Definitions of retention and churn vary from company to company, but typically require weeks if not months of continuous observation. Furthermore, it is possible for a churned customer to recover (often known as resurrection), so it is important to clarify the precise metric definitions and assumptions before answering any questions (e.g. first time churning, most recent time churning, ever churned, etc.).
- Difficulty of measuring incrementality for referral initiatives: When new referral programs are launched, network effects are typically introduced because any new customer can be simultaneously a referrer and a referee.
- Differences between B2B and B2C settings: In B2B companies, acquisition/activation/retention etc. are typically defined at the “company / business” level, because a customer is a company / business. There could be analogous granular user-level metrics as well.
- Challenges of international expansion: These include additional costs associated with localization (adapting a product appropriately to better fit the customers in a particular market), regulatory / compliance requirements, understanding the competitive landscape, etc.
Platform Metrics (Customer Support, Trust and Safety, Payments, Infrastructure, Operations, etc.)
Although Growth lies at the core of a business, “platform teams” that carry out operational support (related to customer help centers, payment flows, fraud detection, infrastructure, etc.) are no less essential, particularly as a company scales. Indeed, effective platform support is often an indirect driver of growth. The metrics of platform teams are also equally rich as those of Growth teams, examples of which include:
- Costs of operations / infrastructure: A common example here revolves around understanding the average cost of a help center ticket / escalation, which is an essential quantity to track for many types of product changes in customer-facing companies. Similarly, a granular understanding of various infrastructure costs allows one to effectively evaluate tradeoffs for expansion.
- Success / failure rates: In the Payments context, this can refer to the percentage of transaction attempts that succeed / fail out of all transaction attempts. These metrics are strongly correlated with conversion, particularly in the Monetization context (since payments are typically the final step of the monetization funnel).
- False positive / false negative / true positive rates (for fraud detection): When working with rules and models for detecting bad user behavior, there are always costs associated with incorrect predictions, such as the loss of a good customer’s LTV associated with false positives, and fraud losses associated with false negatives.
- Fraud loss metrics: As a company scales, fraud losses typically follow from a mixture of “born bad users” (users who abuse the product for inappropriate financial gains) and third party fraud (users who take over the accounts of good users). The percentage of fraudulent users is typically very small, but can nonetheless result in large financial losses.
- Vendor costs: Platform teams frequently partner with external vendors who provide relevant services (e.g. partnering with Stripe for payments processing); these costs need to be carefully weighed against other core business metrics.
Other Business Domains
A few other prevalent business domains include Finance, Pricing, Logistics and Supply Chains, Marketing and Sales, and Search Ranking. Each has its own commonly used metrics frameworks for tracking business performance and related tradeoffs. We recommend curious readers to explore further on their own, both through readings and through interactions with data scientists working in these areas.
Conclusion
This concludes the first part of our guide to Cracking Business Case Interviews for Data scientists.
Thanks for Reading!
If you like this post and want to support me…
- Subscribe to my YouTube Channel!
- Follow me on Medium!
- Connect on Linkedin!
- Head over to emmading.com/resources for more free resources on data science interview tips and strategies!
We also recommend the following related articles:
