How to Hire a Data Scientist
Today I’m writing about how I hired a data scientist for my company, Interview Query, by applying data science to the hiring process. My first time hiring a data scientist really opened my eyes to being on the other side of the table after spending years interviewing as a candidate.
There were three main take-aways that I want to dive into.
- Define the data science role and responsibilities. Know exactly what the data scientist will be working on.
- Evaluate candidates at scale with automated testing and rubrics, and not endless resume reviewing and hiring manager calls.
- Creating a metrics based hiring process for all parts of the funnel.
First, why do we need a data scientist?
It’s ironic that a data science startup founded by a data scientist needs another data scientist. Data scientists are useful for businesses to increase revenue by analyzing data to influence product direction, or by implementing machine learning on a large scale to increase metrics like engagement, retention, and revenue.
This is usually done on huge platforms, marketplaces, e-commerce- basically all of the things Interview Query isn’t. As a small business, our current priority for new feature development is talking to our customers, as they usually have strong opinions about what we should be building.
But as a data science education business, it seemed necessary that someone else’s entire job-besides mine-be focused on applying analytics and discovering interesting product insights and features. On top of providing educational content geared towards data science, I wanted to actually apply data science to the business to improve our users’ learning goals.
Defining the Data Science Role
If you’ve ever chatted with data science managers or recruiters, you’ve probably heard of how painful the hiring process is. They’re scanning through thousands of resumes, all the candidates they’re interviewing are getting rejected, someone gets hired on and quits within the first month- take your pick.
Honestly, in my experience, the issue can often be chalked up to a misalignment in expectations in the job description and interview process. You can’t interview a data scientist, gush about product insights, and then drill them on Leetcode.
When writing out the job description for our open position, I made sure to list a few projects they would be working on- improving our newsletter, creating an AB testing system, etc. I made it really clear why we needed a data scientist and exactly what the role would entail.
The second part of this process involves creating your own hiring strategy. Here are some tips I’ve learned over the years.
Eliminate 50% of the candidates at every step of the funnel. For example, if you’re hiring for one role and want to make two offers, you would schedule four onsite interviews, eight technical screens, and sixteen hiring manager interviews based on resumes or take-home exercises. It’s like a binary search tree, where the goal is to optimize your interview process for finding the best candidate in the shortest amount of time.
Understand the distribution of skills in the candidate pool. Once you set requirements for the job, including salary, hours, work style, and skillset expectations, you will likely receive a ton of candidates across a spectrum. To understand what a good candidate looks like, you have to see what a bad candidate looks like as well.
In other words, define your baseline requirements in a candidate that is a yes-hire. For Interview Query, we needed a data science intern that was well versed in product analytics and could work independently. Our strengths as an employer lay in the fact that we were a startup, where I could mentor data scientists directly, but honestly, we weren’t sure how much of a draw that was for applications.
Amazingly, after a week, we had over 500 candidates that applied! It was insane!
Then, we ran into the classic problem with hiring nowadays- measuring and filtering all the candidates without spending hundreds of hours interviewing everyone.
How do we evaluate candidates accurately, without bias, and at scale?
What we did next was…assign take-home challenges.
Evaluating Candidates at Scale
Yes. I know what you’re thinking. Jay, you’ve written about how you HATE take-home challenges. Oh my god. What have you done?
I’ll admit it: my attitude was always predisposed towards hating take-home challenges because I was interviewing on the candidate side. Truthfully, although it pains me to admit it, take-home challenges are great for employers because they allow the employer to evaluate candidates at scale. Instead of conducting five individual technical screens, you can realistically review 50 take-home challenges in the same period of time.
Additionally, I was motivated to find undervalued candidates. I think the easiest way to get false positives in hiring individual contributors is when employers overweight resumes and previous experience versus actual skill. Take-home challenges are best when used as a filtering and distribution mechanism while minimizing the conversion rate loss. For example:
The x-axis on the graph represents the length of the take-home challenge in hours and the vertical axis represents the signal in data science skills we could theoretically obtain if the take-home complexity increased. I assume that while the signal probably increases with take-home length, at some point, it plateaus in effectiveness. Many times, data scientists will also overspend the allotted time on the take-home challenge, which can muddle the candidate comparisons.
This second line is the conversion rate of candidates that would take the challenge. As the length of the challenge increases, the number of candidates that would complete the take-home challenge decreases, as expected.
Our goal was to find the point between these two lines that maximizes the challenge’s effectiveness in evaluating skill while retaining the most potentially qualified candidates.
Given my previous negative experiences on take-home challenges. I decided that this maximized value should be a take-home that would last one hour and be simple enough for a qualified candidate to breeze through.
Our goal is that the candidate we hire has some understanding of initiative plus grit.
I hosted the take-home on Typeform so that I could send the link to different candidates on Indeed, and created three different problems in the take-home. At the end of the day, we sent the take-home to 100 candidates after doing 200+ resume reviews and received 70 assignment finishes for around a 70% completion rate on the take-home.
Constructing the Take-Home
As you can probably guess, 70 take-home assignments are still a lot of candidates to review. Our issue was now continuing to filter candidates down while assessing skill.
We approached this in two parts and created three questions that each served a different purpose.
The first question was around basic analytics.
For anyone that has ever worked even remotely with analytics, I thought this question would be a freebie. And yet, only 58% of candidates got this question correct.
For those who got this question wrong, the rest of their take-home challenge was discarded. A wrong answer here demonstrated that the candidate either wasn’t observant enough, didn’t care about the take-home, or was just plain bad at analytics. A filtering mechanism here was necessary to trim down the candidate field.
The second question was based on SQL. This question was designed to test if candidates could write a relatively simple query using a JOIN. If they couldn’t write correct SQL, then they wouldn’t be able to pull data from our database. This question was also relatively easy to score. I had the example tables already in Interview Query. I just had to run their solution in our editor, and if the right result came back, then they passed this SQL round.
The only issue with this is that there are different SQL engines, meaning that many people got the question right but missed some syntax. I gave them a pass if the solution was mostly there, so about half of the candidates got this part of the question right.
Lastly, the final and most important question was a product analytics case question.
This problem allowed me to evaluate the candidate on a few key aspects.
- Could they come up with metrics and analyses that made sense for the case study?
- Do they understand analytics within the framework of a content business? This question is great because it involves a case study on a product everyone would know- Youtube- and was about a situation that was tangentially similar to Interview Query. No, we do not have creators, but we do have lots of content, and I wanted to know how a candidate would think about content within a business.
- Lastly, I wanted to understand their level of communication of technical concepts. Communication and structure are both some of the most important skills a data scientist needs to know. If I can’t communicate with you, then I can’t work with you.
Here’s an example of a good answer. The candidate describes the metrics that they wanted to use, outlines and clarifies who the amateur and superstar creators are, and understands how to set up two timelines to compare these metrics against each other.
Let us first specify a time period of 6 months to a year to analyze and remove all those channels that are inactive over the time period. Then let us assign all channels having no. of subscribers over 100,000 as superstar youtubers and the rest as amateur.
Metric 1 : Average increase in subscribers in time period / No of subscribers before start of time period) for Amateur Youtubers / Average (Increase in subscribers in time period/No of subscribers before start of time period) for Superstar Youtubers.
The first metric should preferably be higher than 1 as it is easier to grow smaller channels than bigger channels but a value less than 1 signifies that only Superstar channels are growing.
Metric 2 :( Ratio( Amatuer Channel Videos in Suggested Tab/ Total no of Amatuer Channel Videos ) / Ratio( Superstar Channel Videos in Suggested Tab/ Total no of Superstar Channel Videos )) in the time period.
This metric should be closer to 1 as that indicates that youtube is suggesting both types of channels equally.
Metric 3 : ( Ratio( Amatuer Channel Views obtained via Youtube search / Total no of Amatuer Channel Views ) / Ratio( Superstar Channel Videos via Youtube Search/ Total no of Superstar Channel Views )) in the time period.
This metric should be close to 1 as well. It indicates whether smaller channels are accessible via youtube search or not.
Here is a bad answer. The candidate is essentially saying absolutely nothing. What are trending videos? What are categories? I don’t walk away with any insight.
To start with, I will take the metrics ‘trending videos’, ‘views’ and ‘categories’. By analyzing them identify the number of amateurs and superstars in each category. If the same analysis is performed across time, we can identify the trend.
As you can see above, a major factor in whether the candidate passed the product interview question was based on the length of their entry. And that’s what we want! We want more ideas- albeit good ideas-and ultimately someone that can think about the problem and identify a way to solve it with data. (For more info on how to ace the data science case study, check out our article here!)
With this take-home, we narrowed down the candidate list from the original 100 candidates down to 10, eliminating 50% of the candidates with each subsequent interview question. Instead of calling 100 different candidates or just calling the top 10 based on experience, we found the most qualified candidates without talking to a single person.
Final Onsite Data Science Interviews
The final step in the hiring process is to interview each candidate. This was probably the most surprising step for me because, while I thought that I had found really qualified candidates, a lot of them weren’t that great in-person. (For data science newcomers, here’s what you need to succeed in a data science internship interview.)
A few didn’t schedule an interview since they either got another internship or weren’t interested anymore. For the in-person interview, I was grading each candidate on their communication skills, interest in the job, and a case study. Additionally, I was trying to sense if the candidate in person matched the skill assessment gathered from their take-home.
The case study I asked was about improving the email newsletter that Interview Query sends out each week. It wasn’t so much a real interview question, but rather the first project that the data scientist was supposed to work on. The better they performed on this interview question, the more effective they would be on the job.
When I talked to the eight or so candidates I eventually interviewed, about half of them weren’t very strong in-person. Some of them didn’t know what Interview Query actually did, or had not signed up to even use the website. One of them even told me his dad helped him with the SQL part. Other weak candidates just didn’t have strong communication skills, and while they could string together thoughts in the writing section, I realized in-person communication can be much harder as it requires you to think on the spot in a discussion.
Eventually, though, I interviewed one candidate that I liked so much that, after a week, I gave him the offer and he accepted.
Notes and Tips
Here are the key takeaways from this process.
Make sure that your time is spent effectively when hiring.
Recruiters and hiring managers waste their time emailing candidates, scheduling interviews, and fixing take-home assignments. It’s important to always have a streamlined hiring process that you can replicate for each candidate in the future. I’m going to be using this process from now on whenever we hire anyone because I feel like I maximized the effective use of my time and still received strong candidates.
Define the role and set expectations upfront.
Play to your strengths as an employer. I told each candidate in the interview that we can’t pay that much compared to larger companies, but that they’d receive direct mentorship and their contributions will actually move the needle at Interview Query. Our weakness as a startup ended up also being our strength and we ended up with an awesome intern.
You know how much useless noise there is. In the beginning, I added a simple piece of text at the bottom of the Indeed job description that said, if you see this text, please write “I read the job description” in the text box along with your resume. Only five people ended up writing it in. Many candidates do not ultimately care about each job they apply to. If you put in a little bit of effort, you can definitely stand out.
For employers, if you’re interested in learning more about hiring data scientists effectively, please reach out to learn more about how Interview Query can help.
Thanks for Reading!
- If you’d like more exclusive interview explanations, check out Interview Query!
- Check out my Youtube channel for more interviewing guides, and tips & tricks for solving problems.
- Find more interview guides like the Robinhood Data Scientist Interview and the ByteDance Data Scientist Interview on the Interview Query blog.
Originally published at https://blog.interviewquery.com on November 16, 2020.
