The world’s leading publication for data science, AI, and ML professionals.

Hire better data scientists: A field guide for hiring managers new to data science

Part 2. Create a clear interviewing process.

If you’re new to hiring data scientists, it’s likely you’re terrible at it. Hiring is a huge responsibility and easily one of the most important things you do as a manager. Even a single bad hire can drag down your team’s productivity, kill morale, decrease team retention, and generate a significant amount of work (and headache) for you as a manager. From the job description to the interview to the offer, this series of posts is intended to memorialize hard won wisdom in hiring data scientists over the last few years, focusing on how to enhance the candidate interview process and get less…you know…terrible.

N.b., these are my opinions and not representative of those of my employer. This (these) posts are simply to memorialize some of the things that I’ve learned in an informative manner. While I hope you find them useful, they’re not official statements of any kind.

Authors: Eli Goldberg, PhD, MSc, Director of Data Science in Clinical Analytics, Analytics and Behavior change at CVS-Aetna Benjamin Goldberg, PhD, Instructor in Humanities and Cultural Studies at USF, currently trapped in Bulgaria on a Fulbright.

If you’ve read the last post, it’s possible that you’re less terrible at crafting job descriptions. Let’s keep your skillset monotonically increasing by taking the next step: Create a clear Interview process.

Here’s some organization for these articles. I’ve bolded the part you’re on for convenience.

Part 1. Creating better job descriptions brings in better talent.

Part 2. Create a clear interviewing process.

Part 3. Good interviews have structure.

Part 4. Good companies respect their candidates time.

Part 5. Don’t get emotional about compensation.


Part 2. Create a clear interviewing process.

Congrats! You read the last article on how to craft a better data scientist job description and your dream data scientist is ready to interview. Now what? The interview process, that’s what.

An interview process can be intimidating and stressful for both candidates and hiring managers. For candidates, it can be a brutal and frustrating process filled with doubt and stress. For hiring managers (and the internal team), it can be exhausting, as you wade through an endless sea of CVs. By definition, you have more to do than the current team can support – a problem compounded by the time you and the team spend trying to hire.

Given said stresses, and acknowledging you’re terrible at hiring data scientists, here are 3 actionable principles to building high efficiency process:

Principle 1: Make time for hiring and use your shift in priorities to your advantage.

Principle 2. Don’t wing it. Write your process down and engineer it to be data driven.

Principle 3: Modify the process, not your adherence to it.

I’ll provide some guidance/thoughts for each principle, below.


Principle 1. Make time for hiring and use your shift in priorities to your advantage.

Particularly for managers that have recently been individual contributors, hiring can be viewed as a distraction or a burden. This couldn’t be farther from the truth. Yes, hiring is work. However, your job as a manager is to support the team. While hiring doesn’t result in code, it is critical to the success of your team. Take it seriously and plan for success by doing the following from the get-go.

  1. Make time for hiring. If you don’t want your life to be turned into a steaming pile of garbage, signal to your internal stakeholders, and team, that you’re setting aside time to focus on hiring. Not only does this signal to your team that this is important, but it helps proactively manage your workload. Remember, hiring is part of your job. That said, hiring is not (or should not) be an excuse for late deliverables, missed meetings/appointments, etc. . The fraction of time you set aside should vary depending on the number of open roles. However, plan for at least 1 day of the week to be dedicated to hiring. While this seems like a lot, remember that work is force times distance displaced… it’s important that you become a driving force. Also remember that you want your maximize displacement by having your force be collinear with the direction you need to go. In other words, the more closely your day to day is aligned with hiring, the less your efforts will be wasted pushing the process forward (W = Fdcos(Θ)).

  2. Use your focus shift to create opportunities for leadership within the team. Your shifting priorities create an opening for data scientists on your team to assume new responsibilities. Do you have someone on the team who would thrive with a bit of extra leadership responsibility? I bet you do (and in general, most data scientists are metaphorical ‘gunners’). . One caveat concerning creating incremental responsibilities for the team is the perception of fairness. I tend to favor the ‘IDF’ philosophy that talent trumps tenure. However you end up making your decision on who ‘steps up’, make sure that you’re clear and concise in your messaging and directions to the team.

If you clearly articulate (1) and ‘tee up’ (2) with your team, you’ll find yourself with clear priorities, time and space to focus and execute, and your team stepping up to fill in the gaps to maintain momentum.


Principle 2. Write your process down and engineer it to be data driven.

This seems obvious, but you’d be surprised at the number of hiring managers that I know who think they’re some kind of Han Solo hiring manager. Odds are that you can’t just show up on the death star and ‘wing it’. Also, and to unnecessarily continue this star wars metaphor, you don’t want your hiring process to be the Millenium Falcon. You want consistent, predictable success. Be a tie Interceptor or X-wing.

Don't wing it - you're not the Han Solo of hiring and the Millenium falcon is actually a piece of hot garbage that will let you down. Photo by Josué AS on Unsplash
Don’t wing it – you’re not the Han Solo of hiring and the Millenium falcon is actually a piece of hot garbage that will let you down. Photo by Josué AS on Unsplash

I’m not super in to sports (shocking, I know), but hiring is a team sport. **** Socialize, and ask for feedback, on your process with the team. Once you’re in agreement, make sure that folks know how to execute it in your absence.

Some structure for this section: .. Subpart 1. Key components of a good hiring process. .. Subpart 2. A review of my hiring process.

Subpart 1. Key components of a good hiring process.

Here are a few ‘must haves’ when crafting a hiring process:

  1. Create a ‘source of truth’ for hiring.Google docs, cloud-hosted one drive word doc, compiled latex and git repo, it doesn’t matter. Articulate your hiring process in prose. Along these lines, it is my opinion that powerpoint presentations should really only be a means or mechanisms of delivering information, not storing it. Don’t have your ‘living’ document be a presentation. #opinions. . Also, make sure your team knows, and has bookmarked, your ‘source of truth’. If you have a large team, it may be difficult to keep everyone up-to-date individually.

  2. Provide guidance on where your reqs ‘live’ and which ones are live. Keep it up to date.Administrative tracking systems are notoriously fickle, terrible things (I’m looking at you, Brassring). Make everyone’s life easier by providing a section in your document that contains a table, organized by job title, with links to the job descriptions. As folks get hired, announce it to the team, link the ‘source of truth’, and keep the req list up to date.
  3. Provide guidance on how to source and refer candidates.Internal referrals are an incredibly valuable source of talent. However, make sure your team is well aware of the needs of the future candidate. It’s great if your team member thinks highly enough of you to want their friends/colleagues to join the team. Just make sure that the role needs are well communicated to your team and the candidate clearly. . Make sure that you give clear guidance on what information should accompany a referral (e.g., req#, the corresponding JD, a cover letter, a CV, a github link, letters of recommendation, school transcripts, minimum skillsets not explicitly covered in the JD, etc.). Absence of sufficient information, or sufficient information that exhibits low quality, should be rejected.

  4. Establish time commitment and expectations.Make it clear to your team that you respect their time, but when you ask for them to conduct an interview it should be prioritized. A lot goes into the actual interview process. However, I’ll talk about this at length in Part 3: good interviews have structure.
  5. Establish what happens in each step of the hiring process. From the technical interview, to leveling, to the onsite interview, to the offer, you need to make sure everyone understands the steps. Along these lines, and it may be a bit contentious, but I think less senior hires should have an abbreviated hiring process. That’s not to say that you forgo technical due diligence, but that the more senior your data scientist, the more institutional leadership that you should involve to help "sell".

  6. Create a graphical abstract for your hiring process to serve as a ‘quick reminder’ (See below).
  7. Be data driven and craft a process that minimizes avoidable bias. Track your successes and failures equally. This should go without saying, but I’ll say it anyway – however you end up reviewing feedback from your onsite panel, make sure that you don’t ‘spoil’ each-others interview. Group think is real and you should deliberately engineer against it!

Subpart 2. A review of my hiring process.

Here’s an example graphical abstract from my hiring process with commentary.

This is a version of my hiring process I feel comfortable sharing.
This is a version of my hiring process I feel comfortable sharing.

Step 1. HR Screen.

All candidates start out with an HR screen to collect basic information. Between the HR screen and Step 2, all candidates get a short ‘mission flyer’. This flyer is, essentially, an extension of the job description. It dives a bit deeper into the mission and helps candidates overcome the question of ‘why us’?

Controlling for my institution, and this pandemic, ~65% of all resumes are filtered out at this step.

Step 2. Technical interview. I have a lot of thoughts (and guidance) on how to craft a technical interview for data scientists (see below). However, everyone must pass a technical interview. That’s not to say that everyone should have the same level of difficulty – it should be matched to the candidates level and ‘sphere’. n.b., I use ‘sphere’ to describe the ‘flavor’ of data scientist I’m looking for e.g., consultatory, heavy research and development, a clinical data expert, etc.

I run a large clinical analytics team so my 'sphere's and leveling are overlapping. A 'P5' is essentially a principal data scientist. My expectation is that senior members of the team accumulate skills from other spheres as they advance (or are more senior hires because they already possess them).
I run a large clinical analytics team so my ‘sphere’s and leveling are overlapping. A ‘P5’ is essentially a principal data scientist. My expectation is that senior members of the team accumulate skills from other spheres as they advance (or are more senior hires because they already possess them).

Depending on the quality of the HR screening, and once again controlling for things, ~15% of my candidates are filtered out at this step. For my own technical interviews, I’ve observed that only ~25% of candidates that I’ve interviewed have passed their technical interview.

Step 3. Make a go-no go decision. Good candidates do not struggle with the technical interview. If you’re on the fence, it’s a no. Your team’s time is too valuable to put someone in the process that you don’t think will make it. Also, don’t forget about the ‘soft’ skills, as well. Do control for introversion, and don’t be unnecessarily harsh, but confidence, poise, brevity, and clarity of communication are important.

Step 4. Review the technical interview notes to confirm candidate level and ‘sphere’. Prior to inviting candidates for an onsite interview, make sure to review the level with respect to the strength of their technical interview. In contrast to the guidance above, it’s possible that you could have ‘mis-leveled’ someone. As a consequence, they could have gotten a much harder technical interview. If you have a more junior level role open, it may be worth considering putting them in the process, only after clearly communicating to the candidate that you’re modifying their expected role.

It is rare, but sometimes the technical interview is so impressive that you may be tempted to ‘bump up’ the candidates level. Don’t. On the margin, you want folks who cost one level and ‘act’ another level. This leads to more rapid advancement, which keeps people happy.

Step 5. Provide a level and ‘sphere’ matched interview panel. If you’re hiring a senior data scientists, make sure to setup the onsite interview to showcase the strength of the team and the vision of the company as a whole. Having a diverse team, including non data scientists, will help to sell the big impact and vision. I have a lot of thoughts here on how to make the best out of a candidates ‘day of’, but these will be embedded in the next article, specifically.

Step 6. Review the onsite interview data and make a go/no-go decision. After all is said and done, you (the hiring manager) need to decide. Remember, people will forgive you for passing on a ‘good candidate’, but hate you for hiring a ‘bad candidate’. The contrapositive is fickle beast.


Principle 3: Modify the process, not your adherence to it.

Use hiring to create opportunities for your team to step up and assume extra leadership responsibility. This picture is completely unrelated to the point I'm trying to emphasize. However, I like this picture of the sword... Photo by Krys Amon on Unsplash
Use hiring to create opportunities for your team to step up and assume extra leadership responsibility. This picture is completely unrelated to the point I’m trying to emphasize. However, I like this picture of the sword… Photo by Krys Amon on Unsplash

If you live by the sword, you should die by the sword… more appropriately said, if you create a process follow it. Give it a chance to succeed, or fail, on its own merits. That said, make sure that you track critical aspects of your process. Here are some key performance metrics to track:

  1. Feedback on the process from the team. What’s working? what’s not? Everyone on your team has been hired, at least once, and you’d be surprised how many opinions and terrible hiring horror stories folks will have.
  2. Success rate at each step. Is your level matching wrong? Do you have a ‘go-to’ technical interviewer that’s burnt out or ‘over zealous’?
  3. The cycle time for each step. Where are you bottle-necked?
  4. Reasons why people said ‘yes’.
  5. Reasons why people said ‘no’.
  6. Your and your team’s time. Is it well spent?
  7. A bias check. How often did subsequent panelists disagree? Are some team members consistently too hard? Too easy?
  8. Total time to hire. From HR screen to offer, how fast are you?

Summary:

Making a good interview process is iterative, but there are some key tenants to follow:

Principle 1: Make time for hiring and use your shift in priorities to your advantage.

Principle 2. Don’t wing it. Write your process down and engineer it to be data driven.

Principle 3: Modify the process, not your adherence to it.

If you’re interested in having me proof read your job descriptions, or review your hiring process, let me know! Feel free to reach out to me on LinkedIn (https://www.linkedin.com/in/eli-goldberg/).


Related Articles