Data science career advice
Data science job-seeking advice to my younger self
I’ve changed jobs several times in my career. This is what I know now that I wish I had known during those searches.
I wrote while ago about things I wish I had known earlier about pursuing a career in data science. Several readers reached out personally to say that they had struggled with the same issues or were worried about the same decisions. During those conversations, I realized that my advice focused more on how to decide what kind of job you wanted but not how to actually find and get that job. Every career decision I’ve made, except for my first job out of school, involved the decision to leave the position I was in at the time and search for something better. I haven’t always gone about my job search in the most effective way, and each job search has taken quite a toll on my emotional health.
So this post is about what I would have done differently.
Reject more opportunities
This lesson took me longer to learn than any others, and it stems from the fact that HR practices in the data science industry (among others) are really very broken. Much of a successful job search depends not only on how urgently you need a new job, but also on how much mental and emotional stamina you have. The internet application process generates a whole lot of smoke and not much heat: you’ll send out 100 applications and, if you’re lucky, you hear back from maybe 20 of them, get an initial phone screen for maybe 10, get to the next round for maybe 4, and get to final interview for 1 or 2. That amount of rejection — especially the passive rejection of simply not hearing back from most opportunities — can be exhausting. In most of my job searches, that exhaustion actually made me perform worse in those relatively rare cases where I did get an interview.
It took me too long to realize that a lot of that rejection was unnecessary, because I could be the first one to call it quits. I was once recruited by a search agency looking to fill a new data science instruction position in a prominent hedge fund. I worked through several pre-screen interviews and coding challenges before being brought in to a full day of interviews. Most of those interviews were stimulating and cordial. But then I was brought into into interview with the company’s head of data science. He was incredibly rude. I got the impression that he alone among the hiring committee hadn’t wanted to bring me in. He asked a bunch of nonsense brain-teaser-type questions, repeatedly disparaged my experience, and ended the interview by simply dismissing me, refusing even to shake my hand. It was really ridiculous.
I should have called the recruiter and told her I wasn’t interested anymore. That’s what I would have done if I was a hiring official and a candidate had acted that way. But I really wanted a job. So I raised my concerns about that interaction, got a second interview with the guy’s boss who basically told me I shouldn’t be so thin-skinned, and eventually they informed me that they would pass on my candidacy. Between the time I had the red-flag interview an the time they cut me loose, I wasted a lot of time and, more important, I wasted a lot of energy.
That wasn’t the only time that sort of thing happened to me. While that executive’s behavior was particularly egregious, most opportunities that got to the interview stage raised some sort of major concern: I was given repeated ambiguous coding challenges without any explanation, even after asking, of what they were trying to assess; my references were treated unprofessionally by having calls scheduled then cancelled at the last minute; I would make it clear that something was an important part of what I was looking for in a job and then in the next interaction the hiring officials made clear that they either hadn’t listened or hadn’t remembered or hadn’t cared. As the people offering the job, they certainly had the right to do all those things. I had the right to step away. I didn’t exercise that right often enough.
Only expend effort when it is worth it
Related to my previous point: just because a job was available didn’t mean I had to apply for it. The more applications I turned in, the more generic my applications became, and the more tired I became from turning out just slightly-altered cover letters and answering canned questions through an application portal. I’ve come to settle on a particular process for finding high-quality opportunities:
First, I look online and find jobs that are relatively fresh (posted in the last two weeks) and look interesting. I don’t apply to any at this point. Rather, I look on LinkedIn and find if there is anyone in my network that can recommend me to anyone at all at the companies offering those jobs. I don’t just find companies I have connections to — I contact those connections, tell them I’m thinking about applying to the company, and ask them to introduce me to their contact (who isn’t necessarily the hiring official) and explain specifically what job I am thinking of applying for, so the contact knows that I wan to talk to them specifically in order to decide whether to apply for the job. Incidentally, this is why I don’t accept most connection requests I receive on LinkedIn — I keep my network only to those people who know me well enough that I can recommend them or they can recommend me.
Next, I talk to the contact and listen for information that could get me noticed if I applied for the job: at worst, they’ll mention a project the team is working on and I can make that project the basis of my eventual cover letter; at best, they know the hiring manager for the position and I can ask them to connect me. I don’t try to campaign for the job during this stage. I just want information or an additional introduction. I do this until I either reach the hiring manager, determine I can’t reach the hiring manager, or decide I’m no longer interested. If still interested and I haven’t reached the hiring manager, I fill out the online application and write a cover letter that references my past conversation(s) and explains why those conversations made me decide to apply. That makes it relatively easy to write a custom letter for every single job.
If I haven’t already spoken to the hiring manager by this time but I can figure out who they are, I send them an email and tell them I’ve applied. At times I’ve found it worthwhile to temporarily invest in a job-searcher LinkedIn account so I can send InMail to people I’m not connected to. I make messages incredibly short: “I recently applied for the [position] at [company]. I wanted to drop you a quick line to personally express how interested I am in the position. If you have any questions about my qualifications, or I can give you any additional information about myself, please do not hesitate to contact me.” That’s it. I give them my phone and email.
There are exceptions to these rules. If I come across a job that looks great — not just good — and I can’t find any contacts to help me find inroads, I sometimes go ahead and apply. I don’t do this until I’ve learned something about the company or the position that makes it easy for me to write a cover letter explaining my interest. I do this rarely, mostly because there’s really very small chance that it will lead to anything.
All of this may seem like a lot of work for relatively few applications. I know for certain that it is less draining than slowly falling into despair as more and more copies of my resume fail to return from the void.
A report (or paper or presentation) is where analytic work goes to die
All of the above begs the question of what jobs are worth considering in the first place. Early in my career, I saw it as a major part of my job description to perform analyses and write up the results and accompanying recommendations. I often became frustrated when my recommendations didn’t get much of a hearing. Over time, I learned to blame the decision makers within my organization less for this. I realize now that they weren’t the problem. They were the symptom.
The impact of an analytic report depends upon a whole lot of things: how much of the report the decision maker actually paid attention to; finances, logistics, and other operational concerns that limit the range of decisions that are actually viable at any given point in time; how many other, conflicting reports of varying authority and credibility the decision maker receives on the same subject; how much of the report the decision maker forgot between receiving the report and making the decision the report was supposed to inform; the rhetoric/performance/design in which the report is dressed in order to grab people’s attention and get them invested in the analytic results.
It took me much longer than I wish it had to realize that I needed to create products, not Powerpoints. That wasn’t just for my own sense of accomplishment. The difference between an organization that productionizes an analysis and one that wants that analysis in a report is the difference between an organization that supports individual contributors and one that doesn’t. If my employers trusted me enough to ask me to do an analysis, they should have trusted that analysis enough to have a well-defined path for those results to automate or at least augment concrete decisions within the business.
If there’s a gatekeeper that can pick and chose which results from an analysis get disseminated or acted upon, then the analysis isn’t productionized. If it’s not productionized, the analysis is a political tool, not a technical one. Political tools are fine. They’re just not what I want to build. I spent too much of my career thinking I just needed to figure out how to present better. I actually did learn how to present better, which has certainly come in useful, but I no longer consider a job worth pursuing if it isn’t the default for my work to go into production. That filters a lot of data science jobs out of my search from the very beginning.
There is no way to fix a bad manager
No job is worth an abusive, manipulative, or un-supportive boss unless I’m in immediate and dire need of money. I don’t just mean that I try to avoid working with difficult people. A difficult person can make your life miserable and the cost of continuing to work with a difficult person shouldn’t be downplayed. But working with a difficult manager means you can put months or years of your professional life into something that dies as soon as you aren’t there to constantly fight for it (and, often, even if you are still there to fight for it). Working on something that’s going to die is a great way to burn out.
When I see flip-flopping on expectations, abdicating responsibility but then calling it “management style”, blaming employees for not understanding non-instructions, or simple dismissal of honest concerns when they are brought up, I look elsewhere for employment. The worst thing about bad managers is that they are so hard to fix. If the people who hired them recognized how bad they were, they wouldn’t have been hired, but those same people are insulated from learning of their mistake because the people who might enlighten them are at risk of losing their job if they complain during their employment, and are seen as simply expressing sour grapes if they mention it when they leave.
A bad manager is a relatively easy thing to spot, and it’s a sign of an incompetent organization. If the organization were capable of handling your job function well it wouldn’t have put a bad boss in charge of you because it would have known better. A bad manager is the only sure sign I’ve ever seen that I need to stop pursuing a job. Anything else I may decide is worth fighting or trying to change, but not bad managers.
A great team can just make things harder
Teams amplify. When I was in a good situation, a good team made me happier, more productive, and more focused. In fact, when the other aspects of the job were good, a strong team was the difference between a good work environment and a great work environment. When I was in a bad situation, however, even (in fact, especially) a very good team made things decidedly worse.
In my first job out of graduate school, a bunch of us were initially hired to create new ways of tackling the organization’s problems, but after a relatively short period of giving us freedom to innovate, the people who hired us started to consistently and actively oppose our doing anything different from how things had been done in the past. We on the team fed off of each other’s negativity. Everything became a grievance. We pulled each other down as much as we had built each other up during the good times. Our core technical talent saw the writing on the wall and, one by one, left. I didn’t realize it at the time, but that team breaking up was the best thing that could have happened to me. We had become so toxic that it was much better to go our separate ways. But I didn’t see it that way at the time.
As I’ve searched for jobs, I’ve often been so excited about the team I would work with that I lost sight of the larger context in which that team would need to operate. The components of a job are multiplicative, not additive. A +10 team and a -2 organization doesn’t equal a +8 experience. It equals a -20 experience. But it’s hard to see that in a job search because team dynamics are a lot easier to see than organizational dynamics. This is a difficult problem to spot during an interview process. It’s still worth looking for.
A bad job may be the best way to get a good job
There’s one type of situation where I ignore everything I’ve written above. Early in my career, to get out of an unhealthy work situation, I took a relatively low-paying job that I viewed as almost surely a dead-end position. They’d asked for someone to pull customer lists based on certain selection criteria and I had pitched the idea of doing more with the data they had, but I figured they probably didn’t have any appetite for an analysis that couldn’t be done in a spreadsheet. My first couple months at the company confirmed by initial impressions.
After one day of using their horrible database front-end, I talked to the IT department, got direct access to the database, and started to teach myself SQL. Then I started to create scripts in R that called and combined specific combinations of queries based on common use cases. After a few months, my full-time duties took up between three and ten hours a week. As long as the sales team got their customer lists, I could do whatever I wanted with the rest of my time. Over the two years of that job, I went from having a barely rudimentary knowledge of SQL to being able to unabashedly list it as one of my core competencies on my resume. I started to teach myself Python. I started to use machine learning to model things like customer attrition. A new executive came on who momentarily had similar interests, so that work actually got used. At the end of about a year, my job was still a dead-end but I had a much more marketable profile.
Even a really bad job can be good for your career. In my case, I was junior enough and competent enough that they left me enough freedom to develop new skills. Of course a good job would have been even better, and allowed me to grow even more, but that wasn’t an option for me at that time in my life. I’m far enough along in my career that I don’t know that I’ll need to take a stepping-stool job again. I hope I don’t. But it was a useful thing when I had less experience.
Fear, in moderate doses, is healthy
Searching for a job is stressful. When you have a family to support or a deadline by which you need to get new employment, is can be terrifying. I’ve found the fear inherent in a job search is really pretty useful. Most often, I didn’t really know what I wanted out of my next job any more than my prospective employers knew what they wanted out their next hire, so I’d often feel torn about accepting a decent offer versus looking for a more promising opportunity. In most cases, fear prompted me to take the decent offer, which moved me along in my career.
There have been a couple times when the fear became so big that it eclipsed every other consideration. For example, at one point in my early career, I had difficulty making rent because I had to spend the money on diapers and baby formula instead. I don’t really have any good advice on how to navigate those kinds of situations — I was a constant wreck during that time. But for all other times, when I could get food on the table but wanted a little more satisfaction from my job or a little more growth in my career, I often took a big hit to my emotional health and self-respect because I didn’t realize that I had the power to assert a fair amount of authority in my job search.
In my experience, it pays to approach a job as an applicant, not a supplicant. By deciding there were certain conditions I simply would not accept, and by cutting off prospective employers as soon as they failed to demonstrate that they could accommodate those must-haves, I was able to keep myself energized and clear-headed for those opportunities that really deserved my attention.
