Data science career advice

As near as I can figure, Data Science started to become a popularly-used term around 2012 when an article in the Harvard Business Review called it "the sexiest job of the 21st century." As I read that article, I remembered thinking that that was the kind of stuff I’d been trying to do for three or four years. I started to refer to myself as a data scientist, and over the course of my career I’ve held data science positions and founded and managed data science teams across the security and intelligence, travel, asset management, education, and advertising industries.
My ideas about where and how to work as a data scientist have shifted a lot over the course of my career. Things that used to be very important to me aren’t that important anymore. Some things I used to avoid or even looked down upon have ended up bringing me a lot of satisfaction. With the benefit (and bias) of hindsight, I can see how I would have appreciated knowing a few things then that I know now. This post is my attempt to encapsulate some of those lessons. I hope they might be useful to some people coming into the profession.
Only take a step up if it isn’t also a step sideways
I once worked my way up to a pretty authoritative role at an organization that was one the largest, best-funded, and best-positioned of it’s kind. The CEO was one of the biggest players in the industry. I made it onto the executive committee and ended up taking leadership of all technology issues in the organization. I held the highest technology-related position in the organization, was a required attendee at board meetings, blah, blah, blah. It took me farther from where I wanted to be, career-wise, than anything else I’ve ever done.
Don’t get me wrong: I learned tons of stuff and had tons of opportunities that have been useful in getting me to where I am now. But it wasn’t a promotion. It was a career change. I wrote less and less code, and writing code is something I genuinely love doing. I had to turn an ever-larger number of design decisions over to others because I didn’t have time to work on them myself. I ran all of the things I loved to do but I didn’t get to actually do any of the things I love to do. And it increasingly took time away from my family. I got to the point where almost nothing I did was intentional – it was all reaction. It was simply unhealthy.
I’m sure other people can get to high positions and handle those heights better than I did or could. Also, looking back, I can see how much of the organization was dysfunctional. In some ways, it was even toxic. But those issues aside, I have found much more satisfaction as an implementer than as a manager/supervisor/director. I really like to design things. I haven’t always chosen positions that maximized the time I spent designing. For much of my career, it didn’t occur to me to think about all things things I did in my job and identify the things that made me happiest even if they didn’t pay more or give me a better title. I wish I had done that earlier.
Don’t try to fix the world
For a lot of my career, I looked for ways that my job would let me work to fix some sort of major social problem. That’s what appealed to me about working in security and education. When I worked for a travel agency, I worked for an educational travel agency that sent students on immersive experiences. When I worked in asset management, I worked for a firm that wanted to invest in emerging economies.
The social-good focus was a mistake, I think. Organizations vary in how broad their visions are and in their ability to execute. In my personal experience, broader visions tend to be coupled with weaker abilities to execute. The company I work for now was founded with a pretty focused purpose: figure out how to get advertisements in front of a desired number of prospects within a desired timeframe and within a desired budget. I used to refer to that kind of vision, pejoratively, as "widget making." I’ve learned the joy of making widgets. Because the company I work for has learned to make darn good widgets, those of us who work there have the freedom to branch out into a number of R&D efforts.
The work I do now is almost all what I would consider interesting R&D, and almost none of it would be possible if the company hadn’t, for many years, had a myopic focus on building a super good advertising widget, as well as all the infrastructure to support that. I’m now working directly on the intersection of offline and online behavior, which I think is one of the most interesting things I could be working on. I think my chances of working on such an interesting topic would be much less if I had looked for a company with an explicit change-the-world focus, or even a company that was specifically founded to understand the intersection of offline and online behavior. When I stopped trying to work on something big, I got the opportunity to work on something big.
Stick to organizations that value individual contributors
In most organizations I’ve seen, there’s a hierarchy of managers but the implementer landscape is pretty flat: some individual contributors are recognized as more experienced or competent than others, and their pay and task assignments often reflect that, but there’s not much difference in organizational authority. At my current job, there’s an explicit non-management career ladder and the difference between the higher and lower rungs has to do with design authority: as you get more senior, you’re given more decision-making power over what gets implemented and how it gets implemented. Managers are focused on business need prioritization, inter-personal relationships, ensuring access to resources, and things like that.
The whole experience has spoiled me. I don’t think I can go back to an organization where managers have control over what gets implemented. I don’t think I can go back to a non-engineering organization. It’s tough to be a technical employee in organizations where promotion can only mean management. I wish I had spent less of my career in those types of organizations.
Location matters, but not how you think
Looking back now, I think I got sidetracked when I moved to New York City. I recognized that Charlottesville, Virginia, where I was living, just didn’t offer me many growth opportunities: it was too small, particularly when it came to data science jobs. So I looked around and saw that the real centers of the industry, at least in the U.S. at that time, were San Fransisco and New York. I didn’t realize that those markets were as over-saturated as they were large. There are tons of jobs available, but there are even more applicants competing for those jobs.
I took a job in New York, and eventually moved to a second and more senior job also in New York. Then it came time to leave that job. That’s where I saw the downsides to working in a major hub. I either had to (1) take an entry-level position with it’s entry-level salary simply because there was so much supply that I couldn’t demand any more for an individual contributor position, (2) make a career change and become a full-time manager, (3) trust that my luck and my savings account would hold out as I waited for a company that was looking for my exact profile, (4) settle back into a largely tech-ignorant company like some I had worked at before, or (5) move to a place where the market wasn’t so saturated.
My family finances (and, yes, pride) wouldn’t allow me to take the entry-level position. I no longer wanted to be a manager. I wasn’t willing to deplete our savings waiting for a stroke of luck (because it really is mostly luck). And I didn’t want to be the smartest tech person in the room. That’s when I decided to look for jobs outside of New York but in places larger than Charlottesville, and that’s when I realized the desirability of second- and third-tier locations.
I place the Bay Area and New York City in the first tier, along with maybe Seattle and perhaps Boston. I put place like Chicago, Atlanta, Austin, Washington DC, Denver and the North Carolina Research Triangle in the second tier. I put places like Columbus, Salt Lake City, Indianapolis, Nashville, and others in the third tier. (Don’t yell at me if you’re from one of these places and don’t like your tier – it’s personal heuristic, not a validated classification).
All those next-tier places have tech industries that are really large compared to the rest of the country while small compared to the first-tier hubs. They don’t have as many experienced people as they need, partially because the first-tier locations are eating up the more experienced people (many of whom, I suspect, are taking management jobs or jobs that pay less than what their experience warrants because they’re competing against a huge pool of people equally desperate for employment). While there are fewer companies to choose from in the next-tier locations, those companies are still doing really interesting things, and there are enough of them that there are still opportunities to move jobs in the future without moving location.
Work/life balance outweighs almost everything else
I’ve come to recognize how essential it is to my own happiness to work at a place that recognizes my right to spend time with my family. In my current job, there’s no ominous shaking of heads or raising of eyebrows when I need to work from home or go home early or come in late to take care of things. As long as I meet expectations about what I’ll deliver and when I’ll deliver it, and don’t get in the way of other people meeting their expectations, I largely get to order my time and the rest of my life however I need to. That’s hasn’t always been true at other jobs I’ve had. I didn’t realize until relatively recently just how much that lack of flexibility counter-balanced a whole host of perks and benefits. That’s a personal preference, of course. My point is that I took longer than I would have liked to acknowledge that preference.
A summary
As I look back at the companies I’ve interviewed with or worked for over the years, I recall very few that:
- Had a very specific definition of what they were trying to accomplish.
- Had invested in the policies, personnel, and infrastructure to consistently do the stuff they planed to do.
- Were first and foremost an engineering organization.
- Explicitly supported individual contributors who wanted to grow without becoming managers.
- Paid a decent wage for people who have families and want to do things like save for retirement and/or buy a home.
- Were interested in hiring senior people into a non-management track.
- Recognized an employee’s right to have a substantial life outside of work.
- Offered consistently interesting and challenging analytic work.
When I interviewed at big tech companies, I found limited support for #4 and #6, and practically no support for #7. When I interviewed in tech startups, they were usually incredibly early stage and therefore fell down on #1, #2, #5, #6, and #7. When I interviewed with big non-tech companies with heavy or growing tech presences (basically giving up on getting #3), I found they almost always failed on #1 and #2, and practically never offered #8. I’ve found the companies that most consistently appeal to me are late-stage startups. In many of those cases, I still found they fell down on #4 and #7, and sometimes #5 and #8, but that seemed to offer the most amount of signal and least amount of noise.
A smell test
This might be overfitting on my part, but I feel like those companies that best exemplified the priorities I outlined above were those that were most flexible on the topic of their technical interviews. In my last job search, I got to the point where I actually refused to do white-boarding interviews or other technical tasks.
I still think that was a good idea – except for extremely junior candidates, in which cases the questions should be really simple and general, I think technical challenges are an inhumane as well as ineffective HR practice. When I started telling companies that if they insisted on a technical task then I could tell that it was a poor culture fit, I had several agree to do without the whiteboard interview, and then they would essentially try to do the whiteboard interview without the whiteboard (which is even harder than doing it with the whiteboard).
I’m still incredibly impressed with MaxPoint (my current employer, though we’re now part of Valassis Digital), where the interviewer offered me the option of walking them through any of my own code that I thought would be informative. My gold standard is still what I experienced at Enigma, where I sat in the room with another data scientist and together we planned out how we would start addressing a real business problem that they themselves didn’t know the answer to. I think I can tell a lot about a company by looking at how how flexible they are in evaluating me as a candidate. If I were targeting other priorities, that probably wouldn’t be as strong an indicator.
An ideal is important
So if I suddenly found myself looking for a job, I personally would look for late-stage startups in next-tier locations that appealed to me. I’d look for companies that had enough funding that they weren’t going to try to work me to death or skimp on pay, that had already had enough success in the market that I could gauge whether they were good at what they did, and that were focused enough that they didn’t pretend they were trying to change the world. And I’d cut candidate companies loose as soon as they exhibited inflexibility in how they wanted to evaluate me. I know I haven’t always been in a financial position were I could afford to be that picky about where I worked. I still think the ideal is an important thing to aim for.
My professional goals have changed a lot. I’ve learned to err on the side of exaggerating the cost to my family rather than exaggerating the professional benefit of a particular career move. I’ve learned to get a whole lot of satisfaction out of my job but I get most of my happiness from things that have nothing to do with my career at all. That’s a pretty big change for me, and I think it’s a healthy one. Like I said, I have no idea if that’s a good way for other people to structure their lives. But, looking back now, I can see that many of the choices I made precluded my ability to think about anything but my job, and that wasn’t a healthy place to be.
At the same time, I don’t think I’d be able to derive so much satisfaction from my current career situation without having experienced the ambition and frustration of the different jobs I’ve had in the past. Maybe all this advice to my younger self wouldn’t have helped at all without the experiences that helped me formulate the advice in the first place. All the same, I would have liked to have known some of these things earlier. If nothing else, I think it would have given me a clearer vision of where I wanted my career to lead.
Edit: I’ve written an additional post specifically on job-search advice to my younger self.