Why Data Science Fails
Just as mergers often fail to live up to their promise, so have many data science teams. A KPMG study indicated that 83% of merger deals were unable to boost shareholder returns. And, while I think data science has received general success in companies that have built teams, at many companies it has far underwhelmed its promise.
Data science often fails to achieve its full promise in many companies for the same reasons mergers do not. It is like turkeys voting for Christmas. The people required to participate in the change will very likely, or see themselves as very likely, to lose control as part of their participation.
Typically there is:
- A lack of leadership and management, and;
- A lack of proper incentives with existing employees to integrate and extract value from the new people or technology.
Turf wars ensue, and few people win (especially the shareholders).
Change Agents
Just like mergers and acquisitions, introducing data science to a new organisation is hard, data scientists are likely to be viewed as outsiders. They have new technology in their hands that few understand — and in many cases can be inexplicable. As a result, data scientists end up being unwilling change agents fighting an entrenched base.
Entrenched people in most organisations fear change, and there are typically few leaders that have the necessary experience or capability in managing the uncertainty and fears that come with change. One of the most common manifestations of that fear is scepticism. That scepticism becomes a self-fulfilling prophecy that leads to failure. Compounding this issue is the fact that executives and senior managers in the company do not adequately support, nor understand, what data science is really about.
There are two obvious ways to solve this leadership or change management gap.
- First, the company’s leadership team can recognise their inexperience with new technology as well as the need for stellar change management. They could appoint and empower a trusted advisor or appoint a strong leader to drive implementation.
- Second, the data scientists themselves could learn how to support change, support the organisations they are a part of, and ensure projects gain traction through understanding and support rather than adversity.
While I believe that both sides should be a vital part of any new integration, it is ultimately the responsibility of the leadership team to invoke change. Any organisation that wishes to truly benefit from the technological advancements of the last ten years needs to lead from the front. Executives need to understand the technology, the benefits, the costs, but they also need to understand people, know that this is a significant change, and realise that few of their employees will have had this type of experience in their lifetimes. Uncertainty and fear will be sown. Retraining, upskilling, knowledge transfer and open conversations are a great way to prevent that.
Driving Change
It is effortless to go to data scientists and put the onus on them — either that or to leave them to their own devices. Leadership could tell their people that if they wish to enact change, then they should speak out! After all, this is an agile company where decisions are made democratically and not by a select few.
While the sentiment is nice, it reminds one of something akin to anarchy if the organisation is entrenched, the actual probability of this succeeding is exceedingly low. As anyone who has ever tried to alter the course of an organisation from an un-empowered position will know, it is frustrating, unlikely to succeed, and very likely to lead one to burn out. Moreover, while I think many data scientists are capable of being great technical people, have the necessary business acumen, the right domain knowledge, and the ability to be capable leaders, this sets an unrealistic formula or expectation.
Data Science is F***ing Hard
I am not trying to be self-promoting. When I became a CEO of a medium-sized organisation, I felt like it would ultimately be an easier job than when I was purely a data scientist — sure there would be more stress, but the actual complexities of the situation would be reduced.
While data science is the best job I’ve ever had and it is one I still love to this day, it is incredibly demanding. To be sure, the level of interest I have dulls the difficulties, but it is objectively difficult. One needs to juggle many time-consuming tasks at the same time:
- Stay on top of advances in technology through reading papers every day
- Deeply understand the domain they work in, often through research and long conversations with experts
- Constantly create, implement and judge themselves by objective metrics
- Manage complex integrated systems that are often poorly managed (bad data quality is incredibly prevalent)
- Understand how a business operates
- Know how to integrate new technology into those companies
- Be capable at presenting results to people will far less statistical and mathematical background
- Understand visualisation and how data can be used to tell a complex story in a simple way
- Know their limits and be emotionally intelligent enough to know what they don’t know.
- Question everything
Tacking on all the qualities of being a strong leader, a change agent, an orator, and a few other hats and you’ve asked for the impossible. Business leaders need to understand that data scientists are not, and will not be everything. Those same leaders need to go out of their way to better understand the difficulties that organisations face in successfully implementing data science.
It is time to stop thinking someone else will do it.
