Sir Edmund Hillary wouldn’t have made it up Everest without Tenzing Norgay.

Data analysis has a serious Last Mile problem

A quick and simple explanation of a very important issue.

In order to innovate, business analysts, doctors, and researchers need to ask iterative, detailed questions about their customers, patients, and experiments, respectively — but analytics dashboards are only giving them the same old query tools, KPIs and visualizations.

It’s not working.

Hence, those folks turn to data specialists for help: i.e. statisticians, data scientists, database administrators.

Data specialists are akin to Sherpas like Tenzing Norgay pictured above — they alone know the terrain, and they alone have the skills to get you up and down the mountain (of data).

1. The first advantage of asking a human data specialist for help is that you can effectively communicate your question, in your language to them. Maybe it takes a few back-and-forth emails, maybe it takes an in-person meeting. But afterwards you can (usually) trust they’ll produce an answer with value.
2. The second advantage is that the data specialists do all the work of writing code, queries, data blending and computation — all you need to do is wait. To many folks without data skills, this part can feel like magic.
3. The third advantage is that data specialists can explain the analysis results back to you, in your language, helping you understand if and how the results answer your question.

The crazy thing about the process is how long it takes. Just writing the first email or scheduling the first meeting with a data specialist takes minutes to hours. Then there’s all the time they have to spend gathering data, slicing it, invoking algorithms, and preparing results. This usually takes a data specialist days to weeks, sometimes months. It becomes impossible to iterate through a series of follow-up questions, because it takes too damn long just to get the first answer.

This is the Last Mile of data analysis, and it’s baffling why people take this human-dependent bottleneck for granted. There’s no magic in the process — it can be codified, automated, and accelerated via software. Getting an answer to a complex question should be faster than the time it takes to send an email or schedule a meeting with a data specialist.

We have machines to automatically climb mountains (albeit smaller ones than Everest).

How can we solve this Last Mile problem with software? Look to the three advantages of human data specialists described above:

1. The system must allow the end user to ask a complex data question in their own language. This doesn’t necessarily mean using natural language processing like Siri or Alexa. “Speaking their own language” is more often solved by good User Experience. Guide the user into a point-and-click workflow where they understand the topic, parameters, and actions available.
2. The system must be able to do all the sophisticated+messy work necessary to answer the question, quickly. This means proper integration and slicing of data, and utilizing proper algorithms-statistics-machine learning. In other words — integrating the right data and tools for each successive question that arises, quickly and at scale.
3. The system must be able to explain data analysis results back to the user in their own language. User Experience, again, but with less insistence that visualization can do it all — contrary to conventional wisdom, visualization is often useless — and instead more dynamic text that describes each result clearly and natively in the context of the question asked (i.e. automated and concise figure captions).

I had hoped to keep this short, so I’m just going to leave it at that.

If you’d like to know more about how Tag.bio solves this problem in business, healthcare and life sciences, drop me a line.

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by 289,682+ people.

Subscribe to receive our top stories here.