How anecdotal evidence can make or break your insights
You’ve been waiting for a special occasion to take your partner to the fancy new restaurant in town. You’ve done your research: checked the yelp reviews, asked a few friends, even searched newspaper and magazine articles to make sure it would make for the perfect evening.
Just before you make the reservation, you tell your friend, let’s call her Debbie, about your plans. “You’re making a big mistake,” she warns “I ate there last Friday and it was terrible. I had to wait 20 minutes even though I had a reservation, it was too loud by our table, the bread was stale, the service was slow, and my meal was overcooked.”
Debbie wouldn’t lie to you about her experience, but your exhaustive research indicates it’s a fine establishment. What do you do?!
The issue at hand is one which businesses struggle with constantly: How should we properly take insights from anecdotal evidence? Viewed incorrectly, anecdotes can lead you and your organization to the wrong conclusions. Utilized properly, they can be essential to connecting with your customer.
Before we get into what works well here, let’s come up with a definition of “anecdote” through the lens of analytics.
How can we define an “anecdote” as it relates to analytics?
You know that an anecdote is basically a short story, but let’s come up with a definition that speaks to its value in the context of data collection.
“An anecdote is a description of a data point that goes beyond providing the metric(s) being collected.”
Taking a look at the example of a restaurant experience, let’s first identify which metric or metrics you might collect. On-line reviews and articles offer the entire scope of detail, from “great experience” to descriptions of every course and element of service. (No doubt in each yelp review and article you read details about the dining experience at the restaurant in question. However, if you were to read a yelp review that simply read “It was great!”, you would still take that data point into account.)
What are you really measuring? If you chose to measure the experience as a whole and (you were measuring the experience that people had as a whole. If Debbie simply said “I really didn’t like it,” that data point would have still provided a lot of value, since it speaks to the metric you are targeting. Unless you were specifically tracking the feedback on the details on service and food quality, you didn’t need the additional information.
Therefore, Debbie’s “I really didn’t like it” would constitute a data point, but not an anecdote. What she actually said was an anecdote according to our definition, since it contained information that went beyond the metric you were collecting. Keep in mind, that’s not to say there is no value to additional input, we will get to that later.
How anecdotes can blow up your data insights
One of the well-known challenges with Big Data is information overload. What do you use? What can you toss?
Anecdotes are a microcosm of the same problem: you are given too much information! How do you compare a yelp review that gave the restaurant 5 stars and read “Meal of the year!” with Debbie’s account? The valuable data points are those that relate to the variables you prioritize.
Anecdotes resonate with us on an emotional level, which can lead to an overemphasis on those data points. When Debbie takes you through each step of the evening, it’s almost as if you are experiencing it yourself. You imagine yourself feeling the annoyance of waiting for a table, the displeasure of tasting stale bread, the frustration of not being able to get a server’s attention, and the disappointment when your long-awaited meal is sub-par.
Instead of comparing the data itself, let’s compare your emotional reaction to each data point. Does the general on-line review or Debbie’s description give you a greater magnitude of emotion? Clearly Debbie’s! We naturally associate a greater significance to the account that elicited a stronger emotional response.
In actuality, do you think the person who wrote the five star on-line review had a strong emotional response? Probably so, particularly if they were moved to post a rave review. That person was exposed to the same variables as Debbie, but the description simply was not as detailed. That same review could have been written to match or exceed the level of emotion in Debbie’s account.
In a nutshell, when you experience a data point yourself, you will associate greater value to it than if it was simply reported to you. Because you experience more emotions from a data point, you naturally (often subconsciously) assign it more importance.
How does this blow up your data insights? Too many times, these anecdotes are descriptions of data points that are no different than data points that provide you with your desired metric and your desired metric alone. The anecdote is a drop in bucket, and it can lure you into making it seem of greater significance than it actually is.
This occurs every day with customer service. The people most likely to call in will be dissatisfied customers, and you can bet they will providing descriptions at length as to how a product or service gave them a horrendous experience. If executives were to listen to these stories and compare them to raw data that showed that their sales were getting stronger, they might be prone to making ill-advised changes because of the emotional weight attached to consumer anecdotes. This is not to say that those dissatisfied customers don’t count as data points, or that there is nothing to be done to improve the product or service. It’s simply stating the fact that each customer should be weighted the same, even if the dissatisfied data point is more emotionally engaging.
Emotions are useful in all areas of life, including business. They allow us to connect with our customers so we can market effectively and develop products and services that meet their needs. However, they can cloud our judgement and prevent us from staying objective when drawing conclusions from data sets.
Without a data collection process set in place ahead of time, anecdotal evidence will make you susceptible to the basic human instinct to assign greater value to inputs that move you emotionally. Incorrectly weighted inputs lead to incorrect outputs. This could result in implementing the wrong strategy, costing you resources or worse, customers.
Connecting with customers and end-users
During any political debate, you will hear each candidate give an anecdote about themselves, or someone they’ve met on the campaign. They understand that information is not enough to invest, people want an emotional bond.
Data by itself is cold and impersonal. The numbers and figures they present are nothing more than pixels on a screen or ink on a page. In actuality, those numbers often represent people and experiences. The meaning behind the data is essential to connecting with people both internally and externally.
Internally, end-users must believe the insights from the data are significant enough to alter their actions. Without their buy-in, your insights are useless. Anecdotes can be used to motivate internal clients to take action, when the cold data may not.
For example, if a product developer for an auto manufacturer was determining how much quality should be invested into safety features, they may have access to a figure like the one below.
This developer may see the decline in annual deaths per billion miles (the red line) as justification for not paying a lot of attention to safety, as people seem to be better drivers. However, if introduced to a few stories about how a car’s safety features drastically affected people’s lives, they may take a new course of action due to the connection they now feel to safety. Raw data cannot form that bond, anecdotes can.
Externally, anecdotes can connect you with customers. This is the purpose of focus groups. It’s not enough to know people either approve or disapprove of your product. Understanding the emotions they experience as a result of your product is essential to making a connection to your target market and drive sales.
Anecdotes also help in explaining data points that do not make sense. Hearing stories about data points that do not make intuitive sense can uncover the hidden variables that are really driving the result.
Insights from your data are only useful if your end-users utilize them in practice. Anecdotal evidence can be an effective way to achieve the buy-in you need so your analytics provide advantages to your organization.
Where do we go from here?
Anecdotal evidence can be a powerful tool to derive unique insights from your data if used properly. Remember our definition of an anecdote? It’s a data point that provides additional information. If you have a disciplined approach, that information won’t mislead you, it can generate a better understanding of what you’re researching.
If there’s one takeaway, it’s this: let anecdotal evidence drive your questions, and data analytics support the answers.
The scientific method plays a crucial role in your data analytics process. Systematically testing your hypothesis and communicating the conclusions from the results is how data helps us make more informed strategic decisions. However, do not be misled by the term, “data science,” as there is an art to it as well.
There are practically an infinite number of hypotheses to develop when approaching a data set. Therefore, choosing the one in which to test is an art form, it allows and oftentimes requires creativity to gain the most useful insights.
Anecdotal evidence provides us with additional information beyond that of a typical data point. There could be counter-intuitive patterns present in those stories, or variables you hadn’t thought to take into account.
Let’s go back to our restaurant example. After reading many reviews and listening to Debbie’s story, you notice there may be a pattern when taking the day of the week into account. Debbie dined there on a Friday, and perhaps the only negative reviews you saw were on Fridays as well. That should not be enough to conclude that Fridays should be avoided, but it can drive you to now approach your data with a new question: is there any correlation between day of the week and customer experience? The anecdote provided the question, now you let your analytics provide you with an objective answer. Whether you find correlation or not, you can now make a more informed decision from your data set, thanks to the additional insight provided by anecdotal evidence. So while Debbie is complaining about her night out, remember how to approach anecdotes:
- Keep emotions aside
- Listen for variables you have not yet considered
- Develop novel, testable hypotheses from your new variables
- Systematically and objectively test those hypotheses
- Maintain or change your strategy based on the results
Now, you can feel more confident your night out will be an enjoyable one.
This article was originally published on www.strataquant.com
