Introducing Streaming Business Intelligence
A new form of self-service analytics helps business users with operational visibility
Business intelligence hasn’t had a breakthrough in perspective for over a decade.
Self-service business intelligence tools like Tableau, QlikView, and PowerBI help humans more easily understand their data. But those tools only analyze what already happened. Analyzing historical data assumes the patterns, anomalies, and mechanisms observed in the past will happen in the future.
Now, business intelligence is changing. By combining streaming analytics and self-service visual exploration, business analysts can ask and answer questions about the future. Like the Precrime Division in The Minority Report, this new type of business intelligence can help business analysts anticipate future conditions, detect them in real-time and act before it’s too late.
The Bet That Led to Motion Pictures
The creation of Streaming BI is similar to the invention of moving pictures in the late 1800s.
In 1872, Leland Stanford hired photographer Edward Muybridge to help him settle a bet that horses leave the ground as they run. It took years, but Muybridge perfected a method to take a series of snapshots from multiple cameras, tripped by thin pieces of line broken by the horse.
His snapshots revealed what the human eye could not see: horses leave the ground as they run. Slides two and three show snapshots of this fact. Stanford won his bet.
But more importantly, Muybridge’s technique captured a sense of motion. The first motion picture camera was created in 1892, influenced by his work.
Like cameras in the 1880s, existing BI tools are built for snapshots too. They show what already happened, like a still photograph. And like Stanford’s bet, bets on Wall Street ushered in a new way of thinking about Business Intelligence.
The Birth of Streaming Business Intelligence
Five years ago, “flash crashes” were alarmingly common on Wall Street. A flash crash happens when trading algorithms overreact to a stimulus, like a news event. For example, in 2012, one firm lost $440M in 45 minutes.
Banks needed a new approach to detect and stop flash crashes. As one head of trading put it —
“I need a live data warehouse where I can ask hundreds of questions about the future and be informed in real-time when those conditions are about to happen so I can act before it’s too late.” — Electronic trading VP
This request was like Stanford’s bet, but for business intelligence and data warehousing. Technologists from MIT and Cambridge University set out to answer Wall Street’s “query the future” challenge.
The solution was continuous query processing (CQP), a technique that combines streaming analytics and a push-based query model. A CQP lets Business Intelligence users inject questions into a stream of data. The query is evaluated on each and every data change in the stream. When the result set of the query changes, the BI visualization is automatically updated. Or, if the user isn’t staring at the screen, alerts are fired to raise their awareness of the current conditions.
Streaming BI is like a queryable surveillance camera for business data. It helps augment human intelligence with algorithms.
Streaming Business Intelligence in Action
Let’s examine a day in the life of Streaming BI. The short movie below shows Streaming BI analyze IoT data streaming from sensors embedded in a Formula One race car.
A business analyst can monitor the car’s position in real-time by asking the system to continuously plot the position of the car. Real-Time analysis of throttle, RPM and brake pressure is shown here, but hundreds of factors could be considered. Streaming BI reveals what static snapshots could never show: motion, direction, relationships, momentum.
But how does Streaming BI query the future?
In this example, the user creates a map for the car’s position. The BI tool registers this continuous query:
Select Continuous * [Location, RPM, Throttle, Brake] from IoT-Stream
The CQP registers this question and evaluates every tick on the data stream. When the car moves, the data changes, and the map changes. You just set it and forget it. Computations change. Relationships change. Visual elements change.
This is the simplest example of a future query: “When the car moves, show me where it is.” But thousands of these continuous questions can be registered: tell me when the driver takes a suboptimal path into a hairpin turn; tell me when tire wear degrades; tell me when the weather forecast changes.
Like traditional BI, Streaming BI is self-service. So it’s as easy to query the future as it is to query an Excel spreadsheet: you just open a connection to streaming data, create some charts, and explore.
Streaming BI Augments Human Judgement with Algorithmic Awareness
In Thinking in Bets, World Series of Poker champion Annie Duke explains how computers and humans are good different kinds of decisions. Humans excel when context, judgment, and intuition are needed. Algorithms are better at correlating massive amounts of data.
For example, algorithms will commonly mistake an image of a turtle with whipped cream on top for a Cappuccino, as Gary Marcus describes in Rebooting AI: Building Artifical Intelligence We Can Trust. See for yourself how algorithms guessed at the 40 images in the image above.
On the other hand, humans can’t correlate massive amounts of data. If I flashed a million images at you in a minute, you’d be useless. But a computer can take a pretty good guess.
Streaming BI balances these decision-making strengths: algorithmic insights from the computer and human judgment from context.
Continuous queries aren’t just for raw data; you can inject algorithms into the query. For example, “Tell me when my car is in position to pass,” detects an algorithmic condition: “In a position to pass” could be, for example, a predictive model written in R.
Another change in approach is that the user experience of Streaming BI is push-based, not request-response based. The CQP constantly evaluates streaming data; when query result changes, the new information is pushed to the visualization. Or, an analyst can ask to be alerted of particularly important queries via a text message, email, or alert. Push-based BI makes analysis more immersive than ever before.
In this way, Streaming BI balances human understanding and algorithmic insight.
Turn Your Thinking Around
Querying the future is so new that I’m often asked, “What do I do with it?” Here’s how to think about the problems it helps to solve.
First, think about the data you have that changes frequently: sales leads, transactions, connected vehicles, mobile apps, wearable devices, social media updates, customer calls, robots, kiosks, social media activity, websites, customer orders, chat messages, supply chain updates, file exchanges.
Next, think of questions that start with the words “tell me when.” These questions can contain math, rules, or even a data science model written in Python and can be answered millions or billions of times a day.
Streaming Business Intelligence questions start with: “Tell me when.”
These questions are answered when you’re not looking. Streaming BI will notify you the question is answered; you don’t have to sit around and wait. Go have coffee.
Streaming BI questions are questions about the future: Tell me when a high-value customer walks in my store. Tell me when a piece of equipment shows signs of failure. Tell me when a plane is about to land with a high-priority passenger aboard at risk of missing their connection.
Streaming BI is Like an Algorithmic Surveillance Camera
Streaming BI is like an algorithmically-aware surveillance camera for digital business. Here are some examples of companies that have put it to use today.
Rome and Melbourne’s airports capture streaming data from reservations, check-in, security cameras, autonomous vehicles, planes, and maintenance equipment. Their analysts can ask: tell me when a plane is about to land that does not have a gate clear; tell me when a bag is stuck that will might miss its flight, or tell we when the wait at any security line is longer than 5 minutes. This algorithmic awareness helps airport operations staff react and fix problems before they get out of control.
The algorithms help judge the urgency of the issue, so the human can decide what to do.
Streaming BI helps separate information needles from the streaming haystack.
Bank risk officers can monitor trades, orders, market data, client activity, account activity, and risk metrics to find suspicious transactions. Algorithms can evaluate algorithms according to risk, and humans can judge if that risk is acceptable.
Supply chain, logistics, and transportation firms can analyze thousands of connected vehicles, containers, and people in real time. Streaming BI helps analysts optimize deliveries by identifying the most impactful routing problems.
Smart City analysts can monitor streams of GPS data on emergency vehicles, traffic data, and police report data to help them react quickly to emergencies.
Energy companies like Anadarko monitor IoT-enabled industrial equipment to avoid oil production problems before they happen and predict when to perform maintenance on equipment. ConocoPhillips says that these systems could lead to “billions and billions of dollars” in savings.
And those flash crashes on Wall Street? They don’t happen much anymore.
Snapshots Aren’t Dead. They’re Just Old News
Snapshot BI isn’t dead. Like the still photograph, they continue to be essential for reports, forecasts, and monthly statements. But Streaming business intelligence can shift your point of view to the future and balance algorithmic insight with human judgment.
Every piece of moving data becomes an opportunity to change how you do business. Rather than waiting for conditions to control you, you seize control. You anticipate. You act.
Streaming business intelligence is the future of BI for digital business, and the future is here today.
To Learn More
For more, check out this 3-minute How to Query the Future Medium Series. For more on how to apply data science models inside Streaming BI, read Why You Should Learn About Streaming Data Science.
About the Author
Mark Palmer is the SVP of Analytics at TIBCO software. As the CEO of StreamBase, he was named one of the Tech Pioneers that Will Change Your Life by Time Magazine.
Image credit for the headline article: Jesse Bowser on Unsplash. Thank you!
