How to Query the Future
Imagine you could ask questions about the future like a bear in a stream searching for salmon over and over and over? The tech that does this is called streaming analytics--it makes it easy to #querythefuture using real-time data in motion.
By contrast, conventional databases are request/response systems. That is, you send a query to a "lake" of data; you get a response. First invented over 50 years ago, databases are the bedrock of the computing industry today.
But while so much has changed, so much remains the same. The request/response database query model looks at what's stored on disk--what already happened. Using only historical data to make decisions is like trying to drive your car by only looking in the rearview mirror.
#streamqueryprocessors were invented just 20 years ago to flip this way of thinking upside down. Researchers envisioned a future of data in motion, not only at rest, generated by embedded #sensor technology, robots, and connected devices. Their idea: inject continuous queries into the stream of data; let business users pose questions about what's happening NOW; REMEMBER those queries and ask them continuously, in the FUTURE.
I've been fortunate to run TWO of the preeminent commercial #streaminganalytics companies (Apama and StreamBase) during that time. On my 20th anniversary, it's still fun to explain how it works. Here's a NEW explainer video about how #streamqueryprocessors work. The innovative things our customers do with them today still blow my mind!
What a blast it's been to work with #CEP visionaries like John Bates, Mark Spiteri and Richard Bentley of Apama, Richard Tibbetts, Eddie Galvez at StreamBase. Fun to keep writing patents with Larry Derany and Thomas Hill. Michael Stonebraker, founder of StreamBase and visionary from Massachusetts Institute of Technology. Early investors at In-Q-Tel, Highland Capital Partners, Accel and Bessemer Venture Partners. And of course, Roy Schulte and Mike Gualtieri, two analysts who helped make #streaminganalytics technology known to so many and who continue to do it so well to this day!
And finally, David Luckham from Stanford University, whose book "The Power of Events" set the tone way back in 2002.
Here is the transcript for the movie above!
Enjoy a modern take on "how to query the future" and an innovative technology that's STILL not as well known as it should be, but still as fun as it was back in 2002.
Almost every business on the planet uses a database. But for many applications, conventional databases have a blind spot: they only help you look backward at what already happened in the rearview mirror. That works, but only when conditions stay the same! When conditions are volatile, you need to query the future, not only the past.
You can do this with an unconventional technology - a Stream Query Processor.
Here's how they work.
First, you connect streaming data to a streaming platform. Streaming data comes from IoT sensor readings, financial transactions, Kafka events, database log updates, Twitter tweets, and REST API events, just to name a few. But these data sources aren't easily queryable because they're in motion, and that's where a stream query processor comes in.
For example, imagine we run drilling operations for an oil company and want to be alerted whenever a well is likely to experience danger in the next few minutes or hours, given historical patterns and short-term trends in one query.
A stream processor takes a question like "Select All Oil Wells Showing Signs of Failure for Greater than Five Minutes: and creates live tables that match.
There are two unconventional parts to this query. First, "signs of failure" can be anything a domain expert might specify that compares historical failure indications to real-time conditions-- temperature readings that are too high, voltage readings too low.
The other is the TEMPORAL nature of streaming queries. Real-time systems are notoriously "spikey" -- you get a lot of inaccurate readings or occasional outliers. The trick is to smooth them out so the system isn't effectively crying wolf.
Now the query processor goes to work. First, a dashboard or client application gets an immediate snapshot—here, we get a table of wells currently at risk. But continuous queries stick like glue--it holds the question in memory.
As IoT sensor readings change, the query processor re-evaluates questions continuously. The Live Table is updated as the query is rematched over and over and over. With conditional formatting in modern BI tools, you can make them pop on a dashboard or trigger actions and alerts.
As NEW wells match the query, rows are added to the table. "Well G" is beginning to show danger signs. It's added to the table.
Rows are DELETED from the live table as conditions improve: now, only Well C and G are of concern.
Live tables act like normal database tables in some ways. For example, you can sort, aggregate, filter, and sum values. And, of course, you can drill down.
But also, you can set alerts that trigger action.
Here's one used by an insurance company to continuously track a storm as it approaches Florida; we analyze a streaming weather feed forecast in real-time.
Streaming queries fire with each change in the forecast to predict where damage will be the worst, decide where to deploy staff, fix supply chain issues, or warn customers.
You can see the big picture of the projected financial damage and take action right from this dashboard.
It helps teams sense and respond to changing conditions like a bear in the stream, searching for salmon over and over and over.
Not every enterprise application needs to query the future, but it can be a game-changer for environments where conditions change frequently.
Stream query processors were invented on Wall Street for high-frequency trading. And it's a must for industrial IoT applications like predictive maintenance in drilling.
Or Formula one, since every race, every track, and every weekend has volatile and changing conditions, and F1 cars have 100's of embedded sensors that can be queried to anticipate the best race-day strategies.
High-tech robotic manufacturing systems query IoT sensor streams to ensure quality output and efficient operation.
In retail, knowing what to offer your customer next based on history requires foresight.
So flip your thinking upside down about data. Don't only look in the rearview mirror. Query the future to become more proactive in today's volatile and unpredictable business world.
