Streaming Data: A New Imperative

Download the PDF:

 Click Here

 

Read Time:

 5 min.

 

Prepared for:

Confluent-logo

 

Font Size:

 

Font Weight:

Analyst Viewpoint

Organizations’ use of data and information is evolving as the amount of data and the frequency with which that data is collected both grow. Data now streams into organizations from a variety of sources such as social media feeds and internet of things devices. The ever-increasing number of devices and streams of data offer new ways to capture information about a business and improve its operations. If an organization doesn’t capture this information and utilize it, it would be analogous to running a business without recording its sales or capturing its expenses. We at Ventana Research project that within two years the collection and analyses of streams of event data will be a standard component of enterprise information architectures.

The availability of this real-time data is changing the way we live and the types of systems we build. Historical data stores, while still relevant, are becoming a thing of the “past,” meaning they will become repositories of data that was collected and processed in real time and then stored. Streaming data is becoming the norm. Our Internet of Things benchmark research shows that nearly half (46%) of participants said they consider processing events in seconds or sub-seconds to be essential to their organization. Another fifth (21%) said they consider processing time measured in no more than minutes essential. The research also shows that speeding the flow of information is crucial to more than nine in 10 organizations. Clearly it will become a competitive necessity to capture and utilize these streams of data.

However, many organizations are not ready to deal with real-time data. Fewer than one-third (31%) said they are fully satisfied with their ability to capture and correlate events today. The research suggests a number of reasons for this dissatisfaction. Lack of resources and lack of experience and education are the most significant barriers in establishing a business case for deploying software infrastructure technology that can perform event or stream processing, detection and correlation.

Organizations must overcome these barriers in order to capitalize on the opportunities that real-time stream processing creates. To accomplish this, they should consider adopting an event-centric approach to information architecture by including in it a streaming data platform. A streaming data platform provides a common pipeline of real-time event information that can be shared instantly among a variety of applications and services. This common pipeline eliminates the need for point-to-point communications between these applications and services and provides greater agility to an organization seeking to expand its real-time processing capabilities.

Streaming data platforms also support greater scalability of event data handling. As real-time event streams grow, users can add more nodes to the cluster to increase its processing capacity. Without a common platform, scalability is often a game of “whack-a-mole.” First a user must find the particular point of stress in the myriad of point-to-point communications, then expand capacity, often only to find out that another downstream process breaks under the weight of the increased flow of information. Then the process is repeated.

Handling real-time event data efficiently and effectively creates opportunities to digitally transform organizations. A streaming data platform is critical because it provides the agility to experiment and innovate with event data. In some cases, event data enables entire new business models, such as ride-sharing businesses. In other cases, it can enhance the operations of established businesses with new opportunities and innovations, such as the automobile industry’s connected-car initiatives. More than half (56%) of the participants in our Internet of Things research said they expect their event processing deployments to identify opportunities for improvement. And, as organizations succeed with one initiative, they can leverage their platform to tackle additional opportunities.

However, most streaming data efforts will not transform entire organizations overnight. Many organizations start with smaller, focused efforts to improve their awareness of event data streams in order to better process them. As they learn and identify opportunities to improve the business with real-time data, they typically deploy pilot projects; some then become mission-critical to the organization. With more than four in five participants in our research reporting that it is vital to their functional area of responsibility to monitor and analyze patterns and relationships between events, we expect to see heightened demand for mission-critical real-time data processing.

The demand to capture and process events is growing across both business and IT functions. Within two years, 95 percent of organizations plan to capture streams of IT event data and 92 percent plan to capture streams of business event data. As organizations process more streams of event data, those that can tie all the streams together into a cohesive framework will be at an advantage. They will have a real-time view into operations across their organization, enabling them to react and respond more readily to opportunities as they arise in the market.     

Dave_2016_Circle_Zoomed

David Menninger

SVP and Research Director

David Menninger is responsible for the overall direction of research on data and analytics technologies at Ventana Research. He covers major areas including artificial learning and machine learning, big data, business intelligence, collaboration, data science and information management along with the additional specific research categories including blockchain, data governance, data lakes, data preparation, embedded analytics, natural language processing (NLP) and IoT.