Stream Processing with InfluxDB
Process continuous streams of large volumes of data to detect conditions and anomalies in an instant, inspect results and analyze data from multiple streams simultaneously.
What is stream processing?
Stream processing is the processing of data in motion. It is a big data technology used to query continuous streams of incoming data and detect conditions, quickly.
Why InfluxDB for stream processing?
Stream processing unifies applications and analytics by processing data as it arrives, in real-time, and detects conditions within a short period of time from when data is received. The key strength of stream processing is that it can provide insights faster, often within milliseconds to seconds.
With that being said, stream processing naturally fits with time series data, as most continuous data series are time series data. And time series data needs a purpose-built database to ingest, store and process it. This is exactly what InfluxDB is. And this is why, given its high-write throughput and the scalability it allows, InfluxDB suits stream processing.
Providing material and contextual analysis regarding a significant amount of data, coming in at high rates, is difficult. Therefore, big stream processing perspective follows the quest for the optimal trade-off between latency and throughput.
Stream processing, from an information-need perspective, requires considering order, context and responsiveness.
For these three characteristics, state-of-the-art stream processing provides the necessary theoretical foundations. But a language that reaches the right level of expressiveness is needed. This is where Flux, InfluxDB’s scripting and query language, comes in.
- Turing-complete and meets the need, in stream processing, for a programming language that reaches the right level of expressiveness.
- Functional, which makes it inherently declarative; it enables planning before execution and enables logical/physical optimizations.
- Task-driven. The target programs are Continuous Queries that analyze input streams, gluing together applications.
- Portable and testable. It is designed to be shared and easy-to-read.
- Can serve the design of SQL-like, domain-specific languages on top, like InfluxQL.
Components of the InfluxData Platform
Flux is InfluxData’s new functional data scripting language designed for querying, analyzing, and acting on time series data. Its takes the power of InfluxQL and the functionality of TICKscript and combines them into a single, unified syntax.
InfluxDB is a time series database designed to handle high-write and query loads. It is an integral component of the TICK Stack. InfluxDB is meant to be used as a backing store for any use case involving large amounts of time-stamped data, including DevOps monitoring, application metrics, IoT sensor data, and real-time analytics.
Kapacitor is an open source data processing framework that makes it easy to create alerts, run ETL jobs and detect anomalies. Kapacitor is the final piece of the TICK Stack.
Aleksandr Tavgen, Technical architect, Playtech
Related resources for developers
Free InfluxDB training