Stream Processing with InfluxDB
Process continuous streams of large volumes of data to detect conditions and anomalies in an instant, inspect results and analyze data from multiple streams simultaneously.
What is stream processing?
Stream processing is the processing of data in motion. It is a big data technology used to query continuous streams of incoming data and detect conditions, quickly.
Why InfluxDB for stream processing?
Stream processing unifies applications and analytics by processing data as it arrives, in real-time, and detects conditions within a short period of time from when data is received. The key strength of stream processing is that it can provide insights faster, often within milliseconds to seconds.
With that being said, stream processing naturally fits with time series data, as most continuous data series are time series data. And time series data needs a purpose-built database to ingest, store and process it. This is exactly what InfluxDB is. And this is why, given its high-write throughput and the scalability it allows, InfluxDB suits stream processing.
Providing material and contextual analysis regarding a significant amount of data, coming in at high rates, is difficult. Therefore, big stream processing perspective follows the quest for the optimal trade-off between latency and throughput.
Stream processing, from an information-need perspective, requires considering order, context and responsiveness.
For these three characteristics, state-of-the-art stream processing provides the necessary theoretical foundations. But a language that reaches the right level of expressiveness is needed. This is where Flux, InfluxDB’s scripting and query language, comes in.
- Turing-complete and meets the need, in stream processing, for a programming language that reaches the right level of expressiveness.
- Functional, which makes it inherently declarative; it enables planning before execution and enables logical/physical optimizations.
- Task-driven. The target programs are Continuous Queries that analyze input streams, gluing together applications.
- Portable and testable. It is designed to be shared and easy-to-read.
- Can serve the design of SQL-like, domain-specific languages on top, like InfluxQL.
Components of the InfluxData Platform
InfluxDB is a time series database designed to handle high-write and query loads. It is an integral component of the TICK Stack. InfluxDB is meant to be used as a backing store for any use case involving large amounts of time-stamped data, including DevOps monitoring, application metrics, IoT sensor data, and real-time analytics.
“Why InfluxDB? For Playtech, it was very important to have observability, to understand system behavior to predict possible outages and problems in the very early stages.”
Technical architect • Playtech
Read case study
“The amount of data you collect, how precise you want it to be, when you want to expire it. These are things to consider when building out your solutions because the TICK Stack offers the flexibility to make those changes.”
Head of Engineering • Worldsensing
Read case study
In this webinar, Albert Zaragoza from industrial IoT company Worldsensing will share how he and his team built an end-to-end IoT solution for cities: from traffic flow management, smart parking, emergency & security response, and critical infrastructure monitoring—he will focus this talk on how the team integrated a real-time platform featuring data science and a huge variety of data sources including InfluxDB, a time series database to store a city’s critical IoT time series data.
Is your data in an exotic format stored in Kafka? Let’s write a Telegraf Plugin! This session is a step-by-step presentation on how to develop a Telegraf parser plugin. The focus will be on Apache Avro, a popular data serialization format widely used in Kafka-based data pipelines.