Use Cases for Time-Series Data

For some businesses, windows of opportunity can be measured in seconds. To react immediately and intelligently to these opportunities, the data from disparate systems and services needs to be available for processing in milliseconds. Whether the organization needs real-time analytics to buy and sell equities, debug code before whole systems go offline, adjust prices based on customer behavior or conduct A/B testing inside a MMORPG, processing, analyzing and acting on the time-series data in real-time is the problem to solve.

InfluxData helps companies get insight into what matters to them, in real-time, by helping them effectively manage their time-series data.

Why are companies building real time analytics solutions?

At InfluxData, our interviews with customers and users have surfaced many cases already in wide use, for example:

  • Content personalization and targeted marketing by online retailers based on browser and transaction history
  • A/B testing or in-app purchases in gaming based on real time consumption
  • Predictive analysis on a pipeline of usage data (utilities, bandwidth, storage, servers) for capacity planning
  • Network traffic analysis for detecting anomalies or malicious attacks
  • GPS tracking in fleet management systems.

get-started__graphic-4
There aren’t many out-of-box real-time analysis solutions that can serve so many diverse industries and niches. And more importantly, that don’t require the heavy lifting of a mini supercomputer or a complex, sprawling system that needs to implemented and maintained at a very high cost. Legacy analytics products were based on historical analysis and some queries can take hours to run. This clearly doesn’t meet the needs and pace of today’s business climate.

Some legacy products could in theory be extended, for example enhance a network packet capture tool to detect anomalies in traffic patterns. But it wouldn’t be any less than rewriting the core solution in order for it to scale and compute real-time analysis on the data.

As businesses do more of real time tracking and smart selling, many of them are building their own data management solutions.

Challenges in building a real-time analytics solutions

Technically real time processing can be achieved in a few different ways including:

  • In-memory – Read or write data to RAM instead of disk
  • In-database – Build analytics logic into the database itself to compute streaming metrics during the persistence period
  • On-chip – bake the processing into the memory chip itself
  • MPP style – Massive parallel processing of jobs


All approaches are valid, but present their own sets of challenges:

MPP implementations are very complex with multiple moving parts and need a lot of compute machines chained together over a shared network. In-memory processing is definitely getting popular with RAM cost decreasing steadily. In-database analytics and On-chip processing are a bit risky to build without deep subject matter expertise.

InfluxData for Real-Time Analytics

InfluxData helps companies get insight into what matters to them, in real-time, by helping them effectively manage their time-series data with targeted features like :

  • Open source platform to collect, store, manage and process time-series data at scale
  • Easy install with no external dependencies
  • In-Database processing (Streaming, derivatives, percentiles) with Continuous queries
  • In-Memory processing (Only values are written to / read from disk. Tag and measurement elements written to WAL in memory and SS tables)
  • MPP style processor logic, Pattern matching, Anomaly detection and static/dynamic thresholding. Additional capability to run ETL type jobs.
  • Full customizable layout and template driven dashboarding with query builder to extract any arbitrary metrics realtime

 

Collecting Real-Time Analytic Data

Real-time analysis means having to collect data from disparate systems, applications, datasources, services and infrastructure components. InfluxData’s Telegraf collector supports 30+ inputs and 10+ outputs and can be easily extended to support your sources of data. Telegraf makes collecting data in a format InfluxDB can consume, simple. Here’s why:

  • MIT License
  • Minimal memory footprint
  • Extensible plugin design with 40+ input and output plugins
  • Support for datasources like MongoDB, MySQL and Redis
  • Messaging systems like Apache Kafka and RabbitMQ
  • Third party APIs like Mailchimp, AWS CloudWatch and Google Analytics
  • Collects system metrics like CPU, Memory, I/O, etc


However, the InfluxData platform is extensible by design so you can easily integrate other collection agents like collectd, in conjunction with Telegraf.

Learn more about Telegraf

Storing Real-Time Analytic Data

The most popular data type in any real-time analytics system is going to be in a time-series format. InfluxDB is designed from the ground up to handle just time-series data and to do it better than any other database. InfluxDB is the “I” in the TICK stack. More specifically, InfluxDB is an open source database written in Go to handle time-series data with high availability and high performance requirements. InfluxDB installs in minutes without external dependencies, yet is flexible and scalable enough for complex deployments. Here’s why InfluxDB is the best choice for storing a custom monitoring solution’s time-series data:

  • MIT License
  • Simple to install, yet highly extensible
  • Purpose built for time-series data, no special schema design or custom app logic required
  • Thousands of writes per second with the new TSM1 storage engine
  • Horizontal clustering for high availability in active development
  • A native HTTP API means no server side code to manage
  • Time centric functions and an easy to use SQL­-like query language
  • Data can be tagged, allowing very flexible querying
  • Answer queries in real­time with every data point indexed as it comes in and immediately available in less than 100ms


Learn more about InfluxDB

Visualizing Real-Time Analytic Data

If you don’t already have a dashboarding or graphing UI in place, InfluxData provides Chronograf. It’s the “C” in the TICK stack. Chronograf is a downloadable binary you install behind your firewall to collaboratively, yet securely, perform ad-hoc visualizations on your time-series data. Features include:

  • Simple installation and configuration
  • Tight integration with InfluxDB making getting connected to data easy
  • Support for ad-hoc visualizations
  • Smart query builder designed to work with large datasets
  • Collecting multiple graphs into dashboards
  • Templating, new graph types and visualizations coming!


Another visualization UI choice that offers tight integration with InfluxDB is the open source Grafana project. Either choice makes connecting to and visualizing time-series data, simple.

Learn more about Chronograf
Learn more about InfluxDB & Grafana

Processing Real-Time Analytic Data

Inevitably, you are going to want to either alert on or in some way process the time-series data in your real-time analytics system. You’ll want to do this either before it gets written to InfluxDB or when it is retrieved. To address this need, the InfluxData platform ships with the open source Kapacitor project. Kapacitor is the “K” in the the TICK stack. It’s an alerting and data processing engine specifically designed for time-series data. It lets you define your own custom pipeline to aggregate, select, transform or otherwise process data and then store it back in InfluxDB or trigger an event. Features include:

  • MIT licensed
  • Stream data from InfluxDB or query from InfluxDB
  • Trigger events/alerts based on complex or dynamic criteria
  • Perform any transformation currently possible in InfluxQL, for example: SUM, MIN, MAX, etc.
  • Store transformed data back into InfluxDB
  • Process historical data, for example: backfill data using a processing pipeline


Learn more about Kapacitor

Testimonials

InfluxData is used for real-time analytics by startups and large enterprises alike. Visit our Testimonials page for a comprehensive list.


Next: IoT and Sensor Data

InfluxCloud

InfluxDB Clusters + Grafana on AWS

14 Day Free Trial

InfluxEnterprise

Highly-Scalable InfluxDB Clusters on Your Infrastructure with a Management UI

Learn More