InfluxData Blog - Gunnar Aasen

How to Use Grafana with InfluxDB to Monitor Time Series Data

Gunnar Aasen (InfluxData) — Thu, 02 Nov 2017 10:39:38 -0700

The “open platform for beautiful analytics and monitoring,” Grafana supports various backends that store time series data. One of those backends is InfluxDB. InfluxDB is a time series database built specifically for storing time series data, and Grafana is a visualization tool for time series data. Given this perfect match, Grafana has a tight integration with InfluxDB.

Introduction to time series monitoring with Grafana

Grafana is definitely one of the most popular time series data visualization tools that we recommend using with InfluxDB. Below are some basics on how to set up your Grafana dashboard with InfluxDB, how to use the Grafana InfluxDB solution to get the most out of your time series data, and how to visualize what you want - the way that you want - using InfluxDB time series database.

Grafana dashboard setup for InfluxDB

For starters, download InfluxDB and Grafana. The basic setup is to have InfluxDB and Grafana monitoring connected together. InfluxDB has an API, and typically, that defaults to port 8086 while Grafana’s API is on port 3000. And Grafana will call the InfluxDB API whenever it wants to query data.

When you set up the InfluxData time series platform, you will need a collection agent collecting your metrics. For InfluxDB, use Telegraf which already has over 200 plugins.

Some Grafana InfluxDB setup basics to remember:

InfluxDB is essentially a time series database process that runs on a server. That process can also run on the same box that Grafana runs on. Grafana has a very lightweight server-side application, and most of Grafana monitoring runs in the browser.
It's easiest to set up Grafana and InfluxDB on the same instance, yet if you find you have a very large installation of InfluxDB, a number of Grafana users, or a certain security, or deployment profile within your organization, then setting up InfluxDB and Grafana on separate servers is also perfectly acceptable.
InfluxDB will be the more memory-intensive and CPU-intensive application of the two, simply because a lot of Grafana's work happens in the browser. To maintain best performance, we recommend for Grafana monitoring that you use the latest browsers.

Grafana and InfluxDB setup configurations

A lot of the defaults set up for InfluxDB can be maintained. Note that you can enable query log—which will log all queries when executed or when sent to the InfluxDB API, which can be useful for debugging Grafana issues.

One important thing to note when setting up InfluxDB with Grafana is to set up the coordinator section of the config—specifically, set the maximum number of concurrent queries:

If you have issues with different Grafana users all hitting InfluxDB with different open browsers and sending a bunch of time series database queries through, we recommend setting the max concurrent queries. You can also set query time-outs and set queries to be logged if they take longer than a certain amount of time.
The settings around max-select point, max-select series, and max-select buckets are also very useful in limiting the amount of results that can be returned — thereby preventing a particularly expansive query from potentially taking down the InfluxDB server or causing a slowdown for everyone else.

When exploring Grafana’s configuration file, you’ll find a wealth of other great ways to configure Grafana monitoring to make it more usable. These include setting Grafana http port, router logging, and enabling the user to make your browser pages load faster.

Learn about Grafana security settings

As for security settings, every Grafana instance has a default admin user and default password. If you are setting up Grafana monitoring on anything where anyone is able to get into your instances, set your own custom username or at least password to prevent people from getting access.

Here are some basic Grafana security settings:

By default, Grafana will allow users to sign up and register and also allows non-admin users to create organizations. So we recommend setting the "enable anonymous access" option as "false" to prevent people from setting up users on organizations if you didn't want them to.
Also, anonymous access is disabled by default, but you can enable anonymous access, which is very useful if you have a public Grafana dashboard that you'd like to promote.
If you do a password reset that requires a user to receive an email, you'll want to set that up in the SMTP section.
Grafana also has a number of log levels, so if you're trying to debug, definitely bump up the log level to debug.
Grafana will expose metrics about itself — Telegraf has a Prometheus input built-in so you can direct it towards that and receive or collect internal Grafana metrics, put them into InfluxDB, then graph them again in Grafana.

If you are using InfluxDB Cloud and need to configure access to different groups and users on Grafana to view the time series data that InfluxDB Cloud collects, view the webinar short “InfluxDB Cloud with Multi-Tenant Grafana.”

Setting up graphs for Grafana Metrics

InfluxData’s “How to use Grafana with InfluxDB” webinar explains how to use Grafana UI to set up graphs and use InfluxDB Query Builder.

Here are some webinar highlights on what to expect in building your Grafana dashboard and customizing your graphs:

Once in config, add a data source, select the InfluxDB Type, and give that source a name. You might want to make it the default data source for your Grafana monitoring instance. You pick the InfluxDB URL and enter the database, user, and credentials. Once you get the "success" notification on that, then you add a Grafana dashboard.
Grafana dashboards are based off rows and panels. You can have multiple panels on a row, and you can edit panels.
While in edit mode, you see the Grafana Metrics path, where you specify your InfluxDB query. You can also specify the InfluxDB policy that you want to use.
The default is to use the "value" as a field and "mean", but you can use any aggregator of your choice. You can also select multiple fields, or select the same field twice and use different aggregators.
You can add various kinds of transformation like moving_average.
Another property in the Grafana Metrics tab is how they're GROUPED BY. By default, Grafana GROUPS BY a predefined interval value, which is calculated based on the screen and the panel width. You can also specify how InfluxDB should handle null values.
You can change how to display the graph (as lines, bars, points, etc.) and take advantage of the different display settings.

Optimize time series monitoring with Grafana

InfluxDB and Grafana perform well together, but you can definitely hit some performance issues. If you’re collecting a lot of time series data, or alternatively querying a lot of data and putting some stress on InfluxDB, you’ll typically find that InfluxDB — since it is doing a lot of the server-side work — is the first piece of the Grafana InfluxDB chain to experience issues.

Each Grafana graph is actually a query to InfluxDB. The more graphs you have, the more load there is on the InfluxDB server. So if you have a Grafana dashboard of 30 graphs, that’s 30 queries that you’re sending to InfluxDB and 30 queries that need to get the results collated by InfluxDB, and then sent back through Grafana monitoring. If you have a lot of queries or graphs, think about what you’re graphing and whether all the graphs that you have displayed are actually useful to you.

To further improve the performance of InfluxDB and Grafana, here are some helpful tips:

Set your rows to be collapsible - when you collapse rows, Grafana will not show or render that graph, and therefore won't produce the queries to InfluxDB. So if you happen to have a Grafana dashboard that has a lot of useful graphs, that are perhaps not all useful at once, a great way to reduce or speed up your Grafana monitoring experience with InfluxDB is to use collapsible rows.
There are also several graph types and other options you can add to graphs. For example, if you care only about the number of queries executed, you can just return the latest number of queries that was executed and not a graph over time. You can also do templating and annotations.
Every query generated by Grafana graphs typically has a GROUP BY option, so you'll see a lot of recalculation on each loading of the graph.
If you do a very expansive query to calculate, and InfluxDB takes a while to respond and the query times out and you get an error in your Grafana graph, wait a little before you hit the Refresh button.
InfluxDB and Grafana monitoring both provide many logging options. If Grafana is displaying an error for a graph, go through and figure out why it's producing that error if it doesn't show up after one or two refreshes. InfluxDB doesn't cache queries, so every time you send a query, it recalculates the results. Typically, if you're zooming out or selecting larger time ranges, such as 24 hours, it'll take longer for InfluxDB to pull out all those results. The more data you accumulate, the more data Grafana and InfluxDB have to process to load.

Conclusion

There is no limitation on the number of Grafana metrics displayed on one graph, but remember to track those most useful to you for time series monitoring performance, extracting insights or enabling forecasts. If you’re thinking about how many Grafana metrics you want to display at one time, you might also want to think about what that time series data is actually showing you and what you’re trying to accomplish by reading all that data.

Introduction to InfluxData's InfluxDB and TICK Stack

Gunnar Aasen (InfluxData) — Fri, 22 Sep 2017 11:49:45 -0700

InfluxData provides a Modern Time Series Platform, designed from the ground up to handle metrics and events. InfluxData’s products are based on an open source core. This open source core consists of the projects Telegraf, InfluxDB, Chronograf, and Kapacitor - collectively called the TICK Stack.

What is a Time Series?

A time series is simply any set of values with a timestamp where time is a meaningful component of the data. The classic real world example of a time series is stock currency exchange price data. For example, the chart below (from Coinbase Charts) shows the US dollar to bitcoin exchange price for the past month.

As you can see in the graph above, the USD price of Bitcoin is displayed over time. The underlying data set used to create this graph is composed of many timestamped values, which together form a set of values. This set of values is a time series.

At this point you may be wondering…

Why is This Special?

The graph above displays a couple of thousand points across a month’s worth of data. However, cryptocurrencies like Bitcoin are traded frequently, so the underlying data set used to derive the graph likely contains several tens of millions of data points. In fact, there is usually so much data in most time series that it’s almost always summarized when it is displayed, and a screen doesn’t have enough pixels to display the full granularity of the data. There are other cryptocurrencies too, and thousands of regular currencies being exchanged.

The amount of time series data being generated is simply enormous. This scale has been one of the primary drivers behind the creation of specialized data stores for time series data.

What Exactly is a Time Series Database?

Time Series Databases typically need to solve two problems: high write throughput and high query rates. Let’s go into more depth on these points.

Write throughput: The amount of time series generated by, say, monitoring a fleet of servers can quickly grow from thousands of new values per second to millions of new values per second. Inserting that amount of data into a regular relational database, like MySQL and PostgreSQL, quickly knocks over most systems without careful tuning. Even then, there are limits to the amount of data a system can accept at any one time.
Query throughput: One aspect of time series data is that new data is almost always more valuable than the old data. Again, think of a fleet of servers. Knowing that one server has started to use up all its CPU is much more useful than looking at the evidence even minutes after the CPU spiked. The real-time nature of time series makes it necessary to expose new data in queries as fast as possible. Additionally, not every point matters in a time series. Like the bitcoin chart above, most time series data is summarized into intermediate values because trends provide more information than individual data points.

Because a Time Series Database specializes in processing timestamped data, there are many optimizations available to address the two fundamental issues above along with a few others.

Compression: Since all time series consist of timestamped data, a significant amount of compression is possible.
Query functions: Speed is not the only important thing in queries.

What is InfluxDB?

InfluxDB is a high performance Time Series Database. It can store hundreds of thousands of points per second. The InfluxDB SQL-like query language was built specifically for time series. Check out the InfluxDB documentation to start learning more.

What is the TICK Stack?

The TICK Stack is an acronym for a platform of open source tools built to make collection, storage, graphing, and alerting on time series data incredibly easy. The “I” in TICK stands for InfluxDB. The other components in the platform are:

Telegraf: A metrics collection agent. Use it to collect and send metrics to InfluxDB. Telegraf's plugin architecture supports collection of metrics from 100+ popular services right out of the box.
Chronograf: A UI layer for the whole TICK stack. Use it to set up graphs and dashboards of data in InfluxDB and hook up Kapacitor alerts.
Kapacitor: A metrics and events processing and alerting engine. Use it to crunch time series data into actionable alerts and easily send those alerts to many popular products like PagerDuty and Slack.

The entire TICK Stack is interoperable, yet each component can provide significant value as a standalone installation.

The rest of this guide will explain how to set up the TICK stack on macOS for development and testing.

Step 1: Install the TICK Stack

On macOS, installation of the TICK stack is a breeze using the Homebrew package manager.

brew install telegraf

brew install influxdb

brew install chronograf

brew install kapacitor

The above commands will install several binaries into your path. The important ones are telegraf, influxd (the InfluxDB server), influx (the InfluxDB CLI), chronograf, kapacitord (the Kapacitor server), and kapacitor (the Kapacitor CLI).

Step 2: Configure the TICK Stack

Telegraf, InfluxDB, and Kapacitor all use configuration files. Most configuration values in the TICK Stack do not need to be changed out of the box on macOS. In fact, Chronograf does not require a configuration file and can be configured entirely via CLI flags. Here are the commands to generate a new config file for the other three components:

# The telegraf config generated here will be already set up to only gather CPU, memory, and system metrics.

telegraf --input-filter cpu:mem:system --output-filter influxdb config > /usr/local/etc/telegraf.conf

influxd config > /usr/local/etc/influxdb.conf

kapacitord config > /usr/local/etc/kapacitor.conf

On macOS, InfluxDB, Chronograf, and Kapacitor will store their data in the default directories at ~/.influxdb, ~/.chronograf, and ~/.kapacitor, respecitvely. These locations can be overridden in the configuration files, but do not need to be changed right now.

Step 3: Run the TICK Stack

Now that configuration files have been generated, spinning up the TICK Stack is very easy using Homebrew services.

brew services start telegraf

brew services start influxdb

brew services start chronograf

brew services start kapacitor

This will start all the TICK component processes in the background. The logs for Homebrew services can be found in the /usr/local/var/log directory if you run into any trouble.

Voilà! The TICK Stack is now running. In this setup, Telegraf is collecting CPU and memory metrics and writing them to InfluxDB’s telegraf database, which was automatically created.

There are several ways to interact with the stack. InfluxDB and Kapacitor have APIs available, along with CLI tools to make it easier to interact with their APIs. Additional Telegraf plugins can be configured to gather additional data as well. Finally, Chronograf provides a UI for data in InfluxDB.

Let’s open Chronograf in the browser to see the data being collected. Navigate to http://localhost:8888. This will bring up a configuration page. Using the default settings will automatically connect InfluxDB to Chronograf and allow you to begin exploring the local system data you’ve started collecting in InfluxDB.

Next Steps

To learn more about InfluxDB and the TICK Stack, read through the InfluxDB Getting Started guide. Then take a look at the InfluxDB key concepts doc, which is a great overview of the InfluxDB data model.

Check out the Getting Started guides for Telegraf and Kapacitor. If you have questions or hit errors, we recommend checking out the InfluxData Community. Finally, the TICK Stack is open source and always welcomes new contributors.