cubo-profile

Downsample Your Data in InfluxDB

Template built by

Telegraf Plugins used:

  • Any Telegraf plugin

Included Resources:

  • Downsampling Buckets:
    `telegraf`,
    `telegraf_1m`,
    `telegraf_10m`,
    `telegraf_1h`,
    `telegraf_1d`
  • Telegraf Configurations for each input plugin
  • 1 Dashboard: `Downsampling Status`
  • 1 Task: `00. Downsample Monitoring Task`

Downsample your Data in InfluxDB Dashboard

Downsampling is the process of reducing the sampling rate in your data while maintaining its length with respect to time. This can help reduce the disk space needed for your data yet maintain the data’s shape for viewing and understanding your historical data.

Why downsample your data in InfluxDB?

One of the most common use cases for InfluxDB tasks is downsampling data to reduce overall disk usage as data is collected over time. This InfluxDB Template provides a set of tasks that allow you to downsample your data for common Telegraf input plugins. It also includes a dashboard and a custom monitoring task to ensure everything is running smoothly.

Structure

This isn't a template that you install with a single command. Each subfolder contains a telegraf configuration as well as a set of tasks to perform downsampling on that telegraf configuration. The following assumptions have been made:

  • The bucket name for all Telegraf metrics is simply telegraf. If you are storing your Telegraf metrics in a different bucket, you will need to update the fromBucket variable at the top of each Task.
  • The naming convention for the downsampled buckets is telegraf_interval. For example, for 1h rollups, the data is stored in telegraf_1h.
  • The metrics being calculated for each field are as follows:
    • String Fields: last, count
    • Numeric Fields: last, count, mean, min, max>, sum, p99.9, p99, p95, p90
    • Bool Fields: last, count, ratio (count(true)/count())
  • All data is downsampled from the raw data bucket (telegraf) which is kept forever but can (and should) be changed. The minimum required time to keep the bucket to get the rollups is 24h + 5m to get the downsampled values.
  • The tasks are named num.
    downsample_plugin_every

How to use the Downsample Your Data in InfluxDB Template

Once your InfluxDB credentials have been properly configured in the CLI, you can install the Downsample Your Data in InfluxDB Template using the Quick Install command. There are two paths you can take with this template:

  • If you have a specific Telegraf plugin you would like to downsample your data for, you can leverage the specific downsampling tasks for that plugin located in the folders (see the list below). You might also choose this path if the task for downsampling everything is timing out.
  • If you don't really care and would like to downsample all data from all plugins, you can use the tasks in the all_inputs directory. It will grab all the measurements this template can downsample data for and process them.

Note: You should not install the all inputs tasks as well as the individual plugin downsampling tasks. Just choose one or the other.

Telegraf Plugin Subdirectory
all all_inputs
cpu default_inputs
disk default_inputs
diskio default_inputs
internal internal_input
kernel default_inputs
mem default_inputs
net net_inputs
netstat net_inputs
processes default_inputs
procstat procstat_input
smart smart_input
swap default_inputs
system default_inputs
Setup instructions

General instructions on using InfluxDB Templates can be found in the use a template document.

The first thing you need to do to use the templates that are included for each of the Telegraf plugins is to set up the buckets, dashboard and monitoring tasks. You can do that with the following command:

./influx apply -f buckets.yml -f dashboard.yml -f monitoring_task.yml

This will lay the foundation for the other parts of this template. Next, choose either the all plugins approach or a directory for the specific Telegraf input plugin metrics you are interested in downsampling. If you don't see the plugin listed, open a PR and add it.

You can then add the downsampling tasks using the following command:

./influx apply -f default_inputs/downsampling_tasks.yml

You can also host the telegraf config in InfluxDB if you like, but it will probably be more useful to compare that to your telegraf inputs already configured and make any adjustments. If you do want to use the Telegraf Configs provided, you will need to set $INFLUX_TOKEN and $INFLUX_ORG env variables.

  • This can be found on the Load Data > Tokens page in your browser: export INFLUX_TOKEN=TOKEN
  • Your Organization name can be found on the Settings page in your browser: export INFLUX_ORG=my_org

Key Downsample Your Data in InfluxDB metrics to monitor

Some of the most important metrics that you should proactively monitor include:

  • Downsampling Jobs
  • Telegraf Series Counts
  • Telegraf Value Counts

Related Resources

Open Source InfluxDB 2.0 Monitoring Template

Guarantee that your InfluxDB instance is properly functioning with this InfluxDB Template.

Telegraf Monitoring Template

Track the health of your Telegraf plugins in your architecture with this InfluxDB Template.

Open Source InfluxDB 1.x Monitoring Template

Monitor your InfluxDB 1.x instance to guarantee performance and availability.

Scroll to Top