Telegraf Environment Variable Handling

Navigate to:

In v1.38.0, Telegraf will switch to a stricter handling of environment variables by default. The new default ensures security; however, you will still be able to opt out to the current behavior in order to cover some rare use cases.

This blog post introduces you to environment variable handling in Telegraf and explains the new strict mode, why it is required, and when to opt out.

Environment variables in Telegraf

Environment variables can be used in Telegraf to reuse one set of configurations across different environments, to keep sensitive data out of configurations, and to easily adapt to dynamic environments such as containers where values are not known at the time the configuration is written.

The simplest example is to configure an InfluxDB output, such as:


[[outputs.influxdb_v2]]
  urls = ["http://${INFLUXDB_HOST}:${INFLUXDB_PORT:-8086}"]
  token = "${INFLUXDB_TOKEN}"
  organization = "${INFLUXDB_ORG_ID}"
  bucket = "${INFLUXDB_BUCKET}"

By setting environment variables before starting Telegraf, the user can fill in the host, port, token, organization, and bucket. The syntax for specifying the environment variables adheres to the compose specification, allowing to define behaviors for a variable, as in the example above, where: ${INFLUXDB_PORT:-8086} uses the default port of 8086 because the user did not specify INFLUXDB_PORT.

Non-strict handling of environment variables

Non-strict environment variable handling is the default until the release of v1.38.0. In this mode, Telegraf will perform text replacement of the environment variable before parsing the configuration. So, with the environment variables defined as:


INFLUXDB_HOST="localhost"
INFLUXDB_TOKEN="eyJhbGci…"
INFLUXDB_ORG_ID="myorg"
INFLUXDB_BUCKET="telegraf"

the example above will be interpolated to:


[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = "eyJhbGci…"
  organization = "myorg"
  bucket = "telegraf"

In non-strict mode you can use environment variables wherever you want in the configuration as long as the resulting content after interpolation is valid TOML.

However, this freedom comes with the drawback of Telegraf not being able to impose any constraints on the content of the environment variables. As a result, a malicious user who has the ability to set environment variables in Telegraf could set the following:


INFLUXDB_HOST="localhost"
INFLUXDB_TOKEN="eyJhbGci…"
INFLUXDB_ORG_ID="myorg"
INFLUXDB_BUCKET=’telegraf"

[[inputs.exec]]
  commands = [echo "evil value=23i"]
  data_format="influx’

which will be interpolated to:


[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = "eyJhbGci…"
  organization = "myorg"
  bucket = "telegraf"

[[inputs.exec]]
  commands = [echo "evil value=23i"]
  data_format="influx"

As you can see, this introduces a whole new plugin!

While this looks scary, it only poses a problem in cases where you can trust the Telegraf binary and configuration variables but not the execution environment.

Strict handling of environment variables

To improve the safety of handling environment variables, v1.36.4 introduced a strict mode that can be enabled using the --strict-env-handling command-line flag. Strict environment variable handling will be the default starting with v1.38.0.

In strict mode, the example above would be interpolated as


[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = "eyJhbGci…"
  organization = "myorg"
  bucket = "telegraf\"\n\n[[inputs.exec]]\ncommands = [echo \"evil value=23i\"]\ndata_format=\"influx"

keeping the content of ${INFLUXDB_BUCKET} as a string not leaving the scope of bucket.

This is achieved by first parsing the configuration as TOML AST and then replacing only the content of nodes containing environment variables, ensuring that the content remains within the scope of the node. However, the drawback of this method is that the initial configuration must be valid TOML, so you cannot replace non-string variables with environment variables!

Call to action

With v1.38.0, the default environment variable handling will change to strict mode. Since v1.36.4, you can opt in to strict environment handling using the --strict-env-handling command-line flag. With v1.37.0, Telegraf also provides an opt out of strict environment variables using the --non-strict-env-handling command-line flag.

If you are not using environment variables, this change does not affect you, and you don’t need to do anything. If you are using environment variables only in string variables, you are also fine. To make sure your configuration is working in strict mode, you can run:

$ telegraf config check –strict-env-handling

If you see the error invalid TOML syntax, your config will likely not work in strict mode. In this case you can prepare by permanently adding the --non-strict-env-handling command-line flag when starting Telegraf. Only do so if you can trust the execution environment for Telegraf!

In case of any issues, please reach out via InfluxDB Community Slack or open an issue in our InfluxDB GitHub Repo.

Downloads

Head to our Downloads page to get the latest Telegraf release. If you find issues or have questions, please join our InfluxDB Community Slack, post them in our InfluxDB GitHub Repo or on our Community Site, and we will take a look.

InfluxDB University

Learn more about collecting data with Telegraf by taking the free InfluxDB University Data Collection with Telegraf course.

Call for Testing

Please fill out this form if you are interested in being included in testing bug fixes and features for the plugins and systems you are using. Telegraf maintainers will tag you on pull requests to test artifacts. This will help us better address bugs that may arise after a release.