InfluxData Blog - Josh Powers

Telegraf 1.31 Release Notes

Josh Powers (InfluxData) — Wed, 12 Jun 2024 08:00:00 +0000

A new feature-bearing release for Telegraf is now available:

Telegraf 1.31 — Release notes

You can find the binaries for the latest Telegraf release on our Downloads page. Many thanks to all the open source community members who contributed to this effort!

New plugins

These are the newest plugins, first available in this version:

OpenMetrics Parser (parsers.openmetrics)
- Parse OpenMetrics-style Prometheus metrics
- Contributed by srebhan
Parquet Parser (parsers.parquet)
- Generate metrics from parsing Parquet files
- Contributed by powersj
Smartctl Output (inputs.smartctl)
- Collect metrics from smartctl JSON output
- Contributed by powersj
Timestamp Processor (processors.timestamp)
- Parse fields containing timestamps into timestamps in other formats
- Contributed by powersj

Important changes

Here are some changes to highlight:

CLI Option: Remote URL retries
- A new --url-config-retry-attempts CLI option lets users set the number of retry attempts starting from a URL-based configuration file. The default remains 3.
- Users can set this -1 to try forever, however, note that Telegraf will block starting until it obtains the configuration.
CLI Option: Reload URL interval
- Introduces the --url-config-check-interval CLI option that checks that the remote URL is updated. The existing --watch-config option is only used for local files.
- This option checks the Last-Modified header to determine if an update is required.
Deprecation Removals
- All deprecation removal messages were updated to either v1.35.0 (June 2025) or v1.40 (September 2026).
- If you find any issues with the replacements, please file issues.
Retry Errors on Startup
- In general, if Telegraf detects errors connecting to inputs, it will fail to start. This is to ensure the user is aware of the error(s) and prevent the user from assuming everything is working when Telegraf is not collecting everything configured.
- Some users may want to collect whatever is possible. To enable this, a new set of startup error behavior configuration options was introduced and added to a number of plugins. These options allow the user to either ‘error’ the current behavior, ‘retry’ to allow additional retries of the plugin, or ‘ignore’ to disable that plugin from further attempts.
Procstat with Multiple Selection Criteria
- The proctstat input plugin now allows the user to define multiple filter selection criteria.
- With this new capability, users can define more finely tuned search parameters to reduce the output and generated metrics.
Procstat Fields
- PR #15186 changes the meaning of inputs.procstat fields read_bytes and write_bytes on Linux to now contain all I/O operations for consistency with other operating-systems. The previous values are output as disk_read_bytes and disk_write_bytes measuring only the I/O on the storage layer.

Downloads

Head to our Downloads page to get the latest Telegraf release. If you have issues or questions, please join our InfluxDB Community Slack, post them in our InfluxDB GitHub Repo, or our Community Site, and we will take a look.

InfluxDB University

Learn more about collecting data with Telegraf by taking the free InfluxDB University Data Collection with Telegraf course.

Telegraf 1.30 Release Notes

Josh Powers (InfluxData) — Fri, 15 Mar 2024 08:00:00 +0000

A new feature-bearing release for Telegraf is now available:

Telegraf 1.30 — Release notes

You can find the binaries for the latest Telegraf release on our Downloads page. Many thanks to all the open source community members who contributed to this effort!

Removal of deprecated plugins and options

For the past few years, we warned users that we would remove a number of very old, deprecated plugins and plugin options in v1.30. That time has finally come. Users have replacements available to them and can use the migrate command to attempt to auto-migrate their configuration.

The full list of removed plugins and options includes:

Plugins
- Inputs.cassandra replaced with inputs.jolokia2
- Inputs.httpjson replaced with inputs.http
- Inputs.io replaced by inputs.diskio
- Inputs.jolokia replaced by inputs.jolokia2
- Inputs.kafka_consumer_legacy replaced by inputs.kafka_consumer
- Inputs.snmp_legacy replaced by inputs.snmp
- Inputs.tcp_listener replaced by inputs.socket_listener
- Inputs.udp_listener replaced by inputs.socket_listener
- Outputs.riemann_legacy replaced by outputs.riemann
Config Options
- Inputs.disk: mountpoints replaced by mount_points
- Inputs.mqtt_consumer: metric_buffer is ignored
- Inputs.nats_consumer: metric_buffer is ignored
- inputs.influxdb : url replaced by urls

New plugins

These are the newest plugins, first available in this version:

Zabbix Output (outputs.zabbix)
- Send metrics to Zabbix Network Monitoring
- Contributed by adrianlzt
SNMP Lookup Processor (processors.snmp_lookup)
- Look up and cache extra tags using SNMP, replaces ifname processor
- Contributed by hipska
Binary Serializer (serializers.binary)
- Serialize metrics into binary protocol based on user-configuration
- Contributed by ignatella

Important changes

Here are some changes to highlight:

Deprecations
- A number of long, deprecated plugins were removed.
- Please see the section below for additional details.
Option to skip re-running processors after aggregators
- Introduces a new agent config option to skip re-running processors after aggregators.
- By default, after aggregators are run, processors are run a second time. This can lead to unexpected results.
Kernel Pressure Stall Info
- The kernel input plugin was updated to collect PSI information via /proc/pressure
syslog Input Timeout
- The default read-timeout of syslog was not sensible and could have closed the connection between messages exceeding the timeout.
- The new default sets this to infinite (i.e. zero).
Postgresql Input Tag
- The postgresql plugin was not correctly sanitizing parameter values containing spaces or equal signs.
- This has been resolved, but could result in new tag values.
loongarch64 builds
- Introduces loongarch64 builds for nightly builds.
- Will not be a part of release packages initially.

Downloads

Head to our Downloads page to get the latest Telegraf release. If you find issues or have questions, please join our InfluxDB Community Slack, post them in our InfluxDB GitHub Repo, or our Community Site, and we will take a look.

InfluxDB University

Learn more about collecting data with Telegraf by taking the free InfluxDB University Data Collection with Telegraf course.

Telegraf 1.29 Release Notes

Josh Powers (InfluxData) — Wed, 13 Dec 2023 08:00:00 +0000

A new feature-bearing release for Telegraf is now available:

Telegraf 1.29 — Release notes

You can find the binaries for the latest Telegraf release on our Downloads page. Many thanks to all the open source community members who contributed to this effort!

New Plugins

These are the newest plugins, first available in this version:

Filter Processor (processors.filter)
- Set rules with the ability to keep or drop specific metrics
- Contributed by srebhan
LDAP Input (inputs.ldap)
- Monitor LDAP servers’ monitoring backend
- Contributed by srebhan
OpenSearch Output (outputs.opensearch)
- Send metrics to OpenSearch
- Contributed by mannukalra
Systemd Secret Store (secretstores.systemd)
- Secret store to use systemd as a backend
- Contributed by srebhan

Important Changes

Here are some changes to highlight:

Secrets Global Option
- Introduced a new --unprotected global flag.
- When set, this flag instructs Telegraf to not attempt to reserve locked memory pages. This option might pose a security risk as decrypted secrets might be swapped out to disk, but allows for execution in environments where reserving locked memory is not possible.
Deprecate fieldpass and fielddrop modifiers
- In order to better align the naming conventions, the metric modifiers fieldpass and fielddrop are replaced with fieldinclude and fieldexclude.
- All options will continue to work, but the documentation and examples were updated with the new terms.
Oracle and IBM Netezza SQL Support
- The SQL input plugin now has drivers for both Oracle and IBM Netezza.
Stackdriver Distributions
- The Stackdriver output can now take histograms and send them as a Stackdriver distribution.
- This only supports Prometheus metric version 1 histograms. The sparse layout of version 2 histograms requires using the merge aggregator.
Removed all-zero fields from inputs.procstat
- Up to now, Telegraf reports the fields cpu_time_guest, cpu_time_guest_nice, cpu_time_idle, cpu_time_irq, cpu_time_nice, cpu_time_soft_irq and cpu_time_steal, which the underlying library never sets. As a consequence, those fields were always zero.
Introduction of Migrations
- Users can migrate several deprecated plugins and plugin options using the new telegraf config migrate command. This works for all plugins and options scheduled for removal in v1.30.0 (see next section).
- For each given configuration file, the command generates a migrated version with a .migrated suffix.
- Please check the modifications and test the migrated version before using it!
- Please also check the output of the command carefully as some migrations require further manual intervention due to ambiguous settings, adapted metric formats, or complex setups.
- Note: The migration command will not keep comments for migrated plugins!

Deprecation Removals

This is the final release before the planned removal of several plugins and options. Users have been warned about these removals via warnings for multiple years. The list of removals includes:

Plugins
- inputs.cassandra replaced with inputs.jolokia2
- inputs.httpjson replaced with inputs.http
- inputs.io replaced by inputs.diskio
- inputs.jolokia replaced by inputs.jolokia2
- inputs.kafka_consumer_legacy replaced by inputs.kafka_consumer
- inputs.snmp_legacy replaced by inputs.snmp
- inputs.tcp_listener replaced by inputs.socket_listener
- inputs.udp_listener replaced by inputs.socket_listener
- outputs.riemann_legacy replaced by outputs.riemann
Config Options
- inputs.disk: mountpoints replaced by mount_points
- inputs.mqtt_consumer: metric_buffer is ignored
- inputs.nats_consumer: metric_buffer is ignored
- inputs.influxdb : url replaced by urls

Downloads

InfluxDB University

Learn more about collecting data with Telegraf by taking the free InfluxDB University Data Collection with Telegraf course.

Using the Common Expression Language for Metric Filtering with Telegraf

Josh Powers (InfluxData) — Thu, 22 Jun 2023 07:35:00 +0000

Telegraf is an open-source plugin-driven agent for collecting, processing, aggregating, and writing time series data. When collecting metrics it is common to filter out or pass through metrics with specific names, tags, fields, or timestamp values.

The Common Expression Language (CEL) is an open-source language that provides a set of semantics for expression evaluation. Kubernetes users may already be familiar with the language as it is used to declare validation rules, policy rules, and other constraints or conditions.

In v1.27, Telegraf now has the ability to parse CEL expressions to filter metrics. The following post outlines how users can take advantage of CEL to filter metrics easier than ever with Telegraf.

Telegraf configuration

The new metricpass option enables CEL filtering. This is similar to the existing namepass or tagpass options.

Telegraf evaluates the metrics based on the expression. Metrics that match the expression are passed on, while other metrics are dropped.

[[inputs.cpu]]
  metricpass = “fields.usage_idle < 90”

This option is valid for any input, output, processor, and aggregator plugin. For inputs the filtering takes place at the very end of an input, while for the other plugin types the filter is run before those plugins are run.

While the CEL language is incredibly powerful, it does not alter metrics. It only determines which metrics should continue on from an input or to other plugins.

CEL syntax by example

The following are a set of examples to introduce CEL usage for Telegraf. For the complete set of types, logic, and functions users can reference the cel-spec language definition in the CEL repo. There is even a CEL Discussion Forum for help with specific questions about the language.

Referencing metric components

In order to filter metrics, users can reference specific components of a metric. Consider the following example metric:

example,host=a value=42 1686252961000000000

The following table shows how to reference the specific components of the metric:

Component	CEL Reference	Example
Metric name	name	name == “example”
Tags	tags	tags.host == “a”
Fields	fields	fields.value > 0
Timestamp	time	time >= now() - duration(“24h”)

Logical operators

CEL supports C- and Go-style logic operators to make comparisons. The following example checks that the field “id” exists and the field “id” contains the substring “nwr”.

"id" in fields && fields.id.contains("nwr")

This example ensures the metric name does not start with a “t” and that the tag “state” is equal to “on”.

!name.startsWith("t") || tags.state == "on"

Numeric operations

For numeric values, users can do both arithmetic and equality operations. For example:

fields.value / 8 >= 1024
	fields.bits % 2 == 0

String operations

To aid in testing strings, users have a number of helper functions available. The following table lists a few common ones:

Function	Test Description	Example
contains	String contains substring	tags.host.contains(“domain”)
endsWith	Test string suffix	tags.id.endsWith(“.com”)
lowerAscii	Lower case string	metric.name.lowerAscii()
matches	Compare string against a regular expression	tags.source.matches(“^[0-9a-zA-z-_]+@mycompany.com$”)
startsWith	Test string prefix	tags.host.startsWith(“subdomain”)
size	String length	tags.id.size()

For the full list of functions, check out string.go source, which includes examples as well.

Time operations

A possible use case for this type of filtering would be to remove metrics older than a specific date. CEL includes a variety of helper functions to pick apart a date. For example:

time.getFullYear() == 2023

There is a large list of these helper functions available on the CEL list of standard definitions.

Additionally, a user can compare the current time to the timestamp. With this a user can see if a timestamp is less than a day old:

time >= now() - duration(“24h”)

Types

To test for a specific type, the type() function returns what a particular tag or field is. This might be useful to ensure a field is a numeric type:

type(fields.bits) in [double, int, uint]
type(fields.cluster) == string

The first line ensures that the bits field is of a numeric type, while the second ensures the cluster field is a string.

Performance impact

Users should keep in mind that CEL usage comes at a cost. CEL is an interpreted language and uses overhead when implemented. If you only filter based on a specific tag or metric name, continue to use the namepass/namedrop and tagpass/tagdrop configuration options to achieve the best performance. However, in most cases, these CEL filters should be faster than using the Starlark processor to do the same task.

Get started with metric filtering and Telegraf

With the addition of the Common Expression Language for metric filtering in v1.27, it is now even easier for users to customize their metrics. Download Telegraf and give the new metricpass option a try today!

Storing Secrets with Telegraf

Josh Powers (InfluxData) — Thu, 15 Jun 2023 07:35:00 +0000

Telegraf is an open source plugin-driven agent for collecting, processing, aggregating, and writing time series data. Telegraf relies on user-provided configuration files to define the various plugins and flow of this data. These configurations may require secrets or other sensitive data.

The new secret store plugin type allows a user to store secrets and reference those secrets in their Telegraf configuration file. These stores alleviate the need to have secrets directly in the Telegraf configuration file.

The following post provides an overview of the Secret Store feature of Telegraf and includes a few real-life examples.

Secret Stores

As of v1.27, Telegraf has a few Secret Store plugins to choose from:

Docker
- When running Docker, this plugin can read Docker provided secrets.
- These are values stored in /run/secrets on the container.
HTTP
- Query secrets from an HTTP endpoint.
- The format of the data is expected to be a flat JSON object.
- Supports a variety of encryption methods and authentication.
JOSE
- Local encrypted files using the JavaScript Object Signing and Encryption algorithm.
- Users can use the telegraf secrets set to create secrets.
OS
- Interact with OS-specific secret stores.
  - Linux uses kernel keyrings.
  - macOS works with the macOS Keychain.
  - Windows interacts with the Windows Credential Manager control panel.

Plugin support

Telegraf plugins support secret stores only with specific fields. To learn if a plugin supports secret stores, look at the plugin’s README and look for the “Secret-store support” section. This details the plugin’s secret store capabilities and which configuration options it supports.

User access

When setting up secrets, another key item to consider is that the user running Telegraf needs access to the secrets. When creating a secret store and running Telegraf as a service, be sure to give the commonly used telegraf user access to those secrets.

Getting started checklist

Below is a set of steps to migrate an existing Telegraf configuration to one that supports secret stores:

Verify plugin support
- As mentioned above, ensure that the plugin(s) you want to use supports secret stores.
- Then, ensure that plugin(s) support the configuration options you need.
Choose a secret store
- Decide if you’re going to integrate with the OS secret stores or use an external store.
Add secret store to Telegraf config
- Add the appropriate secret store plugin to the Telegraf config.
- Each secret store requires a unique id that configurations using the secrets can reference.
- Set up any configuration options necessary to reference the secrets.
Save secrets to that store
- Telegraf’s secrets subcommand can write secrets for some, but not all the stores.
- Ensure that the plugin successfully stored your secrets and that the secrets are accessible by the user running Telegraf. Keep in mind that some stores may not persist over reboots.
Update config to use secrets
- Finally, update the Telegraf configuration to switch from hard-coded credentials.
- Use the @{secretstore_id:secret_key} syntax to reference the secret stores for a configuration value.

Examples

Below are a few examples using different secret store plugins:

Linux OS example

The following sets up secrets using a user’s kernel keyring on a Linux machine.

First, update the Telegraf config to include the OS secret store plugin. The following gives the secret store the ID “mystore”, which we will use to reference the secrets later. Because you can have multiple secret stores, the ID is what makes the stores unique for identification purposes. The keyring value is the user’s kernel keyring to use:

[[secretstores.os]]
   id = "mystore"
   keyring = "telegraf"

To create secrets, use keyctl or the Telegraf binary. The secrets set subcommand takes the keyring name used above and the secret name. Then the user is prompted for the secret information to avoid putting the secret in their shell history:

$ telegraf --config config.toml secrets set mystore influx_token
Enter secret value:

Finally, to update the Telegraf configuration to reference the new secret:

[[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:8086"]
  token = "@{mystore:influx_token}"
  organization = "myorg"
  bucket = "mybucket"

Docker example

The following is an example of pulling secrets via Docker secrets. The Docker secret store differs from the other stores in that it only reads secrets provided by Docker itself at runtime. These secrets mount to the /run/secrets directory of a container. A user cannot use Telegraf to set Docker secrets.

Below is an example of a user storing the secrets in a file, which the Docker Compose config then references under secrets. Then update the Telegraf service to use the that secret:

services:
  telegraf:
    image: telegraf:latest
    secrets:
      - influx_token
    user: “${USERID}” # Use the value of `id -u` of the user launching the docker-compose

secrets:
  influx_token:
    file: influx_token.txt

The final Telegraf config used to access the Docker secret is similar to the code below:

[[secretstores.docker]]
  id = "docker_store"

[[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:8086"]
  token = "@{docker_store:influx_token}"
  organization = "myorg"
  bucket = "mybucket"

HTTP example

The following is an example of pulling secrets via an HTTP endpoint.

The HTTP secret store plugin can call out to an HTTP endpoint with a variety of authentication or encryption options. The plugin expects the secrets to be in a flat key-value JSON file where the key is the secret name and the value is the actual secret. Here is an example of the format:

{
    “influx_token_dev”: “6702bf65b31941f59dfcf09afd94b1aa”,
    “influx_token_test”: “1af461d979344253824cbbaf88dd2563”,
    “influx_token_prod”: “56372127fe944e68a497bad0a6e33663”
}

To add this to Telegraf, provide a unique ID for the store and the URL to query for the secrets:

[[secretstores.http]]
  id = "http_store"
  url = "http://127.0.0.1/secrets"

Again, there are many options available for the plugin, including: custom headers, bearer tokens, basic authentication, oauth2, proxy, TLS config, cookie support, and much more. You can send encrypted secrets to Telegraf and, with the appropriate cipher and key, decrypt the secrets themselves.

Finally, add the reference to the secrets to the config:

token = "@{http_store:influx_token_prod}"

JOSE example

The following is an example of pulling secrets via the JavaScript Object Signing and Encryption (JOSE) algorithm. This secret store stores the secrets locally in encrypted files, one for each secret. To decrypt the files, you have to provide the password either in the Telegraf configuration file or when running Telegraf, interactively enter the password. As a result, this secret store option is better thought of as a security through obscurity.

[[secretstores.jose]]
  id = "jose_store"
  path = "/etc/telegraf/secrets"

To create secrets users can use the Telegraf binary:

$ telegraf --config config.toml secrets set jose_store influx_token
Enter secret value:
Enter passphrase to unlock "/etc/telegraf/secrets":
$ ls /etc/telegraf/secrets
influx_token

Then, like other secret stores, update the Telegraf configuration with the reference to the secret:

token = "@{jose_store:influx_token}"

On start, if the password to the store is not provided in the configuration file, the user is prompted to enter the password:

$ ./telegraf --config config.toml 
Enter passphrase to unlock "/etc/telegraf/secrets":

Get started with Secret Stores and Telegraf

This post laid out a number of ways to use the new secret store functionality in Telegraf. Consider where you might be able to use this functionality in Telegraf and give it a shot today!

Telegraf 1.27 Release Notes

Josh Powers (InfluxData) — Tue, 13 Jun 2023 07:35:00 +0000

A new feature-bearing release for Telegraf is now available:

Telegraf 1.27 — Release notes

You can find the binaries for the latest Telegraf release on our Downloads page. Many thanks to all the open source community members who contributed to this effort!

Key new features

Below are a few new features of v1.27 worth highlighting:

Common Expression Metric Filtering
- A new metricpass filter option that accepts a filter expression using the Common Expression Language (CEL).
- Provides users with a powerful way to filter any aspect of a metric.
Configuration File Parameter Expansion
- Config file environment variable substitution now supports additional parameter expansion.
- This allows for setting default values and error handling when a value is not set.
Netflow sFlow decoder
- The Netflow Input Plugin can now parse sFlow v5 messages.
- This will eventually replace the sFlow specific input plugin.
SQL FlightSQL support
- The SQL Input Plugin added support for FlightSQL.
- The allows users to query against the new InfluxDB v3.
MQTT layouts
- The MQTT output now supports specifying the layout format of metrics.
- This option can allow the user to select from a set of high-level layouts that may better represent their data.

New plugins

These are the newest plugins, first available, in this version:

ctrlX Data Layer Input (inputs.ctrlx_datalayer)
- Gather data from communication middleware running on ctrlX CORE devices.
- Contributed by boschrexroth
Intel Baseband Accelerator Input (inputs.intel_baseband)
- Collects metrics from both dedicated and integrated Intel devices that provide Wireless Baseband hardware acceleration.
- Contributed by bkotlowski
Clarify Output (outputs.clarify)
- Write data to Clarify.
- Contributed by bbergshaven
Nebius Cloud Monitoring Output (outputs.nebius_cloud_monitoring)
- Send aggregated metrics to Nebius.Cloud Monitoring.
- Contributed by abrekhov
Scale Processor (processors.scale)
- Processor to scale values from a specified range into another range.
- Contributed by PaddyKe
Docker Secret Store (secretstores.docker)
- Secret store to use Docker Secrets as a backend.
- Contributed by shantanoo-desai
HTTP Secret Store (secretstores.http)
- Secret store to query secrets from an HTTP server.
- Contributed by srebhan
CloudEvents Serializer (serializers.cloudevents)
- Produce metrics into the CloudEvents JSON format.
- Contributed by srebhan

Important changes

There were two changes that could possibly impact a small number of users:

Time parsing with abbreviated time zones
- Up until this release, Telegraf was not parsing date times that included an abbreviated time zone (e.g. mst) correctly.
- If a user is using this type of date time, a warning will print as the behavior will now correctly parse these.
Parser and Serializer Structure
- This was an internal-only change.
- End users should not see any differences unless they use Telegraf like a library.

Deprecation removals

Users may have already seen some deprecation warnings for v2.0. Many of these messages were updated to set removal for v1.30, ~March 2024. This will be the first set of removals focusing on settings and plugins deprecated before v1.0.

Downloads

InfluxDB University

Learn more about collecting data with Telegraf by taking the free InfluxDB University Data Collection with Telegraf course.

Telegraf 1.26.0 Release Includes Static Linux Builds, State Persistence, OpenSearch Query, and Windows Management Instrumentation

Josh Powers (InfluxData) — Thu, 16 Mar 2023 07:00:00 +0000

A new feature-bearing release for Telegraf is now available:

Telegraf 1.26.0 — Release notes

You can find the binaries for the latest Telegraf release on our Downloads page.

Many thanks to all the open source community members who contributed to this effort!

Key new features

Static Builds
- Linux builds are now statically built. Other operating systems were cross-built in the past and as a result, already static.
- Users should not notice any change in behavior.
- The _static specific Linux binary is no longer produced as a result.
Default Configuration
- The telegraf config command and default config file provided by Telegraf now includes all plugins and produces the same output across all operating systems.
- Plugin comments specify supported platform(s).
State Persistence
- State persistence is now available in select plugins.
- This allows plugins to start collecting data where they left off. A configuration with state persistence cannot change or it will not be able to recover.

New inputs

OpenSearch Query (opensearch_query)
- Gather metrics from OpenSearch query endpoints.
- Contributed by NullOranje.
P4 Runtime (p4runtime)
- Collects metrics from P4 programmable pipelines over gRPC.
- Contributed by jokuniew.
Radius (radius)
- Collects authentication response time metrics from Radius.
- Contributed by Hr0bar.
Windows Management Instrumentation (win_wmi)
- Enable queries of Windows Management Instrumentation (WMI) classes.
- Contributed by julesroussel3.

New parsers

Apache Avro (avro)
- Support the Apache Avro format.
- Contributed by athornton.

New processors

Lookup (lookup)
- Read specific files and apply a table of annotations to metrics.
- Contributed by srebhan.

Downloads

InfluxDB University

Learn more about collecting data with Telegraf by taking the free InfluxDB University Data Collection with Telegraf course.

Call for testing

Please fill out this form if you are interested in being included in testing bug fixes and features for plugins and systems you are using. Telegraf maintainers will tag you on pull requests to test artifacts. This will help us better address bugs that may arise after a release.

InfluxDB SQL Queries with Python

Josh Powers (InfluxData) — Tue, 21 Feb 2023 07:00:00 +0000

Recently InfluxData announced SQL support in InfluxDB Cloud. Users can now use familiar SQL queries to explore and analyze their time series data. The SQL support was introduced along with the usage of Apache Arrow.

Apache Arrow is an open source project used as the foundation of InfluxDB’s SQL support. Arrow provides the data representation, storage format, query processing, and network transport layers. Apache Flight SQL provides a method for interacting with Arrow via SQL.

The Flight SQL DB API library provides a seamless user experience for users interacting with Apache Flight SQL in Python. This library can be used directly with InfluxDB with SQL support.

The following demonstrates an end-to-end example of using the library with InfluxDB Cloud. Users who need to execute other operations, like inserting data, querying with Flux, or any other API requests, can continue to use the existing influxdb-client-python client library.

Note: FlightSQL is useful for writing database agnostic code for different databases that support it. This more general approach is what this article covers. For example, if you are creating a UI or other user experience that might query different kinds of databases. If your intent is to only use InfluxDB, we suggest using the upstream Flight libraries directly, because they have simpler dependencies and support both InfluxQL and SQL.

Install the library

The library is published on PyPI in order to make it easy to install:

pip install flightsql-dbapi

To use the library, users need to import the FlightSQLClient in their code:

from flightsql import FlightSQLClient

Connecting to InfluxDB Cloud

To create a client for interacting with InfluxDB, users need to provide three pieces of information:

host: The hostname of the InfluxDB Cloud instance — note that this does not require a protocol (e.g. “https://”) or a port (e.g. 8086). The underlying library builds the full connection string.
token: This is the InfluxDB token string with access to the bucket.
bucket: In the client metadata, users need to provide what InfluxDB bucket to use with the connection; each connection then belongs to a specific bucket.

Below is an example of this connection process, where a user connects to InfluxDB Cloud and uses a token from the environment called “INFLUX_TOKEN” to connect to the “telegraf-monitoring” bucket:

client = FlightSQLClient(
    host="us-east-1-1.aws.cloud2.influxdata.com",
    token=os.environ["INFLUX_TOKEN"],
    metadata={"bucket-name": "telegraf-monitoring"},
)

From here users can use the returned client to interact with InfluxDB’s SQL support. For example, a user could execute queries on, or query information about the bucket’s tables and schema.

Executing a query

Passing a query to the client is a straightforward process of passing the SQL query string to the execute() function. However, the following step is unique to Flight SQL, which requires gathering a ticket from an endpoint:

info = client.execute("select * from cpu limit 10")
reader = client.do_get(info.endpoints[0].ticket)

In Apache Flight SQL, once a query is passed to the server, the server returns a FlightInfo object. The object contains a list of endpoints describing locations containing the query result. In the case of InfluxDB Cloud, there will only be a single endpoint. The client can then gather the query result from the information provided by the single endpoint’s ticket.

Query result handling

The response from the client is of the type FlightStreamReader. This allows the user to stream the data to avoid filling up memory and to be more efficient:

for batch in reader:
    print(batch)

If the user only has a limited number of results from the query, they can use the read_all() function to collect all data chunks into a Table:

data = reader.read_all()

The return value from this is a PyArrow Table. These tables are groups of arrays representing columns of data in tabular form. The table provides a huge number of performant operations and abilities to convert to other data types.

For example, a user can convert the table directly into a Pandas DataFrame. This allows users to quickly manipulate, transform, and analyze their data using the powerful Pandas library:

reader.read_all().to_pandas()

Users of the Polars library can also take advantage of Arrow support directly. A user can create a Polars DataFrame directly from the resulting Arrow Table:

polars.from_table(reader.read_all())

Schemas and tables

When working with SQL it is helpful to know what schema or tables are available. To find that information when working with InfluxDB Cloud, users can use the following scripts to gather the tables:

info = client.get_tables()
tables = client.do_get(info.endpoints[0].ticket).read_all()

Or the following for the full schema:

info = client.get_db_schemas()
schema = client.do_get(info.endpoints[0].ticket).read_all()

As the table and schema information is relatively small, using read_all() here gets the user the data immediately.

Try out SQL with InfluxDB and Python today

This post demonstrated how users could take advantage of the Python Flight SQL library to connect and retrieve data from InfluxDB Cloud, powered by IOx. Users can now take advantage of familiar SQL queries with the leading time-series database and report out using Python.

Consider where you can use InfluxDB Cloud with SQL support and the Python Flight SQL library and give it a shot today!

Telegraf 1.25.2 Release Includes Secret Stores, libvirt, NetFlow, and OPC UA Event Subscriptions

Josh Powers (InfluxData) — Thu, 16 Feb 2023 07:00:00 +0000

A new feature-bearing release for Telegraf is now available:

Telegraf 1.25.0, 1.25.1, and 1.25.2 — Release notes

The binaries for the latest Telegraf release can be found on our Downloads page.

Many thanks to all the open source community members who contributed to this effort!

Key new features

Secret Store - Users can now use secret stores to fill in credentials or other secrets in their config. A user needs to configure one or more secret store plugins and then reference those secrets in their plugin configuration. See the Secret Store configuration docs for more information.

New inputs

Azure Monitor (azure_monitor)
- Gather metrics from Azure Monitor API.
- Contributed by ShiranAvidov.
Google Cloud Storage (gcs)
- Collects metrics by iterating files located on a cloud storage bucket.
- Contributed by gkatzioura.
Intel DLB (intel_dlb)
- Reads metrics from DPDK using the telemetry v2 interface.
- Contributed by bkotlowski.
Libvirt (libvirt)
- Collects statistics from virtualized guests using the virtualization libvirt API.
- Contributed by p-zak.
NetFlow (netflow)
- Collects metrics from NetFlow v5, NetFlow v9, and IPFIX collectors.
- Contributed by srebhan.
OPC UA Event Subscriptions (opcua_listener)
- Gather metrics from subscriptions to OPC UA devices.
- Contributed by LarsStegman.

Downloads

InfluxDB University

Learn more about collecting data with Telegraf by taking the free InfluxDB University Data Collection with Telegraf course.

Call for testing