InfluxData Blog - Samantha Wang

Telegraf 1.24.3 maintenance release addresses processor regressions and Modbus fixes

Samantha Wang (InfluxData) — Mon, 07 Nov 2022 10:00:00 +0000

This new maintenance release for Telegraf is available now.

Telegraf 1.24.3 — Release Notes

Input plugin updates

Directory Monitor Input
- Plugin now allows moving files across filesystem directories.
Kafka Consumer Input
- Kafka input plugin now uses Sarama library’s new consumer group rebalance strategy setting.
Modbus Input
- We added the associated client ID to a failing connection.
- We improved the duplicate field checks in the plugin.
- Plugin now handles field-measurement definitions correctly on a duplicate field check.
OPC-UA Input
- We added the associated client ID to a failing connection.
Syslog Input
- Plugin now prints a message when no error or message is returned.
Zookeeper Input
- We fixed an issue where floats were being parsed as strings.

Processor plugin updates

Fixed a regression where processors were potentially missing metrics when used in combination with aggregators due to the way they were initialized.
Parser Processor
- Processor now keeps the name of the original metric if the parser doesn’t return one.

Parser updates

JSON v2 Parser
- Telegraf now removed BOM before parsing with the JSON v2 parser.

Serializer updates

Splunkmetric Serializer
- The serializer now provides the option to remove the event metric tag.

Head to our Downloads page to get the latest Telegraf release. If you find issues or have questions, please join our InfluxDB Community Slack, post them in our InfluxDB GitHub Repo or our Community site, and we will take a look.

Please fill out this form if you are interested in being included in testing bug fixes and features for plugins and systems you are using. Telegraf maintainers will tag you on pull requests to test artifacts. This will help us better address bugs that may arise after a release.

How to Reduce Telegraf Binary Size with a Customized Telegraf Agent

Jay Clifford, Samantha Wang (InfluxData) — Fri, 04 Nov 2022 07:00:00 +0000

Is Telegraf too big for your device? Too many plugins in one binary getting you down?

Let me introduce you to the Telegraf custom builder – the new tool for reducing Telegraf’s overall memory and disk footprint. In this blog, we will discuss the “what, why, when” and also how to use the new custom builder.

What, why, and when?

What?

The Telegraf custom builder is a tool written in Go and found within the tools directory of the Telegraf repo. The custom builder allows you to select which plugins to build the Telegraf binary with. It takes your Telegraf configuration file as input, scans the configuration files for the plugin files being defined, and builds a new binary including only these plugins (more on this in the how-to section).

Why?

The Telegraf binary has grown substantially over the last few years. The diagram below shows the growth of the Telegraf binary on a Linux amd64 build from versions 1.14.0 to 1.24.0. With this growth, many users have requested ways to build a slimmer Telegraf binary with just the plugins they want to download on their lightweight IoT devices.

Although stripping the Telegraf binary is not a new concept, it’s never been an easy one for the Telegraf community to perform. It required knowledge of the source code and specific build configuration files. The Telegraf custom builder is here to streamline this process for all users. This custom builder also provides scope for Telegraf to be used more performantly on resource-constrained devices, even some embedded devices/controllers which support running the Telegraf binary. The Telegraf custom builder also helps with the future scalability of Telegraf. Naturally, the more plugins the community contributes, the more Telegraf bloats in size (the nature of adding more features over time). This tool provides the scope to build a binary relevant to your use case.

When?

The Telegraf custom builder is a powerful tool to use within the right situations and stages of your solution design. For instance, it makes sense to use the custom builder when you have made a firm decision on which plugins you will be using. The custom builder also makes sense if you are comfortable building the Telegraf binary from its source.

Before you start:

Create your Telegraf configuration with the plugins you want to use. This could be stored locally or remotely. In this blog, we’ll be using the Telegraf configuration.

How to use it

In this section, let’s use the Telegraf custom builder to build Telegraf for this demo: MQTT Emergency Generator Simulator.

Before you start Make sure your Go version is up to date. At the time of publishing, this tool requires Go version 1.18.0+. Create your Telegraf configuration with the plugins you want to use. This could be stored locally or remotely. In this blog we’ll be using the Telegraf configuration from the MQTT Emergency Generator Simulator.

Clone the Telegraf repo

git clone https://github.com/influxdata/telegraf.git

Build custom builder
We now need to build the custom builder. To do this open a terminal within the top-level directory of the Telegraf repository and enter the command:
```
make build_tools
```
Point the custom builder at your Telegraf configuration(s) that include the plugins you want to build
Depending on how you have built/stored your Telegraf config(s), you have a few options here:
- Single Telegraf config
```
./tools/custom_builder/custom_builder --config /etc/telegraf.conf
```
- Single Telegraf config + Telegraf config directory
```
/tools/custom_builder/custom_builder --config     /etc/telegraf.conf --config-dir /etc/telegraf/telegraf.d
```
- Remote Telegraf config
```
./tools/custom_builder/custom_builder --config http://myserver/telegraf.conf
```
For our Emergency Generator demo, we can find the Telegraf configuration here. So let's try out the remote feature with:
```
./tools/custom_builder/custom_builder --config https://raw.githubusercontent.com/InfluxCommunity/InfluxDB-Roadshow-Training/master/telegraf/telegraf.conf
```
This will run both phases of the custom builder (configuration and building), resulting in the creation of your customized Telegraf binary.
View customized Telegraf binary
You should now be able to find your new smaller Telegraf binary within the top level of your Telegraf repository.
The table below gives a comparison of the overall binary size on MacOS:

MQTT Emergency Generator custom build Full binary

MQTT input plugin, InfluxDB_V2 output plugin, JSON_V2 parser All Plugins

17.6 MB 207.9 MB
Add or remove plugins from your customized Telegraf If you wish to add or remove plugins from your customized Telegraf build, you can just edit your configuration and rerun the `./tools/custom_builder/custom_builder --config path/to/config.conf` command.

MQTT Emergency Generator custom build	Full binary
MQTT input plugin, InfluxDB_V2 output plugin, JSON_V2 parser	All Plugins
17.6 MB	207.9 MB

Conclusion

With Telegraf 2.0 in development, we are making a conscious effort to provide a Telegraf binary that delivers great functionality on all devices, including those with constrained resources. The Telegraf custom builder provides you with the flexibility to save vital memory and storage space by only bringing the plugins you need to the deployment phase. We are excited to hear your feedback on the new tool. Head over to the InfluxData Slack and Community forums. Let’s continue the discussion there with the rest of the team!

Telegraf 1.24.1 maintenance release fixes memory leak for plugins

Samantha Wang (InfluxData) — Mon, 19 Sep 2022 10:00:00 +0000

This new maintenance release for Telegraf is available now.

Telegraf 1.24.1 — Release Notes

Configuration

Telegraf will now let you know if the configuration file passed is not a text file.

Builds

We fixed an issue where window tests were timing out by enabling global confirmation for installing the mingw package

Input plugin updates

Ceph Input

The Ceph plugin is now modernized to work with metrics in version 15.0.

Modbus Input

Plugin no longer fails if a single server reports errors.

NTPQ Input

Plugin now handles duration fields that only contain -.

Processor plugin updates

Parser Processor

We added the ability to parse tags in the processor parser.

Parser updates

We fixed a memory leak for plugins using ParserFunc.
We removed some special handling of parsers and unwrapped the remaining parsers.
CSV Parser

We removed direct checks for the parser type.

XPath Parser

We added an array index when expanding names.

Dependency updates

Bump cloud.google.com/go/pubsub from 1.24.0 to 1.25.1</code>
Bump github.com/urfave/cli/v2 from 2.14.1 to 2.16.3
Bump github.com/aws/aws-sdk-go-v2/service/ec2 from 1.52.1 to 1.54.4
Bump github.com/wavefronthq/wavefront-sdk-go from 0.10.2 to 0.10.4
Bump cloud.google.com/go/bigquery` from 1.33.0 to 1.40.0

Telegraf 1.24.0 release includes new AWS CloudWatch Metric Streams, Linux CPU, and PostgreSQL plugins

Samantha Wang (InfluxData) — Thu, 15 Sep 2022 07:00:00 +0000

A new feature-bearing release for Telegraf is now available:

Telegraf 1.24.0 — release notes

The binaries for the latest Telegraf release can be found on our Downloads page.

Many thanks to the 40+ open source community members who contributed to this effort!

Key features

Breaking Change: Set default minimum TLS version to v1.2 for security reasons on both server and client connections.

This is a change from the previous defaults (TLS v1.0) on the server configuration and might break clients relying on older TLS versions.
Older versions can be manually reverted on a per-plugin basis using the tls_min_version option in the plugins required.

Added new json_transformation option to the JSON serializer to transform outputted JSON

This new option can be used to transform the JSON output using the JSONata language to accommodate for requirements on the receiver side.
Setting can also filter and process JSON data points.
JSONata examples and transformations

New inputs

AWS CloudWatch Metric Streams Input Plugin (cloudwatch_metric_streams)

This plugin is a service input plugin that listens for metrics sent via HTTP and performs the required processing for metric streams from AWS. Contributed by Cillian McCabe.

Linux CPU Input Plugin (linux_cpu)

This plugin gathers CPU metrics exposed on Linux-based systems. Contributed by Fabian Mastenbroek.

Supervisor Input Plugin (supervisor)

This plugin gathers information about processes that are running under supervisor using XML-RPC API. Contributed by @niasar

UPSD Input Plugin (upsd)

This plugin reads data of one or more Uninterruptible Power Supplies from an upsd daemon using its NUT network protocol. Contributed by Anton Malinskiy.

New serializer

CSV Serializer

The csv output data format converts metrics into CSV lines. Contributed by @influxdata.

New outputs

PostgreSQL Output Plugin (postgresql)

This plugin writes metrics to PostgreSQL (or compatible database. Contributed by Patrick Hemmer.

RedisTimeSeries Plugin (redistimeseries)

This plugin writes metrics to the RedisTimeSeries server. Contributed by Guy Korland.

Stomp (ActiveMQ) Plugin (stomp)

This plugin writes to an ActiveMQ Broker for STOMP. Contributed by @amus-sal.

New external plugins

NSDP Input

The NSDP external plugin gathers switch network statistics via Netgear Switch Discovery Protocol. Contributed by @hdecarne.

Downloads

InfluxDB University

Learn more about collecting data with Telegraf by taking the free InfluxDB University Data Collection with Telegraf course.

Call for testing

Announcing Native Collectors: Bringing Native Data Collection to InfluxDB Cloud

Samantha Wang (InfluxData) — Tue, 23 Aug 2022 07:00:00 +0000

Streaming time series data from brokers and services that are on-premises or in the cloud to a cloud-based database is a resource-intensive process requiring third-party software and heavy customizations. Today we’re announcing InfluxDB Native Collectors to make it easy for developers to collect, process, and analyze data by subscribing directly to supported message brokers. With this new suite of capabilities, we are expediting cloud-to-cloud data transfers so developers can focus on building and scaling applications with their time series data.

Today we’re excited to announce the immediate availability of InfluxDB Native Collectors, a new suite of capabilities that enable developers building with InfluxDB Cloud to subscribe to, process, transform, and store real-time data from messaging brokers. InfluxDB Native Collectors give developers the fastest way to get data from third-party brokers into InfluxDB Cloud without the need for additional software or new code. Beginning with our Native MQTT collector, this new capability allows InfluxDB users to subscribe to MQTT topics directly from the InfluxDB Cloud interface. The InfluxDB Cloud service will then communicate directly with their MQTT broker to immediately start to receive any message posted to those topics.

InfluxDB Cloud already offers many data ingestion methods to support different data sources, protocols, and formats:

InfluxDB Client Libraries: For many popular development languages allow you to easily make API calls to write data to InfluxDB Cloud.
Telegraf Collection Agent: Offers 300+ plugins to collect metrics from cloud services, apps, IoT sensors, and more.
Third-party integrations: Technology ecosystem integrations for technologies such as FluentD, Vector, and proprietary IT and OT protocols.
Scrapers: Scrape Prometheus-formatted metrics from an HTTP-accessible endpoint and store them in InfluxDB.
Or developers can build their own collection agent with the InfluxDB API, which provides a programmatic interface for interactions with InfluxDB.

These existing options provide excellent ways for developers to get their data into InfluxDB. However, they typically require writing code or installing software. For developers using message brokers, especially those installed in the cloud or serviced by a third party, writing software or installing additional software may not always be desirable, or even feasible. The introduction of InfluxDB Native Collectors adds an option to give developers the fastest and simplest way to get their data into a time series database.

Faster Time to Awesome for builders

Three configuration steps are all it takes to get your time series data flowing from your message broker into InfluxDB Cloud:

Broker Details: Specify the IP Address, Port, and Auth parameters for your Message Broker.
Topic Name: Provide the topic name for the topic you want to subscribe to.
Parsing Rules: Set up some simple parsing rules to map elements in your messages to measurements, timestamps, fields, and tags in InfluxDB.

In most cases, the configuration only takes a few minutes. Once completed, any messages pushed to your MQTT broker will flow into InfluxDB as time series data. You can then immediately start to query and visualize that data and can even set up monitoring alerts for specific metrics and events. And using the Flux scripting language, you can easily set up downsample tasks and other data transformations.

Optimize workflows for broker-based data sources

InfluxDB Native Collectors use a brand-new InfluxDB Cloud service purpose-built for the unique nature of broker subscriptions. We built a number of conveniences into the collector’s User Interface that allow you to save time and effort during configuration. For example:

Subscription Wildcards: Use the wildcard characters + and # in your Topic Configuration to dynamically subscribe to one or many specific topics on a particular broker. This adheres to the MQTT Specification for Topic Names and Topic Filters.
Dynamic Data Routing and Orchestration: Using Subscription Wildcards and Write Destination, you can dynamically route events to different InfluxDB Buckets depending on the broker source. For example, write all data from “Energy” related topics to an “Energy” bucket, and all data from “Weather” related topics to a “Weather Bucket”.
JSON and String Parsing: As the message content of MQTT messages is truly unstructured, we introduced methods for using JSON Path expressions and Regular Expressions to dynamically parse JSON and string payloads, respectively. Mine your MQTT data for timestamp, measurement, field, and tag sets dynamically, no matter the text format.

Reimagining device-to-cloud data transfers

InfluxDB Native Collectors gives you a faster way to ingest time series data in the cloud in one step and without customization, orchestration, or additional hosting services.

Data stays in the cloud: If your message broker is already cloud-based, you can seamlessly send your data from your broker to InfluxDB Cloud. With Native Collectors in InfluxDB Cloud, you can have a completely end-to-end serverless and managed solution.
Simple low-code setup: InfluxDB Native Collectors eliminate coding and troubleshooting complex scripts and allow you to focus your engineering resources on other areas of your application.
Meeting developers where they are: InfluxDB Native Collectors support the most common cloud subscriptions, brokers, and services. Support for MQTT is immediately available, with additional support for Kafka, AMQP, and more coming soon.

Get started with InfluxDB Native Collectors today

If you are unsure whether a time series database or InfluxDB Cloud is for you, Native Collectors let you realize benefits right away. You can evaluate it without needing to spend much time getting things set up or kick off a complex project to evaluate it. This results in both a low cost of adoption and a low cost of evaluation.

For more information on how to get started, sign up for InfluxDB Cloud if you haven’t already and make sure you register for our upcoming webinar on August 30.

Telegraf 1.23.4 released with improvements in Kubernetes and AWS CloudWatch plugins

Samantha Wang (InfluxData) — Tue, 16 Aug 2022 07:00:00 +0000

This new maintenance release for Telegraf is available now.

Telegraf 1.23.4 — Release Notes

Input plugin updates

AWS CloudWatch Input

We added the ability to customize batch size when querying.

Kube Inventory Input

The plugin will now pass a token file location instead of having Telegraf read it directly to enable token auto-refresh.

Kubernetes Input

The Kubernetes plugin will now refresh the token from a file to be re-read at every interval. This enables the Telegraf to get the new token if it’s updated or refreshed.

MongoDB Input

We updated a version check in the plugin to check that the MongoDB version is 5.0+.

OPC-UA Input

The plugin now returns an error for mismatched types.

SQL Server Input

We set a lower deadlock priority on the remaining SQL server queries.

Stackdriver Input

We updated the plugin handling when no buckets are available.

Output plugin updates

OpenTelemetry Output

We added a Coralogix dialect to OpenTelemetry output plugin for the ability to send metrics to Coralogix.

AWS Timestream Output

The plugin now works with endpoint_url provided by the customer.

MQTT topic parsing, SNMP trap plugin, and SQL Server input improvements with Telegraf 1.23.3

Samantha Wang (InfluxData) — Mon, 25 Jul 2022 02:00:00 +0000

This new maintenance release for Telegraf is available now.

Telegraf 1.23.3 — Release Notes

Common

Cookie common plugin now uses reader over readcloser and regenerates cookie-jar during reauthentication.

Input plugin updates

OpenStack Input

The OpenStack plugin now uses the v3 volume library to fix an issue with storage volume API fails.

MQTT Consumer Input

We fixed an issue with topic parsing that would produce an error if the topic had a prefix of /.

SNMP Trap Input

We resolved a nil map panic that occurred when running the plugin with the netSNMP translator.

SQL Server Input

We set a lower deadlock priority on queries in the plugin.

Parser plugin updates

Prometheus

The Histogram infinity bucket is now always present in the Prometheus parser.

Dependencies

Updated github.com/antchfx/jsonquery from 1.1.5 to 1.2.0

Telegraf 1.23.2 release resolves faulty deprecation warnings

Samantha Wang (InfluxData) — Mon, 11 Jul 2022 11:00:00 +0000

This new maintenance release for Telegraf is available now.

Telegraf 1.23.2 — Release Notes

Agent

We fixed an issue in Telegraf 1.23.1 where Telegraf outputed deprecation warnings for non-deprecated plugins and options by restoring old behavior and adding tests.

Input plugin updates

HTTP Input

The HTTP Input Plugin now allows for both a 200 and 201 response code when generating authentication using the cookie authentication plugin.

SQL Server Input

Plugin now uses bigint for backupsize.

GNMI Input

We reworked how to handle tag-only subscriptions to support more complex cases/lookups and to improve concurrency.

SNMP Input

Plugin now sets gosnmp’s UseUnconnectedUDPSocket to true when using UDP.

Dependencies

Updated github.com/docker/docker from 20.10.14 to 20.10.17

Updates to MongoDB, RabbitMQ, and Jolokia plugins with new Telegraf 1.23.1 release

Samantha Wang (InfluxData) — Tue, 05 Jul 2022 05:00:00 +0000

This new maintenance release for Telegraf is available now.

Telegraf 1.23.1 — Release Notes

Configuration

Restored sample configurations that were broken during the initial migration to template-based generation in 1.23.0.

Input plugin updates

RabbitMQ Input

The plugin no longer requires listeners to be present in the RabbitMQ /api/overview JSON.

MongoDB Input

The plugin now filters out views in MongoDB lookup.

Jolokia2 Input

We fixed an issue where Telegraf would panic and exit when it received a jolokia response with null values.

Parser updates

XPath Parser

We brought back the old XPath section names (xml, xpath_json, xpath_msgpack and xpath_protobuf) to fix a regression.

Dependencies

Updated cloud.google.com/go/monitoring from 1.2.0 to 1.5.0.
Updated github.com/aws/aws-sdk-go-v2/credentials from 1.12.2 to 1.12.5.
Updated google.golang.org/grpc from 1.46.2 to 1.47.0.
Updated k8s.io/client-go from 0.23.3 to 0.24.1.
Updated github.com/go-logfmt/logfmt from 0.5.0 to 0.5.1.
Updated github.com/aws/aws-sdk-go-v2/service/dynamodb from 1.15.3 to 1.15.7.
Updated go.mongodb.org/mongo-driver` from 1.9.0 to 1.9.1.
Updated github.com/gophercloud/gophercloud from 0.24.0 to 0.25.0.
Updated google.golang.org/api from 0.74.0 to 0.84.0.
Updated github.com/fatih/color from 1.10.0 to 1.13.0.
Updated github.com/aws/aws-sdk-go-v2/service/timestreamwrite from 1.3.2 to 1.13.6.
Updated github.com/shopify/sarama from 1.32.0 to 1.34.1.
Updated github.com/dynatrace-oss/dynatrace-metric-utils-go from 0.3.0 to 0.5.0.
Updated github.com/nats-io/nats.go from 1.15.0 to 1.16.0.
Updated cloud.google.com/go/pubsub from 1.18.0 to 1.22.2.
Updated go.opentelemetry.io/collector/pdata` from 0.52.0 to 0.54.0.
Updated github.com/jackc/pgx/v4 from 4.15.0 to 4.16.1.
Updated cloud.google.com/go/bigquery from 1.8.0 to 1.33.0.
Updated github.com/Azure/azure-kusto-go from 0.6.0 to 0.7.0.
Updated cloud.google.com/go/pubsub from 1.22.2 to 1.23.0.
Updated github.com/aws/aws-sdk-go-v2/service/kinesis` from 1.13.0 to 1.15.7.
Updated github.com/aws/aws-sdk-go-v2/service/ec2 from 1.1.0 to 1.46.0.
Updated github.com/golang-jwt/jwt/v4 from 4.4.1 to 4.4.2.
Updated github.com/vmware/govmomi from 0.27.3 to 0.28.0.
Updated github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs from 1.15.4 to 1.15.8.
Updated github.com/influxdata/influxdb-observability/otel2influx from 0.2.21 to 0.2.22.
Updated k8s.io/api from 0.24.1 to 0.24.2.
Updated github.com/prometheus/client_golang from 1.12.1 to 1.12.2.