InfluxData Blog - Sam Dillard

InfluxDB OSS Gains Improved User Model, Lots of Flux Enhancements, and Big CLI Additions

Sam Dillard (InfluxData) — Thu, 18 Aug 2022 07:00:00 +0000

Release Announcement: InfluxDB OSS 2.4

InfluxDB OSS 2.4 – Release Notes

This version of InfluxDB comes with Flux additions and a couple of enhancements to the API that support features of the CLI. The CLI received the most work this time around. See the CLI release notes for more information on that.

Enhancements:

Added back the v1 InfluxQL shell/REPL. See this blog post for more information.
Supported v1 dbrp names as bucket IDs in replications API. This makes the creation of replications easier and adds support for replicating to an Enterprise cluster’s v2 /write endpoint.
Changed to operator model such that admin user has instance-level permissions without operator token.
Upgraded Flux from 0.171.0 to 0.179.0

Flux advancement highlights:

Added multiple new join functions to the join package such as join.full()
Promoted experimental.to() to influxdata.influxdb.wideTo()
Added built-in function time to the date package to convert any timeable into datetime
Promoted the following functions from experimental.array into the array package: array.concat(), array.filter(), array.map().
Promoted the following functions from experimental.http.requests into the http.requests: http.requests.do(), http.requests.get(), http.requests.peek(), http.requests.post()
Promoted experimental.bitwise into the bitwise package
Supported conditional expressions, constants, and literals in vectorized map()

Influx CLI Introduces Invokable Scripts, Auth Enhancements, and RE-INTRODUCES the InfluxQL Shell!

Sam Dillard (InfluxData) — Thu, 18 Aug 2022 07:00:00 +0000

Release Announcement: Influx CLI 2.4

Influx CLI 2.4 – Release notes

The InfluxDB CLI is a standalone open source binary that works with both OSS and InfluxDB Cloud. On rare occasions some updates may apply to one or the other. In this release, each gets their own respective applicable enhancement.

Documentation for the CLI is contained in both OSS and InfluxDB Cloud documentation.

Enhancements

Allowed setting membership type with influx org members add where only member was available before
Added back the InfluxQL Shell (REPL) from v1 with an enhanced experience
[Cloud only] Allowed for calling Invokable Scripts
[OSS only] Added username and password authentication as alternative to token
Supported v1 dbrp names as bucket IDs in replications API. This makes the creation of replications easier and adds support for replicating to an Enterprise cluster’s v2 /write endpoint.

Edge Data Replication is GA and Flux performance improvements are now available with InfluxDB OSS 2.3

Sam Dillard (InfluxData) — Wed, 22 Jun 2022 07:00:00 +0000

Check out the GA of Edge Data Replication and Flux improvements here.

We’ve been busy bringing the latest improvements to this release. Some of these have been available within InfluxDB Cloud, and now we are pleased to offer them within the open source offering for those of you deploying the OSS version at the edge or developing against it on your local machine.

Note that releases may be a little lighter as we speed up the release cadence. Development rate should remain or improve, but releases will get tighter.

For more detail on the release, view the release notes.

Maintenance

Upgraded Flux from 0.161.0 to 0.171.0
Upgraded Go to 1.18.3

Flux advancement

Flux was upgraded from 0.161.0 to 0.171.0. This includes the following features and enhancements:

Performance:

Vectorized arithmetic operators in map()
Vectorized logical operations in map()
Enabled movingAverage() and cumulativeSum() optimizations by default

Other:

Added preview() to experimental package for limiting return rows and tables (as opposed to just rows with limit())
Added date.scale() to let users dynamically scale durations in dates
Added OpenTracing spans to Flux transformations. This allows developers to more precisely monitor Flux scripts.
Added trace option to Flux CLI
Moved addDuration() and subDuration() out of experimental to date package.
Added location support to date.truncate()
Added _status Tag to PagerDuty record
Refactored operator profile to be in the query statistics

Bug fixes

If all of that feature advancement isn’t enough, there are a handful of bug fixes which also have been addressed. Check out the release notes for all the details or if you are looking for a specific fix.

If you find issues or have questions, please join our InfluxDB Community Slack, post them in our InfluxDB GitHub Repo or our Community Site and we will take a look.

Announcing InfluxDB Edge Data Replication: Combining the Power of the Cloud with the Precision of the Edge

Sam Dillard (InfluxData) — Wed, 15 Jun 2022 06:00:00 +0000

There are technical and business reasons to have a time series data presence both at the edge and in the cloud – InfluxDB has always played a key role in both contexts. Today, we’re announcing Edge Data Replication, a new feature that combines these two deployment strategies. With this announcement, InfluxData begins a greater initiative to accommodate both edge and cloud data workloads in one unified solution.

The edge is becoming more critical than ever. Businesses requiring new solutions at the edge, combined with a rapid increase in data from these devices, creates enormous challenges for developers to manage and use that data effectively. Data has gravity and that gravity gets stronger as applications become more distributed and data volumes at the edge grow exponentially.

Thousands of businesses use InfluxDB Open Source (OSS) at the edge for local collection, storage, and processing of time series data. This gives InfluxDB the role of managing the edge data layer, allowing API access, locally served dashboards, integration with automation and controls, and delivery to alerting systems. In addition, thousands more organizations and developers have adopted InfluxDB Cloud to build cloud-native time series applications.

The needs for edge and cloud data, respectively, are strong – and even strict at times. There are many considerations when choosing the edge or cloud for storage and compute, but ultimately, developers want to ensure these two data layers work together in a simple and efficient way that delivers centralized business insights in near real-time. Up until now, architecting an edge to cloud integration required significant upfront and ongoing investment that is expensive, time-consuming and error-prone.

Introducing InfluxDB Edge Data Replication

Today we’re excited to announce the immediate availability of Edge Data Replication, a solution that unifies time series data processing between edge and cloud environments so developers can provide consolidated intelligence across widely distributed environments. Edge Data Replication (EDR) makes it possible to safely replicate data from InfluxDB OSS buckets to InfluxDB Cloud buckets in real time, meaning users can now collect, store and analyze high-precision time series data from the edge, and view that data in the cloud.

This new feature builds upon two key properties of InfluxDB OSS. First, InfluxDB OSS can run very efficiently on resource-constrained systems. Second, InfluxDB OSS runs Flux, a functional data scripting language and task engine that can analyze and transform time series data in any way developers choose. At InfluxData, we’re focused on simplifying the developer experience around times series data, so developing a way to connect these OSS properties to enable a full-featured edge-to-cloud data pipeline was a no-brainer.

Version 2.2 of InfluxDB introduces a new feature to automate the replication of time series data to other InfluxDB instances. Durable, disk-backed queues add buffering to withstand planned and unplanned disruptions in network connectivity. In addition, this feature configures replication at the bucket level so developers and operators can precisely define which sets of data to replicate and where to send them.

This replication happens on-write. When time series data arrives at the edge and matches a replication rule, OSS immediately mirrors the data to the remote bucket defined in the configuration. When InfluxDB OSS cannot send the data to the remote instance, all events buffer in the local durable queue until connectivity is restored. The size and retention of the durable queue are configurable.

Tangibly, the feature consists of two new API endpoints (/remotes and /replications) and two new CLI commands (remote and replication). Each replicated bucket also gets a disk-backed queue for buffering data safely in case of disruptions.

Giving distributed applications an edge

Edge Data Replication gives developers a fast and durable way to stream edge data to the cloud, with minimal configuration. This capability gives users the data they need in the cloud to build a global view of all edge activity. Benefits include:

Integrate edge and cloud workloads: Enables businesses to quickly engage in both edge and cloud computing for their time series workloads, without requiring engineering to introduce or manage any third-party software.
Reduce network costs: Edge Data Replication helps users intelligently reduce the volume of data they send to the cloud by leveraging built-in capabilities of the Flux tasks engine. Operators of OSS nodes at the periphery can pre-compute aggregations and filtering so they can limit the data sent over the internet to the most critical data. They may also use this feature to filter or compute interesting events and forward only those events to the cloud for monitoring.
Transform edge data: Dynamically add context to edge data that is necessary for processing in the cloud, which may require extra dimensions to satisfy queries or feature engineering.
Enforce edge-cloud consistency: Withstand planned and unplanned disruptions in network connectivity by queueing data at the edge until it can be safely transferred to the cloud.

Reimagining the TSDB architecture

Data architects need a robust and repeatable way to design systems that work within the constraints of both the edge and the cloud; systems that understand each of their respective strengths and weaknesses. With Edge Data Replication, we are unlocking a way to accommodate both edge and cloud data workloads with flexibility in the collection, analysis, and storage of time series data – no matter its point of origin or intended destination.

Meeting developers where they are

Today’s announcement is an early step in InfluxData’s broader vision to support tomorrow’s technologies, applications, and developer challenges with the InfluxDB platform. It is always our goal to meet developers where they are. With this launch, we’re helping them deliver faster Time to Awesome so they can uncover critical edge insights and move on to other aspects of building their applications.

Get started with Edge Data Replication today

To start building your edge-cloud pipeline, sign up for InfluxDB Cloud if you haven’t already. Otherwise, check out the documentation. If you’re interested in learning more and enabling edge-cloud duality of time series data, sign up for our upcoming EDR webinar on June 28.

InfluxDB Enterprise 1.9.7 released with enhancements to Flux, Passive Nodes, and Cluster Stability

Sam Dillard (InfluxData) — Thu, 26 May 2022 04:30:00 +0000

InfluxDB Enterprise 1.9.7 — Release notes

Key highlights for this InfluxDB Enterprise release – available here – include:

Enhancements

Expose passive nodes to influxd-ctl command. Passive nodes allow for increased horizontal scaling in the write path.
Throttle replication of incoming writes and hinted hand-off when errors are encountered. This improves cluster stability.

Flux Advancement

Add requests package for a more full-featured HTTP requests experience
Add isType() function. This convenience function will allow Flux to check types of columns which will help dynamically apply mathematical operations on a set of Fields that may contain non-numeric Fields.
Added display() function. This is like a print statement and will help in debugging Flux scripts as users compose them.
Various enhancements to increase(), sort(), derivative(), union(), timeshift().
Added vectorization to applicable functions like map().
Continued performance enhancements (i.e., sort() |> limit()).
Enhanced mqtt.publish() to reuse TCP connections when being called in a loop. This allows for functions like map() run mqtt.publish() on each row of a table stream without re-establishing the handshake for each record.

Maintenance updates

Upgrade to Go 1.17.9
Upgrade to Flux v0.161.0

Bug fixes can be found in release notes.

There is no corresponding InfluxDB OSS 1.9 release. While we continue to make improvements to the InfluxDB 1.x code base – and these are being included within InfluxDB Enterprise – we are guiding our community users to InfluxDB 2.x at this time. Of course, you can always build InfluxDB 1.x from source code, if you need to. InfluxDB 2.x does include 1.x compatible interfaces to allow for reads and writes using the 1.x API. The latest open source release can be found on our downloads page.

For our InfluxDB Enterprise customers, log in to the InfluxDB Enterprise portal and download the binaries from there. Need more help? Contact support.

If you find issues or have questions, please join our InfluxDB Community Slack, post them in our InfluxDB GitHub Repo or our Community Site and we will take a look. Thank you.

Release Announcement: InfluxDB OSS 2.2.0

Sam Dillard (InfluxData) — Thu, 07 Apr 2022 10:43:34 -0700

OSS data remote replication, security enhancements, and improvements in Flux available now.

We’ve been busy bringing the latest improvements to this release. Some of these have been available within InfluxDB Cloud, and now we are pleased to offer them within the open source offering for those of you developing against the OSS edition on your laptop or deploying this at the edge.

InfluxDB 2.2.0 – release notes

Feature advancement

This release includes the following new features:

Technical Preview: OSS to Cloud data replication
Flux advancement

OSS to Cloud data replication

Replication between remote buckets was introduced in this version. A bucket in OSS can be configured to replicate to a bucket in a remote service. This is a critical capability in topologies combining edge and cloud deployments of InfluxDB.

This feature is noted as a “technical preview” because it is not yet intended for production use. It is not listed as “experimental” because it is planned as an officially supported feature.

Security

Private IP Validation for Flux and Template Fetching

Template fetching and some Flux functions make HTTP requests over the network. This feature’s optional configuration can limit InfluxDB’s access to private IPs and therefore the possibility of Server Side Request Forgery (SSRF) attacks. By default, private IPs are still reachable, so there is no breaking change. They can be made inaccessible by invoking the new --hardening-enabled flag.

Flux advancement

Flux was upgraded from 0.139.0 to 0.161.0. This includes the following features and enhancements:

Added new experimental http/requests package which increases flexibility of Flux to make HTTP calls and use their response data.
Add isType() function. This convenience function will allow Flux to check types of columns which will help dynamically apply mathematical operations on a set of Fields that may contain non-numeric Fields.
Added display() function. This is like a print statement and will help in debugging Flux scripts as users compose them.
Various enhancements to increase(), sort(), derivative(), union(), timeshift().
Added vectorization to applicable functions like map()
Continued performance enhancements (i.e., sort() |> limit())
Enhanced mqtt.publish() to reuse TCP connections when being called in a loop. This allows for functions like map() run mqtt.publish() on each row of a table stream without re-establishing the handshake for each record.

Bug fixes

If you find issues or have questions, please join our InfluxDB Community Slack, post them in our InfluxDB GitHub Repo or our Community Site and we will take a look.

Usability improvements in the new release of Chronograf

Sam Dillard (InfluxData) — Tue, 29 Mar 2022 09:05:54 -0700

Chronograf 1.9.4 – Release Notes

This latest release of Chronograf includes the following new improvements and fixes:

Flux

Chronograf 1.9.4 now includes a Flux Builder to support users who are new to Flux and want a more guided experience building up Flux queries. For those who are new to Flux and starting to use Flux with InfluxDB Enterprise, this new experience provides an easy on ramp for building your first Flux query. The Flux Builder now in Chronograf mirrors what is offered via InfluxDB Cloud and InfluxDB OSS.

Kapacitor integration

Within Chronograf, you have access to view all of the scripts and tasks that Kapacitor is executing. We've improved the browsing experience to improve the performance of loading and listing TICKscripts and Flux Tasks. This includes the ability to filter by name.

Grab the latest bits from our downloads page under “Are you interested in InfluxDB 1.x Open Source?”

If you find issues or have questions, please join our InfluxDB Community Slack, post them in our Chronograf Github Repo or our Community Site and we will take a look.

Security improvements now available in Kapacitor 1.6.4

Sam Dillard (InfluxData) — Wed, 23 Mar 2022 15:47:56 -0700

Kapacitor 1.6.4 – release notes

Includes the following fixes and improvements:

Security
- For those integrating Kapacitor with Kafka, we've added support for SASL-based security authentication. This is similar to the security provided by Telegraf for integrating with Kafka.
- We've deprecated additional ciphers that were identified as "weak" as part of the sweet32 attack. These include: DES and RC4. We have also deprecated support for TLS 1.0 and 1.1.

Grab the latest bits from our downloads page under “Are you interested in InfluxDB 1.x Open Source?”

If you find issues or have questions, please join our InfluxDB Community Slack, post them in our Kapacitor Github Repo or our Community Site, and we will take a look.

MQTT Topic and Payload Parsing with Telegraf

Sam Dillard (InfluxData) — Mon, 20 Dec 2021 04:00:38 -0700

Buckle up, this one isn’t short…but I’m hoping it will be thoroughly informative! This post is about Telegraf as a consumer of MQTT messages in the context of writing them to InfluxDB. If you are interested in and unfamiliar with Telegraf, you can view docs here. Unsure if Telegraf aligns with your needs? I make a case for it in the Optimizing Writes section of this blog post.

It may also help to have an understanding of Line Protocol, InfluxDB’s default accepted format.

In this post, I want to impart the efficacy of Telegraf in collecting data over MQTT by a) drilling into a feature of the latest release and b) discussing parsing methodology. As many readers know, MQTT message payloads can be…quite literally whatever you want them to be, so it’s paramount that a subscriber to many topics be able to handle whatever you throw at it.

Telegraf can subscribe to any number of topics while giving its users control over how it shapes incoming data. Controlling data shape allows users to write out data that is not only compatible for the destination data store, but optimized as well.

Data shape control in this context comes in two forms, and each will get its own section in this post:

Topic parsing
Payload parsing

Housekeeping:

All Telegraf behavior below assumes a standard [Agent] level configuration with omit_hostname = true
Timestamps are omitted in examples

Topic parsing

In Telegraf’s latest release (v1.21), the MQTT Consumer Input Plugin added the ability to parse information out of topics dynamically and append that information to the payload. This means a Telegraf operator can use data from both the MQTT message payload and the incoming topic.

Below is a diagram to exemplify four different MQTT message packets being laid out the same way in the InfluxDB data model.

This is possible with the insertion of Telegraf, and the rest of this post explains how.

I’ll first explain the problem this solves with examples. Suppose I have a basic sensor emitting a single temperature value to a topic that includes its unique ID; a topic like: sensors/00191. The MQTT messages will be sent to this topic with a single raw value — for this example, let’s say a sample of float 10.0.

No third-party subscriber to this information in sensors/00191 will know what to do with this temperature value:

It is not valid LP on its own, so Telegraf will reject it.
If you help Telegraf with a data_format="value" and data_type="float", it would generate LP like:

mqtt_consumer,topic=sensors/00191 value=10.0 <timestamp>

(This is default behavior of input plugins that are given no measurement name).

Notice that Telegraf adds the whole topic as a LP tag by default. This is Telegraf being helpful so you retain any potentially necessary context. Including the whole topic as a single string, however, would necessitate some regex later to be made useful. The reason it’s there is that without it:

This value field has no meaning to anyone/anything consuming the data.
Each sensor emitting data like this would overwrite the last sensor's data if written at the same time (tags are needed to differentiate fields like value).

The information needed, in this case, is in the topic. This can happen if a vendor programmed their sensor to emit data this way or if users are working with highly constrained resources and need to keep their payloads as small as possible.

Enter topic parsing! Telegraf solves this problem without needing to store the whole topic as a tag by allowing users to specify which portions of a topic they want to bring in as measurement, tags, or fields. Before this release, this would have been accomplished with the Regex Processor Plugin but the new experience is simpler and more convenient.

A simple configuration change to the mqtt_consumer input plugin will give us what we need:

[[inputs.mqtt_consumer]]
    servers = ["<address:port>"]
    topics = ["sensors/#"]
    [[inputs.mqtt_consumer.topic_parsing]]
        topic = "sensors/+"
        measurement = "measurement/_"
        tags = "_/sensor_id"

The above snippet of Telegraf configuration instantiates a client subscribed to sensors/# (any topic with parent sensors). If you’re familiar with Input Plugins like SNMP or parsers like JSON_v2 (going over this next if not!), you’ll recognize the nested TOML table defined by [[inputs.mqtt_consumer.topic_parsing]]. There can be any number of these but we only need one. In this table, you tell Telegraf which topics to match on and — with the information that follows — you will parse those topics as their messages arrive. Following that is where you tell Telegraf what to make a measurement, tag/s, and/or Field/s. In this case, we are only pulling out the Measurement and a single tag (there isn’t much information to be parsed, after all). No Field present…but that won’t always be the case.

We use the syntax measurement/_ in a pair of quotes to telegraph (pun intended) to Telegraf which segments of the topic name will be parsed into what LP elements. Keys map to their respective indexed (by segment) value in the topic, and underscores map to segments that are not relevant to that Line Protocol element (placeholders).

In the example configuration above, we tell telegraf that the first segment will be our measurement and the second (last) segment will be a tag with key sensor_id. The actual information in that segment will be populated as the value of that tag key. If we want to now take out the default topic tag, we can include a topic_tag = "" parameter in the config and get this output:

mqtt_consumer,sensor_id=00191 value=10.0 <timestamp>

Modifications like this matter at scale and with more complicated topics and payloads. It also improves analytics experience downstream…a lot.

Quick note: it’s a best (not necessary) practice to have field names contain useful information so “value” is not ideal. To fix this, you can simply use the Pivot Processor (out of scope for this post).

Anyway, speaking of more complicated topics and payloads, let’s look at two more quick brief examples:

Topic: detroit/0f801/42.33/-83.04/front_left/09fse21/rpm Message: 846

We want our LP output to have all this information in it. The first segment is the city (site) this vehicle belongs to. The second is the vehicle’s ID. Third is the latitude at that given sample time. Fourth is the longitude, fifth is location of tire on the vehicle, sixth is the tire’s ID, and last is the rpm of that tire. All potentially important information!

I won’t go into reasoning for what segments should which LP elements — for that, you can refer to the blog post I linked at the top. For this example, we’ll be generic with the measurement and let Telegraf name it. Our measurement will be mqtt_consumer (Telegraf can rename this in other plugins if necessary). Every string will be a tag and every numerical element will be a Field. Ultimately, we want our resulting LP to look like:

mqtt_consumer,site=detroit,vehicle_id=0f801,tire_id=09fse21 lat=42.33,lon=-83.04,rpm=846 <timestamp>

The following relevant [[inputs.mqtt_consumer]] configuration will get us there:

...snipped...
    [[inputs.mqtt_consumer.topic_parsing]]
        topic = "+/+/+/+/+/+/+"  # all topics with 7 segments
        tags = "site/vehicle_id/_/_/orientation/tire_id/_"
        fields = "_/_/_/_/_/_/rpm"

There you have it — a concise configuration for pulling topic-encoded information into Line Protocol records from every single topic that follows a 7-segment structure!

Payload parsing!

Parser Plugins are not new to Telegraf and neither is specifically JSON parsing but I feel the MQTT consumption topic deserves a drill-down into JSON parsing. First, note that Telegraf offers a list of other parsers as well. Parsers are invoked inside of plugins with the data_format parameter. They are configured slightly differently as their respective formats vary in semantics. For this post, we will focus on JSON parsing.

JSON parsing in Telegraf has been around a long time, but it was revamped in a new version a few releases ago. Just try to throw at it some JSON it can’t parse!

Before I dive in here, if you want a thorough walkthrough of this upgraded parser, this blog post by the Telegraf Product Manager, Samantha Wang, is fantastic. Additionally, if you prefer to parse in the Python dialect, there is another way I wrote about here.

As you may know, MQTT will accept just about any type of payload. JSON is the most common in this context. That said, JSON can also be just about any shape so configurable parsing is paramount!

To bridge the sections, we’ll stick with the last example message but change it up slightly by putting some info that was in the topic into a JSON payload:

Topic: detroit/0f801/09fse21 Message:

{
    "orientation": "front_left"
    "latitude": 42.33
    "longitude": -83.04
    "rpm": 846
}

For this, the user will want to parse the topic as it comes just as done before. That configuration will look like:

...snipped...
    [[inputs.mqtt_consumer.topic_parsing]]
        topic = "+/+/+"  # all topics with 3 segments
        tags = "site/vehicle_id/tire_id"

For this one, we are keeping the Measurement as we had it (Telegaf’s default per Input) and we are setting no fields. The reason for the latter is we know we can get our fields from the payload this time. So let’s parse it up!

To make JSON parsing flexible enough to handle heavily nested blobs and other complexities, this new JSON_v2 parser uses nested TOML tables and GJSON querying for accessing desired blobs. The above message payload is small and flat so the parser will be heavily underutilized this time.

From the combination of the above topic and JSON body, we want the same output that we had in the topic parsing example before it. As a refresher that LP looked like:

mqtt_consumer,site=detroit,vehicle_id=0f801,tire_id=09fse21 lat=42.33,lon=-83.04,rpm=846 <timestamp>

Keeping in mind the shorter topic we just parsed above, here is the accompanying JSON parsing configuration:

...snipped...
data_format = "json_v2" # invokes the parser -- lines following are parser config
[[inputs.mqtt_consumer.json_v2]]
    [[inputs.mqtt_consumer.json_v2.tag]]
        path = "orientation" # GJSON path: JSON is flat -- all keys at root
    [[inputs.mqtt_consumer.json_v2.field]]
        path = "rpm"
        type = "int"
    [[inputs.mqtt_consumer.json_v2.tag]]
        path = "latitude"
        rename = "lat"
        type = "float"
    [[inputs.mqtt_consumer.json_v2.field]]
        path = "longitude"
        rename = "lon"
        type = "float"

That will produce the same result as the example where all the metadata was in the topic, itself. Either case is totally within reason.

I can understand if the JSON parsing looks a little verbose for such a simple JSON blurb. The verbosity glows in more complicated cases. For this case, here is an example of the same result using a different way to parse tags and a different way to parse fields:

...snipped...
data_format = "json_v2" # invokes the parser -- lines following are parser config
[[inputs.mqtt_consumer.json_v2]]
    [[inputs.mqtt_consumer.json_v2.object]]
        tags = ["site"] # can list all Tags -- our case only has one
    [[inputs.mqtt_consumer.json_v2.object.field]] # fields by key and type
        rpm = "int"
        latitude = "float"
        longitude = "float"

That way of defining fields is my favorite part about this parser. So concise! Also notice the object table used. If you just need to define your tags and fields and don’t need to do special things with each, these are good methods for keeping your configuration shorter!

Wrapping up

So that’s the nuts and bolts. You now know what it looks like to have Telegraf consume from an MQTT broker, subscribe to many topics, and parse data from the topic and payload simultaneously, merging that data into LP record/s. InfluxDB will love you for this!

If you’re at a stage where you’re either beginning to develop a pipeline of IoT data and MQTT is in the mix, inserting Telegraf as a consumer (even a producer as well if you feel so inclined) to distribute data to your data stores could be a good move. If you’re using or planning to use InfluxDB as the time series component to your data store layer, Telegraf is optimally designed for feeding InfluxDB. Regardless of where your data ends up, if you need a relatively simple way to get data of various forms and sources from an MQTT broker/cluster, Telegraf can do it.