InfluxData Blog - Michael Hall

Congratulations to the Third Annual InfluxDays Community Awards Winners

Michael Hall (InfluxData) — Fri, 04 Nov 2022 07:00:00 +0000

Every year since I started at InfluxData, it has been my honor to recognize those community members who have made outstanding contributions to both the InfluxDB community and to our products over the past year. It’s always amazing to see people sharing their knowledge, their skills, and their passion with each other. Our community is an integral part of our success; we couldn’t build Telegraf, InfluxDB or Flux without their feedback and perspectives.

InfluxDB is the purpose-built time series data platform with over 705,000 daily active instances of InfluxDB OSS running. There are 75,000 + developers using InfluxDB Cloud globally — the number of people using the serverless platform has nearly doubled since last year. We’re excited to see the community growing every year.

We just wrapped up InfluxDays 2022 — our annual user conference. InfluxDays 2022 included watch parties, conference sessions, and InfluxDB University trainings. Additionally, developers were able to sharpen their skills with the InfluxDB University Challenge throughout the conference. It was a blast getting to hang out with our community members in person again, and share all of the exciting developments we’ve been making to the platform. Learn more about InfluxDB IOx, InfluxDB’s new storage engine, here.

The InfluxData Community Award winners are nominated by the people they interact with, both InfluxData staff and other users in the community. Because there are so many places and ways to contribute to the InfluxDB Community, we choose one winner for each of the following categories:

InfluxDB - For general contribution to InfluxDB itself, including the UI, deployment configurations, documentation or tutorials

Telegraf - For contributing bug fixes, new features, plugins, documentation, or tutorials

Flux Language - For contributing to the design and implementation of Flux, improvements to the language, extensions, documentation, tutorials, or support

Community - For providing community support, events, engagement, promotions or any other activity that has helped us grow our global community

The 2022 InfluxData Community Award winners are:

The InfluxDB Award: Arvid Teichtmann (also known as Franky1) – For his consistent support in the community forums, especially for answering questions about InfluxDB and sharing his knowledge of its tools.
The Telegraf Award: Thomas Casteleyn – For his continued outstanding support of the Telegraf community. Nobody has provided more support, answered more questions, or helped more people than Thomas has.
The Flux Language Award: Alfonso Acosta – For making many significant fixes and contributions to the Flux language itself and engaging with the Flux engineering team to help improve the language for everyone.
The Community Award: Giovanni Luisotto – For contributions across our community forums, Slack, Github and Stack Overflow, and across a broad range of topics, which highlights both the depth and breadth of his knowledge.

Congratulations to this year’s winners! It’s not easy picking just four winners out of the tens of thousands that have been active in our community, but these four stand out for their sustained involvement and commitment to helping other users of InfluxDB, Telegraf, and Flux. I would like to personally thank them for their contributions to the community and congratulate them on receiving the recognition and appreciation of their peers. We invite you to join the Built on InfluxDB program to share your InfluxDB projects and solutions here.

Using InfluxDB as an IoT Edge Historian

Michael Hall (InfluxData) — Thu, 17 Mar 2022 04:00:20 -0700

InfluxDB is increasingly being used in IoT solutions to store data from connected devices. Now it can also be used on IoT edge gateways as a data historian to analyze, visualize and eventually transmit aggregated IoT data up to a centralized server. In this article we’re going to look at three simple ways you can connect an instance of InfluxDB on your IoT Edge device to another instance of InfluxDB in the cloud.

Preparing your edge instance of InfluxDB

To begin with, we need a local instance of InfluxDB to use as our edge historian. The simplest way to do this is in your local development to run the official Docker image:

docker run -p 8086:8086 influxdb

Then you will need to complete the setup by going to http://localhost:8086 and following the on-screen instructions. Remember what you use for your Organization during this setup – you’ll need that information later!

Next, navigate to the Load Data -> API Tokens screen, and grab the token string that was generated for you during setup – you’ll need that later too.

Tip: The default token lets you control everything in InfluxDB — it’s very powerful. When deploying your solution into production, you’re going to want to create a new, less powerful token that only has permissions to read or write from specific buckets.

Then use these values to setup your Influx CLI, which we will be using extensively in the rest of this article:

influx config create –active –name "edge" –host "http://localhost:8086" –org "your_edge_org" –token "your_edge_token"

Now it’s time to setup a bucket on this instance to store our device data in:

influx bucket create --name "devices" --retention 1h

We’re using a 1 hour retention policy on this bucket. In a real world scenario where you’re gathering a lot of high-frequency sensor data on a relatively low-powered IoT edge device, you’re not going to have a lot of space to store it indefinitely. InfluxDB’s retention policies make it easy to put a limit on that. Don’t worry about losing data though — this article will show you how to keep that data around long-term by transmitting it up to your cloud instance.

Faking an IoT device

While the rest of this article will work for any IoT data you are collecting, we’re going to keep the examples simple by using a mock device to generate some artificial sensor data. To do that we’re going to use the new inputs.mock plugin from the Telegraf project. We will generate two fields: temperature and humidity, like you might get from an inexpensive DHT sensor.

First we’ll generate a new Telegraf configuration using just the plugins we need for this article:

telegraf --aggregator-filter none --processor-filter none --input-filter mock --output-filter influxdb_v2 config > telegraf.conf

This will create a new file for you named telegraf.conf with sections for inputs.mock and outputs.influxdb_v2.

You will need to fill these sections in with the following data:

[[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:8086"]
  token = "your_edge_token"
  organization = "your_edge_org"
  bucket = "devices"

And

[[inputs.mock]]
  ## Set the metric name to use for reporting
  metric_name = "dht"

[[inputs.mock.stock]]
  name = "temperature"
  price = 20.0
  volatility = 0.05

[[inputs.mock.stock]]
  name = "humidity"
  price = 50.0
  volatility = 0.05

[inputs.mock.tags]
  "device_id" = "001"

The first section tells Telegraf how to connect to your edge instance to write data to it. Use the organization name and token string you captured during the setup of your edge instance.

The second section defines our mock device with temperature and humidity fields, and device_id tag. Here I’m using the inputs.mock.stock type to generate a field with a value that changes randomly but gradually over time, as temperature and humidity would. There are other generator types that you can use to more accurately simulate other forms of data.

Tip: You can use multiple inputs.mock sections to simulate multiple devices at the same time, giving them each a different device_id tag value.

We’re also going to change the default Telegraf collection interval to one second, because IoT devices don’t typically wait 10 seconds between records, so scroll up to the agent section and change the interval to 1s like this:

[agent]
  ## Default data collection interval for all inputs
  interval = "1s"

Now you can start Telegraf using this configuration and let it start sending data to your edge instance while we get your cloud instance ready:

telegraf –config ./telegraf.conf

Preparing your cloud instance of InfluxDB

Your cloud instance is going to be where you aggregate data from multiple edge devices at once. It doesn’t have to be “in the cloud” specifically, it can be running in your private data center or anywhere else, the key thing is that it’s somewhere all of your edge devices can talk to it. If you don’t already have such an instance available, you can sign up for a free account on the InfluxDB Cloud platform. Everything in this article will run within the free tier, so it won’t cost you anything to follow along.

Just like with our edge instance, you will need to gather the host, organization name, and access token from your cloud instance. Then we’ll use them to create a second configuration in the CLI, so you can switch back and forth between your edge instance and your cloud instance:

influx config create –active –name "cloud" –host "your_cloud_host" –org "your_cloud_org" –token "your_cloud_token"

Tip: You can see both of your configurations by running influx config list, and switch between them using influx config edge or influx config cloud respectively. Then we’re going to create a bucket to store data coming in from our edge devices:

influx bucket create --name "edge_devices" --retention 30d

Unlike our devices bucket on our edge instance, here we will use a 30 day retention policy, because our cloud instance has room for that (and it fits within the free tier use limits for InfluxDB Cloud).

Store cloud authentication in your edge host

This step is optional, but highly recommended. You’ll need to use your cloud credentials in Flux tasks to connect to your cloud instance from your edge instance, but keeping those credentials in plain text in your scripts is poor security.

So instead we’re going to make use of InfluxDB’s built-in secret store to put those in a write-only space that we can reference from our scripts, using the following commands:

influx config edge
influx secret update --key cloud_host  --value "your_cloud_host"
influx secret update --key cloud_org   --value "your_cloud_org"
influx secret update --key cloud_token --value "your_cloud_token"

Basic Store and Forward

The simplest setup is to take all of the data from your edge instance, and forward it as-is up to your cloud instance. To do that, create a new Task on your edge instance, set to run every minute.

You can do this from the UI at http://localhost:8086 or using the VisualStudio Code plugin for Flux, which is the way I did it:

The we’ll use the following Flux script for the task:

import "influxdata/influxdb/tasks"
import "influxdata/influxdb/secrets"

option task = {name: "northbound", every: 1m, offset: 0s}

cloud_host = secrets.get(key: "cloud_host")
cloud_org = secrets.get(key: "cloud_org")
cloud_token = secrets.get(key: "cloud_token")

from(bucket: "devices")
|> range(start: tasks.lastSuccess(orTime: -1h))
|> set(key: "edge_id", value: "001")
|> to(host: cloud_host, org: cloud_org, token: cloud_token, bucket: "edge_devices")

Let’s walk through this step by step:

import "influxdata/influxdb/tasks"
import "influxdata/influxdb/secrets"

The import lines bring some key functionality into our Task’s script. The first line gives us a tasks package that we will use to find the last time this task ran successfully. The second line gives us a secrets object which will give us access to the cloud credentials we saved to the InfluxDB secret store previously.

option task = {name: "northbound", every: 1m, offset: 0s}

This line was automatically included for us when we created this new task, using the values we entered in the VSCode form. This is how InfluxDB Tasks store these values.

cloud_host = secrets.get(key: "cloud_host")
cloud_org = secrets.get(key: "cloud_org")
cloud_token = secrets.get(key: "cloud_token")

The next three lines retrieve our saved secrets, and place them in Flux variables. We will pass these variables to our Flux to() function so that it can connect to our cloud instance.

from(bucket: "devices")
|> range(start: tasks.lastSuccess(orTime: -1h))

The first part of our query specifies what data to fetch from our local edge instance. We’re reading data from the devices bucket that we configured Telegraf to write to. Here we’re using the tasks package we imported earlier to find the last time that this task ran, this will let us only fetch new data that hasn’t already been sent to the cloud, saving us some bandwidth. It also makes this task more robust because if it fails for whatever reason (such as a connectivity issue between your edge and cloud) the next run will include all the data from the previous one.

The orTime parameter provides a default time, which will be used the first time the task is run.

|> set(key: "edge_id", value: "001")

This next line adds a bit of extra information to our data. In a typical edge IoT scenario you’re going to have more than one edge location, and each location is going to have multiple devices. We’re already collecting a device_id from Telegraf, but there’s nothing in that data that says what edge location it was being collected from. Once we have data from multiple locations being mixed together in the cloud it would be difficult to distinguish between them. So here we’re going to inject an edge_id into the datastream before sending it, to identify our edge location.

|> to(host: cloud_host, org: cloud_org, token: cloud_token, bucket: "edge_devices")

Finally we’re ready to send the data up to the cloud! This last step only takes a single function call, the built in to() function, which can write your query data to a local bucket or, in this case, ro a remote bucket by passing in the authentication credentials for our remote InfluxDB instance.

That’s it! Save your task and from now on all of your local data will be copied up into your cloud instance, every minute, automatically!

Downsampling before Forwarding

While the task we just wrote is simple and works well, it’s still going to be sending a lot of data up to your cloud. In a real world scenario you’re going to have dozens, if not hundreds, of devices on each edge, and some of them will be producing data multiple times per second, or even microsecond! Even with our mock device we’re producing a new value every second. That’s a lot of very detailed data that you most likely don’t need for the high level use cases you’ll be running in the cloud, it’s just going to take up extra bandwidth to transfer and extra space to store.

Instead, what if we could reduce the volume of data before we send it, even if that means losing some of the granularity? This process is known as downsampling or materialized views, and is extremely common when moving data up to higher and higher levels. It’s also extremely easy to do with Flux, right inside our task! Here’s what it looks like:

import "influxdata/influxdb/tasks"
import "influxdata/influxdb/secrets"

option task = {name: "northbound", every: 1m, offset: 0s}

cloud_host = secrets.get(key: "cloud_host")
cloud_org = secrets.get(key: "cloud_org")
cloud_token = secrets.get(key: "cloud_token")

from(bucket: "devices")
|> range(start: tasks.lastSuccess(orTime: -1h))
|> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
|> set(key: "edge_id", value: "001")
|> to(host: cloud_host, org: cloud_org, token: cloud_token, bucket: "edge_devices")

As you can see, we only needed to add a single line to our Flux code:

|> aggregateWindow(every: 5m, fn: mean, createEmpty: false)

This line takes the data we’re reading from our bucket, and calculates the average (mean) value of each field over a rolling 5 minute window, reducing 300 individual measurements into just one. Then that dowsampled data is sent to our cloud instance just like before.

Flux provides a number of built-in aggregation functions beyond mean, you can instead calculate the min, max, integral, standard deviation, and much more! You can also define your own aggregation function if you want something more complex, like a swinging door compression algorithm, it’s entirely up to you.

Adapting for high resiliency at the Edge

By now we have a task that is both efficient in what data it is sending to the cloud, and resilient to short connectivity problems. But what if those connectivity problems persist for more than an hour? That may not be a concern for a smart building or factory, but if your edge is a wind turbine or oil rig it’s a real possibility. Because our devices bucket that gathers raw data only has a 1 hour retention policy, any outage longer than that will result in lost data. We can extend that retention, but then storage space becomes a concern as you’re having to save more and more of that high volume data.

The solution to this problem is to store our smaller downsampled data on the edge for a longer period of time. To do that, we first need a new bucket:

influx config edge
influx bucket create --name northbound --retention 1d

For our new bucket, which we called northbound, we’re using a 1 day retention policy, instead of just one hour. That will allow our edge instance to be disconnected for a full 24 hours before we start to lose data.

The next step is to split our task into two. First, create a new task named aggregate_local, and set it to run every minute. This task will perform the downsampling of data from the devices bucket, and store that in the northbound bucket.

import "influxdata/influxdb/tasks"

option task = {name: "aggregate_local", every: 1m, offset: 0s}

from(bucket: "devices")
|> range(start: tasks.lastSuccess(orTime: -1h))
|> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
|> set(key: "edge_id", value: "001")
|> to(bucket: "northbound")

This task no longer needs to import influxdata/influxdb/secrets, or fetch credentials from the secret store, because it’s writing our downsampled data to a local bucket. In this task we’re performing both the downsampling with aggregateWindow, and injecting our edge_id with the set function, so that the data being written to northbound is exactly what we want being sent to the cloud.

The second task will then read data from the northbound bucket and send it, without modification, to our cloud instance. Create this task with the name sync_northbound and set it to run every 5 minutes.

import "influxdata/influxdb/tasks"
import "influxdata/influxdb/secrets"

option task = {name: "sync_northbound", every: 5m, offset: 0s}

cloud_host = secrets.get(key: "cloud_host")
cloud_org = secrets.get(key: "cloud_org")
cloud_token = secrets.get(key: "cloud_token")

from(bucket: "northbound")
|> range(start: tasks.lastSuccess(orTime: -1h))
|> to(host: cloud_host, org: cloud_org, token: cloud_token, bucket: "edge_devices")

Here we once again use the secret store to retrieve our credentials, but our query doesn’t need the aggregateWindow or set calls, because we’ve already performed those steps before writing data to the northbound bucket.

Conclusion

Now we have InfluxDB running at the edge, collecting data from connected devices (mock ones in this case), storing that data temporarily and the edge while also transmitting it up to another InfluxDB instance running in the cloud. We went through multiple iterations of task scripts to make this process more efficient with downsampling, and more robust to connectivity issues by extending local retention. All of this was accomplished using only InfluxDB and just a handful of lines of Flux script.

But this is only a small taste of what you can do! Once you start tailoring this setup to your own specific needs you can enhance your Flux scripts with smarter downsampling algorithms, enhance your device data with meta-data from SQL servers, trigger alerts based on thresholds or loss of data from devices, and so much more.

How to Pivot Your Data in Flux: Working with Columnar Data

Michael Hall (InfluxData) — Tue, 12 Oct 2021 04:00:33 -0700

Relational databases are by far the most common type of database, and as software developers it’s safe to say that they are the kind of database most of us got started on, and probably still use on a regular basis. And one thing that they all have in common is the way they structure data. InfluxDB, however, structures data a little bit differently.

RDBMS default table structure

With a relational database you have a collection of tables, each with its own fixed layout of columns, and you read and write data in rows. Each row has a column that holds a value for each field, and you get all of those values at the same time, giving you a table that looks like this:

time	field1	field2	host
<data_tstamp>	123	789	myhost1
<data_tstamp>	456	101	myhost2

InfluxDB result data

InfluxDB doesn’t follow this pattern. It returns data in what’s known as a columnar format, which just means that you will get the data one column at a time, rather than one row at a time. This is often a source of confusion for developers who are used to the row format they worked with from a relational database, but it has many benefits when working with time series data.

When you run a basic Flux query like this:

from(bucket: "telegraf")
  |> range(start: -1d)

You will get data returned like this:

table	_measurement*	_field*	_value	_start*	_stop*	_time	host*
0	mydata	field1	123	<query_start>	<query_stop>	<data_tstamp>	myhost1
0	mydata	field1	456	<query_start>	<query_stop>	<data_tstamp>	myhost2
1	mydata	field2	789	<query_start>	<query_stop>	<data_tstamp>	myhost1
1	mydata	field2	101	<query_start>	<query_stop>	<data_tstamp>	myhost2

The first thing you’ll notice is that the table’s column headers are NOT your field names like they would be if you were querying a SQL database. Instead your query will return your results with these same columns, and each record (represented by r in the query below) in the result will represent a single field’s value at that point in time.

This is why when you filter your Flux query, it looks like this:

|> filter(fn: (r) => r["_measurement"] == "mydata")
|> filter(fn: (r) => r["_field"] == "field1")

This data structure allows for faster queries and aggregations in the most common usage patterns for InfluxDB. Because InfluxDB is processing one column’s values at a time, it can perform calculations on the data for just that field instead of performing multiple calculations at the same time.

Pivoting between column-like and row-like data layouts

There are times, however, when you are going to want your data to look more like the results of a traditional database query. To do that you need to perform a “pivot” operation. Pivoting data is the act of flipping it from having the fields in separate rows to having the fields in separate columns.

Flux provides a pivot function that does precisely this:

|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")

This function will flip your data so that every row represents a moment in time (rowKey:["_time"]), your fields will each become a column (columnKey: ["_field"]), and the value in each column will be your field’s value (valueColumn: "_value"). Giving you a result that looks like this:

table	_measurement*	_start*	_stop*	_time	host*	field1	field2
0	mydata	<query_start>	<query_stop>	<data_tstamp>	myhost1	123	789
0	mydata	<query_start>	<query_stop>	<data_tstamp>	myhost2	456	101

Using fieldsAsCols instead

This is the most common way people want to pivot their data, because it puts it into the same shape as they’re used to working with from a traditional database. It’s so common, in fact, that we provide a shortcut function in the schema package just for that use case.

After importing schema into your Flux script, the above line of code can be replaced with:.

Flux
  |> fieldsAsCols()

Other ways to pivot

Tags as columns

Even though it’s the most common way to pivot, fields are not the only thing you can flip up into column headers. Suppose you want to compare values not against different fields, but against the same field from different sources. For example, if you want to compare readings from different server hosts taken at the same time.

In our previous example, our data had a tag called host that contained this value.

table	_measurement*	_field*	_value	_start*	_stop*	_time	host*
0	mydata	field1	123	<query_start>	<query_stop>	<data_tstamp>	myhost1
0	mydata	field1	456	<query_start>	<query_stop>	<data_tstamp>	myhost2
1	mydata	field2	789	<query_start>	<query_stop>	<data_tstamp>	myhost1
1	mydata	field2	101	<query_start>	<query_stop>	<data_tstamp>	myhost2

If we wanted to combine the values of each field from all of our hosts into a single record, we could do that with:

|> group()
  |> pivot(rowKey:["_time", "_field"], columnKey: ["host"], valueColumn: "_value")

The first line re-groups our data because InfluxDB by default will put values from different hosts (or any difference in tag values) into separate result tables. Since pivot works on one result table at a time, we have to call group first to put the values from both hosts into a single table, then we can pivot that combined table around the host values.

This query gives us a table structure that looks like this:

table	_field	_start	_stop	_time	myhost1	myhost2
0	field1	<query_start>	<query_stop>	<data_tstamp>	123	456
0	field2	<query_start>	<query_stop>	<data_tstamp>	789	101

Combined columns

You can also use pivot to combine field names with tag values. Building off the above example, suppose you want to see the values of both field1 and field2 for each host in a single row?

We can do that with a slight modification to our previous function call:

|> group()
  |> pivot(rowKey:["_time"], columnKey: ["host", "_field"], valueColumn: "_value")

By moving the _field data from the rowKey to the columnKey, along with the host data, we get a table that combines both into unique columns.

table	_start	_stop	_time	myhost1_field1	myhost1_field2	myhost2_field1	myhost2_field2
0	<query_start>	<query_stop>	<data_tstamp>	123	789	456	101

Flux Aggregation in InfluxDB: Now or Later

Michael Hall (InfluxData) — Fri, 10 Sep 2021 04:00:06 -0700

Aggregations are a powerful tool when processing large amounts of time series data. In fact, most of the time you’re going to care more about the min, max, mean, count or last values of your dataset than you will about the raw values you’re collecting.

Knowing this, InfluxDB and the Flux language make it as easy as possible to run these aggregations, whenever and wherever you need to, and sometimes that leads people to running them in ways that aren’t as efficient as they could be. Here are some ways to ensure your aggregation query runs as fast as possible.

Don't aggregate too early

As great as aggregate functions are and as much as you want to use them, be careful not to use them too soon. Oftentimes, we’ll see somebody with a query like this:

from(bucket: "myBucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "cpu")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> filter(fn: (r) => r["host"] == "myHost")
  |> filter(fn: (r) => r["cpu"] == "cpu-total")
  |> yield(name: "mean")

While this query will return the exact results you want, in this case the average total CPU for a specific host, it’s also doing a lot more work than it needs to.

Because the aggregateWindow function is being called before the filter calls on host and cpu, InfluxDB ends up calculating the average for all host and cpu values first, and then dropping the raw data after it’s done all that hard work.

Instead, perform all the filtering you can before using an aggregate function. This reduces the total amount of data going into those calculations, which will give your query a big speed boost especially on large data sets.

from(bucket: "myBucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "cpu")
  |> filter(fn: (r) => r["host"] == "myHost")
  |> filter(fn: (r) => r["cpu"] == "cpu-total")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")

You can still filter data after calling an aggregate, which is useful when you want to filter on the results of the aggregation. But for filters that only check the raw data in your bucket, it’s always best to apply them first. In fact, that’s what the InfluxDB UI’s query builder does automatically!

Don't aggregate too late either

While you don’t want to get ahead of your data when it comes to aggregation, it’s also possible to call them too late in your query, resulting in a slower response time. For example:

from(bucket: "myBucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "cpu")
  |> filter(fn: (r) => r["host"] == "myHost")
  |> filter(fn: (r) => r["cpu"] == "cpu-total")
  |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
  |> last(column: "_time")

Again this will do what you want: give you the last value of every field in the matching series presented as a single row with fields as columns. However this query will have InfluxDB doing the pivot on the entire data set only to once again throw away that hard work immediately afterwards.

Here it is better to call the last aggregate function before pivoting the data, to reduce the amount of data that has to be transformed to just the data you want in the end.

from(bucket: "myBucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "cpu")
  |> filter(fn: (r) => r["host"] == "myHost")
  |> filter(fn: (r) => r["cpu"] == "cpu-total")
  |> last(column: "_time")
  |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")

Pushdown opportunities

When your Flux query is executed, the first thing it does is read your measurement data from the storage layer of InfluxDB before performing your calculations and transformations on that data in memory. Some of those steps can be performed by the storage layer instead, meaning less data being read into memory to begin with.

When this happens we call it “pushing down” those steps to the layer below Flux, the storage engine layer, and query patterns that can do this are called pushdown patterns. Not only do these result in less data being read into memory, they reduce the amount of work the Flux engine has to do.

The first and most basic pushdown pattern is the common from() |> range() |> filter() that most Flux queries use. If you can put your aggregate function immediately after one of these, some of them can also be pushed down to the storage layer. Let’s take a look at our last query:

from(bucket: "myBucket")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "cpu")
  |> filter(fn: (r) => r["host"] == "myHost")
  |> filter(fn: (r) => r["cpu"] == "cpu-total")
  |> last(column: "_time")
  |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")

Because we put the last function immediately after a from() |> range() |> filter() pattern, it will also be pushed down to the storage layer. This means the only data that will actually be read into the Flux runtime is the last values of our cpu measurement for the selected series, and the only step executed in memory is the pivot step at the end. If we had left it calling last after the pivot, not only would all of that extra data have to be read into memory, but both the pivot and last functions would have been executed in the Flux layer.

Putting it all together

So to recap, aggregation is great and Flux is really good at it, and you can make it work better for you by doing as much filtering as possible before applying the aggregation, waiting on performing any transformations of the data until after the aggregation, and keeping an eye out for opportunities to use pushdown patterns that let the storage layer do more of the heavy lifting.

InfluxData Community Awards

Michael Hall (InfluxData) — Tue, 24 Aug 2021 02:00:21 -0700

We at InfluxData are pleased to announce that we’re once again going to be giving out Community Awards to those who have made significant contributions to our products or community over the past year.

If somebody has helped you answer a question, solve a problem, or has built something cool and interesting that you think deserves some extra recognition, go and nominate them for an award now!

Learn more.

Previous Community Awards

The past 18 months have been quite a change for all of us, and with our normally in-person conference InfluxDays going virtual we wanted a way to give a little bit of extra focus on the people in the community who have been helping both InfluxData staff and InfluxDB users online over that time.

That led us to introducing the InfluxData Community Awards as a way for our staff and users to say “thank you” to the people who are helping to make our products and our community better. Award winners get a beautiful, personalized trophy highlighting their contribution, and more importantly the recognition and appreciation of the whole community.

Last year’s award winners have continued their contributions, with some even taking on larger roles. One joined the inaugural Telegraf Maintainers team, a joint project between InfluxData and community contributors to maintain and develop the Telegraf project together. The Founder’s Choice winner even took a career change and joined us here at InfluxData to continue helping our users full-time!

2021 Community Award nominations

Now another year has gone by and we want to again recognize our community contributors, so we’re seeking nominations for the following award categories:

InfluxDB - For general contribution to InfluxDB itself, including the UI, deployment configurations, documentation or tutorials
Telegraf - For contributing bug fixes, new features, plugins, documentation or tutorials
Flux Language - For contributing to the design and implementation of Flux, improvements to the language, extensions, documentation or tutorials
Community - For providing community support, events, engagement, promotions or any other activity that has helped us grow our global community

Anybody in the community can be nominated, and anybody in the community (or InfluxData) can nominate somebody. You can even nominate multiple people, for the same category or different ones. Once the nomination period closes on September 30th, the results will be reviewed and winners selected by InfluxData staff.

Submit a nomination today!

This is your chance to really say “thank you” and recognize somebody for the work they’ve done or help they’ve given you. Nominations have already started coming in, and we will be taking more up until September 30th, so go make your nominations now!

Don’t forget to register for your free spot for InfluxDays North America so you can see the winners announced live on October 27, 2021! See you there!

Submit nomination.

InfluxDB Latest Tag Updated in Docker Hub

Michael Hall (InfluxData) — Fri, 26 Feb 2021 12:32:47 -0700

The latest InfluxDB OSS (version 2.0.4) has officially been released on Docker Hub! While this will allow a smooth upgrade path for both new and existing users of InfluxDB, we found out that some users were being unexpectedly upgraded to the new images due to having configured their deployments to always use the :latest tag of these images.

In our effort to ensure the best experience possible, we’ve documented here what happened to cause this, how to avoid it, and what to do if you’ve found yourself in a similar situation.

What happened

Over the past several weeks, we have been working with Docker Hub to update their official InfluxDB images to the new InfluxDB OSS 2.0 release. The final step in that process was updating the :latest tag reference to point to the new InfluxDB 2.0.4 version of the container image. That update was accepted by Docker Hub a couple of days ago, and went live later that night. Since then, anybody who pulled a copy of the image, either using the :latest tag explicitly or using no tag at all, started using the InfluxDB 2.0.4 image.

This came as a surprise to some users, who were updating an environment that was originally deployed with the InfluxDB 1.8 version of the container, and now found themselves running InfluxDB 2.0 — especially because their data files from InfluxDB 1.8.x are not compatible with the InfluxDB 2.0 codebase.

If this happened to you, don’t worry, YOUR DATA IS SAFE! InfluxDB OSS has built-in migration tooling to update your data files to the new format, but it will not be run automatically. Depending on your setup, you can either go back to the InfluxDB OSS 1.8 version of the Docker image, or run the command to upgrade your data to work with the latest version. For a list of key differences between InfluxDB 1.x and InfluxDB 2.x, please check out our documentation.

The next section will explain in more detail why this happened and what we’re going to do going forward to help our users avoid it. If you just want to fix your setup and get back to work, scroll down to the “How to Fix It” section

Why this happened

The :latest tag is a convenience provided by Docker Hub that makes it easy to pull the most up to date version of an image without having to know what that version is. It’s useful for automating things like CI where you always want to use the most recent version of your software, and for developers who just want to get started running your software as quickly and easily as possible.

As the default when no tag is specified, it also allows for shorter instructions such as docker run influxdb, which we can put in our documentation and not have to update every time the release version changes.

However, it is NOT meant to be used for deployments, since the version it refers to will constantly be changing. The official Kubernetes instructions warn against this use explicitly:

"You should avoid using the :latest tag when deploying containers in production as it is harder to track which version of the image is running and more difficult to roll back properly." https://kubernetes.io/docs/concepts/configuration/overview/#container-images

Even if you are only deploying a personal or hobby project you should only use :latest in your development and testing environment, and switch to an explicit release tag once you’ve got it working the way you want.

Going forward

The :latest tag for InfluxDB currently points to the 2.0.4 release images. As we release new versions of InfluxDB OSS 2.x and beyond, this tag will be updated to point to those new versions. This is how Docker Hub intends for the :latest tag to be used, and is the expected behavior among users.

We will also continue to publish docker images for the 1.x codebase, and will put out new images for each new release of that codebase. These images, however, will never again have the :latest tag pointing to them, so you will need to give the specific version tag every time you want to run or update them. We have identified several places where we can improve our documentation, both on our site and in Docker Hub, to be more explicit about when the :latest tag should be used, and when a specific version should be given instead. We will also add more documentation on how to upgrade your Docker environment from InfluxDB 1.8 images to influxDB 2.0 images.

How to fix it

If you have existing data from your InfluxDB OSS 1.8 deployment, you can either upgrade that to work with the latest InfluxDB OSS 2.0 release, or roll your deployment back to using the InfluxDB OSS 1.8 release.

Upgrade to InfluxDB OSS 2.0

If you want to stay on InfluxDB OSS 2.0, you can do an in-place upgrade of your existing data using the new version of the InfluxDB image. To do that, you will need to run your container with some additional environment variables set, these will tell InfluxDB OSS 2.0 that you want to upgrade your data and configuration files. We have updated the instructions on the InfluxDB Docker Hub page to walk you through this process.

Rollback to InfluxDB OSS 1.8

If you’d rather just get everything back to how it was before this happened, all you need to do is tell docker to use the 1.8 tag explicitly.

In the Docker CLI, you can do this with:

docker run --rm influxdb:1.8

If you are using docker-compose, add the 1.8 tag to the image: parameter in your docker-compose.yml:

image: influxdb:1.8

Likewise if you are using Kubernetes, update your pod spec to use:

image: influxdb:1.8

Acknowledgements

Lastly I’d like to thank Matthew Iverson, an InfluxDB customer and one of our community InfluxAces, who let us know that this tag update was causing confusion among users of our open source release, even though it didn’t affect him directly. I’d also like to thank Srajan Bhagat in our Customer Success team, who passed Matthew’s message on to me immediately after hearing it.

And I would especially like to thank the InfluxData Product and Engineering teams, in particular Phillip Steinbachs, Daniel Moran, Russ Savage, Jonathan Sternberg and Ryan Betts, who all dropped what they were doing and jumped onto calls with me to help find a resolution as soon as I said that our open source users were having an issue. Their commitment to ensuring the best possible experience for all of our users is what makes InfluxData and the InfluxDB community so great.

As always, you can connect with us directly in the InfluxData community by joining our Slack or visiting our forums.

Resource links

Save the Date: InfluxDays Virtual Experience is Coming Back November 10-11, 2020

Michael Hall (InfluxData) — Wed, 02 Sep 2020 06:00:38 -0700

2020 has been one crazy year! Like so many tech communities, we had planned for an exciting year filled with travel, conferences, and talking to people in person only to have all of those plans upended by the COVID-19 pandemic. InfluxData has traditionally hosted InfluxDays, the industry’s premier conference dedicated to time series data, twice a year in Europe and North America. Even though we can’t hold our editions in-person, we weren’t going to let that stop us from bringing you the kind of content and networking opportunities that you’ve come to rely on.

From in-person to virtual

We held our first InfluxDays Virtual Experience back in June, taking the place of what was originally going to be an in-person event in London. With only a few months to prepare for the change, we were not only able to hold the event successfully online, but with a record turnout of nearly a thousand people watching and interacting with our speakers live. The people who came to watch the InfluxDays sessions have remained active in our Slack and Discourse, learning from, helping and making connections with others in the community. Our speakers were engaging, their talks ranged from high level to deeply technical, and in some cases the sessions were quite literally on fire!

<figcaption> Daniel Putz presented the History of Monitoring at Volvo Cars at InfluxDays London 2020 Virtual Experience</figcaption>

Our next conference edition

Now we’re taking everything we’ve learned from all of our past InfluxDays, including that first Virtual Experience, and using that to make our next InfluxDays even better. On November 10th and 11th, we have another lineup of amazing speakers with both live and on-demand sessions, who will tell you all about the new InfluxDB 2.0, the ecosystem around it and what we have in store for the future. As always, InfluxDays will be kicked-off by Paul Dix, our CTO and the creator of InfluxDB, along with a selection of speakers from InfluxData as well as our wider ecosystem who will be there to tell you about the innovations happening in and around InfluxDB, Telegraf and Flux.

Registration for InfluxDays North America is now open and once again free for all attendees! Whether you’ll be attending for the first time or coming back again for more, I look forward to seeing you all there!

Can’t wait for InfluxDays in November? We will be holding a virtual edition of our hands-on Flux Training October 27th-28th with limited seating available. You can also start talking to us today by joining our community chat!

Become an InfluxDB Inventor

Michael Hall (InfluxData) — Mon, 15 Jun 2020 06:00:24 -0700

As InfluxDB continues to grow and improve, more and more developers are using it as the platform of choice for their own solutions. Most of these solutions range from filling niche internal needs to solving common problems faced by entire industries, but some of them are extraordinary for their impact, innovation, or how much they push the boundaries of what is possible with the technology. In order to recognize those, we are launching the InfluxDB Inventors program.

Program eligibility and two types of inventors

Anybody who builds something with, or on top of, InfluxDB is eligible to become an InfluxDB Inventor. We will specifically look for two types of Inventors: those who pioneer the use of some new product or feature of InfluxDB, and those who create something novel and impactful that we haven’t even considered before.

For the first group, we will define specific challenges or milestones that need to be met in order to qualify and will place a cap on the number of people who can become an InfluxDB Inventor by meeting that challenge. The purpose of this approach is to push forward the cutting edge parts of the InfluxDB platform and recognize those innovators who help us improve both the technology and the developer experience behind it.

But we also know that innovation isn’t limited to just what we can think of. That’s why the second group allows for InfluxDB Inventors to be nominated by anyone in our community, for any solution they feel is deserving of that recognition. These nominations will be reviewed by the InfluxDB Community team for approval based on the creativity, impact, and experience that their solutions provide. There is no limit to the number of people who can become InfluxDB Inventors this way. We have an amazingly creative community, and we plan to recognize as many of them with this program as we can.

InfluxDB Inventor recognition and benefits

The biggest prize an InfluxDB Inventor gets is the recognition of the value and impact of their work — both from InfluxData and the community at large — but Inventors will also receive some special benefits. Every InfluxDB Inventor will receive an exclusive InfluxDB Inventor t-shirt available exclusively through this program — you can’t purchase these, and they won’t be given away at conferences.

Each InfluxDB Inventor, as well as their invention(s), will also be listed in an InfluxDB Inventors Directory on our website. We will also send you a Certificate of Invention, signed by InfluxData founder and CTO Paul Dix, so you can display your accomplishment in your home or office.

As part of our kickoff of the InfluxDB Inventors, we’ve chosen the InfluxDB Templates as our first technology focus.

InfluxDB Templates allow you to create and share a comprehensive monitoring solution.

Launched earlier this year, InfluxDB Templates are a new feature available in InfluxDB 2.0 that allows you to export your entire configuration, from Dashboards to Tasks to Telegraf configurations, into a shareable and reusable format. The first 20 community contributors of an InfluxDB Template that can be reused by anybody will become InfluxDB Inventors! In fact, we already have community contributors who have done this, and who will be the very first Inventors added to this program.

Our first batch of InfluxDB Inventors t-shirts has been ordered, the directory webpage is being built, and our inaugural class of InfluxDB Inventors has already been selected. The only question remaining is: What will you invent next using InfluxDB technology?

Introducing Community InfluxDB Templates

Michael Hall (InfluxData) — Wed, 19 Feb 2020 06:00:57 -0700

With InfluxDB 2.0 we added the ability to export a configuration of your entire stack, and import it again into another instance of InfluxDB. This includes your InfluxDB buckets, dashboards, queries, alerts and even Telegraf configurations.

Since many people have the same or similar use cases, we wanted to provide a way for you to share your configurations with other users, and work together to enhance and improve them over time, just like you would any other open source project. So we’re launching a Community InfluxDB Templates project where you can publish your templates, see what other InfluxDB users have built, and try their templates yourself!

We’ve made using a Template as easy as possible, with just a few commands you’ll have everything you need up and running. You can try it now on either the InfluxDB Cloud or by downloading and running InfluxDB 2.0 yourself.

Setting up your environment

In this article I’ll be using the InfluxDB Cloud free tier to walk through using a Template. Using this hosted service is identical to using a self-hosted InfluxDB 2.0 instance, with just a couple of exceptions. First, the hosted service gives you a default Organization name, which will be the email address you used to sign up. Second, the hosted service doesn’t create an All Access token for you at the start, and you’ll need one in order to apply and use a Template.

To create a token, log into your InfluxDB Cloud instance and go to the Load Data section, and then click the Tokens tab. From there you can create a new All Access token. Be sure not to share this token with anybody you don’t want to have full access to your account! The token I’ll be showing in this article was created just for this demo, and has already been deleted so that it can no longer be used.

Next you’ll want to create some environment variables. This isn’t strictly necessary, as you can pass the same information as parameters to the influx command-line we’ll be using later, but it’ll save you some typing and make it easier to follow along if we set them at environment variables instead. Here’s what mine look like:

export INFLUX_HOST=https://us-west-2-1.aws.cloud2.influxdata.com 
export INFLUX_ORG=mhall@influxdata.com 
export INFLUX_TOKEN=MEMHwunGerdCRFG1_9Yi_GLpvGC3sr03DV1lUuvevhHJ88mqS5I7JmkyjlpMVpCDFv8Lf7rNq6MhUiavNw7Ceg==

Notice that my INFLUX_HOST is the us-west-2 zone on Amazon AWS. When you sign up for InfluxDB Cloud, you’ll be asked to pick where you want to run it. You can use any of the available options, and can find the proper value for this variable under the Load Data > Client Libraries section of the UI.

Pick your Template

The Community InfluxDB Templates repository provides a list of the available template. One of the simplest ones to start with is the Linux System Monitoring template, which will gather a number of system resource metrics from a Linux machine and display them all for you in a nice dashboard.

Apply the Template

Since we’ve already set our environment variables, the only thing we need to specify on the command-line is the location of the Template file. If you’ve cloned the Git repository with all of the Templates in it, you can point to the manifest file like this:

influx pkg --file ~/Projects/community-templates/linux_system/linux_system.yml

But an even easier way is to skip cloning, and just point directly to the manifest file on Github using the –url parameter like this:

influx pkg --url https://raw.githubusercontent.com/influxdata/community-templates/master/linux_system/linux_system.yml

This will read the Template file and present you with a list of changes it will make to your InfluxDB setup before applying it. This gives you the chance to review what the Template is going to do before it makes any changes.

Once you confirm that you want to make those changes, it will create all the specified resources for you. That’s all you need to do!

Running Telegraf configurations

While the above instructions created your dashboard and supporting resources, you’ll notice that there’s no data there yet. That’s because the Linux System Monitor Template, like most Templates, uses Telegraf to read data from some source and write it into InfluxDB. But running Telegraf from a Template is easy too, because the configuration is part of it!

So far you could have run these instructions on any machine you liked. But you’ll need to install run Telegraf on the Linux host you want to monitor.

Each Template will provide instructions for how to setup and run any additional resources, such as Telegraf. The Linux System Monitor, for example, first tells you to setup your environment variables (the same ones we set above, but this time on the Linux host we’ll be monitoring), and then tells you to get the Telegraf configuration from Load Data > Telegraf as follows:

Because the Telegraf configuration is already in your InfluxDB instance, using them is simply a matter of pointing Telegraf to it. This command will connect to your InfluxDB instance using the credentials in your environment variables, download the Linux System Monitoring configuration, and start Telegraf running with it.

telegraf --config https://us-west-2-1.aws.cloud2.influxdata.com/api/v2/telegrafs/05423ced3db76000

Now when you view your Linux System dashboard, you’ll start to see data appearing in your graphs! Be sure to check that your filters at the top are using the right bucket (telegraf) and linux_host (this will come from the host itself). Because the dashboard lets you filter by host, you can run the same Telegraf command on multiple Linux machines and monitor them all through the same dashboard!

Join the InfluxDB Community

To learn more about InfluxDB Templates and other community projects, join the InfluxDB Community Slack and Forums. There you will find InfluxData engineers and InfluxDB users working together, sharing ideas, and supporting each other.