InfluxData Blog - Nate Isley

Exploring Geo-Temporal Flux

Nate Isley (InfluxData) — Wed, 01 Jul 2020 07:00:20 -0700

Flux recently added geo-temporal capabilities to its arsenal, and I have been exploring how to effectively use this new combination of time series and geolocation data. To help get you started, we’ll cover a geo-temporal overview and then look into a few examples.

If you would like to follow along using InfluxDB 2.0 OSS (open source) or InfluxDB Cloud, deploy this AWS Lambda to collect hourly earthquake data from the U.S. Geological Survey (USGS).

Geo-temporal Flux: an overview

Flux’s geo-temporal powers are derived from a Golang S2 Geometry library. S2 is based on spherical geometry, which makes it well-suited for answering geolocation questions.

S2 has many details, but the key concepts in relation to Flux are S2 Cells and their level (from 0 to 30). S2 Cells are bounded by four spherical geodesics. At level 0, an S2 Cell represents approximately 85M km², and as the level increases, S2 Cells are subdivided into smaller areas. At level 30, a single subdivision represents an area of a little less than 1 cm².

Technically, geo-temporal analysis in Flux requires only latitude and longitude. However, tagging your data with S2 Cell Identifiers (IDs) helps to efficiently analyze data at scale. S2 Cell IDs - sometimes referred to as S2 Cell tokens - uniquely identify an S2 Cell at the designated level. To use S2 Cell IDs, you can either calculate them at query time or as the data is written to InfluxDB.

Calculating at query time uses Flux to calculate the S2 Cell ID. Calculating the S2 Cell ID with each query takes time and is inefficient when multiple queries execute on the same dataset. In contrast, preparing the S2 Cell ID while ingesting the data skips query time calculations at the cost of increased cardinality. For those that use Telegraf for data acquisition, there is an S2 Geo processor plugin that allows you to calculate and add the S2 Cell ID to your latitude and longitude data.

If you decide to add the S2 Cell ID at ingest time, consider the precision of your geo-temporal calculations and cardinality, and adjust your S2 level accordingly. For example, at level 0, six S2 Cell IDs represent earth, but at level 16, over 25 billion are required. Of course, if you are only tracking the movement of something on land, this can significantly reduce the number of S2 Cells you need; meaning you can likely up the precision — assuming the latitude and longitude data you’ve captured has sufficient precision as well.

The earthquake Lambda calculates and tags earthquakes with level 9 S2 Cell IDs at ingest time?. At level 9, each S2 Cell ID is roughly 12 miles (19.31 km) wide and there are 1.5M possible S2 Cell IDs on earth. For analyzing earthquake data, this strikes a nice balance between granularity and cardinality. For your use case, you might need a higher or lower level of precision.

Geo-temporal Flux examples

Let’s dig into a real-time analysis of USGS earthquake data. This initial example finds earthquakes that occurred in the last 24 hours within a region that roughly represents the west coast of the continental United States:

import "experimental/geo"

from(bucket: "Earthquake")
  |> range(start: -24h)
  |> filter(fn: (r) => r["_measurement"] == "geo")
  |> filter(fn: (r) => r["_field"] == "lat" or r["_field"] == "lon")
  |> geo.filterRows(
    region: {
      minLat: 32.245914,
      maxLat: 49.259394,
      minLon: -126.860380,
      maxLon: -114.715901
    }
  )

We defined a box with a maximum and minimum latitude and longitude, and the geo function makes sure to only provide us the list of earthquakes within those borders.

For those times when a simple box does not fit your use case, Flux provides circles and general polygons. For example, here I investigate the number of earthquakes in the last 7 days in an area roughly bounded by the five San Francisco Bay Area counties I regularly travel to (San Francisco, Marin, Alameda, Contra Costa, and San Mateo County):

import "experimental/geo"

from(bucket: "Earthquake")
  |> range(start: -7d)
  |> filter(fn: (r) => r["_measurement"] == "geo")
  |> filter(fn: (r) => r["_field"] == "lat" or r["_field"] == "lon")
  |> geo.filterRows(
    region: {
        points: [
        {lat: 38.296395, lon: -123.022551}, {lat: 38.320102, lon: -122.879729}, {lat: 38.190695, lon: -122.580351}, {lat: 38.060455, lon: -122.394375},
        {lat: 38.043152, lon: -121.751675}, {lat: 38.112338, lon: -121.581387}, {lat: 37.825442, lon: -121.536068}, {lat: 37.546042, lon: -121.552121},
        {lat: 37.482863, lon: -121.466977}, {lat: 37.451254, lon: -121.931149}, {lat: 37.454980, lon: -122.112390}, {lat: 37.214760, lon: -122.148095},
        {lat: 37.054918, lon: -122.406274}, {lat: 37.990496, lon: -123.027002}
        ]
    }
  )

Though, if you happen to live in the Bay Area, it’s probably best not to dwell too long on the results.

This last example illustrates an iterative query by first finding the strongest earthquake the USGS observed in the last 24 hours worldwide. Then, we look at the last 7 days and count how many earthquakes occurred within 200 km of that event:

import "experimental/geo"

max_eq = 
   from(bucket:"Earthquake")
       |> range(start: -24h)
       |> filter(fn: (r) => r._measurement == "geo")
       |> geo.toRows()
       |> group()
       |> max(column: "mag")
       |> findRecord(fn: (key) => true, idx: 0)

from(bucket:"Earthquake")
    |> range(start: -7d)
    |> geo.filterRows(region: {lat: max_eq.lat, lon: max_eq.lon, radius: 200.0}, strict: false)
    |> group()
    |> count(column: "mag")

What's next?

To explore more of InfluxDB’s emerging geo-temporal capabilities, check out Tim Hall’s InfluxDays roadmap presentation. Tim discusses more details about the work we’ve done on geo-temporal data acquisition, queries, and visualization. Also, I will continue publishing new blogs exploring Flux’s advanced geo-temporal capabilities.

Want to analyze your own data today? Check out Flux’s geo-temporal features in InfluxDB Cloud 2.0, InfluxDB Enterprise 1.8, and InfluxDB OSS 2.0.

As always, if you have questions or feedback, please let us know in our community forum or join us in Slack!

How to Expand Data Collection for InfluxDB with CloudFormation Templates

Nate Isley (InfluxData) — Tue, 23 Jun 2020 07:00:42 -0700

In a previous post, I demonstrated how to call InfluxDB APIs from AWS Lambda, but the setup is fairly manual and the results are not portable. Ideally, we as a community can expand and share ways to collect and process time series data. To that end, I want to share a CloudFormation template.

CloudFormation is AWS’ infrastructure as code service that lets you define almost any AWS component in a configuration file. The result is a service that provides a concrete, repeatable definition of your environment that can prove useful in many scenarios. For example, if someone accidentally deletes your Simple Queue Service topic or an EC2 instance, you can easily recreate them by running your CloudFormation template.

A geo-temporal CloudFormation template

CloudFormation templates create AWS stacks that wrap the AWS components defined in the template. AWS stacks are easy to create, update, recreate, and remove.

The CloudFormation template that I’ll walk through below pulls earthquake data from the United States Geological Survey (USGS) every hour and writes it to InfluxDB. I will not attempt to give you a full CloudFormation tutorial, but it’s helpful to understand a breakdown of the different sections if you’d like to use this template to create your own:

The first ~20 lines define variables that the template asks for when it's installed.
Lines 21?–120 handle a quirk of Lambda deployments that requires the Lambda assets to be in your region before deployment. As there is no elegant workaround, these 100 lines create an S3 bucket in your account in the region you are creating the stack and copies in the resources.
Lines 121?–132 define a role with basic permission to run the Lambda.
Lines 133?–144 define a Python library layer. Lambda layers are really handy at decoupling your functions from supporting libraries. In this layer, I packaged the Python HTTP library, a Python S2 Geometry library, and the InfluxDB Python client library.
Lines 146?–165 define the Lambda function. The Lambda is a short Python script that is zipped up in a file called geo_lambda.zip.
Lines 166?–188 define an event rule with permission to run the Lambda. The rule will run the Lambda every hour.

Deploying the Lambda Stack

Now that we have explored the template, let’s deploy it. First, log into your free AWS account and search for the CloudFormation service. Make sure you’re in the AWS region you want to deploy the Lambda to? — I have deployed more stacks in the incorrect region than I would like to admit. The following screenshot shows a search for CloudFormation while in the N. Virginia region (upper right):

Once you navigate to CloudFormation, click the Create Stack button. Select Template is ready and Amazon S3 URL options, and then copy and paste the following URL into the Amazon S3 URL field:

https://influxdata-lambda.s3.amazonaws.com/GeoLambda.yml

Click Next, and then name your stack. Provide the following InfluxDB details:

Your organization ID
Bucket ID of the bucket the Lambda writes to
Token with permission to write to the bucket
Your InfluxDB URL

There are also two other parameters (QSS3BucketName, QSS3KeyPrefix) with default values that you should not change. These parameters are helpers for asset copying.

On the Specify stack details page, leave the fields blank and then click Next. Select I acknowledge creating this stack will create resources, and then click Create stack.

After a few minutes, the stack deploys to your region. To navigate to your new Lambda, click Services -> Lambda. On the Lambda functions page, you should see your new Lambda listed there. The CopyZipsFunction is the helper copy function, and the GeoPythonLambda does the data collection and writing work:

GeoPythonLambda should run every hour based on the AWS Rule we set up, but you should test and confirm it works. To test, click GeoPythonLambda, and then click Test. Test requires an input definition, but this Lambda has no input requirements, so click through and save the default dataset.

Here is what you should expect after a successful test — a green Execution result: succeeded.

With the data points written, when you log into your InfluxDB UI, you’ll be able to explore the geolocation earthquake data.

What else?

As an aside, now that the layer asset with the Python client library is in your environment/region, you can easily create new Lambdas and experiment with the client. Create a new Python 3.8 Lambda and select Layers and Add a layer.

In Add layer to function, “my-layers” is an option under the list of runtime compatible layers:

Once added, import and use the client library in any of your Lambda code.

What's next?

This blog covered how to distribute a data collector on AWS via CloudFormation. If you have data collection needs and workloads that run on AWS, this template could jumpstart your AWS stack creation efforts.

Apart from a data collection discussion, this Lambda also directly supports an in-depth review of Flux’s new geo-temporal query capabilities. Now that you have the USGS earthquake data reporting in, you can explore using Flux to analyze your data over time and space.

As always, if you have questions or feedback, please reach out to me in our community forum or join us in Slack!

Extending InfluxDB with Serverless Functions

Nate Isley (InfluxData) — Thu, 04 Jun 2020 07:00:22 -0700

Data ingestion and data analysis are the yin and yang of a time series platform. There are many resources to help you ingest data. Typical ingestions are agent-based, imports via CSVs, using client libraries, or via third-party technologies. Once your time series data arrives, analysis completes the circle and often leads to additional data collection, and so on and so forth.

Native InfluxDB tools such as Tasks, Dashboards, and Flux typically provide the needed insight within the platform, but some use cases necessitate extending the platform. For those, InfluxDB’s unified API provides a complete set of integration endpoints. This extensibility is one of the things developers love about InfluxDB.

To illustrate, I’ve provided two simple examples implemented on Amazon Web Services’ (AWS) functions as a service offering - Lambda. Serverless functions such as Lambda are useful for almost any architecture because they are decoupled units of work that are relatively easy to connect.

The setup

My example incorporates a Flux Task that synthetically monitors the response time of a web application. InfluxDB gathers the data and acts as the backend analytics engine that the service processor uses to power its business processes.

I replicated the synthetic monitoring Task three times with different URLs and stored the results in a bucket with the name of the monitored company. For example, one of my tasks monitors Uber’s response time, and the response time is stored in a bucket called “Uber”. The end result? Three InfluxDB tasks capturing website performance in a uniform data set differentiated only by bucket name.

Using AWS Lambda to query InfluxDB

The first decision when creating a Lambda is to pick which language to use. To make it easier to walk through development, I used NodeJS. AWS’ NodeJS Integrated Development Environment support means the first example can be created entirely within the browser.

To follow along, sign up for a free AWS account and complete Lambda’s getting started steps. Create a hello world NodeJS Lambda and copy/paste Example 1 into the console. On the InfluxDB side, sign up for a free InfluxDB Cloud account, create the Flux task, and grab your org-id and token to insert into Example 1’s code:

var https = require('https');

exports.handler = function(event, context) {
    var queryForCount = "from(bucket: \"Pingpire\") |> range(start: -1h) |> filter(fn: (r) => r[\"_measurement\"] == \"PingService\") |> filter(fn: (r) => r[\"_field\"] == \"response_time_ms\") |> filter(fn: (r) => r[\"method\"] == \"GET\") |> count()"

    // An object of options to indicate where to post
    var post_options = {
        host: 'us-west-2-1.aws.cloud2.influxdata.com',
        path: '/api/v2/query?orgID=<your-ord-id>',
        method: 'POST',
        headers: {
            'Content-Type': 'application/vnd.flux',
            'Authorization': 'Token <your-user-token>'
        }
    };

    // Set up the request
    var post_req_uber = https.request(post_options, function(res) {
        res.on('data', function(chunk) {
            console.log('Response: ' + chunk);
            context.succeed();
        });
        res.on('error', function(e) {
            console.log("Got error: " + e.message);
            context.done(null, 'FAILURE');
        });
    });
    // post the data
    post_req_uber.write(queryForCount);
    post_req_uber.end();
}

Here is a screenshot of the AWS IDE output when I tested the Lambda in my browser:

This first example is a straightforward use of a NodeJS HTTPS POST request to demonstrate direct use of InfluxDB’s API. Using the API is powerful and simple, but many developers would rather use libraries instead of making HTTP calls. My second example invokes InfluxDB via an InfluxDB client library to illustrate this approach.

Setting up the second example takes a little more work as the base AWS NodeJS library is limited. To proceed, you will need the AWS command-line interface (CLI) to add node packages - follow installing/using the AWS CLI.

Once you have the CLI, update the available library/packages for your lambda function with

>npm install @influxdata/influxdb-client

in a local directory. You can zip that directory and use the CLI to import it to an AWS Lambda directly, or do what I did and use the NodeJS IDE’s “upload a zipfile” option under code entry type.

Once the Lambda has the integrated InfluxDB client library, copy/paste Example 2’s’ code:

const { InfluxDB } = require('@influxdata/influxdb-client')

const token = '<your-user-token>'
const org = '<your-org-id'

const client = new InfluxDB({ url: 'https://us-west-2-1.aws.cloud2.influxdata.com', token: token })
const queryApi = client.getQueryApi(org)

exports.handler = function(event, context, callback) {
  var customerJSON = JSON.parse(JSON.stringify(event));

  var customers = [];
  var x;
  for (x in customerJSON) {
    customers.push(customerJSON[x]);
  }

  var y = 0;
  for (var i = 0; i < customers.length; i++) {
    var query = `from(bucket: "${customers[i]}" ) |> range(start: -10m) |> filter(fn: (r) => r.method == "GET" ) |> count(column: "_value")`;
    console.log(query)

    queryApi.queryRows(query, {
      next(row, tableMeta) {
        const o = tableMeta.toObject(row);
        console.log(`Customer "${customers[y++]}" consumed ` + o._value + ` pings.`);
      },
      error(error) {
        console.error(error)
        console.log('\nFinished ERROR')
      },
      complete() {
        console.log('\nFinished SUCCESS')
      },
    })
  }
}

As you can see, using the client library is not drastically different from API access. You still need the token, org-id, and query, but native libraries almost always make developers more efficient.

To execute and test this example, recall that my InfluxDB has several tasks for monitoring company websites, and the results are differentiated by bucket name. My Lambda takes in a JSON list of companies and loops over the list to query InfluxDB for results. Here is my test JSON:

{
  "Company1": "Microsoft",
  "Company2": "Uber",
  "Company3": "Pingpire"
}

Here is a screenshot of the AWS IDE output for the second Lambda:

Exploring serverless integrations continues...

These initial examples demonstrate the ease with which one can get started extending InfluxDB with Lambda. Over the coming weeks, I will be expanding on this foundation to further explore using Serverless with InfluxDB.

As I mentioned, please try out the above examples today by signing up for free InfluxDB Cloud and AWS accounts. As always, if you have questions, feedback, or want to explore these specific examples please reach out in our community forum or via Slack!

Using Serverless Flux to Monitor Website Response Times

Nate Isley (InfluxData) — Wed, 08 Jan 2020 07:00:01 -0700

My holiday challenge to explain serverless InfluxDB to my family produced a useful Flux script anyone can put to work today. Before we dive into the code, let me outline the high-level approach to gathering and visualizing how long it takes a website to respond to an HTTP request:

Create a Flux script to measure a website's performance.
- Decide on the URL to monitor and assign it to theURLtoMonitor variable.
- Record the current time.
- Make an HTTP call to the website.
- Record the new current time.
- Calculate the website response time by subtracting b. from d.
- Save the URL monitored, the response time, and the HTTP status code into an InfluxDB bucket.
Create a task to run this Flux script every 5 minutes.
Build a dashboard to visualize the performance of the website.

The Flux script

import "strings"
import "csv"
import "experimental/http"
import "system"

differenceInNanosec = (earlierTime, laterTime) => {
	return uint(v: laterTime) - uint(v: earlierTime)
}

theURLtoMonitor = "https://v2.docs.influxdata.com/v2.0/"

timeBeforeCall = system.time()
response = http.get(url: theURLtoMonitor)
timeAfterCall = system.time()

responseTime = differenceInNanosec(earlierTime: timeBeforeCall, laterTime: timeAfterCall)

data = "#group,false,false,true,true
#datatype,string,long,string,string
#default,mean,,,
,result,table,service,response_code
,,0,http_ping,${string(v: response.statusCode)}"

csv.from(csv: data)
	|> map(fn: (r) => ({r with _time: now()}))
	|> map(fn: (r) =>
		({r with _measurement: "PingService", url: theURLtoMonitor, method: "GET"}))
	|> to(bucket: "<bucket_name>", orgID: "<orgID>", fieldFn: (r) =>
		({"response_time_ms":  uint(v: responseTime) / uint(v: 1000000)  }))

Deciphering the Flux script

After boilerplate imports @1-4, we have a custom function named differenceInNanosec @6. This is a helper function that converts time into integers for subtraction and helps keep straight which time measurement to subtract from which.

The bulk of the algorithm work comes next and is completed in just a few lines @10-16. @10 assigns the website URL to the variable. Lines @12-14 perform the actual HTTP request in between recording times before and after the call, and we use the times in differenceInNanosec to calculate the website’s response time @16.

After securing the response time, we need to save it to InfluxDB. However, there are a couple of hoops we must jump through. The complication is that our variables are simple data values, but Flux’s InfluxDB connector only writes Flux tables to InfluxDB.

Saving a Flux table sounds simple enough, but we need to create it first and as of today (Flux version 0.57), there are only two ways to create a Flux table. The first is from(bucket:...); Flux automatically creates a table when querying InfluxDB data. The second way to create a Flux table is via csv.from(csv:...).

This second option is our only viable option, and we put it to work @18-22 by constructing csv.from()’s raw input to build the csv table structure. The wrinkle @22 is that I have decided to use the Strings package to create a more dynamic table. You can see this dynamism in action with the use of the string replacement function ${string(v: response.statusCode)} to insert the HTTP status code.

Once the raw string is put together, we create a table object @24 using csv.from(). The table is close to ready for saving but needs a little more shaping to fulfill InfluxDB write requirements. @25-26 adds the required timestamp and _measurement name, and while not a strict requirement, I also add the url and http method as tags. At last, @29 we save the table into the target bucket with the response time in a response_time_ms field — fieldFn: (r) => ({"response_time_ms": uint(v: responseTime) / uint(v: 1000000) })).

(Note the conversion to milliseconds to make it easier for humans to understand the measurement.)

Creating the Flux task

A lot happened in the Flux script, but pulling together the code just gets us to the starting line. We want to gather the response time every 5 minutes, and that is the perfect job for a Task.

The first step in creating a task is to paste/write your code into a script editor and ensure your code successfully saves a singular data point. You can verify the above website response time Flux by copying and pasting the above code into the script editor and changing the URL, organization ID, and bucket name:

After confirming the script runs successfully, copy and paste your updated script into a task. To do this, go to the Tasks tab on the left and then select Create Task -> New Task in the upper right to open a blank task editor. Paste the code, name the task, and specify that it runs every 5 minutes with “5m”:

After saving, the task will run every 5 minutes to collect and store the response time in response_time_ns.

Build a dashboard to visualize response time

There are almost limitless ways to visualize the collected data, but the easiest is to go to the Data Explorer, find the response time metric, and use the upper right Save As button to create a new dashboard.

The result:

Tell us what you think!

Try out the above example today by signing up for a free Cloud 2 account or download the latest OSS release to run it locally. As always, if you have questions or feedback, please let us know in our community forum or join us in Slack!

Explaining InfluxData at Family Holidays

Nate Isley (InfluxData) — Fri, 13 Dec 2019 07:00:46 -0700

After trying to explain InfluxDB to family between hors d’oeuvres, I quickly realized I could not simply dive into Flux. I needed to show, not tell.

Over two blogs, I’ll dig into how I explained InfluxData for non-technical folks, a serverless example, and deep dive into serverless Flux code so you can wow your own in-laws.

Deciphering techno-speak

This Thanksgiving, much of our family was in town to visit and fill up on turkey and ribs. Everyone was curious about InfluxData, but conveying InfluxDB to anyone not working in technology is no small feat.

While a serverless time series database platform might roll off the tongue, I would be more successful describing the weather in Basque than expecting anyone who does not use databases to comprehend techno-marketing-speak. I tried different ways to share InfluxDB with my family and friends, and by Sunday, I settled on:

“Folks measure almost everything constantly now, like ocean tides, house temperatures, or internet speeds. They’re measuring so much, so often that they have found they need specialized time-based collection and analysis tools. InfluxDB is an open source software project to capture and analyze this massive amount of data. InfluxData’s newest iteration includes all the tools already built to help handle this huge time-based measurement load, and now we’ve added support for running serverless code. The platform does the heavy lifting of collecting data, storing it, and providing computing power to analyze the data so software builders can focus on implementing solutions.”

This elicited a few head nods, but I could tell it was still fuzzy. Not to be deterred, I retreated to my desk to create a concrete Flux example. I settled on a use case I thought most folks could relate to — how long it takes to load your favorite website.

I started by tinkering with the Flux REPL and proceeded to my InfluxDB Cloud script builder. I ran into a few snags, but after a couple of workarounds (covered in the deep dive), I managed to distill the data into a suitable dataset.

With my Flux program generating and saving a proper dataset every time it was invoked, I put InfluxDB Cloud’s serverless capabilities to work. By saving the script as a task running every minute, my data started to flow in. From there, it was a simple matter to create and share a dashboard:

A concrete example did the trick and a lightbulb turned on in my brother’s head - “Ah, I can gather and process any information over time with a few lines of Flux.”

Exactly! - of course, now I’m stuck creating Flux to help him gather the price of Brent crude oil…

Unlock Flux's power for your use case!

Flux is a powerful language for working with data and it’s available now on top of our time series platform. It is unlocking exciting new time series use cases and allowing developers to work with data where it resides both within InfluxDB and other data sources such as mySQL, MariaDB, and Postgres. Pick up InfluxDB’s OSS or sign up for InfluxDB Cloud to start exploring. If you are looking for a starting point for Flux, I highly recommend Scott Anderson’s outstanding Flux training.

As always, if you have questions or feature requests for Flux, head on over to Flux’s community forum or join us in Slack!

Using Secret Stores to Secure Flux Access

Nate Isley (InfluxData) — Tue, 22 Oct 2019 07:00:14 -0700

Multi-data source Flux opens up endless variations in time series analysis, but as with all data access, new communications open potential attack vectors for shady actors. To secure your sensitive credentials, use InfluxDB’s integrated secret store and Flux’s new secrets library.

Keep the keys to the kingdom secret

If you follow my multi-data source Flux journey you have seen some exciting possibilities introduced by combining multiple data stores and time series analysis. However, if you look closely at my SQL Flux scripts, you may notice something unsettling — the Flux to connect external data stores avoids usernames and passwords and is in plaintext.

Simple data access methods when developing are acceptable, but when it comes to sensitive production data stores we all must be vigilant in protecting our information. Flux assists in this protection by integrating with secret stores to help you keep the secrets to your kingdom secret.

InfluxDB Cloud 2's secret store

Our InfluxDB Cloud 2 service is integrated with the industry-leading Vault secret store to provide every customer with a secure secret store. Users can store secrets, list their keys, and delete secrets using our fully documented API. As an example, these curl commands use an organization’s ID and all access token to insert and retrieve secrets:

Putting a secret into your secret store:

>curl --data '{"myuser_password": "my_password_value"}' --request PATCH --url https://us-west-2-1.aws.cloud2.influxdata.com/api/v2/orgs/<My_Org_ID>/secrets --header 'Authorization: Token <All_Access_Token>'

Listing out all the secret keys in Organization

>curl --request GET --url https://us-west-2-1.aws.cloud2.influxdata.com/api/v2/orgs/<My_Org_ID>/secrets --header 'Authorization: Token <All_Access_Token>'

Using secrets in Flux

My SQL.to() blog used this bit of Flux code to perform SQL insert magic:

// Store result in Postgres
|> sql.to(driverName: "postgres", dataSourceName: "postgresql://localhost?sslmode=disable", table: "adjustment_billing")

Locally, that works, but a production example would use a user/password to write to a remote database with Flux:

// Store result in remote Postgres database
	|> sql.to(driverName: "postgres", dataSourceName: "postgresql://myuser:mypass@127.0.0.1:5432/nisley?sslmode=disable", table: "sensor_billing")

Without secrets, usernames and passwords are hardcoded in plaintext. Fortunately, we can leverage Cloud 2’s Vault to keep sensitive information safe. Here, I revamp the full fish tank vendor billing script to use the secret store for the actual password stored in myuser_password:

import "sql"
import "influxdata/influxdb/secrets"

myuser_secret = secrets.get(key: "myuser_password")

from(bucket: "tank-health")
	|> range(start: -1w)
	|> filter(fn: (r) => r._measurement == "water_sensor" and r.type == "adjustment")
	|> group(columns: ["type", "device_id"], mode:"by")
|> count(column: "change")
|> rename(columns: {change: "adjustment_actions"})
|> keep(columns: ["device_id", "adjustment_actions"])
|> sql.to(driverName: "postgres", dataSourceName: "postgresql://myuser:" + myuser_secret + "@127.0.0.1:5432/nisley?sslmode=disable", table: "adjustment_billing")

InfluxDB's common API and OSS

If you are developing on-premise with OSS instead of Cloud 2, don’t fret! One of our core tenets is to develop one common API across all products and services. The API examples above work with any local InfluxDB 2 OSS URLs:

Get secrets from local OSS build:

>curl --request GET --url https://localhost:9999/api/v2/orgs/<My_Org_ID>/secrets --header 'Authorization: Token <All_Access_Token>'

Note that InfluxDB 2 OSS’s underlying key-value store is currently BoltDB. BoltDB is not focused on being a secret key-value store, so be mindful that if folks have access to the machine, values should not be considered secure.

Come explore Flux with us

Join us in exploring the new Secret packages in Flux by signing up for InfluxDB Cloud 2, or use it locally with the latest OSS InfluxDB 2 alpha. As always, if you have questions or feature requests for Flux, head on over to Flux’s community forum and let us know!

Using Flux to Write Data Out to SQL Data Stores

Nate Isley (InfluxData) — Wed, 28 Aug 2019 11:22:42 -0700

As discussed in multi-data source Flux, Flux can now get data out of RDBMs with SQL.from() and use it to perform deeper analysis. In many cases, results from crunching data lend themselves naturally to time series storage. In others, the results are more appropriately leveraged on different data stores. In order to fulfill Flux’s general-purpose analysis engine vision, data must flow both into and out of InfluxDB.

To that end, InfluxDB 2.0 Alpha 15 introduced SQL.to(). SQL.to() mirrors Flux’s ability to pull data out of a SQL store by enabling writes to MySQL, MariaDB, and Postgres. Below, I outline using SQL.to() and walk through an example of counting events and storing them into Postgres.

Tables make the world go round

Once I understood that Flux results are truly just tables of data, I wondered how close a Flux result table was to an RDBMs table. It turns out that they are close enough that Flux can leverage their kinship to make SQL writes exceedingly elegant.

Once you shape and name your data to conform to the destination table definition, you simply pipe forward your data to SQL.to(). Let’s build up a use case for a new billing system to demonstrate this new functionality.

Fishtank health vendor - a new business model

My favorite fictional fish tank health vendor notices a large variation in the cost of supporting different customers. After studying their data closely, they learn costs are almost entirely driven by the number of corrective actions their automated water system performs. They decide to adjust their subscription model to one that charges customers for the number of weekly corrective events.

Luckily, every time the devices deployed into their customers’ fish tanks take corrective action, like adjusting pH or nitrate levels, the devices send an event to InfluxDB. Events are similar to the stream of sensor data, except for being emitted on demand instead of every X seconds. These on-demand events are also known as “irregular time series” because they can occur at any time.

The specific format and content of events can be anything you want, but let’s assume the events look like these two examples below (displayed in line protocol):

# Adjustment events - note timestamp will be added by InfluxDB at arrival time
water_sensor,type=adjustment,adjust_what=nitrates,change=up,device_id=TLM0101 triggering_value=5
water_sensor,type=adjustment,adjust_what=pH,change=down,device_id=TLM0102 triggering_value=9

Why use a SQL data store?

The events are already in InfluxDB. Why introduce a SQL data store? There are several characteristics of a time series database that, generally speaking, do not make them an ideal foundation for a billing system.

InfluxDB is designed to flexibly adjust to large volumes of data that change quite often, either due to endpoint changes or data variability because clients can do things like schema at write. Often, users pour tens of millions of data points in very quickly but then downsample or delete most of them after a few weeks. This flexibility is great when you have, say, millions of different sensors or thousands of servers you constantly upgrade.

In contrast, anyone trying to create a billing system wants to keep many years of stable customer billing data. While capturing the raw data at scale makes sense to store within a time series database like InfluxDB, the aggregated and processed data represents a “finished product” that may make more sense to store with other customer data such as billing address and telephone number. A relational database designed for stable, infrequent table changes could be a much better foundation for a billing system.

Come on already, crunch the numbers!

To collect the data for the new subscription billing model, our fish tank vendor decides to create a simple weekly task to count adjustments for each device and save them to their Postgres database. (Note: this naive scenario will not scale — at some large count of devices, both Postgres and InfluxDB will bottleneck. A more comprehensive approach might be to create a task per customer.)

The first step for storing the adjustment count is to create a table in Postgres. Here is the Postgres SQL command I used to create a simple example table:

CREATE TABLE adjustment_billing 
     (billing_record SERIAL PRIMARY KEY,  device_id VARCHAR NOT NULL, 
             adjustment_actions integer,  created_on TIMESTAMP);

Armed with a table to store results, the vendor begins to put together the pieces of the Flux script. We start by getting the last week of data from the tank-health bucket, filtering in only the adjustment events, and grouping the data:

import "sql"

adjustmentActions = from(bucket: "tank-health")
  |> range(start: -1w)
  |> filter(fn: (r) => r._measurement == "water_sensor" and r.type == "adjustment")
  |> group(columns: ["type", "device_id"], mode:"by")

The result of grouping by type and device IDs is a table for every device. To get a count of adjustments for each device (table), we pipe forward the result to count the number of rows with a value in the change column:

// Count the number of changes for each device
  |> count(column: "change")

Note that at this point, only the columns called out by grouping and the count are retained. (Technically, _start and _stop times exist, but they are dropped in the next step).

This result is very close to the format of the SQL billing table we created, but it doesn’t quite line up and a final adjustment is required.

// Adjust the table to match the SQL destination schema/format
  |> rename(columns: {change: "adjustment_actions"})
  |> keep(columns: ["device_id", "adjustment_actions"])

Now, the data columns perfectly match the billing table, and the Flux script can simply store the result.

// Store result in Postgres
|> sql.to(driverName: "postgres", dataSourceName: "postgresql://localhost?sslmode=disable", table: "adjustment_billing")

After the task is saved and set to run weekly, the adjustment event counts for every device will be stored in Postgres. If you query Postgres directly with a “SELECT * from adjustment_billing” after a task runs, you can see the fruit of all your Flux labors.

Get started with Flux today

Hopefully, this example of using Flux to write data to a data store other than InfluxDB illustrates the power and flexibility of multi-data source Flux. I am excited to explore all the different possibilities multi-data source Flux is sure to open up over the next few months.

Come join me in exploring Flux by signing up for InfluxDB Cloud 2 or by downloading the latest OSS alpha. As always, if you have questions or feature requests for Flux, head on over to Flux’s community forum and let us know!

Use Flux to Group, Shape, and Analyze Your Time Series Data

Nate Isley (InfluxData) — Mon, 29 Jul 2019 08:00:39 -0700

Flux is a programming language designed from the ground up for time series analysis. Traditionally, grouping, shaping, and performing mathematical operations across large dynamic time series datasets is cumbersome. Flux’s goal is to make it working with these datasets much more elegant.

Several new exciting Flux features were added to the InfluxDB 2 OSS Alpha that compelled me to jump in with two feet to see how they would work in practice. My initial foray was promising, but something wasn’t quite right.

My key insight to understanding Flux

As I researched how to crunch data across multiple data stores, I quickly realized I was missing something fundamental. I kept getting unexpected answers and many of my graphs looked off.

Looking for answers, I reached out to Adam Anthony on the Flux team and he walked me step- by-step through a more complex use case. After a few steps, it clicked — the key part I had to pay close attention to was table results.

Tables as data

The first few examples you encounter in Flux seem very straightforward. This is a typical first query you might see on system metrics collected by a local telegraf agent:

from(bucket:"my-bucket")
  |> range(start: -10m)
  |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system" and r.cpu != "cpu-total" )

This Flux query pulls all of the data that a Telegraf agent wrote into my-bucket in the prior ten minutes and filters it down to cpu system usage metrics excluding cpu-total values.

When I saw the result of this query in my InfluxDB Cloud 2 UI for the first time, I jumped to the conclusion that Flux simply returned four arrays — each with a litany of numbers. Turns out this initial impression was very wrong and the source of my missteps when trying to solve more complex scenarios.

After Adam’s lessons, I realized that using and understanding Flux depended on understanding each part of the Flux results. Flux results are tables, and at times, results are a rather extensive set of tables. I learned that the most useful tool to lean on when stepping through Flux analysis is not the graphing tool, but the UI toggle to View Raw Data.

Raw view is exactly that — the complete raw output from Flux. When I first looked at raw view, I thought perhaps the UI was adding things like labels, timestamps, or expanded descriptions. In fact, the UI adds nothing to the result — every label, row, and column here is straight from the Flux response.

Digging into the response, the easiest thing to overlook is the table column — the 3rd column in this example. The UI above is scrolled to display data from two tables (table 0 and table 1). We can see the above system cpu Flux query did not return four arrays at all, but rather four tables — one table for each of the four CPUs Telegraf is monitoring.

As you manipulate, analyze, and shape your data, if you keep the fact that every in-between calculation and result is a table at the forefront of your mind, you will find yourself much closer to the answers you seek. The final example below walks through a more complex use case to demonstrate stepping towards an answer by focusing on tables.

Revisiting the fish tank health vendor

Let’s continue the use case of a fish tank health vendor that stored IoT metadata in Postgres and sensor data in InfluxDB. In that example, the vendor wanted their customer success reps to know the pH balances of all the customer’s fish tanks. After a join, the table result looked like:

This result is simple enough for a customer success rep to read and digest, but add a couple of dozen tanks and sensors and reps will be swimming in data. To make it simpler, management realized what reps really need to know is the location of the fish tanks with a pH higher than 8.

Luckily, they are using Flux and can use the awesome built-in pipe forward operator to simply push the initial table result above into additional calculations to find the highest average pH.

//  group by location so that we can analyze each location independently of the others. (creates 3 tables out of 1)
|> group(columns: ["customer_location"])

// find the mean value inside of each table
 |> mean(column: "_value")

// grouping by no columns puts all of the tables together into one table
|> group(columns: [])

// filter out all rows of data below a value of 8
|> filter(fn: (r) => r._value >  8)

// sort the list to ensure the highest priority fish tanks are at the top of the list
|> sort(columns: ["_value"], desc: true)

After crunching and filtering all the data, the customer success rep should have a manageable list of fish tanks to focus on. In this case, at the top of the list is the Lobby fish tank, which looks like it needs some serious attention.

Flux's future is bright

Flux is starting to hit its stride as many powerful features such as conditionals and multi-data stores are released. Now is a great time to jump in and have a close look at Flux if you are tackling any time series analysis use cases. Easily get started by signing up for InfluxDB Cloud 2, or by downloading the latest OSS alpha.

As always, if you have questions or feature requests for Flux, head on over to Flux’s community forum and let us know.

Multi-Data Source Flux Introduced in InfluxDB 2 Alpha 14

Nate Isley (InfluxData) — Thu, 18 Jul 2019 07:00:31 -0700

As our co-founder Paul wrote extensively in InfluxDB 2.0’s Alpha Release announcement, we believe Flux will play an important role in combining time series data with many different data sources. Flux’s first official multi-data store step is InfluxDB 2.0 Alpha 14’s from SQL release. This Flux addition is significant given the increasing number of data stores and the need to cater to them.

It's a multi-data store world

Years ago, as a consultant for Wily Technology during the nascent APM industry explosion, I dealt with hundreds of applications in varying states of distress. Time and again, I would see applications ignoring the nature of their particular data, which invariably led to slowdowns and errors from data stores that were abused in ways they simply were not suited for.

As different application datasets scale, they are imbued with characteristics that make them suitable for specific data store handlers. Working with, storing, and processing each dataset with tools and storage mechanisms built specifically for them results in well-built systems that are maintainable and scale with growth.

Here are some examples of different datasets and data store types built for them:

Relational data like an e-commerce store's inventory, customer orders, and customer data and their relationships are best handled by an RDBMS system built to model that relationship.
Asset information related to IoT devices is finding great success with specialized data stores like IBM's Maximo and SAP's Intelligent Asset Management. Asset metadata is available via an exposed integration API from these applications.
Dependency data such as business processes, integrated applications, and underlying network topology is best handled by a [graph database](https://www.influxdata.com/graph-database/)
Time series data such as the millions of streaming data points from stacks, sensors and systems are best handled by a time series database.

In the distant past, some argued using different data stores put too much strain on programmers to learn different data tools. Before today’s widespread and uniform API use that may have been true. But nowadays, developers are quite comfortable using data source drivers and APIs to pull data from different places to mold them together.

Relying on developers to shape and crunch data works great for purpose-built custom software solving specific problems. But there are never enough developers to go around, and general-purpose data analytics and visualization tools have proven their value for many analysts and data scientists. Single data source analytic tools work at times, but analysts often want to merge data from different data stores and frequently find pulling data sources together is a monumental task.

Providing a tool to flexibly merge together data sources and analyze them across time is one of Flux’s primary use cases. Open source Flux is meant to empower every query and visualization tool so that they may bring together related data sets to generate insights using a common, powerful and unified language.

Using time series data with data from SQL data stores

Flux’s first implementation of a function that can extract data out of non-InfluxDB data stores is InfluxDB 2.0 Alpha 14’s SQL.from() function. Let’s dive into a specific IoT use case to explore this function.

IoT asset data store

Recently, I have had several InfluxDB deep dives with customers using InfluxDB in IoT use cases. These IoT practitioners have tens of thousands of devices and sensors streaming all the data they capture into InfluxDB, and dealing with all that data presents some unique challenges.

Without a way to connect different data stores, many IoT practitioners reuse all their metadata as tags on their time series metrics so they can crunch metadata-based time series analysis such as:

“What is the error rate reported over the last day for each device type?”

Of course, prior to SQL.from(), the only way to have InfluxDB answer that question is to add device type as a tag to every time-series metric coming from their sensors.

This works ok but has issues reacting to changes in metadata sets over time. For example, if an IoT device maker initially tags time series metrics with a certain security classification but the classification changes, only new data shows up when querying with the new classification as the old data has the old value in the tag.

A better solution is a clean separation of data by keeping metadata in a relational store and time-series data in InfluxDB. The key to making a two data store solution work is a Flux language that can seamlessly query both time series data and relational data.

Fishtank health service vendor

To flesh out an example, let's suppose there is a vendor that provides a service for automatically monitoring and adjusting customer fish tank water health. This vendor leases thousands of IoT devices that monitor many aspects of fishtank water and streams their data to InfluxDB. In this example, instead of tagging the streaming metrics with all the metadata, the vendor tags every metric with only the unique device ID the data originates from. The ID ties the device's metrics to sensor metadata that might include information such as the device type, purchaser's customer id, installation street city and state, device revision number, device name, etc. Our fishtank vendor has implemented a homegrown asset tracker in a MySQL database to keep track of all this relational information. Many of this vendor's customers are large corporate customers with dozens of fish tanks. Whenever a customer calls, it's very important for their customer success representative to know the pH balance of their customer's fish tanks over the last hour and where they are located. Here is what a Flux script might look like for finding one customer's fishtank pH balances:

import "sql"

phBalances = from(bucket: "PH_Balances")
  |> range(start: now-60)
  |> filter(fn: (r) => r._measurement == "water_sensor" )

deviceIds = sql.from(driverName: "mysql", dataSourceName: "username:password@tcp(host)/dbname", query: "SELECT customer_name, customer_location, device_id AS device FROM DeviceTable WHERE customer_name = 'InfluxData'")

join(tables: {t1: phBalances, t2: deviceIds}, on: ["device"])
	|> aggregateWindow(every: v.windowPeriod, fn: mean)
 	 |> yield(name: "mean")

This result of this script is a table that might look like this: The data in green is from MySQL, the orange data is from InfluxDB, and the blue data is the joined data in both data stores. Several things to note from this simple example: First, this is a very naive example as the first query that builds the phBalances table pulls the entire data set from InfluxDB. There is an implicit dropping of data when the two data tables join on device id. However, until the join completes, the whole dataset is in query memory and it is easy to imagine tens of millions of datapoints overwhelming all available memory. Second, there is no special need for a join per se. What we are really trying to do is filter down the pH Balances data set to a subset of metrics I am interested in. The Flux team has several initiatives underway that simplify these kinds of queries so I can directly use the SQL result in a filter. Lastly, the example Flux script is all low-level SQL and Flux. It requires a user to piece together their own rather complete prior knowledge of the data stores. Flux's long-term plan is to deliver libraries and tooling that could ease low-level knowledge prerequisites and provide a more approachable experience; for example, ORM tooling or data store visualizations to help users identify, merge, and shape their data and analysis.

The multi-data store road ahead

Flux has entered an exciting new chapter as a multi-data store time-series-focused programming language with the debut of SQL.from(). This initial SQL release enables pulling data from MySQL and Postgres, but we have a long list of data stores we would like to pull data from and write data to. The list of possible connectors on the Flux team's radar is long and dynamic. High on the list are SQLite, MS SQL Server, connectors for our open source Google Cloud Compute partners and, of course, for Google's data sources as well such as BigQuery. But we want to hear from you — what top data sources do you want Flux to connect to? Head over to Flux's community forum and let us know!