InfluxData Blog - Sonia Gupta

Using the Telegraf Elasticsearch Input Plugin

Sonia Gupta (InfluxData) — Thu, 30 May 2019 10:32:16 -0700

The Telegraf Elasticsearch plugin is really easy to use, and it can show you a lot of useful information. To begin, let’s get the InfluxDB sandbox up and running as well as an Elasticsearch node that we can populate with some data using Kibana.

Installing the InfluxDB Sandbox

For this tutorial I’ve opted to use the InfluxDB Sandbox, which is really easy to get running. Just clone the repository and run the ./sandbox up command and you’ve got all four components of the TICK stack running in their own containers. Follow the instructions here to clone the repository and get it running.

Installing Elasticsearch and Kibana

Elasticsearch requires at least Java 8 to be running on your machine, so I first ensured that was installed. Then, since I’m running Mac OS, I used Homebrew to install Elasticsearch and Kibana on my machine: brew install elasticsearch and brew install kibana. As of the time of this article, using Homebrew will install Elasticsearch version 6.7 and Kibana version 6.7. If you want to install the most recent version of Elasticsearch, you will need to follow the instructions available here.

If you used Homebrew to install both Elasticsearch and Kibana, you can start them up by entering elasticsearch in one terminal tab and entering kibana in a second terminal tab. You will keep them running through the duration of setting up the Elasticsearch plugin.

Then navigate to localhost:5601 and click on “Dev Tools” in the left sidebar and you should see something like this:

This is the Kibana UI from which you can make requests to your Elasticsearch node.

I recommend playing around with your Elasticsearch node and Kibana using the tutorial here. It will show you how to populate your node with data and check cluster health (a cluster consists of one or more nodes, so you also have a cluster). For example, running

PUT /user/_doc/1?pretty
{
  "name": "Malala Yousafzai"
}

in Kibana will create a user index and insert a user with an id of 1. And running GET /user/_doc/1?pretty in Kibana will fetch that user and return

{
  "_index" : "user",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 2,
  "found" : true,
  "_source" : {
    "name" : "Malala Yousafzai"
  }
}

Monitoring Elasticsearch

Now that you’ve got some indexes populated with data in your Elasticsearch node, it’s time to monitor the node! First, if you’ve got the sandbox running, shut it down with ./sandbox down.

Then open up the sandbox code in your favorite editor and navigate to telegraf/telegraf.conf. This is where you will find the various input and output plugins that come configured with Telegraf, and also where you will add the configuration for the Elasticsearch plugin.

Copy and paste the configuration from the Elasticsearch plugin README at the bottom of your telegraf.conf file, like so:

Note that I have made a change to the configuration. Instead of servers = ["http://localhost:9200"] I have servers = ["http://host.docker.internal:9200"]. This is because we are running our instance of Telegraf in a docker container and we need it to be able to communicate with the Elasticsearch node that we are running locally, outside of that docker container.

Save the file and spin up the sandbox again with ./sandbox up and navigate to the Chronograf tab that opens up at localhost:8888. Click “Explore” in the left-hand navigation bar. Then click “Flux” to switch to using Flux for queries.

Then, under “SCHEMA,” click “telegraf,” then “MEASUREMENTS,” then “elasticsearch_indices,” then “Add Filter” next to “docs_count.” This will construct a Flux query in the SCRIPT block:

Your query should look like this:

from(bucket: "telegraf")
  |> range(start: dashboardTime)
  |> filter(fn: (r) => r._measurement == "elasticsearch_indices" and r._field == "docs_count")

Click “Visualization” at the top of the page and select “Single Stat.” Then click “Queries” to get back to the query block, and click “Run Script.” You should see in the top block on the page the number of documents (at least one) that you created using the Elasticsearch tutorial. This is the kind of data that you can see with the Elasticsearch plugin.

There are several other metrics available to you, and you can actually visualize some of those metrics by pulling in the pre-built dashboard. To do so, click “Configuration” in the left hand navigation bar and you will see your InfluxDB connection. Click on the name of that connection:

Then click “Update Connection” in the window that pops up:

Then scroll until you see “Elasticsearch,” and click on that box:

Click “Create 1 Dashboard.” Click “Skip” when you are asked about your Kapacitor Connection then click “Finish.” Then click “Dashboards” in the left-hand navigation bar, and you should see your Elasticsearch dashboard in the list. Clicking on that will take you to the dashboard itself:

And that’s it! You’ve now successfully installed Elasticsearch and are able to monitor its performance using the Telegraf plugin as well as use the included dashboard to visualize some critical Elasticsearch metrics.

Getting Started with InfluxDB 2.0: Scraping Metrics, Running Telegraf, Querying Data, and Writing Data

Sonia Gupta (InfluxData) — Mon, 06 May 2019 10:12:48 -0700

Now that we’ve released InfluxDB 2.0 Alpha, you’re probably curious to try it but might not know where to start. This post will guide you through the process of getting it running locally on your machine. By the end of this post you’ll have a good idea of some of the new features that distinguish 2.0 from 1.x, and you’ll be able to load data and navigate the UI with ease.

Scraping metrics with InfluxDB 2.0

First, navigate to the Getting Started docs and follow the instructions to either download the Alpha or run a Docker container with the Alpha. You can run 2.0 natively on Linux or Mac, but if you’re on Windows you will need to use the Docker container. To run the Docker container, you will need to have Docker installed on your machine. For the purposes of this tutorial I have opted to download the Alpha and run it natively.

Once you’ve got InfluxDB running, either by running the influxd command or running the Docker container, navigate to http://localhost:9999 and you should be greeted with this page:

Clicking “Get Started” will take you to this page:

I recommend naming your organization “InfluxData” so that you can follow along with what I’m going to show you. Organizations are an umbrella under which your data and queries are organized in 2.0. A bucket is the equivalent of a database. You can choose whatever bucket name, username, and password you like. Clicking “Continue” will take you to this page:

The “Quick Start” option configures a data scraper. The data scraper will pull metrics from your local instance of InfluxDB 2.0. It’s the fastest way to get up and running with some data to play with. “Advanced” brings you to the Telegraf agent config area. “Configure Later” brings you to the homepage, without anything created. Click “Quick Start” and you will be taken to the following page:

This is the homepage from which you can access every part of your InfluxDB 2.0 instance. You can access your organizations, dashboards, and documentation from here.

Click “Data Explorer” in the left sidebar, and you’ll be taken to this view:

What you’re seeing in the second column on the bottom half are the measurements for the data collected from the scraper that was automatically initialized when you clicked “Quick Start.” That scraper collects data about the InfluxDB 2.0 instance by pulling metrics from localhost:9999/metrics, which you can visit to see those metrics in Prometheus data format.

We’re just going to pick an arbitrary data series. Click “boltdb_reads_total” which will open a block for the fields associated with that measurement. Under the “_field” heading, click “counter,” then click “Submit:”

Congratulations! You have just constructed your first query using the query builder, and what you’re now seeing is a visualization of that query result.

But if you want to see an even better series of visualizations, click on “Dashboards” in the sidebar and you’ll see “Local Metrics,” a predefined dashboard created from the metrics being collected by the scraper. Click on it and you should see something like this:

Running Telegraf with InfluxDB 2.0

Now we’re going to create a bucket, set up the Telegraf agent, and view dashboards displaying data collected by the agent. Click “Settings” in the left sidebar and click “Buckets.” Then click “Create Bucket” and make a new bucket which will store all the time series data collected by the Telegraf agent:

Now click “admin” in the left sidebar and then click “Configure a Data Collector”, which will take you to this page:

Click “Create Configuration” and you will be taken to this page, where you will see a number of collections of plugins that are currently available in 2.0. More collections will be added as 2.0 is further developed. Select the “System” collection of plugins (make sure you click the “Bucket” dropdown on the left side and select the bucket that you just created):

The “System” collection of plugins collects metrics from your Linux machine and is one of the best ways to start playing around with data so you can see how InfluxDB 2.0 works. Click “Continue” and you’ll be taken to this page:

Go ahead and name your configuration “System” so you can follow along with this tutorial. You can keep the description blank. Click “Create and Verify” and you’ll be taken to this page:

If you have not installed Telegraf, now is the time to do so. You can click on the link to visit the downloads page and install Telegraf following the instructions there depending on your system and whether or not you want to use Docker.

Once you have Telegraf installed, you will want to set an environment variable with your telegraf export token, as explained in Step 2. The export token is a secure way to provide another application with access to write data to your instance. Open a new terminal tab, and then copy and paste the command in Step 2 into the prompt for that tab.

In that same tab (you should have two tabs open now, the first one is where the Alpha is running either from your download or in a Docker container) you will run the command in Step 3.

This will start the Telegraf agent which will collect metrics from your computer using the “System” collection of input plugins that we selected earlier.

Click “Listen for Data” and if everything is wired up correctly, you will get a “Connection Found!” notification and see this page:

Click “Finish” and you will be taken to this page:

Here you can see the Telegraf configuration that you just made. Clicking on the name of the configuration (“System”) will show you the configuration file:

If you scroll down to the bottom of the configuration file, you will see the collection of input plugins we added to our config file, one of which is [[inputs.cpu]]. You can also download the config here to make changes to it and add other input and output plugins manually. You could then run Telegraf from the downloaded config file instead of through the API. You can close the config by clicking the “x” in the upper right corner of the popup.

Now for some fun! The “System” collection of plugins includes a predefined dashboard that you can now see by clicking “Dashboards”:

Click “System” and you’ll be taken to a page that looks something like this:

Here you can see the results of the “System” collection of plugins having collected various metrics about your machine.

Querying data with InfluxDB 2.0

Next, click “Data Explorer” in the left sidebar:

You are now in the query builder. Under “_measurement” select “cpu.” This should pop up another block which contains the fields associated with that measurement, like so:

Click “usage_idle” and another block will pop up which contains the tag key “cpu” and all of its possible values:

At any point in this process of building out the query you can click “submit” and see the results of the query you have built:

So what is the query we’ve actually built here? We can see it by clicking “script editor.”

Doing so will take us to this view:

What you’re seeing here is the underlying Flux query that was built in the query builder and that was run when you clicked “submit.”

Writing data with InfluxDB 2.0

Now we can try manually writing some data to the database (as opposed to collecting it through a scraper or the Telegraf agent) and querying that data using the UI.

Under “Configuration” in the sidebar, click “Buckets.” You should see this view, with the buckets that we created earlier.

Click “Create Bucket” in the upper right-hand corner.

Name your bucket “trains” and keep the retention policy as it is (leave “How often to clear data?” set to “Never”). Now click “Add Data” for your “trains” bucket.

Select “Line Protocol” from the dropdown, and you’ll see this view:

Select “Enter Manually” and paste this line into the window:

passengers_in_car,train=a,car=one,driver=gupta count=125

Now click “Continue.” You’ve just written your first point to the database!

Let’s query the database to retrieve that data point. Click on “Data Explorer” in the left sidebar. From the “Bucket” dropdown, select “trains.”

You should now be able to see “passengers_in_car” as a measurement. Click on it, and another block will pop up, like so:

At this point you can click “Submit” and you will be shown a graph of the single data point we just wrote. If you click on the dropdown that says “Graph” in the upper right corner, you’ll find an option to see your data as a Table. Select that option. You should now see this view:

You can now look at the underlying query by clicking “Script Editor:”

This shows you the Flux query that was built for you as you created your query in the query builder. You can modify the query from this view.

We’re at the end of this tutorial, so if you’d like to shut down the influxd daemon and the Telegraf agent you can exit them using ctrl + c on a Mac.

InfluxDB 2.0 walkthrough summary

You’ve now seen a few of the major features of InfluxDB 2.0 including scraping metrics, running Telegraf, querying data, and writing data. There are many other features that we haven’t covered here, so you can look forward to seeing more posts on those features in the future. Please note that InfluxDB 2.0 is not yet production-ready, but we welcome your feedback and encourage you to experiment with the Alpha as new versions are released. Have fun!

InfluxDB: Reorganizing Data with the Pivot Function in Flux

Sonia Gupta (InfluxData) — Tue, 05 Feb 2019 10:44:58 -0700

If you’ve been following along with my Flux journey as I implement Flux queries with Rails and GraphQL, you’ll know that I’ve been using train data when implementing various functions in Flux. In order for the GraphQL gem in my Rails project to work properly (it calls a method on an object for each of the attributes), I need to create a Ruby object for each train data point that has each of my tags and fields separated out into their own attributes, like so:

class Train
   attr_reader :time, :measurement, :driver, :location, :train, :outdoor_temp, :speed, :track_temp

   def initialize(train)
      @time = train["_time"]
      @measurement = train["_measurement"]
      @driver = train["driver"]
      @location = train["location"]
      @train = train["train"]
      @outdoor_temp = train["outdoor_temp"]
      @speed = train["speed"]
      @track_temp = train["track_temp"]
   end

end

Without the right Flux function, when I query my database to return train data points, my tags will be separated out into their own columns, but my fields will be aggregated into a single column, _field. My data won’t be shaped the way I want it to be with a base query, as you can see below with the following query, and the result it returns:

from(bucket: "trains")
    |> range(start: -60d)
    |> filter(fn: (r) =>
        r._measurement == "train_speed"
    )
    |>group(columns: ["driver"])

Do I have to write code to re-shape the data? That doesn’t seem right. So how can I make a clean Ruby object if I can’t extract the fields into their own columns for easier parsing? The pivot function allows me to do just that.

Flux provides a pivot function which allows you to re-shape data that exists in a single column into multiple columns, thus ensuring that your data is returned from the query engine in a way which is easier to use without actually changing the underlying data itself. Where you would normally be resigned to a predetermined set of columns and rows, pivot allows you to designate how data should be organized and displayed and can make it easier to read and directly use without having to write additional application code. Let’s take a look at how it works. From the docs:

The output is constructed as follows:

A new row is created for each unique value identified in the input by the rowKey parameter.
The initial set of columns for the new row is the row key unioned with the group key, but excluding columns indicated by the columnKey and valueColumn parameters.
A set of value columns are added to the row for each unique value identified in the input by the columnKey parameter. The label is a concatenation of the valueColumn string and the columnKey values using _ as a separator.
For each row key + column key pair, the appropriate value is determined from the input table by the valueColumn. If no value is found, the value is set to null.

To explore the pivot function, I used InfluxDB v1.7 with Flux technical preview (but the workflow is the same for the v2.0 Alpha release). First, I seeded a bucket named trains with data points containing train data using the write endpoint. The measurement is train_speed. The tags are driver, location, and train. The fields are outdoor_temp, speed, and track_temp. I would like to see all of my fields broken out into their own columns, rather than aggregated in a single column, so that I can make a clean Ruby object as I mentioned above. As a reminder, here is a sample of the data shape I’m starting with, grouped by driver, where you can see that my fields are all aggregated into a single column, _field:

The table visualization for the above grouping by driver was achieved using the following query:

from(bucket: "trains")
    |> range(start: -60d)
    |> filter(fn: (r) =>
        r._measurement == "train_speed"
    )
    |>group(columns: ["driver"])

In this query, I’m fetching data from my trains bucket, setting a range of the last 60 days (setting a time range is required in Flux), and asking for the broadest possible filter, the measurement. Then I group my data by driver to see all of the available data associated with each driver. This is about as broad of a filter as you can create, and it probably isn’t wise to use on a large dataset, but it’s useful to illustrate at a high level what filtering does.

As a result of this query, my measurement and tags all get sorted into their own columns, but all of my fields are collected together in the same _field column. I can confirm this by looking at the raw data output in the UI, which is useful for confirming whether or not I’m getting the data shape I want. What if I want a table where I see all of those fields separated out into their own columns? I can run the following query to implement the pivot function:

from(bucket: "trains")
    |> range(start: -60d)
    |> filter(fn: (r) =>
        r._measurement == "train_speed"
    )
    |>group(columns: ["driver"])
    |> pivot(
        rowKey:["_time"],
        columnKey: ["_field"],
        valueColumn: "_value"
      )

This query returns a result that looks like this:

You can see that each of my fields now has its own column, but I no longer see my measurement or tags. What if I want to see those fields separated out into their own columns while still grouping by driver and retaining rows for the measurement and tags? Unless those original columns are present in the group function or in the rowKey, they will be dropped. So I can implement the pivot function to ensure that every field will have its own column by setting its columnKey value to _field and setting my rowKey to contain not only _time but also location, train, and _measurement:

from(bucket: "trains")
    |> range(start: -60d)
    |> filter(fn: (r) =>
        r._measurement == "train_speed"
    )
    |>group(columns: ["driver"])
    |> pivot(
        rowKey:["_time", "location", "train", "_measurement"],
        columnKey: ["_field"],
        valueColumn: "_value"
      )

This query returns a result that looks like this:

I now have my fields broken out into their own columns and have also retained my measurement and tags as row keys. As a reminder, this is what I started with:

Later we’ll see what happens when we add additional values to the columnKey array.

But what if I need to sort the data according to the values which existed at a particular timestamp? You can modify the pivot as shown below:

from(bucket: "trains")
    |> range(start: -60d)
    |> filter(fn: (r) =>
        r._measurement == "train_speed"
    )
    |>group(columns: ["driver"])
    |> pivot(
        rowKey:["_field", "location", "train", "_measurement"],
        columnKey: ["_time"],
        valueColumn: "_value"
      )

By making _time my columnKey, and _field a rowKey, I’m now able to produce data that looks like this:

I now have a row for each of my fields and a column for each timestamp. Scrolling to the right allows me to see all the timestamps with their various row values.

Another useful pivot allows you to have each of the tags appear in the column name along with the field, which increases readability by allowing you to have more specific column names. This can come in handy if you have multiple values for a particular tag, but should be used with caution if you have a lot of different possible values for a tag. You can run the following query:

from(bucket: "trains")
    |> range(start: -60d)
    |> filter(fn: (r) =>
        r._measurement == "train_speed"
    )
    |>group(columns: ["driver"])
    |> pivot(
        rowKey:["_time", "location", "_measurement"],
        columnKey: ["_field", "train"],
        valueColumn: "_value"
      )

This query produces a result where driver Gupta has driven both train a and train b. You get separate columns with all possible train values (there are only two, so this query isn’t resource intensive) included in the column names, and separated from the field key by an underscore:

So now you have an idea of how pivot works in Flux. It’s a useful function for returning data in a different format than that prescribed by default constraints. There are a number of different ways you can implement the function, and I find it useful to experiment with changing the values in the rowKey and columnKey arrays to see how data is returned differently. I encourage you to do the same, and to enjoy playing around with all of the functionality offered in Flux!

Using GraphQL with InfluxDB and Flux

Sonia Gupta (InfluxData) — Thu, 17 Jan 2019 11:48:43 -0700

In a previous post, we showed you how to make Flux queries in a Rails app. In this post, we’ll extend that knowledge to use GraphQL in a Rails app which leverages the power of Flux to query InfluxDB. This allows the client to fine-tune the results of an underlying Flux query. You’ll start to understand the powerful capabilities you can access when you pair GraphQL with Flux.

Before we begin, let’s talk a bit about GraphQL and why you would use it over a traditional REST API. I highly recommend doing the tutorials and looking through the documentation at How to GraphQL to better understand what GraphQL is and how to implement it in your apps. I did a lot of my own learning there.

What is GraphQL?

GraphQL is a query language that allows the client to specify exactly what data it wants in a single request rather than having to make multiple requests to fetch the same data or deal with over-fetching or under-fetching of data. It is database agnostic in that it does not itself directly query the database. Backend logic still handles that, as usual (in our case, a Flux query will be doing the heavy lifting). GraphQL uses resolvers to handle incoming queries and then implements the backend logic to fetch data. It allows the client to have more power over fetching data than it does with a traditional REST API that returns pre-determined data structures.

What are We Building?

We’re going to build a backend that triggers Flux to query our InfluxDB database when a client initiates a GraphQL query. In this case, we are a train company that uses sensors to capture data about our trains along with other pertinent information, such as outdoor temperature. We want our client to be able to choose exactly which data it wants to retrieve, such as speed and/or track temperature, rather than over-fetching or under-fetching data. We will therefore give our client a few GraphQL queries that it can customize to suit its needs.

Trains, Planes, and Data Points

In the last post, I used a measurement called cpu_load_short with various tags and fields. In this post I’ve seeded my InfluxDB database, called trains, with train data. The measurement is train_speed. My tags are driver, location, and train. My fields are outdoor_temp, speed, and track_temp. Below you can see a sample of data for three different drivers.

Below is a sample of data for one driver.

The method to format the annotated CSV response to our Flux query, which you saw in the previous post, now looks like this after some refactoring. It performs the same function as before—except I’m now generating a train object rather than a cpu load object:

def parse_trains(response)
 header, *rows = response.body.split("\r\n").drop(3)
 column_names = parse_row(header)
 rows.map do |row|
   values = parse_row(row)
   train_data = column_names.zip(values).to_h
   Train.new(train_data)
 end
end

def parse_row(raw_row)
 raw_row.split(",").drop(3)
end

I also made a new PORO (plain old ruby object) for my train object:

class Train
   attr_reader :time, :measurement, :driver, :location, :train, :outdoor_temp, :speed, :track_temp

   def initialize(train)
      @time = train["_time"]
      @measurement = train["_measurement"]
      @driver = train["driver"]
      @location = train["location"]
      @train = train["train"]
      @outdoor_temp = train["outdoor_temp"]
      @speed = train["speed"]
      @track_temp = train["track_temp"]
   end

end

This PORO, generated at the end of my parse_trains method, comes in handy because the GraphQL gem simply fetches the respective instance variable value when resolving the incoming GraphQL query.

Adding GraphQL to a Rails Project

I found the How to GraphQL Rails tutorial to be really useful in getting started, and I recommend following the steps outlined there to add the graphql gem and the GraphiQL tool to your project.

The rest of that tutorial uses Active Record queries to fetch data from a traditional relational database using GraphQL resolvers. It’s helpful to understand how that works, but what if we want to fetch data from an InfluxDB instance? We can’t use Active Record for that, but as we learned in the last post, we can use a Flux query inside an HTTP call to fetch that data (and when the InfluxDB v2.0 client libraries are ready, we can use those).

For my new train data points, my HTTP request to the query endpoint of my InfluxDB database now looks like this:

def query_influxdb(time, filter_param = nil)
 uri = URI.parse("http://localhost:8086/api/v2/query")
 request = Net::HTTP::Post.new(uri)
 request.content_type = "application/vnd.flux"
 request["Accept"] = "application/csv"
 request.body = flux_query(time, filter_param)

 req_options = {
 use_ssl: uri.scheme == "https",
 }
 response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
   http.request(request)
 end
end

This is a generic HTTP request that is invoked for every GraphQL query. A Flux query is then dynamically generated and passed into its request.body.

Depending on the GraphQL query that is being resolved, the Flux query is constructed as follows:

def flux_query(time, filter_param = nil)
 %Q[
   #{flux_from}
   |> #{flux_range(time)}
   |> #{flux_filter(filter_param)}
   |> #{flux_pivot}
   |> #{flux_yield}
 ]
end

def flux_from
 "from(bucket:\"trains/autogen\")"
End

# There are two possible options in the range method because for most queries I just
# want a default range of the last 60 days, but in order to search by
# timestamp, I need to set my range to that exact timestamp. Flux
# allows me to do that by choosing an inclusive start time, which is my timestamp,
# and an exclusive stop time, which is one second later.
def flux_range(time)
 if time == TIME
   "range(start: #{time})"
 else
   start_time = Time.parse(time)
   stop_time = start_time + 1.second
   "range(start: #{start_time.rfc3339}, stop: #{stop_time.rfc3339})"
 end
end

def flux_filter(filter_param = nil)
 if filter_param
   "filter(fn: (r) => r.driver == \"#{filter_param}\" and
                      r._measurement == \"train_speed\")"
 else
   "filter(fn: (r) => r._measurement == \"train_speed\")"
 end
end

def flux_pivot
 "pivot(
       rowKey:[\"_time\"],
       columnKey: [\"_field\"],
       valueColumn: \"_value\"
     )"
end

def flux_yield
 "yield()"
end

Defining GraphQL Types

After following the initial setup instructions outlined in the How to GraphQL Rails tutorial, I created a new file, app/graphql/types/train_type.rb, into which I placed the following code:

# defines a new GraphQL type

Types::TrainType = GraphQL::ObjectType.define do
   # this type is named 'Train'
   name 'Train'

   # it has the following fields
   field :time, types.String
   field :measurement, types.String
   field :driver, types.String
   field :location, types.String
   field :train, types.String
   field :outdoor_temp, types.String
   field :speed, types.String
   field :track_temp, types.String

end

So what’s going on here, exactly? I’m defining an object type. I know I’ve got an InfluxDB database populated with train data points, with my measurement being train_speed, tags of driver, location, and train, and my fields being speed, outdoor_temp, and track_temp. I need the shape of my GraphQL object type to mimic the shape of my data points in order to render each of the fields. Each of the GraphQL fields thus corresponds to a column of my InfluxDB data points. Note that the Flux pivot function is what allows me to have my InfluxDB fields appear as their own separate columns, which makes the data more readable.

In addition to object types, there are also root types in GraphQL. Root types include a query type and a mutation type. Since we will be querying data, we need to establish our query type in which we will place the code for all of our queries. When we ran rails generate graphql:install in the initial setup, we generated a template file called app/graphql/types/query_type.rb. We now place into this file our various queries, which are just defined as fields, similar to the object type. My entire file looks like this (for easier reading I placed the aforementioned HTTP call, parsing method, and Flux query generator methods at the bottom as private methods):

require 'net/http'
require 'uri'

TIME = "-60d"

Types::QueryType = GraphQL::ObjectType.define do
 name "Query"

   field :allTrains, !types[Types::TrainType] do
     description "Return all train data"
     resolve -> (obj, args, ctx) {
       parse_trains(query_influxdb(TIME))
     }
   end

   field :trainByDriver, !types[Types::TrainType] do
     argument :driver, !types.String
     description "Find train stats by driver"
     resolve -> (obj, args, ctx) {
       parse_trains(query_influxdb(TIME, args[:driver]))
     }
   end

   field :trainByTime, !types[Types::TrainType] do
     argument :time, !types.String
     description "Find train stats by time"
     resolve -> (obj, args, ctx) {
       parse_trains(query_influxdb(args[:time]))
     }
   end

end

private

def parse_trains(response)
 header, *rows = response.body.split("\r\n").drop(3)
 column_names = parse_row(header)
 rows.map do |row|
   values = parse_row(row)
   train_data = column_names.zip(values).to_h
   Train.new(train_data)
 end
end

def parse_row(raw_row)
 raw_row.split(",").drop(3)
end

def query_influxdb(time, filter_param = nil)
 uri = URI.parse("http://localhost:8086/api/v2/query")
 request = Net::HTTP::Post.new(uri)
 request.content_type = "application/vnd.flux"
 request["Accept"] = "application/csv"
 request.body = flux_query(time, filter_param)

 req_options = {
 use_ssl: uri.scheme == "https",
 }
 response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
   http.request(request)
 end
end

def flux_query(time, filter_param = nil)
 %Q[
   #{flux_from}
   |> #{flux_range(time)}
   |> #{flux_filter(filter_param)}
   |> #{flux_pivot}
   |> #{flux_yield}
 ]
end

def flux_from
 "from(bucket:\"trains/autogen\")"
end

def flux_range(time)
 if time == TIME
   "range(start: #{time})"
 else
   start_time = Time.parse(time)
   stop_time = start_time + 1.second
   "range(start: #{start_time.rfc3339}, stop: #{stop_time.rfc3339})"
 end
end

def flux_filter(filter_param = nil)
 if filter_param
   "filter(fn: (r) => r.driver == \"#{filter_param}\" and
                      r._measurement == \"train_speed\")"
 else
   "filter(fn: (r) => r._measurement == \"train_speed\")"
 end
end

def flux_pivot
 "pivot(
       rowKey:[\"_time\"],
       columnKey: [\"_field\"],
       valueColumn: \"_value\"
     )"
end

def flux_yield
 "yield()"
end

The allTrains query returns, as expected, all trains. My resolver is just returning that entire data set, and behind the scenes, the grapqhl gem is simply using dot syntax to call the methods to access each of the instance variables I defined in my Train PORO.

If we completed the initial setup properly, we can use the GraphiQL tool to simulate client queries. Let’s head over to http://localhost:3000/graphiql to test out our queries. I can return all of the fields with the allTrains query, as you see below:

I can also return a selection of only the fields I want:

This is why GraphQL is so useful. The client gets to decide what fields it wants returned rather than having a pre-determined data set returned by a conventional REST API endpoint. On the backend, I can provide my client with a number of options to fetch the desired data, and these options are highly customizable.

For example, I can allow my client to pass in an argument to return all trains for a single driver using the trainByDriver query. In this query, the transformation is passed down to Flux to do the work of returning the train data associated with only the requested driver while GraphQL handles the field selection. The same functionality of only returning the requested fields exists in this query as well.

Since this is time series data we’re working with, it makes sense to allow our client to search by a single timestamp using the trainByTime query. As you can see in the Flux query generator code we discussed earlier, we again push the work of transforming the data into a Flux query to retrieve the data associated with that timestamp, and then have GraphQL return only the fields requested by the client.

You can probably come up with a number of other queries that would be useful to a client, all of which will have the benefit of providing only the information the client specifically requests.

Summary

GraphQL makes a powerful companion to your InfluxDB database. It allows the client to specify exactly what information it needs without over-fetching or under-fetching data. Combined with the results of a Flux query, you now have two loci of control: the range and filter functions of your Flux query, and the types of GraphQL queries you open up to the client. The client can simplify its code and avoid superfluous code, particularly when you push the work of transformation down to the database query rather than requiring the client to make that transformation. Furthermore, you no longer require your client to change the endpoint it hits as requirements change because it is using GraphQL to query the API instead of directly hitting your endpoints. This creates a lot of flexibility in the backend while empowering the client to make data decisions on its own.

One thing that’s worth noting is that you could expose a Flux endpoint to the front end developer and they could write Flux rather than GraphQL to get the same result. However, the advantages of GraphQL are that you have client libraries and the other tooling (like the explorer) that are built on top of it. Also, if you end up hooking into other databases, those results can be joined together in a single API call (i.e. InfluxDB + Postgres).

Making Flux Queries in Rails

Sonia Gupta (InfluxData) — Tue, 04 Dec 2018 10:30:23 -0700

Now that we’ve launched Flux, a combination query and scripting language, you’re probably eager to start playing around with it in your apps. We’re in the process of building libraries for a number of languages, but in the meantime, you can start making Flux queries using a simple raw HTTP request. In this post, we’ll walk through using the sandbox to get the stack up and running and then we’ll query our database using Flux in a Rails app. In a follow-up post, we’ll add GraphQL to our Rails app to enable us to allow the client to have control over the information it wants to retrieve from our Flux query.

First, clone the sandbox repo. Next, install Docker on your machine if you don’t already have it. The fastest way to do this is running homebrew cask install docker using Homebrew. Then you can start Docker using spotlight. Once you’ve got docker running, you can cd to the file path of the cloned down repo and run ./sandbox to see all of the commands available to you in the sandbox.

Running ./sandbox up will get you the latest stable versions of software loaded up within the sandbox and will open two tabs running localhost in your browser. One of the tabs includes tutorials and documentation for the sandbox, and the other tab is where most of the fun is happening in Chronograf, which is InfluxData’s data visualization and querying layer. One of the best parts of using the sandbox is that all four parts of the stack (InfluxDB, Chronograf, Telegraf, and Kapacitor) are automatically wired up to communicate with each other.

You can click around in Chronograf to see what’s already working right out of the box, but you’re here to learn how to query an InfluxDB database using Flux in a Rails app, so let’s get to it!

We can use the InfluxDB CLI to create the database we intend to query. The sandbox makes this easy by allowing us to run ./sandbox influxdb. Once you’re in the CLI, you can create your database by running CREATE <database name>.

We can also use Chronograf to create the database we intend to query. In the left-hand Navigation panel, select the “Admin” icon and then “InfluxDB.”

Now you should see the available databases within the InfluxDB sandbox instance. Click the “Create Database” button to create your database.

I recommend you follow the steps in the Getting Started docs to seed some data into it using line protocol (you can also use the InfluxDB API to write multiple points to your database). I followed the docs and created the “mydb” database and seeded it with several cpu load values.

You can now navigate to Chronograf’s Explore tab and actually see the database you created, along with the values you inserted into it. You also have the ability to start querying your database using Flux by selecting the Flux option (as opposed to InfluxQL which is the default).

Note that Flux is enabled by default in the sandbox, so you don’t have to alter the influxdb.conf file to enable it.

In the screenshot below, I used Flux to query for all of the points in my “cpu_load_short” measurement. In my case, the “cpu_load_short” measurement has two tags, “host” and “region,” and one field, “value.” I only had 31 points in this measurement, so it’s not a big deal, but as a general rule you probably won’t be running a SELECT * in InfluxQL or a |> group(by: ["_measurement"]) in Flux. I added the “group” function to my Flux query so all my entries will visually appear in the same table view in Chronograf, but without the “group” function, you will still get back all of your points and can see them when you toggle “View Raw Data”. I’m using it here merely to illustrate a simple Flux query on a small dataset.

Now for the fun part. Let’s run this same Flux query in a Rails app! The Flux docs give us a good clue of where to start, given that we’re just going to be making a raw HTTP request (until the client libraries are ready, at least).

Per the docs we can simply query InfluxDB’s query endpoint. We will make a POST request to the /api/v2/query endpoint. Our request will set the accept header to application/csv and the content-type header to application/vnd.flux. The response we receive will be in annotated CSV format. The docs give us a curl example that we can paste into curl-to-ruby to convert to Ruby’s net/http. Let’s use the example in the Flux docs to see how this will work:

It really is that simple. So to make my query, I need only add the net/http and uri gems to my Rails app and replace the request.body above with my Flux query, properly formatted to account for the multiple strings within a string.

I encapsulated the HTTP request in a method that looks like this, based on the format of the above curl-to-ruby conversion. You can place something like this wherever in your app seems most suitable. Because I’m also building GraphQL into my app, I placed my code in the same file as my query types, but it may make sense for me to refactor this out into a helper later:

def all_cpu_loads
 uri = URI.parse("http://localhost:8086/api/v2/query")
 request = Net::HTTP::Post.new(uri)
 request.content_type = "application/vnd.flux"
 request["Accept"] = "application/csv"
 request.body = "from(bucket:\"mydb/autogen\")
 |> range(start:-30d)
 |> filter(fn: (r) => r._measurement == \"cpu_load_short\")
 |> group(by: [\"_measurement\"])"

 req_options = {
 use_ssl: uri.scheme == "https",
 }

 response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
   http.request(request)
 end
end

See the body of my request? I literally cut and pasted it from the query I generated above in Chronograf and simply fixed the string format and added an actual time range (since my app obviously doesn’t know about the Chronograf variable dashboardTime).

If we look at the response.body of that request, it’s pretty messy in its annotated CSV format. We can clean that up with a PORO (plain old Ruby object) and a simple parsing method. In my case, I know that each point (each “load,” so to speak) in my “cpu_load_short” measurement has a timestamp, a value, a field, a measurement (“cpu_load_short” for each point), a host, and a region. I want to be able to access each of those attributes in a neat package, so I’ve turned each point into a hash and placed it into a new “Load” object:

def formatted_response(response)
 column_names = response.body.split(",0,")[0].split("\r\n")[3]
 column_names.slice!(",result,table,_start,_stop,") 
 arrayed_column_names = column_names.split(",")

 entries = response.body.split(",0,").drop(1)
 formatted_entries = entries.map do |entry|
   entry.gsub(/\r\n?/, "").split(",").drop(2)
 end
 hashed = formatted_entries.map do |load|
   arrayed_column_names.zip(load).to_h
 end
 hashed.map do |hash|
   Load.new(hash)
 end
end

This leaves me with a nice array of Load objects that will come in handy when I need to implement GraphQL later. For reference, my Load looks like this:

class Load
   attr_reader :time, :value, :field, :measurement, :host, :region

   def initialize(load)
      @time = load["_time"]
      @value = load["_value"]  
      @field = load["_field"]
      @measurement = load["_measurement"]
      @host = load["host"]
      @region = load["region"]
   end

end

And that’s it! Until our client libraries are ready, this is how you can implement any number of Flux queries in a Rails app and render usable objects from the results of your query. When the libraries are published, we’ll create another post showing you how to replace the raw HTTP request with library functions. And in a follow-up post, we’ll be showing you how to use GraphQL to allow a client to query for the exact information it wants from the results of your Flux queries. Happy Fluxing!

When Values Align: Why I Joined InfluxData

Sonia Gupta (InfluxData) — Thu, 01 Nov 2018 10:07:52 -0700

I’m excited to announce that I’ve joined the InfluxData team as a Developer Advocate! Prior to joining the team I worked as a backend developer, and prior to becoming a developer I practiced law in Southern Louisiana, mostly doing criminal work. The combination of my advocacy and engineering skills makes this position a perfect fit for me, and I’m honored to be joining a team of passionate and innovative individuals.

It’s a tradition here at InfluxData to publish a post about why you joined the team. I think it’s a wonderful tradition, and I’m happy to be a part of it. So here goes!

As a developer, I used to think it didn’t matter what product I worked on as long as I worked with great people. While it’s true that having a great team is important, I’ve learned over time that I have to be working on something that contributes to the industry, and the world, in a meaningful way. InfluxData builds products that do exactly that.

Large datasets are becoming commonplace, and the tools we build help developers to store, parse, and understand that data to extract meaningful information. This is incredibly valuable in an era where the responsible and efficient handling of data is of paramount importance. I’m really lucky to be working with a team that understands this, and am excited to make my contributions in line with the company ethos, which brings me to another area of alignment: the InfluxData values.

I believe that the core values of a company truly do shape its trajectory and inform the decisions of its leadership. Two of our values resonate most with me: “We believe humility drives learning.” and “We embrace failure.” These are values that I’ve struggled to embody in the past, coming from a background as a lawyer where so many of my colleagues thrived on bravado and where failure wasn’t an option. I realized that their kind of mentality is toxic and I’ve learned to shift my mindset as a developer, and now as a Developer Advocate. I’m heartened to find a company that understands the importance of humility and failure as foundational aspects of learning and thriving as an individual and as an organization.

InfluxData understands the importance of maintaining strong relationships with the developers who use its products, and I’m honored to be part of a team composed of people dedicated to that mission. I look forward to getting to know our users and helping them to get the most out of our products.