InfluxData Blog - Regan Kuchan

TL;DR Tech Tips: New Query Language Features in Version 1.3

Regan Kuchan (InfluxData) — Thu, 22 Jun 2017 04:00:11 -0700

In this post, we give a preview of some new query language features in InfluxQL version 1.3.

New Operators

Version 1.3 introduces several new mathematical operators:

Modulo (%)
Bitwise AND (&)
Bitwise OR (|)
Bitwise Exclusive-OR (^)

The following query calculates the moduli between two fields: cod and pod.

> SELECT "cod" % "pod" FROM "prod"

name: prod
time                  cod_pod
----                  -------
2017-06-21T22:55:25Z  0
2017-06-21T22:56:25Z  1
2017-06-21T22:57:25Z  3

New time zone clause

InfluxQL’s new time zone clause returns the UTC offset for the specified timezone. The query below returns the UTC offset for Chicago’s time zone:

> SELECT "water_level" FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:18:00Z' tz('America/Chicago')

name: h2o_feet
time                       water_level
----                       -----------
2015-08-17T19:00:00-05:00  2.064
2015-08-17T19:06:00-05:00  2.116
2015-08-17T19:12:00-05:00  2.028
2015-08-17T19:18:00-05:00  2.126

New duration unit

Duration units specify the time precision in InfluxQL queries and when writing data to InfluxDB. InfluxQL Version 1.3 introduces a new nanosecond duration unit: ns.

The query below uses a GROUP BY time() clause to group averages into 1000000000 nanosecond buckets:

> SELECT MEAN("value") FROM "gopher" WHERE time >= 1497481480598711679 AND time <= 1497481484005926368 GROUP BY time(1000000000ns)

What's next:

Downloads for the TICK-stack are live on our "downloads" page.
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring, and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 300 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition hoodie as a thank you.

TL;DR Tech Tips: New InfluxQL Functions in Version 1.3

Regan Kuchan (InfluxData) — Thu, 15 Jun 2017 04:00:54 -0700

In this post, we give a preview of the new InfluxQL functions in version 1.3. Check in next week for the most interesting TICK Stack related issues, workarounds, how-tos, and Q&A from GitHub, IRC, and the InfluxData Community.

New Function: INTEGRAL()

The INTEGRAL() function returns the area under the curve for subsequent field values. The query below returns the area under the curve (in seconds) for the field values associated with the water_level field key and in the h2o_feet measurement:

> SELECT INTEGRAL("water_level") FROM "h2o_feet" WHERE "location" = 'santa_monica' AND time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z'

name: h2o_feet
time                  integral
----                  --------
1970-01-01T00:00:00Z  3732.66

New Function: NON_NEGATIVE_DIFFERENCE()

The NON_NEGATIVE_DIFFERENCE() function returns the non-negative result of subtraction between subsequent field values. Non-negative results of subtraction include positive differences and differences that equal zero. The query below returns the non-negative difference between subsequent field values in the water_level field key and in the h2o_feet measurement:

> SELECT NON_NEGATIVE_DIFFERENCE("water_level") FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica'

name: h2o_feet
time                  non_negative_difference
----                  -----------------------
2015-08-18T00:06:00Z  0.052000000000000046
2015-08-18T00:18:00Z  0.09799999999999986
2015-08-18T00:30:00Z  0.010000000000000231

Updated Functions: TOP() and BOTTOM()

Version 1.3 introduces three major changes to the TOP() and BOTTOM functions:

TOP() and BOTTOM() no longer support other functions in the SELECT clause. The following query returns an error:

> SELECT TOP(value,1),MEAN(value) FROM "gopher"
  
ERR: error parsing query: selector function top() cannot be combined with other functions

TOP() and BOTTOM() now maintain tags as tags if the query includes a tag key as an argument. The query below preserves location as a tag in the newly-written data:

> SELECT BOTTOM("water_level","location",2) INTO "bottom_water_levels" FROM "h2o_feet"

name: result
time                 written
----                 -------
1970-01-01T00:00:00Z 2

> SHOW TAG KEYS FROM "bottom_water_levels"

name: bottom_water_levels
tagKey
------
location

TOP() and BOTTOM() now preserve the timestamps in the original data when they're used with the GROUP BY time() clause. The following query returns the points' original timestamps; the timestamps are not forced to match the start of the GROUP BY time() intervals:

> SELECT TOP("water_level",2) FROM "h2o_feet" WHERE time >= '2015-08-18T00:00:00Z' AND time <= '2015-08-18T00:30:00Z' AND "location" = 'santa_monica' GROUP BY time(18m)

name: h2o_feet
time                   top
----                   ------
                           __
2015-08-18T00:00:00Z  2.064 |
2015-08-18T00:06:00Z  2.116 | <------- Greatest points for the first time interval
                           --
                           __
2015-08-18T00:18:00Z  2.126 |
2015-08-18T00:30:00Z  2.051 | <------- Greatest points for the second time interval
                           --

What's next:

Downloads for the TICK Stack are live on our "downloads" page.
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring, and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 300 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition hoodie as a thank you.

TL;DR Tech Tips: Managing the Work History in Chronograf

Regan Kuchan (InfluxData) — Thu, 01 Jun 2017 04:00:13 -0700

In this post, we recap some new material that you might have missed in the past week or so. Check in next week for the most interesting TICK-stack related issues, workarounds, how-tos, and Q&A from GitHub, IRC, and the InfluxData Community.

Managing alert history in Chronograf

Q: I’m using Chronograf to create alert rules and monitor my infrastructure. My Alert History page shows a lot of alert occurrences; I can narrow down the occurrences with the filter box and the time range selector, but is there a way to expire the occurrences that I no longer need?

A: There is! Chronograf stores the information on the Alert History page as time series data in InfluxDB. It stores it in the chronograf database and in the alerts measurement. By default, those data are subject to an infinite retention policy (RP), that is, InfluxDB stores them forever. To expire your alert history data, just shorten the duration of that RP.

We recommend using Chronograf’s Admin page to modify the retention policy in the chronograf database. In the Databases tab:

1: Locate the `chronograf` database and click on the infinity symbol (?)

2: Enter a different duration

The minimum allowable duration is one hour (1h) and the maximum is infinite (INF). See the InfluxDB documentation for the list of acceptable duration units.

3: Click the green check mark to save your changes

That’s it! InfluxDB only keeps data in the chronograf database that fall within that new duration; the system automatically deletes any data with timestamps that occur before the duration setting.

Writing data with Chronograf

Q: I’ve been using the deprecated web admin interface to write data to InfluxDB. Is there a way to write data using Chronograf?

A: Starting with version 1.3.2.0, Chronograf’s Data Explorer supports writing data to your InfluxDB databases.

To write data with Chronograf, click the Write Data icon at the top of the Data Explorer page and select your target database. Next, enter your line protocol in the main text box and click the Write button.

What's next:

Downloads for the TICK-stack are live on our "downloads" page.
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring, and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 300 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition hoodie as a thank you.

TL;DR InfluxDB Tech Tips: Identical Field Keys & Tag Keys, Existing Kapacitor TICKscripts in Chronograf & More

Regan Kuchan (InfluxData) — Thu, 18 May 2017 04:00:56 -0700

Identical field keys and tag keys

Q: My InfluxDB schema consists of the feet-attire measurement, the shoes tag which tracks shoe color, and the shoes field which tracks shoe quantity. When I attempt to query both the shoes tag and the shoe field, I see duplicate rows of the shoe field but nothing for the shoe tag. Is there a way to see both the field and tag?

> SELECT "shoes","shoes" FROM "feet-attire"

name: feet-attire
time                            shoes shoes_1
----                            ----- -------
2017-05-17T20:34:32.143717027Z  12    12
2017-05-17T20:34:39.949844001Z  15    15
2017-05-17T20:34:47.325195089Z  16    16

A: There is! Use the :: syntax to specify if the identifier is a field or tag:

> SELECT "shoes"::field,"shoes"::tag FROM "feet-attire"

name: feet-attire
time                            shoes shoes_1
----                            ----- -------
2017-05-17T20:34:32.143717027Z  12    yellow
2017-05-17T20:34:39.949844001Z  15    blue
2017-05-17T20:34:47.325195089Z  16    pink

Field keys or tag keys called `time`

Q: I’m using InfluxDB version 1.2.2 and I can’t seem to query my data. It looks like the write succeeds but my queries don’t return any results. Can you help me out? What am I doing wrong?

~# curl -i -XPOST "http://localhost:8086/write?db=tldr&u=<username>&p=<password>" --data-binary 'creatures time="1h"'
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: b76456c6-3b47-11e7-8f86-000000000000
X-Influxdb-Version: 1.2.1-c1.2.2
Date: Wed, 17 May 2017 21:28:06 GMT

~# curl -G "http://localhost:8086/query?db=tldr&u=<username>&p=<password>" --data-urlencode 'q=SHOW MEASUREMENTS'
{"results":[{"statement_id":0}]}


~# curl -G "http://localhost:8086/query?db=tldr&u=<username>&p=<password>" --data-urlencode 'q=SELECT * FROM creatures'
{"results":[{"statement_id":0}]}

A: Your syntax looks good! It’s time that’s causing the issue. time is a special keyword in InfluxDB. time can be a continuous query name, database name, measurement name, retention policy name, subscription name, and username. It can’t be a field key or tag key in InfluxDB.

In version 1.2.2 (more specifically, versions 1.2.0-1.2.3), InfluxDB accepts writes with time as a field or tag key but it silently drops the field key or tag key and its associated value. In version 1.2.4, we’ve changed that behavior; InfluxDB rejects writes with time as a field key or tag key and returns an error. Check out our FAQ page for more information!

Existing Kapacitor TICKscripts in Chronograf

Q: I just started using Chronograf and I have some Kapacitor tasks that I created prior to using Chronograf. Will those tasks show up on Chronograf’s Alert Rules page?

A: Currently, Kapacitor tasks that you created outside of Chronograf have limited functionality in the user interface.

In Chronograf, you can:

View pre-existing tasks the Alert Rules page
View pre-existing tasks' activity on the Alert History page
Enable and disable pre-existing tasks on the Alert Rules page (this is equivalent to the kapacitor enable and kapacitor disable commands)
Delete pre-existing tasks the Alert Rules page (this is equivalent to the kapacitor delete tasks command)

You cannot edit pre-existing tasks on Chronograf’s Alert Rules page. The mytick task in the image below is a pre-existing task; its name appears on the Alert Rules page but you cannot click on it or edit its TICKscript in the interface. Currently, you must manually edit your existing tasks and TICKscripts on your machine.

What's next:

Downloads for the TICK-stack are live on our "downloads" page.
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring, and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 300 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition hoodie as a thank you.

TL;DR InfluxDB Tech Tips: Configuring Telegraf & Kapacitor When auth is Enabled on InfluxDB & Tips on Chronograf

Regan Kuchan (InfluxData) — Thu, 11 May 2017 04:00:18 -0700

In this post, we recap some new material that you might have missed in the past week or so. Check in next week for the most interesting TICK stack related issues, workarounds, how-tos, and Q&A from GitHub, IRC, and the InfluxData Community.

Understanding Chronograf's status icon

Q: I’m using Chronograf to monitor my infrastructure and one of the status icons on the Host List page is red. What does this mean exactly?

My Host List page:

A: The status icon on Chronograf’s Host List page is a high-level measure of your hosts’ health. If Chronograf has not received data from a host’s Telegraf instance for the past minute, the status icon turns red. If Chronograf has received data from a host within the past minute, the status icon remains green.

On your Host List page, it looks like Chronograf hasn’t received Telegraf data from node-abode-02 for at least the past minute.

Configuring Telegraf and Kapacitor when authentication is enabled on InfluxDB

Q: I’m trying to connect a Telegraf instance and a Kapacitor instance to InfluxDB. I’ve enabled authentication on my InfluxDB instance and I get the following errors in the Telegraf and Kapacitor logs when I try to start those processes. How to I specify the authentication credentials in Telegraf and Kapacitor?

Log errors:

# Telegraf
InfluxDB Output Error: {"error":"unable to parse authentication credentials"}

# Kapacitor
run: open server: open service *influxdb.Service: failed to link subscription on startup: unable to parse authentication credentials

A: Specify your InfluxDB username and password in the Telegraf and Kapacitor configuration files. On most systems, Telegraf’s configuration file is located at /etc/telegraf/telegraf.conf and Kapacitor’s configuration file is located at /etc/kapacitor/kapacitor.conf. I’ve printed out the relevant configuration settings below. Once you’ve updated those files, just start the Telegraf and Kapacitor processes and you’re good to go!

Telegraf configuration file:

[[outputs.influxdb]]
  ## The full HTTP or UDP endpoint URL for your InfluxDB instance.

  [...]

  username = "administer" #????
  password = "supersupersecret" #????

Kapacitor configuration file:

[[influxdb]]
  # Connect to an InfluxDB cluster
  # Kapacitor can subscribe, query and write to 

  [...]

  username = "administer" #????
  password = "supersupersecret" #????

Introducing Chronograf's visualization types

Q: I’m thinking about using Chronograf to create a dashboard; what visualization types do you support?

A: Currently, Chronograf supports five visualization types: line, stacked, step-plot, singleStat, and line+stat.

Check out Chronograf’s FAQ page for more information about our visualization types and the Create a Dashboard guide to get started with your customized dashboard!

What's next:

Downloads for the TICK-stack are live on our "downloads" page.
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring, and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 300 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition hoodie as a thank you.

TL;DR InfluxDB Tech Tips - Template Variables, Query Building, & User Management in Chronograf

Regan Kuchan (InfluxData) — Thu, 04 May 2017 04:00:21 -0700

Template variables in Chronograf

Q: I have a tag key with fifteen different tag values and I want to view data for each tag key-value pair on its own graph. I’m using Chronograf’s dashboards to visualize my data - is there a way to do what I want without having to create fifteen different graphs that have nearly identical queries?

A: There is! Starting in version 1.2.0-beta10, Chronograf supports dashboard template variables. Dashboard template variables offer the functionality that you’re looking for; they allow you to alter specific components of a cell’s query without having to navigate away from the dashboard or create several nearly-identical cells.

Here’s an example of a dashboard template variable in action. The template variable :frenchie-name: stores the three tag values (brinkely, princess, and speckles) associated with a specific tag key. Change the data displayed in the dashboard graph by choosing the different tag values from the template variable dropdown.

Query building in Chronograf

Q: I’ve been using Chronograf to create dashboards. I recently upgraded to version 1.2.0-beta9 and I’ve noticed that the raw query editor is missing; I can no longer select the InfluxQL option when creating a new query in the Explorer. Is this feature gone? Can I manually edit a query in Chronograf in the current version?

A: Starting with version 1.2.0-beta9, we’ve effectively merged the raw query editor and the builder. We did this to improve the query creation process and make it more interactive. You can now move seamlessly between using the builder and manually editing the query; when possible, the interface automatically populates the builder with the information that you’ve entered manually.

Here’s an example of how this might work. First, I build a query that selects just the usage_idle field key. Next, I manually add the usage_user field key to the SELECT clause. Notice that the builder automatically updates the Field column with my change. Finally, I use the builder to remove that second usage_user field key.

InfluxDB user management in Chronograf

Q: I’m using Chronograf version 1.2.0-beta10 to manage users in my InfluxDB instance and I’ve noticed that I’m only able to assign admin status to users. The InfluxDB documentation discusses assigning READ and WRITE access to non-admin users - is this something I can do in Chronograf?

I can’t see a way to do it in the Users section on the Admin page. Here’s what my setup looks like:

A: Currently, Chronograf does not support assigning database READ or WRITE access to non-admin users. This is a known issue. As a workaround, grant READ, WRITE, or ALL (READ and WRITE) permissions to non-admin users with the following curl commands, replacing anything inside < > with your own values:

Grant `READ` permission:

~# curl -XPOST "http://<InfluxDB-IP>:8086/query?u=<username>&p=<password>" --data-urlencode "q=GRANT READ ON <database-name> TO <non-admin-username>"

Grant `WRITE` permission:

~# curl -XPOST "http://<InfluxDB-IP>:8086/query?u=<username>&p=<password>" --data-urlencode "q=GRANT WRITE ON <database-name> TO <non-admin-username>"

Grant `ALL` permission:

~# curl -XPOST "http://<InfluxDB-IP>:8086/query?u=<username>&p=<password>" --data-urlencode "q=GRANT ALL ON <database-name> TO <non-admin-username>"

In all cases, a successful GRANT query returns a blank result:

{"results":[{"statement_id":0}]}   <--- Success!

Remove READ, WRITE, or ALL permissions from non-admin users by replacing GRANT with REVOKE in the curl commands above.

What's next:

Downloads for the TICK-stack are live on our "downloads" page.
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring, and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 300 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition hoodie as a thank you.

TL;DR InfluxDB Tech Tips - Querying Data in a Non-DEFAULT Retention Policy, Checking Field Types, and Querying vs. Writing Booleans

Regan Kuchan (InfluxData) — Thu, 27 Apr 2017 04:00:23 -0700

Querying data in a non-DEFAULT retention policy

Q: I created a new retention policy called secret_garden. I started writing data to that retention policy and the logs show that the writes are successful, but when I query the data I get an empty response. What am I doing wrong?

Results of the SHOW RETENTION POLICIES query:

> SHOW RETENTION POLICIES ON "lands"
name	       duration	  shardGroupDuration   replicaN   DEFAULT
autogen	       0          168h0m0s             1          true
secret_garden  24h0m0s	  1h0m0s               1          false

Results of a SELECT query:

> SELECT * FROM "foliage"
>

A: The retention policy secret_garden is not the DEFAULT retention policy for your database (see the fifth column in the SHOW RETENTION POLICIES output).

You’ll need to fully qualify the foliage measurement if you’re querying data in a retention policy that is not the DEFAULT retention policy. Fully qualify a measurement by specifying the database and retention policy in the FROM clause: <"database">.<"retention_policy">.<"measurement">.

Example:

> SELECT * FROM "lands"."secret_garden"."foliage"
name: foliage
-------------
time                             roses
2016-05-31T17:09:04.697144667Z   3
2016-05-31T17:09:07.096708707Z   5

Checking field types

Q: Is there a way to see the data type of a field?

A: Starting with InfluxDB version 1.0, the SHOW FIELD KEYS query also returns the field’s data type.

Example:

> SHOW FIELD KEYS FROM all_the_types
name: all_the_types
-------------------
fieldKey  fieldType
blue      string
green     boolean
orange    integer
yellow    float

Querying vs. writing booleans

Q: I’m able to successfully write booleans to my database, but I can’t seem to use them in my query’s WHERE clause.

> INSERT hamlet tobe=t
> INSERT hamlet tobe=f
> SELECT * FROM "hamlet" WHERE "tobe"=t
>

A: Acceptable boolean syntax differs for writes and for queries.

In your case, InfluxDB understands that t means true when you write your data but it doesn’t understand that t means true when you query your data. You’ll need to use true, True, or TRUE to reference that boolean value in a query.

Example:

> SELECT * FROM "hamlet" WHERE "tobe"=true
name: hamlet
------------
time                             tobe
2016-05-31T18:07:32.93151244Z    true

What's next:

Downloads for the TICK-stack are live on our "downloads" page.
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring, and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 300 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition hoodie as a thank you.

TL;DR InfluxDB Tech Tips - Chronograf's Template Queries and Function Documentation's Make-over!

Regan Kuchan (InfluxData) — Thu, 20 Apr 2017 04:00:41 -0700

Chronograf has template queries

Starting with version 1.2.0-beta8, Chronograf’s data explorer supports query templates. Just click on the Query Templates dropdown and select a CREATE, DROP, or SHOW query to enter it in the query input.

Function documentation gets a make-over

Last week we released an all-new version of the functions documentation. Functions are an essential part of InfluxDB’s query language; they allow you to aggregate, transform, and analyze your time-series data. We hope the revised documentation helps you get to know functions and allows you to seamlessly integrate them into your monitoring solution.

Among the various improvements, the new documentation includes:

In-depth syntax descriptions of each InfluxQL function
Numerous practical examples using a sample dataset
Insights into common issues with syntax and behavior
Extended sections on specifying multiple functions in the SELECT clause and on using the AS clause

Tech tips get a webinar

Check out our Tech Tips webinar to get an overview of the blog entry highlights in the past quarter. The webinar goes into detail about InfluxQL’s subqueries, INTO queries, new CLI features in version 1.2, and it offers a couple miscellaneous tips for working with InfluxDB and Chronograf.

Quarterly Tech Tips - April 2017 from InfluxData Platform on Vimeo.

What's next:

Downloads for the TICK-stack are live on our "downloads" page.
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring, and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 300 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition hoodie as a thank you.

TL;DR InfluxDB Tech Tips - Missing Data with DISTINCT() & the INTO Clause, Unexpected Timestamps with MAX() & More

Regan Kuchan (InfluxData) — Thu, 13 Apr 2017 04:00:54 -0700

What happens when you cross a function with an INTO clause or a GROUP BY time() clause? Occasionally, some curious results. In this post, we describe the unexpected when working with InfluxQL functions in the SELECT statement.

Missing data with DISTINCT() and the INTO clause

Q: I’m trying to write the results of the DISTINCT() function to a different measurement. The only thing is - the results of the query without the INTO clause and the results of the query with the INTO clause are different. Is this the expected behavior? I have a feeling I’m missing something.

Just DISTINCT():

> SELECT DISTINCT("personality_type") FROM "twins"

name: twins
time                  distinct
----                  --------
1970-01-01T00:00:00Z  a                 
1970-01-01T00:00:00Z  b                   <--- Three results!
1970-01-01T00:00:00Z  uncategorizable

DISTINCT() with INTO:

> SELECT DISTINCT("personality_type") INTO "twin_types" FROM "twins"

name: result
time written
---- -------
1970-01-01T00:00:00Z 3      <--- Three written points!

> SELECT * FROM "twin_types"

name: twin_types
time                  distinct
----                  --------
1970-01-01T00:00:00Z  uncategorizable  <--- Just one point!

A: The behavior that you’re seeing is the expected behavior; it comes down to how InfluxDB identifies a single point and how it handles duplicate points.

Notice that each result of the DISTINCT() query without the INTO clause has the same timestamp. InfluxDB assumes that points in the same series and with the same timestamp are duplicate points.

When you add the INTO clause to the query, InfluxDB writes three duplicate points to the twin_types measurement. When the system encounters duplicate points, it simply overwrites the previous point with the most recent point so you end up with only one result in the destination measurement. It’s a little unexpected but that’s the explanation for what you’re seeing.

Unexpected timestamps with MAX() and the GROUP BY time() clause

Q: I’m trying to find the maximum number of eggs for specific time intervals. The results in the max column are accurate but the timestamps in the time column aren’t what I expected to see. I’ve included a simplified version of my data below. Why do the timestamps seem off?

Raw data:

> SELECT * FROM "house"

name: house
time                  eggs
----                  ----
2017-04-12T20:00:00Z  1     <--- Max value in the first 20-minute interval
2017-04-12T20:10:00Z  0
2017-04-12T20:20:00Z  3
2017-04-12T20:30:00Z  5     <--- Max value in the second 20-minute interval

My query:

> SELECT MAX("eggs") FROM "house" WHERE time >= '2017-04-12T20:00:00Z' AND time <= '2017-04-12T20:30:00Z' GROUP BY time(20m)

name: house
time                  max
----                  ---
2017-04-12T20:00:00Z  1
2017-04-12T20:20:00Z  5     <--- I'd expect this timestamp to be 2017-04-12T20:30:00Z

A: The timestamps that you’re seeing are a consequence of the GROUP BY time() clause. That clause automatically overrides MAX()’s original timestamps with timestamps that mark the start of the relevant time interval.

In your case, the second timestamp in the results (2017-04-12T20:20:00Z) refers to the interval between 2017-04-12T20:20:00Z and just before 2017-04-12T20:30:00Z, not to when the maximum value occurred in the raw data. Almost all* queries that use an InfluxQL function with a GROUP BY time() clause overwrite timestamps in this way.

The only exception is the SAMPLE() function. See the SAMPLE() documentation for additional information.

Unexpected consistency with ELAPSED() and the GROUP BY time() clause

Q: I’m using ELAPSED() to calculate the time gap between median values in my data. The results I get (see below) always show the same time gap. I’m having a hard time believing that my median values are always exactly ten minutes apart. Am I using the ELAPSED() function correctly?

> SELECT ELAPSED(MEDIAN("clicks"),1m) FROM "puppies" WHERE time >= '2017-04-12T18:30:00Z' AND time <= '2017-04-12T19:10:00Z' GROUP BY time(10m)

name: puppies
time                  elapsed
----                  -------
2017-04-12T18:40:00Z  10
2017-04-12T18:50:00Z  10
2017-04-12T19:00:00Z  10

A: You’re using the ELAPSED() function correctly from a syntax perspective but your query wont give you the results that you’re looking for. When you use ELAPSED() with a nested function and a GROUP BY time() clause, InfluxDB always returns elapsed values that are equal to the GROUP BY time() interval.

When you run your query, InfluxDB first calculates the results for the nested function (MEDIAN()) at the specified GROUP BY time() intervals and then it applies the ELAPSED() function to those results. In the first step, the GROUP BY time() clause automatically overrides MEDIAN()’s original timestamps with timestamps that mark the start of the each GROUP BY time() interval. When the system applies the ELAPSED() function to the results of that first step, it encounters timestamps that are always ten minutes apart and not the original timestamps of the MEDIAN() values.

What's next:

Downloads for the TICK-stack are live on our "downloads" page.
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial. of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring, and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 300 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition hoodie as a thank you.

InfluxData Blog - Regan Kuchan

TL;DR Tech Tips: New Query Language Features in Version 1.3

New Operators

New time zone clause

New duration unit

What's next:

TL;DR Tech Tips: New InfluxQL Functions in Version 1.3

New Function: INTEGRAL()

New Function: NON_NEGATIVE_DIFFERENCE()

Updated Functions: TOP() and BOTTOM()

What's next:

TL;DR Tech Tips: Managing the Work History in Chronograf

Managing alert history in Chronograf

1: Locate the chronograf database and click on the infinity symbol (?)

2: Enter a different duration

3: Click the green check mark to save your changes

Writing data with Chronograf

What's next:

TL;DR InfluxDB Tech Tips: Identical Field Keys & Tag Keys, Existing Kapacitor TICKscripts in Chronograf & More

Identical field keys and tag keys

Field keys or tag keys called time

Existing Kapacitor TICKscripts in Chronograf

What's next:

TL;DR InfluxDB Tech Tips: Configuring Telegraf & Kapacitor When auth is Enabled on InfluxDB & Tips on Chronograf

Understanding Chronograf's status icon

Configuring Telegraf and Kapacitor when authentication is enabled on InfluxDB

Introducing Chronograf's visualization types

What's next:

TL;DR InfluxDB Tech Tips - Template Variables, Query Building, & User Management in Chronograf

Template variables in Chronograf

Query building in Chronograf

InfluxDB user management in Chronograf

Grant READ permission:

Grant WRITE permission:

Grant ALL permission:

What's next:

TL;DR InfluxDB Tech Tips - Querying Data in a Non-DEFAULT Retention Policy, Checking Field Types, and Querying vs. Writing Booleans

Querying data in a non-DEFAULT retention policy

Checking field types

Querying vs. writing booleans

What's next:

TL;DR InfluxDB Tech Tips - Chronograf's Template Queries and Function Documentation's Make-over!

Chronograf has template queries

Function documentation gets a make-over

Tech tips get a webinar

What's next:

TL;DR InfluxDB Tech Tips - Missing Data with DISTINCT() & the INTO Clause, Unexpected Timestamps with MAX() & More

Missing data with DISTINCT() and the INTO clause

Unexpected timestamps with MAX() and the GROUP BY time() clause

Unexpected consistency with ELAPSED() and the GROUP BY time() clause

What's next:

1: Locate the `chronograf` database and click on the infinity symbol (?)

Field keys or tag keys called `time`

Grant `READ` permission:

Grant `WRITE` permission:

Grant `ALL` permission: