InfluxData Blog - Nathaniel Cook

What to Expect from Flux 1.0

Nathaniel Cook (InfluxData) — Thu, 03 Nov 2022 07:00:00 +0000

(Update: InfluxDB 3.0 moved away from Flux and a built-in task engine. Users can use external tools, like Python-based Quix, to create tasks in InfluxDB 3.0.)

This week at InfluxDays we announced that Flux 1.0 is coming soon. Version 1.0 of Flux lang is a commitment to no longer make breaking changes to the Flux language. Importantly, today’s Flux scripts will work on Flux 1.0, and no breaking changes will be introduced between now and the release of Flux 1.0. Along with version 1.0, we have some features we are also releasing soon. Here are the features we have coming and a short explanation of why you might want to leverage them:

Modules: Share Flux code within your organization and build useful abstractions over common queries.
Editions: Opt into new features of Flux without the risk of breaking changes.
Label Polymorphism: Write Flux code that can abstract over the schema of your data.
Dynamic Type: Work with JSON data that is fully dynamic without limitations.

The Dynamic Type feature is available today and you can try it out in the cloud. Now let’s dive into some of these features in more detail.

How to use modules

Flux modules are a system that allows you to publish Flux code to a registry that’s private to your organization. Then you can import that code and reuse it anywhere from tasks to dashboard cells and within your application’s Flux code.

Graphic illustrating Flux modules and sharing code

Here’s an example of how modules will work once they are released. A common use case we see is that organizations will tag their data with a value that really represents much more data than is present in the tag alone. For example, you might have an env tag on your data that represents the data center where the metrics originated. There are properties of that environment not present in other tags, for example which cloud provider hosts that environment or whether the environment is production or not. Using modules, you can encode this information into a Flux dictionary and then leverage it to annotate data with that extra information.

The first step to using modules is to define a Flux script:

// Notice we give this script a package name since it will be part of a module.
package env

import "dict"

// Map envs to their cloud provider.
providers =
	[
    	"prod01": "aws",
    	"prod02": "aws",
    	"prod03": "gcp",
    	"prod04": "azure",
    	"stag01": "aws",
    	"stag02": "gcp",
	]

// Map envs to their purpose.
// We could just regex match on the name,
// but sometimes it’s important to be explicit.
kinds =
	[
    	"prod01": "production",
    	"prod02": "production",
    	"prod03": "production",
    	"prod04": "production",
    	"stag01": "staging",
    	"stag02": "staging",
	]

// annotate adds extra information about the env to the table.
//
// ## Parameters
// - tables: Input data. Default is piped-forward data (`<-`).
//
annotate = (tables=<-) =>
	tables
    	|> map(
        	fn: (r) =>
            	({r with provider: dict.get(dict: providers, key: r.env, default: "unknown"),
                	kind: dict.get(dict: kinds, key: r.env, default: "unknown"),
            	}),
    	)

Then we publish this module using the API. More details on the publish process will be coming with the release of modules. After publishing, we can import this module in a dashboard cell and make it easy to group the data by cloud provider and filter out the staging environments.

import "modules/env"

// Render median request duration grouped by provider in only production envs
from(bucket: "request")
	|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
	|> filter(fn: (r) => r._measurement == "duration")
	|> env.annotate()
	|> filter(fn: (r) => r.kind == "production")
	|> group(columns: ["provider"])
	|> aggregateWindow(every: v.windowPeriod, fn: median)

Editions and breaking changes

Editions are a key component of the Flux 1.0 release. With 1.0 we will no longer make breaking changes to Flux. However, on occasion there are valuable features that we should add to Flux that would constitute breaking changes. We need a way to safely add these features without breaking Flux scripts. This is where editions come in. An edition is a set of features that are disabled by default. You can opt-in to an edition which will enable those features, but if you do not opt-in nothing will change and your scripts will continue to work without breaking. Because editions are opt-in, you can update your scripts before updating to a new edition to ensure they continue to work as expected. An edition can be selected on a per-script basis or enabled via the API or organization-level settings, providing flexibility in upgrading scripts.

Creating new editions will be a rare event and we anticipate creating at most one new edition per year. Therefore we are naming editions after the year in which they are created. The first edition of Flux will be called 2022.1. We’ll likely cut a second edition of Flux next year and it will be called 2023.1.

Users can enable a new edition by specifying it in their scripts like this:

// Enable a specific edition with this line as the first line in the script
#edition("2023.1")

// The rest of the script follows

What to expect from label polymorphism

Label polymorphism is one of the first features that we will release behind an edition because the feature adds a new syntax to Flux that is breaking.

Label polymorphism allows users to abstract over the schema of their data. Have you ever wanted to write a Flux function that takes a column name as an argument and then uses that name dynamically within the function? This has been a long requested feature of Flux and it’s coming soon as part of the second edition of Flux.

Let’s look at the same example we used earlier showing modules. We wrote this function in the env package:

// annotate adds extra information about the env to the table.
//
// ## Parameters
// - tables: Input data. Default is piped-forward data (`<-`).
//
annotate = (tables=<-) =>
	tables
    	|> map(
        	fn: (r) =>
            	({r with provider: dict.get(dict: providers, key: r.env, default: "unknown"),
                	kind: dict.get(dict: kinds, key: r.env, default: "unknown"),
            	}),
    	)

If we look closely at this function, we realize that it hard-codes a lot of information about the schema of the data. It assumes that the env tag can be accessed as r.env, and additionally it hard-codes the columns provider and kind on the output table. What if we have data that comes from inconsistent systems that sometimes use env and other times use environment? We can use labels to abstract away which columns we are using like this:

// We need to enable the edition in order to use labels
#edition("2023.1")

// annotate adds extra information about the env to the table.
//
// ## Parameters
// - env: Label for the environment column. Default `env`.
// - provider: Output label for the provider column. Default `provider`.
// - kind: Output label for the kind column. Default `kind`.
// - tables: Input data. Default is piped-forward data (`<-`).
//
annotate = (tables=<-, env=.env, provider=.provider, kind=.kind) =>
	tables
    	|> map(
        	fn: (r) =>
            	({r with [provider]: dict.get(dict: providers, key: r[env], default: "unknown"),
                	[kind]: dict.get(dict: kinds, key: r[env], default: "unknown"),
            	}),
    	)

There are three new syntax elements here. First we have .env which defines a label with the value env and the . which tells Flux it’s a label. We use a label value as the default for each of the three column parameters we added to the function. Second, we see r[env]. This is how we access the env column on the record r. Finally, in the output record we see [provider] and [kind], which adds columns on the output record based on the values of the provider and kind labels.

Here’s what it looks like to call this new function, given we are working with a legacy system that uses environment instead of env and already has a tag named kind.

// We need to enable the edition in order to use labels
#edition("2023.1")

import "modules/env"

// Render median request duration grouped by provider in all production envs
from(bucket: "legacy_request")
	|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
	|> filter(fn: (r) => r._measurement == "duration")
	// We can specify the column names when calling the annotate function
	|> env.annotate(env: .environment, kind: .env_kind)
	|> filter(fn: (r) => r.env_kind == "production")
	|> group(columns: ["provider"])
	|> aggregateWindow(every: v.windowPeriod, fn: median)

Neither labels nor edition 2023.1 have been released yet, so the above script will error on existing Flux, but keep an eye out for these features coming soon.

Dynamic type for JSON data

Dynamic type is another new type added to Flux that allows any JSON data to be accurately represented with Flux data types. Flux is a statically typed language that requires container types (i.e. arrays, records, and dictionaries) to contain data of a consistent type. For example, a list of integers is a consistent type, but a list of strings and integers is not. JSON doesn’t have these restrictions, and with the new dynamic type feature we can represent JSON within Flux correctly. Flux is still statically typed, but we’ve added a new type called “dynamic” that allows Flux to handle dynamically typed data.

For example, this JSON was not possible to represent in Flux previously:

{
  "items":  [
  	{"name": "a", "pos": {"x": 10, "y": 10}},
  	{"name": "b", "pos": {"x": 30}},
  	{"name": "c", "pos": {}},
  	{"name": "d"}
	]
}

Unlike these other features, dynamic type is available today.

You can now parse the above JSON without error using the new experimental/dynamic package.

import "experimental/dynamic"

parsed = dynamic.parseJSON(data: jsonDataAsBytes)

This is possible because the function dynamic.parseJSON returns a Flux value with type dynamic.

A dynamic value represents a value whose type is not known until runtime. Flux also can have containers (i.e. arrays etc.) of dynamic values.

Dynamic values aren’t allowed inside Flux tables, so we explicitly convert a dynamic value to a value with a statically known type. This conversion makes use of the array package helper functions and some new functions in the dynamic package.

import "array"
import "experimental/dynamic"

parsed = dynamic.parseJSON(data: jsonDataAsBytes)

// parsed is dynamic so parsed.items is also dynamic
parsed.items
	// convert the plain dynamic value to an array of dynamic values
	|> dynamic.asArray()
	// map over each dynamic value in the array and construct a record
	|> array.map(fn: (r) => ({
    	name: string(v: x.name),
    	posX: int(v: x.pos.x),
    	posY: int(v: x.pos.y),
	}))
	// convert the array of records (no longer containing any dynamic values) into a table
	|> array.from()

For the JSON rows that did not have a pos value, the table will contain nulls.

Final thoughts

There’s a lot included in Flux 1.0. Many new features are coming soon and dynamic type is available today. Keep an eye out for the Flux 1.0 release and new editions of Flux. In the meantime, you can check out my session from InfluxDays 2022.

Flux and Static Analysis

Nathaniel Cook (InfluxData) — Mon, 01 Jun 2020 07:00:55 -0700

Have you seen this when using the Flux code editor

Pretty neat to be able to get that much help from an editor while writing code.

Have you ever wondered how that worked? Today, I’ll take us on a bit of a deep dive on the behind-the-scenes that enable these autocompletion features in the editor.

How does a code editor know enough about the code you write in order to help you out

The answer is something called static analysis through which we can analyze a Flux script and learn lots of information about what the code does without actually running the code. It is important that we do not have to run the code to learn about it because - as the author of the code - you may not be ready to run the code yet as it may edit a file or write data to a database. That is why the technique is called static analysis; the analysis is performed on only static inputs, i.e. the Flux code.

What do we want to learn about the code when we analyze it

The primary information we are trying to learn when we analyze the code are the types of the expressions in the code. For example, if you have this line in the code threshold = 42, we learn that there is a variable named threshold and that its type is an integer. In the animation above, you can see that the editor was able to suggest the arguments to the aggregateWindow function. This is possible because we know the type of the function aggregateWindow and the type includes the information about what arguments the function accepts.

Types are a way of specifying which values make sense in what contexts. So when you type aggregateWindow we know that the name aggregateWindow is a variable in the program scope that has the function type, so we can make suggestions for how to complete the rest of the code using that type.

How does static analysis algorithm learn the types of the expressions and variables in the code

We use a static analysis technique called type inference, which is the idea that looking at the code you can infer the types based on how you see the code used. For example given this code, what do you think the type of the m variable is

m = "Hello World"
print(m)

The variable m is a string. It is clear from the code even though the word string is not present in the code. Contrast that code with the Go code equivalent:

var string m = "Hello World"
print(m)

The string keyword tells the Go compiler that the variable m has the string type.

Type inference is the idea that Flux can infer the type of the values within the program based on the context that those values are used. This means that authors of the code do not need to explicitly state the types of variables in their code.

If you are familiar with Go, you may be thinking that you can use the := syntax instead. That is correct and is in fact a limited case of type inference within Go as well.

Flux does not require type annotations in the code because it can infer the type of any value without them.

While the above examples are simple to infer the types, in general it’s not easy to determine what the types are for an arbitrary Flux script by just looking at it. In order to consistently and accurately determine the types in any Flux program, we need a set of rules (i.e. an algorithm) to define how types can be inferred. The Flux team uses a specific algorithm for type inference known as Algorithm W.

The specifics of Algorithm W are very nuanced, so we will hand-wave away most of those details. Instead, we will cover here just the raw structure of the algorithm.

To understand how Algorithm W works, let’s first look at another problem that may be familiar to you. Remember algebra and solving a system of equations? It’s those problems where they gave two equations that had two variables, and you had to solve for both variables. Those problems are solved in a very similar manner to how Flux solves for the types in a program using Algorithm W. For example, given these two equations with variables x and y, you can solve for the variables.

2x + y = 5
y = 2 + x

The process of solving these variables involves substituting one variable equation into the other equations. We have an equation of what y is equal to, so we can substitute all values of y in the first equation with the right-hand side of the y = equation like this:

2x + (2 + x) = 5

Now that we have only a single variable x in the equation, we solve for x using the rules of algebra. Here are the steps explicitly:

2x + (2 + x) = 5
3x + 2 = 5
3x = 3
x = 1

We have solved for x and learned its value is 1. Now we can use the equation for y to determine its value.

y = 2 + x
y = 3

Using this process, we have solved for the values of x and y.

Type inference is similar except that instead of solving a system of algebraic equations, we solve equations about types. We use substitution and the rules of the types to solve for the type of every expression in a Flux program.

Here is an overview of the process:

?Assign a type variable to each Flux expression in the program.
Generate all the equations about the types.
Solve that set of equations.

First, for each expression in a Flux script, we assign it a new type variable. (We call them type variables because they represent types; we do not know which type until we solve for them.)

Our goal is to solve for those type variables. To do so, we need some equations. We get those equations by looking at how each type variable is used within the Flux program.

Once we have collected all the equations, we can solve them. We do this the same way we solve an algebraic equation, by rearranging the equations according to the rules and using substitution to replace variables in the equations until we have a simple X = Type equation for each type variable.

Here is a simple hello world Flux program that will print “hello world” when run via the REPL.

a = "hello "
b = "world"
a + b

We can follow the process that type inference takes to know the types of all of the expressions in this program. I have chosen this example because all expressions in this program have the type string, and this makes it easy to see the solution we are working towards as we take each step.

The first step is to assign the type variables to the expressions. This program has three expressions: one for each line. Let’s give these type variables the names X, Y and Z for each line in order.

a = "hello " // X
b = "world"  // Y
a + b        // Z

Notice that we are talking about two different kinds of variables:

Variables in the Flux program itself like a and b
Type variables like X, Y and Z

To help differentiate between the two, I will use upper case letters for type variables and lowercase letters for program variables.

Next, we need to gather all the equations we know about those type variables.

First, we have the type variable X with the expression:

a = "hello "

Since we see that only a string literal is used, we know that the expression must have a string type. We can write down an equation for X stating that X must be equal to the type string.

X = string

We also see that we assigned a value to the program variable a, so we write down the type variable that is associated with the normal variable like this:

a -> X

We track the association of the program variables to their corresponding type variables so we can look them up whenever they are used in other expressions.

The next expression for type variable Y is:

b = "world"

This is the same as X from a type perspective; therefore, we get the following equation and type variable association.

Y = string
b -> Y

Finally, we need to write down an equation for Z, remember its expression:

a + b

This expression doesn’t have any hints as to what type it is because only program variables are used. However, we can still learn something useful from the expression.

We know that both sides of the + operator must be the same type. This is what we mean by using the rules of types: certain operators must be used in specific ways, and we write down equations to capture those constraints.

Since we don’t know the type yet, we create a new type variable W as a placeholder for now. Now, we can write down some more equations using W and the lookup function to indicate we need to lookup the association of a program variable to its type variable.

W = lookup(a)
W = lookup(b)
Z = W

We can lookup the type variable for a and b and update the equations.

W = X
W = Y
Z = W

Now we have examined all the expressions in the program, and we can move onto the next step to solve those equations. This part proceeds in the same way as solving the algebraic equations. We start substituting variables for their equation until we learn the type of every variable.

Here are all the equations we know at this point:

X = string
Y = string
W = X
W = Y
Z = W

We can substitute the type variables X and Y and get these simplified equations.

W = string
W = string
Z = W

Now we know a value for W and can substitute it in the equation for Z like so:

Z = string

We now have a set of equations and have solved for all type variables.

X = string
Y = string
Z = string
W = string

We can now say that all expressions in the Flux script are strings as we expected when we first saw the program.

?Taking a step back, we can start to see how to systematically solve for the types of expressions within a Flux program. We can also see how we can find errors in programs. For example, take this slightly modified example program:

a = "hello "
b = 1
a + b

When creating the type equations for this program, we would get the following:

X = string
Y = integer
W = X
W = Y
Z = W

Using substitution, we would get:

W = string
W = integer

At this point, we have discovered a type error because W cannot be both equal to a string and an integer. When there is not a valid solution to the type equations, then there is a type error in the program. This error is valuable because it tells us about a mistake in our program before we ran it.

That’s type inference in a nutshell; generate a bunch of equations based on how the expressions are used within the Flux program and then solve that set of equations for all the type variables. Hopefully, this has shed some light on how we perform static analysis (i.e. type inference) in order to learn the types in a Flux program. Armed with the knowledge of types, an editor can then use those types to provide helpful suggestions and error messages when developing Flux scripts.

Want to learn more about type inference

Naturally, type inference is more complex in practice since even simple Flux programs have lots of expressions, and therefore, lots of equations to solve. Solving a large number of equations can be very difficult to do efficiently and accurately. Imagine trying to solve a system of hundreds of algebra equations. That would be very tedious and easy to make a mistake.

Additionally, as you may remember from algebra class, picking which equations to substitute into which other equations can make solving the problem either easy or extremely difficult. For algebra this becomes a bit of an intuitive art that students get good at with practice. Unfortunately, computers are not good at intuition. That is where Algorithm W comes in. It is a specific algorithm for how to generate and solve type equations efficiently and correctly.

If you want to learn more, there are decades of research and literature on these topics. Here are a few pointers to get you started.

Algorithm W is a class of type inference algorithms known as Hindley-Milner.
We also use an extension to Algorithm W for extensible records see the paper.
Try out the language SML it has a very simple type system based on Hindley-Milner and is a quick language to learn. It's similar to Haskel and OCaml but drastically easier to learn and simpler to understand.
There are several implementations of various type inference algorithms along with their research in a GitHub repo.

I personally found no shortcuts in trying to learn type inference; most of the literature assumes some basic understanding of the concepts of types and computational theory. I would start with lambda calculus as all literature about Hindley-Milner assumes familiarity with lambda calculus. Then follow up with experimentation and practice on your part.

Additionally, I haven’t mentioned yet the Flux language server which is responsible for using the type information learned from static analysis and translating that into suggestions the editor can then present. It’s the critical piece that makes this all work. Check out the Flux LSP implementation on GitHub.

That’s all for today, thanks for following along.

Announcing Flux (formerly IFQL) v0.0.5 with 12+ New Functions

Nathaniel Cook (InfluxData) — Tue, 27 Feb 2018 11:31:35 -0700

This release represents a big leap forward for Flux.

In the first part of the release cycle, we focused on formalizing the way query operations are chained within Flux. Once that was complete we tried it out, and quickly found that it was easy to add new functions to Flux by composing existing functions together. We were able to add nearly a dozen functions to the language using only Flux code—no Go code was required. In addition we added a handful of new functions backed by specialized Go implementations.

The new functions in the release include:

top
bottom
highestMax
highestAverage
highestCurrent
lowestMin
lowestAverage
lowestCurrent
stateCount
stateDuration
distinct
percentile
median

From that list only stateCount, stateDuration, percentile, and distinct required specialized Go implementations. The rest are pure Flux functions.

This large batch of functions is in part due to how we have formalized the way functions are chained in Flux. Every query function accepts a table object and returns a table object. In order to chain them, the functions are called using the return values of the previous functions.

For example, to chain a range operation off the from operation it looks like this:

range(table:from(db:"telegraf"), start:-1m)

Nesting functions in this way has the potential to result in a very long query which is difficult to read and even harder to debug. That is why we have introduced a new syntax to the language to make it easier to describe these chains of operations. The pipe forward operator looks like this |> and allows chaining functions by passing the left hand result to the right hand function call.

// These two lines are equivalent
range(table:from(db:"telegraf"), start:-1m)
from(db:"telegraf") |> range(start: -1m)

Then to apply a filter to the above operations simply add a new |> and filter function call.

from(db:"telegraf") |> range(start: -1m) |> filter(fn: (r) => r._measurement == "mem")

By using this simple model it becomes easy for new functions to be added to the language (both built-in and by the user), that are really just a combination of existing functions. For example, the top operation is a combination of the sort and limit operations. As such the top function can be implemented using this small Flux code snippet:

top = (n, cols=["_value"], table=<-) =>
    table
        |> sort(cols:cols, desc:true)
        |> limit(n:n)

The top function is defined as having three arguments n, cols, and table.

The n argument specifies how many records the result should contain.

The cols argument specifies the list of columns on which to sort the table—it has a default of ["_value"].

The table argument is the source table on which to apply the operation. The table’s default value is the special <- syntax, meaning that it comes from the left hand result of the pipe forward |> operator.

The function body simply chains the sort and limit functions off the table with the provided arguments.

Many of the new functions in this release have been implemented using a similar strategy. If you want to learn more about these more advanced features check out the Flux README as it details what is going on here. Also, some of you may be concerned that implementing top as a sort and limit operation is expensive and slow. You are right, but do not worry as the query planner is able to find these kinds of patterns and rewrite them with a specialized implementation that is efficient.

I’d like to highlight one other new function in this release, the percentile function. It has long been a requested feature to be able to compute percentiles more efficiently using an approximation method instead of computing exact values.

The Flux percentile function by default will return a t-digest approximation unless explicitly asked to compute the exact percentile. The t-digest is an efficient and accurate method for approximating percentiles. Using Flux, the default will be to compute results efficiently by returning good approximations and returning exact results when specifically requested.

This has been a busy release with lots of great additions and development – check out the complete CHANGELOG for all the details. Thanks to all who contributed valuable input throughout the design process this week. We are excited to hear your feedback on the Github project.

Announcing Flux (formerly IFQL) v0.0.4

Nathaniel Cook (InfluxData) — Mon, 22 Jan 2018 09:00:41 -0700

We recently released v0.0.4 of IFQL - Influx Query Language [now called Flux]. With the holidays at the end of the year, we were a little slow getting this release out. Going forward, we expect to release every two weeks. This release has some exciting developments, focused almost entirely in new features for the Flux language itself. Here is a quick list of highlights:

Support for defining/calling functions within Flux
Several new built-in functions: map, shift, integral, derivative and difference
Support for multiple values, with the covariance function being the first function to use multiple values

Adding support for functions into the language enables users to create small reusable components in their queries, building up libraries over time of useful snippets. Functions are also used when logic needs to be applied to the records of the data. Functions replace the previous expressions with a clear well-scoped function to evaluate. We hope you find them to be a clear and powerful tool. A Flux function follows this syntax:

(parameter list) => <function body>

Example function which adds two values:

// define the function add
add = (a,b) => a + b
// call the add function
add(a:1, b:5) // 6

Here is an example of functions in action:

// serviceData gets the field from the requests measurement for a given service.
serviceData = (service,field) =>
    from(db:"myapp")
        // Use an anonymous function as the filter predicate, where "r" is the record.
        .filter(fn: (r) => r._measurement == "requests" and r._service == service and r._field == field)
// We want the last hour of data
start = -1h
// Get the requests for the "cart" service
reqs = serviceData(service:"cart", field:"requests").range(start:start)
// Get the errors for the "cart" service
errs = serviceData(service:"cart", field:"errors").range(start:start)
// Compute the correlation between the reqs and errs for the "cart" service
pearsonr(x:reqs, y:errs, on:["region"])

The above script will compute the pearson correlation coefficient between errors and requests of the cart service for the past hour. Interestingly the pearsonr function is defined using native Flux functions and is built into the language. Here is a look at the definition of pearsonr as it appears in the source.

// Covariance computes the covariance between x and y.
covariance = (x,y,on,pearsonr=false) =>
    join(
        tables:{x:x, y:y},
        on:on,
        fn: (t) => ({x:t.x._value, y:t.y._value}),
    )
    .covariance(pearsonr:pearsonr)


// Pearsonr computes the Pearson R correlation coefficient of x and y.
pearsonr = (x,y,on) => covariance(x:x, y:y, on:on, pearsonr:true)

The pearsonr function is really just the covariance function with the pearsonr argument set to true. The covariance function in turn is a join operation which joins the x and y tables before passing the data to the covariance method which computes the actual covariance between the multiple values on the joined table. Over time, more functions will be added to the language, and we hope that these built-in functions can be a good source of examples for new users.

We are excited about the power that functions and multiple value support will bring to Flux. Have any feedback or want to see support for a certain function? Open an issue or pull request on the Flux repo. For a full list of changes in v0.0.4, see the CHANGELOG.

Announcing Flux (formerly IFQL) v0.0.3

Nathaniel Cook (InfluxData) — Wed, 13 Dec 2017 06:05:36 -0700

The Flux (formerly IFQL) team releases a new version every two weeks. We want to get our changes into the hands of our community quickly so we can hear of your experiences early. Please submit issues and PRs on the repo. Check out the Flux release announcement for details on the vision behind Flux .

These last two weeks we have focused on performance and resource management (i.e. managing CPU and memory resources so that a single query cannot overwhelm the Flux process).

Benchmarking

We have put together benchmarks comparing Flux to InfluxQL. These benchmarks have been put in place early so that we can ensure going forward in continuing to maintain strong performance. Here are some initial charts showing where Flux is compared to equivalent InfluxQL queries.

Five representative queries have been run on both systems¹. To see details of the benchmarks, check out the PR.

Here are the five queries:

Query #1

IFQL - from(db:"db").range(start:2017-11-01T00:00:00Z, stop:2017-11-02T00:00:00Z).filter(exp:{"_measurement" == "m0" and "_field" == "v0" and $ > 0}).sum()
InfluxQL - SELECT sum(v0) FROM m0 WHERE time >= 2017-11-01T00:00:00Z AND time < 2017-11-02T00:00:00Z AND v0 > 0 GROUP BY *

Query #2

IFQL - from(db:"db").range(start:2017-11-01T00:00:00Z, stop:2017-11-05T00:00:00Z).filter(exp:{"_measurement" == "m0" and "_field" == "v0" and $ > 0}).sum()
InfluxQL - SELECT sum(v0) FROM m0 WHERE time >= 2017-11-01T00:00:00Z AND time < 2017-11-05T00:00:00Z AND v0 > 0 GROUP BY *

Query #3

IFQL - from(db:"db").range(start:2017-11-01T00:00:00Z, stop:2017-11-05T00:00:00Z).filter(exp:{"_measurement" == "m0" and "_field" == "v0" and $ > 0}).group(by:["tag0"]).sum()
InfluxQL - SELECT sum(v0) FROM m0 WHERE time >= 2017-11-01T00:00:00Z AND time < 2017-11-05T00:00:00Z AND v0 > 0 GROUP BY tag0

Query #4

IFQL - from(db:"db").range(start:2017-11-01T00:00:00Z, stop:2017-11-13T00:00:00Z).filter(exp:{"_measurement" == "m0" and "_field" == "v0" and $ > 0}).sum()
InfluxQL - SELECT sum(v0) FROM m0 WHERE time >= 2017-11-01T00:00:00Z AND time < 2017-11-13T00:00:00Z AND v0 > 0 GROUP BY *

Query #5

IFQL - from(db:"db").range(start:2017-11-01T00:00:00Z, stop:2017-11-13T00:00:00Z).filter(exp:{"_measurement" == "m0" and "_field" == "v0" and $ > 0}).group(by:["tag0"]).sum()
InfluxQL - SELECT sum(v0) FROM m0 WHERE time >= 2017-11-01T00:00:00Z AND time < 2017-11-13T00:00:00Z AND v0 > 0 GROUP BY tag0

First, let’s look at execution time as that is most important when considering the performance of a query engine.

For most queries, Flux is already faster than InfluxQL. Note Query #4 did not complete using InfluxQL and so has no time. We have an issue open already that should improve Query #5.

Next examining the heap usage of the processes, we can see that Flux is significantly more efficient in its usage.

The heap usage of both ifql processes barely registers in comparison to the heap usage of InfluxQL. Note, there are two ifql processes, ifqld which can be any number of sidecar processes, and influxd running with the new IFQL API enabled.

Resource Management

With version v0.0.3, it is now possible to limit the resources a query consumes. Our goal is to ensure that primarily a really hungry query can’t crash the process and secondarily the hungry query will fairly share resources with other concurrent queries according to its priority.

Now with these limits in place, a bad query that tries to consume too much memory is killed and an appropriate error is returned, instead of allowing the query to crash the process.

We also ran tests comparing v0.0.2 with v0.0.3. The test submitted more queries than the processes had resources. We then compared the execution times of the queries and could see that v0.0.3 reduced the overall execution time by 15%. This is accomplished by queuing and scheduling the work instead of letting all the queries concurrently fight over resources. The long tail latencies increase in v0.0.3 because of the queueing, but we have a plan to solve that issue later.

Overall v0.0.3 is a solid iterative release, and we will see you again in two weeks with another release. For a complete list of all changes see the CHANGELOG.

Footnotes:

[1] Note that InfluxDB master now has a Prometheus “/metrics” endpoint we used to collect these stats. We also used a new tool ingen to create the test datasets.

Announcing Kapacitor 1.0 - A Data Processing Engine for InfluxDB

Nathaniel Cook (InfluxData) — Thu, 08 Sep 2016 03:15:47 -0700

Kapacitor 1.0 GA is here. Kapacitor is the brains of the TICK stack. You can leverage Kapacitor to process your data for various business needs and use it to find changes or anomalies within your time series data. We have come a long way since the 0.13.1 release of Kapacitor back in mid May 2016. Since then, we’ve added 33 new features and 42 bug fixes. We had many PRs from the community, from simple bug fixes to major features. Thanks to everyone for helping improve Kapacitor!

What's new?

Of the new features in 1.0 there are eight we want to highlight, three of which are community contributions…

HTTP based subscriptions: Goodbye to the complexity of managing a unique UDP port per database, HTTP subscription are here and provide a simpler, more reliable transport from InfluxDB to Kapacitor.
Template Tasks: You can now define templates so that multiple tasks that share common behavior can easily be managed together.
Holt-Winters Forecasting: Start using Kapacitor for more complex anomaly detection tasks, Holt-Winters is the first of many powerful algorithms to be added to the TICK stack.
Alert Reset Expressions: Noise from alerts is a plague. Thanks to @minhdanh, you can now define reset expressions for your alert levels to reduce alert noise.
Group By Fields: Convert any field into a tag so that you can use it to group your data streams.
Telegram Alerting: You can now send messages to Telegram thanks to @burdandrei.
Better and Faster Lambda Expressions: @yosiat greatly improved the performance of lambda expressions and added support for conditional logic.
Live Replays: Replay data directly from InfluxDB without going through any intermediate steps.

Beyond 1.0

On the roadmap are improvements to alert management, UDF snapshots and more powerful algorithms. We plan to add API endpoints where you can query the state of alerts. Snapshots are going to be revamped so that your UDFs can easily save their state and even restore from a previous version. This will make saving your trained models a simple managed process within Kapacitor. For more details on what we are planning check out the planned milestones.

What's next

Download: 1.0 GA downloads for the TICK-stack are live on our "downloads" page
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 100 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition InfluxDB hoodie as a thank you.

Combining Kapacitor and Continuous Queries

Nathaniel Cook (InfluxData) — Tue, 10 May 2016 06:00:56 -0700

Kapacitor can be used to do the same work as Continuous Queries in InfluxDB. Today we are going to explore reasons to use one over the other, Kapacitor vs Continuous Queries, and the basics of using Kapacitor for CQ type workloads.

What’s Kapacitor? It’s the “K” in the TICK stack. Kapacitor is InfluxDB’s native data processing engine. It can process both stream and batch data from InfluxDB. Kapacitor lets you plug in your own custom logic or user defined functions to process alerts with dynamic thresholds, match metrics for patterns or compute statistical anomalies.

An Example

First, lets take a simple CQ and rewrite it as a Kapacitor TICKscript.

Here is a CQ that computes the mean of the cpu.usage_idle every 5m and stores it in the new measurement mean_cpu_idle.

CREATE CONTINUOUS QUERY cpu_idle_mean ON telegraf BEGIN SELECT mean("usage_idle") as usage_idle INTO mean_cpu_idle FROM cpu GROUP BY time(5m),* END

To do the same with Kapacitor here is a streaming TICKscript.

stream
    |from()
        .database('telegraf')
        .measurement('cpu')
        .groupBy(*)
    |window()
        .period(5m)
        .every(5m)
        .align()
    |mean('usage_idle')
        .as('usage_idle')
    |influxDBOut()
        .database('telegratelegraff')
        .retentionPolicy('default')
        .measurement('mean_cpu_idle')
        .precision('s')

The same thing can also be done as a batch task in Kapacitor.

batch
    |query('SELECT mean(usage_idle) as usage_idle FROM "telegraf"."default".cpu')
        .period(5m)
        .every(5m)
        .groupBy(*)
    |influxDBOut()
        .database('telegraf')
        .retentionPolicy('default')
        .measurement('mean_cpu_idle')
        .precision('s')

All three of these methods will produce the same results.

Questions

At this point there are a few questions we should answer:

When should we use Kapacitor instead of CQs?
When should we use stream tasks vs batch tasks in Kapacitor?

When should we use Kapacitor instead of CQs?

There are a few reasons to use Kapacitor instead of CQs.

You are performing a significant number of CQs and want to isolate the work load. By using Kapacitor to perform the aggregations InfluxDB's performance profile can remain more stable and isolated from Kapacitor's.
You need to do more than just perform a query, for example maybe you only want to store only outliers from an aggregation instead of all of them. Kapacitor can do significantly more with the data than CQs so you have more flexibility in transforming your data.

There are a few use cases where using CQs almost always makes sense.

Performing downsampling for retention policies. This is what CQs are designed for and do well. No need to add another moving piece (i.e. Kapacitor) to your infrastructure if you do not need it. Keep it simple.
You only have a handful of CQs, again keep it simple, do not add more moving parts to your setup unless you need it.

When should we use stream tasks vs batch tasks in Kapacitor?

Basically the answer boils down to two things, the available RAM and time period being used.

A stream task will have to keep all data in RAM for the specified period. If this period is too long for the available RAM then you will first need to store the data in InfluxDB and then query using a batch task. A stream task does have one slight advantage in that since its watching the stream of data it understands time by the timestamps on the data. As such there are no race conditions for whether a given point will make it into a window or not. If you are using a batch task it is still possible for a point to arrive late and be missed in a window.

Another example: Create a continuous query to down sample across retention policies

CREATE CONTINUOUS QUERY cpu_idle_median ON telegraf BEGIN SELECT median("usage_idle") as usage_idle INTO "telegraf"."sampled_5m"."median_cpu_idle" FROM "telegraf"."default"."cpu" GROUP BY time(5m),* END

The stream TICKscript:

stream
    |from()
        .database('telegraf')
        .retentionPolicy('default')
        .measurement('cpu')
        .groupBy(*)
    |window()
        .period(5m)
        .every(5m)
        .align()
    |median('usage_idle')
        .as('usage_idle')
    |influxDBOut()
        .database('telegraf')
        .retentionPolicy('sampled_5m')
        .measurement('median_cpu_idle')
        .precision('s')

And the batch TICKscript:

batch
    |query('SELECT median(usage_idle) as usage_idle FROM "telegraf"."default"."cpu"')
        .period(5m)
        .every(5m)
        .groupBy(*)
    |influxDBOut()
        .database('telegraf')
        .retentionPolicy('sampled_5m')
        .measurement('median_cpu_idle')
        .precision('s')

Summary

Kapacitor is a powerful tool, if you need more power use it. If not keep using CQs until you do. For more information and help writing TICKscripts from InfluxQL queries take a looks at these docs on the InfluxQL node in Kapacitor. Every function available in the InfluxDB query language is available in Kapacitor, so you can convert any query into a Kapacitor TICKscript.

What's next

Download of the TICK stack!
Need help migrating from 0.8x or 0.9x to 0.11? We are here to help! Drop us a line at contact@influxdata.com to get your migration project started.
Looking to level up your InfluxDB knowledge? Check out our FREE and economically priced virtual and public trainings.

Announcing Kapacitor 0.11 RC with Big Performance Gains and Simplified API

Nathaniel Cook (InfluxData) — Tue, 15 Mar 2016 13:36:41 -0700

We are excited to announce that Kapacitor v0.11 RC is here with big changes. Kapacitor is significantly more performant, exposes internal performance metrics, and has a simplified API for working with functions from InfluxQL.

What's Kapacitor? Kapacitor is InfluxDB's native data processing engine. It can process both stream and batch data from InfluxDB. Kapacitor lets you plug in your own custom logic or user defined functions to process alerts with dynamic thresholds, match metrics for patterns or compute statistical anomalies.

Much of the effort that went into this release was on improving the performance of Kapacitor and providing users the needed tools to right-size their Kapacitor instance. Kapacitor is now up to 4x faster in some cases. A user can now also get stats about how fast their task is running and which steps are slowest. With this data now readily available, a user can now decide how large to make a Kapacitor instance, eliminating any guesswork.

In order to make Kapacitor easier to use we have changed the way TICKscripts use functions from InfluxQL. Now you can write:

`stream.from().measurement(...)
.window().period(1m).every(1m)
.count('value')`

…directly instead of having to use the mapReduce function. The old syntax will continue to work throughout this release, but will be dropped in version 0.12.

Kapacitor can also communicate with multiple InfluxDB clusters. Update the configuration with multiple InfluxDB sections and specify which cluster to use in the TICKscript. With this simple addition, Kapacitor can now act as a Continuous Query engine between different InfluxDB clusters.

What's next

Get started with Kapacitor here and check out the official release notes here.
Looking to level up your InfluxDB knowledge? Check out our economically priced virtual trainings.

Announcing Kapacitor v0.10: Support for Custom Functions, More Integrations and a Dead Man's Switch

Nathaniel Cook (InfluxData) — Thu, 28 Jan 2016 11:00:55 -0700

Since the initial release of Kapacitor v0.10 in December of last year there have been a few common themes among the community on what features they’d most like to see…

Custom User Functions & Anomaly Detection

This release gives users the ability to write custom functions in any programming language. This is done via a Protocol Buffer RPC system with Kapacitor managing the running process. If your language supports Protobuf serialization, you’ll be able to write custom monitoring, alerting, and processing code that can be injected into your TICKScripts.

The big news is that you can now do more advanced anomaly detection with Kapacitor through these User Defined Functions (UDFs). It is now possible to integrate any new or existing algorithm with Kapacitor via UDFs. Check out this guide to get started integrating your favorite anomaly detection algorithm with Kapacitor.

Integrations

Secondly, we have had lots of attention on alerting integrations. Thanks to the community Kapacitor can now send alerts to HipChat, OpsGenie, Alerta and Sensu in addition to previous integrations with PagerDuty, Slack and VictorOps. If you want to integrate with yet another alerting system check out these PRs as they layout how easy it is.

In addition there have been a dozen bug fixes and another dozen new features, most around improving the alerting experience within Kapacitor. Check out the release notes for a complete list of features since the last major release.

Dead Man's Switch

Lastly, people naturally wanted to know if their task stopped receiving data. We have built in a dead man’s switch, that can be configured by individual task or globally for all tasks, that will trigger an alert if a task doesn’t receive data for a configurable duration. See this PR for details.

Where is Kapacitor going from here?

For the next release we are focusing on two areas. First, more integrations for UDFs, we want to make it easy to plugin your algorithms in whatever language you are using. Second, we are focusing on stability and performance. Kapacitor is a young project but we already like the foundation it has built and are ready to take it to the next level. We will be designing a set of benchmarks and failure tests suites for Kapacitor to ensure ongoing stability and performance.

We want to be able to make strong statements about how Kapacitor behaves under real load, and to be able to definitively answer questions around instance sizing and resource consumption.

What's next?

Download the code and get started
Visit the Kapacitor project on GitHub to report a bug, make a feature request or better yet contribute some code!