InfluxData Blog - Jonathan Sternberg

Time Zones in Flux

Jonathan Sternberg (InfluxData) — Mon, 01 Nov 2021 10:13:06 -0700

As a language for processing time series data, Flux has an important role in how we understand that data. As we create and process data, we do it for ourselves and others.

The concept of time and how we as people interact with time isn’t always simple. Most of us have to adjust our schedules at least twice a year because of daylight savings time, and sometimes we have to adjust our clocks in more complex ways because — for important political, cultural, or economic reasons — we need to have our clocks read a certain time such as skipping a day.

Flux before time zone support

Flux is a language for time series data and, until recently, Flux wasn’t able to understand these subtleties. Flux would do all of its work in UTC, which is a simple linear clock, and leave it to the user to figure out the display. This works well for most circumstances! It’s what Flux should be doing! Flux shouldn’t be involved in figuring out the correct way to display the time zone. The InfluxDB UI is very capable of figuring that out better than Flux would ever be able to. In situations like monitoring servers, UTC is perfect and should be used over a specific time zone.

But…what happens when we start to aggregate data for other purposes? What if I want to aggregate by days, weeks, or months and I want these aggregations to be in my own time zone for a business reason? I don’t live in the UK and, even if I did, the UK isn’t on UTC time the entire year anyway. I can just pretend and remember that my aggregations aren’t 100% accurate all the time and they are “good enough”. Sometimes “good enough” isn’t good enough. In those, I can get closer by using an offset to shift the time to my time zone. But, when daylight savings time happens, I might be back to “good enough” aggregation by time.

Time zone support in Flux

Flux now features full time zone support for those who need to perform their aggregations within a specific time zone.

To enable time zones, import the timezone package and set the location option to your location…By default, Flux uses the UTC time zone.

import "timezone"

option location = timezone.location(name: "America/Los_Angeles")

After setting the location option, the window() and aggregateWindow() functions become location-aware and utilize dynamic offsets to account for the clock moving forwards or backwards. In this way, you can use either of the following in your query:

from(bucket: "mybucket")
  |> range(start: -1y)
  |> filter(fn: ...)
  |> aggregateWindow(every: 1mo, fn: mean)

from(bucket: "mybucket")
  |> range(start: -1mo)
  |> filter(fn: ...)
  |> aggregateWindow(every: 1d, fn: mean)

Both of these will always result in the time starting and stopping at midnight in the time zone specified.

When the clock skips forward or moves backwards, the time window will either shrink or stretch to accommodate the change in offset.

For parity, a fixed offset can also be specified in the location option.

import "timezone"

option location = timezone.fixed(offset: -8h)

This sets the time zone with a fixed offset of -8h. This fixed offset works the same as UTC, but midnight will always be 8 hours after midnight in UTC.

In a more complex situation, we might need to mix time zones or only use time zones for a single call to a window function. We can use parameters for that rather than a global option.

import "timezone"

option location = timezone.location(name: "America/Los_Angeles")

from(bucket: "mybucket")
  |> range(start: -1y)
  |> filter(fn: ...)
  |> aggregateWindow(
    every: 1h,
    fn: mean,
    timeSrc: "_start",
    location: timezone.utc, // overrides the global option to use utc
  )
  |> aggregateWindow(every: 1w, fn: max) // global option is used

We hope that this feature will enable many new use cases for processing time series data in Flux.

Aggregating by Months or Years in InfluxDB with Flux

Jonathan Sternberg (InfluxData) — Mon, 02 Dec 2019 13:54:57 -0700

Over a year ago, we made our first commit and commitment to supporting a much wanted and needed feature for Flux. It was a feature that we have wanted as far back as the 1.x versions of InfluxDB, but it was one we were never able to implement as part of the database. Until now.

Recently, we merged and deployed a change to the Flux language to support months and years as special intervals when windowing. Today, I am going to show you how you can use this feature and the power it can help bring to analyzing your data.

This demo uses the NOAA sample dataset located here. I modified the input slightly to remove the lines related to 1.x and used curl to write the data to a new bucket. I then wrote the following code in Flux.

from(bucket: "NOAA")
	    |> range(start: 2015-08-01T00:00:00Z, stop: 2015-10-01T00:00:00Z)
	    |> filter(fn: (r) => r._measurement == "h2o_feet" and r._field == "water_level")
	    |> aggregateWindow(every: 1mo, fn: mean)

And it worked! When I ran this query, I get two points per series. One of them gives me the mean for the month of August and one gives me the mean for the month of September. But this doesn’t only apply to 1 month. I can also use 1y to aggregate across years and it will account for leap years. If I wanted to aggregate how many requests happened each quarter, I would just use 3mo instead.

We hope that this feature will help people to better understand their data and we hope to continue expanding the functionality of Flux to help enable a better understanding of data.

InfluxData Docker on ARM

Jonathan Sternberg (InfluxData) — Tue, 26 Sep 2017 05:34:12 -0700

In the beginning, Docker only supported amd64 and only ran on Linux. Later, Docker started supporting different architectures like Windows 64-bit and ARM. You could run Docker on these platforms, but the image support has been a work in progress. For a long time, some images would create a fork that supported armhf, but then you would have a separate tag for the armhf architecture. Since the default for Docker is to pull the latest tag, running something like docker pull ubuntuwouldn’t pull the correct architecture.

Docker recently created a new manifest format for the registry that allows for multiple images of different architectures to share the same tag. One of our users, @longquan, made a pull request on our Docker images offering a method to have a single Dockerfile build multiple architectures. We adapted this pull request and, with the additional help of @jcberthon, we merged support for ARMv7 32-bit and ARMv8 64-bit.

That means the InfluxData stack can be run with Docker out of the box on platforms like the Raspberry Pi. This is perfect if you want a cheap and easy computer to set up for monitoring something in your home! In this post, we are going to run InfluxDB on our own laptop and configure a Telegraf instance to report to our laptop over a local network.

What You Will Need

The Raspberry Pi
A computer (such as a laptop)
A router with both computers being connected to the same router
An internet connection

Install Docker on the Laptop

Install Docker on your laptop using the appropriate method from the Docker Documentation. I am using Mac OS X, so I am using docker-machine as a virtual machine.

Once you have installed Docker on your computer, verify it works by using ping and then pulling down the InfluxDB image.

$ docker ping
$ docker pull influxdb

We can now run InfluxDB and Chronograf on our laptop with few simple commands.

# Create a shared network for influxdb and chronograf to communicate over.
$ docker network create influxdb
# Launch the influxdb image.
$ docker run -d --name=influxdb --net=influxdb -p 8086:8086 influxdb
# Launch the chronograf image.
$ docker run -d --name=chronograf --net=influxdb -p 8888:8888 chronograf --influxdb-url http://influxdb:8086

We create a shared network for influxdb and chronograf to communicate over. While it looks complicated, this simplifies networking the two containers. We then launch each one of the images using this network and expose the ports that need to be exposed to the outside world. For InfluxDB, this is port 8086. For Chronograf, this is 8888. Now, try visiting Chronograf using your computer’s web browser. If you are running Docker natively on your computer, this should be as simple as going to http://localhost:8888. If you are using docker-machine (such as on Mac OS X), you can use the https://github.com/jsternberg/docker-machine-proxy I created for this demo. It will proxy ports from the Docker Machine to localhost so internetwork connections between computers work correctly.

When this is working and you are able to access Chronograf at localhost, you should see the Chronograf dashboard and Chronograf should say that no hosts were found.

Before we add a host, there is one more thing we must do.

Retrieve the IP Address

Retrieve the IP address that your router gave you for your computer. If you are on Linux or Mac OS X, you can use ifconfig. Here is an example of my output:

> ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether ac:bc:32:cd:96:f3
        inet6 fe80::1c:121a:e9d5:fe98%en0 prefixlen 64 secured scopeid 0x4
        inet6 2605:6000:1522:c065:854:a83:ba74:7e05 prefixlen 64 autoconf secured
        inet6 2605:6000:1522:c065:9c91:edf6:d09a:e266 prefixlen 64 autoconf temporary
        inet 192.168.1.113 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=201<PERFORMNUD,DAD>
        media: autoselect
        status: active
en1: flags=963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX> mtu 1500
        options=60<TSO4,TSO6>
        ether 6a:00:01:52:70:20
        media: autoselect <full-duplex>
        status: active

The en0 interface is the only one that looks like it has an IP address so we will use 192.168.1.113 for this example. For your own computer, you need to use your own IP address! I am running this command on Mac OS X so the name of the interface may be different if you are using another operating system. On Windows computers, you can use cmd.exe and ipconfig to find your local IP address.

Remember, you want the IP address given to the computer by your router. You do not want your public IP address to the internet itself. Most home networks have their networks start with 192.168.*.*, but some may also start with 10.*.*.* or 172.*.*.*.

Boot Your Raspberry Pi

Connect your Raspberry Pi to your router. If you have never installed an operating system, I suggest Raspbian. It’s easy to install and use and the one I used for this article.

You can then run the following to install Docker. First, download the script that Docker offers for making things easy.

$ curl https://get.docker.com > get-docker.sh
$ sudo bash get-docker.sh

After this command finishes, you will see a message like this in the output:

If you would like to use Docker as a non-root user, you should now consider
adding your user to the "docker" group with something like:

  sudo usermod -aG docker pi
  
Remember that you will have to log out and back in for this to take effect!

There will be more and a warning. If you are planning to use this in production and security is something you must consider, it might be best to skip this step. For our purposes, we are going to enter the above command so we don’t have to type sudoin front of the docker commands. Run the above command and then reboot your Raspberry Pi.

After the Pi is back up and running, open the Terminal again and run the following:

$ docker run --rm telegraf telegraf config > telegraf.conf

You just ran Telegraf on your Raspberry Pi for the first time! But we’re not quite done. We generated a configuration file. Open telegraf.conf and go to the [[outputs.influxdb]] section. It should look like this:

[[outputs.influxdb]]
  ## The HTTP or UDP URL for your InfluxDB instance.  Each item should be
  ## of the form:
  ##   scheme "://" host [ ":" port]
  ##
  ## Multiple urls can be specified as part of the same cluster,
  ## this means that only ONE of the urls will be written to each interval.
  # urls = ["udp://localhost:8089"] # UDP endpoint example
  urls = ["http://localhost:8086"] # required
  ## The target database for metrics (telegraf will create it if not exists).
  database = "telegraf" # required

Modify the urls option so it looks like this, but replaced with your own IP address that we found earlier:

[[outputs.influxdb]]
  ## The HTTP or UDP URL for your InfluxDB instance.  Each item should be
  ## of the form:
  ##   scheme "://" host [ ":" port]
  ##
  ## Multiple urls can be specified as part of the same cluster,
  ## this means that only ONE of the urls will be written to each interval.
  # urls = ["udp://localhost:8089"] # UDP endpoint example
  urls = ["http://192.168.1.13:8086"] # required
  ## The target database for metrics (telegraf will create it if not exists).
  database = "telegraf" # required

Now we need to execute Telegraf using our custom configuration file.

$ docker run -d --name=telegraf -v $PWD/telegraf.conf:/etc/telegraf/telegraf.conf:ro telegraf

Now wait a bit. If you want to check if Telegraf is working, you can use docker logs -f telegraf and it will follow along with the log messages. You might need to refresh the Chronograf page. If all goes well, you should see your Raspberry Pi and metrics coming in from the container on the Raspberry Pi!

While this demo showed how to use Telegraf on the Raspberry Pi, all of the InfluxData products now support running them with Docker on ARM.

Help and Caveats

For those who are using an older ARM architecture without native float support, docker pull will not work. This is because multiple architecture support in Docker is still ongoing. When pulling from an ARM 32-bit (armhf), it may pull down the version built with emulated floating point support (armel). Because of this limitation, the Dockerfiles support building on armel, but we do not upload these images to the official repository. If you need to use the Docker images and have armelas your architecture, you can build the images from source here.

To determine which architecture you have, you can run dpkg --print-architecture. If you see armhf, that’s the one that is uploaded and we support. If you see armel, that’s the one you need to build from source.

For those who need help, please visit our https://community.influxdata.com website. If you think something isn’t working correctly with the images, please file a bug in the repository.