InfluxData Blog - Noah Crowley

Synthetic Monitoring with Telegraf

Noah Crowley (InfluxData) — Mon, 16 Sep 2019 07:22:42 -0700

There are two main modes for collecting data about your systems and software: the first is by collecting data from within the application itself, often called white-box monitoring, and the second is by querying the system from the outside and collecting data about the response.

Google’s SRE book defines white-box monitoring as “Monitoring based on metrics exposed by the internals of the system,” and black-box monitoring as “testing externally visible behavior as a user would see it.” Synthetic monitoring is an implementation of a black-box monitoring system which involves creating requests which simulate user activity.

With synthetic monitoring, the aim is to expose any active issues that the user might be experiencing with a system, such as a website being inaccessible. Since it represents real user pain, this data is especially useful as an alerting signal for paging.

This complements the white-box approach, which allows developers and operators to get insight into the internal functioning of the system, providing insight into issues that may be obscured to the user, such as failures that result in a successful retry, and providing invaluable information for debugging purposes.

Telegraf can gather many white-box metrics using application-specific plugins like the ones for NGINX or MySQL, and you can instrument your applications using the InfluxDB client libraries, but we can also use Telegraf as a synthetic monitoring tool to monitor the status of our systems from the outside.

HTTP Response Input Plugin

Telegraf’s http_response input plugin checks the status of HTTP and HTTPS connections by polling an endpoint with a custom request, and then recording information about the result. The configuration for the plugin allows you to specify a list of URLs to query, define the request method, and send a custom request body or headers to simulate actions that might be taken by external users and systems. It also allows you to verify the behavior of those endpoints by verifying that the responses to these requests match certain predefined strings using regular expressions. These options give us a lot of flexibility in terms of how we monitor our applications.

For each target server that is being polled, the plugin will send a measurement to InfluxDB with tags for the server (the target URL), request method, status code, and result, and fields with data about response times, whether the response string matched, the HTTP response code, and a numerical representation of the result called the result code.

We can create a new block in our Telegraf configuring for each endpoint we want to monitor. Telegraf will collect data for each config block once per collection interval.

Monitoring influxdata.com

Let’s look at a quick example: we’ll create a simple synthetic monitoring check that will tell us whether influxdata.com is up or not. Because we want these monitoring checks to come from outside of the system, we’ll need to set up some kind of independent infrastructure, separate from the rest of our systems, for running Telegraf. This could mean running in a different Availability Zone on AWS, or using a different cloud provider altogether. Since I don’t actually need long-lived infrastructure for this example, I’ll configure Telegraf to run on my Mac, which is external to the influxdata.com infrastructure.

I already have Telegraf installed using Homebrew, so the next step will be to create a new config file with our http_response settings. Here’s a snippet of what the inputs.http_response block would look like:

# HTTP/HTTPS request given an address a method and a timeout
[[inputs.http_response]]
  ## List of urls to query.
  urls = ["https://www.influxdata.com"]

[...]

  ## Optional substring or regex match in body of the response (case sensitive)
  response_string_match = "InfluxDB is the open source time series database"

This queries the InfluxData homepage and looks to match the phrase “InfluxDB is the open source […]”.

One thing to note is that Telegraf’s collection interval is especially important for this plugin because it determines how often to make requests to the endpoint in question. Individual plugins can define their own collection interval by including a interval parameter in the appropriate config block. For the sake of example we’ll use the Telegraf defaults, but you’ll need to decide what an appropriate interval is for your own systems. You can find a complete configuration file in this gist.

We can then launch a copy of Telegraf using the new config, and should see some output, as follows:

$ telegraf --config synthetic-telegraf.conf --debug
2019-07-01T11:51:52Z I! Starting Telegraf 1.10.4
2019-07-01T11:51:52Z I! Loaded inputs: http_response
2019-07-01T11:51:52Z I! Loaded aggregators: 
2019-07-01T11:51:52Z I! Loaded processors: 
2019-07-01T11:51:52Z I! Loaded outputs: influxdb
2019-07-01T11:51:52Z I! Tags enabled: host=noah-mbp.local
2019-07-01T11:51:52Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:noah-mbp.local", Flush Interval:10s
2019-07-01T11:51:52Z D! [agent] Connecting outputs
2019-07-01T11:51:52Z D! [agent] Attempting connection to output: influxdb
2019-07-01T11:51:52Z D! [agent] Successfully connected to output: influxdb
2019-07-01T11:51:52Z D! [agent] Starting service inputs
2019-07-01T11:52:10Z D! [outputs.influxdb] wrote batch of 1 metrics in 9.118061ms
2019-07-01T11:52:10Z D! [outputs.influxdb] buffer fullness: 0 / 10000 metrics. 
2019-07-01T11:52:20Z D! [outputs.influxdb] wrote batch of 1 metrics in 7.672117ms
2019-07-01T11:52:20Z D! [outputs.influxdb] buffer fullness: 0 / 10000 metrics.

Next steps

The http_reponse plugin provides a lot of flexibility in terms of creating monitoring requests which you can use to more accurately model how users and applications might interact with your site. For example, on influxdata.com you might want to verify that your search page is working by submitting a POST request and verifying that the response includes text from the search result page, rather than just checking if the page loads. Because synthetic monitoring is intended to model the user experience, the specific number, frequency, and implementation of your checks will depend on the design and function of your product, but in general you’re looking for things like slow response times or high rates of errors - things that effect user happiness.

Since black-box monitoring often exposes issues that are already impacting users, you’ll also want to create a sane alerting strategy based on this data. That might mean paging an engineer, or dropping a notification in Slack, at which point they will have to turn to data better suited for debugging: white box metrics and events.

Black box monitoring isn’t a replacement for data captured from your application, but it provides end-to-end coverage that’s useful in the most catastrophic of scenarios. When used in conjunction with white box tools, it can give you that extra bit of confidence in the functioning of your software, making it a critical component of your monitoring system.

Reading Maxim 1-Wire Temperature Sensors

Noah Crowley (InfluxData) — Thu, 05 Sep 2019 07:00:04 -0700

One of my favorite sensors is the Maxim DS18*20 series, a variety of inexpensive temperature sensors which communicate over a “single contact serial interface” called 1-Wire. I’ve been integrating them into a variety of projects for years now, and they’ve always been reliable, accurate, and extremely easy to use. That’s partly due to their use of a single contact for communication — valuable when you’re looking to save pins on an embedded computer or microcontroller — but also because of the built in w1_therm driver in the Linux kernel, which means they’ll work on any Linux machine with the appropriate physical interface.

For the sake of example, we’ll use a Raspberry Pi with a DS18B20 connected to the GPIO pins. You can find a wiring diagram in this blog post. We’ll load the w1_therm kernel module as well as the w1_gpio module, which uses the GPIO API to provide a 1-Wire interface:

modprobe w1_gpio
modprobe w1_therm

With the kernel modules enabled, the sensor is now accessible through the sysfs interface as individual directories. We can use the ls command with some wildcards to find information about our sensors:

$ ls -laH /sys/class/hwmon/hwmon*
total 0
drwxr-xr-x 3 root root    0 Apr 24 23:59 .
drwxr-xr-x 3 root root    0 Apr 24 23:59 ..
lrwxrwxrwx 1 root root    0 Apr 24 23:59 device -> ../../../28-000002e36be9
-r--r--r-- 1 root root 4096 Apr 24 23:59 name
drwxr-xr-x 2 root root    0 Apr 24 23:59 power
lrwxrwxrwx 1 root root    0 Apr 24 23:59 subsystem -> ../../../../../class/hwmon
-r--r--r-- 1 root root 4096 Apr 24 23:59 temp1_input
-rw-r--r-- 1 root root 4096 Apr 24 23:59 uevent

Within this directory you’ll find a variety of files, including a symlink to the device itself, the sensor name, and a file called temp1_input, which you can read to get the temperature of the sensor in units of Celsius*1000:

$ cat /sys/class/hwmon/hwmon0/temp1_input
27562

In addition, you can access more info about the sensor by reading the /sys/class/hwmon/hwmon0/device/w1_slave file, which looks something like this:

$ cat /sys/class/hwmon/hwmon0/device/w1_slave
bb 01 4b 46 7f ff 05 10 c6 : crc=c6 YES
bb 01 4b 46 7f ff 05 10 c6 t=27678

This shows the output of the device in hex on the left, with the results of the CRC error checking and temperature reading on the right.

Telegraf's Temp plugin

Telegraf’s temp input plugin reads data about temperature sensors from the filesystem during each collection. You can install Telegraf and InfluxDB on a Raspberry Pi using the official InfluxData repositories. Once installed, we can configure Telegraf by uncommenting the following section in /etc/telegraf/telegraf.conf:

# # Read metrics about temperature
[[inputs.temp]]
# # no configuration

Since there is no configuration beyond enabling the plugin, we can restart Telegraf and we should see data begin to appear in the temp measurement of the telegraf database (since we’re using the configuration defaults.)

Using multiple sensors

Unfortunately, there is one shortcoming with the current implementation of the temp plugin: it only supports a single sensor. This is because of an issue with the DS18*20 sensors and the library that Telegraf’s temp plugin uses to access the filesystem: gopsutil.

When reading temperatures using gopsutil, the library generates a SensorKey using data from the filesystem, including a label file. This is what the kernel documentation has to say about those files:

temp[1-*]_label	Suggested temperature channel label.
		Text string
		Should only be created if the driver has hints about what
		this temperature channel is being used for, and user-space
		doesn't. In all other cases, the label is provided by
		user-space.
		RO

Because the application of a DS18B20 and other similar sensors can vary, the driver is generic and does not have any information about what the temperature channel is being used for, so this file is not created. That means when Telegraf collects data from multiple sensors their tag values are identical, as they all have the same SensorKey. So even though we’re collecting data from each sensor, the data points have the same metadata and timestamp, and so will overwrite one another. We can verify this by running Telegraf using the --test argument, which will print the line protocol to the command line.

Multiple sensor workaround using Python

One way we can work around this issue is by using a Python script to read the data (you can find an example gist here), and Telegraf’s exec plugin to execute the script.

Download the script and copy it to a more permanent location; for now, let’s drop the file in /usr/local/bin. Because we installed Telegraf using the official InfluxData repository, Telegraf runs as the telegraf user, which means we’ll also want to change the permissions on the script so it can be executed by Telegraf, as follows:

sudo chown telegraf:telegraf /usr/local/bin/read_multiple_ds18b20.py
$ sudo chmod 755 /usr/local/bin/read_multiple_ds18b20.py

Then we can configure Telegraf’s exec plugin to execute the script by enabling the plugin and adding the file to the commands parameter:

# # Read metrics from one or more commands that can output to stdout
[[inputs.exec]]
#   ## Commands array
  commands = [
    "/usr/local/bin/read_multiple_ds18b20.py"
  ]

The full configuration file is here, and includes the comments that describe the default values. Since multiple DS18B20s can share a single data line, connecting an additional device is fairly straightforward: it will share the connections to ground, power, and data that the first sensor uses. Restart telegraf and it should start collecting data from each device.

Future improvements

I have a few applications in mind that use multiple temperature sensors per device, so it would be nice to iterate on the built-in temp plugin if possible, rather than relying on our Python script workaround.

One possibility would be to make a small change to the gopsutil library which would add an additional field to the SensorKey struct for the device name (unique for each DS18*20). Telegraf could then use that field to add an additional tag to the data and prevent the points from overwriting each other. I’m going to put together a Pull Request and get some feedback from the gopsutil maintainers. In the meantime, feel free to use my Python script, and reach out on our community site or to me directly at noah@influxdata.com or on Twitter @noahcrowley if you have any issues!

Telegraf Socket Listener Input Plugin

Noah Crowley (InfluxData) — Fri, 30 Aug 2019 07:00:00 -0700

One of Telegraf’s biggest strengths is the large collection of plugins it offers that can be used to immediately to start collecting data from a variety of applications. This covers a lot of common infrastructure components like databases and web servers, but if you want to get data into Telegraf using your own scripts or applications, there are also a variety of more general-purpose input plugins that you can use. There’s a file plugin, which will read from or tail a file on disk, an exec plugin which will execute your own scripts or commands and parse their output, and plugins to listen for data via HTTP or poll external endpoints (http_listener_v2 and http).

One of the plugins I end up using most often is the socket_listener plugin, which allows you to send data to Telegraf using UDP, TCP, or Unix sockets. The simplicity of InfluxDB’s line protocol and the ease with which you can write to a socket using most programming languages makes this a powerful tool for both quick prototyping and more long-term solutions.

Let’s take a look at some examples of the plugin in action. We’ll use a host system running Ubuntu, with Telegraf and InfluxDB installed from the official InfluxData repository.

UDP: User Datagram Protocol

For this first example, let’s configure the socket_listener plugin to accept UDP packets. The UDP protocol is a connectionless, best-effort service: messages are sent to their recipient with no guarantee that they will be received. One benefit of this is that it means there is less processing overhead for UDP relative to something like TCP, which is more reliable. That makes UDP well-suited for things like interprocess communication or situations in which some loss of data is acceptable.

In order to set up Telegraf to receive UDP messages, first configure this block in your telegraf.conf:

[[inputs.socket_listener]]
  service_address = "udp://:8094"
  data_format = "influx"

This configures Telegraf to listen for data on UDP port 8094 in InfluxDB line protocol format. A quick restart of Telegraf and it should be ready to receive data:

sudo systemctl restart telegraf

Now we can start writing some data to Telegraf via UDP. To send the data, we’ll use two common command line utilities, echo and nc, or netcat. Enter the following commands into your terminal:

$ echo "m1,tag1=tag_value value=1" | nc -u -4 -q localhost 8094
$ echo "m1,tag1=tag_value value=2" | nc -u -4 -q localhost 8094
$ echo "m1,tag1=tag_value value=3" | nc -u -4 -q localhost 8094

The InfluxDB Line Protocol is a plain text format, which means we can use the echo command to print messages in that format directly to stdout, and use the | character to “pipe” that text into nc, which will transmit the data over a network connection (in our case, on the same machine via the loopback interface). We provide several arguments to nc: -u, -4, and -q. The first argument tells nc to send data using UDP; the second argument says that we should use IPv4, and the third argument tells nc to quit once the data has been sent.

Because UDP doesn’t have any guarantees about whether the messages will be delivered, let’s verify that the data has actually arrived in the database using the Influx CLI:

$ influx
Connected to http://localhost:8086 version 1.7.4
InfluxDB shell version: unknown
Enter an InfluxQL query
> use database telegraf
> SELECT * FROM mymeasurement WHERE time > now()-5m
name: socketwriter
time                host           tag1       value
----                ----           ----       --------
1562602580414360000 noah-mbp.local tag_value  1
1562602581418350000 noah-mbp.local tag_value  2
1562602582420300000 noah-mbp.local tag_value  3

Success! Our three datapoints have been written to the database.

Unix socket

While UDP and TCP are network protocols, the socket_listener plugin also has the ability to listen on a Unix socket, which provides a method for exchanging data between two processes on the same host via the kernel. Even when communicating on the same host, IP sockets like TCP and UDP will send data through the loopback networking interface. A Unix socket, on the other hand, can avoid this extra work and send data directly to the receiving socket buffer, which gives it a performance edge when sending data locally. You can read more about the differences between Unix sockets and network sockets in this mailing list post titled unix domain sockets vs. internet sockets.

This time, instead of using command line tools, we’ll write a small Python script that will send one-dimensional random walk data over a Unix socket to Telegraf. First, we’ll configure Telegraf as follows (full config here):

[[inputs.socket_listener]]
  service_address = "unixgram:///tmp/telegraf.sock"

Then we’ll run Telegraf with the new config:

$ telegraf --config socket-telegraf.conf --debug

We can see that Telegraf has created the socket by listing the contents of the /tmp directory:

$ ls /tmp
telegraf.sock

Next we’ll create our Python script. The full script can be found here. Following the import statements and an interrupt handler which will close the socket when we use Ctrl+C to exit our script, we’ll create a socket object and connect to the socket Telegraf has opened.

# The file handler for the Telegraf process.
telegraf_socket = "/tmp/telegraf.sock"

# Connection to Telegraf, over a network socket.
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)

sock.connect(telegraf_socket)

And then we create a loop that first sends the current data point to InfluxDB before updating the position variable for the next run through the loop, and sleeps for a second in order to rate limit the number of points we send:

while True:
    message = "socketwriter position=%d\n" % x
    print(message)
    sock.send(message.encode('utf8'))
    x = x + random.randint(-1, 1)
    time.sleep(1)

With Telegraf running, let’s start our script:

$ python3 unix_socket_writer.py 
socketwriter position=0

socketwriter position=0

socketwriter position=1

socketwriter position=0

Once again, we can open up the InfluxDB CLI and verify that our data has been written:

$ influx
Connected to http://localhost:8086 version 1.7.4
InfluxDB shell version: unknown
Enter an InfluxQL query
> use database telegraf
> SELECT * FROM socketwriter WHERE time > now()-1m
name: socketwriter
time                host           position
----                ----           --------
1562602580414360000 noah-mbp.local 0
1562602581418350000 noah-mbp.local 0
1562602582420300000 noah-mbp.local 1
1562602583422080000 noah-mbp.local 0

Looks good!

Use it in your own projects!

Whether you’re using Unix sockets for their local performance or TCP sockets for their network robustness, Telegraf’s socket_listener plugin provides one of the easiest ways to get data from your scripts and applications and into Telegraf. And if that doesn’t work for you, check out some of the other general-purpose plugins that Telegraf has to offer — you’re sure to find something that suits your needs.

If you have questions or need help with getting the socket_listener plugin set up, check out some of our community resources, like our public Slack or Discourse forums, or reach out to me directly on Twitter @noahcrowley.

Writing Data to InfluxDB with Python

Noah Crowley (InfluxData) — Thu, 22 Aug 2019 09:45:24 -0700

One of the questions we see fairly often from Python developers who are using InfluxDB is how to improve the write performance of programs which use the InfluxDB client library. Usually they’re trying to import or transfer large amounts of data and want to make sure it can be inserted into the database quickly enough for their business needs. In this post, we’ll go over the common pitfalls that people encounter, as well as some performance best practices.

The script

Let’s look at an example of a Python script that writes data to InfluxDB. You can find it here. The first part of the script (up until Line 56) is used for creating the data we’ll use to populate our measurements; there is a list of locations, a list of fruits, and a function which generates 1000 UUIDs.

We create an array, and append the first value to it using our data as follows:

data = []
data.append("{measurement},location={location},fruit={fruit},id={id} x={x},y={y},z={z}i {timestamp}"
            .format(measurement=measurement_name,
                    location=random.choice(location_tags),
                    fruit=random.choice(fruit_tags),
                    id=random.choice(id_tags),
                    x=round(random.random(),4),
                    y=round(random.random(),4),
                    z=random.randint(0,50),
                    timestamp=data_start_time))

This will append a string that looks like:

m1,location=location,fruit=fruit,id=id x=10,y=1,z=42i 1562458785618

At the beginning of the script we defined a variable which contains the number of points we’d like to write: in this case, 250,000. Once that data has been generated and loaded into the array, we can write it to the database as follows:

client.write_points(data, database='python-throughput', time_precision='ms', batch_size=10000, protocol='line')

Measuring performance

Most Unix systems have some version of the time command, which measures the amount of time it takes for a process to execute and then prints it to the command line.

$ time echo "hello world"
hello world

real	0m0.000s
user	0m0.000s
sys	0m0.000s

It’s a tool that many Linux users are familiar with, but the issue with measuring write performance this way is that you end up timing the entirety of the program. If you’re generating random data as we are, or parsing a CSV, this can be a meaningful chunk of time. In order to measure individual parts of the program, we’ll make use of the time module’s perf_counter() function.

time.perf_counter() accesses the CPU's hardware performance counter and returns a value in fractional seconds. The performance counter is the highest resolution timer available for your system. It starts when the CPU is powered on, and runs continuously, which means that to measure the elapsed time of something you can get the time of the counter before and after an event occur and then subtract the start value from the end value.

Since we want to understand the performance of writing data, we use time.perf_counter() to time the write_points() function.

client_write_start_time = time.perf_counter()

client.write_points(data, database='python-throughput', time_precision='ms', batch_size=10000, protocol='line')

client_write_end_time = time.perf_counter()

print("Client Library Write: {time}s".format(time=client_write_end_time - client_write_start_time))

This is perfect for our use case, but if your needs are different, Python also offers a module called timeit, which “provides a simple way to time small bits of Python code”. For larger, more complex applications, you might want to look into using a profiler, which will give you detailed information about the entirety of your program’s execution.

Let’s run the script! We’ll go ahead and use the time function to time the execution of the entire script so we can compare that to the execution of the write_points() function.

$ time python3 write_test.py 
Client Library Write: 3.4734167899999995s

real	0m6.060s
user	0m2.387s
sys	0m0.114s

I ran the script 5 times, dropping the database and starting fresh between each run: the average write time for 250,000 points was 3.81 seconds on my 2018 dual-core MacBook Pro (which has an admitedly fast SSD), while the average elapsed time for the entire script (including generating the data and the time required to initialize and exit the program) was 6.35 seconds.

Favor Line Protocol over JSON

The Python client library’s write_points function can take data in either JSON or InfluxDB Line Protocol. Looking at the code, though, we discover that any data provided in JSON format gets converted to Line Protocol anyway before being sent to InfluxDB:

if protocol == 'json':
  data = make_lines(data, precision).encode('utf-8')

The make_lines() function can be found here, and does the work of converting from JSON to Line Protocol.

If we can avoid that work by providing data in Line Protocol to being with, we’ll save some time. Let’s run a few tests and see what the data looks like. You’ll find a JSON version of the script here. I ran it 5 times and averaged the results again. With the data in JSON instead of Line Protocol, it took an average of 7.65 seconds to insert all the data, about two times slower than using Line Protocol directly.

Optimization 2: Batch Your points

InfluxDB performs best when data is written to the database in batches. This helps minimize the network overhead of opening and closing HTTP connections by transmitting more data at once. The ideal batch size for InfluxDB is 5,000-10,000 points.

By default, the write_points() function will transmit all of the points in the array passed to the function, but you can also provide a parameter which defines the batch size. In the example script, we generate 250,000 points up front, and then write those points in batches of 10,000:

client.write_points(data, database='python-throughput', time_precision='ms', batch_size=10000, protocol='line')

For comparison, I ran the script with a batch size of 50; with that setting, the write_points portion of our code averaged about 29 seconds! An order of magnitude higher!

Next steps: Measure more things!

If your program is still executing too slowly for you, it’s always worthwhile to examine other properties of the system where your code is running. How do your resources look? RAM, CPU, disk space? What is the throughput of your network connection, or disk I/O? There are a lot of external factors which can influence the execution of your code: be methodical about testing them and collecting data to rule out the cause.

If you have more ideas for optimizing write performance with the Python library, we’d love to hear them! Open a pull request, comment on our community forums, or reach out directly to me at noah@influxdata.com.

Working with Irregular Time Series

Noah Crowley (InfluxData) — Fri, 26 Oct 2018 09:00:06 -0700

One of the benefits of InfluxDB is the ability to store raw events, which might come in at varying intervals, as an irregular time series. However, irregular time series present some unique challenges, and in some cases common operations on the data simply will not work. Fortunately, InfluxDB allows you to convert an irregular time series to a regular one on the fly by calculating an aggregate of individual values for arbitrary windows of time. This gives you the best of both worlds when capturing events from your systems and working with that data.

We can take a look at a few actual data points in order to get a better understanding of what considerations need to be made when working with irregular time series. For the sake of example, we’ll use five data points, and give them values of 10, 20, 30, 40, and 50.

If this data if collected at regular intervals, then calculating the mean of the values will also give you the average power usage over time.

If your data is collected at irregular intervals, however, then calculating the mean of the values will not give you the expected results.

Adding those data points to InfluxDB and playing around with them will give us a better feel for what is going on. Using the Influx CLI, I inserted those five values into a measurement called m1:

> select * from m1 where time > '2018-08-14T17:22:14Z' and time < '2018-08-14T17:23:20Z'
name: m1
time                         value
----                         -----
2018-08-14T17:22:14.159637Z  10
2018-08-14T17:22:16.3561521Z 20
2018-08-14T17:22:18.2251241Z 30
2018-08-14T17:22:20.18086Z   40
2018-08-14T17:23:19.8976057Z 50

Since I inserted the data manually, the timestamps for the values are not evenly spaced, which is just what we want for this example. The first four occur during the same minute, while the fifth occurs nearly a minute later. We can calculate the mean of these five values:

> select mean(*) From m1
name: m1
time                 mean_value
----                 ----------
1970-01-01T00:00:00Z 30

and we get a result of 30, with the timestamp value in the result being the beginning of the time window for which we’re calculating the mean.

If this were a regular time series, then an average value of 30 would make sense, intuitively. Because our example is an irregular time series, though, the amount of time between measurements matters when calculating the final result. This becomes more clear when visualizing the values; here is what those points look like on a graph:

We can see that the majority of the time is spent between the value of 40 and the value of 50 (we’re doing a linear interpolation between those points to draw a graph). We can make a reasonable guess, then, that the mean over time is probably closer to 40 than it is to 30.

Clearly, calculating the mean of all the values in an irregular series directly isn’t going to work, since that operation ignores the values’ distribution in time. We need to impose some kind of regularity on our data.

To do this, we can break up the time range for our data into discreet units using InfluxQL’s GROUP BY clause, and then aggregate the values of the data within those windows. Not every interval will have a value, and those that do might have more than one value. We’ll need to make a decision, as users and operators of our systems, about how we want to handle those cases. As is often the case when working with data, human judgement and interpretation play a significant role.

If we take small enough windows, and use the mean() function to calculate aggregates, we should get a fairly accurate representation of our time series. This is what our data looks like after we’ve grouped the points into ten-second intervals and calculated the mean for each group:

> select mean(*) from m1 where time > '2018-08-14T17:22:14Z' and time < '2018-08-14T17:23:20Z' group by time(10s)
name: m1
time                 mean_value
----                 ----------
2018-08-14T17:22:10Z 20
2018-08-14T17:22:20Z 40
2018-08-14T17:22:30Z 
2018-08-14T17:22:40Z 
2018-08-14T17:22:50Z 
2018-08-14T17:23:00Z 
2018-08-14T17:23:10Z 50

For the 10-second period beginning 2018-08-14T17:22:10Z, we have three values, 10, 20, and 30, and the mean of those is 20. For the next window, we have a single value, and then we have a number of windows with no value before we see the final value of 50. As before, we’ll have to make a judgement call as to how we want to handle these empty windows.

The fill() option of GROUP BY will allow us to fill in data for any of those empty windows. fill() can take the options null, previous, number, none, or linear. In our case, linear is the right choice, since we graphed the values using linear interpolation, but if this were real data some of the other options might be more appropriate.

This is what our data looks like using fill(linear):

> select mean(*) from m1 where time > '2018-08-14T17:22:14Z' and time < '2018-08-14T17:23:20Z' group by time(10s) fill(linear)
name: m1
time                 mean_value
----                 ----------
2018-08-14T17:22:10Z 20
2018-08-14T17:22:20Z 40
2018-08-14T17:22:30Z 42
2018-08-14T17:22:40Z 44
2018-08-14T17:22:50Z 46
2018-08-14T17:23:00Z 48
2018-08-14T17:23:10Z 50

We can then calculate the mean of our new, regular time series, using subqueries:

> select mean(*) from (select mean(*) from m1 where time > '2018-08-14T17:22:14Z' and time < '2018-08-14T17:23:20Z' group by time(10s) fill(linear))
name: m1
time                 mean_mean_value
----                 ---------------
1970-01-01T00:00:00Z 41.42857142857143

There’s another caveat, though: the window that we choose will have an impact on our final results. What if we use a 1s window instead of a 10s one?

> select mean(*) from (select mean(*) from m1 where time > '2018-08-14T17:22:14Z' and time < '2018-08-14T17:23:20Z' group by time(1s) fill(linear))
name: m1
time                 mean_mean_value
----                 ---------------
1970-01-01T00:00:00Z 42.95454545454546

We get a slightly different result! Working with irregular time series isn’t exact, and your approach will vary depending on the data in question. Different methods of aggregation, interval size, and interpolation method will be more appropriate than others.

If the data in our examples above was some kind of environmental data, perhaps ambient temperature, and the irregularity of our series was due to unreliable data collection, then the approach we used might be appropriate.

If the data was instead collected whenever there was a state change, perhaps we have a number of LEDs and we’re switching them on in blocks of ten, then the linear fill that we performed in the query no longer makes sense; instead, using fill(previous) would be more appropriate based on the behavior of the system that our data intends to model.

Ultimately, this is the biggest challenge in working with irregular time series: you need to understand your data well enough that you can make educated decisions about how to work with that data.

If you have questions about your irregular time series, our community site is a great place to get help from other users of the InfluxData Platform, or feel free to reach out to me directly on Twitter @noahcrowley!

Writing Logs Directly to InfluxDB

Noah Crowley (InfluxData) — Fri, 12 Oct 2018 11:35:58 -0700

Back in June we published a blog on our “metrics-first” approach to log analysis; we had found that just over a quarter of our users were already using the InfluxData platform for storing logs and non-numerical events. We consider logs to be just another form of time series data, and so we wanted to give those users better tools for ingesting and viewing that data. At the end of the day, logs are extremely valuable for debugging and troubleshooting; they provide detailed insight into the goings-on of your systems, and let you explore problems in ways that predefined dashboards don’t allow.

We released a plugin for Telegraf that allows you to ingest logs using the syslog protocol, which was based on work we were doing internally, and then back in July, we added log viewing functionality to Chronograf.

If you’re interested in getting rsyslog and the Telegraf plugin up and running on macOS or Linux, you should check out the Get Your Syslog On blog post. Once you start writing data, you’ll be able to explore it through the Log Viewer panel in Chronograf.

Not everyone wants to run a syslog instance, however, and we’ve gotten a few questions here and there about how to write logs directly to InfluxDB so that they can be viewed in Chronograf, but without using syslog or the Telegraf plugin.

Good news! This is totally possible!

First, let’s dig a little deeper into the implementation so we can understand what’s really going on. The first thing to understand is that syslog is a protocol; it describes how messages should be formatted and transmitted. This is described in RFC5424 - The Syslog Protocol. Part of that specification includes details on how to use structured data in your logs, which we strongly encourage!

When you’re using syslog and Telegraf, the latter is responsible for accepting messages in syslog format and converting them to line protocol to be written to InfluxDB. It inserts all syslog messages into a measurement called syslog which is what Chronograf looks for when it is populating the log viewer with data. Since the syslog protocol is well-defined, we know that we’ll always have certain fields and tags present in the data, which is how Chronograf knows how to format everything. The viewer has drop-down menus at the upper right for selecting the InfluxDB instance and database to use.

Once data has been written to the database, the schema looks like this:

Tags:

appname
facility
host
hostname
severity (needs to match the syslog severity level keyword to display properly in Chronograf)

Fields:

facility_code (integer)
message (string)
procid (string)
severity_code (integer)
timestamp (integer)
version (integer)

But there’s no reason why you can’t write data like this directly to the database! All you have to do is write points to InfluxDB that adhere to this schema, and land in the syslog database. Once they’re there, they’ll appear in the Log Viewer in Chronograf.

Here are a few examples of logs in line protocol:

syslog,appname=myapp,facility=console,host=myhost,hostname=myhost,severity=warning facility_code=14i,message="warning message here",severity_code=4i,procid="12345",timestamp=1534418426076077000i,version=1i

syslog,appname=mysecondapp,facility=console,host=myhost,hostname=myhost,severity=crit facility_code=14i,message="critical message here",severity_code=2i,procid="12346",timestamp=1534418426076078000i,version=1i

If you want, you can open up the Influx CLI and write these points directly using the insert command; just be sure you update the timestamps to something recent so that you don’t have to go searching through history to find the data.

Some of the fields in the schema above are specific to the syslog format, and help you identify the severity of the logs (are they informational, or critical errors), as well as the applications that wrote them. This is extremely helpful data for troubleshooting, so don’t leave it out! You can get more information on things like severity codes and facilities on the syslog Wikipedia page.

Using this approach, you can create logs entries from any application which can make an HTTP connection to InfluxDB. We’ve even got a few users already using this technique in the wild, so if it fits your use case, give it a try! One thing to remember is that InfluxDB works best when you batch up large numbers of points before sending them to the database (5,000-10,000), so if you’re sending lots of individual log lines you might overwhelm the database with HTTP requests.

If you end up writing logs directly to InfluxDB, or if you have other success stories about integrating metrics and logs, we’d love to hear about it! Tweet us @InfluxDB on Twitter!

OpenCensus Metrics and InfluxDB

Noah Crowley (InfluxData) — Thu, 06 Sep 2018 09:00:41 -0700

OpenCensus is the latest in a series of projects which have emerged from Google as a result of their decades of experience running “planet-scale” systems; it is a collection of libraries, implemented in a number of languages, designed to help developers increase the observability of their systems. OpenCensus provides APIs for gathering metrics and trace data, adding tags, and then sending that data to a back-end for storage, as well as for the creation of zPages, HTTP endpoints which expose additional information about the application in question.

Instrumentation is a key part of any observability strategy, and the OpenCensus project provides a standards-based approach to adding those features to your codebase. As developers and cloud providers throw their support behind OpenCensus, we wanted to ensure a seamless interaction with the rest of the TICK Stack.

Fortunately, OpenCensus embraces a number of other open source technologies and standards which make that integration easy. The project provides a number of “exporters”, the component which sends data to the back-end, including one for metrics which uses the Prometheus exposition format and one for traces that uses Zipkin.

The Prometheus exposition format is something that we’ve supported for some time now in various parts of the TICK Stack, including a Prometheus input plugin for our collection agent, Telegraf, which allows it to scrape metrics from Prometheus-style endpoints.

As a result, we can easily collect data from any applications that have been instrumented with OpenCensus by making a few configuration changes to the Telegraf configuration file and enabling the Prometheus plugin.

Let’s dive in! This post assumes that you have Telegraf and InfluxDB already installed. If you don’t, you can also check out the InfluxData Sandbox to quickly get up and running with Docker and Docker Compose.

Instrumenting Go Code with OpenCensus

First, we need some code to instrument. Fortunately, we can make use of the example provided in the OpenCensus Quickstart guides. The libraries for the Go programming language are probably the farthest along, so we’ll use Go for this blog post.

The quickstart tutorial involves building a simple REPL which capitalizes any strings provided to it as input, while collecting metrics about the application. We’ll use this example pretty much line-for-line, with one exception—we’ll replace the the StackDriver exporter with the Prometheus one.

Working through the tutorial, the first thing we do is create a simple REPL, followed by adding instrumentation code, including metrics and tags, and then recording those metrics. We create a few views and then set up the exporter. You can scroll down to the last snippet of code and then click the “All” tab to grab the full application.

We’ll need to make a few changes to use the Prometheus exporter; fortunately, OpenCensus provides us some example code for that as part of the documentation on exporters. We’ll do a bit of code surgery; replacing a few import statements and updating the initialization and registration of the exporter. I added the existing code to a GitHub Gist, then updated it so you can view the diff if you’re interested.

Save the modified code and we can run the REPL:

go run repl.go

Enter a few lines into the REPL and then visit http://localhost:9091; you should see some metrics exposed in Prometheus format. As you enter new lines into the REPL, those metrics should change.

Collecting OpenCensus Metrics

As I mentioned earlier, Telegraf, the InfluxData collection agent, has built-in support for both scraping and exposing metrics in the Prometheus exposition format. All it takes to enable Prometheus metrics collection is to edit a few lines in the configuration file.

If you don’t already have a config file, you can generate one with the following command:

telegraf --sample-config --input-filter prometheus --aggregator-filter --output-filter influxdb

The --sample-config argument creates a new config file, while the various filter options filter the configuration sections which will be added to the config file. To enable the collection of Prometheus metrics, we’re using --input-filter prometheus, which results in an [[inputs.prometheus]] section being added to our config.

We need to edit that section and point it to the URL our local REPL is using as its metrics endpoint, http://localhost:9091/metrics. We can use all of the other defaults, as follows:

[[inputs.prometheus]]
  ## An array of urls to scrape metrics from.
  urls = ["http://localhost:9091/metrics"]

  ## An array of Kubernetes services to scrape metrics from.
  # kubernetes_services = ["http://my-service-dns.my-namespace:9100/metrics"]

  ## Use bearer token for authorization
  # bearer_token = /path/to/bearer/token

  ## Specify timeout duration for slower prometheus clients (default is 3s)
  # response_timeout = "3s"

  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  # tls_cert = /path/to/certfile
  # tls_key = /path/to/keyfile
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Because we’re using all of the defaults at this point, Telegraf is writing to a telegraf database, collecting data every 10 seconds and flushing it to the outputs at the same interval. We’re also running InfluxDB on the same machine, so the default InfluxDB output configuration should be sufficient.

Start Telegraf using the config file:

telegraf --config telegraf.conf

You should see metrics showing up in the telegraf database after the first collection interval has elapsed! Let’s verify this using the influx command-line tool to execute a query.

Launch the CLI and use the telegraf database:

$ influx
Connected to http://localhost:8086 version 1.6.1
InfluxDB shell version: v1.6.1
> USE telegraf
Using database telegraf
>

Then, run a query to get the last 5m of data for the demo_demo_latency measurement:

> SELECT * FROM demo_demo_latency WHERE time > now()-5m

You should see a number of points, depending on how long your application has been running.

Next Steps

As you can see, it’s extremely easy to collect metrics from an OpenCensus-instrumented application using Telegraf and the OpenCensus Prometheus exporter. If you already have an application that’s been instrumented with OpenCensus, then you might want to consider installing Chronograf and adding a few dashboards, or setting up Kapacitor and configuring alerts.

OpenMetrics to Join the CNCF

Noah Crowley (InfluxData) — Tue, 21 Aug 2018 12:00:32 -0700

Last Friday, the Cloud Native Computing Foundation announced the acceptance of OpenMetrics, an open source metrics exposition format and evolution of the Prometheus exposition format, into the CNCF Sandbox.

The CNCF is an “open source software foundation dedicated to making cloud native computing universal and sustainable”; they provide a home and support for a variety of projects that are used for building “cloud-native” software—scalable, durable distributed systems that provide functionality over an internet connection. The first project to “graduate” from the CNCF was Kubernetes; OpenMetrics enters the foundation as a “sandbox” project, their classification for early-stage efforts.

It’s great to see this kind of effort getting support from the broader community. The Prometheus metrics exposition format has become the de facto format for pull-based metrics, so it’s nice to see the standard broken out into its own project, opening the door for additional collaboration and further integration into the broader vendor ecosystem.

The Prometheus exposition format is something we’ve supported in the TICK Stack for some time now, both in Telegraf, our collection agent, and in Kapacitor, our stream and batch processing engine.

Telegraf has the ability to both scrape and expose metrics in the Prometheus format. Here’s an example of some of Telegraf’s system metrics being exposed via the /metrics endpoint:

# HELP system_load1 Telegraf collected metric
# TYPE system_load1 gauge
system_load1{host="demo-host.local"} 2.61
# HELP system_load15 Telegraf collected metric
# TYPE system_load15 gauge
system_load15{host="demo-host.local"} 2.19
# HELP system_load5 Telegraf collected metric
# TYPE system_load5 gauge
system_load5{host="demo-host.local"} 2.23
# HELP system_n_cpus Telegraf collected metric
# TYPE system_n_cpus gauge
system_n_cpus{host="demo-host.local"} 4
# HELP system_n_users Telegraf collected metric
# TYPE system_n_users gauge
system_n_users{host="demo-host.local"} 5
# HELP system_uptime Telegraf collected metric
# TYPE system_uptime counter
system_uptime{host="demo-host.local"} 230446

Metrics exposed in the Prometheus format can be scraped by enabling the Prometheus Input Plugin, by making sure the following section is present in your config:

# Read metrics from one or many prometheus clients
[[inputs.prometheus]]
  ## An array of urls to scrape metrics from.
  urls = [
    "http://localhost:9100/metrics",
    "http://remotehost.com:9100/metrics",
    "http://192.168.1.199:9100/metrics",
  ]

  ## An array of Kubernetes services to scrape metrics from.
  # kubernetes_services = ["http://my-service-dns.my-namespace:9100/metrics"]

  ## Use bearer token for authorization
  # bearer_token = /path/to/bearer/token

  ## Specify timeout duration for slower prometheus clients (default is 3s)
  # response_timeout = "3s"

  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  # tls_cert = /path/to/certfile
  # tls_key = /path/to/keyfile
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

If you need to generate a new Telegraf config for use with Prometheus, you can use the following command, adding in any additional plugins you’d like to enable, such as the system plugin, separated by a :, as follows:

$ telegraf --sample-config --input-filter prometheus:system --output-filter prometheus_client

Kapacitor can also scrape Prometheus-style endpoints. You can find more information about this functionality in the documentation for Kapacitor Scraping and Discovery.

Moving forward, InfluxData will support the OpenMetrics format as a first-class citizen and work with other open source companies and organizations to move the standard forward. So what can we expect from the future?

The best way to keep up to date on the project’s progress, and to contribute to the discussion, is via the OpenMetrics GitHub page, and specifically, the GitHub Issues. There are a number of interesting discussions going on at the moment, such as this discussion about how to version metrics.

You can read the full announcement here.

Running the TICK Stack on a Raspberry Pi

Noah Crowley (InfluxData) — Tue, 03 Jul 2018 09:00:17 -0700

<figcaption> The Raspberry Pi is a great platform for the TICK Stack!</figcaption>

Since its introduction in 2013, the Raspberry Pi platform has grown by leaps and bounds, gaining popularity and showing up in projects all over world, from quick hacks to fully commercialized products. It’s an inexpensive, low power, open-source, single-board computer with a variety of interfaces for connecting with sensors and actuators that can be used to interact with the physical world.

It’s also a solid choice when you need an always-on device for collecting data, and it’s powerful enough to run the complete TICK Stack. In fact, one of the most popular posts on our community site is about running the TICK Stack on a Raspberry Pi, so we wanted to provide some update instructions for installing and things to consider while operating the stack.

The Hardware

Your Choice of Pi

The first thing you’ll need to do is decide which Pi you’re going to use. The Raspberry Pi platform has gone through a few iterations since it was first released in 2013:

Model	Cores	Clock Speed	RAM
Zero	1, 32-bit	1000 MHz	512 MB
A/B/+	1, 32-bit	500 MHz	256 MB
2B	4, 32-bit	900 MHz	1024 MB
3B	4, 64-bit	1200 MHz	1024 MB
3B+	4, 64-bit	1400 MHz	1024 MB

There have been some significant architectural improvements over the years, and your best bet is to go with one of the model 3 boards, with quad-core, 64-bit CPUs for the database itself, although I’ve tested running the complete TICK Stack on the model 2 B, 3 B, and 3 B+. In addition, the single-core Zero makes a great platform for collecting sensor data with Telegraf.

Storage

The storage system on the Raspberry Pi, which uses an SD or microSD card, is the slowest by far relative to the systems on a laptop or desktop computer, so it’s important to keep in mind storage performance when you’re considering a Raspberry Pi for your application. Which SD card you choose will have a major impact on the performance of your Raspberry Pi, especially if you’re planning on running a database like InfluxDB.

Get at least a Class 10 Card—the faster, the better. But understand that even with the fastest SD card, performance won’t come close to what you would get with an SSD, and this can often be the bottleneck for your applications.

Some manufacturers are also releasing “High Endurance” SD cards; I haven’t had a chance to do much research about them, but if you plan on writing large volumes of data, it might be worth investigating.

The Raspberry Pi 3 B and B+ also have the ability to enable a USB boot mode, although this is a permanent change for your device. While you’ll still be constrained by the speed of a USB 2.0 port, this would enable you to run a large-capacity SSD with better speed and performance.

Installation

The most common way to use a Pi is with the Raspbian operating system provided by the Raspberry Pi foundation, so that’s what we’ll be using in this blog post. Raspbian is based off the Debian distribution of Linux, with some modifications and utilities like raspi-config that contain functionality specific to the hardware. The latest version of Raspbian is April release based on Debian Stretch.

You can download Raspbian and find installation instructions on the Raspberry Pi foundation’s website.

Recommended: Install from the Official InfluxData Repository

The easiest way to install the TICK Stack is to set up the InfluxData package repository (repo) and use your platform’s package manager, which on Raspbian is apt. Packages contain the application binaries themselves, as well as some information about where and how to install and configure the application on your system. That means you can get everything up and running with just a few commands, and InfluxData regularly publishes the latest versions of the TICK Stack to its repository, so applying updates is easy as well.

The first step is to set up the InfluxData repo. You’ll need to identify which version of Raspbian you’re running, which you can do by taking a look at the /etc/os-release file. Run the following command:

$ cat /etc/os-releases
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

We need the version codename, which you can find in the VERSION field, in parentheses. For the version I’m running, 9, the code name is stretch. We’ll use that when we set up our repo so that we can get the correct versions for our OS.

Next, add the repo’s GPG key and add the repo itself:

$ curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -

and then add the repository:

$ echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

Finally, run sudo apt-get update to refresh the package list with data from the new repository, and install the TICK Stack using sudo apt-get install telegraf influxdb chronograf kapacitor. If you want to only install some components of the stack, you can exclude the applications you don’t want from that command.

It’s also possible to download packages and install them yourself, without setting up the InfluxData repository. You probably don’t have a good reason for doing this, but for those who do, head over to our downloads page.

Docker

Docker is another great option for running the TICK Stack on a Raspberry Pi, and is what I use at home. Docker is an OS-level virtualization technology, which means that it isolates various applications by making it appear as though they each have their own Linux kernel and environment to run in. This provides isolation for your applications, while the Docker runtime provides easy-to-use tooling for creating and managing containers.

The recommended method for installing Docker on Raspbian is using the convenience script; you’ll also probably want to install docker compose.

You can refer to my Docker blog post from last week for more details about running TICK in docker containers, and I’ve written a simple Docker Compose file to help you get started.

Manual Options: Binaries and Building from Source

A few other options exist which provide additional level of manual control over how you want to build and install your applications. You can download a Linux binary from influxdata.com/downloads, or you can visit the respective Git repositories (Telegraf, InfluxDB, Chronograf, and Kapacitor), check out the code, and build the applications yourself.

Other Considerations

Since the Pi is a relatively constrained computing environment, you might have to make some tradeoffs in terms of how much software you can run and how much data you can process and store.

One thing to consider is that you don’t have to run Chronograf on the Pi; you can run a local copy on your primary computer, and set it up to talk to the InfluxDB instance on your Raspberry Pi over the network. This is the setup I’m currently running at home, and I find that it gives a nice performance bump over running everything on the Pi.

Another thing to keep in mind is the amount of storage space you have on your SD card. While InfluxDB is quite efficient at storing data, depending on the size of your card, the other software you have installed, and how much data you’re writing, it is possible to run out of space. You should definitely set up some kind of monitoring and alerting so that you can be informed when you run out of space. Telegraf’s disk plugin and Kapacitor’s alerts functionality would be a good fit for this!

You also might not need to keep your data around forever; a few weeks or months might be good enough for your use case. We often see users taking advantage of into InfluxData’s downsampling and retention policy functionality to reduce the resolution of their data and automatically delete it when it has reached a certain age. For more information on why you might want to do this, check out our recent training webinar, Downsampling Your Data.

That's It!

Hopefully this post got you up and running, but if you have any questions or comments please feel free to reach out to me directly on Twitter @noahcrowley or feel free to post your comments on our community site at community.influxdata.com. And if you build something great with Influx, don’t hesitate to let us know, there might be a hoodie in it for you!