InfluxData Blog - Cameron Sparr

Announcing Telegraf 1.0 - The Plugin-Driven Agent for Collecting Metrics

Cameron Sparr (InfluxData) — Thu, 08 Sep 2016 03:00:14 -0700

Telegraf 1.0 GA is here. Telegraf is InfluxDB’s native data collector, the first step of the TICK stack. You can use Telegraf to collect metrics from a wide range of inputs and write those metrics into InfluxDB or other supported outputs.

1.0 is a big date in Telegraf’s journey, and as an open-source project it has come a very long way. I took over as the maintainer of the project just over a year ago (my first commit was August 3rd), when there were 18 input plugins and only 1 output plugin (InfluxDB). Today, there are over 80 input plugins and 18 output plugins, and it’s still growing.

Since the release of 0.13.1 we’ve had nearly 40 features or enhancements added and more than 45 bugfixes. Many of the features and PRs came from the community, thank you to everyone who contributed!

New in this release

Of the new features in 1.0 there are three we want to highlight, all of which are community contributions…

New SNMP Plugin: The previous SNMP plugin was difficult to use and configure, and was quite limited in it's output schema. Thanks to @phemmer for writing a brand new one.
New Webhooks Plugin: There is a new webhooks plugin that allows for multiple webhooks services to run on a single port. Thanks to @francois2metz and @cduez. More can be read about it here.
Windows Service: Telegraf now supports being installed as a real Windows Service, much thanks to @butitsnotme for this PR. You can read more here.

Beyond 1.0

On the roadmap are an HTTP input listener (a Telegraf clone of the InfluxDB http endpoint), NATS output plugin, Histogram (aggregation) support, and improvements to the existing documentation and handling of external plugins. For more details on what we are planning check out the planned milestones.

What's next

Download: 1.0 GA downloads for the TICK-stack are live on our "downloads" page
Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring and rebalancing clusters, plus managing backups and restores.
Tell Your Story: Over 100 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition InfluxDB hoodie as a thank you.

Parsing Logs into InfluxDB Using Telegraf's Logparser Plugin

Cameron Sparr (InfluxData) — Thu, 23 Jun 2016 06:15:34 -0700

Warning! This post is a little old. Please review the latest Telegraf documentation for complete updates.

InfluxData is excited to announce that it now supports the ability to stream log event data to InfluxDB with out-of-the-box patterns for popular servers like Nginx and Apache. Log monitoring support is the latest addition to Telegraf’s already impressive list of 90+ input and output data plug-ins. In addition to standard format logs, Telegraf logparser, custom log parsing and patterns based on the popular “grok” patterns are also supported.

What is Telegraf? Telegraf is a metric collection daemon that can collect metrics from a wide array of inputs and write them into a wide array of outputs. It is plugin-driven for both collection and output of data so it is easily extendable. It is also written in Go, which means that it is a compiled and standalone binary that can be executed on any system with no need for external dependencies, no npm, pip, gem, or other package management tools required.

With Telegraf you can not only parse logs to create metrics and store them as time series data—InfluxDB also has the ability to store raw log lines alongside the metrics. We should caution that this new capability is not intended to be a replacement for full-text search tools, rather for aiding in root cause analysis during specific windows of triage. So, if you are investigating performance metrics within a band of time, you can search via regex for any log errors that occurred in that same band. All the data is conveniently in one place, making it fast to lookup and correlate data.

The logparser plugin can be utilized in Telegraf to structure any log data into queryable and searchable metrics. In this guide we will use one of Telegraf’s built-in patterns to parse logs that are in the “Combined Log Format”, which is a common setup in Apache and Nginx access logs. It is an extension of the Common Log Format with the addition of two quoted strings for “Referrer” and “User-Agent”:

lang:default decode:true">127.0.0.1 ident auth [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://localhost:8083/" "Chrome/51.0.2704.84"

In Nginx, configuring your server to output logs like this would look something like this in /etc/nginx/nginx.conf:

log_format vhosts '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';

access_log /var/log/nginx/access.log vhosts;

And like this in Apache (see here for more details):

"%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\""

Step 1 -- Install Telegraf

Telegraf packages are available for many different platforms from the InfluxData downloads page:

# Install on Ubuntu:
wget https://dl.influxdata.com/telegraf/releases/telegraf_1.0.0_amd64.deb
sudo dpkg -i telegraf_1.0.0_amd64.deb
service telegraf stop

Step 2 -- Running Through a Test Metric

First create a temp log file:

touch /tmp/test.log

Next, setup a simplified telegraf.conf file by running the following echo command.

echo '
[[inputs.logparser]]
  ## file(s) to tail:
  files = ["/tmp/test.log"]
  from_beginning = false
  name_override = "test_metric"
  ## For parsing logstash-style "grok" patterns:
  [inputs.logparser.grok]
    patterns = ["%{COMBINED_LOG_FORMAT}"]

[[outputs.file]]
  ## Files to write to, "stdout" is a specially handled file.
  files = ["stdout"]' >> /tmp/telegraf.conf

The file tells Telegraf to run using the logparser input plugin and the file output plugin. The logparser input will tail the /tmp/test.log file, parsing and outputting metrics with the name test_metric. The output will be written using the file output plugin to stdout.

Now we will use that config file to run Telegraf:

telegraf --config /tmp/telegraf.conf

While Telegraf is running, we will echo the example log line to the /tmp/test.log file.

bash
echo '127.0.0.1 ident auth [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://localhost:8083/" "Chrome/51.0.2704.84"' >> /tmp/test.log

In the Telegraf output, you should see the above log line get parsed into an InfluxDB line-protocol metric and printed to STDOUT.

test_metric,host=localhost,http_method=GET agent="Chrome/51.0.2704.84",auth="auth",client_ip="127.0.0.1",http_status_code=200i,http_version=1,ident="ident",referrer="http://localhost:8083/",request="/apache_pb.gif",response_bytes=2326i 971211336000000000

Parsing Access Logs and Writing Real Data into InfluxDB

The next steps will build on the previous test data and parse real access log data into InfluxDB.

These steps will deal with parsing an Nginx log file. If you don’t have one handy, you can download an example here.

Step 3 -- Install InfluxDB

InfluxDB packages are available for many different platforms from the downloads page.

# Install on Ubuntu:
wget https://dl.influxdata.com/influxdb/releases/influxdb_1.0.0_amd64.deb
sudo dpkg -i influxdb_1.0.0_amd64.deb
service influxdb start

Step 4 -- Configure Telegraf

First we will rename the default Telegraf config file:

mv /etc/telegraf/telegraf.conf /etc/telegraf/telegraf.conf.orig

Now write a new config file:

echo '
[[inputs.logparser]]
  ## files to tail.
  files = ["/var/log/nginx/access.log"]
  ## Read file from beginning.
  from_beginning = true
  ## Override the default measurement name, which would be "logparser_grok"
  name_override = "nginx_access_log"
  ## For parsing logstash-style "grok" patterns:
  [inputs.logparser.grok]
    patterns = ["%{COMBINED_LOG_FORMAT}"]

[[outputs.influxdb]]
  ## The full HTTP or UDP endpoint URL for your InfluxDB instance.
  urls = ["http://localhost:8086"] # required
  ## The target database for metrics (telegraf will create it if not exists).
  database = "telegraf" # required
  ## Write timeout (for the InfluxDB client), formatted as a string.
  timeout = "5s"
' >> /etc/telegraf/telegraf.conf

This time our config file will assume that you are running an InfluxDB instance locally. We will also be tailing an nginx log file at /var/log/nginx/access.log. If you don’t have an Nginx access log handy, you can download an example here.

Step 5 -- Parsing the Nginx Access Log

Now we can start the Telegraf service daemon:

service telegraf start

We won’t see metrics getting written to STDOUT, but we should see log lines indicating that metrics are getting written to InfluxDB.

The Telegraf service log is located at /var/log/telegraf/telegraf.log.

2016/06/16 15:32:50 Output [influxdb] buffer fullness: 600 / 10000 metrics. Total gathered metrics: 600. Total dropped metrics: 0.
2016/06/16 15:32:50 Output [influxdb] wrote batch of 600 metrics in 5.809µs

Step 6 -- Exploring the Data in InfluxDB

When we installed InfluxDB, we also installed the Influx CLI client for connecting to and querying the database. We can launch this by simply running influx and using the telegraf database.

$ influx
Connected to http://localhost:8086 version 1.0.0
InfluxDB shell version: 1.0.0
> USE telegraf
Using database telegraf
> precision rfc3339
>

First let’s query our parsed data and see how many metrics we’ve got:

> SELECT count(resp_code) FROM nginx_access_log
name: nginx_access_log
----------------------
time			           count
1970-01-01T00:00:00Z	           600

Perfect! This is the number of parsable metrics in the example log file.

Now let’s see how many errors we’ve returned using the count() function:

> SELECT count(resp_code) FROM nginx_access_log WHERE resp_code >= 400
name: nginx_access_log
----------------------
time			           count
1970-01-01T00:00:00Z	           71

71 errors is quite a bit for only 600 requests in total! Let’s take a peek at a few of those errors using * to grab all fields and LIMIT to limit the number returned.

> SELECT * FROM nginx_access_log WHERE resp_code >= 400 LIMIT 5
name: nginx_access_log
----------------------
time				agent			auth	client_ip	host	http_version	ident	referrer	request	resp_bytes	resp_code	verb
2011-08-22T01:02:01.005Z	okhttp/2.4.0		-	192.168.1.2	tars	1.1		-	-		/huh	0		404	GET
2011-08-22T01:02:01.008Z	Python-urllib/2.7	-	127.0.0.1	tars	1.1		-	-		/huh	0		404	GET
2011-08-22T01:02:02.005Z	Firefox/5.0		-	127.0.0.1	tars	1.1		-	-		/huh	0		404	GET
2011-08-22T01:02:03.006Z	Firefox/5.0		-	127.0.0.1	tars	1.1		-	-		/huh	0		404	GET
2011-08-22T01:02:04.009Z	Firefox/5.0		-	127.0.0.1	tars	1.1		-	-		/huh	0		404	GET

Looks like we’ve got some 404s. If we look closely, we can see that each of the 404s are coming from requests on the /huh endpoint. Let’s see if every 404 is on that endpoint—for this we will use the distinct() function.

> SELECT distinct(request) FROM nginx_access_log WHERE resp_code >= 400
name: nginx_access_log
----------------------
time			          distinct
1970-01-01T00:00:00Z	          /huh
1970-01-01T00:00:00Z	          /foo

Looks like there might be a few people getting 404 from the /foo endpoint as well, so let’s see how many:

> SELECT count(resp_code) FROM nginx_access_log WHERE resp_code >= 400 AND request = '/foo'
name: nginx_access_log
----------------------
time			           count
1970-01-01T00:00:00Z	           1

Great! Since there’s only one, this means that the main issue is users querying the /huh endpoint, and now we can deal with finding out where that errant endpoint is coming from.

Customizing the Logparser Plugin

Now that we’ve covered how to parse data from standard “Combined Log Format”, let’s look into parsing logs that might have additional fields. Apache and Nginx allow for a high level of customization to their log output, and likewise so does the Logparser.

Let’s say that our Apache admin added 2 fields on to the end of each log line:

the ID of the request and
the response time in microseconds.

While the response time is likely very useful from an analytics perspective, the ID is probably not as useful, so let’s come up with a grok pattern to deal with this.

Say the log lines look like this (see the full file here):

127.0.0.1 - - [22/Aug/2011:10:02:59 +0900] "GET /huh HTTP/1.1" 404 0 "-" "Firefox/5.0" f18064bf-a4c5-4fd9-9cf8-37aff27382c5 3648

Which is the Combined Log Format with the addition of a UUID and integer at the end. So, how do we go about parsing this?

The Telegraf Grok Logparser

In order to parse a custom log file we must create a custom “grok” pattern for Telegraf. The logparser plugin uses a slightly modified version of logstash ”grok” patterns, with the format:

%{<capture_syntax>[:<semantic_name>][:<modifier>]}

Telegraf has many of its own built-in patterns, as well as supporting Logstash’s built-in patterns. The best way to get acquainted with grok patterns is to read the Logstash docs, which are available here.

Setting Up a Custom Log Pattern

Telegraf allows for specifying custom patterns in a separate file, or directly in the config file. Let’s change our config file to include our custom pattern.

service telegraf stop
vim /etc/telegraf/telegraf.conf

telegraf.conf:

[[inputs.logparser]]
  ## files to tail.
  files = ["/var/log/myapp/access.log"]
  ## Read file from beginning.
  from_beginning = true
  ## Override the default measurement name, which would be "logparser_grok"
  name_override = "my_log"
  ## For parsing logstash-style "grok" patterns:
  [inputs.logparser.grok]
    patterns = ["%{CUSTOM_LOG}"]
    custom_patterns = '''
      CUSTOM_LOG %{COMBINED_LOG_FORMAT} %{UUID} %{NUMBER:response_time_us:int}
    '''

[[outputs.influxdb]]
  ## The full HTTP or UDP endpoint URL for your InfluxDB instance.
  urls = ["http://localhost:8086"] # required
  ## The target database for metrics (telegraf will create it if not exists).
  database = "telegraf" # required
  ## Write timeout (for the InfluxDB client), formatted as a string.
  timeout = "5s"

Notice that we need to add the CUSTOM_LOG pattern, as well as telling Telegraf to use the %{CUSTOM_LOG} pattern when parsing.

Let’s go over the fields in this pattern:

CUSTOM_LOG is the name of the pattern.
%{COMBINED_LOG_FORMAT} is the name of the built-in pattern.
%{UUID} uses the Logstash built-in UUID pattern to recognize this portion of the log file, but does not store it.
%{NUMBER:response_time_us:int} uses the Logstash built-in NUMBER pattern. It names this field response_time_us and casts it to an int.

Parsing the Log & Analyzing the Data

Now that our config file is setup, we can run the Telegraf service again (but first make sure that you have a file at /var/log/myapp/access.log, you can grab the example here.)

service telegraf start

And once the metrics are written we can enter the influx CLI again:

$ influx
Connected to http://localhost:8086 version 1.0.0
InfluxDB shell version: 1.0.0
> USE telegraf
Using database telegraf
> precision rfc3339

And now that we have access to the response times, let’s see what our average & 95th percentile response times are:

> SELECT mean(response_time_us),percentile(response_time_us,95) FROM my_log
name: my_log
------------
time				mean		percentile
1970-01-01T00:00:00Z	        912411.36	10078023

Yikes, seems a bit high, so let’s peek at some of the metrics that have a very large response time:

> SELECT * FROM my_log WHERE response_time_us > 10078023 LIMIT 5
name: my_log
------------
time				agent			auth	client_ip	host	http_version	ident	referrer	request		resp_bytes	resp_code	response_time_us	verb
2011-08-22T01:02:01.008Z	Firefox/5.0		userb	127.0.0.1	tars	1.1		-	-		/large.jpg	10485760	200		10181259		GET
2011-08-22T01:02:04.004Z	Firefox/5.0		usera	127.0.0.1	tars	1.1		-	-		/large.jpg	10485760	200		10110514		GET
2011-08-22T01:02:06.004Z	Python-urllib/2.7	-	127.0.0.1	tars	1.1		-	-		/large.jpg	10485760	200		10132756		GET
2011-08-22T01:02:10.002Z	Firefox/5.0		userb	192.168.1.2	tars	1.1		-	-		/large.jpg	10485760	200		10101485		GET
2011-08-22T01:02:11.006Z	Firefox/5.0		-	127.0.0.1	tars	1.1		-	-		/large.jpg	10485760	200		10144209		GET

Aha! These are serving up a very large image file. Let’s ignore these and view what our response time is in cases with smaller requests:

> SELECT mean(response_time_us),percentile(response_time_us,95) FROM my_log WHERE resp_bytes < 4096
name: my_log
------------
time				mean				percentile
1970-01-01T00:00:00Z	4003.456043956044	7633

>

Great! That looks a bit better.

What's Next?

Download and get started with InfluxDB 1.3.1.
Looking for InfluxDB clustering on your infrastructure? Get started with InfluxDB Enterprise, now available for evaluation.
Schedule a FREE 20 minute consultation with a Solutions Architect to review your InfluxDB project.

Announcing Telegraf 0.11 with UDP & TCP Input Plugins Plus NTP Query & Processes Inputs

Cameron Sparr (InfluxData) — Tue, 15 Mar 2016 13:30:24 -0700

We are excited to announce the immediate availability of Telegraf 0.11, which includes support for TCP and UDP input plugins. These plugins are able to parse multiple data formats (including InfluxDB line protocol and graphite) and write them to any configured output.

We are also releasing three new plugins for gathering additional system metrics:

NTP Query runs the ntpq command and returns NTP query metrics such as offset, delay, jitter, etc.
Processes gathers information on the overall processes running on the system. It counts total processes, as well as processes in states such as zombie, sleeping, running, etc. On Linux it will also gather thread information.
Kernel gathers generic kernel metrics available in /proc/stat, this plugin is only available on Linux.

Additional features and bugfixes in this release below…

Features

#692: Support InfluxDB retention policies
#771: Default timeouts for input plugns. Thanks @PierreF!
#758: UDP Listener input plugin, thanks @whatyouhide!
#769: httpjson plugin: allow specifying SSL configuration.
#735: SNMP Table feature. Thanks @titilambert!
#754: docker plugin: adding docker info metrics to output. Thanks @titilambert!
#788: -input-list and -output-list command-line options. Thanks @ebookbug!
#778: Adding a TCP input listener.
#797: Provide option for persistent MQTT consumer client sessions.
#799: Add number of threads for procstat input plugin. Thanks @titilambert!
#776: Add Zookeeper chroot option to kafka_consumer. Thanks @prune998!
#811: Add processes plugin for classifying total procs on system. Thanks @titilambert!
#235: Add number of users to the system input plugin.
#826: "kernel" linux plugin for /proc/stat metrics (context switches, interrupts, etc.)
#847: ntpq: Input plugin for running ntp query executable and gathering metrics.

Bugfixes

#748: Fix sensor plugin split on ":"
#722: Librato output plugin fixes. Thanks @chrusty!
#745: Fix Telegraf toml parse panic on large config files. Thanks @titilambert!
#781: Fix mqtt_consumer username not being set. Thanks @chaton78!
#786: Fix mqtt output username not being set. Thanks @msangoi!
#773: Fix duplicate measurements in snmp plugin. Thanks @titilambert!
#708: packaging: build ARM package
#713: packaging: insecure permissions error on log directory
#816: Fix phpfpm panic if fcgi endpoint unreachable.
#828: fix net_response plugin overwriting host tag.
#821: Remove postgres password from server tag. Thanks @menardorama!

What's next

Get started with Telegraf here.

Announcing Telegraf 0.10.3 with Support for more Message Brokers and Arbitrary Data Formats

Cameron Sparr (InfluxData) — Fri, 19 Feb 2016 07:00:30 -0700

Here’s what’s new in the Telegraf 0.10.3 release…

New Message Brokers

Telegraf now supports consuming metrics from MQTT and NATS message brokers (in addition to the existing Kafka consumer). In addition to these new message brokers, Telegraf can now also consume data that is in a variety of formats, rather than only line-protocol. Read on to learn more…

Arbitrary Data Formats

Input plugins

For incoming textual data, such as via the exec or any of the *_consumer input plugins, users can now specify the data format that they would like to parse, current option are:

influx (line protocol)
graphite
JSON

Each of these options has a unique set of configuration options. More info on each of them can be found here.

Output Plugins

For outgoing textual data, such as via the file or any message broker output (NSQ, MQTT, Kafka, etc.), users can now specify the data format they would like to serialize metrics to, current options are:

influx (line protocol)
graphite

Each of these options has a unique set of configuration options. More info on each of them can be found here.

Future Work

We would like to support more data formats and more powerful parsing and serializing templates in the future. Members of the community are encouraged to submit PRs for new parser (input) and serializer (output) plugins.

flush_buffer_when_full Configuration Option

In order to better support message broker consumers, Telegraf now has the option to flush it’s internal buffer whenever it fills up, regardless of the specified flushing interval. This will allow users to guarantee that metrics will not be overwritten no matter how high the load gets.

This option is available under the [agent] configuration section.

New Plugins

The following is a list of new plugins added in 0.10.3:

dovecot (input)
raindrops (input)
mqtt_consumer (input)
nats_consumer (input)
couchdb (input)
mesos (input)
file (output)

View the changelog for a full list of changes.

Announcing Telegraf 0.10.0

Cameron Sparr (InfluxData) — Tue, 12 Jan 2016 21:20:56 -0700

Today we are officially releasing Telegraf 0.10.0. Going forward, all InfluxData products under the same major release will be guaranteed compatible with one another, and thus Telegraf is jumping forward to 0.10.x to coincide with the upcoming 0.10.0 InfluxDB release (late-January).

Release 0.10.0 brings a raft of new changes and improvements, many of which are breaking changes. There are a few reasons why we are making a breaking release, which I’ll go over below. See the end of this blog post for a guide on migrating from Telegraf 0.2.x to 0.10.0.

Fields instead of measurements

Telegraf previously put almost every metric it collected into a single measurement with a single field called value, ie, cpu_usage_idle value=100, cpu_usage_guest value=0. Going forward, Telegraf will (for the most part) put all plugins into a single measurement with a field for each metric, ie, cpu usage_guest=0,usage_idle=100. This has many benefits, one of which is that users can easily make math queries on relationships between different metrics within a plugin. For example, with the memory plugin, you can calculate the percent of total memory buffered like this:

SELECT buffered/total FROM mem Another considerable benefit is that this will reduce the amount of network traffic that each Telegraf instance generates. The reason for this is primarily that timestamps and tags are now only sent once for each plugin, rather than once for each measurement. The reduction that you see will depend on which plugins you use, but most users can expect at least a 50% reduction in network traffic.

Renaming plugins -> inputs

Telegraf began as an agent for collecting metrics from plugins and writing them into InfluxDB. Naturally, the input plugins were simply called “plugins”. Telegraf, however, has outgrown this name, since we now have “output plugins” in addition to “input plugins”. There are also plans for the future to have things like “aggregator plugins” and “filter plugins”. For this reason, we are renaming ”plugins” to “inputs”, and restructuring where they appear in the repo. Inputs and Outputs are now both under the plugins/ directory in the root repo. This also means that the config for each input has changed from [[plugins.cpu]] to [[inputs.cpu]]

Input Configuration

For 0.10.0 there are a few new input configuration options:

name_override: Override the base name of the measurement. (Default is the name of the plugin).
name_prefix: Specifies a prefix to attach to the measurement name.
name_suffix: Specifies a suffix to attach to the measurement name.
tags: A map of tags to apply to a specific input's measurements.

Examples

This plugin will emit measurements with the name cpu_total instead of only cpu

[[inputs.cpu]]
  name_suffix = "_total"
  percpu = false
  totalcpu = true

This will emit measurements with the name foobar instead of cpu

[[inputs.cpu]]
  name_override = "foobar"
  percpu = false
  totalcpu = true

This plugin will emit measurements with two additional tags: tag1=foo and tag2=bar

[[inputs.cpu]]
  percpu = false
  totalcpu = true
  [inputs.cpu.tags]
    tag1 = "foo"
    tag2 = "bar"

Plugin Parallelization

Over the past 9 months, Telegraf has added a lot of new functionality, and some of that didn’t really fit into the way that Telegraf was structured and organized. As time progressed, it became clear that we would need to create an avenue for multiple instances of the same plugin to run.

Many contributors solved this problem by setting up internal configuration lists and threads. Take the exec plugin, which was originally configured like this:

# telegraf.conf

[inputs.exec]
  # specify commands via an array of tables
  [[inputs.exec.commands]]
    # the command to run
    command = "/usr/bin/mycollector --foo=bar"
    # name of the command (used as a prefix for measurements)
    name = "mycollector"
    # Only run this command if it has been at least this many
    # seconds since it last ran
    interval = 10

  [[inputs.exec.commands]]
    command = "/usr/bin/othercollector"
    name = "othercollector"
    interval = 15

This is a good solution, and allows users of the exec plugin to specify multiple binaries to be executed. But there are some problems here. Since each exec.command is defined within the plugin, it does not have access to Telegraf’s general plugin configuration options. The interval parameter that you see above is a custom implementation specific to the exec plugin. Another problem is that it requires the plugin writer to internally handle threading and errors, when that functionality is already builtin to the Telegraf agent.

So going forward, this internal threading has largely been removed from inputs that were using it. Instead, users can now define inputs as lists, allowing the above configuration to change to this:

# telegraf.conf

[[inputs.exec]]
  # the command to run
  command = "/usr/bin/mycollector --foo=bar"
  # name_prefix is a new parameter in 0.10.0 available to all inputs
  name_prefix = "mycollector_"
  # This is the general interval parameter available to all inputs.
  interval = "10s"

[[inputs.exec]]
  command = "/usr/bin/othercollector"
  name_prefix = "othercollector_"
  interval = "15s"

This also means, unfortunately, that users of these inputs will need to rewrite their configuration to work with Telegraf 0.10.0. Use telegraf -sample-config to see examples of the new config for your particular input.

The following inputs are affected by this change:

exec
httpjson
jolokia
postgresql
procstat
rabbitmq
twemproxy

Linux package moving out of `/opt`

By popular demand, we are moving away from using the opt directory for Telegraf (as we’ve already done with InfluxDB and Kapacitor).

Below is a summary of the linux package changes:

Binaries are now located at `/usr/bin/`
Configuration is now located at `/etc/telegraf`
A few fixes and updates to the `init.sh` script
Minor update to the `telegraf.service` unit file to correct usage of variable
Added a `post-install.sh` and `pre-install.sh` to handle upgrades and other housekeeping

Migrating from Telegraf 0.2.x

Config File

If you are currently using Telegraf 0.2.x, you should start by installing version 0.10.0 and generating a new config file:

$ telegraf -sample-config > /tmp/test.0.10.0.conf

Open the file in the editor of your choice and find the plugins that you wish to configure. Inspect the differences between the old and new configuration and merge the new config with yours. Make sure you rename plugin -> input. Most plugins will have identical configuration options, but some are substantially different. The following plugins have substantially different configurations.

exec
httpjson
jolokia
postgresql
procstat
rabbitmq
twemproxy

If you need additional help configuring these, please refer to the Plugin Parallelization section above, and see the changelog.

Fields Instead of Measurements

When you start using Telegraf 0.10.0, you will notice that your previous measurements will no longer be getting new metrics. This is because almost all plugins have changed to report metrics as multiple fields on a single measurement.

To get an idea of the change, try running telegraf in test mode in both versions:

# Telegraf 0.2.x
$ telegraf -config /etc/opt/telegraf/telegraf.conf -test

# Telegraf 0.10.0
$ telegraf -config /etc/telegraf/telegraf.conf -test

And compare the output, you should see that there are less measurements but more fields. The mem plugin, for example, used to report all metrics as their own measurements (ie, mem_free, mem_used, mem_buffered). So querying your buffered memory from influxdb would look like this:

# Telegraf 0.2.x
SELECT value FROM mem_buffered

As of Telegraf 0.10.0, the plugin will report a single measurement called mem, with fields for each metric. So querying your buffered memory from influxdb now looks like this:

# Telegraf 0.10.0
SELECT buffered FROM mem

You will need to change these queries in your InfluxDB instrumentation and clients. For more information on the differences between fields, measurements, and tags, see the docs.

Additional Changes

View the changelog for a rundown of all changes made.

Announcing Telegraf 0.2.4 with MailChimp API Support

Cameron Sparr (InfluxData) — Mon, 07 Dec 2015 12:00:03 -0700

MailChimp is an email-marketing platform that provides a robust JSON API of metrics on email campaigns. Since Telegraf can make HTTP API calls and parse JSON, it was a natural step to allow Telegraf to parse data from services that provide metrics via an API key.

The MailChimp API provides many metrics such as emails_sent, clicks_total, and open_rate for all email campaigns via their /reports API. More details on the available MailChimp metrics are documented on the MailChimp website.

With Telegraf 0.2.4, you can now setup the mailchimp plugin to gather metrics and forward those onto InfluxDB or any other configured output sink. Below is a full configuration example that will collect MailChimp metrics and send them to a local InfluxDB instance.

# Configuration for telegraf agent
[agent]
  # Default data collection interval for all plugins
  interval = "30s"

###############################################################################
#                                  OUTPUTS                                    #
###############################################################################

[outputs]

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
  # The full HTTP or UDP endpoint URL for your InfluxDB instance.
  urls = ["http://localhost:8086"] # required
  # The target database for metrics (telegraf will create it if not exists)
  database = "telegraf" # required
  # Precision of writes, valid values are n, u, ms, s, m, and h
  # note: using second precision greatly helps InfluxDB compression
  precision = "s"

###############################################################################
#                                  PLUGINS                                    #
###############################################################################

[plugins]

# Gathers metrics from the /3.0/reports MailChimp API
[[plugins.mailchimp]]
  # MailChimp API key
  # get from https://admin.mailchimp.com/account/api/
  api_key = "" # INSERT API KEY HERE
  # Reports for campaigns sent more than days_old ago will not be collected.
  # 0 means collect all.
  days_old = 14

The configuration above is using the days_old = 14 argument to tell Telegraf to only collect metrics on email campaigns that were sent in the past 14 days. If you set this to 0, it will collect metrics for all email campaigns.

Paste the above into a file of your choice (I’ll use ~/telegraf.conf) and insert your MailChimp API Key and run a test output, which will output your MailChimp metrics converted into the InfluxDB line protocol format.

$ telegraf -config ~/telegraf.conf -test
* Plugin: mailchimp, Collection 1
> mailchimp_emails_sent,campaign_title=MyCampaign,id=1aa2bb34c5 value=7640i 1449103122757261374
> mailchimp_abuse_reports,campaign_title=MyCampaign,id=1aa2bb34c5 value=0i 1449103122757261374
> mailchimp_unsubscribed,campaign_title=MyCampaign,id=1aa2bb34c5 value=38i 1449103122757261374
> mailchimp_hard_bounces,campaign_title=MyCampaign,id=1aa2bb34c5 value=38i 1449103122757261374
> mailchimp_soft_bounces,campaign_title=MyCampaign,id=1aa2bb34c5 value=71i 1449103122757261374
> mailchimp_syntax_errors,campaign_title=MyCampaign,id=1aa2bb34c5 value=0i 1449103122757261374
[...]

You can see from the output that the plugin will collect available metrics for each campaign, as well as collecting the industry and list stats metrics.

Telegraf can now be run and start outputting it’s metrics to InfluxDB: $ telegraf -config ~/telegraf.conf

Future Work

MailChimp is just the first of many 3rd-party APIs that we plan on supporting in Telegraf. Users are encouraged to try their hand at contributing a plugin, and we plan to support AWS Metrics soon. You can view the source for the MailChimp plugin here.

Hosted InfluxDB + Grafana

Get configured and into production in under a minute. Try it free for 14 days. Click to get started!

Announcing Telegraf 0.2.1 with Support for Writing UDP to InfluxDB and Multiple Outputs of the Same Type

Cameron Sparr (InfluxData) — Tue, 17 Nov 2015 08:00:58 -0700

Release Notes

UDP InfluxDB output now supported
Telegraf will now compile on FreeBSD
Users can now specify outputs as lists, allowing multiple outputs of the same type.
Telegraf will no longer use docker-compose for "long" unit test, it has been changed to run docker commands in the Makefile. See make docker-run and make docker-kill. make test will still run all unit tests with docker.
Long unit tests are now run in CircleCI, with docker & race detector
Redis plugin tag has changed from host to server
HAProxy plugin tag has changed from host to server

Features

#325: NSQ output. Thanks @jrxFive!
#318: Prometheus output. Thanks @oldmantaiter!
#338: Restart Telegraf on package upgrade. Thanks @linsomniac!
#337: Jolokia plugin, thanks @saiello!
#350: Amon output.
#365: Twemproxy plugin by @codeb2cc
#317: ZFS plugin, thanks @cornerot!
#364: Support InfluxDB UDP output.
#370: Support specifying multiple outputs, as lists.
#372: Remove gosigar and update go-dockerclient for FreeBSD support. Thanks @MerlinDMC!

Bugfixes

#331: Dont overwrite host tag in redis plugin.
#336: Mongodb plugin should take 2 measurements.
#351: Fix continual "CREATE DATABASE" in writes
#360: Apply prefix before ShouldPass check. Thanks @sotfo!

Announcing Telegraf 0.2.0 with StatsD, MQTT, and Improved Scalability

Cameron Sparr (InfluxData) — Wed, 28 Oct 2015 08:00:03 -0700

We are excited to announce that Telegraf 0.2.0, InfluxDB’s native, open source data collecting agent, is now available!

What's New

PR #205: Include per-db redis keyspace info
PR #241: MQTT Output. thanks @shirou!
PR #240: procstat plugin, thanks @ranjib!
PR #244: netstat plugin, thanks @shirou!
PR #262: zookeeper plugin, thanks @jrxFive!
PR #237: statsd service plugin
PR #273: puppet agent plugin, thanks @jrxFive!
PR #286: bcache plugin, thanks @cornerot!
PR #287: Batch AMQP output, thanks @ekini!
PR #301: Collect on even intervals
PR #300: aerospike plugin. thanks @oldmantaiter!
PR #322: Librato output. thanks @jipperinbham!

Release Notes

The -test flag will now only output 2 collections for plugins that need it.
There is a new agent configuration option: flush_interval. This option tells Telegraf how often to flush data to InfluxDB and other output sinks. For example, users can set interval = "2s"and flush_interval = "60s" for Telegraf to collect data every 2 seconds and flush every 60 seconds.
precision and utc are no longer valid agent config values. precision has moved to the InfluxDB output config, where it will continue to default to s
Debug and test output will now print the raw line-protocol string.
Telegraf will now, by default, round the collection interval to the nearest even interval. This means that interval="10s" will collect every :00, :10, etc.
To address scaling concerns Telegraf now adds a small amount of "jitter" to the flush period. This ensures that if a large number of Telegraf instances are deployed, they do not all flush at the same instant. The amount of jitter can be controlled via the configuration file.
Telegraf will now retry metric flushes twice.

Get your InfluxDB hoodie while they last!

Does your company have a cool app or product that uses InfluxDB in production? We’d love to feature it on influxdb.com and give you a shout out on social media. As a “thank you” gift, we’ll send you an InfluxDB hoodie and a pack of stickers. Just fill out this simple form and we’ll let you know when your entry is live and the hoodie is in the mail. Drop us a line at contact@influxdb.com if you have any questions. Thanks for the support!

Announcing Telegraf 0.1.9 with Clustering, OpenTSDB and AMQP support

Cameron Sparr (InfluxData) — Wed, 23 Sep 2015 08:00:32 -0700

We are excited to announce that Telegraf 0.1.9, InfluxDB’s native, open source data collecting agent, is now available!

What's new

Support for InfluxDB Clusters

PR #143: Telegraf can now write data to InfluxDB clusters by listing out your instances in the URL parameter of the Telegraf config file. Telegraf will randomly select servers until it successfully makes a write.

Support for filtering telegraf outputs on the CLI

PR #217: Telegraf will now allow filtering of output sinks on the command-line using the -outputfilterflag, much like how the -filter flag works for plugins.

Support for filtering on config-file creation

PR #217: Telegraf now supports filtering to -sample-config command. You can now run telegraf -sample-config -filter cpu -outputfilter influxdb to get a config file with only the cpu plugin defined, and the influxdb output defined.

Makefile GOBIN support

PR #181: make will now work despite a custom bin directory, courtesy of @Vye.

AMQP output plugin

PR #200: Telegraf now supports AMQP output, courtesy of @ekini.

OpenTSDB output plugin

PR #182: Telegraf now supports output to OpenTSDB using telnet mode, courtesy of @rplessl.

Retry failed server connections

PR #187: Telegraf can now be configured to retry the connection to InfluxDB if the initial connection fails.

Add a port tag to the Apache plugin

PR #220: You can now select your Apache port number depending on the URL scheme, courtesy of @neezgee.

Release notes

Config change

InfluxDB output config change: url is now urls, and is a list. Config files will still be backwards compatible if only url is specified.

`-test` flag output

The -test flag will now output two metric collections

Breaking change: CPU collection plugin

The CPU collection plugin has been refactored to fix some bugs and outdated dependency issues. At the same time, I also decided to fix a naming consistency issue, so cpu_percentageIdle will becomecpu_usage_idle. Also, all CPU time measurements now have it indicated in their name, so cpu_idle will become cpu_time_idle. Additionally, cpu_time measurements are going to be dropped in the default config.

Breaking change: Memory plugin

The memory plugin has been refactored and some measurements have been renamed for consistency. Some measurements have also been removed from being outputted. They are still being collected bygopsutil, and could easily be re-added in a “verbose” mode if there is demand for it.

…plus 11 bug fixes! Get the full story by checking out the official 0.1.9 changelog on GitHub.

InfluxData Blog - Cameron Sparr

Announcing Telegraf 1.0 - The Plugin-Driven Agent for Collecting Metrics

New in this release

Beyond 1.0

What's next

Parsing Logs into InfluxDB Using Telegraf's Logparser Plugin

Step 1 -- Install Telegraf

Step 2 -- Running Through a Test Metric

Parsing Access Logs and Writing Real Data into InfluxDB

Step 3 -- Install InfluxDB

Step 4 -- Configure Telegraf

Step 5 -- Parsing the Nginx Access Log

Step 6 -- Exploring the Data in InfluxDB

Customizing the Logparser Plugin

The Telegraf Grok Logparser

Setting Up a Custom Log Pattern

Parsing the Log & Analyzing the Data

What's Next?

Announcing Telegraf 0.11 with UDP & TCP Input Plugins Plus NTP Query & Processes Inputs

Features

Bugfixes

What's next

Announcing Telegraf 0.10.3 with Support for more Message Brokers and Arbitrary Data Formats

New Message Brokers

Arbitrary Data Formats

Input plugins

Output Plugins

Future Work

flush_buffer_when_full Configuration Option

New Plugins

Announcing Telegraf 0.10.0

Fields instead of measurements

Renaming plugins -> inputs

Input Configuration

Examples

Plugin Parallelization

Linux package moving out of `/opt`

Migrating from Telegraf 0.2.x

Additional Changes

Announcing Telegraf 0.2.4 with MailChimp API Support

Future Work

Hosted InfluxDB + Grafana

Announcing Telegraf 0.2.1 with Support for Writing UDP to InfluxDB and Multiple Outputs of the Same Type

Release Notes

Features

Bugfixes

Announcing Telegraf 0.2.0 with StatsD, MQTT, and Improved Scalability

What's New

Release Notes

Get your InfluxDB hoodie while they last!

Announcing Telegraf 0.1.9 with Clustering, OpenTSDB and AMQP support

What's new

Support for InfluxDB Clusters

Support for filtering telegraf outputs on the CLI

Support for filtering on config-file creation

Makefile GOBIN support

AMQP output plugin

OpenTSDB output plugin

Retry failed server connections

Add a port tag to the Apache plugin

Release notes

Config change

-test flag output

Breaking change: CPU collection plugin

Breaking change: Memory plugin

Get your InfluxDB hoodie while they last!

`-test` flag output