Category Archives: Telegraf

Kubernetes monitoring and autoscaling with Telegraf and Kapacitor


With the 1.1 release of Telegraf and Kapacitor, InfluxData is improving the ease of use, depth of metrics and level of control we provide in maintaining and monitoring a Kubernetes cluster. InfluxDB has been a part of Kubernetes’ monitoring since v0.4 when the first version of Heapster was released and InfluxDB remains the default data sink.

Continue reading Kubernetes monitoring and autoscaling with Telegraf and Kapacitor

Announcing Telegraf 1.0 – The Plugin-Driven Agent for Collecting Metrics


Telegraf 1.0 GA is here. Telegraf is InfluxDB’s native data collector, the first step of the TICK stack. You can use Telegraf to collect metrics from a wide range of inputs and write those metrics into InfluxDB or other supported outputs.

1.0 is a big date in Telegraf’s journey, and as an open-source project it has come a very long way. I took over as the maintainer of the project just over a year ago (my first commit was August 3rd), when there were 18 input plugins and only 1 output plugin (InfluxDB). Today, there are over 80 input plugins and 18 output plugins, and it’s still growing.

Since the release of 0.13.1 we’ve had nearly 40 features or enhancements added and more than 45 bugfixes. Many of the features and PRs came from the community, thank you to everyone who contributed!

New in this release

Of the new features in 1.0 there are three we want to highlight, all of which are community contributions…

  • New SNMP Plugin: The previous SNMP plugin was difficult to use and configure, and was quite limited in it’s output schema. Thanks to @phemmer for writing a brand new one.
  • New Webhooks Plugin: There is a new webhooks plugin that allows for multiple webhooks services to run on a single port. Thanks to @francois2metz and @cduez. More can be read about it here.
  • Windows Service: Telegraf now supports being installed as a real Windows Service, much thanks to @butitsnotme for this PR. You can read more here.

Continue reading Announcing Telegraf 1.0 – The Plugin-Driven Agent for Collecting Metrics

Announcing InfluxDB, Telegraf, Kapacitor and Enterprise 1.0 RC1


InfluxData team is excited to announce the 1.0 release candidate (RC1) of the Telegraf, InfluxDB, Chronograf, Kapacitor projects and InfluxEnterprise, our clustering and monitoring product. These projects have been years in the making, so are pleased to have finalized the API and all the features for the 1.0 release. From now until the 1.0 final release we’ll only merge bug fixes into the 1.0 branch.

Robust & Stable APIs

We’ve already done extensive testing on both the single server and clustered versions of InfluxDB, so these releases should be considered very stable. We’ll be rolling out the 1.0 RC to all of our customers in InfluxCloud over the next week or two.

Purpose Built for Time-Series

InfluxDB 1.0 will be shipping with our purpose built storage engine for time series data, the Time Structured Merge tree. It gives incredible write performance while delivering compression better than generalized solutions. Depending on the shape of your data, each individual value, timestamp pair can take as little as 2 bytes, including all the tag and measurement metadata. Continuous queries and retention policies give users the ability to have the database manage the data lifecycle automatically by aggregating data for longer term trends while dropping older high precision data.

Continue reading Announcing InfluxDB, Telegraf, Kapacitor and Enterprise 1.0 RC1

Parsing Logs into InfluxDB using Telegraf’s logparser Plugin


InfluxData is excited to announce that it now supports the ability stream log event data to InfluxDB with out-of-the-box patterns for popular servers like Nginx and Apache. Log monitoring support is the latest addition to Telegraf’s already impressive list of 90+ input and output data plug-ins. In addition to standard format logs, custom log parsing and patterns based on the popular “grok” patterns are also supported.

What is Telegraf? Telegraf is a metric collection daemon that can collect metrics from a wide array of inputs and write them into a wide array of outputs. It is plugin-driven for both collection and output of data so it is easily extendable. It is also written in Go, which means that it is a compiled and standalone binary that can be executed on any system with no need for external dependencies, no npm, pip, gem, or other package management tools required.

With Telegraf you can not only parse logs to create metrics and store them as time-series data, InfluxDB also has the ability to store raw log lines along-side the metrics. We should caution that this new capability is not intended to be a replacement for full text search tools, rather for aiding in root cause analysis during specific windows of triage. So, if you are investigating performance metrics within a band of time, you can search via regex for any log errors that occurred in that same band. All the data is conveniently in one place, making it fast to lookup and correlate data. Continue reading Parsing Logs into InfluxDB using Telegraf’s logparser Plugin

Improved MQTT Support in InfluxDB with SurgeMQ


As InfluxDB has grown in popularity over the past two and a half years, we’ve seen it used in many different use cases across custom DevOps monitoring, real-time analytics, and the Internet of Things (IoT). Each of these domains has its own patterns, standards, and protocols, so we’ve been working hard to make Telegraf, our open-source data collection agent, support as many services and systems as possible. This helps our users get data into and out of InfluxDB easily and enables developers to keep building awesome new applications.

MQTT is an example of one of the aforementioned protocols, which is heavily used in industrial monitoring and has become increasingly popular in IoT applications. We recently added support to Telegraf for consuming data from MQTT brokers, but given the amount of segmentation in the MQTT broker space and the specific requirements of high-performance brokers (e.g. very high numbers of concurrent connections), we started to pursue the possibility of building our own MQTT broker to ensure the tightest possible integration with the rest of our products and allow us to more effectively work with customers and partners to build IoT solutions backed by InfluxDB.

We’re huge supporters of the Go programming language, as we’ve discussed in the past, so it was important to us that the project be written in Go for optimal productivity, performance, and community engagement. When we began our investigation, we discovered the SurgeMQ project, written in Go by the brilliant and incredibly talented Jian Zhen. After a few weeks of initial discussions, we received Jian’s blessing to officially take over the project. Says Jian, “I’ve been watching the InfluxDB project from its early days, and I’m a huge fan. I’m happy to see SurgeMQ helping people use InfluxDB in new ways.”

Effective immediately, we’ll be working to finish turning SurgeMQ into a standalone server and begin building packages for it alongside the rest of our products. Continue reading Improved MQTT Support in InfluxDB with SurgeMQ

Announcing InfluxDB 1.0 Beta – A Big Step Forward for Time-Series


The team at InfluxData is excited to announce the immediate availability of InfluxDB 1.0 Beta plus the rest of the components in the TICK stack: Telegraf, Chronograf, and Kapacitor. While there are many new features like exponential smoothing via Holt Winters queries, templates for Kapacitor TICKscripts, Telegraf Cloudwatch integration, and dozens of bug fixes, this release marks a significant point in the development of these projects.

We’ve had customers and community members running the TICK-stack in production at significant scale for months and we are therefore comfortable that the quality of the codebase is worthy of the 1.0 moniker. Second, we’re ready to lock down the API and make a commitment to zero breaking changes for a significant length of time. This is especially important for organizations building products and services on top of the InfluxData stack whose products may have longer development cycles or require a higher degree of stability from the code base to ensure continuity for their customers and users.

Getting to 1.0 GA

This release is the first Beta of the upcoming 1.0 GA release. We still have some known bugs to fix, but from here until 1.0 we’ll be focused on testing, benchmarking and bug fixes. What about new features? They’ll come in the point releases after 1.0. For community members, this Beta is what you should be testing against. For some users, the Beta may even be suitable for production use. Many fixes have gone into all the projects since the 0.13 release nearly 4 weeks ago. Continue reading Announcing InfluxDB 1.0 Beta – A Big Step Forward for Time-Series

Announcing InfluxDB v0.13 Plus Clustering Early Access


The team at InfluxData is excited to announce the immediate availability of v0.13 of the TICK stack!

What’s the TICK stack? It’s InfluxData’s end-to-end platform for collecting, storing, visualizing and alerting on time-series data at scale. Learn more.

We are also pleased to announce early access to InfluxEnterprise. If you have been waiting for InfluxDB clustering to run on your infrastructure with the ability to rebalance nodes, plus a slick UI to deploy, manage and monitor the deployment, InfluxEnterprise is for you.

enterprise_gif

Like what you see? Contact us to request a demo, get early access plus pricing from one of our solutions architects.

InfluxDB v0.13

v0.13 is a significant InfluxDB release delivering robust features and stability to the single server experience. This is also our final major milestone before we launch 1.0. New in this release:

  • Upgrade to Go v1.6.2 delivering performance gains
  • Support for DELETE FROM
  • Time Zone Support – offset argument in the GROUP BY time(…) call

There’s a total of 27 new features and 32 bug fixes! Detailed release notes can be found here and downloads here.

Telegraf v0.13

New in this release:

  • New Input Plugin: “filestat” to gather metrics about file existence, size, etc.
  • HAProxy socket support
  • Support for Docker container IDs to track per container metrics

There’s a total of 17 new features and 21 bug fixes! Detailed release notes can be found here and downloads here.

Chronograf v0.13

New in this release:

  • Design update and performance improvements to the visualization index pages
  • “Visualization cards” have been replaced with more performant list items

chrono
Downloads are here.

Kapacitor v0.13

InfluxDB’s native data processing and alerting engine now supports BYOA (bring your own algorithm) analytics. UDFs written in languages of your choice running in Docker containers can be launched independent of Kapacitor by exposing a Unix socket. On startup Kapacitor will connect to the socket and begin communication, applying the UDF logic to the time series data being processed.

New in this release:

  • UDFs can be connected over a Unix socket. This enables UDFs from across Docker containers to be applied to the Influx data pipeline.
  • CLI features to ID, list and delete tasks, recordings and replays
  • API lock down to a stable 1.0 version

There’s a total of 18 new features and 8 bug fixes! Detailed release notes can be found here and and downloads here.

But wait, there’s more…

FREE Technical Papers

Today we also launched a new “Technical Papers” landing page on influxdata.com where developers and architects can download long reads that dive deep into a variety of InfluxDB related topics. Right now we’ve got three papers posted:

Make sure to check back often as we’ll be uploading new papers every other week!

FREE Virtual Training Summer Schedule is Live!

We’ve just added eight sessions to the virtual training schedule. In addition, we’ll be debuting three new topics including:

  • Benchmarking InfluxDB and Elasticsearch for Time-Series Data Management
  • Migrating from InfluxDB 0.8 and Up to 0.13
  • Intro to Kapacitor for Alerting and Anomaly Detection

What’s Next?

Announcing InfluxDB, Telegraf and Kapacitor 0.12 – Kill Query and New Functions


We are excited to announce that InfluxDB, Telegraf and Kapacitor 0.12.0 GA have been released and ready for download.

InfluxDB improvements

This is the first version of the stand-alone server of InfluxDB. It’s important to note that to upgrade you’ll have to do a small migration of the metadata. See the upgrade details here. For users of the open source products that are looking for an HA option, we’ve also completed the first version of the InfluxDB Relay.

One of the more important features of this release are query management and limitations. You can now see what queries are running by issuing the following query:

`SHOW QUERIES`

You can then kill one of those queries by running:

`KILL QUERY`

This is one of the long requested features and will now give administrators the ability to kill a query that has been running for too long. We’ve also added some configuration options that lets administrators limit how many points, series, or group by intervals a query can hit.

If a user issues a query that passes over those limits it will be terminated and an error returned back. You can see the settings in the example config file.

This release also added support for two functions that have long been on the request list: difference and moving average!

This release has 22 features and 13 bug fixes, see the CHANGELOG for full details.

Telegraf Enhancements

What’s Telegraf? It’s InfluxDB’s native data collector that leverages a plugin architecture with over 60 inputs and outputs supported already!

Lots of great features shipping with 0.12, including:

  • Parse environment variables in the config file: you can now specify env variables in your config file (such as `$USER` or `$MYSQL_HOST`). This can be used anywhere in the config file as strings, booleans, or integers for configuration items or tags.
  • JSON serializer: Telegraf now has the ability to output data in JSON.
    Parse singular values: Telegraf can parse individual values from scripts or executables. This means that you can specify something as simple as `cat /proc/sys/kernel/random/entropy_avail` as an `exec` command, and get parsed data!
  • Nagios parser: allows you to run nagios scripts using the exec plugin.
  • IPMI hardware sensors: If you have ipmitool installed on your system, you can now utilize the `ipmi_sensor` plugin to grab data from each sensor.
  • Couchbase input plugin: gather stats from your Couchbase servers.

Kapacitor Enhancements

What’s Kapacitor? It’s InfluxDB’s native alerting, pre/post data processing engine that supports UDFs (via TICKscripts) to enable things like anomaly detection in real-time.

Kapacitor v0.12.0 brings significant improvements to the TICKscript syntax. Now with two new operators to distinguish different kinds of methods you will never be left guessing what structure the task pipeline has or when a UDF is being used. In addition to the new syntax a utility for formatting TICKscripts to a common standard has been included. Now reading and writing TICKscripts is a cleaner more precise task, so you can focus on the problem at hand. Along with these changes several bugs have been squashed, see the CHANGELOG release notes for more details.

What’s next

  • Download 0.12 of the TICK stack!
  • Need help migrating from 0.8x or 0.9x to 0.11? We are here to help! Drop us a line at contact@influxdata.com to get your migration project started.
  • Looking to level up your InfluxDB knowledge? Check out our economically priced virtual and public trainings.

InfluxDB, Telegraf and Kapacitor 0.11 GA Now Available!


We’re excited to announce that InfluxDB, Telegraf and Kapacitor 0.11 GA are now available for immediate download.

New in InfluxDB 0.11 GA – Query Performance Gains

InfluxDB 0.11 has huge improvements to the query engine to improve performance, stability, and solve some of the out of memory issues that some users were seeing on larger queries. The new query engine is anywhere from 1.4x to 3.8x faster for many queries. By popular request, we’ve also started creating ARM builds! This release has many other improvements including 22 Features and 42 bug fixes. (A list of breaking changes are in the release notes section here.)

The new query engine is important because it lays the foundation for many exciting new query features. New functions and ways to transform and combine series will be in the upcoming releases as a result of this effort. We’ll also give administrators the ability to see and even kill long running queries along with other controls limiting resource utilization.

Now let’s take a look at some of the performance gains from this query engine work. The first has a 10,000 unique series each with 1,000 points. We then ran queries to count all of the points in these series. With this query we saw a 29% decrease in the query response time!

Continue reading InfluxDB, Telegraf and Kapacitor 0.11 GA Now Available!