Apache Kafka is an open-source publish-subscribe messaging system that receives data from disparate source systems and makes the data available to target systems in real time. It is written in Scala and Java and is often associated with real-time event stream processing for big data.
Why use a Telegraf plugin for Apache Kafka Consumer?
Pulling in all your time series data (metrics and events) from your applications, infrastructure, and even sensors makes it easy for your developers to use this data. This data can be used for monitoring your applications and infrastructure as well as enhancing your applications by being queried in a real-time visualization in your application — a single source of truth for all your time series data.
In addition, InfluxDB can handle the large write and query load for not just the data collected with an Apache Kafka Consumer Telegraf Plugin, but combined with the other Telegraf plugins and client libraries so that all the data you need is easy to access.
How to use the Apache Kafka Consumer Telegraf Plugin
The Apache Kafka Consumer Input Plugin polls a specified Kafka topic and adds messages to your InfluxDB instance. The Consumer Group is used to talk to the Kafka cluster so multiple instances of Telegraf can read from the same topic in parallel.
Key use cases using the Apache Kafka Consumer Telegraf Plugin
With Apache Kafka Consumer, you can stream data between different applications, application monitoring, fraud detection and live leader-boards.