Kafka Monitoring Tool
Apache Kafka™ is a distributed streaming platform with three key capabilities. It lets you publish and subscribe to streams of records (similar to a message queue or enterprise messaging system), store streams of records in a fault-tolerant way, and process streams of records as they occur. Kafka was originally built at LinkedIn and is now part of the Apache Software Foundation and used by a variety of companies. Telegraf and InfluxDB can be used as a Kafka monitoring UI.
Telegraf Input Plugin: Kafka
The Kafka consumer plugin polls a specified Kafka topic and adds messages to InfluxDB. The plugin assumes messages follow the line protocol. Consumer Group is used to talk to the Kafka monitoring tool cluster so multiple instances of Telegraf can read from the same topic in parallel.
Telegraf Output Plugin: Kafka
This plugin writes to a Kafka Broker acting a Kafka Producer.
Apache Kafka Basics
Kafka gets used mainly for building real-time streaming data pipelines that reliably get data between systems or applications, and for building real-time streaming applications that transform or react to the streams of data.
Kafka is run as a cluster on one or more servers. The Kafka cluster stores streams of records in categories called topics. Each record consists of a key, a value, and a timestamp.
Kafka has four core APIs:
- The Producer API
- The Consumer API
- The Streams API
- The Connector API