Apache-kafka

Kafka Monitoring Template

Telegraf Plugins used:

Included Resources:

  • 2 label: kafka, jvm
  • 1 Telegraf Configuration: kafka-zk-jolokia
  • 2 Dashboards: Kafka Metrics, Kafka JVM Metrics
  • 3 Variables: bucket, kafka_broker, and kafka_topic

Quick Install

If you have your InfluxDB credentials configured in the CLI, you can install this template with:

https://raw.githubusercontent.com/influxdata/community-templates/master/kafka/kafka-template.yml

Kafka monitoring is important given the nature of what Kafka is and how it works, and the Kafka Monitoring Template was built to serve that need.

What is Kafka?

At its core, Apache Kafka is an open-source distributed even streaming platform that companies all over the world have come to depend on for not only high-performance data pipelines but also streaming analytics, data integration and even to act as the foundation of mission-critical applications as well.

Why monitor Kafka?

A major reason why Kafka monitoring is so important is the sheer volume of information — as well as potential data sources — that you're working with. Keep in mind that Kafka's native "Connect" interface allows you to integrate with just about everything in terms of event sources — from AWS S3 to Elasticsearch to Postgres, JMS and more. It can be difficult to manage this volume of real-time event processing on your own, and a Kafka monitoring dashboard can visualize everything in a way that ensures you don't miss anything important as it is happening.

Additionally, because of the potential critical nature of the data that is flowing through the Kafka event streams, it is important to monitor the health of this stream processing system. It is important to monitor the rate of requests from your producers, consumers, and followers to ensure your Kafka deployment is communicating efficiently. You can expect Kafka's request rate to rise as producers send more traffic or as your deployment scales out, adding consumers or followers that need to fetch messages.

Kafka Monitoring Dashboard

How to use the Kafka Monitoring Template

Once your InfluxDB credentials have been properly configured in the CLI, you can install the Kafka monitoring template using the Quick Install command. Once installed, the data for the dashboard will be populated by the two included Telegraf configurations, which includes the relevant Jolokia and Zookeeper Input. Note that you might need to customize the input configuration to better serve your needs, including by specifying a new input value. All of this will depend on how your organization is currently running Kafka.

To find out more information about environmental variables within the Telegraf configuration, consult the following link.

Key Kafka monitoring metrics

With the Kafka Monitoring Template, you now have access to a web-based solution that was built to help you manage your cluster, monitor system health in easy-to-use and visualized dashboards, get automated alerts and triggers, and more — all via a mechanism that puts all critical information front and center so that you can act on it as quickly as possible.

Some of the most important Kafka monitoring metrics that you should proactively monitor include:

  • Controller state
  • Active Controller count
  • Under Replicated partitions
  • Under Replicated partitions in Kafka topic
  • Bytes per broker/second
  • Bytes in per topic per second
  • Fetch Requests per Topic per second
  • Bytes per minute per broker
  • Latency
  • Alive Connections
  • Uptime
  • Physical Memory
  • Heap Memory
  • CPU Load
  • Elapsed time
  • Thread count
  • Memory pool

Related Resources

Kafka Telegraf plugin

Learn how the Kafka consumer plugin polls a specified Kafka topic and adds messages to InfluxDB.

IoT monitoring and analytics

The solution to provide real-time insight and analytics from IoT sensors and devices.

Scroll to Top