Burrow is a method for monitoring Kafka consumers, specifically for keeping track of consumer status in Apache Kafka. Consumer status is determined by evaluating the consumer’s behavior over a sliding window to determine whether the:
- consumers are committing offsets
- offset commits are increasing
- lag is increasing
- lag is increasing in a consistent or fluctuating manner
Burrow monitors every consumer group that is committing offsets to either Kafka or Zookeeper, and monitors every topic and partition consumed by those groups.
Why use a Telegraf plugin for Burrow?
Monitoring your Apache Kafka infrastructure is important because it could be your core pipeline for your application’s data, and therefore will help ensure 100% availability. The Burrow Telegraf Plugin can help achieve this by monitoring whether or not the consumers are keeping up with the messages — through giving you a view of the offsets that the consumers are committing and the broker’s state.
You can collect these metrics into your InfluxDB instance and build alerts based on thresholds that you set to help you maintain this critical piece of your application stack.
How to monitor Kafka consumers using the Burrow Telegraf Plugin
Burrow Telegraf Plugin configurations allow you to set response times, limit concurrent connections, filter clusters, consumer groups, and topics.
Key Burrow metrics to use for monitoring
Some of the important Burrow metrics that you should proactively monitor include:
burrow_group(one event per each consumer group) and
burrow_partition(one event per each topic partition)
burrow_topic(one event per topic offset)