Prometheus Telegraf Input Plugin

Prometheus is a systems monitoring and alerting toolkit hosted by the Cloud Native Computing Foundation, which also hosts Kubernetes, a container orchestration project. It’s open source and is a popular monitoring solution for Kubernetes metrics. To monitor services using Prometheus, your services need to expose a Prometheus endpoint, which is an HTTP interface with a list of metrics and the current value of the metrics.

The Prometheus server polls the interface periodically to gather metrics, store data, and send alerts. This architecture is called polling-based or pull-based monitoring. This system doesn’t require you to install an agent to collect metrics, but you do need to deploy “exporters” to expose the metrics from the system you are collecting metrics from.

Why use the Prometheus Telegraf Plugin?

The Prometheus Telegraf plugin lets you collect data from HTTP servers exposing metrics in Prometheus format. You can then store your data in InfluxDB or send it to another output destination. Pull-based systems like Prometheus work best for metrics with regular intervals, rather than irregular event-driven time series. Using Telegraf and InfluxDB with Prometheus gives you more options to work with both events and metrics. InfluxDB has built-in tools to analyze time series data and allows you to set up tasks and alerts.

How to monitor Prometheus using the Telegraf plugin

To configure this plugin you first need to set urls to an array of the URLs you want to gather metrics from. Then you need to decide which metric configuration version you want to use. With metric_version = 1 the Prometheus metric name is set as the Telegraf metric name, Prometheus labels are set as Telegraf tags, and Prometheus values are set as Telegraf field values. With metric_version = 2 the Prometheus metric name is set as a Telegraf field key, metrics hold more than one value, and field keys aren’t generic. This option can be more efficient for querying column-oriented data.

There are options you can configure to scrape for Prometheus annotation on Kubernetes pods, to monitor the Caddy HTTP server, and more on the GitHub page for this project.

Key Prometheus metrics to use for monitoring

The metrics you monitor will depend on the HTTP server you’re collecting data from.

To collect metrics you need an array of URLs to scrape metrics from, for example:

urls = ["http://localhost:9100/metrics"]

You can also set an array of Kubernetes services to scrape metrics from and/or set a Kubernetes config file to create a client from, for example:

kubernetes_services = ["http://my-service-dns.my-namespace:9100/metrics"]

kube_config = "/path/to/kubernetes.config"

Project URL   Documentation