Consul and Apache Druid Integration

Input and output integration overview

The Consul Input Plugin collects health check metrics from a Consul server, allowing users to monitor service statuses effectively.

This plugin allows Telegraf to send JSON-formatted metrics to Apache Druid over HTTP, enabling real-time ingestion for analytical queries on high-volume time-series data.

Integration details

Consul

The Consul Input Plugin is designed to gather health check statuses from all services registered with Consul, a tool for service discovery and infrastructure management. By querying the Consul API, this plugin helps users monitor the health of their services and ensure that they are operational and meeting service level agreements. It does not provide telemetry data, but users can utilize StatsD if they want to collect those metrics. The plugin offers configuration options to connect to the Consul server, manage authentication, and specify how to handle tags derived from health checks.

Apache Druid

This configuration uses Telegraf’s HTTP output plugin with json data format to send metrics directly to Apache Druid, a real-time analytics database designed for fast, ad hoc queries on high-ingest time-series data. Druid supports ingestion via HTTP POST to various components like the Tranquility service or native ingestion endpoints. The JSON format is ideal for structuring Telegraf metrics into event-style records for Druid’s columnar and time-partitioned storage engine. Druid excels at powering interactive dashboards and exploratory queries across massive datasets, making it an excellent choice for real-time observability and monitoring analytics when integrated with Telegraf.

Configuration

Consul

[[inputs.consul]]
  ## Consul server address
  # address = "localhost:8500"

  ## URI scheme for the Consul server, one of "http", "https"
  # scheme = "http"

  ## Metric version controls the mapping from Consul metrics into
  ## Telegraf metrics. Version 2 moved all fields with string values
  ## to tags.
  ##
  ##   example: metric_version = 1; deprecated in 1.16
  ##            metric_version = 2; recommended version
  # metric_version = 1

  ## ACL token used in every request
  # token = ""

  ## HTTP Basic Authentication username and password.
  # username = ""
  # password = ""

  ## Data center to query the health checks from
  # datacenter = ""

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = true

  ## Consul checks' tag splitting
  # When tags are formatted like "key:value" with ":" as a delimiter then
  # they will be split and reported as proper key:value in Telegraf
  # tag_delimiter = ":"

Apache Druid

[[outputs.http]]
  ## Druid ingestion endpoint (e.g., Tranquility, HTTP Ingest, or Kafka REST Proxy)
  url = "http://druid-ingest.example.com/v1/post"

  ## Use POST method to send events
  method = "POST"

  ## Data format for Druid ingestion (expects JSON format)
  data_format = "json"

  ## Optional headers (may vary depending on Druid setup)
  # [outputs.http.headers]
  #   Content-Type = "application/json"
  #   Authorization = "Bearer YOUR_API_TOKEN"

  ## Optional timeout and TLS settings
  timeout = "10s"
  # tls_ca = "/path/to/ca.pem"
  # tls_cert = "/path/to/cert.pem"
  # tls_key = "/path/to/key.pem"
  # insecure_skip_verify = false

Input and output integration examples

Consul

1. Service Health Monitoring Dashboard: Utilize the Consul Input Plugin to create a comprehensive health monitoring dashboard for all services registered with Consul. This allows operations teams to visualize the health status in real time, enabling quick identification of service issues and facilitating rapid responses to service outages or performance degradation.

2. Automated Alerting System: Implement an automated alerting system that uses the health check data gathered by the Consul Input Plugin to trigger notifications whenever a service status changes to critical. This setup can integrate with notification systems like Slack or email, ensuring that team members are alerted immediately to address potential issues.

3. Integration with Incident Management: Leverage the health check data from the Consul Input Plugin to feed into incident management systems. By analyzing the health status trends, teams can prioritize incidents based on the criticality of the affected services and streamline their resolution processes, improving overall service reliability and customer satisfaction.

Apache Druid

1. Real-Time Application Monitoring Dashboard: Use Telegraf to collect metrics from application servers and send them to Druid for immediate analysis and visualization in dashboards. Druid’s low-latency querying allows users to interactively explore system behavior in near real-time.

2. Security Event Aggregation: Aggregate and forward security-related metrics such as failed logins, port scans, or process anomalies to Druid. Analysts can build dashboards to monitor threat patterns and investigate incidents with millisecond-level granularity.

3. IoT Device Analytics: Collect telemetry from edge devices via Telegraf and send it to Druid for fast, scalable processing. Druid’s time-partitioned storage and roll-up capabilities are ideal for handling billions of small JSON events from sensors or gateways.

4. Web Traffic Behavior Exploration: Use Telegraf to capture web server metrics (e.g., requests per second, latency, error rates) and forward them to Druid. This enables teams to drill down into user behavior by region, device, or request type with subsecond query performance.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.