Kibana and Apache Hudi Integration

Input and output integration overview

The Kibana plugin enables users to obtain status metrics from Kibana, a data visualization tool for Elasticsearch. By connecting to the Kibana API, this plugin captures various performance indicators and the health status of the Kibana service.

Writes metrics to Parquet files via Telegraf’s Parquet output plugin, preparing them for ingestion into Apache Hudi’s lakehouse architecture.

Integration details

Kibana

The Kibana input plugin is designed to query the Kibana API to gather service status information. This plugin allows users to monitor their Kibana instances effectively by pulling metrics related to its health, performance, and operational metrics. By querying the Kibana API, this plugin provides insights into key parameters such as the current health status (green, yellow, red), uptime, heap memory usage, and request performance metrics. This information is crucial for administrators and operational teams looking to maintain optimal system performance and quickly address any issues that may arise. The configuration settings allow for flexible integration with other components in a microservices architecture, facilitating comprehensive monitoring solutions aligned with organizational needs, making it an essential tool for those leveraging the Elastic Stack in their infrastructure.

Apache Hudi

This configuration leverages Telegraf’s Parquet plugin to serialize metrics into columnar Parquet files suitable for downstream ingestion by Apache Hudi. The plugin writes metrics grouped by metric name into files in a specified directory, buffering writes for efficiency and optionally rotating files on timers. It considers schema compatibility—metrics with incompatible schemas are dropped—ensuring consistency. Apache Hudi can then consume these Parquet files via tools like DeltaStreamer or Spark jobs, enabling transactional ingestion, time-travel queries, and upserts on your time series data.

Configuration

Kibana

[[inputs.kibana]]
  ## Specify a list of one or more Kibana servers
  servers = ["http://localhost:5601"]

  ## Timeout for HTTP requests
  timeout = "5s"

  ## HTTP Basic Auth credentials
  # username = "username"
  # password = "pa$$word"

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false
 
  ## If 'use_system_proxy' is set to true, Telegraf will check env vars such as
  ## HTTP_PROXY, HTTPS_PROXY, and NO_PROXY (or their lowercase counterparts).
  ## If 'use_system_proxy' is set to false (default) and 'http_proxy_url' is
  ## provided, Telegraf will use the specified URL as HTTP proxy.
  # use_system_proxy = false
  # http_proxy_url = "http://localhost:8888"

Apache Hudi

[[outputs.parquet]]
  ## Directory to write parquet files in. If a file already exists the output
  ## will attempt to continue using the existing file.
  directory = "/var/lib/telegraf/hudi_metrics"

  ## File rotation interval (default is no rotation)
  # rotation_interval = "1h"

  ## Buffer size before writing (default is 1000 metrics)
  # buffer_size = 1000

  ## Optional: compression codec (snappy, gzip, etc.)
  # compression_codec = "snappy"

  ## When grouping metrics, each metric name goes to its own file
  ## If a metric’s schema doesn’t match the existing schema, it will be dropped

Input and output integration examples

Kibana

1. Kibana Health Monitoring: Implement a dedicated dashboard to periodically poll the metrics from Kibana. This setup allows operations teams to have a real-time view of their Kibana instances’ health and metrics, enabling proactive performance management and immediate response capabilities in case of service degradation or failure.

2. Automated Alerting System: Integrate the metrics gathered from the Kibana plugin with an alerting system using tools like Prometheus or PagerDuty. By setting thresholds for key metrics (e.g., response time or heap usage), this integration can automatically notify the relevant personnel of performance issues, thereby reducing downtime and improving the response time for operational issues.

3. Resource Optimization Strategy: Use the memory usage and response time metrics collected by this plugin to formulate strategies for optimizing resource allocation in Kubernetes or other orchestration platforms. By analyzing trends over time, teams can adjust resource limits and requests dynamically, ensuring that Kibana instances function efficiently without over-provisioning resources.

Apache Hudi

1. Transactional Lakehouse Metrics: Buffer and write Web service metrics as Parquet files for DeltaStreamer to ingest into Hudi, enabling upserts, ACID compliance, and time-travel on historical performance data.

2. Edge Device Batch Analytics: Telegraf running on IoT gateways writes metrics to Parquet locally, where periodic Spark jobs ingest them into Hudi for long-term analytics and traceability.

3. Schema-Enforced Abnormal Metric Handling: Use Parquet plugin’s strict schema-dropping behavior to prevent malformed or unexpected metric changes. Hudi ingestion then guarantees consistent schema and data quality in downstream datasets.

4. Data Platform Integration: Store Telegraf metrics as Parquet files in an S3/ADLS landing zone. Hudi’s Spark-based ingestion pipeline then loads them into a unified, queryable lakehouse with business events and logs.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.