NATS and Apache Hudi Integration

Input and output integration overview

The NATS Consumer Input Plugin enables real-time data consumption from NATS messaging subjects, integrating seamlessly into the Telegraf data pipeline for monitoring and metrics gathering.

Writes metrics to Parquet files via Telegraf’s Parquet output plugin, preparing them for ingestion into Apache Hudi’s lakehouse architecture.

Integration details

NATS

The NATS Consumer Plugin allows Telegraf to read metrics from specified NATS subjects and create metrics based on supported input data formats. Utilizing a Queue Group allows multiple instances of Telegraf to read from a NATS cluster in parallel, enhancing throughput and reliability. This plugin also supports various authentication methods, including username/password, NATS credentials files, and nkey seed files, ensuring secure communication with the NATS servers. It is particularly useful in environments where data persistence and message reliability are critical, thanks to features such as JetStream that facilitate the consumption of historical messages. Additionally, the ability to configure various operational parameters makes this plugin suitable for high-throughput scenarios while maintaining performance integrity.

Apache Hudi

This configuration leverages Telegraf’s Parquet plugin to serialize metrics into columnar Parquet files suitable for downstream ingestion by Apache Hudi. The plugin writes metrics grouped by metric name into files in a specified directory, buffering writes for efficiency and optionally rotating files on timers. It considers schema compatibility—metrics with incompatible schemas are dropped—ensuring consistency. Apache Hudi can then consume these Parquet files via tools like DeltaStreamer or Spark jobs, enabling transactional ingestion, time-travel queries, and upserts on your time series data.

Configuration

NATS

[[inputs.nats_consumer]]
  ## urls of NATS servers
  servers = ["nats://localhost:4222"]

  ## subject(s) to consume
  ## If you use jetstream you need to set the subjects
  ## in jetstream_subjects
  subjects = ["telegraf"]

  ## jetstream subjects
  ## jetstream is a streaming technology inside of nats.
  ## With jetstream the nats-server persists messages and
  ## a consumer can consume historical messages. This is
  ## useful when telegraf needs to restart it don't miss a
  ## message. You need to configure the nats-server.
  ## https://docs.nats.io/nats-concepts/jetstream.
  jetstream_subjects = ["js_telegraf"]

  ## name a queue group
  queue_group = "telegraf_consumers"

  ## Optional authentication with username and password credentials
  # username = ""
  # password = ""

  ## Optional authentication with NATS credentials file (NATS 2.0)
  # credentials = "/etc/telegraf/nats.creds"

  ## Optional authentication with nkey seed file (NATS 2.0)
  # nkey_seed = "/etc/telegraf/seed.txt"

  ## Use Transport Layer Security
  # secure = false

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

  ## Sets the limits for pending msgs and bytes for each subscription
  ## These shouldn't need to be adjusted except in very high throughput scenarios
  # pending_message_limit = 65536
  # pending_bytes_limit = 67108864

  ## Max undelivered messages
  ## This plugin uses tracking metrics, which ensure messages are read to
  ## outputs before acknowledging them to the original broker to ensure data
  ## is not lost. This option sets the maximum messages to read from the
  ## broker that have not been written by an output.
  ##
  ## This value needs to be picked with awareness of the agent's
  ## metric_batch_size value as well. Setting max undelivered messages too high
  ## can result in a constant stream of data batches to the output. While
  ## setting it too low may never flush the broker's messages.
  # max_undelivered_messages = 1000

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "influx"

Apache Hudi

[[outputs.parquet]]
  ## Directory to write parquet files in. If a file already exists the output
  ## will attempt to continue using the existing file.
  directory = "/var/lib/telegraf/hudi_metrics"

  ## File rotation interval (default is no rotation)
  # rotation_interval = "1h"

  ## Buffer size before writing (default is 1000 metrics)
  # buffer_size = 1000

  ## Optional: compression codec (snappy, gzip, etc.)
  # compression_codec = "snappy"

  ## When grouping metrics, each metric name goes to its own file
  ## If a metric’s schema doesn’t match the existing schema, it will be dropped

Input and output integration examples

NATS

1. Real-Time Analytics Dashboard: Utilize the NATS plugin to gather metrics from various NATS subjects in real time and feed them into a centralized analytics dashboard. This setup allows for immediate visibility into live application performance, enabling teams to react swiftly to operational issues or performance degradation.

2. Distributed System Monitoring: Deploy multiple instances of Telegraf configured with the NATS plugin across a distributed architecture. This approach allows teams to aggregate metrics from various microservices efficiently, providing a holistic view of system health and performance while ensuring no messages are dropped during transmission.

3. Historical Message Recovery: Leverage the capabilities of NATS JetStream along with this plugin to recover and process historical messages after Telegraf has been restarted. This feature is particularly beneficial for applications that require high reliability, ensuring that no critical metrics are lost even in case of service disruptions.

4. Dynamic Load Balancing: Implement a dynamic load balancing scenario where Telegraf instances consume messages from a NATS cluster based on load. Adjust the queue group settings to control the number of active consumers, allowing for better resource utilization and performance scaling as demand fluctuations occur.

Apache Hudi

1. Transactional Lakehouse Metrics: Buffer and write Web service metrics as Parquet files for DeltaStreamer to ingest into Hudi, enabling upserts, ACID compliance, and time-travel on historical performance data.

2. Edge Device Batch Analytics: Telegraf running on IoT gateways writes metrics to Parquet locally, where periodic Spark jobs ingest them into Hudi for long-term analytics and traceability.

3. Schema-Enforced Abnormal Metric Handling: Use Parquet plugin’s strict schema-dropping behavior to prevent malformed or unexpected metric changes. Hudi ingestion then guarantees consistent schema and data quality in downstream datasets.

4. Data Platform Integration: Store Telegraf metrics as Parquet files in an S3/ADLS landing zone. Hudi’s Spark-based ingestion pipeline then loads them into a unified, queryable lakehouse with business events and logs.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.