Jenkins and Apache Hudi Integration

Input and output integration overview

The Jenkins plugin collects vital information regarding jobs and nodes from a Jenkins instance through its API, facilitating comprehensive monitoring and analysis.

Writes metrics to Parquet files via Telegraf’s Parquet output plugin, preparing them for ingestion into Apache Hudi’s lakehouse architecture.

Integration details

Jenkins

The Jenkins Telegraf plugin allows users to gather metrics from a Jenkins instance without needing to install any additional plugins on Jenkins itself. By utilizing the Jenkins API, the plugin retrieves information about nodes and jobs running in the Jenkins environment. This integration provides a comprehensive overview of the Jenkins infrastructure, including real-time metrics that can be used for monitoring and analysis. Key features include configurable filters for job and node selection, optional TLS security settings, and the ability to manage request timeouts and connection limits effectively. This makes it an essential tool for teams that rely on Jenkins for continuous integration and delivery, ensuring they have the insights they need to maintain optimal performance and reliability.

Apache Hudi

This configuration leverages Telegraf’s Parquet plugin to serialize metrics into columnar Parquet files suitable for downstream ingestion by Apache Hudi. The plugin writes metrics grouped by metric name into files in a specified directory, buffering writes for efficiency and optionally rotating files on timers. It considers schema compatibility—metrics with incompatible schemas are dropped—ensuring consistency. Apache Hudi can then consume these Parquet files via tools like DeltaStreamer or Spark jobs, enabling transactional ingestion, time-travel queries, and upserts on your time series data.

Configuration

Jenkins

[[inputs.jenkins]]
  ## The Jenkins URL in the format "schema://host:port"
  url = "http://my-jenkins-instance:8080"
  # username = "admin"
  # password = "admin"

  ## Set response_timeout
  response_timeout = "5s"

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use SSL but skip chain & host verification
  # insecure_skip_verify = false

  ## Optional Max Job Build Age filter
  ## Default 1 hour, ignore builds older than max_build_age
  # max_build_age = "1h"

  ## Optional Sub Job Depth filter
  ## Jenkins can have unlimited layer of sub jobs
  ## This config will limit the layers of pulling, default value 0 means
  ## unlimited pulling until no more sub jobs
  # max_subjob_depth = 0

  ## Optional Sub Job Per Layer
  ## In workflow-multibranch-plugin, each branch will be created as a sub job.
  ## This config will limit to call only the lasted branches in each layer,
  ## empty will use default value 10
  # max_subjob_per_layer = 10

  ## Jobs to include or exclude from gathering
  ## When using both lists, job_exclude has priority.
  ## Wildcards are supported: [ "jobA/*", "jobB/subjob1/*"]
  # job_include = [ "*" ]
  # job_exclude = [ ]

  ## Nodes to include or exclude from gathering
  ## When using both lists, node_exclude has priority.
  # node_include = [ "*" ]
  # node_exclude = [ ]

  ## Worker pool for jenkins plugin only
  ## Empty this field will use default value 5
  # max_connections = 5

  ## When set to true will add node labels as a comma-separated tag. If none,
  ## are found, then a tag with the value of 'none' is used. Finally, if a
  ## label contains a comma it is replaced with an underscore.
  # node_labels_as_tag = false

Apache Hudi

[[outputs.parquet]]
  ## Directory to write parquet files in. If a file already exists the output
  ## will attempt to continue using the existing file.
  directory = "/var/lib/telegraf/hudi_metrics"

  ## File rotation interval (default is no rotation)
  # rotation_interval = "1h"

  ## Buffer size before writing (default is 1000 metrics)
  # buffer_size = 1000

  ## Optional: compression codec (snappy, gzip, etc.)
  # compression_codec = "snappy"

  ## When grouping metrics, each metric name goes to its own file
  ## If a metric’s schema doesn’t match the existing schema, it will be dropped

Input and output integration examples

Jenkins

1. Continuous Integration Monitoring: Use the Jenkins plugin to monitor the performance of continuous integration pipelines by collecting metrics on job durations and failure rates. This can help teams identify bottlenecks in the pipeline and improve overall build efficiency.

2. Resource Allocation Analysis: Leverage Jenkins node metrics to assess resource usage across different agents. By understanding how resources are allocated, teams can optimize their Jenkins architecture, potentially reallocating agents or adjusting job configurations for better performance.

3. Job Execution Trends: Analyze historical job performance metrics to identify trends in job execution over time. With this data, teams can proactively address potential issues before they grow, making adjustments to the jobs or their configurations as needed.

4. Alerting for Job Failures: Implement alerts that leverage the Jenkins job metrics to notify team members in case of job failures. This proactive approach can enhance operational awareness and speed up response times to failures, ensuring that critical jobs are monitored effectively.

Apache Hudi

1. Transactional Lakehouse Metrics: Buffer and write Web service metrics as Parquet files for DeltaStreamer to ingest into Hudi, enabling upserts, ACID compliance, and time-travel on historical performance data.

2. Edge Device Batch Analytics: Telegraf running on IoT gateways writes metrics to Parquet locally, where periodic Spark jobs ingest them into Hudi for long-term analytics and traceability.

3. Schema-Enforced Abnormal Metric Handling: Use Parquet plugin’s strict schema-dropping behavior to prevent malformed or unexpected metric changes. Hudi ingestion then guarantees consistent schema and data quality in downstream datasets.

4. Data Platform Integration: Store Telegraf metrics as Parquet files in an S3/ADLS landing zone. Hudi’s Spark-based ingestion pipeline then loads them into a unified, queryable lakehouse with business events and logs.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.