Kubernetes and Parquet Integration

Input and output integration overview

This plugin captures metrics for Kubernetes pods and containers by communicating with the Kubelet API.

This plugin writes metrics to parquet files, utilizing a schema based on the metrics grouped by name. It supports file rotation and buffered writing for optimal performance.

Integration details

Kubernetes

The Kubernetes input plugin interfaces with the Kubelet API to gather metrics for running pods and containers on a single host, ideally as part of a daemonset in a Kubernetes installation. By operating on each node within the cluster, it collects metrics from the locally running kubelet, ensuring that the data reflects the real-time state of the environment. Being a rapidly evolving project, Kubernetes sees frequent updates, and this plugin adheres to the major cloud providers’ supported versions, maintaining compatibility across multiple releases within a limited time span. Significant consideration is given to the potential high series cardinality, which can burden the database; thus, users are advised to implement filtering techniques and retention policies to manage this load effectively. Configuration options provide flexible customization of the plugin’s behavior to integrate seamlessly into different setups, enhancing its utility in monitoring Kubernetes environments.

Parquet

The Parquet output plugin for Telegraf writes metrics to parquet files, which are columnar storage formats optimized for analytics. By default, this plugin groups metrics by their name, writing them to a single file. If a metric’s schema does not align with existing schemas, those metrics are dropped. The plugin generates an Apache Arrow schema based on all grouped metrics, ensuring that the schema reflects the union of all fields and tags. It operates in a buffered manner, meaning it temporarily holds metrics in memory before writing them to disk for efficiency. Parquet files require proper closure to ensure readability, and this is crucial when using the plugin, as improper closure can lead to unreadable files. Additionally, the plugin supports file rotation after specific time intervals, preventing overwrites of existing files and schema conflicts when a file with the same name already exists.

Configuration

Kubernetes

[[inputs.kubernetes]]
  ## URL for the kubelet, if empty read metrics from all nodes in the cluster
  url = "http://127.0.0.1:10255"

  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## If both of these are empty, we'll use the default serviceaccount:
  ## at: /var/run/secrets/kubernetes.io/serviceaccount/token
  ##
  ## To re-read the token at each interval, please use a file with the
  ## bearer_token option. If given a string, Telegraf will always use that
  ## token.
  # bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  ## OR
  # bearer_token_string = "abc_123"

  ## Kubernetes Node Metric Name
  ## The default Kubernetes node metric name (i.e. kubernetes_node) is the same
  ## for the kubernetes and kube_inventory plugins. To avoid conflicts, set this
  ## option to a different value.
  # node_metric_name = "kubernetes_node"

  ## Pod labels to be added as tags.  An empty array for both include and
  ## exclude will include all labels.
  # label_include = []
  # label_exclude = ["*"]

  ## Set response_timeout (default 5 seconds)
  # response_timeout = "5s"

  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  # tls_cert = /path/to/certfile
  # tls_key = /path/to/keyfile
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Parquet

[[outputs.parquet]]
  ## Directory to write parquet files in. If a file already exists the output
  ## will attempt to continue using the existing file.
  # directory = "."
  
  ## Files are rotated after the time interval specified. When set to 0 no time
  ## based rotation is performed.
  # rotation_interval = "0h"
  
  ## Timestamp field name
  ## Field name to use to store the timestamp. If set to an empty string, then
  ## the timestamp is omitted.
  # timestamp_field_name = "timestamp"

Input and output integration examples

Kubernetes

1. Dynamic Resource Allocation Monitoring: By utilizing the Kubernetes plugin, teams can set up alerts for resource usage patterns across various pods and containers. This proactive monitoring approach enables automatic scaling of resources in response to specific thresholds—helping to optimize performance while minimizing costs during peak usage.

2. Multi-tenancy Resource Isolation Analysis: Organizations using Kubernetes can leverage this plugin to track resource consumption per namespace. In a multi-tenant scenario, understanding the resource allocations and usages across different teams becomes critical for ensuring fair access and performance guarantees, leading to better resource management strategies.

3. Real-time Health Dashboards: Integrate the data captured by the Kubernetes plugin into visualization tools like Grafana to create real-time dashboards. These dashboards provide insights into the overall health and performance of the Kubernetes environment, allowing teams to quickly identify and rectify issues across clusters, pods, and containers.

4. Automated Incident Response Workflows: By combining the Kubernetes plugin with alert management systems, teams can automate incident response procedures based on real-time metrics. If a pod’s resource usage exceeds predefined limits, an automated workflow can trigger remediation actions, such as restarting the pod or reallocating resources—all of which can help improve system resilience.

Parquet

1. Data Lake Ingestion: Utilize the Parquet plugin to store metrics from various sources into a data lake. By writing metrics in parquet format, you establish a standardized and efficient way to manage time-series data, enabling faster querying capabilities and seamless integration with analytics tools like Apache Spark or AWS Athena. This setup can significantly improve data retrieval times and analysis workflows.

2. Long-term Storage of Metrics: Implement the Parquet plugin in a monitoring setup where metrics are collected over time from multiple applications. This allows for long-term storage of performance data in a compact format, making it cost-effective to store vast amounts of historical data while preserving the ability for quick retrieval and analysis later on. By archiving metrics in parquet files, organizations can maintain compliance and create detailed reports from historical performance trends.

3. Analytics and Reporting: After writing metrics to parquet files, leverage tools like Apache Arrow or PyArrow to perform complex analytical queries directly on the files without needing to load all the data into memory. This can enhance reporting capabilities, allowing teams to generate insights and visualization from large datasets efficiently, thereby improving decision-making processes based on accurate, up-to-date performance metrics.

4. Integrating with Data Warehouses: Use the Parquet plugin as part of a data integration pipeline that feeds into a modern data warehouse. By converting metrics to parquet format, the data can be easily ingested by systems like Snowflake or Google BigQuery, enabling powerful analytics and business intelligence capabilities that drive actionable insights from the collected metrics.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.