Kubernetes and DuckDB Integration

Input and output integration overview

This plugin captures metrics for Kubernetes pods and containers by communicating with the Kubelet API.

This plugin enables Telegraf to write structured metrics into DuckDB using SQLite-compatible SQL connections, supporting lightweight local analytics and offline metric analysis.

Integration details

Kubernetes

The Kubernetes input plugin interfaces with the Kubelet API to gather metrics for running pods and containers on a single host, ideally as part of a daemonset in a Kubernetes installation. By operating on each node within the cluster, it collects metrics from the locally running kubelet, ensuring that the data reflects the real-time state of the environment. Being a rapidly evolving project, Kubernetes sees frequent updates, and this plugin adheres to the major cloud providers’ supported versions, maintaining compatibility across multiple releases within a limited time span. Significant consideration is given to the potential high series cardinality, which can burden the database; thus, users are advised to implement filtering techniques and retention policies to manage this load effectively. Configuration options provide flexible customization of the plugin’s behavior to integrate seamlessly into different setups, enhancing its utility in monitoring Kubernetes environments.

DuckDB

Use the Telegraf SQL plugin to write metrics into a local DuckDB database. DuckDB is an in-process OLAP database designed for efficient analytical queries on columnar data. Although it does not provide a traditional client-server interface, DuckDB can be accessed via SQLite-compatible drivers in embedded mode. This allows Telegraf to store time series metrics in DuckDB using SQL, enabling powerful analytics workflows using familiar SQL syntax, Jupyter notebooks, or integration with data science tools like Python and R. DuckDB’s columnar storage and vectorized execution make it ideal for compact and high-performance metric archives.

Configuration

Kubernetes

[[inputs.kubernetes]]
  ## URL for the kubelet, if empty read metrics from all nodes in the cluster
  url = "http://127.0.0.1:10255"

  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## If both of these are empty, we'll use the default serviceaccount:
  ## at: /var/run/secrets/kubernetes.io/serviceaccount/token
  ##
  ## To re-read the token at each interval, please use a file with the
  ## bearer_token option. If given a string, Telegraf will always use that
  ## token.
  # bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  ## OR
  # bearer_token_string = "abc_123"

  ## Kubernetes Node Metric Name
  ## The default Kubernetes node metric name (i.e. kubernetes_node) is the same
  ## for the kubernetes and kube_inventory plugins. To avoid conflicts, set this
  ## option to a different value.
  # node_metric_name = "kubernetes_node"

  ## Pod labels to be added as tags.  An empty array for both include and
  ## exclude will include all labels.
  # label_include = []
  # label_exclude = ["*"]

  ## Set response_timeout (default 5 seconds)
  # response_timeout = "5s"

  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  # tls_cert = /path/to/certfile
  # tls_key = /path/to/keyfile
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

DuckDB

[[outputs.sql]]
  ## Use the SQLite driver to connect to DuckDB via Go's database/sql
  driver = "sqlite3"

  ## DSN should point to the DuckDB database file
  dsn = "file:/var/lib/telegraf/metrics.duckdb"

  ## SQL INSERT statement with placeholders for metrics
  table_template = "INSERT INTO metrics (timestamp, name, value, tags) VALUES (?, ?, ?, ?)"

  ## Optional: manage connection pooling
  # max_open_connections = 1
  # max_idle_connections = 1
  # conn_max_lifetime = "0s"

  ## DuckDB does not require TLS or authentication by default

Input and output integration examples

Kubernetes

1. Dynamic Resource Allocation Monitoring: By utilizing the Kubernetes plugin, teams can set up alerts for resource usage patterns across various pods and containers. This proactive monitoring approach enables automatic scaling of resources in response to specific thresholds—helping to optimize performance while minimizing costs during peak usage.

2. Multi-tenancy Resource Isolation Analysis: Organizations using Kubernetes can leverage this plugin to track resource consumption per namespace. In a multi-tenant scenario, understanding the resource allocations and usages across different teams becomes critical for ensuring fair access and performance guarantees, leading to better resource management strategies.

3. Real-time Health Dashboards: Integrate the data captured by the Kubernetes plugin into visualization tools like Grafana to create real-time dashboards. These dashboards provide insights into the overall health and performance of the Kubernetes environment, allowing teams to quickly identify and rectify issues across clusters, pods, and containers.

4. Automated Incident Response Workflows: By combining the Kubernetes plugin with alert management systems, teams can automate incident response procedures based on real-time metrics. If a pod’s resource usage exceeds predefined limits, an automated workflow can trigger remediation actions, such as restarting the pod or reallocating resources—all of which can help improve system resilience.

DuckDB

1. Embedded Metric Warehousing for Notebooks: Write metrics to a local DuckDB file from Telegraf and analyze them in Jupyter notebooks using Python or R. This workflow supports reproducible analytics, ideal for data science experiments or offline troubleshooting.

2. Batch Time-Series Processing on the Edge: Use Telegraf with DuckDB on edge devices to log metrics locally in SQL format. The compact storage and fast analytical capabilities of DuckDB make it ideal for batch processing and low-bandwidth environments.

3. Exploratory Querying of Historical Metrics: Accumulate system metrics over time in DuckDB and perform exploratory data analysis (EDA) using SQL joins, window functions, and aggregates. This enables insights that go beyond what typical time-series dashboards provide.

4. Self-Contained Metric Snapshots: Use DuckDB as a portable metrics archive by shipping .duckdb files between systems. Telegraf can collect and store data in this format, and analysts can later load and query it using the DuckDB CLI or integrations with tools like Tableau and Apache Arrow.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.