Intel PowerStat and Databricks Integration

Input and output integration overview

Monitor power statistics on Intel-based platforms and is compatible with Linux-based operating systems. It helps in understanding and managing power efficiency and CPU performance.

Use Telegraf’s HTTP output plugin to push metrics straight into a Databricks Lakehouse by calling the SQL Statement Execution API with a JSON-wrapped INSERT or volume PUT command.

Integration details

Intel PowerStat

The Intel PowerStat plugin is designed to monitor power statistics specifically on Intel-based platforms running a Linux operating system. It offers visibility into critical metrics such as CPU temperature, utilization, and power consumption, making it essential for power saving initiatives and workload migration strategies. By leveraging telemetry frameworks, this plugin enables users to gain insights into platform-level metrics that help with monitoring and analytics systems in the context of Management and Orchestration (MANO). It facilitates the ability to make informed decisions and perform corrective actions based on the state of the platform, ultimately contributing to better system efficiency and reliability.

Databricks

This configuration turns Telegraf into a lightweight ingestion agent for the Databricks Lakehouse. It leverages the Databricks SQL Statement Execution API 2.0, which accepts authenticated POST requests containing a JSON payload with a statement field. Each Telegraf flush dynamically renders a SQL INSERT (or, for file-based workflows, a PUT ... INTO /Volumes/... command) that lands the metrics into a Unity Catalog table or volume governed by Lakehouse security. Under the hood Databricks stores successful inserts as Delta Lake transactions, enabling ACID guarantees, time-travel, and scalable analytics. Operators can point the warehouse_id at any serverless or classic SQL warehouse, and all authentication is handled with a PAT or service-principal token—no agents or JDBC drivers required. Because Telegraf’s HTTP output supports custom headers, batching, TLS, and proxy settings, the same pattern scales from edge IoT gateways to container sidecars, consolidating infrastructure telemetry, application logs, or business KPIs directly into the Lakehouse for BI, ML, and Lakehouse Monitoring. Unity Catalog volumes provide a governed staging layer when file uploads and COPY INTO are preferred, and the approach aligns with Databricks’ recommended ingestion practices for partners and ISVs.

Configuration

Intel PowerStat

[[inputs.intel_powerstat]]
  # package_metrics = ["current_power_consumption", "current_dram_power_consumption", "thermal_design_power"]
  # cpu_metrics = []
  # included_cpus = []
  # excluded_cpus = []
  # event_definitions = ""
  # msr_read_timeout = "0ms"

Databricks

[[outputs.http]]
  ## Databricks SQL Statement Execution API endpoint
  url = "https://{{ env "DATABRICKS_HOST" }}/api/2.0/sql/statements"

  ## Use POST to submit each Telegraf batch as a SQL request
  method = "POST"

  ## Personal-access token (PAT) for workspace or service principal
  headers = { Authorization = "Bearer {{ env "DATABRICKS_TOKEN" }}" }

  ## Send JSON that wraps the metrics batch in a SQL INSERT (or PUT into a Volume)
  content_type = "application/json"

  ## Serialize metrics as JSON so they can be embedded in the SQL statement
  data_format = "json"
  json_timestamp_units = "1ms"

  ## Build the request body.  Telegraf replaces the template variables at runtime.
  ## Example inserts a row per metric into a Unity-Catalog table.
  body_template = """
  {
    \"statement\": \"INSERT INTO ${TARGET_TABLE} VALUES {{range .Metrics}}(from_unixtime({{.timestamp}}/1000), {{.fields.usage}}, '{{.tags.host}}'){{end}}\",
    \"warehouse_id\": \"${WAREHOUSE_ID}\"
  }
  """

  ## Optional: add batching limits or TLS settings
  # batch_size = 500
  # timeout     = "10s"

Input and output integration examples

Intel PowerStat

1. Optimizing Data Center Energy Usage: Monitor power consumption metrics across all CPUs in a data center. By capturing real-time data, administrators can identify which servers consume the most power and implement shutdowns or load balancing strategies during low demand periods, effectively reducing operational costs.

2. Dynamic Workload Migration Based on Power Efficiency: Integrate this plugin with a cloud orchestration tool to enable dynamic migration of workloads based on power usage metrics. If a particular server is recorded as consuming excessive power without corresponding output, the orchestrator can seamlessly migrate workloads to more efficient nodes, ensuring optimal resource utilization and lower energy expenses.

3. Monitoring and Alerting Mechanism for Overheating CPUs: Implement an alerting system using the CPU temperature metrics captured by Intel PowerStat. Setting thresholds for temperature can alert system administrators when a CPU is prone to overheating, allowing proactive measures to be taken before hardware damage occurs, ultimately extending the life of the components.

4. Performance Benchmarking for CPU-intensive Applications: Use the metrics provided to benchmark the performance of CPU-intensive applications. By analyzing the cpu_frequency, cpu_temperature, and power metrics under load, developers can optimize application performance and make informed decisions regarding scaling and resource allocation.

Databricks

1. Edge-to-Lakehouse Telemetry Pipe: Deploy Telegraf on factory PLCs to sample vibration metrics and post them every second to a serverless SQL warehouse. Delta tables power PowerBI dashboards that alert engineers when thresholds drift.
2. Blue-Green CI/CD Rollout Metrics: Attach a Telegraf sidecar to each Kubernetes canary pod; it inserts container stats into a Unity Catalog table tagged by deployment_id, letting Databricks SQL compare error-rate percentiles and auto-rollback underperforming versions.
3. SaaS Usage Metering: Insert per-tenant API-call counters via the HTTP plugin; a nightly Lakehouse query aggregates usage into invoices, eliminating custom metering micro-services.
4. Security Forensics Lake: Upload JSON batches of Suricata IDS events to a Unity Catalog volume using PUT commands, then run COPY INTO for near-real-time enrichment with Delta Live Tables, producing a searchable threat-intel lake that joins network logs with user session data.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.