Google Cloud PubSub and OSI PI Integration

Input and output integration overview

This plugin ingests metrics from Google Cloud PubSub, allowing for real-time data processing and integration into monitoring setups.

This setup converts Telegraf into a lightweight PI Web API publisher, letting you push any Telegraf metric into the OSI PI System with a simple HTTP POST.

Integration details

Google Cloud PubSub

The Google Cloud PubSub input plugin is designed to ingest metrics from Google Cloud PubSub, a messaging service that facilitates real-time communication between different systems. It allows users to create and process metrics by pulling messages from a specified subscription in a Google Cloud Project. One of the critical features of this plugin is its ability to operate as a service input, actively listening for incoming messages rather than merely polling for metrics at set intervals. Through various configuration options, users can customize the behavior of message ingestion, such as handling credentials, managing message sizes, and tuning the acknowledgment settings to ensure that messages are only acknowledged after successful processing. By leveraging the strengths of Google PubSub, this plugin integrates seamlessly with cloud-native architectures, enabling users to build robust and scalable applications that can react to events in real-time.

OSI PI

OSI PI is an data management and analytics platform used in energy, manufacturing, and critical infrastructure. The PI Web API is its REST interface, exposing endpoints such as /piwebapi/streams/{WebId}/value that accept JSON payloads containing a Timestamp and Value. By pairing Telegraf’s flexible HTTP output with this endpoint, any metric Telegraf collects—SNMP counters, Modbus readings, Kubernetes stats—can be written directly into PI without installing proprietary interfaces. The configuration above authenticates with Basic or Kerberos, serializes each batch to JSON, and renders a minimal body template that aligns with PI Web API’s single-value write contract. Because Telegraf already supports batching, TLS, proxies, and custom headers, this approach scales from edge gateways to cloud VMs, allowing organizations to back-fill historical data, stream live telemetry, or mirror non-PI sources (e.g., Prometheus) into the PI data archive. It also sidesteps older SDK dependencies and enables hybrid architectures where PI remains on-prem while Telegraf agents run in containers or IIoT devices.

Configuration

Google Cloud PubSub

[[inputs.cloud_pubsub]]
  project = "my-project"
  subscription = "my-subscription"
  data_format = "influx"
  # credentials_file = "path/to/my/creds.json"
  # retry_delay_seconds = 5
  # max_message_len = 1000000
  # max_undelivered_messages = 1000
  # max_extension = 0
  # max_outstanding_messages = 0
  # max_outstanding_bytes = 0
  # max_receiver_go_routines = 0
  # base64_data = false
  # content_encoding = "identity"
  # max_decompression_size = "500MB"

OSI PI

[[outputs.http]]
  ## PI Web API endpoint for writing a single value to a PI Point by Web ID
  url = "https://${PI_HOST}/piwebapi/streams/${WEB_ID}/value"

  ## Use POST for each batch
  method       = "POST"
  content_type = "application/json"

  ## Basic-auth header (base64-encoded "DOMAIN\\user:password")
  headers = { Authorization = "Basic ${BASIC_AUTH}" }

  ## Serialize Telegraf metrics as JSON
  data_format           = "json"
  json_timestamp_units  = "1ms"

  ## Render the JSON body that PI Web API expects
  body_template = """
  {{ range .Metrics -}}
  { "Timestamp": "{{ .timestamp | formatDate \"2006-01-02T15:04:05Z07:00\" }}", "Value": {{ index .fields 0 }} }
  {{ end -}}
  """

  ## Tune networking / batching if needed
  # timeout     = "10s"
  # batch_size  = 1

Input and output integration examples

Google Cloud PubSub

1. Real-Time Analytics for IoT Devices: Utilize the Google Cloud PubSub plugin to aggregate metrics from IoT devices scattered across various locations. By streaming data from devices to Google PubSub and using this plugin to ingest metrics, organizations can create a centralized dashboard for real-time monitoring and alerting. This setup allows for immediate insights into device performance, facilitating proactive maintenance and operational efficiency.

2. Dynamic Log Processing and Monitoring: Ingest logs from numerous sources via Google Cloud PubSub into a Telegraf pipeline, utilizing the plugin to parse and analyze log messages. This can help teams quickly identify anomalies or patterns in logs and streamline the process of troubleshooting issues across distributed systems. By consolidating log data, organizations can enhance their observability and response capabilities.

3. Event-Driven Workflow Integrations: Use the Google Cloud PubSub plugin to connect various cloud functions or services. Each time a new message is pushed to a subscription, actions can be triggered in other parts of the cloud architecture, such as starting data processing jobs, notifications, or even updates to reports. This event-driven approach allows for a more reactive system architecture that can adapt to changing business needs.

OSI PI

1. Remote Pump Stations Telemetry Bridge: Install Telegraf on edge gateways at oil-field pump stations, gather flow-meter and vibration readings over Modbus, and POST them to the PI Web API. Operations teams view real-time data in PI Vision without deploying heavyweight PI interfaces, while bandwidth-friendly batching keeps satellite links economical.

2. Green-Energy Micro-Grid Dashboard: Export inverter, battery, and weather metrics from MQTT into Telegraf, which relays them to PI. PI AF analytics can calculate real-time power balance and feed a campus dashboard; historical deltas inform sustainability reports.

3. Brownfield SCADA Modernization: Legacy PLCs logged to CSV are ingested by Telegraf’s tail input; each row is parsed and immediately sent to PI via HTTP, creating a live data stream that co-exists with archival files while the SCADA upgrade proceeds incrementally.

4. Synthetic Data Generator for Training: Telegraf’s exec input can run a script that emits simulated sensor patterns. Posting those metrics to a non-production PI server through the Web API supplies realistic datasets for PI Vision training sessions without risking production tags.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.