Google Cloud PubSub and SigNoz Integration

Input and output integration overview

This plugin ingests metrics from Google Cloud PubSub, allowing for real-time data processing and integration into monitoring setups.

This configuration turns any Telegraf agent into a Remote Write publisher for SigNoz, streaming rich metrics straight into the SigNoz backend with a single URL change.

Integration details

Google Cloud PubSub

The Google Cloud PubSub input plugin is designed to ingest metrics from Google Cloud PubSub, a messaging service that facilitates real-time communication between different systems. It allows users to create and process metrics by pulling messages from a specified subscription in a Google Cloud Project. One of the critical features of this plugin is its ability to operate as a service input, actively listening for incoming messages rather than merely polling for metrics at set intervals. Through various configuration options, users can customize the behavior of message ingestion, such as handling credentials, managing message sizes, and tuning the acknowledgment settings to ensure that messages are only acknowledged after successful processing. By leveraging the strengths of Google PubSub, this plugin integrates seamlessly with cloud-native architectures, enabling users to build robust and scalable applications that can react to events in real-time.

SigNoz

SigNoz is an open source observability platform that stores metrics, traces, and logs. When you deploy SigNoz, its signoz-otel-collector-metrics service exposes a Prometheus Remote Write receiver (default :13133/api/v1/write). By configuring Telegraf’s Prometheus plugin to point at this endpoint, you can push any Telegraf collected metrics, SNMP counters, cloud services, or business KPIs—directly into SigNoz. The plugin natively serializes metrics in the Remote Write protobuf format, supports external labels, metadata export, retries, and TLS or bearer-token auth, so it fits zero-trust and multi-tenant SigNoz clusters. Inside SigNoz, the data lands in ClickHouse tables that back Metrics Explorer, alert rules, and unified dashboards. This approach lets organizations unify Prometheus and OTLP pipelines, enables long-term retention powered by ClickHouse compression, and avoids vendor lock-in while retaining PromQL-style queries.

Configuration

Google Cloud PubSub

[[inputs.cloud_pubsub]]
  project = "my-project"
  subscription = "my-subscription"
  data_format = "influx"
  # credentials_file = "path/to/my/creds.json"
  # retry_delay_seconds = 5
  # max_message_len = 1000000
  # max_undelivered_messages = 1000
  # max_extension = 0
  # max_outstanding_messages = 0
  # max_outstanding_bytes = 0
  # max_receiver_go_routines = 0
  # base64_data = false
  # content_encoding = "identity"
  # max_decompression_size = "500MB"

SigNoz

[[outputs.prometheusremotewrite]]
  ## SigNoz OTEL-Collector metrics endpoint (Prometheus Remote Write receiver)
  ## Default port is 13133 when you install SigNoz with the Helm chart
  url = "http://signoz-otel-collector-metrics.monitoring.svc.cluster.local:13133/api/v1/write"

  ## Add identifying labels so you can slice & dice the data later
  external_labels = { host = "${HOSTNAME}", agent = "telegraf" }

  ## Forward host metadata for richer dashboards (SigNoz maps these to ClickHouse columns)
  send_metadata = true

  ## ----- Authentication (comment out what you don’t need) -----
  # bearer_token   = "$SIGNOZ_TOKEN"          # SaaS tenant token
  # basic_username = "signoz"                 # Basic auth (self-hosted)
  # basic_password = "secret"

  ## ----- TLS options (for SaaS or HTTPS self-hosted) -----
  # tls_ca                  = "/etc/ssl/certs/ca.crt"
  # tls_cert                = "/etc/telegraf/certs/telegraf.crt"
  # tls_key                 = "/etc/telegraf/certs/telegraf.key"
  # insecure_skip_verify    = false

  ## ----- Performance tuning -----
  max_batch_size = 10000      # samples per POST
  timeout        = "10s"
  retry_max      = 3

Input and output integration examples

Google Cloud PubSub

1. Real-Time Analytics for IoT Devices: Utilize the Google Cloud PubSub plugin to aggregate metrics from IoT devices scattered across various locations. By streaming data from devices to Google PubSub and using this plugin to ingest metrics, organizations can create a centralized dashboard for real-time monitoring and alerting. This setup allows for immediate insights into device performance, facilitating proactive maintenance and operational efficiency.

2. Dynamic Log Processing and Monitoring: Ingest logs from numerous sources via Google Cloud PubSub into a Telegraf pipeline, utilizing the plugin to parse and analyze log messages. This can help teams quickly identify anomalies or patterns in logs and streamline the process of troubleshooting issues across distributed systems. By consolidating log data, organizations can enhance their observability and response capabilities.

3. Event-Driven Workflow Integrations: Use the Google Cloud PubSub plugin to connect various cloud functions or services. Each time a new message is pushed to a subscription, actions can be triggered in other parts of the cloud architecture, such as starting data processing jobs, notifications, or even updates to reports. This event-driven approach allows for a more reactive system architecture that can adapt to changing business needs.

SigNoz

1. Multi-Cluster Federated Monitoring: Drop a Telegraf DaemonSet into each Kubernetes cluster, tag metrics with cluster=<name>, and Remote Write them to a central SigNoz instance. Ops teams get a single PromQL window across prod, staging, and edge clusters without running Thanos sidecars.

2. Factory-Floor Edge Gateway: A rugged Intel NUC on the shop floor runs Telegraf to scrape Modbus PLCs and environmental sensors. It batches readings every 5 seconds and pushes them over an intermittent 4G link to SigNoz SaaS. ClickHouse compression keeps costs low while AI-based outlier detection in SigNoz flags overheating motors before failure.

3. SaaS Usage Metering: Telegraf runs alongside each micro-service, exporting per-tenant counters (api_calls, gigabytes_processed). Remote Write streams the data to SigNoz where a scheduled ClickHouse materialized view aggregates usage for monthly billing—no separate metering stack required.

4. Autoscaling Feedback Loop: Combine Telegraf’s Kubernetes input with the Remote Write output to publish granular pod CPU and queue-length metrics into SigNoz. A custom SigNoz alert fires when P95 latency breaches 200 ms and a GitOps controller reads that alert to trigger a HorizontalPodAutoscaler tweak—closing the loop between observability and automation.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.