Google Cloud PubSub and GroundWork Integration

Input and output integration overview

This plugin ingests metrics from Google Cloud PubSub, allowing for real-time data processing and integration into monitoring setups.

This plugin writes to a GroundWork Monitor instance, allowing for effective metrics management and monitoring in a centralized manner.

Integration details

Google Cloud PubSub

The Google Cloud PubSub input plugin is designed to ingest metrics from Google Cloud PubSub, a messaging service that facilitates real-time communication between different systems. It allows users to create and process metrics by pulling messages from a specified subscription in a Google Cloud Project. One of the critical features of this plugin is its ability to operate as a service input, actively listening for incoming messages rather than merely polling for metrics at set intervals. Through various configuration options, users can customize the behavior of message ingestion, such as handling credentials, managing message sizes, and tuning the acknowledgment settings to ensure that messages are only acknowledged after successful processing. By leveraging the strengths of Google PubSub, this plugin integrates seamlessly with cloud-native architectures, enabling users to build robust and scalable applications that can react to events in real-time.

GroundWork

The GroundWork plugin enables Telegraf to send metrics to a GroundWork Monitor instance, specifically supporting GW8 and newer versions. This integration allows users to leverage the robust monitoring capabilities of GroundWork, enabling comprehensive oversight of metrics collected from diverse sources. Users can specify various parameters such as the GroundWork instance URL, agent IDs, and authentication credentials, allowing for a tailored fit within their existing monitoring setups. It also supports secret-store secrets to enhance security for sensitive fields like username and password. Tags used within the plugin provide fine-grained control over how metrics are categorized and displayed within the GroundWork interface, allowing for custom configurations that adapt to different monitoring needs. However, users should be aware that string metrics are currently not supported by GroundWork, impacting how they manage their data.

Configuration

Google Cloud PubSub

[[inputs.cloud_pubsub]]
  project = "my-project"
  subscription = "my-subscription"
  data_format = "influx"
  # credentials_file = "path/to/my/creds.json"
  # retry_delay_seconds = 5
  # max_message_len = 1000000
  # max_undelivered_messages = 1000
  # max_extension = 0
  # max_outstanding_messages = 0
  # max_outstanding_bytes = 0
  # max_receiver_go_routines = 0
  # base64_data = false
  # content_encoding = "identity"
  # max_decompression_size = "500MB"

GroundWork

[[outputs.groundwork]]
  ## URL of your groundwork instance.
  url = "https://groundwork.example.com"

  ## Agent uuid for GroundWork API Server.
  agent_id = ""

  ## Username and password to access GroundWork API.
  username = ""
  password = ""

  ## Default application type to use in GroundWork client
  # default_app_type = "TELEGRAF"

  ## Default display name for the host with services(metrics).
  # default_host = "telegraf"

  ## Default service state.
  # default_service_state = "SERVICE_OK"

  ## The name of the tag that contains the hostname.
  # resource_tag = "host"

  ## The name of the tag that contains the host group name.
  # group_tag = "group"

Input and output integration examples

Google Cloud PubSub

1. Real-Time Analytics for IoT Devices: Utilize the Google Cloud PubSub plugin to aggregate metrics from IoT devices scattered across various locations. By streaming data from devices to Google PubSub and using this plugin to ingest metrics, organizations can create a centralized dashboard for real-time monitoring and alerting. This setup allows for immediate insights into device performance, facilitating proactive maintenance and operational efficiency.

2. Dynamic Log Processing and Monitoring: Ingest logs from numerous sources via Google Cloud PubSub into a Telegraf pipeline, utilizing the plugin to parse and analyze log messages. This can help teams quickly identify anomalies or patterns in logs and streamline the process of troubleshooting issues across distributed systems. By consolidating log data, organizations can enhance their observability and response capabilities.

3. Event-Driven Workflow Integrations: Use the Google Cloud PubSub plugin to connect various cloud functions or services. Each time a new message is pushed to a subscription, actions can be triggered in other parts of the cloud architecture, such as starting data processing jobs, notifications, or even updates to reports. This event-driven approach allows for a more reactive system architecture that can adapt to changing business needs.

GroundWork

1. Centralized Monitoring Dashboard: Use the GroundWork plugin to aggregate metrics from multiple Telegraf instances into a single GroundWork Monitor dashboard. This configuration offers complete visibility into system health across various components, enabling swift identification of performance bottlenecks and improved incident response times.

2. Service Health Monitoring with Alerts: Configure this plugin to send critical service metrics to GroundWork, establishing a robust alerting system. Metrics such as CPU usage and service statuses can trigger alerts based on threshold values, informing administrators of potential issues before they escalate, thereby enhancing system reliability.

3. Historical Data Analysis: Leverage the historical metric capabilities of GroundWork using this plugin to conduct trend analysis over time. This application allows organizations to make data-driven decisions based on comprehensive historical performance metrics, which can assist in capacity planning and optimize resource allocation.

4. Custom Service Tags for Enhanced Monitoring: Extend the functionality of this plugin by utilizing custom tags for different services and hosts. By customizing these tags, users can filter and categorize metrics more effectively within their monitoring framework, leading to tailored monitoring experiences that align specifically with business objectives.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.