Google Cloud PubSub and AWS Redshift Integration

Input and output integration overview

This plugin ingests metrics from Google Cloud PubSub, allowing for real-time data processing and integration into monitoring setups.

This plugin enables Telegraf to send metrics to Amazon Redshift using the PostgreSQL plugin, allowing metrics to be stored in a scalable, SQL-compatible data warehouse.

Integration details

Google Cloud PubSub

The Google Cloud PubSub input plugin is designed to ingest metrics from Google Cloud PubSub, a messaging service that facilitates real-time communication between different systems. It allows users to create and process metrics by pulling messages from a specified subscription in a Google Cloud Project. One of the critical features of this plugin is its ability to operate as a service input, actively listening for incoming messages rather than merely polling for metrics at set intervals. Through various configuration options, users can customize the behavior of message ingestion, such as handling credentials, managing message sizes, and tuning the acknowledgment settings to ensure that messages are only acknowledged after successful processing. By leveraging the strengths of Google PubSub, this plugin integrates seamlessly with cloud-native architectures, enabling users to build robust and scalable applications that can react to events in real-time.

AWS Redshift

This configuration uses the Telegraf PostgreSQL plugin to send metrics to Amazon Redshift, AWS’s fully managed cloud data warehouse that supports SQL-based analytics at scale. Although Redshift is based on PostgreSQL 8.0.2, it does not support all standard PostgreSQL features such as full JSONB, stored procedures, or upserts. Therefore, care must be taken to predefine compatible tables and schema when using Telegraf for Redshift integration. This setup is ideal for use cases that benefit from long-term, high-volume metric storage and integration with AWS analytics tools like QuickSight or Redshift Spectrum. Metrics stored in Redshift can be joined with business datasets for rich observability and BI analysis.

Configuration

Google Cloud PubSub

[[inputs.cloud_pubsub]]
  project = "my-project"
  subscription = "my-subscription"
  data_format = "influx"
  # credentials_file = "path/to/my/creds.json"
  # retry_delay_seconds = 5
  # max_message_len = 1000000
  # max_undelivered_messages = 1000
  # max_extension = 0
  # max_outstanding_messages = 0
  # max_outstanding_bytes = 0
  # max_receiver_go_routines = 0
  # base64_data = false
  # content_encoding = "identity"
  # max_decompression_size = "500MB"

AWS Redshift

[[outputs.postgresql]]
  ## Redshift connection settings
  host = "redshift-cluster.example.us-west-2.redshift.amazonaws.com"
  port = 5439
  user = "telegraf"
  password = "YourRedshiftPassword"
  database = "metrics"
  sslmode = "require"

  ## Optional: specify a dynamic table template for inserting metrics
  table_template = "telegraf_metrics"

  ## Note: Redshift does not support all PostgreSQL features; ensure your table exists and is compatible

Input and output integration examples

Google Cloud PubSub

1. Real-Time Analytics for IoT Devices: Utilize the Google Cloud PubSub plugin to aggregate metrics from IoT devices scattered across various locations. By streaming data from devices to Google PubSub and using this plugin to ingest metrics, organizations can create a centralized dashboard for real-time monitoring and alerting. This setup allows for immediate insights into device performance, facilitating proactive maintenance and operational efficiency.

2. Dynamic Log Processing and Monitoring: Ingest logs from numerous sources via Google Cloud PubSub into a Telegraf pipeline, utilizing the plugin to parse and analyze log messages. This can help teams quickly identify anomalies or patterns in logs and streamline the process of troubleshooting issues across distributed systems. By consolidating log data, organizations can enhance their observability and response capabilities.

3. Event-Driven Workflow Integrations: Use the Google Cloud PubSub plugin to connect various cloud functions or services. Each time a new message is pushed to a subscription, actions can be triggered in other parts of the cloud architecture, such as starting data processing jobs, notifications, or even updates to reports. This event-driven approach allows for a more reactive system architecture that can adapt to changing business needs.

AWS Redshift

1. Business-Aware Infrastructure Monitoring: Store infrastructure metrics from Telegraf in Redshift alongside sales, marketing, or customer engagement data. Analysts can correlate system performance with business KPIs using SQL joins and window functions.

2. Historical Trend Analysis for Cloud Resources: Use Telegraf to continuously log CPU, memory, and I/O metrics to Redshift. Combine with time-series SQL queries and visualization tools like Amazon QuickSight to spot trends and forecast resource demand.

3. Security Auditing of System Behavior: Send metrics related to system logins, file changes, or resource spikes into Redshift. Analysts can build dashboards or reports for compliance auditing using SQL queries across multi-year data sets.

4. Cross-Environment SLA Reporting: Aggregate SLA metrics from multiple cloud accounts and regions using Telegraf, and push them to a central Redshift warehouse. Enable unified SLA compliance dashboards and executive reporting via a single SQL interface.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.