Amazon CloudWatch and AWS Redshift Integration

Input and output integration overview

This plugin will pull Metric Statistics from Amazon CloudWatch, streamlining the process of monitoring and analyzing AWS resources.

This plugin enables Telegraf to send metrics to Amazon Redshift using the PostgreSQL plugin, allowing metrics to be stored in a scalable, SQL-compatible data warehouse.

Integration details

Amazon CloudWatch

The Amazon CloudWatch Plugin allows users to pull detailed metric statistics from Amazon’s CloudWatch service. As a monitoring solution, CloudWatch enables users to track various metrics related to AWS resources and applications, facilitating improved operational and performance insights. The plugin uses a structured authentication method that prioritizes security and flexibility through a combination of STS (Security Token Service), shared credentials, environment variables, and EC2 instance profiles, ensuring robust access control to AWS resources. Key features include the ability to define specific metric namespaces, aggregated periods for metrics, and optional inclusion of linked accounts for cross-account monitoring. A significant aspect of this plugin is its capacity to handle both sparse and dense metric formats, allowing for varied output structures depending on user preference. Thus, it supports versatile use cases in cloud monitoring and analytics by providing comprehensive, timely data directly from CloudWatch.

AWS Redshift

This configuration uses the Telegraf PostgreSQL plugin to send metrics to Amazon Redshift, AWS’s fully managed cloud data warehouse that supports SQL-based analytics at scale. Although Redshift is based on PostgreSQL 8.0.2, it does not support all standard PostgreSQL features such as full JSONB, stored procedures, or upserts. Therefore, care must be taken to predefine compatible tables and schema when using Telegraf for Redshift integration. This setup is ideal for use cases that benefit from long-term, high-volume metric storage and integration with AWS analytics tools like QuickSight or Redshift Spectrum. Metrics stored in Redshift can be joined with business datasets for rich observability and BI analysis.

Configuration

Amazon CloudWatch

[[inputs.cloudwatch]]
  region = "us-east-1"
  # access_key = ""
  # secret_key = ""
  # token = ""
  # role_arn = ""
  # web_identity_token_file = ""
  # role_session_name = ""
  # profile = ""
  # shared_credential_file = ""
  # include_linked_accounts = false
  # endpoint_url = ""
  # use_system_proxy = false
  # http_proxy_url = "http://localhost:8888"
  period = "5m"
  delay = "5m"
  interval = "5m"
  #recently_active = "PT3H"
  # cache_ttl = "1h"
  namespaces = ["AWS/ELB"]
  # metric_format = "sparse"
  # ratelimit = 25
  # timeout = "5s"
  # batch_size = 500
  # statistic_include = ["average", "sum", "minimum", "maximum", sample_count]
  # statistic_exclude = []
  # [[inputs.cloudwatch.metrics]]
  #  names = ["Latency", "RequestCount"]
  #  [[inputs.cloudwatch.metrics.dimensions]]
  #    name = "LoadBalancerName"
  #    value = "p-example"

AWS Redshift

[[outputs.postgresql]]
  ## Redshift connection settings
  host = "redshift-cluster.example.us-west-2.redshift.amazonaws.com"
  port = 5439
  user = "telegraf"
  password = "YourRedshiftPassword"
  database = "metrics"
  sslmode = "require"

  ## Optional: specify a dynamic table template for inserting metrics
  table_template = "telegraf_metrics"

  ## Note: Redshift does not support all PostgreSQL features; ensure your table exists and is compatible

Input and output integration examples

Amazon CloudWatch

1. Cross-Account Monitoring: Utilize this plugin to monitor resources across multiple AWS accounts by enabling the include_linked_accounts option. This scenario allows companies managing multiple AWS accounts to aggregate metrics into a central monitoring dashboard, providing a unified view of all metrics while ensuring secure data access and compliance through proper role management.

2. Dynamic Alerting System: Integrate this plugin with alerting tools to create an automated system that triggers alerts based on defined thresholds for CloudWatch metrics. For instance, if latency metrics exceed specified limits, alerts can be sent to relevant teams, enabling proactive responses to performance issues and reducing downtime.

3. Cost Management Dashboard: Use the metrics gathered from the plugin to build a cost management dashboard that visualizes AWS service usage metrics over time. By correlating these metrics with billing data, organizations can identify high-cost services and take informed actions to optimize their resource usage and spending.

4. Performance Benchmarking for Applications: Leverage the metrics collected from applications running on AWS to perform performance benchmarks. For example, by tracking latency and request count metrics for an ELB, developers can assess the impact of application changes on its performance, making data-driven decisions for optimization.

AWS Redshift

1. Business-Aware Infrastructure Monitoring: Store infrastructure metrics from Telegraf in Redshift alongside sales, marketing, or customer engagement data. Analysts can correlate system performance with business KPIs using SQL joins and window functions.

2. Historical Trend Analysis for Cloud Resources: Use Telegraf to continuously log CPU, memory, and I/O metrics to Redshift. Combine with time-series SQL queries and visualization tools like Amazon QuickSight to spot trends and forecast resource demand.

3. Security Auditing of System Behavior: Send metrics related to system logins, file changes, or resource spikes into Redshift. Analysts can build dashboards or reports for compliance auditing using SQL queries across multi-year data sets.

4. Cross-Environment SLA Reporting: Aggregate SLA metrics from multiple cloud accounts and regions using Telegraf, and push them to a central Redshift warehouse. Enable unified SLA compliance dashboards and executive reporting via a single SQL interface.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.