Apache Aurora and TimescaleDB Integration

Input and output integration overview

This plugin gathers metrics from Apache Aurora schedulers, providing insights necessary for effective monitoring of Aurora clusters.

This output plugin delivers a reliable and efficient mechanism for routing Telegraf collected metrics directly into TimescaleDB. By leveraging PostgreSQL’s robust ecosystem combined with TimescaleDB’s time series optimizations, it supports high-performance data ingestion and advanced querying capabilities.

Integration details

Apache Aurora

The Aurora plugin is designed to gather metrics from Apache Aurora schedulers. This plugin connects to specified schedulers using their respective URLs and retrieves operational metrics that help in monitoring the health and performance of Aurora clusters. It primarily captures numeric data from the /vars endpoint, ensuring key metrics related to task execution and resource utilization are monitored. The plugin enhances operational insights by utilizing HTTP Basic Authentication for secure access. With optional TLS configuration, it further bolsters security when transmitting data. The plugin provides a robust way to interface with Apache Aurora, reflecting a focus on operational reliability and ongoing performance assessment across distributed systems.

TimescaleDB

TimescaleDB is an open source time series database built as an extension to PostgreSQL, designed to handle large scale, time-oriented data efficiently. Launched in 2017, TimescaleDB emerged in response to the growing need for a robust, scalable solution that could manage vast volumes of data with high insert rates and complex queries. By leveraging PostgreSQL’s familiar SQL interface and enhancing it with specialized time series capabilities, TimescaleDB quickly gained popularity among developers looking to integrate time series functionality into existing relational databases. Its hybrid approach allows users to benefit from PostgreSQL’s flexibility, reliability, and ecosystem while providing optimized performance for time series data.

The database is particularly effective in environments that demand fast ingestion of data points combined with sophisticated analytical queries over historical periods. TimescaleDB has a number of innovative features like hypertables which transparently partition data into manageable chunks and built-in continuous aggregation. These allow for significantly improved query speed and resource efficiency.

Configuration

Apache Aurora

[[inputs.aurora]]
  ## Schedulers are the base addresses of your Aurora Schedulers
  schedulers = ["http://127.0.0.1:8081"]

  ## Set of role types to collect metrics from.
  ##
  ## The scheduler roles are checked each interval by contacting the
  ## scheduler nodes; zookeeper is not contacted.
  # roles = ["leader", "follower"]

  ## Timeout is the max time for total network operations.
  # timeout = "5s"

  ## Username and password are sent using HTTP Basic Auth.
  # username = "username"
  # password = "pa$$word"

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

TimescaleDB

# Publishes metrics to a TimescaleDB database
[[outputs.postgresql]]
  ## Specify connection address via the standard libpq connection string:
  ##   host=... user=... password=... sslmode=... dbname=...
  ## Or a URL:
  ##   postgres://[user[:password]]@localhost[/dbname]?sslmode=[disable|verify-ca|verify-full]
  ## See https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING
  ##
  ## All connection parameters are optional. Environment vars are also supported.
  ## e.g. PGPASSWORD, PGHOST, PGUSER, PGDATABASE
  ## All supported vars can be found here:
  ##  https://www.postgresql.org/docs/current/libpq-envars.html
  ##
  ## Non-standard parameters:
  ##   pool_max_conns (default: 1) - Maximum size of connection pool for parallel (per-batch per-table) inserts.
  ##   pool_min_conns (default: 0) - Minimum size of connection pool.
  ##   pool_max_conn_lifetime (default: 0s) - Maximum connection age before closing.
  ##   pool_max_conn_idle_time (default: 0s) - Maximum idle time of a connection before closing.
  ##   pool_health_check_period (default: 0s) - Duration between health checks on idle connections.
  # connection = ""

  ## Postgres schema to use.
  # schema = "public"

  ## Store tags as foreign keys in the metrics table. Default is false.
  # tags_as_foreign_keys = false

  ## Suffix to append to table name (measurement name) for the foreign tag table.
  # tag_table_suffix = "_tag"

  ## Deny inserting metrics if the foreign tag can't be inserted.
  # foreign_tag_constraint = false

  ## Store all tags as a JSONB object in a single 'tags' column.
  # tags_as_jsonb = false

  ## Store all fields as a JSONB object in a single 'fields' column.
  # fields_as_jsonb = false

  ## Name of the timestamp column
  ## NOTE: Some tools (e.g. Grafana) require the default name so be careful!
  # timestamp_column_name = "time"

  ## Type of the timestamp column
  ## Currently, "timestamp without time zone" and "timestamp with time zone"
  ## are supported
  # timestamp_column_type = "timestamp without time zone"

  ## Templated statements to execute when creating a new table.
  # create_templates = [
  #   '''CREATE TABLE {{ .table }} ({{ .columns }})''',
  # ]

  ## Templated statements to execute when adding columns to a table.
  ## Set to an empty list to disable. Points containing tags for which there is
  ## no column will be skipped. Points containing fields for which there is no
  ## column will have the field omitted.
  # add_column_templates = [
  #   '''ALTER TABLE {{ .table }} ADD COLUMN IF NOT EXISTS {{ .columns|join ", ADD COLUMN IF NOT EXISTS " }}''',
  # ]

  ## Templated statements to execute when creating a new tag table.
  # tag_table_create_templates = [
  #   '''CREATE TABLE {{ .table }} ({{ .columns }}, PRIMARY KEY (tag_id))''',
  # ]

  ## Templated statements to execute when adding columns to a tag table.
  ## Set to an empty list to disable. Points containing tags for which there is
  ## no column will be skipped.
  # tag_table_add_column_templates = [
  #   '''ALTER TABLE {{ .table }} ADD COLUMN IF NOT EXISTS {{ .columns|join ", ADD COLUMN IF NOT EXISTS " }}''',
  # ]

  ## The postgres data type to use for storing unsigned 64-bit integer values
  ## (Postgres does not have a native unsigned 64-bit integer type).
  ## The value can be one of:
  ##   numeric - Uses the PostgreSQL "numeric" data type.
  ##   uint8 - Requires pguint extension (https://github.com/petere/pguint)
  # uint64_type = "numeric"

  ## When using pool_max_conns > 1, and a temporary error occurs, the query is
  ## retried with an incremental backoff. This controls the maximum duration.
  # retry_max_backoff = "15s"

  ## Approximate number of tag IDs to store in in-memory cache (when using
  ## tags_as_foreign_keys). This is an optimization to skip inserting known
  ## tag IDs. Each entry consumes approximately 34 bytes of memory.
  # tag_cache_size = 100000

  ## Cut column names at the given length to not exceed PostgreSQL's
  ## 'identifier length' limit (default: no limit)
  ## (see https://www.postgresql.org/docs/current/limits.html)
  ## Be careful to not create duplicate column names!
  # column_name_length_limit = 0

  ## Enable & set the log level for the Postgres driver.
  # log_level = "warn" # trace, debug, info, warn, error, none

Input and output integration examples

Apache Aurora

1. Dynamic Resource Allocation Monitoring: Utilize the Aurora plugin to build a real-time dashboard displaying metrics related to resource allocation in your Aurora clusters. By aggregating data from multiple schedulers, you can visualize how resources are distributed among various roles (leader and follower), enabling proactive management of resource utilization and helping prevent bottlenecks in production workloads.

2. Alerting on Scheduler Health: Implement alerting mechanisms where the Aurora plugin checks the health of schedulers periodically. If a scheduler role responds with a status that indicates a failure to communicate (non-200 status), alerts can be automatically generated and sent to the operations team via email or messaging apps, ensuring immediate attention to critical issues and maintaining availability in service management.

3. Performance Benchmarking Over Time: By continuously collecting metrics such as job update events and execution times, this plugin can assist teams in benchmarking the performance of their Apache Aurora deployment over time. Relevant metrics can be logged into a time-series database, enabling historical analysis, trend identification, and understanding how changes in the system, such as configuration adjustments or workload changes, impact performance.

4. Integration with CI/CD Pipelines: Integrate the metrics collected via the Aurora plugin with CI/CD pipeline tools to monitor how deployments affect runtime metrics in Aurora. Teams can thereby ensure that new releases do not adversely impact scheduler performance and can roll back changes seamlessly if any metric exceeds predefined thresholds after deployment.

TimescaleDB

1. Real-Time IoT Data Ingestion: Use the plugin to collect and store sensor data from thousands of IoT devices in real time. This setup facilitates immediate analysis, helping organizations monitor operational efficiency and respond quickly to changing conditions.

2. Cloud Application Performance Monitoring: Leverage the plugin to feed detailed performance metrics from distributed cloud applications into TimescaleDB. This integration supports real-time dashboards and alerts, enabling teams to swiftly identify and mitigate performance bottlenecks.

3. Historical Data Analysis and Reporting: Implement a system where long-term metrics are stored in TimescaleDB for comprehensive historical analysis. This approach allows businesses to perform trend analysis, generate detailed reports, and make data-driven decisions based on archived time-series data.

4. Adaptive Alerting and Anomaly Detection: Integrate the plugin with automated anomaly detection workflows. By continuously streaming metrics to TimescaleDB, machine learning models can analyze data patterns and trigger alerts when anomalies occur, enhancing system reliability and proactive maintenance.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.