Apache Aurora and New Relic Integration

Input and output integration overview

This plugin gathers metrics from Apache Aurora schedulers, providing insights necessary for effective monitoring of Aurora clusters.

This plugin allows the sending of metrics to New Relic Insights using the Metrics API, enabling effective monitoring and analysis of application performance.

Integration details

Apache Aurora

The Aurora plugin is designed to gather metrics from Apache Aurora schedulers. This plugin connects to specified schedulers using their respective URLs and retrieves operational metrics that help in monitoring the health and performance of Aurora clusters. It primarily captures numeric data from the /vars endpoint, ensuring key metrics related to task execution and resource utilization are monitored. The plugin enhances operational insights by utilizing HTTP Basic Authentication for secure access. With optional TLS configuration, it further bolsters security when transmitting data. The plugin provides a robust way to interface with Apache Aurora, reflecting a focus on operational reliability and ongoing performance assessment across distributed systems.

New Relic

This plugin writes metrics to New Relic Insights utilizing the Metrics API, which provides a robust mechanism for sending time series data to the New Relic platform. Users must first obtain an Insights API Key to authenticate and authorize their data submissions. The plugin is designed to facilitate easy integration with New Relic’s monitoring and analytics capabilities, supporting a variety of metric types and allowing for efficient data handling. Core features include the ability to add prefixes to metrics for better identification, customizable timeouts for API requests, and support for proxy settings to enhance connectivity. It is essential for users to configure these options according to their requirements, enabling seamless data flow into New Relic for comprehensive real-time analytics and insights.

Configuration

Apache Aurora

[[inputs.aurora]]
  ## Schedulers are the base addresses of your Aurora Schedulers
  schedulers = ["http://127.0.0.1:8081"]

  ## Set of role types to collect metrics from.
  ##
  ## The scheduler roles are checked each interval by contacting the
  ## scheduler nodes; zookeeper is not contacted.
  # roles = ["leader", "follower"]

  ## Timeout is the max time for total network operations.
  # timeout = "5s"

  ## Username and password are sent using HTTP Basic Auth.
  # username = "username"
  # password = "pa$$word"

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

New Relic

[[outputs.newrelic]]
  ## The 'insights_key' parameter requires a NR license key.
  ## New Relic recommends you create one
  ## with a convenient name such as TELEGRAF_INSERT_KEY.
  ## reference: https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#ingest-license-key
  # insights_key = "New Relic License Key Here"

  ## Prefix to add to add to metric name for easy identification.
  ## This is very useful if your metric names are ambiguous.
  # metric_prefix = ""

  ## Timeout for writes to the New Relic API.
  # timeout = "15s"

  ## HTTP Proxy override. If unset use values from the standard
  ## proxy environment variables to determine proxy, if any.
  # http_proxy = "http://corporate.proxy:3128"

  ## Metric URL override to enable geographic location endpoints.
  # If not set use values from the standard
  # metric_url = "https://metric-api.newrelic.com/metric/v1"

Input and output integration examples

Apache Aurora

1. Dynamic Resource Allocation Monitoring: Utilize the Aurora plugin to build a real-time dashboard displaying metrics related to resource allocation in your Aurora clusters. By aggregating data from multiple schedulers, you can visualize how resources are distributed among various roles (leader and follower), enabling proactive management of resource utilization and helping prevent bottlenecks in production workloads.

2. Alerting on Scheduler Health: Implement alerting mechanisms where the Aurora plugin checks the health of schedulers periodically. If a scheduler role responds with a status that indicates a failure to communicate (non-200 status), alerts can be automatically generated and sent to the operations team via email or messaging apps, ensuring immediate attention to critical issues and maintaining availability in service management.

3. Performance Benchmarking Over Time: By continuously collecting metrics such as job update events and execution times, this plugin can assist teams in benchmarking the performance of their Apache Aurora deployment over time. Relevant metrics can be logged into a time-series database, enabling historical analysis, trend identification, and understanding how changes in the system, such as configuration adjustments or workload changes, impact performance.

4. Integration with CI/CD Pipelines: Integrate the metrics collected via the Aurora plugin with CI/CD pipeline tools to monitor how deployments affect runtime metrics in Aurora. Teams can thereby ensure that new releases do not adversely impact scheduler performance and can roll back changes seamlessly if any metric exceeds predefined thresholds after deployment.

New Relic

1. Application Performance Monitoring: Use the New Relic Telegraf plugin to send application performance metrics from a web service to New Relic Insights. By integrating this plugin, developers can collect data such as response times, error rates, and throughput, enabling teams to monitor application health in real-time and resolve issues quickly before they impact users. This setup promotes proactive management of application performance and user experience.

2. Infrastructure Metrics Aggregation: Leverage this plugin to aggregate and send system-level metrics (CPU usage, memory consumption, etc.) from various servers to New Relic. This helps system administrators maintain an comprehensive view of infrastructure performance, facilitating capacity planning and identifying potential bottlenecks. By centralizing metrics in New Relic, teams can visualize trends over time and make informed decisions regarding resource allocation.

3. Dynamic Metric Naming for Multi-tenant Applications: Implement dynamic prefixing with the metric_prefix option to differentiate between different tenants in a multi-tenant application. By configuring the plugin to include a unique identifier per tenant in the metric names, teams can analyze usage patterns and performance metrics per tenant. This provides valuable insights into tenant behavior, supporting tailored optimizations and enhancing service quality across different customer segments.

4. Real-time Anomaly Detection: Combine the New Relic plugin with alerting mechanisms to trigger notifications based on unusual metric patterns. By sending metrics such as request counts and response times, teams can set thresholds in New Relic that, when breached, will automatically alert responsible parties. This user-driven approach supports immediate responses to potential issues before they escalate into larger incidents.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.