How NetApp Uses InfluxDB to Minimize Service Downtime in Build & Test

Overview

Supporting Engineering at Scale with Unified System Telemetry

Within NetApp, the BAERO team (Build, Automation, and EngineeRing Operations) supports the build, test, and automation infrastructure used by developers and QA engineers to validate changes and maintain the quality of NetApp ONTAP, the operating system behind NetApp’s storage products.

Across two of its largest testing environments alone, BAERO supports more than half a million hours of ONTAP testing every month. Keeping those systems available requires Site Reliability Engineers (SREs) to continuously understand service health, detect degradation, and respond before problems become outages.

To create that uptime visibility, the BAERO team built a unified telemetry platform using the Telegraf, InfluxDB, Grafana (TIG) reference architecture. The platform brings infrastructure, service, and automation telemetry together so SREs can measure reliability, investigate performance, and coordinate incident response across NetApp’s globally distributed team.

Challenge

Maintaining Reliability Across NetApp’s Test Infrastructure

To maintain the quality of ONTAP, NetApp developers and QA engineers rely on two large-scale testing platforms: the Common Test Lab (CTL) and the Continuous Integration Test Environment (CIT). CTL spans approximately 32,000 compute nodes, 35,000 client VMs, and 90 unique hardware configurations, supporting roughly 100,000 hours of testing each month. CIT runs more than 1,100 continuous integration tests on recurring schedules, adding another 440,000 hours. Together, the two environments support more than half a million hours of testing every month.

The BAERO) team’s SREs are responsible for keeping these systems reliable and available. As Dustin Sorge points out, “If you were to boil SRE down to one thing, SREs care about service uptime.” That means continuously monitoring for service degradation, investigating latency and failed jobs, and responding before emerging problems turn into downtime.

For BAERO, SRE also helped bridge a longstanding divide between development and operations. The Common Test Lab’s software was built by a development team in Pittsburgh and operated by a team in Raleigh. When issues emerged after a release, they often moved back and forth between the two groups. Creating an SRE function gave NetApp a dedicated team responsible for operating the service and responding to issues, while allowing developers to stay focused on building features.

But SREs still needed better visibility into the systems they were responsible for. As Sorge, recalled: “All the pieces were very disjointed, so we did the best we could.” Operational metrics was stored across multiple MySQL and MariaDB databases, resource utilization was investigated using host-level Linux tools, and custom scripts recorded service and automation metrics.

That fragmentation also limited alerting. Nagios could warn the team about basic infrastructure conditions, such as a filesystem filling up, but BAERO needed more flexible alerts tied to health of its services:API latency, SLI thresholds, and failed automation jobs, while also tracking performance against SLOs. Those signals were harder to monitor consistently when the underlying telemetry lived across different tools and systems.

As NetApp’s testing environments and telemetry workloads grew, the SRE team needed a scalable monitoring foundation, not more disconnected tools.

Solution

Building a Unified Telemetry Foundation for SRE

After evaluating Prometheus, NetApp decided to standardize on InfluxDB directly. Sorge recalled that during the evaluation, his team saw architectures that sent metrics through Prometheus before moving the data into InfluxDB for scalability. BAERO chose a simpler path.

With InfluxDB as its common telemetry repository, the SRE team simplified how metrics were collected and delivered using Telegraf, which has over 200 input plugins and gave BAERO a consistent collection layer across its environment. Scripts and automation can write directly to InfluxDB, while a Jenkins integration records job status so SREs can immediately identify failed jobs and trigger custom alerts.

Grafana queried InfluxDB to populate dashboards with real-time and historical views of infrastructure and service health. Stream processing transformed incoming telemetry into new time series, continuously evaluated it against custom conditions, and routed alerts directly to service-specific Slack channels used by NetApp’s globally distributed SRE team.

BAERO even monitors its monitoring infrastructure. The team runs what Sorge calls a “watcher of watchers”: a separate, open source InfluxDB instance that monitors the health of the production Enterprise cluster itself. If the primary cluster ever has a problem, SREs still have a dashboard to diagnose it, instead of losing visibility into their own monitoring platform at the worst possible time.

Result

Turning Telemetry into Faster, More Proactive Response

Moving from disconnected monitoring tools to a common telemetry foundation built on InfluxDB has changed how BAERO manages reliability, detects problems, and coordinates response.

• Catch regressions that earlier testing misses. By tracking production API latency, BAERO can identify performance changes that passed staging and unit tests, correlate them with a software release, and trace them to the responsible code in minutes.
• Measure and track reliability. BAERO tracks Service Level Indicators (SLIs) against Service Level Objectives (SLOs) in one place, giving SREs a consistent way to measure service performance and improve over time
• Coordinate incident response globally. Alerts flow into Slack giving SREs across locations a shares place to investigate and resolve service issues.
• Address issues proactively. SREs identify and investigate service degradation before users report an issue.
• Bring new hosts under monitoring quickly with Telegraf and configurable dashboards and alerting in place, Sorge estimates that adding a host could take “a minute, maybe.”

The Common Test Lab’s reservation-add API shows the system in action.ONTAP developers and QA engineers use the API to reserve test-environment. Telegraf extracts latency measurements from the service logs and writes them to InfluxDB. BAERO then computes an SLI based on API latency and compares it against a 99% Service Level Objective (SLO); if the SLI falls below the SLO, the platform alerts the SRE team in Slack immediately.

That visibility has already caught regressions that testing missed.In one case, the team saw a spike in reservation-add latency and traced the beginning of the spike to a recent software release. From there, the team traced it back to the specific code change responsible in minutes rather than hours.

For Sorge, that ability to turn telemetry into action is now fundamental to how BAERO operates. Powered by InfluxDB, the platform has become the “rock-solid foundation” for how the team measures reliability, detects problems, and responds to incidents.