Deadman Alerts with Grafana and InfluxDB Cloud 3.0

By Anais Dotis-Georgiou / Developer
Jun 05, 2024

Flagging failures or inactivity in your monitoring system are crucial for maintaining operational reliability. This blog will guide you through setting up deadman alerts using Grafana and InfluxDB Cloud, tools that help you detect issues before they become critical. We’ll be integrating Grafana’s visualization capabilities with InfluxDB Cloud’s data management features to create a robust monitoring system. By the end of this guide, you’ll understand how to configure and use these tools to keep your systems running smoothly.

This tutorial is a follow-up to Alerting with Grafana and InfluxDB Cloud Serverless, a blog post about configuring threshold alerts in Grafana with data from InfluxDB Cloud. That post leverages the Flight SQL data source, which you can still use to connect to InfluxDB Cloud 3.0. However, Grafana has since unleashed the official InfluxDB v3 data source, which this tutorial uses.

Requirements

To follow this tutorial, you’ll need the following:

• A Telegraf instance installed on your machine
• An InfluxDB Cloud Serverless account
• A local Grafana or Grafana Cloud instance

In this example, we’ll use Telegraf to write CPU metrics to InfluxDB Cloud. I chose this dataset solely to demonstrate how to alert with Grafana, but you can use any Telegraf plugin or dataset you desire. Configure your Telegraf agent by following this documentation. Alternatively, you can use the InfluxDB UI to configure Telegraf by following the instructions in this blog post. (Note that while the post references InfluxDB 2.x, the UX for configuring Telegraf is the same in InfluxDB Cloud 3.0).

You can also follow these steps to configure the Telegraf agent from the UI:

• From the home page, click on Add Data
• Under Telegraf Reporting Agent, select Configure Agent
• Click on + CREATE CONFIGURATION
• Select a bucket to write data to (I already created a bucket called “cpu”)
• Search for and select the CPU plugin
• Follow the pop-up instructions

Next, I verified that I can successfully query data from InfluxDB Cloud 3.0 with the Data Explorer.

Because we’ll be creating a deadman alert, technically you don’t need to send any data to Grafana at all.

Setting up an InfluxDB v3 data source in Grafana

To set up an InfluxDB v3 data source in Grafana, follow these steps. First, navigate to the Data sources page and search for InfluxDB.

Then select InfluxDB and configure it. This means including the database that you want to query data from and monitor, the authentication token, and the URL.

Once you’ve successfully configured an InfluxDB data source, you can test it out by building a panel in a dashboard. Here, we’re querying for the usage_system and time columns. We’ll be creating a deadman alert on this data.

Configuring a deadman alert in Grafana

Now that we’ve successfully connected Grafana to InfluxDB Cloud 3.0 we can configure a deadman alert. Navigate to Alerts & IRM.

Next, select Alerting > Alert rules > New alert rule and begin configuring your deadman alert by providing a rule name.

Now define the query and alert condition. We query for the same data we did to build the visualization above.

To create the deadman expressions, we can count the values and set a threshold so that when the count is less than 1 we know we aren’t returning any data.

Next, set the evaluation behavior. For the sake of testing, we’ll run this test every 1m to ensure that we’re successfully getting deadman alerts.

Now we can configure the labels and notifications. We’ll use the public webhook URL that is available through the Influx Community Slack organization under the #notifications channel.

Finally, we can configure the annotations. A simple “No data incoming” will suffice for this example.

After saving the alert and stopping the Telegraf agent, we successfully received an alert for “no data” in the #notifications channel.

Congratulations! You have successfully configured a deadman alert in Grafana with data from InfluxDB Cloud 3.0.

Final thoughts

Grafana is a robust visualization and alerting tool. While this tutorial barely scratches the surface of what’s possible with Grafana and InfluxDB, it’s worth exploring them further to see how they can improve your workflows and business operations. As always, get started with InfluxDB Cloud 3.0 here. If you need help, please contact our community site or Slack channel.