MAD-Based Anomaly Detection

• InfluxDB 3 Core/Enterprise: with the Processing Engine enabled.
• Python packages:
  • requests (for notification delivery)
• Notification Sender Plugin (optional): Required if using the senders parameter. See the influxdata/notifier plugin.

Installation steps

1. Start InfluxDB 3 with the Processing Engine enabled (--plugin-dir /path/to/plugins):

   influxdb3 serve \
     --node-id node0 \
     --object-store file \
     --data-dir ~/.influxdb3 \
     --plugin-dir ~/.plugins

2. Install required Python packages:

   influxdb3 install package requests

3. (Optional) For notifications, install the influxdata/notifier plugin and create an HTTP trigger for it.

Plugin parameters may be specified as key-value pairs in the --trigger-arguments flag (CLI) or in the trigger_arguments field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s config_file_path parameter.

If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use.

Plugin metadata

This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the InfluxDB 3 Explorer UI to display and configure the plugin.

Required parameters

MAD threshold parameters

Multiple thresholds are separated by @: temp:2.5:20:5@load:3:10:2m

Optional parameters

Default notification templates

• Count: “MAD count alert: Field $field in $table outlier for $threshold_count consecutive points. Tags: $tags”
• Time: “MAD duration alert: Field $field in $table outlier for $threshold_time. Tags: $tags”

Notification channel parameters

Slack

Discord

HTTP

SMS/WhatsApp (via Twilio)

TOML configuration

To use a TOML configuration file, set the PLUGIN_DIR environment variable and specify the config_file_path in the trigger arguments. This is in addition to the --plugin-dir flag when starting InfluxDB 3.

Example TOML configuration

https://github.com/influxdata/influxdb3_plugins/blob/main/influxdata/mad_check/mad_anomaly_config_data_writes.toml

For more information on using TOML configuration files, see the Using TOML Configuration Files section in the influxdb3_plugins/README.md.

Real-time anomaly detection

Detect anomalies as data is written:

influxdb3 create trigger \
  --database mydb \
  --path "gh:influxdata/mad_check/mad_check_plugin.py" \
  --trigger-spec "all_tables" \
  --trigger-arguments 'measurement=cpu,mad_thresholds="temp:2.5:20:5@load:3:10:2m",senders=slack,slack_webhook_url="$SLACK_WEBHOOK_URL"' \
  mad_anomaly_detector

Set SLACK_WEBHOOK_URL to your Slack incoming webhook URL.

Example 1: Basic count-based anomaly detection

Detect when temperature exceeds 2.5 MADs from the median for 5 consecutive points:

# Create trigger for count-based detection
influxdb3 create trigger \
  --database sensors \
  --path "gh:influxdata/mad_check/mad_check_plugin.py" \
  --trigger-spec "all_tables" \
  --trigger-arguments 'measurement=environment,mad_thresholds="temperature:2.5:20:5",senders=slack,slack_webhook_url="$SLACK_WEBHOOK_URL"' \
  temp_anomaly_detector

# Write test data with an anomaly
influxdb3 write \
  --database sensors \
  "environment,room=office temperature=22.1"
influxdb3 write \
  --database sensors \
  "environment,room=office temperature=22.3"
influxdb3 write \
  --database sensors \
  "environment,room=office temperature=45.8"  # Anomaly
# Continue writing anomalous values...

Set SLACK_WEBHOOK_URL to your Slack incoming webhook URL.

Expected output

• Plugin maintains a 20-point window of recent temperature values
• Computes median and MAD from this window
• When temperature exceeds median ± 2.5*MAD for 5 consecutive points, sends Slack notification
• Notification includes: “MAD count alert: Field temperature in environment outlier for 5 consecutive points. Tags: room=office”

Example 2: Duration-based anomaly detection with multiple fields

Monitor CPU load and memory usage with different thresholds:

# Create trigger with multiple thresholds
influxdb3 create trigger \
  --database monitoring \
  --path "gh:influxdata/mad_check/mad_check_plugin.py" \
  --trigger-spec "all_tables" \
  --trigger-arguments 'measurement=system_metrics,mad_thresholds="cpu_load:3:30:2m@memory_used:2.5:30:5m",senders=slack.discord,slack_webhook_url="$SLACK_WEBHOOK_URL",discord_webhook_url="$DISCORD_WEBHOOK_URL"' \
  system_anomaly_detector

Set SLACK_WEBHOOK_URL and DISCORD_WEBHOOK_URL to your webhook URLs.

Expected output

• Monitors two fields independently:
  • cpu_load: Alerts when exceeds 3 MADs for 2 minutes
  • memory_used: Alerts when exceeds 2.5 MADs for 5 minutes
• Sends notifications to both Slack and Discord

Example 3: Anomaly detection with flip suppression

Prevent alert fatigue from rapidly fluctuating values:

# Create trigger with flip suppression
influxdb3 create trigger \
  --database iot \
  --path "gh:influxdata/mad_check/mad_check_plugin.py" \
  --trigger-spec "all_tables" \
  --trigger-arguments 'measurement=sensor_data,mad_thresholds="vibration:2:50:10",state_change_count=3,senders=http,http_webhook_url="$HTTP_WEBHOOK_URL",notification_count_text="Vibration anomaly detected on $table. Field: $field, Tags: $tags"' \
  vibration_monitor

Set HTTP_WEBHOOK_URL to your HTTP webhook endpoint.

Expected output

• Detects vibration anomalies exceeding 2 MADs for 10 consecutive points
• If values flip between normal/anomalous more than 3 times in the 50-point window, suppresses notifications
• Sends custom formatted message to HTTP endpoint

Files

• mad_check_plugin.py: The main plugin code containing the handler for data write triggers
• mad_anomaly_config_data_writes.toml: Example TOML configuration file

Logging

Logs are stored in the trigger’s database in the system.processing_engine_logs table:

influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'"

Log columns:

• event_time: Timestamp of the log event
• trigger_name: Name of the trigger that generated the log
• log_level: Severity level (INFO, WARN, ERROR)
• log_text: Message describing the action or error

Main functions

process_writes(influxdb3_local, table_batches, args)

Handles real-time anomaly detection on incoming data.

Key operations:

1. Filters table batches for the specified measurement
2. Maintains in-memory deques of recent values per field
3. Computes MAD for each monitored field
4. Tracks consecutive outliers and duration
5. Sends notifications when thresholds are met

Key algorithms

MAD (Median Absolute Deviation) Calculation

median = statistics.median(values)
mad = statistics.median([abs(x - median) for x in values])
threshold = k * mad
is_anomaly = abs(value - median) > threshold

Flip Detection

Counts transitions between normal and anomalous states within the window to prevent alert fatigue from rapidly changing values.

Common issues

Issue: No notifications being sent

Solution:

1. Verify the Notification Sender Plugin is installed and running
2. Check webhook URLs are correct:bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%notification%'"
3. Ensure notification channel parameters are provided for selected senders

Issue: “Invalid MAD thresholds format” error

Solution: Check threshold format is correct:

• Count-based: field:k:window:count (e.g., temp:2.5:20:5)
• Duration-based: field:k:window:duration (e.g., temp:2.5:20:2m)
• Multiple thresholds separated by @

Issue: Too many false positive alerts

Solution:

1. Increase the k multiplier (e.g., from 2.5 to 3.0)
2. Increase the threshold count or duration
3. Enable flip suppression with state_change_count
4. Increase the window size for more stable statistics

Issue: Missing anomalies (false negatives)

Solution:

1. Decrease the k multiplier
2. Decrease the threshold count or duration
3. Check if data has seasonal patterns that affect the median

Debugging tips

1. Monitor deque sizes:

influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%Deque%'"

1. Check MAD calculations:

influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%MAD:%'"

1. Test with known anomalies: Write test data with obvious outliers to verify detection

Performance considerations

• Memory usage: Each field maintains a deque of window_count values
• Computation: MAD is computed on every data write for monitored fields
• Caching: Measurement and tag names are cached for 1 hour
• Notification retries: Failed notifications retry up to 3 times with exponential backoff

Schema requirement

The plugin assumes that the table schema is already defined in the database, as it relies on this schema to retrieve field and tag names required for processing.

Using TOML Configuration Files

This plugin supports using TOML configuration files to specify all plugin arguments.

Important Requirements

To use TOML configuration files, you must set the PLUGIN_DIR environment variable in the InfluxDB 3 host environment.

Setting Up TOML Configuration

1. Start InfluxDB 3 with the PLUGIN_DIR environment variable set:

PLUGIN_DIR=~/.plugins influxdb3 serve \
  --node-id node0 \
  --object-store file \
  --data-dir ~/.influxdb3 \
  --plugin-dir ~/.plugins

1. Copy the example TOML configuration file to your plugin directory:

cp mad_anomaly_config_data_writes.toml ~/.plugins/

1. Edit the TOML file to match your requirements:

# Required parameters
measurement = "cpu"
mad_thresholds = "temp:2.5:20:5@load:3:10:2m"
senders = "slack"

# Notification settings
slack_webhook_url = "$SLACK_WEBHOOK_URL"
notification_count_text = "Custom alert: $field anomaly detected"

Set SLACK_WEBHOOK_URL to your Slack incoming webhook URL.

1. Create a trigger using the config_file_path argument:

   influxdb3 create trigger \
     --database mydb \
     --path "gh:influxdata/mad_check/mad_check_plugin.py" \
     --trigger-spec "all_tables" \
     --trigger-arguments config_file_path=mad_anomaly_config_data_writes.toml \
     mad_toml_trigger