InfluxData Blog - Anais Dotis-Georgiou

How to Use the Power BI Desktop InfluxDB 3 ODBC Connector

Anais Dotis-Georgiou (InfluxData) — Fri, 24 Oct 2025 08:00:00 +0000

The challenge of storing, processing, and alerting on your time series data is only part of the battle when it comes to deriving value from time-stamped data. While InfluxDB 3 addresses those hurdles with the database and Python processing engine, data analytics teams still need to be able to visualize their data and build dashboards to complete the time series story.

A Power BI connector for InfluxDB is a long-desired feature, and I’m happy to present you with today’s tutorial, in which we’ll learn how to use the Power BI InfluxDB 3 ODBC (Open Database Connectivity) connector to bring your time series data stored in InfluxDB 3 into Power BI. This blog post assumes you have the following requirements; if not, please download and install them:

A Docker instance of InfluxDB 3 Core or Enterprise running, or a free trial of InfluxDB 3 Cloud
Windows (For Mac users, consider using a VM like Parallels)
Power BI Desktop (a free trial is also available)
Arrow Flight SQL ODBC Driver
The Power BI Desktop InfluxDB 3 connector

Please note that this tutorial is for Power BI Desktop only.

Setup walk through

In case you’re entirely new to InfluxDB 3, Docker, and Power BI, this section is for you as we’ll walk through the installation process for all the requirements above. Otherwise, feel free to skip this section and move on to the next.

I’m a Mac user, so the first thing I did was to install Parallels and activate their free trial. The docs and driver are easy to follow so I won’t go into detail here. I’m assuming that most people reading this post are Windows users anyways.

Install Power BI Desktop and the Arrow Flight SQL ODBC Driver

First, I downloaded Power BI Desktop. Activating your free trial is easy; just follow the prompts after installation. A screenshot of Power BI Desktop’s Free Trial Home page.

To install the Arrow Flight SQL ODBC Driver, you can run the following commands in PowerShell, making sure to correct the pathways by specifying your user:

# 1) Download
Invoke-WebRequest -Uri "https://download.dremio.com/arrow-flight-sql-odbc-driver/arrow-flight-sql-odbc-LATEST-win64.msi"
  -OutFile "C:\Users\"YOUR USER" \Downloads\arrow-flight-sql-odbc-win64.msi"

# 2) Unblock (mark as trusted)
Unblock-File "C:\Users\"YOUR USER" \Downloads\arrow-flight-sql-odbc-win64.msi"

# 3) Install (GUI)
Start-Process msiexec.exe -Wait -ArgumentList '/i "C:\Users\"YOUR USER"\Downloads\arrow-flight-sql-odbc-win64.msi"

From there, follow the installation instructions with default settings as per the driver instructions.

Setting Up InfluxDB

Next, I set up InfluxDB 3 Core. I decided to use InfluxDB 3 Core as it is InfluxData’s OSS version of InfluxDB. However, you can also use InfluxDB 3 Enterprise and register for a free trial during setup. I used Docker to run this instance; follow that link to install Docker if you don’t already have it. I also used VSCode and the Dev Containers extension to connect to the running Docker container to run shell commands directly within the container. However, for simplicity, I’ll include Docker container exec commands here instead. A screenshot of running commands directly within the container, after connecting with the Dev Containers extension in VS Code.

You can find the complete install docs for InfluxDB 3 Core here. First, we need to pull the image with:

docker pull influxdb:3-core

Next, we can serve InfluxDB 3 Core with:

docker run --rm --name influxdb3-core -p 8181:8181 \
  -v $PWD/data:/var/lib/influxdb3/data \
  -v $PWD/plugins:/var/lib/influxdb3/plugins \

  influxdb:3-core influxdb3 serve \
    --node-id=my-node-0 \
    --object-store=file \
    --data-dir=/var/lib/influxdb3/data \
    --plugin-dir=/var/lib/influxdb3/plugins

This command runs an InfluxDB 3 Core container and removes it when stopped (--rm). It maps port 8181 from the container to your local machine so you can access the database at localhost:8181. Two local directories are mounted into the container: one for persistent data ($PWD/data) and one for plugins ($PWD/plugins). Finally, it starts the InfluxDB 3 server with a specific node ID, using the filesystem as the object store (see all the options here), and points it to the mounted data and plugin directories. While we won’t be using the Python Processing Engine and plugins as a part of this tutorial, I like to serve all my instances of InfluxDB this way so it is available for future use.

Now, we need to create a token for InfluxDB 3 Core and write some data to it. Let’s create that admin token with:

# With Docker — in a new terminal:
docker exec -it influxdb3-core influxdb3 create token --admin

Copy the first token you see; we’ll use it in the next command. In order to write data into InfluxDB, we first need to create a database to write data to:

docker exec -it influxdb3-core influxdb3 create database --token AUTH_TOKEN DATABASE_NAME

Replace DATABASE_NAME with the name of your choice—I used mydatabase, for example. Congratulations! You’ve successfully set up InfluxDB 3 Core.

Writing data to InfluxDB 3 Core

Now we’re ready to write some data to InfluxDB 3 Core so we can query and visualize it in Power BI Desktop. But before we do, I want to showcase a useful feature in the docs:

Editing code snippets in the InfluxDB Docs to more easily write data to InfluxDB 3.

Notice the red text: there you can edit any variables you need directly within the docs, like tokens and database names, to more easily copy code snippets.

Right, let’s get back to actually writing some sample data with the following:

influxdb3 write \
  --database DATABASE_NAME \
  --token AUTH_TOKEN \
  --precision s \
'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000
home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000
home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600
home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600
home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200
home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200
home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800
home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800
home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400
home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400
home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000
home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000
home,room=Living\ Room temp=22.3,hum=36.1,co=0i 1641045600
home,room=Kitchen temp=22.8,hum=36.3,co=1i 1641045600
home,room=Living\ Room temp=22.3,hum=36.1,co=1i 1641049200
home,room=Kitchen temp=22.7,hum=36.2,co=3i 1641049200
home,room=Living\ Room temp=22.4,hum=36.0,co=4i 1641052800
home,room=Kitchen temp=22.4,hum=36.0,co=7i 1641052800
home,room=Living\ Room temp=22.6,hum=35.9,co=5i 1641056400
home,room=Kitchen temp=22.7,hum=36.0,co=9i 1641056400
home,room=Living\ Room temp=22.8,hum=36.2,co=9i 1641060000
home,room=Kitchen temp=23.3,hum=36.9,co=18i 1641060000
home,room=Living\ Room temp=22.5,hum=36.3,co=14i 1641063600
home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063600
home,room=Living\ Room temp=22.2,hum=36.4,co=17i 1641067200
home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200'

The sample datasets consist of sensor data that records the temperature of various rooms in a house. Alternatively, consider writing a file of line protocol data with the file flag instead, using a client library or Telegraf.

Enabling the Power BI Desktop InfluxDB 3 Connector

We’re ready to set up the gateway to query our InfluxDB 3 instance, connect to our InfluxDB 3 Core instance, query our data, and create visualizations in Power BI Desktop.

Connector Extensibility

For this tutorial, we’re connecting InfluxDB 3 to Power BI through a custom connector. To enable custom connectors, I followed the Power BI Desktop custom Connector Extensibility documentation. Make sure you have downloaded the .pqx file from the Power BI Desktop InfluxDB 3 Connector download page. Copy or move it to [Documents]\Microsoft Power BI Desktop\Custom Connectors (make sure to create those folders if they don’t exist) with:

# 1. Create the Power BI custom connectors folder if it doesn't exist
mkdir -p ~/Documents/Power\ BI\ Desktop/Custom\ Connectors

# 2. Move your connector into that folder
mv InfluxDB.pqx ~/Documents/Power\ BI\ Desktop/Custom\ Connectors/

Now, return to Power BI Desktop. From the home page, select Options and Settings in the bottom left, then Options. Screenshot of the options and settings page in Power BI Desktop, the first step in enabling the InfluxDB Connector. On the Options page in Power BI Desktop, click on “(Not Recommended) Allow any extension to load without validation or warning.”

Navigate to Security and select (Not Recommended) under Data Extensions. As per the Connector Extensibility docs, “The default Power BI Desktop data extension security setting is ‘(Recommended) Only allow Microsoft certified and other trusted third-party extensions to load.’ With this setting, if uncertified custom connectors are on your system, the Uncertified Connectors dialog box appears at startup and lists the connectors that can’t load.”

After you’ve changed these settings make sure to restart Power BI Desktop so that the connector can be discoverable.

Connect to InfluxDB 3

Return to the Power BI Desktop home page. Select Get Data from the Home ribbon in Power BI Desktop. Search and select InfluxDB 3. A screenshot of Power BI Desktop, searching for the InfluxDB 3 Custom connector.

We’re ready to use the connector to get data from our InfluxDB 3 Core instance. First, we need to specify the server URL. Since this is running on a Docker container on our machine, we can use http://localhost:8181 for the Server field. However, since I was running Windows on Parallels, I specified the IP directly with http://10.211.55.2. Make sure to include your database name and port (8181). Finally, we’ll select the Native Query option and include the data that we want to pull into Power BI.

Important Note: Because InfluxDB can handle high throughput and dimensional data, make sure to limit the size of the query request for each data source collection, so that Power BI can successfully handle data volume. Limit the size by using a limit clause or specifying time ranges, and optionally filtering by columns (or tags).

Querying InfluxDB 3 Core with a Native Query In Power BI Desktop.

Select DirectQuery as your query type and make sure to apply Native Query. Select Direct Query as your Data Connectivity mode when configuring the Power BI Desktop InfluxDB3 connector.

Finally, make sure to add your token on the next page in order to complete the connection to your InfluxDB 3 instance: The final step in connecting to your InfluxDB 3 instance with a custom connector in Power BI Desktop.

We are then offered a preview of the data we want to load into Power BI Desktop, confirming a successful connection. Yay! We did it! Click Load to load the data. Preview of the time series data from InfluxDB 3 to be loaded into Power BI Desktop.

Visualizing data from InfluxDB 3 in Power BI Desktop

Congratulations! We’re now able to see our data in Power BI, apply filters, and create an assortment of visualizations. In the screenshot below we have a table view of all our data. We can apply filters to that data through the Filters panel. Make sure to specify the columns you want to include in your table on the right-most Query column. In the screenshot below, I select for the sum of co, room, and time. Fields are summed by default. Select the field in the Fields pane or the Visualizations pane, go to the Modeling tab, and change the Default Summarization to “Don’t Summarize,” if you desire raw values. Viewing summarized time series data (default) in Power BI Desktop from InfluxDB 3.

Click on different visualization types in the middle Visualizations control panel, i.e., a line graph:

A line graph with data from InfluxDB 3 in Power BI Desktop.

Final thoughts

I hope this tutorial has helped you load your time series data from InfluxDB 3 to Power BI Desktop. This integration is a long awaited feature request, and I’m delighted that we’ve been able to deliver it to our users. For those who are new to InfluxDB, I want to take this opportunity to share other visualization tools that InfluxDB 3 offers in case you’re still exploring visualization and dashboarding options:

Grafana
Superset (for InfluxDB Cloud)
Tableau (for InfluxDB Cloud)
Streamlit

As always, if you need help, please contact us on our community site or Slack channel. If you are also working on a visualization project with InfluxDB, I’d love to hear from you! Please share your feedback on those channels as well.

Edge Data Replication: Contributions and Status Updates for InfluxDB 3

Anais Dotis-Georgiou (InfluxData) — Tue, 03 Jun 2025 08:00:00 +0000

If you’ve ever stood up multiple edge InfluxDB instances in remote locations and wished you could consolidate their data into a centralized instance for analysis, you’re not alone. That’s exactly why we designed Edge Data Replication (EDR) in InfluxDB v2. Now, with InfluxDB 3 Core and 3 Enterprise, we’re seeing new ways to handle replication using the brand-new Python Processing Engine.

In this post, I’ll walk you through:

What EDR looked like in InfluxDB v2 and why it matters for TSDBs
The new Python plugin engine for InfluxDB 3 Core and 3 Enterprise
A community-built EDR plugin ready for immediate use
Setup instructions and requirements
A sneak peek at official Support and how to find both official and community plugins

EDR in InfluxDB v2

In InfluxDB v2, Edge Data Replication gave you a built-in way to forward writes from an edge instance (like InfluxDB OSS) to a remote InfluxDB Cloud bucket. It handled durability, retries, compression, filtering, and optional aggregation. This made it easy to collect data from the edge and then stream it back to a centralized system for backup, analysis, or visualization.

In InfluxDB 3 Core and Enterprise, the architecture has changed. There’s no built-in EDR. Instead, we have a Python-based Processing Engine, which is even more flexible.

Introducing the Python Processing Engine for InfluxDB 3

InfluxDB 3 Core and Enterprise now support a Python-based plugin engine. It lets you run Python code inside the database, triggered by either:

a WAL flush (new data arriving)
a schedule (like a cron job)
or a manual request (an HTTP API call you define)

This means you can process data as it’s written, transform or downsample it, send alerts—or, yes, replicate it elsewhere. To activate it, you pass --plugin-dir when starting the server and drop your .py plugin files in that directory. We’ll walk through how to use a data replicator plugin later in this post.

A DevRel-contributed data replicator plugin

The official EDR plugin for InfluxDB 3 is expected to be released soon—it’s currently undergoing some final testing. In light of that, I wanted to share Developer Advocate @suyashcjoshi ’s early version of a data replicator plugin, which is available today, and inspired development details of the official plugin.

You can find his InfluxDB 3 Custom Data Replication Plugin in our public plugin repository. It works with both InfluxDB 3 Core and 3 Enterprise and includes:

Replication to InfluxDB Cloud (Serverless or Dedicated) or InfluxDB 3 Enterprise
Table filtering
Optional downsampling via aggregate_interval
A compressed queue (edr_queue.jsonl.gz) for durability

Like all plugins, it works with any data source, whether you’re writing via Telegraf, a script, or your own app.

Requirements

Here’s what you’ll need to run this Plugin:

InfluxDB 3 Core or 3 Enterprise
A free-trail version of InfluxDB Cloud Serverless
InfluxDB 3 Python client: pip install influxdb3-python
Python 3.7+
Optional: Telegraf for simulating system metrics

How to use the plugin

Here’s a quick-start guide for testing this plugin locally and replicating the data to InfluxDB Cloud Serverless.

Start InfluxDB with the Plugin Directory.

mkdir -p ~/.plugins
cp data-replicator.py ~/.plugins/

influxdb3 serve \

  --node-id host01 \
  --object-store file \
  --data-dir ~/.influxdb3 \
  --plugin-dir ~/.plugins

Download and install the InfluxDB 3 Python Client.

(Optional) Run Telegraf. You can elect to run the Telegraf config that’s provided in the data replicator directory to quickly write system stats from your machine to your local InfluxDB 3 instance. You simply have to provide an authorization token. Run the following commands to generate a token and run the Telegraf config.
```
influxdb 3 create token 
telegraf --config telegraf.conf
```

Create and enable the Replication Trigger. Here’s how you replicate all tables to a cloud instance:

influxdb3 create trigger \
  -d mydb \
  --plugin-filename data-replicator.py \
  --trigger-spec "all_tables" \
  --trigger-arguments "host=YOUR_HOST,token=YOUR_TOKEN,database=TARGET_DB" \
  data_replicator_trigger

influxdb3 enable trigger --database mydb data_replicator_trigger

If you want to replicate only some tables or downsample the data to one-minute average values, you can add the following trigger arguments to the “create trigger” command above.

--trigger-arguments "tables=cpu,aggregate_interval=1m"

What’s next: official EDR and Downsampling Plugin status

We’re working on an officially-supported EDR Plugin that will:

Data Replication: Replicate data from a local InfluxDB 3 instance to a remote one.
Filtering: Specify which tables to replicate and which fields to exclude.
Renaming: Rename tables and fields during replication.
Downsampling: When enabled, downsample all data within the specified time window for scheduled triggers or for each individual run for data write triggers.
Queue Management: Use a compressed JSONL queue file for reliable delivery.
Retry Logic: Handle errors and rate limits with retry mechanisms.

We’re also developing a robust Downsampling Plugin that you can use alongside the EDR plugin that offers the following features:

Flexible Downsampling: Aggregate data over specified time intervals (e.g., seconds, minutes, hours, days, weeks, months, quarters, or years) using functions like avg, sum, min, max, or derivative.
Field and Tag Filtering: Select specific fields for aggregation and filter by tag values.
Scheduler and HTTP Support: Run periodically via InfluxDB triggers or on-demand via HTTP requests.
Retry Logic: Configurable retries for robust write operations
Batch Processing: Process large datasets in configurable time batches for HTTP requests.
Backfill Support: Downsample historical data within a specified time window.
Task ID Tracking: Each execution generates a unique task_id included in logs and error messages for traceability in late-arriving data.
Metadata Columns: Each downsampled record includes three additional columns to verify accurate downsampling and easily re-downsample data in the face of late-arriving data.

Final thoughts

Edge Data Replication isn’t just back; it’s more flexible than ever thanks to the Python Processing Engine. With InfluxDB 3 and the new Processing Engine, you can write plugins that transform, downsample, and replicate data in real-time, right from the edge. You’ll also be able to take the official plugins and easily adapt them to meet your own needs. I hope this post encourages you to contribute a plugin of your own. Check out our Getting Started Guide for Core and Enterprise, and share your feedback with our development team on Discord in the #influxdb3_core channel, Slack in the #influxdb3_core channel, or our Community Forums.

Forecasting with InfluxDB 3 and HuggingFace

Anais Dotis-Georgiou (InfluxData) — Tue, 20 May 2025 07:00:00 +0000

Machine learning models must do more than make accurate predictions; they also need to adapt as the world around them changes. In real-world systems, data distributions shift due to seasonality, equipment wear, user behavior changes, or other external forces. If your models can’t keep up, the result is poor predictions. This can lead to outages, inefficiencies, or missed opportunities. That’s why forecasting systems need to be monitored and resilient, not just accurate. In this post, we’ll walk through a full-stack ML demo that shows exactly how to do that. We combine ML model monitoring with tooling to forecast time series data, detect when models start to drift, and automatically retrain them in response.

For example, in an industrial IoT setting, a predictive maintenance model might learn to detect motor failure based on temperature and vibration data. But over time, the equipment wears down, operators change procedures, or sensors begin to degrade, causing the model’s assumptions about “normal” to drift. If the model isn’t updated, it may miss early failure warnings or raise false alarms, leading to costly downtime or unnecessary maintenance. In finance, a trading model might rely on patterns that no longer hold true after market volatility shifts, leading to poor decisions if it isn’t retrained. These are the kinds of scenarios where detecting and responding to drift in real time becomes critical.

The demo uses a PyTorch-based LSTM model for forecasting, InfluxDB 3 for storing time series data and model metrics, and Hugging Face Hub for cloud-based model storage and versioning. It’s all wrapped in a Streamlit app so you can interactively explore the pipeline, from synthetic data generation to drift-aware retraining. InfluxDB, a purpose-built time series database (TSDB), is particularly well suited for this kind of pipeline. Time series workloads are naturally indexed by time and benefit from TSDB features like fast temporal queries, downsampling, retention policies, and streaming ingestion. By storing everything from raw inputs to forecasts and retraining events in InfluxDB, you get efficient storage and retrieval and a transparent audit trail of your model’s behavior over time.

The pipeline follows a modular ML lifecycle: generating sine wave-based time series data, writing that data into InfluxDB 3 Core (open source), and training an initial model. The model is then saved to Hugging Face. You can then simulate concept drift by injecting noise or offsets, and the system will automatically detect degraded performance using MSRE and MSE thresholds. When drift is detected, the pipeline allows you to trigger a retraining process, uploads the new model to Hugging Face, and logs the event for traceability. Every stage, from data generation to initial training, forecasts, drift detection, and retraining events, is captured in InfluxDB 3 Core to ensure full visibility of the model’s health over time.

For this demo, we’re using InfluxDB 3 Core, the open source version for recent data. It’s ideal for development, prototyping, and recent time series workloads. However, if you plan to scale this pipeline and include high availability, scalability, enhanced security, and long-term storage, then InfluxDB 3 Enterprise is a better fit. Luckily, switching between products is easy. InfluxDB 3 Enterprise is a superset of InfluxDB 3 Core, and migration happens in-place (for this project, it’s as simple as pulling the Enterprise docker image instead of the Core image).

This project was built with Replit; you can find the corresponding project repo here. A screenshot from the Streamlit application showing the difference between the original time series and the drifted data.

Requirements

To run this project locally, you’ll need the following:

Python 3.11+: For model training, inference, and app execution
InfluxDB: To store your time series data
Docker: Used to spin up a local instance of InfluxDB 3 Core
Hugging Face Account: To upload and download models from your repository
PyTorch: Required libraries for LSTM model implementation and persistence
Streamlit: To power the interactive UI that orchestrates the pipeline

Quick start

1. Clone the repository and set up the environment.

git clone https://github.com/yourusername/lstm-forecasting-drift-detection.git
cd lstm-forecasting-drift-detection
python -m venv venv
source venv/bin/activate  # On Windows, use: venv\Scripts\activate
pip install -r requirements.txt

2. Start InfluxDB 3 Core with Docker.

Note: We are mounting our plugin directory here to prepare for this project’s next evolution, which utilizes the InfluxDB 3 Python Processing Engine.

docker run -it --rm --name influxdb3core \
  -v ~/influxdb3/data:/var/lib/influxdb3 \
  -v ~/influxdb3_huggingface_forecasting_monitoringplugins:/plugins \
  -p 8181:8181 \
  quay.io/influxdb/influxdb3-core:latest serve \
  --node-id my_host \
  --object-store file \
  --data-dir /var/lib/influxdb3
  --plugin-dir /plugins

3. Create the database and auth token for InfluxDB 3 Core.

docker exec -it influxdb3core influxdb3 database create --name timeseries
docker exec -it influxdb3core influxdb3 auth create --name my-token

4. Set up environment variables. Create a .env file in the project root.

env
INFLUXDB_HOST=http://localhost:8181
INFLUXDB_TOKEN=your_influxdb_token_here
INFLUXDB_DATABASE=timeseries
HF_TOKEN=your_huggingface_token_here
HF_REPO_ID=your_username/your_repo_name

5. Run the app.

streamlit run [app.py](http://app.py)

The Streamlit dashboard will be live at http://localhost:5000. The Streamlit app: the first step runs you through configuring your InfluxDB instance. This will automatically be populated with environment variables, but you can choose to configure it through the UI instead.

App walkthrough

Tab 0 InfluxDB Configuration: First, we need to connect our application to an InfluxDB instance to store our time series data and model metrics. The InfluxDB Configuration tab makes this straightforward. The app will also automatically populate your Connection Settings with env variables if you provide them.

Tab 1 Data Generation: Now that we’re connected, let’s generate some synthetic time series data to work with. This tab allows us to create a clean sine wave with customizable parameters.

Using the sidebar sliders, you can adjust:

Number of data points: Controls how many data points to generate
Noise level: Adds random variation to the sine wave
Frequency factor: Changes how quickly the wave oscillates
Amplitude: Sets the height of the wave

Click Generate New Data, and the application will:

Create a synthetic time series based on your parameters
Split it into training and testing sets
Write it to InfluxDB (if connected)
Display a visualization of the generated data

Tab 2 Initial Model Training: Click Train LSTM Model to begin training. The application will:

Prepare sequences from your time series data
Scale the data for better convergence
Create and train an LSTM model with your chosen parameters
Display training progress and final metrics

The charts show:

Training and validation loss curves (to check for overfitting)
Model predictions on the test data
A detailed view of the last portion of predictions vs. actuals

The model accuracy metrics give you immediate feedback on how well your model learned the patterns in your data.

Pro Tip: If your training loss decreases but validation loss increases, you might be overfitting. Try increasing the dropout rate or reducing the number of LSTM units.

Tab 3 Drift Injection: Now comes the interesting part! In real-world scenarios, data patterns often change over time, causing models to become less accurate. This phenomenon is known as data drift. This tab lets you simulate drift to see how it affects model performance.

From the left-hand panel, you can configure:

Drift type: Choose between “offset” (a vertical shift) or “noise” (increased randomness)
Drift start point: When the drift should begin (as a percentage of the dataset)
Drift magnitude: How severe the drift should be

Click Inject Drift to apply these changes to your data. The application will:

Take your original data and add drift at the specified point
Store the drifted data in InfluxDB 3 Core
Show a comparison between the original and drifted data

The visualization clearly highlights where the drift begins and how it affects the data pattern. The red portion represents the drifted section. Behind the scenes, the drifted data is stored in the drifted_data measurement in InfluxDB 3 Core, preserving both the original and modified versions.

Tab 4 Drift Detection: Once drift is present in our data, we need a systematic way to detect it. This tab demonstrates using error metrics to identify when model performance deteriorates.

You can configure:

Drift metric: Choose between MSRE (Mean Squared Relative Error) or MSE (Mean Squared Error)
Drift threshold: The error value above which drift is considered detected
Window size: Number of points to include in each sliding window for error calculation

Click Detect Drift to analyze model performance on the drifted data. The application will:

Run the original model on drifted data
Calculate error metrics using sliding windows
Determine when/if drift is detected
Visualize error metrics over time

The chart shows error metrics with a horizontal line representing your threshold. When the metrics cross this line, the system flags drift detection. You’ll also see the exact window where drift was first detected.

Pro Tip: MSRE is often more sensitive to relative changes in pattern, while MSE is better for detecting absolute magnitude changes. Choose based on what’s more important for your use case.

Tab 5 Model Retraining: Once drift is detected, the appropriate response is usually to retrain the model on more recent data that includes the new patterns. This tab demonstrates how to retrain and compare model performance.

Click Retrain Model with Drifted Data to:

Create a new training dataset that includes drifted data
Train a new model with the same architecture but on updated data
Compare predictions from both the original and retrained models
Calculate improvement metrics

The visualization shows three lines:

Actual values (ground truth)
Predictions from the original model
Predictions from the retrained model

You can clearly see how the retrained model adapts to the new pattern while the original model continues to follow the old pattern. The metrics quantify this improvement in terms of reduced error.

Tab 6 Model Persistence: Our final tab handles saving and loading models to/from Hugging Face, enabling model versioning and sharing capabilities.

You’ll see fields for:

Hugging Face Repository ID: Where to store your models (format: “username/repo-name”)
Model Name: Identifier for this specific model

Click Save Model to Hugging Face to:

Serialize the current model (either original or retrained)
Upload it to your Hugging Face repository
Record metadata about the saved model

Code overview

The project is organized into modular utility scripts that mirror the structure of the ML pipeline.

data_generator.py handles synthetic data creation and drift injection, allowing you to simulate real-world scenarios with evolving patterns.
model.py defines the LSTM architecture, training loop, forecasting logic, and model persistence methods.
drift_detection.py implements statistical drift detection using MSRE and MSE metrics with support for sliding windows.
huggingface_utils.py manages saving and loading models from the Hugging Face Hub, enabling easy model versioning and sharing.
influxdb_utils.py handles all interactions with InfluxDB 3 Core, from writing training metrics to querying time series data.

Together, these scripts form a complete forecasting monitoring loop—each one loosely coupled, testable, and easy to extend.

Limitations and Python Processing Engine

While this demo provides a full walkthrough of forecasting, drift detection, and retraining, it’s currently designed as a static application. Each step runs once when manually triggered via the Streamlit UI. In real-world production systems, forecasting and monitoring need to happen continuously and autonomously. The natural next step for this project is to integrate with the InfluxDB 3 Python Processing Engine, which allows you to schedule these scripts to run on a user-defined schedule or triggered on request. This would enable automated forecasting, real-time drift monitoring, and hands-free retraining. All driven directly by new data arriving in InfluxDB. Moving from manual to scheduled execution is what transforms this project from an educational tool into a robust model monitoring observability pipeline. For example, you could generate new data every hour with a schedule trigger, monitor for drift in real-time with a WAL-flush trigger, trigger a model retrain when needed with a HTTP request trigger, and push updated models to Hugging Face—all without user intervention. This shift would turn the project from a static demo into a dynamic model monitoring pipeline.

Looking ahead: from static scripts to plugins

For example, you could easily create a Data Generator Schedule Trigger with the following code:

"""
Simple Sine Wave Generator for InfluxDB 3 Processing Engine

This plugin generates a sine wave with noise and writes it to InfluxDB.
"""

import numpy as np
from datetime import datetime, timedelta
def process_scheduled_call(influxdb3_local, call_time, args=None):
    """
    Generate a sine wave with noise and write it to InfluxDB.

    Parameters:
    -----------

    influxdb3_local : InfluxDB3Local
        Interface for interacting with InfluxDB
    call_time : str
        The time the scheduled call was made
    args : dict, optional
        Arguments for data generation:
        - measurement: Name of the measurement/table (default: 'synthetic_data')
        - periods: Number of data points to generate (default: 168)
        - amplitude: Sine wave amplitude (default: 10.0)
        - noise: Noise level (default: 0.5)
    """

    # Parse arguments with defaults
    args = args or {}
    measurement = args.get("measurement", "synthetic_data")
    periods = int(args.get("periods", "168"))
    amplitude = float(args.get("amplitude", "10.0"))
    noise = float(args.get("noise", "0.5"))

    influxdb3_local.info(f"Generating {periods} data points for measurement '{measurement}'")

    # Get current time and calculate start time (hourly data)
    end_time = datetime.now()
    start_time = end_time - timedelta(hours=periods)

    # Generate sine wave with noise
    for i in range(periods):
        # Calculate timestamp for this point
        timestamp = start_time + timedelta(hours=i)
        unix_nano = int(timestamp.timestamp() * 1e9)

        # Generate sine value (full cycle every 24 points)
        x = (i / 24) * 2 * np.pi
        value = amplitude * np.sin(x)

        # Add noise
        if noise > 0:`
            value += np.random.normal(0, noise)

        # Create and write point
        line = LineBuilder(measurement)
        line.tag("source", "generator")
        line.float64_field("value", value)
        line.time_ns(unix_nano)

        influxdb3_local.write(line)

    influxdb3_local.info(f"Successfully generated and wrote {periods} data points")

    # At the end of your process_scheduled_call function, before the return:
    stats = {
        "measurement": measurement,
        "points_generated": periods,
        "start_time": start_time.isoformat(),
        "end_time": end_time.isoformat(),
        "amplitude": amplitude,
        "noise_level": noise
    }
    influxdb3_local.info(f"Statistics: {stats}")
    return stats

This plugin is designed to run on a schedule within the InfluxDB 3 Python Processing Engine. Each time it’s triggered, it generates a configurable number of sine wave data points with added noise, timestamps them at hourly intervals, and writes them to a specified measurement in InfluxDB. The core logic lives in the process_scheduled_call() function, which is the required entry point for scheduled plugins. It uses the influxdb3_local interface provided by the Processing Engine to log messages and write data. Data points are formatted using the LineBuilder class to construct line protocol records with tags, fields, and nanosecond-precision timestamps. At the end, it logs a summary of what was written, making this plugin a useful tool for simulating time series data streams in automated workflows.

Then you would create with:

influxdb3 create trigger \
  --trigger-spec "every:1m" \
  --plugin-filename "plugins/data_generator.py" \
  --database timeseries \
  data_generator_trigger
```
And enable it with:
```bash
influxdb3 enable trigger  --database timeseries data_generator_trigger

It’s also worth mentioning that you don’t have to use the LineBuilder class to write data in an InfluxDB 3 Core or Enterprise Python Processing Plugin. You can use the InfluxDB 3 Python Client Library instead, especially in scheduled or on-request plugins. This approach can be more convenient when you’re writing larger datasets, such as full Pandas DataFrames, all at once. For example, when model drift is detected and a retraining step is triggered, it often makes sense to write the resulting metrics or retrained model outputs in batch rather than looping through row-by-row with LineBuilder. This should also make converting our existing scripts to an InfluxDB 3 Python Processing Engine pipeline easier since many already operate on DataFrames.

Conclusion

This project helps to start bridging the gap between machine learning and model monitoring. Combining InfluxDB 3 for time series storage and Hugging Face for model versioning shows how you can monitor and adapt models as data evolves over time. While the current app runs in a controlled, step-by-step environment, the architecture lays the groundwork for a fully automated system. With just a few additions, like scheduled plugins powered by the InfluxDB 3 Processing Engine, you can turn this demo into a more robust workflow for real-time forecasting. Whether you’re monitoring sensor data, energy usage, or financial trends, this approach gives you the tools to stay ahead of model drift and keep your predictions reliable.

I encourage you to look at the InfluxData/influxdb3_plugins as we add examples and plugins. Also, please contribute your own! To learn more about building your plugin, check out these resources:

Additionally, check out these resources for more information and examples of other alert plugins:

I invite you to contribute any plugin that you create. Check out our Getting Started Guide for Core and Enterprise, and share your feedback with our development team on Discord in the #influxdb3_core channel, Slack in the #influxdb3_core channel, or our Community Forums.

Build a Time Series Forecasting Pipeline in InfluxDB 3 Without Writing Code

Anais Dotis-Georgiou (InfluxData) — Thu, 17 Apr 2025 08:00:00 +0000

Curious how time series forecasting fits into your InfluxDB 3 workflows? Let’s build a complete forecasting pipeline together using InfluxDB 3 Core’s Python Processing Engine and Facebook’s Prophet library.

InfluxDB 3 Core’s Python Processing Engine dramatically lowers the barrier to entry—not just for experienced developers but for anyone with a basic understanding of time series data and Python. It turns what used to be a complex, multi-day task into something you can prototype in a few hours, making advanced forecasting and data processing far more accessible and accelerating the path from idea to insight.

One of the most exciting aspects of this project is how quickly it came together using a large language model (LLM). By simply providing InfluxDB 3 Core’s Python Processing Engine the documentation and FB Prophet quick start guide, the LLM generated working plugin code, wired up the full pipeline, and even suggested improvements.

Now, let’s build a pipeline that predicts daily pageviews for the Wikipedia article on Peyton Manning over an entire year, starting with historical data and ending with an interactive Plotly visualization. To build alongside this project, download InfluxDB 3 Core or Enterprise. This project will cover:

Details on what I provided my LLM (ChatGPT-4o) with to execute this
The requirements and setup required to run this end-to-end forecasting pipeline examples
Creating necessary InfluxDB 3 Core and Enterprise resources
Writing data, making a forecast, and visualizing the results

The forecasting pipeline includes three purpose-built plugins: one to load historical data, another to generate daily forecasts, and one to visualize results on demand:

load_peyton (load_peyton_data.py): An HTTP-triggered plugin that loads sample Wikipedia pageview data from a CSV and writes it to the peyton_views table.
peyton_forecast (forecast_peyton.py): A scheduled plugin (e.g., every:1d) that queries the peyton_views table, fits a Prophet model and writes a full 365-day forecast to the prophet_forecast table.
forecast_plot (plot_forecast_http.py): An HTTP-triggered plugin that queries both peyton_views and prophet_forecast, merges them, and returns an interactive Plotly graph as HTML.

A visualization of the historical page view data (blue) and forecasted page view data (green) that you can generate by following this tutorial. Accessible by visiting http://localhost:8181/api/v3/engine/plot_forecast.

From coding to AI validation

I generated the code for this project by providing the following resources to ChatGPT-4o: The Processing engine and Python plugins documentation and the Prophet Quickstart example. From there, I gave it a handful of natural-language prompts to build the components I needed iteratively. Here are the core prompts I used:

“Can you write a plugin for InfluxDB 3 that uses Facebook Prophet to forecast time series data?” This got the conversation started and helped ChatGPT understand the target use case and library.
“Create a plugin that loads historical data from a public CSV (like the Peyton Manning Wikipedia views) and writes it to InfluxDB.” This became the HTTP-triggered loader plugin that populates the peyton_views table.
“Now, write a scheduled plugin that queries that table, fits a Prophet model, and writes the forecast to another table. Make sure the logic writes all forecast rows.” This became the daily forecasting plugin, which outputs yhat, yhat_lower, and yhat_upper to prophet_forecast.
“Can you build a third plugin that reads both the historical and forecasted data, plots it using Plotly, and returns the result over HTTP as an HTML page?” This plugin allows me to visualize the entire pipeline in my browser without setting up an external dashboard.

By layering the prompts above, I built a fully functioning, interactive forecasting pipeline inside InfluxDB without writing any code manually. What began as traditional coding quickly became a process of prompt engineering and AI validation.

What began as traditional coding quickly became a process of prompt engineering and AI validation. I described intent, reviewed generated code, and iterated until everything worked end-to-end.

That said, using the test command for the Processing Engine was incredibly helpful—not just for this project, but for building and validating any plugin. The test command quickly triggers a plugin with cURL or verifies results with SQL queries. It made the feedback loop tight and efficient, especially when paired with AI-generated code.

Requirements and setup

You can either run this example locally or within a Docker container. Follow the InfluxDB 3 Core or Enterprise installation guides, as this post applies to both products. I recommend using Docker, so this post will assume you’re running InfluxDB 3 in a containerized environment for ease of setup, isolation, and cleanup. Make sure Docker is installed on your system and you’ve pulled the latest InfluxDB 3 image for your chosen edition. I’ll use InfluxDB 3 Core as it’s the OSS version. Enterprise is also freely available for local development and testing if you select at-home use.

Once you pull this repo, save the file in your configured plugin directory (e.g., /path/to/plugins/). Then run the following command:

docker run -it --rm --name test_influx -v ~/influxdb3/data:/var/lib/influxdb3   -v /path/to/plugins/:/plugins -p 8181:8181 quay.io/influxdb/influxdb3-core:latest serve --node-id my_host --object-store file --data-dir /var/lib/influxdb3 --plugin-dir /plugin

This command runs a temporary InfluxDB 3 Core container named test_influx using the latest image. It mounts your local data directory to persist database files and the plugin directory containing the deadman check plugin. It also exposes port 8181 so you can access the database locally, and start the server using the serve command with file-based object storage (you could also easily use an S2 bucket for file-based storage), a custom node ID, and the mounted plugin directory.

Writing data, making a forecast, and visualizing our data

Before we create and enable any triggers that use the plugins mentioned above, we need to install all dependencies. This project depends on Plotly and Prophet. Install them using:

influxdb3 install package plotly 
influxdb3 install package prophet

Next, create a database to write page view data to:

influxdb3 create database prophet

Now, we can create our triggers. First, we’ll create Plugin 1 and load data via HTTP:

influxdb3 create trigger \
  --trigger-spec "request:load_peyton" \
  --plugin-filename "load_peyton_data.py" \
  --database prophet \
  load_peyton

The load_peyton plugin:

Is a HTTP-triggered plugin
Downloads a public CSV of daily Wikipedia views
Writes rows to the peyton_views table in InfluxDB 3 Core

We can then execute the following cURL command to trigger the execution of the load_peyton plugin:

curl http://localhost:8181/api/v3/engine/load_peyton

You should see the following output, which confirms a successful write of the data:

{"status": "success", "rows_written": 2905}

Next, we can create forecasts on schedule:

influxdb3 create trigger \
  --trigger-spec "every:1m" \
  --plugin-filename "forecast_peyton.py" \
  --database prophet \
  peyton_forecast

The peyton_forecast plugin:

Scheduled plugin (runs daily or on your schedule)
Reads data from peyton_views table
Fits a Prophet model
Forecasts 365 days into the future
Writes summary forecast results to prophet_forecast table

I’ve set the peyton_forecast plugin to run every minute to get a forecast quickly. However, since we’re looking at daily data, you’d more likely run this type of pipeline on a daily interval. After you’ve successfully generated and written a forecast, you should see the following output:

processing engine: Running Prophet forecast on 'peyton_views'INFO - Chain [1] start processing
INFO - Chain [1] done processing

To prevent the plugin from running indefinitely and recruiting the same forecasting workload, disable it with:

inflxudb3 disable trigger --database prophet peyton_forecast

Finally, we can visualize our data by enabling the final plugin with:

influxdb3 create trigger \
  --trigger-spec "request:plot_forecast" \
  --plugin-filename "plot_forecast_http.py" \
  --database prophet \
  forecast_plot

The forecast_plot plugin:

Is a HTTP-triggered plugin
Reads data from both peyton_views and prophet_forecast
Creates an interactive Plotly chart combining historical data and forecasts
Returns the chart as HTML for browser viewing

Visit http://localhost:8181/api/v3/engine/plot_forecast to view the historical data and forecast shared in the first section of this tutorial.

Final thoughts and considerations

This collection of plugins for InfluxDB 3 Core and Enterprise provides a basic example of how to build an end-to-end data collection, forecasting, and visualization pipeline using the InfluxDB 3 Python Processing Engine. While this project focuses on a simple FB Prophet-based forecast, it can easily be extended into a more robust and production-ready system. For example, you could load a pre-trained forecasting model from Hugging Face for faster inference, to monitor forecast accuracy over time to detect model drift, and to schedule automated retraining when performance degrades.

Pairing this pipeline with InfluxDB 3’s Processing Engine alerting capabilities allows you to proactively respond to anomalies or drift events by sending notifications or triggering remediation workflows. With these building blocks, you can create intelligent, self-healing time series pipelines tailored to your use case.

I encourage you to look at the InfluxData/influxdb3_plugins as we add examples and plugins. Also, please contribute your own! To learn more about building your plugin, check out these resources:

Additionally, check out these resources for more information and examples of other alert plugins:

Deadman Alerts with the Python Processing Engine

Anais Dotis-Georgiou (InfluxData) — Wed, 09 Apr 2025 07:00:00 +0000

Sometimes silence isn’t golden; it’s a red flag. Whether you’re monitoring IoT sensors, system logs, or application metrics, missing data can be just as critical as abnormal data. Without visibility into these gaps, you risk overlooking potential failures, security threats, or operational inefficiencies. In time series workflows, detecting silence is often the first sign of trouble—whether it’s a network issue, device failure, sensor failure, or stalled process.

In a previous blog post, we learned to use the Python Processing Engine with InfluxDB 3 Core and Enterprise to build threshold alerts and send notifications to Slack, Discord, or HTTP endpoint using a Wall Sync trigger. In this post, we’ll learn how to build a deadman check and alert by leveraging a schedule trigger known as a deadman trigger. Deadman triggers are a powerful alerting strategy that notifies you immediately when expected data stops arriving.

The deadman check plugin can be found here. The plugin monitors a target table for recent writes and sends a Slack alert if no new data is received within a configurable time threshold.

This blog post will cover:

Requirements and Setup for InfluxDB 3 Core and Enterprise (this post works with both Core and Enterprise)
Getting a Slack Webhook URL
Creating InfluxDB 3 Core and Enterprise resources
Testing the plugin and sending a deadman alert

Requirements and setup

Download InfluxDB 3 Core or Enterprise and follow the appropriate Core or Enterprise installation guides to work alongside this tutorial. You can either run this example locally or within a Docker container but I recommend using Docker for ease of setup, isolation, and cleanup. That said, this post assumes you’re running InfluxDB 3 in a Docker containerized environment.

Before you can run this in Docker, make sure Docker is installed on your system and pull the latest InfluxDB 3 image for your chosen edition (Core or Enterprise). I’m going to use InfluxDB 3 Core, as it’s the OSS version. You can use Enterprise for no cost by specifying “at-home use.”

Once you pull this repo, save the file as deadman_alert.py in your configured plugin directory (e.g., /path/to/plugins/). Then run the following command:

docker run -it --rm --name test_influx -v
~/influxdb3/data:/var/lib/influxdb3   -v /path/to/plugins/:/plugins -p 
8181:8181 quay.io/influxdb/influxdb3-core:latest serve --node-id 
my_host --object-store file --data-dir /var/lib/influxdb3 --plugin-dir 
/plugin

This command runs a temporary InfluxDB 3 Core container named test_influx using the latest image. It mounts your local data directory to persist database files and mounts the plugin directory containing the deadman check plugin. It also exposes port 8181 so you can access the database locally and start the server using the serve command with file-based object storage (you could also easily use an S2 bucket for file-based storage), a custom node ID, and the mounted plugin directory.

Follow this documentation on how to create a Slack webhook URL. You’ll need to include the webhook as an argument during the trigger creation process. Alternatively, you can use acpublic webhook that offers users an opportunity to test out InfluxDB-related notifications. It’s pinned to the #notifications-testing channel in the InfluxDB Slack.

Creating InfluxDB 3 resources and generating a deadman alert

Create a database to monitor for a heartbeat:

influxdb3 create database my_database

Now, write some data to it:

influxdb3 write --database my_database "sensor_data temp=20"

Next, create and enable the trigger:

influxdb3 create trigger \’
  --trigger-spec "every:10s" \
--plugin-filename "deadman_alert.py" \
  --trigger-arguments table=sensor_data,threshold_minutes=1,slack_webhook=https://hooks.slack.com/services/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 \
  --database my_database \
  sensor_deadman

The deadman check plugin runs every 10 seconds. It checks for data written to the sensor_data table in the my_database within the last minute. If data was written, you’ll see the following output in the InfluxDB logs:

INFO influxdb3_py_api::system_py: processing engine: Data exists in 'sensor_data' in the last 1 minutes.

If no data written in the last minute, you’ll receive the following notification in Slack:

The trigger will continue firing until you disable it with:

influxdb3 disable trigger --database my_database sensor_deadman
Trigger sensor_deadman disabled successfully

Final thoughts and considerations

This deadman check and alert plugin for InfluxDB 3 Core and Enterprise provides a powerful and flexible way to monitor data pipeline durability in real-time. I hope this tutorial helps you start alerting with Python Plugins and enabling triggers in InfluxDB 3 Core and Enterprise with Docker. I encourage you to look at the InfluxData/influxdb3_plugins as we add examples and plugins there. Also, please contribute your own! To learn more about building your own plugin, check out these resources:

Additionally, check out these resources for more information and examples of other alert plugins:

I invite you to contribute any plugin that you create there. Check out our Getting Started Guide for Core and Enterprise, and share your feedback with our development team on Discord in the #influxdb3_core channel, Slack in the #influxdb3_core channel, or our Community Forums.

Alerting with InfluxDB 3 Core and Enterprise

Anais Dotis-Georgiou (InfluxData) — Tue, 11 Mar 2025 07:00:00 +0000

Monitoring is only as good as the alerts that surface critical issues before they spiral out of control. With InfluxDB 3 Core and Enterprise, you can extend alerting capabilities beyond built-in solutions by leveraging custom Python processing plugins. Whether you need real-time notifications when thresholds are exceeded or advanced anomaly detection tailored to your infrastructure, developing custom alerting logic ensures you get the right alerts at the right time. In this post, we’ll dive into how to build and integrate alerting plugins within InfluxDB 3. Specifically, we’ll learn how to incorporate a custom alerting plugin that notifies you via Slack, Discord, or other HTTP endpoints when key metrics exceed thresholds. This post will walk through how to set up and configure the alert plugin, covering its features, setup, and real-world use cases.

Why custom alerts matter in InfluxDB 3

InfluxDB 3 provides powerful time-series storage and query capabilities, but detecting anomalies in real-time requires an efficient alerting system. The alert plugin enables dynamic notifications when a sensor value, CPU usage, or any monitored field crosses a threshold.

Key features include:

Multi-platform notifications – Supports Slack, Discord, and generic HTTP endpoints
Configurable thresholds – Set custom conditions for triggering alerts
Flexible messages – Customize alert content with dynamic variables
Retries with backoff – Ensure delivery with exponential retry mechanisms
Environment variable support – Securely configure webhook URLs

While this plugin focuses on sending alerts for straightforward threshold-based conditions, you can extend its functionality by building a custom processing plugin that performs sophisticated anomaly detection or forecasting. For example, you could develop a plugin that analyzes trends, detects anomalies, and writes flagged events to a separate database. Then, by setting the threshold to 0 and pointing this alert plugin at the anomaly database, you can trigger alerts whenever new anomalies are detected. This modular approach allows you to integrate advanced anomaly detection with real-time notifications, ensuring a more intelligent and proactive monitoring system.

Getting started: prerequisites

This blog post assumes that you have installed and are using InfluxDB 3 Core or InfluxDB 3 Enterprise. Additionally, before setting up alerts, install the required package for HTTP requests:

influxdb3 install package httpx

This tutorial also assumes that you have created two databases with:

influxdb3 create database my_databaseinflxudb3 create database alerts_history

Setting up notifications

To send alerts to Slack, set up a webhook URL. For testing, use the public webhook (not for production):

export SLACK_WEBHOOK_URL= "https://hooks.slack.com/services/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"

This posts alerts to the #notifications-testing channel in the InfluxData Slack community. For production, use your own Slack webhook:

export SLACK_WEBHOOK_URL="[your slack webhook]"

Similarly, to leverage the Discord Integration, set up a a Discord webhook:

export DISCORD_WEBHOOK_URL="[your discord webhook]"

Testing the influxdata/Anaisdg/Alerts/alert.py plugin

Make sure to place the alert.py script in ~/influxdb3/plugins before running the command below.

Once configured, test alerts using sample data. For example, to send a Slack alert when the temperature exceeds 20°C, run:

influxdb3 test wal_plugin \
  --database my_database \
  --lp="sensor_data,sensor=TempSensor1,location=living_room temperature=25 123456789" \
  --input-arguments "name=temp_alert,endpoint_type=slack,threshold=20,field_name=temperature,notification_text=Alert: Temperature ${temperature} exceeds threshold,webhook_url=$SLACK_WEBHOOK_URL" \
  alert.py

Expected behavior: The temperature field exceeds 20, so a notification is sent to Slack with the message:

"Temperature 25 exceeds threshold"

The input-arguments flag defines all of the configuration options for your plugin. Here’s how you can fine-tune this specific plugin:

Parameter	Required?	Description
name	Yes	Unique identifier for the alert instance
endpoint_type	Yes	Choose slack, discord, or http for alerts
field_name	Yes	Field to monitor (e.g., temperature, cpu_usage)
threshold	Yes	Numeric value that triggers an alert
notification_text	Optional	Custom alert message (supports variables)
webhook_url	Optional	Override environment variable webhook
headers	Optional	Base64-encoded headers for HTTP endpoints
alerts_db	Optional	Database to store alert history for tracking and analysis

Sending and tracking alerts

Now that we’ve tested our plugin, we’re ready to set up and enable an alert trigger. For this example, we’ll send alert notifications to Discord and use the alerts_db option:

influxdb3 create trigger \
  --database my_database \
  --plugin-filename alert.py \
  --trigger-spec "table:sensor_data" \
  --trigger-arguments "name=temp_alert_trigger,endpoint_type=discord,threshold=20,field_name=temperature,notification_text=Value is \${temperature},webhook_url=$DISCORD_WEBHOOK_URL,alert_db=alerts_history" \
  temperature_alert
influxdb3 enable trigger --database my_database temperature_alert

Now, we can write test data and check the alerts_history database to confirm our notification sent and that we’re writing alert metrics successfully:

influxdb3 write --database my_database "sensor_data,sensor=TempSensor1,location=living_room temperature=25"
influxdb3 query --database alerts_history "select * from sensor_data"

The output of your query should look like this:

time	alert_message	sensor	location	plugin_name	processed_at	temperature
2025-02-25T00:25:40Z	Value is 25.0	TempSensor1	living_room	temp_alert	2025-02-25T00:25:45Z	25.0

Final thoughts

The influxdata/Anaisdg/alert plugin for InfluxDB 3 provides a powerful and flexible way to monitor key metrics and trigger notifications in real-time. Whether you need Slack, Discord, or custom HTTP alerts, this plugin helps ensure that critical issues never go unnoticed. I hope this tutorial helps you start alerting with Python Plugins and enabling triggers in InfluxDB 3 Core and Enterprise with Docker. I encourage you to look at the InfluxData/influxdb3_plugins as we add examples and plugins there. Also, please contribute your own! To learn more about building your own plugin, check out these resources:

Building Your First Python Plugin for the InfluxDB 3 Processing Engine

Anais Dotis-Georgiou (InfluxData) — Tue, 04 Mar 2025 07:00:00 +0000

**Note: This blog runs the InfluxDB 3 Core CLI not Enterprise.

One of the most compelling features of InfluxDB 3 is its built-in Python Processing Engine, a versatile component that adds powerful, real-time processing capabilities to InfluxDB 3 Core. For those familiar with Kapacitor in InfluxDB 1.x or Flux Tasks in 2.x, the Processing Engine represents a more streamlined, integrated, and scalable approach to acting on data. With the ability to run Python code directly within the database, you no longer need external servers or complex data pipelines to process incoming information.

The Processing Engine can trigger actions as data arrives, on-demand, or on a schedule, making it ideal for real-time transformation, normalization, alerting, downsampling, and edge data replication. In this blog, we’ll build a Python plugin that standardizes IoT data from diverse sources. Standardization is critical because IoT devices often produce data in inconsistent formats—different units, structures, or naming conventions—complicating analysis and decision-making. By normalizing this data at the point of ingestion, you simplify downstream queries, ensure consistency across datasets, and improve the reliability of your analytics.

Requirements

To follow this tutorial, you’ll need:

Docker installed on your machine.
A code editor like Visual Studio Code (VS Code) or another integrated development environment (IDE) of your choice.

Note: Using Docker for this tutorial ensures you can easily spin up a compatible environment without complex setup steps, allowing you to focus on building and testing your Python plugin. We’ll walk through the process of creating the plugin from scratch with Docker, but you can also run this tutorial locally without the Docker commands.

This tutorial also assumes you have some familiarity with Docker fundamentals, such as running containers, managing images, and using Docker Compose. Additionally, it helps to have a basic understanding of InfluxDB, including the line protocol ingest format and the InfluxDB 3 Core CLI. If you’re new to any of these concepts, we recommend reviewing the InfluxDB documentation or Docker’s getting-started guide before proceeding:

Process overview

Inconsistencies in units, field names, and measurement structures are common when dealing with IoT data from various devices. Different sensors might report temperature in Fahrenheit or Celsius, pressure in pascals or kilopascals, and inconsistent naming conventions like humidity_percent vs. humidity. This variability makes querying, analyzing, and correlating data unnecessarily complex, leading to errors in reporting, delayed insights, and increased maintenance overhead.

Standardizing units and field names during data ingestion ensures a consistent and reliable dataset for downstream analysis. Beyond simplifying analytics, consistent naming conventions are essential for maintaining compliance with industry standards, regulatory requirements, and internal data governance policies. With standardized, well-structured data, teams can more confidently generate reports, audit historical records, and integrate with other systems that rely on clear, predictable data formats.

Our plugin will solve this by standardizing units and names as the data is ingested, ensuring a consistent and reliable dataset for downstream analysis.

Here’s a high-level look at the process:

Create a Plugin Directory: Set up a plugin directory in your local InfluxDB 3 environment and give it read/write permissions.
Start InfluxDB 3 with Docker: Pull the InfluxDB 3 Core Docker image and launch a container, mounting the plugins directory.
Write the Python Processing Script: Create a Python script to convert measurements into standardized units and naming conventions.
Create Source and Destination Databases: Use the CLI to create databases for raw and standardized data.
Test and Enable the Plugin: Test the script with sample data, then enable the plugin.
Write Data and Verify the Transformation: Ingest sample IoT data with inconsistent formats and query the destination database to confirm successful standardization.

Building your first plugin

As mentioned in the overview, start by creating a plugin directory at:

mkdir -p ~/influxdb3/plugins

Ensure the directory has the necessary read and write permissions:

chmod 755 ~/influxdb3/plugins

Next, pull the latest InfluxDB 3 Enterprise Docker image:

docker pull quay.io/influxdb/influxdb3-enterprise:latest

Refer to the official documentation for details on running the Enterprise or Core editions, depending on your setup. Now, we’re ready to start the InfluxDB 3 container with the following command:

docker run -it \
  -v ~/influxdb3/data:/var/lib/influxdb3 \
  -v ~/influxdb3/plugins:/plugins \
  -p 8181:8181 \
  --user root \
  quay.io/influxdb/influxdb3-enterprise:latest serve \
  --node-id my_host \
  --object-store file \
  --data-dir /var/lib/influxdb3 \
  --plugin-dir /plugins

Let’s break down this command:

-v ~/influxdb3/data:/var/lib/influxdb3: Mounts the local data directory as the database’s persistent storage location.
-v ~/influxdb3/plugins:/plugins: Mounts the plugins directory where our Python plugin will live, making it accessible to the Processing Engine.
-p 8181:8181: Maps port 8181 from the container to the host, allowing access to the InfluxDB 3 API.
--user root: Ensures the container runs with root privileges, which are required for the Processing Engine to access the plugins.
serve: Starts the InfluxDB 3 server.
--node-id my_host: Assigns a unique node ID, which can be customized based on your environment.
--object-store file: Configures the database to use the local filesystem for object storage.
--data-dir /var/lib/influxdb3: Points to the directory where InfluxDB will persist its data.
--plugin-dir /plugins: Instructs InfluxDB to load any available plugins from the mounted plugins directory.

Note: If you’re running InfluxDB 3 Core or Enterprise locally, you’ll want to serve your InfluxDB 3 instance and set the plugin directory with the following command (learn more about the serve command options with the CLI Operations for InfluxDB 3 Core and Enterprise or the docs):

influxdb3 serve --object-store file --data-dir ~/.influxdb3/data --node-id my_host --plugin-dir ~/influxdb3/plugins

With this setup, the Processing Engine will have access to your Python plugins and be ready to apply transformations to incoming data.

Now, we’re ready to write the Python script to handle our data standardization. We’ll name our script hello_world.py. It will contain a function called process_writes that performs the core transformation logic. This function processes incoming batches of sensor data, standardizing field names, tags, and units to ensure consistency across datasets. It iterates through each table batch, logs key information, and skips tables that match a predefined exclusion rule. For each row, it converts sensor names and locations to lowercase and replaces spaces with underscores to maintain uniform naming conventions. It also enriches the data by adding a timestamp field indicating when the record was processed. Finally, the function writes the transformed data to a new InfluxDB 3 database called unified_sensor_data, ensuring all sensor records share a consistent structure for easier querying, analysis, and compliance with data standards.

import datetime

def process_writes(influxdb3_local, table_batches, args=None):
    # Log the provided arguments
    if args:
        for key, value in args.items():
            influxdb3_local.info(f"{key}: {value}")

   # Process each table batch
    for table_batch in table_batches:
        table_name = table_batch["table_name"]
        influxdb3_local.info(f"Processing table: {table_name}")

      # Skip processing a specific table if needed
        if table_name == "exclude_table":
            continue

      # Analyze each row
        for row in table_batch["rows"]:
            influxdb3_local.info(f"Row: {row}")

            # Standardize sensor names (lowercase, no spaces)
            sensor_name = row.get("sensor", "unknown").lower().replace(" ", "_")
            influxdb3_local.info(f"Standardized sensor name: {sensor_name}")

            # Standardize location and other tags by replacing spaces with underscores
            location = row.get("location", "unknown").lower().replace(" ", "_")

            # Add enriched field (e.g., timestamp)
            line = LineBuilder(table_name)
            line.tag("sensor", sensor_name)
            line.tag("location", location)
            line.float64_field("temperature_c", row.get("temperature", 0))
            line.string_field("processed_at", datetime.datetime.utcnow().isoformat())

            # Write the enriched data to a different database
            influxdb3_local.write_to_db("unified_sensor_data", line)

    influxdb3_local.info("Processing completed")

Create the source and destination databases with the influxdb3 create database command:

docker exec {container id} influxdb3 create database my_databasedocker exec {container id} influxdb3 create database unified_sensor_data

Testing your plugin

You can test your plugin on a target database with the influxdb3 test command:

docker exec {container id} influxdb3 test wal_plugin \
-d my_database \
--lp="sensor_data,location=living\\ room temperature=22.5 123456789" \
/plugins/hello-world.py

The command outputs a JSON object with logging information, database writes, and errors. It shows us that the trigger is parsing data from the sensor_data measurement, standardizing the data, and writing the transformed line protocol to the unified_sensor_data database without errors:

{
  "log_lines": [
    "INFO: Processing table: sensor_data",
    "INFO: Row: {'location': 'living room', 'temperature': 22.5, 'time': 123456789}",
    "INFO: Processing completed"
  ],
  "database_writes": {
    "my_database": [],
    "unified_sensor_data": [
      "sensor_data,sensor=unknown,location=living\\ room temperature_c=22.5,processed_at=\"2025-02-13T21:33:44.117195\""
    ]
  },
  "errors": []
}

Create and enable your trigger

Now that the test passes successfully, let’s create a trigger and enable it to run our Python plugin. The following command runs the influxdb3 create trigger command. The -d option specifies the database where the trigger will be applied. The --plugin-filename="/plugins/hello-world.py" option points to the plugin script that will be executed when the trigger is activated. The --trigger-spec="all_tables" option indicates that the trigger should apply to all tables within the specified database. Finally, hello_world_trigger is the name assigned to the trigger.

docker exec {container id} influxdb3 create trigger \
-d my_database \
--plugin-filename="/plugins/hello-world.py" \
--trigger-spec="all_tables"  \
hello_world_trigger

Now we can enable it with influxdb3 enable trigger command:

docker exec {container id} influxdb3 enable trigger \
--database my_database  \
hello_world_trigger

Verify your trigger and plugin are working

To verify that our trigger is enabled correctly and that our plugin is working as expected, we can write a line to the my_database source database and query the unified_sensor_data database to verify that the standardization is working as expected. Use the influxdb3 write command to accomplish the former:

docker exec {container id} influxdb3 write \
--database my_database \
"sensor_data,sensor=TempSensor1,location=living\\ room temperature=22.5 123456789"

Finally, we verify that our data has been transformed to our standard with the influxdb3 query command:

docker exec {container id} influxdb3 query \
--database unified_sensor_data \
"SELECT * FROM sensor_data"

The output confirms that the spaces in our location tag value were correctly replaced with underscores, a processed_at field was added, our sensor tag values were converted to lowercase, and our temperature field key now contains the temperature unit.

Final thoughts

I hope this tutorial helps you start creating Python Plugins and enabling triggers in InfluxDB 3 Core and Enterprise with Docker. I encourage you to look at the InfluxData/influxdb3_plugins as we start to add examples and plugins there. I also invite you to contribute any plugin that you create there! Get started by downloading Core or Enterprise. Share your feedback with our development team on Discord in the #influxdb3_core channel, Slack in the #influxdb3_core channel, or our Community Forums.

CLI Operations for InfluxDB 3 Core and Enterprise

Anais Dotis-Georgiou (InfluxData) — Tue, 04 Feb 2025 07:00:00 +0000

Note: This blog was written for InfluxDB 3 Core and Enterprise Alphas. Please check the documentation as there might be slight changes in its current version, Beta.

This blog covers the nitty-gritty of essential command-line tools and workflows to effectively manage and interact with your InfluxDB 3 Core and Enterprise instances. Whether you’re starting or stopping the server with configurations like memory, file, or object store, this guide will walk you through the process. We’ll also look at creating and writing data into databases using authentication tokens, exploring direct line protocol input versus file-based approaches for tasks like testing. You’ll learn how to query data efficiently using SQL and InfluxQL, set up performance-boosting features like the last value cache and meta-cache, and tidy up by deleting databases and tables when necessary. Let’s get started with mastering the InfluxDB 3 Core and Enterprise CLI!

Requirements

This blog post assumes you meet the following requirements: InfluxDB 3 Core or InfluxDB Enterprise. If you’re just getting started, I recommend beginning with InfluxDB 3 Core. As the open-source version of InfluxDB 3 Enterprise, it’s a solid foundation for learning and an excellent choice as an edge data collector. Alternatively, you can use the free trial of Enterprise or upgrade from Core to Enterprise—the CLI commands in Core are identical to those in Enterprise. To upgrade from Core to Enterprise, users need to download and install InfluxDB 3 Enterprise in free trial mode or with a valid license in the same environment where they installed Core, ensuring it points to the same object store.

Image of the CLI after you install it with a single cURL command. There are options to install as a simple download or with docker.

Naming Conventions Note: If you’re a previous InfluxDB user, some changes in nomenclature might be confusing; please note that the following are synonymous:

Bucket ↔ Database

Measurement ↔ Table

Field, Tags ↔ Columns

Start by getting `–-help`

Once you have InfluxDB v3 OSS installed, the influxdb3 command-line interface (CLI) becomes your go-to tool for interacting with the database. This versatile CLI allows you to manage your InfluxDB instance, create resources like databases and tokens, and perform essential operations such as querying and writing data. Running influxdb3 –-help provides a complete list of commands and options, with examples to get you started. Whether you’re setting up a server with influxdb3 serve, writing data with influxdb3 write, or running queries using influxdb3 query, this tool makes it simple to work with your time series data.

Here’s a detailed explanation of each command listed in the influxdb3 –-help output to help you understand their purpose and usage:

serve: Starts the InfluxDB 3 Core or Enterprise server, the central process that makes the database operational.
query: Executes queries against a running InfluxDB 3 Core or Enterprise server, allowing you to retrieve and analyze stored data.
write: Writes time series data to an InfluxDB 3 Core or Enterprise server. This command allows you to add data manually or programmatically from other tools.
create: Helps you create new resources in your InfluxDB instance, such as databases or tokens.
help: Displays help information for the influxdb3 CLI or a specific command. For example, influxdb3 query –-help provides detailed usage instructions for the query command.

Starting the server with different configurations

When you first install InfluxDB 3, the CLI walks you through server configuration options. After that initial setup, use the serve command to start the instance:

influxdb3 serve --object-store file --data-dir ~/.influxdb3 --node-id node0

InfluxDB 3 runs on port 8181 by default, but you can add --http-bind='0.0.0.0:8181 to specify a different port. You must also specify the --node-id identifier that determines the server’s storage path within the configured storage location. It must be unique for any nodes sharing the same object store configuration, such as the same bucket. Parquet files serve as the durable, persisted data format for InfluxDB 3 Core and Enterprise, enabling object storage to become the preferred solution for long-term data retention. This approach significantly lowers storage costs while maintaining excellent performance. The --object-store option allows users to specify where they want to write those Parquet files. You can choose to write these to memory, local file system, Amazon S3, Azure Blob Storage, Google Cloud Storage, or any cloud storage. Cloud storage options include:

memory (default): Effectively no object persistence.
memorythrottled: Like memory but with latency and throughput that somewhat resembles a cloud object store. Useful for testing and benchmarking.
file: Stores objects in the local filesystem. Must also set --data-dir.
s3: Amazon S3. Must also set --bucket, --aws-access-key-id, --aws-secret-access-key, and possibly --aws-default-region.
google: Google Cloud Storage. Must also set --bucket and --google-service-account.
azure: Microsoft Azure blob storage. Must also set --bucket, --azure-storage-account, and --azure-storage-access-key.

For example, I could use the serve command to start the instance using memory as the “object store” with:

influxdb3 serve --object-store memory --node-id node0

In-memory storage doesn’t provide a permanent object store, and data clears on restart, but it is the fastest way to get running with InfluxDB 3.

Create a database and write to it with a plain token

Now we’re ready to create a database and write data with one command:

influxdb3 write --database [your database name] --file [path to your line protocol data]`

For example, if we wanted to write the following line protocol data we might use:

influxdb3 write --database airSensors --file /Desktop/airsensors.lp

Where airsensors.lp is a file that contains line protocol data, the ingest format for InfluxDB. You can find a selection of line protocol real-time datasets here. For example, you could download some Air Sensor data which looks like:

airSensors,sensor_id=TLM0100 temperature=71.24021491535241,humidity=35.0752743309533,co=0.5098629816173851 1732669098000000000
airSensors,sensor_id=TLM0101 temperature=71.84309523593232,humidity=34.934199682459,co=0.5034259382294339 1732669098000000000
airSensors,sensor_id=TLM0102 temperature=71.95391915782443,humidity=34.92433120092046,co=0.5175197455105179 1732669098000000000

This dataset contains temperature, carbon monoxide, and humidity data from eight different sensors.

Or, we could write those few lines of data directly, instead of pointing to a file with:

influxdb3 write --database [your database name]
'airSensors,sensor_id=TLM0100 temperature=71.24021491535241,humidity=35.0752743309533,co=0.5098629816173851 1732669098000000000
airSensors,sensor_id=TLM0101 temperature=71.84309523593232,humidity=34.934199682459,co=0.5034259382294339 1732669098000000000
airSensors,sensor_id=TLM0102 temperature=71.95391915782443,humidity=34.92433120092046,co=0.5175197455105179 1732669098000000000'

After writing the data with the influxdb3 write command, you should see the following confirmation:

success

In this example, and by default, we are serving InfluxDB 3 Core with the plain Token. In the next section, we’ll learn about how to write data to InfluxDB 3 Core or Enterprise with the Hashed Token and the difference between the two.

You create a database on write with the influxdb3 write command. However, you can also elect to create a database with the influxdb3 create database [your database name] command.

Writing into the database with authentication tokens

You’ll need to create a token if you want to use the HTTP API or SDKs to access your data. To create a token with the InfluxDB CLI, you can use the following command: influxdb3 create token

You should see the following output:

Token: apiv3_xxx
Hashed Token: zzz
Start the server with `influxdb3 serve --bearer-token zzz
HTTP requests require the following header: "Authorization: Bearer apiv3_xxx"
This will grant you access to every HTTP endpoint or deny it otherwise

Now, you can elect to serve influxdb3 with the bearer-token:

influxdb3 serve --object-store memory --bearer-token zzz --node-id node0

You can also elect to serve the instance and store objects in the local filesystem:

influxdb3 serve --object-store file --data-dir ~/.influxdb3 --bearer-token zzz --node-id node0

The hashed token is a cryptographic representation of the plain token. By passing the hashed token to the server, you avoid exposing the plain token in the command line, logs, or configuration files. So when a client sends a plain bearer token in an HTTP request, the server hashes the received token and compares the hashed result to the hashed token you provided at startup. This ensures that the server can validate the plain token securely without needing to store or process it directly. It’s best practice to serve InfluxDB 3 Core and Enterprise with the Hashed Token.

Now that we’re serving with the Hashed Token, we can use the same CLI commands above to write data to the database. Alternatively, if we wanted to write with cURL we do the following:

curl \
"http://127.0.0.1:8181/api/v2/write?bucket=[your database name]&precision=s" \
--header "Authorization: Bearer zzz" \
--data-binary 'home,room=kitchen temp=72 1732669098'

Querying data with InfluxQL and SQL

Now we’re ready to query our InfluxDB 3 Core instance with SQL:

influxdb3 query --database=[your database name] "SELECT * FROM airSensors LIMIT 10"

You can also elect to set the language to InfluxQL instead of SQL if you’re an existing InfluxDB user and more familiar with that language. Simply specify the language with: --language=influxql:

influxdb3 query --database=[your database name] --language=influxql "SHOW MEASUREMENTS"

Output from querying InfluxQL with the InfluxDB 3 CLI.

Setting up last value cache and distinct value cache

InfluxDB 3 Core supports a last-n values cache (LVC), which stores the last N values in a series or column hierarchy in memory, and a distinct value cache, which retains unique values for a single column or a hierarchy of columns in RAM. The last value cache allows InfluxDB 3 to answer last-n values queries in under 10 milliseconds, while the distinct value cache is great for fast metadata lookups.

In future blog posts, we’ll dive into benchmarking the performance of these caches, but for this blog, let’s focus on creating and using them. The process for creating both caches is almost identical, so let’s demonstrate the process by creating a last-value cache with:

influxdb3 create last_cache --database [your database name] --table [your database table] [CACHE_NAME]

The [CACHE_NAME] is optional, and the command automatically generates a name if not provided. So, for example, if we wanted to create a last cache for the airSensors table/measurement.

influxdb3 create last_cache --database [your database name] --table airSensors airSensorsLVC

You should see the following output:

new cache created: {
  "table": "airSensors",
  "name": "airSensorsLVC",
  "key_columns": [
    0
  ],
  "value_columns": "all_non_key_columns",
  "count": 1,
  "ttl": 14400
}

Now we can return last 10 values that utilize the cache with:

influxdb3 query --database=[your database name] "SELECT * FROM last_cache(airSensors, airSensorsLVC) LIMIT 10"

The last-value cache is helpful because it enables fast, efficient access to the most recent data for specific combinations of key column values. This helps users to ensure they have up-to-date data for alerts or decision-making. It’s also extremely useful when working with an event-based time-series application where you aren’t writing subsets of data regularly. In this instance, you want to avoid full scans of historical data for queries focused on the latest values.

Creating a Last Value Cache for foo

To better understand that output, we need to dive into the additional create last_cache options that we didn’t utilize during our LVC creation. The create last_cache command has the following options, as per the documentation:

--key-columns: A comma-separated list of columns to use as keys in the cache. For example: foo,bar,baz. This provides the top level of the cache.
--value-columns: A comma-separated list of columns to store as values in the cache. For example: foo,bar,baz. At the leaf (or terminal) nodes of the hierarchy, a buffer is maintained to store the values. The buffer size is determined by the --count parameter.
--count: The number of entries per unique key column combination to store in the cache. The maximum number here can be 10.
--ttl: The cache entries’ time-to-live (TTL) in Humantime form–for example: 10s, 1min 30sec, 3 hours. If any entry in the buffer has been there for longer than the TTL, it will be removed, regardless of the buffer size and how many entries it contains.

When data is written, the values are added to the buffer corresponding to the matching combination of key column values. For example, imagine we create a cache with: influxdb3 create last_cache --database [your database name] --table [your database table] --key-columns t1,t2 --value-columns f1 --count 5

Consider the following line protocol data, where 1 denotes data written at the first timestamp:

foo,t1=A,t2=A f1=1 1 
foo,t1=A,t2=B f1=2 1 
foo,t1=B,t2=C f1=3 1 
foo,t1=B,t2=D f1=4 1

Data would be added to the buffer in the cache in the following way: A hierarchical cache structure for organizing data with Last Value Cache based on key columns and storing values in buffers for InfluxDB 3.

Now imagine we write another line of data: foo,t1=A,t2=A f1=2 2. Now we see it added to the following buffer:

Values are only buffered if their time is newer than the most recent value of their respective buffer.

The distinct value cache operates similarly to the last value cache with the following option differences, as per the documentation:

--columns (required): A comma-separated list of columns to cache distinct values. For example: col1,col2,col3.
--max-cardinality: Maximum number of distinct value combinations to hold in the cache.
--max-age: Maximum age of an entry in the cache entered as a human-readable duration. For example: 30d, 24h.

As mentioned above, the distinct value cache is optimal for fast metadata lookups. They help guide queries toward the correct subset of data by traversing the cache structure, and looking for the leaf nodes that match the keys in the query. For example, in an IoT scenario, this could help developers quickly check if a particular sensor (perhaps one of thousands of the same sensors across multiple factories) is reporting data and operating as expected.

Deletes

We can also use the influxdb3 delete command to delete databases, distinct value cache, file index for a database, last value cache, plugins, tables in a database, and triggers. We’ll learn more about plugins and triggers in the following blog post. But let’s delete the database we created with:

influxdb3 delete database [your database name]

Stopping the server

Finally, if you want to stop running the influxdb3 server, you can kill the process first by returning the PID with:

pgrep influxdb3

And then using:

kill [PID]

Final thoughts

I hope this blog post helps you get started with InfluxDB 3 Core or Enterprise. In an upcoming blog post, we’ll learn how to create Python Plugins for the Processing Engine for InfluxDB 3 Enterprise, a feature that is also controlled through the CLI. As always, get started with InfluxDB 3 Cloud here and Core and Enterprise here. If you need help, please contact us on our community site or Slack channel. If you are also working on a data processing project with InfluxDB, I’d love to hear from you!

Get Started with the TIG Stack and InfluxDB Core

Anais Dotis-Georgiou (InfluxData) — Fri, 24 Jan 2025 07:00:00 +0000

Note: This blog was written for InfluxDB 3 Core and Enterprise Alphas. Please check the documentation as there might be slight changes in its current version, Beta.

Time series data is everywhere—from IoT sensors and server metrics to financial transactions and user behavior. To collect, store, and analyze this data efficiently, you need tools purpose-built for the job. That’s where the TIG Stack comes in: Telegraf for data collection, InfluxDB for storage and analytics, and Grafana for visualization. Together, these tools offer a powerful solution for real-time analytics, observability, and monitoring.

At the heart of the TIG Stack is InfluxDB, a database platform specifically designed to collect, process, transform, and store time series data. InfluxDB is widely used for applications requiring real-time monitoring, lightning-fast queries, and interactive dashboards—whether you’re tracking IoT sensor data, monitoring server performance, analyzing financial markets, or ensuring network reliability.

The latest version, InfluxDB Core, introduces significant improvements in performance, usability, and cost-efficiency. These enhancements make it easier than ever to handle high-volume time series data and deliver real-time insights.

In this blog post, you’ll learn how to get started with the TIG Stack. We’ll guide you through:

Writing data to InfluxDB 3 Core with Telegraf
Visualizing your data with Grafana

Requirements

You’ll need to meet the following requirements to run this tutorial yourself. Please install the following software (or use Grafana Cloud):

Telegraf
InfluxDB Core
Grafana: for example, if you’re on MacOS using homebrew, you can use brew install grafana and brew services start grafana. Grafana will now be available on http://localhost:3000/.

Get InfluxDB up and running

After installing InfluxDB 3 Core, run the following command to start a server:

influxdb3 serve --host-id=local01 --object-store file --data-dir ~/.influxdb3

Next generate a token with:

influxdb3 create token

You should see the following output:

Token: apiv3_xxx
Hashed Token: zzz
Start the server with `influxdb3 serve --bearer-token zzz`
HTTP requests require the following header: "Authorization: Bearer apiv3_xxx"
This will grant you access to every HTTP endpoint or deny it otherwise.

Now, we can use the plain Token with Telegraf. The Hashed Token is a cryptographic representation of the plain token. By passing the hashed token to the server, you avoid exposing the plain token in the command line, logs, or configuration files. So when a client sends a plain bearer token in an HTTP request, the server hashes the received token and compares the hashed result to the hashed token you provided at startup. This ensures that the server can validate the plain token securely storing or processing it directly.

Writing data to InfluxDB v3 OSS with Telegraf

First, you’ll need to install Telegraf on your machine. Review the requirements, download, and installation guide to get started. Once Telegraf is installed on your machine, we can create a configuration to write data to our InfluxDB instance. However, before diving into the configuration, let’s briefly discuss what Telegraf is and why its configuration is essential.

Telegraf is a lightweight, open-source server agent for collecting, processing, and sending metrics and events from various sources to a data store like InfluxDB. It works by using plugins that define what data to gather (inputs), how to process it (processors), and where to send it (outputs).

The Telegraf configuration file plays a central role in this process. It specifies which plugins to use, sets parameters like authentication credentials, and defines the data flow. Without a proper configuration file, Telegraf wouldn’t know:

Where to collect the data from (e.g., a database, MQTT, system metrics)
How to process or transform the data (optional)
Where to send the data (in this case, to InfluxDB v3 OSS).

With Telegraf installed, you can use the following configuration to write data to your InfluxDB instance. This example sets up Telegraf to send system CPU metrics to InfluxDB v3 using the CPU Input Plugin and the InfluxDB v2 Output Plugin (the HTTP API endpoints are the same for v2 as they are for v3). Here’s what our configuration would look like:

# Global configuration
[agent]
  interval = "10s"           # Collection interval
  flush_interval = "10s"     # Data flush interval
# Input Plugin: CPU Metrics
[[inputs.cpu]]
  percpu = true              # Collect per-CPU metrics
  totalcpu = true            # Collect total CPU metrics
  collect_cpu_time = false   # Do not collect CPU time metrics
  report_active = true       # Report active CPU percentage
# Output Plugin: InfluxDB v2
[[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:8181"]
  token = "your plain Token apiv3_xxx"
  organization = ""
  bucket = "cpu"

Note: You don’t need to provide an Organization ID with InfluxDB Core

Now, we can run Telegraf with the following:

telegraf --config pwd/telegraf.conf --debug

The output helps us verify that we are successfully writing data with the following Telegraf logs:

2025-01-09T23:34:02Z I! Loading config: ./telegraf.conf
2025-01-09T23:34:02Z I! Starting Telegraf 1.26.2
2025-01-09T23:34:02Z I! Available plugins: 235 inputs, 9 aggregators, 27 processors, 22 parsers, 57 outputs, 2 secret-stores
2025-01-09T23:34:02Z I! Loaded inputs: cpu
2025-01-09T23:34:02Z I! Loaded aggregators: 
2025-01-09T23:34:02Z I! Loaded processors: 
2025-01-09T23:34:02Z I! Loaded secretstores: 
2025-01-09T23:34:02Z I! Loaded outputs: influxdb_v2
2025-01-09T23:34:02Z I! Tags enabled: host=MacBook-Pro-4.local
2025-01-09T23:34:02Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"MacBook-Pro-4.local", Flush Interval:10s
2025-01-09T23:34:02Z D! [agent] Initializing plugins
2025-01-09T23:34:02Z D! [agent] Connecting outputs
2025-01-09T23:34:02Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2025-01-09T23:34:02Z D! [agent] Successfully connected to outputs.influxdb_v2
2025-01-09T23:34:02Z D! [agent] Starting service inputs
2025-01-09T23:34:12Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2025-01-09T23:34:23Z D! [outputs.influxdb_v2] Wrote batch of 13 metrics in 792.507791ms

Verifying a successful write with the InfluxDB CLI

If you want to, you can now verify that we are successfully writing CPU metrics to InfluxDB v3 OSS with the following CLI command:

influxdb3 query --database=cpu "SELECT * FROM cpu LIMIT 10"

Configuring the Grafana source

To configure InfluxDB Core as a new source in Grafana visit http://localhost:3000/

and navigate to connections > datasources > new > search and select for InfluxDB. Select SQL as the language type. Then provide the following credentials in the configuration:

URL: http://localhost:8181
Database: cpu
Insecure Connection: toggle on

Hit Save&Test to verify that you can connect to InfluxDB Core. Now, you can make visualizations and add them to your dashboards as you usually would in Grafana by navigating to Dashboards > +Create Dashboard > +Add Visualization > Select Datasource > influxdb. And using the Builder to generate a SQL query for you. For example, the Builder generated the following code and visualization:

SELECT "cpu", "usage_user", "time" FROM "cpu" WHERE "time" >= $__timeFrom AND "time" <= $__timeTo AND "cpu" = 'cpu0'

Final thoughts

I hope this blog post helps you get started using InfluxDB Core, Telegraf, and Grafana. If you have any feedback about InfluxDB Core, we’d love to hear it in the #influxdb3_core channel on Discord. Your experience and opinion are important to us during the Alpha release of InfluxDB Core. Get started with InfluxDB v3 Cloud here. If you need help, please get in touch with us us on our community site or Slack channel. If you are also working on a data processing project with InfluxDB, I’d love to hear from you!

InfluxData Blog - Anais Dotis-Georgiou

How to Use the Power BI Desktop InfluxDB 3 ODBC Connector

Setup walk through

Install Power BI Desktop and the Arrow Flight SQL ODBC Driver

Setting Up InfluxDB

Writing data to InfluxDB 3 Core

Enabling the Power BI Desktop InfluxDB 3 Connector

Connector Extensibility

Connect to InfluxDB 3

Visualizing data from InfluxDB 3 in Power BI Desktop

Final thoughts

Edge Data Replication: Contributions and Status Updates for InfluxDB 3

EDR in InfluxDB v2

Introducing the Python Processing Engine for InfluxDB 3

A DevRel-contributed data replicator plugin

Requirements

How to use the plugin

What’s next: official EDR and Downsampling Plugin status

Final thoughts

Forecasting with InfluxDB 3 and HuggingFace

Requirements

Quick start

1. Clone the repository and set up the environment.

2. Start InfluxDB 3 Core with Docker.

3. Create the database and auth token for InfluxDB 3 Core.

4. Set up environment variables. Create a .env file in the project root.

5. Run the app.

App walkthrough

Code overview

Limitations and Python Processing Engine

Looking ahead: from static scripts to plugins

Conclusion

Build a Time Series Forecasting Pipeline in InfluxDB 3 Without Writing Code

From coding to AI validation

Requirements and setup

Writing data, making a forecast, and visualizing our data

Final thoughts and considerations

Deadman Alerts with the Python Processing Engine

Requirements and setup

Creating InfluxDB 3 resources and generating a deadman alert

Final thoughts and considerations

Alerting with InfluxDB 3 Core and Enterprise

Why custom alerts matter in InfluxDB 3

Getting started: prerequisites

Setting up notifications

Testing the influxdata/Anaisdg/Alerts/alert.py plugin

Sending and tracking alerts

Final thoughts

Building Your First Python Plugin for the InfluxDB 3 Processing Engine

Requirements

Process overview

Building your first plugin

Testing your plugin

Create and enable your trigger

Verify your trigger and plugin are working

Final thoughts

CLI Operations for InfluxDB 3 Core and Enterprise

Requirements

Start by getting –-help

Starting the server with different configurations

Create a database and write to it with a plain token

Writing into the database with authentication tokens

Querying data with InfluxQL and SQL

Setting up last value cache and distinct value cache

Creating a Last Value Cache for foo

Deletes

Stopping the server

Final thoughts

Get Started with the TIG Stack and InfluxDB Core

Get InfluxDB up and running

Writing data to InfluxDB v3 OSS with Telegraf

Verifying a successful write with the InfluxDB CLI

Configuring the Grafana source

Final thoughts

Start by getting `–-help`