InfluxData Blog - Jay Clifford

Building the Next Generation of Smart PLC's With the ctrlX CORE and InfluxDB

Jay Clifford (InfluxData) — Wed, 28 Feb 2024 08:00:00 +0000

This article was originally published on IIoT World and is reprinted here with permission.

Program Logic Controllers (PLCs) have played an integral role in industrial automation since their initial creation during the 1960s. Since then, we’ve seen incremental improvements to their form and function throughout the years. The problem? While PLCs are exceptional at managing and controlling industrial processes, the evolving demands of modern manufacturing require these devices to do more than control machinery. Manufacturers now seek to leverage the computational power of PLCs for more complex tasks, particularly in monitoring the health and status of machines. The need for greater efficiency, predictive maintenance, and real-time data analysis drives this shift. So, does this mean we need to revisit how organizations design and use PLCs?

In this blog, we’ll explore a next-generation PLC called the ctrlX core by Rexroth and how we can utilize this platform to integrate best-in-class open source projects such as InfluxDB and Grafana to create a modern, interconnected anomaly detection stack.

What exactly is a PLC?

Before we get ahead of ourselves, let’s break down a standard PLC’s architecture and role. The diagram above breaks down a PLC into its core components:

Input Module: receives signals from input devices like sensors, switches, and buttons and converts these signals into a format that a PLC CPU can process.
Output Module: sends control signals from the PLC to output devices like motors, valves, lights, and relays.
Power Supply: converts the main AC (alternating current) to low-voltage DC (direct current) to power the PLC components.
CPU (Central Processing Unit): processes program data and instructions, executing control operations based on programmed logic. It handles logic operations, arithmetic operations, sequencing, timing, counting, and data manipulation.
Memory: used to store the control program loaded by the user. This typically includes both volatile and non-volatile memory types. Volatile memory (like RAM) temporarily stores data during operation. Non-volatile memory (like ROM, EEPROM) stores the PLC program and retains it even when power is off.
Programming Device: used to input the desired program into the PLC’s memory.

I believe breaking down the architecture of a PLC helps to remove the ambiguity behind its design. In its simplest form, it’s a rugged computer that performs and executes a program repeatedly.

To cement this idea, let’s take a look at a scenario to illustrate a PLC function: In the scenario above, the PLC automates the movements of a conveyor:

Start/Stop Control:
- Pressing the Start button sends a signal from the PLC to the motor control relay, activating the conveyor belt.
- Pressing the Stop button or activating the Emergency Stop immediately sends a signal to deactivate the motor control relay, stopping the conveyor.
Object Detection and Handling:
- Photoelectric sensors detect objects on the conveyor.
- The PLC can count objects, control their spacing, and halt the belt for specific operations like inspection or packaging.
Speed Control and Monitoring:
- The speed sensor provides feedback to the PLC.
- The PLC adjusts the motor speed to maintain a consistent conveyor speed, which can be crucial for synchronized operations with other machinery.
- The PLC can also respond to manual speed adjustments made by the user.

With this basic understanding of how PLCs work, how are next-generation PLCs like the ctrlX core iterating upon this design—and where does InfluxDB come into all of this?

Next generation of PLC

As we now know, most standard PLCs have an embedded processor that carries out the devised program. Usually, this means PLCs ship with their own proprietary OS (e.g., Siemens’ SIMATIC Step 7 for their S7 PLCs and Rockwell’s RSLogix for Allen-Bradley PLC) to run on the embedded processor. This is great for providing a fault-tolerant and performant execution engine but leaves very little room for extensibility.

The ctrlX core, on the other hand, provides a flexible hardware specification, which allows manufacturers to control power levels for their PLC according to the needs of certain tasks. The PLC also deploys a Linux operating system called Ubuntu IoT Core, providing the flexibility to extend the core functionality and services users can install on the PLC. Let’s revisit the original PLC architecture and modify it to reflect the design of the ctrlX core: We’ve introduced two new aspects to our PLC design:

Data plane: this replaces the standard operation data in memory. It still provides a way for our PLC program to access input and output data, as well as a gateway to allow third-party applications to subscribe and publish data.
Third-Party Apps: because our PLC now uses a Linux distribution, we can leverage built-in Linux services, such as containerization. This allows developers to build and deploy dockerized applications onto the ctrlX core to interface with the data plane.

Where do InfluxDB and Grafana play into all of this? How can we leverage these new services and the extra compute of the ctrlX core?

Open source data historian

InfluxDB, an open source time series database, is specifically engineered from the ground up to handle vast amounts of time series data. In recent years, it has become the foundation for many next-generation data historians that are replacing standard relational databases. We can attribute this shift to several of InfluxDB’s distinctive qualities:

High-Throughput Data Ingestion: InfluxDB efficiently manages data intake from multiple parallel sources, making it ideal for environments with extensive data generation.
Schema-on-Write Flexibility: it offers a “schema on write” approach, allowing for the flexible and rapid storage of thousands of machine tags without requiring pre-designed schema. This makes it adaptable to varying data structures.
Low System Requirements: with minimal installation and operational demands, InfluxDB is an effective solution for local data storage, conserving hardware resources for other critical PLC processes.
Advanced Time-Based Queries: performs efficient time-based aggregations (like average, sum, min, max) across various time granularities (seconds, minutes, hours, days, etc.).
High-Resolution Timestamps: highly granular timestamps (up to nanosecond precision) make it suitable for precise machinery monitoring at a detailed level.

Users can install InfluxDB directly onto the ctrlX core via its third-party app store. This automatically installs the latest open source instance. So, what benefits does this provide to my site engineers? Let’s revisit our conveyor example.

Machine Anomaly Detection

In this example, we deployed the ctrlX core to control our conveyor. Like before, the PLC program includes functions such as:

Stop/Start control
Object detection and handling
Speed control and monitoring

On top of this, we’ll also deploy InfluxDB to act as a local edge data historian. InfluxDB will connect to the ctrlX core data plane and ingest the following health metrics from the conveyor servos:

Vibration
Speed
Temperature

In the illustration below, we can see that during standard operation, the system stores these sensor metrics in InfluxDB. Grafana connects directly to InfluxDB to display these raw metric readings directly on the HMI to the user: This is standard practice within the industry. However, using the InfluxDB OSS (Open Source Software) built-in task engine, we can perform automated analysis on both historical and current telemetry data to look for trends. Within this InfluxDB task, we compare historic readings to current metrics to learn that there is an upward climb in both temperature and vibration. We use this data to send alerts through Grafana to indicate to the onsite engineer that our current telemetry readings indicate one of the motors is under increasing stress and may require servicing.

This scenario provides a basic way for InfluxDB and Grafana to introduce predictive maintenance and anomaly detection directly onto the PLC. The benefit of this approach is lower latency when identifying and alerting on potential faults because these processes occur closer to the data source. We also use the hardware capabilities of the PLC rather than introducing another edge device to carry out this analysis.

Edge Data Replication

Lastly, as an added extra, InfluxDB OSS has a built-in feature that can write data from one InfluxDB instance to another remote instance of InfluxDB. This can be either InfluxDB Serverless, Dedicated, Clustered (on-prem), or even another remote instance of InfluxDB OSS. This feature allows you to aggregate data from different production lines into a single aggregated repository for further analysis. Some assets include:

Anomaly detection between production lines of the same type—discover if one production line begins to deviate from the others.
Greater organizational visibility. For instance, if one of your data scientists needs real machine data to train an ML model, this feature provides a non-invasive way to offload the necessary data.
Inherent security. InfluxDB acts as the data writer and receiver between OT and IT-based networks—data movement only occurs one way.

Conclusion

The evolution of Industry 4.0 is forcing many manufacturers to think creatively about how they can extract, store, and analyze data from PLCs on the shop floor. For many older systems, this practice won’t change (in these cases, combining PTC Kepware and InfluxDB will do the trick with most PLCs). However, I hope this blog serves you well when you consider bringing new machinery into your production line. A next-gen PLC like the ctrlX core is built from the ground up to accommodate the flexibility and adaptability of performing both as a robust machine controller and a powerful edge computing device. Pair this with open source projects such as InfluxDB and Grafana, and you’ll be able to develop a cost-effective solution for automating the health analysis of your machines.

To see how easy it is to configure InfluxDB, I highly recommend checking out this interactive demo. I also suggest this walkthrough by Rexroth on how to install and configure InfluxDB onto the ctrlX core. If you have any questions about this blog or InfluxDB in general, I highly recommend joining the InfluxDB community.

InfluxDB 3 Python Client Update: Adding Polars Support

Jay Clifford (InfluxData) — Fri, 26 Jan 2024 08:00:00 +0000

It’s been a while since we posted about the InfluxDB 3 Python Client. Let’s take a look at what’s new!

Polars Dataframe ingest

2023 saw the popularity of a new kid on the block within the data analytics space, Polars. The Polars Data Frame library is an alternative data frame package to the original OG Pandas. Although both cater to the same use cases, each is fundamentally built on different technologies.

One of the most frequent community requests we received was to provide greater compatibility with Polars. Since Polars is built on Apache Arrow, we extended the mode function to include polars. Simply query the data and modify the mode to polars as below:

import polars as pl
from influxdb_client_3 import InfluxDBClient3

with InfluxDBClient3(
    token="",
    host="eu-central-1-1.aws.cloud2.influxdata.com",
    org="6a841c0c08328fb1") as client:

        sql = 'SELECT * FROM caught LIMIT 100000'
        df = client.query(database="pokemon-codex", query=sql, language='sql', mode='polars')
        print(df, flush=True)

We call the Polars function from_arrow() within the underlying client code. This automatically converts our Arrow table into a Polars Dataframe. Note: you must install the Polars Dataframe library to use this mode.

Ingestion was a slightly different story. Like V1 and V2, InfluxDB 3 expects line protocol (LP) as its primary ingestion method. This means we build out converters to LP in our client libraries. Polars provides an extremely efficient UDF feature, which made creating this new converter straightforward to implement. We built the new Polars data frame converter into the preexisting data frame converter class. Here is an example:

import polars as pl
from influxdb_client_3 import InfluxDBClient3,InfluxDBError,WriteOptions,write_client_options

class BatchingCallback(object):

    def success(self, conf, data: str):
        print(f"Written batch: {conf}, data: {data}")

    def error(self, conf, data: str, exception: InfluxDBError):
        print(f"Cannot write batch: {conf}, data: {data} due: {exception}")

    def retry(self, conf, data: str, exception: InfluxDBError):
        print(f"Retryable error occurs for batch: {conf}, data: {data} retry: {exception}")

callback = BatchingCallback()

write_options = WriteOptions(batch_size=10000,
                                        flush_interval=10_000,
                                        jitter_interval=2_000,
                                        retry_interval=5_000,
                                        max_retries=10,
                                        max_retry_delay=15_000,
                                        exponential_base=2, max_close_wait=900_000)

wco = write_client_options(success_callback=callback.success,
                          error_callback=callback.error,
                          retry_callback=callback.retry,
                          WriteOptions=write_options 
                        )

client = InfluxDBClient3(
    token="token",
    host="eu-central-1-1.aws.cloud2.influxdata.com",
    org="6a841c0c08328fb1", enable_gzip=True, write_client_options=wco)

pl_df =pl.read_parquet('pokemon_100_000.parquet')

client.write(database="pokemon-codex", 
             record=pl_df, data_frame_measurement_name='caught', 
             data_frame_tag_columns=['trainer', 'id', 'num'], 
             data_frame_timestamp_column='timestamp')

client.close()

In this case, you can see it includes the same parameters we would use when writing a Pandas data frame. We distinguish the data frame type and call the right converter within the write API.

Top Tip: Make sure to include the data_frame_timestamp_column= and specify your timestamp column. Polars does not provide an index method like Pandas, so we cannot automatically distinguish which is the correct column.

Custom Arrow Flight headers

Another requested feature was the inclusion of custom Arrow Flight Call Options for queries. This allows users familiar with Arrow Flight to use the underlying configuration parameters. A simple example could be increasing the timeout for a particular query:

df = client.query(database="pokemon-codex", query=sql, language='sql', mode='polars', timeout=5)

Bugs and miscellaneous

Lastly, here is a minor change history list:

Version	Change
0.3.4 / 0.3.3	Merged V2 Write API into V3 and removed the V2 client library as a dependency.
0.3.4 / 0.3.3	Added custom port declaration for clustered users
0.3.2	Fixed Pandas as an optional dependency issue
0.3.1	Added flight errors readme
0.3.1	Added community and cookbook example
0.3.0	Added custom certificates parameter. (To fix Windows-based gRPC SSL issue)

What’s next?

We hope you find the new features added to the InfluxDB 3.0 Python Client library useful. If you have any feature requests or bugs to report, please do not hesitate to open an issue via the client repo. We are always looking for community contributors for our 3.0 client libraries. You can always discuss your contribution with us on Slack.

Grafana Unleashes Official InfluxDB V3 Data Source: A Quick-start Guide to Configuration and Usage

Jay Clifford (InfluxData) — Mon, 22 Jan 2024 08:00:00 +0000

Yes, the title says it all: Grafana released the official V3 plugin for InfluxDB Data Source!

Before delving into the tutorial, we’d like to thank Ismail Simsek, a Tech Lead at Grafana. Ismail was pivotal in adding the V3 SQL plugin to the InfluxDB data source and making significant backend code improvements.

To clarify, this release isn’t an entirely new data source. Rather, it integrates the community-developed Flight SQL plugin into the official InfluxDB data source, incorporating several highly requested features and performance improvements. This blog teaches you how to configure the new data source plugin with your InfluxDB V3 instance.

Where can I find the V3 data source?

The V3 Flight SQL plugin has been added to the official InfluxDB data source within Grafana. Below is a table of the Grafana edition and the version you need to install to use it:

Edition	Version
Grafana OSS	10.3.0+
Grafana Enterpise	10.3.0+
Grafana Cloud	Latest

Let’s jump into Grafana Cloud and locate the official InfluxDB V3 Datasource:

Here is our Grafana Cloud instance. On the left-hand navigation panel, we select Data Source. From there, we select Add data source and choose the official InfluxDB data source from the time series database list.

So no big changes there! We find and create our new InfluxDB data source in the same way we always have. Let’s move on to configuration and learn how to tap into the V3 SQL plugin.

How do I configure my V3 data source?

As usual, we need to give the data source a useful name. In this case, I aptly named the data source InfluxDB V3. Next, we need to select our Query language—this is also where we enable our V3 plugin.

Within the drop-down list, you will now see a third option, SQL. Selecting SQL as a new language automatically utilizes the newly integrated Flight SQL API within the Grafana backend. This also transforms the rest of the form parameters we need to fill out.

You will need to define the following:

Parameter	Description
URL	The InfluxDB V3 host you wish to connect to. In this case, if you omit the port it will automatically default to the protocol's default port (http -> 80, https ->443). Here are some accepted examples: Serverless: https://eu-central-1-1.aws.cloud2.influxdata.com Serverless: https://eu-central-1-1.aws.cloud2.influxdata.com:443 Dedicated: https://b0c7cce5-8dbc-428e-98c6-7f996fb96424.b.influxdb.io Dedicated: https://b0c7cce5-8dbc-428e-98c6-7f996fb96424.b.influxdb.io:443 Note: Port definition is currently only important for clustered and eventually OSS / Community users.
Database	Define the database/bucket name you wish to query from. Note that this is a one-to-one ratio as part of the Grafana data source implementation. In this case, we are connecting to a database called factory.
Token	Lastly, you need to create a token with at least read-only privilege for the specific database you defined.

After completing the form, select Save & test. This should result in a successful connection to the InfluxDB V3 data source.

A quick 101

Now that we connected Grafana to InfluxDB V3, let’s build a simple visualization and discuss some features along the way. For this demo, I used data generated from the Arrow Task Engine demo.

Data Explorer

Within the Explore panel, we select our InfluxDB V3 data source. This provides an SQL explorer for navigating our current database schema as well as building basic SQL queries.

Let’s start by creating a table that shows all columns from the last hour.

To generate this table, we simply select—via the drop-down panels—the table (measurement) and columns (tags and fields) we would like to view. We can also formulate more specific SQL queries as well:

After generating some SQL using the basic explorer, can I edit the SQL? The quick answer is yes; simply toggle from Builder to Code.

From Table to Time Series

Because InfluxDB 3.0 now returns tables rather than table streams, the V3 plugin defaults to the Grafana table format. Let’s build a query and transform it into the time series format for visualization as a line graph.

Here is our SQL:

SELECT "vibration", "machineID", time 
FROM iox.machine_data 
WHERE time >= $__timeFrom AND time <= $__timeTo AND "machineID" = 'machine1' 
ORDER BY time

Note that we must add ORDER BY time to our query to ensure the order of our timestamps.

Now we can simply change the Format using the dropdown menu from table to Time series.

We will save this query to a dashboard to explore our final feature in this blog: query-based dashboard variables.

Dashboard Variables

One feature missing from the Flight SQL community plugin was query-based dashboard variables. These allow you to define a query to populate a Grafana dashboard dropdown. You can then use these variables within your other queries to create a more dynamic experience.

In this example, we created a dashboard variable to list individual machine tags. This can be utilized within our line graph query by referencing $machineID (the name of the dashboard variable).

SELECT "vibration", "machineID", time 
FROM iox.machine_data 
WHERE time >= $__timeFrom AND time <= $__timeTo AND "machineID" = '$machineID' 
ORDER BY time

At the top left of the image, you can see our populated dashboard variable. We selected machine2, which appears to show an anomaly…

Next steps

We are extremely excited to see the new and improved InfluxDB data source in the hands of the community. We hope the new plugin provides a streamlined experience to start interacting with InfluxDB V3 via its Flight SQL interface. As you may have noticed, this new plugin is still alpha, so if you have any issues/feature requests, please let us know via our Slack community or open an issue within the Grafana repo. We have only scratched the surface of the changes made within the InfluxDB data source rewrite. Make sure to check out the next blog, in which we will dive deeper into some of the features and implementation, including moving the data source to the Grafana backend.

You can see all this in action here:

Quix Community Plugins for InfluxDB: Build Your Own Streaming Task Engine

Jay Clifford (InfluxData) — Wed, 18 Oct 2023 08:00:00 +0000

With our plans for InfluxDB 3.0 OSS laid out, both myself and the rest of the DevRel team have been actively searching for ecosystem platforms that would be logical integrations for the future of InfluxDB. One of these platforms is Quix!

Quix is a comprehensive solution tailored for crafting, launching, and overseeing event streaming applications using Python. If you’re looking to sift through time series or event data in real-time for instant decision-making, Quix is your go-to. What we discovered is that it makes a great alternative task engine for InfluxDB 3.0. If you would like to see our initial project with Quix to create an anomaly detection pipeline, check out this blog.

Progressing from this, we are extremely pleased to announce the release of two InfluxDB Community plugins for Quix. In this blog post, we will break down what each plugin does and provide an example of a use case you can try out yourself.

The plugins

The community contributions comprise two plugins:

InfluxDB Source: This plugin allows users to query InfluxDB 3.0 using Apache Arrow Flight on a user-defined interval, parse the data into a pandas DataFrame, and publish to a Quix stream topic.
InfluxDB Destination: This plugin allows users to ingest a DataFrame from a Quix stream topic, and define its structure (measurement and tags) before writing the data to an InfluxDB instance (this is compatible with both InfluxDB 2.x and 3.x).

You can find these plugins in the Quix platform by navigating to “Code Samples” via the menu on the left and searching for “InfluxDB”. Let’s take a look at the configuration of each of these plugins individually.

InfluxDB Source

InfluxDB Source is a Python-based plugin, so within the Quix environment it’s highly customizable based on your needs. You may simply edit the Python script within their cloud-based editor before deployment. Let’s start with an overview of the “out of the box” plugin.

The InfluxDB Source plugin requires the following environment variables to be defined in order to run:

output: This is the output topic that will receive the stream (Default: influxdb, Required: True)
task_interval: Interval to run query. Must be within the InfluxDB notation; 1s, 1m, 1h, 1d, 1w, 1mo, 1y (Default: 5m, Required: True)
INFLUXDB_HOST: Host address for the InfluxDB instance. (Default: eu-central-1-1.aws.cloud2.influxdata.com, Required: True)
INFLUXDB_TOKEN: Authentication token to access InfluxDB. (Default: <TOKEN>, Required: True)
INFLUXDB_ORG: Organization name in InfluxDB. (Default: <ORG>, Required: False)
INFLUXDB_DATABASE: Database name in InfluxDB where data is stored. (Default: <DATABASE>, Required: True)
INFLUXDB_MEASUREMENT_NAME: The InfluxDB measurement to read data from. If not specified, the name of the output topic will be used (Default: <INSERT MEASUREMENT>, Required: True)

Once these parameters are set the script will run as follows:

As seen in the diagram, this plugin loops until shutdown. Based on the provided interval the script sleeps before repeating the query. The interval is also used to formulate the query since we will query only the latest data based on how long the plugin has slept for. Note that the plugin pushes the resulting data using the DataFrame format.

InfluxDB Destination

The InfluxDB Destination plugin – like the Source plugin, is purely Python-based – enables you to customize how the plugin writes data to InfluxDB (a feature we will use in the demo). For now, let’s start with another overview of the “out of the box” plugin.

The InfluxDB Destination plugin requires you to define the following environment variables in order to run:

input: This is the input topic (Default: detection-result, Required: True)
INFLUXDB_HOST: Host address for the InfluxDB instance. (Default: eu-central-1-1.aws.cloud2.influxdata.com, Required: True)
INFLUXDB_TOKEN: Authentication token to access InfluxDB. (Default: <TOKEN>, Required: True)
INFLUXDB_ORG: Organization name in InfluxDB. (Default: <ORG>, Required: False)
INFLUXDB_DATABASE: Database name in InfluxDB where data should be stored. (Default: <DATABASE>, Required: True)
INFLUXDB_TAG_COLUMNS: Columns to be used as tags when writing data to InfluxDB. (Default: ['tag1', 'tag2'], Required: False)
INFLUXDB_MEASUREMENT_NAME: The InfluxDB measurement to write data to. If not specified, the name of the input topic will be used. (Default: <INSERT MEASUREMENT>, Required: False)

Once these parameters are set the script will run as follows:

As the diagram shows, on startup the script loads the given environment variables and initializes the InfluxDB Client. It then waits for the chosen topic to stream data to the plugin. The streamed data is received as a pandas DataFrame so we need to apply some basic transformations (rename time column, set time as index) before writing to InfluxDB. From there, we write the DataFrame to InfluxDB based upon the schema specified within the environment variables.

The demo

Now that we have discussed each plugin and its respective design, let’s apply them to an industrial IoT use case.

In this example, we have three machines on a production line producing sensor data and writing to an MQTT broker (for this use case we are using HiveMQ). The payload for each machine is within a JSON structure which looks like this.

{"metadata": {"machineID": "machine1", "barcode": "31856669", "provider": "Miller-Phillips"}, "data": [{"temperature": 40}, {"load": 100}, {"power": 204}, {"vibration": 90}]}

We will start by connecting the Quix MQTT client to the HiveMQ broker.

MQTT Client -> Quix -> InfluxDB

First, I located the MQTT plugin, which connects to the broker and writes the data to a Quix Stream (the one on the right).

On inspection of the code, I realized I needed to make a minor code change because I was connecting to a broker that didn’t require TLS authentication. I needed to remove these lines:

# we'll be using tls
mqtt_client.tls_set(tls_version = mqtt.client.ssl.PROTOCOL_TLS)
mqtt_client.username_pw_set(os.environ["mqtt_username"], os.environ["mqtt_password"])

Next, I used the provided environment variables to establish my connection to the broker.

Then click new deployment. Configure our resource limits (default is fine) and click deploy. With that our first stage is complete.

Next, we need to write this data to InfluxDB. For this, we make use of the new destination plugin with some alterations.

We follow a similar process to select the InfluxDB 3.0 destination plugin and generate a project. Now we have a slight issue to overcome. Currently, the destination plugin only supports ingesting Quix DataFrames. In our case, we are writing Event Data in JSON. So we need to write a small transformation function for event-based data, which you can see here:

def on_event_data_received_handler(stream_consumer: qx.StreamConsumer,data: qx.EventData):
    with data:
        jsondata = json.loads(data.value)
        metadata = jsondata['metadata']
        data_points = jsondata['data']
        fields = {k: v for d in data_points for k, v in d.items()}
        timestamp = str(data.timestamp)

        point = {"measurement": measurement_name, "tags" : metadata, "fields": fields, "time": timestamp}

        print(point)
        client.write(record=point)

def on_stream_received_handler(stream_consumer: qx.StreamConsumer):

    # subscribe to new DataFrames being received
    # if you aren't familiar with DataFrames there are other callbacks available
    # refer to the docs here: https://docs.quix.io/sdk/subscribe.html
    stream_consumer.timeseries.on_dataframe_received = on_dataframe_received_handler

    stream_consumer.events.on_data_received = on_event_data_received_handler

The majority of the code you will use is already there. The main element we added was the on_event_data_received_handler. Now that we have done this, we define our environment variables like we did for the MQTT connector.

A note on two of these environment variables:

You can modify InfluxDB_token to a secure environment variable to secure your token.
We are not using InfluxDB_tag in this example because we are using the metadata within our JSON payload as our tags.

Click on new deployment. Configure our resource limits (default is fine) and click deploy.

We are now writing raw machine data to our chosen measurement within InfluxDB. Let’s move on to how we can utilize Quix as a task-based engine to transform our stored raw data.

InfluxDB -> Quix (Transform) -> InfluxDB

We will utilize the out-of-the-box configurations for both community plugins to piece together a transformation task. The transformation task is simple:

Query the last 1-minute worth of data
Add a new column that checks if the vibration over that interval surpassed a user-defined threshold. This column will contain true or false based on the outcome.
Write the data back to a new table within InfluxDB

Let’s start with querying the data from InfluxDB. This time we are going to utilize our InfluxDB 3.0 Source plugin. Like our previous examples, we search for and select this from the Code Samples library and create a project.

We do not need to modify any of the plugin code for this one. Just simply define our environment variables:

We then click new deployment. Configure our resource limits (default is fine) and click deploy.

Our query data is now being written directly as a DataFrame for the topic InfluxDB. We can now deploy and create a series of transformation plugins to reshape our data. For this example, we are going to keep it simple:

Ingest the DataFrame
Use basic conditional logic within the vibration column to check if it surpasses our predefined threshold
Create the new boolean column and write the DataFrame to a new topic reader for ingestion.

The code looks like this:

import quixstreams as qx
import os
import pandas as pd

client = qx.QuixStreamingClient()

topic_consumer = client.get_topic_consumer(os.environ["input"], consumer_group = "empty-transformation")
topic_producer = client.get_topic_producer(os.environ["output"])

def on_dataframe_received_handler(stream_consumer: qx.StreamConsumer, df: pd.DataFrame):

        vibration_limit = int(os.environ["vibration_limit"])
        df['over_limit'] = df['vibration'] > vibration_limit

        stream_producer = topic_producer.get_or_create_stream(stream_id = stream_consumer.stream_id)
        stream_producer.timeseries.buffer.publish(df)

def on_event_data_received_handler(stream_consumer: qx.StreamConsumer, data: qx.EventData):
    print(data)
    # handle your event data here

def on_stream_received_handler(stream_consumer: qx.StreamConsumer):
    stream_consumer.events.on_data_received = on_event_data_received_handler # register the event data callback
    stream_consumer.timeseries.on_dataframe_received = on_dataframe_received_handler

topic_consumer.on_stream_received = on_stream_received_handler

print("Listening to streams. Press CTRL-C to exit.")

qx.App.run()

Based on this basic transformation plugin we have three environment variables to define.

We then click new deployment. Configure our resource limits (default is fine) and click deploy.

Our final step is writing the data back to InfluxDB. For this task, we deploy another instance of our InfluxDB Destination plugin. This time, because we ingest the DataFrame from our transformation topic, we only need to define the environment variables.

Notes on two of these environment variables:

InfluxDB_measurement writes our transformed data to a new table called transformed. InfluxDB creates this table on demand.
We are providing a string array for the columns we wish to define as tags:['machineID', 'barcode', 'provider']

We then click new deployment. Configure our resource limits (default is fine) and click deploy.

Conclusion

There you have it! We successfully deployed our event streaming pipeline and task engine for InfluxDB. From a birds-eye view it looks like this.

(Thanks Quix for the great user interface)

Here is what our raw machine data and transformed data look like in InfluxDB.

(Raw Machine Data)

(Transformed Machine Data)

In summary, we utilized the Quix platform along with the new InfluxDB Community plugins to ingest ‘live’ raw machine data from three MQTT topics, store this data within InfluxDB, and then derive new value from our stored data. So where could you go next?

My hope is you can take this example, adapt it to your own needs, and scale it using the Quix platform. The major benefit of using Quix is we can efficiently scale the number of Sources, Transformations, and Destinations to fit our needs. A great example of this would be to expand our task engine:

Downsampling script
Anomaly detection algorithm
Check and Alert script

Each script would subscribe to our InfluxDB topic and work in parallel with one another, making efficient use of the data within the event streaming pipeline compared to conventional task methods which would re-query the data.

You can find the source code for a similar project here. If you have any questions or would like to discuss InfluxDB or Quix further, come hang out with us within our Slack channel.

Infrastructure Monitoring Basics with Telegraf, InfluxDB, and Grafana

Jay Clifford (InfluxData) — Tue, 29 Aug 2023 07:35:00 +0000

Earlier this year, I had the pleasure of speaking at the Open Source Summit North America. When choosing a topic, I felt it was time to return to our roots and discuss the subject that originally put InfluxDB on the map: infrastructure monitoring.

What was especially exciting was the opportunity to showcase the new capabilities of InfluxDB 3.0 to the open source community and explain their significance for the future of infrastructure monitoring use cases.

This blog breaks down the key points from that presentation and delves deeper into those topics, offering further insights and discussion.

Monitoring vs observability

InfluxDB has the ability to tackle both monitoring and observability use cases. 3.0 not only improves the performance of both but also makes Observability use cases viable at scale. Before we jump into the details, let’s level set and discuss what exactly the difference is between monitoring and observability:

Monitoring:

Monitoring involves the collection and analysis of metrics, logs, and events to keep track of system performance. Using predefined rules and thresholds, the monitoring process detects potential issues and generates alerts when threshold breaches occur, thereby helping to maintain system health. You can apply this approach across various types of infrastructure, in both the physical and the digital realms.

Observability:

Observability takes monitoring a step further to include the instrumentation of both code and infrastructure to expose pertinent data. This empowers teams to deeply understand the behavior of their systems. By correlating data from diverse sources, it facilitates the diagnosis of issues and the identification of root causes. This, in turn, provides actionable insights for effective problem-solving. Tracing, which maps the journey of requests or transactions through components of a system, is the quintessential observability tool.

At a glance, monitoring and observability might appear to serve the same purpose, but they approach system health from distinct angles. Monitoring is proactive, setting predefined rules and thresholds to ensure systems are operating within desired parameters. It’s about ensuring everything is “on track” and alerting when it’s not. On the other hand, observability is more diagnostic in nature. It’s about understanding “why” something happened and drilling down into system behavior. While both aim to maintain system health and performance, monitoring is more about detecting known issues, whereas observability focuses on exploring unknown issues. However, in the landscape of modern system management, they are complementary. Together, they provide a holistic view of system health, performance, and behavior, ensuring both robustness and resilience.

Monitoring and observability fields

We can further categorize these concepts into several distinct fields, each with its specific focus and application:

Network monitoring: Observing the performance of network components such as routers, switches, and firewalls to ensure efficient data transmission, detect bottlenecks, and identify security threats.

Server monitoring: Tracking the performance and availability of physical or virtual servers, including CPU usage, memory consumption, disk space, and response times, to ensure optimal performance and reduce downtime.

Application performance monitoring (APM): Monitoring the performance of software applications to identify issues, bottlenecks, and inefficiencies in the code, databases, or infrastructure components. (Application performance monitoring has been highlighted in blue as it can also fall into the realm of observability which we will cover later on).

Cloud infrastructure monitoring: Tracking the performance and availability of cloud-based services, such as virtual machines, storage, and databases, to optimize resource allocation and minimize costs.

A problem to solve

Now, onto the fun part! I always find that when you have a problem to solve, it is easier to do so if you first learn about the solution you wish to employ. Let’s ask ChatGPT to create an infrastructure problem which involves creating a solution to monitoring each field we discussed earlier.

So based on this problem, we can break the architecture down into our monitoring and observability subfields:

Network monitoring: Monitor network traffic, especially requests to models.

Server monitoring: Track the performance of Whisper GPT servers hosting their primary models, focusing on CPU and GPU metrics.

Application performance monitoring (APM): This encompasses two aspects:

Monitoring our Kubernetes cluster on barebone infrastructure.
Providing developers with tools to proactively analyze code within the Whisper platform.

Cloud infrastructure monitoring: Utilize services like App Runner or Amazon EKS for front-end management.

Solving the problem

Now, I am going to show you how we can tackle each of these problems using the TIG stack (Telegraf, InfluxDB 3.0 and Grafana) and OpenTelemetry. Before we get ahead of ourselves let’s start by creating an architectural blueprint.

Data collection

Telegraf is our go-to open source data collection agent designed specifically for gathering metrics and events. Equipped with more than 300 plugins for both ingesting, transforming, and outputting data, it is a versatile agent for time series data. The community refers to it as the Swiss army knife of monitoring and observability data collection due to its ability to deploy both pull and push collection methods based on the plugin’s use. It is also equipped to handle the parsing of a considerable number of data formats, including Prometheus, JSON, XML, CSV, and many more.

Let’s take a look at some of the plugins we might use to solve our Whisper GPT problem:

Field	Plugins
Network monitoring	gNMI Net SNMP
Server monitoring	CPU Disk Diskio Mem Processes Nvidia SMI System
APM	Kubernetes Inventory Kubernetes OpenTelemetry Prometheus
Cloud infrastructure monitoring	CloudWatch Kubernetes Inventory Kubernetes

By design, Telegraf acts like a data pipeline that you can route through different plugins to process and aggregate the data before reaching its final output. The following architecture diagram visualizes this nicely.

At this point of the presentation, I delved into Telegraf best practices and initial deployment. I highly recommend checking out our InfluxDB University course on Telegraf to learn more about this part.

Data storage

Having checked data collection off our list, now let’s move on to establishing the keystone within our infrastructure monitoring architecture… data storage.

InfluxDB 3.0 is a purpose-built time series database built for handling metrics, traces, and logs at a massive scale for real-time analytics. This is driven by the three core open source technologies we use to create our database engine: Apache Arrow, Parquet and DataFusion. If you would like to learn more about how we deploy these technologies, I highly recommend checking out this blog.

At its very core, InfluxDB offers us some considerable benefits when it comes to our use case:

Benefit	Description
Schema on write	This is a no-brainer when it comes to monitoring use cases. Schema design is one of the most costly and time-consuming tasks developers need to focus on when using a conventional database. It is also an issue that will not go away because, depending on how Whisper GPT evolves, the schema will as well. InfluxDB constructs the schema on initial data ingest, removing the need to build out a schema from scratch. This capability allows the schema to evolve along with our solution.
Write and query performance	In most monitoring use cases, users require near-real-time visibility on the data they are ingesting and this can come from hundreds of data sources generating gigabytes of time series data a day. InfluxDB can ingest over 4 million values per second while providing millisecond query return times. I highly recommend checking out this blog to see some of our performance stats.
Single data store	One of the most interesting issues to solve within the monitoring and observability space is the storage of different types of time series data: traces, metrics and logs. Most seasoned providers use different data storage technologies for each and then provide an interface for joining these results at query time. InfluxDB 3.0 allows us to keep everything in a single store, reducing the overall cost of ownership.
Query support	With InfluxDB 3.0, we wanted to emphasize meeting developers where they are. This meant providing query languages that most users, whether current and new, can engage with. InfluxQL and SQL give developers performant options for interfacing with their data. They also provide a rich ecosystem to third-party solutions that make use of both languages.

At this point, I, again, discussed best practices and getting started with InfluxDB 3.0. I highly recommend checking out the InfluxDB 3.0 Essentials course to catch up on this content.

Data in action

We have reached our second milestone. At this point, our Whisper GPT infrastructure monitoring process is collecting and storing data.

Now we need to do something with this data. Depending on your own initiatives or current company infrastructure, you might have a pretty good idea how this part is going to shape out. For the sake of completeness, let’s discuss some ideas.

Grafana is an open source data visualization and monitoring platform. It allows users to create interactive dashboards for real-time data analysis and tracking of metrics across various data sources. It is one of the most widely used platforms with InfluxDB. There are hundreds of blogs and articles on utilizing both Grafana and InfluxDB so let’s focus on the parts that are new with 3.0.

The FlightSQL plugin provides a new connection method between InfluxDB and Grafana allowing users to build dashboards with native SQL. The table below provides some useful SQL queries within Grafana.

Note how we can deploy $__ variables to make our queries dynamic. The example above shows methods for monitoring our CPU usage over time and the last known total memory reading.

You can find the full dashboard here.

OpenTelemetry

The last point I wanted to touch on is OpenTelemetry. InfluxDB 3.0 provides one datastore for metrics, logs, and traces. We are working hard to provide the integrations required to make InfluxDB a plug-and-play solution for your OpenTelemetry stack. The ultimate goal is to provide the ability to visualize and inspect traces alongside metrics using a single pane of glass through Grafana.

We use Killercoda to provide an online interactive demo, which you can try out here.

The finishing touch

Our three-step milestones to building an infrastructure monitoring platform.

I took the liberty of adding some further integrations to the Data Action list. Let’s conclude by applying this to our Whisper GPT platform.

Through the integration of Telegraf, InfluxDB, and Grafana (aka the TIG stack), we architected a scalable solution adept at collecting, storing, and processing infrastructure data across diverse domains.

It’s my hope that this blog not only enlightens you about the journey of InfluxDB 3.0 and infrastructure monitoring, but also kindles your interest in the expansive world of Open Source. Open architecture offers a wealth of benefits, and the deeper you dive the more you will find. If you have any questions or comments on InfluxDB, Telegraf, or infrastructure monitoring in general please do not hesitate to reach out to me via Slack. I would love to hear from you.

Tutorial: Modifying Grafana's Source Code

Jay Clifford (InfluxData) — Fri, 25 Aug 2023 07:35:00 +0000

This article was originally published on dev.to and is reposted here with permission.

A story of exploration and guesswork

So this blog is a little different from my usual tutorials…

A little background: I have been working with Jacob Marble to test and “demo-fy” his work with InfluxDB 3.0 and the OpenTelemetry ecosystem (If you would like to learn more, I highly recommend checking out this blog).

During the project, we identified a need to enable specific Grafana features for InfluxDB data sources, particularly the trace to logs functionality. Grafana is an open source platform, and one of its major advantages is the ability to modify its source code to suit our unique requirements. However, diving into the codebase of such a robust tool can be overwhelming, even for the most seasoned developers.

Despite the complexity, we embraced the challenge and dove headfirst into Grafana’s source code. We tumbled, we stumbled, and we learned a great deal along the way. And now, having successfully modified Grafana to meet our specific project needs, I believe it’s time to share this acquired knowledge with you all.

The purpose of this blog is not just to provide you with a step-by-step guide for tweaking Grafana’s source code, but also to inspire you to explore and adapt open source projects to your needs. It’s about imparting a method and a mindset, cultivating a culture of curiosity, and encouraging more hands-on learning and problem-solving.

I hope that this guide inspires you to modify Grafana’s source code for your projects, thereby expanding the horizons of what’s possible with open source platforms. It’s time to roll up your sleeves and venture into the depths of Grafana’s code.

The problem

So our problem lies within the Trace visualization of Grafana.

As you can see the visualization performs rather well with InfluxDB except for one disabled button: Logs for this span. If we don’t configure a log data source with our trace data source (in this case, Jaeger with InfluxDB 3.0 acting as the gRPC storage engine), then Grafana automatically disables this button. Grafana usually represents a log data source by default using the log explorer interface. Common log data sources include Loki, OpenSearch, and Elasticsearch. So let’s head across to the Jaeger data source and configure that…

You can navigate data sources via Connections -> Data Sources. We currently have three data sources configured: FlightSQL, InfluxDB, and Jaeger. If we open the Jaeger configuration and navigate to the Trace to Logs section we want to be able to select either InfluxDB or FlightSQL as our Data source.

Houston, we have a problem. It appears Grafana doesn’t recognize InfluxDB as a log data source. Fair enough. InfluxDB only recently became a viable option for logs. So, what are our options?

We lie down, accept the issue, and hope that in the future this feature becomes generic enough to support more data sources.
Take action and make the change ourselves.

Well, by now you know what option we chose.

The solution

This section summarizes the steps I took to discover the changes I needed to make, how to implement the changes for your own data source, and, finally, how to build your own custom build of Grafana OSS.

Discovery

So the first step is to understand where to even begin. Grafana is a huge open source platform with many components so I needed to narrow down the search. So the first thing I did was search the Grafana repository for signs of life.

As you can see I made this little discovery by using the keyword trace, which led me to the directory TraceToLogs. This led me to this section of code within TraceToLogsSettings.tsx:

export function TraceToLogsSettings({ options, onOptionsChange }: Props) {
  const supportedDataSourceTypes = [
    'loki',
    'elasticsearch',
    'grafana-splunk-datasource', // external
    'grafana-opensearch-datasource', // external
    'grafana-falconlogscale-datasource', // external
    'googlecloud-logging-datasource', // external
  ];

This section of code seems to create a static list of data sources supported by the Trace to Logs feature. We can confirm this by some of the common suspects within the list (Loki, Elasticsearch, etc.). Based on this finding, our first alteration to the Grafana source code is to add our data sources to this list.

Now, as the coding pessimist that I am, I knew this probably wouldn’t be the only change we needed to make but it’s a good place to start. So, I did the following:

I forked the Grafana repo
Cloned the repo:

git clone https://github.com/InfluxCommunity/grafana

Before I made those modifications I wanted to do some more searching to see if there were any other changes I should make. One line stood out to me in TraceToLogsSettings file:

const updateTracesToLogs = useCallback(
    (value: Partial<TraceToLogsOptionsV2>) => {
      // Cannot use updateDatasourcePluginJsonDataOption here as we need to update 2 keys, and they would overwrite each
      // other as updateDatasourcePluginJsonDataOption isn't synchronized
      onOptionsChange({
        ...options,
        jsonData: {
          ...options.jsonData,
          tracesToLogsV2: {
            ...traceToLogs,
            ...value,
          },
          tracesToLogs: undefined,
        },
      });
    },
    [onOptionsChange, options, traceToLogs]
  );

It was TraceToLogsOptionsV2. When I searched for places where Grafana used this interface, I found the following entry.

It appears we might also have work to do in the createSpanLink.tsx file. Within this section I found the following piece of code. At this point, my question was “what exactly is this code doing?”

To cut a long story short, the case statement essentially tells the trace visualization to check the defined log data source (if any) and to define a query interface relevant to that data source. If the specified data source is not found within this case statement, then Grafana simply disables the button. This meant that changing the original file won’t be enough as we suspected.

Okay, with our investigation complete, let’s move on to the code changes.

Modification

We have two files to modify:

Let’s start with the simplest to tackle and go from there.

TraceToLogsSettings

This file was relatively simple to change. All we needed to do was modify the static list of supported log input sources like so:

export function TraceToLogsSettings({ options, onOptionsChange }: Props) {
  const supportedDataSourceTypes = [
    'loki',
    'elasticsearch',
    'grafana-splunk-datasource', // external
    'grafana-opensearch-datasource', // external
    'grafana-falconlogscale-datasource', // external
    'googlecloud-logging-datasource', // external
    'influxdata-flightsql-datasource', // external
    'influxdb', // external
  ];

As you can see, I added two data sources. I ran a quick build of the Grafana project to see how this affected our data source configuration (we will discuss how to build at the end).

Hey presto! We have a result. Now, this still didn’t enable the button within our Trace View but we already knew this would require more work.

createSpanLink

Now, let’s move on to the meat of our modification. For the record, I am not a TypeScript developer. What I do know is that the file has a whole bunch of examples we can use to attempt a blind copy-and-paste job with a few modifications. I ended up doing this for both plugins but to keep the blog short we will focus on the InfluxDB official plugin.

My hypothesis was to use the Grafana Loki interface as the basis for the InfluxDB interface. The first included adding data source types:

import { LokiQuery } from '../../../plugins/datasource/loki/types';
import { InfluxQuery } from '../../../plugins/datasource/influxdb/types';

These are easy to locate when Grafana has an official plugin for your data source since it’s embedded within the official repository. For our community plugin I had two options: define a static interface within the file or provide more query parameters. I chose the latter.

The next step was to modify the case statement:

// TODO: This should eventually move into specific data sources and added to the data frame as we no longer use the
    //  deprecated blob format and we can map the link easily in data frame.
    if (logsDataSourceSettings && traceToLogsOptions) {
      const customQuery = traceToLogsOptions.customQuery ? traceToLogsOptions.query : undefined;
      const tagsToUse =
        traceToLogsOptions.tags && traceToLogsOptions.tags.length > 0 ? traceToLogsOptions.tags : defaultKeys;
      switch (logsDataSourceSettings?.type) {
        case 'loki':
          tags = getFormattedTags(span, tagsToUse);
          query = getQueryForLoki(span, traceToLogsOptions, tags, customQuery);
          break;
        case 'grafana-splunk-datasource':
          tags = getFormattedTags(span, tagsToUse, { joinBy: ' ' });
          query = getQueryForSplunk(span, traceToLogsOptions, tags, customQuery);
          break;
        case 'influxdata-flightsql-datasource':
            tags = getFormattedTags(span, tagsToUse, { joinBy: ' OR ' });
            query = getQueryFlightSQL(span, traceToLogsOptions, tags, customQuery);
          break;
        case 'influxdb':
            tags = getFormattedTags(span, tagsToUse, { joinBy: ' OR ' });
            query = getQueryForInfluxQL(span, traceToLogsOptions, tags, customQuery);
          break;
        case 'elasticsearch':
        case 'grafana-opensearch-datasource':
          tags = getFormattedTags(span, tagsToUse, { labelValueSign: ':', joinBy: ' AND ' });
          query = getQueryForElasticsearchOrOpensearch(span, traceToLogsOptions, tags, customQuery);
          break;
        case 'grafana-falconlogscale-datasource':
          tags = getFormattedTags(span, tagsToUse, { joinBy: ' OR ' });
          query = getQueryForFalconLogScale(span, traceToLogsOptions, tags, customQuery);
          break;
        case 'googlecloud-logging-datasource':
          tags = getFormattedTags(span, tagsToUse, { joinBy: ' AND ' });
          query = getQueryForGoogleCloudLogging(span, traceToLogsOptions, tags, customQuery);
      }

As you can see I added two new cases: influxdata-flightsql-datasource and influxdb. Then, I copied the two function calls within the case from Loki: getFormattedTags and getQueryFor. I determined that I could leave getFormattedTags alone because it appeared to be the same for the majority of the cases. However, I still needed to define my own getQueryFor function.

Let’s take a look at the new getQueryForInfluxQL function that’s called in the influxdb case statement:

function getQueryForInfluxQL(
  span: TraceSpan,
  options: TraceToLogsOptionsV2,
  tags: string,
  customQuery?: string
): InfluxQuery | undefined {
  const { filterByTraceID, filterBySpanID } = options;

  if (customQuery) {
    return {
      refId: '',
      rawQuery: true,
      query: customQuery,
      resultFormat: 'logs',
    };
  }

  let query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE time >=${__from}ms AND time <=${__to}ms';

  if (filterByTraceID && span.traceID && filterBySpanID && span.spanID) {
            query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE "trace_id"=\'${__span.traceId}\' AND "span_id"=\'${__span.spanId}\' AND time >=${__from}ms AND time <=${__to}ms';
    } else if (filterByTraceID && span.traceID) {
            query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE "trace_id"=\'${__span.traceId}\' AND time >=${__from}ms AND time <=${__to}ms';
    } else if (filterBySpanID && span.spanID) {
            query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE "span_id"=\'${__span.spanId}\' AND time >=${__from}ms AND time <=${__to}ms';
  }

  return {
    refId: '',
    rawQuery: true,
    query: query,
    resultFormat: 'logs',
  };
}

There is quite a lot here, but let me highlight the important parts. First of all, I started with an exact copy of the Loki function. Then, I made the following changes:

I changed the return interface from LokiQuery | undefined to InfluxQuery | undefined. This is the data source type we imported earlier.
Next, I focused on the return payload. After some digging in the InfluxQuery type file, I came up with this:
```
return {
    refId: '',
    rawQuery: true,
    query: query,
    resultFormat: 'logs',
  };
```
The InfluxDB data source had a resultFormat parameter which allowed me to define the result format (usually metrics). This also informed me that the data source expected a raw query rather than an expression.
Lastly, I defined the queries that would run when the user clicked the button. These depended on what filter features the user toggled within the data source settings (filter by traceID, spanID or both). I modified the if statement defined within the Loki function and constructed static InfluxQL queries. From there, I used the Grafana placeholder variables found within other data sources to make the queries dynamic. Here is an example:
```
if (filterByTraceID && span.traceID && filterBySpanID && span.spanID) {
            query = 'SELECT time, "severity_text", body, attributes FROM logs WHERE "trace_id"=\'${__span.traceId}\' AND "span_id"=\'${__span.spanId}\' AND time >=${__from}ms AND time <=${__to}ms';
```
Full disclosure, it took me a good minute to find out about the >=${__from}ms and <=${__to}ms. This ended up being a brute force build and error case.

Building

Phew! We’re past the hard bit. Now onto the build process. I have quite a few years of experience with Docker, so this part was stress-free for me, but I imagine it could be daunting for new Docker users. Luckily, Grafana has some easy-to-follow documentation for the task. To paraphrase, these are the steps:

Run the following build command (this can take a while and make sure your docker VM has enough memory if using macOS or Windows)
```
make build-docker-full
```
The build process produces a Docker image called: grafana/grafana-oss:dev. We could just use this image, but as a formality, I like to retag the image and push it to my Docker registry.
```
docker tag grafana/grafana-oss:dev jaymand13/grafana-oss:dev2
docker push jaymand13/grafana-oss:dev2
```
This way I have checkpoints when I am brute forcing changes.

There we have it! A fully baked Grafana dev image to try out with our changes.

The results and conclusion

So after investigating, making the changes, and building our new Grafana container, let’s take a look at our results:

With our changes, the Logs for this span button is now active. We also have this neat little Log button that appears next to each span. A confession: the blue Logs for this span button currently only works within the Grafana Explorer tab, but the new Log link works within our dashboard.

To quickly explain the differences, users build custom Grafana Dashboards and can include 1 or many data sources with a variety of different visualizations. Data Explorers, on the other hand, provide an interface for drill-down and investigation activities like you see in the screenshot above. Still, it’s not a huge problem compared to how little we needed to change to get here.

And so, we’ve reached the end of our dive into the intricacies of modifying Grafana’s source code. Over the course of this tutorial, I hope you’ve not only gained a practical understanding of how to customize Grafana for your specific requirements, but also an appreciation for the flexibility and potential of open source platforms in general.

Remember, in the realm of open source, there’s no limit to how much we can tweak, adjust, and reimagine to suit our needs. I hope this guide serves you well as you delve deeper into your own projects, and that it brings you one step closer to mastering the powerful tool that is Grafana. For me, my journey continues as I now plan to add exemplar support to this OSS build. If you would like to try this out yourself you can find the OpenTelemetry example here.

Client Library Deep Dive: Python (Part 2)

Jay Clifford (InfluxData) — Fri, 28 Jul 2023 07:35:00 +0000

Working with the new InfluxDB 3.0 Python CLI and Client Library

Okay, we are back for Part 2! Last time we discussed the new community Python library for InfluxDB 3.0. If you missed it, you can also watch it in video form.

Now for Part 2, let’s talk about a bolt-on application that uses the client library as the core of its development, the InfluxDB 3.0 Python CLI.

Python CLI

Okay, so following the same format as before, what were the reasons for building the CLI? Well, there are two primary reasons:

We wanted to give users a data browsing tool that leveraged the new Flight endpoint. Python gave us the opportunity to prototype fast before we invested work in a more robust CLI offering. It also allowed us to leverage some interesting data manipulation libraries that could extend the scope of the Python CLI.
We wanted a robust way to test the newly created InfluxDB 3.0 Python Client library, as you will see most of the tooling and functionality in use.

Install

Let’s talk about the installation process because, I must admit, Python doesn’t provide the most user-friendly packaging and deployment methods unless you use it daily. I recommend installing the CLI in a Python Virtual Environment first for test purposes:

$ python3 -m venv ./.venv
$ source .venv/bin/activate
$ pip install –upgrade pip
$ pip install influxdb3-python-cli

This set of commands creates our Virtual Python Environment, activates it, updates our Python package installer, and finally installs the new CLI.

If you would like to graduate from a Python Virtual Environment and move the CLI to your path, you can do so with a sudo install (You have to be careful here not to cause permission issues with packages):

sudo python3 -m pip install influxdb3-python-cli

Creating a CLI config

The first thing you want to do is create a connection config. This feature acts like the current InfluxDB influx CLI by saving your connection credentials for InfluxDB to use later.

influx3 create config \
--name="poke-dex" \
--database="pokemon-codex" \
--host="us-east-1-1.aws.cloud2.influxdata.com" \
--token="<your token>" \
--org="<your org ID>"

–name	Name to describe your connection config. This must be unique.
–token	This provides authentication for the client to read and write from InfluxDB Cloud Serverless or Dedicated. Note: you need a token with read and write authentication if you wish to use both features.
–host	InfluxDB host — this should only be the domain without the protocol (https://)
–org	Cloud Serverless still requires a user’s organization ID for writing data to 3.0. Dedicated users can just use an arbitrary string.
–database	The database you wish to query from and write to.

Config commands

Config commands also exist to activate, update, delete, and list current active configs:

`influx3.py config update --name="poke-dex" --host="new-host.com"`	The update subcommand updates an existing configuration. The --name parameter is required to specify which configuration to update. All other parameters (--host, --token, --database, --org, --active) are optional.
`influx3.py config use --name="poke-dex"`	The use subcommand sets a specific configuration as the active one. The --name parameter is required to specify which configuration to use.
`influx3.py config delete --name="poke-dex"`	The delete subcommand deletes a configuration. The --name parameter is required to specify which configuration to delete.
`influx3.py config list`	The list subcommand lists all the configurations.

Writing and querying

You can use the CLI to either directly call the application, followed by the commands you wish to run, or run it through an interactive REPL. I personally believe the REPL approach provides a better flow, so let’s demo some of the features.

Once you created your config you simply enter the following to activate the REPL:

influx3

Which leads to:

influx3
InfluxDB 3.0 CLI.

(>)

Query

Let’s first take a look at the query options. Within the REPL you have 3 query options: SQL, InfluxQL, and chatGPT (more on this later). Let’s drop into the SQL REPL and run a basic query against the Trainer data we generated in the previous blog:

InfluxDB 3.0 CLI.

(>) sql
(sql >) SELECT * FROM caught

Now I wouldn’t normally recommend querying without some form of time-based WHERE clause, but I wanted to highlight how the CLI can handle large datasets. It uses mode = chunk from the Python Client Library to break large datasets into manageable Arrow batches. From there we have three options.

We can either hit TAB to see the next portion of data, if one exists.
Press F to save the current Arrow batch to a file type of our choosing (JSON, CSV, Parquet, ORC, Feather).
Press CTRL-C to return back to the SQL REPL.

Let’s take a look at the option 2:

| 3961 |       82 | Venusaur                  |        83 |   80 | 0003 |      12 |     7 |      80 | 2023-07-06 13:41:36.588000 | ash       | Grass    | Poison   |
| 3962 |       64 | Dratini                   |        45 |   41 | 0147 |       6 |     7 |      50 | 2023-07-06 14:30:32.519000 | jessie    | Dragon   |          |

Press TAB to fetch next chunk of data, or F to save current chunk to a file
Enter the file name with full path (e.g. /home/user/sample.json): ~/Desktop/all-trainer-data.csv
Data saved to ~/Desktop/all-trainer-data.csv.

Here is a sample of the CSV file created:

"attack","caught","defense","hp","id","level","num","speed","time","trainer","type1","type2"
49,"Bulbasaur",49,45,"0001",12,"1",45,2023-07-06 14:30:41.886000000,"ash","Grass","Poison"
62,"Ivysaur",63,60,"0002",7,"1",60,2023-07-06 14:30:32.519000000,"ash","Grass","Poison"
62,"Ivysaur",63,60,"0002",8,"1",60,2023-07-06 14:30:38.519000000,"ash","Grass","Poison"

Once we reach the end of our dataset, it prompts us to press ENTER to drop back into the SQL REPL. Just remember if you feel like you’re pressing TAB forever, you can always drop out of the query with CTRL-C.

Now, let’s look at a more interesting example with the InfluxQL REPL:

(sql >) exit
(>) influxql
(influxql >) SELECT count(caught) FROM caught WHERE time > now() - 2d GROUP BY trainer
|    | iox::measurement   | time                | trainer   |   count |
|---:|:-------------------|:--------------------|:----------|--------:|
|  0 | caught             | 1970-01-01 00:00:00 | ash       |     625 |
|  1 | caught             | 1970-01-01 00:00:00 | brock     |     673 |
|  2 | caught             | 1970-01-01 00:00:00 | gary      |     645 |
|  3 | caught             | 1970-01-01 00:00:00 | james     |     664 |
|  4 | caught             | 1970-01-01 00:00:00 | jessie    |     663 |
|  5 | caught             | 1970-01-01 00:00:00 | misty     |     693 |

(influxql >) SELECT count(caught) FROM caught WHERE time > now() - 2d  GROUP BY time(1d),trainer ORDER BY time
|    | iox::measurement   | time                | trainer   |   count |
|---:|:-------------------|:--------------------|:----------|--------:|
|  0 | caught             | 2023-07-05 00:00:00 | ash       |     nan |
|  1 | caught             | 2023-07-06 00:00:00 | ash       |     625 |
|  2 | caught             | 2023-07-07 00:00:00 | ash       |     148 |
|  3 | caught             | 2023-07-05 00:00:00 | brock     |     nan |
|  4 | caught             | 2023-07-06 00:00:00 | brock     |     673 |
|  5 | caught             | 2023-07-07 00:00:00 | brock     |     180 |
|  6 | caught             | 2023-07-05 00:00:00 | gary      |     nan |
|  7 | caught             | 2023-07-06 00:00:00 | gary      |     645 |
|  8 | caught             | 2023-07-07 00:00:00 | gary      |     155 |
|  9 | caught             | 2023-07-05 00:00:00 | james     |     nan |
| 10 | caught             | 2023-07-06 00:00:00 | james     |     664 |
| 11 | caught             | 2023-07-07 00:00:00 | james     |     157 |
| 12 | caught             | 2023-07-05 00:00:00 | jessie    |     nan |
| 13 | caught             | 2023-07-06 00:00:00 | jessie    |     663 |
| 14 | caught             | 2023-07-07 00:00:00 | jessie    |     144 |
| 15 | caught             | 2023-07-05 00:00:00 | misty     |     nan |
| 16 | caught             | 2023-07-06 00:00:00 | misty     |     693 |
| 17 | caught             | 2023-07-07 00:00:00 | misty     |     178 |

We will save this one as a Parquet file for later.

Write

Moving on from using the CLI for querying, let’s talk about the write functionality. Now, this feature set isn’t as fleshed out as I would like it to be but it covers the basics. We can drop into the write REPL and write data to InfluxDB using line protocol like so:

(influxql >) exit
(>) write
(write >) caught,id=0115,num=1,trainer=brock attack=125i,caught="KangaskhanMega Kangaskhan",defense=100i,hp=105i,level=13i,speed=100i,type1="Normal" 1688741473083000000

Next let’s have a look at the write_file feature. For this we need to drop out of the REPL entirely and use flag commands when calling ‘influx3’. Let’s load our count results into a new table:

(write >) exit
(>) exit

Exiting …

influx3 write_file --help
usage: influx3 write_file [-h] --file FILE [--measurement MEASUREMENT] --time TIME [--tags TAGS]

options:
  -h, --help            show this help message and exit
  --file FILE           the file to import
  --measurement MEASUREMENT
                        Define the name of the measurement
  --time TIME           Define the name of the time column within the file
  --tags TAGS           (optional) array of column names which are tags. Format should be: tag1,tag2

influx3 write_file --file ~/Desktop/count.parquet --time time --tags trainer --measurement summary

Here is the result:

(influxql >) SELECT count, trainer, time  FROM summary
|    | iox::measurement   | time                |   count | trainer   |
|---:|:-------------------|:--------------------|--------:|:----------|
|  0 | summary            | 2023-07-05 00:00:00 |     nan | ash       |
|  1 | summary            | 2023-07-05 00:00:00 |     nan | brock     |
|  2 | summary            | 2023-07-05 00:00:00 |     nan | gary      |
|  3 | summary            | 2023-07-05 00:00:00 |     nan | james     |
|  4 | summary            | 2023-07-05 00:00:00 |     nan | jessie    |
|  5 | summary            | 2023-07-05 00:00:00 |     nan | misty     |
|  6 | summary            | 2023-07-06 00:00:00 |     625 | ash       |
|  7 | summary            | 2023-07-06 00:00:00 |     673 | brock     |
|  8 | summary            | 2023-07-06 00:00:00 |     645 | gary      |
|  9 | summary            | 2023-07-06 00:00:00 |     664 | james     |
| 10 | summary            | 2023-07-06 00:00:00 |     663 | jessie    |
| 11 | summary            | 2023-07-06 00:00:00 |     693 | misty     |
| 12 | summary            | 2023-07-07 00:00:00 |     148 | ash       |
| 13 | summary            | 2023-07-07 00:00:00 |     180 | brock     |
| 14 | summary            | 2023-07-07 00:00:00 |     155 | gary      |
| 15 | summary            | 2023-07-07 00:00:00 |     157 | james     |
| 16 | summary            | 2023-07-07 00:00:00 |     144 | jessie    |
| 17 | summary            | 2023-07-07 00:00:00 |     178 | misty     |

Experimental feature (ChatGPT)

So with chatGPT and OpenAI being all the rage these days, I looked to see if their Python package could benefit the CLI. Interestingly it does… Because InfluxDB has been open source since its inception, chatGPT has become pretty well-versed in building InfluxQL queries. Take a look at this example:

(chatgpt >) give me a list of the top 10 caught with an attack higher than 100 from caught
Run InfluxQL query: SELECT * FROM caught WHERE attack > 100 LIMIT 10
|    | iox::measurement   | time                       |   attack | caught                    |   defense |   hp |   id |   level |   num |   speed | trainer   | type1    | type2   |
|---:|:-------------------|:---------------------------|---------:|:--------------------------|----------:|-----:|-----:|--------:|------:|--------:|:----------|:---------|:--------|
|  0 | caught             | 2023-07-06 13:09:36.095000 |      110 | Dodrio                    |        70 |   60 | 0085 |      19 |     1 |     100 | jessie    | Normal   | Flying  |
|  1 | caught             | 2023-07-06 13:09:36.095000 |      125 | Pinsir                    |       100 |   65 | 0127 |       6 |     1 |      85 | brock     | Bug      |         |
|  2 | caught             | 2023-07-06 13:10:53.995000 |      130 | CharizardMega Charizard X |       111 |   78 | 0006 |       6 |     1 |     100 | brock     | Fire     | Dragon  |
|  3 | caught             | 2023-07-06 13:10:53.995000 |      150 | BeedrillMega Beedrill     |        40 |   65 | 0015 |      12 |     1 |     145 | jessie    | Bug      | Poison  |
|  4 | caught             | 2023-07-06 13:10:53.995000 |      102 | Nidoking                  |        77 |   81 | 0034 |      20 |     1 |      85 | gary      | Poison   | Ground  |
|  5 | caught             | 2023-07-06 13:10:53.995000 |      105 | Primeape                  |        60 |   65 | 0057 |      16 |     1 |      95 | misty     | Fighting |         |
|  6 | caught             | 2023-07-06 13:10:53.995000 |      120 | Golem                     |       130 |   80 | 0076 |       8 |     1 |      45 | ash       | Rock     | Ground  |
|  7 | caught             | 2023-07-06 13:10:53.995000 |      105 | Muk                       |        75 |  105 | 0089 |       5 |     1 |      50 | brock     | Poison   |         |
|  8 | caught             | 2023-07-06 13:10:53.995000 |      105 | Muk                       |        75 |  105 | 0089 |      19 |     1 |      50 | james     | Poison   |         |
|  9 | caught             | 2023-07-06 13:10:53.995000 |      105 | Muk                       |        75 |  105 | 0089 |      16 |     2 |      50 | james     | Poison   |         |

This feature currently only uses ChatGPT 3.5 and requires an OpenAPI token. If you would like instructions on how to use this feature, check out this part of the README.

Future hopes

The future is bright for the Python CLI as our development team pushes forward with tooling for InfluxDB 3.0. For now, the scope is to keep it as a bolt-on tool for Python developers and those who want an easily extendable CLI. Here is my current laundry list for the project:

Feature	Status
Improve OpenAI functionality: Upgrade to chatgpt 4 Add call functions Extend to SQL	TO DO
Find a better way to package and distribute. Currently looking into Pyinstaller as an option.	TO DO
Extended write functionality.	TO DO
Provide post query exploration support (Pandas functions)	TO DO
Integrate delta sharing	TO DO

Wrapping up

So there you have it, Part 2 done and dusted. I really enjoyed writing this blog series on both the Python Client Library and CLI. Having such a heavy hand in the inspection makes writing about them far more exciting and easy. I hope these blogs inspire you to join our new community-based libraries and tooling. If you want to chat about how to get involved, you can reach me via Slack or Discourse.

Client Library Deep Dive: Python (Part 1)

Jay Clifford (InfluxData) — Wed, 26 Jul 2023 07:35:00 +0000

Working with the new InfluxDB 3.0 Python CLI and Client Library

Community Client libraries are back with InfluxDB 3.0. If you would like an overview of each client library then I highly recommend checking out Anais’s blog on their status.

In this two-part blog series, we do a deep dive into the new Python Client Library and CLI. By the end, you should have a good understanding of the current features, how the internals work, and my future ideas for both projects. From there my hope is that it gives you the opportunity to contribute to, and have your say in their future.

In this post (Part 1), we will focus primarily on the Client Library because it underlies the Python CLI.

If you prefer, you can watch this tutorial in video form.

Python client library

So, let’s start off with the Python client library. The scope was simple: build a library that could write to and query InfluxDB 3.0. Because the write endpoint didn’t change inInfluxDB 3.0, we could bring forward much of the functionality from the V2 library, such as batch writes, data parsing, point objects, and much more. However, on the query side of things, we had to completely remake it. We wanted to focus on the capabilities of Arrow Flight and bring support for both SQL and InfluxQL-based queries. PyArrow also opened up better ecosystem support for libraries such as Pandas and Polars, but I’ll have more on this later.

Let’s build a simple Python application together that writes and queries InfluxDB 3.0.

Install

To install the client library (I recommend making a Python Virtual Environment first):

$ python3 -m venv ./.venv
$ source .venv/bin/activate
$ pip install –upgrade pip
$ pip install influxdb3-python

This set of commands creates our Virtual Python Environment, activates it, updates our Python package installer, and, finally, installs the new client library.

Creating a client

In this section, we import our newly installed library and establish a client. I also discuss some configuration parameters and the reasoning behind them.

Let’s create a main.py file with the following code:

from influxdb_client_3 import InfluxDBClient3, Point
import pandas as pd
import numpy as np
import datetime

client = InfluxDBClient3( token="",
    host="eu-central-1-1.aws.cloud2.influxdata.com",
    org="6a841c0c08328fb1",
    database="pokemon-codex")

This example shows a minimal configuration for the client. Like previous clients, it requires the following parameters:

token	This provides authentication for the client to read and write from InfluxDB Cloud Serverless or Dedicated. Note: you need a token with read-and-write authentication if you wish to use both features.
host	InfluxDB host — this should only be the domain without the protocol (https://)
org	Cloud Serverless still requires the users’ organization ID for writing data to 3.0. Dedicated users can just use an arbitrary string.
database	The database you wish to query and write from.

I recommend creating a client on a per-database basis, though you can update the _database instance variable if you only want to create one client.

Next, let’s take a look at the advanced parameters of the client:

flight_client_options	This provides access to parameters for the flight query protocol. You can find configuration options here. Example.
write_client_options	This provides access to the parameters used by the V2 write client, which you can find here. Example.
**kwargs	Lastly, this provides access to the parameters used by the V2 client, which you can find here. Example. (gzip compression)

Let’s continue our original example by discussing the write functionality.

Writing data

So now that we established our client, in this section we look at the different methods you can use to write data to InfluxDB 3.0. Most will be familiar to you as they follow the same ingestion method as V2.

Let’s start off with basic point building:

# Continued from the Client's example

now = datetime.datetime.now(datetime.timezone.utc)

data = Point("caught").tag("trainer", "ash").tag("id", "0006").tag("num", "1")\
                                             .field("caught", "charizard")\
                                             .field("level", 10).field("attack", 30)\
                                             .field("defense", 40).field("hp", 200)\
                                             .field("speed", 10)\
                                             .field("type1", "fire").field("type2", "flying")\
                                             .time(now)

try:
    client.write(data)
except Exception as e:
    print(f"Error writing point: {e}")

In this example, you can see we build our line protocol using an instance of the Point class, which then translates into line protocol:

Point,trainer=ash,id=0006,num=1 caught="charizard",level=10i,attack=30i,defense=40i,hp=200i,speed=10i,type1="fire",type2="flying" <timestamp>

You can also format this as an array of points:

data = []
# Adding first point
data.append(
    Point("caught")
    .tag("trainer", "ash")
    .tag("id", "0006")
    .tag("num", "1")
    .field("caught", "charizard")
    .field("level", 10)
    .field("attack", 30)
    .field("defense", 40)
    .field("hp", 200)
    .field("speed", 10)
    .field("type1", "fire")
    .field("type2", "flying")
    .time(now)
)

# Adding second point
data.append(
    Point("caught")
    .tag("trainer", "ash")
    .tag("id", "0007")
    .tag("num", "2")
    .field("caught", "bulbasaur")
    .field("level", 12)
    .field("attack", 31)
    .field("defense", 31)
    .field("hp", 190)
    .field("speed", 11)
    .field("type1", "grass")
    .field("type2", "poison")
    .time(now)
)

You can also write via dictionary encoding and structured data methods. One of my favorite ingest methods is via Pandas DataFrame.

Let’s take a look at an example utilizing this method:

# Convert the list of dictionaries to a DataFrame
caught_pokemon_df = pd.DataFrame(data).set_index('timestamp')

# Print the DataFrame
print(caught_pokemon_df)

try:
    client.write(caught_pokemon_df, data_frame_measurement_name='caught',
             data_frame_tag_columns=['trainer', 'id', 'num'])
except Exception as e:
    print(f"Error writing point: {e}")

This example creates a Pandas DataFrame of our caught Pokemon for this session. We set the index of our dataframe to the timestamp of when the Pokemon was caught and then provide the dataframe plus the following write parameters to the ‘write()’ function:

data_frame_measurement_name	The name of the measurement you wish to write your Pandas DataFrame into.
data_frame_tag_columns	A list of strings containing the column names you wish to make tags.
data_frame_timestamp_column	Use this parameter to set the timestamp column if your index is not set to the timestamp.

Make sure to check out the full example here. You can also find a batching example here.

Writing data from a file

A much-requested feature of the previous client library was more ways to upload and parse different file data formats. Leveraging the utilities of PyArrow, we can now support the upload of files in the following formats:

CSV	Example here.
JSON	Example here.
Feather	Example here.
ORC	Example here.
Parquet	Example here.

Querying data

Now that we wrote some data into InfluxDB 3.0, let’s talk about how to query it back out. 3.0 provides a fully supported Apache Arrow Flight endpoint, which allows users to query using SQL or InfluxQL.

Let’s first take a look at a basic time series query in both SQL and InfluxQL;

from influxdb_client_3 import InfluxDBClient3
import pandas as pd

client = InfluxDBClient3(
    token="",
    host="eu-central-1-1.aws.cloud2.influxdata.com",
    org="6a841c0c08328fb1",
    database="pokemon-codex")

sql = '''SELECT * FROM caught WHERE trainer = 'ash' AND time >= now() - interval '1 hour' LIMIT 5'''
table = client.query(query=sql, language='sql', mode='all')
print(table)

influxql = '''SELECT * FROM caught WHERE trainer = 'ash' AND time  > now() - 1h LIMIT 5'''
table = client.query(query=influxql, language='influxql', mode='pandas')
print(table)

As you can see in this example we used the same client to query both with InfluxQL and SQL. Let’s take a quick look at the query parameters to see how they shape our returned result.

query	This parameter currently accepts the string literal of your SQL or InfluxQL query. We hope to add prepared statements to this soon.
language	This parameter accepts a string literal of either ‘sql’ or ‘influxql’
mode	There are currently 5 return modes: 1. ‘all’: this returns all the data queried as a PyArrow Table 2. ‘pandas’: Returns all data as a Pandas DataFrame 3. ‘chunk’: Returns a flight reader so a user can iterate through large queries in smaller sample sizes (see example) 4. ‘reader’: Attempts to convert the stream to a RecordBatchReader 5. ‘schema’: returns the query payload schema

Future hopes

Rome wasn’t built in a day, and there are plenty of quality-of-life improvements and new features to add. Here is a table outlining a few:

Feature	Status
Merge the Write API from the V2 Client to remove the external library dependency.	In progress
Prepared Statements for queries	TO DO
Arrow table writer for InfluxDB	TO DO
Improve Polars support	TO DO
Integrate delta sharing	TO DO

Try it out for yourself

We built the foundations of what I hope will be a great community-driven client library for InfluxDB 3.0 in Python. My call to action is if you haven’t already done so, try out the library and put it through its paces. There are so many edge cases we might not be aware of and we won’t find those without community help. I am eagerly awaiting issues and feature requests.

The Rebirth of InfluxQL in 3.0: A Quick Start Guide to Configuration and Usage

Jay Clifford (InfluxData) — Fri, 30 Jun 2023 07:35:00 +0000

If we turn the clocks back to September 2013, we released InfluxQL alongside InfluxDB. InfluxQL is a SQL-like query language, specifically designed to query time series data. For many of our users, InfluxQL still remains the primary way they interact with InfluxDB. Based on this feedback, InfluxQL has been reborn in InfluxDB 3.0 alongside native support for the SQL query language.

So what do I mean by reborn? Well, in case you didn’t know, we built InfluxDB 3.0 on three key open-source projects:

Apache DataFusion acts as the foundational query engine for InfluxDB 3.0, providing our native SQL support. Our engineers extended the query engine to natively support InfluxQL as well. This allows developers to leverage the full performance of Apache Arrow Flight while using InfluxQL-based queries.

In this blog, we will look at how you can take advantage of InfluxQL via our new v3 client libraries. We will discuss how to configure the v1 InfluxQL API for Serverless and Dedicated, which provides backward compatibility for InfluxQL plugins, such as Grafana and NodeRed.

v3 Client Libraries

We currently have five v3 community-based client libraries:

Client Library	Status	Query Languages
C#	Ready	SQL, InfluxQL
Go	Ready	SQL, InfluxQL
Python	Ready	SQL, InfluxQL
Java	Ready	SQL, InfluxQL
JavaScript	Ready	SQL, InfluxQL

Each of these client libraries support writing and querying with InfluxDB. I highly recommend checking out this blog if you would like to deep dive into their current status.

Let’s take a look at a few client examples utilizing the new InfluxQL query feature.

Python

Let’s start off with a Python example:

import influxdb_client_3 as InfluxDBClient3

client = InfluxDBClient3.InfluxDBClient3(
    token="<INSERT TOKEN>",
    host="eu-central-1-1.aws.cloud2.influxdata.com",
    org="6a841c0c08328fb1",
    database="database")

table = client.query(
    query="SELECT * FROM <MEASUREMENT> WHERE time > now() - 4h",
    language="influxql")

print(table.to_pandas())

As you can see, we create a new client instance called ‘client’. We then call query() which takes the following parameters;

Query: String literal representation of the query you would like to perform. This can be SQL- or InfluxQL-based.
Language: This parameter indicates whether your query string literal is InfluxQL or SQL.

Note: It is important to set our language parameter to influxql because SQL is the default query language.

Go

Next, we look at a Go example:

import (
  "context"
  "encoding/json"
  "fmt"
  "os"
  "github.com/InfluxCommunity/influxdb3-go/influx"
)

url := os.Getenv("INFLUXDB_URL")
token := os.Getenv("INFLUXDB_TOKEN")
database := os.Getenv("INFLUXDB_DATABASE")

// Create a new client using an InfluxDB server base URL and an authentication token
client, err := influx.New(influx.Configs{
    HostURL: url,
    AuthToken: token,
})
// Close client at the end and escalate error if present
defer func (client *influx.Client)  {
    err := client.Close()
    if err != nil {
        panic(err)
    }
}(client)

query := `SELECT * FROM <MEASUREMENT> WHERE time > now() - 4h`;

iterator, err := client.QueryInfluxQL(context.Background(), database, query, nil)

if err != nil {
    panic(err)
}

for iterator.Next() {
    value := iterator.Value()

    fmt.Printf("avg is %f\n", value["avg"])
    fmt.Printf("max is %f\n", value["max"])
}

In this example, we follow a similar practice to the Python client library except instead of using a language parameter we use the client.QueryInfluxQL function.

JavaScript

Finally, let’s consider a JavaScript example:

import {InfluxDBClient, Point} from '../index' // replace with @influxdata/influxdb3-client in your project

type Defined<T> = Exclude<T, undefined>

/* allows to throw error as expression */
const throwReturn = <T>(err: Error): Defined<T> => {
  throw err
}

async function main() {
  // Use environment variables to initialize client
  const url = 'INFLUXDB_URL'
  const token = 'INFLUXDB_TOKEN'
  const database = 'INFLUXDB_DATABASE'

  // Create a new client using an InfluxDB server base URL and an authentication token
  const client = new InfluxDBClient({url, token})

   // Prepare flightsql query
    const query = `SELECT * FROM <MEASUREMENT> WHERE time > now() - 4h`
    // Execute query
    const queryResult = await client.query(database, query, ‘influxql’)

    for await (const row of queryResult) {
      console.log(`avg is ${row.get('avg')}`)
      console.log(`max is ${row.get('max')}`)
    }
  } catch (err) {
    console.error(err)
  } finally {
    await client.close()
  }
}

main()

Like the Python library, we provide credentials to instantiate the client. We call client.query(), including influxql as the queryType parameter.

v1 InfluxQL API

We covered the client libraries, which make use of Arrow Flight to communicate with InfluxDB 3.0. This works great for building new applications but doesn’t provide a universal communication method for current InfluxQL applications and plugins because they operate using the InfluxDB API. This is where the v1 API comes into play.

The v1 API has a v1 query endpoint, which takes the incoming query request, passes it to the query scheduler, and then runs the query. From there the result is passed back to the v1 API endpoint and then returned to complete the API request.

Note: As of writing this blog there is currently a distinct difference in how the V1 API is set up between Serverless and Dedicated.

InfluxDB Cloud Dedicated

Each database created is inherently compatible with the v1 API due to the v1 API bridge.

InfluxDB Cloud Serverless

Serverless currently requires the creation of DBRP mappings to initialize the execution of v1 API InfluxQL queries against InfluxDB 3.0 databases.

Next, let’s take a look at how to configure the DBRP mappings for Serverless.

Configure the bridge with DBRP (Serverless)

Note: Always check the documentation for updates because this feature is in active development. Cloud Dedicated users can skip this step!

Currently, you must manually create DBRPs for each of the databases you wish to use via the v1 API Bridge. There are two ways you can do this:

InfluxDB CLI: With this method, you must have the InfluxDB CLI installed on your host computer. Make sure that you also configure an initial configuration profile, which points to your InfluxDB 3.0 instance. Next run the following command:
```
influx v1 dbrp create \
  --token API_TOKEN \
  --db DATABASE_NAME \
  --rp RETENTION_POLICY_NAME \
  --bucket-id BUCKET_ID \
  --default
```

API token to authenticate. We recommend setting your token to your active InfluxDB connection configuration in the influx CLI, so you don’t have to add these parameters to each command. To set up your active InfluxDB configuration, see influx config set.
database name to map
retention policy name to map
Bucket ID to map to
Default — this flag sets the provided retention policy as the default retention policy for the database.

InfluxDB API: If you wish to interact with the API directly, you can do this via the following curl request:

curl --request POST https://us-west-2-1.aws.cloud2.influxdata.com/api/v2/dbrps \
  --header "Authorization: Token API_TOKEN" \
  --header 'Content-type: application/json' \
  --data '{
        "bucketID": "BUCKET_ID",
        "database": "DATABASE_NAME",
        "default": true,
        "orgID": "ORG_ID",
        	   "retention_policy": "RETENTION_POLICY_NAME"
     		 }'

As you can see, curl requires the same parameters as the CLI. If you have a lot of databases you want to map in Serverless then I highly recommend setting up the InfluxDB CLI and utilizing this bash script:

#!/bin/bash

# Run influx bucket list and parse the output
influx bucket list | awk '
BEGIN {
    # Skip the header line
    getline
}
{
    # Extract the values
    bucket_id = $1
    database_name = $2

    # Construct and run the influx v1 dbrp create command
    cmd = "influx v1 dbrp create --db " database_name " --rp " database_name " --bucket-id " bucket_id " --default"
    system(cmd)
}'

This will map each database/bucket to its own DBRP.

Grafana InfluxQL datasource

Now that we created our first v1 mapping, let’s utilize it with the InfluxQL datasource in Grafana. Let’s take a look at the configuration in two stages:

Stage 1: Authentication

At this stage of the plugin configuration, we must modify three parameters (this excludes creating a Name and specifying InfluxQL as your Query Language).

URL: Make sure to add your protocol and domain to this form. An example would look like: https://eu-central-1-1.aws.cloud2.influxdata.com

Basic auth: Toggle this to true.

User/Password: The username can be any string; it is not used for authentication purposes but cannot be empty. Password must be an InfluxDB API token that has enough privileges to query from the database you are going to use.

Stage 2: Database details

In this final stage, you only have one parameter to change. Note that the configuration doesn’t use the username and password parameters.

Database: Specify the name of the database you wish to query from.

Node-RED Plugin

Let’s take a look at one more example — the node-red-contrib-influxdb plugin for Node-RED. This contribution from one of our community members makes InfluxDB accessible for both querying and writing within Node-Red. Using the v1 API, we can make use of this plugin once again. Let’s split the setup into two stages:

Stage 1: Authentication

Like with the Grafana data source, we need to configure our connection and authentication to the v1 bridge first. To do this, we have a few parameters to configure:

Version: Make sure this is set to 1.X

Host & Port: Strip your domain URL of any protocol (https://) and specify port ‘443’.

Database: Name of the database to query from.

User/Password: The username can be given any string; it is not used for authentication purposes but cannot be empty. Password must be an InfluxDB API token that has enough privileges to query from the database you are going to use.

Enable Secure: Make sure to enable SSL/TLS

Stage 2: Query details

Now you may form your query in Node-RED. There are only two parameters to note:

Time Precision: Make sure to set this parameter appropriately based on the timestamp stored.

Query: Provide your InfluxQL query.

Key takeaways

InfluxQL is back and better than ever. When using the v3 client libraries, InfluxQL power users can continue utilizing the language while reaping the performance gains of InfluxDB 3.0 and the vectorized, columnar DataFusion query engine. The v1 API also provides a much-needed stepping stone to backward compatibility for well-used ecosystem products, such as Grafana and Node-RED.

Now for some hard truths — InfluxQL is still under active development, so there isn’t yet a like-for-like feature representation for v1 InfluxQL. My advice is to check out the InfluxQL reference documentation and see what is currently possible. At present, it supports most core querying functions or will support them soon. My call to action is to start pointing your InfluxQL applications at InfluxDB 3.0 and tell us about your experience in our community. In later blogs, we will deep-dive into some of the core InfluxQL functions.