InfluxData Blog - Susannah Brodnitz

The Immutability of Time Series Data

Susannah Brodnitz (InfluxData) — Wed, 14 Dec 2022 07:00:00 +0000

This article was originally published in The New Stack and is reposted here with permission.

Time series data often comes in large volumes that need to be handled carefully to produce insights in near real time.

We’re constantly moving through time. The time it took you to read this sentence is now forever in the past, unchangeable. This leads to something unique about data with a time dimension: It can only go in one direction. Time series data is different from other data for many reasons. It often comes in large volumes that need to be handled carefully to produce insights in near real time. This blog post focuses on the unchangeable, immutable nature of time series data.

The past is the past

In our world, time is immutable, which means once it’s in the past, it can’t be changed, like an immutable object in programming. In a perfect world, data reflects that; you can’t change time series data any more than you can rewind the clock. Data should reflect reality, but sometimes bad data points get written into a database. Bad data points don’t reflect reality either, and it makes sense in this case to delete those points.

Deletion needs to be handled with care.

You need a database that can delete points without shifting other points around.
You might also need to edit the historical record by adding late-arriving points.

There’s a balance to strike so you’re not constantly rewriting the past in such a way that your data loses meaning, but you can still make necessary changes that enhance the context presented by time. When deciding if it makes sense to make an edit, consider whether the edit brings the data closer to reflecting reality or further from it.

Time keeps moving

The other thing about time is that it never stops. The present is always continuously moving forward. Because time is always moving, time series data updates continuously. When you think of a database, you might think of a place for storage where you write data and later read that data without changing it very often. A time series database is constantly being changed and updated because time is always moving and changing.

You can’t collect data with the infinite precision of reality, but you can choose the level of precision that makes sense for your application. For example, averages are one of the most common and useful calculations. If you’re working with data that isn’t a time series, you might average the number of people per square mile in a state. With time series data, you might average the number of people entering a building every hour. The difference here is that at each moment, the start and end of the last hour changes. Here’s some example code for this sort of calculation:

from(bucket: "sample")
    |> range(start: 2022-01-01, stop: 2022-01-31)
    |> filter(fn: (r) => r["_measurement"] == "foot_traffic")
    |> aggregateWindow(column: "number_of_people",every: 1h, fn: mean, createEmpty: false)
    |> yield(name: "running mean")

When you take a moving average, you calculate a new average at specified intervals so you can see how your calculation changes over time, resulting in a new time series. You need to consider your data set to know what sort of interval makes sense. If you choose too broad an interval, you lose information and context, but if you choose one that is too precise, you’ll have windows without any data points, and your results will drop to zero in a way that doesn’t make sense and isn’t helpful.

The context of time

No matter how close to real time your data architecture is, there will be some infinitesimal amount of lag between when your data is collected and when it lands in a database, ready to be queried. If you have automated queries or processing set up, this can skew your results. For example, if you calculate the mean of a metric from the last five minutes, the data that arrived at the database in the last five minutes might not include the full set of measurements that were taken at the edge in the last five minutes. InfluxDB lets you handle this with task offsets.

You can schedule tasks to run calculations like this while including some extra buffer time to allow all data to arrive in the database first. This is important to preserve the full context of when each point was collected. Telegraf, InfluxData’s open source data-collection agent, also allows for offsets.

There are many reasons to downsample data. Sometimes you don’t have enough storage space for the full raw data set. Sometimes an averaged signal cuts through the noise and gives you more valuable information. When you take an average, some information is lost and some new information is added. Averages aren’t the only way of downsampling either. Sometimes instead of maintaining the shape of the data, it might make more sense to count the number of times a metric goes above a set threshold.

Whatever downsampling method you use, everything you do should be intentional so you don’t lose data you realize later is important. If you aren’t careful while downsampling and aren’t handling all your timestamps properly, downsampling can skew your time series. InfluxDB is built to handle downsampling using a variety of tools and processes, and it creates multiple backup copies of your data in InfluxDB Cloud, so you don’t accidently delete a point you need. In order to keep as much context as possible InfluxDB also supports nanosecond precision. Here’s some example code for downsampling in Flux:

from(bucket: "sample")
    |> range(start: 2022-01-01, stop: 2022-01-31)
    |> filter(fn: (r) => r["_measurement"] == "foot_traffic")
    |> aggregateWindow(column: "number_of_people",every: 1h, fn: mean, createEmpty: false)
    |> to(bucket: "sample-downsampled")

Of course, time isn’t the only important context to consider, and time series data isn’t the only important kind of data. Information like customer details, location or the version of a machine being used aren’t time series data, but they are important to record. Fortunately, InfluxDB allows you to join these other types of data with time series data to produce deeper insights into your systems and processes.

Time is one of the fundamental building blocks of our reality, and understanding its nature helps you better understand the world and get more useful information out of your data.

How Prescient Devices Uses Time Series Data for IoT Automation

Susannah Brodnitz (InfluxData) — Wed, 07 Dec 2022 07:00:00 +0000

This article was originally published in The New Stack and is reposted here with permission.

Companies need to consider both how fast they can put edge applications into action and update them, and how quickly they can process incoming data.

Industrial processes are becoming increasingly automated as sensors on machines collect a growing amount of data. Much of this data is time-stamped and can help companies improve processes. This large volume of sensor data can become unwieldy if companies don’t manage it properly.

Companies need to be able to handle the life cycle of time series data, from real-time analysis to downsampling of historical data. Many companies use purpose-built data management tools and products to help with this.

Prescient Devices built an industrial Internet of Things (IoT) data management platform, powered by InfluxDB, that helps companies build applications to manage edge devices quickly and easily.

Connecting edge and cloud

Prescient Devices created Prescient Designer, a SaaS platform to help businesses manage distributed edge devices. It has an agent that users install on edge devices and an interface where they build and implement applications. InfluxDB is the backbone of Prescient Designer and handles time-stamped data.

Prescient Devices built the platform using Node-RED, an open source project that has a low-code visual interface. Prescient Designer also uses InfluxDB Cloud, TensorFlow, and Grafana and can integrate with other tools that companies require.

Prescient Edge is Prescient Devices’ edge management software. It uses Node-RED, InfluxDB OSS, TensorFlow, and a runtime system. Prescient Edge deployments can include applications built by users in Prescient Designer. Data from edge devices come in any number of formats. Prescient Devices enables users to collect a wide variety of sensor data, such as temperature, humidity, and acceleration, as well as data from cameras, APIs, and industrial equipment.

Data come into Prescient Designer in two separate streams, one for user data and one for application management and deployment data. Prescient Devices includes a broker to connect Prescient Designer and devices running Prescient Edge.

Their system also supports custom brokers if users need support for compliance requirements like HIPAA or GDPR. For data visualization, Prescient Designer leverages Grafana dashboards to provide users insight into their applications in real time.

Low-code framework

Prescient Designer is based on Node-RED and lets users create applications using minimal code. This creates a lower barrier to entry and makes it easier for companies to develop applications and solutions quickly.

This approach also enables companies to include subject-matter experts who aren’t developers more directly in building applications. So, key stakeholders like data engineers, system integrators, and other innovators can all use Prescient Devices to build better edge-to-cloud data solutions, without having to be experienced coders. Prescient Designer’s visual workspace also lets users get a view of their whole system in one place.

Saving time

The two key issues with data that Prescient Devices helps solve are scale and speed. IoT environments have huge volumes of data coming from hundreds of devices that each process a significant amount of data. Companies need to consider both how fast they can put edge applications into action and update them and how quickly they can process incoming data.

Prescient Devices initially used Telegraf, InfluxData’s open source plug-in–driven agent for collecting metrics, to collect and process data from edge devices. They ultimately chose to use InfluxDB Cloud as their time series database because it supports Node-RED and runs well on edge devices with limited resources. Many of Prescient Devices’s customers already used InfluxDB and specifically requested InfluxDB support in the platform.

Prescient Designer lets IoT companies create distributed systems quickly. Users can build applications within weeks and update and deploy changes to those applications within hours to seconds. Because users can build applications within a visual interface, experts can build applications even if they aren’t developers. This lets IoT manufacturers automate their systems quickly and get better insights from their data.

TL;DR Python, Pandas Dataframes, and InfluxDB

Susannah Brodnitz (InfluxData) — Wed, 09 Nov 2022 07:00:00 +0000

InfluxDB has over a dozen client libraries so developers can get started more easily and program in the language they’re most comfortable with. One of our most popular options is the Python client library. InfluxDB supports not just Python but pandas, a tool popular with data scientists for analyzing and manipulating data. You can use the client library to output data from InfluxDB into a DataFrame format pandas can ingest, and you can write pandas DataFrames directly to InfluxDB. This makes it simple to incorporate InfluxDB into your data science applications and take advantage of real-time monitoring and alerting.

Querying data from InfluxDB to a Pandas Dataframe

To query and return a pandas DataFrame in InfluxDB you first need to install the Python client library. To set it up you need to download that library into your Python project and set your credentials, including your URL, token, and the name of the bucket you want to send data to and read data from. Once you’ve set up your credentials you can query data.

Here’s an example query:

query= ‘’’
from (bucket: “system”)
|> range(start: -5m, stop: now())
|> filter(fn: (r) => r._measurement = “cpu”)
|> pivot(rowKey: [“_time”], ColumnKey: [“tag”], ValueColumn: “value”)
|> keep(columns: [“_time”, “usage_user”, “cpu”])
‘’’
client = InfluxDBClient(url, token, org, debug = false)
system_stats = client.query.api().query_data_frame()
display(system_stats.head())

This query is written in Flux, InfluxDB’s scripting and query language. It asks for data from the bucket “system” from the past five minutes and filters on the measurement “cpu.” The two most important functions to keep in mind when you’re working with InfluxDB and pandas are pivot and keep. Pivot changes the data format to a column. Pandas dataframes are two-dimensional tables so pandas functions and tools expect data to be in that format and you’ll need to pivot the data you query from InfluxDB. Keep tells the query what data to keep in this column and gets rid of any other columns you don’t want to include in the output pandas DataFrame.

After you’ve written your query, you need to call the InfluxDB client with your credentials and use the query API with the query_data_frame() function to get the DataFrame format. Then you can display the data inside of the pandas DataFrame with .head.

Writing data from a Pandas Dataframe to InfluxDB

You can also write data from a pandas dataframe into InfluxDB. Here’s some example code that does that:

system_stats.drop(columns = [‘result’, ‘table’, ‘start’, ‘stop’])
system_stats.set_index(“_time”)
_write_client.write(bucket.name, record = system_stats)

Any data that you don’t remove using the drop function from your pandas DataFrame becomes a field when you write the data to InfluxDB. In this example query we’ve removed columns we don’t want to send as fields to InfluxDB because we want to make them tags. You also need to set your index, which will be your timestamp in InfluxDB. Then you need to use the Python client library write function to write the data to InfluxDB. It takes in a bucket name and the record you want to write to that bucket. You also need to set up credentials for the client, as in the first example.

One thing to consider is the timestamp. Pandas comes with a datetime format. You can use that or you can use one of the following time formats:

“2018-10-26”, “2018-10-26 12:00”, or “2018-10-26 12:00:00-05:00”

Flux is a powerful analytics language, but you might want to use pandas because it’s very popular within the Python community. It has a lot of support and many people are already familiar with it and comfortable working with DataFrames. Combining pandas with the InfluxDB Python client library lets you easily incorporate InfluxDB into your applications to get the benefits of a purpose-built time series database.

To learn more about using pandas DataFrames with InfluxDB you can watch our Meet the Developer video on this topic.

Intro to InfluxDB IOx

Susannah Brodnitz (InfluxData) — Thu, 03 Nov 2022 07:00:00 +0000

Last week InfluxData announced IOx, the new time series engine for InfluxDB. We’ve revisited the core of our database to achieve big things with the underlying technology. Users can expect higher performance and more options for querying data. Here’s a quick intro to some of the most exciting things coming with InfluxDB IOx.

Unbounded cardinality

At InfluxDB when we say cardinality we mean the number of unique measurement, tag set, and field key combinations in a bucket. In the past, we discouraged using large numbers of tags or tags that contained unbounded data in InfluxDB because it would slow down performance. This limited use cases, like observability and tracing, that relied on high-cardinality data like logs and traces. InfluxDB IOx removes cardinality limits. Now users can write in data with infinite cardinality and monitor and query their time series data along any dimension they want without impacting performance.

Real-time analytics

InfluxDB IOx is a columnar time series database. Columnar databases represent their data as tables and let you execute queries very quickly and at scale. Basing our new engine on a columnar database solves performance problems in several ways. Per-column compression helps with space on disk. Dictionaries make sure you’re not repeatedly storing strings when you have tag values that apply to many timestamps and run length encoding helps with efficient storage for legitimate repeated values. Vectorized execution lets you organize data so that the CPU can run through it very quickly. And queries are parallelized to run faster.

One of the most important parts of the new engine is partitioning. With IOx, time series data is broken into partitions based on time (such as by days) and also by tags (such as regions). This lets you easily filter out sections of data that don’t fall into a query’s time range or other specification. Queries operate on all rows from included partitions one-by-one, and because of vectorized execution, IOx lets you query around 1 billion rows per second per core. This partitioning and incredibly fast query response means you can have large numbers of tags and still get real-time results to build dashboards and monitoring and alerting systems.

SQL support

InfluxDB IOx was written in the Rust programming language and uses Apache Parquet files for on-disk storage and Apache Arrow for operations between components. Apache Arrow is an in-memory specification for columnar data that makes analytical queries very fast. IOx also uses the DataFusion library, a native SQL query engine, as its parser, planner, optimizer, and execution engine. This means that for the first time InfluxDB supports the PostgreSQL dialect and wire protocol allowing you to connect to third-party libraries and BI tools like PSQL, Grafana, Tableau and Apache Superset. Compatibility is a key focus at InfluxData and this new engine supports many querying options. In addition to the new SQL support, InfluxDB IOx continues to support versions of the API and our own query and scripting languages. In the API layer you can communicate with the new engine using the 1.x or the 2.0 API, and you can query the database using Flux, InfluxQL, SQL, or any of the 12+ client libraries we offer.

You can sign up to be a part of the InfluxDB IOx Beta program here. You’ll get the SQL compatibility, unbounded cardinality, and faster performance within InfluxDB Cloud. If you use OSS or InfluxDB Enterprise you can expect builds next year with the new engine, these compatibility layers, and migration tooling.

Getting Started with Python and Geo-Temporal Analysis

Susannah Brodnitz (InfluxData) — Mon, 31 Oct 2022 07:00:00 +0000

This article was originally published in The New Stack and is reposted here with permission.

Working with geo-temporal data can be difficult. In addition to the challenges often associated with time-series analysis, like large volumes of data that you want real-time access to, working with latitude and longitude often involves trigonometry because you have to account for the curvature of the Earth. That’s computationally expensive. It can drive costs up and slow down programs. Fortunately, InfluxDB’s geo-temporal package is designed to handle these problems.

The package does this with S2 geometry. The S2 system divides the Earth into cells to help computers calculate locations faster. Unlike a lot of other projections, it’s based on a sphere instead of a flat surface, so there are no gaps or overlapping areas. You can choose different levels to vary the size of each cell. With this system, a computer can check how many cells away two points are to get an estimate of their distance.

In a lot of cases, the estimates you can get from S2 calculations are precise enough and much faster for computers than trigonometry. In cases where you need exact answers, using S2 geometry can still speed things up because computers can get rough estimates first and then do only the expensive calculations that are truly needed.

Getting started with an example

For this example, we’re going to calculate the average surface temperature of the ocean over a specified region and at different time windows, as well as the standard deviation. We’re going to be working in InfluxDB’s Python client library in a Jupyter notebook. Here are some other blog posts on these topics.

I used Jupyter notebooks through Anaconda, so the first step for me was to install InfluxDB in Anaconda by typing the following in the command window.

conda install -c conda-forge influxdb

Then, as prompted, I had to update anaconda with

conda update -n base -c defaults conda

The file used in this example is in NetCDF format, so I also had to install the following to read it

conda install -c conda-forge netcdf4

NetCDF files are a common format for scientific data. The data used in this example is the Roemmich-Gilson Argo temperature climatology, which is available here from the second link on the page entitled “2004-2018 RG Argo Temperature Climatology.” This data comes from measurements made by thousands of floats throughout the ocean taking measurements at irregular times and in irregular locations, which are averaged into a gridded product with monthly values on a 1° grid.

In your Jupyter notebook, start by running the following commands to import various packages needed for this example.

import matplotlib.pyplot as plt
import numpy as np
import datetime
import pandas as pd
import influxdb_client, os, time
from influxdb_client import InfluxDBClient, Point, WritePrecision, WriteOptions
from influxdb_client.client.write_api import SYNCHRONOUS
import netCDF4 as nc

Cleaning up the data

Run the following commands to read the file.

file_name = '/filepath/RG_ArgoClim_Temperature_2019.nc'
data_structure = nc.Dataset(file_name)

If you run

print(data_structure.variables.keys())

Then the following should be the output.

dict_keys(['LONGITUDE', 'LATITUDE', 'PRESSURE', 'TIME', 'ARGO_TEMPERATURE_MEAN', 'ARGO_TEMPERATURE_ANOMALY', 'BATHYMETRY_MASK', 'MAPPING_MASK'])

These are the various arrays within the file, which store the data. The ones we’re interested in are LONGITUDE, LATITUDE, TIME, ARGO_TEMPERATURE_MEAN, and ARGO_TEMPERATURE_ANOMALY. To read the data from the file run

lon = data_structure.variables['LONGITUDE']
lat = data_structure.variables['LATITUDE']
time = data_structure.variables['TIME']
temp_mean = data_structure.variables['ARGO_TEMPERATURE_MEAN']
temp_anom = data_structure.variables['ARGO_TEMPERATURE_ANOMALY']

To see the dimensions of each array, run

print(lon.shape)
print(lat.shape)
print(time.shape)
print(temp_mean.shape)
print(temp_anom.shape)

You should see

(360,)
(145,)
(180,)
(58, 145, 360)
(180, 58, 145, 360)

There are 360 longitude values, 145 latitude values and 180 time values. There are also 58 pressure values. Since we’re only interested in the ocean temperature at the surface, which is the lowest pressure, we’re going to subset the array to take the first pressure index.

You will also notice that the mean temperature doesn’t have a time dimension. This dataset separates out temperature into the mean value over the whole time series and the monthly differences from the mean, called anomalies. To get the actual temperature for each month, we just add the mean and anomaly.

If you try to display the value of a random point within the array, you should see

temp_anom[0,0,0,0]

masked_array(data=1.082,
             mask=False,
       fill_value=1e+20,
            dtype=float32)

Right now, each value is within an array as an artifact of being within a netCDF file. Run the following to get simple arrays of values:

lon = lon[:]
lat = lat[:]
time = time[:]
temp_mean = temp_mean[:]
temp_anom = temp_anom[:]

temp_anom[0,0,0,0]

1.082

The unit of time is the number of months since the start of the dataset. To make an understandable date-time vector, run the following.

time_pass=pd.date_range(start='1/1/2004', periods=180, freq='MS')

To create an array of the temperature just at the surface, add together the mean and anomaly arrays at the first pressure index with the following code.

ocean_surface_temp=np.empty((180,145,360))
for itime in range(180):
ocean_surface_temp[itime,:,:]=temp_mean[0,:,:]+temp_anom[itime,0,:,:]

Writing data to InfluxDB Cloud

Now that the data we want is in a nicely formatted array, we can get started sending it to InfluxDB Cloud. To save storage space, for this example we’re not going to upload the whole thing, just 10 years of data from a 10-degree-by-10-degree box in the Atlantic Ocean. The coordinates I picked for this are 15.5°N to 25.5°N and 34.5°W to 44.5°W, or latitude indices 80 to 90 and longitude indices 295 to 305.

To send data to InfluxDB most efficiently, we’re going to create an array of data points. We’re going to call each point ocean_temperature, and that name will be set as its measurement.

To use the geo-temporal package in InfluxDB, you need to send in your data with latitude and longitude as fields, so each of our points will have latitude, longitude and temperature fields. In InfluxDB, if there are two field values at the same time stamp, the next value you upload will overwrite the previous one. This is a problem for us because our data has many values for latitude, longitude and temperature at the same time.

A simple way to prevent data from being overwritten is by giving each point a location tag. In our data, there are many measurements at the same time but no two at the same location and time. You can develop other unique tags for points at the same time you don’t want to be overwritten, as described in more detail here.

points_to_send = []
for itime in range(120):
    for ilat in range(80, 90):
        for ilon in range(295, 305):
            p = Point("ocean_temperature")
            p.tag("location", str(lat[ilat]) + str(lon[ilon]))
            p.field("lat", lat[ilat])
            p.field('lon', lon[ilon])
            p.field('temp', ocean_surface_temp[itime,ilat,ilon])
            p.time(time_pass[itime])
            points_to_send.append(p)

Then in the InfluxDB UI, create a bucket and token from the “load data” sidebar, as shown in these screenshots.

Your org is your email for the account, and the url is the url of your cloud account.

token=token
org = org
url = url
bucket = bucket

Here we set the batch size to 5,000 because that makes things more efficient, as described in the docs here.

with InfluxDBClient(url=url, token=token, org=org) as client:
    with client.write_api(write_options=WriteOptions(batch_size=5000)) as write_api:
        write_api.write(bucket=bucket, record=points_to_send)

Querying the data

Now that the data is in InfluxDB, we can query it. I’m going to use several queries to show you what each command does step by step. First, set up the query API.

client = influxdb_client.InfluxDBClient(url=url, token=token, org=org)
query_api = client.query_api()

Within your Jupyter notebook, each query is a string of Flux code, which you then call with the query API. In all of these cases, we have many options for what to do with the results of our queries. For the first ones, I’m going to print the first few results to make sure they’re working, and then finally we’ll end with a plot.

This query simply gathers all of the latitude, longitude and temperature fields.

query1='from(bucket: "sample_geo")\
  |> range(start: 2003-12-31, stop: 2020-01-01)\
  |> filter(fn: (r) => r["_measurement"] == "ocean_temperature")\
  |> filter(fn: (r) => r["_field"] == "lat" or r["_field"] == "temp" or r["_field"] == "lon")\
  |> yield(name: "all points")'

This query returns latitude.

query2='from(bucket: "sample_geo")\
  |> range(start: 2003-12-31, stop: 2020-01-01)\
  |> filter(fn: (r) => r["_measurement"] == "ocean_temperature")\
  |> filter(fn: (r) => r["_field"] == "lat")\
  |> yield(name: "lat")'

To execute either of these queries, and print the amount of points there are and the first few results, you can run the following, changing what query you’re passing in.

result = client.query_api().query(org=org, query=query2)
results = []
for table in result:
    for record in table.records:
        results.append((record.get_value(), record.get_field()))
print(len(results))
print(results[0:10])

Now we’re going to get started using the geo-temporal package. The geo.shapeData function reformats the data and assigns each point an S2 cell ID. You specify what your latitude and longitude field names are, “lat” and “lon” in this case, and what S2 cell level you want. In this case, I’ve chosen 10, which corresponds to an average of 1.27 kilometers squared. You can read about the cell levels here.

Next we’re going to use the geo.filterRows function to select the region we want to calculate the average temperature of. I’m picking a 150 kilometer circle centered around 20.5°N and 39.5°W, but you can pick any sort of box, circle or polygon as described here.

By default, the data is grouped by s2_cell_id, so to calculate a running mean over this whole region, we have to run the group function and tell it to group by nothing so all the data in the region is grouped together. Then you can use the aggregateWindow function to calculate running means and standard deviations over time windows of your choice.

Putting this all together, the code below calculates and plots the mean over this circle every three months and every year, and the standard deviation every three months, which I’ve put as error bars in the plot below.

query3='import "experimental/geo"\
from(bucket: "sample_geo")\
    |> range(start: 2003-12-31, stop: 2020-01-01)\
    |> filter(fn: (r) => r["_measurement"] == "ocean_temperature")\
    |> geo.shapeData(latField: "lat", lonField: "lon", level: 13)\
    |> geo.filterRows(region: {lat: 20.5, lon: -39.5, radius: 150.0}, strict: true)\
    |> group()\
    |> aggregateWindow(column: "temp",every: 3mo, fn: mean, createEmpty: false)\
    |> yield(name: "running mean")\
    '

query4='import "experimental/geo"\
from(bucket: "sample_geo")\
    |> range(start: 2003-12-31, stop: 2020-01-01)\
    |> filter(fn: (r) => r["_measurement"] == "ocean_temperature")\
    |> geo.shapeData(latField: "lat", lonField: "lon", level: 13)\
    |> geo.filterRows(region: {lat: 20.5, lon: -39.5, radius: 150.0}, strict: true)\
    |> group()\
    |> aggregateWindow(column: "temp",every: 3mo, fn: stddev, createEmpty: false)\
    |> yield(name: "standard deviation")\
    '

query5='import "experimental/geo"\
from(bucket: "sample_geo")\
    |> range(start: 2003-12-31, stop: 2020-01-01)\
    |> filter(fn: (r) => r["_measurement"] == "ocean_temperature")\
    |> geo.shapeData(latField: "lat", lonField: "lon", level: 13)\
    |> geo.filterRows(region: {lat: 20.5, lon: -39.5, radius: 150.0}, strict: true)\
    |> group()\
    |> aggregateWindow(column: "temp",every: 12mo, fn: mean, createEmpty: false)\
    |> yield(name: "running mean")\
    '

result = client.query_api().query(org=org, query=query3)
results_mean = []
results_time = []
for table in result:
    for record in table.records:
        results_mean.append((record["temp"]))
        results_time.append((record["_time"]))

result = client.query_api().query(org=org, query=query4)
results_stddev = []
for table in result:
    for record in table.records:
        results_stddev.append((record["temp"]))
result = client.query_api().query(org=org, query=query5)

results_mean_annual = []
results_time_annual = []
for table in result:
    for record in table.records:
        results_mean_annual.append((record["temp"]))
        results_time_annual.append((record["_time"]))

plt.rcParams["figure.figsize"] = (10,7)
plt.errorbar(results_time,results_mean,results_stddev)
plt.plot(results_time_annual,results_mean_annual)
plt.xlabel("Time")
plt.ylabel("Degrees C")
plt.title("Average Ocean Surface Temperature")

Further resources

Using InfluxDB in Python makes geo-temporal analysis more efficient. I hope this example of the kinds of calculations you can do with this package sparks some ideas for you. This is really the tip of the iceberg. You can also use the package to calculate distances, find intersections, find whether certain regions contain specific points and more. And it can make an even bigger difference in saving computations with more complicated data sets with more points.

The combination of a platform built for time-series data and the S2 cell system is very powerful. For more information, you can read about the Flux geo-temporal package in our docs here and watch our Meet the Developers mini-series on the subject here.

A Sneak Peek at InfluxDB University’s Upcoming Live Telegraf Training

Susannah Brodnitz (InfluxData) — Mon, 24 Oct 2022 07:00:00 +0000

Telegraf, InfluxDB’s open source data collection agent, uses hundreds of plugins to gather data, transform it, and send it where you want it to go. It’s a very powerful tool to incorporate into your data pipeline. If you’ve been wanting to get started with Telegraf but haven’t known where to start, a great way to learn is at InfluxDB University’s live virtual training on November 1st. You can register here for free to learn from experts in a hands-on lab setting. By the time the training is done you’ll be a Telegraf pro and have a digital badge to show for it. This training is part of InfluxDays, our free virtual user conference on November 2-3, 2022. Anyone who signs up for the training will be automatically registered for InfluxDays. Here’s a sneak peek at some of the topics covered in the training.

Understanding Telegraf

During the training you’ll learn about all the things you can do with Telegraf. There are hundreds of plugins for collecting data from sensors, systems, and stacks, but Telegraf goes beyond that. You’ll learn about processor and aggregator plugins that let you filter and transform data before it reaches its endpoint. Processors plugins can do things like print metrics as they pass through Telegraf and add tags to metrics. Aggregator plugins create new metrics, such as running means, standard deviations, or quantiles. You can choose within Telegraf whether to pass through both the original metrics and the aggregates or to drop the originals and just send the aggregate values to your endpoint.

You’ll also get hands-on experience configuring and running Telegraf. The instructors leading this course have designed exercises to give you the skills you need to set up your local environment, configure your cloud environment, and enable plugins to collect and filter data. This training includes several exercises you can complete live, including learning to debug and learning dual writes to send data to two different outputs. The great thing about taking a course live is that you can get your questions answered as they come up, and you can also hear the responses to questions other participants bring up that you might not have thought to ask.

Edge Data Replication

This training will also show you how to use InfluxDB with Edge Data Replication (EDR). Edge Data Replication lets you automatically replicate data from an edge OSS instance of InfluxDB to InfluxDB Cloud. This is great for situations where you want to work with data in detail locally and also in aggregate in the Cloud. For example, a factory floor manager might want to keep tabs on the status of a machine in detail, but a data scientist at the company might want to work with downsampled data from many factories in the Cloud. Edge Data Replication uses a durable queue so it’s also great for situations with intermittent network connectivity. In this training you’ll learn to work with InfluxDB OSS locally and replicate data to InfluxDB Cloud.

If you can’t take the live course but still want to learn more about Telegraf, InfluxDB University has you covered! Data Collection with Telegraf is a free self-paced course you can take to become a Telegraf expert and earn a shareable badge. InfluxDB University also has courses on Flux, InfluxDB Cloud, InfluxDB Enterprise, and more. Don’t forget to register for InfluxDays, There will be some exciting announcements in InfluxDB University in the session on November 3rd. And the Become an InfluxDB Pro in 20 Minutes and Become a Flux Pro are a must see. If you can’t make it live, register for free to get a link to all the on-demand sessions when they become available.

TL;DR InfluxDB’s MQTT Native Collector

Susannah Brodnitz (InfluxData) — Tue, 18 Oct 2022 07:00:00 +0000

There are a lot of ways to get data into InfluxDB, including over a dozen client libraries, hundreds of Telegraf plugins, and Native Collectors. Native Collectors let you directly ingest data from a cloud broker to InfluxDB Cloud so you don’t need to install anything else or write extra code.

This blog will focus on the MQTT Native Collector. MQTT is one of the most popular IoT message brokers because it’s lightweight and works well in environments with limited bandwidth and intermittent network connectivity. The MQTT Native Collector is the fastest way to get your data into InfluxDB so you can analyze it, transform it, create visualizations, and set up alerts. That speed is very important for many IoT applications that rely on real-time analysis to keep smart devices running. To get started with the MQTT Native Collector you just need to have a message broker in place and an InfluxDB Cloud account.

When to use InfluxDB MQTT Native Collector

Because there are so many ways to get data into InfluxDB, it’s helpful to know when to use each method. A Native Collector is useful when you’re really busy and don’t want to go through the additional step of writing software using a client library to ingest data. But in that case you may also want to use Telegraf, and it includes processor and aggregator plugins to transform and downsample your data before it reaches InfluxDB.

However, Telegraf requires you to install it somewhere. If your message broker is in the cloud and you’re using InfluxDB Cloud there might not be a convenient place to install Telegraf, especially if you’re using a third party cloud service. And it can also be more efficient to keep your data pipeline in the cloud rather than needlessly downloading your data onto a server before uploading it back to InfluxDB Cloud. You can sometimes install Telegraf in a virtual machine in your cloud environment, such as if you’re using HiveMQ or Mosquito, but if you’re using a third party cloud-based message broker like HiveMQ Cloud that’s not possible.

The MQTT Native Collector is a great solution for people who want to keep their data in the cloud and get it into InfluxDB quickly. You save on time by getting rid of download or upload speed and bandwidth issues because you don’t need to download data to wherever you’re running Telegraf. You also save compute time associated with running Telegraf on a virtual machine.

Using a client library gives you the most flexibility because the code you write is specifically designed for your application, and that may be a good option if there are complex transformations you need to do on your data. But if you’re trying to ingest raw data directly from IoT devices to InfluxDB and speed matters most, the MQTT Native Collector is the simplest and quickest method.

Configuring & parsing MQTT data with InfluxDB’s Native Collector

It’s simple to configure native collection and get started. First you need to configure the IP address or URL of your MQTT message broker and any authentication parameters. Then you need to set the topic(s) you want to subscribe to. If there are multiple topics you want to subscribe to, you can repeat this process for each topic or you can use wildcards.

Topics generally come in the format my/fav/set/topics separated by tree levels. At any tree level you can use a + to indicate that you want to subscribe to anything from that tree level, such as my/+/set/topics or a # to indicate you want to subscribe to anything from that tree level down, such as my/fav/#.

Finally, you need to set parsing rules. This tells InfluxDB how to read raw data from the MQTT broker and where to put it. Data in an MQTT broker can be in any format, and InfluxDB needs to know the measurement, timestamp, fields, and tags. You don’t need parsing rules if you’re subscribing to a topic where the data is already in line protocol format.

One of the most common data formats is JSON, so InfluxDB has a specific parser already set up for it. If your data is in another format you can use string parsing to let InfluxDB know what variables in your data should be read in as measurements, fields, timestamps, and tags. It’s that simple and can be set up in minutes.

To learn more about InfluxDB’s MQTT Native Collector you can watch our series of videos on this topic.

Part one is an introduction to Native Collectors.

Part two explains when to use Native Collectors.

And part three gives examples for how to set up the MQTT Native Collector.

We’re Going to AWS re:Invent 2022

Susannah Brodnitz (InfluxData) — Mon, 17 Oct 2022 07:00:00 +0000

From November 28–December 3, InfluxData is heading to Las Vegas for AWS re:Invent 2022. We’re super excited to connect with our community in person and take part in this huge event. You can get InfluxDB on AWS and we have some great demos prepared to show how well they work together. Stop by our booth (#3648) in the Data and Analytics Competency Area, to get your questions answered by our experts. We also have a virtual booth if you can’t make it in person.

Watch live demos

Every day of the expo, we will have live demos at our booth showing how you can use InfluxDB to build powerful time series applications. Come check out what we have planned. Ever wondered how much energy it takes to power the coffee needs of an events team? We even have a connected Lego robot for the IoT fans out there. These demos are super fun and a great way to learn about all the built-in developer tools that make InfluxDB so powerful.

Meet with InfluxDB experts

Stop by our booth to meet with InfluxDB pros. These events are exciting for us because we love to talk directly with people like you! If you’re interested in InfluxDB and want to ask how it can help with your time series applications, swing on by. You can book a 1:1 onsite meeting from our event page and get a free IoT weather station. Plus, if you come to the booth and set up a Cloud account and register an OSS instance with us you can win a drone.

You can also stop by the Marketplace Pavilion Wednesday and Thursday at Kiosk #3525-9 to ask any and all questions about running InfluxDB in AWS. Be sure to ask one of our team members how getting InfluxDB through the AWS Marketplace can simplify billing and help meet AWS annual spend commitments.

Interviews you can use

Catch some of InfluxData’s executives on theCUBE and Screaming in the Cloud discussing the latest InfluxDB enhancements. This includes Edge Data Replication, as well as the company’s broader focus on expediting time series data pipelines from edge to cloud.

Win a free conference pass

Enter to win a free full conference pass for AWS re:Invent! Simply visit our contest page to sign up for, and complete a qualified meeting with our sales team by November 2, 2022. The page is also packed with resources for using InfluxDB with AWS like blogs, videos, case studies, templates, and more.

See you soon!

We hope to see you at AWS re:Invent November 28–December 3! We love talking about the cool things you can do with InfluxDB and we hope you’ll stop by our booth so we can clear up any questions, spark some ideas for using InfluxDB with AWS, and give out swag (Did someone say socks?!). See you soon!

Why Use a Purpose-Built Time Series Database?

Susannah Brodnitz (InfluxData) — Thu, 13 Oct 2022 07:00:00 +0000

This article was originally published in The New Stack and is reposted here with permission.

For many workloads, using a time series database is a smart choice that saves time and storage space.

Developers and companies have more database choices than ever. Choosing the right database for a project saves time when writing and querying data. As companies work with larger datasets to make increasingly intelligent and automated systems, efficiency is key. For many workloads, using a time series database is a smart choice that saves time and storage space.

How time series data is different

Time series data is any metric with a timestamp. It includes many kinds of variables, from weather patterns to CPU usage. It often comes from sensors, systems or applications that need to make real-time decisions. This data is vital to understanding past performance and creating models to predict future outcomes. The amount of data involved in these calculations can quickly add up, and it’s important not to lose resources to an inefficient data architecture.

Time series databases are designed to handle typical time series workloads. They’re optimized to measure change over time, rather than relationships between data points. The two main kinds of time series data are metrics, which are taken at regular intervals, and events, which are taken at irregular intervals due to outside events or user measurements. It’s important that a time series database is able to handle both metrics and events, and is able to average events and convert them to metrics.

Storing data

A good database needs to store data securely and efficiently. Users have to be able to write data to it quickly and feel confident that it can handle the volume of data they plan to store in it. Time series data can have huge volumes, and the databases it’s stored in need to be built to accommodate that. Time is linear, and time series databases can take advantage of this by appending new data to existing data. They’re optimized to quickly write in time-stamped data the way it’s most commonly used to save time from the moment users begin writing in data.

Time series databases also may have lifecycle management built in. It’s common for developers or companies to initially collect and analyze highly detailed data and, as time goes on, to want to store smaller, downsampled datasets that describe trends without taking up as much storage space. Time series databases can take this into account and automatically aggregate and delete data as necessary for each application. If developers use a more basic database, they often need to create new systems to manage data in this way. With a time series database, it’s already taken care of and developers can focus on their applications.

Time series databases also need to be easily scalable. For example in IoT use cases, as more sensors are added and projects expand, data increases exponentially. This is common in time series workloads, and the databases used for these projects need to be able to accommodate it.

Querying data

Using a time series database also speeds up query time for time series workloads. One of the most common things to do with time series data is to summarize it over a large period of time. This kind of query is very slow when storing data in a typical relational database that uses rows and columns to describe the relationships of different data points. A database designed to process time series data can handle queries exponentially faster. Time series databases also may have built-in visualization tools or advanced functions to make it simpler to do common kinds of time series analysis.

Choosing a time series database

There are a few time series databases out there that you can explore. For this blog post, we will look at the leading time series database according to DB-Engines, InfluxDB. InfluxDB assigns a measurement name and timestamp to data and uses key/value pairs for data values and metadata. It keeps measurement names and sets of tags in an inverted index, which speeds up queries. Users can write queries based on measurement, tag and/or field across a time range and receive results in milliseconds. A single InfluxDB server can handle over 2 million writes per second. Compared to a NoSQL database like Cassandra, InfluxDB writes data 4.5 times faster, uses storage space 2.1 times less and returns queries 45 times faster.

Databases are the backbones of many applications and working with time-stamped data in a time series database saves developers time and storage. Choosing the right database for an application lets developers focus on building cool projects, rather than spending time managing architecture before they can get started.

InfluxData Blog - Susannah Brodnitz

The Immutability of Time Series Data

The past is the past

Time keeps moving

The context of time

How Prescient Devices Uses Time Series Data for IoT Automation

Connecting edge and cloud

Low-code framework

Saving time

TL;DR Python, Pandas Dataframes, and InfluxDB

Querying data from InfluxDB to a Pandas Dataframe

Writing data from a Pandas Dataframe to InfluxDB

Intro to InfluxDB IOx

Unbounded cardinality

Real-time analytics

SQL support

Sign up for the InfluxDB IOx Beta program

Getting Started with Python and Geo-Temporal Analysis

Getting started with an example

Cleaning up the data

Writing data to InfluxDB Cloud

Querying the data

Further resources

A Sneak Peek at InfluxDB University’s Upcoming Live Telegraf Training

Understanding Telegraf

Edge Data Replication

TL;DR InfluxDB’s MQTT Native Collector

When to use InfluxDB MQTT Native Collector

Configuring & parsing MQTT data with InfluxDB’s Native Collector

We’re Going to AWS re:Invent 2022

Watch live demos

Meet with InfluxDB experts

Interviews you can use

Win a free conference pass

See you soon!

Why Use a Purpose-Built Time Series Database?

How time series data is different

Storing data

Querying data

Choosing a time series database