InfluxData Blog - Getting Started

What Is Predictive Analytics? A Complete Guide for 2026

Company (InfluxData) — Tue, 24 Feb 2026 08:00:00 +0000

In simple terms, predictive analytics is a form of analytics that tries to predict future events, trends, or behaviors based on historical and present data. You can achieve this goal in different ways, each involving trade-offs between accuracy and cost.

Why is predictive analytics important?

Predictive analytics enables organizations to be more efficient and accurate in how they plan for the future. The end result of a properly implemented predictive analytics system will depend on the industry, but at a high level, here are some common benefits:

Improved Strategic Decision-Making

Predictive analytics provides insight into future trends, so business leaders can make better decisions faster rather than relying on reactivity.

Increased Operational Efficiency

Using predictive analytics can help businesses improve their profit margins and efficiency by predicting equipment failures and reducing downtime.

Improved Risk Management

By looking at historical data where things went wrong, a business can reduce its risk by finding data that correlates with negative outcomes and avoiding them proactively. An example would be a bad investment in the finance industry.

Happier customers

Predicting potential churn and reaching out to customers, or ensuring items are in stock by having more accurate predictions for inventory management help enhance customer experience.

How does predictive analytics work?

The end goal of predictive analytics is to make accurate predictions based on historical data. Here is a general outline of the process for building a predictive analytics system:

1. Determine the goal for the project. The first step is to identify the problem or opportunity you are trying to address via predictive analytics. Define your goals and success metrics upfront.

2. Organize and collect data. The next step will be gathering the data to build your predictive analytics model, as well as the pipeline that will send fresh data to your model for generating predictions. This will typically be a combination of public data similar to your own, 3rd-party data relevant to your use case, and your own unique business data for fine-tuning your model.

3. Process data. Once you have your data, one of the biggest challenges is often processing and cleaning it so it’s ready for your model. This can involve removing invalid data, filling in missing data, or transforming data into a standard format.

4. Develop a predictive analytics model. Now that your data has been collected and cleaned, you are ready to actually develop your predictive model. The model you use will depend on your business requirements, including accuracy requirements and the type of modeling you will be doing.

A predictive model can be used for trend detection, classification, clustering, and more. You can create these models using statistical methods or modern machine learning techniques.

5. Validate results. Creating and deploying your model is just the first step; once the model is live, you will need to validate the results to confirm it works as expected. This generally involves testing against a separate dataset for accuracy, as well as running the model against live production data and evaluating the results based on the output. If the results aren’t as good as desired, you may need to return to the previous steps and modify factors like how data is processed and the type of model used.

6. Deploy to production. If your predictive analytics model produces accurate, valuable results, you can now deploy it to production, where people will actually use the results. The system may need a human to confirm the action, or it may be fully automated, taking action solely based on the model.

7. Update and improve the model over time. Predictive analytics isn’t a one-time deal. You will want to constantly feed your model recent data so it stays up to date and can be aware of potential changes that need to be integrated. Typical tasks would involve retraining the model, adjusting parameters, or providing it with additional data to improve accuracy. The entire system can also be fine-tuned over time to be more efficient and affordable.

Predictive analytics use cases

Predictive analytics are useful across almost every industry, but let’s take a look at a few specific examples where predictive analytics are particularly valuable. An ideal use case for predictive analytics is any situation where data is relatively easy to collect and having more accurate predictions will generate a significant business impact, such as revenue or cost reduction.

Manufacturing

In the manufacturing sector, predictive analytics can be used to predict and prevent machinery malfunctions before they occur. This reduces maintenance costs and improves factory efficiency of factories, resulting in higher profit margins.

Healthcare

Governments and businesses both use predictive analytics to improve the healthcare industry. Governments create predictive models to try to predict and prevent the spread of diseases and also determine investments in healthcare programs. Hospitals can use predictive models to look at patient medical records to create personalized treatment plans.

Marketing

Predictive analytics can be used for marketing purposes to predict trends in consumer demand, improve customer engagement to prevent churn, and improve sales by recommending products customers might like based on their past purchases compared to those of similar customers.

Supply Chain Management

Predictive analytics can help with supply chain management by forecasting changes in product supply and demand driven by factors such as time of year or location.It can also be used to optimize logistics and manage risk.

Finance

The finance industry uses predictive analytics in a number of ways, ranging from predicting stock prices to detecting fraudulent transactions. Banks can use predictive analytics to assess loan applicants’ risk by comparing historical data with the applicant’s personal history.

Predictive analytics challenges

While predictive analytics can offer many business benefits, implementing it can be challenging, especially if a company lacksin-house expertise or infrastructure. Here are some of the key roadblocks to consider when getting started.

Data Quality

To make accurate predictions, you will need a large volume of high-quality data relevant to your predictive analytics use case. This means you need to have a way to collect data and store it in a long-term format that is easy to access for teams creating predictive analytics models.

Integration with Legacy Systems

Many established businesses will have systems that may not be seamlessly integrated. This means engineering effort will be required to ensure that data is not siloed and that the predictive analytics team can access the systems and data they require.

Accuracy of Results

The biggest challenge with predictive analytics will be creating a model that produces results accurate enough to justify the investment in creating them and that drives business value.

This will require not only the initial creation of the model but also constant updates with new data to keep it accurate as conditions change.

Hiring Talent

All of the above problems require highly skilled employees to be solved. These skills are in demand across many industries, making it difficult to attract and retain the workers needed to implement a predictive analytics system.

Security

Another challenge with predictive analytics is ensuring that all the new data collected and stored is secure. This data can contain sensitive information about customers or about your business, so security must be a top priority.

Predictive analytics techniques

There are a number of models available for generating insights via predictive analytics. The type of model to use for your organization depends on the data you are working with, as well as factors such as the cost to develop the model and your accuracy requirements. Let’s take a look at some of the most common predictive analytics techniques and models.

Machine Learning/AI Models

In the past, classical statistical models have dominated predictive analytics and forecasting because of their ease of interpretation, lower computational costs, and accuracy. However, in recent years, ML/AI-based models have begun to surpass traditional forecasting methods in accuracy. They also offer the benefit of being easier to generalize across different predictions and of requiring less fine-tuning by highly trained statisticians.

Time Series Models

Time series models are used to analyze temporal data and forecast future values. They are particularly useful when data shows sequential patterns or seasonality, such as stock prices, weather patterns, or sales data.

Time series models are ideal for data that has seasonal variations and time-based dependencies, making them useful for forecasting.

Some downsides of time series models are that they can struggle when the data isn’t at regular intervals and may assume past trends will continue, which can make them inaccurate at predicting drastic changes.

ARIMA and exponential smoothing are examples of time series models. An easy way to start testing these models for predictive analytics is to use a library like Python Statsmodels.

Regression Models

Regression models predict a continuous outcome variable based on one or more predictor variables. They are widely used in predictive analytics, from predicting house prices to estimating stock returns.

Regression models are useful for providing results that are easy to interpret and for identifying clear relationships between variables. Some downsides of regression models are that they do require a decent level of statistics knowledge and can struggle with non-linear relationships and datasets with many variables.

Linear and logistic regression are examples of regression models. You can get started with regression models using the Python scikit-learn library.

Decision Tree Models

Decision tree models make predictions by learning simple decision rules from the data. They can be used for both regression and classification problems. Decision tree models offer results that are easier to understand than those from machine learning models. A challenge is that they can be easily over- or underfit and be affected by small changes in the data.

Gradient Boosting Model

Gradient boosting involves creating an ensemble of prediction models, typically from decision tree models. This method can be extremely accurate and has been used in recent years to win many machine learning competitions.

Gradient boosting is good at providing accurate predictions for data with non-linear relationships between variables and datasets with high dimensionality.

One weakness is that they can be overfit when they aren’t tuned properly and are more of a black box compared to traditional statistical models. XGBoost and LightGBM are libraries that can be used to create gradient boosting models.

Random Forest Models

Random forests are similar to gradient boosting in that they are ensemble models that use decision trees for making predictions. The main difference is that gradient boosting models generally use far more decision trees, and they are also trained sequentially so that errors from previous trees can be corrected.

In comparison, random forest decision trees make predictions independently, and then the final prediction is created by aggregating those predictions. This makes the results easier to interpret because each decision tree’s prediction can be analyzed. You can test out random forest models on your data using a library like scikit-learn.

Clustering Models

Clustering models, such as k-means clustering, can be used to group data points. While this is generally used for data analysis, these clusters can also serve as input features for predictive models like the ones mentioned above.

Cluster modeling can help identify hidden patterns or relationships in your data, but to work, it requires a way to measure how similar data points are, and the number of clusters ‌must be chosen ahead of time.

Future trends in predictive analytics

The predictive analytics landscape is changing rapidly as technology advances and impacts all industries. Here are a few trends to look out for in the future:

Increased demand for real-time data. To get the most accurate results, models need to be updated as frequently as possible so they aren’t out of sync with reality. This means that real-time data and systems that support it will become increasingly important.
Prescriptive analytics. The term prescriptive analytics refers to the next step beyond predictive analytics. This involves taking action based on a predicted outcome before it occurs to try to influence the outcome.
Synthetic data. Data is the key to making accurate predictions. The problem is that many businesses haven’t collected the data they need. A number of tools have been created to generate “synthetic” data, which can help get a predictive analytics system off the ground using artificial data that mimics the use case.
Further adoption of machine learning and AI. While most businesses still rely on traditional methods for prediction, cutting-edge practitioners are using ML/AI to win competitions because of its accuracy.
Easier to use predictive analytics tools. Currently, implementing and using predictive analytics requires specialized skills. But domain knowledge is very important for making accurate predictions.

Future tools will focus on usability and enabling non-technical users to make predictions based on their data. This will make implementation more affordable and drive more business value.

Best practices

Here are some helpful tips for using predictive analytics.

Have a well-defined objective. Predictive analytics can only generate value when it influences a decision, and hence, the why should be the first thing followed by the model. Without a goal, you’ll maximize the things that make no difference. To implement this, you must clearly state what you want to predict, where you will apply the prediction, and what action you will take.
Focus more on feature engineering than model complexity. Features are used to convert raw data into signals that the model can learn, and this step can be what makes the difference in determining success, more than the algorithm used. To do this effectively, design domain-aware features such as rolling averages, lagged values, and behavioral features like frequency and recency.
Measure models based on business impact. Conventional measures such as accuracy may be misleading, particularly in skewed problems. It is significant because the technically correct model can be expensive or hazardous to implement. Use measures of actual trade-offs, like accuracy and accuracy of fraud detection, or average misplacement of demand forecasting.
Choose an easy, performance-dependent model. Complex models may be appealing; however, they are more difficult to maintain, debug, and explain. This is important in production situations where stability and interpretability are paramount. It is better to start with baselines and simple models, and add complexity only as performance improves.
Provide quality, time-accurate data. Predictive models use patterns in past records, and poor quality or poorly ordered records can lead to misleading results. Problems such as lost values, data leakage, or irregular timestamps may only inflate model performance during testing but not in production.

Common pitfalls to avoid in predictive analytics projects

Overfitting the Model

Overfitting occurs when a model fits noise rather than any general patterns, usually because of too much complexity or too little data. This is important because these models are useful for training data but not for new data.

An example of this is that a deep neural network trained on a small sample of customers might work flawlessly at elucidating the past, but would not help predict what customers would purchase in the future, whereas a simpler model would be more generalizable.

Data Leakage

Data leakage occurs when the information of the future accidentally affects the model during training. This will happen when it has features with data that cannot be known at prediction time, achieving unrealistically high test performance but failing in practice.

One such example is the use of the account closed date or an order completion status as an input into the churn or demand prediction model, which makes the model seem very accurate, but is not usable in practice.

Using the Wrong Evaluation Metrics

Accuracy alone can be a bad way to measure model performance, especially for use cases where positives are rare and costly when missed. An example would be fraud detection, a model that simply classifies all transactions as non-frauds would be very accurate(due to over 99% of transactions being legitimate), but in reality it’s still missing every case of fraud. For use cases like this teams need to use metrics that track actual business impact when evaluating their models.

Ignoring Changes in Data Patterns

Predictive models assume that future data will behave like past data; however, in reality, systems continue to evolve. This is particularly problematic in areas such as retail or finance, where seasonality, promotions, or changes in user behaviour often change.

FAQs

Predictive Analytics vs Predictive Maintenance

Predictive analytics is a broad field that uses statistical algorithms, machine learning, and data to anticipate future events across many domains. It identifies patterns in historical and current data to predict future trends, behaviors, and activities. Predictive analytics is used across industries such as finance, healthcare, and marketing to inform decision-making and develop proactive strategies. Predictive maintenance, on the other hand, is a specific application of predictive analytics in maintenance and asset management. It uses predictive analytics techniques to anticipate when equipment might fail or require maintenance. By analyzing data from sensors, logs, and historical maintenance records, predictive maintenance models can forecast equipment failures before they happen. The goal is to perform maintenance in time to prevent failures, improving efficiency and reducing downtime. In short, predictive maintenance is a subset of the broader predictive analytics ecosystem.

Traditional Statistical Models vs Machine Learning and AI Models for Predictive Analytics

More traditional techniques, such as regression models and decision trees, have been used for decades in predictive analytics. This is due to their simplicity, lower computational requirements, and ability to show the relationship between specific variables and the impact of changing them on business outcomes. In recent years, AI/ML techniques like neural networks and gradient boosting have grown in popularity for predictive analytics use cases. The primary reason is that ML techniques can perform better with higher-dimensional data, where relationships among numerous variables are harder to define. These AI/ML models can learn from data without explicit tuning and can uncover relationships between variables that aren't obvious, resulting in higher accuracy. Some downsides of AI/ML for predictive analytics are that they tend to require more hardware for computation and are harder to interpret in terms of how they produce results, in some ways acting as black boxes.

Getting Started with InfluxDB and Pandas: A Beginner's Guide

Community (InfluxData) — Tue, 27 Jan 2026 08:00:00 +0000

InfluxData prides itself on prioritizing developer happiness. A key ingredient to that formula is providing client libraries that let users interact with the database in their chosen language and library. Data analysis is the task most broadly associated with Python use cases, accounting for 58% of Python tasks, so it makes sense that Pandas is the second most popular library for Python users. The InfluxDB 3 Python client library supports Pandas DataFrames, making it easy for data scientists to use InfluxDB.

In this tutorial, we’ll learn how to query our InfluxDB instance and return the data as a DataFrame. We’ll also explore some data science resources included in the Client repo. To learn about how to get started with the InfluxDB 3 Python client library, please take a look at this video.

Me eagerly consuming Pandas and InfluxDB Documentation. Photo by Sid Balachandran on Unsplash.

Data science resources

A variety of data science resources have been included in the InfluxDB Python client repo to help you take advantage of the Pandas functionality of the client. I encourage you to take a look at the example notebooks.

Dependencies

pyarrow (automatically comes with influxdb3-python installation)
pandas
certifi (if you are using Windows)

Installations

pip install influxdb3-python pandas certifi

Import Dependencies


import influxdb_client_3 as influxDBclient3
import pandas as pd
Import certifi #if you are on Windows
from influxdb_client_3 import flight_client_options

Initialization

Direct Initialization

Take note that the “database” argument in the function is the bucket name if you are using InfluxDB cloud:

client = InfluxDBClient3(token="your-token",
                         host="your-host",
                         database="your-database or your bucket name")

For Windows Users

Include certifi within the “flight_client_options” argument within the client initialization to fix certificate issues:

with open(certifi.where(), "r") as fh:
    cert = fh.read()
client = InfluxDBClient3(token="your-token",
                         host="your-host",
                         database="your-database or your bucket name”,
flight_client_options=flight_client_options(tls_root_certs=cert)

Prepare a pandas Dataframe

Let’s use simple weather data:


# Example weather data
df = pd.DataFrame({
    "timestamp": pd.date_range("2025-09-01", periods=4, freq="h", tz="UTC"),
    "city": ["Lagos", "Illinois", "Chicago", "Abuja"],
    "temperature": [30.5, 15, 16, 32],
    "humidity": [20, 10, 10, 19]
})

# ensure timestamp dtype is datetime64[ns] and (optionally) timezone-aware
df['timestamp'] = pd.to_datetime(df['timestamp'], utc=True)
print(df.head())

Weather timestamp data.

Write the pandas Dataframe to InfluxDB


client._write_api.write(
    bucket="my_bucket",
    record=df,
    data_frame_measurement_name="weather",
    data_frame_tag_columns=["city", "temperature"],
    data_frame_timestamp_column="timestamp"
)
print("DataFrame written to bucket=new-test-bucket, measurement=weather")

Below is confirmation in your InfluxDB Cloud that the Pandas DataFrame was successfully written to the bucket.

Pandas Dataframe written to InfluxDB.

Query InfluxDB and return a Pandas DataFrame

query = "SELECT * FROM weather"
table = client.query(query=query, language="influxql")
result_df = table.to_pandas()

print("Loading from InfluxDB:")
print(result_df.head())

Returning a Pandas DataFrame from InfluxDB

Start building with InfluxDB and Pandas

InfluxDB makes it easy to integrate with your existing data analysis tools and frameworks, such as Pandas, to get insights from your time series data. Under the hood, you get the benefits of Apache Arrow for fast data transfers into Pandas DataFrames without any performance hits. Whether you’re building dashboards, machine learning models, or just exploring your metrics, combining InfluxDB 3 with Pandas gives you the best of both worlds in terms of performance and developer experience. As always, if you run into hurdles, please share them on our community site or Slack channel. We’d love to get your feedback and help with any problems you run into.

System Tables Part 1: Introduction and Best Practices

Chunchun Ye (InfluxData) — Tue, 29 Oct 2024 08:00:00 +0000

As an InfluxDB Cloud Dedicated or Clustered user, you may want to inspect your cluster to gain a better understanding of the size of your databases, tables, partitions, and compaction status. InfluxDB stores this essential metadata in system tables (described in Section 1), which help inform decisions about cluster performance and maintenance.

1. What are system tables?

System tables are “virtual” tables that present metadata for a specific database and provide insights into database storage. Each system table is scoped to a particular database and is read-only, meaning it cannot be modified.

System tables are hidden by default, as high-frequency access to these tables can interfere with the ongoing operations of the database. Thus, querying system tables requires a special debug header with the request. Once the debug header is added (described in Section 2), you can query system tables using SQL, similar to any other table in InfluxDB.

Here are the system tables that InfluxDB provides:

+---------------+--------------------+-------------+------------+
| table_catalog | table_schema       | table_name  | table_type |
+---------------+--------------------+-------------+------------+
| ...           | ...                | ...         | ...        |
| public        | system             | compactor   | BASE TABLE |
| public        | system             | partitions  | BASE TABLE |
| public        | system             | queries     | BASE TABLE |
| public        | system             | tables      | BASE TABLE |
| ...           | ...                | ...         | ...        |
+---------------+--------------------+-------------+------------+

In this blog, we will focus on three tables:

system.tables
system.partitions
system.compactor

Table	Description	Schema
`system.tables`	Contains information about tables, such as table name and partition template in the specific database.	Link
`system.partitions`	Contains information about partitions, partition sizes, file count, etc.	Link
`system.compactor`	Contains detailed information about compacted partitions at different compaction levels.	Link

Warning: System tables are not part of InfluxDB’s stable API. They are subject to change, and compatibility is not guaranteed.

Warning: Querying system tables may impact write and query performance. Use them only for debugging purposes and use filters to optimize queries and minimize their impact on your cluster.

2. Accessing system tables

To access system tables, you must provide a debug header with the request. The specific commands to add this header vary depending on the client you are using.

influxctl CLI

For influxctl, set the --enable-system-tables header:

influxctl query \
  --enable-system-tables \
  --database DATABASE_NAME \
  --token DATABASE_TOKEN \
  "SQL_QUERY"

Arrow Flight SQL or other client libraries

For Arrow Flight SQL or other client libraries, such as Go and Python, set the iox-debug header to true.

3. Querying system tables: Examples

1. View the partition template of a specific table

SELECT  *  FROM  system.tables  WHERE  table_name  =  'TABLE_NAME'

Example Result:

+-----------------+--------------------------------------------------------+
| table_name      | partition_template                                     |
+-----------------+--------------------------------------------------------+
| your_table_name | {"parts":[{"timeFormat":"%Y-%m"},{"tagValue":"col1"}]} |
+-----------------+--------------------------------------------------------+

If a table doesn’t include a partition template in the output of this command, the table uses the default (1 day) partition strategy and doesn’t partition by tags.

2. View the number of partitions and total size per table

SELECT
  table_name,
  COUNT(*) AS partition_count,
  SUM(total_size_mb) AS total_size_mb
FROM system.partitions
WHERE table_name IN ('foo', 'bar', 'baz')
GROUP BY table_name

Example Result:

+------------+-----------------+---------------+
| table_name | partition_count | total_size_mb |
+------------+-----------------+---------------+
| foo        | 1               | 2             |
| bar        | 4               | 5             |
| baz        | 10              | 23            |
+------------+-----------------+---------------+

3. View the size for different levels of compacted files*

SELECT
  table_name,
  SUM(total_l0_files) AS l0_files,
  SUM(total_l1_files) AS l1_files,
  SUM(total_l2_files) AS l2_files,
  SUM(total_l0_bytes) AS l0_bytes,
  SUM(total_l1_bytes) AS l1_bytes,
  SUM(total_l2_bytes) AS l2_bytes
FROM system.compactor
WHERE table_name IN ('foo', 'bar', 'baz')
GROUP BY table_name

*Compacted files are compressed Parquet files processed by the Compactor to optimize storage. These files have different compaction levels: L0, L1, and L2. L0, or “Level 0”, represents newly ingested, uncompacted small files, while L2, or “Level 2”, represents compacted, non-overlapping files.

Example Result:

+------------+----------+----------+----------+----------+----------+----------+
| table_name | l0_files | l1_files | l2_files | l0_bytes | l1_bytes | l2_bytes |
+------------+----------+----------+----------+----------+----------+----------+
| foo        | 0        | 1        | 0        | 0        | 20659    | 0        |
| bar        | 0        | 1        | 0        | 0        | 7215     | 0        |
| baz        | 0        | 1        | 0        | 0        | 10784    | 0        |
+------------+----------+----------+----------+----------+----------+----------+

4. Optimize queries to reduce cluster impact

Querying system tables can degrade the performance of other common queries, especially if you are trying to view every detail in clusters with hundreds of tables, hundreds of thousands of partitions, and millions of Parquet files.

To reduce the performance impact, we suggest selecting information for a specific table or a particular partition by adding filters as follows:

WHERE table_name = '...'
WHERE table_name = '...' AND partition_key = '...'
WHERE table_name = '...' AND partition_id = ...
WHERE partition_id = ...

See documents on how to obtain partition_key and partition_id.

5. Use the most efficient filters

Among the above filters, the following filters are specially optimized and significantly reduce query latency to about 20 ms, even on our largest clusters:

WHERE table_name = '...' AND partition_key = '...'
WHERE table_name = '...' AND partition_id = ...

6. Bringing it home

In this first post, we introduced system tables, explained how to access them, and discussed how to optimize your queries with filters. In the next post, we will explain how we improved the performance of system tables.

References:

Getting Started with Kafka, Telegraf, and InfluxDB v3

Anais Dotis-Georgiou (InfluxData) — Tue, 15 Oct 2024 08:00:00 +0000

In the world of smart gardening, keeping track of environmental conditions like humidity, temperature, wind, and soil moisture is key to ensuring your plants thrive. But how do you bring all this data together in an efficient and scalable way? Enter the powerful trio of Kafka, Telegraf, and InfluxDB Cloud v3. In this guide, we’ll walk you through setting up a seamless pipeline that collects real-time data from garden sensors, streams it through Kafka, and stores it in InfluxDB for monitoring and analysis through the use of Telegraf. Whether you’re new to these tools or looking to expand your IoT toolkit, this example will show you how to get started. The corresponding repo for this tutorial can be found here.

Requirements and Run

Before we dive into the setup, there are a few requirements to have in place. First, you’ll need Docker and Docker Compose installed on your system, as the example relies on containerized services to simplify deployment. You should also have an InfluxDB Cloud v3 account, with your URL, token, organization, and bucket information readily available. These details will be crucial for configuring Telegraf to write garden sensor data to InfluxDB. Additionally, ensure you have Python installed, as the garden sensor gateway script relies on Python’s Kafka package to simulate and send sensor data. Finally, familiarity with the basic concepts of Kafka, Telegraf, and InfluxDB will help you follow along more easily.

To run this example, follow these steps:

Clone the project and navigate to the directory.
Open the resources/mytelegraf.conf file and insert your InfluxDB Cloud v3 URL, token, organization, and bucket name. You can also use environment files if you desire instead.
Start the containers by changing “directory” to resources and running the command docker-compose up --build -d.
Wait approximately 30 seconds for Telegraf to initialize and begin writing metrics.
Once everything is up and running, the garden sensor gateway will start generating random humidity, temperature, wind, and soil data, sending it through Kafka, and storing it in your InfluxDB Cloud v3 instance for monitoring and analysis.

Code Explained

In this section, we’ll break down the example’s components and explain how each piece fits together to create a seamless data pipeline for monitoring garden sensor data using Kafka, Telegraf, and InfluxDB Cloud v3.

1. app/Dockerfile

The Dockerfile in the app directory is responsible creates a containerized environment to run the garden_sensor_gateway.py script.

2. app/garden_sensor_gateway.py

import time
import json
import random

from kafka import KafkaProducer

def random_temp_cels():
    return round(random.uniform(-10, 50), 1)

def random_humidity():
    return round(random.uniform(0, 100), 1)

def random_wind():
    return round(random.uniform(0, 10), 1)

def random_soil():
    return round(random.uniform(0, 100), 1)

def get_json_data():
    data = {}

    data["temperature"] = random_temp_cels()
    data["humidity"] = random_humidity()
    data["wind"] = random_wind()
    data["soil"] = random_soil()

    return json.dumps(data) 

def main():
    producer = KafkaProducer(bootstrap_servers=['kafka:9092'])

    for _ in range(20000):
        json_data = get_json_data()
        producer.send('garden_sensor_data', bytes(f'{json_data}','UTF-8'))
        print(f"Sensor data is sent: {json_data}")
        time.sleep(5)

if __name__ == "__main__":
    main()

This Python script simulates garden sensor data and sends it to a Kafka topic. Let’s look at how it works:

Importing Libraries: The script imports necessary libraries like time, json, random, and KafkaProducer from the kafka-python package.
Data Generation Functions: Functions like random_temp_cels(), random_humidity(), random_wind(), and random_soil() generate random values for temperature, humidity, wind, and soil moisture, respectively. These values are rounded to one decimal place to simulate realistic sensor readings.
Data Formatting: The get_json_data() function collects these generated values into a dictionary and converts it into a JSON string using json.dumps(data).
Kafka Producer: The main() function initializes a Kafka producer with KafkaProducer(bootstrap_servers=[‘kafka:9092’]), pointing it to the Kafka broker running in the container. It then enters a loop where it generates sensor data, sends it to the Kafka topic garden_sensor_data, and prints the data to the console. The loop runs 20,000 times, with a 5-second delay between each iteration.

3. resources/docker-compose.yml

The docker-compose.yml file in the resources directory defines the services required for the project, orchestrating the containers for Kafka, Zookeeper, Telegraf, and the garden sensor gateway. Here’s what each service does:

Kafka and Zookeeper: These services set up the Kafka broker and Zookeeper, which Kafka relies on for distributed coordination. Kafka is exposed on port 9092, and Zookeeper on port 2181.
Garden Sensor Gateway: This service builds the container for the garden_sensor_gateway.py script using the Dockerfile in the app directory. It depends on Kafka to ensure that Kafka is up and healthy before the script starts running.
Telegraf: The Telegraf service is configured to consume messages from the Kafka topic garden_sensor_data and write them to InfluxDB Cloud v3. The Telegraf configuration file, mytelegraf.conf, is mounted into the container to provide the necessary settings.

4. resources/mytelegraf.conf

[[inputs.kafka_consumer]]
  ## Kafka brokers.
  brokers = ["kafka:9092"]
  ## Topics to consume.
  topics = ["garden_sensor_data"]
  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "json"

This configuration file is where Telegraf is set up to process the garden sensor data:``

InfluxDB Output: The [[outputs.influxdb_v2]] section configures Telegraf to write data to InfluxDB Cloud v3. You must replace the placeholders with your InfluxDB URL, token, organization, and bucket details.
Kafka Consumer Input: The [[inputs.kafka_consumer]] section configures Telegraf to subscribe to the garden_sensor_data topic on Kafka. It consumes the JSON-formatted sensor data, which is sent to InfluxDB for storage and analysis.

Together, these components create a robust pipeline where garden sensor data is generated, sent to Kafka, processed by Telegraf, and stored in InfluxDB Cloud v3, allowing you to monitor your garden’s environment in real-time.

Conclusion

This blog post describes how to start using InfluxDB, Kafka, and Telegraf. A Python script that generates garden data and sends it to a Kafka topic, Telegraf reads the data from the Kafka topic and writes it to InfluxDB. As always, get started with InfluxDB v3 Cloud here. In the next post, we’ll cover how to run the project, dive into the architecture and logic, and discuss some of the pros and cons of the selected stack. If you need help, please contact us on our community site or Slack channel. If you are also working on a data pipelining project with InfluxDB, I’d love to hear from you!

An Introductory Guide to Cloud Security for IIoT

Community (InfluxData) — Thu, 12 Sep 2024 08:00:00 +0000

The state of industries has come a long way since the Industrial Revolution with new technologies such as smart devices, the internet, and the cloud. The Industrial Internet of Things (IIoT) is a network of industrial components that share and process data to gain insights. But as IIoT involves sensitive data and life-critical operations, this also comes with various IIoT cloud security challenges. Therefore, it is important to strengthen security.

In this post, we’ll look into the benefits and challenges of IIoT and understand the common threats IIoT faces. Finally, we’ll discuss IIoT cloud security best practices and the tools you need to secure your IIoT infrastructure.

Benefits of IIoT

Minimized Downtime

Organizations can get real-time data on machine performance and environmental conditions and plan for maintenance. This proactive measure minimizes downtime and enhances operational efficiency.

Data-Driven Decision-Making

Advanced analytics provide actionable insights to make data-driven decisions. Real-time analytics help in responding swiftly to changing conditions and competitiveness.

Safety and Compliance

IIoT detects hazardous conditions and trigger automatic shutdowns or alerts to protect workers. It also helps comply with regulatory requirements.

Cost Reduction and Resource Optimization

IIoT provides insights to create strategies for more efficient use of resources, leading to cost reductions. Stakeholders can identify demand patterns and track inventory to increase resource optimization.

Enhanced Customer Satisfaction

IIoT plays a crucial role in enhancing customer satisfaction as it leads to high quality products and on-time delivery.

Innovation and New Business Models

IIoT opens doors for innovation and the development of new business models as it helps organizations identify what upgrades will be helpful and how they can make better profits.

The benefits of IIoT make it so attractive that one might want to jump right in and implement it. While IIoT has a lot of benefits, it also comes with some challenges.

Key challenges in IIoT

If the challenges of IIoT are not addressed, it might turn out to be a bane more than a boon. So, let’s look into some common challenges in IIoT.

Security Vulnerabilities and Threats

A significant challenge in IIoT is securing the network, connected devices, and data as each component represents a potential entry point for cyber attackers.

Data Privacy and Compliance

IIoT devices generate sensitive data that are subject to strict regulatory requirements. Ensuring compliance with these regulations is a complex task.

Scalability and Network Latency

For large-scale industrial and fast-growing units, managing the increased volume of data and network performance and ensuring real-time processing become challenging.

Skills Gap

Successful deployment and management of IIoT solutions require skills in both IT and operational technology (OT), which can be challenging to find.

Legacy Systems

Many industrial facilities still rely on legacy systems, and integrating these systems with IIoT devices is a challenge.

These key challenges can be a make-or-break for any organization. It’s important to understand how to deal with them. Since the focus of this post is IIoT cloud security, let’s now look at that.

Common attack vectors and types of threats

IIoT is an attractive target for attackers because of the impact it has on organizations and how profitable it can be for attackers. Attackers can ask for unreasonable ransoms and create leverage over these organizations. If something goes wrong due to poor practices, it can cause catastrophic damage to organizations. Let’s look into some common attack vectors and types of threats that IIoT cloud security faces.

Device Compromise and Malware

Attackers can exploit vulnerabilities in device firmware or software to install malware, which can then be used to steal data, disrupt operations, or infiltrate cloud systems.

Man-in-the-Middle (MITM)

Attackers can steal or manipulate sensitive data by intercepting the communication between devices and the cloud. Manipulation of critical data or commands can cause operational disruptions and unreliable data insights.

Denial of Service (DoS)

DoS and DDoS attacks aim to overwhelm a system, network, or service, making it unavailable and impacting the operations.

Attackers try to trick employees into giving out sensitive information that they can use to breach the security of the IIoT infrastructure.

Insider Threats

Insider threats involve intentional or unintentional actions that cause harm by individuals having legitimate access to the organization’s IIoT infrastructure.

Supply Chain Attacks

Supply chain attacks can involve compromising software updates, hardware components, or third-party service providers.

Now that we’ve looked into the common security issues, let’s look into the best practices for IIoT cloud security.

IIoT cloud security best practices

Every technology has security concerns, and following best practices can help eliminate threats and reduce risks. While every organization is different and needs a tailored security posture, here are some IIoT cloud security best practices to start with.

Authentication and Access Control

Enforce the use of strong passwords and multi-factor authentication (MFA), and regularly review and update access using the principle of least privilege.

Encrypt Data

Encrypt data at rest and in transit to prevent unauthorized access to sensitive data. Use strong and industry-standard encryption protocols such as AES-256 and implement the latest TLS.

Regularly Update and Patch

Keep track of all components, check regularly for updates, and apply them. Regular updates help protect against exploitation of known security weaknesses.

Network Segmentation

Network segmentation helps contain a potential breach by limiting the spread. Segment your network based on the criticality of components and how easy they are to reach by an attacker. Enforce strong firewall and access control rules.

Continuous Monitoring and Incident Response Plan

Deploy IIoT cloud security monitoring tools to track system activity and detect suspicious, malicious, and anomalous activity. Create extensive SOPs for active response. This helps in remediating an attack before it could cause harm. Create an incident response plan including roles, responsibilities, and procedures, and regularly update it.

Security Awareness

Hold regular security training and awareness programs for all employees. Encourage reporting of suspicious activities and provide clear guidelines on how to address potential security incidents.

These best practices are just something to begin with. You have to additionally evaluate your IIoT infrastructure and implement IIoT cloud security accordingly. Taking a proactive security approach can help you reap the benefits of IIoT while mitigating the associated risks.

Useful tools for IIoT cloud security

To implement a strong security posture, it is important to choose the right tools. So, let’s look into some useful tools for IIoT cloud security.

Endpoint Security Tools

Endpoint security solutions are critical for protecting IIoT devices. Deploy endpoint security solutions across all IIoT devices to ensure continuous protection and monitoring.

Network Security Tools

Network security tools help secure communications between IIoT devices and cloud systems, preventing unauthorized access and data breaches.

Identity and Access Management (IAM) Solutions

IAM helps enforce strong authentication protocols and access controls. These solutions help ensure that only authorized personnel can access sensitive IIoT data and cloud resources.

Continuous Monitoring and SIEM Solutions

Continuous monitoring and Security Information and Event Management (SIEM) solutions provide real-time visibility into IIoT cloud security, ensuring rapid detection and response to security incidents.

Vulnerability Management Tools

Vulnerability management tools help identify and remediate security weaknesses within IIoT devices and cloud systems. Regularly scan IIoT devices and cloud systems for vulnerabilities, and promptly address identified issues.

Cloud-Specific Security Solutions

Cloud-specific security solutions provide comprehensive security controls, visibility, and compliance management tailored for cloud environments. Most cloud service providers have several security features and solutions and also allow you to integrate additional tools to enhance IIoT cloud security.

Secure Data Storage Solutions

Data is the core of IIoT, and you need a tool that can handle IIoT-specific data securely. InfluxDB is one such solution. It provides real-time insights from any time series data with a single, purpose-built database. InfluxDB focuses on security with its security program including:

regular third-party penetration tests
continuous security management and monitoring
industry-standard encryption

Try InfluxDB for free

Conclusion

IIoT is a great technology for industries, and it’s our responsibility to make the best use of it. By connecting industrial devices, machines, and systems, IIoT facilitates real-time data collection, analysis, and automation. However, IIoT also comes with some security challenges. In this post, we discussed some IIoT cloud security challenges and best practices to secure IIoT infrastructures. These best practices are a baseline, the bare minimum, and organizations must implement additional security measures for enhanced security. The path to embracing the true power of IIoT is through proactive security!

This post was written by Omkar Hiremath. Omkar is a cybersecurity team lead who is enthusiastic about cybersecurity, ethical hacking, and Python. He is keenly interested in bug bounty hunting and vulnerability analysis.

Home Assistant Tutorial: A Beginner’s Guide to Automation

Community (InfluxData) — Fri, 05 Jan 2024 08:00:00 +0000

In this post, we’ll be taking a closer look at Home Assistant, an open source platform for connecting your smart devices at home. We’ll walk through every important section of Home Assistant: dashboards, integrations, add-ons, devices and entities, automation, scripts, and scenes. In addition, we’ll be walking through how to set up your Home Assistant and create automation using Home Assistant’s graphical user interface.

How Does Home Assistant Work?

Home Assistant is an open source smart home platform that allows you to connect your smart home devices like your TV, fan, cameras, thermostats, lights, and sensors. As a user, you can build intricate automation using Home Assistant’s user-friendly, unified web-based user interface.

Home Assistant means you don’t need to be a programmer or a computer scientist to get a device working with your smart home. You can simply build and test automation without writing a single line of code. Home Assistant can also be as complex as you want it to be, just depending on how much time you’re willing to put into it.

Important Parts of Home Assistant

Home Assistant has some very important features, which include the following:

Dashboard
Integrations
Add-ons
Devices and entities
Automation
Scripts
Scenes

Now let’s take a closer look at each of them.

Home Assistant dashboard

The first thing you see once you’ve installed and logged into Home Assistant is the dashboard. The dashboard is a page in Home Assistant used to display information that is available in Home Assistant. The more devices and services you add to Home Assistant, the more information will be available for display. Whenever devices or services are added, Home Assistant automatically adds their information to a default dashboard. But if you’d like, you can take control and display exactly what you want.

Dashboards are composed of cards that display information about the devices and services that are in Home Assistant. These cards can be added via the user interface or by editing YAML scripts. Even though Home Assistant is largely configurable through the user interface, you’re still going to run the YAML scripts. It is, therefore, paramount that you understand it properly.

Home Assistant integrations

Integrations are additional software that can be installed within Home Assistant and allow it to connect to different platforms, which bring in data and devices. When integrations are installed, the data from these integrations are represented in Home Assistant as devices and entities. There are currently over 1,000+ built-in integrations in Home Assistant that are fully supported by the Home Assistant community. Built-in integrations are those that are directly supported by Home Assistant, and they’re often automatically discovered on your Wi-Fi network. For custom integrations, there is also a Home Assistant community store, which can be installed as an integration itself, and it gives you access to thousands of these custom integrations as well.

Home Assistant add-ons

Sometimes integrations and add-ons are confused with each other, but we should keep them separate because they are, in fact, different from one another. Depending on your installation type, you may or may not have the ability to install add-ons. Add-ons are applications that run alongside Home Assistant and whatever hardware your Home Assistant is running on. They can be easily and quickly installed, configured, and run within Home Assistant.

While integrations only connect Home Assistant with many applications, devices, and services, add-ons provide additional functionality.

An example of an add-on is the Z-Wave JS server, which runs alongside Home Assistant to act as a server for all Z-Wave devices. Those devices are then connected to Home Assistant using the Z-Wave JS integration. Other examples are file editors, which can allow you to edit your Home Assistant configuration files right within Home Assistant.

Home Assistant devices and entities

Entities in Home Assistant represent logical groupings of functions within the system. A device, on the other hand, signifies a physical device connected to Home Assistant through integration. However, entities can be added independently without grouping them into devices. They encompass not only devices but also automation, scripts, and scenes. For instance, the Philips Hue motion sensor in Home Assistant comprises temperature, illuminance, occupancy sensors, and a motion detector, each represented as separate entities within the device grouping. Entities can have different types, such as binary sensors and motion sensors, and exhibit various states. For example, a motion sensor may have an “ON” state when motion is detected and an “OFF” state when no motion is detected. Numeric states are also possible, such as a temperature sensor with a state of 70°F. Additionally, entities can possess attributes that provide further information about their state.

Home Assistant automation

Automation is a process of making certain actions happen automatically. Automation in Home Assistant typically include the following:

Triggers: This is a prompt for those actions to happen.
Conditions: These are rules dictating if the actions should happen.
Actions: This represents the actions that Home Assistant will do when the automation is triggered.

An example of simple automation is as follows.

A motion sensor device like the Philips Hue sensor has a motion sensor entity. When that entity turns on (meaning there is motion), it triggers an automation that turns the light on. The automation has a condition that it has to be after 8 pm. So the light only comes on at night.

Home Assistant scripts

Scripts are similar to automation in that they allow you to run multiple actions in a row, step by step. The main difference here is that scripts don’t have triggers, meaning that they cannot automatically run. They have to be called upon by automation. For instance, if you have a series of actions you want to run in multiple automation, you can create a single script, and each automation can call that single script. In that way, if an action needs to be changed, you’re only changing one script instead of two automation. Within a script, delays can be added between steps if needed. You can also trigger scripts from your Home Assistant dashboard with cards.

An example of a script is a projector setup. We choose this as a script because, let’s say, there are two different ways that you want to trigger your projector setup to run. Firstly, you want to use a dashboard button and a smart physical button that anyone can press to run the setup. Now when you press the start button, it triggers automation to call the projector setup script. This script lowers the projector’s screen, turns on the projector, and also selects HDMI as the input.

Home Assistant scenes

Scenes are a set of saved states of entities that can be used as Home Assistant automation or scripts. For example, you can set up two different scenes, one for the morning and one for the evening, that both have specific brightness and light colors and then you can run an automation that changes the scene based on the time of day. This may seem like automation or a script can do what a scene can do, so there is no point in having scenes. However, contrary to that, automation and scripts are for actions, while scenes are for setting the state of entities. Automation and scripts follow a sequential flow and can be interrupted if triggers are not met, whereas scenes execute simultaneously.

Setting Up Home Assistant

There are two steps in setting up Home Assistant:

Installation
Initial setup wizard

Installation

The first step in setting up Home Assistant on your system is by installing it. There are different ways of getting Home Assistant up and running on your device, some of which include the below:

Raspberry Pi
Virtual machine
Docker
Windows
macOS
Linux

To get Home Assistant up and running, you need to choose from the list of options for installation by visiting the official Home Assistant website, www.home-assistant.io, then following the instructions provided for the respective installation options.

Initial setup wizard

Here at the initial setup wizard step, you need to fill in the blank fields (name, username, password) with your details to create an account. After filling in your details, you can now click “CREATE ACCOUNT.” You can see the details below.

The next page shows you fields and options for you to set a name for your home, your location, and your unit system. You can manually get your preferred location using the map, or you can simply detect your location by clicking the “DETECT” option. Furthermore, the setting of your time zone, currency, and unit system will be required based on your location. Afterward, you click “NEXT.” You can see the details in the image below.

The next screen asks you if you would like to share any data with Home Assistant. This is entirely up to you to decide. Afterward, click “NEXT.” You can see the details below.

Now the next screen shows a bunch of devices that we can import into Home Assistant. You have the option to set these up now or later on. Lastly, you click on “FINISH.” You can see the details in the image below.

Bravo! We can now see our Home Assistant Dashboard!

Setting Up Automation in Home Assistant

There are two ways to configure automation in Home Assistant. You can use the built-in automation editor right in the Home Assistant user interface, or you can manually write it yourself in a YAML script. In this post, we’ll be making use of the built-in automation editor within Home Assistant.

The following steps show you how to build a basic automation that sends a simple notification when the front door is open.

Automation and scenes

You’ll find the automation editor under Settings, and under this, you’ll find Automations & Scenes. Now when Settings is opened, click on Automations & Scenes.
Click Create Automation in the lower right
Now you have an option of using a blueprint or starting with empty automation. Select Start with empty automation since our objective is to create a new automation.
Now you have the three parts of automation: Triggers, Conditions, and Actions, which were defined earlier in the post. Now select + ADD TRIGGER under Trigger to add a trigger.

Triggers

For Triggers, under which you’ll see a lot of trigger options to pick from, we’ll be making use of State. Now select State from the list. Under the blank space for Entity, type in Front Door. In the search results, select your entity for the front door. In From and To, fill the blanks with Closed and Open, respectively. Now we’re done with Trigger, so next is to add the conditions. Now select + ADD CONDITION, which can be found under Condition. You can see the details below.

Conditions

Under Conditions, you’ll find a list of options to choose from. Any of these conditions need to be true for the automation to continue. Since, in our case, we’re only sending a notification, we’ll make sure the automation happens only when people are away. For that reason, we’ll be making use of the sun as a proxy for that context. For the Condition, we’ll use State from the list of options provided. Now we’ll fill in the blanks for entity and state withSun andAbove Horizon, respectively. Remember that automation will continue only if these conditions are true. Now let’s proceed with the actions since we’ve finished with the conditions. Now select + ADD ACTION, which can be found under Action. You can see the details below.

Actions

Under Actions, you’ll find a list of options to choose from. Now we’ll select Call Service. To fill in the blank for Service, type mobile into the search bar (since we intend to use the mobile application for receiving the notification). Unfortunately, you may not find a notification service in the search results. But then, select any of the services from the search results that have “mobile_app” in the name. Furthermore, now that you have made the selection, the system will prompt you with other attributes you can use. It’s important to note that the options without a check box are mandatory, whereas the ones with a check box are optional. Let’s put a check on the Message option and assign it the label Front door was opened. You can see the details in the image below.

Now, if we want, we can check the Title to give it the title Door Notification. Lastly, hit save.

Bravo! The process is complete. So, whenever someone opens the front door, you’ll receive a notification like this:

Summary

Home Assistant enables users to automate and control home appliances like cameras, thermostats, lighting, and sensors. To get the Home Assistant software up and running on your system, you must choose the most convenient option for your installation. The list of options for installation and the necessary steps to successfully install Home Assistant on your system are provided on the Home Assistant official website. After completing the installation, you should set up your personal Home Assistant to connect your smart home devices by using the initial setup wizard. You can create automation using the GUI feature within Home Assistant, where you need to put in place triggers, conditions, and actions, which are essential features for your automation to run successfully.

This post was written by Theophilus Onyejiaku. Theophilus has over 5 years of experience as data scientist and a machine learning engineer. He has garnered expertise in the field of Data Science, Machine Learning, Computer Vision, Deep learning, Object Detection, Model Development and Deployment. He has written well over 660+ articles in the aforementioned fields, python programming, data analytics and so much more.

Grafana Dashboard Tutorial: How to Get Started

Community (InfluxData) — Fri, 08 Dec 2023 09:30:00 +0000

Grafana is an open-source web application for visualizing data. You can query your data, create visuals, and receive alerts to better understand what you have. Some people think of Grafana as a Kubernetes-only tool, but in reality, it’s simply a data visualization tool that became popular within the Kubernetes ecosystem, especially when combined with Prometheus.

In this post, I’ll focus on a very specific part of Grafana: the dashboards. To do so, I’ll use Grafana’s tutorial environment setup to help you get up to speed and learn about dashboards using Docker Compose.

Before I get into the specifics, let’s explore what you can do with Grafana dashboards.

What Does a Grafana Dashboard Do?

Dashboards, in general, are a visual representation of data. Raw data itself might not tell you much, but when you use a dashboard you can glean valuable information. Say, for example, you’re hosting a site about the latest gaming industry news. From time to time, you get customer complaints that the site is not loading or is taking too much time. How would you identify the problem? One way could be logging into the servers and reading some logs. That’s doable if you only have a few servers or containers. But when you have dozens, you need a better approach.

Grafana dashboards can help you better understand what’s happening in advance. You can configure Grafana to use any data source you want, and the dashboards will help you with a visual representation so you can fix problems more rapidly. You can also annotate events and explain what happens after certain actions, like a recent release or a marketing campaign. I’ll show you how to do that in the next section.

How Do I Use a Grafana Dashboard?

Setting up

Let’s give you some hands-on experience using the sample application from Grafana. To do so, you need to have Git, Docker, and Docker Compose installed. Then clone the repository to have all the files locally:

git clone github.com/grafana/tutorial-environment

Go to the repository folder and start all the services you’re going to use. This could take a few minutes.

cd tutorial-environment
docker-compose up -d

You can now browse the news site’s sample application on localhost:8081 and Grafana on localhost:3000. Grafana’s credentials are “admin” for both the username and password. The first time you log in, you’ll need to change that password. Make sure you can browse both services and news, as that’s what you’ll use as the data source for the dashboard.

Importing data

Before you get started creating dashboards, you’ll need to let Grafana know which data sources you want to use. You can add multiple types of data sources, such as time series databases, logging and document databases, distributed tracing, SQL databases, and cloud providers. One common data source is Prometheus, which we’re going to use that one today.

Note: You can easily send Prometheus data to InfluxDB using Telegraf plugins. This enables you to combine Prometheus data with other data sources to create a more holistic view of your data sources. InfluxDB works with Grafana natively, enabling you to query and visualize all your data in one place.

In Grafana, click on the sidebar to open the settings menu, then click “Data sources” below “Connections.”

Next, click the “Add data source” button, then click “Prometheus.” In the “Prometheus server URL” text box, enter: http://prometheus:9090. Scroll down to the bottom, then click the “Save & test” button.

You’re going to use Prometheus to get metrics such as latency and the number of requests from the news site. To enrich your dashboards, you can add another data source to get logs from the news site.

Go to the “Data sources” page again and click the “Add new data source” button, scroll down a little bit, and click “Loki” in the “Logging & document databases” section.

Once there, enter http://loki:3100 in the URL text box. Scroll down to the bottom, then click the “Save & test” button.

Open the news site’s sample application to simulate traffic by adding new links, voting, or simply refreshing the browser. We need to get some fresh data to see something in the dashboard.

Creating a dashboard

Now that you have two data sources and have generated some data from the news site, you can visualize the data to get more insights about what’s happening. Go back to the Grafana site and click “Dashboards” in the sidebar menu. Next, click the “New” button in the top-right corner and select “New Dashboard” in the drop-down list. Now click on the “+ Add visualization” button. You should see the two data sources you added earlier, Prometheus and Loki. Add the metrics data source first by clicking the “Prometheus” option.

The screen below has several buttons, text boxes, and a big section where you should see a graph. Configure this screen step by step to create your first visualization of the Prometheus data. Click the “Query” tab below the graph and then click the “Code” option on the right side of the screen. Next, in the “Metrics browser” text box, type the following query:

sum(rate(tns_request_duration_seconds_count[5m])) by(route)

Now click the “Run queries” button—you should see a visual representation of the query on the screen.

You can customize the title of the graph on the right side of the screen. Go to the “Panel options” and type a title like “Traffic” in the “Title” text box. Notice that the visualization title is different now. To finish creating your dashboard, click the “Apply” button in the top-right corner.

Annotating events

Grafana dashboards allow you to add annotations. This tool is especially useful when things go wrong. You can create annotations automatically or manually. You can add annotations every time you release a new version of the site or a marketing campaign starts. With annotations, if the latency decreases, you will have more context as to why.

Let’s suppose you initiated a deployment a few minutes ago and want to manually add this annotation to the dashboard. Before you can annotate events, you need to save the dashboard. To do so, click on the gear icon in the dashboard header. Next, type a name for the dashboard in the “Name” text box, then click the “Save dashboard” button. Click anywhere in the graph from the dashboard, then click the “Add annotation” button.

Add a description and a tag, then click the “Save” button.

If you hover your mouse over the base of the annotation, in the line above the x-axis, you’ll see the annotation. Every time something happens while there’s an annotation in the dashboard, you’ll get extra context about what’s happening on your application. This information can help you make decisions, such as rolling back or quickly pushing a fix.

When you’re done playing with Grafana, you can stop all the container services with the following command:

docker-compose down -v

Wrapping Up

Creating dashboards in Grafana is straightforward. Make sure you have the data sources you need, then build queries that allow you to visualize this data. Adding multiple data sources can help you enrich your dashboards and you can store all this data in InfluxDB. Moreover, when you annotate events, it becomes easier to find the root cause of problems and resolve issues faster.

This post was written by David Snatch. David is a cloud architect focused on implementing secure continuous delivery pipelines using Terraform, Kubernetes, and any other awesome tech that helps customers deliver results.

An Introduction to Apache Superset: An Open Source BI solution

Jay Clifford (InfluxData) — Wed, 28 Dec 2022 07:00:00 +0000

With native SQL support coming to InfluxDB, we can broaden the scope of developer tools used to analyze and visualize our time series data. One of these tools is Apache Superset. So let’s break down the basics of what Superset is, look at its features and benefits, and run a quick demo of Superset in action.

What is Apache Superset?

Apache Superset is an open-source data exploration and visualization platform. Originally started as a hack-a-thon project by Maxime Beauchemin while working at Airbnb, Superset entered the Apache Incubator program in 2017. Apache Superset is similar to enterprise business intelligence solutions, such as Power BI and Tableau, as opposed to other dashboarding software. This is due to its emphasis on data analytics and exploration, so bear this in mind if you are looking for a simple real-time dashboard solution.

Apache Superset is based on a Dataset-Centric methodology, which lies firmly in the middle of a query-centric and semantic-centric architecture.

Credit to The Case for Dataset-Centric Visualization

This architecture promotes the use of datasets that are similar to a Pandas Dataframe but with some further enrichment. Essentially, what we end up with is an enriched tabular structure that contains a subset of semantic characteristics:

Labels and descriptions of the dataset and columns
Metrics as aggregated SQL expressions (AVG, MAX, COUNT)
Timezone and time granularity support
Definitions for which columns can be aggregated and filtered on

This quote from The Case for Dataset-Centric Visualization ties up the dataset approach neatly:

“The dataset metaphor offers a simple and safe “dimensional” playground. In the dataset-centric approach, all charts are built from these datasets that contain a comprehensive collection of relevant dimensions and metrics. This enables users to self-serve within that context. Your team members can typically slice and dice, which entails superpowers like applying arbitrary filters, drilling into dimensional details, drilling through to atomic rows, and choosing the right visualization.”

Apache Superset features and benefits

Now that we have an understanding of what Apache Superset is, let’s take a look at some of its core features and benefits.

Rich visualization library

Apache Superset currently supports 50+ visualization types to experiment with. These cover a variety of data types and use cases. For time series use cases specifically, I would make sure to check out the following visualization types:

Basic: Time series Bar Chart, Time-series Line Chart, Time-series Scatter Plot
Intermediate: Histogram, MapBox, Big Number with Trendline, Time-series Table
Advanced: Calendar Heatmap, Radar Chart, Nightingale Rose Chart

Remember Apache Superset visualizations present a “swiss army knife” of options, so make sure to choose the best visualization to fit your use case. We will explore how and when to use certain visualizations in another blog.

Versatile backend support

Apache Superset supports a large range of databases through SQLAlchemy (plus any required drivers). Support for Apache Superset in the new InfluxDB IOx storage engine is currently in development (specifics will be discussed in a future blog). Apache Superset also provides direct file ingest for formats such as JSON, CSV, Excel, and columnar files.

Customization and deployment

Apache Superset’s cloud-native design makes it scalable. It promotes high availability and deployment support for distributed architectures. Superset also promotes a flexible modular architecture, which lets you choose the right components for your deployment:

webserver (Gunicorn, Nginx, Apache)
metadata database engine (MySQL, Postgres, MariaDB, etc)
message queue (Redis, RabbitMQ, SQS, etc)
results backend (S3, Redis, Memcached, etc),
caching layer (Memcached, Redis, etc),

Apache Superset also allows users to create their own custom visualizations via their visualization plugins feature. These plugins can be created in JavaScript or TypeScript.

Apache Superset tutorial

With the new InfluxDB IOx storage engine quickly approaching general release, it won’t be long until you can deploy Apache Superset as part of your solution. In the meantime here is a small taste of what is possible with InfluxDB and Apache Superset:

First of all, we use the integrated SQL Lab to query against InfluxDB:

Our dataset is based on an IoT simulator for Emergency Power Generators. We save this as an Apache Superset dataset for further manipulation and visualization.

The graph creator provides a low-code interface for selecting and creating your visualization for your dataset. In this case, to keep it simple I am using the Mixed-Time Series visualization:

You can see that based on the dataset, we can allocate fields as metrics, tags as dimensions or filters with a simple drag-and-drop interface. From there we can save our new visualizations directly to a dashboard:

Dashboards offer you extended functionality such as auto-refresh, advanced filtering, and email reports, but we will save that for another blog.

Conclusion

I hope this blog sparks your interest to investigate Apache Superset. It is a powerful business intelligence platform that provides equivalent functionality to the industry giants (Power BI and Tableau). The best part? It’s all based on open-source technology. Are you using Apache Superset? We would love to hear from you, so come join us on Slack and the forums. Share your thoughts — I look forward to seeing you there!

Getting Started with InfluxDB and Grafana

Charles Mahler (InfluxData) — Tue, 20 Dec 2022 07:00:00 +0000

At some point if you’re working with data, you’ll probably want to be able to visualize it with different types of charts and organize those charts with dashboards. You’ll also need somewhere to store that data so it can be queried efficiently.

One of the most popular combinations for storing and visualizing time series data is Grafana and InfluxDB. InfluxDB serves as the data store and Grafana is then used to pull data from InfluxDB (and potentially other data sources) to create dashboards to visualize the data.

In this article you will learn how to:

Install Grafana and InfluxDB using Grafana
Connect Grafana to InfluxDB
Create your first Flux query
Visualize financial data with more advanced queries

InfluxDB overview

InfluxDB is an open source time series database that is optimized for fast and highly available data storage for time series data in use cases like monitoring, application metrics, IoT sensor data, real-time analytics, and more.

Grafana overview

Grafana is an open source data visualization and monitoring platform. It is used to create dashboards and visualize data from a variety of sources like Prometheus or InfluxDB.

Grafana allows users to quickly create visualizations of their data, such as graphs, tables, and heatmaps. It also provides alerting capabilities, allowing users to be notified when certain conditions are met.

Setting up InfluxDB and Grafana

Now let’s get started with installing InfluxDB and Grafana. First you’ll need to make sure you have the following on your computer:

Docker setup

This tutorial will use docker-compose to manage Grafana and InfluxDB. Before running the following script make sure to navigate to the directory where you want the project to be located.

Once you are in the desired folder, run the following script in the command line:

mkdir influxdb-getting-started-with-grafana
cd influxdb-getting-started-with-grafana
cat > ./docker-compose.yml <<EOF
version: "3"

networks:
  monitoring:

services:
  influxdb:
    image: influxdb:2.3.0
    ports:
      - 8086:8086
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:9.0.4
    ports:
      - 3000:3000
    networks:
      - monitoring
EOF

This script will create a docker-compose.yml file which defines the network and images used for InfluxDB and Grafana. To start the containers you just need to run the following command:

docker-compose up -d

InfluxDB setup

Setup of credentials required when initiating connection to InfluxDB is as follows.

Create initial super-admin credentials, organization, bucket and the all-access security token. Run the influx setup command:

influx setup --name myinfluxdb2 --host http://localhost:8086 \
  -u admin -p admin54321 -o my-org \
  -b my-bucket -t my-token -r 0 -f

The superuser, organization, bucket, and access token have been created. In addition, the influx command creates a new server configuration object and stores it into the active config named myinfluxdb.

Note: You can define as many configs as you want to work with multiple InfluxDB2 servers easily. The configuration objects are stored locally in the $HOME/.influxdbv2/configs file on your computer. To list all available server configurations use the command influx config ls.

Grafana and InfluxDB connection setup

Add data source in Grafana UI

Open in browser http://localhost:3000/datasources
Sign in as user admin, password admin.
Click on Skip to skip the question about the new password.
In the left menu, click on the Gear icon, to open Data Sources.
Click on Add data source.
Select InfluxDB.
Replace InfluxQL with Flux in the dropdown called Query Language.
Type http://influxdb:8086/ at the URL field in the section called HTTP.
Write my-org into the Organization field in the InfluxDB Details section.
Type my-token in the Token field. (Once the save and test button is clicked, the password is hidden and replaced with configured.)
Save & Test: Success will display two green notifications (3 buckets found + Datasource updated). Please see below.

Note: The address http://influxdb:8086/ is the address visible from the Grafana container in the internal network orchestrated by docker-compose.

Your first Flux query from Grafana

(Update: InfluxDB 3.0 moved away from Flux and a built-in task engine. Users can use external tools, like Python-based Quix, to create tasks in InfluxDB 3.0.)

Open Grafana Explorer located in the left menu of Grafana GUI as the Compass icon.

Type the following simple query on line 1 of the Explorer:

buckets()

Hit the Run Query button located at the top right corner. The result is the table listing 3 buckets.

The buckets _monitoring and _tasks are internal InfluxDB buckets. The bucket is called my-bucket, which was created with the influx setup command run earlier.

InfluxDB is also running at http://localhost:8086. You can login to the InfluxDB UI with the following credentials:

User: admin
Password: admin54321

Doing more with Flux and Grafana

In this section you will learn more about how to use Flux and how to create more advanced visualizations using Grafana Dashboards.

Focus points:

Importing the CSV file of financial data into InfluxDB
Usage of simple Flux queries to extract time series
Data visualization in the candlestick chart saved as a Panel in Grafana Dashboard

The data represents a historical time series of daily stock prices from selected companies, with a time range between the years 2010 and 2016.

The following figure shows the head of the CSV, to preview how the spreadsheet looks:

The data set is a 50MB file, containing daily values which are all double precision numbers, for __open, low, high and close__prices and traded volume, of 502 companies between the years 2010 and 2016.

To analyze this size of data in a spreadsheet processor is a tedious process.

Import data into InfluxDB

First you will need to download the financial data CSV onto your computer. Open your terminal and navigate to the folder where the CSV file has been downloaded. Before using influx CLI, make sure you are working with the correct active configuration.

Use the following command to see the available configurations:

influx config ls

The output should produce a line with the config from InfluxDB setup:

Active  Name         URL                       Org
*       myinfluxdb2  http://localhost:8086     my-org

Note: The asterisk signifies a config is active, and this means the –host and –org arguments don’t need repeated entry on CLI each time the influx command is used.

Import CSV data into InfluxDB

Run this influx CLI Data Import & Annotation CMD:

influx write -b my-bucket -f ./stock-prices-example.csv \
  --header "#constant measurement,stocks" \
  --header "#datatype dateTime:2006-01-02,tag,double,double,double,double,double"

Note: This influx write command takes roughly around 12-15 seconds to complete. To print the run time duration in the results printout of importing the data, add this “dry run” script to the query: time influx write -b my-bucket

Visualize time series data in Grafana Explorer

To get started with creating your dashboard, choose InfluxDB as your data source using the dropdown selector:

Query and visualizing

Run the following Flux query

from(bucket: "my-bucket")
  |> range(start: 2016-01-01T00:00:00Z, stop: 2016-01-31T00:00:00Z)
  |> filter(fn: (r) => r["_measurement"] == "stocks")
  |> filter(fn: (r) => r["symbol"] == "AAPL")
  |> aggregateWindow(every: 1d, fn: mean, createEmpty: false)
  |> yield(name: "mean")

The graph area is still empty because the result is out of data range. Click on the Zoom to data button to see the AAPL raw data in the right time range, January 2016.

Note: Adapting time frame of queries to user needs is streamlined in the Grafana UI. Making an adjustment to data time range in Grafana UI is interactive and simple to adjust (see figure below).

Create your first Grafana dashboard

Click on the Add to Dashboard button located at the top of the page.

Message box with options appears. Select New Dashboard and open the Grafana Dashboard in a New Window not to lose the Explorer tab.

After the new browser tab opens:

Type e on your keyboard, wait a few seconds, and the edit mode of the Panel will appear on the page.
Select Candlestick type on the right side.

Then:

Modify the previous Flux command, and extend the time range on line 2 from 2010-01-01 to 2016-12-31.
Click the Apply button.

To zoom out, you can use the magnifying glass button with a minus sign in the Grafana toolbar. The keyboard shortcut is Ctrl+Z.

To zoom in, select a bigger time range with your mouse or just use the time range selector (dropdown control).

Enter edit mode and try out other possibilities. Enjoy your new dashboard.

Doing more with Grafana dashboards and Flux

Using Flux with Grafana provides a number of options and functionality that aren’t possible using other query languages, such as joining data from multiple sources and doing advanced manipulation and transformation of your data before returning the query.

Joins - create graphs that span multiple buckets. For example, you might want a Grafana chart that displays both bytes transferred and requests per second; Flux allows you to query these two measurements and join them into a single table.
Multiple data sources - enrich time series data with metadata from relational databases such as MySQL, MariaDB, Postgres, Microsoft SQL Server, Snowflake, SQLite, AWS Athena, and Google BigTable; or from CSV files. This is useful when, for example, your time series data includes customer number fields but not customer names. Flux allows you to pull in the customer name so that it can be displayed in your Grafana dashboards.

Some additional Flux features you might find useful:

Sort by tags
Pivot data
Group by any column
Window functions by date

Additional resource on using InfluxDB and Grafana: How to Integrate Grafana with Home Assistant