InfluxData Blog - Use Cases

Community Spotlight: A Native iOS App for Your InfluxDB Data

Ashley Fowler (InfluxData) — Tue, 19 May 2026 08:00:00 +0000

One of the things we love most about building an open source platform is seeing what the community creates with it, and independent developer Anton Havekes recently built something we just had to share.

Anton put together Influx Dashboard, a native iOS app that connects to your InfluxDB instance and brings your time series data straight to your phone. We’re genuinely thrilled to see this kind of work come out of the community.

A quick note before we dive in: this is entirely Anton’s project, built and published by him. InfluxData has no commercial relationship or financial stake in the app. We’re sharing it simply because we think it’s a great piece of community work and because surfacing what people build on InfluxDB is something we’ll always make time for. So, thank you, Anton, for the awesome work you’ve done!

Here’s how it works.

What is Influx Dashboard?

Influx Dashboard is a mobile-first visualization tool for InfluxDB. It supports InfluxDB versions 1, 2, and 3—including Core, Enterprise, and Cloud editions—and gives you a full-featured charting experience on your phone.

Think of it as the community-built, mobile companion to the InfluxDB 3 Explorer UI. Instead of opening a browser on your laptop, you open an app. Instead of sitting at a desk, you monitor your data from wherever you are.

Key features include:

Real-time line, bar, gauge, and pie charts
Support for SQL queries
An intuitive drag-and-drop dashboard builder
Direct, secure connections to your InfluxDB server
Works with any kind of time series data: server metrics, IoT sensors, weather stations, factory equipment, and more

Influx Dashboard is available on the Apple App Store for $4.99 (InfluxData is not commercially affiliated with this app and receives no proceeds from purchases). Once downloaded, open the app, and you’ll be ready to connect to your InfluxDB server.

Step 2: Connect to Your InfluxDB Server

Before you can see any data, you’ll need to connect the app to your InfluxDB instance. Tap Add Server on the main screen.

You’ll need a few things from your InfluxDB setup:

Server URL: the address of your InfluxDB instance
API Token: your authentication token (found in InfluxDB under Management → Tokens)

If you’re using InfluxDB Cloud, your server URL will look something like https://us-east-1-1.aws.cloud2.influxdata.com. If you’re connecting to a self-hosted instance, your admin can provide the URL and token.

Once connected, the app loads your available databases, and you’re ready to build your first dashboard.

Step 3: Build Your Dashboard

Tap + New Dashboard and give it a name. Then tap Add Chart to create your first visualization.

The chart builder is straightforward, even if you’ve never written a database query before. You’ll see options for:

Chart Title: give your chart a descriptive name

Chart Type: choose from line, bar, gauge, or pie

Database: select which InfluxDB database to query

Table: select the specific measurement or table within that database

Column: choose which field to visualize

Time Range: pick how far back to look (last hour, last 24 hours, last 7 days, and more)

Aggregation Window: control how data points are grouped over time to smooth out your chart

Step 4: Query Your Data

One of the things that makes Influx Dashboard powerful is that it works with virtually any kind of time series data. During our testing, we connected to several different databases, each representing a completely different real-world use case:

Server monitoring: CPU usage and memory metrics from a live server, showing system health at a glance.

Industrial IoT: Factory sensor data streamed via MQTT, the kind of data you’d see in a manufacturing or industrial environment.

Weather stations: Real-time weather observations, including temperature, humidity, wind speed, and barometric pressure, from stations across multiple locations.

Personal IoT sensors: A live “feels like” temperature sensor from a personal weather station in Holualoa, Hawaii.

The app handled all of them identically: same interface, same workflow, completely different data. That versatility is one of Influx Dashboard’s biggest strengths.

Step 5: Explore Your Charts

Once your charts are built, they appear on your dashboard and refresh automatically with live data.

Tap any chart to expand it to full screen for a closer look. From the full-screen view, you can inspect trends, zoom in on specific time windows, and get a clearer picture of what your data is doing.

Use the time range selector at the top of the dashboard to zoom in or out, from the last hour to the last 30 days. This is especially useful when you’re looking for a specific event or trying to understand a longer trend.

Step 6: Manage Your Servers

From the Settings screen, you can manage your server connections, add new servers, or update your credentials. If your organization uses multiple InfluxDB instances, for example, a production server and a staging server, you can easily switch between them. You can also rename or delete servers you no longer need, keeping your workspace clean as your infrastructure evolves.

Why this matters

DevOps engineers, IoT developers, data engineers, and system administrators often need visibility into their time series data when they’re nowhere near a laptop, from a factory floor, at a remote deployment, or just away from their desk. Influx Dashboard closes that gap. Not through an official InfluxData product, but through something a community member decided to go build.

That’s exactly the kind of activity an open source ecosystem should produce, and it’s exactly why we wanted to share it.

Get started

Download Influx Dashboard on the Apple App Store (independently developed by Anton Havekes—InfluxData is not affiliated with this app and receives no proceeds from purchases).
Don’t have InfluxDB yet? Get started free with InfluxDB 3 Core.
Explore the InfluxDB 3 Explorer UI at docs.influxdata.com
Built something on InfluxDB you want to share? Join the conversation at the InfluxData Community Forums.

IoT Made Easy with Node-RED and InfluxDB

Charles Mahler (InfluxData) — Thu, 17 Apr 2025 10:04:00 +0000

In this article you will learn about Node-RED, a popular tool for rapidly gluing together different types of hardware and software. You’ll learn about some of the core concepts of Node-RED and then learn how to make some workflows like storing data from a sensor using an MQTT broker and InfluxDB.

Node-RED and InfluxDB for IoT webinar

Watch the webinar

What is Node-RED?

Node-RED is an open-source, low-code, visual programming tool based on the concept of flow-based development. The idea behind it is to make it very easy to connect APIs, hardware devices, and anything else accessible over some type of network connection.

Node-RED provides a number of pre-built “nodes” which allow non-developers to create custom integrations and automations using a visual interface, while also allowing developers to write their own custom nodes using JavaScript or external service written in another language when needed. In many ways Node-RED can be seen as one of the earliest examples of the low-code movement that has been rising in popularity.

Node-RED was initially developed by IBM and then contributed to the OpenJS foundation. Arguably the biggest strength of Node-RED is the ecosystem and community with over 14,000 stars on GitHub and a community library with over 3000 pre-built nodes and 2000 customizable flows that cover many common use cases.

In short, the benefits of using a tool like Node-RED are:

No reinventing the wheel - tons of pre-built functionality
Huge community with solutions to many common problems available
Easy to extend with custom functionality for your use case

Node-RED core concepts

In this section I will cover a few of the main components of Node-RED that will help you hit the ground running.

Nodes

As you might expect based on the name, nodes are an important part of Node-Red. They are the fundamental building blocks for everything you do when working with Node-Red. Nodes are triggered by either receiving a message object from a previous node or an external event like an HTTP request. The node processes the message or event and then passes it on to the next node.

Here are a few of the most common and useful nodes provided by Node-RED:

Inject - The inject node allows you to manually start a flow by injecting a message payload. The payload can be a string, number, or object with multiple properties. This node can also be set to run at a set time or regular interval.
Change - The change node is a nice alternative to writing a custom function to do basic transformations or modifications like adding or removing properties on the message object.
Debug - The debug node can be used to help develop flows and find where any potential bugs are occurring. By default it will send the message payload to the debug sidebar but can also be customized to send different information and also print to the terminal where you started Node-RED.
Switch - The switch node allows messages to be routed to different branches of a flow based on defined rules. For example, if you have a sensor detecting temperature you might want to create an alert based on a certain threshold. If the temperature isn't over that threshold you can end the flow.
Function - The function node allows you to write custom JavaScript code for use cases where the pre-built nodes provided by Node-RED don't support your use case

Beyond these nodes, there are a number of other nodes for working with messages, storing data, and handling different types of network requests.

Flows

Flows can be thought of as containers for organizing a sequence of nodes. They are represented in the Node-RED editor as an independent tab. There are a number of different strategies and best practices you can use to determine when you should divide a flow into multiple different flows or keep it all together.

Messages

Messages are JavaScript objects that are passed along from the beginning to the end of a flow through nodes. Properties on the object can be added or modified by a node. Convention is to store the most relevant information in a property called payload on the message object, which itself is generally called msg.

Context

Context within Node-RED is essentially the ability to define global variables that can be accessed by nodes directly rather than from within the message being passed to the node. Context can be scoped at 3 different levels. The first is node level, where only the node that set the value can access it. The second is flow level where the value is visible to all nodes within that flow. The final context level is truly global, any node within the Node-RED instance can access the value.

Node-RED Flow Tutorial

Now let’s jump into the actual tutorial and make a few flows using Node-Red. If you want to follow along you will need a running instance of Node-RED. You have multiple options for how to install Node-RED, if you already have NPM installed that is probably the easiest option. You can also use Docker or install from source using Git. You can see all available options in the Node-Red documentation.

Once Node-RED is installed you can open your web browser and visit localhost at port 1880 and see the Node-Red editor. We’ll start out making a basic “hello world” flow to confirm everything is working properly.

First, drag an inject node from the left hand sidebar and put it onto the editor. Click on the node and change the payload to a string with value and whatever input you want, I’ll follow tradition and use “hello world” for my value.

Now grab a debug node and place it on the editor, by default it will output the message payload passed by the previous node. Click on the debug node and check the box for “system console” so our message will also be printed to the terminal as well as the debug sidebar. Your editor should look something like this:

Now you can click deploy in the top right corner of the Node-Red editor. Click the box on the left side of your inject node to start your flow. If everything worked properly you should see values appearing in both your terminal and the Node-Red debug sidebar:

Node-RED MQTT flow

Now we’ll move onto something a little more complicated. For this flow we will connect Node-RED to an MQTT broker, transform the data using a Node-Red function node, and then store that data using InfluxDB for long term storage and analysis.

The first thing you need to do is install the InfluxDB Node-Red package. To do this click on the hamburger menu icon on the top right of the editor next to the deploy button. Then click on ‘manage palette’ and then click to the Install tab. Type InfluxDB into the search box and select the package called node-red-contrib-influxdb.

With that package installed if you scroll to the bottom of your node palette on the left hand of your screen you should see three new nodes added for working with InfluxDB.

You’ll now want to create an InfluxDB instance. You can create a free InfluxDB Cloud account or install InfluxDB locally. Create a bucket and then grab an API token with access to that bucket. You can find instructions on how to do this and much more in the InfluxDB Documentation.

For this flow instead of using an inject node to start things, we will use the MQTT In node and connect to a free public MQTT broker provided by HiveMQ. If you already have an MQTT broker installed locally you can also use that, the configuration is the same. Click on the MQTT node after dragging it onto the editor and add a new broker by providing the server URL and port number. For HiveMQ the URL is broker.hivemq.com and the port is 1883. Leave the version as MQTT V3.1.1.

Set the Action for the node to subscribe to a single topic and put in the topic you want to listen to, it can be whatever you want in this case. I’ll be setting the topic as nodered for simplicity. Once you deploy this flow, Node-RED will connect to the broker and listen to any messages under that topic name.

We will be passing a temperature number through MQTT to simulate some sort of IoT sensor recording air temperature. By default this will be passed as a string, so we need to create a function to convert it into an integer before storing it with InfluxDB. Add a function node to the page and put the following code into the node:

msg.payload = Number(msg.payload)
return msg;

Now add the InfluxDB Out node and connect it to the function. Create an InfluxDB server instance by adding in your URL and API token. Then add in your organization ID and the name of the bucket where you want to store your data.

To test everything out you can use any MQTT client, HiveMQ also provides a websocket client that is easy to use here. Set the topic to match the topic in your MQTT node and then publish a message with a number as the value, something like this:

If everything worked properly you should start seeing data in your InfluxDB instance every time you submit an MQTT message to the broker:

The flow itself should look similar to this, note the MQTT node provides a notification box letting you know if you are able to connect to the broker:

Node-RED project ideas

So now you have a basic understanding of how to work with Node-RED, but what are some actual real-world use cases? In this section I’ll give you a few potential areas you can explore to build out your own custom projects with Node-Red.

Data visualization and analysis

Node-Red works best as the glue between different components, making it easy to move data around. While Node-RED does provide some basic storage and visualization capabilities it makes sense to use specialized tools for these things. As you saw with the MQTT example InfluxDB can be used for storing and visualizing data. For analysis the Flux scripting language provides tons of functionality for analyzing time series data. The Node-RED InfluxDB package also gives you a node to query data from InfluxDB into Node-Red, so you have access to other tools in the ecosystem if you prefer those.

Smart home

Home Assistant is a popular home automation framework that provides integrations with numerous smart devices. But many users find it difficult to build out more customized workflows using the tools provided by Home Assistant. Node-RED is a common choice for these situations and there are a ton of potential project options that will help you learn more about Node-RED and also be useful from a practical perspective.

Node-RED as an event processing framework

While a lot of the focus in this article has been IoT related, I think that looking at Node-RED as a tool to process any kind of event is the best way to see how much you can really do with it. Node-RED gives you the ability to quickly connect and listen to HTTP, websockets, TCP, MQTT, and numerous other network protocols.

As a result it can be used to move data to and from almost any type of application and allows you access to the entire NodeJS ecosystem via NPM modules. You also always have the option to reach out to other services via API or running local commands via the Exec node. With Node-RED your only limitation is really your own creativity for how to combine the tools it provides.

Additional resources

Check out these additional resources to learn how you can use Node-RED with InfluxDB

Introducing the Time Series Buying Guide for IIoT

Jessica Wachtel (InfluxData) — Wed, 29 Jan 2025 07:00:00 +0000

All machinery and equipment, including their controls and sensors, tell a story through the data they collect. This data, or Industrial Internet of Things (IIoT) data, provides a detailed narrative about the machines, offering actionable insights to improve operations. IIoT data empowers businesses to optimize and enhance industrial processes by detailing operational status, performance metrics, usage patterns, health diagnostics, and environmental conditions.

The value of leveraging IIoT data

Harnessing IIoT data can significantly improve operational and business efficiency. When fully utilized, it transforms raw information into tangible benefits, such as:

Real-time anomaly detection and intervention
Maximized productivity, minimized waste and downtime
Reduced unplanned outages and improved maintenance forecasting

Extracting the full value from machines, equipment, industrial controls, and sensor data drives increased revenue by reducing outages, optimizing processes, and lowering error rates. However, achieving these benefits requires tools that preserve and enhance the quality of datasets, ensuring they remain actionable and accurate.

The challenges of managing IIoT data

IIoT data captures changes over time, from subtle fluctuations to catastrophic shifts. This temporal context makes it “time series” data—a sequence of data points collected or recorded at regular intervals. Managing time series data poses unique challenges due to its scale, speed, and complexity:

Massive scale: Continuous high-speed, high-volume data streams
Real-time action: The need for immediate analysis and response within data streams
Data cardinality: High numbers of tags collected result in high cardinality, which can strain system performance

Businesses can easily generate billions of IIoT time series data points per second, demanding ingestion and storage solutions that keep pace with real-time analytics.

Why high-fidelity data matters

Successfully managing these challenges results in high-fidelity data—complete and accurate datasets that provide precise insights. High-fidelity data enables businesses to detect issues early, build accurate analytics, and create reliable models.

Conversely, failing to address these challenges results in incomplete datasets, which lack the necessary precision to capture early-stage anomalies or build actionable insights.

Safeguard and maximize value from your data

“Time Series Buying Guide for IIoT” explores how manufacturers can identify the tools and architectural practices necessary to achieve high-fidelity data sets. Readers will gain insight into the unique characteristics of IIoT time series data compared to other data types. The guide highlights which tools can maximize the value of IIoT data and warns against those that may appear beneficial but prove detrimental over time. Additionally, it offers practical advice on seamlessly integrating specialized IIoT tools into existing architectures and environments, eliminating the need for costly, large-scale system overhauls.

Download the “Time Series Buying Guide for IIoT” now.

Maximizing IIoT Impact with Open Data, AI, and Advanced Analytics: A Comprehensive Guide

Jessica Wachtel (InfluxData) — Wed, 18 Dec 2024 07:00:00 +0000

This tech paper was created by IIoT World and InfluxDB. This post was originally published on IIoT World.

The Industrial Internet of Things (IIoT) is revolutionizing industries like manufacturing, energy, and logistics by creating more intelligent, interconnected systems that elevate productivity and efficiency. With IIoT, machines, systems, and sensors are linked in real-time, streamlining industrial automation and making predictive maintenance a reality—all while reducing downtime and costs.

Transforming industry with the power of IIoT

In today’s competitive landscape, an open data ecosystem is crucial for maximizing IIoT’s potential. Open data allows companies to modify their systems to meet unique needs without being locked into proprietary solutions. This flexibility lets companies control expenses and integrate new technologies, making infrastructure management more streamlined as they scale or expand operations.

Benefits of open data ecosystems in IIoT

Unlike rigid, closed systems, open data solutions empower manufacturers to innovate and customize, adapting to specific operational needs. This flexibility improves resource allocation and data flow, giving companies a significant edge by eliminating vendor lock-in. With open data, industrial firms can make faster adjustments and implement changes more economically, ultimately leading to smoother management and a more agile approach to new technologies.

Key tools: AI and time series databases for real-time insights

As IIoT systems grow, so do the demands for real-time data analysis. Artificial intelligence (AI) and time series databases are indispensable for managing vast amounts of time-stamped data generated by IIoT devices. This booklet showcases how AI tools like TensorFlow, PyTorch, and Apache Spark MLlib optimize industrial processes and deliver real-time insights. Time series databases, such as InfluxDB, play a pivotal role, enabling predictive maintenance, anomaly detection, and enhanced quality control to keep operations running smoothly.

Check out the guide for smarter industrial automation

“The Value of Open Data AI, ML, and Analytics Tools for IIoT” dives into how manufacturers can leverage these tools for a competitive edge. Readers will learn about: AI Tools for Intelligent Data Insights: Discover how advanced AI tools can forecast maintenance needs and spot anomalies before they cause issues, boosting efficiency across the board. Flexible and Scalable Open Data Ecosystems: Learn how open data solutions offer customization options that avoid vendor lock-in, empowering companies to choose the tools that best suit their needs. Real-Time Analytics with Time Series Databases: Explore how time series databases like InfluxDB manage large-scale, time-stamped data, allowing immediate insights and faster, data-driven decisions. Advanced Analytics for Smarter Decision-Making: See how IIoT data, paired with AI, helps decision-makers spot trends, enhance efficiency, and optimize production.

Download the guide today to explore how these cutting-edge tools and open data ecosystems are paving the way for a more efficient, flexible industrial future. Embrace the power of IIoT to keep your operations optimized and ahead of the competition.

Case Study: Modernizing SPEN's Tech Stack with Capula and InfluxDB

Jessica Wachtel (InfluxData) — Tue, 05 Nov 2024 08:00:00 +0000

Background

Scottish Power Energy Networks (SPEN) started a journey to improve its technology by working with Capula and using InfluxDB. Let’s explore the use case, goals, and the role of InfluxDB in this modernization effort.

In 2020, SPEN looked at its data historian systems and realized it needed a better solution to meet future data storage needs. With a focus on modernizing their systems and addressing business challenges through data, they sought a new technology solution.

Capula, a seasoned systems integrator familiar with SPEN’s infrastructure, recommended InfluxDB, a time series database that can manage high volumes of diverse data types. Capula provided comprehensive consultancy services, employing a layered approach to understanding organizational needs, design solutions, and apply technologies that enhance the convergence of operational technology (OT) and information technology (IT). Capula’s efforts included conducting tests and demonstrations to ensure system reliability and cybersecurity, such as Telegraf buffering, visualizing alarm data, and ingesting real-time data using OPC-UA protocols.

Capula undertook a two-phase proof-of-concept project focusing on learning the intricacies of InfluxDB’s architecture and its integration within SPEN’s systems. They performed functional benchmarking, addressed issues related to high data cardinality, optimized data ingestion processes, and tested the distributed architecture of InfluxDB to ensure system stability. Additionally, they assisted in designing data retention policies and preparing for future updates and cybersecurity enhancements.

This ultimately led to a successful initial proof of concept in 2021, which expanded into a more comprehensive integration with SPEN’s existing systems that aligned with their modernization goals and enhanced data quality, reliability, and scalability.

Goals

The specific goals SPEN aimed to achieve by modernizing its data historian systems included:

Handling dramatic increases in data storage requirements.
Consolidating analog and digital data into one system for easier comparison and analysis.
Ensuring the system’s reliability and stability.
Preparing for future data volume and requirements.
Testing new technologies to build knowledge on technical requirements and inform future purchases.
Enhancing decision-making processes by integrating diverse data types and improving data quality and integrity.

Role of InfluxDB

InfluxDB played a crucial role in modernizing SPEN’s tech stack by providing a scalable and efficient solution for handling high volumes of diverse data types. It enabled real-time data ingestion and processing, essential for managing energy sensor data. InfluxDB’s ability to blend different data types, such as field, asset, and operational data, facilitated easier access and integration, enhancing decision-making processes. Additionally, its advanced features for data cleaning, system interoperability, and the capability to design custom data retention policies further demonstrate its effectiveness.

Benefits

The new system ‌significantly improved SPEN’s data quality, reliability, and scalability. It allows for the ingestion of diverse data types without a specific data model (aka schema-on-write), which enhances data quality by enabling the blending of various data sources. The system’s distributed architecture ensures reliability by maintaining database access even if a node becomes unavailable. The system achieves scalability through its ability to handle high volumes of data and its optimization for write-intensive workloads, making it suitable for SPEN’s growing data requirements.

Wrapping up

Overall, the journey involved extensive testing, functional benchmarking, and the development of a scalable and efficient data management solution to support SPEN’s goal of achieving net zero carbon by 2050. InfluxDB stands to play a pivotal role in SPEN’s ongoing transformation, providing the necessary tools and capabilities to meet its current and future data management needs.

Go here to learn more about using InfluxDB for energy and utilities.

How WOW! Modernized Legacy Infrastructure Monitoring with InfluxDB and Kafka

Jessica Wachtel (InfluxData) — Wed, 01 Nov 2023 08:00:00 +0000

With over 500,000 residential, business, and wholesale customers across multiple markets in the United States, WideOpenWest (WOW!) is one of the United States’ largest broadband providers. They aim to connect homes and businesses to the world with fast and reliable internet, TV, and phone services.

WOW!’s challenge

As a data-driven organization, WOW!’s support team wanted to detect network outages using data from the network nodes (devices), rather than customer calls. But WOW!’s support engineering team faced challenges from the diversity of their distributed field equipment. WOW!’s networks consist of both brownfield and greenfield builds, which are cable and fiber, or all-fiber networks, respectively. WOW!’s nodes (modems and other DOCSIS devices) are different models, created by different vendors, and come with their own set of specifications. It’s cost prohibitive for WOW! to make every network a greenfield build and buy uniform technology because WOW! has networks all over the United States and monitors upwards of 800,000 modems. Monitoring this massive number of varying technologies kept WOW!’s engineering team locked in a disjointed legacy observability solution.

WOW!’s legacy monitoring platforms offered insight into their infrastructure, but diverse vendor-locked data collection, monitoring restrictions, and technical requirements led to separate platforms and dashboards for different groups of networks and nodes. The lack of a centralized observability platform meant WOW! engineers had no way of analyzing all their nodes and networks in one place. In addition to the complexities added by WOW!’s disjointed dashboards and platforms, WOW!’s engineering team used a time series database that failed regularly.

WOW!’s engineering team needed to better understand the health of each node and their networks overall. The engineers hypothesized that a centralized datastore with real-time and historical analytic viewing capabilities will provide more visibility into the node and network health.

The solution

Time series benchmark testing introduced WOW! engineers to InfluxDB’s superior write speeds. When WOW! engineers decided to modernize their legacy system, they selected InfluxDB Enterprise (now available as InfluxDB Clustered) as the time series database backend. InfluxDB offered WOW! engineers something no other vendors or monitoring solutions could – the flexibility to work around all restrictions to create a single monitoring platform.

The architecture

WOW!’s monitoring platform consists of a four-node InfluxDB cluster in production and a two-node InfluxDB cluster running on OpenStack for testing. WOW! engineers use InfluxDB to derive insight from real-time analytics, create visualizations, and trigger alerts and the troubleshooting process. WOW! engineers leverage InfluxDB’s alerting frameworks to send alerts via Slack, email, and ServiceNow, their automatic ticketing platform. WOW! engineers use Grafana to create customized dashboarding. This includes custom dashboarding for real-time analytics and historical trend analysis.

The engineers built a Kafka cluster and placed it between the data sources and InfluxDB. This provides an additional layer of redundancy and control of the data flow. WOW! engineers use Simple Network Management Protocol (SNMP) polling and traps to collect data from roughly 650,000 cable modems in five-minute cycles. WOW! engineers collect data from most of their virtual machines (VMs) and containers using Telegraf. After the modernization, WOW! engineers implemented an infrastructure-as-code system using Ansible. The engineers now use Ansible to automate cluster setup and installation.

Leveraging Telegraf, InfluxDB, and Grafana, also known as the TIG stack, created a complete picture of WOW!’s device and network health. This led to higher functionality and lower downtimes.

To learn more about WideOpenWest, read the full case study here.

Getting Started with Infrastructure Monitoring

Charles Mahler (InfluxData) — Wed, 11 Oct 2023 07:35:00 +0000

This article was originally published on The New Stack and is reposted here with permission.

By taking advantage of monitoring data, companies can ensure their infrastructure is performing optimally while reducing costs.

While building new features and launching new products is fun, none of it matters if your software isn’t reliable. One key part of making sure your apps run smoothly is having robust infrastructure monitoring in place. In this article you will learn about the following:

The different components of infrastructure monitoring.
Popular tools used for infrastructure monitoring.
How to set up monitoring for an application.

If you prefer video, you can also check out this presentation, which covers some of the themes discussed in this article.

Components of infrastructure monitoring

Infrastructure monitoring consists of a number of different architecture components that are needed to serve a modern application. To ensure software is reliable, all of these components need to be properly monitored.

Network monitoring — Network monitoring focuses on hardware-like routers and switches and involves tracking things like bandwidth usage, uptime and device status. It is used to identify bottlenecks, downtime and potentially inefficient network routing.
Server monitoring — Server monitoring is focused on monitoring the performance and health of physical and virtual server instances. Metrics like CPU, RAM and disk utilization are common. Server monitoring is important for capacity planning.
Application performance monitoring (APM) — APM is focused on software and is used to track how an application is performing at every layer from the UI to how data is stored. Common metrics are things like error rates and response times.
Cloud infrastructure monitoring — Cloud monitoring, as the name implies, is about monitoring cloud infrastructure like databases, different types of storage and VMs. The goal is to track availability and performance, as well as resource utilization to prevent over or under provisioning of cloud hardware.

Each of these types of monitoring act as a different lens for teams to view and manage their infrastructure. By taking advantage of all of this data, companies can ensure their infrastructure is performing optimally while reducing costs.

Tools for infrastructure monitoring

Choosing the right tools for the job is critical when it comes to creating an infrastructure monitoring system. There are a number of open source and commercial options available. You also have the option of choosing a full-service solution or creating your own custom solution by combining specialized tools. Regardless, there are three main questions to consider: How are you going to collect your data, how to store the data and what will you do with the data? Let’s look at some of the tools available for accomplishing each one.

Data collection tools

One of the biggest challenges with infrastructure monitoring is collecting data that may be coming from many different sources, often with no standardized protocol or API. The key goal here should be to choose a tool that saves you from having to reinvent the wheel, doesn’t lock you in and is extensible so you can scale or modify data collection as your app changes.

Telegraf

Telegraf is an open source server agent that is ideal for infrastructure monitoring data collection. Telegraf solves most of the problems mentioned above. It has over 300 different plugins for inputs and outputs, meaning you can easily collect data from new sources and output that data to whichever storage solution works best for your use case.

The result is that Telegraf saves you a ton of engineering resources by not having to write custom code for collecting data and prevents vendor lock-in because you can change storage outputs easily. Telegraf also has plugins for data processing and transformation, so in some use cases it can simplify your architecture by replacing stream-processing tools.

OpenTelemetry

OpenTelemetry is an open source set of SDKs and tools that make it easy to collect metrics, logs and traces from applications. The primary advantage of OpenTelemetry is that it is vendor agnostic, so you don’t have to worry about getting locked into an expensive APM tool with high switching costs. OpenTelemetry also saves your developers time by providing tools to make instrumenting applications for data collection easy.

Data storage tools

After you start collecting data from your infrastructure, you’ll need a place to store that data. While a general-purpose database can be used for this data, in many cases you will want to look for a more specialized database designed for working with the types of time series data collected for infrastructure monitoring. Here are a few available options:

InfluxDB

InfluxDB is an open source time series database designed for storing and analyzing high volumes of time series data. It offers efficient storage and retrieval capabilities, scalability and support for real-time analytics. With InfluxDB, you can easily capture and store metrics from various sources, making it a good fit for monitoring and analyzing the performance and health of your infrastructure.

Prometheus

Prometheus is an open source monitoring and alerting toolkit built for collecting and storing metrics data. It is specifically designed to monitor dynamic and cloud native environments. Prometheus provides a flexible data model and powerful query language, making it well-suited for storing infrastructure monitoring data. With its built-in alerting and visualization capabilities, Prometheus enables you to gain insight into the performance and availability of your infrastructure.

Graphite

Graphite is a time series database and visualization tool that focuses on storing and rendering graphs of monitored data. It is widely used for monitoring and graphing various metrics, making it a suitable option for storing infrastructure monitoring data. Graphite excels at visualizing time series data, allowing you to create interactive and customizable dashboards to monitor the performance and trends of your infrastructure. Its scalable architecture and extensive ecosystem of plugins make it a popular choice for monitoring and analyzing infrastructure metrics.

Data analysis tools

Once you’ve got your data stored, it’s time for the fun part, actually doing something with it to create value. Here are a few tools that you can use for analyzing your data.

Grafana

Grafana is a powerful open source data visualization and analytics tool that allows users to create, explore and share interactive dashboards. It is commonly used for analyzing infrastructure monitoring data by connecting to various data sources such as databases, APIs and monitoring systems. With Grafana, users can create visualizations, set up alerts and gain insights into their infrastructure metrics, logs and traces.

Apache Superset

Apache Superset is a modern enterprise-ready business intelligence web application that enables users to explore, visualize and analyze data. It provides a user-friendly interface for creating interactive dashboards, charts and reports. When it comes to analyzing infrastructure monitoring data, Apache Superset can be used to connect to monitoring systems, databases or other data sources to explore and visualize key metrics, generate reports and gain insights into the performance and health of the infrastructure.

Jaeger

Jaeger is an open source, end-to-end distributed tracing system that helps users monitor and troubleshoot complex microservices architectures. It can be used for analyzing infrastructure monitoring data by providing detailed insights into the interactions and dependencies between different components of the infrastructure. Jaeger captures and visualizes traces, which represent the path of requests as they travel through the system, allowing users to identify bottlenecks, latency issues and performance optimizations in the infrastructure.

Infrastructure monitoring tutorial

Now let’s look at an example of how to implement a monitoring system for an application. This tutorial will focus on a combination of open source tools known as the TIG stack: Telegraf, InfluxDB and Grafana. The TIG stack allows developers to easily build an infrastructure monitoring solution that is scalable and extensible in the long term.

Architecture overview

The example application for this tutorial is a chat app powered by an AI model that returns responses based on user input. The app has a hybrid architecture with the backend hosted on AWS, and the AI model is run on dedicated GPUs outside the cloud. The primary challenge is ensuring reliability of the service while also scaling infrastructure due to rapid user growth. Doing this requires collecting large amounts of data to track resource utilization in real time for monitoring and also for future capacity planning based on user growth.

Infrastructure monitoring setup

Now let’s look at how to set up and configure monitoring for this application. The first step will be configuring Telegraf to collect the data we want from each part of our infrastructure. We’ll take advantage of the following Telegraf plugins:

SNMP input — The SNMP plugin is used to collect the metrics needed for network monitoring.
CPU, Disk, Nvidia SMI, DiskIO, mem, swap, system input — These plugins are used to collect server monitoring metrics.
OpenTelemetry input — OpenTelemetry is used to collect application performance metrics like logs, metrics and traces.
AWS Cloudwatch input — The AWS CloudWatch plugin makes it easy to collect all the cloud infrastructure metrics we need from AWS.
InfluxDB V2 output — The InfluxDB output plugin will send all of these collected metrics to the specified InfluxDB instance.

And here’s an example of a Telegraf configuration TOML file for this setup:

[global_tags]
  # dc = "us-east-1" # will tag all metrics with dc=us-east-1
  # rack = "1a"
  # user = "$USER"

[agent]
  interval = "10s"
  round_interval = true

  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"

  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""

  # debug = false
  # quiet = false
  # logtarget = "file"
  # logfile = ""
  # logfile_rotation_interval = "0d"
  # logfile_rotation_max_size = "0MB"
  # logfile_rotation_max_archives = 5

  hostname = ""
  omit_hostname = false

[[inputs.snmp]]
  agents = ["udp://127.0.0.1:161"].
  timeout = "15s"
   version = 2
   community = "SNMP"
  retries = 1

  [[inputs.snmp.field]]
    oid = "SNMPv2-MIB::sysUpTime.0"
    name = "uptime"
    conversion = "float(2)"

  [[inputs.snmp.field]]
    oid = "SNMPv2-MIB::sysName.0"
    name = "source"
    is_tag = true

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

[[inputs.diskio]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[nvidia-smi]]

[[inputs.opentelemetry]]
service_address = "0.0.0.0:4317"

  timeout = "5s"

 metrics_schema = "prometheus-v2"

  tls_cert = "/etc/telegraf/cert.pem"
   tls_key = "/etc/telegraf/key.pem"

[[inputs.cloudwatch_metric_streams]]

  service_address = ":443"

[[inputs.cloudwatch]]
  region = "us-east-1"

[[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:8086"]

  ## Token for authentication.
  token = ""

  ## Organization is the name of the organization you wish to write to.
  organization = ""

  ## Destination bucket to write into.
  bucket = ""

  ## The value of this tag will be used to determine the bucket.  If this
  ## tag is not set the 'bucket' option is used as the default.
  # bucket_tag = ""

  ## If true, the bucket tag will not be added to the metric.
  # exclude_bucket_tag = false

  ## Timeout for HTTP messages.
  # timeout = "5s"

  ## Additional HTTP headers
  # http_headers = {"X-Special-Header" = "Special-Value"}

  ## HTTP Proxy override, if unset values the standard proxy environment
  ## variables are consulted to determine which proxy, if any, should be used.
  # http_proxy = "http://corporate.proxy:3128"

This Telegraf configuration takes care of both the data collection and data storage steps by collecting all the designated data and sending it to InfluxDB for storage. Let’s go over some ways you can use that data.

Data visualization

One of the first steps for many companies is to create dashboards and data visualizations for their infrastructure monitoring system. These dashboards can be used for everything from high-level reports to detailed analysis by engineers monitoring things in real time. Here’s an example of a Grafana dashboard built using the data collected for this tutorial:

Alerting

While dashboards are nice, it’s impossible to manually track everything happening with your infrastructure at scale. To help with this problem, setting up automated alerting is a common feature of infrastructure monitoring systems. Here’s an example of how Grafana can be used to set value thresholds for metrics and create automated alerts if those thresholds are violated.

Grafana integrates with third-party tools like PagerDuty and Slack so engineers can be notified if something goes wrong. In some cases, alerting like this could be used to completely automate certain actions, like automatically scaling cloud capacity if hardware utilization hits a certain level.

Predictive analytics and forecasting

Predictive analytics and forecasting are probably the ideal end goal for many engineering teams. While alerting is a reactive approach that only works after something has gone wrong, predictive analytics and forecasting allow you to take action before the issue occurs. Creating accurate forecasts is obviously easier said than done, but it has huge benefits when done right.

Next steps

Hopefully this article helped you to better understand infrastructure monitoring and some of the tools that are available for building your own system. If you want to play around with some real data you can check out the following resources:

A Long Time Ago, on a Server Far, Far Away…

Jason Myers (InfluxData) — Mon, 09 Oct 2023 07:35:00 +0000

This article was originally published on The New Stack and is reposted here with permission.

Here is a brief case study that explores the logistics and motivations that would lead a successful company to spend time and resources completely rewriting the core of their flagship product in Rust.

Calling a programming language Rust almost seems like a misnomer. Rust is the brittle byproduct of corrosion — not something that would typically inspire confidence. But fortunately, software developers have very different concerns from metallurgists. In the digital realm, Rust is a game changer.

The following is a brief case study that explores the logistics and motivations that would lead a successful company to spend time and resources completely rewriting the core of their flagship product in Rust. InfluxData makes InfluxDB, the leading time series database in the world. When it comes to time series data use cases, the 1.x and 2.x iterations of InfluxDB are great for metrics. They’re able to handle analytics use cases to a certain extent, but there was always a danger of high-cardinality data impacting database performance.

The vision for InfluxDB is not to simply master metrics, but to provide solutions for all time series use cases. To achieve this, developers needed to solve the cardinality problem. Doing so would throw open the floodgates for time series data and InfluxDB.

As developers sought solutions for the cardinality problem, it became clear that to achieve their desired end they needed to rewrite significant portions of InfluxDB’s core. They needed to build a columnar database and, as a company with roots in, and a commitment to open source, turned to Apache Arrow for the columnar framework of the new database version. Versions 1.x and 2.x were written in Go, but for this new stack InfluxData founder and CTO Paul Dix saw an opportunity to try something different. Enter Rust.

Why Rust?

Rust has many attractive features for developers. The real-time nature of time series data brings with it significant performance demands. Rust has the inherent performance capabilities to support the characteristics of time series data and use cases. For example, Rust relies on fearless concurrency. This is an approach to systems programming that enforces discipline around different programming paradigms that helps developers mitigate or eliminate subtle bugs in their code. Another benefit of this fearless concurrency approach is that it makes applications easy to refactor without introducing new bugs. The borrow checker is another critical aspect of Rust. It helps users manage memory and initializes all variables before using them. This prevents users from unintentionally using the same value more than once.

Some additional perks of using Rust include the fact that its libraries can export a foreign function interface (FFI) compatible with many different programming languages. This provides extensibility and interoperability that makes Rust a major potential value-add to a wide range of applications. Rust uses the Crates.io packaging system, which gives developers everything they need right out of the box. In Rust, errors are first-class citizens and developers don’t have to deal with a garbage collector.

Rust also gives developers more control over runtimes than many other languages. Its async/await tool is much more advanced than order languages like JavaScript. In JavaScript, for example, users can’t control the order of asynchronous functions when they execute in Node.js. Async runtimes are runtimes optimized to execute async functions in specific environments. In Rust, however, developers have granular control over the execution order of asynchronous functions using async runtimes.

This just scratches the surface for the advantages of Rust. However, memory management and runtime control are two contributing factors that led to InfluxData’s decision to build its new database engine in Rust.

Rust challenges

While Rust presents a lot of advantages, it has its share of challenges as well. The most significant is that it has a high learning curve. It is a uniquely designed programming language, complete with its own design patterns. In some cases, these unique qualities are driven by the very capabilities that make Rust appealing, like the borrow checker. Developers with a background limited to dynamic programming languages, such as Python, Ruby or JavaScript tend to have a harder time learning Rust than developers with a background in static programming languages, like C++ or Swift.

Another sticking point that developers must adapt to is Rust’s lengthy compile time. This puts pressure on developers to write code that optimizes compile time. But the uphill climb might just be worth it because Dix believes that developers and companies will write more and more high-performance server software in Rust moving forward.

Supporting a shift to Rust

Hiring seasoned Rust developers may not always be an option as demand for them continues to increase. So, it’s important for individuals and companies alike to tap into available resources that will help mitigate the language’s steep learning curve. Rust is an open source language with a growing community supporting it, so leaning on the community is a great starting point for motivated developers.

Rust results

InfluxData set out to expand the analytical capability of its leading time series database and used Rust to accomplish that task. InfluxDB 3.0 is the result of years of research and development. It takes several key database concepts and applies them to the time series use case. Columnar databases aren’t new. Neither is the idea of separating storage and compute. But combining these concepts for the time series use case results in a database that can drive both monitoring and real-time analytics projects at scale.

InfluxDB 3.0 can handle data with unlimited cardinality, can scale compute and storage separately and supports native SQL queries. Its performance gains compared to previous versions of InfluxDB OSS are seismic. With a “hot” storage tier for leading-edge data, users can perform real-time analytics. And the combination of a columnar database and the use of Apache Parquet as its persistence format ushers in drastic data compression gains. Using low-cost cloud object storage for “cold” data can save users up to 90% on storage costs, all while enabling them to keep more, high-granularity data for longer periods.

Rust was a key difference maker in the creation of InfluxDB 3.0. While the decision to rewrite the database core was a major one, the end results speak for themselves. Thanks to Rust, InfluxDB is poised to remain atop the time series category for the foreseeable future.

Try InfluxDB for yourself and see what a difference Rust makes.

Time Series Is out of This World: Data in the Space Sector

Jason Myers (InfluxData) — Mon, 18 Sep 2023 07:35:00 +0000

This article was originally published on The New Stack and is reposted here with permission.

While time series data is critical for space industries, managing that data is not always straightforward.

While humans have yet to develop light-speed travel, teleportation or lots of the other cool things we see in movies or read in books, that doesn’t mean we aren’t making progress. Advances in technology are starting, ever so slowly, to blur the lines between science fiction and reality when it comes to outer space.

There are many different types of technology that make space industries “go,” but virtually all of them rely on time series data in some capacity. This article highlights several industries within the space sector that use time series data. Then, we’ll look at some of the sources that produce the data for these organizations. Finally, we’ll consider some of the challenges that exist when focusing business interests in outer space.

Uses for time series data in outer space

Satellite communications

We’ve long relied on satellites to relay communications around the world. Whether we’re talking about a phone call or a live broadcast on television, everyone has benefitted from satellite-based communications at some point.

Thales Alenia Space designs and delivers satellite systems to power telecommunications, navigation, Earth observation, environmental management, exploration, science and orbital infrastructures.

Just like you’ve probably been to places with spotty reception for your mobile phone, communications companies monitor signal quality from their satellites. Using time series data, these organizations can track changes in signal strength and quality over time and use that data to optimize performance.

Satellite operators also need to constantly be aware of the physical locations of their devices. Combining time series data with radar, signal Doppler and laser reflector data helps pinpoint satellite location and movement over time.

Furthermore, time series data is critical for predicting signal interference and the effects of space weather. Having the ability to anticipate space phenomena like solar flares or meteor showers can have a significant impact on satellite performance.

Earth observation and remote sensing

It’s not enough to simply watch the watchmen, so to speak. While it’s important to ensure that space equipment like satellites is in good working order, the type of work these devices do also makes heavy use of time series data. Loft Orbital provides space infrastructure as a service and collects telemetry data on all its equipment in orbit to ensure everything functions properly.

Remote sensing devices can monitor environmental changes over time. Whether that’s sea levels, the area of polar ice caps or changes in the sizes of the planet’s forests, there are almost endless environmental elements to track.

A common use case for time series data is studying climate patterns and weather forecasting. Having accurate and timely climate data can affect human safety. That’s why this data is also useful for assessing natural disasters and formulating emergency response and restoration efforts when crises occur.

Thanks to sci-fi, exploration is the most common thing we think about when it comes to outer space. Mission planning and execution rely heavily on accurate data and monitoring systems.

First, there’s the actual spacecraft needed to get into orbit. Time series data is critical for trajectory planning, setting thresholds and spacecraft telemetry for launches. Of course, the need for data doesn’t end after a successful launch. Data streams enable monitoring spacecraft health and many other performance metrics for space missions.

The recent deployment of the Webb space telescope highlights another use of time series data in space, which is analyzing and predicting celestial events. Not only are scientists able to see things in greater detail than ever before, but they are also able to track these events in greater detail.

While these examples aren’t exhaustive, they paint a consistent picture of how critical data is for monitoring and understanding aspects of the space sector.

Time series data sources in space industries

We’ve touched a little bit on data sources, but it’s worth going a bit deeper to highlight the variety of tools and technology used in the space industries.

Satellite sensors and instruments are some of the most common data sources we see. Public and private organizations alike use a wide range of devices to track events in space and on Earth. The data received from imaging sensors, like optical or infrared sensors, helps us visualize and track phenomena. Spectroscopy sensors help with many aspects of space exploration, like measuring radiation from the sun to understand solar events. These sensors can also track the movements of distant objects, providing a more thorough glimpse of the far reaches of the universe.

Radar and Lidar sensors measure distances using radio waves and lasers, respectively. Astronauts used Lidar to map the surface of the moon. These sensors have wide applications for outward-facing initiatives, as well as inward, Earth-facing goals. For example, weather monitoring systems rely on these and other sensors to monitor space environments, to predict and mitigate the effects of space weather on spacecraft and to monitor and forecast radiation levels in space.

Plenty of ground-based data sources help spur space exploration as well. Telescopes and observatories collect data on and monitor objects and events in space. For example, the Vera C. Rubin Observatory uses time series data to keep its equipment in good running order. These Earth-based devices and systems augment devices in orbit, or even those further afield, to expand the area of the cosmos that humans can monitor. They also help track space debris and identify potential collisions.

Challenges in handling time series data in space industries

While time series data is critical for space industries, managing that data is not always straightforward. This is especially true for devices in or beyond Earth’s orbit. Doing routine maintenance on satellites isn’t possible because the cost of sending a crew up for repairs would break almost any company’s budget. Therefore, these data-producing devices need built-in redundancies and automated processes for managing data.

One challenge with time series data is that sources generate a lot of it. High-volume data is fine when you have cloud infrastructure, but if you need to collect data on a device and then transmit it, you need to make sure you have enough resources to handle that data.

It’s not like you can simply swap out a larger hard drive on a satellite. So organizations need to make sure their devices have sufficient storage and compute power. The latter is key because, while you may want to transmit data from orbital sources to storage on Earth, you likely want to process that data to clean and optimize it before transmitting. This cuts down on total data throughput.

Of course, device malfunctions or space weather events may affect a device’s ability to transmit data. In these events, organizations need to be sure that they have tools on the device and in their central data hub that can handle transmission delays and late-arriving data. A time series database like InfluxDB provides solutions for many of these challenges.

It is available for on-device deployment and can collect and process data at the point of origin. A feature like edge data replication creates a durable queue so that connectivity issues don’t cause you to lose data. Instead, that data automatically gets pushed to a cloud instance of the database once connectivity returns. InfluxDB upserts late-arriving data to ensure you have as complete of a dataset as possible.

Businesses are just starting to scratch the surface when it comes to leveraging space. As more companies explore the possibilities in space, InfluxDB will be there to provide the tools and capabilities that enable observation outward toward the cosmos and inward to the Earth.

InfluxData Blog - Use Cases

Community Spotlight: A Native iOS App for Your InfluxDB Data

What is Influx Dashboard?

Step 1: Download the App

Step 2: Connect to Your InfluxDB Server

Step 3: Build Your Dashboard

Step 4: Query Your Data

Step 5: Explore Your Charts

Step 6: Manage Your Servers

Why this matters

Get started

IoT Made Easy with Node-RED and InfluxDB

Node-RED and InfluxDB for IoT webinar

What is Node-RED?

Node-RED core concepts

Nodes

Flows

Messages

Context

Node-RED Flow Tutorial

Node-RED MQTT flow

Node-RED project ideas

Data visualization and analysis

Smart home

Node-RED as an event processing framework

Additional resources

Introducing the Time Series Buying Guide for IIoT

The value of leveraging IIoT data

The challenges of managing IIoT data

Why high-fidelity data matters

Safeguard and maximize value from your data

Maximizing IIoT Impact with Open Data, AI, and Advanced Analytics: A Comprehensive Guide

Transforming industry with the power of IIoT

Benefits of open data ecosystems in IIoT

Key tools: AI and time series databases for real-time insights

Check out the guide for smarter industrial automation

Case Study: Modernizing SPEN's Tech Stack with Capula and InfluxDB

Background

Goals

Role of InfluxDB

Benefits

Wrapping up

How WOW! Modernized Legacy Infrastructure Monitoring with InfluxDB and Kafka

WOW!’s challenge

The solution

The architecture

Getting Started with Infrastructure Monitoring

Components of infrastructure monitoring

Tools for infrastructure monitoring

Data collection tools

Telegraf

OpenTelemetry

Data storage tools

InfluxDB

Prometheus

Graphite

Data analysis tools

Grafana

Apache Superset

Jaeger

Infrastructure monitoring tutorial

Architecture overview

Infrastructure monitoring setup

Data visualization

Alerting

Predictive analytics and forecasting

Next steps

A Long Time Ago, on a Server Far, Far Away…

Why Rust?

Rust challenges

Supporting a shift to Rust

Rust results

Time Series Is out of This World: Data in the Space Sector

Uses for time series data in outer space

Satellite communications

Earth observation and remote sensing

Space exploration and navigation

Time series data sources in space industries

Challenges in handling time series data in space industries