InfluxData Blog - Brian Gilmore

Managing Time Series Data in Industrial IoT

Brian Gilmore (InfluxData) — Wed, 13 Apr 2022 04:00:08 -0700

Thsi article was originally published in The New Stack. Posted here with permission.

The industrial revolution was a watershed period in human history. The shift from piecemeal, cottage-industry work to mechanized manufacturing transformed the way humans work. Since the 18th century, successive waves of innovation, such as the assembly line and the computer, continued to alter and change the nature of manufacturing. Today, we find ourselves in the midst of another industrial transformation. Typically referred to as Industry 4.0, this latest wave of innovation involves feeding data – either raw or as trained machine learning models – to autonomous systems that enhance manufacturing processes.

Manufacturers seek to generate consistent and predictable output. To do that, they have embraced physical instrumentation, which involves putting sensors on equipment to measure different aspects of a process. These sensors are the basis for the industrial Internet of Things (IIoT) and record critical data about how industrial machinery functions.

Critical context: time

Industrial operators need context for that data to begin to make sense of it. No matter what type of reading a sensor collects, it always includes a timestamp. This time-series data, therefore, provides a shared context for these readings and becomes the critical fulcrum for processing and understanding Industry 4.0 IoT data.

Fortunately, the basic principles underpinning Industry 4.0 mesh with the characteristics of time-series data. Industry 4.0 strives for:

Interconnection – The ability to have devices, sensors and people connect and communicate with each other.
Information transparency – Interconnection allows for the collection of large amounts of data from all points of the manufacturing process. Making this data available to industrial operators provides them with an informed understanding that helps identify areas for innovation and improvement.
Technical assistance – The ability to aggregate and visualize the collected data using a centralized dashboard allows industrial operators to make informed decisions and solve urgent issues on the fly. Furthermore, centralized data views help industrial operators avoid performing a range of tasks that are unpleasant or unsafe.
Decentralized decisions – The ability for systems to perform their tasks autonomously based on data collected. These systems only require human input for exceptions.

Compare these concepts and goals with some of the IIoT use cases for time-series data, and you can start to see how time-series data touches almost every aspect of industrial operations, both physical and virtual.

IIoT use case
Metrics	Temperature, pressure, flow, valve state, etc.
Resolution	(Sub) seconds
Retention	5 to 10 years (or longer), no downsampling
Main goals	Quality guarantee, Overall Equipment Effectiveness (OEE), predictive maintenance

Incorporating time

Industrial operators want greater observability into their machinery and processes, and time-series data provides the raw data for that. Transforming that raw data into actionable insights is one of the key objectives for time-series data in Industry 4.0. Having the right tools in place for processing, transforming and analyzing that data can make or break Industry 4.0 initiatives.

The challenge here is that many factories and manufacturers use legacy data historians, the time-series databases common in Industry 3.0. There are several reasons why these solutions are not ideal for an Industry 4.0 system.

Cost – These solutions are expensive to set up and maintain, plus they charge annual license and support fees. Most installations of legacy data historians require custom development work to fit the needs of a specific business or process and may require external consulting resources. The proprietary nature of these systems means this work is time consuming and expensive.
Vendor lock-in – These solutions are often Windows-based and do not offer a simple, open API to interface with other software. Therefore, you need to buy all integrations and components from a single vendor, locking you into a proprietary solution.
Scalability – Scalability issues can stem from both commercial and technical problems. On the technical side, these legacy data historians were built with a limited dataset in mind. This creates problems when introducing advanced capabilities like artificial intelligence or machine learning (AI/ML). These capabilities require a lot more data to train the models, which legacy systems cannot handle.
Poor developer experience – Most legacy solutions have a traditional closed design with limited API support. As a result, it takes a lot of time and money to implement or integrate these systems. These closed-design solutions provide few built-in tools, no developer community and do not support a modular development approach, thereby limiting developers' ability to pick and choose the tools that best fit the needs of their organization.
Siloed data – SCADA makers may provide a data historian for their devices, but most industrial organizations that use a traditional manufacturing execution system (MES) consolidate all their data to a single on-premises data historian. However, the lack of microservices architecture and open APIs, and an extensive use of firewalls and subnets, typically separate the data at the site level.

Without the ability to integrate with modern IT, cloud, or open source software (OSS) solutions, legacy data historians do not provide the flexibility and connectivity necessary to evolve industrial operations. This significantly reduces the efficacy of the operational technology and IT systems involved, and the data they contain, in an Industry 4.0 context because the lack of interoperability between the data historian and other systems inhibits innovation and limits observability.

Replacing legacy data historians

So, if legacy data historians aren’t the answer, what should companies use instead?

It may be tempting for manufacturers to fall back on familiar technology, like relational databases, to replace their legacy data historians. However, relational databases can’t scale for the high-volume data and lack of fixed schema that characterize time-series data.

A more suitable replacement for legacy data historians is an open source time-series platform. InfluxDB, for example, is purpose-built to handle the volume and velocity of time-series data. It uses APIs so it’s able to integrate with virtually any other connected device. InfluxDB is a schemaless platform, so it automatically adjusts to changes in the shape of incoming IIoT data.

Another open source tool that complements InfluxDB is Telegraf, a plugin-based collection agent. Telegraf is written in Go, compiles into a single binary with no external dependencies and requires a minimal memory footprint. With hundreds of plugins, many of which cater to the most popular IIoT technologies and protocols, such as OPC-UA, MQTT, Modbus, AMQP and Kafka, Telegraf connects directly to, or scrapes data from, virtually any database, application, system or sensor.

Managing and leveraging time series data

This broad connectivity also enables manufacturers to monitor and manage distributed systems and networks and remote devices in the field more easily. For example, if a manufacturer has three different facilities spread across the country, Telegraf and InfluxDB allow them to collect data from every sensor on every machine in each facility.

The data generated at each facility can be aggregated and stored on site. Those aggregations can also be sent to a central storage instance that collects data from all three sites and be rolled up to generate company-wide insights.

These same principles apply to any connected devices on the edge, whether that includes rural solar panels or ocean buoys with GSM (Global System for Mobile) connectivity. No matter how your company defines “the edge”, Telegraf and InfluxDB can handle data collection from the devices there.

InfluxDB has a mature ecosystem for IIoT systems. Some of the leading industrial systems, including PTC Kepware, PTC ThingWorx, Siemens WinCC OA and Bosch ctrlX use InfluxDB as their time-series platform. Preconfigured integrations already exist with these systems, so companies, like those making the transition to Industry 4.0, can quickly and easily upgrade their time-series database to InfluxDB.

Getting data into InfluxDB is one part of the equation to managing and leveraging time-series data. Analyzing and acting on that data are equally important parts. The Flux query language works with all components of the InfluxDB platform (i.e., InfluxDB, Telegraf). Flux allows you to slice and dice time-series data to produce actionable insights, set thresholds and alerts, and output data to any desired end points. By using Flux and InfluxDB, you can create visualizations for your data that assist with identifying usage patterns and areas for optimization or predicting maintenance.

The bottom line

Ultimately, incorporating an open source time-series database into your Industry 4.0 technology stack helps to bridge the gap between operational technology and information technology, delivering greater observability for both physical and virtual plant and providing critical data about all aspects of the manufacturing process. A solution like InfluxDB empowers industrial operators to harness data, providing critical information to workers on the factory floor and adding measurable value throughout the manufacturing process.

Where Will Process Historians Fit in the Modern Industrial Technology Stack?

Brian Gilmore (InfluxData) — Mon, 28 Mar 2022 04:00:14 -0700

When Rolls Royce Power Systems recently needed to improve its operational efficiency within its manufacturing plants, it didn’t expand its use of a legacy process historian or purchase historian connectors to export data to their business intelligence systems. Instead, it decided to go with a modern time series database, InfluxDB.

Graphite Energy, another customer we featured in our recent IIoT announcement, also chose InfluxDB over the legacy process historian vendors. Why? It was simply a better value to monitor and optimize its zero-emissions energy platform.

These are just two recent examples of this trend. Going a bit further back, Ausgrid, the organization that owns, maintains, and operates the electrical networks in Sydney, the Central Coast, and Hunter regions of New South Wales, Australia, selected InfluxDB over the OSIsoft Pi System for storing and analyzing information from tens of thousands of distributed smart cities devices. Its reasoning? Faster time to install, faster time to implement, and faster time to value. We like to call the combination of these “Faster Time to Awesome,” and it fits here. Ausgrid’s performance testing suggested nearly 4000x better query performance than the Pi System for complex, multi-sensor use cases.

So what has changed? Legacy process historians used to be a “must-have” for any critical industrial operation. Open-source software and cloud services were unwelcome just ten years ago, but it now looks as if these technologies may disrupt the old-guard vendors, even as they scramble to keep up. Here are just a few of the reasons we are seeing this shift:

Silos are collapsing

It was an easy if misleading story to tell – the big industrial software companies made the machines. Therefore, their software was the only responsible and safe way to interact with them. Historians came bundled with the assets and automation applications, usually selected by the design-build engineering firms long before the end-users could give input. Like the grade of concrete or the piping schedule, certain technologies became part of the boilerplate, with no consideration of the future needs of the customer or operator. This selection process was truly a one-size-fits-all approach, and organizations found themselves locked into the standards and vendor contracts.

Over the last decade, these standard technology stacks have begun to show cracks. First were the cracks created by interoperability standards, like OPC-UA. Many third-party applications have used these standards to connect newer, modern, and cost-effective solutions. Kepware Technologies KepServerEX is a great example of this, now owned by PTC (one of our premier IIoT partners). Kepware built inexpensive software that could run on commodity hardware and bridge the gap between the legacy industrial protocols and OPC-UA. Over time, this evolved into other interfaces like HTTP and MQTT. Now, any organization can securely capture data from their industrial assets in real-time and send it to a platform like InfluxDB. Customers remark again and again that it “just works.” This new stack does not introduce artificial or unnecessary obstacles to pilots or production, a total paradigm shift for industry. Self-service data pipelines are now a reality.

The disruption started by Kepware directly led to similar approaches from new platforms like the HighByte Intelligence Hub, the Akenza IoT Platform, Losant, and Cogent Data Hubs. In addition, the more forward-looking of the established industrial software vendors like Siemens, Bosch, Rockwell, Schneider and PTC also collaborate with InfluxData to bring the InfluxDB time series database to their platforms and customers. These efforts ultimately provide a more powerful and cost-effective solution than existing options.

The workforce is evolving

A 2019 research study by the Manufacturing Institute, partnered with the Alfred P. Sloan Foundation, identified several concerning findings related to the aging workforce in US manufacturing.

With the median age of manufacturing workers approaching 45, it is clear that there are a high number of near-retirement operators in the workforce. However, those older workers are quickly joined by younger workers who are truly “digital natives.” This new generation of operators, analysts, and engineers is tech-savvy and obstacle-adverse. To them, an instruction manual is a troubleshooting tool, and technologies that are intuitive, powerful, and, most importantly, stimulating are the ones that grab their attention. With few exceptions, the legacy process historians appear more like a rotary phone or typewriter than a piece of modern technology to them. As the Manufacturing Institute study shows, a combination of apprenticeship and adaptation is the best way to navigate the generational change. This approach presents the perfect opportunity for any industrial organization to prioritize technology modernization projects. It’s a special surprise when the newer technologies turn out to be more cost-effective as well.

The new world is one of microservices, edge and cloud hybrid networks, and access-anywhere information users can consume through new interactions and user interfaces. This is the technology landscape into which millennial makers, developers and engineers were born. It is where they learned the basics, innovated, and it’s where and how they are building their own successful companies. As a result, we see fundamental changes and accommodations coming within industrial sectors to put this new generation in their element, and 40-year-old software isn’t the answer.

A new mindset is emerging

The emergence of new, open-source technology stacks designed specifically for industrial operations is a telltale sign of an important change in mindset. Consider, for example, the Libre Stack – a fast, flexible, infinitely scalable data hub for manufacturing. Libre is “revolutionizing the industry with the world’s only completely open Manufacturing Operating System.”

Libre believes that their platform disrupts the legacy hybrid MES/Historian applications by taking novel approaches to data governance, platform extensibility, open-source and open standards. It provides a truly scalable, microservices-based platform for data collection, analytics, application building, and API-first access, all incredibly novel approaches that fly in the face of previous solutions. With an active Discord community, tight interlock with the Industry 4.0 online community, Docker and Kubernetes support, and regular commits on GitHub, this type of vendor-agnostic, capabilities-focused community presents novel and serious competition to the technology and business models of the legacy process historian.

The team behind Libre chose InfluxDB as the historian component for the platform. According to Geoff Nunan, Chief Technology Officer at Libre Technologies, “The one word that defines success in Industrial IoT for many companies is Agility. Being able to look at a new problem, explore a new angle or prove a new concept fast. Competitive advantage is often found in shortening the time-to-market for new products. InfluxData lives their “Time to Awesome” vision, and that’s why we chose InfluxDB as the time series database behind the Libre Manufacturing platform.”

The times are truly changing.

A Platform Gaining Momentum: Announcing New InfluxDB Features for Industrial IoT

Brian Gilmore (InfluxData) — Tue, 15 Feb 2022 04:10:07 -0700

Data – specifically time series data – continues to be the key ingredient for successful digital transformation. No matter the industry, time series data helps companies understand the activities and output of people, processes and technologies impacting their business. The effective management and use of time series data has emerged as the best path towards this goal. Nowhere is this more apparent than when IoT or industrial devices and applications are part of the technology landscape.

In IIoT, time series data can be used to detect operational anomalies, like storage tank leaks and changes in wind direction. With InfluxDB, this data can trigger alerts to inform operators and engineers when anomalies or problematic conditions occur. Other IIoT use cases which require time series data include predictive maintenance, optimized traffic routing, and enhanced water conservation.

We built InfluxDB to handle the variety, volume and velocity of IoT data, making it an ideal foundation for IoT and IIoT use cases. As our business grows, we find our platform at the center of the “IoT Insights” value proposition, with almost a half-million installs of the InfluxDB platform to date. We’re confident that the substance of our solutions and the community that supports them foster this level of adoption in the IoT and IIoT spaces. Check out our announcement about how we’re making an impact in IoT and the new features we’re rolling out to support developers, our community members, and our customers and partners in that field. Here are a few highlights to get you started:

InfluxDB includes powerful dashboarding tools for visualizing and analyzing sensor data.

InfluxDB enhancements to build IoT applications

Today we’re announcing new capabilities to store IoT data at the edge, to process and analyze that data locally, and to push raw, aggregated, and otherwise actionable data to a network of InfluxDB hubs in the datacenter or cloud. Together, these features expedite the development of IoT and IIoT solutions and enable accelerated insights for engineers working with industrial data at the edge. These updates include:

New and expanded support for edge analytics

Developers can now perform analytics at the edge, which shifts processing loads to the edge and increases the efficiency of overall data management and processing for large, distributed architectures. Developers can now also replicate data generated from an instance of InfluxDB at the edge to InfluxDB Cloud. This gives developers more control over how to process and handle high velocity and high-resolution data. The replication update makes it easier to collect and view data in a central location, and to perform additional analytic processing on both finite and unified data sets. You can learn more about working with InfluxDB to collect and analyze data from edge devices in this video.

Expanded flexibility for resource-constrained edge devices

We’ve added a new InfluxDB feature that allows developers to distribute processed data with custom payloads to thousands of devices all at once from a single Flux script. This is part of our broader focus on reducing the memory footprint of our edge-capable software components, while also making them more modular. We want developers to feel confident and comfortable using our databases, Telegraf agents, and client libraries, and SDKs in situations with limited compute, storage, and network resources.

Contextualize IoT data across multiple sources

We know that MQTT is a key protocol for IoT and IIoT use cases, so we improved our support for data residing in MQTT brokers. Customers can use Telegraf, our collection and processing agent, to connect to these key data sources. Keep an eye out for future announcements related to cloud-native MQTT collection, a hosted service connecting your InfluxDB Cloud directly to your cloud or on-premises MQTT brokers. Join our community Slack channel to be first to know when we’re looking for beta testers and early adopters!

New MQTT conveniences include parsing of topic names and message payloads, as well as new actions for sending information and setpoints to thousands of devices via an MQTT broker.

Check out today’s announcement for more information on the new features we’ve built and how they’re helping our IoT customers be successful.

A growing ecosystem of IIoT partners

We’ve expanded our partner ecosystem with a host of new technology partners and integrations, connecting the leading Industrial and IoT platforms and middleware software with InfluxDB. PTC ThingWorx, Inductive Automation’s Ignition, Bosch ctrlX, PTC Kepware, HighByte’s Intelligence Hub, Akenza, Cogent Data Hub and many more now support direct connection to InfluxDB’s hybrid edge-cloud APIs for time series data storage. Several of these integrations also open the opportunity for their customers to analyze and visualize the time series data stored within InfluxDB as part of their own applications.

Check out our tech partners page for additional information on our partner ecosystem, and reach out if you would like to join.

Time series in action

Along with our success in open source and edge adoption, we’ve built and extended commercial relationships with several organizations leading the Industrial, Enterprise, and Consumer categories of the Internet of Things. A few examples:

- Tesla uses InfluxDB as a key component of its internal and customer-facing analytics time series data pipeline for its consumer and commercial solar power solutions.
- Rolls-Royce Power Systems uses InfluxDB to improve operational efficiency at its industrial engine manufacturing facility. They collect sensor data from the engines of ships, trains, planes, and other industrial equipment to monitor performance in real time, identify trends, and to predict maintenance needs.
- Graphite Energy helps its clients collect industrial sensor data about energy usage, fuel consumption, temperature, solar panels, wind farms, process steam and air dryers. Graphite Energy chose InfluxDB over a legacy data historian to collect IIoT data used to monitor its zero-emission energy platform. You can read more about our work with Graphite Energy in today's blog.

New InfluxDB capabilities join existing geospatial and time series features to expand relevance for IoT solution and application builders.

We’re seeing new adoption in Energy, Transportation, Manufacturing, Consumer Goods and beyond. All of this is a good sign for the future, and we’re honored to be a part of this digital revolution. If you’d like to join us, sign up for an InfluxDB Cloud trial or download one of our open source distributions.

Industry 4.0 Defined and Explained

Brian Gilmore (InfluxData) — Tue, 21 Sep 2021 04:00:31 -0700

With Industry 4.0 fundamentally transforming manufacturing systems and processes through IIoT technologies, manufacturers large and small are seeking the most efficient ways to reap its benefits. Potential gains include optimizing operations, generating data-driven insight, creating new revenue streams, and accelerating innovation. To paint the big picture, let’s start with a definition of Industry 4.0, followed by an explanation of what adopting it involves.

What Industry 4.0 means

Industry 4.0 – the Fourth Industrial Revolution also known as the Industrial Internet of Things (IIoT) – is the automation of traditional manufacturing and industrial practices, using cloud-native technology and data analytics. In Industry 4.0, industrial machine-to-machine communication and IoT are integrated for increased automation, communication and autonomous monitoring.

Industry 4.0, which focuses on cyber physical systems to reinvent the manufacturing process around the power of data, was preceded by the:

1st industrial revolution (mechanization, water power, steam power)
2nd (mass production, assembly line, electricity)
3rd (introduction of computers and automation to enhance existing processes)

Although coined as a phrase back in 2012, the term “Industry 4.0” has gained popularity in recent years with its promise to revolutionize manufacturing through the power of data.

Overcoming the data challenges of Industry 4.0

With data at the center of Industry 4.0 environments, the challenge becomes designing a data architecture that meets industrial environments’ demanding performance, scalability, and availability requirements. As such, a common challenge is adopting new data flow systems and connecting them to existing legacy solutions. This is where IIoT comes into play, offering heavy asset industries reduced cost, easy installation, improved data accuracy and remote monitoring. As IIoT is a newer technology than industrial automation mainstays such as Supervisory Control and Data Acquisition (SCADA) and Programmable Logic Controller (PLC) systems, its capabilities adapt well to current industry demands.

IIoT adoption requirements

Industry 4.0 marks the convergence of operational technology (OT) and information technology (IT) as well as the real-time interdependence between the process and analytics. Achieving this convergence necessitates reinventing the manufacturing process with an architecture that can ingest the huge volumes of real-time data generated by IoT sensors and other devices, and enable nano-second control of the entire environment. Transitioning to Industry 4.0 requires:

Real-time data ingestion and querying capabilities — The magnitude of data that IIoT applications process and the real-time processing they require impose distinct demands on the database used to ingest and process their data. This is why a data historian (a time series database for industrial IoT) is a centerpiece of Industry 4.0. The data historian needs to allow for fast ingestion and query of time series data in near real-time; compression to minimize storage; and storage at the required precision for maintaining production line efficiency and minimizing downtime.
A data architecture that allows seamless integration across data sources — Integrations maintain data availability for real-time process optimization. The role of data integrations in managing industrial assets is captured well by Nortal: "The road to your digitalization journey and the Fourth Industrial Revolution is paved with data integrations...Without data silos, you have a complete, accurate, and centralized view of your assets and their current state and health. The ability to visualize your asset performance and process efficiencies anytime and anywhere makes you agile and able to react quickly."

Designing an Industry 4.0 environment

In sync with the data requirements discussed above, an Industry 4.0 environment can be based on four design principles:

Interconnection — Enabling communication between devices, sensors, and people
Information transparency — Gathering large data volumes from all points of the manufacturing process
Technical assistance — Aggregating and visualizing data in a centralized dashboard to solve problems in real time
Decentralized decisions — Enabling systems to perform tasks autonomously based on collected data and only exceptionally require human interference

From cost efficiency to data democratization, operational agility, traceability, and customer retention through better products and services, Industry 4.0 adoption is empowering manufacturers to realize their productive potential. In contrast, failing to adopt Industry 4.0 technologies puts manufacturers at risk of falling behind as their competitors race to capitalize on automation and digitization.

Why I Joined InfluxData - Brian Gilmore

Brian Gilmore (InfluxData) — Fri, 06 Aug 2021 00:17:46 -0700

I’ve worked on golf courses, in music stores, as a hospital administrator, and in Marine Husbandry and Life Support. I never finished college, have collected several comically dated tattoos, and am proud of all of it. My family and friends have oscillated between shock and curiosity with each new adventure. They whisper quietly that I’m still finding myself. I’m fine with saying it out loud.

When I found my way into Tech, I knew I had found a professional home. For the most part, Tech is a place where substance, experience, and expertise are valued over pedigree. I wish there were fewer obstacles to inclusion for others, but that’s a problem easier fixed from the inside. I need to work harder there myself.

While I’ve used technology and data to make an impact in each of my “careers,” this is my second act in software and my first at a startup. InfluxData was a no-brainer for me — I’m passionate about the democratization of data, integrating diverse human experience and tribal knowledge within analytics strategies, and I believe that we’re actively deciding our future with the technology we invent, build, and deploy today. I want to be here for that.

From my first conversation with the team, it was clear to me that InfluxData was one of the “good ones”. Valuing each other as individuals. Being great and humble. Embracing failure in the interest of progress. Building great software that people love to use. A position of leadership in a segment of technology that is critical to the availability, performance, security, and efficacy of every other innovative technology in Healthcare, Manufacturing, Energy, Public Sector, and beyond. It all just seemed to make sense and I jumped at the opportunity.

My first month has made it clear that I’ve made the right choice, and I’m looking forward to much more of what I’ve experienced so far. Together, let’s use technology to make the world a better, safer, more inclusive, and equitable place. I’m here for you — just let me know how I can help!