InfluxData Blog - Bria Jones

InfluxDB is Once Again a Leader in G2’s Fall 2022 Reports

Bria Jones (InfluxData) — Fri, 30 Sep 2022 07:00:00 +0000

G2 has released their Fall 2022 reports, and we are thrilled to share that InfluxDB – the purpose- built time series platform, has once again ranked #1 in the G2 Grid for Time Series Databases.

InfluxDB has also held its leading position in the Momentum Grid for Time Series Databases. The Momentum Grid® identifies products that are on a high growth trajectory based on user satisfaction scores, employee growth, and digital presence.

Time series intelligence

More excitingly, InfluxDB is now included among the leaders in an additional G2 Grid Report for Best Time Series Intelligence, as well as ranking #2 in the Time Series Intelligence Momentum Grid.

To qualify for inclusion in the Time Series Intelligence category, a product must:

Ingest and consume time series data from time series databases
Continuously monitor, and provide visualizations of, time series data
Pull out trends, patterns, and insights from time series data via machine learning
Allow users to explore, forecast, and predict future business outcomes based on the data

We are thrilled to not only be included, but rank among leaders in the Time Series Intelligence report.

Why is this a big deal?

For starters, G2 is a globally recognized and trusted software review website. More than 60 million people annually — including employees at all of the FORTUNE 500 — use G2 to make smarter software decisions based on authentic peer reviews.

InfluxDB earned leadership rankings in all reports by receiving positive reviews from verified users compared to similar products. For inclusion in the report, a product must have received 10 or more reviews.

In both the Time Series Databases and Time Series Intelligence reports, customers ranked InfluxDB #1 in four categories:

Ease of Setup
Market Presence
Ease of Admin
User Satisfaction

“InfluxDB is the Go-To Time Series Database” - Verified G2 Reviewer

Being an open source company, our community is incredibly important to us so we are thrilled that our users have voted us the leader in Time Series.

Our users have said some pretty great things about us in our G2 reviews, but we’d like to return the favor. Head over to our Customer Page to see some of the incredible things our customers are doing with InfluxDB.

For the latest InfluxDB reviews on G2, visit the G2 website. To learn more about InfluxDB, the most powerful time series platform, sign up for a free Cloud trial.

Webinar Highlights: Improving Clinical Data Accuracy - How to Streamline a Data Pipeline Using Node.js, AWS and InfluxDB

Bria Jones (InfluxData) — Fri, 26 Aug 2022 07:00:00 +0000

Given the global health crises the world has faced over the last few years, the need for expeditious but accurate medical trials has never been more important. The faster clinical trial data is validated, the faster medicines get approved and treatments become available. Pinnacle 21’s customers are driving forces behind creating life-saving treatments. In this webinar, Josh Gitlin, Director of DevOps, shares how Pinnacle 21 is using the purpose-built time series platform, InfluxDB, to help streamline data pipelines for faster and more accurate clinical trial data. If you missed attending the live session, we have shared the recording and the slides for everyone to review and watch at your leisure.

Webinar highlights

Pinnacle 21 overview

Pinnacle 21 by Certara is a software company specializing in life sciences solutions. Their flagship product, Pinnacle 21 Enterprise (P21E for short), is used by major life science and pharmaceutical companies to validate clinical trial data. When developing medical treatments, medicines, and devices, there are specific data standards the CDISC (Clinical Data Interchange Standards Consortium) established that must be met to be approved. Pinnacle 21 customers use P21E to ensure their data will look correct when sent for approval; Josh provided the best layman’s explanation he has heard, “It’s like spell-check for your clinical trial data.”

The need for a solution

When Josh joined Pinnacle 21, Datadog had been selected as the product of choice to monitor their servers. He felt it was lacking critical features such as the ability to label the Y axis, and there were limited visualization options. Plus, it was expensive and not well suited for Pinnacle 21’s use case. Having previous experience with Grafana and InfluxDB, Josh proposed the idea of replacing Datadog. Since Josh had been writing automation software in CINC, the open source version of Chef, it would be fairly easy to replace Datadog with another monitoring solution.

Replacement requirements:

Easy to implement
Ability to capture logs and metrics
Must be an externally-hosted tool
Collect and monitor APM metrics

They needed their data to be hosted elsewhere to be secure and protected from tampering but did not want to manage the infrastructure of another monitoring solution. This requirement would help them adhere to the audit and compliance requirements established by the CDISC.

Why Grafana, InfluxDB, and Telegraf?

Josh considered InfluxDB and Grafana as a potential replacement for their DevOps monitoring needs initially because of the cost efficiency – even when paired with another vendor for their log analytics, it was still significantly cheaper than their previous solution. With InfluxDB’s usage-based plan in place, they evaluated each metric they chose to send to Telegraf to decide what was worth paying for. Another key component for Pinnacle 21 was Telegraf itself. For Josh and his team, Telegraf was an impressive and powerful data ingestion tool with more plugins available, out of the box, than their existing agent.

Pinnacle 21’s customers are located in AWS within their own EC2 security group and each has two instances – a web server and an application server. Telegraf is on both of these instances publishing metrics to an internal InfluxDB instance as well as to a GCP-hosted InfluxDB Cloud instance. The Pinnacle team created a policy file-based workflow for each type of server that defines which pieces of automation to run and what to monitor through Telegraf.

Installation was easy for Pinnacle 21 since Telegraf has a directory that allowed them to write individual plugin configurations. This was useful when automating a Telegraf installation via Chef as they have different policies and roles based on their servers. Josh and team chose to have each plugin write a configuration in the Telegraf directory then establish a node attribute listing which plugins Telegraf should monitor. This enabled them to customize the configuration for each Telegraf input for each server individually.

Josh’s tip: Structure the node attributes as hashes and not arrays, since arrays will be deep merged and not as easy to turn off. Stacking Nodes as hashes will allow better control of what is being monitored at a very granular level.

Josh then decided on an initial base monitoring set in Chef and found that some of the inputs were valuable but not worth the expense. The solution? Telegraf has a filtering option in each output plugin called “Tag Pass”. The team configured the output plugins with specific destination tags and excluded certain tags from being published to a specific server. By putting tags on each Telegraf input, they could select which InfluxDB servers to send those metrics to. This also enabled the team to collect higher granularity metrics for a particular input and fine-tune what they were collecting, how often they were collecting it and where they were sending it to better manage cost.

KPIs and APM

The next key action item was to look at some key performance indicators for their servers. Because they had previously used Datadog, they were already writing NGINX logs in JSON format but Josh recommended parsing the NGINX logs. By parsing their logs, Pinnacle 21’s team is able to convert their logs into metrics, and store this in InfluxDB.

Pinnacle 21’s customers use IBM Aspera to upload large data sets for validation; they have configured Aspera to write logs to a particular path where Telegraf can pick up using a Tail Telegraf Plugin. This enables the team to generate graphs showing specific KPIs – the average customer data set size, the speed at which the customer data is uploading, how many uploads are in progress, etc. which were helpful for the extended Pinnacle 21 team to understand the impact of the data.

Because P21E is a Java application, they needed to use a Java agent for their APM metrics. The team chose InspectIT Ocelot to collect JVM metrics, and they publish those metrics natively using the Telegraf Listener on the machines. This enabled their engineers to write code to capture individual events like function runtimes and other metrics to monitor the performance of the application and optimize the software.

This is incredibly important for Pinnacle 21 as some of the data sets can take hours, sometimes days, to validate. Consider COVID-19 data sets, for example – the longer it takes to validate that data, the longer it takes to get new treatments approved. By giving the engineers the ability to quickly assess and optimize the performance of the application, it can ultimately speed up the validation process.

HTTP monitoring replacement

Initially the team considered Telegraf as a possible replacement for their HTTP monitoring system, and while Telegraf could do the job, they ultimately went with an AWS Lambda instance with a Node.js application. Node was the correct solution for them because of its event-driven infrastructure. Josh uses the Node.js client to publish metrics directly to InfluxDB and triggers AWS CloudWatch events to execute every minute from multiple regions. They used Grafana for alerts, to create heat maps, and to visualize data such as response times, outages, and failure codes from across the globe.

Tips and tricks

Josh provided a number of helpful tips and tricks for anyone starting their monitoring journey with InfluxDB. First, he suggested starting out by evaluating needs and determining the overall objective. Because Telegraf is so powerful, it’s easy to get lost in all of the plugins available. Josh noted that sending data to multiple InfluxDB instances is a powerful option for redundancy and a good option for back-up in case one instance has an issue. Next, he stresses the importance of using the usage dashboard as a way to measure your usage and manage cost. He suggests adding metrics slowly to evaluate their impact to your usage, especially if there is a large fleet of servers. Finally, he recommended using Flux as it is a powerful and full functional language.

Josh mentioned he would be checking out InfluxDB University to sharpen his Flux skills and if you’re interested in learning more about Flux or other InfluxDB topics, start here!

Q&A

Question: Is there an initiative at your organization to be cloud-first?

Answer: Within the Pinnacle 21 department, it was very much cloud first, and that was largely the startup mentality, I think, where the founders needed to host this software and the last thing that the CTO needed to do was manage more infrastructure just to keep the business running. I do really like cloud first, and so I’m sticking with that approach for a significant number of the things like InfluxDB that we rolled out, like our log monitoring solution. But I’m also not afraid of running things internally. Sometimes I think there’s benefits one way or the other. So it depends on team size and resources. We are hiring at the moment, so as we get more DevOps engineers, we’ll have the greater ability to support internally hosted things as well.

Question: You’ve mentioned a few things that you’re hoping to do next — what’s at the top of that punch list regarding metrics that you’re collecting using InfluxDB?

Answer: I would love to start replacing some of the InfluxQL queries we have with Flux queries and improve some of the dashboards that we have. I think we could benefit from some comparative information where we look at comparing one customer or performance week over week. I think there’s definitely some power to be had there. I like to improve some of the things in the HTTP monitoring side. It’s good right now, but I think it could be better, and less on the InfluxDB side but more on the Grafana side. I’ve been playing with Grafana OnCall, really happy with that. We’re not using it in production yet, but we are using it in development, and I would like to switch everything over to that. It allows you to acknowledge alerts right from within Slack. It allows you to configure downtime better. That’s one of the challenges we’ve had with the existing solution is if we know we’re doing maintenance on a particular instance, we get a whole bunch of alerts for that instance because it’s difficult to silence those alerts or schedule downtime. So Grafana OnCall makes that easier and still works with InfluxQL or Flux.

Question: Our community loves Grafana, but have you looked at the visualization tooling in InfluxDB?

Answer: Yes. When I’m building new queries, I will use the InfluxDB Cloud UI and I will write my queries in there, visualize them, make sure that I have them the way I want them. I use the InfluxDB Cloud UI when I’m looking at the usage dashboard. I think some of the visualizations in there are not quite as powerful yet just because Grafana has such a head start on InfluxDB Cloud. But it’s a great system. It’s already more powerful than some of the things I’ve seen with Datadog. For example, you have different kinds of visualizations than just line charts. So yes, we do have some dashboards within the InfluxDB Cloud UI itself.

To learn more about how Pinnacle 21 is using InfluxDB, click here.

Webinar Highlights: How Olympus Controls Automates Predictive Maintenance with Telit, MQTT, and InfluxDB

Bria Jones (InfluxData) — Fri, 12 Aug 2022 07:00:00 +0000

Unplanned outages are problematic for any organization, and can be especially costly for manufacturers when they involve parts that have a long lead time. In this webinar, Nick Armenta – Automation Engineer at Olympic Controls, shares how InfluxDB provides insights into when parts may fail to help predict the maintenance windows for their robots. Focused on robot joints and monitoring the “current,” which is essentially the force of the motor, Nick shares details of his experience using the purpose-built time series platform.

Webinar highlights

Olympus Controls overview

Olympus Controls is a hardware distribution company that primarily sells robotics and motion control technologies, but their ultimate goal is to be an engineering partner to their customers. Driven to find the root cause of a problem, Olympus Controls implements technologies and devices that can connect to a platform, such as InfluxDB, for remote IoT sensor monitoring.

Machine KPIs

In manufacturing, it is all about improving machinery uptime. By collecting and analyzing sensor metrics about machines and production, organizations want to enable better predictive maintenance of their costly machines. When successful, operations costs are reduced and machine uptime increases. Rates are critically important to establish the overall health of production lines and for Olympus Controls, they use InfluxDB to collect robot and machine health data. By using a time series platform, they have been able to automate the monitoring of industrial manufacturing plants and factories. The machines used in factories run for years, sometimes decades, so understanding when they will need maintenance is necessary.

Telit + MQTT + Telegraf + InfluxDB

Robotiq Palletizer is a solution Olympus Controls created to automatically stack products onto a pallet. Each robot has joints, similar to our wrists, and each of those joints has a motor. Nick is monitoring motor health using InfluxDB to help predict when maintenance may be needed. Robot data is obtained using a simple IO device with Telit DeviceWISE edge platform installed onto it. This gateway allows communication with any device regardless of their preferred protocol and offers built-in communication protocols such as Modbus, TCP and MQTT. This is helpful since Olympus Controls may not know which gateway their customers are using. From there, the MQTT broker publishes specific topics that robots in other locations can subscribe to provide what Olympus Controls calls “load balancing” of your automation. By using the MQTT Consumer Telegraf plugin, Nick is able to collect data from the broker and pull it into InfluxDB.

Since Nick wasn’t much of a “database guy,” he enjoyed the quick querying capabilities which allowed him to query and visualize his data. Utilizing custom dashboards is essential to manufacturing, as it makes identifying problems – and solving them, much quicker. To demonstrate an error in real time, Nick playfully interrupted the robot by “punching” it and the error can be seen in the dashboard in real time. This webinar focused on one specific robot but because scalability is easy with InfluxDB, Nick can include multiple robots and easily visualize their data to help get ahead of any costly maintenance issues for his customers.

Why InfluxDB?

Nick had very little database experience before InfluxDB and didn’t quite know where to start, but once he realized how easy data collection was with Telegraf and InfluxDB, it made the most sense. The quick querying with minimal effort on his part was a key factor as well. He also loves a good visualization tool and since the visualization component is already built into InfluxDB, it made it easy to start viewing his data.

(If you are also new to InfluxDB, check out InfluxDB University — there are free on-demand and instructor-led courses.)

Q&A

During the webinar, audience members had some thoughtful questions for Nick — here are a few of them.

Question: Is it necessary to use a separate MQTT broker somewhere for InfluxDB to collect data? What is the role of Telegraf in the schema?

Answer: We actually did some other implementations where we did not need MQTT, because we had a small local development environment and so we could send it to a local database. Once we started to go to more of an enterprise type of deployment, that’s where it made more sense, because now we need to go to a cloud instance, so we couldn’t keep that localized anymore. So that’s where the MQTT broker stepped in. But you’re absolutely right about just MQTT and the protocols. It’s not necessarily what we needed. It just made the most sense in this standpoint, because really in manufacturing, it’s all about simplicity. We did have a broker running locally, using Mosquitto, and a Docker container, as well. So it doesn’t have to be that specific endpoint. But MQTT was simple and safe to communicate, which is why I chose it in this instance. But by no means the only way.

Question: Did you face any pushback in your organization? How did you overcome that?

Answer: Yes, the number one complaint I get in factories is that they don’t have enough people. So generally people are stressed. They’ve got tunnel vision. So really pulling them back to that 10,000 foot view and saying, “Hey, if you look right here, this is a serious pain point. If we can address this, we can alleviate your stress.” And to eliminate those pain points, we need to pull data that allows us to show them where they could be most productive. So it’s really just pulling them back for a little bit, talking them off the ledge, and saying, “Let’s look at this area first.” And from there, we can start generating more traction and more productivity.

Question: Are you considering a machine learning approach, for example, using neural networks, classification algorithms, or something to predict specific events and trying to learn something from available data?

Answer: The number one problem I’ve seen is really just a lack of data availability. So step one, get a good pool of clean, formatted data. Once your data is clean, formatted, and visualized in an understandable way, you can actually make quick judgment calls and understand just by looking at a good, visualized interface. There’s a bunch of different data streams and it’s a very complex type of scenario. But when it comes to things like preventative maintenance or any kind of productivity metrics, those are pretty basic, discrete type signals we can usually look for.

Question: How do you manage loss of connection to the MQTT broker? Is there an MQTT client or other agent buffering data locally in the factory?

Answer: There are actually a couple of different ways to manage the loss of connection. MQTT comes with quality of service (QoS) levels to guarantee the delivery of a specific message. Level 0 is basically similar to UDP (versus TCP), where you send a message and you don’t care what happened. That is the default setting in the protocol. Level 1 sends back an acknowledgment, and Level 2 provides a 4-part handshake which is slower but guarantees message delivery.

There were many more interesting questions! If you’re interested in watching the full webinar to hear the rest of the Q&A, check it out here! Find the presentation here.

It’s BBQ Season! How Pit Bosses Use InfluxDB to Avoid the Stall

Bria Jones (InfluxData) — Thu, 28 Jul 2022 07:00:00 +0000

If you’re ever in Dallas or Kansas City (or really any foodie city) and feel like having a spirited debate, just ask who does BBQ best. Having lived in Texas for most of my life, I can confidently say Texans take their BBQ seriously and, based on our recent internal Slack poll, I’m inclined to believe the folks of Kansas City, the Carolinas and Louisiana do too. The hours of preparation and execution are not for the faint of heart, and those that take on the challenge each have a unique and specific method they use to achieve greatness.

Some call it art, some call it science. Those who include InfluxDB in their culinary toolkit will call it science and I will share with you how these pitmaster scientists perfect their BBQ with time series data.

First, the basics

In the BBQ world, “smoking” is the method used to cook various meats using a specific piece of equipment, at a low temperature, for a long period of time. You’ll hear the phrase “low and slow” and this refers to the tried and true method for achieving tender and flavorful results.

There is a lot that goes into determining a proper smoke, such as the type of smoker and wood choice. You’ll likely start an argument if you claim one is better than the other, but one element that is not debated is the importance of temperature control.

For starters, most meats need to reach an internal temperature of at least 145 degrees Fahrenheit (63 ℃) to be safe to consume. For optimal flavor and tenderness, the desired temperature is around 180 degrees (82 ℃). One of the most common frustrations of smoking meat is the dreaded “stall” or “plateau.” This occurs when the internal temperature reaches a certain point and stops rising. Caused by the evaporation of liquid on the meat’s surface – much like when you sweat while working hard, the stall can last for hours which means a much longer cook time. The remedy to getting out of the stall is simple: just wrap the meat and contain the heat, but what if you could set up alerts to notify you when you’ve hit the stall? You’ll most likely get to enjoy the fruits of your labor a lot sooner.

Did you know that Texas is the home of the world’s largest BBQ pit and it’s on wheels? The Undisputable Cuz is the world’s largest BBQ pit. This pit can grill up to 4 tons of meat in one go, and the man who owns it only uses it for charity. He once fed 55,000 people in 11 days after Hurricane Harvey.

Using InfluxDB to avoid “the stall”

Will Cooke, Director of Engineering for InfluxData’s storage team and Scott Anderson, Senior Writer and Technical Lead for InfluxData’s Docs team both utilize InfluxDB to optimize their smoke performance but each in their own way. Fun fact: they’re regulars in our internal BBQ Slack channel called #influxpits – we take our BBQ seriously at InfluxData. If you want to share your own experience monitoring your BBQ (or anything else), check out the #home-automation channel in our community Slack workspace.

Will began his journey with DIY in mind, attempting to build his own sensors at first. After a few hundred hours and some questionable decisions, he decided on off-the-shelf hardware for his home monitoring needs and a mechanism that he could “hack” to his liking. He settled on an inexpensive thermostat that uses Bluetooth Low Energy, Raspberry Pi and InfluxDB Cloud.

Not one to be confined by the given ecosystem, he used MQTT as his method to get the data from Raspberry Pi into a server that was running Telegraf and into InfluxDB Cloud. Realizing how quickly he could set up dashboards, he created his own to track the temperature inside the smoker and the internal temperature of the meat. Using InfluxDB’s visualization tool, it’s easy to see when the stall began. The dip in temperature indicates the moment Will opened the smoker to wrap the meat and almost instantly, the internal temperature of the meat began to climb back up. Thus, ending the stall.

Not so DIY inclined?

Scott wasn’t interested in building everything from scratch. He opted to purchase a cloud-connected wireless meat thermometer called FireBoard, which collects sensor data from the probes plugged into it and sends the data up to the FireBoard cloud servers. One of the cool things about having a FireBoard is that it comes equipped with out-of-the-box visualization tools and dashboards that can be used on your phone or Apple Watch. So why bother with InfluxDB if the dashboards are already made and available?

For Scott, the real value in storing FireBoard data in InfluxDB is what he could do with that data. Specifically, the ability to detect the stall as it happens and be alerted.

Scott enjoys smoking overnight while still getting a good night’s sleep, so he utilized the monitoring alerting API that’s built into InfluxDB. He assigned statuses based on calculating the difference between 5 minute averages. Anything less than 0.02 degrees every five minutes meant the meat was stalling and was given a “warn” status and that change in status would trigger an alert that would send him a notification directly to his phone, thereby waking him up so he could wrap the meat and contain the heat. Talk about commitment to his craft! If you’re interested in monitoring your FireBoard, check out this InfluxDB Template created by Scott.

Some call it art, but these two masters made it science. I just call it delicious.

If you’re interested in hearing more about Will and Scott’s experience using InfluxDB to monitor their BBQ sessions, check out their presentation and recording here to try it yourself. If you have your own unique and special way you’re using InfluxDB, let us know here!

InfluxData Blog - Bria Jones

InfluxDB is Once Again a Leader in G2’s Fall 2022 Reports

Time series intelligence

Why is this a big deal?

“InfluxDB is the Go-To Time Series Database” - Verified G2 Reviewer

Sharing the love

Webinar Highlights: Improving Clinical Data Accuracy - How to Streamline a Data Pipeline Using Node.js, AWS and InfluxDB

Webinar highlights

Pinnacle 21 overview

The need for a solution

Why Grafana, InfluxDB, and Telegraf?

KPIs and APM

HTTP monitoring replacement

Tips and tricks

Q&A

Webinar Highlights: How Olympus Controls Automates Predictive Maintenance with Telit, MQTT, and InfluxDB

Webinar highlights

Olympus Controls overview

Machine KPIs

Telit + MQTT + Telegraf + InfluxDB

Why InfluxDB?

Q&A

It’s BBQ Season! How Pit Bosses Use InfluxDB to Avoid the Stall

First, the basics

Using InfluxDB to avoid “the stall”

Not so DIY inclined?