InfluxData Blog - Chris Churilo

InfluxDB is 5x Faster vs. MongoDB for Time Series Workloads

Chris Churilo (InfluxData) — Thu, 17 Nov 2022 07:19:00 +0000

This blog post has been updated on November 17, 2022 with the latest benchmark results for InfluxDB v1.8.10 and MongoDB v5.0.6. To provide you with the latest findings, this blog is regularly updated with the latest benchmark figures.

At InfluxData, one of the common questions we regularly get asked by developers and architects alike the last few months is, “How does InfluxDB compare to MongoDB for time series workloads?” This question might be prompted for a few reasons. First, if they’re starting a brand new project and doing the due diligence of evaluating a few solutions head-to-head, it can be helpful in creating their comparison grid. Second, they might already be using MongoDB for ingesting data in an existing application, but would like to now see how they can integrate metrics collection into their system and believe there might be a better solution than MongoDB for this task.

Over the last few weeks, we set out to compare the performance and features of InfluxDB and MongoDB for common time series workloads, specifically looking at the rates of data ingestion, on-disk data compression, and query performance. InfluxDB outperformed MongoDB in all three tests with 1.9x greater write throughput, while using 7.3x less disk space, and delivering 5x higher performance when it came to query speed.

To read the complete details of the benchmarks and methodology, download the “Benchmarking InfluxDB vs. MongoDB for Time-Series Data & Metrics Management” technical paper or watch the recorded webinar.

Our overriding goal was to create a consistent, up-to-date comparison that reflects the latest developments in both InfluxDB and MongoDB with later coverage of other databases and time-series solutions. We will periodically re-run these benchmarks and update our detailed technical paper with our findings. All the code for these benchmarks is available on Github. Feel free to open up issues or pull requests on that repository or if you have any questions, comments, or suggestions.

Now, let’s take a look at the results…

Versions tested

InfluxDB v1.8.10

InfluxDB is an open source time series database written in Go. At its core is a custom-built storage engine called the Time-Structured Merge (TSM) Tree, which is optimized for time-series data. Controlled by a custom SQL-like query language named InfluxQL, InfluxDB provides out-of-the-box support for mathematical and statistical functions across time ranges and is perfect for custom monitoring and metrics collection, real-time analytics, plus IoT and sensor data workloads.

MongoDB v5.0.6

MongoDB is an open source, document-oriented database, colloquially known as a NoSQL database, written in C and C++. Though it’s not generally considered a true time series database per se, its creators often promote its use for time series workloads. It offers modeling primitives in the form of timestamps and bucketing, which give users the ability to store and query time series data.

About the benchmarks

In building a representative benchmark suite, we identified the most commonly evaluated characteristics for working with time-series data. We looked at performance across three vectors:

Data ingest performance - measured in values per second
On-disk storage requirements - measured in bytes
Mean query response time - measured in milliseconds

About the dataset

For this benchmark, we focused on a dataset that models a common DevOps monitoring and metrics use case, where a fleet of servers are periodically reporting system and application metrics at a regular time interval. We sampled 100 values across 9 subsystems (CPU, memory, disk, disk I/O, kernel, network, Redis, PostgreSQL, and Nginx) every 10 seconds. For the key comparisons, we looked at a dataset that represents 100 servers over a 6-hour period, which represents a relatively modest deployment.

Number of servers: 100
Values measured per server: 100
Measurement interval: 10s
Dataset duration(s): 24h
Total values in dataset: 86M per day

This is only a subset of the entire benchmark suite, but it’s a representative example. If you’re interested in additional detail, you can read more about the testing methodology on GitHub.

Write performance

InfluxDB outperformed MongoDB by 1.9x when it came to data ingestion.

On-disk compression

InfluxDB outperformed MongoDB by delivering 7.3x better compression.

Query performance

InfluxDB outperformed MongoDB by 5x when it came to query speed.

Summary

The benchmarking tests and resulting data demonstrated that InfluxDB outperformed MongoDB in data ingestion and on-disk storage by a significant margin. Specifically:

InfluxDB outperformed MongoDB by 1.9x when it came to data ingestion
InfluxDB outperformed MongoDB by delivering 7.3x better compression
InfluxDB outperformed MongoDB by delivering 5x better query performance

It’s also important to note that configuring MongoDB to work with time series data wasn’t trivial. It requires up-front decisions about how to structure your collections and data types, which can be very time consuming and will have long-lasting impacts on how you can interact with your data and what types of queries you can run. InfluxDB, on the other hand, is ready to use for time series workloads out-of-the-box with no additional configuration.

In conclusion, we highly encourage developers and architects to run these benchmarks themselves to independently verify the results on their hardware and data sets of choice. However, for those looking for a valid starting point on which technology will give better time series data ingestion, compression and query performance “out-of-the-box”, InfluxDB is the clear winner across many dimensions, especially when the data sets become larger and the system runs over a longer period of time.

What's Next?

Download the detailed technical paper: "Benchmarking InfluxDB vs. MongoDB for Time Series Data, Metrics & Management".
Check out the video playback of the companion webinar.
Download and get started with InfluxDB.
Join the Community!

InfluxDB Tops Cassandra in Time Series Data & Metrics Benchmark

Chris Churilo (InfluxData) — Thu, 20 Oct 2022 23:06:00 +0000

This blog post has been updated on October 20, 2022 with the latest benchmark results for InfluxDB 1.8.10 and Cassandra v4.0.5. To provide you with the latest findings, this blog is regularly updated with the latest benchmark figures.

At InfluxData, one of the common questions we regularly get asked by developers and architects alike is: “How does InfluxDB compare to Cassandra for time series workloads?” This question might be prompted for a few reasons. First, if they’re starting a brand new project and doing the due diligence of evaluating a few solutions head-to-head, it can be helpful in creating their comparison grid. Second, they might already be using Cassandra for ingesting data in an existing application, but would like to now see how they can integrate metrics collection into their system and believe there might be a better solution than Cassandra for this task.

Over the last few weeks, we set out to compare the performance and features of InfluxDB and Cassandra for common time series workloads, specifically looking at the rates of data ingestion, on-disk data compression, and query performance. InfluxDB outperformed Cassandra in all three tests with 5x greater write throughput, while using 2.4x less disk space, and delivering up to 100x faster response times for tested queries.

Before we dig into the details of the benchmarks, it’s important to point out that doing a head-to-head comparison of InfluxDB and Cassandra for time series workloads was impossible without first writing a significant amount of application code to make up for Cassandra’s lacking functionality. Effectively, you’d need to rewrite portions of InfluxDB in your own application. In order to put these benchmarks together, we had to write some basic versions of these features, but a production application would take significantly more effort.

To read the complete details of the benchmarks and methodology, download the Benchmarking InfluxDB vs. Cassandra for Time Series Data, Metrics & Management technical paper or watch the recorded webinar.

Our overriding goal was to create a consistent, up-to-date comparison that reflects the latest developments in both InfluxDB and Cassandra with later coverage of other databases and time series solutions. We will periodically re-run these benchmarks and update our detailed technical paper with our findings. All of the code for these benchmarks are available on Github. Feel free to open up issues or pull requests on that repository if you have any questions, comments, or suggestions.

Now, let’s take a look at the results…

Versions tested

InfluxDB v1.8.10

InfluxDB is an open source time series database written in Go. At its core is a custom-built storage engine called the Time-Structured Merge (TSM) Tree, which is optimized for time series data. Controlled by a custom SQL-like query language named InfluxQL, InfluxDB provides out-of-the-box support for mathematical and statistical functions across time ranges and is perfect for custom monitoring and metrics collection, real-time analytics, plus IoT and sensor data workloads.

Cassandra v4.0.5

Cassandra is a distributed, non-relational database written in Java, originally built at Facebook and open-sourced in 2008. It officially became part of the Apache Foundation in 2010. It is a general-purpose platform that provides a partitioned row store, which offers features of both key-value and column-oriented data stores.

Though it provides excellent tools for building a scalable, distributed database, Cassandra lacks most key features of a Time Series Database. Thus, a common pattern is to build application logic on top of Cassandra to handle the missing functionality. This is what we’ll be doing to help make a fair assessment of Cassandra for time series workloads.

About the benchmarks

In building a representative benchmark suite, we identified the most commonly evaluated characteristics for working with time series data. We looked at performance across three vectors:

Data ingest performance – measured in values per second
On-disk storage requirements – measured in bytes
Mean query response time – measured in milliseconds

About the dataset

For this benchmark, we focused on a dataset that models a common DevOps monitoring and metrics use case, where a fleet of servers are periodically reporting system and application metrics at a regular time interval. We sampled 100 values across 9 subsystems (CPU, memory, disk, disk I/O, kernel, network, Redis, PostgreSQL, and Nginx) every 10 seconds. For the key comparisons, we looked at a dataset that represents 100 servers over a 24-hour period, which represents a relatively modest deployment.

Number of servers: 100
Values measured per server: 100
Measurement interval: 10s
Dataset duration(s): 24h
Total values in dataset: 87,264,000 per day

This is only a subset of the entire benchmark suite, but it’s a representative example. If you’re interested in additional detail, you can read more about the testing methodology on GitHub.

Write performance

InfluxDB outperformed Cassandra by 5x when it came to data ingestion.

On-disk compression

InfluxDB outperformed Cassandra by delivering 2.4x better compression.

Query performance

InfluxDB outperformed Cassandra by delivering up to 100x better query performance.

Summary

Ultimately, many of you were probably not surprised that a purpose-built time series database designed to handle metrics would significantly outperform a search database for these types of workloads. Especially glaring is that when the workloads require temporal query flexibility, as is the common characteristic of real-time analytics and sensor data systems, a purpose-built time series database like InfluxDB makes all the difference.

It’s important to emphasize that achieving optimal query performance with Cassandra requires a significant amount of additional application-level processing. This could have a huge impact in production deployments where each query places additional load on the application servers.

In conclusion, we highly encourage developers and architects to run these benchmarks for themselves to independently verify the results on their hardware and data sets of choice. However, for those looking for a valid starting point on which technology will give better time series data ingestion, compression and query performance “out-of-the-box”, InfluxDB is the clear winner across all these dimensions, especially when the data sets become larger and the system runs over a longer period of time.

What's next?

Download the detailed technical paper: Benchmarking InfluxDB vs. Cassandra for Time Series Data, Metrics & Management.
Download and get started with InfluxDB.
Check out the video playback of the companion webinar.
Download and get started with InfluxDB.
Join the Community!

Real-Time Energy Management with InfluxDB and eSoftLink IoT Platform

Chris Churilo (InfluxData) — Thu, 21 Jul 2022 07:00:00 +0000

Smart energy IoT platforms are empowering consumers to track energy usage and even control spend based on their next bill’s forecast. Yet eSoftThings, a specialist in the Internet of Things (IoT) and artificial intelligence (AI), set out to push smart energy management even further, for both consumers and utility companies, through its IoT platform eSoftLink.

The solution that the company built deploys InfluxDB and Telegraf within its eSoftLink platform to provide real-time energy management, which allows usage visibility down to the appliance level. The solution is end-to-end — from hardware device, to backend platform, to mobile application. Here’s what drove them to build it and how it works.

Making smart energy meters even smarter

Achieving energy savings has become a major goal for both utility companies and consumers. Under Energy Efficiency Directive 2012/27/EU, the European Union (EU) set a target of 20% energy savings by 2020. Yet to save energy, it’s necessary to assess energy consumption and compare it to energy production.

Smart metering is one of the first steps to address challenges surrounding energy consumption. Smart meters allow consumers to make smarter decisions about their energy usage by providing real-time data on electricity and gas usage.

For example, France implemented the Linky program. Linky (smart meter), connects to a concentrator through Power Line Communication (PLC). This system uploads consumption information once a day (due to a small PLC tunnel and limited data rate) with 30-minute granularity. The concentrator uploads the data it collects from the meters, in GPRS, to the information system.

The utility company offers customers a web portal to track their energy consumption on a yearly, monthly, and daily basis. While this enables customers to make high-level comparisons (such as comparing summer and winter energy consumption), it doesn’t allow them to identify the devices that most contribute to overall consumption. Decreasing power consumption requires the ability to check consumption per device in real-time.

Energy consumption visibility at the appliance level

Major studies, like those conducted by National Science Research Institutes, Oxford University, and the American Council for an Energy-Efficient Economy, suggest that consumer access to real-time energy consumption feedback improves energy savings. Consumers are able to increase control over energy spend when equipped with the availability of in-depth data at the appliance level and a real-time feedback tool. “The more real-time the feedback tool, the highest the energy savings percentage at the end of the day by the consumer,” says Technical Marketing Engineer at eSoftThings Samuel Chevrier. This challenge is what drove eSoftThings to transform the smart meter into an IoT system using their eSoftLink IoT platform for their lead customer Total Direct Energie.

Transforming a smart meter in an IoT node through WiFi connectivity

Energy meters generate consumption data every second and sends it over the IoT link to the backend platform. The smart meter data also sends the data to eSoftLink every second so that consumers can easily view and monitor their energy consumption.

In building this solution, eSoftThings realized that they needed a time series database for eSoftLink to store and process the large volume of time-stamped data generated by the smart meters. IoT data, after all, is time series data. Says Chevrier: “We wanted a solution that can support this data volume, which led us to deploying a time series database.”

InfluxDB to support large volumes of IoT data

eSoftThings chose the InfluxDB time series platform because it needed a performant solution with high write speeds and scalability, and something with easy-to-find documentation, community guidance, and long-term support.

To ensure that InfluxDB and Telegraf fit well into the system, eSoftThings performed load tests that recorded server activity as well as Telegraf and InfluxDB KPI’s. The test results confirmed that InfluxDB and Telegraf were the right time series platform for eSoftLink and that they had the right configuration.

With its implementation of InfluxDB (and its metrics collection agent Telegraf), eSoftThings is able to collect a large volume of time series data that provides value-added services while maintaining GDPR compliance. eSoftThings uses InfluxDB to store the time series data that its energy application collects from sensors, and uses this data to help monitor, alert, and predict in real-time.

How eSoftThings’ IoT real-time energy management system works

The eSoftLink IoT solution uses an in-house-built message broker to manage data flow. This includes Apache Zookeeper and Apache Kafka for handling huge data volumes quickly, InfluxDB for time series data, Telegraf for metrics collection, and Chart.js with PHP for dashboarding.

eSoftLink’s main purpose is to collect the data coming from all the IoT nodes connected to the smart meter and to present this data through visualization for:

Customers on their smartphones to monitor their energy consumption
The utility company to manage the fleet of devices connected to the platform

Platform functions include data storage, data analytics, Smart Contract (an application where people can view and select offers available from producers who want to sell their energy directly to consumers), and device management.

Technical architecture of eSoftLink IoT platform

eSoftLink architecture’s three main pillars are: IoT node, backend platform (with administration panels to manage data securely), and end user frontend (mobile/web application).

The blocks within eSoftLink architecture

eSoftLink can connect to a wide range of IoT sensors and devices. Coupled with integrated over the air (OTA) and device management capabilities, it provides customers with a highly dynamic, flexible, and reactive tool to support their operations and business activities. The integration of a data analytics system and a messaging server provides users with real-time notifications and alerts on events of interest. eSoftLink also allows connection to external data lakes to enable long-term storage and cold-analysis of acquired data. This allows extracting maximum value from any dataset on the fly and over time.

To ensure compliance with GDPR (a regulation in EU law regarding data protection and privacy), eSoftThings implemented features that had an impact on the architecture and the design within the database. These features included the right to be forgotten, the right to consult, data portability, data log access, and anonymization.

While ensuring GDPR compliance, eSoftThings had to collect a lot of personal information to provide powerful insights to consumers and providers. They applied several database retention policies given the huge amount of data, taking advantage of InfluxDB’s flexible built-in retention policy functionality.

Smarter energy management decisions in real time

eSoftLink is a versatile and robust end-to-end IoT platform designed to work as a stand-alone on customer infrastructure, on the Edge, or in the Cloud. Its flexibility means it can serve as the integration platform between a provider’s devices, infrastructure, and the cloud service of their choice. eSoftLink provides real-time data collection, management, analytics, and reporting tools that allow enterprises to benefit from the data that they collect quickly and efficiently.

eSoftLink’s implementation for utility providers delivers real-time energy monitoring and reporting, with secure data storage and access. The eSoftLink stack offers a full range of REST APIs, so connecting frontend UIs, such as mobile phone applications and web browsers, is quick and easy.

The eSoftLink consumer application and provider application are very empowering because they provide previously unavailable information and visibility in real time.

The consumer mobile app provides real-time information about consumption down to the device level, thereby allowing consumers to control home energy use. Consumers can instantaneously see the consumption difference they can make, which motivates behavioral change.

The mobile app provides visualization and configuration options, as well as a home comparison tool, which shows the customer’s home consumption relative to similar households. It also shows consumption in real-time, power usage, and bill-to-date.

Sample consumer-facing mobile app visualization from eSoftThings customer Total Direct Énergie

The provider UI empowers the utility company by allowing them to ensure they can quickly service and support their consumers. It allows easy, centralized configuration of platform and device parameters.

Sample visualization from the admin panel developed by eSoftThings for Total Direct Énergie

Provider UI visualizations include the number of active sessions, connections per day, total energy accumulated the previous day, instantaneous power, number of user accounts created, unique connections per day, and average time per session.

The provider UI allows the solution owner to consult any user ID in order to access details on whether the IoT node has been paired and connected, whether it is plugged into the smart meter, and the date of latest firmware update. The solution owner can view and visualize details regarding the power curve, energy, and spending for a selected period.

For the next iteration, eSoftThings is working on improving two recently deployed functionalities: adding blockchain with the capability to do Smart Contract and adding NILM capabilities.

To read the full story, download this real-time energy management case study.

How Strivve is Helping Credit Card Issuers Capture Lost Revenue and Gain Visibility

Chris Churilo (InfluxData) — Fri, 08 Jan 2021 04:00:33 -0700

“We all get our credit cards replaced a lot. Our online ecosystem is getting more complex with the number of accounts we have. We wanted to simplify that, and make it less complicated to get those cards back on file,” says Katherine Chavez, Director of Marketing for Strivve (formerly Switch, Inc.).

Strivve is a startup that aims to take the pain out of updating credit and debit cards by automating the updating process. This was the need driving Strivve to build a platform that helps issuers get their cards into use immediately, to update online payment profiles, and activate new or replacement cards on cardholders’ behalf.

The task of managing cards to site relationships, however, is complex. To build their credit card updater, Strivve needed anonymized site navigation for machine learning to enable using remote process automation (RPA). Strivve chose InfluxDB Cloud, the hosted version of InfluxDB, as their platform’s time series database. Here’s how they built their platform.

Credit card updater solving card issuer and cardholder needs

Strivve sought to solve both card issuer and cardholder needs.

Cardholders have to replace their card on every single site if they get a new card — a painstaking, time-consuming process.
Card issuers and merchants lose significant revenue when reissuing events (such as fraud) occur and replacement cards are active but not used. A card issuer's goal is not to know what sites their cardholders are using but to get their card on those sites.

With the growing number of cards in every wallet, online merchants are storing on file. Besides the need to store credit card information, Strivve needed to store login credentials on all sites used by each cardholder.

The path of machine learning and remote process automation

Early on in their journey, Strivve attempted to gather as much data as possible but soon realized that classic mining techniques were not the right approach. They narrowed down their massive data volume to only data of interest:

Anonymized site navigation for machine learning — so that they can help their users through their remote process automation (RPA).
Anonymized RPA interaction error data for troubleshooting — whereby the service provider gets errors snapshotted to help resolve them.

Strivve first chose Elasticsearch, put a Logstash server up into Linux, and created an EC2 instance on AWS. However, they learned that it was difficult to get the machine learning and troubleshooting artifacts out of Logstash; that Logstash wouldn’t scale for their needs; and that it didn’t fit their strategic model of leveraging non-intellectual property hosted/cloud services.

As a small team, Strivve had to be creative and efficient in gathering data and extracting value from it. They decided to invest their energy into machine learning and remote process automation. This put an end to using Elasticsearch and set them on a search for an alternative.

Managed time series database for Strivve's credit card updater

“We saw some of the other folks using InfluxDB, and some of the scale points they were using it at. Could we generate a big enough load to bother InfluxDB? It was far beyond what we were doing with Logstash at that time,” says Gary Tomlinson, Strivve’s Director of R&D, “We knew it would work for what we were doing.”

Strivve chose InfluxDB because it is:

Purpose-built for time series data
Scalable
Tag system based
Available in a cloud version (InfluxDB Cloud)
Widely adopted by the community

Strivve modified their machine learning system to ingest time-based site navigation events stored in InfluxDB. Since InfluxDB is time-based, it was easy to use InfluxQL to query the data on time for analysis in their machine learning system. This allowed them to structure their data nicely so when problems arise, they are able to navigate the records to determine cause during any specified time period. The InfluxData platform is well-suited for machine learning because it can easily send time series data to an ML solution for training. The trained model can then be brought back into InfluxDB to provide real-time dashboarding and trigger the appropriate workflow.

Using best-in-class security protocols, Strivve’s platform CardUpdatr™ (formerly called TopWallet), also available in the form of an API called CardSavr™, discovers where cards are used for payments, navigates to login pages, and adds new or updated payment cards on behalf of each user. This unique functionality is enabled by streaming anonymized crowdsourced site artifacts into InfluxDB for learning, analysis and support.

<figcaption> The Strivve Platform is easily integrated into any online environment.</figcaption>

Platform architecture using InfluxDB Cloud and machine learning

Early on, Strivve realized that if they anonymize their data, they could crowdsource site artifacts and learn about visited sites that they have commercial relationships with. The Remote Process Automation, which runs in the virtual or real browser, interfaces with merchant sites on behalf of an end user. So they can log in, impersonate users, navigate forms, update billing information and card information, and replace a card.

<figcaption> Service architecture of Strivve’s platform</figcaption>

Strivve leverages machine learning to automate common tasks across tens of thousands of the most popular websites. They modified their machine learning system to ingest site navigation events. InfluxDB Cloud, being time-series-based, made that ingestion easy to set up and implement.

Strivve’s automation technology allows issuers to capture lost revenue of newly issued cards while enhancing cardholder convenience and security. By choosing InfluxDB Cloud, Strivve successfully built their groundbreaking credit card updater service. Learn more about this machine learning use case deploying InfluxDB Cloud.

Network Usage Visibility from the Free InfluxDB sFlow Monitoring Template

Chris Churilo (InfluxData) — Tue, 05 Jan 2021 10:42:18 -0700

As business-critical applications increasingly rely on network services, even a minor change in network usage can impact network performance and reliability, thereby also impacting business functions and network maintenance costs. sFlow (short for “sampled flow”) — by providing unprecedented visibility into network usage and active routes of high-speed and complex networks — delivers the data needed to effectively control and manage network usage. Here’s why, and how, sFlow monitoring provides network usage visibility.

sFlow monitoring to provide a deeper look

Like NetFlow, sFlow is an industry standard used for network monitoring. It defines a packet sampling (not all traffic) technology to provide continuous statistics on any protocol (L2, L3, L4, and up to L7). As it uses sampling, it can scale to high-speed networks. It is supported by multiple network device manufacturers and network management software vendors. Essentially, it’s a tool used to analyze and manage network traffic, to help guarantee quality of service.

sFlow monitoring allows you to take a deeper look at network traffic patterns in an easy-to-visualize, easy-to- understand way. Applications of sFlow data include fixing network problems, real-time congestion management and capacity planning. sFlow monitoring samples network packets to help accomplish important tasks like:

Finding bandwidth hogs on a network
Detecting, diagnosing and resolving network performance issues.
Tracking response time, availability and uptime of critical assets
Monitoring and analyzing network bandwidth performance and traffic patterns

sFlow monitoring using InfluxDB

The sFlow Monitoring Template can be used to monitor traffic from sFlow sources. It is an InfluxDB Template that lets you quickly define your entire monitoring configuration (data sources, dashboards, and alerts) for any technology in one easily-shared, open-source text file that can be imported into InfluxDB with a single command.

The sFlow Monitoring Template allows you to zoom in on and track key performance-related metrics — as shown below — such as bandwidth by source, by destination, by source port and by destination port.

The sFlow Monitoring Template also gives you better visibility into your “top talkers” in sFlow (as shown in the dashboard below), sorting things based on factors like bytes, packets and records. All of these “top talkers” are listed by their IP address so that you can easily tell one from another.

To install this template, sign up for your free InfluxDB Cloud account.

Additionally, the InfluxDB Template Gallery offers other network monitoring templates such as the Network Interface Performance Template and SNMP Network Monitoring Template. And if you’re interested in submitting your own InfluxDB Template, here’s where you can learn how to build one. Most InfluxDB templates have been contributed by the community, and yours can be next.

Monitoring Telegraf Plugins in Your Architecture

Chris Churilo (InfluxData) — Tue, 29 Dec 2020 04:00:06 -0700

Monitoring Telegraf — the open source, plugin-driven server agent for collecting metrics from stacks, sensors and systems — is important because it allows you to track the health of Telegraf plugins in your stack. It’s for this purpose that the Telegraf Monitoring Template was developed. Before introducing the template, it’s helpful to recall how Telegraf works.

Telegraf in a nutshell

In case you’re not familiar with Telegraf or can benefit from a refresher, Telegraf is a great choice for metrics collection because it’s written entirely in Go; compiles into a single binary with no external dependencies; has a minimal memory footprint; and has 200+ plugins written by subject matter experts in the community.

Telegraf has plugins to source a variety of metrics directly from the system it’s running on, pull metrics from third party APIs, or even listen for metrics via statsd and Kafka consumer services. It also has output plugins to send metrics to other datastores, services and message queues, including InfluxDB, Graphite, OpenTSDB, Datadog, Librato, Kafka, MQTT, NSQ and others.

The template's Telegraf monitoring dashboard

The Telegraf Monitoring Template is an InfluxDB template that has a premade dashboard. Below is the template’s dashboard.

The dashboard shows metrics collected, graph errors, and throughput rates for inputs, outputs, and time spent collecting and writing metrics. See the list of Telegraf monitoring metrics you can proactively track using the template.

Advantage of using the Telegraf Monitoring Template

Since Telegraf is InfluxDB’s native metrics collection agent, it’s much easier to use the Telegraf Monitoring Template — an InfluxDB template — to monitor Telegraf than to use other monitoring solutions or to build a monitoring solution from scratch.

Using this template, like other InfluxDB Templates, is easy:

Once your InfluxDB credentials have been properly configured in the CLI, you can install the Telegraf Monitoring Template using the Quick Install command.
Once installed, the data for the dashboard will be populated by the included Telegraf configuration, which includes the relevant Telegraf Input.

Depending on how your organization is running Telegraf, you might need to customize the input configuration to better serve your needs.

To install this template, sign up for your free InfluxDB Cloud account.

Get started with InfluxDB Templates today

The InfluxDB community templates repository is home to a growing number of InfluxDB templates. InfluxDB Templates let you quickly define your entire monitoring configuration (data sources, dashboards, and alerts) for any technology in one easily-shared, open-source text file that can be imported into InfluxDB with a single command.

To use an InfluxDB template, apply community templates directly from GitHub using a template’s download URL or download the template. See How to use a Template for detailed instructions on how to load a template. The InfluxData Docs website also shows how to view, validate and apply templates. Interested in submitting your own InfluxDB Template? Learn how to build one.

Monitoring Endpoint Security States with InfluxDB

Chris Churilo (InfluxData) — Wed, 09 Dec 2020 04:00:19 -0700

Several factors in recent years have increased endpoint vulnerability — from organizations’ need to make access to data more fluid, to threats targeting mobile device access and networks, to the growing work-from-home and work-on-the-go trends. Endpoints connected to a network — including remote devices, IoT devices, workstations, tablets, laptops and servers — create attack paths for security threats.

While most security breaches in the past had entered through the network, threats today increasingly come in through endpoints, requiring a new layer of security through endpoint protection. Greater control over access points helps prevent vulnerabilities posed by the use of remote devices.

Why is endpoint security important?

When hackers and cybercriminals execute code and exploit existing vulnerabilities in endpoints, they get access not only to the data contained on that device — but also potentially an unrestricted way to get onto the network that endpoint is connected to.

Endpoint security secures the various endpoints on a network to keep them safe from malicious attacks. For many years, organizations relied on antivirus software to secure endpoints, yet as hackers’ methods became more advanced, more sophisticated security measures became necessary.

Monitoring the security state of endpoints helps you to understand the effectiveness of the endpoint detection and response. Having clear visibility in this area allows your security team to take advantage of threat intelligence, to help identify known threats and respond in a timely fashion.

How to monitor endpoint security states

One security mechanism you can use to secure these endpoints is endpoint authentication, which ensures only authorized devices can connect to the endpoint. To monitor endpoint security states, Security Tools Manager at InfluxData Darin Fisher created the InfluxDB Endpoint Security State Template. Like other InfluxDB Templates, this template lets you quickly define your entire monitoring configuration (data sources, dashboards, and alerts) in one easily-shared, open-source text file that can be imported into InfluxDB with a single command.

This InfluxDB Template works by connecting to secure endpoints and attempting to log in. Using the http_response and x509_cert Telegraf plugins, availability, authentication, and certificate information are collected. The dashboard displays the general availability for each endpoint along with the x509 certificate status and authentication state.

Fisher provides a step-by-step walkthrough of how to install and use the template. Read it to learn more about the monitoring metrics you can track using this template, and the endpoint security states represented by dashboard icons.

To install this template, sign up for your free InfluxDB Cloud account.

Getting a Headstart with the Docker Monitoring Template

Chris Churilo (InfluxData) — Tue, 01 Dec 2020 04:00:00 -0700

The growing popularity of Docker has led many enterprises to containerize applications. By 2022, more than 75% of global organizations will be running containerized applications in production, Gartner predicts, up from less than 30% today. Yet the shift to containers has posed new challenges to performing effective monitoring. As more applications move to the cloud and become containerized, the demand for dynamic container monitoring has become more urgent. The Docker Monitoring Template offers a simple solution to this problem, but before presenting it, let’s provide an overview of Docker.

Why monitor Docker?

Docker monitoring, often referred to as container monitoring, is critical to safeguard the performance of your containerized applications. Docker monitoring can help you:

Detect and resolve issues proactively
Implement changes frequently and safely
Optimize resource management and allocation

Monitoring container resource utilization is important because containers have multiple isolated perimeters. You need to consider metrics like the CPU utilization of a container instance and the aggregate usage for the host it is running on. Visibility into resource utilization, gained through exploring high-granularity historical data, will also enable you to detect excess capacity and improve operational costs.

Docker monitoring with a premade dashboard

Start monitoring Docker easily with the Docker Monitoring Template - an InfluxDB Template that improves visibility into your containerized environment’s performance and helps you detect issues quickly. This template uses the Docker Telegraf plugin to collect the metrics stored in InfluxDB, which are then visualized in the included dashboard, as shown below.

InfluxDB Templates let you quickly define your entire monitoring configuration (data sources, dashboards, and alerts) for any technology in one easily-shared, open-source text file that can be imported into InfluxDB with a single command.

Docker monitoring metrics to track using the dashboard

Tracking key Docker monitoring metrics allows you to analyze your containers’ health and manage resources effectively. Metrics you can proactively track with the Docker Monitoring Template include:

Number of Docker containers
Network TX traffic per container/sec
CPU usage per container
Memory usage % per container
Memory usage per container
Network RX traffic per container/sec

To install this template, sign up for your free InfluxDB Cloud account.

Building InfluxDB Templates is easy, and many InfluxDB Templates are contributed by the InfluxDB community. Here’s how to submit and build your own InfluxDB Template.

Tracking CS:GO Stats with InfluxDB

Chris Churilo (InfluxData) — Tue, 24 Nov 2020 04:00:47 -0700

Counter-Strike: Global Offensive (CS:GO) – a multiplayer first-person shooter video game developed by Valve and Hidden Path Entertainment – is the fourth game in the now-legendary Counter-Strike series. Since its release, CS:GO has drawn in an estimated 11 million players per month, and has gathered an active competitive scene, with many tournaments sponsored by Valve themselves. Here’s a quick overview of CS:GO, and why and how to track your CS:GO stats.

CS:GO gameplay overview

CS:GO was first made available for the Windows, Mac, Xbox and PlayStation 3 platforms in 2012, while the Linux version was released in 2014. The game pits two teams against each other: the Terrorists and the Counter-Terrorists. Each side is tasked with eliminating the other while also completing separate objectives.

Depending on the game mode, The Terrorists must either plant the bomb or defend the hostages while the Counter-Terrorists must either prevent the bomb from being planted, defuse the bomb, or rescue the hostages.

The game also has matchmaking support that allows players to play on dedicated Valve servers, as well as allowing members of the community to host their own servers with custom maps and game modes. In December 2018, Valve transitioned the game to a free-to-play model, focusing on revenue from cosmetic items.

Why it's worth tracking your CS:GO stats

Tracking your CS:GO stats will help you remain competitive in your matches and identify which items to use. It is your best opportunity to get the most out of the game, as it allows you to see which of your current techniques are working and which aren’t and also allows you to continue to enjoy the game’s competitive nature well after you’ve “completed” the primary missions.

To really see how you’re doing as a player, it’s worth tracking game metrics over time, such as last match kills, total MVP, stats by weapon, total kill count, and kill-to-death ratio.

How to track your CS:GO stats

An easy way to monitor your CS:GO stats is through the Counter-Strike Gametracker with InfluxDB, an InfluxDB Template created by Ignacio Van Droogenbroeck, InfluxAce and winner of the Founder’s Choice Award at the 2020 InfluxData Community Awards.

InfluxDB Templates are a set of tools that includes a packager and a set of pre-made monitoring solutions that allow users to share their monitoring expertise with their coworkers and other community members around the world. Ignacio created the Counter-Strike Gametracker with InfluxDB to analyze his performance through time and own that data.

Using the InfluxDB CS:GO Gametracker

The CS:GO Gametracker with InfluxDB enables what other CS:GO gametrackers don’t. Says the template’s creator Ignacio: “This template offers me the possibility to track my performance through time and answer questions like: Am I improving compared to last month? With which gun am I performing better than the last month? This kind of data is not available on the default tracker, where you only see the latest data, and of course, you can’t play with it.”

The desire to be data-driven is what led Ignacio to build this template: “It’s in my nature to be competitive and improve over time. Also, I like data because I consider it critical to take decisions and change when necessary. With this template, I learn that I shoot a lot with PP-Bizon, but I hit less compared with MP7, so I change my choice and start to kill more — the same with M4A1 and M4A1-S. I’m more accurate with a silencer than without it.”

Potentially, the template can be adjusted to display game metrics in real time. Says Ignacio: “It can be very interesting for the coaches to understand how your players are performing at the game in a specific moment, with which gun they’re performing better, how much they kill or die and how their economics are performing. The template is not ready to get data in real-time but is easy to adapt.”

In fact, the template can be changed to adapt to any needs: “If you don’t want to know how many windows you broke, see total damage done, or specific data about a gun, you can delete this pane. But the info is there, stored in InfluxDB to be used whenever you need.”

Building InfluxDB Templates

As for the benefit of this template being specifically an InfluxDB template, where game metrics can be stored, Ignacio says: “I find it very easy to build templates for InfluxDB, and also, it’s very easy taking data using Telegraf. The data originally comes in JSON, XML, and VDF and can be stored wherever you want. But to take advantage of this template to understand your game over time, a time series database is recommended.”

Interested in submitting your own InfluxDB Template? Learn how to build one.

InfluxData Blog - Chris Churilo

InfluxDB is 5x Faster vs. MongoDB for Time Series Workloads

Versions tested

About the benchmarks

About the dataset

Write performance

On-disk compression

Query performance

Summary

What's Next?

InfluxDB Tops Cassandra in Time Series Data & Metrics Benchmark

Versions tested

About the benchmarks

About the dataset

Write performance

On-disk compression

Query performance

Summary

What's next?

Real-Time Energy Management with InfluxDB and eSoftLink IoT Platform

Making smart energy meters even smarter

Energy consumption visibility at the appliance level

InfluxDB to support large volumes of IoT data

How eSoftThings’ IoT real-time energy management system works

Technical architecture of eSoftLink IoT platform

Configuring InfluxDB to ensure GDPR compliance

Smarter energy management decisions in real time

How Strivve is Helping Credit Card Issuers Capture Lost Revenue and Gain Visibility

Credit card updater solving card issuer and cardholder needs

The path of machine learning and remote process automation

Managed time series database for Strivve's credit card updater

Platform architecture using InfluxDB Cloud and machine learning

Network Usage Visibility from the Free InfluxDB sFlow Monitoring Template

sFlow monitoring to provide a deeper look

sFlow monitoring using InfluxDB

Monitoring Telegraf Plugins in Your Architecture

Telegraf in a nutshell

The template's Telegraf monitoring dashboard

Advantage of using the Telegraf Monitoring Template

Get started with InfluxDB Templates today

Monitoring Endpoint Security States with InfluxDB

Why is endpoint security important?

How to monitor endpoint security states

Getting a Headstart with the Docker Monitoring Template

Why monitor Docker?

Docker monitoring with a premade dashboard

Docker monitoring metrics to track using the dashboard

Tracking CS:GO Stats with InfluxDB

CS:GO gameplay overview

Why it's worth tracking your CS:GO stats

How to track your CS:GO stats

Using the InfluxDB CS:GO Gametracker

Building InfluxDB Templates