InfluxData Blog - Al Sargent

How InfluxDB Works with IoT Data

Al Sargent (InfluxData) — Mon, 05 Apr 2021 04:00:09 -0700

Over the past few years, the IoT community has embraced InfluxDB as a cornerstone of the solutions they build. Whether modernizing or greenfield, InfluxDB has helped many in working with vast quantities of sensor and device data as we continue to deliver on our promise of time to awesome for IoT.

Today, let’s look at why you should consider InfluxDB for your IoT data architecture, how to plan your IoT architecture, and what we’ve built recently to make InfluxDB even better at handling IoT data.

So, let’s dive in!

Why use InfluxDB for IoT telemetry?

It boils down to a few things:

Proven. Every IoT device generates time series data, and the explosion of these devices and the data they generate, fueled by the desire for more analytics, has driven the requirements for this type of data to be handled by a specialized platform. InfluxDB has proven its chops at some of the most demanding IoT customers in the world, such as Siemens and PTC, as a time series database at the core of their IoT stack.
Scalable. IoT data volumes and the appetite to consume them are rapidly growing. InfluxDB scales to ingest and index massive volumes of IoT telemetry, while providing real-time analytics and fast query response times.
Extensible. IoT telemetry comes in a wide variety of formats across many domains — legacy standards, new datatypes, and everything in between. InfluxDB can ingest a broad range of data formats. If you need a format that isn't supported, InfluxDB's extensibility lets you ingest it.
Flexible. InfluxDB's Flux language is incredibly powerful for building IoT solutions, since it gives you a single language to both shape and query your IoT data, and does so at the database layer so that you don't have to resort to expensive queries to download data to modify it.
Contextual. InfluxDB blends rich metadata, such as IoT asset information, into time series measurements, making it easier to derive insights on past, current and future performance of assets and processes.

Every IoT use case is a time series use case

IoT - Internet of Things - is increasingly broad. The pace of deployment of sensors across nearly every industry from agricultural to oil & gas, power generation, and transportation, allows us access to time-stamped information at about the same efficiency with which products are made or grown, the potential need for maintenance and repairs, and more. Here are just a few examples to explore what’s possible today:

Connected fleets: Solar power companies such as BBOXX regularly send measurements on the voltage produced by each of their installations, so they can tell when a particular one is in need of repair — sometimes before end customers themselves know.
Smart spaces : Office tower managers like Aquicore track the amount of energy consumed across all floors of a building, a "Smart City" example of optimizing the amount of energy required to provide a comfortable environment for occupants.
Smart products: Office water dispensers, for instance Bevi, measure the amount of flavoring used, so that their service personnel can automatically restock flavorings just before they run out.

IoT data architecture

We’ve had a large number of customers successfully use InfluxDB for IoT across a range of industries, but two architectural patterns continually emerge:

Hub only
Edge and hub

Hub-only IoT data architectures are where each of your “things” send their IoT telemetry to a central time series database (such as InfluxDB Cloud) for storage, enrichment and analysis. All devices are assumed to have a reliable, performant Internet connection to send their IoT telemetry, and thus all consumers of that data access it from that centralized database.

For example, tado°, a provider of connected home thermostats for efficient cooling and heating, sends all their thermostat telemetry to InfluxDB Cloud. tado° customers access their home’s temperatures from tado°’s mobile app, which pulls data from InfluxDB. This hub-only data architecture works given that a home thermostat will always be located within coverage of a house’s WiFi router.

Credit: tado°

Edge and hub data architectures are required when a piece of equipment doesn’t have a network connection that is fast and reliable, and where people are onsite and require access to the analytics to assist in operational decision making, while centralized visibility across all of the potential edge sites can be leveraged to understand trend analysis, operational efficiencies or identify potential issues. In these cases, it makes sense to dual-write time series telemetry to an onsite time series database, as well as to a centralized instance.

For example, Equinor, an energy producer, operates oil rigs in the North Sea. Their oil rigs require near real-time visibility into various aspects of the rig so that it can be run properly. In this case, it makes sense for Equinor to deploy InfluxDB on oil rigs, in order to make this data accessible to rig operators.

At the same time, these oil rigs transmit time series telemetry data — often at a lower resolution than is being captured and stored locally — to their corporate headquarters. The corporate HQ folks can analyze the data from all of the rigs and look at the similarities and differences between the operational characteristics of the equipment in the aggregate. This allows for analyses like predictive maintenance to be performed and those recommendations sent back to the rig operators. We find the edge-and-hub pattern especially common in use cases for IIoT (industrial IoT) and factories implementing Industry 4.0.

Credit: Jamie Baikie and Michal Wachucik / Equinor ASA

So, when planning your IoT data architecture, ask yourself which model makes the most sense for you and your organization’s needs. Speaking of questions…

How to plan your IoT data architecture

There are eight types of questions to consider when planning your IoT data architecture; here they are:

Data & decisions: What kind of IoT data are you collecting, and what decisions are being driven by this data? You’ll likely have three kinds of decisions to consider:

Past: How did we perform previously? What impact did this have on quality, efficiency, or quantity, and what changes (if any) does it suggest? This analysis can be especially useful when rolled up across multiple sites (factories, wind turbines, etc.).
Current: How are we performing right now? Are there corrective actions that need to be taken? These kinds of decisions are often taken by onsite operational staff.
Future: How are we likely to perform in the future, based on the data we've seen so far? This can help drive decisions on predictive maintenance, by using data on mileage or usage to determine when a piece of equipment will need to be repaired or replaced.

Data & storage: Which data makes sense to be stored in a relational database, and which should be stored in a time series database? Data that changes over time, such as sensor readings (temperature, pressure, voltage, etc.) is typically best in a time series database.

However, data that is static over time, such as IoT asset metadata like machine IDs, sensor ID, make, model, purchase date, last repair date, or location of fixed equipment, is typically best stored in a relational database. After all, rewriting the same unchanging information over and over into a time series database would be a waste of storage.

When your time series data and metadata are stored in separate databases, it’s important to be able to blend this metadata into your time series data to provide context for better analysis — and we’ll see how to do that below.

It’s also crucial to work with a time series database that gives you a choice of cloud vendors and locations for storing your IoT data. This gives you the freedom to use whichever vendor has the cloud services you need, and the flexibility to address whatever data residency requirements you have. InfluxDB Cloud does this, running on AWS, Azure, and Google, in several regions globally (and we’re continually expanding).

Data & location: If you are using a hub-and-edge pattern, which data do you store at the edge, and which at the hub? For onsite operators to do their job, which data do they need, at what frequency does it need to be collected (once a second, every 10 seconds, etc.), and at what frequency does it need to be visualized to be best understood? Similarly, for analysts at corporate to do their job, what data do they need, how often does it need to be sent from edge to hub, and at what resolution?

Networks & data resolution: How does network performance and availability come into play in terms of when the data is sent, and what kinds of rollups are required? For example, BBOXX provides solar to developing countries and needs to collect data over slow 2G cellular networks; in these cases, downsampling data is often required. This is different from, say, tado°, where each connected thermostat is located next to a fast, reliable WiFi connection, which means that data can be sent more frequently, with less need for downsampling.

Credit: Unsplash via @riiyad

Data volumes: How many sensors are deployed, how often do they send data, how many IoT measurements and events do they send, and what (if any) downsampling is done? All this affects the amount of data you’ll need to ingest in a time series database. If you plan to store data at the edge using an onsite time series database, like InfluxDB OSS, you’ll need to plan your computing resource consumption (memory, disk, CPU) and size your hardware accordingly. And if you plan on storing data at the hub using InfluxDB Cloud, you don’t have to plan on how to scale, since InfluxDB Cloud scales automatically; simply use its usage-based pricing to determine your budgeted spend.

Existing data architecture: What are you currently using to transmit time series data from sensors? What (if anything) are you using to persist that data? And what are you using to store your static data (sensor ID, etc.)? You’ll want to choose a time series platform that provides you with plenty of options to fit with your current architecture, and give you options to upgrade your architecture in the future. (More on that below.)

Existing performance: How well is your existing data architecture performing? In the past, we’ve seen companies and organizations struggle to ingest the sheer volume of time series data they produce. This led Aquicore, which manages office buildings with IoT sensors, to switch off Postgres and move to InfluxDB for time series data. Query speeds also matter, both for onsite operators that need real-time telemetry to make critical adjustments, as well as corporate analysts looking to discern trends in large datasets. Query performance led tado° to switch from MySQL to InfluxDB, since the latter provided a 200x reduction in query times.

Architecture & authority: Even the best architecture can be derailed without taking human and organizational factors into account. You want to understand who has authority and influence over decisions around data architecture, including who has a stake in keeping things as-is, and who is pushing for change. This includes looking broadly at people both in OT (operational technology) and IT (information technology), and both in the field (factories, power plants, etc.) and at corporate offices. If you’re in a regulated industry, it may include people on legal and compliance teams, who may want data transformed to remove PII (personal identifying information) — more on that below.

Credit: Unsplash via @tool_inc

What's new with InfluxDB and IoT

Over the past months, we’ve invested in expanding our product-market fit for IoT customers with devices in motion as well as fixed equipment. This includes investments in geotemporal data enrichment that are available today, and visualizations (maps) that will be landing soon in InfluxDB Cloud.

These IoT-related product investments span our time series data platform, which includes:

Telegraf, to acquire and enrich IoT telemetry data
InfluxDB Cloud, to persist, enrich, and analyze IoT data at the hub
InfluxDB OSS, to persist, enrich, and analyze IoT data at the edge

And they break into four areas: Acquire, Enrich, Operate, and Analyze.

Let’s jump in and look at each.

Acquire IoT telemetry data

We’ve worked hard to enhance Telegraf and InfluxDB so that you have the flexibility to connect to a broad range of existing IoT sensors and devices. With these enhancements, InfluxDB can:

Acquire data from the majority of commonly used IIoT (industrial IoT) telemetry protocols like OPC-UA;
Support open, standards-based protocols like MQTT; and
Provide extensibility to gather data from other protocols.

Our recent additions include the following:

Telegraf OPC-UA plugin (connects to many systems, including PTC Kepware)
Telegraf Modbus plugin
Telegraf KNX plugin (coming soon in Telegraf 1.19)

These enhancements are in addition to the following longstanding capabilities - a pretty long list that reflects the breadth and diversity of the IoT market shown above, including:

Telegraf MQTT plugin
Telegraf AMQP plugin
Telegraf Azure Event Hub plugin
Telegraf RTI DDS plugin
Native integrations between InfluxDB and Apache NiFi, openHAB, WinCC, Node-RED, Particle.io
..and many more

Also, InfluxDB offers a rich set of APIs enabling virtually any system or device (using a variety of programming languages) to send data to and query from the data store.

Here’s a graphic to summarize our IoT-related data acquisition capabilities:

Even with the above breadth of support and the over 200 Telegraf input plugins, there may still be datasources we don’t yet support. In these cases, submit a feature request on the Telegraf repo or leverage the extensibility through Telegraf’s ExecD integration shim.

Enrich IoT telemetry data

Over the past year, we’ve been enhancing InfluxDB to let you combine IoT asset metadata - again, things like machine IDs and sensor IDs - with IoT time series telemetry to provide a human-readable view of device status and health. There are three ways you can enrich IoT telemetry with InfluxDB:

Metadata enrichment
Geospatial enrichment
Calculated values enrichment

Let’s cover each.

Metadata enrichment. Here’s an example. In InfluxDB line protocol, you could have the following time series data point for a wind turbine:

turbineVoltage, turbineID=1234 voltage=150 1556813561098000000

If you’re a human, this doesn’t tell you a whole lot. How big is this turbine? Is it producing its expected output? If not, when was it last serviced? Does it need to be taken offline soon?

But let’s say that you have a relational database that contains turbineID as a primary key for your turbines table, with columns for turbine make, model, and last service date. In this case, you can query your relational databases for metadata, then join it to your time series data, while keeping your relational database credentials secret. All this you can do using Flux, InfluxDB’s language for working with time series data.

Once you have this richer relational-plus-time-series view of data, questions like the above are much easier to answer. Over the past year, we’ve added support for a broad range of relational databases, including PostgreSQL, MySQL, Snowflake, SQLite, Microsoft SQL Server, AWS Athena, Google BigQuery, MariaDB, and CockroachDB. This allows you to add metadata from a broad set of sources, but if there are others you’d like us to support, please let us know by submitting an issue in the Flux repository.

Geospatial enrichment. Often, IoT telemetry will contain the latitude and longitude of a device, and you’ll want to use this data to filter telemetry by a particular location. Because IoT data volumes can be huge, it’s important to ensure that these geospatial queries run as quickly as possible.

Flux’s geotemporal query functions can leverage a specialized geospatial index, which can be calculated at data collection time or at query time from the raw lat/lon data dynamically, if necessary.

Credit: Unsplash via @dead__artist

Calculated values enrichment. Sometimes you need to enrich time series telemetry with calculated values. For instance, you might want to use your time series telemetry on electrical current and resistance to calculate voltage, using the formula voltage = current * resistance. Or convert imperial measurements like Fahrenheit, miles, or pounds to their corresponding metric values. Or reduce the precision of IoT measurements.

At ingest time, you can add calculated values to your incoming time series telemetry using the Telegraf Starlark plugin. And you can add calculated values to your existing time series data using InfluxDB Tasks and the Flux map() function.

Here’s a graphical summary of what we’ve covered so far around IoT data acquisition and enrichment:

Operate using local IoT telemetry data

As mentioned, sometimes it’s helpful to store IoT telemetry data onsite or on-vehicle, so equipment operators have high resolution data available in real-time, without any concerns around network performance.

To support these use cases, InfluxDB OSS (open source software) is a good option, since it provides a complete time series data platform (storage, analysis, tasks, dashboards, and alerting) in a single lightweight binary. This full range of functionality makes InfluxDB OSS a good choice for intelligent edge use cases.

Since there are a broad range of IoT devices that run on ARM processors, we’ve recently ported InfluxDB to ARM 64 to provide you with more deployment flexibility. Now you can go to our download page to get InfluxDB binaries for Ubuntu, Debian, RHEL, and Centos running on ARM 64-bit systems such as a Raspberry Pi 4 series, and there are ARM 64 Docker images for InfluxDB as well.

And if you’re running Windows 10, InfluxDB OSS will soon run on that as well - so stay tuned!

Here’s what we’ve covered so far:

Analyze IoT telemetry data

Once your data is ingested and enriched, it’s time to analyze it to drive decisions, such as those listed above. And for that, we’ve recently made some significant strides; here are two.

Geotemporal data. InfluxDB now lets you run geotemporal queries, both for fixed and mobile assets. For instance, you might want to understand which vehicles’ engines are performing properly? What’s been the output of all our factories, solar farms, or wind turbines? And so on.

In these cases, you can query InfluxDB by geographic location using the Flux Geo package. And soon, you’ll be able to visualize those results on a map in your InfluxDB Cloud Data Explorer and Dashboards. Below is an example.

Maps now in InfluxDB Dashboards & Data Explorer

The Flux Geo package is very flexible: you can find all points within a radius of a particular point, a rectangle, a single point, or a polygon of an arbitrary shape (such as the boundary of a city, state, or country). And you can combine geotemporal filtering with many other types of Flux functions.

Extensible analytics. One of the reasons we created Flux was that we wanted a language that was easy to extend. Across the range of IoT domains above, there is a correspondingly broad range of analytic functions we can add to Flux. One of these is a Flux function to calculate OEE (overall equipment effectiveness), a standard for measuring manufacturing productivity. For instance, an OEE score of 100% means you are manufacturing only good parts, as fast as possible, with no stop time.

Credit: Unsplash via @lennykuhne

OEE is but one example of Flux’s extensibility. If there are other analytical functions you’d like us to add to Flux, submit an issue in the Flux repository.

To sum up, here’s a diagram of all four categories of IoT-related investments in our product line:

Our IoT product philosophy

What you’ve seen above are the first steps in a longer journey of IoT capabilities, so I’d like to share the thinking behind what we’re building so that InfluxDB works as a foundational element of your IoT data architecture:

Time to awesome for IoT: Since our founding, InfluxData has been focused on making it easy to work with time series data. We’ve continued to follow this principle as we’ve expanded deeper into meeting the needs of our IoT and IIoT customers, by striving to provide the right mix of ease-of-use, power, and extensibility to our time series platform.

Self-managed and as-a-Service: We realize that there’ll be a range of requirements and use cases across our customer base. Some of you will want the flexibility to assemble your own IoT telemetry platform using the components above. Or you might want a pre-assembled IoT platform; if so, check out PTC ThingWorx, which uses InfluxDB as its time series database.

Greenfield and brownfield: Some of our customers have the luxury of starting with a blank page when it comes to formulating their IoT data architecture; we call them the greenfield customers. But many others, the brownfield, have decades-old legacy IIoT historians that can’t handle the exponential growth of IoT telemetry data, and the appetite across the organization to consume it. We have to meet the needs of both our green- and brownfield customers, by supporting both older and newer technologies. And this includes coexisting with legacy historians when necessary.

Incremental modernization: In live industrial settings, no one ever changes everything all at once. It’s too disruptive to operations. Instead, we see incremental modernization, for instance, when new PLCs (programmable logic controllers) that arrive as old equipment is replaced. We need to provide an IoT telemetry platform that grows with our customers.

Partnering with domain experts: We understand that IoT is a broad range of domains, and so we partner with a range of IoT experts to deliver IoT solutions to our joint customers. In addition to PTC, these include Nortal, Factry RTI, Particle, Seeq, and Rockwell. Do reach out to them as needed!

To sum up, we see InfluxDB as a core piece of a modern IoT stack that meets the above requirements.

Conclusion

Now that you’ve seen common IoT deployment patterns, architectures, use cases, and how InfluxDB addresses them, here’s what to do next:

Sign up for a free InfluxDB Cloud account and follow our getting started guide, which provides a tutorial.
Review our whitepapers on IoT and IIoT.
Learn from other IoT and IIoT customers.

And of course, feel free to engage our ever-helpful InfluxDB community and Slack for questions. Enjoy!

InfluxDB Cloud is on Microsoft Azure Marketplace

Al Sargent (InfluxData) — Tue, 16 Feb 2021 04:00:20 -0700

Here at InfluxData, we’ve been focusing recently on deepening our support for Microsoft Azure. First we turned on InfluxDB Cloud on Azure West Europe, in Amsterdam, back in July. Then we launched InfluxDB Cloud on Azure East US, in Virginia, in September. Today, we’re pleased to announce that InfluxDB Cloud joins InfluxDB Enterprise on Azure Marketplace.

How does InfluxDB work with Azure and other Microsoft technologies?

InfluxDB integrates with a broad range of technologies related to Azure and Microsoft more broadly; here’s a visual representation. Microsoft-related technologies are in gray, while InfluxDB components are in purple.

Let’s dive deeper into each of these integration points.

To accumulate time series data, Telegraf, our open source data collection agent, has the following input plugins for Azure and other Microsoft technologies:

Flux, our open source time series query language, lets you enrich your time series data by pulling in metadata from SQL Server. This lets you, for instance, resolve an IoT device’s serial number in your time series data (in InfluxDB) to data in SQL Server, such as make, model, and mileage. Here’s what that looks like in Flux:

import "sql"
sql.from(
  driverName: "sqlserver",
  dataSourceName: "sqlserver://user:password@localhost:1234?database=examplebdb",
  query: "GO SELECT * FROM Example.Table"
)

To analyze your time series data, you have the full power of InfluxDB — visualization, dashboarding, data processing, anomaly detection, and alerting — all running on Azure Cloud and billed using Azure Marketplace.

We provide the following InfluxDB Templates via our template gallery that let you quickly set up monitoring datasources (Telegraf configurations) and dashboards:

InfluxDB Templates are an incredible time saver. In just one minute, they let you import a Telegraf configuration and dashboard that you can easily customize so you don’t have to spend hours building from scratch. For instance, here’s a snippet of the dashboard that comes with the SQL Server template that contains over two dozen query and associated charts:

As part of this launch, we have also added a Microsoft SSO login option to InfluxDB Cloud. This allows InfluxDB Cloud users to login with their Azure Active Directory credentials.

And to act on your time series data, Flux lets you send alerts to Microsoft Teams, as well as other popular endpoints such as PagerDuty, Slack, webhooks, and more. For instance, here’s how to use Flux to post a message to Microsoft Teams:

import "contrib/sranka/teams"
teams.message(
  url: "https://outlook.office.com/webhook/example-webhook",
  title: "Example message title",
  text: "Example message text",
  summary: "",
)

We also provide a C# client library for InfluxDB.

Why access InfluxDB on Azure Marketplace?

When you sign up for InfluxDB through Azure Marketplace, you can consolidate your InfluxDB Cloud and Enterprise expenditures across all teams, onto a single Azure bill. Cloud computing is all about eliminating tedious tasks so you can focus on what matters. Doing this provides the following benefits:

Focus on your day job — Without Azure Marketplace billing, your developers and SREs have to put Azure charges on their credit cards and later expense them. That’s one more administrative task to pull them from their work. With our new Marketplace integration, you give them a fast, easy way to spin up the time series database instances they need to be productive. And they don’t need to deal with time-consuming expense reports.

Pain-free reporting — All your InfluxDB charges, for all accounts, appear on your existing Azure bill, so they’re easier to track. For engineering and IT managers, this makes it easier to generate spending reports so they can focus on what matters — delivering projects on time, and keeping critical services running.

Streamlined purchasing — There’s no need to work with your purchasing department to onboard InfluxData as a new vendor or deal with the hassle of setting up a new contract. If you’re already using Azure, then you remain with a single vendor for both your Azure usage and your InfluxDB usage.

Streamlined support — When you’re trying to keep a development project on track, or a critical service running, you know that any downtime can throw a wrench into your plans. When outages occur, the InfluxDB and Microsoft teams work closely to quickly pinpoint the root cause to minimize downtime.

To sum up, we hope that accessing InfluxDB through Azure Marketplace means less stress for you and your team.

What's the pricing of InfluxDB Cloud on Azure?

Pricing for InfluxDB Cloud on Azure Marketplace is based on usage with no minimum spend. It offers the same, simple low rates available directly through InfluxDB Cloud.

InfluxDB Cloud Pricing Vector	Price
Data In	$0.002/MB
Queries	$0.01 per 100 query executions
Storage	$0.002/GB-hour
Data Out	$0.09/GB

The pricing of InfluxDB Cloud on Azure actually is pretty cool. Here’s why:

With most “cloud” databases, you have to pick the size of the machine you want to run the database on. Pick too small a machine, and your database will fall over. Pick too large, and your budget will take a hit. Migrating your data from one size of machine to another is a time sink. This means you miss many of the advantages of the cloud, specifically cost savings and time efficiency.

InfluxDB Cloud is different. The reason is that, under the hood, InfluxDB Cloud is a multitenant database cluster running on Kubernetes. This lets it dynamically scale to provide whatever ingest, storage, and query capabilities you need, on the fly. It means that you only pay for what you use — specifically, the data you send us, the data you store, and how long your queries run. That’s why we call it usage-based pricing.

So, you are freed from the hassle of trying to pick a big enough machine, the risk of overpaying for a database, and the frustration of having to migrate from one machine to another.

Here’s more on how this pricing gives you complete control over your time series database spend on Azure, as well as how to understand and forecast your charges.

<figcaption> Credit Alexey Ruban via Unsplash</figcaption>

Where does InfluxDB run on Azure?

As of the time of this writing, InfluxDB Cloud currently runs in two Azure locations: West Europe (Amsterdam) and East US (Virginia). Here’s the most current list of InfluxDB Cloud locations.

InfluxDB Enterprise runs in 33 Azure locations worldwide:

Region	Location
Europe, Middle East, & Africa	Amsterdam, Netherlands
Europe, Middle East, & Africa	Dubai, UAE
Europe, Middle East, & Africa	Dublin, Ireland
Europe, Middle East, & Africa	Paris, France
Europe, Middle East, & Africa	Frankfurt, Germany
Europe, Middle East, & Africa	Liverpool, England
Europe, Middle East, & Africa	Oslo, Norway
Europe, Middle East, & Africa	Portsmouth, England
Europe, Middle East, & Africa	Pretoria, South Africa
Europe, Middle East, & Africa	Zurich, Switzerland
Americas	Blue Ridge, Virginia
Americas	Boydton, Virginia
Americas	Chicago, Illinois
Americas	Des Moines, Iowa
Americas	Quebec City, Canada
Americas	Quincy, Washington
Americas	Salt Lake City, Utah
Americas	San Antonio, Texas
Americas	San Francisco, California
Americas	Sao Paulo, Brazil
Americas	Toronto, Canada
Asia-Pacific	Busan, Korea
Asia-Pacific	Canberra, Australia
Asia-Pacific	Chennai, India
Asia-Pacific	Hong Kong, China
Asia-Pacific	Melbourne, Australia
Asia-Pacific	Mumbai, India
Asia-Pacific	Osaka, Japan
Asia-Pacific	Pune, India
Asia-Pacific	Seoul, Korea
Asia-Pacific	Singapore
Asia-Pacific	Sydney, Australia
Asia-Pacific	Tokyo, Japan

Conclusion

The quickest way to get started is to get InfluxDB Cloud on Azure Marketplace. And for those who prefer to run on-prem, here’s InfluxDB Enterprise on Azure Marketplace. Once signed up, check out our getting started guide and feel free to engage our ever-helpful InfluxDB community and Slack for questions. Enjoy!

InfluxData is SOC 2 Certified

Al Sargent (InfluxData) — Thu, 28 Jan 2021 04:00:51 -0700

At InfluxData, we focus on our customers’ productivity - time to awesome, as we call it. Usually this is about product capabilities - InfluxDB’s features, speed, scalability, etc.

But for some, your project will grow in size to the point where you need to purchase InfluxDB. And in some cases, you’ll need your compliance and/or security teams to sign off on the purchase. While that can be a slow process in the best of times, choosing a vendor with SOC 2 certification can help move things along faster, allowing you to go back to focusing on building apps and systems.

So, we’re pleased to announce that InfluxDB Cloud is now SOC 2? Type 1 and Type 2 certified. Our SOC 2 attestation report is available to customers, under a non-disclosure agreement given the sensitive nature of the information. To request a copy, please contact your InfluxData account manager, or sales@influxdata.com.

What is SOC 2 compliance?

Service Organization Control 2 (SOC 2) is a component of the American Institute of CPAs (AICPA)’s Service Organization Control reporting platform. SOC 2 is a technical auditing process and certification that serves as an assurance to customers that their data is being managed in a controlled and audited environment.

When a business is SOC 2 compliant, it means that an independent auditor has reviewed their processes to ensure the security, availability, confidentiality, integrity, and privacy of customer data, and validated that they meet the standards of the AICPA.

SOC 2 compliance is essential for technology-based service organizations that store customer data in the cloud. This makes it applicable to most SaaS businesses, and any business that relies on the cloud to store its customers’ information.

There are two types of SOC 2 audits:

Type I: The report describes a vendor's systems and whether their design is suitable to meet relevant trust principles.
Type II: Essentially, Type II is Type I over time; it includes a historical element that shows how controls were managed by a business over a minimum period of six months.

InfluxDB Cloud became SOC 2 Type I compliant in March 2020, and as of January 2021, became SOC 2 Type II compliant.

<figcaption> Credit: Fractional CISO</figcaption>

Why does SOC 2 compliance matter?

Customer trust

A typical company typically has between 100 to 300 SaaS applications. Let’s suppose you evaluate three vendors for any given category of SaaS app. That means 300 to nearly 1000 vendors. Looking at vendors with SOC 2 compliance provides a way to quickly weed out vendors that haven’t invested in the policies and processes to earn their customers’ trust.

Customer compliance requirements

Thousands of our customers store massive amounts of data in InfluxDB Cloud. This data includes everything from the details of their internal computing systems to the behavior of their users. Understandably, this much data in one place has sometimes required our customers, especially larger ones, to require that InfluxDB Cloud become SOC 2 compliant. Often, this comes from in-house compliance and procurement teams. Now that InfluxDB Cloud has SOC 2 certification, approval from these teams should come faster.

Foundation for additional certifications

SOC 2 requirements are similar to those of ISO 27001, which is on our roadmap of additional certifications to tackle.

<figcaption> Credit: @wocintechchat on Unsplash</figcaption>

What did our SOC 2 audit include?

Our SOC 2 audit focused on a number of criteria, one of which is availability. In the context of SOC 2, availability means, simply, whether InfluxDB Cloud is accessible as stipulated by a contract or service level agreement (SLA).

We take numerous steps to ensure continual availability of InfluxDB Cloud. For data durability, InfluxDB Cloud replicates all data in the storage tier across three availability zones in a cloud region, automatically creates backups, and verifies that replicated data is consistent and readable. You can track InfluxDB Cloud availability on our status page, and view more of our availability practices in this video:

Other aspects of our SOC 2 report included:

Monitoring: In an upcoming post, we'll describe how we use InfluxDB Cloud to monitor access to our own systems.
Auditing: We use InfluxDB Cloud to audit system access. Using a time series database, rather than a traditional SIEM, allows us to store many more events at a lower cost, since a time series database avoids the noisy data inherent in log files.
Alerting: InfluxDB Cloud supports a range of alerting options, which we use to become notified when we detect a potential breach.

<figcaption> Credit: @tool_inc on Unsplash</figcaption>

Need to become SOC 2 compliant?

Since many of you reading this are developers, you might be asking yourself, How do I make my own cloud service or SaaS application SOC 2 compliant?

Here are some steps you can take that can simplify your process of getting SOC 2 certification and improve your overall security posture:

Use Single Sign-On (SSO) using Okta, Google Cloud Identity, or similar, to sign into your applications and VPN, and integrate multi-factor authentication (MFA) into your SSO.
Secure your repos by locking your deploy branch, requiring pull requests to merge to it, and using continuous integration / continuous deployment (CI/CD) to automate your deployment.
Centralize your logging, using a SIEM and/or a time series database.
Automate provisioning, using Terraform or similar, storing configs in GitHub or similar, in a secured repository.
Secure your cloud configurations, for instance by using CloudTrail and AssumeRole on AWS.
Vendor security, by tracking all the software that you use and understanding their security postures.
Secure admin consoles, by putting your admin consoles behind a VPN accessible only with SSO and thus MFA.

SOC 2 certification requires much more than this, of course, but implementing the above should help. There’s much, much more in this blog post about the realities of SOC 2 compliance, and it’s worth a read.

Conclusion

SOC 2 compliance is but one more step on our journey to make InfluxDB Cloud easier to purchase, which is part of our overall user experience. Earlier steps in that journey include placement of InfluxDB Cloud in the AWS Marketplace, the Google Cloud marketplace, the Azure marketplace, and our usage-based pricing that puts you in control of your cloud database spend.

If you have further questions about our SOC 2 compliance, or would like to see our SOC 2 report, please contact us so we can help you out.

InfluxDB Cloud is on AWS US East

Al Sargent (InfluxData) — Mon, 23 Nov 2020 04:00:31 -0700

We’d like to let you know that InfluxDB Cloud is now on AWS US East, also called us-east-1, based in northern Virginia. This is our third AWS region, after initially launching InfluxDB Cloud in AWS Oregon and later AWS Frankfurt.

This brings InfluxDB Cloud’s effortless scaling, flexible usage-based pricing, AWS marketplace integration, and a broad range of AWS connectivity points to customers that want to manage their time series data in the eastern United States.

Where InfluxDB Cloud runs

Where does InfluxDB run now? Below is a list of all the locations as of when this post was written, but here is the current list of InfluxDB Cloud Regions.

Vendor	Region Name	Location
AWS	us-east-1	Virginia, United States
AWS	us-west-2	Oregon, United States
AWS	eu-central-1	Frankfurt, Germany
Microsoft Azure	East US	Virginia, United States
Microsoft Azure	West Europe	Amsterdam, Netherlands
Google Cloud	us-central-1	Iowa, United States

As you can see, InfluxDB Cloud is broadly distributed across North America and Europe:

<figcaption> InfluxDB Cloud is available in multiple regions across all three major cloud providers.</figcaption>

Integrate InfluxDB Cloud with AWS

InfluxDB Cloud has a broad range of integration points with AWS. Before we dive in, it’s helpful to think of our time series data platform as doing three main things:

Accumulating time series data by ingesting it from numerous locations
Analyzing that data, for visualization, anomaly detection and alerting
Acting on that data, by sending to alerting systems, custom applications or third-party tools

With this framework, there are a number of ways in which InfluxDB integrates with AWS; AWS technologies are marked with an AWS logo, InfluxDB technologies are in purple, and other technologies are in gray.

<figcaption> InfluxDB has a wide range of integration points with AWS.</figcaption>

To accumulate time series data, InfluxDB provides a number of options for data ingest, but I’d like to focus on two: Telegraf and Flux. Telegraf, our open source server agent for collecting metrics, can pull data from the following AWS services:

AWS CloudWatch: Import CloudWatch metrics with the CloudWatch Telegraf plugin from 97 different AWS services, including S3, VPC and DynamoDB. InfluxDB provides more powerful and flexible dashboarding than what's offered natively than CloudWatch. The easiest way to set this is up is with our AWS CloudWatch Monitoring Template; here's a tutorial on how to quickly import templates into your InfluxDB Cloud instance.
AWS EC2, AWS Elastic Container Service (ECS) and AWS Fargate: Monitor performance metrics inside your EC2 virtual machines, ECS containers, and Fargate containers, using the Telegraf AWS ECS plugin or Telegraf's generalized plugins for system monitoring and Docker monitoring. The Telegraf ECS plugin is Fargate compatible, and uses the Amazon ECS v2 metadata and stats API endpoints to gather stats on running containers in a task.
AWS Elastic Kubernetes Services (EKS): Inventory the components of your Kubernetes clusters running on AWS, and track the performance of each pod and container, using the Telegraf Kubernetes and Kubernetes Inventory plugins.
AWS Kinesis: Ingest data from AWS Kinesis for analysis, visualization, and alerting, using the Telegraf Kinesis consumer plugin. You can also use Telegraf to publish data to Kinesis as well.
AWS synthetic monitoring: Track the response times of all your AWS API HTTP endpoints.
AWS Programmatic Billing: Import your Programmatic Billing data into InfluxDB to better understand AWS spend.
IoT data collection & analysis: If you're pulling data from IoT devices and sensors into AWS, use the Telegraf OPC-UA plugin, Modbus plugin, MQTT plugin, AMQP plugin, JSON plugin, HTTP listener plugin, Kinesis consumer, DDS plugin, and other IoT plugins.

Using Flux, our data scripting language purpose-built for time series data, you can pull data directly from SQL data stores, such as AWS RDS Postgres, RDS MySQL, RDS MariaDB, RDS SQL Server, and AWS Athena. The specific Flux function to do this is called sql.from(), and is part of the Flux SQL package. This function lets you enrich time series data to provide additional context. Here are a few examples:

For IoT users, Flux lets you combine time series data from sensors with information from relational tables like manufacturer, model number, device age or mileage.
For real user monitoring (RUM), you can use Flux to look up customer IDs and connect customer name, pricing plan, and firmographics like company name and location.
In IT infrastructure monitoring, Flux can resolve server IDs to software version and configuration parameters.

In fact, Flux lets you enrich from much more than just RDS — from just about any relational database:

<figcaption> InfluxDB and Flux let you enrich time series data using relational databases</figcaption>

In addition to Telegraf and Flux, InfluxDB Cloud can also ingest data from Vector, Apache NiFi, OpenHAB, JMeter, Node-RED, and Jaeger. All in all, a broad range of options for data ingest, for making observability reality.

The analyze piece of the equation is handled by InfluxDB itself. Our main vehicle for this is Flux. Beyond data enrichment, Flux provides a broad range of functions for statistical analysis and forecasting, such as:

Calculating percentiles to detect SLA compliance failures
Windowing and aggregating data to pick out insights from noisy data sets
Forecasting with Holt-Winters to predict outages and capacity issues
Geographically tracking monitoring metrics to better determine which regions are experiencing problems
Sophisticated anomaly detection, using techniques like Median Absolute Deviation (MAD) and Balanced Iterative Reducing & Clustering (BIRCH), to help you find and fix problems faster and reduce your mean time to resolution (MTTR)
Machine learning using Naive Bayes Classification

To act on time series data, you can use Flux running in InfluxDB Tasks to send alerts to a broad range of endpoints: Amazon Simple Email Service (SES) and other email services such as Twilio SendGrid, Mailjet, and Mailgun; IT alerting systems such as PagerDuty, Atlassian Opsgenie, Sensu, and soon ServiceNow and BigPanda; messaging services such as Microsoft Teams, Slack, Discord, Telegram; and general-purpose solutions such as Pushbullet, MQTT, and webhooks.

<figcaption> InfluxDB and Flux let you send alerts to a broad range of endpoints.</figcaption>

You can also use Flux to pull data into your code running on AWS Lambda, EC2, or other services, using our client libraries for JavaScript/Node.js, Python, Go, Arduino, Java, Scala, C#, Ruby, PHP, and Kotlin. These client libraries were previously provided by our open source community; they are now fully maintained and supported by InfluxData.

<figcaption> InfluxDB has ten fully maintained client libraries</figcaption>

How to access InfluxDB Cloud on AWS Virginia

Here’s a step-by-step guide on how to get InfluxDB Cloud on AWS us-east-1:

First, visit cloud2.influxdata.com and sign up for a free InfluxDB Cloud account.

Once you hit the Create Account button, check your email for a message that looks like this:

Click Verify Your Email, and you’ll see the following page asking you where to store your data.

Once you select AWS, choose US East (N. Virginia), and you’ll use that instance of InfluxDB Cloud.

There are a few more steps that are self-explanatory, but one thing I want to point out is the InfluxDB Cloud URL for AWS East US:

https://us-east-1-1.aws.cloud2.influxdata.com

This is the endpoint of the InfluxDB Cloud API that you’ll be using when you interact with the InfluxDB API, InfluxDB client libraries, Influx CLI, and Telegraf. Here is our documentation page with a complete list of InfluxDB Cloud URLs if you are running on a different region.

Customized code snippets for AWS US East

Speaking of our documentation, here’s a handy trick to customize all our code samples to use the AWS US East URL. In the InfluxDB Cloud documentation, click on the gear icon in the upper-right corner:

Then specify that you’re running in AWS US East (Virginia):

And now code samples in our documentation, such as on this page, will have the correct InfluxDB Cloud URL already inserted so you have one less step to worry about. See the first line of the curl command below:

Expanding our reach

Even though InfluxDB Cloud is available on six regions, we’re still looking to expand our footprint to best serve you. However, we can’t simply deploy across the combined ~75 regions offered by Amazon Web Services, Microsoft Azure, and Google Cloud.

The reason: InfluxDB Cloud is available as an instant-on, always available cloud service; you don’t spin up and maintain a single-tenant virtual machine. This means that for each region, we’re always running a fleet of InfluxDB instances, ready to accept your incoming data, run your queries and tasks, display your dashboards, and fire off your alerts. And THAT means each region incurs a significant monthly cloud bill that needs to be offset by revenue — just like any business.

So, we need YOUR help in telling us where to expand next by filling in this form. It’s literally five fields, anonymous, and will take you all of 30 seconds to complete:

AWS US East is just part of the story

Expanding into AWS US East is just one of the many recent improvements to InfluxDB Cloud. In this video from the InfluxDays North America 2020 Virtual Experience, our VP of Engineering, Ryan Betts, goes into more detail about all we’ve done recently and some of our roadmap:

?

Conclusion

Here’s more about our partnership with AWS. If you’d like to check out InfluxDB Cloud for yourself, sign up for a free InfluxDB Cloud account. Once you do, feel free to ask any questions in our ever-helpful InfluxDB community and community Slack channel. Enjoy!

3 ways InfluxDB OSS Complements InfluxDB Cloud

Al Sargent (InfluxData) — Mon, 16 Nov 2020 04:00:50 -0700

As you may know, InfluxDB is available in multiple forms, including InfluxDB Open Source and InfluxDB Cloud. Customers will sometimes ask us, “If I’m using InfluxDB Cloud, is there any reason I should also consider using InfluxDB open source?”

It’s a fair question. InfluxDB Cloud has several capabilities not in InfluxDB OSS: InfluxDB Cloud is serverless so there are no servers to manage, it can automatically accommodate traffic surges, and we’re continually updating it to automatically deliver you the latest features.

We continually hear from customers that they prefer InfluxDB Cloud because it frees them from tedious database administration tasks so they can focus on their code. Even so, there are a few use cases where an InfluxDB Cloud customer can use InfluxDB Open Source to help manage their time series data. Let’s explore.

InfluxDB development environment

You can install InfluxDB OSS on your local machine to write code anywhere without having to worry about a network connection. During these days of shelter-in-place, you might have network issues due to Zoom-induced bandwidth saturation. Post-pandemic, when we can return to cafes, trains, and airplanes, it’ll be handy to have a database that easily runs on a laptop.

The other day, I drove my kid to go surfing in a remote spot on the California coast without cell coverage. While I watched him from the car, I was able to easily import sample data from a local CSV file into my local InfluxDB instance, then write some Flux code to query it. Coding from the beach – one of the silver linings of shifting to a remote work environment!

InfluxDB OSS lets you work on your time series data even where there's no network

Because the InfluxDB UI, CLI, API, and client libraries between InfluxDB OSS and Cloud are practically identical, you can always shift resources like Flux scripts, InfluxDB Templates, InfluxDB Dashboards, Tasks, Alerts, and Grafana dashboards from OSS to Cloud (and vice-versa) with no modifications, facilitating a smooth DevOps pipeline.

The differences between InfluxDB Open Source and Cloud are rare, and highlighted in our documentation’s left-hand navigation.

Differences between InfluxDB OSS and Cloud are noted in our documentation.

We also make it easy to switch your Telegraf agents, client code using the InfluxDB client libraries, and InfluxDB CLI from pointing at a local InfluxDB open source instance, to an InfluxDB Cloud instance. Just change your InfluxDB Open Source URL from localhost:8086 to the appropriate InfluxDB Cloud URL, update your org, token, and bucket, and you’re good to go.

When using the CLI, you can create connection configurations, and easily switch between configurations as you move from Open Source to Cloud and back. And, we just added a -c flag to set your profile on the fly.

InfluxDB edge processing

A common pattern in IoT is where people set up Telegraf to dual-write to both a local InfluxDB OSS instance on an edge device, as well as a central InfluxDB Cloud instance. This edge device might be as small as an Arduino board or as large as a windmill. Because InfluxDB OSS is a single lightweight binary, it can easily be deployed on devices with minimal compute resources.

@harrisonbroadbent via Unsplash

In these situations, the edge instance of InfluxDB OSS often holds full-resolution telemetry for just the edge device, while the central InfluxDB Cloud instance holds telemetry across all devices. The degree to which InfluxDB Cloud’s telemetry is downsampled is often a function of business requirements, as well as connection reliability and bandwidth from the edge devices.

With this approach, folks responsible for managing the local environment – which would be a ferry, oil rig, farmland, etc. – have full telemetry to troubleshoot issues, without having to worry about connectivity to a central cloud instance. And folks in headquarters can analyze trends and issues across an entire IoT fleet using InfluxDB Cloud.

For example, one of our customers runs offshore oil rigs. All of the various time series metrics that they are gathering are stored in a local InfluxDB OSS instance and the operators of the rig have access to the information through dashboards. This information provides critical near real time visibility into what is occurring on the rig itself.

For some of the equipment being monitored, there is a desire to have access to this data – at a lower resolution than is being captured and stored locally – at the corporate headquarters. The corporate HQ folks can analyze the data from all of the rigs and look at the similarities and differences between the operational characteristics of the equipment in the aggregate. This allows for analyses like predictive maintenance to be performed and sent back to the rig operators.

Because InfluxDB OSS and Cloud share the same UI, API, and CLI, and both use Flux, teammates in the field and central office can easily share results and findings with each other.

To implement this pattern, you’ll need to dual-write to both InfluxDB Cloud and InfluxDB OSS to have a local backup. To do this, you’d put text similar to what’s below into your Telegraf configuration file (typically called telegraf.conf):

# Write to InfluxDB open source
[[outputs.influxdb_v2]] # For InfluxDB 2.0
 urls = ["http://127.0.0.1:8086"]
token = "your-token"
organization = "your-organization-name"
bucket = "your-bucket-name"

# Write to InfluxDB Cloud
[[outputs.influxdb_v2]]
# Write to AWS Oregon instance.
# Other cloud URLs: https://docs.influxdata.com/influxdb/cloud/reference/regions/
urls = ["https://us-west-2-1.aws.cloud2.influxdata.com"]
token = "your-token"
organization = "email-you-use-to-sign-into-influxdb-cloud"
bucket = "your-bucket-name"

With this configuration, every write goes to both your on-prem InfluxDB OSS instance and InfluxDB Cloud instance, at the same time. Because of this, data in both instances is kept in sync, up-to-the-second; there’s no batch process to periodically run backups.

Now that InfluxDB 0SS 2.0 is GA, look for us to make further engineering investments in this area of edge processing to simplify and support edge-to-cloud connection across these editions.

Air-gapped monitoring

But what about those cases where you can’t dual-write, because there’s no Internet connection. These include servers and equipment that are disconnected from the Internet (or “air-gapped”) for security purposes and thus can’t send monitoring telemetry to InfluxDB Cloud. For instance, power plants are often disconnected to prevent a wide range of threat vectors.

If any of your servers or equipment are air-gapped, you can install InfluxDB OSS on them and use it to collect monitoring data in order to provide local observability and control. However, if you need higher levels of scalability or reliability than a single node of InfluxDB OSS can deliver, you should consider InfluxDB Enterprise. InfluxDB Enterprise provides horizontal scale-out and a high availability configuration.

So how does this usage of InfluxDB OSS complement InfluxDB Cloud? Because Cloud and OSS share the same API, CLI, web user interface, and client libraries, you can easily share assets like Dashboards, Tasks, and Alerts between your air-gapped InfluxDB OSS instances, and your InfluxDB Cloud instances. For example, if your company has standardized on Kubernetes, you build one set of k8s monitoring dashboards used at both your air-gapped and Internet-connected locations.

InfluxDB Cloud compared to InfluxDB OSS

Here’s a detailed comparison of how InfluxDB Cloud compares to InfluxDB open source.

	InfluxDB Cloud	InfluxDB Open Source
Free Plan	✓	✓
Data Explorer	✓	✓
Dashboards	✓	✓
Tasks	✓	✓
Alerts	✓	✓
Common client libraries	✓	✓
Unified REST API	✓	✓
InfluxQL queries	✓	✓
Flux queries	✓	✓
Multiple users	✓	✓
Local developer instance	X	✓
Edge deployment	X	✓
Real-time on-prem backups	X	✓
Air-gapped monitoring	X	✓
Prometheus scrapers	X	✓
Serverless (no servers to manage)	✓	X
Automatically accommodate traffic surges	✓	X
High availability	✓	X
Built-in disaster recovery	✓	X
Fault tolerance	✓	X
Automated upgrades	✓	X
Integrated Google social sign-on (Oauth)	✓	X

Conclusion

At InfluxDB, we provide you with multiple options for deploying our database, including lightweight, self-managed InfluxDB Open Source that you can deploy anywhere, and fully-managed, serverless InfluxDB Cloud that frees you from database infrastructure concerns so you have more time to focus on your code.

We’re also grateful to our open source community of nearly 400 contributors to InfluxDB, who bring innovation to the shared InfluxDB Open Source and Cloud codebase. With them, we’re able to provide all of you with a range of options to suit your needs.

To get started with the patterns above, download InfluxDB Open Source and sign up for a free InfluxDB Cloud account. Then check out our getting started guide and feel free to engage our ever-helpful InfluxDB community, Slack, and GitHub repository for questions. Enjoy!

Introducing the InfluxDB Template UI: Monitoring Made Simple

Al Sargent (InfluxData) — Tue, 06 Oct 2020 06:00:22 -0700

At InfluxData, we’re obsessed with time to awesome — how quickly can you start working productively with time series data? What can we do to make things better?

InfluxDB Templates are a great example of this mindset. Back in April, we announced Templates as a way to package up everything you need to monitor a particular technology — Telegraf configurations and InfluxDB Dashboards, Tasks, Alerts, and related artifacts — into a single configuration file. This had two benefits:

First, it allowed those of you who are experts with monitoring a particular technology to easily share your expertise — with your teammates, and with the world.

Second, it allowed those of you ramping on a new technology to set up your own monitoring in just minutes, so you don’t need to recreate the wheel.

InfluxDB Templates were a big step forward because while InfluxDB has always been more flexible than dedicated monitoring platforms, those platforms come with prebuilt dashboards for various technologies.

With InfluxDB Templates, we’re closing that gap. Since we announced Templates, we now have over 40 Templates for dozens of popular application technologies — from AWS to Zookeeper — and our community is continually adding more. Each one of these templates shaves hours, if not days, of configuration time spent setting up everything from data acquisition to alerting. That’s a massive accelerator of time to awesome for many of the technologies in your stack.

<figcaption> Image credit: George Hodan via Needpix.com</figcaption>

Even so, we knew we could do more.

Up until now, to import Templates into your InfluxDB Cloud instance, you needed to use the influx CLI command. But the whole point of any cloud product is to avoid the hassle of installing software.

So we built Template import straight into the InfluxDB Cloud UI.

No software to download. No command line syntax to memorize.

Just copy a link to the Template you want, paste it into the page in your InfluxDB Cloud UI, and you’re done.

Let me show you how.

How to use the InfluxDB Template UI

Let’s start at the very beginning. Go to cloud2.influxdata.com and sign in.

Next, click on Settings on the left side of your browser:

Then click on Templates:

You can probably guess what to do next, given the big blue button: Browse Community Templates.

You’ll get dropped onto our GitHub page listing all our templates. Let’s click on the Network Interface Monitor template:

This is a nice template to start with, since it can use actual network monitoring data from your Mac or Linux machine.

Every InfluxDB Template is defined by a YAML, JSON, or Jsonnet file. This particular network monitoring template is defined by a YAML file, network_interface_performance.yml. Click the link:

Copy the link from your browser’s address bar:

https://github.com/influxdata/community-templates/blob/master/network_interface_performance/network_interface_performance.yml

… then switch back to InfluxDB Cloud, and paste in the link:

You should see a message stating that you’re good to go.

(Digging into this, I learned that both the URL to the HTML page and the URL to the raw file both work. That’s our engineering team sweating the details — so you don’t have to. )

Click on Lookup Template, and you’ll see all the various resources associated with the template:

Now hit that big green button and install your template! ????

Once you do, you’ll see the template in your installed list:

Set up Telegraf to monitor network performance

At this point, our Dashboard, Check, Bucket, Variables, and Labels are all installed as part of our template. But we need to get Telegraf to start sending data to our InfluxDB Cloud instance.

If you’re not familiar, Telegraf is our open-source, plugin-driven server agent for collecting & reporting metrics. Telegraf has plugins to source a variety of metrics directly from the system it’s running on, pull metrics from third party APIs, or even listen for metrics via StatsD or Kafka consumer services. It also has output plugins to send metrics to a variety of other datastores, services, and message queues, including InfluxDB.

To start the process of setting up Telegraf, click on Data on the left side of your browser window:

Then click Telegraf:

We’ll see our newly installed Network Interface Monitoring configuration:

Click on Setup Instructions. You’ll see the following:

Here we need to perform some non-cloudy tasks, installing software — Telegraf — on your machine. This could be your laptop, an EC2 instance, or whatever server for which you want to track network traffic.

The reason we need to do this local software install is that our machine is (hopefully!) behind a firewall and thus not reachable by InfluxDB Cloud. By installing Telegraf on a local machine, it can send data through our firewall and out to InfluxDB Cloud.

Let’s walk through each of these steps.

First, get Telegraf. If you’re monitoring a Mac, get Homebrew if you don’t have it (trust me, it’s handy), then run brew install telegraf. If you’re monitoring something else, here are instructions for how to install Telegraf on other operating systems.

Next, get an InfluxDB API token by clicking Generate New Token:

This token is sort of like a password that Telegraf uses when sending network performance data (or other telemetry) to InfluxDB.

If you’re on Mac or Linux, press Copy to Clipboard:

… and a token will get generated for you. Then open a terminal window (here’s how on Mac) and paste the command you just copied:

export INFLUX_TOKEN=MrYzeGC6C1m9ZGY852ZZgaKai8t1iQBRk9BlAh8oRkUM2f1RhNgM_UJgsFeTzrRYhU7k1Ynl472DqRhjgFWYM-A-Wg==

(Of course, replace the token value with whatever’s associated with your cloud account.)

You also need to specify two more environment variables, INFLUX_URL and INFLUX_ORG.

The INFLUX_URL environment variable specifies the URL of your InfluxDB Cloud instance. This is necessary since InfluxDB Cloud runs multiple cloud providers — AWS, Google, and Microsoft Azure — as well as different regions on each. You can find the appropriate URL for your instance on the InfluxDB Cloud URLs page.

For example, if you’re running on our new Azure East US instance, enter this into your terminal window:

export INFLUX_URL="https://eastus-1.azure.cloud2.influxdata.com"

Note that I put quotes around the cloud URL, whereas I didn’t with the token.

Next, set the INFLUX_ORG environment variable. If you created your own InfluxDB Cloud instance, this will be the email you use to sign in. On the other hand, if you got an invite to InfluxDB Cloud, here’s how to find your org name. You’ll type something like:

export INFLUX_ORG="asargent+09.29.2020@influxdata.com"

You can confirm whether it’s set by typing the env command. Pipe it through sort to quickly pick out the variable from a long list:

env | sort

You should see something like this:

If you’re on Windows, you’ll need to export these variables using a different command syntax. Here’s how to set and view environment variables in PowerShell and how to set and view them in the Windows command prompt.

Run Telegraf to monitor network performance

Anyways, once your InfluxDB environment variables are set, switch back to the InfluxDB Cloud UI, and copy your command to start Telegraf:

You should see something like this:

Parsing this output, you see the following has happened:

Telegraf version 1.15.3 started up.
One Telegraf input plugin loaded: the Net Input Plugin.
One Telegraf output plugin loaded, the InfluxDB v2 plugin (this is what you use for InfluxDB Cloud and InfluxDB OSS v2); for InfluxDB OSS v1 and InfluxDB Enterprise, use the InfluxDB output plugin.
Telegraf is collecting metrics every ten seconds.

View your Template dashboards

Now that Telegraf is sending data to InfluxDB Cloud, let’s see our dashboards in action!

Switch back to InfluxDB Cloud, and click on Boards on the left side. You’ll see the Network Interface Performance dashboard that was imported with our template.

Click Network Interface Performance and you’ll see charts of your network performance:

Note that some of your network interfaces might be inactive and show no data. My laptop defaulted to network interface awd10, so it was a simple matter of choosing en0:

Send InfluxDB alerts to Slack

Dashboards are great. But what if we get network errors — how will we know?

For that, let’s set up alerting to Slack. To do this, click on Alerts:

You’ll see that our template included an alert checking for Network Errors:

If you click Network errors, you’ll see that this check runs every 10 minutes, and displays a warning message when there have been 10 network errors in that time, and a critical message when there’ve been 20 errors.

But, we need to send these notifications somewhere. Thankfully, the InfluxDB Community Slack has a sandbox you can play with. If you haven’t already done so, get Slack, then join the InfluxDB Community on Slack.

In Slack, head to the channel #notifications-testing, and you’ll see a webhook URL that you should copy:

As I write this, the webhook URL is the following:

https://hooks.slack.com/services/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

Flip back to InfluxDB Cloud, then click Notification Endpoints, then Create:

In the next screen, name your endpoint and fill in the URL above:

Click Create Notification Endpoint and you’ll see your newly created endpoint:

Now switch over to Notification Rules and press Create:

Name your notification rule. To be polite in the sandbox, include your personal name in the notification rule name, like I did below:

Press Create Notification Rule, and you’ll see list of notification rules:

When you have more than 20 network errors in a 10-minute period, a message will get posted to InfluxDB Community Slack’s #notifications-testing channel.

Now — this is important — when you’re done testing, TURN OFF your notification rule by clicking the toggle button on the left.

Share your expertise with InfluxDB Templates

We’re grateful that we’ve gotten 40 templates in just seven months, including:

Apache JMeter
Apache + Postgres
Apex Legends
AWS Cloudwatch
Counter Strike Global Offensive / CSGO
Currency Exchange Rates
Docker
Downsampling
Endpoint Security State
Enviro+
Fortnite
GitHub
Google Cloud
HAProxy
InfluxDB 1.x
InfluxDB OSS 2.0
JBoss Wildfly
Jenkins
Kafka
Kubernetes
Linux
Microsoft SQL Server
MinIO
Modbus
MongoDB
MySQL / MariaDB
Network Interface
Nginx + MySQL
Postgres
Redis
Sensu Go
sFlow
SNMP
Speedtest (local Internet connection speed)
Telegraf (use Telegraf to monitor itself)
Tomcat
VMware vSphere
Windows
X509
Zookeeper

But we’d love to get even more templates. So, for anyone who builds an InfluxDB Template that gets committed to our repository, we’re offering InfluxDB Inventor status and a cool, limited-edition Inventor t-shirt — even our CEO can’t get one! (Sorry, Evan.)

So, if you’ve created a Telegraf configuration, dashboard, tasks and/or alerts for any technology not listed above, create your own InfluxDB Template and then follow these steps to submit your template.

Conclusion

As mentioned, we continually focus on time to awesome. That’s because we’re inspired by others that have followed a similar path of making technology accessible:

Friendly reminder to startups: customers don't care about how brilliant your technology is. They only care about how simple your solution is. Focus on the latter at all costs.
— Aaron Levie (@levie) September 29, 2020

In the time series database market, plenty of vendors focus on speeds and feeds — but little else. InfluxDB handles massive amounts of time series data — for instance, millions of metrics per second at Wayfair — but we realize that the most brilliant time series technology is nothing if you can’t use it. That’s why we’ve invested in making time series data easy to use — and will continue to do so. Our new templates UI is just one step in that process.

If you’d like to check out the InfluxDB Template UI for yourself, sign up for a free InfluxDB Cloud account, and feel free to ask any questions in our ever-helpful InfluxDB community and community Slack channel. Enjoy!

InfluxDB Cloud Now on Azure East US

Al Sargent (InfluxData) — Wed, 23 Sep 2020 06:00:46 -0700

We’re pleased to announce that InfluxDB Cloud is now on Azure East US, based in Virginia. With this expansion, our time series data platform is now on Microsoft Azure on two continents — Europe and North America — and runs in five locations across all three major cloud providers.

This brings InfluxDB Cloud’s effortless scaling, flexible usage-based pricing, and Azure time series data integration to customers that want time series data storage in the United States.

Where InfluxDB Cloud runs

Where does InfluxDB run? Here’s a list of all the locations as of when this post was written. Here is the current list of InfluxDB Cloud Regions.

Vendor	Region Name	Location
Microsoft Azure	East US	Virginia, United States
Microsoft Azure	West Europe	Amsterdam, Netherlands
AWS	us-west-2	Oregon, United States
AWS	eu-central-1	Frankfurt, Germany
Google Cloud	us-central1	Iowa, United States

As you can see, InfluxDB Cloud is now broadly distributed across North America and Europe:

<figcaption> InfluxDB Cloud is now on five regions on all three major cloud providers.</figcaption>

How to access InfluxDB Cloud on Azure East US

Here’s a step-by-step guide on how to get InfluxDB Cloud on Azure East US:

First, visit cloud2.influxdata.com and sign up for a free InfluxDB Cloud account. You can do 1-click signup with your Google account, or enter your details.

Once you hit the Create Account button, check your email for a message that looks like this:

Click Verify Your Email, and you’ll see the following page asking you where to store your data.

Once you select Microsoft Azure, choose Virginia, and you’ll use that instance of InfluxDB Cloud.

There are a few more steps that are self-explanatory, but one thing I want to point out is the InfluxDB Cloud URL for Azure East US:

https://eastus-1.azure.cloud2.influxdata.com

This is the endpoint of the InfluxDB Cloud API that you’ll be using when you interact with the InfluxDB API, InfluxDB client libraries, influx CLI, and Telegraf. Here is our documentation page with a complete list of InfluxDB Cloud URLs if you are running on a different region.

Customized code snippets for Azure US East

Speaking of our documentation, here’s a handy trick to customize all our code samples to use the Azure East US URL. In the InfluxDB Cloud documentation, click on the gear icon in the upper right corner:

Then specify that you’re running in Azure East US:

And now code samples in our documentation, such as on this page, will have the correct InfluxDB Cloud URL already inserted so you have one less step to have to worry about. See the first line of the curl command below:

Integrate InfluxDB Cloud with Microsoft Azure

InfluxDB Cloud has a broad range of integration points with Azure, and Microsoft products in general. First, let’s take a look at how InfluxDB Cloud ingests and processes time series data.

InfluxDB provides a number of options for data ingest, but I’d like to focus on two: Telegraf and Flux. Telegraf, our open source server agent for collecting metrics, can pull data from the following Microsoft technologies:

Azure IoT Hub, using the Telegraf plugins for Azure Event Hub, AMQP, MQTT, and HTTPS, to collect IoT device telemetry data.
Windows servers, using the Telegraf Windows Performance Counter plugin and Windows Services plugin, to ensure that your Windows servers are performant.
GitHub, using the Telegraf plugin and InfluxDB Template for GitHub, to track stats for each of your repositories, such as the number of forks, open issues, etc.
Azure Storage Queues, using the eponymous Telegraf plugin, to track queue sizes.
SQL Server, using the Telegraf plugin and InfluxDB Template for SQL Server, to track the health and performance of your databases.

Telegraf has over 200 plugins, covering technologies that aren’t Microsoft-specific but which you might find on Azure, such as Docker and Kubernetes. You can also pull in data from over 700 FluentD plugins, thanks to the Telegraf FluentD plugin.

Also, our Flux data scripting and query language has a SQL library that lets you enrich your time series telemetry with metadata from relational databases, including SQL Server. This metadata can include IoT asset information — for example, the year, make, and model of a piece of IoT equipment. The specific Flux function you use to do this is called sql.from().

In addition to Telegraf and Flux, InfluxDB Cloud can also ingest data from Vector, Apache NiFi, OpenHAB, and JMeter. All in all, a broad range of options.

Analysis of time series data is handled by InfluxDB itself. As mentioned, InfluxDB Cloud runs on Azure infrastructure running in the West EU region in Amsterdam, and by early September, East US in Virginia.

Additionally, when you analyze your data using Flux from within the InfluxDB UI, you do so in an embedded Monaco editor that delivers an experience that is similar to Microsoft Visual Studio Code with capabilities like autocompletion, syntax validation and highlighting, variable references and renaming, and more. There’s also a Flux plugin for VS Code, for developers building applications on top of InfluxDB. If you have features you’d like us to add, file an issue on the Flux LSP repo.

<figcaption> Autocompletion in InfluxDB Cloud’s embedded Monaco editor</figcaption>

These editors are two of many ways we sweat the details to ensure that you have a great experience working with InfluxDB, and is part of our time to awesome mantra.

Flux provides a broad range of functions for statistical analysis and forecasting, such as:

Calculating percentiles to detect SLA compliance failures
Windowing and aggregating data to pick out insights from noisy data sets
Enriching monitoring data with business data in SQL databases — like account name, type or size — to detect anomalies by business measures
Forecasting with Holt-Winters to predict outages and capacity issues
Geographically tracking monitoring metrics to better determine which regions are experiencing problems
Sophisticated anomaly detection, using techniques like Median Absolute Deviation (MAD) and Balanced Iterative Reducing & Clustering (BIRCH), to help you find and fix problems faster and reduce your mean time to resolution (MTTR)

To act on time series data, you can:

Send alerts to Microsoft Teams, PagerDuty, Slack, Discord, Telegram, Pushbullet, and via webhooks.
Pull data into your custom applications using our client libraries for Javascript/Node.js, C#, and more.

Up next: Azure Marketplace billing, more regions

In the coming months, you’ll see InfluxDB Cloud in the Azure Marketplace, allowing you to put your InfluxDB Cloud charges straight onto your Azure bill. This lets you consolidate all your Azure expenses, providing simpler administration, easier expense tracking, and streamlined purchasing. Plus, it lets you take advantage of any committed use discounts that you have arranged with Microsoft. We’ve already done this for InfluxDB Cloud and AWS Marketplace and Google Cloud Marketplace, and we’re looking forward to bringing it to our customers on Azure.

Also, you’ll see InfluxDB Cloud on additional regions, based on input that customers like you have provided us. If you’d like to provide us with input on where we launch future InfluxDB Cloud instances, please fill in this short form. Your vote will help us best serve our entire customer base.

<figcaption> Credit: Headway via Unsplash</figcaption>

Conclusion

If you’d like to check out InfluxDB Cloud for yourself, sign up for a free InfluxDB Cloud account, and feel free to ask any questions in our ever-helpful InfluxDB community and community Slack channel. Enjoy!

Putting You in Control of Your InfluxDB Cloud Spend

Al Sargent (InfluxData) — Tue, 01 Sep 2020 06:00:05 -0700

We recently changed the pricing of InfluxDB Cloud to let you control your cloud database spend so you spend only as much as you need to run your software and systems — with no wasted budget.

If you just want a summary, check the InfluxDB Cloud pricing page. But if you’d like to nerd out on the changes we made, why we made them, and how to estimate your monthly spend on InfluxDB, then buckle up for a deep dive.

Our search for the right pricing metric

Before we talk about the changes we made, let’s look at how we arrived at our pricing for InfluxDB Cloud.

Our first principle is that we want to price fairly, based on how much value we deliver. That begs the question: what’s a good proxy for value delivered?

A second principle is that we want to put you in control of your spend. You’ll see below how we let you tune your cloud database usage through increased transparency.

Let’s address the first question: because InfluxDB is used across a wide range of use cases — IoT, software monitoring, scientific research, user experience, and more — we have to find a unit of value that’s common across all of them.

We can’t charge by “endpoints monitored” because that doesn’t work in the context of IoT: an oil rig or windmill will emit orders of magnitude more telemetry than a networked lightbulb.

Nor did it make sense to price based on database host size, since InfluxDB Cloud has a cloud-native architecture that defies any simplistic machine-based pricing. Here’s what we mean.

Some hosted databases run on one or two nodes, then slap “Cloud” on the product name even though they’re little more than their open-source code running on an EC2 instance.

That’s nothing like InfluxDB Cloud; it runs across hundreds of containers, each working to process data in a horizontally-scaled manner. Those containers are dynamically orchestrated with Kubernetes and Istio, continually starting up and shutting down as needed. They’re fronted by Kafka to stream in massive volumes of data. All these resources are multitenant, shared across multiple customer accounts. Because of this, InfluxDB Cloud is a cloud service that’s too complex to price by “machine size”.

<figcaption> InfluxDB Cloud: cloud-native architecture that runs on many machines — not just one. Source.</figcaption>

How InfluxDB Cloud is priced

When you look across our hundreds of customers across dozens of industries, the one thing they have in common is data: how much time series data runs through InfluxDB, how often they access that data via queries, and how long they choose to retain it.

For this reason, InfluxDB Cloud pricing is based on data, which allows us to use a value metric that’s applicable to all our customers: as you store more data in InfluxDB Cloud, the more value you should be getting from it. We want to incent you to store as much time series data as needed to run your business — no more, no less.

This thinking manifests in four vectors of usage, which cover any imaginable use case for InfluxDB and ensure fair pricing for everyone:

InfluxDB Cloud Pricing Vector	Definition
Data In	How much data you're bringing into InfluxDB
Queries	How much you're querying in InfluxDB
Storage	How much data you're storing
Data Out	How much data you're pulling from InfluxDB

Usage-based pricing vs per-server pricing

Pricing InfluxDB Cloud by usage lets you avoid the hassles of per-server pricing that some databases impose.

What’s per-server pricing? Per-server pricing is where you’re charged by the size of the machine your database runs on. Machines with small amounts of memory, CPU, and disk are cheaper, and those with more are more expensive. There are a few problems with this approach:

Workload mapping: It's hard to map a particular workload profile to a specific machine size. For example, if you're monitoring 50,000 IoT devices or 5,000 containers, it's not immediately clear what size machine you need.
Estimation penalties: If you overestimate your workloads, you waste precious budget. And if you underestimate, you set yourself up for a painful migration to a larger machine — imagine being a retailer and having to do this in the middle of the holiday shopping season!
No peak provisioning: Per-server pricing naively assumes that businesses don't have surge seasons: Cyber Monday for retail, tax season for investment and payroll websites, and so on. A machine that works fine for most of the year might be too small for your surge period.

To sum up, usage-based avoids the risk of getting locked into the wrong sized machine. Our cloud-native architecture seamlessly scales as your workload demands, so you pay only for what you actually use. Let’s explore how we do that.

Putting you in control of your spend

In contrast to inflexible per-server pricing, the above four pricing vectors provide you with multiple ways to fine-tune your budget spend to your usage and business cycles.

<figcaption> Credit: Alexey Ruban via Unsplash</figcaption>

For example, suppose you’re a retailer monitoring your applications during the Cyber 5 shopping days. During this time, you might collect server performance data and run alert checks more frequently than usual — say, every 5 seconds — since downtime can easily stretch into thousands of dollars per minute. This increases your Data In and Query spend — a worthwhile investment to ensure that revenue continues uninterrupted.

When the holiday season is over and customer visits decline, you throttle back your data collection and alert check frequency — perhaps once a minute. Your Data In and Query spend declines accordingly, in real time. You reduce cloud database spend to match reduced application revenue. (How great would it be if all your costs could be adjusted this easily?!)

You can easily track your consumption of all four of these pricing vectors on the InfluxDB Cloud usage page, which gives you full visibility into what’s driving your cloud database spend across all four vectors:

Usage status: Overall usage for the current billing period, to see which vectors are driving your costs.
Usage trends: Trend graphs for all pricing vectors, to see if usage is accelerating or slowing.
Usage overages: Any recent rate limit overages related to Data In or Queries.

You can see these features in the screenshot below.

<figcaption> InfluxDB Cloud lets you determine exactly what is driving your spend</figcaption>

What's the price of InfluxDB Cloud?

Here’s a summary of InfluxDB Cloud monthly pricing before and after September 1:

InfluxDB Cloud Pricing Vector	Price before 9/1/20	Price starting 9/1/20
Data In	$0.0015/MB	$0.002/MB
Queries	$0.0015 per query second	$0.01 per 100 query executions
Storage	$0.0015/GB-hour	$0.002/GB-hour
Data Out	No charge	$0.09/GB

The main changes are in the second and fourth rows — Queries and Data Out. We’ll walk through both changes. But first, let’s look at a simple example of how InfluxDB Cloud charges are calculated.

How to calculate InfluxDB Cloud charges

Now let’s get to the meat of the matter: How much does InfluxDB Cloud cost?

Let’s answer by way of example. Suppose your time series data profile looks like this — we’ll use round numbers for simple math:

Ingest one GB per month
Store two GB of data before it's evicted
100,000 queries run per month across all InfluxDB Dashboards, Tasks, and Alerts
One GB of query response data per month

In a 30-day month (with 720 hours), your InfluxDB Cloud charges would be $14.97, calculated as follows:

InfluxDB Cloud Pricing Vector	Price	Usage	Subtotal
Data In	$0.002/MB	1000 MB	$2.00
Queries	$0.01/100 queries	100,000 queries	$10.00
Storage	$0.002/GB-hour	2 GB, stored all month	$2.88
Data Out	$0.09/GB	1 GB	$0.09
Total			$14.97

With that out of the way, let’s dig into the two changes to our InfluxDB Cloud pricing, Query Count and Data Out.

Understanding Query Count pricing

InfluxDB Cloud is no longer priced by query duration, i.e., the total seconds that all queries took to run. Instead, we price InfluxDB Cloud by the number of queries. If you run 10,000 queries in a month, it’s the same price whether they take an average of 50ms each or one second each. This holds whether your queries are written in Flux or InfluxQL.

<figcaption> Credit: Crissy Jarvis via Unsplash</figcaption>

Why did we make this change? It was the right thing for our customers.

By pricing based on a simple query count, it’s now much easier to forecast your InfluxDB spend. And, by moving away from duration-based pricing, what you pay for a given query will be the same each time you run that query.

To estimate your queries, look at queries that run regularly, as that will dominate your spend. These include:

InfluxDB Tasks, where each task execution is a query execution.
InfluxDB Alerts, specifically the alert checks.
InfluxDB Dashboards that are continually displayed, such as on a wall monitor. Note that each chart in a dashboard, when refreshed, counts as at least one query execution for billing purposes.
Queries from InfluxDB client libraries and InfluxDB API.

Conversely, ad-hoc queries from the Influx query command, Influx REPL, and InfluxDB Data Explorer probably won’t drive a significant portion of your spend.

Understanding Data Out pricing

The other change is that we now charge for Data Out, albeit a small amount. Data Out refers to the amount of data transferred out of InfluxDB Cloud. This is referred to by most cloud providers as “Data Transfer”. Those of you familiar with cloud provider Data Transfer charges will recognize that our Data Out pricing is simply based on cost-recovery. You would incur the same charges directly from your cloud provider when transferring that same amount of data.

Going a step further, we’ve built mechanisms to reduce your Data Out charges when you use InfluxDB Data Explorer. Because it makes no sense to pull down a million datapoints if the InfluxDB UI can only display 1000 of them, Data Explorer now automatically aggregates your query results based on some function (min, max, mean). Not only does this make your queries snappier — it reduces your Data Out charges.

Here’s how auto-aggregation looks in Data Explorer; you can read more in this post.

<figcaption> Auto-aggregation in InfluxDB Data Explorer throttles your Data Out usage</figcaption>

On the other hand, if you want to download ALL data points, without aggregation, use Script Editor, InfluxDB Client Libraries, or the InfluxDB command-line interface.

How to forecast your InfluxDB Cloud charges

<figcaption> Credit: Vishwarajsinh Rana via Unsplash</figcaption>

Here’s how to estimate your InfluxDB Cloud monthly usage and price:

Sign up for the free tier of InfluxDB Cloud and send us data.
Set up the InfluxDB Cloud Tasks, Alerts, Dashboards, and calls to our client libraries and API that will drive Query Count and Data Out.
Run InfluxDB Cloud for whatever length of time you need to get a representative amount of data Storage. If your data volume is steady throughout the day, an hour should be enough. If it rises and falls over a 24-hour period, then that's the amount of time you need.
Check your usage in InfluxDB Cloud, extrapolate to 30 days, then apply the price points above.

Is InfluxDB Cloud free?

Thus far we’ve been talking about what the price of InfluxDB Cloud is, but to clarify: InfluxDB Cloud also has a free tier that doesn’t expire. We provide this because we believe that free tiers provide the best experience for developers:

Pacing: You can explore InfluxDB Cloud at your own pace, as your schedule allows, without having to worry about a time-limited trial. We understand that when you present new technologies to your team, you want to have enough time to prepare.
Professional growth: For many developers, hobby projects are a key part of their professional growth, since they allow them to work with new technologies that aren't yet adopted by their employer. A free tier lets you try out InfluxDB for hobby projects.
Sizing: You have enough time to set up realistic sizing experiments to forecast your spend.

When you create your InfluxDB Cloud account on InfluxData.com, you automatically start in our free tier. As you add data and increase the number of dashboards, tasks and buckets, you may exceed the free tier limits. At that point, you’ll be prompted for your credit card so that you can switch over to our monthly usage-based pricing plan for InfluxDB Cloud. (Some vendors call these a pay-as-you-go or PAYG plan).

The InfluxDB Cloud pricing page describes what both the free and usage-based pricing tiers provide.

How to pay less for InfluxDB Cloud

In addition to our usage-based pricing plan, we also provide an annual plan. Usage-based price points are listed above, while annual plans offer you an additional discount.

Annual plans also accommodate variations in usage that might occur due to an initial migration of data into InfluxDB, as well as the seasonality of your business cycle. They also have the added benefit of advance visibility into when an annual commit will be exceeded, versus being surprised at the end of the month due to exceeding a monthly budget.

To learn more about our annual plan, contact our sales team.

Focus on what matters

At InfluxData, we’re obsessed with making time series data easy to work with. That’s why “time to awesome” is our motto. This mantra works its way into everything we do. We strive to put the user experience first wherever you interface with our products and services — including our pricing — to take the hassle out of time series.

<figcaption> Credit: Tyler Lastovich via Unsplash</figcaption>

We hope that our new InfluxDB Cloud pricing provides the right balance of simplicity and control, so that you can focus on what really matters: building great software and systems that use time series data. But if you have further questions, be sure to contact us so we can get them answered.

How to Use Starlark with Telegraf

Al Sargent (InfluxData) — Tue, 04 Aug 2020 05:00:10 -0700

Our Telegraf Starlark Processor Plugin is an exciting new processor in Telegraf 1.15 that gives you the flexibility of performing various operations in Telegraf using the Starlark language.

What is Starlark, you ask? Starlark (formerly known as Skylark) is a language intended for use as a configuration language. Starlark is a dialect of Python. Like Python, it is a dynamically typed language with high-level data types, first-class functions with lexical scope, and garbage collection. Independent Starlark threads execute in parallel, so Starlark workloads scale well on parallel machines. Starlark is a small and simple language with a familiar and highly readable syntax.

The Telegraf Starlark Processor calls a Starlark function for each matched metric, allowing for custom programmatic metric processing. Since the Starlark language is a dialect of Python, it will be familiar to those who have experience with the Python language. However, there are major differences and existing Python code is unlikely to work unmodified. In addition, the execution environment is sandboxed and cannot access file system, network, system resources.

Even with these restrictions, the Starlark processor has the following capabilities:

Math operations
String operations
Renaming tags
Logic operations

Using the Starlark processor, you can perform real-time aggregates (min, max) on your data in Telegraf before sending it on to InfluxDB. This helps reduce the load on your InfluxDB instance by distributing processing to your servers running Telegraf.

Since we’ve had a lot of requests for math capabilities within Telegraf, we’re glad to report that this plugin lets you perform arithmetic functions. A useful IoT example was requested by a community member who wanted to calculate power locally from the current and voltage fields read from the Telegraf Modbus Input Plugin. Here’s the Starlark function within the processor to do that:

[[processors.starlark]]
 # Reads the Starlark script embedded
 source = '''
def apply(metric):
 # Get the field called current and put into variable I
 I = metric.fields['current']
 # Get the field called voltage and put into variable V
 V = metric.fields['voltage']
 # Create a new field, power, which is I times V
 metric.fields['power'] = I * V
 # Return power as part of your Telegraf metrics stream
 return metric
'''

You can also put the above into a file, give it a “.star” file extension, and call it from your Telegraf configuration file as follows:

[[processors.starlark]]
 # File containing a Starlark script.
 script = "/usr/local/bin/myscript.star"

How to run multiple Starlark functions on Telegraf data

Only one Starlark source or script can be run per processor. So, if you would like to include multiple Starlark functions to your data, just make sure you use multiple processors and include namepass = ["measurement_name"] so all the processors will make changes to a specific data set you’re targeting. Here’s an example of that that looks like:

[[inputs.cpu]]
[[processors.starlark]]
namepass = ['cpu']
 # First Starlark script 
 source = '''
def apply(metric)
 return metric
'''
[[processors.starlark]]
namepass = ['cpu']
 # Second Starlark script 
 source = '''
def apply(metric)
 return metric
'''

Here are more examples of the Starlark Processor in action.

Credit: Roman Mager via Unsplash

How to calculate a percentage ratio in Telegraf

Use this Starlark code to calculate the ratio of two integers, then multiply by 100 to get a percentage. The ratio will get returned as part of the Telegraf metric stream.

[[processors.starlark]]
 source = '''
# Compute the ratio of two integer fields.
def apply(metric):
 used = float(metric.fields['used'])
 total = float(metric.fields['total'])
 # Create a new field called usage.
 metric.fields['usage'] = (used / total) * 100
 # Return usage as part of your Telegraf metrics stream
 return metric
'''

How to scale metrics in Telegraf using Starlark

A common need is to take metrics that are in bytes, and scale them to kilobytes, megabytes, gigabytes, or terabytes. Here’s how to use Telegraf Starlark Processor to convert bytes to kilobytes:

[[processors.starlark]]
 source = '''
# Convert bytes to kilobytes
def apply(metric):
	# k stands for key, v for value
	for k, v in metric.fields.items():
 	 if type(v) == "int":
	 # 1000 bytes in a kilobyte
 	 metric.fields[k] = v / 1000
	return metric
'''

Here’s how to use Telegraf Starlark Processor to convert bytes to megabytes:

[[processors.starlark]]
 source = '''
# Convert bytes to megabytes
def apply(metric):
	# k stands for key, v for value
	for k, v in metric.fields.items():
 	 if type(v) == "int":
	 # 1000000 bytes in a megabyte
 	 metric.fields[k] = v / 1000000
	return metric
'''

And here’s how to use Telegraf Starlark Processor to convert bytes to gigabytes:

[[processors.starlark]]
 source = '''
# Convert bytes to gigabytes
def apply(metric):
	# k stands for key, v for value
	for k, v in metric.fields.items():
 	 if type(v) == "int":
	 # 1000000000 bytes in a gigabyte
 	 metric.fields[k] = v / 1000000000
	return metric
'''

(If you prefer to use base-2 values to convert your storage, then you’re talking mebibytes, gibibytes, etc. This article explains the difference and how to calculate them.)

One last example: sometimes it’s helpful to scale metrics to ensure that they display correctly in a chart. Here’s how you can use the Telegraf Starlark Processor to scale up a number by 10:

[[processors.starlark]]
 source = '''
# Multiply any float fields by 10
def apply(metric):
	# k stands for key, v for value
	for k, v in metric.fields.items():
 	 if type(v) == "float":
 	 metric.fields[k] = v * 10
	return metric
'''

Transform values in Telegraf using Starlark

Suppose you need to transform one value to another in your Telegraf telemetry. For instance, F5 network devices return their status as an integer with values 0 through 5, where the value of 1 means healthy and the rest are variations of unhealthy status. Suppose I want to collapse this to a binary state — just one or zero — where one means healthy and zero means unhealthy, as shown in this table:

Original value	New value
0	0
1	1
2	0
3	0
4	0
5	0

Note that we don’t need to change anything if the values are zero or one; we only need to make changes if the values are 2 through 5. Here’s how I’d make those changes in Telegraf using Starlark:

[[processors.starlark]]
 namepass = ['measurementname']
 source = '''
def apply(metric):
	# v stands for value
	v = metric.fields.get('status')
	# If no value, do nothing:
	if v == None:
 	return metric
	# When v is 2, 3, 4, or 5:
	if 1 < v and v < 6:
 		metric.fields['status'] = 0
	return metric
'''

How to calculate the percent of healthy devices using Flux

Let’s continue with this network monitoring example. Performing this kind of transformation locally simplifies working with the data upstream. Now that this kind of transformation has occurred, we can perform a much simpler mathematical calculation like sum(status)/count(status)*100 to get the percentages of healthy devices.

Using Flux, the query looks like this:

// First, create a variable called data
data = from(bucket: "device_data")
 |> range(start: -10m)
 |> filter(fn: (r) => r._measurement == "network_devices" and r._field == "status")

// Create a sum of the data across one minute windows
sum = data
 |> aggregateWindow(every: 1m, fn: sum)
 |> map(fn: (r) => ({ r with _field: "sum" })

// Create a count of the data across one minute windows
count = data
 |> aggregateWindow(every: 1m fn: count)
 |> map(fn: (r) => ({ r with _field: "count" })

// Create a union of the data, using the sum and the counts to calculate percent of healthy devices
union(tables: [sum,count])
 |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
 |> map(fn: (r) => ({ r with
 _field: "percent_healthy",
 _value: (float(v: r.sum) / float(v: r.count)) * 100.0
 }))

In one pass, this Flux script collects the data and then uses it to calculate both the sum and count while storing those values in separate variables. The data is then combined via a union of the two variables and the percent_healthy result is calculated producing a single value in 1 min intervals over the past 10 minutes.

How to rename tags in Telegraf using Starlark

If you need to rename tags in your line protocol, here’s how you’d use Starlark:

[[processors.starlark]]
 source = '''
# Rename any tags using the mapping in the renames dict.
renames = {
	'lower': 'min',
	'upper': 'max',
}
def apply(metric):
	# k stands for key, v for value
	for k, v in metric.tags.items():
 	if k in renames:
 	metric.tags[renames[k]] = v
 	metric.tags.pop(k)
	return metric
'''

How to use conditional logic with Telegraf and Starlark

Starlark also supports changing metrics only if certain logical conditions are met. For example, suppose you wanted to set any metric with a value between 1 and 6 to zero. Here’s how you’d do that with the Telegraf Starlark Processor:

[[processors.starlark]]
 source = '''
# Set any 'status' field between 1 and 6 to a value of 0
def apply(metric):
	# v stands for value
	v = metric.fields.get('status')
	if v == None:
	 # If there is no status field
 	 return metric
	if 1 < v and v < 6:
	 # If status is greater than 1 and less than 6, set status to zero
 	 metric.fields['status'] = 0
	# Return your modified metric to the Telegraf metrics stream
	return metric
'''

How to drop metrics based off a numeric condition

Here’s how to store metrics in InfluxDB if they meet a particular numeric condition, such as detecting a 500 error:

def apply(metric):
    if metric.fields["value"] <= 10:
      return metric // keep
    else:
      return None // drop

Demo of Starlark and Telegraf

If you’d like to see a demo of how to write Starlark scripts for Telegraf using Visual Studio Code, check out this demo video:

Conclusion

As you can see, there’s a lot you can do with the Starlark Processor. For more, check out the list of examples and uses for the Starlark Processor if you would like to see how it could enhance your data collection. Also if you would like to read more about Starlark, take a look at the Starlark specifications for more details about the syntax and available functions you can use with this processor.

Got your own Starlark functions that you think others would find useful? Please share them in our public InfluxData Slack channel and InfluxDB community site. And feel free to post any questions you have there if you need help. Enjoy!

InfluxData Blog - Al Sargent

How InfluxDB Works with IoT Data

Why use InfluxDB for IoT telemetry?

Every IoT use case is a time series use case

IoT data architecture

How to plan your IoT data architecture

What's new with InfluxDB and IoT

Acquire IoT telemetry data

Enrich IoT telemetry data

Operate using local IoT telemetry data

Analyze IoT telemetry data

Our IoT product philosophy

Conclusion

InfluxDB Cloud is on Microsoft Azure Marketplace

How does InfluxDB work with Azure and other Microsoft technologies?

Why access InfluxDB on Azure Marketplace?

What's the pricing of InfluxDB Cloud on Azure?

Where does InfluxDB run on Azure?

Conclusion

InfluxData is SOC 2 Certified

What is SOC 2 compliance?

Why does SOC 2 compliance matter?

Customer trust

Customer compliance requirements

Foundation for additional certifications

What did our SOC 2 audit include?

Need to become SOC 2 compliant?

Conclusion

InfluxDB Cloud is on AWS US East

Where InfluxDB Cloud runs

Region Name

Location

Integrate InfluxDB Cloud with AWS

How to access InfluxDB Cloud on AWS Virginia

Customized code snippets for AWS US East

Expanding our reach

AWS US East is just part of the story

?

Conclusion

3 ways InfluxDB OSS Complements InfluxDB Cloud

InfluxDB development environment

InfluxDB edge processing

Air-gapped monitoring

InfluxDB Cloud compared to InfluxDB OSS

InfluxDB Cloud

InfluxDB Open Source

Conclusion

Introducing the InfluxDB Template UI: Monitoring Made Simple

How to use the InfluxDB Template UI

Set up Telegraf to monitor network performance

Run Telegraf to monitor network performance

View your Template dashboards

Send InfluxDB alerts to Slack

Share your expertise with InfluxDB Templates

Conclusion

InfluxDB Cloud Now on Azure East US

Where InfluxDB Cloud runs

How to access InfluxDB Cloud on Azure East US

Customized code snippets for Azure US East

Integrate InfluxDB Cloud with Microsoft Azure

Up next: Azure Marketplace billing, more regions

Conclusion

Putting You in Control of Your InfluxDB Cloud Spend

Our search for the right pricing metric

How InfluxDB Cloud is priced

Usage-based pricing vs per-server pricing

Putting you in control of your spend

What's the price of InfluxDB Cloud?

How to calculate InfluxDB Cloud charges

Understanding Query Count pricing

Understanding Data Out pricing

How to forecast your InfluxDB Cloud charges

Is InfluxDB Cloud free?

How to pay less for InfluxDB Cloud

Focus on what matters

How to Use Starlark with Telegraf

How to run multiple Starlark functions on Telegraf data

How to calculate a percentage ratio in Telegraf

How to scale metrics in Telegraf using Starlark

Transform values in Telegraf using Starlark