Announcing Native Collectors: Bringing Native Data Collection to InfluxDB Cloud

By Samantha Wang / Company, Product
Aug 23, 2022

Streaming time series data from brokers and services that are on-premises or in the cloud to a cloud-based database is a resource-intensive process requiring third-party software and heavy customizations. Today we’re announcing InfluxDB Native Collectors to make it easy for developers to collect, process, and analyze data by subscribing directly to supported message brokers. With this new suite of capabilities, we are expediting cloud-to-cloud data transfers so developers can focus on building and scaling applications with their time series data.

Today we’re excited to announce the immediate availability of InfluxDB Native Collectors, a new suite of capabilities that enable developers building with InfluxDB Cloud to subscribe to, process, transform, and store real-time data from messaging brokers. InfluxDB Native Collectors give developers the fastest way to get data from third-party brokers into InfluxDB Cloud without the need for additional software or new code. Beginning with our Native MQTT collector, this new capability allows InfluxDB users to subscribe to MQTT topics directly from the InfluxDB Cloud interface. The InfluxDB Cloud service will then communicate directly with their MQTT broker to immediately start to receive any message posted to those topics.

InfluxDB Cloud already offers many data ingestion methods to support different data sources, protocols, and formats:

• InfluxDB Client Libraries: For many popular development languages allow you to easily make API calls to write data to InfluxDB Cloud.
• Telegraf Collection Agent: Offers 300+ plugins to collect metrics from cloud services, apps, IoT sensors, and more.
• Third-party integrations: Technology ecosystem integrations for technologies such as FluentD, Vector, and proprietary IT and OT protocols.
• Scrapers: Scrape Prometheus-formatted metrics from an HTTP-accessible endpoint and store them in InfluxDB.
• Or developers can build their own collection agent with the InfluxDB API, which provides a programmatic interface for interactions with InfluxDB.

These existing options provide excellent ways for developers to get their data into InfluxDB. However, they typically require writing code or installing software. For developers using message brokers, especially those installed in the cloud or serviced by a third party, writing software or installing additional software may not always be desirable, or even feasible. The introduction of InfluxDB Native Collectors adds an option to give developers the fastest and simplest way to get their data into a time series database.

Faster Time to Awesome for builders

Three configuration steps are all it takes to get your time series data flowing from your message broker into InfluxDB Cloud:

• Broker Details: Specify the IP Address, Port, and Auth parameters for your Message Broker.
• Topic Name: Provide the topic name for the topic you want to subscribe to.
• Parsing Rules: Set up some simple parsing rules to map elements in your messages to measurements, timestamps, fields, and tags in InfluxDB.

In most cases, the configuration only takes a few minutes. Once completed, any messages pushed to your MQTT broker will flow into InfluxDB as time series data. You can then immediately start to query and visualize that data and can even set up monitoring alerts for specific metrics and events. And using the Flux scripting language, you can easily set up downsample tasks and other data transformations.

Optimize workflows for broker-based data sources

InfluxDB Native Collectors use a brand-new InfluxDB Cloud service purpose-built for the unique nature of broker subscriptions. We built a number of conveniences into the collector’s User Interface that allow you to save time and effort during configuration. For example:

• Subscription Wildcards: Use the wildcard characters + and # in your Topic Configuration to dynamically subscribe to one or many specific topics on a particular broker. This adheres to the MQTT Specification for Topic Names and Topic Filters.
• Dynamic Data Routing and Orchestration: Using Subscription Wildcards and Write Destination, you can dynamically route events to different InfluxDB Buckets depending on the broker source. For example, write all data from “Energy” related topics to an “Energy” bucket, and all data from “Weather” related topics to a “Weather Bucket”.
• JSON and String Parsing: As the message content of MQTT messages is truly unstructured, we introduced methods for using JSON Path expressions and Regular Expressions to dynamically parse JSON and string payloads, respectively. Mine your MQTT data for timestamp, measurement, field, and tag sets dynamically, no matter the text format.

Reimagining device-to-cloud data transfers

InfluxDB Native Collectors gives you a faster way to ingest time series data in the cloud in one step and without customization, orchestration, or additional hosting services.

• Data stays in the cloud: If your message broker is already cloud-based, you can seamlessly send your data from your broker to InfluxDB Cloud. With Native Collectors in InfluxDB Cloud, you can have a completely end-to-end serverless and managed solution.
• Simple low-code setup: InfluxDB Native Collectors eliminate coding and troubleshooting complex scripts and allow you to focus your engineering resources on other areas of your application.
• Meeting developers where they are: InfluxDB Native Collectors support the most common cloud subscriptions, brokers, and services. Support for MQTT is immediately available, with additional support for Kafka, AMQP, and more coming soon.

Get started with InfluxDB Native Collectors today

If you are unsure whether a time series database or InfluxDB Cloud is for you, Native Collectors let you realize benefits right away. You can evaluate it without needing to spend much time getting things set up or kick off a complex project to evaluate it. This results in both a low cost of adoption and a low cost of evaluation.

For more information on how to get started, sign up for InfluxDB Cloud if you haven’t already and make sure you register for our upcoming webinar on August 30.