StreamNative
Founded by the original creators of Apache Pulsar, StreamNative provides a cloud-native, scalable, and secure messaging and event streaming solution powered by Apache Pulsar. StreamNative Cloud empowers teams and simplifies operations with a fully managed, flexible Pulsar®-as-a-Service offering.
Enriching your streaming data
StreamNative and InfluxData
StreamNative makes it easy to ingest, transform, filter and enrich a large variety of streaming data sources into InfluxDB via native Apache Pulsar connector and MQTT Telegraf Input Plugin. Combined as a team, StreamNative and InfluxDB allow any streaming sources including logs, sensors and events to be routed, buffered, aggregated, multiplexed, transformed and filtered to InfluxDB while also populating any number of other enterprise data stores.
Integrating Apache Pulsar with InfluxDB
StreamNative’s Pulsar to InfluxDB open source connector streams data from Apache Pulsar clusters. It creates a fast, scalable data pipeline to InfluxDB Buckets. The FLiP-InfluxDB connector is a no-code solution to send data. The InfluxDB sink connector pulls messages from Pulsar topics and persists the messages to InfluxDB.
To integrate Apache Pulsar with InfluxDB, you merely need to download the InfluxDB connector to your Pulsar nodes.
Steps:
- Download Pulsar to InfluxDB connector.
- Copy the nar to the connectors directory.
- Create an Influx token.
- Build a configuration file.
- Create the influxdb sink pointing to the nar and configuration file utilizing REST endpoint or command line interface.
- Send data.
- Visualize in InfluxDB Cloud.
For a demo of how to filter and enrich data from Apache Pulsar to InfluxDB, check out this Pulsar Function that outputs Line Protocol data.
Key resources
- Video: Using FLaNK with InfluxDB for EdgeAI IoT at Scale
- Integration: FLiP-InfluxDB
- Webinar: Using FLiP with InfluxDB for EdgeAI IoT at Scale
- Documentation: Writing data with no-code technologies
- Integration: InfluxDB sink connector
- Documentation: Pulsar functions
Unified messaging and streaming to InfluxDB
StreamNative provides a platform for ingesting data from a variety of sources, filtering, routing and transforming them in real-time to feed InfluxDB via its no-code data sink.
StreamNative can also act as a source for InfluxDB to read from with Telegraf via the MQTT protocol for seamless IoT integration.
StreamNative and Apache Pulsar are typically used to ingest, route, transform and enrich data streams for devices, databases, events, logs, sensors and REST endpoints. This data can be streamed one event at a time or in bulk to InfluxDB depending on the client’s needs.
Typical Dashboard
VIDEO
Using FLiPN with InfluxDB for EdgeAI IoT at Scale | InfluxDays NA 2021
Timothy Spann from StreamNative leads this video walkthrough about how his team ran deep learning models, and capture real-time GPS and sensor data on edge devices. They used Pulsar, Apache NiFi + Edge Flow Manager + MiniFi Agents with Apache MXNet, OpenVino, TensorFlow Lite, and other Deep Learning Libraries on edge devices including Raspberry Pi with Movidius 2, Google Coral TPU and NVidia Jetson Nano. Their low-coding IoT applications provided easy edge routing, transformation, data acquisition and alerting before they decide what data to stream real-time to their data space. These edge applications classified images and sensor readings real-time at the edge and then sent Deep Learning results to Flink SQL and Apache NiFi for transformation, parsing, enrichment, querying, filtering and merging data to InfluxDB.
What's next?
Questions? Get Answers