Announcing: Native MQTT Integration with HiveMQ and InfluxDB Cloud

Session date: Aug 30, 2022 06:00pm (Pacific Time)

InfluxData is excited to announce the launch of a new Native MQTT Collector which enables developers to configure InfluxDB Cloud to subscribe to an MQTT topic with no additional software or agents. InfluxDB Cloud will natively convert MQTT messages to Line Protocol — resulting in a faster and simplified process. Discover how to get IoT data from a HiveMQ MQTT Broker into InfluxDB with a few easy steps. Explore how to subscribe to MQTT topics and how to parse MQTT messages to determine the relevant metrics you want to ingest into InfluxDB. Learn how you can use the Native MQTT collector to ingest industrial IoT metrics quickly and start visualizing, analyzing, and transforming your data.

Join this webinar as Kudzai Manditereza and Gary Fowler discuss:

• Feature deep-dive into InfluxDB’s new MQTT functionality
• How to configure HiveMQ and InfluxDB to quickly start ingesting data
• Demo — see a live demo of the new feature and MQTT messages flowing from sensors to an HiveMQ Broker to InfluxDB Cloud

Click here for presentation

Watch the Webinar

Watch the webinar “Announcing: Native MQTT Integration with HiveMQ and InfluxDB Cloud” by filling out the form and clicking on the Watch Webinar button on the right. This will open the recording.

[et_pb_toggle _builder_version=”3.17.6” title=”Transcript” title_font_size=”26” border_width_all=”0px” border_width_bottom=”1px” module_class=”transcript-toggle” closed_toggle_background_color=”rgba(255,255,255,0)”]

Here is an unedited transcript of the webinar “Announcing: Native MQTT Integration with HiveMQ and InfluxDB Cloud”. This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.

Speakers:

• Caitlin Croft: Senior Manager, Customer and Community Marketing, InfluxData
• Kudzai Manditereza: Developer Advocate, HiveMQ & Founder, Industry40.tv
• Gary Fowler: Product Manager, InfluxData
• Helen Weller: Software Engineer, InfluxData
• Cannon Palms: Software Engineer, InfluxData

Caitlin Croft: 00:00:04.595 Hello, everyone. And welcome to today’s webinar. My name is Caitlin Croft. Real excited to have you all here today to learn more about our new native MQTT integration with HiveMQ and InfluxDB Cloud. This session is being recorded and will be made available for replay later today. And don’t be shy. We have tons of people on the line to answer your questions. So we have Samantha and Gary from our product team. We have Cannon and Helen who are part of our engineering team, so. And of course, we also have Kudzai from HiveMQ. So Gary and Kudzai are going to be presenting, but don’t be shy. Ask questions for the rest of them. And I just want to remind everyone to please be friendly and courteous to all attendees and speakers. Without further ado, I’m going to hand things off to Gary.

Gary Fowler: 00:01:04.425 All right. Thanks, Caitlin. Excited to talk to everyone about this new feature that we’ve come out with, our native cloud ingestion of MQTT data using our native MQTT data collector. So here’s what we’re going to talk about today. I’ll give you a real quick introduction to InfluxData and HiveMQ, talk about the new feature, and then we’re going to mainly focus on seeing it in action. So Kudzai has a couple of sensors that he’s going to start generating some MQTT data. We’re going to see it flow all the way through the Hive MQTT broker. And we’re going to see it land in InfluxDB, where we can do some visualization, set up some alerts, and things like that. So here we go.

Gary Fowler: 00:01:52.098 So a little bit about InfluxData, if you’re not familiar. Most of you probably are if you joined this webinar. But if you’re not, InfluxData is where developers build IoT, real-time analytics, and cloud applications. Most people know us for our time series database. Here’s a quick fact sheet on us and our open source roots. Founded in 2013. We have about 265 full-time employees, with nearly half of those all in a very technical role. So we are very software-engineer-heavy. Mainly, we’re a company that’s focused on building great software. I’ll turn it over to Kudzai to give an overview of HiveMQ.

Kudzai Manditereza: 00:02:33.744 Okay. Thank you so much, Gary. And thanks, Caitlin, for the introduction. I’m excited to be here on this platform today to talk about this new functionality. So again, my name is Kudzai Manditereza. I’m a developer advocate at HiveMQ. So just to give you a brief background about HiveMQ, we’re a company that was founded in 2012 just outside of Munich. And the whole basis of the company is that in today’s hyper-connected world, we see that more and more people, things, and services are communicating with each other. And HiveMQ provides a platform based on the MQTT protocol to achieve this kind of hyper-connectivity in a very reliable, highly scalable, and flexible way. And as you may be aware, the MQTT protocol is already used in many IoT use cases today, like connected assets, smart manufacturing, transportation, and logistics, and at HiveMQ are helping companies like BMW, Telecom, and Netflix to achieve their goals and provide a flexible solution so that they can focus on their core business. Back to you, Gary.

Gary Fowler: 00:03:44.016 All right, thanks, Kudzai. Just a quick plug for HiveMQ. Been using it a long time, love the HiveMQ products, and great group of people over there to work with as well. So they’ve been a great partner of ours.

Gary Fowler: 00:03:58.882 So what is this native collection? The ability to directly ingest data from all these popular data sources without the need to do any transformation to light protocol or normal format for ingesting data or to run our Telegraf agent. So I think those of you that have been working with Influx for a while, you already know we have multiple ways to get data into InfluxDB from other sources. We have our client libraries, which is very common. You just write some code, use our APIs to get the data in. Very flexible, really no limits on what you can do there. A lot of our customers also use the Telegraf agent. So the open source Telegraf agent has 200 to 300 plugins. It has a lot of plugins for a lot of different sources. MQTT is one of them we’re going to talk about today. But you can use Telegraf for all sorts of different things to get data in. Now we’re adding this other method, native collection, where we interface directly with cloud data sources. In this case, we’re going to be talking about MQTT, but we do have plans to do other data sources like Kafka and AMQP, etc.

Gary Fowler: 00:05:19.031 So why so many options? You say, “Hey, well, if you could use client library, you can use Telegraf, why do you need this native collector option?” So that’s a good question. There is great reason to use our client libraries and our Telegraf, and they are perfect options in a number of cases, but there’s a couple of times, instances, where they don’t necessarily be — they’re not necessarily the perfect choice. In one instance, if your data is already in the cloud and you’re a cloud-first customer, you may not have a good place to install a Telegraf agent. You may not have developers ready to write any code, right? Your developers may be working on your core business, and you don’t want to have to have them use the client library just to write some code. If you’re using a cloud-based MQTT broker, there may not even be a good place to install your software or the Telegraf agent’. So in those cases, this native collection becomes a really great option because you don’t need to write a single line of code, you don’t need to install an agent, you don’t need any extra compute time or anything like that in your cloud environment. You just set up the Influx software to communicate directly to your MQTT broker.

Gary Fowler: 00:06:44.476 There’s another case where it becomes a really nice solution. And that’s when you’re just trying to do a quick POC, maybe you’re trying to evaluate solutions. You don’t necessarily want to go to the time of setting up some additional software. If you already have an MQTT broker that is available, you can really get up and running and have a POC with InfluxDB in a matter of minutes.

Gary Fowler: 00:07:14.840 So with this new feature from the Influx Cloud, you can directly subscribe to MQTT topics. So anything that you are — any topic that your devices are publishing to that is being serviced by your MTT broker, you can set InfluxDB to subscribe to those topics. So you can subscribe to a single topic. You can subscribe to multiple topics. You can even use the wildcard facilities that MQTT provides to you, hashtags and plus signs, to take a specific point in the subscription tree or anything below that in the subscription tree. So a number of options there. So how does this compare with the previous ingestion that you might have been doing with Telegraph or with client libraries? This is just a diagram that just shows you’re eliminating one extra step here. You’re going directly from the MQTT broker to InfluxDB Cloud without the need for a Telegraf agent.

Gary Fowler: 00:08:20.663 All right, so today we’re going to show you how all this works, how easy it is to get set up. Kudzai’s going to show you the great HiveMQ MQTT broker. I’m going to show you InfluxDB and we’re going to show you data flowing all the way through there. So we’re going to start with Kudzai. I’m going to turn it over to him. He’s going to show you the sensors and the things that he is working with to generate the data that we’re going to use for today’s demo.

Kudzai Manditereza: 00:08:52.538 Okay, if you can — yeah, sure. Thanks, Gary. Yeah. So today I’m going to take you through the demo setup that I’ve got here that will be generating all these data so I could show you how the new functionality works. So what we’re going to do is we are going to demonstrate a remote monitoring use case. So this is a scenario where you might have some industrial assets located in a factory or in some remote location. And you have some devices that are MQTT-enabled, that are collecting telemetry data that is being generated by these assets and then publishing them to the back end system. So in our case, what we have here is I’ve got a groov RIO, remote IO device, right? So that is collecting temperature data; I’ve got a temperature prop, which I’m going to show you shortly. And then it’s publishing that as MQTT messages to the HiveMQ cloud broker that I’ve deployed on AWS Cloud. And then I’ve also got a Raspberry Pi device that is collecting temperature and humidity telemetry data and also publishing it to the HiveMQ MQTT broker in the AWS Cloud. And then we’re going to have InfluxDB instance subscribe to that same MQTT broker and then pull all those tags and then passes them onto the database.

Kudzai Manditereza: 00:10:23.775 So what I’m going to do here is I’ll switch my video just to show you the kind of setup that I have here. So here you can see I’ve got this groov RIO. So that’s the remote IO device, that’s industrial remote IO device. And then here what I’ve got is a temperature probe. So that is currently measuring ambient temperature. So we’re going to use a heat source to kind of generate some changes to that value. And then over here, I’ve got a Raspberry Pi. And then as you can see, I’ve got the DHT11 temperature and humidity sensor that is also generating temperature and humidity data. And then that Raspberry Pi is then publishing that information to the HiveMQ broker that’s in the cloud. So if I could share my screen here, okay, so what you’re looking at here is the — that’s the dashboard of my Hive MQbroker that I’ve deployed. So it’s actually a simple one-click deployment. And then you just specify your instance and the amount of computing resources that you need to dedicate to that cluster. If you need to have more than one broker cluster, you could actually specify. So in this instance, I’ve only provisioned one cluster, which is enough for this demonstration.

Kudzai Manditereza: 00:11:56.031 So what this dashboard gives you is this ability to see what’s going on with your MQTT network. So as you can see here, we see the number of connected clients that are currently connected to that. And you can see if there’s messages that has been dropped. As you can see here, we had a message that has dropped at 1515. So it gives you insights into the health of your network. And here this is the IP address on which my broker is actually publicly available, which is what Gary is going to be using to connect to this MQTT broker. And then on my devices, I’ve got Node-RED installed on both the Raspberry Pi and the groov RIO, which comes with Node-RED already pre-built into it. So here this is a basic flow where I’m collecting temperature from this sensor. And then I’m then converting that into a JSON format as you can see here. And then I’m publishing that JSON payload and this InfluxDB/JSON telemetry topic.

Kudzai Manditereza: 00:13:05.635 Okay, so as you can see here, I’m using a debug window to show you the actual JSON payload, right? And then because the native connector actually supports multiple formats, you can actually use JSON. You can use the InfluxDB’s line protocol and then you can use the string. So what I’m going to be showing you again here is how you’re going to actually ingest a line protocol. So here I’ve got another function block that takes the JSON format and then converts it into InfluxDB’s line protocol here. So basically that’s what is happening. I’m just using some string concatenation methods here to just build up a line protocol, a string, and then publish it to the HiveMQ broker that’s on AWS.

Kudzai Manditereza: 00:13:57.997 Same applies here if I come here to my Node-RED. So I’m also running a Node-RED. So here I’ve got a flow where I’m using an Opto 22 node to read temperature from my module, channel number zero, which is where my temperature prop is connected. And then I’m passing this on to create a JSON payload again. So you’d notice here that my JSON payload has got some properties such as device ID, device type, [inaudible] groov RIO, remote IO, and then it’s also got the error state and then the module ID. And then I’ve got the same for my Raspberry Pi, right? So here the only difference is that on the Raspberry Pi, I’m actually publishing two data points, and on the Node-RED, I’m publishing only one data point, which is the temperature. And I’m doing the same here, creating a line protocol. So we’re actually publishing a line protocol and a JSON payload from my groov RIO. And the line protocol is actually being published to this topic called line protocol telemetry. And the JSON payload is published to the InfluxDB JSON telemetry. So the same applies with the Raspberry Pi. So that’s our setup that we have going on here. So I will turn it over to you, Gary, so that we can start to see that information come through.

Gary Fowler: 00:15:30.057 All right. Thanks, Kudzai. Appreciate it. Let me re-share my screen here. We’ll talk about what does it take to actually configure this, then, to get InfluxDB communicating with the MQTT broker that Kudzai has set up? So it’s really a simple three-step configuration. First is configuring the broker. We’re going to tell it the IP address or URL that we want to use. We’re going to give it the port number, and then we’re going to give it the broker authentication type. And in this case, there’s no authentication required, but if there was a username and password, we could configure it. We’re going to be releasing certificate support here in the next few days. So you can paste your certificate information in there. That’s all it takes to configure the broker.

Gary Fowler: 00:16:22.584 The second step is configuring the topic. You saw Kudzai just put in a topic when he was showing you the topics that were set up that he’s publishing on; we’re just going to subscribe to the same topic, and then we’re going to tell it what InfluxDB bucket that we want to write to. And then we’re going to define some parsing rules. And so the parsing rules tells us where we want this information to go. So if we want it to go — for InfluxDB, everything goes into an InfluxDB element. We have measurements, timestamps, fields, and tags. So the parsing rules tell us where to go for each item.

Gary Fowler: 00:17:07.037 All right, I’m going to stop this. And we’re going to go over to InfluxDB Cloud. First thing that I’m going to do is I’m going to go in and create a new bucket that we’re going to write to. So I’m going to call this bucket — we’ll say “Webinar Today.” We’re going to say, “Hey, I want to automatically delete data after 30 days.” We want this for 30 days. So I could change this, but we don’t need to for today’s purpose. And I’ve created that bucket. So now I’m going to go over here to native subscriptions, which I can either choose from here, or I can go from the main navigation menu on the left here and say “native subscriptions.” But I’ll go from here and I’m going to create a new subscription.

Gary Fowler: 00:17:58.334 So the first thing that I’m going to do is give it a name. So we’ll call this “Webinar Today” and we’ll call this one “Line Protocol.” We’ll set the line protocol up first. We can give this optionally a description, and then I am going to use an IP address, but I could use a new URL here if I wanted to, to configure my parameters here. And then I’m going to give it the port number that we’re using that was configured. And like I said before, we’re not going to do any security details here. Then what I’m going to do is choose the topic. So in this case, I’m going to say “InfluxDB Demo/Line Protocol Telemetry,” because that is the topic name that Kudzai had configured when he was setting up on his end. So I’ll just ask, because I had to look at that, make sure that matches what he has. And then I’m going to pick the bucket that I’m going to do. And my monitors are overlapping here, so I have to look over it to find the bucket that I picked.

Gary Fowler: 00:19:35.680 And the last step is I’m going to specify the data format. So I could do line protocol. I could do JSON. I could do string. So the neat thing about MQTT is it doesn’t specify what format the data has to be in. So you can put anything in an MQTT message. Probably the most common format that we do see is JSON. A lot of the devices will put the information in JSON format, but as Kudzai showed you, a lot of the devices you can set to whatever you want. So he set a payload in line protocol. He set one in JSON. We’re going to configure line protocol first, but if it was not in line protocol and maybe it was just a bunch of text. You can also do string parsing. And in string parsing, you set a regular expression to be able to find your data. But for this case, we’re going to say line protocol and we’re going to save this subscription.

Gary Fowler: 00:20:35.039 And now we have a subscription running. You can see I have a number of other subscriptions here from different things that I’ve done, and we have a line protocol here. So I’m going to go and I’m going to switch over to my data explorer here and I’m going to go and look and see if I have any data flowing in yet. And of course, since I’m doing a demo, I’m not seeing any data right at this moment. So I’m going to show you the data that I had — we’re already capturing with a previous selection. So I’d already would have walked through this before and set up a subscription to Kudzai’s data. I can go in here and see, “Oh, I want to look at the temperature settings from the Raspberry Pi for the last 3 hours,” and I can go and do a graph and see that information that is flowing through.

Gary Fowler: 00:21:41.025 All right, so let’s go back. I’ll show you something else here about parsing rules. So we talked a little bit already about the line protocol, JSON, and string formats. For JSON specifically, if we want to set up JSON rules, it’s very simple. We’re going to specify the JSON path to the key that we are looking for, the key value pair that we are looking for, so that it can take the value from the key value pair as the element that we want to bring in. So this is a sample of the payload from the Raspberry Pi that Kudzai is doing. So if you look at the JSON, it has a temperature setting here. It has a device ID, time, a device type, error state, model ID, and humidity.

Gary Fowler: 00:22:37.519 And so what we need to do is take each one of those, if we want them; we can ignore data, if we don’t, if we’re not interested in it. But in this case, we’re going to say, “Yes, all of these metrics are interesting to us and we’re going to specify the path to them.” So we’re going to say the measurement. So every element needs a measurement within InfluxDB. We’re going to say the measurement is this device type. And so when we configure this, the JSON path for device type is dollar.device type. That’s just a convention for JSON path, is dollar dot, to start. For the timestamp, we’re going to say that’s dollar.time, because we see the time here. For error state, is a field; model ID, humidity is a field and we’re going to do device ID up here as a tag. So you can see this is a very simple JSON format. It’s flat. But let’s say this temperature was in an embedded object called “thermostat” or something like that. You could simply — in the JSON path, you could say “dollar.thermostat.temperature.” So if it’s an embedded object or if it’s an array, you still have the ability to reference it.

Gary Fowler: 00:23:54.252 All right, so let’s go back and let’s show you what this looks like. So I’m going to go back here to native subscriptions. This time I’m going to create a one with our JSON data. I’m going to configure the same broker. For the topic this time, I’m going to say “InfluxDB Demo/JSON Telemetry.” I’m going to give it the same “Webinar Today” bucket I’m going to write to, but this time I’m going to say “JSON” instead of “line protocol.” And I’m going to pick my — first thing I’m going to do is pick my path to the timestamp, which was dollar.time. And I’m going to say the precision was milliseconds, and I think I forgot to do that on the line protocol and that’s why my data was not showing up, is I didn’t set the right timestamp precision.

Gary Fowler: 00:25:22.607 Now I’m going to specify my measurement. I’m going to do dollar dot — for my measurement. I am going to do dollar.device_type, and I’m going to set it — that looks like a dollar comma. Dollar dot. I’m going to set it as the data type is string. This is the format of the data coming in. Notice there is a little button up here where you can say a JSON path or name. So you can also choose to just hard code your measurement and specify a name. So if you don’t want to pull the measurement out of the data, you can specify a name. In this case we’re going to pull the measurement out of the device type field in the JSON. I am going to add a tag as well. So besides just the measurement, I’m going to do a tag. The tag I’m going to add is “device ID.”

Gary Fowler: 00:26:20.249 Now, notice for device type, for the name here, I’m putting device ID. This doesn’t have to match the JSON key value pair; only the JSON path has to. But if I wanted the name to be something different than what appears in the JSON, that’s okay. I could call it anything that I wanted here. I’m going to call this one an integer because the device ID comes in as an integer. Then I’m going to say dollar.device ID. And now I’m going to configure my fields. So I’m going to configure my temperature first. Again, I could call this “my” temperature if I wanted to. Doesn’t have to be the same as what is in the data, but I’m going to make it the same. We’re going to say this one is a float and we’ll do dollar.temperature here.

Gary Fowler: 00:27:19.934 Now I’m going to add another field. So we’re going to do error state. [inaudible] string. I’ll add another field. My monitor is hiding my other monitor here. So scroll down here, say “dollar.humidity —” oops, humidity here. Miss another float. And then the last one is model ID. Model ID is a string as well. All right, so now I have created my subscription.

Gary Fowler: 00:28:36.104 So if you look down here, I see “Webinar Today JSON.” It’s running notification. Same thing. I can go in and now query and pull graphs on this. So what I did previously is I set up dashboards. So I have a dashboard just for this webinar that it shows — a dashboard that just shows the temperature, shows the current temperature, and it shows the high and low median average temperature today. And I can refresh this and always get the up to date information. So I can change from the groov RIO remote to the Raspberry Pi, see the information on it. You can see all this. So it’s very easy to set up these dashboards. Dashboard literally just takes a few seconds to set up. Come in, and you can see the query behind it, and you can change the type of graph that you want. Those of you that have used this platform, you know there’s a whole bunch of different graph types. And it’s very easy to set up dashboards showing us data.

Gary Fowler: 00:29:47.544 In addition, I have some alerts set up as well. So let’s say I wanted an alert set up for if I was not getting any data in. So if data is not getting reported, I can set up an alert for that. Maybe I want to set up an alert on temperature, if something is getting too hot, so I can set a threshold alert. So in this case, I’ve said, “Hey, when the value is greater than 35, set this check to critical.” And then I have a notification rule that basically says, “When it’s critical, sent a note to my notification endpoint,” which in this case, I have set up for Slack, so I can have Slack set up to send me that information. So it’s very easy to set up these alerts. It’s very easy to set up the dashboard. It’s very easy to query on the data that you’re loading.

Gary Fowler: 00:30:51.393 So actually, one thing that I wanted to do is, Kudzai, would you be willing to try to vary the temperature a little bit and we’ll see what that does?

Kudzai Manditereza: 00:31:04.129 Okay. Sure.

Gary Fowler: 00:31:13.124 You can see that I do have data flowing in here to “Webinar Today” now. Are you going to do the Raspberry Pi, or the —?

Kudzai Manditereza: 00:31:34.429 [The?] groov Rio, yeah. [Get the groov RIO configured?] [inaudible] [shot up there?].

Gary Fowler: 00:31:50.307 All right. You could see the temperature spiked as — what did you say? You were going to use some boiling water or something near your sensor?

Kudzai Manditereza: 00:31:59.706 Yeah. Put a cup of coffee. Just dropped that temperature prop into the hot coffee. So that’s like four to five degrees Celsius, and — that’s Celsius, by the way, for our American audience.

Gary Fowler: 00:32:14.217 All right, thank you. All right, so that’s basically our demo showing you how to get data from sensors in through HiveMQ and then all the way into InfluxDB. So now we’re going to talk just a couple of minutes about what it takes to get started with all this. We’ll talk about HiveMQ and MQTT broker first, so go ahead, Kudzai.

Kudzai Manditereza: 00:32:47.386 Okay, thanks, Gary. So to get started with HiveMQ, there’s many different options for provisioning the broker. So we’ve got — first of all, the option that I showed you here is a public cloud, self-hosted. So in the event that you want to host this broker yourself, maybe in a platform like Azure or AWS, you could actually just deploy it there like the one-click and then configure your information, and then you can deploy it onto your public cloud, right? And then in the event that maybe you don’t want to deploy this in the public cloud — you’ve got IT infrastructure within your premise — you could use some tools that your [inaudible] to deploy this broker cluster on prem, and other options that you’ve got is a managed service HiveMQ cloud. So perhaps if you’re maybe working as a one-man developer engineer, you just want to try out a few things, or maybe a small team or even big companies — there are some big companies that don’t want the hassle of having to manage IT infrastructure — we do have a managed service HIVEMQ cloud. So that’s a free service actually for up to 100 devices.

Kudzai Manditereza: 00:34:07.607 So those are the three options that we’ve got. And we do have some extensions. Some of them are open source; some of them are enterprise. So as you can see, one of the reasons why actually I’m excited about this InfluxDB new functionality is that we actually had an extension. As you can see, the InfluxDB that lives on the broker that was responsible for actually capturing that information and forwarding it to HiveMQ, which means this functionality actually makes that redundant. So that’s one of the things that actually makes us excited about this new InfluxDB functionality. Back to you, Gary.

Gary Fowler: 00:34:48.336 All right, thank you very much, Kudzai. So getting started from the InfluxDB standpoint is really simple. This is a feature that’s only available at the pay-as-you-go level. Doesn’t require a big expenditure if you don’t have a lot of data, though, but it does require a pay-as-you-go account. So you just need to upgrade. You just essentially need an InfluxDB Cloud 2.0 account upgraded. Go and create a bucket just like I did. Configure your MQTT parameters, just like I did. And away you go. You’re starting to ingest MQTT data. All right, I’m going to turn it over to Caitlin here to talk again a little bit about InfluxDB University.

Caitlin Croft: 00:35:42.634 Fantastic. Thank you, Gary. Thank you, Kudzai. So I just want to remind everyone of InfluxDB University. We launched it in the last six months. It’s completely free and there is a ton of free instructor-led as well as on-demand training. So like I said, if you’re brand new to InfluxDB, or if you just need a refresher or a certain component you need help with, definitely be sure to check it out. We’re always adding more courses, so it’s a really fantastic resource to get even more familiar with our platform.

Caitlin Croft: 00:36:20.095 And then I want to just let everyone know that Influx Days is coming up soon, November 2nd and 3rd. It’s going to be a hybrid event. We’re still finalizing the agenda, but you can register today for free. A lot of it is going to be virtual, even though it is going to be a hybrid event. So regardless of where you live, you’ll be able to join the sessions and learn from what’s going on at InfluxData and all the amazing things that we are working on. So register for it for free and you’ll get updates about InfluxDays.

Caitlin Croft: 00:36:59.488 All right, so there have been a ton of questions, and I realize some of them have been answered in the Q&A in the chat, but I just want to read them out just so that if you guys haven’t seen them or whatever else, that everyone gets it. So, “There’s no two-way MQTT communication here. Any input on this regards to the InfluxDB integration?”

Gary Fowler: 00:37:29.157 Yeah. So right now this is intended to be the InfluxDB Cloud as a subscriber, not a publisher. So it definitely is only one way.

Caitlin Croft: 00:37:40.349 Cool. And this feature is currently only available in the InfluxDB Cloud pay-as-you-go. So definitely if you’re using OSS, look into converting to pay-as-you-go if you’re interested in this feature. “Good support for JSON timestamps with different time formats. Had some challenges with this with Telegraph.” So I think they’re just kind of curious about getting best practices on dealing with how can you handle different types of time formats with JSON.

Gary Fowler: 00:38:20.404 Yeah, so —

Kudzai Manditereza: 00:38:21.327 Go ahead, Gary.

Gary Fowler: 00:38:23.462 I wasn’t sure if this was a question or comment, so hopefully it’s a comment that it’s a good support for JSON timestamps with different formats, but it may also be anticipating a question where the incoming data may have multiple time formats in it and not just a single format. Is that what you’re asking on?

Caitlin Croft: 00:38:52.732 Anders, I’m happy to unmute you if you would like to talk a little bit more. So you should be able to unmute yourself and chat with Gary and Cannon, because I know you, Cannon, also provided some information.

Attendee #1: 00:39:14.681 Hello?

Caitlin Croft: 00:39:15.930 Hello.

Attendee #1: 00:39:18.838 Hello. Can you hear me?

Caitlin Croft: 00:39:21.097 Yes.

Gary Fowler: 00:39:22.522 We can.

Attendee #1: 00:39:26.990 Can you hear me?

Caitlin Croft: 00:39:28.417 Yes, we can hear you, Anders.

Gary Fowler: 00:39:29.589 We can hear you. Okay, well, I —

Attendee #1: 00:39:38.554 [crosstalk] same timestamp format, but different sources have different formats.

Gary Fowler: 00:39:47.933 Yeah, so what way did you —

Attendee #1: 00:39:50.486 [crosstalk] JSON versus version 1.0. Still got some error messages using the timestamps from JSON.

Gary Fowler: 00:40:01.284 Okay. Cannon or Helen, do you see any issue in that case of just setting up a different — multiple subscriptions with the different formats?

Cannon Palms: 00:40:12.681 No, I don’t. Anything that is a numeric type, so basically a counter of the number of seconds, milliseconds, microseconds, nanoseconds, or ISO8601/RFC3339, we will support. So if you have multiple different formats, yeah. You can create multiple subscriptions, actually.

Caitlin Croft: 00:40:44.003 Cool. Anders, does —

Gary Fowler: 00:40:45.027 Thanks, Cannon.

Caitlin Croft: 00:40:46.413 —Anders, does that answer your question? Oh, he says, “Same timestamp from one source, but different time format for different sources.”

Cannon Palms: 00:41:02.469 Gotcha.

Gary Fowler: 00:41:02.531 Yes.

Cannon Palms: 00:41:04.036 I’m going to make a wild assumption here that a source is probably either a distinct topic or a distinct broker, or the same topic on a different broker. But provided it is possible to target your source, so it’s on a compatible MQTT broker and with a topic that sends this data in the same format repeatedly, yes. We’d be able to support any of those three timestamp formats that I mentioned.

Caitlin Croft: 00:41:40.141 And, Anders, I’m happy to connect with you over email and connect you with some of our DevRels who can help troubleshoot any further issues, as well as with Cannon and Helen. Cool. Let’s see. So the next question is, “Do you support MQTT 5.0, and how do you handle user properties?” And so Helen chimed in and said, “We currently support MQTT version 3.1.1,” and there was a follow-up. “Are there any plans for supporting MQTT 5.0?”

Gary Fowler: 00:42:19.669 Yeah, so not any short-term plans, no, but long-term plans, of course.

Caitlin Croft: 00:42:26.306 Cool. Let’s see. “First of all, want to thank you very much for the demo. Very good work, both Kudzai and Gary and the Influx Data team. This feature is a very useful one. Is it possible to configure the mapping of MQTT messages to an InfluxDB bucket fields and tags through the API? Is it in the roadmap to release this native subscription system for self-hosted instances of InfluxDB Enterprise or InfluxDB OSS? Thank you.”

Gary Fowler: 00:43:00.364 So I’ll answer the last question first. We don’t have it on our roadmap to port this to the OSS instance or InfluxDB Enterprise. Doesn’t mean that we won’t do it someday, just not on our roadmap right now. The first part is interesting. No, it’s not possible right now to configure this via API, but it is an interesting suggestion, one that I’ll put on our backlog for consideration in the future.

Caitlin Croft: 00:43:34.058 “Does the line protocol function allow the bucket tag that Telegraf allows, or do you have to specify the bucket for each topic you’re subscribing?”

Gary Fowler: 00:43:47.113 You do have — oh.

Helen Weller: 00:43:48.444 Yeah, I was just going to say you’d have to specify a bucket per subscription, so for each one, you’re going to have to find where it’s going.

Caitlin Croft: 00:43:58.941 Gary, anything else you want to add there?

Gary Fowler: 00:43:59.385 Thanks, Helen. Nope, she said what I was going to say.

Caitlin Croft: 00:44:05.897 Perfect. “What happens if the field name is changed or you add new with existing data? With the fixed hard coded measure and field names like Temp 1, Temp 2, Temp X, we will run into problems with historical data. How can we manage transform old data to new data models in the future?”

Gary Fowler: 00:44:31.539 Well, that’s one of the neat things about Flux, right, and the power of Flux in the platform. So you can use Flux to set up transformations. So if you have some historical data that doesn’t necessarily match what you’re doing right now, you can set that up to do the transformation. One of the great things about MQTT, right, is that you can pause your subscription, your messages are held, you could do your transformation, you could change to your new format for your subscription, restart, and your data starts flowing in. You’re not losing data. So it becomes a nice way to migrate data from one format to another.

Caitlin Croft: 00:45:16.083 Fantastic. Thank you. All right, if anyone has any further questions for our amazing panel today, please feel free to post them in the Q&A or in the chat; we’ll stay on here just for a couple more minutes. David, we’ll connect with you over email just to follow up with your very specific questions, so don’t worry about that. And if anyone else has questions, don’t be shy. I know this team is really excited to have this feature out there in the wild, and they spent months working on it. So congrats guys, for getting this out the door. In the meantime, I’m just kind of curious. Kudzai, Gary, Helen, Cannon, are there any tips or tricks related to this feature that you’d like the community to know about?

Gary Fowler: 00:46:16.286 Good question. Off the top of my head — go ahead, if you have one, Cannon.

Cannon Palms: 00:46:24.445 I’d say the power of topic wildcards is understated. So we support the traditional methods of expressing wildcard topics. In a lot of cases, it depends on how you’re designing your topic structure, but some folks will have some kind of shared prefix and then individual devices firing off messages to a distinct topic. Let’s say you want to collect data from all of those different devices, but you don’t want to create a new subscription for every single one of those topics. That’s where a wildcard comes in, would support the full extent of the MQTT specification there.

Helen Weller: 00:47:15.936 Yeah. Another, I guess, hint or tip is if you’re not seeing data flowing, take a look at your JSON parsing or regex parsing and we’ve got little links to places where you can test with some of your sample data and the either JSON path or regex you’re providing and make sure that it’s doing what you expect, as that’s a common pitfall that people can come across.

Gary Fowler: 00:47:40.526 And I know that firsthand, as I started playing around with the great software that Helen and Cannon and team wrote. And then I fat-fingered several things when I was first setting it up, and they would have to hit me over the head with a mallet and say, “Hey, did you test your parsing rules using a parsing tester?” And then I did and I was like, “Yes, user error.” But yeah, good tip, Helen.

Caitlin Croft: 00:48:06.495 That’s great to hear. And if anyone on the call — if you guys ever have any questions or need help, definitely check out our community forums as well as the community Slack workspace. Most of us, I think, on the call are in there and so we’re happy to answer questions. So going to put them on the spot. You can go bug Helen and Cannon directly if you have any issues or go bug Gary or Kudzai. I’m sure they’d all be happy to help. Are there any additional resources for MQTT error message? I keep running into a vague MQTT Exception 128.

Cannon Palms: 00:48:49.697 I’m not sure off the top of my head what that one would be, but the experience around error handling and the ability for a user to sort of self-service there in terms of what’s wrong is something that we would like to work on. It’s coming up. But if I had to guess, an MQTT exception is most likely a problem connecting to your broker. So double-check the host, the port number, any credentials you’re providing us, that kind of thing.

Gary Fowler: 00:49:24.345 There is, on our Docs page — so if you go to Docs.Influxdata.com and you go to the Low Data section, there is a place where we talk about MQTT. It has some of the sample parsing and different things like that set up, but it also mentions the monitor bucket, so monitoring buckets. So I don’t know how much you’ve used InfluxDB in the past, but there is an _monitoring bucket that’s automatically created for you and we do write some diagnostic information there. So if you go into a subscription and you say “view data,” it will actually take you to a notebook inside the software that will do a query of the bucket that you are writing to. But also it’ll pull up the underscore monitoring bucket and there can be some additional information available there to you. It might not apply in this particular case, but it can be helpful sometimes.

Caitlin Croft: 00:50:32.043 Great. Let’s see. Error handling if the field comes in as a type not defined in parsing or if field type is different in current chart in InfluxDB.

Gary Fowler: 00:50:47.245 Yeah. So it is a fairly important thing to get right, because this — not necessarily a function of this particular feature, but just a function of InfluxDB. Once you’ve established your schema, what that data type is, that’s what you want it to be, so when data starts coming in with something different, you may have some trouble writing that data.

Caitlin Croft: 00:51:14.420 Are there any resources for offensive penetration testing and security references and defensive strategies available?

Gary Fowler: 00:51:24.905 Why don’t we take that one and let me follow up with our security team on that one, because I don’t know the answer to that and I doubt Cannon or Helen knows the answer to that either. So let me capture that and take it offline.

Caitlin Croft: 00:51:45.108 All right. Well, thank you everyone for joining today’s webinar. It’s been recorded and will be made available for replay later today or early tomorrow morning and the slides will also be made available as well. Thank you so much to Gary and Kudzai for presenting. Kudzai, I love that you had a camera pointing towards your desk and showing up the setup. I thought that was a fantastic addition to the demo component of the webinar. And thank you so much to Canon and Helen and Samantha for being online and answering people’s questions. Really appreciate it.

Gary Fowler: 00:52:22.843 Yeah, thanks, everyone, for joining. I really appreciate it.

Kudzai Manditereza: 00:52:24.536 Thanks, everyone.

Caitlin Croft: 00:52:27.623 Thank you. Bye

[/et_pb_toggle]