Saving the Holidays with Quix and InfluxDB: The OpenTelemetry Anomaly Detection Story

Session Date: Dec 12, 2023
Time: 8:00am (PT) | 4:00pm (GMT)

The holiday season is upon us—make sure you don’t get pulled away from the holiday fun because of a broken data pipeline!

OpenTelemetry is an open-source tool that offers a standardized way for instrumenting your code and producing telemetry data. Collect, Store, and Act are the three pillars to building any time series-based solution. These pillars simultaneously present questions about data querying, anomaly detection, and forecasting. Python developers use Quix to integrate their data sources and build event-driven apps with AI features that scale. InfluxDB is the purpose-built time series database.

Join this session to explore best practices for crafting a scalable time series data pipeline with OpenTelemetry, Quix, and InfluxDB. We will walk through the process of creating, training, and deploying a machine learning model, utilizing the power and flexibility of Keras and Hugging Face for anomaly detection.

In this webinar, Tun Shwe and Jay Clifford dive into:

• OpenTelemetry’s key concepts and how it leads to observability best practices.
• Tips and tricks for managing time series data with InfluxDB.
• Model training and storage: Dive into model creation and deployment using Keras and Hugging Face.
• Building and deploying a resilient data pipeline with built-in open source connectors in Quix.

Save Christmas, Hanukkah, and New Year’s Eve by building scalable architectures with Quix and InfluxDB!

View the Presentation

Watch the Webinar

Watch the webinar “Saving the Holidays with Quix and InfluxDB: The OpenTelemetry Anomaly Detection Story” by filling out the form and clicking on the Watch Webinar button on the right. This will open the recording.

[et_pb_toggle _builder_version=”3.17.6” title=”Transcript” title_font_size=”26” border_width_all=”0px” border_width_bottom=”1px” module_class=”transcript-toggle” closed_toggle_background_color=”rgba(255,255,255,0)”]

Here is an unedited transcript of the webinar “Saving the Holidays with Quix and InfluxDB: The OpenTelemetry Anomaly Detection Story.” This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors. Speakers:

• Caitlin Croft: Director of Marketing, InfluxData
• Jay Clifford: Developer Advocate, InfluxData
• Tun Shwe: VP of Data, Quix

CAITLIN CROFT: 00:00
Hello everyone. And welcome to today’s webinar. My name is Caitlin, and I’m joined today by Jay and Tun. And today, they’re going to be talking about how to save the holidays with Quix and InfluxDB, the open telemetry anomaly detection story. So regardless of what holidays you guys celebrate, Quix and InfluxDB will be able to help. Please post any questions you may have for Tun and Jay in the Q&A, which you can find at the bottom of your Zoom screen. This session is being recorded. I know you guys are all here to hear from Jay and Tun, not myself. So, without further ado, I’m going to hand things off to Jay and Tun.

JAY CLIFFORD: 00:44
Thank you so much, Caitlin, as always, and thank you all for joining. I think myself and Tun has felt very festive building this webinar. So, without further ado, there’s some quick introductions to be had. So, Tun, my friend, would you like to introduce yourself first?

TUN SHWE: 01:01
Sure. Thank you, Jay. Thank you, Caitlin, and thank you, everyone, for joining us today. My name is Tun Shwe. I’m the VP of data at Quix. And in my past life, I used to be a head of data and data engineer for many years. So, I helped companies determine and implement their data strategies with streaming technologies at the forefront. So, I was responsible for building teams and delivering solutions such as analytics systems and AI features. So, it’s no surprise that my passion is around the real-time and streaming technology spaces. But I’m also really interested in music. So, I play guitar, and I love making any sort of different sounds, and I got into music production over lockdown. So, if you also are interested in that kind of intersection between software engineering and music, please connect with me. We’d love to work on you on something together. I’m driven to create a new normal where data is processed as soon as it is generated. So, oftentimes, we collect the data. And we introduce a delay before we process that data and derive insights from it. And I’d love to be a part of that mission to change that mindset. My belief is that less is more. So, as humans, we tend to believe that success lies in the domain of complication. You have to make things complicated to succeed. But having more is better because keeping things simple, it’s easier to understand, easier to build on. And also believe in getting started sooner. So, education is the path to success as well. So, you should always keep your skills fresh keep an eye on the changing landscape. So that’s me. I’m going to hand it over to Jay. And first of all, I see you’re wearing a Star Wars jumper, so I have to say I would like to see the baby. [laughter]

JAY CLIFFORD: 02:41
There he is.

TUN SHWE: 02:42
Oh, yeah, there it is. Over to you. Thank you very much.

JAY CLIFFORD: 02:45
And I have to say, we need one demo where we’re actually using your guitar-playing skills as the source of data. I think that would be pretty interesting.

TUN SHWE: 02:55
Yeah. I’m playing a lot of Prague Metal at the moment, so yeah, lots of anomalies will be detected, so yeah, let’s come up with something. Thanks.

JAY CLIFFORD: 03:03
So, I’ll keep mine short. I used to be a sales engineer in industrial IoT solutions. So, I was a data plumber, and I didn’t even know it. So basically, connecting machines on the shop floor together using InfluxDB and Grafana a lot. And then swapped over to the dark side and became a full developer advocate for InfluxDB. And that’s kind of where my passions lie now, within the Apache ecosystem, looking at things like Parquet, Arrow and how we can integrate those into big data systems. And then I’m also a massive demo tinkerer at home. So, anything from plant monitoring to building my own homegrown ring system. Yeah, that’s kind of my weekly activities. I’m driven to make observability and IoT solutions accessible to all. And my belief is that an industry success comes from the domain experts. I’m a strong believer in this. I believe our platforms are there to enhance the knowledge and the expertise of those within the industry and allow them to deliver their own solutions. So, we have quite a bit to cover today. We’ve kind of built out a data pipeline for you here of an agenda, so you can see the flow of how things are going to work. We’re going to start off with a really quick intro to sort of what is data plumbing. And then we’re going to talk about how it relates to event streaming and time series databases. We’re then going to have a very festive problem. The idea here is we’re going to create a problem within our festive factory, and we’re then going to solve that problem using Quix and InfluxDB. Once we’ve kind of solved that problem, we are going to be a villain in our own solution and work out how we can basically observe that solution from afar using open telemetry and make sure that solution stands the test of time. And then finally, of course, we will give you the source code so you can try it out yourself at home.

JAY CLIFFORD: 05:01
So, the first thing is, what exactly is a data pipeline? And I kind of stole a quote a little bit from Tun earlier. And that’s the idea of being a data plumber. And this is kind of where we got this idea from, with the idea of this essentially a water facility. So, when you think about a data pipeline, you got to think about it in sources, transformations, and destinations. So, in this real-world example, sources are like our pumps. We’re drawing on that data from, say, a reservoir, a river, and we’re bringing that into our system. We’re then routing that data to different services or microservices or containers that operate on that data in different ways. So, you can imagine those transformations from anything like data cleaning to filtering upon that data, even to enriching that data, say adding further transformations, adding new features to that data as we go through. And we’ll do that a lot within the demo as we go forth. And then finally we have destinations. And destinations in this case, is your faucet or is your tap, this is where you’re getting that lovely water or that lovely data at the end, and you as the end user are consuming that data or consuming that lovely water for your end purposes. So hopefully this gives you quite a holistic view of what a data pipeline is. Now, Tun, could you sort of take this overall concept and relate it to sort of event streaming for us?

TUN SHWE: 06:33
Yeah, definitely. So, let’s get into an introduction on event streaming. So, when you hear about streaming technologies, these are kind of like the three logos that you’re going to encounter most commonly. So, you have Apache Kafka, you have Apache Pulsar, and you have Redpanda. So, Kafka has been around for about a decade and a half. Pulsar, about a decade, and Redpanda roughly half a decade. But they’ve gained traction in the market in their various ways because they each provide kind of very different solutions to the same challenges of stream processing. And when we talk about streaming technologies, these are stream brokers, they’re message brokers. And streaming technologies are kind of split into two parts. So, the first part are the producers. So, these are the applications that either capture or generate data and then produce it into these message brokers. So that’s where you send the data. And then you have, on the other side, the consumers, the applications that will consume from that message broker, consume that data, generate it, get you insights, and get value from that data. So, I think the best thing is to always describe things in the context of batching, because most people would be accustomed to batching. So, let’s go into an example for batching and I’ll how it relates to streaming. So, in this use case, if you can see that little cyclist at the top, that cyclist has sensors on a fitness app. So, measuring things like temperature, speed, heart rate, various data points. And as you can see in the animation, as the cyclist moves along, it generates data points. And in batching, data is generally collected into a database. And once all these data points are collected, a short time later, an orchestrator or scheduler, something like Airflow or Prefect, will kick in and read in the data. And when it reads in the data, it’s called bounded data. And you call it bounded because it has got a start and it’s got an end. So, you can see those timestamps over there, which—

CAITLIN CROFT: 08:24
Oh, sorry to interrupt. We’re seeing—

TUN SHWE: 08:26
Yeah. No worries.

CAITLIN CROFT: 08:27
We’re seeing the batching slide. Is this the correct slide that we’re supposed to be seeing?

TUN SHWE: 08:31
Yeah, that’s correct. Yeah. So, we’re talking about batching right now. Yeah, hopefully, you can see the animations as well. There are some animations there. So, we’ve—

CAITLIN CROFT: 08:38
We cannot see the animations, unfortunately.

TUN SHWE: 08:43
Oh, interesting. Okay.

JAY CLIFFORD: 08:45
You should move backwards—

CAITLIN CROFT: 08:47
Oh, there it is. Yeah, there it is.

TUN SHWE: 08:49
Hopefully. Yeah. Maybe it’s just catching up. Yeah. Okay, we’re going to have to look into this real-time situation here. But you’ve got bounded data here. So, you’ve got the timestamps T1 through to T3, etc. So, you’re loading in those rows of data. And we’ve selected the data we’re interested in in this case, which is gravitational forces. So, we’ve got columns for gravitational X, the Y, and the Z axis. So now you are processing that data, and now that you have this data together in memory, what you can do is you can start computing new columns. So, you might compute gravitational totals, for example, where you sum up the X, Y, and the Z, so you get the totals. And then additionally, you might also be interested in the deltas, how does that total change over time? So, you will compute it that way. So, the three main points there are that computation is on a range of historical data, and it’s stateless because you have all the data there in front of you. So, you don’t need to maintain state. The data processed is at rest and the results are not in real time because you’ve introduced a delay. Now, hopefully what you’ll see on the next slide is the streaming example. So, it’s the exact same use case cyclist generates a data point, but rather than introducing a delay, that message is consumed instantaneously and is processed instantaneously too. So, you have that row come in immediately, and we’ve computed the totals straight away. Now, that is essentially one task. And because between events, you lose the state, you have to maintain an internal state score. So, in this case, we’re going to keep the state of the highest observed value for gravitational total. So here it’s 1.3. So, we’re going to save that into a state. We’re going to continue. The difference here is that data is being collected into a broker or transport, like a Kafka topic.

TUN SHWE: 10:31
My CEO, Mike, he calls these message brokers. They’re kind of databases that have fallen on their side, which is an interesting analogy, because not just pictorially, but with these, as you consume the data, they’re not actually being removed, they actually are retained in memory. So, a new consumer application could come along and rewind and reread all the events at any time. But right now, as we go through and the cyclist is generating new data that flows through the message broker and is instantaneously consumed. So much like Jay’s water analogy with the reservoir, this data is continuously flowing like that water. And we are updating the state with the highest observed total as we do. So, the three points there are that computation is on each event, and it’s stateful where necessary. And we process that data in motion, so we get real-time results.

JAY CLIFFORD: 11:25
Yeah, I mean, I love this analogy because in this case, when we look at the difference between sort of near real-time and real-time, in the case of, say F1 racer or something like that, then you need that instant calculation and that literal subsecond, millisecond response, where with the idea of basically doing bounded data, you basically draw it in, do a calculation, and it’s great for historic viewership. But yeah, you don’t have that sub-millisecond change that you need. So, it’s very cool. So, one of the things that we’re going to quickly cover here, I know there’s going to be a lot of influxes on the call, so I’m just going to speed through this. But one of the main things that we do in an event streaming is essentially we’re going to be processing a lot of time series data. And the formal way of looking at time series data is, it’s a sequence of data points consisting of successive measurements made from the same source over time. So that’s kind of the formal definition of time series data. But if we really unpacked it—the idea here is, if we take an example, say, stock market, basically what we’re doing there is monitoring the stocks over a period of time. So, when the stocks open for the day, we’ll look at a particular company’s stock, we’ll look at where that company’s stock is, and that will be our successive measurement. And we’ll monitor that company’s stock throughout the day, whether it’s going up and down. And what we really want to see is the trends in that data, rather than a single value point, which isn’t very interesting, whether we should be buying or selling. So, in this case, in this instance, the x-axis for us is always the timestamp.

JAY CLIFFORD: 13:13
So, when we talk about time series data, we can put it into two categories. And we’ve sort of covered a little bit of both here with metrics and events. But the idea with metrics here is essentially it’s at a regular interval. We know when it’s come in, it could be every second, every minute, if we base it back on the cycle example. Essentially, what we’re doing there is, say, if we had an accelerometer on the bike, and we’re monitoring that every minute to see what speed the cyclist is going, we know we’re going to get that metric every second, and we can monitor that as such. Events, a little bit different. Events come at irregular intervals. We can still store and capture these, but we don’t know when they’re coming. Essentially, you can think about it like an emergency stop from the cyclist. Basically, when the cyclist stops at an emergency, we log when that emergency happened. But that could be an hours time, could be in a days time. We just don’t know when that emergency stop is going to come. Now, the cool thing is that we can actually turn events into metrics through aggregation, something that a time series database is extremely good at doing. And the idea there is, say, with our emergency stop, we could basically go, okay, give me a count of how many emergency stops happened within one day. And no matter even if that emergency stop count was zero, we know we’re going to get a regular metric each day of zero, even if there’s no emergency stops. And so, to really hit it home, where are we storing all this time series data in a long-range historic store? And that is a time series database. And there’s four qualities, I believe, that make a good time series database. Of course, the first-class citizen in any time series database is time series data. So, timestamps form part of your primary key. It allows you to retrieve and order your data in timestamp order.

JAY CLIFFORD: 15:09
Another key quality is high write throughput. So, the ability to ingest data from multiple parallel sensors, Kafka Streams all at the same time and store all of this data in real-time, then we also have the ability to query that data and perform effective query ranges upon that data, really looking at trends in that data. So, aggregations on, say, the average mean of data over the course of a day, looking at account, looking at a derivative upon that data. So really returning groups of metrics or groups of points and performing some form of aggregation upon them. And then, finally is of course inherent scalability, being able to ramp up your database as your solution expands. So as the number of cyclists come online, we should be able to scale accordingly. And I would not be a DevRel for InfluxData if I did not talk about InfluxDB 3.0. So just for people that are currently using open-source or haven’t heard of InfluxDB 3.0 yet, this is the new architecture of our time series database. We’ve built it upon open-source technologies. You have DataFusion here as your query scheduler, Apache Arrow as your in-memory data formats. This gives you millisecond return times on your queries. And then we have Parquet as our open data format for storage. So rather than using things like TSM and TSI, this gives us an open data plane where we can interact with things like Delta Sharing, Apache Iceberg. So, you can interface with things like Snowflake and Databricks to really create that real-time data store and merge it with your data warehouse solutions.

TUN SHWE: 16:58
Yeah. I love this architecture. And I should say Influx 2 and 3 are extremely different, right? Because Influx 3 is built on Rust, all these open standards. Love it. Just love seeing that. And like Apache Arrow, having columnar-based memory store, perfect.

JAY CLIFFORD: 17:17
Love it.

TUN SHWE: 17:19
Great. So, I’m going to talk about Quix next. So, these are the two products that we’re focused on. So, we have, first of all, Quix Streams, which is our open-source cloud native library for processing data in Kafka using pure Python. So, anyone that knows about streaming technologies knows that it all comes from a Java-based landscape. So, Java virtual machine type languages. We take the Python-first approach because modern data professionals like data engineers and data scientists, they all work in Python and we wanted to serve that whole ecosystem and that community of developers. You’ll see in our live demo later that it was inspired by Function-as-a-Service, like Containers-as-a-Service, because the concept of each application is that it’s containerized and we are heavily inspired by Pandas. You’re going to see a lot of data frames flying around there. And we have Quix Cloud. So, you’re able to write Quix Streams applications and deploy them yourself, or you can use our managed cloud for that. It’s a fully managed cloud. It comes with lots of nice things, like an inbuilt library, which is also open source, actually. It has its own repo with various connectors and code samples to get you started really quickly. And we’re going to see the InfluxDB connectors that Jay contributed. So, thank you very much for that. So yeah, that’s a great way to get started. Next, I’ll talk about the overview for the platform architecture. So, you have Quix there, which is built on those three kinds of core technologies that you’ll see at the bottom in the logos. So that’s Kafka there. So that’s the data backbone. So, all the messaging inside of Quix is done via topics in Kafka. We dockerize all the applications, so everything is containerized and orchestrated using Kubernetes. And we’ve got Infrastructure as Code, kind of like Pipelines-as-Code and versioning, all handled via Git that we either manage for you or you can bring yourself. So, we’re kind of slowly unbundling all our components as well.

TUN SHWE: 19:11
We’ve got the sources there on the left, destinations on the right. You see Influx’s logo there in the destinations. But as I mentioned, it’s also a source. And if Jay wants to contribute more transformations, then he’s absolutely free to do so. But those are all open source. It’s a place that we’re hoping to grow the community with. So, if you work for a vendor, if you use any of these sources and you want to integrate your data, we’d love to have you get in touch, and I’d love to get you started contributing and connected there. And a bit about why the Quix Cloud exists, because building your own architecture is costly. So, there are a number of times where I’ve landed in a new workplace as the first data hire, and my task was, “Right. We need to build this platform.” And the first thing you need to do is build out a team, maybe upskill some of the teams, and in many cases, you end up with a dedicated platform team. So, it can usually be a very large team that manages Kafka and Kubernetes and all the different data infrastructure components that you need. It takes usually many months to get to even a first release and to start building and realizing your data products and having them out in the public. So, this is where Quix comes in, we aim to speed up that application development. There we go. And we try to bring that down to a matter of weeks. Jay, for example, when you started building, I think weeks is even extremely conservative. It’s more like a day. We’ve got a really good connector ecosystem now, right? So, you can get started quick. So, what was your story there?

JAY CLIFFORD: 20:43
That was the thing for me. I mean, when looking at 3.0, we basically wanted to sort of create sort of a third-party task engine. Came across Quix and literally, I didn’t know Kafka that well. I’ve deployed it once locally open source myself, and I just wanted to focus on building Python code and building out the different components of my data pipeline. And I think that’s what made me fall in love with Quix, right? It was just the ability to basically orchestrate and create containers without even thinking about it. Just focus on the Python code. And then have the data feed between my different containers was literally like 2 hours’ worth of work. It was amazing. So, I hope you can see some of that today.

TUN SHWE: 21:27
Amazing. Thank you. Yeah, I’ve just realized my own personal metric that I have for myself is to increase the number of people that say, “I fell in love with Quix.” What you just said, that’s one. Got to keep going.

JAY CLIFFORD: 21:40
So, I guess the big question here is, how can we combine Quix and InfluxDB to create a solution? And I think the first thing that we need to do to be able to create a solution is to define a problem. So, Tun, can you define a problem for us, maybe make it a little bit festive?

TUN SHWE: 21:58
Sure. Let’s go for it. Yeah, not going to be my first festive problem today. I had one earlier, but here we go. So, Jay, you are now the CEO of Packing Co.

JAY CLIFFORD: 22:07
Great. [laughter]

TUN SHWE: 22:09
Exactly what you’ve always wanted to do. So straight onto the management track. So now you’re CEO of Packing Co. And you’re having recurring issues with your packing machine. So, as we gear up for the festive season, many people are going to be exchanging gifts, and they need those gifts wrapped. So, your machines there with those robot arms are doing all the gift wrapping, and you’re now scaling that out. But unfortunately, any of those machines can enter a failing state, and it requires a manual reset by an engineer. So, you can see one of Jay’s human engineers in the top left. He’s got to run over to that machine and press a button on it to reset it so that it returns to normal operation. And we’ve been speaking to the subject matter expert here, who is the plant manager, who has advised that when running normally, all machine sensors will follow similar output patterns. If a machine is at fault, these will fluctuate abnormally. So, we’re able to detect some anomalies there. So, this could be interesting. So, the question is, how can we use Quix and InfluxDB to solve this?

JAY CLIFFORD: 23:07
So, I’m literally a victim of my own success, and if I can’t get this done in time, there is going to be a lot of sad people in time for the holiday period. Great. Good to know. This is great as my first sort of role as CEO, that I’m already facing problems in my packing line. So, let’s solve that problem then. The idea of what we’re going to build together today is we’re going to keep it as an interactive demo. Here is a snapshot of the architecture me and Tun have built based on the Quix and InfluxDB architecture. So, what you can see is we’re basically feeding our machine data into an MQTT broker. We’re using HiveMQ, in this case. A major thanks to the HiveMQ team. We’re going to be using Quix to collect that data from the HiveMQ broker. We’re then going to write that data into an InfluxDB plugin or a destination plugin that writes that data to a machine data table. This is going to contain our raw data over time. From there, we’re going to draw that data back out, bring it back into Quix using one of the other community plugins called the InfluxDB query plugin, which allows you to query the historic data out. We can then stream the data frames through Quix into an ML model that we’ve been hosting on Hugging Face. And then we’re going to write that data from the destination plugin—sorry, the results from the ML model through the destination plugin, and into the ML results table. And then we’re going to use Grafana to tie all of that together where we’re going to visualize the raw data and also the ML results. So, let’s strip it back. Let’s first have a look at data ingest. So, I’m going to jump into the demo now. What we can see is if I connect our client here, so forth, you should see that we have data flowing into our system.

JAY CLIFFORD: 25:05
For that, I am hoping. There we go. And you can see it’s all in JSON format. We have a series of machines and child topics, and they each have different sensor values such as temperature, load, power, and vibration. And some metadata that we can use as part of our schema. Now, the first step on our journey is actually jumping into Quix and having a look at the pipeline here. So, while I turn these on, Tun, would you like to give them a quick tour of Quix?

TUN SHWE: 25:33
Yeah, absolutely. So, what you’re looking at now is Quix Cloud. So, we’re in the platform. Let’s start off right there at the top. So, we’ve got production there. So, we have the concept of different environments. So, you would connect your Git, or use our own managed Git, and you could have different branches for different environments. So ordinarily, we’d have like a dev, a staging, a demo, a production branch, but we’re pro so we go straight to production. If you click on code samples on the left, let me just do—I’m going to respond in real-time. So, I saw a question come in about code samples because someone had asked about if it has inbuilt transformations. So now you’ve got a filter here for different types of transformations, so you can check that button and you’ll see only transformations. So, you can explore there. We can search for InfluxDB in here. So here are some that. Jay actually wrote for us. So that’s there. Let’s go back to the pipeline view. So, this is a good way to get started. So back in the pipeline view. Let’s just describe what we’re seeing here.

JAY CLIFFORD: 26:32
Sorry, just getting—

TUN SHWE: 26:34
Getting a bit of lag. Yeah.

JAY CLIFFORD: 26:35
The festive gremlins. Sorry, just—

TUN SHWE: 26:37
Oh, here we go.

JAY CLIFFORD: 26:38
There we go.

TUN SHWE: 26:39
No worries, no worries. Okay, so now we’re back in the pipeline view. So, each of those nodes is a container in Quix so it’s backed by a Dockerfile and is containerized and deployed inside the Quix platform. You may see some of those gray arrows turning green, that means that topics are being filled with data. So there, that is the raw data topic that Jay is mouse hovering over. And you’ve got 4.7-ish kilobytes per second flowing through. The connectors there in blue are data sources. The connectors in orange are the destinations. And the ones with the purple edge are transformations. So, if we focus on the bottom left there, we’ve got the MQTT one. So, shout out again to HiveMQ, our friends there, where we are passing machine data through. So, all the robot arms are sending the data to the HiveMQ broker, and we are using the Quix MQTT connector to load that data into Kafka and into Quix. So, you see the data flowing through. So, if we click through, we’re going to see the code. So, this is kind of like the cloud IDE view, so this is really good for teams to collaborate on. You can see there on the left-hand side, it’s very much like a Python project. We’ve got a main.py, which has all the logic in it. How Quix works, and how it’s different to a lot of, maybe, other libraries, is that Quix Streams is based on handlers. So LIKE function as a service or if you’re more familiar with AWS Lambda, you define an on kind of handler. So, whenever a new event flows through the broker and is consumed, it will trigger that and run that function. And we inject in all the things, the consumers, the data frames, everything you need to get going there. So that’s how these work.

TUN SHWE: 28:26
You can bring your favorite Python libraries as well. You may have spotted there’s a requirements.txt file. You can put anything you like in there. You’ll spot that we’ve got some open telemetry libraries coming in there. And we’ve got a build folder, which is where we keep our Dockerfiles. So, we supply one to build for you. So, we have automated CI/CD inside of Quix Cloud and you can modify and change the Docker image yourself if you are well versed with Docker. So, should we go back to the pipeline view so we can see data flowing through? And it’s being written to influx into the machine data.

JAY CLIFFORD: 29:02
Yeah, thank you. I mean, this is one of the main things for me. It’s like you can get started with all of these code samples and just get going building a pipeline. And then I think this is what makes it a little bit different from things, inherently, because to me, when I look at visualization like this, it feels like Node-RED. It’s the complexity you can really get into here and the amount that you’re scaling, it’s all technically building a microservice architecture all in its own containers. And you could really get into the weeds here if you want to. And it’s code first. So, I really, really like this design. So, when we have a look at our data being written in from Quix, let’s jump into our data explorer in InfluxDB. So, we can see our bucket there factoring where we’re writing our data to. If we jump into our measurement there, our machine results, and look at the data from the last five minutes, you can see here that we have the sensor values that we saw before, including our metadata. And our metadata now forms part of our tag keys. And this essentially just forms part of our primary key and allows us to separate one machine’s data from another machine’s data. So, if they share the same timestamp, they’re not going to overwrite one another. And that’s kind of the main function of tags within InfluxDB 3.0. So now that we have the data flowing in, let’s start working with that raw data. And I think the next thing on our step is to actually start choosing a model and how we can operate on that data and start spotting those anomalies within our data.

JAY CLIFFORD: 30:38
Now I am going to steal a quote from you, “Less is more.” And in this case, it’s not always reality, unfortunately. And in this case, what we hope to see a lot of the time is when we’re in a solution like this, is if there’s a problem with the robot, we would see a sharp increase in the vibration data. It would jump over a certain threshold and stay continuously constant at this level. And we could basically just use threshold check-in, which I believe, yes, less is more. If we could just use threshold check-in, we should just use that rather than doing an overkill sort of ML model for that case. So, in reality, we kind of have something that looks a little bit like this.

TUN SHWE: 31:25
Yeah, that’s right. So, I’m going to take your quote now. So, “Success comes from the domain expert.” So, paying it back. So, if you cast your eye on the graph in the middle there, that robot has entered into anomalous behavior and you can’t really see the axes, but if they’re tiny, like kind of point differences, your thresholding and your range query checks aren’t going to catch that. You’re going to get too many false positives. So, what do we normally do in this situation when results become unpredictable by conventional means? You turn to your domain experts, which in this case is a data scientist. So, you may have some of those roaming around in your company, or I like to call them friendly neighborhood data scientists, who are roaming around in coffee shops, just tap them on the shoulder, explain your problem. In this case, we’ve done that, and we found that we need to use auto encoders. So, I used to be a data engineer, and am a data engineer, and I only play a data scientist on TV, so I’m not going to claim to know exactly the intricacies of this, but an auto encoder neural network is an unsupervised learning algorithm. And it’s designed to learn a compressed representation, or an encoding, of that input data. And so, it’s able to form these predictions better. So yeah. We move on to the ML deployment. So, we are going to deploy a model that we have trained using—we have trained this model offline using a notebook, and we’ve pushed it to Hugging Face. So Hugging Face, for those who don’t know, it’s kind of like Docker Hub or GitHub for models and data sets. So, we’ve got an example there on the left of a popular, large language model that’s open source, called Falcon. On there, there’s a model card which is a great overview of what the model does, how it was used to be trained, what data was used to be trained, what data it’s good for working with, things like that. And we’ve got on the right there, this is Jay’s account, where we’ve pushed the vibration auto encoder, and we’re going to be pulling that down into Quix, running through CI/CD, building that image, and deploying that model inside of Quix.

JAY CLIFFORD: 33:29
I always like to look at my Humble downloads compared to Falcon when I see this. Most of the time it’s just me pulling down the vibration auto encoder, so I’ll take the 30 downloads compared to the 161,000.

TUN SHWE: 33:43
[laughter] Yeah. People are finding out about these two models every day. So yeah, it’s going to go up slowly together, right.

JAY CLIFFORD: 33:52
So, when we basically put this together, this is kind of how our deployment will now look like in Quix. We’re going to use that query node to basically pull the data back in. We’re going to use the ML model deployed from Hugging Face. So essentially pulling that model from Hugging Face, bringing it into Quix, and running the model directly in Quix. We’re then going to push the results of that model, using the destination InfluxDB plugin, into the ML results table. So, let’s have a look at that within Quix. So, if I jump back into our Quix architecture here, you can see that we’re actually querying the data out, and you can see, based upon this, this looks very similar to a Pandas DataFrame. So essentially what we’re doing is we’re streaming dataframes from one container service to another. So, we’ve basically got our time, our vibration, and the machine ID, as you can see there. If I jump back out and then jump into our pipeline. If we have a look at our anomaly detection node, what you can see that we’re doing is, based upon the model that Tun sort of explained was the auto encoder. We’re essentially deriving thresholds, basically about the anomalies within our data. So, what we’re doing is we’re calculating an MSE percentage, and then we’re also enriching that with the threshold, and the amount of—how anomalous the model perceives the data to be. And so, you can see in this case, the lower the MSE percentage, the less confident the model is that there’s an anomaly occurring. And the higher, the more anomalous that it is. So yeah.

TUN SHWE: 35:37
So, I mentioned Airflow and Prefect before. So back in the pipeline view, this is kind of how—one way I describe Quix is, we’re a bit like Airflow for streaming, because you have these familiar dags—these directed acyclic graphs—and they’re all joined together in some way, in this case by Kafka. So yeah, we’re able to basically perform these workloads much like Airflow does. So, in that anomaly detection step, we are essentially computing two new columns. And as Jay mentioned, MSE, the mean squared error, we’re able to compute new columns and send that into InfluxDB again into a new result measurement table. So that’s generally how the flow goes. You read data in the model, gives you a prediction, and kind of generally it’s confidence we create new columns from it.

JAY CLIFFORD: 36:25
Yeah. And I think in this case, the awesome thing here is that you can actually run these models directly in Quix. And that’s what that direct integration with Hugging Face gives you, just that ability to automatically push a new model, deploy it directly, which is brilliant.

TUN SHWE: 36:44
Yeah. And it’s all code. So, you can use the library any way you want, and you can configure it any way you want. But we do have turnkey integration for Hugging Face. Yeah, that’s a good point to make.

JAY CLIFFORD: 36:55 Love it.
So, what I did there secretly, as Tun was explaining the pipeline, is I actually asked our festive gremlins to create an anomaly within one of our machines. So, I toggled an anomaly in our machine here. And so, what we should start to see if I now toggle auto refresh on here. If we bring, hopefully in the last data, you can see our vibration data from machine one. So, this data that you see here is the machine functioning in normal process. It’s essentially collecting a box, it’s wrapping, it’s slapping a bow onto it, and then it’s pushing it onto the conveyor. Job well done. We have a happy festive season with a present being delivered. Now, this is not so good. This is when our machine is acting irregular. And you can see the interesting thing here is most of the time the data is within a particular range compared to the positive. So, we really care more about the pattern in our data than a particular range of it being higher or lower. And as you can see here within the anomaly detection in yellow, we’re overlaying where the model perceives the data to be anomalous. So yes, we did get a few false positives, but in general, the model has been trained well enough to understand that this data coming in is extremely anomalous. We can actually aggregate that data here, which is what you can see. So, this is a count over the last five minutes, the amount of anomalies that we’ve seen. And then we’re also plotting the MSE percentage. And you can see a sharp increase as we go of the MSE percentage going higher because we’re seeing far more anomalies being discovered in our data. So, jumping back into the slides. I think it’d be good if you could sort of bring us back home a little bit and kind of sort of take us through what we’ve solved in a human way, which would be great.

TUN SHWE: 38:53
Definitely. Thank you. So yeah. So, Jason Eck was on the line as CEO, but I think with how things have gone and being able to detect these anomalies, he’s back to normal operations. So Pakinko is happy. And what we’ve done as a summary is we’ve taken all that data from those robot arms and thanks to HiveMQ and through MQTT, we’ve been able to ingest and transform and store that machine data. We’ve deployed an initial machine learning model to detect these potential malfunctions using vibration data from the machines. And we’ve provided the foundations of a scalable data pipeline. And I guess the end thing to realize there, the human part of it, is that the plant manager now has a dashboard so he can pass that Grafana to the plant manager and they’re able to see when anomalies have been detected and instruct the engineer. So, yeah, we have saved the holidays. Right?

JAY CLIFFORD: 39:46
Well, Tun, hold on a minute. So, we’ve sort of mentioned here about a scalable data pipeline. So, I think we might become the victims of our own success. And I think we did mention in the title of the video that we would be discussing observability and open telemetry. So, what exactly can we do to solve the visibility problem of our data pipeline?

TUN SHWE: 40:12
Sure. Okay, I’ll give you a little history of observability. It’s kind of more my personal experience of it, and hopefully, it resonates with some of you. Yeah. The thing to know is I’m old, so I’m going to go back a few years to show you what we mean. I’m going to wait for the slides to catch up. So, what is observability? So, there are a few definitions here, and we’re going to go through kind of three different layers. So, the first one is in control theory. Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. Sort of makes sense. Let’s go down to distributed systems. So that’s where it’s the ability to collect that data about programs, execution modules, internal states, and communication among components. I like to take the human-centric approach to observability, which is that it is maintaining a full understanding of our systems. So, this is me back in 2010. I was happy logging files, saving them somewhere. How was I to know this was not observability? But I was like, “I know exactly what’s happening in my system.” So back in 2010, I was already building applications and systems on the cloud. And what normally happens when you encounter an error, or when you need to monitor, is you would SSH into the box. So, you’d open a secure shell, you would connect to that box, you would navigate through the file structure, and you would find where the log files are, and you would read them out, or you would tail them just to see what was happening on that machine. So, this was a very manual process, but it felt good. It was busy work. But it felt good. And you felt like a God, because you’re like, “Right, I can connect anywhere. I can see anything. I have the all-seeing eye.” Except when you start building distributed systems and distributed applications, how do you know which box to SSH onto? You have no idea where any of these logs are going.

TUN SHWE: 42:01
So, it became a bit of luck, really. It’s just like you just hoped that your next jump is going to be on the right box with the right logs telling you what the problem was. So that wasn’t so good. So, let’s fast forward to 2015. And that gave rise to something called the ELK stack. So, which stands for these three services, so Elasticsearch, Logstash, and Kibana. So Logstash was your way of capturing those logs from your machines and then shipping them into a central location, which in this case was a database called Elasticsearch. And as the name would imply, it allowed you to search through those logs so you could easily figure out what was happening. So, you got centralization, you’ve got search, and you’ve got Kibana, which was a way of creating metrics. So, you create dashboards and metrics around those logs so that you can understand performance, latency, and so forth. So, 2015 was a good place because we’re starting to understand our systems, except that the whole world realized that a distributed monolith is not what you want. You actually want to build microservices. So, things became more complicated. So along came Jaga. So, Uber released this in 2015. And then it was followed shortly by OpenTracing, and then Google released OpenCensus towards 2017. So, these are all solutions for understanding distributed tracing. And this is all well and good, but they were all kind of pulling in different directions towards the same mission, which didn’t make sense. So OpenTracing and OpenCensus merged together, and in 2019, OpenTelemetry was born. So that’s the short history, really, of OpenTelemetry, and brings us kind of to current day. And current day it looks like this, it’s the unification. So, I mentioned logs, metrics and traces, and you can have them across various different systems. You’ve got Elasticsearch there, InfluxDB there, Prometheus over there.

TUN SHWE: 43:55
But in 2023, I find it’s much simpler to use something like OpenTelemetry, where you can think of just one protocol like GRPC or HTTP and push them all to the same place, which in our example here is InfluxDB. And yeah, this is a pretty simple illustration of what a distributed architecture with microservices might look like, but you may have a multitude of different services that you can instrument with OpenTelemetry.

JAY CLIFFORD: 44:24
Love it. And I’m going to sort of jump across this a little bit quickly, just make sure we still have time for questions at the end, but if we have a look at a bird’s eye view of OpenTelemetry, it’s not just a standard. What we also have here is a toolkit. It’s a fantastic toolkit for orchestrating, as Tun mentioned, the ability to create traces, logs and metrics, ingest them into a service called the OpenTelemetry Collector, which in itself is its own data pipeline, and then write that data to a variety of endpoints, usually some form of data source, whether it be segregated in this way, where you have a time series database for metrics, somewhere to store your traces and logs, or all-in-one purpose general store like InfluxDB. What we’re going to quickly cover is the ability to orchestrate Quix using the OpenTelemetry SDK. We’re going to write all that data to the OpenTelemetry Collector, and then we’re going to basically interact with those traces stored within InfluxDB. Now, quickly on what a trace is, you can imagine a trace in the human world like a delivery. So, imagine you ordered something on Amazon, and it had to come to your house. So, the parcel would first be tagged in a sorting facility. It would then be driven somewhere. It might go to another sorting facility before it arrives at your final location. That entire process is what we consider a trace. Within that, we have a notion of what we call child spans, and so spans is a particular act within a certain microservice or a certain—or a—sorry, a certain microservice or a certain line—or a certain sort of function within that microservice. JAY CLIFFORD: 46:13 So if we take our human world example here—say we’ve hit the sorting center; the box gets sorted a few times and pushed to a different location to be sorted onto a van—that would be considered a span at the sorting location, and what we can do here is we can actually use the ability of looking at traces and spans to really drill into our microservice architecture, work out what’s going on at different points, and understand where our code might be going wrong, or where services might be slow or bottlenecked because of what we’re feeding to them. So, if you’ve used Telegraf before, then you’ll be quite familiar with how the OpenTelemetry Collector is set up. It’s all written in Go. It’s created with a series of plugins that are all defined in YAML. You have receivers, processors, exporters, connectors. Receivers, of course, ingest data. So OpenTelemetry is push-only in most cases, and then you have this notion of processes for enriching the data. And then you have this idea of connectors, which is an interesting concept, basically connecting pipelines together. So you could do something like span metrics, where you basically create metrics about your traces and then you feed them back into the OpenTelemetry Collector as metrics rather than traces—and that gives you this idea of being able to basically use traces to formulate metrics or to formulate logs—and then finally, of course, exporters, where you want to finally write your traces, logs and metrics to. So how do we combine OpenTelemetry with the rest of our stack? Well, what we’re going to do is we actually orchestrated the OpenTelemetry SDK into each of our services, and we’re writing that to the OpenTelemetry Collector, which we’re just hosting in this instance, in an EC2 on AWS, but it’s something that you could host in Quix, right, Tun? TUN SHWE: 48:13 Yeah, we could do, yeah. It’s something that we’re looking into because—I mean, they’re just Dockerfiles. For security reasons, we tend to lock everything down. So, whilst there is public access, we don’t expose all the ports. So that is on the roadmap, and it’s something that we want to change over time. But yeah, we tried, and I’m hoping for that feature to come in, then we can host any arbitrary container in any language, basically.

JAY CLIFFORD: 48:39
Awesome. I think that would be super cool. So, in this case, I just want to take you through a little bit of an intro to the Python SDK for OpenTelemetry. So essentially here, we’re including the SDK libraries. What we’re doing first is, we’re creating a resource, and that resource attribute is a service name. So, it’s basically a big, fat label for our service that we can use that within our dashboard to understand where our spans and traces are coming from. From there, we’re basically coordinating where we’re going to be sending our traces to. In this case, we’re going to be exporting over HTTP to the collector. And then what we’re doing after that is, we’re actually creating a tracer. So, the tracer object allows us to formulate spans underneath. Now, what we can see as we jump down, we start using this tracer object to create a span. And we define that span here as data frame clean because we’re cleaning the data frame. And then a span can also have child spans associated with it. So, as you can see here, we’re then having a child span, which is called predict, which is us actually predicting with the model. The cool thing that we can do is if we actually have a look, if I can find one that we did earlier, you can add span events. And span events are basically primitive logging within spans. So, you can basically add data to your spans as you go to pick up on particular events within your data. So, we’ve orchestrated all of these spans throughout our services. We’re pushing these all through the OpenTelemetry Collector. We’re then going to send these to InfluxDB. And we’re actually going to use Grafana over the top to interpret these.

JAY CLIFFORD: 50:27
So, what I have here is another Grafana instance. If I refresh, we can actually see the series of service names. So, we have our MQTT, our ML, InfluxDB exporter, query, anomaly detection. If I click on anomaly detection, we can see the duration of each of our traces within our data here. We can see the service latency over time as a histogram. But where it gets really interested is when we can start drilling into our traces. So, let’s click on the most recent trace, so you can see based upon that—based upon the anomaly detection service, these were those nested spans that we created. You can basically see the anomaly detection data frame clean, predict MSE calculation, and publish prediction. Now, all of these are happening within one service. But normally what happens is these span over multiple services. So, you can see the full route of your data. But in this case, we’ve just kept it really simple so you can see what one service is doing. And from there, you can actually see what a visualization of our traces looks like. So, you can see now we have our anomaly detection data frame clean. That has a series of events associated with it. So, I’ve put the data frame size and number of time steps that we’re basically chunking the data into. You can see free prediction. And then in the MSE calculation, you can store just about anything because event logs. In this case, this was probably a little bit too much, but I’ve basically stored the array, the NumPy array of every single prediction that the model was making. And then finally the end result, which was the publish, which I think just has an end result in the data frame size. So that was kind of a whistle-stop tour of OpenTelemetry.

JAY CLIFFORD: 52:19
And the idea, the great thing there for us is that as we expand our data pipeline if we keep this methodology in mind of constantly basically monitoring our pipeline using basically an open-source, open data plane for observability like this, it gives our SRE teams, the observability guys or girls within your team, the ability to be able to monitor your data pipeline as it expands. So, Tun, would you like to take us through next steps and how they can get their hands on the lovely source code?

TUN SHWE: 52:58
Definitely. Thank you. That was very cool and on par on coolness with this next chap in the next slide, Morpheus from The Matrix, who asks, what have I told you we can do better? So here are some things that we could do next to extend this. So, we use the vibration data and we use auto encoders. So, we could potentially use Quix to deploy several models in parallel, focusing on different things. So, we could start analyzing temperature and other bits of telemetry data. So, we could run those in parallel and eventually, we could close the loop. This is Jay’s favorite, where we remove the human completely from the loop. The accuracy gets so good that the machine itself knows when the anomaly is happening, and it presses the button by itself.

JAY CLIFFORD: 53:42
Love it.

TUN SHWE: 53:43
And yeah, it’s all well and good seeing this on Jay’s machine. I had this running on my machine earlier and it was great. So, I implore you to do the same. Try the demo yourself. There’s the QR code in the link. And yeah, that was the last slide. So, Jay, do you want to talk through where people can go to find out more about InfluxDB and learn more?

JAY CLIFFORD: 54:04
Yes. Sweet. I mean, one thing I always suggest taking away from all of this is join our community. As we said, this is where I met Tun, and this is where we’ve collaborated upon this and our platforms together are growing strength to strength. So, if you have any questions about the webinar today or just wanted to chat about your own solution with InfluxDB or Quix, I’m in Tun’s community, he’s in mine. So, this is a fantastic way to chat with us.

TUN SHWE: 54:36
Yeah, my slide says the same thing. So, I think 2023 is the year of bring your own cloud. So that is a potential solution for running Quix cloud as well on your own infrastructure. Several action points here, but the one I would tell you to focus on is the community one. Please join our Slack. I’ve seen some good questions. We will have some time to answer. But if you ask them again in our community, we are there to help you. And please do follow Quix Streams. That’s our open-source library on GitHub. If you like what we’re doing, we’re about to release a lot more new features as we go out of Alpha for version two. So please star us if you’d like to follow us there. So yeah, that’s it from me. Thank you very much.

JAY CLIFFORD: 55:18
Love it. Caitlin, I think—

CAITLIN CROFT: 55:21
There’s a bunch of questions.

JAY CLIFFORD: 55:22
Yeah, let’s go for it.

TUN SHWE: 55:24
You are a Grafana wizard. I love that. That’s true.

CAITLIN CROFT: 55:30
So, there is someone here, Peter, who has raised their hand. So, Peter, I have enabled it so that you can unmute yourself. Looks like you had your hand raised, so feel free to unmute yourself and ask away.

ATTENDEE #1: 55:48
Sorry, I just mistake in the—

CAITLIN CROFT: 55:55
No worries.

ATTENDEE #1: 55:56
I didn’t notice. So, I lower it down.

CAITLIN CROFT: 56:00
Okay, no problem. I’ll meet you and we’ll go through the other questions. All right, so the first question is, can you compare using Quix versus Telegraf? It seems like there is a lot of commonalities between the two, but lately I’ve been hearing a lot more about Quix. It would be great to understand the difference and when to use which one and where.

TUN SHWE: 56:27
Jay, to answer this one. Yeah, because we talked about this actually in the architecture diagram, right? Because I even said, “Where is Telegraf in this?” Because that’s always a possibility.

JAY CLIFFORD: 56:38
Right. And I think that’s the cool thing here, is even with Quix, we could actually even host Telegraf within Quix since it’s a container, and we could basically use that as an input data source. I think when it comes to the differentiation here is you have to host Telegraf. It’s a collection agent that you’ll run on a system or a machine or even host it in your cloud. But the idea here is it’s an agent that pulls in, it’s a single node instance with a variety of plugins. I think where Quix really steps up the mark in terms of—or differentiates itself is it’s not just a data collection agent. It provides the ability to have real-time access, but also that HA ability to be able to collect data without the worry of a single agent falling over. And so, if you need that real-time ability to look at your data and make sure that your collection and also your processing of that data is fault-tolerant, then that’s really where Quix comes into its own. And I think where people really excel with Quix is where they see it as more than just a collection agent, but also something where you can build out a task engine, build out data enrichment, and you have that programming methodology in mind where you can basically build Python code applications from the ground up.

TUN SHWE: 58:02
Thank you for that question. That was a good question. We often get compared to Kapacitor as well, which is part of the TICK stack, too. Quix is a processor, it’s a transformer. So that’s something else to look into, too.

CAITLIN CROFT: 58:17
Tun, I know you answered this earlier, but I’m just going to pose this question again in case you have anything else to add. So, someone said, “Can we say that Quix is an ETL tool? If so, does it have any built-in transformations such as outlier detection and missing value imputation with respect to time series?”

TUN SHWE: 58:44
Right. So, you picked up on the fact that it is ETL because you saw a reference to sources, transformations and destinations. So that comes with the territory. So yes, you can consider us an ETL tool. We don’t have those specific transformations. We have other ones. But that’s why everything there is open source. And that’s why you definitely should join the community. If you’d like to work with us on it or contribute it yourself. We do give rewards for contributions, credits. We’re going to think up a whole new program that involves Slack as well. So yeah, please do get involved. I’d love to know specifically what you’re working on and how we can solve that. So, in order to grow the community, because what we’re finding often is people do ask for the same solutions. It’s just if we can make them open source, it benefits everyone. So please do join the community on Slack and ask again. We’ll pick it up from there.

CAITLIN CROFT: 59:34
Cool. What kind of latencies are occurring at this point from the initial generation?

JAY CLIFFORD: 59:42
That’s a really good question. And I think where we introduce the most amount of latency is actually where we query it from InfluxDB back into Quix, because we’re basically polling that like every 15, 20 seconds we collect the last historic data that’s stored in InfluxDB. Up until that point, I think, basically MQTT has very little lag in terms of writing data from one endpoint to another. And then, as we sort of said, within Quix, it’s all based on a Kafka Streams, which is real-time data processing. So, latency up until that written point at when we were putting the data to rest is, I would say, in milliseconds. I don’t know if you want to push on that, but it’s probably something we could actually measure and reply in an email and say, this is kind of like the latency as we go. But yeah, I would say the biggest latencies introduced is when we query back in, just because of how we’ve done it.

TUN SHWE: 01:00:43
No, nothing to add. You got it.

CAITLIN CROFT: 01:00:47
Someone is asking, are there any open source versions of Quix?

TUN SHWE: 01:00:52
Right, so you’re probably referring to Quix Cloud, the platform itself. So, we did expand a couple of months ago. We offered bring your own cloud. So, we’ve got a few customers opting for that for various reasons, one of which is security. So, we can do that. So please get in touch with us via our website and we can help you with that. In terms of running it in an open-source way. This is much like InfluxDB 3 as well. We are working on a community version. What we’re trying to do is grow the community and find out what the community would want, and we’re guided by that. We have a few people who have asked. Jay is also someone that’s asked about a community version, and it’s something we’d like to bring. It’s on the roadmap. We’d ideally like to do it at some point next year, but the more people we have shouting about it and sharing kind of what they are trying to achieve, the sooner we can work on that. Because, yeah, the person who asked the question, probably very much like in Jay’s position, you have a platform team in InfluxDB you would love to run it yourself. Like you have that expertise. So, we’re seeing that a lot more. So, we’d like to do that.

JAY CLIFFORD: 01:01:55
My dream is just to be able to basically give people a Docker-compose file or some form of platform package that has InfluxDB 3.0 open-source Quix. There you go. Enjoy. Basically, create your home setup. I think that would be epic next year.

TUN SHWE: 01:02:12
Yeah. Because it’s Dockerized, we can get to that point, but not the bells and whistles of the lovely pipelines and the IDE views, but we’re going to work on that.

JAY CLIFFORD: 01:02:21
Yeah, I love it.

TUN SHWE: 01:02:22
Yeah. Thank you for that question.

CAITLIN CROFT: 01:02:23
And Tun already shouted this out, but someone said that you guys are Grafana wizards, so great job with the Grafana dashboard. [laughter]

TUN SHWE: 01:02:30
Yeah, it was all Jay. Jay’s the wizard. I’m learning from him all the time. [laughter]

CAITLIN CROFT: 01:02:35
And let’s see, our friend Angelo said, “Thank you for the great presentation. We already deploy Kafka on Kubernetes using Strimzi, and we write data to InfluxDB similar to how MQTT is used in the demo. If I want to use Quix for anomaly detection, would Quix replace Strimzi in my architecture, or can I use Quix integrated with Strimzi?”

TUN SHWE: 01:03:00
Interesting. Great question. Thank you for that, Angelo. So, the answer is, because you have essentially got a self-hosted Kafka, you can actually bring that with you. So, you can generate keys, get a certificate, and you can actually create a brand-new project inside of Quix using your own Kafka that is hosted on Strimzi. So, you can do that. Or you can obviously use our own, but we enable you to bring you any Kafka instance. So anywhere it’s accessible in the cloud, as long as you make it accessible, you can hook it in, no problem.

CAITLIN CROFT: 01:03:34
Awesome. Thank you everyone for attending today’s webinar. We have gone over time, which is totally fine. Yeah, it’s being recorded, so if you had to step away, don’t worry. Thank you, Jay and Tun, for this fantastic presentation. I think it’s fantastic. I think a lot of people will be checking it out. Love the Christmas sweaters. The holiday sweaters. You can’t go wrong with Star Wars or Mutant Ninja Turtles. So great job. Love the sweaters, love the festivities. To everyone on the webinar still, once again, it has been recorded. It will be made available by tomorrow morning. I really appreciate everyone joining. If anyone has any further follow-up questions, please feel free to email me. You should all have my email address, and I’m happy to loop in Jay and Tun. And of course, you can always find them both in both communities, so there’s plenty of ways to reach out to us if you have any follow-up questions. Jay, Tun, is there anything else you’d like to add? Anything that you’re really excited for this Christmas, this holiday season?

JAY CLIFFORD: 01:04:42
I just think no matter if you’re celebrating or not, have a wonderful, wonderful festive break if you’re taking it. We’ve had a really great fun doing this. It’s got us into the spirit of things. We love to have all feedback. So yes, leave us pull requests, leave us messages. We would love to keep working on this and build a better presentation and demo in the future.

TUN SHWE: 01:05:07
Thank you, Jay. Yeah, we’re nothing without this community. So yeah, please do get involved. We want to help with the education and get everyone to succeed with real-time data and time series databases. So yeah, please do reach out. Yeah, I’m just going to have a lot of rest. You probably saw me coughing on mute back there, so I’m in recovery still because we’re deep into the holiday season for parties. So yeah, I hope everyone has great celebrations and a lovely festive period and we’ll see you in 2024.

CAITLIN CROFT: 01:05:35
Awesome. Thank you. And I’ll just leave on this note. Andre said, “Excellent presentation, very informative and useful. Thank you.” So, on that note, thank you everyone for joining, and we’ll see you for the next webinar. Thank you.

[/et_pb_toggle]