Coming soon! Our webinar just ended. Check back soon to watch the video.
Webinar Date: 2019-01-24 08:00:00 (Pacific Time)
In this webinar, we will describe how to setup InfluxDB & Telegraf to pull metrics into your InfluxDB. We will provide an introduction to querying data with InfluxQL.
Watch the webinar “Intro to InfluxDB & Telegraf” by filling out the form and clicking on the download button on the right. This will open the recording.
Here is an unedited transcript of the webinar “Intro to InfluxDB & Telegraf.” This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.
• Chris Churilo: Director Product Marketing, InfluxData
• Katy Farmer: DevRel, InfluxData
Chris Churilo 00:00:05.140 All right. As promised, it’s three minutes after the hour. Thanks, everybody. My name is Chris Churilo and today we are doing our training on introduction to InfluxDB and Telegraf. And we have the lovely Katy Farmer, who’s one of our developer advocates, who will be conducting the training today. Just want to remind everybody I am recording this session and I’ll post it as quick as I can so that you can take another listen to it. In addition, we do trainings every Thursdays, so come back and join us on some more advanced topics. If you have any questions, feel free to post them either in the chat or the Q&A panel, either will work, and we will definitely get to answering all of them at the end of the session. But if we happen to have a chance midway through, we’ll also take a look and answer your questions as well. So with that, I’m going to take Miss Katy off of mute and hand the ball over to her so she can get started on this training.
Katy Farmer 00:01:04.421 All right. Good morning, everybody. I mean, good morning for me because I’m in Colorado right now. But wherever you are, good day to you or good night. I’m not sure what time it is in Amsterdam with Chris, but I’m Katy, and I’m just going to take you through some of the basics of InfluxDB and Telegraf today and try to answer any questions you might have. So our agenda is to go over some basic concepts—what is InfluxDB, what is Telegraf, and how do they work together? We’re also going to talk a little bit about how to use them inside of our visualization tool, Chronograf, and how to use them without Chronograf. So I hope that’s pretty straightforward. Let’s get started.
Katy Farmer 00:01:46.981 Okay. So InfluxDB is our modern, Time Series Database. Basically it was built to handle time series data specifically and to handle really high write and query loads. It’s got customized performance, specifically for timestamp data—so if we’re thinking about data where time is not only relevant but it adds value to your data. So for example, IoT sensors. It’s just as important you know when the event happened as what happened. So that’s the kind of data we’re talking about here, and InfluxDB was written with that in mind. So you can configure InfluxDB to conserve space on your machine and memory by setting retention policies. So you can only keep data for a defined length of time. If you know you’re going to need it in three months, you can just set it to expire, and then you don’t have to worry about it anymore. And you can just delete unwanted data or old data, and the retention policies are one of our most popular features because I know that it’s not that easy to delete unwanted data from other types of databases. And we also have a SQL-like query language that’s pretty similar. I only knew SQL when I started using it and I found it totally readable and understandable. It has a little bit of difference just to handle some of the time functions, but other than that, pretty good.
Katy Farmer 00:03:25.607 So let’s talk about what is Telegraf. Telegraf is our plugin-driven server agent for collecting and reporting metrics. We set it up using input and output plugins. So we set up an input plugin to collect metrics from somewhere, and then we set up an output plugin to send metrics somewhere. Basically, we’ve set it up so that we can pull metrics from third-party APIs, it can listen for metrics. We want to set it to be push or pull, so we’re kind of agnostic on that front. And on the output side, most of the time people are sending it to another kind of data store or message queue, but there’s a ton of things you could send it to. I recently have been working with MySQL. So I’m using MySQL as an input plugin and then using InfluxDB as my output plugin. So I’m gathering metrics off of monitoring my database and then storing them in InfluxDB using Telegraf.
Katy Farmer 00:04:34.695 So let’s talk about using Telegraf and InfluxDB with Chronograf. So if we’re talking about how to just install and get started, you’ll see the instructions here. If you’re on a Mac and you use Homebrew, you can also brew install our stack. So you could do brew install InfluxDB and that would work perfectly for you. It’s also pretty easy on most UNIX-based systems just using your package manager and your service manager. So an important thing that I sometimes forget is, after you’ve installed it, you need to make sure that you start the service. And you can find more instructions and guides on how to install it in our documentation. So if you start at this downloads page on our website and you’re having trouble, you can always go to our documentation. We have a couple videos available, I think, on how to get started and a lot of documentation from our excellent technical writers. And the same goes with Telegraf. Again, with Homebrew, you can brew install Telegraf. Make sure you start the service. And the point of this is developer happiness. We don’t want you to have to worry about installing and configuring for a long time. You should be able to do this in a couple of minutes without much trouble. And yeah, we just want to make sure that what our CTO calls Time to Awesome is really quick for you. So again, same process with Chronograf. You should be able to get the whole stack running in under five minutes. So brew install Chronograf. If you’re more interested in getting a specific version number or something, you’d go to the downloads page and get the version you would like. You can set it up to get a nightly version if you’re interested in being on the most current one.
Katy Farmer 00:06:46.441 All right. So after you’ve installed and started all of your services, the first time you open Chronograf, which defaults to your local port 8888, you should see this welcome screen. And you can set a username and password if you want, but you’ll see the connection string is localhost 8086 and then you can name it InfluxDB. One of the important things to note here is this bottom part where it says Telegraf database. So automatically, when you set up Chronograf, if you have Telegraf up and running, it’s going to configure Telegraf for you and keep some internal metrics from your machine in a database, and right here is where you’re naming that database. I do prefer to keep it named Telegraf just so that I can remember this is Telegraf using its built-in disk CPU monitoring. It’s just easier for me to remember what it is if leave it as the default name. But again, you could always change it to be my local disk or something. And then your option here will be to make this default source, and then you’ll add it, and you’ll get taken to the Chronograf screen. No, I’m actually going to exit out of the presentation and open up Chronograf here so that you can get a better feel by watching me use it.
Katy Farmer 00:08:29.751 So because I’ve already configured it, it’s taking me to the status page, which is just sort of its default home page. You can see there’s a news feed, there’s a new version of Chronograf, and then I have some various busywork going on in here. If we want to see what’s going on with that Telegraf database, we go to the Host List. And then you’ll see here that there’s a couple hosts from my local machine. Basically what I do is I go over to the Apps column, and I click System. And then now I’m viewing these default, built-in Telegraf dashboards that are collecting my local metrics. So I can see system load—something happened here; I’m not even going to worry about that right now [laughter]—the memory usage, system disk used. So you can see some of these are maybe more useful to you than others, but just having them built in is a nice way to get a feel for how to see your data. So this is how we view sort of the default, but we can also go to the Data Explorer. I clearly have been doing something interesting here before. But if I go to Telegraf, I can select some things here. And we’ll talk more about this later but I just wanted to show you kind of basically how you can navigate. There’s Dashboards button where you can see the dashboards that I’ve made in the past. There’s an Alerting button where you can set up alerts via Kapacitor, which is a different training, so I wouldn’t even worry about it. Admin, and then Configuration, which is where we can see this source that we set up initially.
Katy Farmer 00:10:23.837 So that being said, let’s go back to the slides so we can see if I’ve missed anything. I just wanted to show you what it was like to really use Chronograf and navigate around, but you can see here that these have some more interesting disk usage [laughter]. Mine has been pretty dormant because I’ve been traveling, but that’s okay. All right. So let’s talk a little bit about using InfluxDB. So there are a few things to know when it comes to a time series database schema. It can be a little misleading to think about databases that have no schema, like, “Oh it’s a schema-less design, and so I don’t need to set up migrations and that sort of thing.” But you still need to be very aware of how you’re storing your data and what that schema looks like for you mentally, and sort of if there are multiple people working on this database, you need to make sure that you all agree on what that schema looks like so you don’t end up with confused data and bad queries.
Katy Farmer 00:11:36.658 So some of the key terms we’re going to be talking about is a database that has one or more series, and a series is a time-ordered collection of value/timestamp pairs that share a measurement, tag-set, and field-key. Now don’t worry about that. We’ll talk a little bit more about that later. So if any of this is a little confusing, you can always reference this slide or reference our documentation. So the tag-set is a collection of tag-key and tag-value pairs for a point. A measurement acts as a container for fields and their type. The field set is the collection of field-key and field-value pairs. And a point is a combination of measurement, tag-set, and field-set with a timestamp. So points are written to InfluxDB using Line Protocol, which is something that we wrote here at Influx to make sure that even in the schema-less design data is always going on in in any consistent format. So you can see here the cpu_load is the measurement, the hostnames are the tag-set, field-set is temperature and volts, and then at the end is the timestamp. Again, you can reference all this on our documentation. The important parts to look at here is that you can see, in the tag-set, that you can just use comma-separated values to make more than one tag. So the hostname is server02 and then comma Arizona, us_west. So we have two tags going in. And the same with the field-set. So that was important to us, but although it is unique to InfluxDB, we want to make sure that it’s easy to read and understand. And the timestamp can be as precise as a nanosecond level, so that’s pretty exciting.
Katy Farmer 00:13:48.000 Okay. So let’s look at one example to kind of get some concrete ideas of what this is. So up in the top left, you can see the line protocol, which is these data strings being input in the bottom left. You can see what that would look like inside the database. So the retention policy is set to seven days. The measurement is census, so that’s the very first item in the line protocol, and then it’s the top line of the graph. The field-sets are butterflies and honey bees, the tag-sets are location and scientist, and there are two series in this example. The first series is census, and location, and scientist. And the second one, you’ll notice, which initially could be confusing, is also census, location, and scientist, right, but the locations are different. So one series is the same data but at location one, and then the second series is the scientist at location two. The shard group is all shards in the time range, and you can see the time range listed there. So basically, we’re just talking about a subset of series that are based on a hash of each series measurement and tag-set here. And we’ll talk a little bit more about shards later in case that doesn’t make enough sense. Again, you can always reference this to get an idea of what your schema looks like and what you would need to do to input it correctly.
Katy Farmer 00:15:45.063 So there are some considerations to make sure that your database is going to be optimized and performant. Tags are indexed and fields are not. So if you think that the data you’re putting in is also something you’re going to want to query by, then you want to make sure it’s a tag, because indexed fields are going to be query a lot faster than unindexed ones. Field types are also immutable, so you want to store data in tags if you plan to use them with GROUP BY. And this just goes back to what I was saying before. If you want to use aggregate functions like GROUP BY, you want to make sure that they’re in tags so that they’re indexed. That way you’re optimizing your queries. Store data in fields if you plan to use them with an InfluxQL function. InfluxQL is our query language. And again, we’ll talk a little bit more about that later on and about why you might need to do that. Store data in fields if you need them to be something other than a string. We do have some limitations. Time series data is more often than not numerical. But when you want strings, you want to be careful about your schema and where you’re putting them. So the best schema depends primarily on what queries you want to run. At a high level, you can’t GROUP BY fields, only tags. Especially if you think you’re going to want to GROUP BY, then make sure it’s a tag. You can’t do math on tags or across measurements. You can’t pass tags to functions. Queries that filter by tags are fast, and queries that filter by fields are slower because fields are unindexed. And be aware of series cardinality, which is what we’re going to talk about next.
Katy Farmer 00:17:48.380 So the number of unique database, measurement, and tag-set combinations in an InfluxDB instance is series cardinality. For example, assume that you have a InfluxDB instance with a single database and only one measurement, and the measurement has two tag keys—email and status. It looks like [laughter] some of my text is behind this. That’s okay. Don’t worry. I’ll help you [laughter]. So if you have three different email addresses, right—we have Lorr, Marv, and Cliff, each with their own emails—and they could all have one of two different statuses—start and finish—then the series cardinality for this measurement is six because we have three emails and two different statuses. So we’re going to do a slightly more realistic computation here on the next one [laughter].
Katy Farmer 00:18:53.374 Imagine that you have a measurement that has two tag keys—container ID, process ID—but then each tag has 10,000 possibilities. In this, the series cardinality is like 100 million. And if each series takes up 1K in memory, then you’re looking at 100 gigs of RAM before you begin to even see the problem. And that’s crazy, right? You don’t want to be worrying about that kind of memory usage, and you don’t want to be realizing you have a cardinality problem that far into the problem. Which is why it’s important to think about the way that you insert your data and organize your data inside of a time series database. When you run into this kind of high cardinality, we call it runaway series cardinality. Now we have implemented something called time series indexing, TSI, which is a way to kind of move memory usage onto disk instead of in RAM. And that way it can sort of mitigate the usage, and it’s a much better option for really high cardinality if it can’t be helped if that’s just what you need. TSI, I think, right now, is available but you have to go turn it on. I could be wrong about that in the newest version, but if you need TSI, make sure that you check the configuration settings and see that it is on. And just plan to monitor series cardinality so that it doesn’t sneak up on you and cause you problems.
Katy Farmer 00:20:39.979 So let’s talk about retention policies. I mentioned before, when I talk to people, this is oftentimes one of the features of our database that kind of convinces them to try it. It’s not that easy to set retention policies and delete data in other databases, and we pride ourselves on making this about developer happiness and ease. So a retention policy is how long InfluxDB keeps data. It’s the duration of the data. And it can also describe how many copies of that data are stored in a cluster, which is the replication factor. And then it’s also the time range covered by shard groups, which is the shard group duration. So retention policies are unique per database, and along with the measurement and tag-set, they define a series. When you create a database, InfluxDB automatically creates a retention policy called autogen. It has an infinite duration, a replication factor set to one, and a ShardGroup duration set to seven days. Most of the time, especially if you just want to download it, install, and play around with it, these defaults will be fine for you, especially if you’re just using toy data. But I think it’s always really interesting to set them and play around with them and make sure that their behavior matches your expectations.
Katy Farmer 00:22:13.347 So let’s talk a little bit about shards and shard groups. InfluxDB stores data in shard groups. In general, shorter shard group durations allow the system to efficiently drop data. When InfluxDB enforces a retention policy, it drops entire shard groups, not individual data points. So there is some consideration you need to put into when you’re setting the retention policy on data, making sure that you want to drop the entire shard and not an individual point because the retention policy affects the entire shard group. If your retention policy’s duration is greater than six months, there’s no need to have a short shard group duration. So you can see that you have to think a little bit about how these retention policies are going to interact with each other. Your shard group duration needs to sort of complement and take into account your retention policy duration. Increasing the ShardGroup duration beyond the default seven-day value can improve compression, improve write speed, and decrease the fixed iterator overhead per ShardGroup.
Katy Farmer 00:23:29.103 Basically what we’re talking about here is the same thing I’ve been mentioning. You want to think about how your shard group duration affects your other data. And when you’re too short of a shard group duration or too long of a shard group duration, both have their advantages and disadvantages, but you want to make sure that it’s aligned with the behavior you expect from your retention policy duration. We recommend configuring your shard group so that it’s two times your longest typical query’s time range. We’ll come back to an example of that in a second. Each shard group has at least 10 or—sorry, 100,000 points per ShardGroup, and each ShardGroup has at least 1,000 points per series. So what we’re talking about here is making sure that the shard groups are optimized for compression, and write speed, and overhead so that you don’t end up with conflicting retention policies that affect the performance of your database or are in conflict with your retention policy duration.
Katy Farmer 00:24:43.842 So we’re going to talk a little bit about continuous queries and downsampling since continuous queries are something you can do with InfluxDB. When you work with time series data over a long period of time, this can create some storage concerns. We’re talking about millions of points and if you want to keep your data infinitely—some people do—then clearly you start to come across a storage problem. So one solution is to downsample the data, which means that you keep the high-precision, raw data for a limited time, but you store the summarized, aggregated, lower-precision data for longer or forever. And there are two ways to perform downsampling: one is to set up continuous queries in InfluxDB, and the second one is to use Kapacitor as a continuous-query engine. We haven’t really talked about Kapacitor that much in this training. It usually gets its own training. But Kapacitor is a really powerful part of our stack, and we’re working on some benchmarks to display how much more performant it can be to use it as a continuous-query engine that’s used in InfluxDB. But when we get those out, you’ll know.
Katy Farmer 00:26:02.920 So you need to make some decisions about when to use Kapacitor and when to use continuous queries in InfluxDB. So if you’re performing a significant number of continuous queries and you want to isolate the workload, or when you need to do more data processing beyond just performing a query, those are both good use cases for Kapacitor. It can do much more highly customized tasks, and if you want to isolate the workload, then you can sort of offload the task to Kapacitor and let it run and then have it return to InfluxDB if you wanted to keep the database performant during that time. And you want a continuous query from Influx when you’re performing downsampling for retention policies via query only or if you only have just a handful of continuous queries. So we’re really just trying to minimize the amount of work the database has to do on these tasks and it’s just a little bit like, when is there a tipping point when your database is spending too much of its resources on continuous queries? Then you want to switch them over to Kapacitor.
Katy Farmer 00:27:18.000 Now we’re back at InfluxQL, our InfluxDB query language. It is SQL-like, so you can see from the first line, you can see SELECT count (*). This is entirely possible, but you’re not going to see it as a typical query for time series. We mentioned earlier just how many data points can be in a time series database. Even just in one field, it could be millions of points. So if you do SELECT count (*) and it’s got to go through every one, it’s really going to bombard you with the return of that query. It’s going to be running for a while. I mean, if you’re running it in the terminal, probably you’d just get to watch the results fly by, but it’s not very performant and not a very good idea. So you always want to consider the query scope when you’re performing these kinds of ad hoc analyses. What kind of data do you expect to get back? How many series are in this database? What’s the time range, especially if you’re building dashboards? And am I trying to get a single point? Subqueries are a great way to limit the scope. So you can use something like a WHERE clause, that kind of thing, to make sure that you’re narrowing down the results. And you can also always use a time range on your queries to make them more efficient. This is sort of the most common way you’ll see people limiting their time series queries, is just to say, “We very rarely care about the entire history of the database so only give me where the dates are between yesterday and this morning.” Just gives you a more concrete set of numbers that you can analyze, dashboard, whatever it is you’re going to do with them.
Katy Farmer 00:29:19.492 This is just a nice sort of reference table that you can see the similarities between InfluxQL and SQL. A lot of our vocabulary is the same. So a lot of times—I’m not one to check the documentation first. I’ll usually try something, and then if it doesn’t work, I’ll say like, “Oh I thought that this thing from SQL would work here, and if it doesn’t, then I’ll go check the docs.” But there are some small differences that are very important. However, you can see that a lot of our vocabulary is the same. Also, you can include regular expressions in your InfluxQL queries. So you can use field-keys and tag-keys in the SELECT clause, measurements in the FROM clause, tag-values and string field-values in WHERE clauses, and tag keys in the GROUP BY. You can also use mathematical operators: AND, OR, or—I’m going to say XOR. I know that’s not how you say it, but it makes me happy because of the way it sounds kind of like a He-Man villain [laughter]. And subqueries, they can be nested in the FROM clause of another query. So you can use a subquery to apply a query as a condition to the enclosing query, just, again, to help limit the results that are coming back to make sure that you are being very specific about the results that you want. Subqueries offer functionality similar to nested functions and SQL HAVING clauses. So again, this is one of those important differences between InfluxQL and SQL. If you want to use a HAVING clause, we use subqueries instead. That’s just something that’s easy to look up in our documentation. The documentation for InfluxQL is really good. And you can also do continuous queries from InfluxQL.
Katy Farmer 00:31:38.825 All right. So now we’re going to talk a little bit about Telegraf. Give me one second to get a drink of water before we jump in. I said water—it’s not. It’s caffeine, but all the same. Okay. So Telegraf is our open source, plugin-driven server agent for collecting and reporting metrics. We talked about this a little bit in the beginning, but again, we set up Telegraf specifically so that it can be push or pull. It can be kind of controversial. Some people subscribe to only pushing or pulling. We want to be agnostic and let the developer decide. So we’ve made it so that you can do either. We use Telegraf to batch metrics from multiple sources together in order to reduce the number of atomic write requests. And we have a huge collection of plugins. We have input plugins, output plugins, aggregators, and processors. And Telegraf can act as an agent, a collector, and also an ingest pipeline. Telegraf is one of our products that we don’t really have a way to track the number of installs, but as we interact with the community, it’s clear that this is one of our most popular products because it can do so much and it’s so flexible.
Katy Farmer 00:33:10.421 Telegraf also manages metrics ingestion from the host system. I showed you earlier the CPU usage. And you can use third party APIs, some common programs. Like I said before, you can ingest metrics from—I was ingesting metrics from MySQL and then sending them out to InfluxDB. But we have over 140 plugins, most of which are written by the community. So when people find a need to use Telegraf, and it’s not integrated with the service that they’re using, oftentimes they’ll just write their own plugin. We have a really good system for teaching people how to write the plugins and having them submit PRs. And it’s been really successful for us, especially because I think people really love Telegraf, and they want to use it so much that they’ll write their own plugin for it, which always really amazes me. I didn’t go over some of the examples. You can see listed here we have some examples of the output or—sorry, the plugins that we have. It’s so many. Graphite, and OpenTSDB, and Datadog, and Librato, Kafka, MQTT. You can go to the GitHub page for Telegraf under InfluxData and see a full list grouped by what kind of plugin they are, or you can go to our documentation and find a pretty good list there too.
Katy Farmer 00:34:53.768 The benefits of Telegraf are that it has this minimal memory footprint, it’s really performant, it tags metrics, it is an easy way to add functionality, and like I said before, it has really strong community contributions. A lot of its—most of its plugins are written by the community because they like to use Telegraf and find that it adds value to their projects, which is always just, again, amazing to me because it’s really flexible, you can use it in a lot of different ways, and we’re always surprised and impressed by the way that community members use it. So if you have an idea for a plugin, definitely write it or contact us if you need help. We’re always there to make sure that people access these. So we’ll go over just a few of the Telegraf plugins. We have the StatsD, PostGres, Apache—these are some of the popular ones; they’re just to give you kind of a feel for the variety of the plugins, especially when it comes to just some very popular services—Prometheus, NGINX, Kubernetes. We’re always working on improving these, and if any of these seem like they’ll add value to your projects, then feel free to contribute to them and make PRs. That’s what makes us better, is getting feedback, so.
Katy Farmer 00:36:37.338 I’ve hit the end of the presentation, so we’re going to open it up for questions now. If you want to see any more demos, just ask and we’ll give it a go.
Chris Churilo 00:36:48.548 Nice job, Katy. If you go to the Q&A, we’ve got our first question from Jocelyn.
Katy Farmer 00:36:57.927 There we go, Jocelyn. “Is there an easy way to dump data from InfluxDB and restore it in another instance? For example, let’s say we need to see the data from a customer but for security reasons it cannot connect to their network.” Yeah. So the recommended process for this would be a backup and restore, which we have a really good documentation for. Basically you can create a backup either in the command line tool—I think that’s probably the easiest way to do it, is in the command line tool. And just in the most recent version of InfluxDB, you can now make a backup of a time range. If you don’t want the entire database, you can do a backup of a time range, and then when you create a new instance and then do an import on that. I’m sorry. I’m thinking it through as I’m saying it. I just helped someone with this the other day. There can be some nuances, but yeah, the backups are really easy to do and then you can use the backups to restore them into a different instance of InfluxDB. There’s really good documentation on it on our community site where I just answered a similar question the other day.
Chris Churilo 00:38:25.325 So Jocelyn, hopefully that answers your question. Oh, looks like we’ve got a few more. I’ll let you take those, Katy.
Katy Farmer 00:38:32.838 All right. Gunga says, “I missed the demo. Can you show me how—would you be able to show quickly to understand the core system metrics?” Yeah, of course. So I’m going to navigate back to Chronograf. I’m going to close the questions for a moment. So if this is your first time setting up Chronograf, you’ll come to a page that looks like this. It will say welcome cause it will be your first time. The database here will be the database that tracks your sort of local CPU metrics and stuff. So what we do is we go to the Host List—I can see here my MacBook listed—and then I click on system under the Apps column. And when I go to the—whatever else I have going on. Some of them are blank cause, again, I’ve just been traveling a lot so my computer has been not that active. That’s sort of it, basically. You don’t really have to set up. As long as you have Chronograf, InfluxDB, and Telegraf installed and started with your service manager, it will already be collecting these, and as soon as you set this up, they’ll be there for you to view. I hope that answers that and, again, if you want more demos, we’re always doing new webinars and we have some videos available.
Katy Farmer 00:40:16.808 Daryl asked, “How do you handle time zone processing when handling metrics from different regions?” That’s a good question, Daryl. We get that question a lot. The short answer is, I am fairly certain that we use like a UNIX timestamp, so it goes off of—sorry, I’m trying to think of the right words so I don’t say it wrong. But there is a universal time tracked by UNIX in most operating systems. I’m not sure how it would work on Windows, if there’s something similar. So they all go off of this universal time that’s in the operating system so that way we don’t have to worry about converting time zones as such. We haven’t had any problems with things getting out of order so they use this—I want to say it’s the Unicode timestamp. I would have to check, but.
Chris Churilo 00:41:19.824 Yeah. So it’s the time zones at declaration. I’ll just throw in a community answer into chat for everybody so you could see. Hopefully you guys heard me. Is my Internet okay, Katy?
Katy Farmer 00:41:39.553 Yeah, it’s good.
Chris Churilo 00:41:39.849 Okay. Cause it keeps popping up and down. Okay. There’s another question in the—
Katy Farmer 00:41:46.027 Yeah. “Will recording be available for review later?” Yes. Absolutely. We’re recording this and we’ll do a little bit of editing, and then it will be available. I’m not sure of the time frame on it—Chris will know—but it’s definitely going to be available hopefully later on today. Let’s see. You’re welcome [laughter].
Chris Churilo 00:42:17.245 All right. Ganga, do you have any other questions? I see that you raised your hand in—oh there we go. Yes he does have more questions.
Katy Farmer 00:42:24.139 “How about the network latency metrics?” I think that you have to use another plugin for the network latency metrics. It’s on the Telegraf GitHub page. I don’t have it installed on mine, or I don’t have it configured. That would make a great demo that I don’t know if we have time for it this morning, but that would make a great demo. It’s on the GitHub page. It’s really easy to configure. Basically all you have to do is just alter like one thing in your Telegraf configuration files. Just kind of enable that plugin, and then It would be also available on this Host List page. That’s where it would show up once you had it configured.
Chris Churilo 00:43:16.418 Yeah. And I’ll just throw in the link for your documentation. Yeah. There you go. All right. Don’t be shy, everybody. Please keep your questions coming in either in the chat or in the Q&A. We’ll get those answered for you. And then as Katy mentioned, she sits on our community site, waiting for your questions to come in, so don’t be shy. She’s more than capable of answering your questions beyond just the basics of InfluxDB. And as we’re waiting for you guys to put your questions in, just want to remind everybody. So we do our trainings every Thursday, as you guys know—you’re here today—and we’ll be kicking off some customer-led webinars on Tuesdays. I have a bunch that are lined up that will get started again in March. And if you have any ideas of what you want to hear for either of those sessions, just let us know. We’d be happy to accommodate. In addition, I just want to put out a plug for Influx/Days. We just had our New York conference. Man, it’s already been, I guess, a week and a half. And the presentations are all on SlideShare and all the videos are on YouTube. You can just go to influxdays.com, and if you go to the New York past events, then you can get all those links in a single place. But we’re already getting ready for the London conference, which will be in June, so if you happen to be in the [inaudible] region, then it will be an opportunity for you to come and meet us and hang out.
Katy Farmer 00:45:08.875 So we’ve one more question. Sobhan asked, “Do you have any comparison study with columnary database and InfluxDB?” Yes. And so let me just pull that up. So we actually conduct benchmarks on a regular basis. We’ve got four that I’m just going to pull them up and put them in, that we do on a regular basis. We’ll be doing more than these pretty soon. So we actually have one time series, Elasticsearch, and then we have Cassandra, and also MongoDB as comparisons that we’ve run. And let me just submit this link to everybody. And so in the benchmarks, we have webinars and technical papers. What we describe in there are the capabilities that you’ll have to build in a columnar database in order to be able to mimic the time series capabilities that you would get in a time series database, whether it’s from InfluxData or any other vendor that has a time series database.
Chris Churilo 00:46:12.388 In addition, Katy mentioned that really the three most important capabilities within a time series database is you want to make sure that you have a fast ingest, you want to have fast query, and you also want to minimize disk storage. And so we actually do tests across those three things across these four different tools. So you can take a look at that at your leisure, and hopefully you’ll get a pretty good sense of what a lot of work it would be if you were to do it yourself [laughter].
Katy Farmer 00:46:42.010 Yeah, so much work. Sobhan asked, “How do we accomplish join operations between tables in time series tables? Any performance implications?” So my current understanding is that you wouldn’t—I don’t know if there’s a direct comparison to adjoin because they work a little differently. Because there isn’t really a table as such, right? This is something you have to consider, I think, when you’re building up your data, is what kind of comparisons are you going to want to make. Because, in one of the slides earlier, I think we said that you can’t make comparisons across measurements. But you can make comparisons across tags and possibly fields. So what you want to do is just think about that when you’re setting up sort of the schema for your database. Since there isn’t really a table, you want to make sure that you think about your data, and one of the things I like about time series databases is that they force you to kind of think not just about the data you’re ingesting but what its value is in. So you have to consider how you’re going to compare it later, what you might want to analyze. So a lot of it, there are performance implications when you’re thinking about whether it’s a tag or a field—right?—is it indexed or unindexed? But generally, that’s not necessarily because you could do the comparisons. It’s just that you want to make sure that your schema is optimized for the kind of data it is. So if you’re going to want to compare two pieces of data or two series, then you want to make sure that your tags are appropriate so that you can compare indexed fields because that’s always going to be faster.
Chris Churilo 00:48:51.312 Yeah. And so when I think—we’re almost at the top of the hour, but I think one of the things that might be helpful is if you can put maybe a little bit information about what you’re trying to accomplish with your data, then we can probably explain a little bit more that you wouldn’t need to do such an operation.
Katy Farmer 00:49:13.271 Yeah. Most of the time, there’s another option, I guess, is the best thing. If you’re thinking like, “Oh I really want to do this join, and how can I do that?” Usually, in time series data, the answer is that there’s a simpler way to achieve that just based on the functions that are available.
Chris Churilo 00:49:38.942 Trying to move.
Katy Farmer 00:49:40.373 Sobhan says, “We’re trying to move away from Oracle into Cloudera and thinking to use InfluxDB for analytics.” Yeah, I mean, InfluxDB is great for analytics. I would just say that making sure that your—since you already know what kind of analytics you want to do, I assume, then just making sure that your schema sort of matches up with the things you’re going to want to compare later on. So there’s no join, but you will be able to sort of make sure that the tags and fields and measurements are appropriate since measurements are the only things you can’t compare across.
Chris Churilo 00:50:22.026 Yeah. And Katy’s absolutely right. I mean, if you think about analytics data, analytics data has a timestamp because you’re trying to measure how well something is doing or how well it’s not doing over time. So it’s perfect for a Time Series Database. And I would recommend just getting started by just doing the quick setup that Katy showed you, and just start looking at just the basic metrics that you can collect from your laptop. And I think you’ll quickly be able to see how easy it is to pull that data into a time series database and how you can query, build a little dashboard, and even set some kind of threshold alerts as well.
Katy Farmer 00:51:03.376 Sobhan says thank you. I say you’re welcome. You can always hit us up on the community site. We also have a InfluxDB Slack channel if you’re in the Gopher Slack. And we’re always on Twitter, so we’re just @InfluxDB on Twitter. You can hit us up there. If you’re in the San Francisco area, we have a meetup once a month where we talk about time-series-related things. We don’t always talk about Influx, but if you have questions about Influx, it’s a great place to ask. We also have one in Denver and soon we’ll have one in New York. So if you want any more information or questions, we have a lot of ways for you to ask them because we want to make sure they get answered. So hit us up on community or Twitter, and we’ll do our best to make sure you get what you need.
Chris Churilo 00:51:55.440 That’s a perfect ending to this training. I couldn’t have done it better [laughter]. So thanks, everybody. As Katy mentioned, there’s a lot of different resources that you can take advantage of to get your questions answered. And I think the best way to get started is just start collecting some of those basic metrics and start playing around with InfluxDB. It’s open source. Majority of our users rely on the open source version. So it’s a fantastic set of projects. I will post the recording hopefully before the end—well, definitely before the end of the week. Hopefully I can get to it before the end of today. And we look forward to seeing you guys again in one of our trainings or webinars. Thanks, Katy, for a fabulous training. And—
Katy Farmer 00:52:37.287 You’re welcome.
Chris Churilo 00:52:37.871 —have a safe journey home.
Katy Farmer 00:52:39.377 Thanks.
Chris Churilo 00:52:40.281 Buh-bye.
Katy Farmer 00:52:41.348 Bye.
David Simmons is the IoT Developer Evangelist at InfluxData, helping developers around the globe manage the streams of data that their devices produce. He is passionate about IoT and helped to develop the very first IoT Developer Platform before "IoT" was even ‘a thing.’ David has held numerous technical evangelist roles at companies such as DragonFly IOT, Riverbed Technologies, and Sun.
Track and graph your Aerospike node statistics as well as statistics for all of the configured namespaces.
Knowing how well your webserver is handling your traffic helps you build great experiences for your users. Collect server statistics to maintain exceptional performance.
Collect and graph performance metrics from the MON and OSD nodes in a Ceph storage cluster.
Use the Dovecot stats protocol to collect and graph metrics on configured domains.
Easily monitor and track key web server performance metrics from any running HAProxy instance.
Gather metrics about the running Kubernetes pods and containers for a single host.
Collect and act on a set of Mesos statistics and metrics that enable you to monitor resource usage and detect abnormal situations early.
Gather and graph metrics from this simple and lightweight messaging protocol ideal for IoT devices.
Gather phusion passenger stats to securely operate web apps, microservices & APIs with outstanding reliability, performance and control.
The Prometheus plugin gathers metrics from any webpage exposing metrics with Prometheus format.
Monitor the status of the puppet server – the success or failure of actual puppet runs on the end nodes themselves.