In this session, you will learn how to access the Kapacitor configuration via the API. You can use this to add and update the Kapacitor configuration.
Watch the webinar “Access Kapacitor Configuration via the API” by clicking on the download button on the right. This will open the recording.
Here is an unedited transcript of the webinar “Access Kapacitor Configuration via the API.” This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.
• Chris Churilo: Director Product Marketing, InfluxData
• Michael DeSa: Software Engineer, InfluxData
Michael DeSa 00:00.591 So as Chris noted, we’re going to be talking about the dynamic configuration of the API, the Kapacitor’s HTTP API. There we go. So the agenda for today is to understand the Kapacitor configuration file. So what components are in there, which ones can be controlled via the API, so we’re going to talk about what are called overrides, Kapacitor overrides, that use the HTTP API. We’re going to access specific Kapacitor configuration via the API. We’re going to update the configuration. We’re going to add new configuration. We’re going to delete some of the configuration. And we’ll have two examples that we go through in this process. So the Kapacitor configuration is used to specify parameters for Kapacitor surfaces. Each of the various components that Kapacitor’s built out of are structured in this kind of service architecture, and these services are kind of like internal services or external services. So internal services would be things like the general system configuration or the HTTP service configuration or the logging service configuration. So those are the kind of things that are internal to Kapacitor. And I’m going to make a note here that a lot of these can only be set in the config file. They’re not accessible via the API. So if it’s something that’s internal to Kapacitor, you cannot set it any way other than through the API.
Michael DeSa 01:29.516 There’s also these kinds of external services or things that are external to Kapacitor that Kapacitor needs configuration for, and those are things like the InfluxDB configuration, the HipChat configuration, PagerDuty, Slack. The list keeps going on. And we’ll see some more examples of that when we go through accessing the actual values in the API. So as I noted, there’s two ways you can configure these Kapacitor services. You can do so through the TOML file or the config file, which is in a markup language called TOML. It’s a very common markup language that you see in many Go programs, becoming more and more popular. And for starters, various Debian or Linux systems, you can find this config file at /etc/kapacitor/kapacitor.conf. And you can sort of just update there and restart your process to get the new values to be set.
Michael DeSa 02:28.775 The other way that you can manage configs or add configs or configure Kapacitor is through the API using what are called overrides. So the kind of way you think about this is Kapacitor in initial configuration gets set to the config file, and then you use the API to apply overrides to these config variables or these settings so that you end up with something that is what you intend it to be. There’s a little bit of a disconnect here that you have to be careful of. Configuration should usually be done exclusively through the config file or exclusively through the API. You shouldn’t do both. And the reason for that is it’s possible to get yourself into a little bit of a split-brain situation where your config file is saying one thing, but the API is saying another thing. And you definitely want to try to avoid that just in case you sort of—you don’t want to end up in kind of a bad situation.
Michael DeSa 03:34.033 So when should you use a config file versus the API? So the config file, you can disable accessing values through the API. So if you want to do something that’s relatively secure, you’re worried about people accessing values, configuration values that are stored in the API, you may want to sort of disable the configuration through the API. And you can access a configuration through the API so that no external things can network and nothing on the network can actually get at what those configuration values are. So if you’re in a secure environment, you don’t necessarily trust your network, this is really your only option. If you do have a more secure network and you can kind of control things a bit more, you might want to consider using the API just because you access it anywhere sort of have an HP connection to the instance. And you can kind of dynamically add more services or configure services. So suppose that you’re using PagerDuty one day, but you decide that you don’t want to do that anymore. It’s just a couple sort of clicks away in Chronograf or a couple HTTP requests away from having that configuration without ever having to restart Kapacitor itself. It kind of just takes and everything should just work from there.
Michael DeSa 04:55.607 So one of the downsides here is in the Open Source version of Kapacitor, there is no authorization or authentication. So anyone can access the API, which can be problematic. And again, just to make it clear, you should always be doing either configuration either through the API or through the config file. And try to avoid doing both. So here’s the sort of a little bit of a structure, an example structure of a Kapacitor configuration file. It’s a static TOML file. You can update it with no hubs. It must set some Kapacitor instance level options. So that’s things like the host name, the data directory, the HTTP settings, but it does not have to necessarily set things like the InfluxDB configuration or the SMTP configuration. So these are things that can be controlled via the API. So it’s sort of—if you notice here, I have this sort of the upper section and then the lower section, that’s how they’re structured in the configuration. The upper section are things that you cannot control via the API. The lower section are things that you can control the API.
Michael DeSa 06:10.997 So there’s one particular thing that I want to call attention to here. You’ve heard me use this word before and that word is an override or a configuration override. If you want to access things via the API, you want to be able to update things via the API or just even access them via the API, you have to have this config override enabled true. Otherwise, you cannot access the config options that are available, the service config options that are available via the API. So again, one thing that I noted is it’s possible that you can get into a little bit of a split-brain situation. It’s even possible that things can get a little bit bad that you get into a somewhat unrecoverable state with your overrides. If that ever does happen, we have a config option called skip config overrides, which will skip applying any overrides and just use the values that are in the configuration. So your configuration file will be the source of truth again.
Michael DeSa 07:10.619 So if you start out liking the HTTP API and then decide later on that I don’t want to use this, I can disable the config override and then skip config overrides. And that’ll allow me to just have my config file be the source of truth. Alternatively, if you just want to disable these things, disable config override and enable skip config overrides right off the bat and you know you only want to use the config file, this is the place to do that as well. And as I mentioned earlier, you really do want to pick one or the other. Trying to do both can get you in a little bit of a confusing state.
Michael DeSa 07:50.594 So here we’re going to start going through just kind of an example of walking through the API and the sort of base route for getting the entire configuration is HTTP GET/Kapacitor/v1/config. So this will return the entire config. We see the length—so where it was found —and then some of the sections below it. When you actually make this request, all of the various sections come back together. For space purposes, I’ve included these kind of dot, dot, dots in here. So, at the moment, you can see we have things like Alerta, HipChat, InfluxDB, OpsGenie, PagerDuty, Pushover, Sensu, Slack, SMTP, SNMP Trap, Talk, Telegram, VictorOps. We’re adding more and more of these up every day. Got another one coming up next release for a HTTP post request. So you can add headers and find endpoints that you can contact. So we’re adding more of these all the time but these are the ones that we have for now.
Michael DeSa 09:00.180 To access a specific subsection of that configuration, we just look at the section name and then sort of tack that onto the end of our URL. So to get the InfluxDB configuration, we make a HTTP GET request to Kapacitor/v1/config/InfluxDB and this returns all of the InfluxDB configurations. So the note here is that Kapacitor can have multiple InfluxDBs. So if I want to have a single Kapacitor instance for both my testing and my production, it’s entirely something you can do with Kapacitor. And so, when we make this request, we can see we get the link back and then the various elements that are there. So you should note in this case, we only have a single element that is the default InfluxDB configuration. We’re going to see in just a moment when we add a new section what that looks like.
Michael DeSa 10:01.306 So here we add a new element to the InfluxDB section and what we’re doing here is just making a post request to Kapacitor v1/config/InfluxDB. And so we specify here some parameters. We say add and then all of the associated parameters that we would like to add. So the things that we’re going to add to this is a name, the URLs, whether or not it’s the default, and whether or not disabled subscriptions is true. So when we do this, we can then see—we make a request back to getting the whole InfluxDB config, we can see that we have—in addition to the default route, we have a new route called /example. Now if you recall from previously, the new Kapacitor InfluxDB instance that we added had name example. So the new instance that we had added is indexed by its name. So this is kind of a common concept that comes up in this configuration where the new element that you add is indexed by either a name or some other kind of configuration option that is set in that section.
Michael DeSa 11:19.430 So now that we’ve accessed the whole configuration, the specific configuration for InfluxDB, we want to access a specific subsection of that InfluxDB code or configuration. And to do that, we make a GET request to Kapacitor v1/config/InfluxDB/example. So again, we get the link back with all of the associated options and we can see that kind of everything that we set in that add statement is still true. So we have the default true. We have disabled subscriptions true. And I believe we set URLs, which in this case, I have hidden here just a little bit for space concerns. So one thing to pay note to here is certain elements of the configuration are redacted. So even if they are set via the API you can’t necessarily access them. So in this example here, you may see at the very bottom that we have redacted password. So we actually prevent people from being able to access that password, but we do let them know that it exists, and that maybe the value had been set. But nothing about whether or not it is—what it actually is. That being said, you can still set the passwords remotely. So the main reason for this is if somebody for some reason was able to get access to your instance they can’t set any values but—or, they can’t get any values. They may be able to set some values. And again, this is only for the Open Source product. In the closed source version, we have authentication and authorization on these things.
Michael DeSa 13:05.144 To update a specific subsection, we make a POST request to Kapacitor v1/config/InfluxDB/example with this kind of set, and then the value we’d like to set, and its corresponding configuration value. So, in this case, we’re setting disabled subscriptions to be false, and previously it was true. So then when we re-access that subsection, we can now see that disabled subscriptions is false, which is what we did when we contacted the API. And then finally, suppose that for some reason this InfluxDB instance I’ve been working with is now dead, and we’re not using it or it’s out of commission, and I want to remove it from my instances sort of awareness. I can do that by making a POST request to Kapacitor/v1/config/InfluxDB. And I do that with this kind of removed, and then a list of the elements that I would like to remove. So, in this case, I’m removing a single element called example, which is the one that we had previously created. And then when we do that, we get all the subsections again, we can see that the example Influx DB was removed, and we’re just left with the default configuration that was there.
Michael DeSa 14:31.174 So this is great, but in the InfluxDB section, you actually have an array of different elements. You can have multiple different Influx DB configurations. There’s actually a number of different configuration options that you have in Kapacitor that do not have corresponding multiple sections that can be set. They only have a singular source of kind of truth. There’s only a single config section. So the important thing to note about there is you need this trailing slash to get the specific element because it doesn’t have anything that it is exactly indexed by. So the important thing to keep in mind is that the trailing slash is distinct from the route without it. The route without it will return me all SMTP configurations, but there is only one. And so for that reason, you kind of want to avoid using it. It adds a little bit of confusion. So to access the SMTP configuration, we make a GET request to Kapacitor v1/config/SMTP. And we can see we get all of the associated linking, and any of the options as well as the things that may or may not be redacted. So if you’ll recall previously, in the InfluxDB section we set a single value and kind of that’s all that we did. You can actually do multiple operations in these POST requests that we make when updating values.
Michael DeSa 15:58.293 And so when we update them we can say set, I want to set enable to be true, from to be [email protected], and host to be SMTP.example.com. And then, on top of that, I want you to delete ports. I don’t want access to the port value anymore. Any value that was there, if you could delete it, that would be desirable. So when I do that, we can see that enabled is true, then we have from is [email protected], host is SMTP.example.com, and then you’ll note here that there is no associated port value. So that has been removed from the resulting options. So as a little bit of a summary kind of going through this is Kapacitor service configuration can be controlled through the API. The good thing about this—the nice thing about this is that all configuration options just take place immediately. So if I have a task that wanted to use a new configuration, it just would kind of accept that value as it was running. You don’t have to worry about making sure that any of the configuration is getting kind of updated or stopping a task by pausing processor or anything like that.
Michael DeSa 17:17.425 Whenever you are deciding how you want to configure your instance, you should always be deciding whether or not you’re doing exclusively through the API or exclusively through the config file. And the reason for that is it prevents multiple sources of truth. If both are used together, the API values will take precedence over what is in the configuration file. And so that’s kind of just one thing you want to keep in mind. You can get yourself out of this, if you ever get yourself into a position where you’re not really sure what’s going on, by changing that config override, disabled and then skip config override’s true in the configuration file. Those will be applied and whatever is in your config file will become the source of truth.
Michael DeSa 18:00.005 So you might want to keep that in mind when you’re working with the configuration. The other thing to be aware of is not everything in the configuration is accessible via the API. So there’s a number of options that say the HTTP service for the Kapacitor, the login service for the Kapacitor, those things you cannot control through the API. And sort of the final thing to keep in mind here is whenever you’re asking yourself, “Should I be doing things to the configuration file or should I be doing things through the API?” The API is probably less secure, but it does give you a little bit more flexibility. So if you’re using something like Chronograf and you want to, say, add Alerta, or you want to start using a different SMTP server, or you want to add a new end point that you want to contact, or you want to do something that you want to change immediately, without having to get onto the box and make sure that the config file there is updated and you start everything appropriately, then you may want to use the API. It does give you some niceties, but it is less secure because it’s just kind of publicly available to any machine that has access to it, so anything that’s on the network with it.
Michael DeSa 19:21.073 And that’s kind of what you want to keep in mind when you’re designing your system. So you get the niceties of being able to dynamically change things if you have an accessible view of the API, but you kind of lose a little bit of security and this really just comes down to your own environment and what’s acceptable for you. And this kind of brings me to my section here on questions. I can see that we’ve got kind of a number of things that are coming through here and the question in chat is, “Which languages are these APIs written in? Do we have various languages of support?” So this is actually just an HTPAPI, it’s a restful HTPAPI. So anything that has support for HTTP, which is virtually every programming language in the world, should be able to use this. And the results that you get back and the data that you POST is all JSON, so it should be pretty familiar to anyone that has access to it. “Would you be able to demonstrate?” I can show a couple examples, maybe, of that, but that might be out of the scope of things. If we have some time, maybe at the end here, I’ll share an example. When I was doing this and building up these sort of slides here, I did all of this using Curl and the command client.
Michael DeSa 20:42.232 “Would you be available to show a short demo of Kapacitor?” Sure, I can do that, actually. Let’s do that. Unless there’s other questions that are pressing, I’m happy to give a short demo of Kapacitor. It appears like there is not, so I will go and do that, myself. Let me share my screen right now. All right. So we’re going to—I’ve got a number of TICKscripts. For those of you who aren’t familiar with TICKscripts, I highly recommend you take a look at our introductory sections, where we cover, what is TICKscript, what are tasks, how do these things all come together? We’ll have a little bit of assumption that you’ve seen these things before, but it should be sufficient. What we’re going to do is take a look at the CPU alert tick, and this TICKscript that I have right here is very very long, so we’re not going to use that one. I’ve gone off the cuff a little bit here, so this is kind of all on the whim. Let’s take a look at alert.tick. All right. Here. This should be a good example that we have here. So in this example I’m going to—here we have TICKscript. Hopefully you’ve seen it before and you have this concept of making either batch requests or stream requests, or whether the data is kind of just streamed to Kapacitor, or if Kapacitor is requesting the data from InfluxDB. And then we say—we’re declaring a variable called Data. We actually don’t need to do that in this example.
Michael DeSa 22:46.200 And then we stream data from the measurement CPU, and then we’re going to create a window with five seconds worth of data. We’re going to report that data every five seconds, and once we’re done with that, we’re going to kick off a POST request to some end point. Just for the sake of clarity here, we’re going to make some changes here, and we’re going to do this to http://localHost and I think I have it at 60-60. All right. So this is just an HTTP server that I have running. Essentially, what we’re doing here is we’re making batches of five seconds worth of data, and we’re making a POST request to some external end point. The thing to keep in mind is this HTTP POST and these header examples here are coming out in the 1.3 release, which is currently unreleased, and should be landing some time in the next few weeks. I’m going to save that file, and then I’m going to clean things up. I’m going to define the task by saying, “Kapacitor define,” and then we’re going to do example. And then I want to make it—give it a specific TICKscript, which will be alert.tick. Then I’m going to type. It is a stream task. And then I set up my DB and RP, which my database and retention policy into telegraf.autogen.
Michael DeSa 24:25.133 I want to make sure that I actually have some things running. And I don’t. So I’m going to start my own Telegraf. All right. So here, I’ve defined my Kapacitor task. I can then do Kapacitor list tasks and can see all of my various tasks here. And then if I wanted to, I can do Kapacitor enable example. And I’m going to do some clean up while I’m here. I’m going to do Kapacitor disable TICK and disable TOCK. This was me playing around from earlier in the day. So I’m going to disable those things for now. And if I reissue this list tasks, we can see that example is enabled and executing. And then I’m going to do a Kapacitor show example. And we can actually see some information here. So up at the top, we can see the ID for the thing, whether or not there’s any errors, a template. So it’s currently not a template so we don’t have to worry about that. The type of task it is, it’s status, whether or not it’s enabled essentially, whether or not it’s executing, when it was created, when it was modified, last enabled, any database and then contention policies it has access to, TICKscript, and then a graphical representation, graph.biz, familiar with the representation of the actual TICKscript.
Michael DeSa 26:00.845 And we can see here how much data it’s processed, how much data it’s processed, what the working cardinality is. So working cardinality is a concept similar to general cardinality or sort of if you’re familiar with that from InfluxDB. Kapacitor has similar kind of analogous concept. And any associated errors. So we can see right here we’re getting a bunch of errors. The main reason for that is the server that I have receiving this thing making POST requests too is currently down. So I’m going to go somewhere that actually has one that’s up. And I believe that was in—and we want to go in node. That’s also under 70. 50. Trying all the options here before I get it right. All right. And so we can see actually now that it’s making requests. So this is Kapacitor making requests, should be every five seconds or so with new batches of data. And here, in particular, I’m outputting the series, which is all of the data that was written and any of the associated headers.
Michael DeSa 27:15.376 So if you recall from the TICKscript, I set an example header with its value B. And we can see that being sent over to the actual instance. So TICKscripts can get a lot more involved than what I have here. This is kind of just a short little example that you can sort of see stuff with. Here we can see, if you recall previously, there was a bunch of errors. We’ve reduced the number of errors that are there and just have an average execution time. So any more of examples of this will probably be outside of the scope of what we can do in this time frame. But I think this was a good kind of offshoot that came about. And so I’m going to stop sharing now.
Michael DeSa 28:05.464 That was not a pull example. That was a push example. So the data was being streamed. So any rights that were being written to InfluxDB get streamed onwards to InfluxDB. So it took the stream of all things coming from Kapacitor and then from Telegraf and then bundled them together in five second windows and then made a batch request to this server that I had listening for it.
Track and graph your Aerospike node statistics as well as statistics for all of the configured namespaces.
Knowing how well your webserver is handling your traffic helps you build great experiences for your users. Collect server statistics to maintain exceptional performance.
Collect and graph performance metrics from the MON and OSD nodes in a Ceph storage cluster.
Use the Dovecot stats protocol to collect and graph metrics on configured domains.
Easily monitor and track key web server performance metrics from any running HAProxy instance.
Gather metrics about the running Kubernetes pods and containers for a single host.
Collect and act on a set of Mesos statistics and metrics that enable you to monitor resource usage and detect abnormal situations early.
Gather and graph metrics from this simple and lightweight messaging protocol ideal for IoT devices.
Gather phusion passenger stats to securely operate web apps, microservices & APIs with outstanding reliability, performance and control.
The Prometheus plugin gathers metrics from any webpage exposing metrics with Prometheus format.
Monitor the status of the puppet server – the success or failure of actual puppet runs on the end nodes themselves.