InfluxDB and the /debug/vars Endpoint
By Mark Rushakoff / Apr 03, 2017 / InfluxDB, Community, Kapacitor, Developer
Like many Go programs with an HTTP server, InfluxDB exposes some diagnostic information over the
/debug/vars endpoint. Because the information we expose there is simply JSON, it should be very straightforward to expose InfluxDB’s diagnostics to other custom utilities that you might want to use to monitor your InfluxDB instance.
Package expvar provides a standardized interface to public variables, such as operation counters in servers. It exposes these variables via HTTP at /debug/vars in JSON format.
expvar works great for most use cases. Even though InfluxDB started using
expvar directly, there was, and still is, a missing feature that caused us to stop using the standard
expvar package: it does not allow you to remove published variables. There were at least a couple of issues filed about stats being reported for entities that were deleted. A low signal-to-noise ratio is never helpful when you're trying to debug an active problem.
In InfluxDB PR 6964 (which was first present in the 1.0 release) we moved away from using
expvar directly, in favor of a custom implementation with
and the TICK Stack
The output of InfluxDB’s
/debug/vars endpoint is one JSON object that roughly corresponds with the contents of the
_internal database. The
_internal database stores the values every 10 seconds by default, but the
/debug/vars endpoint, like the
SHOW STATS query, gives you an instant-in-time view of those stats.
First, there are two key-value pairs that match the standard
cmdline is an array of strings representing the command line arguments used to invoke the process. If you're running
influxd as a service on Linux, the value for
cmdline will look something like:
The rest of the
/debug/vars output have keys representing the details of the object being measured, with values in "InfluxDB expvar format". The "InfluxDB expvar format" is an object with this structure:
name: a string describing what's being measured, i.e. the corresponding InfluxDB measurement
tags: an object with string keys and values, corresponding to the tags to use for the fields
values: an object whose keys and values correspond with fields for the measurement
Kapacitor uses a mix of “standard” expvar format with simple top-level key-value pairs, and InfluxDB-formatted expvars for received writes underneath the
Telegraf’s Influxdb input plugin supports ingesting InfluxDB-formatted expvars from a remote HTTP endpoint. It also handles
memstats as a special case. For a single instance of InfluxDB, this will result only in information redundant with the
_internal database, but this is a straightforward approach to monitoring multiple InfluxDB instances in one central location.
Unlike InfluxDB and Kapacitor, Telegraf does not (currently) expose a
/debug/vars HTTP endpoint.
I haven’t seen any standalone projects that allow you to emit InfluxDB-formatted expvars, but it doesn’t take much code to do that yourself if you understand the format. The entire file of our first, pure-expvar implementation is less than 50 lines, and the corresponding code to serve expvars over HTTP is only about 11 lines.