InfluxData Blog - Rick Brown

Continuous Deployment of Telegraf Configurations

Rick Brown (InfluxData) — Mon, 09 Dec 2019 07:00:57 -0700

After I shared my “Using Telegraf as a Gateway” post with Rawkode, he mentioned a talk he gives, where he discusses advanced Telegraf topics, including a requirement he’s seen for automatically configuring running Telegraf instances just by editing config files in GitHub. Watch that talk here.

I had a couple of questions about Telegraf reloading its configs, which he answered, and I wondered whether I have all the toolkit components already running in my home LAN to get this working. I could look at implementing a Continuous Delivery or Release Automation application to do this, but I’ll look at using what I already have.

I found that I already use everything I need! So here’s a Continuous Deployment mechanism for Telegraf. To replicate this, you will need:

Telegraf
- I mostly use Linux servers, but I have a couple of Windows machines and I mention how to include those later in this post
Root access to those machines
- to change configuration files
Node-RED
- other applications that permit the dynamic creation of HTTP endpoints would do just as well, but I use Node-RED
Err, that's it!

When installed on Linux (Debian - other distros might put config locations elsewhere), Telegraf command line options read /etc/telegraf/telegraf.conf and /etc/telegraf/telegraf.d/*.conf. Telegraf also includes the option of reading its configuration from an HTTP endpoint, which you’ll have seen if you use InfluxDB v2. When prompted to configure Telegraf with InfluxDB v2, you are provided with an API Token to store as an environment variable on the machine where your Telegraf runs, and you’re presented with this command line:

telegraf --config https://InfluxDBCloud2Instance/api/v2/telegrafs/NNN

That command line will run Telegraf (assuming it’s in your PATH), connecting to the specified endpoint to read its configuration from there.

So, what happens if I use this configuration?

/usr/bin/telegraf -config http://192.168.1.8:1880/telegraf/myhost

This should connect to that IP address (it’s my Node-RED server) on port 1880 (the Node-RED port), to the endpoint /telegraf/myhost. If I can get Node-RED to respond to web requests on /telegraf/myhost, I should be able to send a configuration to Telegraf.

The Node-RED http-in and http-out nodes are designed to do exactly this.

The next challenge is that each of my Telegraf instances might want a different set of configurations, so I want Node-RED to understand which instance of Telegraf is calling it, and to change configs accordingly. Telegraf will send a GET, so it’ll need to ensure the URL is unique in some way.

After that, I want Node-RED to provide updated configurations, for when I change any config, so I need to have Node-RED refresh its configs periodically. What might be a reasonable refresh time? I’ll go with 10 minutes.

Then I want each instance of Telegraf to refresh configurations, so everything is automated. What might be a reasonable refresh time for this? I’ll try an hour.

Of course, I want all this stuff to run without exposing any sensitive config information, so I need to have Node-Red modify any retrieved configs to add personal information such as the InfluxDB v2 API Token.

This is quite a kit-bag of things I need to do, so let’s get to it!

First, the Telegraf end:

I run Telegraf from systemd in all my LXC containers, so it runs automatically on start-up. Therefore, the configuration of Telegraf is in the systemd directories. If you’re manually running Telegraf, running on a system that uses sysvinit, or running from Docker, you’ll need to modify accordingly, but for my configuration, I need to edit the Telegraf service file located here:

/lib/systemd/system/telegraf.service

Copy that file to /etc, so it’s preserved when Telegraf is updated:

cp /lib/systemd/system/telegraf.service /etc/systemd/system

Now edit the new file. The ExecStart line shows the command line that will run Telegraf, with the -config and -config-directory options. Change it to this:

ExecStart=/usr/bin/telegraf -config http://192.168.1.8:1880/telegraf/%H $TELEGRAF_OPTS

Note the %H. It’s a special property in systemd that means “replace this with the hostname of this machine”, so each machine will request its own set of configs. That should be enough uniqueness for now — if I ever want to implement multiple instances of Telegraf on a single machine, or expand this mechanism to include environment definitions, I’ll investigate abstractions in systemd to retrieve variables from the EnvironmentFile entry rather than using %H.

Look at the next line in that file:

ExecReload=/bin/kill -HUP $MAINPID

This states that a “systemctl reload” command will send a SIGHUP signal to Telegraf, which is designed to cause Telegraf to refresh its configuration. I want to execute that on an hourly schedule. Linux machines run cron for regular automated tasks, so we can use this facility. Create the file /etc/cron.hourly/telegraf and put these lines into it:

#!/bin/bash -e 
# random sleep (max 5 min) to prevent clients from hitting the server at the same time 
SLEEP=$[ ($RANDOM % 300) ] && sleep $SLEEP 
systemctl reload telegraf

This will cause Telegraf to reload its configuration every 60-65 minutes.

Make that file executable:

This will cause Telegraf to reload its configuration every 60-65 minutes.

Make that file executable:

chmod +x /etc/cron.hourly/telegraf

Notes:

If your filename uses punctuation marks, it will be rejected by cron. There's no running a telegraf.runthis file — just keep the filename simple.
If you want your config reload to happen at a different time scale than hourly, other parts of the cron system enable this, so you can edit crontab, cron.daily, etc.

Now run two commands:

systemctl daemon-reload 
systemctl restart telegraf

and Telegraf will now complain in its log file that it can’t get a configuration from anywhere. We’d better jump to Node-RED and sort this out!

Before we run off, a quick note for readers running Telegraf on Windows. Edit the registry, go to:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\telegraf

Change ImagePath to this (change the IP address to that of your own Node-RED server):

"C:\Program Files(x86)\Telegraf\telegraf.exe" --config 
"http://192.168.1.8:1880/telegraf/%ComputerName%"

Either reboot, or run “Services” and restart Telegraf.

This will NOT cause Telegraf to regularly reload its configuration in Windows. Just reboot the PC or restart the Telegraf service for this to happen.

Now, let’s run to Node-RED.

Create a new tab in Node-RED. Add the required nodes into it for configs.

The first node, “timestamp”, updates the configs every 10 minutes.

The next nodes set properties for private settings. These are “Change” nodes, and are where I’ll add all my personal information for connecting and authenticating against my Cloud instances of InfluxDB. They look like this:

Now we need to retrieve the configs from wherever they are stored. I created nodes like this:

This is the time where you can get really creative! You might want to store your configs in GitHub or GitLab, or you might be into Software-Defined Networking (config-as-a-service) and so want to send updates over something like a messaging topic. Updated config data can come from many places, and the only difference in Node-RED is the type of node used to update the config information:

I prefer to use Node-RED to directly store my configs, so my config nodes look something like this:

Note the “mustache” template for the options urls, database, username and password. These pieces of information are retrieved from the “Change” nodes and inserted into here, so the data is always correct and current.

Multiple config snippets can exist in one template node. Create each template node as you would like, for best readability, understanding, and appropriate re-use, such as this one for a set of core monitoring inputs:

After a deployment of the flow, all of our config snippets are loaded into Node-RED, and refreshed every 10 minutes. The final task is to serve a web page for Telegraf to use:

The first node in this flow is a “http-in” node. It listens for GET requests on http://192.168.1.8:1880/telegraf/:hostname.

The :hostname automatically becomes a property in Node-RED, which we will use in the “Select on hostname” Switch node. Remember when we configured Telegraf with %H? That parameter replacement is what we’re using here.

The Switch node matches against the hostname, and sends the request down a specific path for each host.

As an example to follow the workflow, let’s look at what happens for the host “mqtt”. After the Switch node selects the “mqtt” path, it goes next to a “MQTT config” template node, which enables specific monitoring config for my MQTT implementation.

That config contains my ActiveMQ and Jolokia-2 config information (I use ActiveMQ as my MQTT broker because it has monitoring hooks and a built-in view of all the topics I’m using, along with an indication of how many subscribers I have listening to each topic). The workflow then goes to the Linux config node, which inserts all the standard Linux monitoring configs I want to use, appending the MQTT config at the end:

The final step is a “http-out” node, which responds to the original requestor with the required config data.

Every few minutes, Node-Red logs that a connection has been made and a new config has been served:

How does this look from the Telegraf end? Here’s a section of telegraf.log from the server named “mqtt”, which shows what happens when the reload command is sent to Telegraf, how it writes all buffered points to InfluxDB before it restarts and loads all its updated inputs and outputs.

Note that the first outputs after the reload contain more points than the outputs before the reload. This is Telegraf processing inputs that were buffered while it was reloading its config. During testing, I noticed that, if I write to the Telegraf instance over HTTP while it’s reloading its config, the HTTP port isn’t open, so this could be a consideration for determining how frequently to reload configs.

So all my Telegraf instances are now updating their running config every hour, which means I can tune my jitter and buffers as required, and I can update my configs as I want, for every instance of Telegraf, a subset of instances, or individually, all without needing to log in to a server.

Node-RED can even be configured to commit your tabs to version control, so all the configs are safe and secure.

In effect, I’ve just configured Node-RED-to-Telegraf to run as a configuration management / change management / continuous deployment system, all through UI elements using model-based processes, without needing a single line of code.

I don’t know if I’m going to continue to use this mechanism for Telegraf configs, whether I’ll use the Telegraf Gateway mechanism, or whether I’ll use a combination of both. I’ll run a combination for now and revisit this at a later date to determine the most appropriate deployment mechanism for my use case.

Using A Telegraf Gateway

Rick Brown (InfluxData) — Mon, 18 Nov 2019 09:01:11 -0700

If you meet InfluxData at a trade show, or we provide you with a demo, you might be shown the dashboard called “Rick’s House”. I’m that “Rick”, and this is how I’m sending my data to InfluxDB Cloud, which currently looks like this:

My home network contains numerous servers (many of them LXC containers in Proxmox), lots of software components (Node-Red, MQTT, MariaDB, etc), and a variety of smart-home sensors (SmartThings, Xiaomi, IKEA). It’s grown over many years, and I now find that I want to monitor the whole set-up in one place, because without monitoring everything I can’t visualize what’s happening, and without visualizing what’s happening, it’s difficult to understand what automations to execute or what maintenance I might need to do. It’s been a long-running vanity project, and I discovered InfluxDB during my journey. It’s partly because I found my “time to awesome” was so fast when implementing the software that I applied to join InfluxData!

So, I want to visualize my home network from data stored in InfluxDB. I want everything to write directly into my local instance of InfluxDB OSS, but I also want to be able to write to remote instances, such as InfluxDB Cloud. I also want this to be usable for larger use cases than mine. To do this, there are some restrictions:

InfluxDB Cloud can only accept HTTP ingest. Any UDP messages (Proxmox writes UDP to an InfluxDB endpoint) will be dropped before they get through the Internet.
Usernames and passwords might change, so I will need to edit every Telegraf configuration file on every machine on my LAN.
Access tokens might change over time (especially as I migrate to InfluxDB Cloud v2), so I will need to edit every Telegraf configuration file on every machine on my LAN.
I've had instances in corporate networks where I had to go through long security processes to open an outgoing TCP port from a host in a data centre. If I need to do this for many hosts in order to get Telegraf data sent to a remote InfluxDB, the elapsed time for security clearance would be correspondingly longer.
I want to be efficient sending data to remote instances of InfluxDB. This means that I should look at concepts like batching my outputs. If I write to InfluxDB with a Telegraf Gateway, batching and associated settings can be configured in that collator, and those settings are used for all writes. If I write directly to InfluxDB from client libraries, each one might have a different mechanism for batching, so I'll need to configure it in multiple places.

If I can collate all my updates into one place, all the administration of the points above goes away.

Therefore, implementing an interstitial instance of Telegraf — a Telegraf Gateway — is a good solution for these use cases. To create it, I’ll need to do some things:

1. Install a container for Telegraf
2. Enable Telegraf inputs for UDP & HTTP
3. Enable Telegraf outputs for wherever I'll be writing my data

So, I need to create a configuration for my Telegraf gateway. Telegraf reads default configurations from /etc/telegraf/telegraf.conf and /etc/telegraf/telegraf.d/.conf, so let’s create these.

Firstly, here is my /etc/telegraf/telegraf.conf for global settings

[[global_tags]]

[[agent]]
  interval = "10s"
  round_interval = true

  metric_batch_size = 1000   
  metric_buffer_limit = 10000

  collection_jitter = "3s"
  flush_interval = "10s"
  flush_jitter = "5s"

  precision = ""
   
  debug = false
  quiet = false
   
  logfile = "/var/log/telegraf/telegraf.log"   
  logfile_rotation_interval = "0d"
  logfile_rotation_max_size = "1MB"
  logfile_rotation_max_archives = 5

  hostname = ""

Now some specific config files for providing endpoints for Telegraf to act as an InfluxDB server:

/etc/telegraf/telegraf.d/socket_listener.conf

[[inputs.socket_listener]]
  service_address = "udp://:8089"

/etc/telegraf/telegraf.d/influxdb_listener.conf

# Influx HTTP write listener
[[inputs.influxdb_listener]]
  service_address = ":8086"

  read_timeout = "10s"
  write_timeout = "10s"

  max_body_size = "500MiB"
  max_line_size = "64KiB"

  database_tag = "bucket_name"

Note the use of database_tag = "bucket_name" in the config above. When I write directly to an InfluxDB HTTP endpoint using its URL, which I do from Node-Red, I select a database to write to. I don’t want to lose this information, so Telegraf stores it in a tag called bucket_name. Of course, I don’t want that sent as a tag to InfluxDB, because it’ll increase my cardinality for no purpose, so the output configuration will need to remove it. Telegraf does this by the configuration line exclude_database_tag = true. Here’s the output configuration:

/etc/telegraf/telegraf.d/output_influxdb.conf

# Local InfluxDB
[[outputs.influxdb]]
  urls = ["http://ip_address_of_local_InfluxDB_server:8086"]
  database_tag = "bucket_name"
  exclude_database_tag = true
   
# SE Cloud
[[outputs.influxdb]]
  urls = ["https://FQDN_of_Influx_Cloud:8086"]
  database = "database_name_to_write_to"
  username = "my_username"
  password = "my_password"   
  timeout = "30s"

# Cloud 2 instance
[[outputs.influxdb_v2]]
  urls = ["https://FQDN_of_Influx_Cloud_2"]
  token = "The_Token_Generated_within_Influx_Cloud_2"   
  organization = "my_registered_email_address_on_Influx_Cloud_2"   
  bucket = "my_bucket_to_write_to"

At this point, this gives me a working Telegraf gateway. However, it also gives me an extra thing to monitor on my LAN, so capturing internal Telegraf metrics seems like the thing to do. As this is a LXC container, and I want to monitor all my containers in the same way, I should add my usual set of OS monitoring configurations. I shouldn’t enable SMART monitoring or temperature monitoring, as those should come from the underlying server on which I’m running the container. Additionally, if this was a Windows machine rather than a LXC container, I would want to replace “processes” and “system” with “win_processes” and “win_perf_counters”. These are the defaults that I try and use with every server I include in my dashboards:

/etc/telegraf/telegraf.d/cpu.conf

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false

/etc/telegraf/telegraf.d/disk.conf

[[inputs.disk]]  
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

[[inputs.diskio]]

/etc/telegraf/telegraf.d/internal.conf

[[inputs.internal]]  
  collect_memstats = true

/etc/telegraf/telegraf.d/kernel.conf

[[inputs.kernel]]  
  # no configuration

/etc/telegraf/telegraf.d/mem.conf

[[inputs.mem]]  
  # no configuration

/etc/telegraf/telegraf.d/net.conf

[[inputs.net]]

/etc/telegraf/telegraf.d/swap.conf

[[inputs.swap]]  
  # no configuration

The final two sections are for capturing Linux metrics - see the equivalents mentioned above for Windows metrics.

/etc/telegraf/telegraf.d/processes.conf

[[inputs.processes]]  
  # no configuration

/etc/telegraf/telegraf.d/system.conf

[[inputs.system]]  
  ## Uncomment to remove deprecated metrics.  
  # fielddrop = ["uptime_format"]

So now Telegraf will write to all three of my outputs, collating data from all three of the types of input I commonly use (Telegraf, HTTP custom endpoint, UDP).

This is what my Telegraf gateway looks like, showing a system view:

I’ll be running my Telegraf implementation like this for a while, to see if this meets my needs as-is, or if it needs extra configuration. If you deploy Telegraf like this, or use alternative ways to get your data through Telegraf into InfluxDB, let me know - I’m interested in Telegraf deployment patterns!