Bring Your Own Telegraf
By David Flanagan / Jun 06, 2019 / Community, Telegraf, Developer, Telegraf, Kubernetes, Docker
I have the privilege of telling people I work for a truly open source company. What do I mean by that?
90% of the InfluxData code is open source, with that other 10% being our enterprise additions to our database, InfluxDB. That 10% fuels the company and it’s open source commitment.
Our commitment to open source doesn’t stop there. We also actively encourage, maintain, and support contributions that integrate our products with other products, OSS or otherwise; even those which some may consider our “competitors”. Alas, we don’t see it this way. We welcome and encourage an open and supporting ecosystem around time series, monitoring, and alerting.
Of course, this comes at a cost. While Telegraf supports reading and writing to over 200 software components; such as DataDog, Elasticsearch, Kafka, MongoDB, OpenTSDB, Prometheus, Wavefront, and much much more this comes at a cost a cost of bytes.
<figcaption> Plugins everywhere</figcaption>
Presently, the Telegraf binary weighs in at
60MiB (amd64 binary),
244MiB (Docker Image), and
80MiB (Docker Image Alpine).
While not massive, depending on your usage of our plugins, this might be too much for you to chew. As more and more of our community adopts Kubernetes, embracing patterns like sidecars, the size of utilising multiple Telegraf instances can become problematic.
Introducing Bring Your Own Telegraf (BYOT)
BYOT is a Docker-based framework for building Docker Images with a custom, slim-line, Telegraf binary; that only includes the plugins you need. This does not require any special configuration. You just need to provide your Telegraf configuration, as you normally would.
This will result in substantially smaller image sizes. Let’s take a look at a comparison. Below are the image sizes for the normal Telegraf image, and the Alpine variant. Next to it is a custom Kubernetes image I built with only the following plugins:
$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE telegraf 1.10 c14f9454c5aa 13 days ago 244MB telegraf 1.10-alpine f6d6ea3f4972 4 days ago 80MB rawkode/telegraf kubernetes 3437fe686f23 4 days ago 25.4MB
My recommended approach is to provide a Git repository for each Telegraf configuration within your infrastructure. Let’s assume we want to build a Telegraf with the following plugins:
- InfluxDB 2
First, we’d create our Telegraf configuration. The build process expects there to be a
telegraf directory in the root of your build context. This directory is no different from your
/etc/telegraf directory on any other system. You can use a single file,
telegraf.conf, or a directory-based approach,
telegraf.d. For simplicity, this example uses a single file.
[agent] interval = "5s" omit_hostname = true [[inputs.docker]] endpoint = "unix:///var/run/docker.sock" [[outputs.influxdb_v2]] urls = ["http://influxdb2:9999"] token = "$INFLUXDB_2_TOKEN" organization = "InfluxData" bucket = "default"
Next you need to add a
Dockerfile to your repository:
FROM rawkode/telegraf:byo AS build FROM alpine:3.7 AS telegraf COPY --from=build /etc/telegraf /etc/telegraf COPY --from=build /go/src/github.com/influxdata/telegraf/telegraf /bin/telegraf ENTRYPOINT [ "/bin/telegraf" ]
That’s it! Have your CI system automagically build this image on every commit and push it to your registry and you’re good to go. You can specify the version of Telegraf to build with a
docker image build --build-arg VERSION=1.10 -t my_telegraf
A custom Telegraf with only the plugins you require, within a few minutes!
How does it work?
BYOT utilizes the often forgotten
ONBUILD instruction available within a
Dockerfile. This allows us to define instructions that are deferred until a subsequent calling, such as your
docker image build command. Feel free to peruse my code, or checkout my example repository, to understand it a little better.
Any other improvements?
During my initial tests, which are not extensive, I’ve noticed that the memory usage of Telegraf is about a 1/3 (
12MiB) of running the normal Telegraf binary with only 4 plugins (system, cpu, mem, disk) enabled. This could be circumstantial and I will publish more results on this in due course.
Currently, BYOT supports custom building with input and output plugins based on your configuration. I plan to add support for aggregators and processors shortly also, which will bring our image size down even further.
Thanks for listening. I’d love to know what you think, so please reach out in the comments section or via Twitter.