InfluxData Blog - Jack Zampolin

InfluxData On The Road: CloudFoundry Summit

Jack Zampolin (InfluxData) — Mon, 10 Jul 2017 14:36:32 -0700

CloudFoundry Summit

InfluxData was at CloudFoundry summit last week! We met a bunch of users who were very excited to see the new version of Chronograf.

CloudFoundry and InfluxDB

While InfluxDB can ingest metrics from a variety of sources, each one requires something different. In CloudFoundry metrics (application, host, container, etc…) are aggregated in a system called the Loggregator. This is extremely useful, but requires a special piece of infrastructure called a “Nozzle” to read off of the “Firehose”, the stream of metrics coming out of Loggregator.

Prior to the conference I found an existing implementation of a Firehose Nozzle that outputs to InfluxDB. I updated it to work with the latest CF as well as the latest InfluxDB. We also did a webinar on deploying that Firehose that I would encourage you to check out. Also if you have any feedback please let us know! If it is an issue with the code, please open an issue on the repo. Or, if I’m approaching this all wrong, you can let me know over on community.influxdata.com.

While at the show we met a number of InfluxData users who deploy on CloudFoundry. ExpressScripts and the ECSTeam introduced us to some work they had done on an InfluxDB firehose nozzle (complete with a Bosh Release) that they run in production. They also have a tile to deploy an InfluxDB instance in your cluster!

Telegraf Update - 1.3 and How to Write Plugins

Jack Zampolin (InfluxData) — Fri, 09 Jun 2017 10:20:25 -0700

We recently released the 1.3 version of Telegraf.

Have a python script that grabs some custom metrics? Use the exec plugin. Have some logfiles that contain metrics? Use the logparser plugin. Need to measure API response time? The ping plugin has your back. With over 300 different committers and over 100 plugins in Telegraf, it’s the data Swiss Army Knife you need.

What makes Telegraf such an active project? The ease of plugin development. Below I’ll briefly describe that architecture and show how easy it is to contribute to Telegraf.

Telegraf Architecture

Telegraf is a configuration-driven agent. Each of the input plugins satisfies a simple golang interface:

type Input interface {
  // SampleConfig returns the default configuration of the Input
  SampleConfig() string
  // Description returns a one-sentence description on the Input
  Description() string
  // Gather takes in an accumulator and adds the metrics that the Input
  // gathers. This is called every "interval"
  Gather(Accumulator) error
}

The SampleConfig() and Description() functions are used when generating the configuration file. The real magic happens in the Gather() function. The Accumulator that gets passed in is shared by all the plugins. It takes a representation of a single metric from the plugin and gives it to Telegraf:

Accumulator.AddFields(measurement, fields, tags, time)

The Gather function gets called on an interval that is set in the configuration file ([agent]interval). Every ‘interval’ Telegraf calls the Gather function for each of its plugins and stores the resulting metrics from all the AddFields calls in its metric buffer.

The output interface is slightly more complex than the input interface as it needs to deal with database connections and writes:

type Output interface {
  // Connect to the Output
  Connect() error
  // Close any connections to the Output
  Close() error
  // Description returns a one-sentence description on the Output
  Description() string
  // SampleConfig returns the default configuration of the Output
  SampleConfig() string
  // Write takes in group of points to be written to the Output
  Write(metrics []Metric) error
}

The metric buffer then gets cleared by the configured output plugins every flush interval ([agent]flush_interval) by a Write() call. Connect() and Close() help manage the connection to the metric output and are called on startup and shutdown by Telegraf.

This simple, yet powerful architecture is very extensible and makes Telegraf easy to contribute to.

Config Generation

One of the cool features the above architecture allows for is the config file generation which makes Telegraf self-documenting. I’ve found it particularly useful for getting monitoring started on a new project, or seeing what kind of metrics I can get out of existing systems.

For example if you are trying to see what kind of metrics Telegraf can generate from your databases, you might generate the following config:

$ telegraf -sample-config -input-filter elasticsearch:mysql:mongodb -output-filter influxdb

And if you have a RabbitMQ instance you would like to route the data through on the way to InfluxDB, Telegraf makes that easy too:

$ telegraf -sample-config -input-filter elasticsearch:mysql:mongodb -output-filter amqp
$ telegraf -sample-config -input-filter amqp_consumer -output-filter influxdb

I could go on about processor and aggregator plugins, queuing integrations, input and output service plugins, and the countless things that make Telegraf cool, but I’ll leave those topics for another day!

Contributors

I also want to take this opportunity to thank our contributors to the 1.3 release: @ablagoev, @AntoineAugusti, @bullshit, @calerogers, @cosmopetrich, @d-ulyanov, @danielnelson, @ddryden, @djjorjinho, @einhirn, @francois2metz, @gioxchopper, @j-vizcaino, @jackzampolin, @JamesClonk, @jeremydenoun, @johnrengelman, @ldep30, @lpic10, @lpic10, @lrsmith, @martinseener, @nevins-b, @nfirvine, @njwhite, @pdcleyn, @phemmer, @PierreF, @ririsoft, @rossmcdonald, @sebito91, @seuf, @sparrc, @ssorathia, @tjmcs, @vlasad, @vtg

Helpful Resources

We have a number of great resources to help you get started with Telegraf as well as information on how to build your own plugin.

Draft for Kubernetes - A Prototyping Tool

Jack Zampolin (InfluxData) — Tue, 06 Jun 2017 04:00:26 -0700

Overview

Draft for Kubernetes is a tool to help prototype microservices and expose them on a publicly available domain. This makes quickly iterating on things like webhooks or APIs quick and painless. Draft utilizes docker, Helm, and Kubernetes Ingress Controllers to make your ‘drafts’ publicly available in a hurry. Under the hood, it generates a generic Dockerfile and helm chart for your application to get you up and running quickly. The helm chart creates an ingress resource to expose your chart at a url. You will need the following running for this walkthrough:

A Kubernetes Cluster
- I've developed this walkthrough on GKE.
- It's pretty easy to get up and running there.
A domain name you can add A or CNAME records to
An account at quay.io or hub.docker
A working Helm installation
- Install instructions can be found here.
A golang development environment
- Draft also supports basic python, ruby, php, node, and java applications.

In this walkthrough, we are going to first install the dependencies for Draft, then create our sample application, install Draft and deploy our app.

Installing Draft Dependencies

An Ingress Controller creates a loadBalancer with a Public IP (or a CNAME record on AWS) to expose your applications to the internet. It is a quick and secure abstraction for routing traffic into the applications running on your cluster. To install an Ingress Controller, use the Nginx Ingress provided by helm:

# Install Ingress Controller
$ helm install stable/nginx-ingress --namespace=kube-system --name=nginx-ingress
# Wait for load balancer to have IP
$ kubectl --namespace kube-system get services -w nginx-ingress-nginx-ingress-controller

# If you have an existing ingress controller find its IP 
echo $(kubectl get svc --namespace kube-system nginx-ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Once you get the IP or CNAME you can create the appropriate DNS record. Make it something like *.draft.mydomain.mytld. The steps to do this will vary depending on where your domain is registered. I use What’s My DNS to check propagation. Once you see that the change has propagated, you should get the following response from curl:

$ curl foo.draft.mydomain.mytld
default backend - 404

Note: If you want to know more about default backends, I've found this article scintillating reading.

Prepare Golang App

Make a new project directory and create a golang application. I’ve used a simple server:

// main.go
 package main

import (
  "log"
  "net/http"
)

func sayHello(w http.ResponseWriter, r *http.Request) {
  message := "hello world\n"
  w.Write([]byte(message))
}

func main() {
  http.HandleFunc("/", sayHello)
  err := http.ListenAndServe(":8080", nil)
  if err != nil {
    log.Fatal("ListenAndServe: ", err)
  }
}

You will also want to add some type of dependency management to your application. This is because of the script Draft uses to detect different application types. The golang pack looks to see if there is a glide.yaml, Godeps/Godeps.json, or vendor/vendor.json file in your project. If you are using glide, a simple glide create will satisfy this requirement.

Install Draft

Download the draft binary for your OS and move it into your path. After that you will need to run draft init with some arguments:

draft init --set
 registry.url=docker.io,
 registry.org=mydockerorg,
 registry.authtoken=$(echo '{"username":"myDockerUsername","password":"myDockerPassword","email":"my@email.com"}' | base64),
 basedomain=mydomain.mytld

Note: Be sure to run the above command all on one line with no spaces between the different key=value pairs! Also, I had some trouble getting the Google Container Registry to work, but it seems to be supported.

This spins up a draftd deployment and some draft pods in your kube-system namespace. These help manage the connection with the docker image repository.

Deploy Your App!

Phew! That was a lot of setup! Now that this is done, it’s time to use Draft! The first command to run is draft create. That will scaffold out your helm chart (/chart) and /Dockerfile. It will also create a draft.toml that allows you to control different environments for your application as well as where it is installed in your cluster. To give your app a name (the default works just fine too!) edit draft.toml:

[environments]
  [environments.development]
    name = "cool-beans"
    namespace = "draft"
    watch = true
    watch_delay = 2

Then run draft up and watch the magic happen:

$ draft up
 --> Building Dockerfile
 Step 1 : FROM golang:onbuild
 # Executing 3 build triggers...
 Step 1 : COPY . /go/src/app
 Step 1 : RUN go-wrapper download
 ---> Running in 825771ecaf59
 ...
 # Some docker output...
 ...
 --> Deploying to Kubernetes
 --> Status: DEPLOYED
 --> Notes:
http://cool-beans.mydomain.mytld to access your application

Watching local files for changes...

This will build your docker image, push your docker image to the configured container registry under the {{ .environments.development.name }} repository, and run helm install on the preconfigured helm chart at the root of the repo! It might take a second for all the resources to spin up and for the Ingress to register your new application, but you will be able to curl your application at that address:

$ curl http://cool-beans.mydomain.mytld
 hello world

But, to see the real magic, make a change to your application:

// main.go

func sayHello(w http.ResponseWriter, r *http.Request) {
  message := "I don't know Margo...\n"
  w.Write([]byte(message))
}

Draft will rebuild your image and push those changes live with a helm upgrade:

$ curl http://cool-beans.mydomain.mytld
 I don't know Margo...

Hooking in a datasource or performing more advanced actions is just a handler edit away now!

InfluxData and Docker

Jack Zampolin (InfluxData) — Mon, 17 Apr 2017 08:00:52 -0700

Getting ready for the 2017 DockerCon this past week has had me thinking about all of the different ways we use Docker at InfluxData. From our official images, backing InfluxDB Cloud, and in our everyday development work. Furthermore, we share a commitment to open source and here are a couple of ways the two are great together.

Shared Commitment to Open Source

Both Docker and InfluxData are major open source contributors. All of InfluxData’s major projects InfluxDB, Kapacitor, Telegraf, and Chronograf as well as Docker’s Docker Community Edition and ContainerD are open source. Developers trust open source software due to its inherent visibility, transparency, and active communities behind the projects—characteristics shared by both Docker and InfluxData.

InfluxData Sandbox and the Official Docker Images

We recently launched InfluxData’s official Sandbox for experimenting with the full TICK Stack in an easy-to–get-started way. It is at its core a docker-compose.yml file and a script to help users unfamiliar with docker accomplish common tasks.

With millions of pulls, the images behind sandbox are production tested and included in the official Docker image repository. I know this has saved me a bunch of time, and really helps me get started on projects quickly! Want to spin up monitoring for an IOT side project? Need to do a proof of concept for your new monitoring system? No problem. There is also an example docker-compose file in the sandbox if you want to run the full stack together and want a place to start. Here are the links to our official Docker images:

Autoscaling and Docker

We recently did a webinar on how to automatically scale containers running on Docker Swarm, or the infrastructure underlying those containers, using the TICK stack. We wrote a small Golang program called Orbiter to make sending scaling messages to different providers a breeze. For the webinar, we implemented Digital Ocean and Docker Swarm as scaling targets.

Using Kapacitor to trigger autoscaling makes it very easy to setup, and allows you to target different metrics to scale on. Want to scale your RabbitMQ Cluster based on queue depth, or your Nginx servers based on requests per second? No problem with Orbiter and the TICK Stack.

In addition to scaling Docker Swarm we also have a Kapacitor alert handler to scale Kubernetes deployments based on any metric you choose. Want to scale your CI workers based on … the weather outside? Well that might be getting carried away, but with a Raspberry Pi and some gumption you could make it happen!

InfluxDB Cloud

At InfluxData, our cloud service (cloud.influxdata.com—go check it out!) runs on Docker. Even the smallest sized cluster requires at least four different pieces of infrastructure with multiple copies of each. While this can be painful with some configuration management tools, with Docker it is a breeze. By eating our own dogfood we can tighten the iteration cycles on our products and deliver a better experience to all of our users.

DockerCon

Last but not least, InfluxData is at the 2017 DockerCon! Don’t forget to drop by the booth and come say hello! Tim Hall and I will be there all week.

Denver DevOps Days

Jack Zampolin (InfluxData) — Fri, 14 Apr 2017 16:54:38 -0700

We had a great DevOps Days in Denver! The venue had an excellent cloud theme, an energetic crowd, and a hard-working group of organizers, making for a smooth and fun conference.

This DevOps Days continued the tradition of excellent talks and breakout sessions. Of the talks, I found the one about the recent GitLab Database outage from their Community Advocate, Connor Shea most interesting. If you haven’t already, go check out the postmortem.

InfluxData will be at these upcoming events:

Dockercon - Austin, April 17-21
DevOpsDays - Austin, May 4-5

Autoscaling Docker Swarm with the TICK Stack

Jack Zampolin (InfluxData) — Thu, 06 Apr 2017 04:00:37 -0700

TICK Stack and Docker

With each successive release of the TICK Stack, InfluxData is improving the ease of use, depth of metrics, and level of control we provide in maintaining and monitoring Docker installations. InfluxDB has been an integral part of most Docker monitoring solutions.

From its use as a primary sink for CAdvisor, to our native Telegraf plugin for gathering container level metrics, InfluxDB and the TICK stack have a part to play in monitoring any Docker setup. It can also scale from a single host to a complex multi-host cluster.

Docker Background

Docker has also been moving forward. In their last two releases they have added support for orchestration with their Swarm feature in the core Docker daemon. Docker 1.12 was released in June 2016 and many Docker users began using this version. For Docker 1.13, they focused on feedback from the community and added features such as secret manager and the new CLI re-design.

Now that Docker has solved the orchestration problem and you can manage a cluster of Docker hosts without other tools or frameworks, however, you need to have a solid environment. This means that you need to be able to trust how your system behaves and have visibility into each part. This makes logging and monitoring skills mandatory for any Docker Swarm administrator.

That might leave you asking questions like: “How can I scale application components when my queue backs up or my cache fills up?”

Orbiter

Enter Orbiter, an open source tool to scale Docker Swarm clusters. You can use this piece of infrastructure along with Kapacitor to scale services on your Swarm cluster based on any of your monitoring metrics. The README on Orbiter contains the information needed to spin the system up or down. There is also a webinar with a demo showing how to use Orbiter to scale your Docker Swarm.

What are you waiting for? Go try it out!

Resources

Announcing the InfluxData Sandbox

Jack Zampolin (InfluxData) — Wed, 05 Apr 2017 04:00:27 -0700

Today I’m happy to announce the InfluxData sandbox. The sandbox runs the entire suite of InfluxData products for use as a learning tool. With all of our previous full stack tutorials, such as this one for Kapacitor, you need to install each of the products separately and configure them to work together. This sandbox eliminates all the extra work and confusion.

Working with the Sandbox

Many operations have additional steps to be taken at each level of the stack. For example, adding a new input to Telegraf gives additional metrics to alert on, graph, and downsample. The sandbox makes the setup transparent so that you can focus on understanding the task you need to perform.

Working with the sandbox is meant to simulate a production InfluxData installation running on separate machines without the management complexity. Currently the sandbox hosts a number of tutorials for setting up authentication for the TICK Stack, and creating your first alert in Kapacitor using the UI in Chronograf. We will be adding more tutorials to the sandbox as features are added and at users’ requests. To request a new tutorial open an issue on sandbox, or post your request over on community.influxdata.com.

This project was built using Docker. The necessary network and volume configurations are specified in the ./docker-compose.yaml file at the root of the project. That file contains comments to show how to change the versions of each product run by the sandbox.

Get the Sandbox!

The sandbox is available on github (influxdata/sandbox). To get started just clone the repo and run ./sandbox up. This downloads docker images for each of the products and starts them in the proper configuration. You browser will open Chronograf and the tutorials in separate tabs. Just follow the quick instructions in the README.md to connect InfluxDB and Kapacitor to Chronograf and you are ready to go!

The sandbox persists data in the cloned directory to make for easy install/uninstall as well as inspection of the data directories for each product. This means that you can stop the standbox and easily come back to your previous state, or just as easily wipe everything and start from scratch.

So what are you waiting for? Go give it a try!

tick-charts Update

Jack Zampolin (InfluxData) — Wed, 29 Mar 2017 04:00:39 -0700

This is an update to an earlier blog post I did on tick-charts, the project to make the InfluxData stack as easy to run on Kubernetes as possible. While developing the project I’ve learned that some of the original implementation decisions were not the right way to go! Also, Chronograf has matured significantly since that time and has required additions to the chart to support a few different flavors of OAuth: Heroku, Github, and Google.

Telegraf Changes

The primary change from the first version is the re-architecture of the telegraf chart. Initially I thought that a monolithic chart for telegraf would be the best way to go. That chart put host-level monitoring, as well as polling, in one values.yaml file. This becomes complicated because telegraf is primarily configuration driven, and holding both configurations in one file quickly becomes too complex. Another downside of that chart was that spinning up a single telegraf instance required deleting a bunch of configurations for the daemonset.

To reduce this complexity I’ve taken a two-prong approach. First I split up the telegraf chart into two, providing a much cleaner interface for users. Next I reduced the amount of configuration required for the daemonset (telegraf-ds). The defaults provide the basics for host-level monitoring in Kubernetes: Kubelet and Docker polling, cpu, mem, disk, system load, and network statistics. All that is required of the user is to set config.outputs.influxdb.url. If you don’t want to host your InfluxDB, you can easily spin up an InfluxDB Cloud instance to hold the data.

The chart for spinning up single telegraf instances is now called telegraf-s. Currently, it is implemented in much the same way as the old telegraf chart: using some custom golang templates to generate the configuration. This is difficult for plugins that require a substantial amount of configuration such as snmp or jolokia and is error prone. In order to eliminate that complexity and the difficulty of maintaining custom code, I’ve added a toToml template function to Helm. Once the 2.3 release of Helm is available, the telegraf-s chart will be modified to use it. This will make creating telegraf instances in your cluster extremely easy. Using a tool like remarshal to convert the tomloutput by telegraf into yaml, the workflow for generating a new telegraf instance to monitor a piece of your cluster will look like this:

# On mac at least...
$ telegraf -sample-config -input-filter nginx:cloudwatch -output-filter 
kafka_producer | toml2yaml | pbcopy

The resulting blob of yaml can be added directly to your values.yaml file under the config section and edited there.

Next Steps

I plan to continue improving the production readiness, examples, and user experience for deploying the TICK stack. The following items are on my # TODO: for this project:

Production Readiness
- Make InfluxDB deploy with basic authentication enabled
- Add backup/restore job examples to back up InfluxDB to s3 or other object store
- Create job to dynamically reload configuration for telegraf on upgrade
User Experience
- Create a top-level chart so that the whole deployment can be managed from one chart
- Reduce the time from zero to dashboards to as short as possible
- Address any pain points that begin to show with increasing usage
Examples
- Monitoring Prometheus endpoints with Telegraf
- Using queues in Kubernetes with Telegraf
- Monitoring {{ .telegraf_plugin.name }} (suggestions welcome!)

I would also love some input as to what features you would like to see in this integration. Please post your questions/concerns/suggestions or other comments on this post on our Community site. I can’t wait to hear from you!

Packaged Kubernetes Deployments - Writing a Helm Chart

Jack Zampolin (InfluxData) — Mon, 13 Mar 2017 04:00:25 -0700

Throughout my time at InfluxData, and even before, I have been working with containers, and occasionally doing useful things with them! This led me to try out Kubernetes, which I have found to be very useful. In my job, I end up having to deploy random code for things like internal dashboards, demos, reporting, and data collection. Much of this code lives on a small Kubernetes cluster that I maintain.

At first when I started running Kubernetes, I found that keeping track of all my *.yaml files to be a pain. Even though I had a repository where I would keep the current state of my deployments, it was a management nightmare. I would end up copying from old projects to get started with new ones, and keeping the repo up to date was a pain! I even wrote a quick little bash program to help with this.

Bottom line is that when you are running Kubernetes, you quickly get to a place where the management of your deployment files is cumbersome and often not using deployment best practices.

Enter helm.

Helm is a templating packaging and deployment manager for Kubernetes. Think of it like apt-get or brew for Kubernetes. It consists of a server that runs in your cluster and renders your templates, tiller, and a command line interface helm. (If you’re totally new to Helm charts, scroll to the bottom of this article for some handy reference content).

Through this experience, I thought it would make sense to develop a quick guide to writing a simple helm chart with a focus on workflow. If you want to follow along, download helm. Please note: A familiarity with Kubernetes and Docker is required.

1. Find an application to deploy

I’ll be deploying a small node app that has been sitting in my Github. WEC is a small nodejs application that subscribes to the Wikipedia Recent Changes feed. When it receives an event, it parses and inserts the event into an InfluxDB instance. I’m using the excellent node-influx/node-influx Javascript client to write to the database.

You can check out the code in /server.js.

2. Dockerize it!

When creating a helm chart, you first need an image to run. I’ve created a Dockerfile for this server. The next step is to push it to a Docker Registry that is accessible from your cluster. I use gcr.io because of its ease of use in Google Cloud. Other options are Docker Hub and Quay.io. You could also push to a private registry. To build and push the image from this repository run:

$ cd wec
$ export ORG=myorg
$ export REGISTRY=gcr.io
$ docker build -t $REGISTRY/$ORG/wec:latest .
$ docker push $REGISTRY/$ORG/wec:latest

3. Stub out helm chart

The helm cli has a command to stub out your chart. This makes getting projects started quick and easy. To create the chart files for WEC just run helm create wec. A directory, wec, will be created in the current directory with the following files:

$ tree wec/
wec/
??? Chart.yaml
??? charts
??? templates
?   ??? NOTES.txt
?   ??? _helpers.tpl
?   ??? deployment.yaml
?   ??? service.yaml
??? values.yaml

2 directories, 6 files

Chart.yaml contains metadata about the chart. You can use the defaults.
charts/ holds any dependency charts. You can delete this folder now. This is used when building a chart from component charts.
templates/ holds all of your template files! This is where you do your work.
values.yaml holds any variables you want to reference in your templates. These are the configuration options you present to the end user.

I’ve moved the chart into a directory called chart.

4. Write your .yaml files

Helm starts you off with a deployment.yaml and a service.yaml. If you need other types of Kubernetes objects in your chart the kubernetes/charts repo has some great examples:

Helm uses Golang templates as a tempting language. It will expand any variables in {{}}. As a chart author, you have some built-in objects available for reference in your chart files (e.g. {{ .Release.Namespace }}) as well as the values.yaml file (e.g. {{ .Values.database_name }}). There are two sets of template helper functions as well: Sprig and Golang template functions.

Conveniently, for wec we only require a deployment. wec subscribes to the Wikipedia feed and receives the data back without needing to expose an endpoint! Very easy! We will need to set a couple of ENV as well to point to the database, other than that the stock file looks the same. We want to make those database connection parameters available in the values.yaml file to make database configuration easy. Here is our deployment file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  # Template helpers are stored in /templates/_helpers.yaml
  name: {{ template "fullname" . }}
  labels:
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ template "fullname" . }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        env: 
        - name: DATABSE_NAME
          # `quote`-ing strings prevents type errors on yaml parsing
          value: {{ .Values.database_name | quote }}
        - name: INFLUX_URL
          value: {{ .Values.influx_url | quote }}
        - name: INFLUX_PORT
          value: {{ .Values.influx_port | quote }}
        - name: BATCH_SIZE
          value: {{ .Values.batch_size | quote }}
        - name: CONNECTION_URL
          value: {{ .Values.connection_url | quote }}

And the default values.yaml:

replicaCount: 1
image:
repository: gcr.io/influx-perf-testing/wec
tag: latest
pullPolicy: Always
database_name: 'wikipedia'
influx_url: "db-influxdb.prod"
influx_port: "8086"
batch_size: 10
connection_url: 'https://stream.wikimedia.org/rc'

5. Test your chart

Now that we have written a chart, we need to see if it produces the proper result. Luckily helm has some utilities to help with this. To see the deployment files that will be output by helm, run the following command:

$ helm install --dry-run --debug --name test --namespace test chart/wec/

This debug output is very helpful in chart development and is a great way to inspect the output of your chart and check for any errors before trying to install it on your cluster.

Tip: If you run helm install without --dry-run --debug and your chart errors out sometimes there is a ghost release left. You can delete image by passing the --purge option to helm delete {{ .Release.Name }}.

6. Deploy it!

When you are ready to deploy your chart just run helm install --name wec --namespace wec ./chart/wec/ to install!

When you run the install, you should see the output of NOTES.txt. This should give instructions on how to health check, or interact with the deployment:

$ helm install --name wec --namespace wec ./chart/wec/
NAME: foo
LAST DEPLOYED: Fri Mar 10 09:04:37 2017
NAMESPACE: bar
STATUS: DEPLOYED

RESOURCES:
==> extensions/v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
foo-wec 1 1 1 0 0s

NOTES:
Watch for you pods to come up by running the following command:

- kubectl get pods --namespace bar -l app=foo-wec -w

Check the logs on a running pod with the following:

- kubectl logs $(kubectl get pods --namespace bar -l app=foo-wec -o jsonpath='{ .items[0].metadata.name }')

Once the deployment is running you should be able to see the events flowing into your InfluxDB instance. Then you can have fun exploring the data! Here is a graph of the number of bot vs human edits on Wikipedia (the humans are winning - for now):

7. Next steps

This guide just scratches the surface of helm charts. Helm also offers hooks to enable running jobs at different points in chart lifecycle. There is also an excellent tips and tricks guide for debugging common issues. And packaging charts for distribution and building chart repos could be another blog post in and of itself. If you have further questions, the helm community is active on the Kubernetes slack in the helm channel.

If you are interested in experimenting more with the InfluxData stack, we have charts for all of our products available in the main charts repo:

For those of you who are just beginning with Helm charts, here are some answers to basic questions about how Helm charts work.

What is a helm chart?

Helm is designed to help you manage all of your Kubernetes applications in one simple, easy-to-use place — and Helm charts are a big part of what helps you define, install and upgrade even the most complicated applications you happen to be working with. A chart is little more than a collection of files that describe a related grouping of Kubernetes resources.

The directory name of the Kubernetes Helm chart in question will just be the name of the chart itself, without any type of versioning information. Inside that directory, Helm will create a structure that includes all listed file names. One of those will be the Chart.yaml file, which is required for a chart to properly function. Here, you’ll find important information like the name of the chart, the API version, the SemVer 2 version, a list of keywords about the project and more. You can also include other optional information like URLs for the source code of the project, the name and other contact information of the person maintaining the project and more.

Why are Helm charts used?

By design, Kubernetes Helm charts make it easy to create, version, share and publish your applications. It’s also built to support workflows that are far more advanced than you can complete with simple copying and pasting alone. A single chart might be used to deploy something simple, or something far more complex like a web app stack complete with servers, caches and more.

Overall, Helm charts are created as files that are laid out in a very particular type of directory tree that makes sense given the project you’re working on at the time. They can then be packaged into archives depending on the most recently updated version, at which point they can be deployed in any way you see fit.

How to create a Helm chart

The process of creating a Helm chart is very straightforward — you can do so right from the command line. All you have to do is use the following command:

helm create [insert name here]

This single command will create a chart directory, along with all of the different common files and directories that are to be used within the chart. But most importantly, that final portion of the command will be the given name for everything that comes after. So if you wanted to create a Helm chart with the name of the project you’re currently working on, you would include it as the last portion of the command prior to execution.

How do I update a deployed helm chart?

Whenever a new version of a particular Helm chart is released, or if you want to make any changes to the configuration of the current release, you can always use the following command:

helm upgrade

This will take your existing release and upgrade everything according to only the information you provide. It’s also important to note that because Kubernetes charts by nature can be incredibly large and complex, Helm will always try to perform the most straightforward upgrade possible. This means that it will update the things that have changed since the last release whenever you run that command. Because of this, Helm chart updates will happen far faster than you likely expect.

Note that if you’re unhappy with any updates and would like to roll back to a previous version, you can also do so using the following command:

helm rollback

What is the anatomy of a helm chart?

While the internal structure of your Helm chart will obviously vary depending on the deployment, any Helm chart tree will contain the aforementioned .yaml file, template files and a notes.txt file. The notes.txt file in particular will always print on the command line whenever you start your application.

How to create a helm chart from a docker image

After you have obtained the appropriate application source code and have built your docker image, the process of creating a new Helm chart from that image is straightforward. All you have to do is publish the image and use the “helm create” command. Then, you can edit the values.yaml file and edit the deployment.yaml file as you see fit to include your own variables.

Note that to publish your docker image, you will need to upload it to a public registry. Just a few of the options that you can choose from include Google Container Registry, Amazon EC2 Container Registry, Azure Container Registry, Quay and, of course, Docker Hub. Simply log into your public registry of choice and publish the image to your account. Once you have confirmed that the image is in your account and is visible on your dashboard, you can continue with the rest of the process.

How to deploy a helm chart in Kubernetes

Once you have connected to your Kubernetes cluster, you can deploy your Helm chart using the following command:

helm install [INSERT NAME HERE] ./helm-chart/go-k8s/

You should replace the [INSERT NAME HERE] placeholder with the name of your actual Helm chart release. Once the Helm chart has been deployed, you’ll be able to gather a lot of useful information about the chart itself that you can use to better manage your projects moving forward. Once you’ve properly configured your database, you’ll also be able to access your application and continue to work on it.