How to Use InfluxDB with Its Python Client on Kubernetes

By Community / Product, Use Cases, Developer
Jan 07, 2021

This article, first published in The New Stack, was written by Saiyam Pathak, InfluxAce and Director of Technical Evangelism at Civo Cloud. 

In this tutorial, we will discuss InfluxDB and its Python client. We will deploy InfluxDB inside a Kubernetes cluster and then use the InfluxDB Python client to send data to InfluxDB.

What you will learn?

• How to deploy InfluxDB to a Kubernetes cluster (quickstart way)
• How to use the InfluxDB Python client

Prerequisites

• A Kubernetes cluster you control. We'll take advantage of Civo's super-fast managed K3s service to experiment with this quickly. If you don't yet have an account, sign up for the beta now to take advantage of quick deployment times and $70 free credit per month. Alternatively, you could also use any other Kubernetes cluster.
• Install and set up kubectl, and have the kubeconfig file for your cluster downloaded.

Make sure you can connect to your Kubernetes cluster by running:

kubectl get nodes
NAME               STATUS   ROLES    AGE   VERSION
kube-master-18e1   Ready    master   8h    v1.18.6+k3s1
kube-node-4e70     Ready    <none>   8h    v1.18.6+k3s1
kube-node-d58a     Ready    <none>   8h    v1.18.6+k3s1

You should see the names of the nodes in your cluster displayed.

Getting up and running with InfluxDB

Clone the repository: https://github.com/saiyam1814/pyconf.git

git clone https://github.com/saiyam1814/pyconf.git
cd pyconf/deploy
# Apply the influx.yaml file
kubectl create -f influx.yaml
namespace/influxdb created
statefulset.apps/influxdb created
service/influxdb created

The above script creates the Namespace, StatefulSet and service for InfluxDB version 2.0.1.

Now we will create the ingress, and for that, we will modify the ing.yaml file and input the DNS name of the created cluster. You can get the DNS name from the dashboard.

In the host section of ing.yaml, point to influx.{DNS NAME}.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: influxdb
 namespace: influxdb
spec:
 rules:
 - host: influxdb.360dfac7-2adc-4e60-bdcb-0b7a283ac3c8.k8s.civo.com
   http:
     paths:
     - backend:
         serviceName: influxdb
         servicePort: 8086

kubectl create -f ing.yml
ingress.extensions/influxdb created
kubectl get all -n influxdb
NAME             READY   STATUS    RESTARTS   AGE
pod/influxdb-0   1/1     Running   0          6m45s
NAME               TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE
service/influxdb   ClusterIP   192.168.145.255   <none>        8086/TCP   6m45s
NAME                        READY   AGE
statefulset.apps/influxdb   1/1     6m46s

kubectl get ing -n influxdb
'NAME       CLASS    HOSTS                                                        ADDRESS         PORTS   AGE
influxdb   <none>   influxdb.360dfac7-2adc-4e60-bdcb-0b7a283ac3c8.k8s.civo.com   91.211.153.65   80      55s

Navigate to the HOST ADDRESS:

Now, in the Advanced section from the Token, select and copy the generated token — as that would be required for connecting to InfluxDB via the Python client.

Next steps

Create Kubernetes Secret from the token 

kubectl create secret generic influx --from-literal=token="qWpu90WO31r02miedaP7-BXG9hmrtfBCPgncqu3PsU-PzZZ3jrg7eC9RE2yZzLurhNlk8tr_maKYE3zpk1GJ2A=="
secret/influx created

Create config map for the Ingress HOST Address

kubectl create configmap host --from-literal=host="influxdb.360dfac7-2adc-4e60-bdcb-0b7a283ac3c8.k8s.civo.com"
configmap/host created

Create the Daemonset from the Deploy Folder

kubectl create -f ds.yaml
daemonset.apps/pyconf-demo created
kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
pyconf-demo-dq5c9   1/1     Running   0          40m
pyconf-demo-lbxww   1/1     Running   0          40m
pyconf-demo-q65ps   1/1     Running   0          86s

Viewing from InfluxDB UI

Understanding the connection and writing points to InfluxDB via its Python Client

from influxdb_client import InfluxDBClient, Point, WritePrecision
from datetime import datetime
from influxdb_client.client.write_api import SYNCHRONOUS
import schedule
import time
import socket
import os
def influx():
   print("starting")
   org = "demo"
   bucket = "demo"
   client = InfluxDBClient(url="{}".format(os.environ.get('host')), token="{}".format(os.environ.get('token')))
   meminfo = dict((i.split()[0].rstrip(':'),int(i.split()[1])) for i in open('/proc/meminfo').readlines())
   freemem = meminfo['MemFree'] / 1024 /1024
   write_api = client.write_api(write_options=SYNCHRONOUS)
   point = Point("free_mem").tag("host", socket.gethostname()).field("free_memory_Gb", freemem ).time(datetime.utcnow(), WritePrecision.NS)
   write_api.write(bucket, org, point)
schedule.every(5).seconds.do(influx)  
while 1:
   schedule.run_pending()
   time.sleep(1)

• Import influxdb-client.
• Org: Corresponds to the org where data has to be pushed.
• Bucket: Corresponds to the bucket where data has to be pushed.
• Client: The connection created by proving the host and the token.
• Freemem: To get free memory of the host in Gb.
• client.write_api for writing to InfluxDB.
• Point: The actual point that will be creating measurement free_mem, tag host and field free_mem_Gb.
• write_api.write(bucket,org,point)for the insertion to the Db.
• Last section makes it run every 5 seconds.

Use cases

You might have certain scripts that would capture some data, or do some level of processing and then create the data, which can then be sent via the Python client to InfluxDB (for this particular use case). There are a lot of client libraries that are supported and can be used to send the data.

Summary

We have seen how to deploy InfluxDB on an existing K3s Kubernetes cluster. Use the Python client to write data to InfluxDB, and then view the data from InfluxDB UI.