Showcases Category: Kafka

Formula 1 telemetry processing using Kafka Streams

This slides deck comes from the DevConf.cz session about how to build an event streams pipeline running on Kubernetes and using Apache Kafka as ingestion system, Apache Camel for integration and InfluxDB with Grafana for dashboards.

hippo

Hippo is a data ingestor service for gRPC and REST based clients. It publishes your messages on a kafka queue and eventually saves them to influxDB, it is build to be easily scalable and prevent SPOF for your mission-critical data collection.

Horus – GitHub

Horus is a distributed tool to collect snmp and icmp metrics from various network equipments and send the results to Kafka, Prometheus, and/or InfluxDB.

Streaming analysis with Kafka, InfluxDB and Grafana

If you are dealing with the streaming analysis of your data, there are some tools which can offer performing and easy-to-interpret results. First, we have Kafka, which is a distributed streaming platform which allows its users to send and receive live messages containing a bunch of data (you can read more about it here). We will use it as our streaming environment. Then, if we want to visualize, in real time, our results, we need a tool which can capture our data and predictions: it is Grafana, and among its data sources, it can be connected to InfluxDB, an open source time series database. So, through this article we will build an ML algorithm which can extract information and make predictions, in real time, on our data, throughout the following steps:

Monitoring Kafka in Kubernetes

This post focuses on monitoring your Kafka deployment in Kubernetes if you can’t or won’t use Prometheus. Kafka exposes its metrics through JMX. To be able to collect metrics in your favourite reporting backend (e.g. InfluxDB or Graphite) you need a way to query metrics using the JMX protocol and transport them. This is where jmxtrans comes in handy. With a few small tweaks it turns out it’s pretty effective to run this as a sidecar in your Kafka pods, have it query for metrics and transport them into your reporting backend. For the impatient: all sample code is available here.

Doppler Heatmap

This project provides an API for developers to track live geographical data. The process starts by 1) a client sending formatted events to an API which 2) serves the event data onto a messaging queue 3) which is read off by another API, 4) then served up live on a heatmap hosted in the browser. The backend layer is written in Go. Other components used include gRPC, Kafka, Couchbase, InfluxDB, Leaflet Maps, and Heatmap.js.

Scroll to Top