InfluxData Blog - Daniella Pontes

Scaling Kubernetes Monitoring without Blind Spots or Operations Burden

Daniella Pontes (InfluxData) — Thu, 21 Nov 2019 07:00:22 -0700

This article was written by Daniella Pontes and Chris Goller.

Kubernetes has seized center stage in the DevOps world when building and migrating applications to a cloud-native environment. In reality, Kubernetes is a means to an end — to scale operations of containerized microservice architecture. Managing deployment, scaling, upgrading and updating of the various microservices of fragmented applications running on containers is not a trivial task, certainly already beyond manual processes. So automation is the only way to go. Kubernetes came to take the role of orchestrating containerized workloads.

InfluxDB Cloud 2.0 Kubernetes monitoring

At InfluxData, we embraced microservices and Kubernetes when building our InfluxDB Cloud 2.0. But we also know that what makes automation reliable is monitoring. Therefore, between the metrics exposed by SREs (to fulfill the need to keep resources and microservices healthy) and the metrics exposed by developers (resulting from their inclination to instrument the code to provide as much information as possible for a potential failure diagnosis), is an ever-growing number of endpoints exposed for scraping in our Kubernetes environment. The exact number of metric endpoints and the functional endpoints themselves are in a continuous state of change due to the ephemeral nature of Kubernetes atomic structure, Pods. So the first question is: how scalable is scraping metrics via these exposed endpoints in Kubernetes? The answer is: It depends on how scraping is implemented.

We know, from our own experience dealing with metrics generation from multiple teams in a large cloud deployment, that there is a turning point of a “label too far” and as a result, you get a blind spot, missing metrics. The scraper cannot cope with pulling an increasing number of the metrics exposed within the polling interval. Increasing the polling interval means reducing the frequency for obtaining metrics and that reduces the availability of critical information for troubleshooting. So lengthening the polling interval is no true solution. (For more details on our journey to master Kubernetes monitoring, please check Chris Goller’s session at InfluxDays San Francisco Oct 2019.)

We then realized that we must think beyond centralized scraping, it proved not to be scalable for large environments — not even with a federated implementation. Since it presses operations to keep up optimum metrics distribution among federated servers, it burdens Ops with yet another monitoring and balancing act. In other words, it is not easily implemented and maintained.

As a principle, a good strategy for scraping shall neither increase the burden on operations nor impose a barrier to democratizing metrics to anyone who needs it. If the larger your system gets, the more complex it becomes to monitor, then fundamentally you don’t have a scalable solution.

The answer to this apparent DevOps dilemma is actually quite simple and is in the heart of Kubernetes’ containerized structure: you just need to contain (isolate) the impact of each exposed metrics. The scraper should be contained inside the pod of the service or workload, that it is going to scrape. The mechanism to implement this approach is called sidecar deployment.

In InfluxDB Cloud 2.0, we have Telegraf, a lightweight, plugin-based, metrics collection agent, deployed as a sidecar in every pod, so all the metrics exposed by the application, service or microservice are handled by that agent and will not impact the scraping of other workloads.

Each Telegraf agent sends its metrics to InfluxDB in the cloud or on-prem, without creating any burden on IT Operations, or a culture of pushing back on developers and data engineers on their needs for metrics.

Watching the watcher

Sidecar deployment addressed the issue of scaling scraping, but our journey to properly monitor our Kubernetes environment did not stop there. A second question still needs to be answered — we need to guarantee that we don’t have blind spots, missing metrics. So we need to be able to watch the watcher.

The answer is in Telegraf one more time. It is actually a very good fit as a monitoring agent, because it takes nothing for granted. Telegraf monitors how well it is doing its job. It has an input plugin for its internal metrics, collecting data such as:

gather_errors
metrics_dropped
metrics_gathered
metrics_written

The screenshot below is from monitored measurements detecting missing metrics. The next step is just to dig into the Telegraf self-watch to find which had metrics dropped and to gather errors.

Other good reasons

Adding capabilities for self-control is also an important feature that a monitoring solution provides. Monitoring metrics is always a balance between cardinality and indexing. The more tags added for description and visualization grouping, the more series are created. This comes at a cost since having more series will take its toll on resources. Telegraf has a very useful feature to keep well-behaving “metrics” citizens behaving well by putting some guard-rails in the agent.

While in Telegraf, this can be easily configured to limit the number of tags taken as preferential ones. Changing a measurement description would not be as straightforward in the case of trying to make this change in a central Prometheus server. For instance, changing the central Prometheus server would require a restart, impacting the metrics gathering of all microservicers being monitored. On the other hand, with a Telegraf sidecar deployment, a restart of a single pod with the single service and telegraf, there would mean no interruption to anyone else.

Configuring Telegraf can be done in runtime — with no need to recompile — and completely under the developer’s control, freeing ops of additional overhead.

[[processors.tag_limit]]

  limit = 3

  ## List of tags to preferentially preserve

  keep = ["handler", "method", "status"]

Telegraf has 200+ open-source plugins for full-stack monitoring (infrastructure, network, & application), supporting pull, push and metrics streaming, as well as client libraries (C#, Go, Java, JavaScript/Node.js, Python…) for direct instrumentation. Telegraf also monitors kubelet API for metrics exposed in /summary endpoint and Kubernetes Inventory monitors system resources state as for:

daemonsets
deployments
nodes
persistentvolumes
persistentvolumeclaims
pods (containers)
Statefulsets

Key lessons learned

Scaling monitoring is not about adding more manual processes and controls. Scaling cannot be coupled with higher complexity, and surely, must embrace empowering developers — with observability, predictability and prescriptive means - to ensure that monitoring is doing its job.

Of course, Kubernetes is constantly evolving, and so are we. If you have an idea, we’d love to hear about it!

Moving Towards AIOps with Real-Time Full Stack Visibility, Intelligent Alerting and Response Automation

Daniella Pontes (InfluxData) — Tue, 19 Nov 2019 10:46:32 -0700

InfluxData integration with PagerDuty takes you down the road of excellence managing your Kubernetes environments.

There is no one single playbook for all types of incidents — that is granted. Therefore, it is necessary to have the flexibility and adaptive learning about the environment and appropriate human actions to address issues promptly and effectively. Kubernetes’ ephemeral pods and self-healing nature makes even more imminent the need for intelligent alerting because Kubernetes is naturally in constant change, therefore, an isolated failure or error is not enough to make a decision about triggering an alarm. This does not mean to over-complicate things with a myriad of static alerting rules trying to adjust to patterns. And it’s important not to depart from “simple” when simple can do the job, as in well-understood risk thresholds in resource constraint situations. Rather, it means embracing the power of analysis of historic data to be adaptive, handle multidimensional risk scenarios and event alerting when relying solely on static isolated thresholds would be insufficient. Such reliance would leave you with two out-of-measure options: an excess of alerts or a dangerous frugality, which respectively leads to either alert fatigue or ineffective trigger rules.

In modern IT operations, incidents can evolve quickly to a devastating cascading effect with large business impact, both to your company’s finances and brand. Therefore, response and repair time to normality cannot be delayed. Every second counts — in thousands of dollars. Actions should be automated for mature workflows, be planned with a preemptive, prescriptive and predictive approach. That is where the integration of the InfluxData real-time, full-stack monitoring and data analytics platform and the PagerDuty incident response platform becomes powerful. This integration provides real-time visibility of technical and business insights enabling stakeholders to act in time.

Kubernetes can benefit from high-fidelity monitoring data applied to alerting. That is because, although Kubernetes orchestration will continually work towards the desired declared state, problems could be developing in the dark. Therefore, it is necessary to monitor and preemptively alert on problematic workloads or environmental issues such as network load, storage, long response time of services, and so on.

Real-time monitoring, intelligent alerting and orchestration of the appropriate response for each event is the road that leads to excellence in operations and user experience. Having the right solutions in place is important, and having them integrated lets you take full advantage of data-driven workflows, on the operations as well as business levels. Both operations goals related to infrastructure and application environment — and business goals addressing user experience, critical transactions and request failures — should be used to compose effective trigger rules and avoid false positives. Leveraging a time series platform that provides full-stack historic data to dynamic define performance baselines is fundamental to reduce alarm noise and fatigue.

The integration of InfluxData with PagerDuty enables alert thresholds and triggers to be set dynamically using data analytics and correlation of data from multiple monitoring measurements. Indeed, advanced data analytics (such as Holt-Winters forecasting) is natively supported by the InfluxData platform. This enables detection of trends and seasonality, and moreover, shows the impact on high-level business KPIs, to be used to define static and dynamic alerts. It also provides visibility into the metrics, logs, performance indicators and metadata leading the way to a quick diagnosis and faster mean time to resolution (MTTR).

Simple static thresholds (for infrastructure and application metrics) must be complemented with dynamic thresholds (suitable for fluctuations found in ephemeral and highly adaptive environments like Kubernetes, but also in business seasonality and trends). Insights derived from statistical data analytics or machine learning frameworks can provide smarter alerting triggers. Such triggers - when integrated with a modern event handling solution - can direct calls-to-action as well as perform automatic escalation and optimization of digital operations. That means not only metrics but also high-fidelity data must feed data analytics engines for intelligent alerting and must be kept “fluid” to serve multiple frameworks all the way to an end-to-end response automation as an ultimate goal. InfluxData’s integration with PagerDuty empowers organizations to move towards AIOps by applying data-driven action flows from alert triggering through the entire incident lifecycle management.

Learn more about InfluxData’s integration with PagerDuty and about monitoring Kubernetes.

About InfluxData InfluxData is the creator of InfluxDB, the open source time series database. The technology is purpose-built to handle the massive volumes of time-stamped data produced by IoT devices, applications, networks, containers and computers. The company has more than 600 customers and is on a mission to help developers and organizations, such as Cisco, IBM, PayPal, and Tesla, store and analyze real-time data, empowering them to build transformative monitoring, analytics, and IoT applications quicker and to scale. InfluxData is headquartered in San Francisco with a workforce distributed throughout the U.S. and across Europe. For more information, visit www.influxdata.com and follow us @InfluxDB.

About PagerDuty In an always-on world, teams trust PagerDuty to help them deliver a perfect digital experience to their customers, every time. PagerDuty is the central nervous system for a company’s digital operations. PagerDuty identifies issues and opportunities in real time and brings together the right people to respond to problems faster and prevent them in the future. From digital disruptors to Fortune 500 companies, over 12,000 businesses rely on PagerDuty to help them continually improve their digital operations - so their teams can spend less time reacting to incidents and more time building for the future.

IT Monitoring in the Era of Containers: Tapping into eBPF Observability

Daniella Pontes (InfluxData) — Mon, 15 Jul 2019 08:00:43 -0700

Article by Daniella Pontes and Luca Deri

Containers are a game-changer for everyone. An abstraction between the infrastructure and the application layers, containers are a group play and concern IT system engineers, Ops, NetOps and DevOps. However, professionals in different roles approach container monitoring from different perspectives, all valid and important, and necessary to build a complete monitoring strategy.

With the separation of applications into containerized microservices running in clusters, each container packages the necessary resources to execute its part of the deal. However, unless its workload, counterparts, and also the network putting the pieces together are equally performant, the sum of the parts will fall short from delivering an application environment that meets performance and service levels objectives.

New monitoring metrics for containers and networks

Container monitoring — from an infrastructure perspective — has come a long way with systems like Kubernetes. The emphasis has shifted from the health of individual containers to cluster health. This is because Kubernetes simplified orchestration by providing a logical layer to commoditize the infrastructure layer on which microservices run, while automating deployment and optimizing resource assignment. Infrastructure cost lowered, agility up — a great value proposition! However, this new approach has its benefits and drawbacks.

On the one hand, a misbehaving or underperforming microservice, running on a compliant cluster to the desired optimum declared state, can still create a dismantling effect on the network and overall application performance. On the other hand, observing how processes are handling the workload and which services are generating them from within the containers can provide very positive outcomes with regard to application performance improvements. Easier said than done due to the disposable nature of containers and the short window of time to make these observations.

Troubled or struggling microservices can move around from container to container, from host to host, from interface to interface. Finding the epicenter of an underperforming application in a containerized microservice environment by only looking from the outside would require tracking down a moving, intermittent target: the worst nightmare for the diagnosis teams.

IT managers must keep their monitoring sharpened for metrics from the orchestration system, containers and nodes; and NetOps must keep network latency, service availability, responsiveness and bandwidth consumption under control. Unless the health of the microservices running on this containerized infrastructure and their vast, meshed inter-container network is also kept close to heart, containerized application monitoring will suffer from shortsightedness.

It is clear that monitoring only from the outside will not fit the bill, especially for those full-stack teams. To these teams, every layer and moving part needs to work harmoniously. With that in mind, performance monitoring of microservices and how it impacts inter-container flows, more than the availability of resources (accomplished by killing and spinning new pods and containers) will show where trouble is brewing. For that, it is necessary to have visibility into what is going on from the inside.

Measuring resource and service metrics

Baseline resource monitoring metrics — such as CPU, memory and disk space — are well-covered in Kubernetes on three main fronts: containers, nodes and the master node. Additional application- specific metrics and event monitoring can be done deploying sidecars. However, symptoms on resource usage, saturation and failures can be better understood when correlated with information collected from a microservice performance perspective, which includes:

Latency: time to service a request
Traffic generation: communication with other services
Error: how often errors occur

Container activity impacts infrastructure, network utilization and application performance; therefore, it should not be overlooked. Nothing should be taken for granted because speed and complexity don’t leave much room for guessing. A thoughtful monitoring plan for containerized application environments must include ways to verify what is going on inside and between containers — identifying users, processes, pods and containers presenting faulting, abusing or suspicious behavior.

Approaches such as in-and-out packet inspection and grouping packets into flows based on IP, port and protocol provide a viable way to monitor bandwidth utilization as well as detect non-legit, malicious and malformed traffic. However, that was before container proliferation. For containerized applications, the packet paradigm is no longer enough to provide the necessary visibility, as they do not carry context (i.e. application, user, process, pod or container) and thus monitoring them will not provide the pursued understanding of accountability. Sometimes services interact inside a system and not over a network, where it is possible to capture packets. To make matters worse, the rate at which networks and applications run today demands that network packet analyzers crank up the packet processing speed, putting a tremendous load on the CPU. One could also say that packets are used simply because networks are based on them, but applications do not see packets. They operate in terms of data sent/received, action latency and code response. So, instead of using packet-level data to infer application information, it would make more sense to read system metrics directly to extract the desired information without the need for “translations.” And as a nice bonus, taking the system introspection approach incurs lighter computational load on the system.

One way to get system information about the workload running on the container is to use eBPF to collect kernel events. This technology was introduced as Berkeley Packet Filtering (BPF), a Linux kernel technology for in-kernel packet filtering by Steven McCanne and Van Jacobson at Lawrence Berkeley National Laboratory. eBPF was extended to process multiple types of events, executing actions other than packet filtering. When using eBPF, you can correlate kernel events with network flow data to identify which containers are participating in communication sessions objectively. This will point to which users, processes and containers are presenting abnormal behaviors.

InfluxData and its partner ntop are taking the next step in monitoring containerized application environments with the use of the extended Berkeley Packet Filter (eBPF). This work will shed light on activities within the container to guide IT to find out where things are broken or breaking, and who is causing the performance issue.

Correlate network metrics with other with eBPF event data

System introspection via eBPF plays the role of adding context to infrastructure and network monitoring. This allows for root-cause to be quickly identified, and can act as the only viable source of interaction information in container deployment scenarios, for two main reasons:

With current network speeds, any meaningful sampling rate will generate an insurmountable volume of packets to be processed for inspection, consuming precious CPU cycles. If deployed in a hosted cloud environment, those cycles will slowly and surely eat up the budget.

Traffic between microservices could never really reach monitored interfaces. And therefore, there will be no opportunity to capture them.

Adding eBPF monitoring to packet inspection binds system events to network traffic, and by doing so, provides the contextual multi-dimensionality necessary to reduce the entropy of monitored data and target alerts to actionable information. Narrowing down activities to who and what to account for the process being monitored is something neither of the approaches would be able to do in isolation — not to mention doing so efficiently and effectively.

In this new accelerated and complex world of ephemeral container infrastructure and fragmented applications, organizations must become data-driven in order to cope with ever-increasing performance expectations. Monitoring has to go beyond availability, consumption and performance of isolated resources. It must provide binocular and panoramic views in order to provide a good understanding of what is going on and what could be brewing undetected. Monitoring must shed light on trends while providing real-time insights — and ultimately seek anticipation and automation.

In order to pursue such goals, IT should look into inter-container traffic and intra-container events and correlate this data with other metrics and information gathered, thereby identifying where the trouble starts: container, process, and user.

One place for metrics, events and network traffic

eBPF opens one more channel to gain observability in the inner systems required to link anomalous behavior and performance variations in distributed containerized environments. But in order to connect the dots, all the information from metrics, events and intra/inter container events need to be in one platform capable of fully utilizing this data and cross-analyzing it in conjunction.

All data should come to the same place, and reduce the burden from setup, ramp-up, management and gathering of information pieces from multiple siloed sources. The ntop eBPF solution uses InfluxDB as its time series storage engine and is therefore ready to include all types of monitoring data like metrics, kernel events, log, tracing and business KPIs. This converged set of metrics and events is useful when used for alerting and prediction modeling. Container era complexity demands one integrated data source, one engine for multiple data type analysis, and one UI for all visualizations. Bringing it all together will compound insights and perspectives leading to a monitoring solution that enables more intelligent alerts and actionable information.