In the world of DevOps, one size does not fit all when it comes to monitoring unique application and infrastructure deployments. (Not to mention ever evolving data structures.) Raw data is “dumb” when randomly distributed. To make this data work for you, you’ll need to implement smart analytics in order to unlock the hidden insights already present in the data.
Emerging trends like microservices, APIs, containerization, elastic storage, Software Defined Networking, Hybrid clouds, etc, all keep pushing the boundaries of what coverage and depth traditional monitoring solutions can provide out of box.
The unique tooling and data needs of an organization’s DevOps team is often left unmet because the already deployed monitoring systems rarely keep pace, or can even support the new applications and infrastructure constantly being rolled out. If you can’t monitor it, how can you support it? That’s why it often makes sense to create custom monitoring solutions using extensible open source technologies.
DevOps is not just a technology shift, it is also a process and cultural shift that has been happening for years and will continue to do so for the foreseeable future.
Traditional waterfall development methodologies have been replaced by agile development processes with incremental, but shorter Code-Test-Build-Deploy cycles. Builds are pushed much more frequently, sometimes even 5-10 times a day, sometimes for patches, sometimes to support fragmented deployment targets like mobile devices, sensors or servers.
Most off-the-shelf monitoring products were built to give deep visibility into one layer of the stack and hence we have silos of monitoring solutions like APM (Application Performance), IM (Infrastructure), NFM (Network Flows), RUM (Real User), Crash Reporting (Mobile), and Log Analytics to name a few. The challenges with these tools are many.
Correlating events and metrics from each silo is non-trivial and often we see “Frankenstein” monitoring systems built. These systems are traditionally riddled with problems of all large deployments:
Metrics redundancy and alert duplicacy also render these “Frankenstein” monitors ineffective in real triaging scenarios. Correlation between siloed metrics are faulty because of incompatible data formats or the variability of persistence frequencies. Further, each silo may store its metrics in different data stores like an RDBMS, Cloud storage or flat disk files, making cross-silo aggregation impractical or at least an ETL-nightmare.
Other major problems frequently encountered are:
The reality is that we are stuck with 20-30 year old monitoring methodologies and solutions (whether they are on-premise or SaaS) which were not built for today’s dynamic DevOps reality.
Unfortunately, the list is long:
The InfluxData platform is uniquely suited for building custom DevOps monitoring solutions.
Monitoring means having to collect data from disparate systems, applications, datasources, services and infrastructure components. InfluxData’s Telegraf collector supports 30+ inputs and 10+ outputs and can be easily extended to support your sources of data. Telegraf makes collecting data in a format InfluxDB can consume, simple. Here’s why:
However, the InfluxData platform is extensible by design so you can easily integrate other collection agents like collectd, in conjunction with Telegraf.
Learn more about Telegraf
The most popular data type in any monitoring system is going to be in a time-series format. InfluxDB is designed from the ground up to handle just time-series data and to do it better than any other database. InfluxDB is the “I” in the TICK stack. More specifically, InfluxDB is an open source database written in Go to handle time-series data with high availability and high performance requirements. InfluxDB installs in minutes without external dependencies, yet is flexible and scalable enough for complex deployments. Here’s why InfluxDB is the best choice for storing a custom monitoring solution’s time-series data:
Learn more about InfluxDB
If you don’t already have a dashboarding or graphing UI in place, InfluxData provides Chronograf. It’s the “C” in the TICK stack. Chronograf is a downloadable binary you install behind your firewall to collaboratively, yet securely, perform ad-hoc visualizations on your time-series data. Features include:
Another visualization UI choice that offers tight integration with InfluxDB is the open source Grafana project. Either choice makes connecting to and visualizing time-series data, simple.
Inevitably, you are going to want to either alert on or in some way process the time-series data in your monitoring system. You’ll want to do this either before it gets written to InfluxDB or when it is retrieved. To address this need, the InfluxData platform ships with the open source Kapacitor project. Kapacitor is the “K” in the the TICK stack. It’s an alerting and data processing engine specifically designed for time-series data. It lets you define your own custom pipeline to aggregate, select, transform or otherwise process data and then store it back in InfluxDB or trigger an event. Features include:
Learn more about Kapacitor
InfluxData products are used for custom DevOps monitoring by startups and large enterprises alike. Visit our Testimonials page for a comprehensive list.