Time series database (TSDB) explained
What is a time series database?
A time series database (TSDB) is a database optimized for time-stamped or time series data. Time series data are simply measurements or events that are tracked, monitored, downsampled, and aggregated over time. This could be server metrics, application performance monitoring, network data, sensor data, events, clicks, trades in a market, and many other types of analytics data.
A time series database is built specifically for handling metrics and events or measurements that are time-stamped. A TSDB is optimized for measuring change over time. Properties that make time series data very different than other data workloads are data lifecycle management, summarization, and large range scans of many records.
Why is a time series database important now?
Time series databases are not new, but the first-generation time series databases were primarily focused on looking at financial data, the volatility of stock trading, and systems built to solve trading. But financial data is hardly the only application of time series data anymore — in fact, it’s only one among numerous applications across various industries. The fundamental conditions of computing have changed dramatically over the last decade. Everything has become compartmentalized. Monolithic mainframes have vanished, replaced by serverless servers, microservers, and containers.
Today, everything that can be a component is a component. In addition, we are witnessing the instrumentation of every available surface in the material world — streets, cars, factories, power grids, ice caps, satellites, clothing, phones, microwaves, milk containers, planets, human bodies. Everything has, or will have, a sensor. So now, everything inside and outside the company is emitting a relentless stream of metrics and events or time series data.
This means that the underlying platforms need to evolve to support these new workloads — more data points, more data sources, more monitoring, more controls. What we’re witnessing, and what the times demand, is a paradigmatic shift in how we approach our data infrastructure and how we approach building, monitoring, controlling, and managing systems. What we need is a performant, scalable, purpose-built time series database.
What distinguishes the time series workload?
Time series databases have key architectural design properties that make them very different from other databases. These include time-stamp data storage and compression, data lifecycle management, data summarization, ability to handle large time series dependent scans of many records, and time series aware queries.
For example: With a time series database, it is common to request a summary of data over a large time period. This requires going over a range of data points to perform some computation like a percentile increase this month of a metric over the same period in the last six months, summarized by month. This kind of workload is very difficult to optimize for with a distributed key value store. TSDB’s are optimized for exactly this use case giving millisecond level query times over months of data. Another example: With time series databases, it’s common to keep high precision data around for a short period of time. This data is aggregated and downsampled into longer term trend data. This means that for every data point that goes into the database, it will have to be deleted after its period of time is up. This kind of data lifecycle management is difficult for application developers to implement on top of regular databases. They must devise schemes for cheaply evicting large sets of data and constantly summarizing that data at scale. With a Time Series Database, this functionality is provided out of the box.
Independent ranking of top 15 time series databases
Time series databases are the fastest growing segment in the database industry. But which time series database is the best and most popular? There are many ways of determining popularity, but an independent website, DB-Engines, ranks databases based on search engine popularity, social media mentions, job postings, and technical discussion volume. (Read their full methodology). Here are the current results:
Time series – the fastest growing database category
DB-Engines also ranks time series database management systems (Time Series DBMS) according to their popularity. Time series databases are the fastest growing segment of the database industry over the past year.
What makes InfluxDB time series database unique?
The whole InfluxData platform is built from an open source core. InfluxData is an active contributor to the Telegraf, InfluxDB, Chronograf and Kapacitor (TICK) projects — the “I,C,K” from the TICK Stack is being collapsed into a single binary in InfluxDB 2.0 — as well as selling InfluxDB Enterprise and InfluxDB Cloud on this open source core. The InfluxDB data model is quite different from other time series solutions like Graphite, RRD, or OpenTSDB. InfluxDB has a line protocol for sending time series data which takes the following form:
measurement-name tag-set field-set timestamp. The measurement name is a string, the tag set is a collection of key/value pairs where all values are strings, and the field set is a collection of key/value pairs where the values can be int64, float64, bool, or string. The measurement name and tag sets are kept in an inverted index which make lookups for specific series very fast. For example, if we have CPU metrics:
cpu,host=serverA,region=uswest idle=23,user=42,system=12 1464623548s
Timestamps in InfluxDB can be second, millisecond, microsecond, or nanosecond precision. The micro and nanosecond scales make InfluxDB a good choice for use cases in finance and scientific computing where other solutions would be excluded. Compression is variable depending on the level of precision the user needs. On disk, the data is organized in a columnar style format where contiguous blocks of time are set for the measurement, tagset, field. So, each field is organized sequentially on disk for blocks of time, which make calculating aggregates on a single field a very fast operation. There is no limit to the number of tags and fields that can be used.
Other time series solutions don’t support multiple fields, which can make their network protocols bloated when transmitting data with shared tag sets. Most other time series solutions only support float64 values, which means the user is unable to encode additional metadata along with the time series. Even OpenTSDB and KairosDB, which support tags (unlike Graphite and RRD), have limitations on the number of tags that can be used. At around 5 to 6 tags, the user will start seeing hot spots within their cluster of HBase or Cassandra machines.
InfluxDB doesn’t have this limitation because the InfluxDB data model is designed for time series specifically. It pushes the developer in the right direction to get good performance out of the database by indexing tags and keeping fields unindexed. It’s flexible in that many data types are supported, and the user can have many fields and tags. Because of all these factors, a purpose-built time series database like InfluxDB is the best solution for working with time series data.
Available as InfluxDB open source, InfluxDB Cloud & InfluxDB Enterprise