InfluxData Blog - Ryan Betts

InfluxDB Internals 101 - Part Two

Ryan Betts (InfluxData) — Mon, 27 Nov 2017 14:17:08 -0700

Query path: reading data from InfluxDB
- Indexing points for query
- A note on TSI (on disk indexes)
- Executing queries
- A note on IFQL
- DELETE and DROP - removing data from InfluxDB
- Updating points

Introduction

Part One of this series describes the InfluxDB write path: how the database persists and organizes data being written to the database. This part (Part Two) describes the other main interaction with the database: querying data once it has been persisted. Note that Part One also defines the InfluxDB jargon used in this post (tagset, fieldset, measurement, series) which will be helpful to new readers.

InfluxDB is queried using a SQL dialect called influxql. There is quite a bit of documentation for the language as well as a guide to using influxql for different querying tasks. This post focuses on how the query engine works and not on the semantics of the language itself.

Time series applications tend to query in two patterns. Queries either window and produce per-window aggregates (window data into one-minute intervals and calculate the average for each minute). Or, queries search for a specific point (often the last() or most recent point in a series). Both query patterns filter the points in the database by criteria applied to a set of dimensions; for example, all the data where region = us-east or where measurement = 'cpu'. In InfluxDB, these dimensions are stored as tagsets.

Finally, before we get into more detail, it is important to note that influxql supports selection and projection operators but does not support traditional relational joins. Optimizing query performance in InfluxDB requires finding the initial point for each series and then leveraging columnar storage to efficiently scan a sequence of points following that initial point. The use of flexible schema-on-write tagsets vs. pre-defined dimension tables in a star-schema is one of the more interesting differences between InfluxDB and a traditional SQL columnar OLAP database.

Indexing Points for Query

Part One describes the different data structures populated by incoming writes to achieve durability and compact long-term storage. There is one additional data structure populated by writes to make queries efficient: the index. InfluxDB automatically maintains an index to make filtering by tagsets efficient.

The index maintains mappings of measurement name to field keys, of measurement name to series ids (an internal series identifier), of measurement name to tag keys to tag value to series id, and of series id to shards. The index (as of version 1.4) also maintains sketches of series and measurements for fast cardinality estimates. You can read the index implementation on GitHub for more detail.

That’s a lot of different mappings to think about and understand. Personally, I find it easier, and conceptually accurate, to think of the index as a posting list (aka inverted index) that maps tag key/value pairs to a list of series keys. This slight abstraction captures the primary purpose of the index: to make it efficient at query time to identify all series that need to be scanned based on a tagset filter in an influxql WHERE predicate.

A Note on TSI (On-disk Index)

The current default index is stored in-memory. This allows fast lookup for query planning. However, it also means that high-cardinality data, data that include a large number of unique tagsets, requires a lot of memory to index. This is why we suggest that users use tagsets for lower-cardinality dimension data and use unindexed field values for high-cardinality data.

We are developing a new index structure, Time Series Index (TSI), which is now shipping as an opt-in preview. TSI stores the index on SSD, allowing much higher cardinality datasets than the default in-memory index.

Parsing and Planning

Having described the index, it is possible to explain the internal workflow that runs to parse, plan, and execute an example influxql query. The query engine:

Determines the type of query (one with an expression or a raw data query)
Determines and then separates the time range and the condition expression for filtering data
Determines which shards it needs to access using the list of measurements and the time frame
Expands any wildcards
Validates that the query is semantically correct
Directs the storage engine to create the iterators for each shard
And merges the shard iterator outputs, performing any post-processing on the data

Sample query: select user, system from cpu where time > now() - 1h and host = 'serverA'

The database receives the query and parses out the measurements that are accessed, fields returned, grouping time intervals, filter predicates, and other influxql query components. You can read the AST structure for the SELECT statement in the influxdata/influxql GitHub repository.

After parsing, the query engine determines which series are needed to produce an answer. In this example, the query engine uses the index to find all series that are part of the cpu measurement. It then uses the index to find all series that have the tag key, tag value pair host, serverA. The intersection of these sets provide the series that need to be scanned. The time range in the query, now() - 1h, limits the scan to shard groups covering the last one hour.

The query engine instantiates an iterator for each series, for each shard. These iterators are nested, forming a tree. The iterator tree is executed bottom-up, reading, filtering, and merging data to produce a final result set.

The version 1.4 EXPLAIN and EXPLAIN ANALYZE statements provide statistics on iterators created and TSM blocks decoded as part of query execution. There are example outputs in the What’s New in InfluxDB 1.4 blog post.

A Note on IFQL

The combination of schema-on-write, automatic indexing of tagsets, and SQL-like syntax produce a system that allows newcomers to be productive quickly, that feels familiar, and requires minimal setup to get started.

However, the pre-allocation of narrowly scoped iterators means high-cardinality queries, and queries that produce a very large number of groups are expensive to plan. The iterator structures can consume, worst case, GBs of RAM. Secondly, the iterator allocation during planning and other implementation details make multi-query resource management difficult. Finally, while SQL-like syntax is a good fit for simple queries, it becomes cumbersome for more sophisticated analytics. Time series queries are often sets of functions applied to groupings of filtered streams. Expressing these queries using select-project-join logic with advanced SQL partition and over clauses requires an experienced SQL programmer and is no longer beginner- friendly.

We recently announced a prototype query language, IFQL, to explore solutions to these problems: cheaper planning, better resource management, and easier expression of complex queries.

DELETE and DROP: Removing Data from InfluxDB

InfluxDB supports retention policies to enforce time to live policies against data. This is always the preferred way to regularly delete points from the database. However, applications sometimes write bad data to the database. That data needs to be removed to return to normal operation. In these cases, DELETE and DROP can be used to delete unwanted points.

DELETE and DROP statements are processed through the query layer, not the write layer. This allows DELETE and DROP to re-use the selection and expression features of influxql.

Deleting data from a columnar database is expensive. InfluxDB organizes data on disk into immutable runs of values for a single column of a series. A delete operation needs to undo a lot of that work for a subset of points.

In InfluxDB, deleting a row from the database produces a tombstone. A tombstone includes a series key and the min and max time of the deleted range. This allows a very compact expression for the primary delete use case: delete all data for an invalid series between times t1 and t2.

When sufficient tombstones collect, TSM data is re-compacted into a new immutable file with the deleted data removed and tombstone records deleted. At query time, tombstones are checked to avoid processing data marked as deleted.

Over the last six months, substantial work has gone into making tombstone management, compaction based on accumulated deletes, and index updates after deletes, correct and efficient.

Updating Points

InfluxDB does not support an UPDATE statement. However, re-inserting a fully qualified series key at an existing timestamp will replace the old point’s field value with the new field value.

Conclusion

Hopefully this post has added to your mental model of InfluxDB. It discusses four key concepts:

series and tagsets are indexed for query planning.
Query planning uses the index to identify series to scan.
Query planning generates and executes a tree of iterators.
DELETE and DROP statements are part of influxql and result in tombstones to annotate deleted data.

InfluxDB 1.4 Now Available: InfluxQL Enhancements, Prometheus Read/Write, Better Compaction and a Lot More!

Ryan Betts (InfluxData) — Tue, 14 Nov 2017 04:00:56 -0700

What’s New in InfluxDB 1.4

We are announcing the new version of InfluxDB 1.4 in Open Source. This release, unlike our previous releases, is not paired with a corresponding InfluxDB Enterprise release. All of the features and changes described here are available in open source InfluxDB. This blog post is assembled largely from the pull requests and feature descriptions written by the InfluxDB platform team and community members. Thank you!

InfluxQL Enhancements

InfluxDB 1.4 includes new InfluxQL capabilities to make it easier to explore metadata and understand query execution. We’ve added SHOW CARDINALITY queries to make it much easier to query for series cardinality.

The SHOW CARDINALITY commands come in two flavors: estimated and exact. The estimated values are calculated using sketches and are a safe default for all cardinality sizes. The EXACT variations count directly from TSM data and are expensive to run for high cardinality data. We suggest preferring the estimates. We also started adding predicate support (WHERE clause support) to meta-queries. However, filtering by time is only currently supported with TSI. We will continue to improve these capabilities as we develop 1.5.

SHOW MEASUREMENT CARDINALITY — estimates the cardinality of the measurement set for the current database.

SHOW MEASUREMENT CARDINALITY ON db0 — estimates the cardinality of the measurement set on the provided database.

Note: SHOW MEASUREMENT CARDINALITY also supports GROUP BY tag and WHERE tag. However when using these options on the query, then the query falls back to an exact count. 

SHOW SERIES CARDINALITY — estimates the cardinality of the measurement set for the current database.

SHOW SERIES CARDINALITY ON db0 — estimates the cardinality of the measurement set on the provided database.

Note: SHOW SERIES CARDINALITY also supports FROM measurement, GROUP BY tag WHERE tag etc. However when using these options on the query, then the query falls back to an exact count. 

SHOW MEASUREMENT EXACT CARDINALITY —  counts exactly the number of measurements on the current database. 

SHOW SERIES EXACT CARDINALITY —  counts exactly the number of series exactly on the current database.

SHOW TAG KEY CARDINALITY —  estimates the number of tag keys on the current database. Note: this is currently implemented as an exact count.

SHOW TAG VALUES CARDINALITY WITH KEY = "X" —  estimates the number of tag values for the provide tag key, on the current database. Note: this is currently implemented as an exact count.

SHOW TAG KEY EXACT CARDINALITY —  counts exactly the number of tag keys on the current database.  

SHOW TAG VALUES EXACT CARDINALITY WITH KEY = "X" —  counts exactly the number of tag values for the provide tag key, on the current database.

We’ve also added support for EXPLAIN and EXPLAIN ANALYZE to help understand query costs. EXPLAIN parses and plans the query and then prints a summary of estimated costs. Many SQL engines use EXPLAIN to show join order, join algorithms, and predicate and expression pushdown. InfluxQL doesn’t support joins. Instead, the cost of a query in InfluxQL is typically a function of total series accessed, number of iterator accesses to a TSM file, and number of TSM blocks that need to be scanned. Consequently, these are the elements of InfluxQL EXPLAIN:

> explain select sum(pointReq) from "_internal"."monitor"."write" group by hostname;
QUERY PLAN
------
EXPRESSION: sum(pointReq::integer)
NUMBER OF SHARDS: 2
NUMBER OF SERIES: 2
CACHED VALUES: 110
NUMBER OF FILES: 1
NUMBER OF BLOCKS: 1
SIZE OF BLOCKS: 931

Running EXPLAIN ANALYZE executes the query and counts the actual costs during runtime.

> explain analyze select sum(pointReq) from "_internal"."monitor"."write" group by hostname;
EXPLAIN ANALYZE
-----------
.
└── select
├── execution_time: 242.167µs
├── planning_time: 2.165637ms
├── total_time: 2.407804ms
└── field_iterators
├── labels
│   └── statement: SELECT sum(pointReq::integer) FROM "_internal"."monitor"."write" GROUP BY hostname
└── expression
├── labels
│   └── expr: sum(pointReq::integer)
├── create_iterator
│   ├── labels
│   │   ├── measurement: write
│   │   └── shard_id: 57
│   ├── cursors_ref: 1
│   ├── cursors_aux: 0
│   ├── cursors_cond: 0
│   ├── float_blocks_decoded: 0
│   ├── float_blocks_size_bytes: 0
│   ├── integer_blocks_decoded: 1
│   ├── integer_blocks_size_bytes: 931
│   ├── unsigned_blocks_decoded: 0
│   ├── unsigned_blocks_size_bytes: 0
│   ├── string_blocks_decoded: 0
│   ├── string_blocks_size_bytes: 0
│   ├── boolean_blocks_decoded: 0
│   ├── boolean_blocks_size_bytes: 0
│   └── planning_time: 1.401099ms
└── create_iterator
├── labels
│   ├── measurement: write
│   └── shard_id: 58
├── cursors_ref: 1
├── cursors_aux: 0
├── cursors_cond: 0
├── float_blocks_decoded: 0
├── float_blocks_size_bytes: 0
├── integer_blocks_decoded: 0
├── integer_blocks_size_bytes: 0
├── unsigned_blocks_decoded: 0
├── unsigned_blocks_size_bytes: 0
├── string_blocks_decoded: 0
├── string_blocks_size_bytes: 0
├── boolean_blocks_decoded: 0
├── boolean_blocks_size_bytes: 0
└── planning_time: 76.192µs

For the moment, these statistics are provided to help users and our support team understand the “cost” of the queries being executed. It helps to explain what the query engine is actually doing and hopefully provides greater insight into the data set being accessed. Unfortunately, there isn’t much more you can do to act on these results other than confirming that the number of series being accessed make sense based on expected results. But, we believe the insight to be useful by itself.

Support for Prometheus Read and Write Endpoints

As announced earlier, we added Prometheus read and write endpoints. These have been available on master for a while and are shipping with InfluxDB 1.4.

Compaction Performance Improvements

TSM compactions have been improved in a few areas: performance, scheduling, and observability. Performance has improved to better handle higher cardinalities within a shard. These changes include using off-heap memory for TSM indexes, disk-based index buffering when creating TSM files, and reductions in allocations. These changes should reduce GC pressure which lowers CPU and memory utilization in almost all cases. It also prevents OOMs due to very high cardinality compactions.

Compaction scheduling has been improved to better coordinate resources across shards and adapt to changing workloads. Previously, each shard scheduled and limited compactions for the shard independently. The scheduler now uses a weighted queue approach instead of fixed limits on scheduling and works across shards better. This allows higher priority work to take advantage of available cores more dynamically. The max-concurrent-compactions limit that was added in 1.3 is now enabled by default to limit compactions to 50% of available cores. This better controls memory and CPU utilization when many shards are active.

Monitoring of compactions now has a metric for the depth of the queue for each level. Each shard now exposes a gauge style metrics such as tsmLevel3CompactionQueue that indicates how long the queue is for that level and shard. The sum of all levels in a shard indicates if compactions are backing up and you may need more CPU cores. The combination of the *Active, *Err and *Queue metrics provide basic utilization, saturation and error (USE) metrics. The existing *Duration metrics can be used to monitor compaction latencies if you use the four gold signals approach to monitoring.

Client and HTTP Enhancements

There are a number of features that make using the HTTP interface easier.

HTTP responses to the /query endpoint no longer force Connection: close. This allows re-use of HTTP connections by clients. The issue #8525 includes useful discussion of the change.

InfluxDB HTTP responses now include the InfluxDB version in the header X-Influxdb-Build for applications that need to distinguish database versions. Internally, Chronograf will use this to more easily manage combinations of open source and enterprise InfluxDB instances.

Errors from queries and writes are now available via the X-InfluxDB-Error header and 5xx error messages are written to server logs when log-enabled = true is enabled in the [httpd] configuration section.

InfluxDB now honors X-Request-Id header so that callers can pass a correlation id as part of the request. HTTP responses populate both X-Request-Id and Request-Id to maintain backwards compatibility with previous version, and to support the more common X-Request-Id header name. More details are recorded in the pull request.

Finally, thanks to @emluque, the InfluxDB CLI now supports Ctrl+C to cancel a running query.

Message Pack formats for responses

Message pack can now be used for responses by setting application/x-msgpack in the Accept header. The server will respond with message pack serialized responses.

Experimental and Preview Features

TSI Progress

A lot of work has gone into TSI over the course of developing 1.4. However, we are not yet ready to release TSI as the default production index. We’ll be writing more about our TSI progress as we work on 1.5.

This release does include a utility to generate TSI indexes from TSM data. This is allows TSI indexes to be rebuilt even when they are larger than in-memory support would allow; it also allows building TSI indexes for older shards for experimentation.

Further description and a usage example is available in the pull request: #8669.

Preview of uint64 Support

We have added unsigned 64 bit integer (aka uint64) support that can be enabled with an InfluxDB build flag. We are leaving this behind a build flag until we implement uint64 support through the rest of the TICK stack. Telgraf, Chronograf, and Kapacitor do not yet support this field type and there are some client libraries where uint64 values cannot be naturally expressed.

To enable uint64, build InfluxDB with go install -tags uint64 ./.... Write uint64 values by suffixing an integer with u in the write protocol.

create database u64ex
use u64ex
Using database u64ex
insert cpu v1=18446744073709551615u
select v1 from cpu
name: cpu
time                v1
----                --
1510620507267476000 18446744073709551615

IFQL Prototype Interfaces

InfluxDB OSS 1.4 includes the prototype RPC interface to support IFQL. This API will change as we advance the IFQL prototype and we are not establishing any compatibility promises for this new interface. However, you can enable and access storage if you want to explore the interface as an access point to the database. An ifql section is now available on the configuration file.

[ifql]
# Determines whether the RPC service is enabled.
enabled = true
# Determines whether additional logging is enabled.
log-enabled = true
# The bind address used by the ifql RPC service.
bind-address = ":8082"

The IFQL RPC interface is protobuf based; the protobuf file is available for your coding pleasure.

Other Changelog Tidbits

#8426: Add parse-multivalue-plugin to allow users to choose how multivalue plugins should be handled by the collectd service.
#8548: Allow panic recovery to be disabled when investigating server issues.
#8592: Mutex profiles are now available.
#8854: Report the task status for a query.
#8830: Separate importer log statements to stdout and stderr.
#8690: Implicitly decide on a lower limit for fill queries when none is present.

InfluxDB Internals 101 - Part One

Ryan Betts (InfluxData) — Fri, 27 Oct 2017 04:00:12 -0700

Paul Dix led a series of internal InfluxDB 101 sessions to teach newcomers InfluxDB internals. I learned a lot from the talks and want to share the content with the community. I’m also writing this to organize my own understanding of InfluxDB and to perhaps help others who want to learn how InfluxDB is architected. A lot of this information is gathered from InfluxDB documentation as well — the goal with this series is to present a consolidated overview of the InfluxDB architecture.

There’s a lot to digest so it’s presented in three parts. This first post explains the data model and the write path. Post two explains the query path. Post three explains InfluxDB Enterprise clustering.

Series Table of Contents

Data model and write path: adding data to InfluxDB
- Data model terminology
- Receiving points from clients
- Persisting points to storage
- Compacting persisted points
Query path: reading data from InfluxDB
- Indexing points for query
- A note on TSI (on disk indexes)
- Parsing and planning
- Executing queries
- A note on IFQL
- DELETE and DROP - removing data from InfluxDB
- Updating points
Clustering: InfluxDB Enterprise
- Understanding the meta-service
- Understanding data-nodes
- Understanding data distribution and replication

Data model and write path: adding data to InfluxDB

Data model and terminology

An InfluxDB database stores points. A point has four components: a measurement, a tagset, a fieldset, and a timestamp.

The measurement provides a way to associate related points that might have different tagsets or fieldsets. The tagset is a dictionary of key-value pairs to store metadata with a point. The fieldset is a set of typed scalar values — the data being recorded by the point.

The serialization format for points is defined by the line protocol (which includes additional examples and explanations if you’d like to read more detail). An example point from the specification helps to explain the terminology:

’’ temperature,machine=unit42,type=assembly internal=32,external=100 1434055562000000035

The measurement is temperature.

The tagset is machine=unit42,type=assembly. The keys, machine and type, in the tagset are called tag keys. The values, unit42 and assembly, in the tagset are called tag values.

The fieldset is internal=32,external=100. The keys, internal and external, in the fieldset are called field keys. The values, 32 and 100, in the fieldset are called field values.

Each point is stored within exactly one database within exactly one retention policy. A database is a container for users, retention policies, and points. A retention policy configures how long InfluxDB keeps points (duration), how many copies of those points are stored in the cluster (replication factor), and the time range covered by shard groups (shard group duration). The retention policy makes it easy for users (and efficient for the database) to drop older data that is no longer needed. This is a common pattern in time series applications.

We’ll explain replication factor, shard groups, andshards later when we describe how the write path works in InfluxDB.

There’s one additional term that we need to get started: series. A series is a group of points that share a measurement + tag set + field key.

You can refer to the documentation glossary for these terms or others that might be used in this blog post series.

Receiving Points from Clients

Clients POST points (in line protocol format) to InfluxDB’s HTTP /write endpoint. Points can be sent individually; however, for efficiency, most applications send points in batches. A typical batch ranges in size from hundreds to thousands of points. The POST specifies a database and an optional retention policy via query parameters. If the retention policy is not specified, the default retention policy is used. All points in the body will be written to that database and retention policy. Points in a POST body can be from an arbitrary number of series; points in a batch do not have to be from the same measurement or tagset.

When the database receives new points, it must (1) make those points durable so that they can be recovered in case of a database or server crash and (2) make the points queryable. This post focuses on the first half, making points durable.

Persisting Points to Storage

To make points durable, each batch is written and fsynced to a write ahead log (WAL). The WAL is an append only file that is only read during a database recovery. For space and disk IO efficiency, each batch in the WAL is compressed using snappy compression before being written to disk.

While the WAL format efficiently makes incoming data durable, it is an exceedingly poor format for reading — making it unsuitable for supporting queries. To allow immediate query ability of new data, incoming points are also written to an in-memory cache. The cache is an in-memory data structure that is optimized for query and insert performance. The cache data structure is a map of series to a time-sorted list of fields.

The WAL makes new points durable. The cache makes new points queryable. If the system crashes or shut down before the cache is written to TSM files, it is rebuilt when the database starts by reading and replaying the batches stored in the WAL.

The combination of WAL and cache works well for incoming data but is insufficient for long-term storage. Since the WAL must be replayed on startup, it is important to constrain it to a reasonable size. The cache is limited to the size of RAM, which is also undesirable for many time series use cases. Consequently, data needs to be organized and written to long-term storage blocks on disk that are size-efficient (so that the database can store a lot of points) and efficient for query.

Time series queries are frequently aggregations over time — scans of points within a bounded time range that are then reduced by a summary function like mean, max, or moving windows. Columnar database storage techniques, where data is organized on disk by column and not by row, fit this query pattern nicely. Additionally, columnar systems compress data exceptionally well, satisfying the need to store data efficiently. There is a lot of literature on column stores. Columnar-oriented Database Systems is one such overview.

Time series applications often evict data from storage after a period of time. Many monitoring applications, for example, will store the last month or two of data online to support monitoring queries. It needs to be efficient to remove data from the database if a configured time-to-live expires. Deleting points from columnar storage is expensive, so InfluxDB additionally organizes its columnar format into time-bounded chunks. When the time-to-live expires, the time-bounded file can simply be deleted from the filesystem rather than requiring a large update to persisted data.

Finally, when InfluxDB is run as a clustered system, it replicates data across multiple servers for availability and durability in case of failures.

The optional time-to-live duration, the granularity of time blocks within the time-to-live period, and the number of replicas are configured using an InfluxDB retention policy:

CREATE RETENTION POLICY <retention_policy_name> ON <database_name> DURATION <duration> REPLICATION <n> [SHARD DURATION <duration>] [DEFAULT]

The duration is the optional time to live (if data should not expire, set duration to INF). SHARD DURATION is the granularity of data within the expiration period. For example, a one- hour shard duration with a 24 hour duration configures the database to store 24 one-hour shards. Each hour, the oldest shard is expired (removed) from the database. Set REPLICATION to configure the replication factor — how many copies of a shard should exist within a cluster.

Concretely, the database creates this physical organization of data on disk:

'' Database director  /db
    '' Retention Policy directory /db/rp
        '' Shard Group (time bounded). (Logical)
            '' Shard directory (db/rp/Id#)
                '' TSM0001.tsm (data file)
                '' TSM0002.tsm (data file)
                '' …

The in-memory cache is flushed to disk in the TSM format. When the flush completes, flushed points are removed from the cache and the corresponding WAL is truncated. (The WAL and cache are also maintained per-shard.) The TSM data files store the columnar-organized points. Once written, a TSM file is immutable. A detailed description of the TSM file layout is available in the [InfluxDB documentation].

Compacting TSM Data

The cache is a relatively small amount of data. The TSM columnar format works best when it can store long runs of values for a series in a single block. A longer run produces both better compression and reduces seeks to scan a field for query. The TSM format is based heavily on log-structured merge-trees. New (level one) TSM files are generated by cache flushes. These files are later combined (compacted) into level two files. Level two files are further combined into level three files. Additional levels of compaction occur as the files become larger and eventually become cold (the time range they cover is no longer hot for writes.) The documentation reference above offers a detailed description of compaction.

There’s a lot of logic and sophistication in the TSM compaction code. However, the high-level goal is quite simple: organize values for a series together into long runs to best optimize compression and scanning queries.

Concluding Part One

In summary, batches of points are POSTed to InfluxDB. Those batches are snappy compressed and written to a WAL for immediate durability. The points are also written to an in-memory cache so that newly written points are immediately queryable. The cache is periodically flushed to TSM files. As TSM files accumulate, they are combined and compacted into higher level TSM files. TSM data is organized into shards. The time range covered by a shard and the replication factor of a shard in a clustered deployment are configured by the retention policy.

Hopefully this post helps to explain how InfluxDB receives and persists incoming writes. In the next post, we’ll discuss how the system supports query, update, and delete operations.

Why I Joined InfluxData - Ryan Betts

Ryan Betts (InfluxData) — Thu, 30 Mar 2017 04:01:00 -0700

Ever take a long break and think about changing careers or industries only to realize that you still absolutely love building high-performance databases? I did, and I’m thrilled to have recently joined InfluxData to continue that path. I took seven months off and in early March joined InfluxData to lead the team that builds InfluxDB - the ‘I’ in the TICK stack.

I spent eight years as a founding developer and then CTO at VoltDB building a high-velocity ACID relational database. I’ve thought a lot about high-velocity data, and I sought out and joined InfluxData very intentionally. InfluxData is building the right tools for the right users in the right way.

Making high-velocity data simple for developers is an unsolved problem that needs to be solved. Real-time metrics, events, and interactions are the heart of large-scale operational workloads and critical to creating value from IoT. Monetizing IoT often requires meaningful real-time action on real-time data. It’s still too hard to get right.

Easy for developers means thinking about the full stack, not just the database. Teams that set off to build high-performance data applications can quickly lose themselves in the weeds of integration, interoperability, and complex distributed systems management. It is still too hard to do simply.

Open source is the foundation for modern infrastructure. InfluxData is working hard to build a sustainable business (so we’ll be here for you in the future) around our open source tools and stacks. We’re committed to our community and determined to build the heart of our systems in the open.

Finally, a talent-driven business must recruit based on skills and experience, not location. InfluxData embraces distributed development teams. Why should your employer dictate your neighborhood? (Recruiting pitch here: just email me!)

I feel a lot of gratitude for the offer to join InfluxData. The people here are amazingly welcome and focused on the company’s mission. There is a lot of work to do, and I look forward to sharing with you what we learn and build over the next few years.