Meet the Founders Who Rewrote in Rust

By Jessica Wachtel / Developer
Jul 06, 2023

The webinar Meet the Founders: An Open Discussion About Rewriting Using Rust includes an in-depth conversation between Mrinal Wadhwa, CTO and Founder of Ockam and Paul Dix, CTO and Founder of InfluxData about their ground-up rewrites from C and Go, respectively, to Rust. Ockam was in development and InfluxDB, InfluxData’s product, was in production but both CTOs came to the same conclusion: their current coding languages weren’t cutting it. So, they both opted for a total rewrite. Big risk, big reward.

Ockam

Ockam is an open-source tool that provides end-to-end encryption for all data in motion. Developers can run Ockam in each application to create a secure communication channel between all their connected applications. Even though Ockam wasn’t in production yet, the decision to do a full rewrite in Rust wasn’t an easy one. Once the idea was on the table, a few more months of decision making passed before the choice was clear. By that time, developers put in roughly eight months of work and wrote tens of thousands of lines of code in C.

Ultimately, the decision came down to a few things. Ockam’s codebase wasn’t simple to use or easy to maintain. Ockam’s core consists of cryptographic messaging protocols. With cryptographic messaging protocols, one tiny misstep renders an entire system insecure. This was unacceptable to Ockam. They wanted users to have a safe, easy-to-use library to create these secure communication channels.

InfluxDB

InfluxDB is a purpose-built database for time series data. Developed in Go, InfluxDB 1.x and 2.x has many strengths. Working with metric data is one of its strongest use cases. But as with many other successful products, InfluxDB continues to evolve. Dix saw the next evolution of InfluxDB widening its best use cases. Tiered storage — offering customers the opportunity to move data from in-memory to a locally attached, cheaper, SSD disk-backed object store — was one of those new features. Allowing for more high cardinality metadata and native SQL support were also high priority goals for the next iteration of InfluxDB. It soon became clear that this evolution was more than a feature add. It was an entire database rewrite.

The rewrite, now known as InfluxDB 3.0, began in 2019 as a research project. Once Rust’s async/await tool for handling asynchronous functions became available in the fall of 2019, Dix backed the language as his choice for InfluxDB 3.0. In the fall of 2020, the rewrite started to take shape. The small research team, composed of three engineers, selected and understood the software tools they’d include in the new database.

These tools, all previously written in Rust, were Apache Arrow, Apache Parquet, and DataFusion. Apache Arrow is an open source project that provides a standardized columnar memory format for flat and hierarchical data. The primary benefit of adapting Arrow is performance improvement. Apache Parquet is the data persistence format. DataFusion is the SQL parser, planner, optimizer, and executer.

Even with the tools defined, it still wasn’t clear whether the rewrite would ever become a viable product. Only 10% of InfluxDB’s engineering effort went to the rewrite until the summer of 2022 when the team expanded. The result of this rewrite, InfluxDB Cloud Serverless and InfluxDB Cloud Dedicated, launched earlier in 2023 with the launch of InfluxDB 3.0 Clustered coming this autumn. The remainder of InfluxDB 3.0 products will launch later in 2023 and early 2024.

Why Rust?

The upside

Rust is a highly performant language. Because of the nature of the applications, both Ockam and InfluxDB require high performance tools. Rust has fearless concurrency which allows developers to write code free of subtle bugs. This also makes applications easy to refactor without introducing new bugs.

Rust’s borrow checker initializes all variables before using them, meaning the same value can’t get used twice accidentally. Rust works with any programming language that’s compatible with C because Rust libraries can export an interface compatible with the C-calling convention. Other characteristics, such as the package manager, errors as a first-class citizen, and developers not having to deal with a garbage collector, are other reasons why both engineering teams chose to rewrite in Rust.

Rust’s async/await tool is another check in the plus column for the language. It’s a much more advanced version than JavaScript’s async/await tool. In JavaScript, when asynchronous functions execute in Node.js, there’s no way to control their specific execution order. In Rust, however, the developer determines the execution order of asynchronous functions using async runtimes. Async runtimes are different runtimes optimized for executing async functions in specific environments. The result is a high level of runtime control.

Challenges

Rust has a lot of positives, but there are downsides. Rust is a very challenging language to learn. Rust’s design patterns are different. Some of these are a direct result of the borrow checker and others stem from the language in general.

Another potential stumbling block is Rust’s lengthy compile time. Because of this, developers must learn to write code that optimizes compile time. But the uphill climb might just be worth it because Dix believes that developers and companies will write more and more high-performance server software in Rust moving forward.

Developers with a background in dynamic programming languages only, such as Python, Ruby, or JavaScript have a harder time learning Rust than developers with a static programming background, e.g., C++ or Swift.

Dix and Wadhwa recommended using tools such as the free Rust Book online, the O’Reilly book, Programming Rust, and leaning into the expertise of others to ease the transition. Ockam created a buddy system that paired developers with Rust expertise with others who had expertise in things like protocol design. InfluxDB used a consultancy service, Integer 32, which led training sessions and answered questions.

Learn more about Rust and InfluxDB 3.0

There’s much excitement about the InfluxDB rewrite. The InfluxDB community is excited about its features and capabilities. The Rust community is also excited because InfluxDB promotes Rust as a community and a language.

Learn more about InfluxDB Cloud Serverless, InfluxDB Cloud Dedicated, or get started with a free cloud account now.