InfluxData Customer Success Story: Sorj Labs
Storj is an open source decentralized cloud storage platform where anyone can sell their extra hard drive space for STORJ tokens. Unlike traditional cloud storage providers, Storj keeps data spread across a decentralized network, eliminating the problem of having a single point of failure. It also encrypts all data, making it impossible for anyone, including Storj, to gain access to users’ files without possession of the corresponding private encryption key. Since most of Storj’s metrics are time-stamped and since they wanted to track change over time, Storj Labs chose InfluxDB as the time series component of Storj’s Tardigrade service and workflows.
Storj’s goal is to create the world’s largest and most secure, resilient, performant, and economical cloud storage service, comparable to many of the other cloud storage providers and to do that without owning a data center. With decentralized cloud storage, Storj has a large, distributed network comprised of thousands of independently owned and operated nodes across the globe that store data on Storj’s behalf.
Over time, storage nodes might leave the network and hard drives might fail. The satellite (software that Storj runs) is responsible for counting those storage nodes as they leave the network. If it reaches a certain threshold where file availability could be at risk, the satellite is able to download 29 pieces to itself, recreate the missing pieces, and upload them to healthy nodes, ensuring that the file remains available at all times. For Storj, this is an expensive process because it consumes bandwidth. As a function of files being stored on the network, Storj had to consider file durability, retrievability and repair. To improve their network and estimate repair, they needed to extract insight from data, and specifically, the massive volumes of time-stamped data generated by their architecture.
Storj chose InfluxDB time series platform to enable a file segment decay forecast and gain storage repair visibility. As an open source partner that is part of Storj Labs’ Open Source Partner Program, InfluxDB helps Storj adjust the repair threshold to ensure file availability and monitor growth rate in order to maintain network supply-demand balance. This, in turn, helped Storj Labs achieve tradeoffs between file durability, retrievability, and repair while adhering to cost constraints, as well as providing storage performance comparable to leading cloud providers.
“If you can’t measure it, you can’t improve it. So we have a list of key metrics that directly impacted our roadmap progression. We wanted to make sure that the actual performance of the network, not based on theory but on data, was living up to the expectations we set for ourselves.”
Ben Sirb, Senior Data Scientist, Storj