Apache Doris vs M3

Choosing the right database is a critical choice when building any software application. All databases have different strengths and weaknesses when it comes to performance, so deciding which database has the most benefits and the most minor downsides for your specific use case and data model is an important decision. Below you will find an overview of the key concepts, architecture, features, use cases, and pricing models of Apache Doris and M3 so you can quickly see how they compare against each other.

The primary purpose of this article is to compare how Apache Doris and M3 perform for workloads involving time series data, not for all possible use cases. Time series data typically presents a unique challenge in terms of database performance. This is due to the high volume of data being written and the query patterns to access that data. This article doesn’t intend to make the case for which database is better; it simply provides an overview of each database so you can make an informed decision.

Apache Doris Overview

Apache Doris is an MPP-based interactive SQL data warehousing system designed for reporting and analysis. It is known for its high performance, real-time analytics capabilities, and ease of use. Apache Doris integrates technologies from Google Mesa and Apache Impala. Unlike other SQL-on-Hadoop systems, Doris is designed to be a simple and tightly coupled system that does not rely on external dependencies. It aims to provide a streamlined and efficient solution for data warehousing and analytics.

M3 Overview

M3 is a distributed time series database written entirely in Go. It is designed to collect a high volume of monitoring time series data, distribute storage in a horizontally scalable manner, and efficiently leverage hardware resources. M3 was initially developed by Uber as a scalable remote storage backend for Prometheus and Graphite and later open-sourced for broader use.

Apache Doris for Time Series Data

Apache Doris can be effectively used with time series data for real-time analytics and reporting. With its high performance and sub-second response time, Doris can handle massive amounts of time-stamped data and provide timely query results. It supports both high-concurrent point query scenarios and high-throughput complex analysis scenarios, making it suitable for analyzing time series data with varying levels of complexity.

M3 for Time Series Data

M3 is specifically designed for time-series data. It is a distributed and scalable time-series database optimized for handling large volumes of high-resolution data points, making it an ideal solution for storing, querying, and analyzing time-series data.

M3’s architecture focuses on providing fast and efficient querying capabilities, as well as high ingestion rates, which are essential for working with time-series data. Its horizontal scalability and high availability ensure that it can handle the demands of large-scale deployments and maintain performance as data volumes grow.

Apache Doris Key Concepts

• MPP (Massively Parallel Processing): Apache Doris leverages MPP architecture, which allows it to distribute data processing across multiple nodes, enabling parallel execution and scalability.
• SQL: Apache Doris supports SQL as the query language, providing a familiar and powerful interface for data analysis and reporting.
• Point Query: Point query refers to retrieving a specific data point or a small subset of data from the database.
• Complex Analysis: Apache Doris can handle complex analysis scenarios that involve processing large volumes of data and performing advanced computations and aggregations.

M3 Key Concepts

• Time Series Compression: M3 has the ability to compress time series data, resulting in significant memory and disk savings. It uses two compression algorithms, M3TSZ and protobuf encoding, to achieve efficient data compression.
• Sharding: M3 uses virtual shards that are assigned to physical nodes. Timeseries keys are hashed to a fixed set of virtual shards, making horizontal scaling and node management seamless.
• Consistency Levels: M3 provides variable consistency levels for read and write operations, as well as cluster connection operations. Write consistency levels include One (success of a single node), Majority (success of the majority of nodes), and All (success of all nodes). Read consistency level is One, which corresponds to reading from a single nod

Apache Doris Architecture

Apache Doris is based on MPP architecture, which enables it to distribute data and processing across multiple nodes for parallel execution. It is a standalone system and does not depend on other systems or frameworks. Apache Doris combines the technology of Google Mesa and Apache Impala to provide a simple and tightly coupled system for data warehousing and analytics. It leverages SQL as the query language and supports efficient data processing and query optimization techniques to ensure high performance and scalability.

M3 Architecture

M3 is designed to be horizontally scalable and handle high data throughput. It uses fileset files as the primary unit of long-term storage, storing compressed streams of time series values. These files are flushed to disk after a block time window becomes unreachable. M3 has a commit log, equivalent to the commit log or write-ahead-log in other databases, which ensures data integrity. Client Peer streaming is responsible for fetching blocks from peers for bootstrapping purposes. M3 also implements caching policies to optimize efficient reads by determining which flushed blocks are kept in memory.