How InfluxDB 3 Enterprise Delivers 10-Millisecond Queries Over Historical Time Series Data

Navigate to:

Time series data, such as IoT sensor readings or stock market ticks, flow in fast, often at a rate of millions of points per second. Querying this data, especially years of historical records, can be slow and painful if using a nonspecialized database rather than a time series database like InfluxDB. InfluxDB 3 Enterprise consistently delivers query response times under 10 milliseconds for last value and under 30 milliseconds for distinct metadata, making your dashboards faster and more insightful, especially for real-time analytics.

Why speed matters

Imagine monitoring satellite sensor data remotely: each sensor sends thousands of data points per second, including temperature, pressure, and other relevant data. You need real-time insights to catch issues and historical analysis to spot trends. Slow queries can result in missed opportunities and degraded system performance. InfluxDB 3 Enterprise addresses this with a high-performance database engine built on top of the open source stack, diskless architecture, and layers of caching that keep queries blazing fast.

Key features driving 10-millisecond queries

Compactor and Indexing for Historical Data

The compactor is one of InfluxDB 3 Enterprise’s secret elements. It takes the default 10-minute Parquet files created during data ingestion and rewrites them into larger, optimized blocks. These blocks are sorted by series, and a separate index pinpoints exactly which files hold the data you need. This process slashes the number of files scanned for historical queries, keeping response times under 10 ms, even for queries with a longer time horizon.

While InfluxDB Core has a 72-hour/432 Parquet file limitation, Enterprise enables you to query months or years of data without slowdowns, making it ideal for deep historical analysis.

Caching and RAM Optimization with Apache Arrow

Both Core and Enterprise use Apache Arrow for in-memory data processing, which minimizes data copying and boosts CPU efficiency. Extensive buffering and caching means that recent or “hot” data remains in RAM, bypassing slower object storage. For historical data, the compactor’s optimized file layout and indexing reduce unnecessary data loads, keeping queries snappy.

Last Value Cache (LVC) and Distinct Value Cache (DVC)

The LVC and DVC are two unique types of caches that enhance time series query performance. The LVC delivers in under 10 ms, powering real-time dashboards and monitoring systems with minimal latency.

Example - Querying the Latest Temperature:

Suppose you have a satellite table with sensor data and want the latest temperature for the “shield.” Here’s how the LVC makes it lightning-fast:

Create the LVC: Use the ‘influxdb3 create last cache’ command.

influxdb3 create last_cache [OPTIONS] \   
  --database DATABASE_NAME \   
  --table TABLE_NAME \   
  --token AUTH_TOKEN \   
  CACHE_NAME

Example:

influxdb3 create last_cache \
  --database example-db \
  --table satellite \
  --key-columns shield \
  --token AUTH_TOKEN \
  satelliteLastCache

This command creates a last value cache named ‘satelliteLastCache’ for the ‘satellite’ table, using ‘shield’ as the key column. This means the cache will store the most recent values for each unique value of ‘shield’ in-memory

Query the LVC:

SELECT part, temp
FROM last_cache('satellite', 'satelliteLastCache') 
WHERE part = 'shield'

This retrieves the latest ‘temp’ value for the row where part = ‘shield’ from the in-memory cache, delivering results in under 10 ms.

Try it yourself

Want to see 10-millisecond queries in action for your data? InfluxDB 3 Enterprise offers a free 30-day trial (no credit card needed). If you have any questions or would like to share what you’re working on, please reach out to us on Discord, Slack, or our Community Forums.