InfluxData Blog - Reid Kaufmann

A New Way to Debug Query Performance in Cloud Dedicated

Reid Kaufmann (InfluxData) — Tue, 20 Jan 2026 08:00:00 +0000

I’d like to share a new influxctl ease-of-use feature in v2.12.0 that makes it easier to optimize important queries or debug slow ones. influxctl has had the capability to send queries and display the results in JSON or tabular formats for some time. (Note: this CLI utility is specific to Cloud Dedicated and Clustered, as are many of the specifics in this post.) In Clustered, you can monitor querier pods’ logs, and in both Dedicated and Clustered, metrics on individual queries’ performance can be found in the system tables. Both of those options offer a lot of data—enough that it can be hard to digest quickly. Additionally, associating a single execution of a query to its log entry is tedious. A new feature, the --perf-debug flag for the influxctl query command (release notes), accelerates the experimentation cycle by providing real-time feedback, allowing you to stay in the context of your shell as you tweak your query.

Sample output

The new flag, --perf-debug, will execute a query, collect and discard the results, and emit execution metrics instead. When --format is omitted, output defaults to a tabular format with units dynamically chosen for human readability. In the second execution below, --format json is specified to emit a data format appropriate for programmatic consumption: in a nod to the querier log, it uses keys with shorter variable names, delimits words with underscores, and sports consistent units (bytes, seconds as a float).

In the tabular format, you can also see a demarcation between client and server metrics.

$ influxctl query --perf-debug --token REDACTED --database reidtest3 --language influxql "SELECT SUM(i), non_negative_difference(SUM(i)) as diff_i FROM data WHERE time > '2025-11-07T01:20:00Z' AND time " '2025-11-07T03:00:00Z' AND runid = '540cd752bb6411f0a23e30894adea878' GROUP BY time(5m)"
+--------------------------+----------+
| Metric                   | Value    |
+--------------------------+----------+
| Client Duration          | 1.222 s  |
| Output Rows              | 20       |
| Output Size              | 647 B    |
+--------------------------+----------+
| Compute Duration         | 37.2 ms  |
| Execution Duration       | 243.8 ms |
| Ingester Latency Data    | 0        |
| Ingester Latency Plan    | 0        |
| Ingester Partition Count | 0        |
| Ingester Response        | 0 B      |
| Ingester Response Rows   | 0        |
| Max Memory               | 70 KiB   |
| Parquet Files            | 1        |
| Partitions               | 1        |
| Planning Duration        | 9.6 ms   |
| Queue Duration           | 286.6 µs |
+--------------------------+----------+

$ influxctl query --perf-debug --format json --token REDACTED --database reidtest3 --language influxql "SELECT SUM(i), non_negative_difference(SUM(i)) as diff_i FROM data WHERE time > '2025-11-07T01:20:00Z' AND time " '2025-11-07T03:00:00Z' AND runid = '540cd752bb6411f0a23e30894adea878' GROUP BY time(5m)"
{
  "client_duration_secs": 1.101,
  "compute_duration_secs": 0.037,
  "execution_duration_secs": 0.247,
  "ingester_latency_data": 0,
  "ingester_latency_plan": 0,
  "ingester_partition_count": 0,
  "ingester_response_bytes": 0,
  "ingester_response_rows": 0,
  "max_memory_bytes": 71744,
  "output_bytes": 647,
  "output_rows": 20,
  "parquet_files": 1,
  "partitions": 1,
  "planning_duration_secs": 0.009,
  "queue_duration_secs": 0
  }

Notes

Client duration includes the time to open the connection to the server. In the example, you can see a big delta between that and the server’s total duration. When I ran this command, my client and database server were not colocated. Additionally, influxctl may not be tuned for optimal connection latency. Your native client probably caches connections and might not suffer this latency. When tuning your query, it’s more important to look at the durations recorded by the server.

Output size is the size in Arrow format in memory, after gzip inflation (if client and server agree on compression), so this metric does not report the bytes transferred. The network bytes transferred might be more useful, so that’s a potential future enhancement. However, the current metric can still provide a relative metric to compare between different queries.

Ingester metrics are zeroed out if the ingester has no partitions with unpersisted data matching the query. In Serverless, Dedicated, and Clustered, queries always consult ingesters, so the 0 in ingester latency can be misleading.

Parquet files indicate how many files were traversed for the query. However, if the query was optimized by a ProgressiveEvalExec plan (simple sorted LIMIT queries without aggregations, typically; verify with EXPLAIN ANALYZE), this value may not be useful because it is calculated in planning and then reflects the potential number of files to be accessed determined by the time range, as opposed to the actual number accessed before reaching the LIMIT. For most queries, this metric is a handy indicator, but it’s worth noting that the query log also contains a related metric, deduplicated_parquet_files, which tells us how many of the files had overlapping time ranges, requiring the querier to merge/sort/deduplicate data. It’s normal to have a few files at the leading edge, but this operation becomes a serious bottleneck if too much data needs to be deduplicated (the main responsibility of the compactor is to manage this problem).

Query durations vary, and a query can be executed several times (and at different times of day) to get a sense of the variation.

Potential sources of latency or variability

Cache warmup (shows up in Execution Duration): The first few times a query in a particular time frame for a table is executed, the duration may be significantly higher due to parquet cache misses. Queries fan out to multiple queriers in a round robin fashion, and each querier has an independent parquet cache, so expect a few cache misses as each querier may have to incur the delay of retrieving parquet files from the object store. Due to multiple load balancer pods and other clients executing queries, how many query executions it will take to warm up all queries is indeterminate. If the queries are on the “leading” edge of the data, be aware that persistence of new data OR compaction may also periodically cause cache misses. Large L2 file compactions lead to a greater disruption, while latency from typical small incremental persists may be imperceptible.

Corollary: cache eviction. Other queries executing may cause cache eviction to make room for their data. Given a high rate of queries covering a lot of data (many series and/or a wide time frame), it’s possible to thrash the cache. In this case, influxctl can’t provide much context about other queries running at the same time (exception: a non-zero Queue Duration does indicate maximum execution concurrency was reached). You may still need to review the query log or observability dashboards. Some query loads are cyclical, and so is the work of the compactor, depending on ingest and partitioning rates; therefore, you may get better performance in the afternoon than in the morning. When the CPU is maxed out, it tends to increase all recorded server latencies.

Variation in data density or volume will affect all queries to some degree, but will impact computationally intensive queries the most. This shows up in Execution Duration. Monitor the Parquet Files or the Output Rows metrics as possible proxies for this. Be aware that changing a tag value in the WHERE clause or the time constraints may affect latency, depending on the underlying data. Not all writers may employ the same frequency. When tuning aggregate queries, you may occasionally want to add a COUNT() field and drop the --perf-debug flag to see how many records are contributing. For some queries (SELECT DISTINCT, for example), tag cardinality and time range can greatly impact performance.

See documentation for more general information on query optimization.

Other things to try

Check if Planning Duration is substantially higher than Execution Duration. This can be caused by high numbers of tables or partitions, which may be excessive or intended. Custom partitioning can help reduce execution latency, but can increase planning latency—find the right balance for your workload.
Check if Ingester Latency or Response is abnormally high/large. It may indicate a need for, or a problem with, custom partitioning, resulting in excessive delay in persisting partitions.
If Parquet Files is abnormally large, check that the query has a time constraint and that it’s reasonable. Also, check observability dashboards to see if the compactor is not keeping up or look for skipped partitions. If customer partitioning on a tag is in use, make sure the query is specifying a value for that tag in the WHERE clause (also note that regexes that don’t equate to simple equality checks on said field will also prevent partition pruning).
How much does increasing or decreasing the time range of the query change the execution metrics?
Compare similar queries against different tables, schemas, or partitioning schemes.
Compare different means of achieving the same result (SQL ORDER BY time DESC LIMIT 1 vs INFLUXQL LAST)

You can learn a lot through experimentation and finding correlations beyond those suggested here. We hope this minor feature makes it a little easier!

Optimizing Queries in InfluxDB 3 Using Progressive Evaluation

Nga Tran, Reid Kaufmann (InfluxData) — Thu, 14 Nov 2024 07:00:00 +0000

In a previous post, we described the technique that makes the “most recent values” queries hundreds of times faster and has benefited many of our customers. The idea behind this technique is to progressively evaluate time-organized files until we reach the most recent values. Since then, we have received questions like “What queries support progressive evaluation?” “How do we verify that a query is progressively evaluated?” “Are there certain file organizations that progressive evaluation won’t help?” This blog post answers those questions.

Queries that support progressive evaluation

Currently, this technique is only available for SQL queries; it is not yet applicable to InfluxQL queries. Your SQL query must have the clause ORDER BY time DESC (or ASC). In addition, some other limitations include expressions, aliases (including AT TIME ZONE), and aggregations on the SELECT clause. In other words, everything in the SELECT clause must be simple table columns.

Examples of supported queries

SELECT host, temperature 
FROM   machine 
WHERE  time > now - interval ‘1 day’ and region = ‘US’
ORDER BY time ASC;

SELECT host, temperature 
FROM   machine 
WHERE  time > now - interval ‘1 day’ and region = ‘US’
ORDER BY time DESC
LIMIT  10;

Examples of unsupported queries

These queries are not optimized using progressive evaluation yet. We hope to lift the restrictions in a future release.

Query with an expression (temperature + 2)

SELECT host, temperature + 2
FROM   machine 
WHERE  time > now - interval ‘1 day’ and region = ‘US’
ORDER BY time ASC;

Query with an alias (as host_name)

SELECT host as host_name, time 
FROM   machine 
WHERE  time > now - interval ‘1 day’ and region = ‘US’
ORDER BY time ASC;

Query that specifies time zone and then alias (AT TIME ZONE ‘Europe/Oslo’ as time)

SELECT host, time AT TIME ZONE ‘Europe/Oslo’ as time 
FROM   machine 
WHERE  time > now - interval ‘1 day’ and region = ‘US’
ORDER BY time DESC
LIMIT  10;

Query with an aggregate (min(temperature))

SELECT min(temperature)
FROM   machine 
WHERE  time > now - interval ‘1 day’ and region = ‘US’
ORDER BY time DESC
LIMIT  10;

Signal that a query is evaluated progressively

If ProgressiveEvalExec is in your query plan, it is optimized using progressive evaluation (see this post for how to get and read the query plan). However, the absence of progressive evaluation does not mean your query will run slowly. We only apply it when it actually benefits your query.

File organizations that benefit from progressive evaluation

To understand when progressive evaluation benefits your query, we first need to understand data organization, which is one of the most important factors affecting query performance.

Data organization in InfluxDB 3.0

Below is a brief description of how data is organized in InfluxDB 3.0 (see our system architecture and data compaction for the complete data cycle, how data is compacted, and how it benefits query performance).

As a time series database, data from the table machine always includes a time column representing the time of an event, such as temperature at 9:30 am UTC. Figure 1 shows three different stages of data organization. Each rectangle in the figure illustrates a chunk of data. C represents data that is not yet persisted and usually includes the most recent values. L represents the level of different persisted files. L0 is used for files of newly ingested and persisted data. They are usually small and contain recent values. However, L0 files of backfilled data can be as old as desired. L1 files store the results of compacting many small L0 files. We also have L2 files but they are beyond the scope of this topic and do not change how progressive evaluation works.

Figure 1 (borrowed from the compaction blog post): Four stages of data organization after two rounds of compaction.

In stage 1, all data are in small L0 files. In stage 2, data in stage 1 has been compacted to larger L1 files, while some new data are persisted in a few small L0 files and some are in a not-yet-persisted chunk (C). If you ingest new data most of the time, your data organization mostly looks like stage 2 or stage 3. However, if you backfill data, your data organization can combine stages 1 and stage 2 or stage 3. Thus, depending on how you ingest data and how fast the compactor keeps up with your ingest workload, there may be few or many overlapped and small files. Stage 3 is what we call “well-compacted data” and is usually best for query performance. The goal of the compactor is to have most of your data in stage 3. Avoiding frequent backfilling of data also helps keep your data well-compacted.

Application of progressive evaluation in various overlap scenarios

Let’s go over examples of querying different data sets. Figure 2 shows the data organization of the table machine. We use F for the prefix name of the files, which can be either L0 or L1. Files F1, F2, F6, and F7 do not time-overlap with any files. Files F3, F4, and F5 overlap with each other, and file F8 overlaps with F9, which overlaps with chunk C.

Figure 2: Data organization of table machine

Reading Non-Overlapped Files Only

If your query asks for latest data before t1:

SELECT temperature 
FROM   machine 
WHERE  time < t1 and region = ‘US’
ORDER BY time DESC LIMIT  1;

The query will be optimized with progressive evaluation because the files needed, F1 and F2, do not overlap. The simplified query plan is as follows:

ProgressiveEvalExec: fetch=1
    SortExec: TopK(fetch=1), expr=[time DESC], preserve_partitioning=[true]   
         ParquetExec: file_groups={2 groups: [F2], [F1]}

A few essential properties in the query plan are needed for ProgressiveEvalExec to work correctly:

Files in ParquetExec are sorted on time descending F2, F1.
preserve_partitioning=[true] means data of 2 file groups, [F2] and [F1], are sorted in their own group and won’t be merged. This is important for us to be able to fetch data from F2 before fetching F1.
fetch=1 means the query will stop running as soon as it gets a row that meets the query filters. In other words, if there is at least one row in file F2 with US as a region, F1 will never be read.

The same query sorted on time ascending will look like this:

ProgressiveEvalExec: fetch=1
    SortExec: TopK(fetch=1), expr=[time ASC], preserve_partitioning=[true]   
         ParquetExec: file_groups={2 groups: [F1], [F2]}

You will get a similar query plan if your query reads data between t2 and t3 that include only non-overlapped files.

Reading Overlapped Files Only

Now let’s look at the query reading data between t1 and t2.

SELECT temperature 
FROM   machine 
WHERE  time < t2 and time > t1 and region = ‘US’ 
ORDER BY time DESC LIMIT  1;

Because all three needed files, F3, F4, and F5, overlap, we need to merge data for deduplication and they cannot be evaluated one by one progressively. The simplified query plan will look like this:

SortExec: TopK(fetch=1), expr=[time DESC], preserve_partitioning=[false]
  DeduplicateExec:
     SortPreservingMergeExec:
         ParquetExec: file_groups={3 groups: [F4], [F3], [F5]}

Without ProgressiveEvalExec, the files in ParquetExec can be in any order and group because the groups will be read in parallel and merged into one stream for deduplication.

Similarly, progressive evaluation won’t be applied if your query reads overlapped data after t3.

Reading a Mixture of Non-Overlapped and Overlapped Files

When your query reads a mixture of non-overlapped and overlapped data, progressive evaluation is applied, and the data is split and grouped accordingly. Let’s look at a query that reads data before t2.

SELECT temperature 
FROM   machine 
WHERE  time < t2 and region = ‘US’ 
ORDER BY time DESC LIMIT  1;

The simplified query plan will look like this:

ProgressiveEvalExec: fetch=1
    SortExec: TopK(fetch=1), expr=[time DESC], preserve_partitioning=[false]
       DeduplicateExec:
          SortPreservingMergeExec:
              ParquetExec: file_groups={3 groups: [F4], [F3], [F5]}
    SortExec: TopK(fetch=1), expr=[time DESC], preserve_partitioning=[true]   
         ParquetExec: file_groups={2 groups: [F2], [F1]}

Even though files F3, F4, and F5 overlap, they do not overlap with F1 and F2 and contain more recent data. Therefore, the subplan of F3, F4, and F5 is progressively evaluated with F2 and F1. Note that the number of input streams into ProgressiveEvalExec is three: one for the merge of F3, F4, and F5, one for F2, and one for F1. These three streams are evaluated progressively in that order.

If the query is sorted ascending, the progressive order will be opposite:

ProgressiveEvalExec: fetch=1
    SortExec: TopK(fetch=1), expr=[time ASC], preserve_partitioning=[true]   
         ParquetExec: file_groups={2 groups: [F1], [F2]}
    SortExec: TopK(fetch=1), expr=[time ASC], preserve_partitioning=[false]
       DeduplicateExec:
          SortPreservingMergeExec:
              ParquetExec: file_groups={3 groups: [F4], [F3], [F5]}

Three streams still go into ProgressiveEvalExec but in the opposite order: F1, F2, and the F3, F4, and F5 merge.

Similarly, let’s read all data:

SELECT temperature 
FROM   machine 
WHERE  time < now and region = ‘US’ 
ORDER BY time DESC LIMIT  1;

The query plan is getting more complicated but follows the same rules: subplans of overlapped data will be put in the right order and progressively evaluated with non-overlapped files.

ProgressiveEvalExec: fetch=1
    SortExec: TopK(fetch=1), expr=[time DESC], preserve_partitioning=[false]
       DeduplicateExec:
          SortPreservingMergeExec:
              SortExec:
                 RecordBatchExec: {C}
              ParquetExec: file_groups={2 groups: [F8], [F9]}
    SortExec: TopK(fetch=1), expr=[time DESC], preserve_partitioning=[true]   
         ParquetExec: file_groups={2 groups: [F7], [F6]}
    SortExec: TopK(fetch=1), expr=[time DESC], preserve_partitioning=[false]
       DeduplicateExec:
          SortPreservingMergeExec:
              ParquetExec: file_groups={3 groups: [F4], [F3], [F5]}
    SortExec: TopK(fetch=1), expr=[time DESC], preserve_partitioning=[true]   
         ParquetExec: file_groups={2 groups: [F2], [F1]}

There are six non-overlapped streams of data progressively evaluated by ProgressiveEvalExec in this order:

The merge of C, F8 and F9
F7
F6
The merge of F3, F4 and F5
F2
F1

If your query orders data on time ascending, you will have a similar query plan but in the opposite order.

Cache implications: progressive evaluation may help other queries’ latency

For workloads highly dependent on cached files to get the lowest latency possible but running near the cache limit, progressive evaluation can make a significant performance difference. Obviously, not traversing extra parquet files reduces CPU time for the optimized query. Depending on how excessive the time bound is compared to the bounds of the LIMIT, the fact that the database may bring far fewer files into cache means other queries’ files need not be evicted, potentially reducing the latency of other queries on the system.

Summing up

If your query selects pure table columns and orders data on time, InfluxDB 3.0 will automatically use progressive evaluation to improve your query performance, even if a subset of the query data contains non-overlapped files. Progressive evaluation cannot be used when all your data overlaps.