Understanding the M4 Downsampling Issue in Apache IoTDB

When working with high-frequency time series data in Apache IoTDB, downsampling algorithms like M4 are essential for responsive visualization and trend analysis. The M4 algorithm preserves the visual profile of time series charts by extracting four key data points from each window: the first point, the last point, the minimum point, and the maximum point.

However, developers working with Apache IoTDB 2.0.8 (and tree-model aggregation operations) often encounter surprising behavior when configuring overlapping sliding windows (e.g., setting windowSize=4 and slidingStep=2). The query output can move backward in time and return repeated timestamps across window boundaries.

Why Does M4 Emit Out-of-Order and Duplicate Timestamps?

To understand why this occurs, it helps to examine how window functions and M4 operators process data inside IoTDB:

  • Window-Independent Processing: IoTDB evaluates sliding windows independently. For a configuration with windowSize=4 and slidingStep=2, Window 1 processes timestamps 1000 through 4000, while Window 2 processes timestamps 3000 through 6000.
  • Independent Emission of Key Points: Window 1 evaluates its four points (First, Last, Min, Max) and emits them into the result stream. Next, Window 2 evaluates points in its range (3000 to 6000) and emits its four points.
  • Absence of Global Deduplication Across Windows: Because M4 computes per-window results directly without an auxiliary global sort/deduplication operator across overlapping windows, timestamps present in multiple windows (like 3000ms or 4000ms) appear repeatedly and cause temporal rollbacks in the final stream output.

Demonstration: Overlapping vs. Non-Overlapping Windows

Consider the following dataset registered in Apache IoTDB:

CREATE TIMESERIES root.operations_review.edge_lab.chart_a.value WITH DATATYPE=DOUBLE, ENCODING=PLAIN;

INSERT INTO root.operations_review.edge_lab.chart_a(timestamp, value) VALUES (1000, 5.0);
INSERT INTO root.operations_review.edge_lab.chart_a(timestamp, value) VALUES (2000, 1.0);
INSERT INTO root.operations_review.edge_lab.chart_a(timestamp, value) VALUES (3000, 9.0);
INSERT INTO root.operations_review.edge_lab.chart_a(timestamp, value) VALUES (4000, 4.0);
INSERT INTO root.operations_review.edge_lab.chart_a(timestamp, value) VALUES (5000, 8.0);
INSERT INTO root.operations_review.edge_lab.chart_a(timestamp, value) VALUES (6000, 2.0);
INSERT INTO root.operations_review.edge_lab.chart_a(timestamp, value) VALUES (7000, 7.0);
INSERT INTO root.operations_review.edge_lab.chart_a(timestamp, value) VALUES (8000, 3.0);

Executing an M4 query with overlapping step sizes (slidingStep=2):

SELECT value, M4(value, 'windowSize'='4', 'slidingStep'='2') AS sampled
FROM root.operations_review.edge_lab.chart_a;

This query returns non-monotonic output because Window 2 emits timestamp 08:00:03.000 right after Window 1 has already emitted timestamp 08:00:04.000.

By contrast, setting slidingStep equal to windowSize removes overlapping regions completely:

SELECT value, M4(value, 'windowSize'='4', 'slidingStep'='4') AS sampled
FROM root.operations_review.edge_lab.chart_a;

This configuration yields strictly monotonic, ordered timestamps without duplicates.

Recommended Solutions & Workarounds

1. Align slidingStep with windowSize

The M4 algorithm was designed primarily for fixed-pixel trend chart downsampling where time buckets do not overlap. For standard UI visualization, ensure that slidingStep matches windowSize to guarantee monotonic output directly from the database engine.

2. Deduplicate and Sort on the Client Side

If your application architecture specifically requires overlapping window calculations (such as calculating continuous rolling envelopes), sort and deduplicate the result set on the application layer (e.g., using a client-side Map or array sorter before handing off points to plotting tools like ECharts or Grafana).

3. Use Native Time-Based GROUP BY Clauses

If you prefer downsampling based on fixed time intervals rather than row counts, leverage IoTDB's native time aggregation features:

SELECT M4(value) 
FROM root.operations_review.edge_lab.chart_a
GROUP BY ([1000, 9000), 4s);

Summary

In Apache IoTDB, overlapping M4 windows (where slidingStep < windowSize) independently emit per-window key points, resulting in non-monotonic and duplicate timestamps. This is expected execution behavior rather than a bug. For standard downsampling and charting needs, set slidingStep equal to windowSize.