Understanding the IoTDB VARIATION Window Collapse

When working with time-series data in Apache IoTDB, the VARIATION window function is essential for detecting value changes and segmenting continuous metrics like vibration, temperature, or pressure. However, sensor warm-up sequences often emit non-finite NaN (Not a Number) readings. In Apache IoTDB 2.0.8 (Table Model), a leading NaN value can cause the VARIATION function to fail silently, lumping all subsequent readings into a single window_index of 0.

Why Does a Leading NaN Cause Window Collapse?

The VARIATION function segments data by comparing upcoming values to an established baseline. If the change exceeds the specified DELTA, a new window index is created, and the baseline resets.

When the first row in a partition is NaN, IoTDB uses NaN as the initial baseline. Under IEEE 754 floating-point rules, any arithmetic operation or comparison involving NaN (such as |10.0 - NaN| > 5.0) yields false. Because the comparison never evaluates to true, the threshold check never triggers, preventing the algorithm from creating subsequent windows.

How to Fix the Issue

Solution 1: Pre-Filter Non-Finite Values using CTEs

The recommended approach is to remove non-finite or NaN records using a Common Table Expression (CTE) before passing the dataset into the VARIATION function:

WITH clean_samples AS (
    SELECT *
    FROM vibration_samples
    WHERE amplitude IS NOT NULL 
      AND amplitude != 'NaN'
)
SELECT window_index, time, amplitude
FROM VARIATION(
    DATA => clean_samples PARTITION BY machine_id ORDER BY time,
    COL => 'amplitude',
    DELTA => 5.0
)
ORDER BY time;

Filtering out the NaN row ensures that the first valid reading (e.g., 10.0) is chosen as the baseline, allowing subsequent readings (e.g., 20.0) to correctly trigger window index 1.

Solution 2: Cleaning Data at Ingestion

If sensor warm-up data frequently populates your table model, consider filtering out non-finite readings during data ingestion or stream pre-processing pipelines (such as Apache Flink or the IoTDB Pipe framework).

Is this Expected Behavior?

No, a leading NaN is an edge-case behavior in window evaluation functions. Ideally, VARIATION should automatically skip non-finite values until it encounters the first valid finite sample to establish the baseline. Until an official patch handles non-finite baselines natively in IoTDB, explicitly filtering non-finite values is the safest production pattern.