Batch interval, window length and slide duration on Spark Streaming

理解Spark Streaming：批处理间隔与窗口长度滑动持续时间

最新推荐文章于 2025-05-13 23:20:32 发布

原创

最新推荐文章于 2025-05-13 23:20:32 发布 · 861 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark

Spark Streaming中的批处理间隔（Batch interval）是系统接收数据的频率，例如2秒会产生一个数据批次。窗口长度（Window length）定义了窗口的持续时间，而滑动持续时间（Slide duration）指窗口向前滑动的间隔。两者都必须是批处理间隔的倍数。举例说明，当批处理间隔为2秒时，窗口长度为4秒且未指定滑动持续时间，则每2秒产生一个包含过去4秒数据的RDD，窗口之间存在重叠。如果设置滑动持续时间为4秒，那么每4秒产生一个不重叠的10秒数据窗口。

Spark Streaming中Batch interval, window length和slide duration的关系

Batch interval的含义

“Batch interval” is the basic interval at which the system with receive the data in batches. This is the interval set when creating a StreamingContext. For example, if you set the batch interval as 2 second, then any input DStream will generate RDDs of received data at 2 second intervals.

Length of window and slide duration的含义

A window operator is defined by two parameters -
- - the length of the window
- Slide duration - the interval at which the window will slide or move forward
Its a bit hard to explain the sliding of a window in words, so slides may be more useful. Take a look at slides 27 - 29 in the attached slides.