Flink——理解 allowLateness

最新推荐文章于 2024-11-27 18:46:39 发布

原创

最新推荐文章于 2024-11-27 18:46:39 发布 · 661 阅读

1 ·

CC 4.0 BY-SA版权

本文详细解析了Flink中WindowOperator的allowLateness机制，包括其内部逻辑、代码实现及实际应用效果，帮助读者理解如何有效处理迟到数据。

什么鬼

WindowOperator 里面还有有一个叫做 allowLateness 的东西，这个东西什么鬼呢？简单来说就给迟到的数据第二次机会。我允许它迟到一定的时间。在规定的迟到时间内，只要要数据来了，就会触发第二次窗口计算，那到什么时候就没有第二次机会了呢？下面我们来娓娓道来。

allowLateness 的逻辑过程

二话不说，先来看一下下面的代码，在这段代码中，


```java
WindowOperator 中的成员变量
 
  	/**
	 * The allowed lateness for elements. This is used for:
	 * <ul>
	 *     <li>Deciding if an element should be dropped from a window due to lateness.
	 *     <li>Clearing the state of a window if the system time passes the
	 *         {@code window.maxTimestamp + allowedLateness} landmark.
	 * </ul>
	 */
	protected final long allowedLateness;

从上面的代码中，意思是允许元素迟到多长时间。它两个点作用：

根据迟到的时间，决定一个元素是否别丢弃。
如果系统时间超过了运行的最时间，Flink 就会清清理窗口的状态，这个阈值就是 window.maxTimestamp + allowedLateness 。超过这个值就会清楚窗口的状态。

如果想弄清楚这里的逻辑，就要到 WindowOperator onElement 这个方法里面找答案。

先来上代码：

public void processElement(StreamRecord<IN> element) throws Exception {
   
   
		final Collection<W> elementWindows = windowAssigner.assignWindows(
			element.getValue(), element.getTimestamp(), windowAssignerContext);

		//if element is handled by none of assigned elementWindows
		boolean isSkippedElement = true;

		final K key = this.<K>getKeyedStateBackend().getCurrentKey();

		if (windowAssigner instanceof MergingWindowAssigner) {
   
   
			MergingWindowSet<W> mergingWindows = getMergingWindowSet();

			for (W window: elementWindows) {
   
   

				// adding the new window might result in a merge, in that case the actualWindow
				// is the merged window and we work with that. If we don't merge then
				// actualWindow == window
				W actualWindow = mergingWindows.addWindow(window, new MergingWindowSet.MergeFunction<W>() {
   
   
					@Override
					public void merge(W mergeResult,
							Collection<W> mergedWindows, W stateWindowResult,
							Collection<W> mergedStateWindows) throws Exception {
   
   

						if ((windowAssigner.isEventTime() && mergeResult.maxTimestamp() + allowedLateness <= internalTimerService.currentWatermark())) {
   
   
							throw new UnsupportedOperationException("The end timestamp of an " +
									"event-time window cannot become earlier than the current watermark " +
									"by merging. Current watermark: " + internalTimerService.currentWatermark() +
									" window: " + mergeResult);
						} else if (!windowAssigner.isEventTime()) {
   
   
							long currentProcessingTime = internalTimerService.currentProcessingTime();
							if (mergeResult.maxTimestamp() <= currentProcessingTime) {
   
   
								throw new UnsupportedOperationException("The end timestamp of a " +
									"processing-time window cannot become earlier than the current processing time " +
									"by merging. Current processing ti