Flink AsyncFunction time out 导致消费卡主

使用Flink 异步IO去关联维表的时候，当QPS和TPS上去以后，就有可能会出现以后情况，超时，然后程序就会僵住。

java.lang.Exception: An async function call terminated with an exception. Failing the AsyncWaitOperator.
	at org.apache.flink.streaming.api.operators.async.Emitter.output(Emitter.java:137)
	at org.apache.flink.streaming.api.operators.async.Emitter.run(Emitter.java:85)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Async function call has timed out.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
	at org.apache.flink.streaming.api.operators.async.queue.StreamRecordQueueEntry.get(StreamRecordQueueEntry.java:68)
	at org.apache.flink.streaming.api.operators.async.Emitter.output(Emitter.java:129)
	... 2 more
Caused by: java.util.concurrent.TimeoutException: Async function call has timed out.
	at org.apache.flink.streaming.api.functions.async.AsyncFunction.timeout(AsyncFunction.java:97)
	at org.apache.flink.streaming.api.operators.async.AsyncWaitOperator$1.onProcessingTime(AsyncWaitOperator.java:215)
	at org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:281)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more

 

具体涉及到的底层代码，如下 ：在AsyncWaitOperator中

if (timeout > 0L) {
			// register a timeout for this AsyncStreamRecordBufferEntry
			long timeoutTimestamp = timeout + getProcessingTimeService().getCurrentProcessingTime();

			final ScheduledFuture<?> timerFuture = getProcessingTimeService().registerTimer(
				timeoutTimestamp,
				new ProcessingTimeCallback() {
					@Override
					public void onProcessingTime(long timestamp) throws Exception {
						userFunction.timeout(element.getValue(), streamRecordBufferEntry);
					}
				});

			// Cancel the timer once we've completed the stream record buffer entry. This will remove
			// the register trigger task
			streamRecordBufferEntry.onComplete(
				(StreamElementQueueEntry<Collection<OUT>> value) -> {
					timerFuture.cancel(true);
				},
				executor);
		}

涉及到了对timeout的处理，会条用AsyncFunction的timeout方法，这个方法只会抛出一个错误，就是timeoutexception

default void timeout(IN input, ResultFuture<OUT> resultFuture) throws Exception {
		resultFuture.completeExceptionally(
			new TimeoutException("Async function call has timed out."));
	}

解决方式

重写 timeout方法即可，不让他抛出timeout异常即可。

    @Override
    public void timeout(FlowModel input, ResultFuture<FlowModel> resultFuture) throws Exception {
        //超时了。丢弃即可
        System.out.println("超时了：" + input);
        //重新调用处理方法
//        asyncInvoke(input,resultFuture);

    }