Flink 维表异步查询的实现以及问题排查

原创

已于 2022-09-14 20:37:36 修改 · 2.5k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#flink #big data #java

于 2022-06-10 18:25:34 首次发布

本文探讨了Flink 1.13.3中使用VVR版本CloudHBase Connector时遇到的同步查询速度慢和NPE问题，通过分析发现是由于多线程环境下RowData对象共享导致的。解决办法是修改代码，避免对象共享并修复了calc函数对null值的处理。同时揭示了Flink SQL的异步操作实现方式不同于预期，仅需实现eval方法而非asyncInvoke。

背景

本文基于Flink 1.13.3
Flink计算引擎VVR版本的hbase Connector
具体maven依赖如下：

<dependency>
	<groupId>com.alibaba.ververica</groupId>
	<artifactId>ververica-connector-cloudhbase</artifactId>
	<version>1.13-vvr-4.0.7</version>
</dependency>

在基于VVR版本的cloudHbase维表查询的时候，发现同步查询的速度很慢，所以我们打算做基于异步的维表查询。

在运行的过程中发现了NPE问题,具体的报错堆栈如下：

2022-06-08 15:01:05
java.lang.Exception: Could not complete the stream element: Record @ (undef) : org.apache.flink.table.data.binary.BinaryRowData@d8014011.
	at org.apache.flink.streaming.api.operators.async.AsyncWaitOperator$ResultHandler.completeExceptionally(AsyncWaitOperator.java:382)
	at org.apache.flink.table.runtime.operators.join.lookup.AsyncLookupJoinRunner$JoinedRowResultFuture.completeExceptionally(AsyncLookupJoinRunner.java:253)
	at org.apache.flink.table.runtime.operators.join.lookup.AsyncLookupJoinRunner$JoinedRowResultFuture$DelegateResultFuture.completeExceptionally(AsyncLookupJoinRunner.java:275)
	at org.apache.flink.table.runtime.collector.TableFunctionResultFuture.completeExceptionally(TableFunctionResultFuture.java:61)
	at org.apache.flink.table.runtime.operators.join.lookup.AsyncLookupJoinWithCalcRunner$TemporalTableCalcResultFuture.complete(AsyncLookupJoinWithCalcRunner.java:121)
	at org.apache.flink.table.runtime.operators.join.lookup.AsyncLookupJoinRunner$JoinedRowResultFuture.complete(AsyncLookupJoinRunner.java:219)
	at org.apache.flink.table.runtime.operators.join.lookup.DelegatingResultFuture.accept(DelegatingResultFuture.java:48)
	at org.apache.flink.table.runtime.operators.join.lookup.DelegatingResultFuture.accept(DelegatingResultFuture.java:32)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	at org.apache.flink.connector.xx.cloudhbase.source.AsyncHBaseLRURowFetcher.lambda$fetchResult$6(AsyncHBaseLRURowFetcher.java:223)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:443)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.NullPointerExce

最低0.47元/天解锁文章