flink AsyncWaitOperator 采坑 java.io.EOFException: No more bytes left

原创

已于 2022-10-16 10:53:27 修改 · 1.6k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#java #flink #apache

于 2022-10-16 00:15:00 首次发布

本文介绍在Flink项目中使用AsyncWaitOperator时遇到的状态恢复失败问题，由于新增字段导致序列化错误，通过调整类结构解决。

背景

最近有个项目是会用到flink做实时数据流处理，整个流图中算子较多，其中有一个算子依赖外部存储查询的数据，这里显然用AsyncWaitOperator来异步处理比较合适，写完之后就直接上线了，过了一阵子，因为迭代的需求，需要在AsyncWaitOperator的input对象中新增一个double类型，以为flink是直接兼容的，后面上线从state状态恢复启动失败，看异常日志，抛 java.io.EOFException: No more bytes left. 详细日志如下：

[Full restart]java.lang.Exception: Exception while creating StreamOperatorStateContext.
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:199)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:303)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:1170)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:459)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:937)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:692)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.util.FlinkException: Could not restore operator state backend for AsyncWaitOperator_f9688c8e333fe3850523a1cc41cc2cb6_(378/400) from any of the 1 provided restore options.
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.operatorStateBackend(StreamTaskStateInitializerImpl.java:259)
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:147)
	... 6 more
Caused by: org.apache.flink.runtime.state.BackendBuildingException: Failed when trying to restore operator state backend
	at org.apache.flink.runtime.state.DefaultOperatorStateBackendBuilder.build(DefaultOperatorStateBackendBuilder.java:88)
	at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createOperatorStateBackend(RocksDBStateBackend.java:631)
	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$operatorStateBackend$0(StreamTaskStateInitializerImpl.java:250)
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
	at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
	... 8 more
Caused by: java.io.EOFException: No more bytes left.
	at org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:79)
	at com.esotericsoftware.kryo.io.Input.readLong(Input.java:668)
	at com.esotericsoftware.kryo.io.Input.readDouble(Input.java:799)
	at com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeDoubleField.read(UnsafeCacheFields.java:180)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:346)
	at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:227)
	at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:48)
	at org.apache.flink.runtime.state.OperatorStateRestoreOperation.deserializeOperatorStateValues(OperatorStateRestoreOperation.java:191)
	at org.apache.flink.runtime.st