【回顾】flink1.12+作业以per-job模式部署在yarn-cluster的异常，flink日志配置及webui日志输出，重启作业后checkpoint恢复

yxluojiecpp

已于 2022-10-13 16:17:30 修改

阅读量1.5k

点赞数

CC 4.0 BY-SA版权

分类专栏： # yarn集群大数据文章标签： flink yarn

于 2022-09-12 14:09:02 首次发布

本文链接：https://blog.youkuaiyun.com/yxluojiecpp/article/details/126801422

本文介绍了在将Flink作业部署到YARN集群时遇到的异常，包括由于Hadoop依赖冲突导致的问题，以及如何通过Maven Helper排除相关jar包。此外，还分享了flink pom.xml的日志配置，尽管存在日志警告但不影响输出。最后，讨论了作业重启后如何利用checkpoint恢复状态。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一、flink作业部署在yarn-cluster遇到的异常

某年某月某时，flink作业在本地测试运行的7788了，于是打了个jar包，打算部署yarn-cluster运行试试，我遇到过如下但不限于的异常：

异常

hadoop.hdfs 冲突

java.lang.IllegalAccessError:

class org.apache.hadoop.hdfs.web.HftpFileSystem

cannot access its superinterface

org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator

hadoop.comon 冲突

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system......

hadoop.comon 冲突

Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Cannot support file system for 'hdfs' via Hadoop, because Hadoop is not in the classpath, or some classes are missing from the classpath.

hadoop.auth 冲突

Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism(Ljava/lang/String;)V

hadoop-yarn-server-resourcemanager，hadoop-yarn-registry冲突

Caused by: java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$Registe