flink-connector-hbase 任务提交yarn出现类冲突bugfix

最新推荐文章于 2025-01-20 14:39:55 发布

TreeCode

最新推荐文章于 2025-01-20 14:39:55 发布

阅读量1.7k

点赞数 1

CC 4.0 BY-SA版权

分类专栏： flink 文章标签： flink hbase java

本文链接：https://blog.youkuaiyun.com/TreeCode/article/details/120185674

flink 专栏收录该内容

6 篇文章

订阅专栏

本文分析了Apache Flink在连接HBase时遇到的两种运行时异常，这些异常源于配置文件中`hbase.status.listener.class`配置项的不匹配。问题在于Flink的HBase连接器和HBase客户端的配置文件存在冲突。解决方案包括更换Flink的HBase连接器版本或者直接修改`ConnectionImplementation`源码以避免从配置文件读取该配置。建议采用更改依赖为flink-connector-hbase并引入hbase-client的方式来解决此问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

第一种报错信息：

java.lang.RuntimeException: Cannot create connection to HBase.
	at org.apache.flink.connector.hbase.sink.HBaseSinkFunction.open(HBaseSinkFunction.java:99)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
	at org.apache.flink.table.runtime.operators.sink.SinkOperator.open(SinkOperator.java:58)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:437)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:574)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:554)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.reflect.UndeclaredThrowableException
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:65)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:36)
	at org.apache.flink.connector.hbase.sink.HBaseSinkFunction.open(HBaseSinkFunction.java:72)
	... 12 more
Caused by: java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1769)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:347)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:61)
	... 14 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionFactory.lambda$createConnection$0(ConnectionFactory.java:62)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
	... 16 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2299)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:339)
	... 24 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2267)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2291)
	... 25 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2171)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2265)
	... 26 more

第二种报错信息：

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ClusterStatusListener$Listener
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2299)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:339)
	... 37 more
Caused by: java.lang.RuntimeException: class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ClusterStatusListener$Listener
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2293)

配置：

<dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-sql-connector-hbase-2.2_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop-client.version}</version>
            <scope>provided</scope>
            <exclusions>
                <exclusion>
                    <artifactId>commons-cli</artifactId>
                    <groupId>commons-cli</groupId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

产生问题原因：
flink-sql-connector-hbase-2.2_2.11中有一个配置文件hbase-default.xml，在打包的时候会打进来，在这个配置文件中有一个配置项为：

<property>
		<name>hbase.status.listener.class</name>
		<value>org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener</value>
		<description>
			Implementation of the status listener with a multicast message.
		</description>
	</property>

这个配置是导致问题产生的原因，如果引入flink-sql-connector-hbase-2.2_2.11，这个依赖中产生的hbase配置文件和hbase-common依赖自带的配置文件是不一样的，当任务提交到yarn上时，如果依赖的是flink-sql-connector-hbase-2.2_2.11，那么hbase.status.listener.class配置项的value应该为org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ClusterStatusListener $M u l t i c a s t L i s t e n e r ，但是由于种种原因，最后代码读取配置文件时，读到该项配置为 o r g . a p a c h e . h a d o o p . h b a s e . c l i e n t . C l u s t e r S t a t u s L i s t e n e r$ MulticastListener ，导致上述报错信息产生。

解决方法：

将flink-sql-connector-hbase-2.2_2.11改为flink-connector-hbase-2.2_2.11，并且引入hbase-client，解决问题（推荐）
更改ConnectionImplementation源码：将其中的getClass方法写死，直接手动指定listenerClass （不推荐）

 this.metaCache = new MetaCache(this.metrics);
        boolean shouldListen = conf.getBoolean("hbase.status.published", false);
        this.hostnamesCanChange = conf.getBoolean("hbase.resolve.hostnames.on.failure", true);
        // 不去配置文件中获取，直接手动指定类
        Class<? extends Listener> listenerClass = ClusterStatusListener.DEFAULT_STATUS_LISTENER_CLASS;
        //Class<? extends Listener> listenerClass = conf.getClass("hbase.status.listener.class", ClusterStatusListener.DEFAULT_STATUS_LISTENER_CLASS, Listener.class);
        LOG.warn("listenerClass.name = {}",listenerClass.getName());
        this.alternateBufferedMutatorClassName = this.conf.get("hbase.client.bufferedmutator.classname");