flink-connector-hbase 任务提交yarn出现类冲突bugfix

本文分析了Apache Flink在连接HBase时遇到的两种运行时异常,这些异常源于配置文件中`hbase.status.listener.class`配置项的不匹配。问题在于Flink的HBase连接器和HBase客户端的配置文件存在冲突。解决方案包括更换Flink的HBase连接器版本或者直接修改`ConnectionImplementation`源码以避免从配置文件读取该配置。建议采用更改依赖为flink-connector-hbase并引入hbase-client的方式来解决此问题。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

第一种报错信息:

java.lang.RuntimeException: Cannot create connection to HBase.
	at org.apache.flink.connector.hbase.sink.HBaseSinkFunction.open(HBaseSinkFunction.java:99)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
	at org.apache.flink.table.runtime.operators.sink.SinkOperator.open(SinkOperator.java:58)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:437)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:574)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:554)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.reflect.UndeclaredThrowableException
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:65)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:36)
	at org.apache.flink.connector.hbase.sink.HBaseSinkFunction.open(HBaseSinkFunction.java:72)
	... 12 more
Caused by: java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1769)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:347)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:61)
	... 14 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionFactory.lambda$createConnection$0(ConnectionFactory.java:62)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
	... 16 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2299)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:339)
	... 24 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2267)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2291)
	... 25 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2171)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2265)
	... 26 more

第二种报错信息:

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ClusterStatusListener$Listener
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2299)
	at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:339)
	... 37 more
Caused by: java.lang.RuntimeException: class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ClusterStatusListener$Listener
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2293)

配置:

<dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-sql-connector-hbase-2.2_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop-client.version}</version>
            <scope>provided</scope>
            <exclusions>
                <exclusion>
                    <artifactId>commons-cli</artifactId>
                    <groupId>commons-cli</groupId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

产生问题原因:
flink-sql-connector-hbase-2.2_2.11中有一个配置文件hbase-default.xml,在打包的时候会打进来,在这个配置文件中有一个配置项为:

<property>
		<name>hbase.status.listener.class</name>
		<value>org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener</value>
		<description>
			Implementation of the status listener with a multicast message.
		</description>
	</property>

这个配置是导致问题产生的原因,如果引入flink-sql-connector-hbase-2.2_2.11,这个依赖中产生的hbase配置文件和hbase-common依赖自带的配置文件是不一样的,当任务提交到yarn上时,如果依赖的是flink-sql-connector-hbase-2.2_2.11,那么hbase.status.listener.class配置项的value应该为org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.ClusterStatusListenerMulticastListener,但是由于种种原因,最后代码读取配置文件时,读到该项配置为org.apache.hadoop.hbase.client.ClusterStatusListenerMulticastListener,但是由于种种原因,最后代码读取配置文件时,读到该项配置为org.apache.hadoop.hbase.client.ClusterStatusListenerMulticastListenerorg.apache.hadoop.hbase.client.ClusterStatusListenerMulticastListener ,导致上述报错信息产生。

解决方法:

  1. 将flink-sql-connector-hbase-2.2_2.11改为flink-connector-hbase-2.2_2.11,并且引入hbase-client,解决问题(推荐)
  2. 更改ConnectionImplementation源码:将其中的getClass方法写死,直接手动指定listenerClass (不推荐)
 this.metaCache = new MetaCache(this.metrics);
        boolean shouldListen = conf.getBoolean("hbase.status.published", false);
        this.hostnamesCanChange = conf.getBoolean("hbase.resolve.hostnames.on.failure", true);
        // 不去配置文件中获取,直接手动指定类
        Class<? extends Listener> listenerClass = ClusterStatusListener.DEFAULT_STATUS_LISTENER_CLASS;
        //Class<? extends Listener> listenerClass = conf.getClass("hbase.status.listener.class", ClusterStatusListener.DEFAULT_STATUS_LISTENER_CLASS, Listener.class);
        LOG.warn("listenerClass.name = {}",listenerClass.getName());
        this.alternateBufferedMutatorClassName = this.conf.get("hbase.client.bufferedmutator.classname");
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值