spark2.4.7兼容haddoop 3.1.4 和hive 3.1.2

原创已于 2023-04-12 17:25:55 修改 · 4.5k 阅读

12 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #spark #hive #大数据

于 2021-05-17 10:27:35 首次发布

大数据专栏收录该内容

1 篇文章

订阅专栏

背景

公司有一套大数据解决方案平台，包括数据同步平台、大数据管理平台、数据可视平台、数据监控报警平台等。所有平台是基于apache 的 hadoop 2.6.3、hive 1.2.1、spark 2.0.2 、scala 2.11.8（其他组件暂不详述），现在需要搭建一套新的集群，决定新集群组件采用新的版本，hadoop 3.1.4 、hive 3.1.2 、spark 2.4.7、scala 2.12.13。基于以上组件的升级，数据平台项目需要做同步的升级，在升级过程中，遇到一些api改变、版本不兼容、调用各种异常报错等问题，本篇主要介绍上述组件的兼容如何配置和遇到的问题。

环境

hadoop 3.1.4
hive 3.1.2
spark 2.4.7
jdk 1.8.0_271

前提说明

版本的选择是参考cdh 、hdp 采用的组件版本组合，同时结合了hive官网给出的hive on spark 对hive和spark匹配测试过的版本组合。spark采用了自编译的方式手动编译的，内置版本不符合我们对hadoop版本和 hive版本的要求

./dev/make-distribution.sh --name "hadoop3-with-hive" --tgz "-Pyarn,hadoop-3.1,scala-2.12,parquet-provided,orc-provided,hive,hive-thriftserver" -Dhadoop.version=3.1.4 -Dscala.version=2.12.13 -Dscala.binary.version=2.12

hadoop和hive均从apache官网连接直接下载相关预编译的版本，直接解压安装。
hive配置了hive on spark
spark 提交方式是在 spark on yarn 的cluster模式下

下面讲的配置和问题都是基于这些前提之下

配置

1、hive server 端

如果配置hive on spark需要配置 hive-site.xml

  <property>
    <name>hive.execution.engine</name>
    <value>spark</value>
    <description>
      Expects one of [mr, tez, spark].
      Chooses execution engine. Options are: mr (Map reduce, default), tez, spark. While MR
      remains the default engine for historical reasons, it is itself a historical engine
      and is deprecated in Hive 2 line. It may be removed without further warning.
    </description>
  </property>

需要将编译好spark的jars 目录下的一下jar包复制到$HIVE_HOME/lib 目录下

在这里插入图片描述

Hive 执行可能会遇到其他问题，还需要以下两个包，第一个升级guava否则会出现jar包兼容问题，第二个采用mysql来做元数据存储的话需要mysql的driver 在这里插入图片描述

2、hadoop server端

$SPARK_HOME/yarn/spark-2.4.7-yarn-shuffle.jar 复制到$HADOOP_HOME/share/hadoop/yarn/lib

3、提交spark任务client配置

sparkConf中资源配置和性能调优不做详细说明，只说兼容相关的配置

3.1指定hive的metastore版本和lib目录，spark自带默认版本是1.2.1

# 配置自定义 metastore 版本和jar包地址(可以配置hdfs，带验证)
spark.sql.hive.metastore.version=3.1.2
spark.sql.hive.metastore.jars=/data/hadoop-3.1.4/metastorelib/*

$HADOOP_HOME目录下创建metastorelib，并把$HIVE_HOME/lib 下jar复制过去，也可以把路径指定到$HIVE_HOME/lib

http://spark.apache.org/docs/2.4.7/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore

3.2 设置分发到executor上的文件

spark.yarn.dist.files=core-site.xml,hive-site.xml,hdfs-site.xml,yarn-site.xml,og4j.properties
spark.yarn.dist.files=xxx.jar

如果有依赖jar包可以配置，我们项目是把所有依赖打到一个jar包了，所以可以不用设置

http://spark.apache.org/docs/2.4.7/running-on-yarn.html

3.3 设置spark依赖

将$SPARK_HOME/lib 达成一个zip文件，上传到hdfs，然后指定以下配置

spark.yarn.archive=hdfs://hadoop3cluster/user/spark/jars/spark.zip

spark任务提交各种报错和jar包冲突，主要集中在zip中的jar里面。后面会详细说明这个针对zip中jar的选择和处理

3.4 spark任务提交方式

Client client = null;
try {
  client = new Client(clientArgs, sparkConf);
  client.setApplicationId(applicationId); //修改spark源码支持
  // 提交application
  client.submitApplication();
} finally {
  //关闭client连接
  try {
    if (client != null) {
      LOGGER.info("finally stop the spark client.");
      client.stop();
    }
  } catch (Exception e) {
    LOGGER.warn("the yarn client stop exception", e);
    client = null;
  }
}

其中sparkConf自定义参数配置和上面定义的参数配置，clientArgs指定–class 、–jar 等参数

3.5 spark应用

SparkSession sparkSession = SparkSession.builder().config(sparkConf).enableHiveSupport().getOrCreate();

enableHiveSupport() 这一步很重要，很多hive相关配置都可以省掉，后面遇到很多问题，通过查阅资料增加各种配置，事实证明启动了hiveSupport后，那些配置并不需要配置，比如：

spark.sql.catalogImplementation=hive
hive.metastore.warehouse.dir=xxx  ##在配置到hdfs上的hive-site.xml中配置即可
hive.metastore.uris=xxxx   ##在配置到hdfs上的hive-site.xml中配置即可

spark源码修改

1、spark-hive

在这里插入图片描述

修改源码后重新编译spark(我是上传linux服务器，在服务器编译完成)，可以只编译spark-sql模块

./dev/make-distribution.sh --name "hadoop3-with-hive" --tgz "-Pyarn,hadoop-3.1,scala-2.12,parquet-provided,orc-provided,hive,hive-thriftserver" -pl :spark-sql_2.11 -Dhadoop.version=3.1.4 -Dscala.version=2.12.13 -Dscala.binary.version=2.12

编译完成后的jar文件是spark-hive_2.11-2.4.7.jar,不确跟最终的spark-hive_2.12-2.4.7.jar是否一样的，为了保险我采用了替换spark-hive_2.12-2.4.7.jar中相关class文件的方式，复制之前编译好的spark-hive_2.12-2.4.7.jar到目录spark-2.4.7_resource/sql/hive/target/scala-2.12/classes下，通过命令替换class文件。举例替换Shim相关class

 jar uvf spark-hive_2.12-2.4.7.jar org/apache/spark/sql/hive/client/Shim*

用新的jar，替换上面3.3处zip中的spark-hive_2.12-2.4.7.jar ，并重新上传hdfs

2、hive-exec-1.2.1.spark2.jar 替换

原来spark依赖的hive-exec存在和hive高版本不兼容的问题，使用新版本替换zip中的jar，并重新上传hdfs

遇到的问题

2021-04-22 15:22:35,320 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
        at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
        at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1128)
        at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:867)
        at com.ucar.dspider.plugin.sql.action.SqlActionMain.main(SqlActionMain.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:685)
Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.4
        at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
        at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
        at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
        at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
        ... 12 more
2021-04-22 15:22:35,324 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: User class threw exception: java.lang.ExceptionInInitializerError
        at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
        at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1128)
        at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:867)
        at com.ucar.dspider.plugin.sql.action.SqlActionMain.main(SqlActionMain.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:685)
Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.4
        at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
        at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
        at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
        at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
        ... 12 more
)
2021-04-22 15:22:35,333 ERROR yarn.ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
        at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:470)
        at org.apache.spark.deploy.yarn.ApplicationMaster.runImpl(ApplicationMaster.scala:305)
        at org.apache.spark.deploy.yarn.ApplicationMaster.$anonfun$run$1(ApplicationMaster.scala:245)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:780)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
        at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:779)
        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:245)
        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)
        at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.util.concurrent.ExecutionException: Boxed Error
        at scala.concurrent.impl.Promise$.resolver(Promise.scala:87)
        at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:79)
        at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
        at scala.concurrent.Promise.tryFailure(Promise.scala:112)
        at scala.concurrent.Promise.tryFailure$(Promise.scala:112)
        at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:187)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:704)
Caused by: java.lang.ExceptionInInitializerError
        at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
        at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1128)
        at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:867)
        at com.ucar.dspider.plugin.sql.action.SqlActionMain.main(SqlActionMain.java:73)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:685)
Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.4
        at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
        at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
        at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
        at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
        ... 12 more

2021-04-22 15:28:56,651 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
        at org.apache.spark.sql.hive.HiveMetastoreCatalog.<init>(HiveMetastoreCatalog.scala:41)
        at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:56)
        at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
        at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:69)
        at org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:69)
        at org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$build$2(BaseSessionStateBuilder.scala:293)
        at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:79)
        at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:79)
        at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:58)
        at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:56)
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:48)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
        at com.ucar.dspider.plugin.sql.task.SparkSqlExecutorTask.doTask(SparkSqlExecutorTask.java:77)
        at com.ucar.dspider.plugin.sql.task.AbstractSyncTask.call(AbstractSyncTask.java:38)
        at com.ucar.dspider.plugin.sql.action.SqlActionMain.executeSql(SqlActionMain.java:104)
        at com.ucar.dspider.plugin.sql.action.SqlActionMain.main(SqlActionMain.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:685)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging$class
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 22 more