实践数据湖iceberg 第十五课 spark3安装与集成iceberg0.13 (jersey包冲突，安装完成)

Spark与Iceberg集成实战：问题与解决

原创

已于 2022-02-22 20:17:28 修改 · 2.9k 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#spark #数据湖 #iceberg

于 2022-02-13 15:49:49 首次发布

系列文章目录

文章目录

系列文章目录
前言
1.准备安装包spark-3.2.1-bin-hadoop2.7.tgz ，解压
2.配置spark-defalult.conf
3. /etc/profile配置HADOOP_CONF_DIR
4.启动测试报错
5. 解决方法
5.1 把版本改为一致
- 5.2 降低 spark版本
- 5.3 增加个参数
6. 集成iceberg
总结

前言

根据iceberg官网提示，目前iceberg0.13版本，spark对iceberg的支持是最好的，了解iceberg的最好方法是，通过spark

虽然确定公司的架构是flink+iceberg。最快速的学习路径应该是flink+iceberg. 但有时通过曲折迂回的路径才是最快的。
有时为了快就要走一些弯路。

开始安装spark

1.准备安装包spark-3.2.1-bin-hadoop2.7.tgz ，解压

解压
配置spark-defalult.conf

2.配置spark-defalult.conf

spark.master                               yarn
spark.eventLog.enabled                   true
spark.eventLog.dir                        hdfs:///spark-history

3. /etc/profile配置HADOOP_CONF_DIR

vim /etc/profile

export HADOOP_CONF_DIR=/opt/module/hadoop/etc/hadoop

4.启动测试报错

[root@hadoop101 spark]# bin/spark-shell 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/02/13 15:05:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
  at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
  at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
  at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:175)
  at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
  at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220)
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:581)
  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690)
  at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
  ... 55 elided
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 68 more
<console>:14: error: not found: value spark
       import spark.implicits._
              ^
<console>:14: error: not found: value spark
       import spark.sql
              ^
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.1
      /_/
         
Using Scala version 2.12.15 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.

重新解压一个包测试local模式，没问题, 排除安装包有问题

5. 解决方法

5.1 把版本改为一致

参考 https://blog.youkuaiyun.com/zhanglong_4444/article/details/106097216
增加jersey的包，安装文档提示，无效

spark3.1 和Hadoop2.7 jersey的版本不一致
hadoop2.7

[root@hadoop101 jars]# ls /opt/module/hadoop/share/hadoop/common/lib/je
jersey-core-1.9.jar    jersey-json-1.9.jar    jersey-server-1.9.jar  jets3t-0.9.0.jar       jettison-1.1.jar       jetty-6.1.26.jar       jetty-util-6.1.26.jar

spark3.2

[root@hadoop101 jars]# ls jersey-*
jersey-client-2.34.jar  jersey-common-2.34.jar  jersey-container-servlet-2.34.jar  jersey-container-servlet-core-2.34.jar  jersey-core-1.19.1.jar  jersey-guice-1.19.jar  jersey-hk2-2.34.jar  jersey-server-2.34.jar

头都大了！！！

问题很简单，解决很困难

查一下社区，还是要自己编译，又要一天。。。
https://github.com/apache/spark/commit/b6f46ca29742029efea2790af7fdefbc2fcf52de

另外找个文章
https://www.jianshu.com/p/d630582c8108

5.2 降低 spark版本

为了节省一天编译包时间，重新一下一个spark2.4的包，

下面作者用的spark.2.2也有包冲突问题，看来不用降级了。
https://www.jianshu.com/p/d630582c8108

5.3 增加个参数

[root@hadoop101 conf]# spark-shell --conf spark.hadoop.yarn.timeline-service.enabled=false
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/02/13 16:15:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/02/13 16:15:03 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Spark context Web UI available at http://hadoop101:4040
Spark context available as 'sc' (master = yarn, app id = application_1642579431487_0048).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.1
      /_/
         
Using Scala version 2.12.15 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.

5.1的步骤也做了 spark_home/jars 下增加了1.9的jersey