spark 提交任务error：java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/HBaseAdmin

最新推荐文章于 2024-09-04 10:58:32 发布

原创最新推荐文章于 2024-09-04 10:58:32 发布 · 5.3k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#spark #sparkstreaming #打包 #jar包

大数据开发同时被 3 个专栏收录

204 篇文章

订阅专栏

idea

45 篇文章

订阅专栏

SparkStreaming

32 篇文章

订阅专栏

博客详细记录了使用Spark提交包含HBase操作的任务时遇到的NoClassDefFoundError错误，原因是瘦包缺少HBase依赖。作者通过在spark-submit命令中加入HBase的jar包路径解决了问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.spark提交任务

bin/spark-submit --master local[2] \
--class _0924MoocProject.ImoocStatStreamingApp_product \
--packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.1.0 \
/opt/datas/project/scalaProjectMaven.jar \
bigdata.ibeifeng.com:2181/kafka08 test streamingtopic 1

报错：

ERROR JobScheduler: Error running job streaming job 1537890565000 ms.1
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 8.0 failed 1 times, most recent failure: Lost task 0.0 in stage 8.0 (TID 7, localhost, executor driver): java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/HBaseAdmin
        at _0924MoocProject.utils.HBaseUtils2.<init>(HBaseUtils2.java:26)
        at _0924MoocProject.utils.HBaseUtils2.getInstance(HBaseUtils2.java:37)
        at _0924MoocProject.dao.CourseSearchClickCountDao$.save(CourseSearchClickCountDao.scala:26)

2.原因

在idea中有hbase的pom依赖，但是打的jar瘦包里面没有！

    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>
      <version>0.98.6-hadoop2</version>
    </dependency>
   
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-server</artifactId>
      <version>0.98.6-hadoop2</version>
    </dependency>

3.解决

将需要的对应jar包写入spark submit命令

bin/spark-submit --master local[2] \
--jars $(echo /opt/modules/hbase-0.98.6-hadoop2/lib/*.jar | tr ' ' ',') \
--class _0924MoocProject.ImoocStatStreamingApp_product \
--packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.1.0 \
/opt/datas/project/scalaProjectMaven.jar \
bigdata.ibeifeng.com:2181/kafka08 test streamingtopic 1

经测试，成功！