spark submit提交任务报错Exception in thread “main“ java.lang.NoClassDefFoundError: org/apache/spark/stream

1.问题描述

提交spark任务：

bin/spark-submit --master local[2] \
--class _0924MoocProject.ImoocStatStreamingApp_product \
/opt/datas/project/scalaProjectMaven.jar \
bigdata.ibeifeng.com:2181/kafka08 test streamingtopic 1

报错：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils$
        at _0924MoocProject.ImoocStatStreamingApp_product$.main(ImoocStatStreamingApp_product.scala:33)
        at _0924MoocProject.ImoocStatStreamingApp_product.main(ImoocStatStreamingApp_product.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaUtils$
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

2.原因

打jar包的时候，因为打瘦包。所以，一些pom依赖例如KafkaUtils都没有导入jar包，所以导致这个问题。

 

3.解决

参考官网：http://spark.apache.org/docs/2.1.0/streaming-kafka-0-8-integration.html

 ./bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.1.0 ...

添加--packages，所以最后的提交任务为

bin/spark-submit --master local[2] \
--class _0924MoocProject.ImoocStatStreamingApp_product \
--packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.1.0 \
/opt/datas/project/scalaProjectMaven.jar \
bigdata.ibeifeng.com:2181/kafka08 test streamingtopic 1

（如果本文对大家有帮助，希望大家点赞~）