spark 连接 kafka 遇到的问题

最新推荐文章于 2021-02-17 05:31:01 发布

WQQ72600

最新推荐文章于 2021-02-17 05:31:01 发布

阅读量967

点赞数

CC 4.0 BY-SA版权

分类专栏： spark kafka

本文链接：https://blog.youkuaiyun.com/WQQ72600/article/details/90078488

spark 同时被 2 个专栏收录

4 篇文章

订阅专栏

kafka

1 篇文章

订阅专栏

本文记录了在使用Spark过程中，尝试连接Kafka时遇到的问题及解决方案。详细描述了从使用kafka_2.12-2.1.0.jar替换为kafka_2.11-0.8.2.0.jar的过程，探讨了不同版本间可能的兼容性问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

java.lang.NoClassDefFoundError: kafka/api/TopicMetadataRequest
        at org.apache.spark.streaming.kafka.KafkaCluster.getPartitionMetadata(KafkaCluster.scala:130)
        at org.apache.spark.streaming.kafka.KafkaCluster.getPartitions(KafkaCluster.scala:120)
        at org.apache.spark.streaming.kafka.KafkaUtils$.getFromOffsets(KafkaUtils.scala:212)
        at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:485)
        at com.company.CF_from_kudu$.main(CF_from_kudu.scala:42)
        at com.company.CF_from_kudu.main(CF_from_kudu.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635)
Caused by: java.lang.ClassNotFoundException: kafka.api.TopicMetadataRequest
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

将kafka_2.12-2.1.0.jar 替换成kafka_2.11-0.8.2.0.jar

./spark-submit --class com.company.CF_from_kudu --master yarn --deploy-mode cluster --executor-memory 6g --executor-cores 2 --jars /data1/testdata/dependencesjar/kafka_2.11-0.8.2.0.jar,/data1/testdata/dependencesjar/metrics-core-2.2.0.jar,/data1/testdata/dependencesjar/spark-streaming-kafka-0-8_2.11-2.3.0.jar /data1/testdata/executorjar/CF-1.0-SNAPSHOT.jar datanode4:9092,datanode5:9092,datanode6:9092 mytopic