【Spark】提交Spark任务-ClassNotFoundException-错误处理

本文探讨了Apache Spark Streaming与Kafka的集成方法,包括直接方法(Direct Approach)的使用及其实现细节。通过具体案例分析,展示了如何利用Spark Streaming进行实时数据处理,并将结果输出到Kafka中。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

提交Spark任务-ClassNotFoundException-错误处理 

Overview - Spark 2.2.0 Documentation Spark Streaming - Spark 2.2.0 Documentation Spark Streaming + Kafka Integration Guide - Spark 2.2.0 Documentation Spark Streaming + Kafka Integration Guide (Kafka broker version 0.8.2.1 or higher) - Spark 2.2.0 Documentation Spark Streaming - Spark 2.2.0 Documentation spark/DirectKafkaWordCount.scala at v2.2.0 · apache/spark The Central Repository Search Engine spark/KafkaWordCount.scala at v2.2.0 · apache/spark Spark Standalone Mode - Spark 2.2.0 Documentation Spark Programming Guide - Spark 2.2.0 Documentation Spark Master at spark://node-01:7077 【实践】spark streaming中的广播变量应用 - 一寒惊鸿 - 优快云博客 sparkstreaming kafka_百度搜索 Spark Streaming 订单关联案例剖析-youkuaiyun.com Spark Streaming vs. Kafka Stream 哪个更适合你 Apache Kafka Apache Kafka Apache Kafka Apache Kafka Spark Streaming + Kafka 入门实例--万峥,芒果酷,永不止步! 简单之美 | Kafka+Spark Streaming+Redis实时计算整合实践 Spark Streaming和Kafka整合开发指南(一) – 过往记忆 Spark Streaming和Kafka整合开发指南(二) – 过往记忆 sparkstreaming output kafka_百度搜索 SparkStreaming采用直连方式(Direct Approach)获取Kafka数据的研究心得 - Lin_wj1995的博客 - 优快云博客 spark streaming 输出数据到kafka - 优快云博客 整合Kafka到Spark Streaming——代码示例和挑战 - stark_summer的专栏 - 优快云博客 scala - How to write to Kafka from Spark Streaming - Stack Overflow kafka-storm-starter/PooledKafkaProducerAppFactory.scala at develop · miguno/kafka-storm-starter Spark and Kafka integration patterns · allegro.tech jedispool returnresource 遭废弃 用 什么替代_百度知道 spark-stream 访问 Redis数据库示例 - 阿里云 【Redis】Java中使用Jedis操作Redis(Maven导入包)、创建Redis连接池 - 谙忆-大学之旅 - 优快云博客 ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaUtils_百度搜索 第91课:SparkStreaming基于Kafka Direct案例实战和内幕源码解密 java.lang.ClassNotFoundException 踩坑解决问题详细内幕版本 - 优快云博客 sbt - KafkaUtils class not found in Spark streaming - Stack Overflow
[root@node1 server]# hdfs dfs -ls /spark/jars/ Found 146 items -rw-r--r-- 3 hadoop supergroup 232470 2025-08-04 19:54 /spark/jars/JLargeArrays-1.5.jar -rw-r--r-- 3 hadoop supergroup 1175798 2025-08-04 19:54 /spark/jars/JTransforms-3.1.jar -rw-r--r-- 3 hadoop supergroup 325335 2025-08-04 19:54 /spark/jars/RoaringBitmap-0.7.45.jar -rw-r--r-- 3 hadoop supergroup 134044 2025-08-04 19:54 /spark/jars/aircompressor-0.10.jar -rw-r--r-- 3 hadoop supergroup 1168113 2025-08-04 19:54 /spark/jars/algebra_2.12-2.0.0-M2.jar -rw-r--r-- 3 hadoop supergroup 336803 2025-08-04 19:54 /spark/jars/antlr4-runtime-4.7.1.jar -rw-r--r-- 3 hadoop supergroup 27006 2025-08-04 19:54 /spark/jars/aopalliance-repackaged-2.6.1.jar -rw-r--r-- 3 hadoop supergroup 1194003 2025-08-04 19:54 /spark/jars/arpack_combined_all-0.1.jar -rw-r--r-- 3 hadoop supergroup 64674 2025-08-04 19:54 /spark/jars/arrow-format-0.15.1.jar -rw-r--r-- 3 hadoop supergroup 105777 2025-08-04 19:54 /spark/jars/arrow-memory-0.15.1.jar -rw-r--r-- 3 hadoop supergroup 1437215 2025-08-04 19:54 /spark/jars/arrow-vector-0.15.1.jar -rw-r--r-- 3 hadoop supergroup 176285 2025-08-04 19:54 /spark/jars/automaton-1.11-8.jar -rw-r--r-- 3 hadoop supergroup 1556863 2025-08-04 19:54 /spark/jars/avro-1.8.2.jar -rw-r--r-- 3 hadoop supergroup 132989 2025-08-04 19:54 /spark/jars/avro-ipc-1.8.2.jar -rw-r--r-- 3 hadoop supergroup 187052 2025-08-04 19:54 /spark/jars/avro-mapred-1.8.2-hadoop2.jar -rw-r--r-- 3 hadoop supergroup 134696 2025-08-04 19:54 /spark/jars/breeze-macros_2.12-1.0.jar -rw-r--r-- 3 hadoop supergroup 13826799 2025-08-04 19:54 /spark/jars/breeze_2.12-1.0.jar -rw-r--r-- 3 hadoop supergroup 3226851 2025-08-04 19:54 /spark/jars/cats-kernel_2.12-2.0.0-M4.jar -rw-r--r-- 3 hadoop supergroup 58684 2025-08-04 19:54 /spark/jars/chill-java-0.9.5.jar -rw-r--r-- 3 hadoop supergroup 211523 2025-08-04 19:54 /spark/jars/chill_2.12-0.9.5.jar -rw-r--r-- 3 hadoop supergroup 284184 2025-08-04 19:54 /spark/jars/commons-codec-1.10.jar -rw-r--r-- 3 hadoop supergroup 71626 2025-08-04 19:54 /spark/jars/commons-compiler-3.0.16.jar -rw-r--r-- 3 hadoop supergroup 365552 2025-08-04 19:54 /spark/jars/commons-compress-1.8.1.jar -rw-r--r-- 3 hadoop supergroup 134595 2025-08-04 19:54 /spark/jars/commons-crypto-1.0.0.jar -rw-r--r-- 3 hadoop supergroup 284220 2025-08-04 19:54 /spark/jars/commons-lang-2.6.jar -rw-r--r-- 3 hadoop supergroup 503880 2025-08-04 19:54 /spark/jars/commons-lang3-3.9.jar -rw-r--r-- 3 hadoop supergroup 2035066 2025-08-04 19:54 /spark/jars/commons-math3-3.4.1.jar -rw-r--r-- 3 hadoop supergroup 273370 2025-08-04 19:54 /spark/jars/commons-net-3.1.jar -rw-r--r-- 3 hadoop supergroup 197176 2025-08-04 19:54 /spark/jars/commons-text-1.6.jar -rw-r--r-- 3 hadoop supergroup 79845 2025-08-04 19:54 /spark/jars/compress-lzf-1.0.3.jar -rw-r--r-- 3 hadoop supergroup 164422 2025-08-04 19:54 /spark/jars/core-1.1.2.jar -rw-r--r-- 3 hadoop supergroup 18497 2025-08-04 19:54 /spark/jars/flatbuffers-java-1.9.0.jar -rw-r--r-- 3 hadoop supergroup 14395 2025-08-04 19:54 /spark/jars/generex-1.0.2.jar -rw-r--r-- 3 hadoop supergroup 234942 2025-08-04 19:54 /spark/jars/hive-storage-api-2.7.1.jar -rw-r--r-- 3 hadoop supergroup 200223 2025-08-04 19:54 /spark/jars/hk2-api-2.6.1.jar -rw-r--r-- 3 hadoop supergroup 203358 2025-08-04 19:54 /spark/jars/hk2-locator-2.6.1.jar -rw-r--r-- 3 hadoop supergroup 131590 2025-08-04 19:54 /spark/jars/hk2-utils-2.6.1.jar -rw-r--r-- 3 hadoop supergroup 27156 2025-08-04 19:54 /spark/jars/istack-commons-runtime-3.0.8.jar -rw-r--r-- 3 hadoop supergroup 1282424 2025-08-04 19:54 /spark/jars/ivy-2.4.0.jar -rw-r--r-- 3 hadoop supergroup 67889 2025-08-04 19:54 /spark/jars/jackson-annotations-2.10.0.jar -rw-r--r-- 3 hadoop supergroup 348635 2025-08-04 19:54 /spark/jars/jackson-core-2.10.0.jar -rw-r--r-- 3 hadoop supergroup 1400944 2025-08-04 19:54 /spark/jars/jackson-databind-2.10.0.jar -rw-r--r-- 3 hadoop supergroup 46646 2025-08-04 19:54 /spark/jars/jackson-dataformat-yaml-2.10.0.jar -rw-r--r-- 3 hadoop supergroup 105898 2025-08-04 19:54 /spark/jars/jackson-datatype-jsr310-2.10.3.jar -rw-r--r-- 3 hadoop supergroup 34991 2025-08-04 19:54 /spark/jars/jackson-module-jaxb-annotations-2.10.0.jar -rw-r--r-- 3 hadoop supergroup 43740 2025-08-04 19:54 /spark/jars/jackson-module-paranamer-2.10.0.jar -rw-r--r-- 3 hadoop supergroup 341862 2025-08-04 19:54 /spark/jars/jackson-module-scala_2.12-2.10.0.jar -rw-r--r-- 3 hadoop supergroup 44399 2025-08-04 19:54 /spark/jars/jakarta.activation-api-1.2.1.jar -rw-r--r-- 3 hadoop supergroup 25058 2025-08-04 19:54 /spark/jars/jakarta.annotation-api-1.3.5.jar -rw-r--r-- 3 hadoop supergroup 18140 2025-08-04 19:54 /spark/jars/jakarta.inject-2.6.1.jar -rw-r--r-- 3 hadoop supergroup 91930 2025-08-04 19:54 /spark/jars/jakarta.validation-api-2.0.2.jar -rw-r--r-- 3 hadoop supergroup 140376 2025-08-04 19:54 /spark/jars/jakarta.ws.rs-api-2.1.6.jar -rw-r--r-- 3 hadoop supergroup 115498 2025-08-04 19:54 /spark/jars/jakarta.xml.bind-api-2.3.2.jar -rw-r--r-- 3 hadoop supergroup 926574 2025-08-04 19:54 /spark/jars/janino-3.0.16.jar -rw-r--r-- 3 hadoop supergroup 780265 2025-08-04 19:54 /spark/jars/javassist-3.25.0-GA.jar -rw-r--r-- 3 hadoop supergroup 95806 2025-08-04 19:54 /spark/jars/javax.servlet-api-3.1.0.jar -rw-r--r-- 3 hadoop supergroup 1013367 2025-08-04 19:54 /spark/jars/jaxb-runtime-2.3.2.jar -rw-r--r-- 3 hadoop supergroup 16537 2025-08-04 19:54 /spark/jars/jcl-over-slf4j-1.7.30.jar -rw-r--r-- 3 hadoop supergroup 244502 2025-08-04 19:54 /spark/jars/jersey-client-2.30.jar -rw-r--r-- 3 hadoop supergroup 1166647 2025-08-04 19:54 /spark/jars/jersey-common-2.30.jar -rw-r--r-- 3 hadoop supergroup 32091 2025-08-04 19:54 /spark/jars/jersey-container-servlet-2.30.jar -rw-r--r-- 3 hadoop supergroup 73349 2025-08-04 19:54 /spark/jars/jersey-container-servlet-core-2.30.jar -rw-r--r-- 3 hadoop supergroup 76733 2025-08-04 19:54 /spark/jars/jersey-hk2-2.30.jar -rw-r--r-- 3 hadoop supergroup 85815 2025-08-04 19:54 /spark/jars/jersey-media-jaxb-2.30.jar -rw-r--r-- 3 hadoop supergroup 927721 2025-08-04 19:54 /spark/jars/jersey-server-2.30.jar -rw-r--r-- 3 hadoop supergroup 83632 2025-08-04 19:54 /spark/jars/json4s-ast_2.12-3.6.6.jar -rw-r--r-- 3 hadoop supergroup 482486 2025-08-04 19:54 /spark/jars/json4s-core_2.12-3.6.6.jar -rw-r--r-- 3 hadoop supergroup 36175 2025-08-04 19:54 /spark/jars/json4s-jackson_2.12-3.6.6.jar -rw-r--r-- 3 hadoop supergroup 349025 2025-08-04 19:54 /spark/jars/json4s-scalap_2.12-3.6.6.jar -rw-r--r-- 3 hadoop supergroup 33031 2025-08-04 19:54 /spark/jars/jsr305-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 4592 2025-08-04 19:54 /spark/jars/jul-to-slf4j-1.7.30.jar -rw-r--r-- 3 hadoop supergroup 410874 2025-08-04 19:54 /spark/jars/kryo-shaded-4.0.2.jar -rw-r--r-- 3 hadoop supergroup 775174 2025-08-04 19:54 /spark/jars/kubernetes-client-4.9.2.jar -rw-r--r-- 3 hadoop supergroup 11908731 2025-08-04 19:54 /spark/jars/kubernetes-model-4.9.2.jar -rw-r--r-- 3 hadoop supergroup 3954 2025-08-04 19:54 /spark/jars/kubernetes-model-common-4.9.2.jar -rw-r--r-- 3 hadoop supergroup 1045744 2025-08-04 19:54 /spark/jars/leveldbjni-all-1.8.jar -rw-r--r-- 3 hadoop supergroup 12488 2025-08-04 19:54 /spark/jars/logging-interceptor-3.12.6.jar -rw-r--r-- 3 hadoop supergroup 649950 2025-08-04 19:54 /spark/jars/lz4-java-1.7.1.jar -rw-r--r-- 3 hadoop supergroup 33786 2025-08-04 19:54 /spark/jars/machinist_2.12-0.6.8.jar -rw-r--r-- 3 hadoop supergroup 3180 2025-08-04 19:54 /spark/jars/macro-compat_2.12-1.1.1.jar -rw-r--r-- 3 hadoop supergroup 7343426 2025-08-04 19:54 /spark/jars/mesos-1.4.0-shaded-protobuf.jar -rw-r--r-- 3 hadoop supergroup 105365 2025-08-04 19:54 /spark/jars/metrics-core-4.1.1.jar -rw-r--r-- 3 hadoop supergroup 22042 2025-08-04 19:54 /spark/jars/metrics-graphite-4.1.1.jar -rw-r--r-- 3 hadoop supergroup 20889 2025-08-04 19:54 /spark/jars/metrics-jmx-4.1.1.jar -rw-r--r-- 3 hadoop supergroup 16642 2025-08-04 19:54 /spark/jars/metrics-json-4.1.1.jar -rw-r--r-- 3 hadoop supergroup 23909 2025-08-04 19:54 /spark/jars/metrics-jvm-4.1.1.jar -rw-r--r-- 3 hadoop supergroup 5711 2025-08-04 19:54 /spark/jars/minlog-1.3.0.jar -rw-r--r-- 3 hadoop supergroup 4153218 2025-08-04 19:54 /spark/jars/netty-all-4.1.47.Final.jar -rw-r--r-- 3 hadoop supergroup 54391 2025-08-04 19:54 /spark/jars/objenesis-2.5.1.jar -rw-r--r-- 3 hadoop supergroup 423175 2025-08-04 19:54 /spark/jars/okhttp-3.12.6.jar -rw-r--r-- 3 hadoop supergroup 88732 2025-08-04 19:54 /spark/jars/okio-1.15.0.jar -rw-r--r-- 3 hadoop supergroup 19827 2025-08-04 19:54 /spark/jars/opencsv-2.3.jar -rw-r--r-- 3 hadoop supergroup 813783 2025-08-04 19:54 /spark/jars/orc-core-1.5.10.jar -rw-r--r-- 3 hadoop supergroup 48122 2025-08-04 19:54 /spark/jars/orc-mapreduce-1.5.10.jar -rw-r--r-- 3 hadoop supergroup 27749 2025-08-04 19:54 /spark/jars/orc-shims-1.5.10.jar -rw-r--r-- 3 hadoop supergroup 65261 2025-08-04 19:54 /spark/jars/oro-2.0.8.jar -rw-r--r-- 3 hadoop supergroup 19479 2025-08-04 19:54 /spark/jars/osgi-resource-locator-1.0.3.jar -rw-r--r-- 3 hadoop supergroup 34654 2025-08-04 19:54 /spark/jars/paranamer-2.8.jar -rw-r--r-- 3 hadoop supergroup 1097799 2025-08-04 19:54 /spark/jars/parquet-column-1.10.1.jar -rw-r--r-- 3 hadoop supergroup 94995 2025-08-04 19:54 /spark/jars/parquet-common-1.10.1.jar -rw-r--r-- 3 hadoop supergroup 848750 2025-08-04 19:54 /spark/jars/parquet-encoding-1.10.1.jar -rw-r--r-- 3 hadoop supergroup 723203 2025-08-04 19:54 /spark/jars/parquet-format-2.4.0.jar -rw-r--r-- 3 hadoop supergroup 285732 2025-08-04 19:54 /spark/jars/parquet-hadoop-1.10.1.jar -rw-r--r-- 3 hadoop supergroup 1048171 2025-08-04 19:54 /spark/jars/parquet-jackson-1.10.1.jar -rw-r--r-- 3 hadoop supergroup 123052 2025-08-04 19:54 /spark/jars/py4j-0.10.9.jar -rw-r--r-- 3 hadoop supergroup 100431 2025-08-04 19:54 /spark/jars/pyrolite-4.30.jar -rw-r--r-- 3 hadoop supergroup 112235 2025-08-04 19:54 /spark/jars/scala-collection-compat_2.12-2.1.1.jar -rw-r--r-- 3 hadoop supergroup 10672015 2025-08-04 19:54 /spark/jars/scala-compiler-2.12.10.jar -rw-r--r-- 3 hadoop supergroup 5276900 2025-08-04 19:54 /spark/jars/scala-library-2.12.10.jar -rw-r--r-- 3 hadoop supergroup 222980 2025-08-04 19:54 /spark/jars/scala-parser-combinators_2.12-1.1.2.jar -rw-r--r-- 3 hadoop supergroup 3678534 2025-08-04 19:54 /spark/jars/scala-reflect-2.12.10.jar -rw-r--r-- 3 hadoop supergroup 556575 2025-08-04 19:54 /spark/jars/scala-xml_2.12-1.2.0.jar -rw-r--r-- 3 hadoop supergroup 3243337 2025-08-04 19:54 /spark/jars/shapeless_2.12-2.3.3.jar -rw-r--r-- 3 hadoop supergroup 4028 2025-08-04 19:54 /spark/jars/shims-0.7.45.jar -rw-r--r-- 3 hadoop supergroup 302558 2025-08-04 19:54 /spark/jars/snakeyaml-1.24.jar -rw-r--r-- 3 hadoop supergroup 1934320 2025-08-04 19:54 /spark/jars/snappy-java-1.1.7.5.jar -rw-r--r-- 3 hadoop supergroup 9387724 2025-08-04 19:54 /spark/jars/spark-catalyst_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 10167565 2025-08-04 19:54 /spark/jars/spark-core_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 430744 2025-08-04 19:54 /spark/jars/spark-graphx_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 369562 2025-08-04 19:54 /spark/jars/spark-kubernetes_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 59866 2025-08-04 19:54 /spark/jars/spark-kvstore_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 75930 2025-08-04 19:54 /spark/jars/spark-launcher_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 294928 2025-08-04 19:54 /spark/jars/spark-mesos_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 111914 2025-08-04 19:54 /spark/jars/spark-mllib-local_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 5885296 2025-08-04 19:54 /spark/jars/spark-mllib_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 2397699 2025-08-04 19:54 /spark/jars/spark-network-common_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 87027 2025-08-04 19:54 /spark/jars/spark-network-shuffle_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 52512 2025-08-04 19:54 /spark/jars/spark-repl_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 30347 2025-08-04 19:54 /spark/jars/spark-sketch_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 7118845 2025-08-04 19:54 /spark/jars/spark-sql_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 1137816 2025-08-04 19:54 /spark/jars/spark-streaming_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 9478 2025-08-04 19:54 /spark/jars/spark-tags_2.12-3.0.0-tests.jar -rw-r--r-- 3 hadoop supergroup 15149 2025-08-04 19:54 /spark/jars/spark-tags_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 51181 2025-08-04 19:54 /spark/jars/spark-unsafe_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 330670 2025-08-04 19:54 /spark/jars/spark-yarn_2.12-3.0.0.jar -rw-r--r-- 3 hadoop supergroup 79588 2025-08-04 19:54 /spark/jars/spire-macros_2.12-0.17.0-M1.jar -rw-r--r-- 3 hadoop supergroup 8261 2025-08-04 19:54 /spark/jars/spire-platform_2.12-0.17.0-M1.jar -rw-r--r-- 3 hadoop supergroup 34601 2025-08-04 19:54 /spark/jars/spire-util_2.12-0.17.0-M1.jar -rw-r--r-- 3 hadoop supergroup 7188024 2025-08-04 19:54 /spark/jars/spire_2.12-0.17.0-M1.jar -rw-r--r-- 3 hadoop supergroup 178149 2025-08-04 19:54 /spark/jars/stream-2.9.6.jar -rw-r--r-- 3 hadoop supergroup 233745 2025-08-04 19:54 /spark/jars/threeten-extra-1.5.0.jar -rw-r--r-- 3 hadoop supergroup 443231 2025-08-04 19:54 /spark/jars/univocity-parsers-2.8.3.jar -rw-r--r-- 3 hadoop supergroup 281356 2025-08-04 19:54 /spark/jars/xbean-asm7-shaded-4.15.jar -rw-r--r-- 3 hadoop supergroup 99555 2025-08-04 19:54 /spark/jars/xz-1.5.jar -rw-r--r-- 3 hadoop supergroup 35518 2025-08-04 19:54 /spark/jars/zjsonpatch-0.3.0.jar -rw-r--r-- 3 hadoop supergroup 4210625 2025-08-04 19:54 /spark/jars/zstd-jni-1.4.4-3.jar 有什么问题吗
最新发布
08-08
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值