Spark提交作业找不到jar

最新推荐文章于 2022-03-10 10:19:26 发布

原创最新推荐文章于 2022-03-10 10:19:26 发布 · 2k 阅读

1 ·

CC 4.0 BY-SA版权

Spark 专栏收录该内容

36 篇文章

订阅专栏

在使用Spark 2.4.0版本提交作业时，遇到找不到jar文件的问题，尤其是在采用standalone-cluster模式下。问题可能源于集群运行时，未在所有节点上放置jar文件。通过在master和worker节点上放置jar文件，问题得以解决。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

楔子

Spark 提交作业找不到jar
ERROR ClientEndpoint:70 - Exception from cluster was: java.nio.file.NoSuchFileException

出错提示

[root@had4 spark240]# ./bin/spark-submit \
>   --master spark://had1:7077 \
>   --deploy-mode cluster \
>   --supervise \
>    --class cn.sxt.spark.core.WC \
>   wc.jar
2018-12-26 17:01:06 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-12-26 17:01:13 ERROR ClientEndpoint:70 - Exception from cluster was: java.nio.file.NoSuchFileException: /opt/spark240/wc.jar
java.nio.file.NoSuchFileException: /opt/spark240/wc.jar
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
	at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
	at java.nio.file.Files.copy(Files.java:1274)

环境

hadoop是基于3.1.1
spark是基于 2.4.0版本

提交

`采用hdfs存放文件不知道为何存放在本地有问题`

#采用hdfs存放文件 不知道为何存放在本地有问题 `但是后来 在master 
#节点上提交 使用本地的jar 又好用了没报错`
./bin/spark-submit \
--master spark://had1:7077 \
--deploy-mode cluster \
--class cn.sxt.spark.core.WC hdfs://had2/file/wc.jar

在这里插入图片描述

`找不到jar`

测试了2次分别如下

-----成功的提交
./bin/spark-submit   --class cn.sxt.spark.core.WC  --master spark://had1:7077     wc.jar
-- 失败的提交
./bin/spark-submit   --class cn.sxt.spark.core.WC  --master spark://had1:7077   --deploy-mode cluster  wc.jar

第一次是standalone -client 提交，结果提交成功了
第二次是 standalone-cluster提交，结果提示找不到jar了。spark master 节点和worker 节点都放置了jar文件，结果成功了。或许是因为集群运行，需要在各个节点上都放置jar文件

其他提交实例

官网提交实例参考
 hadoop一个旧版本中文文档

# Run application locally on 8 cores
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master local[8] \
  /path/to/examples.jar \
  100

# Run on a Spark standalone cluster in client deploy mode
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://207.184.161.138:7077 \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000

# Run on a Spark standalone cluster in cluster deploy mode with supervise
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://207.184.161.138:7077 \
  --deploy-mode cluster \
  --supervise \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000

# Run on a YARN cluster
export HADOOP_CONF_DIR=XXX
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode cluster \  # can be client for client mode
  --executor-memory 20G \
  --num-executors 50 \
  /path/to/examples.jar \
  1000

# Run a Python application on a Spark standalone cluster
./bin/spark-submit \
  --master spark://207.184.161.138:7077 \
  examples/src/main/python/pi.py \
  1000

# Run on a Mesos cluster in cluster deploy mode with supervise
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master mesos://207.184.161.138:7077 \
  --deploy-mode cluster \
  --supervise \
  --executor-memory 20G \
  --total-executor-cores 100 \
  http://path/to/examples.jar \
  1000

# Run on a Kubernetes cluster in cluster deploy mode
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master k8s://xx.yy.zz.ww:443 \
  --deploy-mode cluster \
  --executor-memory 20G \
  --num-executors 50 \
  http://path/to/examples.jar \
  1000