提交spark的代码的时候出现Exception in thread "main" org.apache.SparkException:Yarn application has already end

最新推荐文章于 2025-06-27 21:32:28 发布

原创最新推荐文章于 2025-06-27 21:32:28 发布 · 7.3k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #error #sparksubmit #spark queue

spark 同时被 2 个专栏收录

5 篇文章

订阅专栏

error

3 篇文章

订阅专栏

本文介绍了在使用Apache Spark过程中遇到的任务无法提交的问题，并详细解释了两种不同模式下的错误信息及其原因。通过调整队列设置成功解决了问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

出现的具体错误是:

Exception in thread "main" org.apache.SparkException:Yarn application has already ended! It might have been killed or unable to launch application master.

我最开始运行的是client模式出现了Spark的任务无法提交的情况.

之后我更换了运行模式为cluster模式

错误信息:org.apache.hadoop.security.AccessControlException: User gd cannot submit application to queue root.default

当看到这个错误信息的时候,我恍然大悟了,原来是我的提交队列没有给定.因此默认提交的就是root.default的队列,但是这个队列我们公司是不能用的.

解决方法:在提交脚本中添加--queue $SPARK_QUEUE.

再次执行的时候就可以正常运行了.

确定要放弃本次机会？

福利倒计时

: :

立减 ¥

普通VIP年卡可用

立即使用

Hikigaya-Hachiman

关注关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
分享

复制链接

分享到 QQ

分享到新浪微博

扫一扫
举报

举报

专栏目录

spark在yarn上运行报错:Yarn application has already ended

mlin的博客

11-01

6103

spark在yarn上运行报错: Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.s...

Exception in thread “main“ org.apache.spark.SparkException: Yarn application has already ended! It m

weixin_38842096的博客

10-12

4432

spark 提交任务遇到的错误如下： 1、错误（1） Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.w

参与评论您还未登录，请先登录后发表或查看评论

JAVA下运行Spark程序报错：A master URL must be set in your configuration

热门推荐

u012893747的博客

07-14

1万+

Exception in thread "main" org.apache.spark.SparkException: Application application_1498149692663_0164 finished with failed status

Spark to yarn启动的时候报错：org.apache.spark.SparkException: Yarn application has already ended!

weixin_43681796的博客

04-13

1838

场景：Spark to yarn启动的时候报错原因：虚拟内存不够解决方案：可通过以下两种方式解决：调大虚拟内存比： yarn.nodemanager.vmem-pmem-ratio 默认2.1倍或者关闭虚拟内存检查： ...

spark streaming restart error: org.apache.spark.SparkException: Yarn application has already ended!

u012075079的专栏

09-06

771

使用 KafkaUtils.createDirectStream，hdfs 设置checkPoint目录， restart spark streaming application; then error: 17/09/06 10:30:39 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.Spark

spark源码跟踪(一)yarn-cluster模式任务提交

weixin_43172032的博客

09-20

1285

一，运行命令 bin/spark-submit \ --master yarn \ --deploy-mode cluster \ --class org.apache.spark.examples.SparkPi \ examples/jars/spark-examples_2.11-2.3.1.3.0.1.0-187.jar 二，启动脚本查看spark-submit 脚本文件，程序入口为 exec "${SPARK_HOME}"/bin/spark-class org.apache.spark

spark常见问题错误汇总

我丶怀念的的博客

08-13

5808

一.经验 1.Spark Streaming包含三种计算模式：nonstate .stateful .window 2.kafka可通过配置文件使用自带的zookeeper集群 3.Spark一切操作归根结底是对RDD的操作 4.部署Spark任务，不用拷贝整个架包，只需拷贝被修改的文件，然后在目标服务器上编译打包。 5.kafka的log.dirs不要设置成/tmp下的目录，貌似...

大数据常见错误及解决方案

lv_hulk的博客

02-16

9639

大数据常见错误及解决方案（转载） 1、用./bin/spark-shell启动spark时遇到异常：java.net.BindException: Can’t assign requested address: Service ‘sparkDriver’ failed after 16 retries! 解决方法：add export SPARK_LOCAL_IP=“127.0.0.1” to spark-env.sh 2、java Kafka producer error:ERROR kafka.util

Spark Event Log (二) 开始和结束

三丰的专栏

08-25

1423

Spark Event Log Start 上一节已经看过操作Spaek Event Log的类是org.apache.spark.scheduler.EventLoggingListener,现在来分析一下start()方法 /** * Creates the log file in the configured log directory. */ def start() { if (!fileSystem.getFileStatus(new Path(logBaseDir))

Exception in thread “main“ org.apache.spark.SparkException: When running with master ‘yarn‘ either

默主归沙的博客

03-14

3611

在执行一个spark程序的时候，明明把yarn-site.xml,core-site.xml,hdfs-site.xml都打包进去了，可是还是报错 Exception in thread "main" org.apache.spark.SparkException: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. at org...

spark-On-Yarn报错application has already ended! It might have been killed or

qq_32736999的博客

11-26

1130

如果使用spark on yarn 提交任务，一般情况，都使用cluster模式，该模式，Driver运行在集群中，其实就是运行在ApplicattionMaster这个进程成，如果该进程出现问题，yarn会重启ApplicattionMaster（Driver），SparkSubmit的功能就是为了提交任务。如果使用交换式的命令行，必须用Client模式，该模式，Driver是运行在S...

Spark in yarn：Yarn application has already ended! It might have been killed or unable ……

厚积薄发

08-11

1564

使用Spark基于yarn进行操作，发生了错误，先附上日志： 18/08/11 20:29:29 INFO yarn.Client: client token: N/A diagnostics: Application application_1533988876407_0004 failed 2 times due to AM Container ...

基于yarn执行spark程序(client模式)报错如下:Yarn application has already ended! It might have been killed or unabl

qq_44160357的博客

02-12

899

在yarn上执行spark程序, 是client模式, 然后冒出了以下错误: 20/02/12 16:08:04 ERROR spark.SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Yarn application has already ended! It might have ...

Exception in thread "main" org.apache.spark.SparkException: Task not serializable

struggling_rong的博客

09-02

5052

运行一个spark应用时报如下错误： Exception in thread &quot;main&quot; org.apache.spark.SparkException: Task not serializable at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298) at o...

spark报错： Yarn application has already ended! killed or unable to launch application master

九师兄

07-04

1564

1.背景背景参考：Spark执行任务卡死：SparkException: Failed to connect to driver! unable to launch application master 报错： Yarn application has already ended! It might have been killed or unable to launch applicatio...

【Spark】解决配置client模式时出现异常：Yarn application has already ended!

hr786250678的博客

01-17

4522

Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler...

Yarn application has already ended! It might have been killed or unable to launch application master

汐朔

11-28

1485

提交SPARK任务时报org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.异常，如下： SLF4J: See http://www.slf4j.org/codes.html#multi...

解决Exception in thread “main“ org.apache.spark.SparkException: When running with master ‘yarn‘ either

学亮编程手记

07-22

6315

Exception in thread “main” org.apache.spark.SparkException: When running with master ‘yarn’ either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment. spark-env.sh export HADOOP_CONF_DIR=/opt/bigdata/hadoop-2.9.2/etc/hadoop/ export YARN_CONF_D

spark出现Exception in thread "main" org.apache.spark.SparkException: Master must either be yarn or start with spark, mesos, k8s, or local

07-27

- *3* [Exception in thread “main“ org.apache.spark.SparkException: Yarn application has already ended! It m](https://blog.csdn.net/weixin_38842096/article/details/109024966)[target="_blank" data-...