spark在yarn集群上执行client模式代码 Container killed on request. Exit code is 143 Container exited with a no

最新推荐文章于 2023-11-30 18:10:30 发布

佳钰小甜心

最新推荐文章于 2023-11-30 18:10:30 发布

阅读量1.1k

点赞数

分类专栏：问题解决方案文章标签：大数据 zookeeper hadoop spark scala

本文链接：https://blog.youkuaiyun.com/weixin_45401713/article/details/114852000

版权

在YARN集群上使用Spark client模式执行代码时遇到Container killed问题，退出代码为143。错误日志显示应用失败。通过检查Spark、YARN和Hadoop的相关配置，发现需要修改/hadoop/etc/hadoop/yarn-site.xml，并重新分发和重启Hadoop集群以解决问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark在yarn集群上执行client模式代码
启动历史服务

bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
./examples/jars/spark-examples_2.12-3.0.0.jar \
10

出现问题：
[2021-03-15 20:36:42.553]Container killed on request. Exit code is 143
[2021-03-15 20:36:42.554]Container exited with a non-zero exit code 143.
For more detailed output, check the application tracking page: http://hadoop103:8088/clusterpp/application_1615792406203_0009 Then click on links to logs of each attempt.
. Failing the application.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(rnClientSchedulerBackend.scala:95)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchulerBackend.scala:62)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:201)
at org.apache.spark.SparkContext.(SparkContext.scala:550)
at org.apache.spark.SparkContext $. g e t O r C r e a t e (S p a r k C o n t e x t . s c a l a : 2555) a t o r g . a p a c h e . s p a r k . s q l . S p a r k S e s s i o n$ Builder. $a n o n f u n$ getOrCreate $1 (S p a r k S e s s i o n . s c a : 930) a t s c a l a . O p t i o n . g e t O r E l s e (O p t i o n . s c a l a : 189) a t o r g . a p a c h e . s p a$

最低0.47元/天解锁文章