使用deploy-mode client 时候遇到的问题

最新推荐文章于 2025-04-18 17:00:32 发布

追风的星星

最新推荐文章于 2025-04-18 17:00:32 发布

阅读量943

点赞数

分类专栏：大数据问题积累文章标签：大数据 hadoop

本文链接：https://blog.youkuaiyun.com/qq_37607026/article/details/106921895

版权

在尝试使用`spark-submit`以client模式部署到YARN时，遇到了报错。设置参数包括executor内存为20GB，executor核心数为4，executor数量为3。错误日志未给出具体解决方案，目前只停留在问题记录阶段。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark-submit \
--master yarn \
--deploy-mode client \
--executor-memory 20g \
--executor-cores 4 \
--num-executors 3 \
--class com.cpic.sparkdemo.downTest.downTxtSelect /cpic/cpicapp/lm_test/jav

报错log


Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:86)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:505)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2508)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.a