Spark通过YARN-client提交任务不成功

最新推荐文章于 2024-11-19 22:43:23 发布

原创

最新推荐文章于 2024-11-19 22:43:23 发布 · 1.3w 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#Hadoop Kerberos CDH #spark #yarn

在尝试使用Spark-1.3.1通过YARN Client提交任务到Kerberos认证的Hadoop CDH集群时，遇到应用状态持续为ACCEPTED的问题。检查发现并非由于内存资源不足，而是ResourceManager日志显示Node Manager状态为UNHEALTHY，导致可用资源为0。解决方法是删除Node Manager的临时目录并重启YARN集群，从而解决这个问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

自己用的Spark-1.3.1, 通过YARN Client提交任务访问Kerberos认证的Hadoop集群。

发现应用提交后始终出现如下循环提示：

15/03/31 09:00:45 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:46 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:47 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:48 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:49 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:50 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:51 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:52 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31