运行spark问题：Initial job has not accepted any resources; check your cluster UI to ensure that workers a

最新推荐文章于 2023-05-25 10:04:56 发布

coffeebreak

最新推荐文章于 2023-05-25 10:04:56 发布

阅读量2.8k

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/coffeebreak/article/details/77725555

本文记录了一次使用Spark提交任务时遇到的资源未被接受的问题，并详细解释了问题的原因及解决办法。通过调整executor-memory参数的大小，最终解决了任务无法启动的问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

运行 spark-submit --master spark://master:7077 --executor-memory 3000g --py-files SparkUtil.py Spark_ModelMatch_eigen.py

出现如下警告，spark无法运行一直处于等待状态

RN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
17/08/30 20:56:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
17/08/30 20:56:56 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
17/08/30 20:57:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

从警告信息翻译如下：
初始化job时没有获取到任何资源；提示检查集群，确保workers可以被注册并有足够的内存资源。

集群应该没问题，之前运行过一个程序，

改成 30G 就行原来--executor-memory 3000g 是每个执行器运行的内存数我理解成总运行的内存数

spark-submit --master spark://master:7077 --executor-memory 30g --py-files SparkUtil.py Spark_ModelMatch_eigen.py

该问题解决

(TID 458, 172.17.0.35): java.io.IOException: Cannot run program "/home/hadoop/anaconda2/bin/python": error=2, No such file or directory Cannot run program "/home/hadoop/anaconda2/bin/python": error=2, No such file or directory
这个文件明明是存在的，在终端运行是能打开python。是35这台集群上没有/anaconda2/bin/python