Spark standalone模式结合虚拟机遇到的问题

最新推荐文章于 2024-06-18 19:33:35 发布

原创最新推荐文章于 2024-06-18 19:33:35 发布 · 667 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#shell #spark #虚拟机

Spark 专栏收录该内容

2 篇文章

订阅专栏

在Spark standalone模式下运行作业时遇到driver URL获取问题，表现为TaskScheduler无法接收worker回复。问题源于executor启动命令中使用的driver IP（192.168.56.1）为虚拟机网卡IP。通过检查worker工作日志，发现executor启动参数包含错误的driver IP。解决方案是启动时设置`spark.driver.host`环境变量，确保使用正确的driver IP地址。

在spark standalone模式启动结合虚拟机遇到的driver url获取问题

学习博客：http://bit1129.iteye.com/blog/2179543

在job启动后，输出如下问题：
17/12/22 10:01:21 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

定位问题：
查看worker ui中executor的stdout和stderr。stderr输出如下：
Caused by: org.apache.spark.rpc.RpcTimeoutException: Cannot receive any reply from192.168.56.1:61266 in 120 seconds. This timeout is controlled by spark.rpc.askTimeout
全部过程中并没有使用该ip，但是考虑到启动该job时并未设置任何参数，所以应该是环境变量设置问题。查看ip地址，发现192.168.56.1为虚拟机网卡ip。
然后查看worker的工作日志，也就是worker链接到master后提示的日志文件，通常在logs目录下，有如下信息：
17/12/21 17:00:55 INFO Worker: Asked to launch executor app-20171221165506-0000/5 for spark_main
17/12/21 17:00:55 INFO SecurityManager: Changing view acls to: user
17/12/21 17:00:55 INFO SecurityManager: Changing modify acls to: user
17/12/21 17:00:55 INFO SecurityManager: Changing view acls groups to:
17/12/21 17:00:55 INFO SecurityManager: Changing modify acls groups to:
17/12/21 17:00:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); groups with view permissions: Set(); users with modify permissions: Set(user); groups with modify permissions: Set()
17/12/21 17:00:55 INFO ExecutorRunner: Launch command:
"/home/user/java/jdk1.8.0_151/bin/java"
"-cp" "/home/user/spark/spark-2.2.1-bin-hadoop2.7/conf/:/home/user/spark/spark-2.2.1-bin-hadoop2.7/jars/*" "-Xmx1024M"
"-Dspark.driver.port=61266"
"org.apache.spark.executor.CoarseGrainedExecutorBackend"
"--driver-url" "spark://CoarseGrainedScheduler@192.168.56.1:61266"
"--executor-id" "5"
"--hostname" "10.16.143.60"
"--cores" "1"
"--app-id" "app-20171221165506-0000"
"--worker-url" "spark://Worker@10.16.143.60:40670"
通过launch command命令发现，该ip是当worker启动job时传递的driver ip，根据官方文档，可以手动设置driver ip。

解决办法：
通过启动时设置“spark.driver.host”解决该问题。