spark集群其中一个节点没有启动成功

最新推荐文章于 2023-04-11 16:00:44 发布

原创最新推荐文章于 2023-04-11 16:00:44 发布 · 3.6k 阅读

0 ·

CC 4.0 BY-SA版权

spark 专栏收录该内容

6 篇文章

订阅专栏

在三节点Spark集群中，遇到一个worker与master在同一节点上，启动情况不稳定，有时只能启动两个或一个worker，缺乏明显规律。问题可能涉及配置、资源分配或系统状态。

Spark Command: /usr/java/jdk1.7/bin/java -cp ::/home/hadoop2/spark/spark-1.0.0/conf:/home/hadoop2/spark/spark-1.0.0/assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop2.2.0.jar -XX:MaxPermSize=128m -Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m org.apache.spark.deploy.worker.Worker spark://master:7077 --webui-port 8081
========================================

14/07/19 17:05:07 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/07/19 17:05:07 INFO SecurityManager: Changing view acls to: hadoop2
14/07/19 17:05:07 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop2)
14/07/19 17:06:11 INFO Slf4jLogger: Slf4jLogger started
14/07/19 17:06:16 INFO Remoting: Starting remoting
Exception in thread "main" java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds]
   at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
   at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
   at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
   at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
   at scala.concurrent.Await$.result(package.scala:107)
   at akka.remote.Remoting.start(Remoting.scala:173)
   at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
   at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
   at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
   at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
   at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
   at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
   at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:104)
   at org.apache.spark.deploy.worker.Worker$.startSystemAndActor(Worker.scala:386)
   at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:367)

at org.apache.spark.deploy.worker.Worker.main(Worker.scala)

我这里是三个节点，一个master和三个worker。其中有一个worker和master在同一个节点上。很奇怪的是有的时候三个worker都能正常启动，有的时候启动两个。有的时候只有一个worker。貌似也没有什么规律。