Spark Executor原理与源码解析

最新推荐文章于 2021-12-15 12:30:57 发布

发布了一场Chat

最新推荐文章于 2021-12-15 12:30:57 发布

阅读量742

点赞数

CC 4.0 BY-SA版权

分类专栏： spark深入学习文章标签： spark executor Spark executor原理 Spark Executor源码解析

本文链接：https://blog.youkuaiyun.com/u013174239/article/details/80380058

①Executor原理示意图

②Executor源码解析

CoarseGrainedExecutorBackend.scala

  override def onStart() {
    logInfo("Connecting to driver: " + driverUrl)
    rpcEnv.asyncSetupEndpointRefByURI(driverUrl).flatMap { ref =>
      // This is a very fast action so we can use "ThreadUtils.sameThread"
      // 获取driver
      driver = Some(ref)

      // 向driver发送注册executor的消息
      ref.ask[Boolean](RegisterExecutor(executorId, self, hostname, cores, extractLogUrls))
    }(ThreadUtils.sameThread).onComplete {
      // This is a very fast action so we can use "ThreadUtils.sameThread"
      case Success(msg) =>
        // Always receive `true`. Just ignore it
      case Failure(e) =>
        exitExecutor(1, s"Cannot register with driver: $driverUrl", e, notifyDriver = false)
    }(ThreadUtils.sameThread)
  }