Spark源码分析（五）：Worker工作原理

最新推荐文章于 2024-01-04 17:06:35 发布

lxlneversettle

最新推荐文章于 2024-01-04 17:06:35 发布

阅读量304

点赞数

CC 4.0 BY-SA版权

分类专栏： spark core 文章标签： spark

本文链接：https://blog.youkuaiyun.com/lxlneversettle/article/details/88873881

Worker工作原理

在之前关于Master调度中我们知道，Master会向Worker发送启动Executor的消息
接下来看一下worker收到启动executor的消息后会怎样处理

// 启动executor
case LaunchExecutor(masterUrl, appId, execId, appDesc, cores_, memory_) =>
  if (masterUrl != activeMasterUrl) {
   
   
    logWarning("Invalid Master (" + masterUrl + ") attempted to launch executor.")
  } else {
   
   
    try {
   
   
      logInfo("Asked to launch executor %s/%d for %s".format(appId, execId, appDesc.name))

      // Create the executor's working directory
      // 创建工作目录
	  // workDir默认是SPARK_HOME/work
      val executorDir = new File(workDir, appId + "/" + execId)
      if (!executorDir.mkdirs()) {
   
   
        throw new IOException("Failed to create directory " + executorDir)
      }

      // Create local dirs for the executor. These are passed to the executor via the
      // SPARK_EXECUTOR_DIRS environment variable, and deleted by the Worker when the
      // application finishes.
      // 通过参数SPARK_EXECUTOR_DIRS来配置app的本地目录
      val appLocalDirs = appDirectories.getOrElse(appId, {
   
   
        val localRootDirs = Utils.getOrCreateLocalRootDirs(conf)
        val dirs = localRootDirs.flatMap {
   
    dir =>
          try {
   
   
            val appDir = Utils.createDirectory(dir, namePrefix = "executor")
            Utils.chmod700(appDir)
            Some(appDir.getAbsolutePath())
          } catch {
   
   
            case e: IOException =>
              logWarning(s"${e.getMessage}. Ignoring this directory.")
              None
          }
        }.toSeq
        if (dirs.isEmpty) {
   
   
          throw new IOException("No subfolder can be created in " +
            s"${localRootDirs.mkString(",")}.")
        }
        dirs
      })
      appDirectories(appId) = appLocalDirs
      // 创建一个ExecutorRunner(之前执行Driver时还有一个DriverRunner)
      val manager = new ExecutorRunner(
        appId,
        execId,
        appDesc.copy(command = Worker.maybeUpdateSSLSettings(appDesc.command, conf)),
        cores_,
        memory_,
        self,
        workerId,
        host,
        webUi.boundPort,
        publicAddress,
        sparkHome,
        executorDir,
        workerUri,
        conf,
        appLocalDirs, ExecutorState.RUNNING)
      // 加入本地缓存
      executors(appId + "/" + execId) = manager
      manager.start()
      // 更新使用资源
      coresUsed += cores_
      memoryUsed += memory_
      // 向master发送executor状态变化信息
      sendToMaster(ExecutorStateChanged(appId, execId, manager.state, None, None))
    } catch {
   
   
      case e: Exception =>
        logError(s"Failed to launch executor $appId/$execId for ${appDesc.name}.", e)
        if (executors.contains(appId + "/" + execId)) {
   
   
          executors(appId + "/" + execId).kill()
          executors -= appId + "/" + execId
        }
        sendToMaster(ExecutorStateChanged(appId,