spark core源码分析5 spark提交框架

最新推荐文章于 2022-03-14 14:31:42 发布

shark.zyq

最新推荐文章于 2022-03-14 14:31:42 发布

阅读量1k

点赞数

本文链接：https://blog.youkuaiyun.com/yueqian_zhu/article/details/47976903

版权

spark源码解析同时被 2 个专栏收录

21 篇文章

订阅专栏

spark 源码分析

17 篇文章

订阅专栏

本文解析了Spark Submit的主要流程，详细介绍了如何通过Spark Submit解析参数，包括main class的配置、Executor和Driver的相关设置等。此外，还讲解了不同部署模式下childMainClass的选择逻辑。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

博客地址: http://blog.youkuaiyun.com/yueqian_zhu/

源码位置：

org.apache.spark.deploy.SparkSubmit.SparkSubmit.scala

这里解析参数，包括提交jar包的mainclass，Executor、Driver的相关配置等等等等。。

def main(args: Array[String]): Unit = {
  val appArgs = new SparkSubmitArguments(args)
  if (appArgs.verbose) {
    printStream.println(appArgs)
  }
  appArgs.action match {
    case SparkSubmitAction.SUBMIT => submit(appArgs)//默认走submit分支
    case SparkSubmitAction.KILL => kill(appArgs)
    case SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)
  }
}

看提交流程

private def submit(args: SparkSubmitArguments): Unit = {
  //这个方法很长，不细讲了，主要是根据参数来设置必要的运行信息
  //需要注意的是，childMainClass是根据部署的模式来区分的
  //deployMode == CLIENT,childMainClass直接取jar包中的mainclass
  //deployMode == cluster且是部署为standalone模式，childMainClass = "org.apache.spark.deploy.Client"
  //部署为Yarn模式，childMainClass = "org.apache.spark.deploy.yarn.Client"
  val (childArgs, childClasspath, sysProps, childMainClass) = prepareSubmitEnvironment(args)

  def doRunMain(): Unit = {
    if (args.proxyUser != null) {
      val proxyUser = UserGroupInformation.createProxyUser(args.proxyUser,
        UserGroupInformation.getCurrentUser())
      try {
        proxyUser.doAs(new PrivilegedExceptionAction[Unit]() {
          override def run(): Unit = {
            runMain(childArgs, childClasspath, sysProps, childMainClass, args.verbose)
          }
        })
      } catch {
        case e: Exception =>
          // Hadoop's AuthorizationException suppresses the exception's stack trace, which
          // makes the message printed to the output by the JVM not very helpful. Instead,
          // detect exceptions with empty stack traces here, and treat them differently.
          if (e.getStackTrace().length == 0) {
            printStream.println(s"ERROR: ${e.getClass().getName()}: ${e.getMessage()}")
            exitFn()
          } else {
            throw e
          }
      }
    } else {
      //一般默认情况下走这里，即运行childMainClass中的main方法
      runMain(childArgs, childClasspath, sysProps, childMainClass, args.verbose)
    }
  }

   // In standalone cluster mode, there are two submission gateways:
   //   (1) The traditional Akka gateway using o.a.s.deploy.Client as a wrapper
   //   (2) The new REST-based gateway introduced in Spark 1.3
   // The latter is the default behavior as of Spark 1.3, but Spark submit will fail over
   // to use the legacy gateway if the master endpoint turns out to be not a REST server.
  if (args.isStandaloneCluster && args.useRest) {
    try {
      printStream.println("Running Spark using the REST application submission protocol.")
      doRunMain()
    } catch {
      // Fail over to use the legacy submission gateway
      case e: SubmitRestConnectionException =>
        printWarning(s"Master endpoint ${args.master} was not a REST server. " +
          "Falling back to legacy submission gateway instead.")
        args.useRest = false
        submit(args)
    }
  // In all other modes, just run the main class as prepared
  } else {
    doRunMain()
  }
}