5. Spark源码解析之worker启动流程解析

启动Spark的Worker，不管是执行start-all.sh还是直接在Worker节点直接启动start-slaves.sh，都会进入start-slave.sh开始，而基本都不会设置参数，所以参数都是默认的--webui-port 8081 spark://host:7077。

脚本阶段跟启动Master步骤差不多，执行流程可以参照Master：4. Spark源码解析之Master启动流程解析

start-slave.sh会调用spark-daemon.sh，spark-daemon.sh会调用spark-class，然后再调用launch.main来解析生成命令，最后返回给spark-class的exec "${CMD[@]}" 执行，执行其实才是真正启动Worker。

最终的命令是：java -cp ../conf/:../jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://host:7077

这里从Worker的开始，仍然先进入伴生对象的main方法。

 

org.apache.spark.deploy.worker.Worker

private[deploy] object Worker extends Logging {
  val SYSTEM_NAME = "sparkWorker"
  val ENDPOINT_NAME = "Worker"

  // 参数:--webui-port 8081 spark://host:7077
  def main(argStrings: Array[String]) {
    Thread.setDefaultUncaughtExceptionHandler(new SparkUncaughtExceptionHandler(
      exitOnUncaughtException = false))
    Utils.initDaemon(log)
    // 加载默认配置
    val conf = new SparkConf
    // 加载参数和配置进行解析,得到启动RPC和worker需要的参数
    // worker-host、worker-port、webUI-port、cores、memory、master、workerDir、properties
    // 这个类在下面另外进行解读
    val args = new WorkerArguments(argStrings, conf)
    // 启动RPC和worker终端
    val rpcEnv = startRpcEnvAndEndpoint(args.host, args.port, args.webUiPort, args.cores,
      args.memory, args.masters, args.workDir, conf = conf)
    // With external shuffle service enabled, if we request to launch multiple workers on one host,
    // we can only successfully launch the first worker and the rest fails, because with the port
    // bound, we may launch no more than one external shuffle service on each host.
    // When this happens, we should give explicit reason of failure instead of fail silently. For
    // more detail see SPARK-20989.
    // 如果开启shuffle服务,只能启动一台worker,因为会绑定端口,导致其他多台worker启动失败,所以这里关闭
    val externalShuffleServiceEnabled = conf.getBoolean("spark.shuffle.service.enabled", false)
    // worker实例,默1个
    val sparkWorkerInstances = scala.sys.env.getOrElse("SPARK_WORKER_INSTANCES", "1").toInt
    // 判断shuffle