spark-core_24:注册RegisterApplication，schedule()分配worker的资源,启动CoarseGrainedExecutorBackend源码分析

基于上文（spark-core_23: TaskSchedulerImpl.start()、SparkDeploySchedulerBackend.start()、CoarseGrainedExecutorBackend.start()、AppClient.start()源码分析）

11，AppClient初始化ClientEndpoint初始化时调用tryRegisterAllMasters，触发

masterRef.send(RegisterApplication(appDescription,self))：查看一下MasterRpcEndPoint,接收到RegisterApplication这个case class做了哪些工作

//receive方法接收EndpointRef send方法发送的信息
override def receive: PartialFunction[Any, Unit] ={
 。。。为了分析流程，将不相关的代码去掉
  //注册appliction,是在SparkContext初始化时启动TaskSchedulerImpl.start()之后由SparkDeploySchedulerBackend的AppClient的ClientEndpoint调用的
    //这里的description包含了app的具体信息，包括command信息,里面有启动类CoarseGrainedExecutorBackend；这里的driver是ClientEndpoint本身
  case RegisterApplication(description, driver) => {
    // TODOPrevent repeated registrations from some driver
    if (state == RecoveryState.STANDBY) {
      // ignore, don't send response
    } else{
      //description.name就 是sparkconf.setAppName的值
      logInfo("Registeringapp " + description.name)
      //driver:NettyRpcEndpointRef(spark://AppClient@192.168.1.152:60112)
      //app： ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息（app-20180404172558-0000）
      val app= createApplication(description, driver)
      //将ApplicationInfo存放到master中有关操作ApplicationInfo的成员变量中
      registerApplication(app)
      logInfo("Registered app " + description.name + " with ID " + app.id)
      //使用ZooKeeperPersistenceEngine，将信息存储到zk中
      persistenceEngine.addApplication(app)
      //ClientEndpoint发送RegisteredApplication(app.id, MasterRpcEndpointRef)
      driver.send(RegisteredApplication(app.id, self))
      schedule()
    }
  }

12,先回到AppClient内部类ClientEndpoint，因为master调用driver.send(RegisteredApplication(app.id, self))

override def receive: PartialFunction[Any, Unit] ={
  //RegisteredApplication(app-20180404172558-0000,masterRpcEndpointRef)
  case RegisteredApplication(appId_, masterRef) =>
    // FIXMEHow to handle the following cases?
    // 1. A master receives multiple registrations and sendsback multiple
    // RegisteredApplications due to anunstable network.
    // 2. Receive multipleRegisteredApplication from different masters because the master is changing.
    /**如何处理下面两种情况：
      * 1，master接收多次注册和返回多次RegisteredApplication给AppClient，因为不稳定的网络造成的
      * 2，AppClient接收多个RegisteredApplication,从不同的master过来的，因为master改变了 */
    //appId: AtomicReference[String] ;registered:AtomicBoolean(false)默认是false，注册后设置true
    appId.set(appId_)
    registered.set(true)
    //master:Option[RpcEndpointRef],将masterRpcEndpointRef引用存放到成员变量中
    master = Some(masterRef)
    //listener就是SparkDeploySchedulerBackend
    //让Semaphore(0)信号量释放一个许可，让阻塞在registrationBarrier.acquire()的线程通过
    //就是让SparkDeploySchedulerBackend.start()方法中阻塞在waitForRegistration()的线程通过
    listener.connected(appId.get)

。。。。

===》进入SparkDeploySchedulerBackend的connected方法

/** 该方法会在AppClient.start之后，向master注册RegisterApplication，注册进去之后，会回复RegisteredApplication给ClientEndpoint
  * 在receive里面会调用这个connected
  * 参数appId:  app-20180404172558-0000
  */
override def connected(appId: String) {
  logInfo("Connected to Spark cluster with app ID " + appId)
  this.appId = appId
  //registrationBarrier:Semaphore(0),通知信号量释放一个许可，让阻塞在registrationBarrier.acquire()的线程通过
  //找到的一下只有SparkDeploySchedulerBackend.start()方法调用waitForRegistration()
  notifyContext()
  //默认不会联接SocketServer
  launcherBackend.setAppId(appId)
}

===》进入notifyContext()方法

private def notifyContext() = {
  //registrationBarrier:Semaphore(0),通知信号量释放一个许可，让阻塞在registrationBarrier.acquire()的线程通过
  //找到的一下只有SparkDeploySchedulerBackend.start()方法调用waitForRegistration()
  registrationBarrier.release()
}

===》查看一下只有阻塞在waitForRegistration()的线程

override def start() {
  super.start()
  launcherBackend.connect()
。。。。
  client = new AppClient(sc.env.rpcEnv, masters, appDesc, this, conf)
  client.start()
  //默认情况下launcherBackend不起作用
  launcherBackend.setState(SparkAppHandle.State.SUBMITTED)

//只有AppClient的ClientEndpoint向master注册RegisterApplication成功之后,master发RegisteredApplication给ClientEndpoint，阻塞在waitForRegistration()线程才能通过
  waitForRegistration()
  launcherBackend.setState(SparkAppHandle.State.RUNNING)
}

13，master发向ClientEndPoint发送RegisteredApplication之后，自己调用了一下scheduler()

override def receive: PartialFunction[Any, Unit] ={
 。。。为了分析流程，将不相关的代码省略了
    case RegisterApplication(description, driver) => {
    ….     

//ClientEndpoint发送RegisteredApplication(app.id,MasterRpcEndpointRef)
      driver.send(RegisteredApplication(app.id, self))
      schedule()
    }
  }

14，scheduler（）会排可用的资源给等待的apps,每次有应用程序加入或资源变得可用时都会调用这个方法

/**
  * Schedule the currently availableresources among waiting apps. This method will be called
  * every time a new app joins or resourceavailability changes.
 */
private def schedule(): Unit= {
  if (state != RecoveryState.ALIVE) { return }
  // Drivers take strict precedence over executors
  //会将workers：HashSet[WorkerInfo]，进行重新调整顺序,
  // 这个Worker集合是在Worker启动之后向master注册RegisterWorker，然后调registerWorker()加进去的
  val shuffledWorkers= Random.shuffle(workers) //Randomization helps balance drivers
  //new WorkerInfo().state的值是WorkerState.ALIVE，waitingDrivers集合是对应Cluster模式的，client模式是没有元素的
  //WorkerInfo里面的成员值如：workerId: worker-20180321165947-luyl153-workrpc的port值, host:work的host
  //port = workrpc.address.port,workerRpcEndPoint  ， publicAddress: 当前worker的主机名
  for (worker<- shuffledWorkers if worker.state ==WorkerState.ALIVE) {
    //waitingDrivers:ArrayBuffer[DriverInfo], DriverInfo针对StandaloneRestServer,即cluster模式,StandaloneRestServer是给RestSubmissionClient
    //当前分析的是client模式，所以for不执行
    for (driver<- waitingDrivers) {
      if (worker.memoryFree>= driver.desc.mem && worker.coresFree >= driver.desc.cores) {
        launchDriver(worker, driver)
        waitingDrivers -= driver
      }
    }
  }
  //会让worker启动CoarseGrainedExecutorBackend
  startExecutorsOnWorkers()
}

15、查看startExecutorsOnWorkers()：分析每个worker是否有足够的core和memery来启动CoarseGrainedExecutorBackend进程

/**
 * Schedule and launch executors onworkers
  * 安排启动executor在worker节点上，executor就是CoarseGrainedExecutorBackend进程
 */
private def startExecutorsOnWorkers(): Unit = {
  // Right now this is a very simple FIFO scheduler. Wekeep trying to fit in the first app
  // in the queue, then the second app,etc.
  //1，刚开始启动start-all.sh时，waitingApps集合是没有元素的
  //2，waitingApps集合元素是在SparkContext初始化时启动TaskSchedulerImpl.start()之后由SparkDeploySchedulerBackend的AppClient的RpcEndPonit调用registerApplication 放进去的
  //app：ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息（app-20180404172558-0000）

  for (app<- waitingApps if app.coresLeft > 0) {
    //desc:ApplicationDescription的coresPerExecutor就是--executor-cores的值或spark.executor.cores的值，没有设置就是None
    val coresPerExecutor:Option[Int] = app.desc.coresPerExecutor
    // Filter out workers that don't have enough resources tolaunch an executor
    //过滤掉没有足够的资源来启动executor即CoarseGrainedExecutorBackend的worker节点。
    //app.desc.memoryPerExecutorMB:它的值对应sc.executorMemory默认是1024MB

    // worker.memoryFree:默认给服务器留1g，其它都拿来使用

// app.desc.memoryPerExecutorMB:默认是1024MB对应sc.executorMemory

//worker.coresFree:默认等于worker节点的cpu最大值
    val usableWorkers= workers.toArray.filter(_.state ==WorkerState.ALIVE)
      .filter(worker =>worker.memoryFree >= app.desc.memoryPerExecutorMB &&worker.coresFree >= coresPerExecutor.getOrElse(1))
      .sortBy(_.coresFree).reverse

    /**  app： ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息（app-20180404172558-0000）
      * spreadOutApps：spark.deploy.spreadOut默认是true
      *
         返回的数组：在--num-executors或SparkConf的"spark.executor.cores"设置了值情况下对应minCoresPerExecutor变量值
         如果不设置值，一个worker节点只会启动一个CoarseGrainedExecutorBackend，只是一个CoarseGrainedExecutorBackend进程会有多个core而以
         如果设置了值：返回的数组的索引对应可用usableWorkers数组，work加一个CoarseGrainedExecutorBackend，索引上元素值+=minCoresPerExecutor的值，
         worker如果有n个CoarseGrainedExecutorBackend，索引对应的值就是n*minCoresPerExecutor个数的core
      */
    val assignedCores = scheduleExecutorsOnWorkers(app, usableWorkers, spreadOutApps)

16,查看一下scheduleExecutorsOnWorkers()是按什么给worker分配一个CoarseGrainedExecutorBackend还是多个CoarseGrainedExecutorBackend进程的

* 安排CoarseGrainedExecutorBackend在worker启动，返回一个array包含一定数量的cores给每个worker
    *
    * 有两种模式来启动CoarseGrainedExecutorBackend，
    * 第一种，尽可能在每个worker节点上进行扩散（默认，基于数据本地性考虑的）；
    * 第二种则是启动部分worker。
    *
    * 每个Executor即CoarseGrainedExecutorBackend对应的core的数量是可以配制的通过--num-executors或SparkConf的"spark.executor.cores"进行显示的设置
    * 如果Worker有足够的cores或内存，可能一个worker会启动多个Executor即CoarseGrainedExecutorBackend,
    * 否则，每个CoarseGrainedExecutorBackend会得到worker中默认的可用的cores数量，在这种情况下一个worker只会启动一个CoarseGrainedExecutorBackend
    *
    *分配coresPerExecutor即--num-executors或SparkConf的"spark.executor.cores"，是很重要的，而不是采用默认情况下不设置值，哪spark.executor.cores的值就是1
    * 如：有4个节点，每个节点16个core，一共有64个core,如果用户要启动3个Executor即CoarseGrainedExecutorBackend,
    * 要设置每个worker：spark.executor.cores是16个（值是几个就会有几个CoarseGrainedExecutorBackend），总共spark.cores.max是48
    * 而不是按默认情况一次一个core
    *
    * //app： ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息（app-20180404172558-0000）
    * // spreadOutApps是boolean默认是true
    *
    *  为Application分配资源选择Worker(Executor),现在有两种策略：
    *  1.尽量打散
    *    将一个Application尽可能的分配到不同的节点，可以通过设置spark.deploy.spreadOut来实现对应方法实参spreadOutApps，默认值为true
    *  2.尽量集中
    *    将一个Application金肯呢个分配到尽可能少的节点，CPU密集型而内存占用比较少的Application比较适合这种策略
   */
  private def scheduleExecutorsOnWorkers(
      app: ApplicationInfo,
      usableWorkers: Array[WorkerInfo],
      spreadOutApps: Boolean): Array[Int] = {
    //如果--num-executors或SparkConf的"spark.executor.cores"不设置每个worker上的coresPerExecutor的值是None
    val coresPerExecutor= app.desc.coresPerExecutor
    //如果不给--num-executors或SparkConf的"spark.executor.cores"设置值，minCoresPerExecutor的默认值是1，每个worker只会启动一个CoarseGrainedExecutorBackend,
    val minCoresPerExecutor= coresPerExecutor.getOrElse(1)
    //不设置--num-executors或SparkConf的"spark.executor.cores"，这个oneExecutorPerWorker的值就是true
    val oneExecutorPerWorker= coresPerExecutor.isEmpty
    //memoryPerExecutor值来至sc.executorMemory，默认是1024MB
    val memoryPerExecutor= app.desc.memoryPerExecutorMB
    //得到可用的Worker个数
    val numUsable= usableWorkers.length
    //按可用的worker长度来设置assignedCores数组
    val assignedCores= new Array[Int](numUsable) // Number of cores to give to each worker。一定数量的cores给每个worker

    val assignedExecutors= new Array[Int](numUsable) // Number of new executors on each worker.一定数量的CoarseGrainedExecutorBackend给每个worker
    //coresLeft的默认值：Int.MaxValue，coresFree默认等于worker节点的cpu最大值 - 0，将所有可用的worker的cpu加起来
    //coresToAssign的值应该是所有worker的cpu之和
    var coresToAssign= math.min(app.coresLeft, usableWorkers.map(_.coresFree).sum)

    /** Return whether the specified worker can launch anexecutor for this app.
      * 返回指定worker是否可以启动CoarseGrainedExecutorBackend给这个app。*/
    def canLaunchExecutor(pos: Int): Boolean = {
      //coresToAssign的值应是所有worker的cpu之和,
      // 如果不给--num-executors或SparkConf的"spark.executor.cores"设置值，minCoresPerExecutor的值是1
      val keepScheduling= coresToAssign >= minCoresPerExecutor
      //coresFree默认等于worker节点的cpu最大值;minCoresPerExecutor的默认值是1
      // assignedCores：new Array[Int](numUsable)如果第一次进来，只是声明了数组的长度，所以每个元素的默认值是0
      val enoughCores= usableWorkers(pos).coresFree - assignedCores(pos) >= minCoresPerExecutor

      // If we allow multiple executors per worker, then we canalways launch new executors.
      // Otherwise, if there is alreadyan executor on this worker, just give it more cores.
      //如果允许在每个worker上启动多个CoarseGrainedExecutorBackend，这样每次启动新的CoarseGrainedExecutorBackend。
      //否则，已经在已有CoarseGrainedExecutorBackend增加更多的cores
      /**
        * 不设置--num-executors或SparkConf的"spark.executor.cores"，这个oneExecutorPerWorker的值就是true，
        * 否则为false，这样就可以一直启动新的CoarseGrainedExecutorBackend
        * assignedExecutors：如果第一次进来，只是声明了数组的长度，所以每个元素的默认值是0
        * 这样launchingNewExecutor的值就是true
        */
      val launchingNewExecutor = !oneExecutorPerWorker ||assignedExecutors(pos) == 0
      if (launchingNewExecutor){
        //第一次：0* memoryPerExecutor:1024 = 0
        val assignedMemory = assignedExecutors(pos) *memoryPerExecutor
        //coresFree默认等于worker节点的cpu最大值 - 0
        val enoughMemory = usableWorkers(pos).memoryFree -assignedMemory >= memoryPerExecutor
        //第一次app.executors的默认值new mutable.HashMap[Int, ExecutorDesc],
        // app.executorLimit的默认值是Integer.MAX_VALUE
        val underLimit = assignedExecutors.sum + app.executors.size < app.executorLimit
        //在这四个变量都为true时，canLaunchExecutor方法参数上索引对应的usableWorkers数组上的worker表示可运行CoarseGrainedExecutorBackend
        keepScheduling && enoughCores &&enoughMemory && underLimit
      } else {
        // We're adding cores to an existing executor, so no needto check memory and executor limits
        //我们将增加core到已存在的CoarseGrainedExecutorBackend，所以没有必有检查内存和CoarseGrainedExecutorBackend的限制
        keepScheduling && enoughCores
      }
    }

    // Keep launching executors until no more workers canaccommodate any  more executors, or if wehave reached this application's limits
    //一直启动CoarseGrainedExecutorBackend直到worker不能来容纳更多的CoarseGrainedExecutorBackend，或达到应用的限制
    //scala> (0 until3).filter(_!=1).foreach(println)
    //打印：0 2，所以会将不符合条件的索引去掉
    var freeWorkers= (0 until numUsable).filter(canLaunchExecutor)
    while (freeWorkers.nonEmpty){
      freeWorkers.foreach { pos =>
        var keepScheduling= true
        while (keepScheduling &&canLaunchExecutor(pos)) {
          //如果不给--num-executors或SparkConf的"spark.executor.cores"设置值，minCoresPerExecutor的默认值是1
          //coresToAssign的值应该是所有worker的cpu之和,每次自减minCoresPerExecutor
          coresToAssign -= minCoresPerExecutor
          //同时给assignedCores= new Array[Int](numUsable)对应索引上元素值，每次自增minCoresPerExecutor
          assignedCores(pos) += minCoresPerExecutor

          // If we are launching one executor per worker, thenevery iteration assigns 1 core  to theexecutor. Otherwise, every iteration assigns cores to a new executor.
          //如果我们将在每个worker上启动一个CoarseGrainedExecutorBackend，后面每次遍历都会赋一个core给CoarseGrainedExecutorBackend
          //否则，每次遍历会给新的CoarseGrainedExecutorBackend赋core
          /**
            * 从这个代码可以很清楚的看出，给--num-executors或SparkConf的"spark.executor.cores"设置值，可以用多个CoarseGrainedExecutorBackend来执行任务
            */
          if (oneExecutorPerWorker) {
            assignedExecutors(pos) = 1
          } else{
            //assignedExecutors = new Array[Int](numUsable),这个数组决定每个Worker，能启动多少个CoarseGrainedExecutorBackend
            assignedExecutors(pos) += 1
          }

          // Spreading out an application means spreading out itsexecutors across as  many workers aspossible.
          // If we are not spreading out,then we should keep scheduling executors on this worker until we use all of itsresources.
          // Otherwise, just move on tothe next worker.
          /**
            * spreadOutApps的值是master初始化时设置的对应：apache.spark.deploy.spreadOut，默认值为true
            * 代码也证明，注释说的
            * 有两种模式来启动CoarseGrainedExecutorBackend，
            * 第一种，尽可能在每个worker节点上进行扩散（默认，基于数据本地性考虑的）；
            * 第二种,如果spreadOut的值是false，则会安排多个CoarseGrainedExecutorBackend在一个worker上直到把所有资源都用完。
            */
          if (spreadOutApps) {
            keepScheduling = false //false会跳出循环
          }
        }
      }
      //在上面freeWorkers.foreach之后，再次过滤一下最开始的可用的worker是否还能再启动CoarseGrainedExecutorBackend
      //即freeWorkers回到while (freeWorkers.nonEmpty)，是否还能再次进入给assignedExecutors数组赋值（代码值得学习）
      //assignedExecutors = newArray[Int](numUsable),这个数组决定每个Worker，能启动多少个CoarseGrainedExecutorBackend
      freeWorkers = freeWorkers.filter(canLaunchExecutor)
    }
    //assignedCores = new Array[Int](numUsable)每增加一个CoarseGrainedExecutorBackend就会给该数组中索引上的元素加minCoresPerExecutor
//    返回的数组：在--num-executors或SparkConf的"spark.executor.cores"设置了值情况下对应minCoresPerExecutor变量值
//      如果不设置值，一个worker节点只会启动一个CoarseGrainedExecutorBackend，只是一个CoarseGrainedExecutorBackend进程会有多个core而以
//    如果设置了值：返回的数组的索引对应可用usableWorkers数组，work加一个CoarseGrainedExecutorBackend，索引上元素值+=minCoresPerExecutor的值，
//    worker如果有n个CoarseGrainedExecutorBackend，索引对应的值就是n*minCoresPerExecutor个数的core
    assignedCores
  }

 

17，再回到startExecutorsOnWorkers()中的allocateWorkerResourceToExecutors方法如何分配资源给CoarseGrainedExecutorBackend

private def startExecutorsOnWorkers(): Unit = {
  。。。

    val assignedCores= scheduleExecutorsOnWorkers(app, usableWorkers, spreadOutApps)

    // Now that we've decided how many cores to allocate oneach worker, let's allocate them
    //现在我们要决定好有多少个core可以分配到每个worker上，assignedCores(pos)如果等于0，
    // 说明内存或core不符合启动CoarseGrainedExecutorBackend，会将该索引过滤掉
    for (pos<- 0 until usableWorkers.length if assignedCores(pos)> 0) {
      /**
        * 方法中会launchExecutor(worker, exec),仔细看这个launchExecutor(worker: WorkerInfo, exec: ExecutorDesc)，它发送了如下两条消息：
        *worker.endpoint.send(LaunchExecutor(masterUrl, exec.application.id, exec.id,exec.application.desc, exec.cores, exec.memory))
        *exec.application.driver.send(ExecutorAdded(exec.id, worker.id, worker.hostPort,exec.cores, exec.memory))
        *
        * //app：ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息（app-20180404172558-0000）
        * //coresPerExecutor就是--executor-cores（yarn模式中使用）的值或spark.executor.cores的值，没有设置就是None
        */
      allocateWorkerResourceToExecutors(app, assignedCores(pos), coresPerExecutor, usableWorkers(pos))
    }
  }
}

===》allocateWorkerResourceToExecutors()方法通过第二个参数assignedCores除spark.executor.cores（即第三个参数）的结果就是每个worker对应的Executer

  *  分配worker的资源给一个或多个CoarseGrainedExecutorBackend
  * app： ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息（app-20180404172558-0000）
  * assignedCores:对应和第四个参数WorkerInfo对应，表示该worker可以有M个core,有多少个CoarseGrainedExecutorBackend等于M 除以 coresPerExecutor
  * coresPerExecutor就是--executor-cores的值或spark.executor.cores的值，没有设置就是None，这样一个CoarseGrainedExecutorBackend就会有多个core
 */
private def allocateWorkerResourceToExecutors(
    app: ApplicationInfo,
    assignedCores: Int,
    coresPerExecutor: Option[Int],
    worker: WorkerInfo): Unit = {
  // If the number of cores per executor is specified, wedivide the cores assigned
  // to this worker evenly among the executorswith no remainder.
  // Otherwise, we launch a singleexecutor that grabs all the assignedCores on this worker.
  //如果每个CoarseGrainedExecutorBackend被指定了--num-executors或SparkConf的"spark.executor.cores"的值，会对该worker总共的assignedCores除coresPerExecutor进行平均分（我没看实现代码，看到下面代码，果真是这样）
  //否则会给一个CoarseGrainedExecutorBackend赋assignedCores
  //可以看出如果coresPerExecutor不赋值numExecutors就是1
  val numExecutors= coresPerExecutor.map { assignedCores / _ }.getOrElse(1)
  //Option语法，getOrElse表示如果coresPerExecutor是None则取参数上的值，否则取自己
  val coresToAssign= coresPerExecutor.getOrElse(assignedCores)
  //to是一个左闭，右闭的集合，也就是有多少个CoarseGrainedExecutorBackend就会循环多少次
  for (i<- 1 to numExecutors) {
    /**
      * 返回ExecutorDesc同时会给ApplicationInfo成员（一个SparkContext对应一个ApplicationInfo）：  executors: mutable.HashMap[Int, ExecutorDesc],这个id就是executor（CoarseGrainedExecutorBackend）自增数值,
      * 就是通过newExecutorId(useID)算的，value就是ExecutorDesc
      * coresGranted： //就是对应--num-executors或SparkConf的"spark.executor.cores"的值，每调用一次addExecutor会按cores自增
      * （每个worker有多少个CoarseGrainedExecutorBackend就会addExecutor就调多少次）
      *
      * ExecutorDesc的对象成员
      * newExecutorId(useID) 给executor（CoarseGrainedExecutorBackend）赋id值，第一次是0,后面自增1
      * app： ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息（app-20180404172558-0000）
      * WorkerInfo：可用来启动的CoarseGrainedExecutorBackend的worker引用
      * cores：每个CoarseGrainedExecutorBackend可以有多少个core,如果没有--num-executors或SparkConf的"spark.executor.cores"的值，就只启动一个CoarseGrainedExecutorBackend，把所有可用的core给它
      * desc.memoryPerExecutorMB:对应sc.executorMemory，默认是1024MB
      */
    val exec = app.addExecutor(worker, coresToAssign)
    //是在SparkContext初始化时启动TaskSchedulerImpl.start()之后由SparkDeploySchedulerBackend的AppClient的RpcEndPonit调用registerApplication 放进去的
    //再调用startExecutorsOnWorkers==》allocateWorkerResourceToExecutors==》launchExecutor(worker, exec)
    launchExecutor(worker, exec)
    app.state =ApplicationState.RUNNING
  }
}

===》顺便看一下applictionaInfo.addExecutor()实现

private[master] def addExecutor(
    worker: WorkerInfo,
    cores: Int,
    useID: Option[Int] = None):ExecutorDesc = {
  /** ExecutorDesc的对象成员
    * newExecutorId(useID) 给executor（CoarseGrainedExecutorBackend）赋id值，第一次是0,后面自增1
    * app： ApplicationInfo里面包括command信息,里面有启动类CoarseGrainedExecutorBackend,appId的信息（app-20180404172558-0000）
    * WorkerInfo：可用来启动的CoarseGrainedExecutorBackend的worker引用
    * cores：每个CoarseGrainedExecutorBackend可以有多少个core,如果没有--num-executors（属于yarn的）或SparkConf的"spark.executor.cores"的值，
    *        就只启动一个CoarseGrainedExecutorBackend，把所有可用的core给它
    * desc.memoryPerExecutorMB:对应sc.executorMemory，默认是1024MB
     */
  val exec= new ExecutorDesc(newExecutorId(useID), this, worker, cores, desc.memoryPerExecutorMB)
  //
  //executors: mutable.HashMap[Int, ExecutorDesc],这个id就是executor（CoarseGrainedExecutorBackend）,就是通过newExecutorId(useID)算的
  //value就是ExecutorDesc。这两成员都是ApplicationInfo，整个应用只有一个ApplicationInfo
  executors(exec.id) = exec
  coresGranted += cores //就是对应--num-executors或SparkConf的"spark.executor.cores"的值，每调用一次addExecutor会按cores自增
  exec
}

18，再回到allocateWorkerResourceToExecutors方法调用lauchExecutor(),查看master让各个Worker启动CoarseGrainedExecutorBackend

private def allocateWorkerResourceToExecutors(
    app: ApplicationInfo,
    assignedCores: Int,
    coresPerExecutor: Option[Int],
    worker: WorkerInfo): Unit = {
  …
  for (i<- 1 to numExecutors) {

…
    launchExecutor(worker, exec)
    app.state =ApplicationState.RUNNING
  }
}

 

private def launchExecutor(worker: WorkerInfo, exec:ExecutorDesc): Unit = {
  //Launching executor app-20180508232008-0000/5 on workerworker-20180508231819-192.168.1.155-33242
  logInfo("Launchingexecutor " + exec.fullId + " on worker " + worker.id)
  /** 一个worker有多少个CoarseGrainedExecutorBackend，该方法就会被执行几次
    * 方法会给WorkerInfo赋如下值：
    * //executors：HashMap[String, ExecutorDesc]它的key是: app-20180404172558-0000/0，对应的值是这个id对应的ExecutorDesc
    * //coresUsed: 每个CoarseGrainedExecutorBackend可以有多少个core,如果没有--num-executors或SparkConf的"spark.executor.cores"的值，
         就只启动一个CoarseGrainedExecutorBackend，把所有可用的core给它
       //memoryUsed: 对应sc.executorMemory，默认是1024MB
    */
  worker.addExecutor(exec)

 ….}

===>先看一下每启动一个CoarseGrainedExecutorBackend时，对WorkerInfo的成员的操作

/**
  * 该方法是master的RegisterApplication被触发==》通过master启动CoarseGrainExecutorBackend的allocateWorkerResourceToExecutors==》launchExecutor（）调用的一个worker有多少个CoarseGrainedExecutorBackend，该方法就会被执行几次
  */
def addExecutor(exec:ExecutorDesc) {
  //executors：HashMap[String, ExecutorDesc]它的key是: app-20180404172558-0000/0，对应的值是这个id对应的ExecutorDesc
  executors(exec.fullId) = exec
  //coresUsed: 每个CoarseGrainedExecutorBackend可以有多少个core,如果没有--num-executors（yarn模式下面用的）或SparkConf的"spark.executor.cores"的值，
  //就只启动一个CoarseGrainedExecutorBackend，把所有可用的core给它
  coresUsed += exec.cores
  //memoryUsed: 对应sc.executorMemory，默认是1024MB
  memoryUsed += exec.memory
}

===》再看一下master给Worker发送LaunchExecutor,启动CoarseGrainedExecutorBackend

private def launchExecutor(worker: WorkerInfo, exec:ExecutorDesc): Unit = {
 ….
  /**
    * 第一条消息是发给worker让其launchExecutor的，worker本身是个消息通讯体，其在接受到消息LaunchExecutor(masterUrl,appId, execId, appDesc, cores_, memory_) 后，
     创建并start了一个（ExecutorRunnerManages the execution of one executor process.）
     masterUrl:spark://luyl152:7077
     ApplicationInfo.desc，里面包括command信息,里面有启动类CoarseGrainedExecutorBackend
     使用Jdk的ProcessBuilder.start()来启动CoarseGrainedExecutorBackend
     exec.application.id:app-20180404172558-0000
     exec.id: 一个自增的数值，默认从0开始
    exec.cores : --num-executors或SparkConf的"spark.executor.cores"的值
    exec.memory : 对应sc.executorMemory，默认是1024MB
    */

  worker.endpoint.send(LaunchExecutor(masterUrl,
    exec.application.id, exec.id, exec.application.desc, exec.cores, exec.memory))
  //ExecutorStateChanged消息给master通知Master executor状态的变化 :sendToMaster(ExecutorStateChanged(appId, execId, manager.state,None, None))
  exec.application.driver.send(
    ExecutorAdded(exec.id, worker.id, worker.hostPort, exec.cores, exec.memory))

  /**
    * master发送的第二条消息是发送给ClientEndpoint这个消息通讯体通知它获得了executor，ClientEndpoint在接受到
    * ExecutorAdded(id: Int, workerId:String, hostPort: String, cores: Int, memory: Int)
    * 消息后会调用listener.executorAdded(fullId,workerId, hostPort, cores, memory)，
    * 实质是调用AppClientListener的具体实现SparkDeploySchedulerBackend.executorAdded(fullId: String, workerId: String,hostPort: String, cores: Int,memory: Int),到此executor注册成功.
    */
}