Spark修炼之道（高级篇）——Spark源码阅读：第七节 resourceOffers方法与launchTasks方法解析

zhouzhihubeyond

于 2015-10-28 21:38:42 发布

阅读量5.3k

点赞数 3

CC 4.0 BY-SA版权

分类专栏： Spark Spark修炼之道文章标签： spark 源码解析

本文链接：https://blog.youkuaiyun.com/lovehuangjiaju/article/details/49474521

本文深入探讨Spark高级篇，重点解析TaskSchedulerImpl的resourceOffers方法和CoarseGrainedSchedulerBackend的launchTasks方法。这两个方法在Executor任务提交中起关键作用，通过resourceOffers分配资源，然后由launchTasks在Worker节点上启动并运行任务。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

在上一节中，我们提到Task提交通过makeOffers提交到Executor上

    // Make fake resource offers on just one executor
    private def makeOffers(executorId: String) {
      // Filter out executors under killing
      if (!executorsPendingToRemove.contains(executorId)) {
        val executorData = executorDataMap(executorId)
        val workOffers = Seq(
          new WorkerOffer(executorId, executorData.executorHost, executorData.freeCores))
        launchTasks(scheduler.resourceOffers(workOffers))
      }
    }

上面的代码依赖于两个重要方法，它们分别是TaskSchedulerImpl resourceOffers方法及CoarseGrainedSchedulerBackend的launchTasks方法

//TaskSchedulerImpl resourceOffers方法
 /**
   * Called by cluster manager to offer resources on slaves. We respond by asking our active task
   * sets for tasks in order of priority. We fill each node with tasks in a round-robin manner so
   * that tasks are balanced across the cluster.
   */
  def resourceOffers(offers: Seq[Work