spark 2.1 TaskSetManager

houzhizhen

于 2017-05-24 11:27:16 发布

阅读量553

点赞数

CC 4.0 BY-SA版权

分类专栏： spark

本文链接：https://blog.youkuaiyun.com/houzhizhen/article/details/72674772

spark 专栏收录该内容

158 篇文章

订阅专栏

本文介绍Spark中TaskSetManager的作用及其实现细节。该组件负责在TaskSchedulerImpl内调度TaskSet中的任务，包括跟踪每个任务的状态、处理任务失败并尝试重试（最多重试指定次数），以及通过延迟调度实现本地性感知的调度。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

/**
 * Schedules the tasks within a single TaskSet in the TaskSchedulerImpl. This class keeps track of
 * each task, retries tasks if they fail (up to a limited number of times), and
 * handles locality-aware scheduling for this TaskSet via delay scheduling. The main interfaces
 * to it are resourceOffer, which asks the TaskSet whether it wants to run a task on one node,
 * and statusUpdate, which tells it that one of its tasks changed state (e.g. finished).
 *
 * THREADING: This class is designed to only be called from code with a lock on the
 * TaskScheduler (e.g. its event handlers). It should not be called from other threads.
 *
 * @param sched           the TaskSchedulerImpl associated with the TaskSetManager
 * @param taskSet         the TaskSet to manage scheduling for
 * @param maxTaskFailures if any particular task fails this number of times, the entire
 *                        task set will be aborted
 */
private[spark] class TaskSetManager(
    sched: TaskSchedulerImpl,
    val taskSet: TaskSet,
    val maxTaskFailures: Int,
    clock: Clock = new SystemClock()) extends Schedulable with Logging {