spark 笔记 8: Stage

最新推荐文章于 2025-09-10 15:58:35 发布

转载最新推荐文章于 2025-09-10 15:58:35 发布 · 114 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/zwCHAN/p/4246918.html

文章标签：

#大数据

本文介绍了Apache Spark中Stage的概念及其重要属性。Stage是Spark作业的一部分，由一组独立任务组成，这些任务执行相同的功能并拥有相同的shuffle依赖。文章还详细解释了Stage的划分依据、跟踪信息及与Job的关系。

spark 笔记 8: Stage

Stage 是一组独立的任务，他们在一个job中执行相同的功能（function），功能的划分是以shuffle为边界的。DAG调度器以拓扑顺序执行同一个Stage中的task。

/**
 * A stage is a set of independent tasks all computing the same function that need to run as part
 * of a Spark job, where all the tasks have the same shuffle dependencies. Each DAG of tasks run
 * by the scheduler is split up into stages at the boundaries where shuffle occurs, and then the
 * DAGScheduler runs these stages in .
 *
 * Each Stage can either be , or a . For shuffle map stages, we also track the nodes
 * that each output partition is on.
 *
 * Each Stage also has a .  When FIFO
 * scheduling is used, this allows Stages from earlier jobs to be computed first or recovered
 * faster on failure.
 *
 * The . For a shuffle map
 * stage, the callSite gives the user code that created the RDD being shuffled. For a result
 * stage, the callSite gives the user code that executes the associated action (e.g. count()).
 *
 * A single stage can consist of multiple attempts. In that case, the latestInfo field will
 * be updated for each attempt.
 *
 */
privateclass val Int,
    val ,
    val Int,
    val , , ,  // Output shuffle if stage is a map stage
    val List,
    val Int,
    val extends

重要属性：

val isShuffleMap = shuffleDep.isDefined
val numPartitions = rdd.partitions.size
val outputLocs = Array.fill[List[MapStatus]](numPartitions)(Nil)
var numAvailableOutputs = 0

/** Set of jobs that this stage belongs to. */
val jobIds = new HashSet[Int]

/** For stages that are the final (consists of only ResultTasks), link to the ActiveJob. */
var resultOfJob: Option[ActiveJob] = None
var pendingTasks = new HashSet[Task[_]]

def addOutputLoc(partition: Int, status: MapStatus) {

/**
 * Result returned by a ShuffleMapTask to a scheduler. Includes the block manager address that the
 * task ran on as well as the sizes of outputs for each reducer, for passing on to the reduce tasks.
 * The map output sizes are compressed using MapOutputTracker.compressSize.
 */
private[spark] class MapStatus(var location: BlockManagerId, var compressedSizes: Array[Byte])