DAGScheduler之Job的提交划分Stage

整体流程图

源码分析 spark 2.3

getOrCreateParentStages 创建所有祖先Stage

/**
   * Get or create the list of parent stages for a given RDD.  The new Stages will be created with
   * the provided firstJobId.
   */
  private def getOrCreateParentStages(rdd: RDD[_], firstJobId: Int): List[Stage] = {
    // getShuffleDependencies 获取RDD的第一层直接宽依赖
    getShuffleDependencies(rdd).map { shuffleDep =>
      //getOrCreateShuffleMapStage 创建rdd对应的所有祖先Stage
      getOrCreateShuffleMapStage(shuffleDep, firstJobId)
    }.toList
  }

getShuffleDependencies 获取RDD的第一层直接宽依赖

  /**
   * Returns shuffle dependencies that are immediate parents of the given RD