spark源码 RangeDependency分析

最新推荐文章于 2020-11-10 23:11:08 发布

原创最新推荐文章于 2020-11-10 23:11:08 发布 · 749 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#spark dependency

spark 专栏收录该内容

17 篇文章

订阅专栏

本文详细解析了RangeDependency类的实现原理及应用，展示了如何通过该类建立父RDD与子RDD之间的分区依赖关系，并以UnionRDD为例介绍了其在实际场景中的使用。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

/**
 * :: DeveloperApi ::
 * Represents a one-to-one dependency between ranges of partitions in the parent and child RDDs.
 * @param rdd the parent RDD
 * @param inStart the start of the range in the parent RDD
 * @param outStart the start of the range in the child RDD
 * @param length the length of the range
 */
@DeveloperApi
class RangeDependency[T](rdd: RDD[T], inStart: Int, outStart: Int, length: Int)
  extends NarrowDependency[T](rdd) {

  override def getParents(partitionId: Int): List[Int] = {
    if (partitionId >= outStart && partitionId < outStart + length) {
      List(partitionId - outStart + inStart)
    } else {
      Nil
    }
  }
}

上面的代码RangeDependency的代码，其中outStart是这个partitionId在子RDD中的开始位置，

比如

RDD1由Partition0，Partition1，Partition2组成，    RDD2由Partition0，Partition1组成

那么新组成的RDD3由Partition0，Partition1，Partition2，Partition3，Partition4组成

在这个新的RDD中，如果要求RDD3中Partition4在RDD2中的PartitionId，先判断区间

其中outStart=3，就是RDD2的Partition在RDD3中开始的位置，if(4 >= 3 && 4< 3+2)

那么返回的partitionId就是 4-3+0,这里inStart为0，具体看UnionRDD的代码，如下：

@DeveloperApi
class UnionRDD[T: ClassTag](
    sc: SparkContext,
    var rdds: Seq[RDD[T]])
  extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies

  // visible for testing
  private[spark] val isPartitionListingParallel: Boolean =
    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)

  override def getPartitions: Array[Partition] = {
    val parRDDs = if (isPartitionListingParallel) {
      val parArray = rdds.par
      parArray.tasksupport = UnionRDD.partitionEvalTaskSupport
      parArray
    } else {
      rdds
    }
    val array = new Array[Partition](parRDDs.map(_.partitions.length).seq.sum)
    var pos = 0
    for ((rdd, rddIndex) <- rdds.zipWithIndex; split <- rdd.partitions) {
      array(pos) = new UnionPartition(pos, rdd, rddIndex, split.index)
      pos += 1
    }
    array
  }

  override def getDependencies: Seq[Dependency[_]] = {
    val deps = new ArrayBuffer[Dependency[_]]
    var pos = 0
    for (rdd <- rdds) {
      deps += new RangeDependency(rdd, 0, pos, rdd.partitions.length)
      pos += rdd.partitions.length
    }
    deps
  }

  override def compute(s: Partition, context: TaskContext): Iterator[T] = {
    val part = s.asInstanceOf[UnionPartition[T]]
    parent[T](part.parentRddIndex).iterator(part.parentPartition, context)
  }

  override def getPreferredLocations(s: Partition): Seq[String] =
    s.asInstanceOf[UnionPartition[T]].preferredLocations()

  override def clearDependencies() {
    super.clearDependencies()
    rdds = null
  }
}