SPARK SHUFFLE中 ShuffleId BlockManagerId 以及 与ESS（External Shuffle Server）交互

背景

本文基于SPARK 3.2.1
用来更好的理解spark shuffle中的点点滴滴

分析

1. 我们直接从SortShuffleManager着手，因为这是个shuffle的纽带：

override def registerShuffle[K, V, C](
     shuffleId: Int,
      dependency: ShuffleDependency[K, V, C]): ShuffleHandle {
...

 override def getReader[K, C](
      handle: ShuffleHandle,
      startMapIndex: Int,
      endMapIndex: Int,
      startPartition: Int,
      endPartition: Int,
      context: TaskContext,
      metrics: ShuffleReadMetricsReporter): ShuffleReader[K, C] = {
    ...

 override def getWriter[K, V](
      handle: ShuffleHandle,
      mapId: Long,
      context: TaskContext,
      metrics: ShuffleWriteMetricsReporter): ShuffleWriter[K, V] = {
...

跟shuffle紧密关联的是这三个方法，

• 其中registerShuffle方法是在ShuffleDependency实例构建出来的时候机会被调用:

 val shuffleId: Int = _rdd.context.newShuffleId()

  val shuffleHandle: ShuffleHandle = _rdd.c