spark shuffle 2 map和reduce

最新推荐文章于 2024-10-25 17:20:04 发布

better_mouse

最新推荐文章于 2024-10-25 17:20:04 发布

阅读量291

点赞数

分类专栏： spark源码

本文链接：https://blog.youkuaiyun.com/better_mouse/article/details/92787558

版权

spark源码专栏收录该内容

12 篇文章

订阅专栏

spark shuffle

shuffle分为map阶段和reducer阶段

map阶段

ShuffleManager

在driver和executor的sparkEnv中被创建.基于spark.shuffle.manager的设置.driver用它注册shuffle,executor(或在driver本地运行的任务)可以请求来读或写数据

/**
 * Pluggable interface for shuffle systems. A ShuffleManager is created in SparkEnv on the driver
 * and on each executor, based on the spark.shuffle.manager setting. The driver registers shuffles
 * with it, and executors (or tasks running locally in the driver) can ask to read and write data.
 *
  *用于shuffle systems的可插拔接口.
  *在driver和executor的sparkEnv中被创建.基于spark.shuffle.manager的设置.driver用它注册shuffle
  * executor(或在driver本地运行的任务)可以请求来读或写数据
  *
  *
 * NOTE: this will be instantiated by SparkEnv so its constructor can take a SparkConf and
 * boolean isDriver as parameters.
  *
  * 注意这将被sparkEnv 实例化,所以它的构造可以将sparkCOnf和boolean inDriver作为参数
 */
private[spark] trait ShuffleManager {

  /**
   * Register a shuffle with the manager and obtain a handle for it to pass to tasks.
    * 用这个manager注册一个shuffle,为它获取一个handle来传递给任务
   */
  def registerShuffle[K, V, C](
      shuffleId: Int,
      numMaps: Int,
      dependency: ShuffleDependency[K, V, C]): ShuffleHandle

  /** Get a writer for a given partition. Called on executors by map tasks.
    * 对于给定的分区得到一个writer,被mapTask在executor上调用 */
  def getWriter[K, V](handle: ShuffleHandle, mapId: Int, context: TaskContext): ShuffleWriter[K, V]

  /**
   * Get a reader for a range of reduce partitions (startPartition to endPartition-1, inclusive).
   * Called on executors by reduce tasks.
    *得到一个关于范围reduce分区的reader(startPartition to endPartition-1包含)
    * 在executor的reduce任务调用
   */
  def getReader[K, C](
      handle: ShuffleHandle,
      startPartition: Int,
      endPartition: Int,
      context: TaskContext): ShuffleReader[K, C]

}

SortShuffleManager

在spark2.0.0以后,使用的都是SortShuffleManager


    // Let the user specify short names for shuffle managers
    val shortShuffleMgrNames = Map(
      "sort" -> classOf[org.apache.spark.shuffle.sort.SortShuffleManager].getName,
      "tungsten-sort" -> classOf[org.apache.spark.shuffle.sort.SortShuffleManager].getName)
    val shuffleMgrName = conf.get("spark.shuffle.manager", "sort")
    val shuffleMgrClass =
      shortShuffleMgrNames.getOrElse(shuffleMgrName.toLowerCase(Locale.ROOT), shuffleMgrName)
    val shuffleManager = instantiateClass[ShuffleManager](shuffleMgrClass)

SortShuffleManager registerShuffle

在这里插入图片描述

sort shuffle manager根据不同的内容选择不同的方式.

SortShuffleManager getWriter[K, V]

shuffle write 用在map阶段在这里插入图片描述

BypassMergeSortShuffleWriter

每个reduce分区写一个文件,最后合并起来,在没有ordering,没有Aggregator的时候使用

SortShuffleWriter

利用上面讲的ExternalSorter进行使用.外部排序.

UnsafeShuffleWriter

reduce

SortShuffleManager getReader

Get a reader for a range of reduce partitions (startPartition to endPartition-1, inclusive). Called on executors by reduce tasks.
得到reduce分区的reader 被executor的reduce task 调用.

在这里插入图片描述

在这里插入图片描述
这里可以发现spark先做聚合,再做排序.也就是说它想避免掉排序的性能消耗.而mapreduce 在reduce端是先排序然后在做聚合.感觉如果是有排序的话,

reduce阶段

在这里插入图片描述

      val manager = SparkEnv.get.shuffleManager
      writer = manager.getWriter[Any, Any](dep.shuffleHandle, partitionId, context)
      writer.write(rdd.iterator(partition, context).asInstanceOf[Iterator[_ <: Product2[Any, Any]]])
      writer.stop(success = true).get

getWriter 方法是在mapTask中执行的,
getReader方法也是在mapTask中执行的.
这里就不像hadoop 的map reduce明显的分了两个阶段.可以想一下是为啥.