在阅读本文之前,请先阅读Spark Sort Based Shuffle内存分析
Spark Shuffle Read调用栈如下:
1. org.apache.spark.rdd.ShuffledRDD#compute()
2. org.apache.spark.shuffle.ShuffleManager#getReader()
3. org.apache.spark.shuffle.hash.HashShuffleReader#read()
4. org.apache.spark.storage.ShuffleBlockFetcherIterator#initialize()
5. org.apache.spark.storage.ShuffleBlockFetcherIterator#splitLocalRemoteBlocks()
org.apache.spark.storage.ShuffleBlockFetcherIterator#sendRequest()
org.apache.spark.storage.ShuffleBlockFetcherIterator#fetchLocalBlocks()
下面是fetchLocalBlocks()方法执行时涉及到的类和对应方法:
6. org.apache.spark.storage.BlockManager#getBlockData()
org.apache.spark.shuffle.hash.ShuffleManager#shuffleBlockResolver()
ShuffleManager有两个子类,如果是HashShuffle 则对应的是org.apache.spark.shuffle.hash.HashShuffleManager#shuffleBlockResolver()方法,该方法返回的是org.apache.spark.shuffle.FileShuffleBlockResolver,再调用FileShuffleBlockResolver#getBlockData()方法返回Block数据
;如果是Sort Shuffle,则对应的是
org.apache.spark.shuffle.hash.SortShuffleManager#shuffleBlockResolver(),该方法返回的是org.apache.spark.shuffle.IndexShuffleBlockResolver,然后再调用IndexShuffleBlockResolver#getBlockData()返回Block数据。
下面是org.apache.spark.storage.ShuffleBlockFetcherIterator#sendRequest()方法执行时涉及到的类和对应方法
7.
org.apache.spark.network.shuffle.ShuffleClient#fetchBlocks
org.apache.spark.network.shuffle.ShuffleClient有两个子类,分别是ExternalShuffleClient及BlockTransferService
,其中org.apache.spark.network.shuffle.BlockTransferService又有两个子类,分别是NettyBlockTransferService和NioBlockTransferService,对应两种不同远程获取Block数据方式,Spark 1.5.2中已经将NioBlockTransferService方式设置为deprecated,在后续版本中将被移除
下面按上述调用栈对各方法进行说明,这里只讲脉络,细节后面再讨论
ShuffledRDD#compute()代码
Task执行时,调用ShuffledRDD的compute方法,其代码如下:
//org.apache.spark.rdd.ShuffledRDD#compute()
override def compute(split: Partition, context: TaskContext): Iterator[(K, C)] = {
val dep = dependencies.head.asInstanceOf[ShuffleDependency[K, V, C]]
//通过org.apache.spark.shuffle.ShuffleManager#getReader()方法
//无论是Sort Shuffle 还是 Hash Shuffle,使用的都是
//org.apache.spark.shuffle.hash.HashShuffleReader
SparkEnv.get.shuffleManager.getReader(dep.shuffleHandle, split.index, split.index + 1, context)
.read()
.asInstanceOf[Iterator[(K, C)]]
}
可以看到,其核心逻辑是通过调用ShuffleManager#getReader()方法得到HashShuffleReader对象,然后调用HashShuffleReader#read()方法完成前一Stage中ShuffleMapTask生成的Shuffle 数据的读取。需要说明的是,无论是Hash Shuffle还是Sort Shuffle,使用的都是HashShuffleReader。
HashShuffleReader#read()
跳到HashShuffleReader#read()方法当中,其源码如下:
/** Read the combined key-values for this reduce task */
override def read(): Iterator[Product2[K, C]] = {
//创建ShuffleBlockFetcherIterator对象,在其构造函数中会调用initialize()方法
//该方法中会执行splitLocalRemoteBlocks(),确定数据的读取策略
//远程数据调用sendRequest()方法读取
//本地数据调用fetchLocalBlocks()方法读取
val blockFetcherItr = new ShuffleBlockFetcherIterator(
context,
blockManager.shuffleClient,
blockManager,
mapOutputTracker.getMapSizesByExecutorId(handle.shuffleId, startPartition),
// Note: we use getSizeAsMb when no suffix is provided for backwards compatibility
SparkEnv.get.conf.getSizeAsMb("spark.reducer.maxSizeInFlight", "48m") * 1024 * 1024)
// Wrap the streams for compressi