Spark的RDD中key-value类型RDD处理函数reduceByKey,aggregateByKey,foldBykey和combineByKey理解

reduceByKey:
让相同的key进行分区内聚合，让相同key分区间聚合，这里涉及到了分区内预聚合，所以与groupByKey区别在于，groupByKey中shuffle过程数据量不会操作，shuffle落盘文件，相同操作reduceByKey的性能要优于groupByKey

def reduceByKey(func: (V, V) => V): RDD[(K, V)] = self.withScope {
    reduceByKey(defaultPartitioner(self), func)
  }

 /**
   * Merge the values for each key using an associative and commutative reduce function. This will
   * also perform the merging locally on each mapper before sending results to a reducer, similarly
   * to a "combiner" in MapReduce.
   */
  def reduceByKey(partitioner: Partitioner, func: (V, V) => V): RDD[(K, V)] = self.withScope {
    combineByKeyWithClassTag[V]((v: V) => v, func, func, partitioner) //分区内和分区间进行同样的计算
  }

aggregateByKey：
每个分区内key的第一个value与初始值ZoreVaue进行分区内计算
分区内进行计算
分区间进行计算
分区内计算规则与分区间计算不同

/**
  * Aggregate the values of each key, using given combine functions and a neutral "zero value".
  * This function can return a different result type, U, than the type of the values in this RDD,
  * V. Thus, we need one operation for merging a V into a U and one operation for merging two U's,
  * as in scala.TraversableOnce. The former operation is used for merging values within a
  * partition, and the latter is used for merging values between partitions. To avoid memory
  * allocation, both of these functions are allowed to modify and return their first argument
  * instead of creating a new U.
  */
 def aggregateByKey[U: ClassTag](zeroValue: U, partitioner: Partitioner)(seqOp: (U, V) => U,
     combOp: (U, U) => U): RDD[(K, U)] = self.withScope {
   // Serialize the zero value to a byte array so that we can get a new clone of it on each key
   val zeroBuffer = SparkEnv.get.serializer.newInstance().serialize(zeroValue)
   val zeroArray = new Array[Byte](zeroBuffer.limit)
   zeroBuffer.get(zeroArray)

   lazy val cachedSerializer = SparkEnv.get.serializer.newInstance()
   val createZero = () => cachedSerializer.deserialize[U](ByteBuffer.wrap(zeroArray))

   // We will clean the combiner closure later in `combineByKey`
   val cleanedSeqOp = self.context.clean(seqOp)
   combineByKeyWithClassTag[U]((v: V) => cleanedSeqOp(createZero(), v), //分区内第一个参数与zorevalue计算分区内计算
     cleanedSeqOp, combOp, partitioner) //
 }

foldBykey:
与aggregateByKey类似，只不过foldBykey分区内和分区间的计算规则是一样的

def foldByKey(
      zeroValue: V,
      partitioner: Partitioner)(func: (V, V) => V): RDD[(K, V)] = self.withScope {
    // Serialize the zero value to a byte array so that we can get a new clone of it on each key
    val zeroBuffer = SparkEnv.get.serializer.newInstance().serialize(zeroValue)
    val zeroArray = new Array[Byte](zeroBuffer.limit)
    zeroBuffer.get(zeroArray)

    // When deserializing, use a lazy val to create just one instance of the serializer per task
    lazy val cachedSerializer = SparkEnv.get.serializer.newInstance()
    val createZero = () => cachedSerializer.deserialize[V](ByteBuffer.wrap(zeroArray))

    val cleanedFunc = self.context.clean(func)
    combineByKeyWithClassTag[V]((v: V) => cleanedFunc(createZero(), v),
      cleanedFunc, cleanedFunc, partitioner)  //分区内和分区间计算规则一致
  }

combineByKey:
combineByKey共有三个参数：
第一个参数表示，把每个分区内的每个key的第一个value进行转化结构
第二个参数表示，每个分区内第一个参数进行 转化后与分区内第二参数进行的计算规则
第三个参数表示，分区间数据的计算规则

 /**
   * Simplified version of combineByKeyWithClassTag that hash-partitions the resulting RDD using the
   * existing partitioner/parallelism level. This method is here for backward compatibility. It
   * does not provide combiner classtag information to the shuffle.
   *
   * @see `combineByKeyWithClassTag`
   */
  def combineByKey[C](
      createCombiner: V => C,//分区内第一个value结构转化
      mergeValue: (C, V) => C,//分区内计算
      mergeCombiners: (C, C) => C): RDD[(K, C)] = self.withScope {
    combineByKeyWithClassTag(createCombiner, mergeValue, mergeCombiners)(null)
  }

从上面可知，上面的key-value类型函数底层都是通过调用combineByKeyWithClassTag 来实现，只不过分区内和分区间的计算规则不同，以及初始值与分区内第一个value的计算方式，规则