spark reduce与reduceByKey的区别

• reduce针对集合是单个元素的,reduceByKey针对有key的
• reduce是行动算子,reduceByKey的转换算子

案例

reduce

	val spark: SparkSession = SparkSession.builder().master("local").getOrCreate()
    val sc: SparkContext = spark.sparkContext
    val rdd: RDD[Int] = sc.parallelize(Array(1, 2, 3))
    println(rdd.reduce(_ + _))//6

val spark: SparkSession = SparkSession.builder().master("local").getOrCreate()
    val sc: SparkContext = spark.sparkContext
    val rdd: RDD[Int] = sc.parallelize(Array(1, 1, 2, 3))
    //比如,希望求所有key的和
    val tuple: (Int, Int) = rdd.map((_, 1)).reduce((x: (Int, Int), y: (Int, Int)) => {
      (x._1 + y._1, 1)
    })
    println(tuple) //(7,1)

reduceByKey

 val spark: SparkSession = SparkSession.builder().master("local").getOrCreate()
    val sc: SparkContext = spark.sparkContext
    val rdd: RDD[Int] = sc.parallelize(Array(1, 1, 2, 3))
    //对相同key的进行聚合,即key一样的计算其value的和
    val value: RDD[(Int, Int)] = rdd.map((_, 1)).reduceByKey(_ + _)
    println(value.collect().mkString("Array(", ", ", ")"))//Array((1,2), (3,1), (2,1))

参考

Spark-reduce和reduceByKey_lisery的博客-优快云博客