spark coalesce

 Cities, if unrestricted, tend to coalesce into bigger and bigger conurbations.
如果不加限制，城市往往会联合发展成越来越大的集合城市。
来自柯林斯例句
3. Gas bubbles coalesce to grow as they rise.
气泡在上升中聚集成长.
来自辞典例句
4. Particles do not coalesce with other particles.
颗粒之间互不聚合.

 coalesce，只能用于减少分区数量，而且可以选择不发生shuffle
 repartiton，可以增加分区，也可以减少分区，必须会发生shuffle，相当于是进行了一次重分区操作

object TypedOperation {

  case class Employee(name: String, age: Long, depId: Long, gender: String, salary: Long)
  def main(args: Array[String]): Unit = {
    println("astron")
    val spark = SparkSession
      .builder()
      .master("local")
      .appName("star")
      .getOrCreate()
    import spark.implicits._
    // DF->DS
    val demoDf = spark.read.json("d://employee.json")
    println(demoDf)
    //demoDf.printSchema()
    val demoDs = demoDf.as[Employee]
    println(demoDf.rdd.partitions.size)
    println(demoDs.rdd.partitions.size)
    //repartion
    val demoDf1 = demoDf.repartition(10)
    val demoDs1 = demoDs.repartition(10)
    println(demoDf1.rdd.partitions.size)
    println(demoDs1.rdd.partitions.size)

    //coalesce 跟预期不一致
    val demoDf2 = demoDf1.coalesce(10)
    val demoDs2 = demoDs1.coalesce(10)
    println(demoDf2.rdd.partitions.size)
    println(demoDs2.rdd.partitions.size)

  }

}

1
1
10
10
1
1