Spark PruneDependency 依赖关系 RangePartitioner

最新推荐文章于 2024-11-20 18:46:12 发布

转载最新推荐文章于 2024-11-20 18:46:12 发布 · 58 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：https://yq.aliyun.com/articles/675417

文章标签：

#大数据

本文探讨了Spark中依赖关系RangePartitioner的作用，特别是在PartitionPruningRDD与其父RDD之间的子集关系。通过实例展示了如何使用Scala处理数据，包括parallelize、sortByKey和filterByRange等操作。

Spark PruneDependency 依赖关系 RangePartitioner

Represents a dependency between the PartitionPruningRDD and its parent. In this
case, the child RDD contains a subset of partitions of the parents’.

youtub视频演示

https://youtu.be/YRQ6OaOXmPY (youtube视频)
https://www.bilibili.com/video/av37442139/?p=4 (bilibile视频)

输入数据

List(("a",2),("d",1),("b",8),("d",3)

处理程序scala


package com.opensource.bigdata.spark.local.rdd.operation.dependency.narrow.n_03_pruneDependency.n_02_filterByRange

import com.opensource.bigdata.spark.local.rdd.operation.base.BaseScalaSparkContext

object Run  extends BaseScalaSparkContext{

  def main(args: Array[String]): Unit = {

    val sc = pre()
    val rdd1 = sc.parallelize(List(("a",2),("d",1),("b",8),("d",3)),2)  //ParallelCollectionRDD
    val rdd1Sort = rdd1.sortByKey()   //ShuffleRDD
    val rdd2 =rdd1Sort.filterByRange("a","b")  //MapParttionsRDD

    println("rdd \n" + rdd2.collect().mkString("\n"))

    sc.stop()
  }

}