BlueLotuss的博客

spark学习笔记—核心算子(二) distinct算子 /** * Return a new RDD containing the distinct elements in this RDD. */ def distinct(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = withScope { def removeDuplicatesInPartition(partition: Iterator[

spark学习笔记—核心算子(一) HashPartitioner的决定分区的逻辑 核心方法 def getPartition(key: Any): Int = key match { case null => 0 case _ => Utils.nonNegativeMod(key.hashCode, numPartitions) } /* Calculates 'x' modulo 'mod', takes to consideration sign of x,