第一种TOPK排序方式
整个排序取 TopK 的实现:
- Case:
输入:文本文件
输出:
(158,)
(28,the)
(19,to)
(18,Spark)
(17,and)
(11,Hadoop)
(10,##)
(8,you)
(8,with)
(8,for) - 算法:
首先实现wordcount,topk实现是以wordcount为基础,在分词统计完成后交换key/value,然后调用sortByKey进行排序。
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.SparkContext._
object TopK {
def main(args: Array[String]) {
if (args.length != 2) {
System.out.println("Usage: <src> <num>")
System.exit(1)
}
val conf = new SparkConf().setAppName("TopK")
val sc = new SparkContext(conf)
val lines = sc.textFile(args(0))
val ones = lines.flatMap(_.split(" ")).map(word => (word, 1))
val count = ones.reduceByKey((a, b) => a + b)
val convert = count.map {
case (key, value) => (value, key)
}.sortByKey(true, 1)
convert.top(args(1).toInt).foreach(a => System.out.println("(" + a._2 + "," + a._1 + ")"))
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 应用场景:
TopK模型常用于分析消费者热门消费分析、网站/博客点击量、用户浏览量分析,最新热词及热门搜索等的分析处理
第二种TOPK排序方式
基于最小堆的实现:
最大/小堆,对应的数据结构优先级队列,PriorityQueue,不光 Java 中有,Scala 中也有,当然 c++ 中也有
object TopK {
val K = 3
val ord = Ordering.by[(String, Int), Int](_._2).reverse
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("TopK")
val spark = new SparkContext(conf)
val textRDD = spark.textFile("hdfs://10.0.8.162:9000/home/yuzx/input/wordcount.txt")
val countRes = textRDD.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
countRes.foreach(println)
val topk = countRes.mapPartitions(iter => {
val heap = new mutable.PriorityQueue[(String, Int)]()(ord)
while (iter.hasNext) {
val n = iter.next
println("分区计算:" + n)
putToHeap(heap, n)
}
heap.iterator
}).collect()
println("分区结果:")
topk.foreach(println)
val heap = new mutable.PriorityQueue[(String, Int)]()(ord)
val iter = topk.iterator
while (iter.hasNext) {
putToHeap(heap, iter.next)
}
println("最终结果:")
while (heap.nonEmpty) {
println(heap.dequeue())
}
spark.stop()
}
def putToHeap(heap: mutable.PriorityQueue[(String, Int)], iter: (String, Int)): Unit = {
if (heap.nonEmpty && heap.size >= K) {
if (heap.head._2 < iter._2) {
heap += iter
heap.dequeue()
}
} else {
heap += iter
}
}
}