【Data Algorithms_Recipes for Scaling up with Hadoop and Spark】Chapter3 Top 10 NonUniqueList

最新推荐文章于 2016-03-31 19:00:45 发布

翻译最新推荐文章于 2016-03-31 19:00:45 发布 · 839 阅读

spark 专栏收录该内容

28 篇文章

订阅专栏

本文介绍了一种使用Apache Spark实现的通用Top-N算法，该算法适用于处理非唯一键的数据集。通过具体示例展示了如何利用Spark API进行数据映射、规约及排序，最终找出每个类别中的前N项。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

：scala版

package com.bbw5.dataalgorithms.spark

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import java.util.PriorityQueue

/**
 * Assumption: for all input (K, V), K's are non-unique.
 * This class implements Top-N design pattern for N > 0.
 * The main assumption is that for all input (K, V)'s, K's
 * are non-unique. It means that you will find entries like
 * (A, 2), ..., (A, 5),...
 *
 * This is a general top-N algorithm which will work unique
 * and non-unique keys.
 *
 * This class may be used to find bottom-N as well (by
 * just keeping N-smallest elements in the set.
 *
 *  Top-10 Design Pattern: “Top Ten” Structure
 *
 *  1. map(input) => (K, V)
 *
 *  2. reduce(K, List<V1, V2, ..., Vn>) => (K, V),
 *                where V = V1+V2+...+Vn
 *     now all K's are unique
 *
 *  3. Find Top-N using the following high-level Spark API:
 *     java.util.List<T> takeOrdered(int N, java.util.Comparator<T> comp)
 *     Returns the first N elements from this RDD as defined by the specified
 *     Comparator[T] and maintains the order.
 * cat id,cat name,cat weight
 * 1,cat1,13
 * 2,cat2,10
 * 3,cat3,14
 * 4,cat4,13
 * 5,cat5,20
 * 6,cat6,24
 * 7,cat7,13
 * 8,cat8,10
 * 9,cat9,24
 * 10,cat10,13
 * 11,cat11,30
 * 12,cat12,14
 * @author babaiw5
 *
 */
object SparkTop10UsingTakeOrdered {
  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("SparkTop10UsingTakeOrdered")
    val sc = new SparkContext(sparkConf)

    val filename = "G:/temp/data/top1.txt"
    val textFile = sc.textFile(filename)

    val topN = 5
    val topNRDD = textFile.map(_.split(",")).map(d =>
      ((d(0), d(1)), d(2).toInt)).reduceByKey((a, b) => a + b).
      takeOrdered(topN)(Ordering.by[((String, String), Int), Int](-_._2))
      
    topNRDD.foreach(println)
    
  }
}