sparkcore实现word count

最新推荐文章于 2023-06-06 14:17:06 发布

0x12A2A7F

最新推荐文章于 2023-06-06 14:17:06 发布

阅读量320

点赞数

CC 4.0 BY-SA版权

分类专栏： scala Big Data Data Analysis 文章标签： spark 大数据

本文链接：https://blog.youkuaiyun.com/LW_GHY/article/details/117772233

Big Data 同时被 3 个专栏收录

77 篇文章

订阅专栏

Data Analysis

25 篇文章

订阅专栏

scala

15 篇文章

订阅专栏

本文介绍使用Apache Spark通过六种不同方法实现WordCount的过程，包括reduceByKey、groupByKey、aggregateByKey、foldByKey和combineByKey等，并提供了基于Spark 3.1.2和Scala 2.12.13的具体代码示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark core几种实现word count的方式（基于spark3.1.2和scala2.12.13）：

import org.apache.spark.{SparkConf, SparkContext}

object WordCount {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[4]").setAppName("WordCount")
    val sc = new SparkContext(conf)
    val rdd = sc.makeRDD(Seq("Apache Spark lightning-fast unified analytics engine",
      "Apache Spark is a unified analytics engine for large-scale data processing"))
      .flatMap(_.split(" ")).map((_, 1))

    println("==========1. reduceByKey==========")
    val words = rdd.reduceByKey(_ + _)
    words.collect().foreach(println)

    val groupRDD = rdd.groupByKey().mapValues(_.sum)
    println("==========2. groupByKey==========")
    groupRDD.collect().foreach(println)

    val aggRDD = rdd.aggregateByKey(0)(math.max, _ + _)
    println("==========3. aggregateByKey1==========")
    aggRDD.collect().foreach(println)

    println("==========4. aggregateByKey2==========")
    val aggRDD2 = rdd.aggregateByKey(0)(_ + _, _ + _)
    aggRDD2.collect().foreach(println)

    val foldRDD = rdd.foldByKey(0)(_ + _)
    println("==========5. foldByKey==========")
    foldRDD.collect().foreach(println)

    val combineRDD = rdd.combineByKey((_, 1),
      (acc: (Int, Int), v) => (acc._1 + v, acc._2 + 1),
      (acc1: (Int, Int), acc2: (Int, Int)) => (acc1._1 + acc2._1, acc1._2 + acc2._2)
    ).mapPartitions(prt => {
      prt.map(f => (f._1, f._2._2))
    })
    println("==========6. combineByKey==========")
    combineRDD.collect().foreach(println)

    sc.stop()
  }
}