SparkStreaming（9）：实例-Streaming整合Spark SQL，进行wordcount功能

最新推荐文章于 2025-07-16 00:16:53 发布

原创最新推荐文章于 2025-07-16 00:16:53 发布 · 1.1k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#SparkStreaming #Spark SQL #wordcount

大数据开发同时被 3 个专栏收录

204 篇文章

订阅专栏

Spark

68 篇文章

订阅专栏

SparkStreaming

32 篇文章

订阅专栏

本文介绍如何结合Spark Streaming和Spark SQL进行实时词频统计，利用DStream和RDD交互操作，通过foreachRDD API将实时数据转换为DataFrame并执行SQL查询。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.功能实现

综合Spark Streaming和Spark SQL，进行word count的统计。核心理解DStream和RDD相互操作，需要通过使用foreachRDD这个API。

2.代码

package Spark

import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession
import org.apache.spark.streaming.{Seconds, StreamingContext, Time}

/**
  * spark streaming 整合spark sql完成词频统计操作
  * https://github.com/apache/spark/blob/v2.1.0/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala
  */
object SqlNetworkWordCount {
  def main(args: Array[String]): Unit = {
    val sparkConf=new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
    /***
      * 创建StreamingContext需要sparkConf和batch interval
      */
    val ssc=new StreamingContext(sparkConf,Seconds(5))

    val lines = ssc.socketTextStream("bigdata.ibeifeng.com", 6789)
    val words = lines.flatMap(_.split(" "))

    // Convert RDDs of the words DStream to DataFrame and run SQL query
    words.foreachRDD { (rdd: RDD[String], time: Time) =>
      // Get the singleton instance of SparkSession
      val spark = SparkSessionSingleton.getInstance(rdd.sparkContext.getConf)
      import spark.implicits._

      // Convert RDD[String] to RDD[case class] to DataFrame
      val wordsDataFrame = rdd.map(w => Record(w)).toDF()

      // Creates a temporary view using the DataFrame
      wordsDataFrame.createOrReplaceTempView("words")

      // Do word count on table using SQL and print it
      val wordCountsDataFrame =
      spark.sql("select word, count(*) as total from words group by word")
      println(s"========= $time =========")
      wordCountsDataFrame.show()
    }

    ssc.start()
    ssc.awaitTermination()
  }

  /** Case class for converting RDD to DataFrame */
  case class Record(word: String)

  /** Lazily instantiated singleton instance of SparkSession */
  object SparkSessionSingleton {

    @transient  private var instance: SparkSession = _

    def getInstance(sparkConf: SparkConf): SparkSession = {
      if (instance == null) {
        instance = SparkSession
          .builder
          .config(sparkConf)
          .getOrCreate()
      }
      instance
    }
  }

}

3.测试

（1）打开nc输入

[root@bigdata hadoop-2.7.3]#  nc -lk 6789
20180808,ww
20180808,ww
20180808,ww

（2）结果：

========= 1537287530000 ms =========
+-----------+-----+
|       word|total|
+-----------+-----+
|20180808,ww|    3|
+-----------+-----+