SparkStreaming之滑动窗口的实现

最新推荐文章于 2024-01-04 06:30:00 发布

CatherineHuangTT

最新推荐文章于 2024-01-04 06:30:00 发布

阅读量894

点赞数

分类专栏： Spark学习随笔

Spark学习随笔专栏收录该内容

32 篇文章

订阅专栏

本文介绍了一个使用Apache Spark Streaming实现的单词计数应用案例，该应用通过接收来自网络的数据流，并利用reduceByKeyAndWindow方法实现每四分钟统计最近六分钟内的单词出现频率。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import org.apache.spark.SparkConf
import org.apache.spark.streaming.dstream.{DStream, ReceiverInputDStream}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.log4j.{Logger,Level}
object Window {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("ReduceByKeyAndWindow").setMaster("local[2]")
    val ssc = new StreamingContext(conf,Seconds(1))
    Logger.getLogger("org.apache.spark").setLevel(Level.WARN)

    val inputDStream: ReceiverInputDStream[String] = ssc.socketTextStream("hadoop02",9999)
    val wordOneDStream: DStream[(String, Int)] = inputDStream.flatMap(_.split(",")).map((_,1))
    /*
      reduceFunc: (V, V) => V,
      windowDuration: Duration,
      slideDuration: Duration
      (_+_) is fault
      每隔4分钟统计近6分钟的单词数
     */
    val resultDStream: DStream[(String, Int)] = wordOneDStream.reduceByKeyAndWindow((x:Int, y:Int)=>x+y,Seconds(6),Seconds(4))
    resultDStream.print()

    ssc.start()
    ssc.awaitTermination()
    ssc.stop()
  }
}