Spark 时间窗口的 worldcount 在 windows 环境下运行

最新推荐文章于 2025-05-10 22:47:49 发布

原创最新推荐文章于 2025-05-10 22:47:49 发布 · 292 阅读

0 ·

CC 4.0 BY-SA版权

BigData 专栏收录该内容

47 篇文章

订阅专栏

本文介绍如何使用 Apache Spark 的 Spark Streaming 模块进行实时数据流处理。通过设置时间窗口和滑动间隔，实现数据流的批处理和状态更新，演示了基于 Socket 的实时数据输入，以及如何使用 reduceByKeyAndWindow 和 updateStateByKey 进行 wordcount 计算。

pom 依赖

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>${spark.version}</version>
</dependency>

scala:

package org.feng.stream.window
import org.apache.spark._
import org.apache.spark.streaming._

/**
* Created by Feng on 2019/12/3 15:26
* CurrentProject's name is spark
* 时间窗口：3秒一个批次，窗口12秒，滑步6秒
*/
object WordCount {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf().setMaster("local[2]").setAppName("WordCount")
val streamingContext = new StreamingContext(sparkConf, Seconds(3))
// 设置检查点
streamingContext.checkpoint(".")

val lines = streamingContext.socketTextStream("localhost", 12345)
val words = lines.flatMap(_.split(" ")).map(x => (x, 1))
val result = words.reduceByKeyAndWindow((x:Int, y:Int) => x + y, Seconds(12), Seconds(6))

result.print()
streamingContext.start()
streamingContext.awaitTermination()
}
}

附加：计算总的wordcount

package org.feng.stream
import org.apache.spark._
import org.apache.spark.streaming._
/**
  * Created by Feng on 2019/12/3 16:04
  * CurrentProject's name is spark
  */
object StateWordCount {
  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setMaster("local[2]").setAppName("StateWordCount")

    val streamingContext = new StreamingContext(sparkConf, Seconds(3))
    streamingContext.checkpoint(".")
    val lines = streamingContext.socketTextStream("localhost", 12345)


    val fun = (values: Seq[Int], state: Option[Int]) => {
      val currentCount = values.sum
      val previousCount = state.getOrElse(0)
      Some(currentCount + previousCount)
    }

    lines.flatMap(_.split(" ")).map(word => (word, 1)).updateStateByKey(fun).print()

    streamingContext.start()
    streamingContext.awaitTermination()
  }
}