SparkStreaming wordcount demo

最新推荐文章于 2024-09-26 20:47:25 发布

原创最新推荐文章于 2024-09-26 20:47:25 发布 · 1k 阅读

0 ·

CC 4.0 BY-SA版权

spark 同时被 2 个专栏收录

57 篇文章

订阅专栏

streaming

3 篇文章

订阅专栏

本文介绍了一个使用Apache Spark Streaming实现的实时流数据统计分析系统。该系统每10秒接收一次数据，并进行单词级别的统计，适用于实时日志分析等场景。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

流数据统计，将每隔10s内的数据做一次单词统计

package com.streaming

import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.SparkContext
import org.apache.spark.api.java.function._
import org.apache.spark.streaming._
import org.apache.spark.streaming.api._
import org.apache.spark.SparkConf

object WordCount {
def main(args: Array[String]): Unit = {
// Create a StreamingContext with a local master
// Spark Streaming needs at least two working thread
   val ssc = new StreamingContext(new SparkConf().setAppName("NetworkWordCount"),Seconds(10))
//val ssc = new StreamingContext("local[2]", "NetworkWordCount", Seconds(10))
// Create a DStream that will connect to serverIP:serverPort, like localhost:9999
val lines = ssc.socketTextStream("172.171.51.131", 9999)
// Split each line into words
val words = lines.flatMap(_.split(" "))
// Count each word in each batch
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_+ _)
wordCounts.print()
ssc.start()
ssc.awaitTermination()
}
}