sparkstreaming消费direct

最新推荐文章于 2024-05-03 17:08:17 发布

转载最新推荐文章于 2024-05-03 17:08:17 发布 · 124 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/heguoxiu/p/10149677.html

文章标签：

#大数据

本文介绍如何使用Apache Spark Streaming从Kafka消费消息并进行词频统计。通过配置Spark上下文和Kafka参数，创建直接Kafka流，读取指定主题的消息，然后将接收到的数据进行词分割和计数。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

package two

/**
* Created by on 上午2:19.
*/
import kafka.serializer.StringDecoder

import org.apache.spark.streaming._
import org.apache.spark.streaming.kafka._
import org.apache.spark.SparkConf

/**
* Consumes messages from one or more topics in Kafka and does wordcount.
* Usage: DirectKafkaWordCount <brokers> <topics>
* <brokers> is a list of one or more Kafka brokers
* <topics> is a list of one or more kafka topics to consume from
*
* Example:
* $ bin/run-example streaming.DirectKafkaWordCount broker1-host:port,broker2-host:port \
* topic1,topic2
*/
object DirectKafkaWordCount {
def main(args: Array[String]) {
if (args.length < 2) {
System.err.println(s"""
|Usage: DirectKafkaWordCount <brokers> <topics>
| <brokers> is a list of one or more Kafka brokers
| <topics> is a list of one or more kafka topics to consume from
|
""".stripMargin)
System.exit(1)
}

val Array(brokers, topics) = args

// Create context with 2 second batch interval
val sparkConf = new SparkConf().setAppName("DirectKafkaWordCount").setMaster("local[2]")
val ssc = new StreamingContext(sparkConf, Seconds(2))

// Create direct kafka stream with brokers and topics
val topicsSet = topics.split(",").toSet
val kafkaParams = Map[String, String]("metadata.broker.list" -> brokers)
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
ssc, kafkaParams, topicsSet)

// Get the lines, split them into words, count the words and print
val lines = messages.map(_._2)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1L)).reduceByKey(_ + _)
wordCounts.print()

// Start the computation
ssc.start()
ssc.awaitTermination()
}
}
// scalastyle:on println

转载于:https://www.cnblogs.com/heguoxiu/p/10149677.html