Spark Streaming消费kafka示例

最新推荐文章于 2025-05-21 15:52:24 发布

原创最新推荐文章于 2025-05-21 15:52:24 发布 · 2.3k 阅读

1 ·

CC 4.0 BY-SA版权

数据技术专栏收录该内容

9 篇文章

订阅专栏

本文介绍如何使用Spark 2.2.0版本与Kafka 0.11.0.0集成进行实时数据流处理。文章提供了详细的POM依赖配置及示例程序代码，包括设置Spark配置、创建直接Kafka流、读取消息并打印等步骤。

一组件版本

Spark版本：spark-2.1.1-bin-hadoop2.7
Kafka版本：kafka_2.11-0.11.0.0

Scala版本：2.11.8

Tips：用scala 2.12.x的版本会报方法不存在错误

二 POM文件内容

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>
</dependencies>

三示例程序

import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import org.apache.spark.streaming.kafka010._


object StreamingTest {
    def main(args: Array[String]): Unit = {
      //获取sparkstreaming

      val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")
      val ssc = new StreamingContext(conf, Seconds(10))

      // Create direct kafka stream with brokers and topics
      val topics="test"
      val brokers="localhost:9092"
      val topicsSet = topics.split(",").toSet
      val kafkaParams = Map[String, String]("bootstrap.servers" -> brokers,
        "value.deserializer" -> "org.apache.kafka.common.serialization.StringDeserializer",
        "key.deserializer" -> "org.apache.kafka.common.serialization.StringDeserializer",
        "group.id" -> "test-consumer-group")
      val messages = KafkaUtils.createDirectStream[String, String](
        ssc,
        LocationStrategies.PreferConsistent,
        ConsumerStrategies.Subscribe[String, String](topicsSet, kafkaParams))

      // Get the lines, split them into words, count the words and print
     
      val lines = messages.map(_.value)
      lines.print()

      // Start the computation
      ssc.start()
      ssc.awaitTermination()
    }
}