【大数据之Kafka】十七、Kafka集成外部系统之集成Spark

Spark 是一个在大数据开发中非常常用的组件。可以用于 Kafka 的生产者，也可以用于Spark 的消费者。

提前准备Scala环境。

Spark环境准备：
（1）创建一个 maven 项目 spark-kafka；
（2）在项目 spark-kafka 上点击右键，Add Framework Support => 勾选 scala；
（3）在 main 下创建 scala 文件夹，并右键 Mark Directory as Sources Root =>在 scala 下创建包名为 com.study.spark
（4）添加配置文件：

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10_2.12</artifactId>
        <version>3.0.0</version>
    </dependency>
</dependencies>

（5）将 log4j.properties 文件添加到 resources 里面，就能更改打印日志的级别为 error

log4j.rootLogger=error, stdout,R
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %5p --- [%50t] %-80c(line:%5L) : %m%n

log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=../log/agent.log
log4j.appender.R.MaxFileSize=1024KB
log4j.appender.R.MaxBackupIndex=1

log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %5p --- [%50t] %-80c(line:%6L) : %m%n

1 Spark 生产者

在 com.study.spark 包下创建 scala Object：SparkKafkaProducer：

package com.study.spark

import java.util.Properties
import org.apache.kafka.clients.producer.{KafkaProducer,ProducerRecord}

 
object SparkKafkaProducer {
     def main(args: Array[String]): Unit = {
         // 0 kafka 配置信息
         val properties = new Properties()
         properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, 
"hadoop102:9092,hadoop103:9092,hadoop104:9092")
         properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, 
classOf[StringSerializer])
         properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, 
classOf[StringSerializer])
 
         // 1 创建 kafka 生产者
         var producer = new KafkaProducer[String, String](properties)
         
         // 2 发送数据
         for (i <- 1 to 5){
             producer.send(new ProducerRecord[String,String]("first","lyx" + i))
         }
         
         // 3 关闭资源
         producer.close()
     }
}

启动 Kafka 消费者：

bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --topic first

执行 SparkKafkaProducer 程序。

2 Spark 消费者

添加配置文件pom.xml:

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10_2.12</artifactId>
        <version>3.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.12</artifactId>
        <version>3.0.0</version>
    </dependency>
</dependencies>

在 com.study.spark 包下创建 scala Object：SparkKafkaConsumer

package com.study.spark

import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}
import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.spark.SparkConf
import org.apache.spark.streaming.dstream.{DStream, InputDStream}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}

object SparkKafkaConsumer {
     def main(args: Array[String]): Unit = {
         //1.创建 SparkConf
         val sparkConf: SparkConf = new 
        SparkConf().setAppName("sparkstreaming").setMaster("local[*]")
        
         //2.创建 StreamingContext
         val ssc = new StreamingContext(sparkConf, Seconds(3))
         
         //3.定义 Kafka 参数：kafka 集群地址、消费者组名称、key 序列化、value 序列化
         val kafkaPara: Map[String, Object] = Map[String, Object](
             ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> 
"hadoop102:9092,hadoop103:9092,hadoop104:9092",
             ConsumerConfig.GROUP_ID_CONFIG -> "atguiguGroup",
             ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> 
classOf[StringDeserializer],
             ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> 
classOf[StringDeserializer]
             )

        //4.读取 Kafka 数据创建 DStream
        val kafkaDStream: InputDStream[ConsumerRecord[String, String]] = 
            KafkaUtils.createDirectStream[String, String](
                 ssc,
                 LocationStrategies.PreferConsistent, //优先位置
                 ConsumerStrategies.Subscribe[String, String](Set("first"), kafkaPara)// 消费策略：（订阅
多个主题，配置参数）
             )
 
         //5.将每条消息的 KV 取出
         val valueDStream: DStream[String] = kafkaDStream.map(record => record.value())
 
         //6.计算 WordCount
         valueDStream.print()

          //7.开启任务
         ssc.start()
         ssc.awaitTermination()
     }
}

启动 SparkKafkaConsumer 消费者，启动 kafka 生产者：

 bin/kafka-console-producer.sh --bootstrap-server hadoop102:9092 --topic first