Flink--初识 DataStream Connector Kafka

最新推荐文章于 2025-01-09 10:45:11 发布

原创最新推荐文章于 2025-01-09 10:45:11 发布 · 383 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#flink #kafka #scala

Flink 专栏收录该内容

6 篇文章

订阅专栏

本文介绍如何在Apache Flink中集成Apache Kafka作为数据源和数据接收方，并提供了具体的配置步骤与示例代码。

1. 增加 POM 文件

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka_2.12</artifactId>
    <version>1.11.2</version>
</dependency>

2. 使用 Kafka 作为 Source

2.1 创建 Topic

[root@bigdatatest03 bin]# ./kafka-topics.sh --create --zookeeper  bigdatatest02:2181,bigdatatest02:2181,bigdatatest03:2181 --partitions 3 --replication-factor 3 --topic flink_kafka_source

创建生产者

[root@bigdatatest03 bin]# ./kafka-console-producer.sh \
> --broker-list bigdatatest01:9092,bigdatatest02:9092,bigdatatest03:9092 \
> --topic flink_kafka_source

创建消费者

[root@bigdatatest03 bin]# ./kafka-console-consumer.sh \
> --bootstrap-server bigdatatest01:9092,bigdatatest02:9092,bigdatatest03:9092 \
> --topic flink_kafka_source \
> --from-beginning

2.2 Code

package com.xk.bigdata.flink.datastream.connector

import java.util.Properties

import org.apache.flink.api.common.serialization.SimpleStringSchema
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer

/**
 * 使用 Kafka 作为数据源
 */
object KafkaSourceApp {

  def main(args: Array[String]): Unit = {
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    val properties = new Properties()
    val bootStrap = "bigdatatest02:9092,bigdatatest03:9092,bigdatatest04:9092"
    val sourceTopic = "flink_kafka_source"
    properties.setProperty("bootstrap.servers", bootStrap)
    properties.setProperty("group.id", "demo")
    val kafkaSource = new FlinkKafkaConsumer[String](sourceTopic, new SimpleStringSchema(), properties)
    val stream = env
      .addSource(kafkaSource)
    stream.flatMap(_.split(","))
        .map((_,1))
        .keyBy(_._1)
        .sum(1)
        .print()
    env.execute(this.getClass.getSimpleName)
  }

}

运行结果
生产者

>spark
>spark,hadoop

IDEA 控制台

1> (spark,1)
11> (hadoop,1)
1> (spark,2)

3. 使用 Kafka 作为 Sink

3.1 Code

package com.xk.bigdata.flink.datastream.connector

import java.util.Properties

import org.apache.flink.api.common.serialization.SimpleStringSchema
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer

/**
 * 数据输出到 Kafka
 */
object KafkaSinkApp {

  def main(args: Array[String]): Unit = {
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    val stream = env.readTextFile("data/wc.txt")
    val bootStrap = "bigdatatest02:9092,bigdatatest03:9092,bigdatatest04:9092"
    val topic = "flink_kafka_sink"

    val myProducer = new FlinkKafkaProducer[String](
      bootStrap, // target topic
      topic,
      new SimpleStringSchema()) // serialization schema

    stream.addSink(myProducer)
    env.execute(this.getClass.getSimpleName)
  }

}