flume与kafka集成

最新推荐文章于 2024-11-24 06:00:00 发布

tugangkai

最新推荐文章于 2024-11-24 06:00:00 发布

阅读量212

点赞数

CC 4.0 BY-SA版权

分类专栏： kafka

本文链接：https://blog.youkuaiyun.com/tugangkai/article/details/83996760

kafka 专栏收录该内容

5 篇文章

订阅专栏

本文详细介绍了Apache Flume如何与Kafka进行集成配置，包括作为生产者将数据发送到Kafka，作为消费者从Kafka接收数据，以及使用Kafka Channel进行数据中转的三种场景。每种场景都提供了具体的配置参数，帮助读者理解如何在实际应用中配置Flume与Kafka的交互。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.KafkaSink

		[生产者]
		a1.sources = r1
		a1.sinks = k1
		a1.channels = c1

		a1.sources.r1.type=netcat
		a1.sources.r1.bind=localhost
		a1.sources.r1.port=8888

		a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
		a1.sinks.k1.kafka.topic = test3
		a1.sinks.k1.kafka.bootstrap.servers = s202:9092
		a1.sinks.k1.kafka.flumeBatchSize = 20
		a1.sinks.k1.kafka.producer.acks = 1

		a1.channels.c1.type=memory

		a1.sources.r1.channels = c1
		a1.sinks.k1.channel = c1

2.KafkaSource

		[消费者]
		a1.sources = r1
		a1.sinks = k1
		a1.channels = c1

		a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
		a1.sources.r1.batchSize = 5000
		a1.sources.r1.batchDurationMillis = 2000
		a1.sources.r1.kafka.bootstrap.servers = s202:9092
		a1.sources.r1.kafka.topics = test3
		a1.sources.r1.kafka.consumer.group.id = g4

		a1.sinks.k1.type = logger

		a1.channels.c1.type=memory

		a1.sources.r1.channels = c1
		a1.sinks.k1.channel = c1

3.Channel

		生产者 + 消费者
		a1.sources = r1
		a1.sinks = k1
		a1.channels = c1

		a1.sources.r1.type = avro
		a1.sources.r1.bind = localhost
		a1.sources.r1.port = 8888

		a1.sinks.k1.type = logger

		a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
		a1.channels.c1.kafka.bootstrap.servers = s202:9092
		a1.channels.c1.kafka.topic = test3
		a1.channels.c1.kafka.consumer.group.id = g6

		a1.sources.r1.channels = c1
		a1.sinks.k1.channel = c1