Kafka实战之整合Flume和Kafka完成实时数据采集

最新推荐文章于 2024-11-24 06:00:00 发布

逛逛1973

最新推荐文章于 2024-11-24 06:00:00 发布

阅读量4.4k

点赞数

CC 4.0 BY-SA版权

分类专栏： Hadoop 文章标签： kafka

本文链接：https://blog.youkuaiyun.com/m0_37156901/article/details/90438579

Hadoop 专栏收录该内容

11 篇文章

订阅专栏

本文介绍如何通过修改Flume配置文件，实现从控制台输入数据到Kafka的实时数据采集流程。具体步骤包括配置Avro源、内存通道及Kafka接收器，将数据发送至指定的Kafka主题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一、整合Flume和Kafka完成实时数据采集流程

二、修改flume配置文件，从控制台输出到kafka

/**
*之前的配置控制台文件
*
*/
[peng@bogon conf]$ cat avro-memory-logger.conf 
vro-memory-logger.sources = avro-source
avro-memory-logger.sinks = logger-sink
avro-memory-logger.channels = memory-channel

avro-memory-logger.sources.avro-source.type = avro
avro-memory-logger.sources.avro-source.bind = localhost
avro-memory-logger.sources.avro-source.port = 44444

avro-memory-logger.sinks.logger-sink.type = logger

avro-memory-logger.channels.memory-channel.type = memory

avro-memory-logger.sources.avro-source.channels=memory-channel
avro-memory-logger.sinks.logger-sink.channel=memory-channel
[peng@bogon conf]$ 

/**
*配置到kafka
*/

avro-memory-kafka.sources = avro-source
avro-memory-kafka.sinks = logger-sink
avro-memory-kafka.channels = memory-channel

avro-memory-kafka.sources.avro-source.type = avro
avro-memory-kafka.sources.avro-source.bind = localhost
avro-memory-kafka.sources.avro-source.port = 44444
    
avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.brokerList = localhost:9092
avro-memory-kafka.sinks.kafka-sink.topic = peng_topic
avro-memory-kafka.sinks.kafka-sink.batchSize = 5
avro-memory-kafka.sinks.kafka-sink.requiredAcks = 1

avro-memory-kafka.channels.memory-channel.type = memory

avro-memory-kafka.sources.avro-source.channels=memory-channel
avro-memory-kafka.sinks.logger-sink.channel=memory-channel