flume连kafka

最新推荐文章于 2021-12-25 20:41:47 发布

14yhuang

最新推荐文章于 2021-12-25 20:41:47 发布

阅读量278

点赞数 1

分类专栏：大数据知识文章标签： flume

本文链接：https://blog.youkuaiyun.com/qq_36459386/article/details/86763778

版权

大数据知识专栏收录该内容

53 篇文章

订阅专栏

本文介绍如何使用Apache Flume收集日志数据，并通过Kafka Sink将其发送到Kafka主题，实现数据的实时传输。配置包括定义Exec Source、Memory Channel和Kafka Sink，以及启动Flume Agent和Kafka消费者。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

exec-memory-kafka2.conf

#agent
exec-memory-kafka.sources = exec-source
exec-memory-kafka.channels = memory-channel
exec-memory-kafka.sinks = kafka-sink1

#source
exec-memory-kafka.sources.exec-source.type = exec
exec-memory-kafka.sources.exec-source.command =tail -F /home/hadoop/app/flume-1.6.0-cdh5.7.1/logs/1.log

#channel
exec-memory-kafka.channels.memory-channel.type = memory

#sink
exec-memory-kafka.sinks.kafka-sink1.type=org.apache.flume.sink.kafka.KafkaSink
exec-memory-kafka.sinks.kafka-sink1.topic=ruozeg5sparkkafka
exec-memory-kafka.sinks.kafka-sink1.brokerList=hadoop000:9092

#bind source and sink to channel
exec-memory-kafka.sources.exec-source.channels=memory-channel
exec-memory-kafka.sinks.kafka-sink1.channel=memory-channel

启动

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/exec-memory-kafka2.conf --name exec-memory-kafka -Dflume.root.logger=INFO,console

启动kafka消费者

bin/kafka-console-consumer.sh --zookeeper 192.168.137.190:2181,192.168.137.190:2182,192.168.137.190:2183/kafka --topic ruozeg5sparkkafka

给1.log加数据，发现消费者可以访问到数据

#agent
exec-memory-kafka.sources = exec-source
exec-memory-kafka.channels = memory-channel
exec-memory-kafka.sinks = kafka-sink1,kafka-sink2,kafka-sink3

#source
exec-memory-kafka.sources.exec-source.type = exec


#channel
exec-memory-kafka.channels.memory-channel.type = memory

#sink
exec-memory-kafka.sinks.kafka-sink1.type=org.apache.flume.sink.kafka.KafkaSink
exec-memory-kafka.sinks.kafka-sink1.topic=ruozeg5sparkkafka
exec-memory-kafka.sinks.kafka-sink1.brokerList=hadoop000:9092

exec-memory-kafka.sinks.kafka-sink2.type=org.apache.flume.sink.kafka.KafkaSink
exec-memory-kafka.sinks.kafka-sink2.topic=ruozeg5sparkkafka
exec-memory-kafka.sinks.kafka-sink2.brokerList=hadoop000:9093

exec-memory-kafka.sinks.kafka-sink3.type=org.apache.flume.sink.kafka.KafkaSink
exec-memory-kafka.sinks.kafka-sink3.topic=ruozeg5sparkkafka
exec-memory-kafka.sinks.kafka-sink3.brokerList=hadoop000:9094

#bind source and sink to channel
exec-memory-kafka.sources.exec-source.channels=memory-channel
exec-memory-kafka.sinks.kafka-sink1.channel=memory-channel
exec-memory-kafka.sinks.kafka-sink2.channel=memory-channel
exec-memory-kafka.sinks.kafka-sink3.channel=memory-channel

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/exec-memory-kafka.conf --name exec-memory-kafka -Dflume.root.logger=INFO,console