flume与kafka整合

最新推荐文章于 2025-01-23 17:22:17 发布

jingqiang521

最新推荐文章于 2025-01-23 17:22:17 发布

阅读量1.6k

点赞数 1

分类专栏： 33、flume

33、flume 专栏收录该内容

4 篇文章

订阅专栏

本文详细介绍了如何将Flume与Kafka进行整合，并提供了具体的配置步骤与验证方法。从Flume配置文件的修改到Kafka主题的创建，再到最终的数据传输验证，全程实战指导。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

flume与kafka整合

前提：

flume安装和测试通过，可参考：http://www.cnblogs.com/rwxwsblog/p/5800300.html
kafka安装和测试通过，可参考：http://www.cnblogs.com/rwxwsblog/p/5800224.html

在上诉条件满足的情况下才能进行flume和kafka的整合。

flume与kafka整合(注：flume和kafka两台机器要能连通)

修改/usr/local/flume/conf/flume-conf.properties

agent.sinks.s1.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.s1.topic = mytopic
agent.sinks.s1.brokerList = localhost:9092
agent.sinks.s1.requiredAcks = 1
agent.sinks.s1.batchSize = 20
agent.sinks.s1.channel = c1

启动kafka

启动zookeeper

/usr/local/kafka/bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

启动kafka

/usr/local/kafka/bin/kafka-server-start.sh config/server.properties

创建主题test

/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

启动flume

/usr/local/flume/bin/flume-ng agent --conf conf -f conf/flume-conf.properties -n agent&

验证

新开一个客户端

telnet localhost 8888

[root@localhost ~]# telnet localhost 8888
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
test
OK
chat
OK
this is a flume message to kafka
OK
from flume message.
OK
it's flume and kafka message。
OK

kafka查看消息接收情况

[root@localhost kafka]# /usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
test
chat
this is a flume message to kafka
from flume message.
it's flume and kafka message。

如果出现上述效果则说明flume和kafka整合成功。