Flume 与 kafka 配置和使用

最新推荐文章于 2025-01-23 17:22:17 发布

原创最新推荐文章于 2025-01-23 17:22:17 发布 · 805 阅读

1 ·

CC 4.0 BY-SA版权

kafka 同时被 2 个专栏收录

1 篇文章

订阅专栏

flume

1 篇文章

订阅专栏

本文介绍如何配置Flume从日志文件收集数据并发送到Kafka的过程。使用Flume 1.7.0、Kafka 2.11.0及Zookeeper 3.4.9版本。配置中详细说明了Flume源、通道和接收器的设置，并展示了通过命令行启动Zookeeper、Kafka、Flume代理和Kafka消费者的步骤。

1，使用版本如下：

flume-1.7.0

kafka-2.11.0

zookeeper-3. 4.9

2, 配置flume, 源数据基于日志文件内容检测， sink为 kafka 的producer, 配置文件如下：

a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /root/test.log

# Describe the sink
a1.sinks.k1.type= org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.brokerList=MASTER:9092
a1.sinks.k1.topic=testKafka
a1.sinks.k1.serializer.class=kafka.serializer.StringEncoder

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

3, 为kafka创建topic --> testKafka

kafka-topics.sh -zookeeper localhost:2181 -describe -topic testKafka

4, 先运行zookeeper 和kafka

[root@MASTER conf]# zkServer.sh start 
[root@MASTER conf]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /root/cluster/opt/zookeeper-3.4.9/bin/../conf/zoo.cfg
Mode: standalone
[root@MASTER conf]# kafka-server-start.sh -daemon ../conf/server.properties

查看进程：
[root@MASTER conf]# jps
13010 SecondaryNameNode
27651 Kafka
12534 NameNode
29207 Jps
23303 QuorumPeerMain
12727 DataNode
24953 ConsoleConsumer
13484 NodeManager
13261 ResourceManager

5, 运行flume

[root@MASTER opt]# flume-ng agent -c . -f kafka.conf -n a1 -Dflume.root.logger=INFO,console

6, 启动kafka 消费者进程

[root@MASTER config]# kafka-console-consumer.sh -zookeeper localhost:2181 --from-beginning --topic testKafka

7，测试与验证

1），往/root/test.log 写入内容
echo "mytestKafka one >> test.log
echo "mytestKafka two >> test.log ...
2) , 查看kafka消费进程输出：
...
mytestKafka one
mytestKafka two
mytestKafka three ..