Flume+Kafka日志传输

最新推荐文章于 2025-08-05 13:23:41 发布

原创

最新推荐文章于 2025-08-05 13:23:41 发布 · 347 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#flume #kafka

本文档详细介绍了如何配置Flume与Kafka进行日志传输。首先，讲解了Flume的配置文件设置；接着，提到了Zookeeper配置的重要性，指出必须先启动Zookeeper再启动Kafka；然后，阐述了Kafka配置，包括多端口启动服务的步骤；此外，还列出了启动Flume、Zookeeper和Kafka的命令；最后，通过测试数据传输验证配置正确性，并提供了常用Kafka命令及官方文档链接。

Flume+Kafka日志传输

一、flume配置

配置文件在conf/flume-conf.properties.template

a1.sources = s1
a1.channels = c1
a1.sink = k1

a1.sources.s1.type = exec
a1.sources.s1.command = tail -F /usr/tmp/logs/test.log
a1.sources.s1.channels = c1

#a1.channels.c1.type = memory
#a1.channels.c1.capacity = 1000
#a1.channels.c1.transactionCapacity = 100
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = 192.168.133.128:9092,192.168.133.128:9093,192.168.133.128:9094
a1.channels.c1.kafka.topic = test
a1.channels.c1.kafka.consumer.group.id = flume-consumer

a1.sinks.k1.channel = c1
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = test
a1.sinks.k1.kafka.bootstrap.servers = 192.168.133.128:9092
a1.sinks.k1.kafka.flumeBatchSize = 20
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.linger.ms = 1
a1.sinks.k1.kafka.producer.compression.t