flume消费kafka到hdfs

最新推荐文章于 2023-02-23 13:53:00 发布

转载最新推荐文章于 2023-02-23 13:53:00 发布 · 898 阅读

博客介绍了Flume和Kafka的数据采集配置。包含指定的Flume 1.7.0和Kafka 0.10.2.1版本，说明了进入conf文件创建flume - hdfs.conf配置文件，还给出启动Flume、Kafka的命令，以及开启发送者发消息和查看文件采集是否成功的操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.压缩包版本
flume版本1.7.0
kafka版本0.10.2.1
2.flume配置文件
进入conf文件
创建 flume-hdfs.conf文件，配置文件代码如下：


# source alias
agent.sources = source_from_kafka
# channels alias
agent.channels = mem_channel
# sink alias
agent.sinks = hdfs_sink

# define kafka source
agent.sources.source_from_kafka.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.source_from_kafka.channels = mem_channel
agent.sources.source_from_kafka.batchSize = 5000
# set kafka broker address  
agent.sources.source_from_kafka.kafka.bootstrap.servers = master:9092,slave1:9092,slave2:9092
# set kafka topic
agent.sources.source_from_kafka.kafka.topics = test
# set kafka groupid
agent.sources.source_from_kafka.kafka.consumer.group.id = test-consumer-group
         
# defind hdfs sink
agent.sinks.hdfs_sink.type = hdfs         
# specify the channel the sink should use  
agent.sinks.hdfs_sink.channel = mem_channel
# set store hdfs path
agent.sinks.hdfs_sink.hdfs.path = /data/flume/kafka/%Y%m%d   
    
# set file size to trigger roll
agent.sinks.hdfs_sink.hdfs.rollSize = 0  
agent.sinks.hdfs_sink.hdfs.rollCount = 0  
agent.sinks.hdfs_sink.hdfs.rollInterval = 3600  
agent.sinks.hdfs_sink.hdfs.threadsPoolSize = 30
agent.sinks.hdfs_sink.hdfs.fileType=DataStream
agent.sinks.hdfs_sink.hdfs.writeFormat=Text    

# define channel from kafka source to hdfs sink 
agent.channels.mem_channel.type = memory
# channel store size
agent.channels.mem_channel.capacity = 100000
# transaction size
agent.channels.mem_channel.transactionCapacity = 10000

3.启动flume命令

./flume-ng agent --conf ../conf/ --name agent --conf-file ../conf/flume-hdfs.conf -Dflume.root.logger=INFO,console

4.启动kafka

./kafka-server-start.sh  -daemon ../config/server.properties

5.开启发送者并发送消息

./kafka-console-producer.sh --broker-list 192.168.88.5:9092 --topic test

6.查看文件是否采集成功

Hadoop fs -cat /data/flume/kafka/20190620/FlumeData.1561088964259.tmp