flume配置离线hdfs实时kafka

最新推荐文章于 2024-04-19 14:56:12 发布

潇洒-人生

最新推荐文章于 2024-04-19 14:56:12 发布

阅读量246

点赞数

分类专栏： flume

本文链接：https://blog.youkuaiyun.com/qq_35744460/article/details/90600603

版权

flume数据数据一路到hdfs 供离线分析，一路经kafka消息中间件配置案例

flume-hdfs-kafka.sources = r1
flume-hdfs-kafka.channels = c1
flume-hdfs-kafka.sinks = hdfs-sink kafka-sink

#配置source
flume-hdfs-kafka.sources.r1.type = TAILDIR
flume-hdfs-kafka.sources.r1.channels = c1
flume-hdfs-kafka.sources.r1.filegroups = f1 
flume-hdfs-kafka.sources.r1.filegroups.f1 = /home/bigdata/flume/data/*.txt
flume-hdfs-kafka.sources.r1.positionFile = /var/log/flume/taildir_position.json


 #配置channel
flume-hdfs-kafka.channels.c1.type  =  memory 


#配置hdfs sink
flume-hdfs-kafka.sinks.hdfs-sink.type = hdfs
flume-hdfs-kafka.sinks.hdfs-sink.channel = c1
flume-hdfs-kafka.sinks.hdfs-sink.hdfs.path = hdfs://spark001:8020/flume/events/%y%m%d%H%M
flume-hdfs-kafka.sinks.hdfs-sink.hdfs.filePrefix = events-
flume-hdfs-kafka.sinks.hdfs-sink.hdfs.rollSize = 10000000
flume-hdfs-kafka.sinks.hdfs-sink.hdfs.round = true
flume-hdfs-kafka.sinks.hdfs-sink.hdfs.roundV