使用flume 读取数据 sink到kafka中消费者读出_flume1.9 kafka sink consumer 拿到header中的信息-优快云博客

本文链接：https://blog.youkuaiyun.com/sunbin11220904/article/details/108107331

本文档展示了如何使用Flume从users.csv和usersfriends.csv文件中读取数据，并将这些数据发送到Kafka，以便进一步的处理和消费。配置文件详细说明了Flume的数据流向。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

用flume 读取users.csv 传出到kafka读取 .conf 文件如下

users.sources=usersSource
users.channels =usersChannel
users.sinks =usersSink

users.sources.usersSource.type = spooldir
users.sources.usersSource.spoolDir=/opt/kb07file/flumeFile/users
users.sources.usersSource.deserializer = LINE
users.sources.usersSource.deserializer.maxLineLength =3000
users.sources.usersSource.includePattern= user_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
users.sources.usersSource.interceptors=head_filter
users.sources.usersSource.interceptors.head_filter.type=regex_filter
users.sources.usersSource.interceptors.head_filter.regex=^user_id*
users.sources.usersSource.interceptors.head_filter.excludeEvents = true

users.channels.usersChannel.type = file
users.channels.usersChannel.checkpointDir= /opt/kb07file/flumeFile/checkpoint/users
users.channels.usersChannel.dataDir=/opt/kb07file/flumeFile/data/users


users.sinks.usersSink.type = org.apache.flume.sink.kafka.KafkaSink
users.sinks.usersSink.batchSize =640
users.sinks.usersSink.brokerList = 192.168.106.107:9092
users.sinks.usersSink.topic=users


users.sources.usersSource.channels =usersChannel
users.sinks.usersSink.channel =usersChannel

用flume 读取usersfriends.csv 传出到kafka读取 .conf 文件如下


user_friends.sources =userFriendSource
user_friends.channels = userFriendChannel
user_friends.sinks = userFriendSink

user_friends.sources.userFriendSource.type = spooldir
//为输入文件目录
user_friends.sources.userFriendSource.spoolDir =/opt/kb07file/flumeFile/user_friends/
user_friends.sources.userFriendSource.deserializer =LINE
user_friends.sources.userFriendSource.deserializer.maxLineLength = 60000
//匹配文件名
user_friends.sources.userFriendSource.includePattern =userFriends_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv


user_friends.channels.userFriendChannel.type = file
user_friends.channels.userFriendChannel.checkpointDir =/opt/kb07file/flumeFile/checkpoint/ufriends/
user_friends.channels.userFriendChannel.dataDirs=/opt/kb07file/flumeFile/data/uFriends/

user_friends.sinks.userFriendSink.type =org.apache.flume.sink.kafka.KafkaSink
user_friends.sinks.userFriendSink.batchSize =640
user_friends.sinks.userFriendSink.brokerList = 192.168.106.107:9092
user_friends.sinks.userFriendSink.topic=user_friends_raw

user_friends.sources.userFriendSource.channels = userFriendChannel
user_friends.sinks.userFriendSink.channel=userFriendChannel