使用flume读取数据，sink到kafka中及hdfs中

最新推荐文章于 2025-03-20 09:59:50 发布

Messi的小迷弟

最新推荐文章于 2025-03-20 09:59:50 发布

阅读量854

点赞数

文章标签： flume spark hdfs

本文链接：https://blog.youkuaiyun.com/wAwT13zA/article/details/108107529

版权

案例一：使用flume读取数据，sink到kafka中

users.sources = usersSource
users.channels = usersChannel
users.sinks = usersSink

users.sources.usersSource.type = spooldir
users.sources.usersSource.spoolDir = /data/flumeFile/users
users.sources.usersSource.deserializer = LINE
users.sources.usersSource.deserializer.maxLineLength = 3000
users.sources.usersSource.includePattern = users_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
# 过滤第一行的脏数据
# 定义一个过滤器，名字叫 head_filter
users.sources.usersSource.interceptors = head_filter
# 使用正则过滤器 regex_filter
users.sources.usersSource.interceptors.head_filter.type = regex_filter
# 以user_id开头的数据过滤掉
users.sources.usersSource.interceptors.head_filter.regex = ^user_id*
# 开启过滤数据
users.sources.usersSource.interceptors.head_filter.excludeEvents = true

users.channels.usersChannel.type = file
users.channels.usersChannel.checkpointDor = /data/flumeFile/checkpoint/users
users.channels.usersChannel.dataDir = /data/flumeFile/data/users

#sink类型为 org.apache.flume.sink.kafka.KafkaSink 读到kafka中
users.sinks.usersSink.type = org.apache.flume.sink.kafka.KafkaSink
# 批处理大小
users.sinks.usersSink.batchSize = 640
# broker的地址
users.sinks.usersSink.brokerList = 192.168.108.181:9092
# topic的名字
users.sinks.usersSink.topic = users

users.sources.usersSource.channels = usersChannel
users.sinks.usersSink.channel = usersChannel

案例二：使用flume读取数据，sink到hdfs中和kafka中，分两个channel传输

train.sources = trainSource
# 创建两个channel
train.channels = kafkaChannel hdfsChannel
# 创建两个sink
train.sinks = kafkaSink hdfsSink

train.sources.trainSource.type = spooldir
train.sources.trainSource.spoolDir = /data/flumeFile/train
train.sources.trainSource.deserializer = LINE
train.sources.trainSource.deserializer.maxLineLength = 3000
train.sources.trainSource.includePattern = train_[0-9]{4}-[0-9]{2}-[0-9]{2}.csv
train.sources.trainSource.interceptors = head_filter
train.sources.trainSource.interceptors.head_filter.type = regex_filter
train.sources.trainSource.interceptors.head_filter.regex = ^user*
train.sources.trainSource.interceptors.head_filter.excludeEvents = true

train.channels.kafkaChannel.type = file
train.channels.kafkaChannel.checkpointDir = /data/flumeFile/checkpoint/train
train.channels.kafkaChannel.dataDirs = /data/flumeFile/data/train

# 使用memory内存当作channel
train.channels.hdfsChannel.type = memory
train.channels.hdfsChannel.capacity  = 64000
train.channels.hdfsChannel.transactionCapacity = 16000

train.sinks.kafkaSink.type = org.apache.flume.sink.kafka.KafkaSink
train.sinks.kafkaSink.batchSize = 640
train.sinks.kafkaSink.brokerList = hadoop1:9092
train.sinks.kafkaSink.topic = train

train.sinks.hdfsSink.type = hdfs
train.sinks.hdfsSink.hdfs.fileType = DataStream
train.sinks.hdfsSink.hdfs.filePrefix = train
train.sinks.hdfsSink.hdfs.fileSuffix = .csv
train.sinks.hdfsSink.hdfs.path = hdfs://hadoop1:9000/user/events/train/%Y-%m-%d
# 必须开启useLocalTimeStamp 才能使用日期当作目录
train.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
train.sinks.hdfsSink.hdfs.batchSize = 6400
train.sinks.hdfsSink.hdfs.rollCount = 0
train.sinks.hdfsSink.hdfs.rollSize = 64000000
train.sinks.hdfsSink.hdfs.rollInterval = 10

train.sources.trainSource.channels =  kafkaChannel hdfsChannel
train.sinks.hdfsSink.channel = hdfsChannel
train.sinks.kafkaSink.channel = kafkaChannel