flume介绍与配置案例

最新推荐文章于 2024-08-02 16:29:39 发布

lihuan1124

最新推荐文章于 2024-08-02 16:29:39 发布

阅读量240

点赞数 1

CC 4.0 BY-SA版权

分类专栏：大数据环境搭建

本文链接：https://blog.youkuaiyun.com/lihuan1124/article/details/80890821

大数据环境搭建专栏收录该内容

4 篇文章

订阅专栏

->使用（解压即用）

安装Flume

下载
解压到~/app
将java配置系统环境变量中: ~/.bash_profile
export FLUME_HOME=/home/hadoop/app/apache-flume-1.6.0-cdh5.7.0-bin
export PATH=$FLUME_HOME/bin:$PATH
source下让其配置生效
flume-env.sh的配置：export JAVA_HOME=/home/hadoop/app/jdk1.8.0_144
检测: flume-ng version
使用Flume的关键就是写配置文件
A）配置Source
B）配置Channel
C）配置Sink
D）把以上三个组件串起来

a1: agent名称
r1: source的名称
k1: sink的名称
c1: channel的名称

Agent选型：exec source + memory channel + logger sink
#example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /home/hadoop/data/data.log
a1.sources.r1.shell = /bin/sh -c

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动agent
flume-ng agent \
--name a1 \
--conf $FLUME_HOME/conf \
--conf-file $FLUME_HOME/conf/exec-memory-logger.conf \
-Dflume.root.logger=INFO,console
Event: { headers:{} body: 68 65 6C 6C 6F 0D hello. }
Event是FLume数据传输的基本单元
Event = 可选的header + byte array

->案例：

1.flume-kafka

----------------------------------------------------------vim kafka.conf------------------------------------------------------
exec-memory-kafka.sources = exec-source
exec-memory-kafka.sinks = kafka-sink
exec-memory-kafka.channels = memory-channel

exec-memory-kafka.sources.exec-source.type = exec
exec-memory-kafka.sources.exec-source.command = tail -F tail -F /root/data/data.log
exec-memory-kafka.sources.exec-source.shell = /bin/sh -c

exec-memory-kafka.channels.memory-channel.type = memory

exec-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
exec-memory-kafka.sinks.kafka-sink.brokerList = hdp01:9092
exec-memory-kafka.sinks.kafka-sink.topic = my-topic
exec-memory-kafka.sinks.kafka-sink.batchSize = 5
exec-memory-kafka.sinks.kafka-sink.requiredAcks = 1

exec-memory-kafka.sources.exec-source.channels = memory-channel
exec-memory-kafka.sinks.kafka-sink.channel = memory-channel

启动：flume-ng agent --name exec-memory-kafka --conf $FLUME_HOME/conf --conf-file /root/software/apache-flume-1.7.0-bin/conf/kafka-conf -Dflume.root.logger=INFO,console

--------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------vim flume-hdfs-------------------------------------------------------------

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /root/data/data.log
a1.sources.r1.shell = /bin/sh -c

# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://hdp03:9000/test/data.log

# Use a channel which buffers events in memory
a1.channels.c1.type = memory

# Bind the source and sink to the channel

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

启动：flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file /root/software/apache-flume-1.7.0-bin/conf/hdfs-conf -Dflume.root.logger=INFO,console

--------------------------------------------------------------------------------------------------------------------------------