Flume Hello World！

最新推荐文章于 2025-08-04 14:00:00 发布

weixin_30808253

最新推荐文章于 2025-08-04 14:00:00 发布

阅读量127

点赞数

CC 4.0 BY-SA版权

文章标签：大数据

原文链接：http://www.cnblogs.com/hanhuilee/p/5221285.html

本文深入解析了Flume日志收集系统的架构，详细介绍了如何配置Flume来实现日志数据的高效收集，并通过实际命令演示了如何使用Flume进行日志数据的发送与接收。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Flume 是 Cloudera 公司开源出来的一套日志收集系统。模型如下所示：

图中Source，Sink分别代表数据源和数据目的地，channel表示Source和Sink之间的通道。配置文件为/path/to/flume/conf/flume.conf

# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
 
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414
 
# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.log-sink1.channel = ch1
agent1.sinks.log-sink1.type = hdfs
agent1.sinks.log-sink1.hdfs.path = hdfs://qaserver:9001/hello/helloworld 
 
# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = log-sink1

启动flume

/path/to/flume/bin/flume-ng agent --conf /path/to/flume/conf/ -f /path/to/flume/conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1

执行以下命令将./hello.txt内容发送到avro-source1，即本地端口41414。

/path/to/flume/bin/flume-ng avro-client -H localhost -p 41414 -F ./hello.txt

./hello.txt内容如下

Hello World!

查看HDFS文件内容
hadoop fs -cat /hello/helloworld/FlumeData.1394013090403
ƒšî‘ÓD‘§´Ùache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable…=>ûù7ög
           Hello World!

转载于:https://www.cnblogs.com/hanhuilee/p/5221285.html