flume笔记(三) 多级agent连接模型

最新推荐文章于 2025-07-09 20:48:48 发布

kkongyu

最新推荐文章于 2025-07-09 20:48:48 发布

阅读量444

点赞数

CC 4.0 BY-SA版权

分类专栏： flume hadoop

本文链接：https://blog.youkuaiyun.com/ting_163/article/details/84900909

hadoop 同时被 2 个专栏收录

16 篇文章

订阅专栏

flume

4 篇文章

订阅专栏

本文介绍了使用Flume构建两层agent连接的实践，通过avroSource和avroSink实现数据在不同agent间的跃点代理。详细步骤包括配置avro_hop.conf，依次启动a2和a1 agent，并验证它们的状态，最后展示了在a1输入数据及a2接收输出数据的过程。

两层agent连接

agent 连接模型

avroSource和avroSink实现跃点agent代理：

配置avro_hop.conf

#a1:
	a1.sources = r1
	a1.sinks = k1
	a1.channels = c1
	
	a1.sources.r1.type = netcat
	al.sources.r1.bind = localhost
	a1.sources.r1.port = 44444
	
	a1.sinks.k1.type = avro
	a1.sinks.k1.bind = localhost
	a1.sinks.k1.port = 88888
	
	a1.channels.c1.type = memory
	a1.channels.c1.capacity = 1000
	a1.channels.c1.transactionCapacity = 100
	
	a1.sources.r1.channels = c1
	a1.sinks.k1.channel = c1
#a2:
	a2.sources = r2
	a2.sinks = k2
	a2.channels = c2
	
	a2.sources.r2.type = avro
	a2.sources.r2.bind = localhost
	a2.sources.r2.port = 88888
	
	a2.sinks.k2.type = logger
	
	a2.channels.c2.type = memory
	a2.channels.c2.capacity = 1000
	a2.channels.c2.transactionCapacity = 100
	
	a2.sources.r2.channels = c2
	a2.sinks.k2.channel = c2

开启a2 agent:

$> bin/flume-ng agent -f conf/avro_hop.conf -n a2 -Dflume.root.logger=INFO,console

验证a2 agent是否开启

$> netstat -anop | grep 88888

开启a1 agent：

$> bin/flume-ng agent -f conf/avro_hop.conf -n a1

验证a1 agent是否开启

$> netstat -anop | grep 44444

a1上输入数据

$> nc localhost 44444

a2 上输出数据

$> .......

参考链接：
flume官方文档