flume 容错和负载均衡3

最新推荐文章于 2025-08-16 11:46:42 发布

weixin_30799995

最新推荐文章于 2025-08-16 11:46:42 发布

阅读量66

点赞数

CC 4.0 BY-SA版权

文章标签：大数据

原文链接：http://www.cnblogs.com/wjsshide/p/5032781.html

本文介绍如何使用flume-ng实现集群负载均衡和故障转移功能，通过配置不同机器的角色，如主机1作为负载均衡器，主机2和主机3作为数据接收节点，实现数据流的高效分配和故障恢复。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

集群采用3台机器：

host1 load-balance设置  host2 机器1  host3 机器2

其中，host1 机器配置：

#Define a memory channel called c1 on a1  
a1.channels = c1  
a1.sources = r1  
a1.sinks = k1 k2  
a1.sinkgroups = g1  
a1.sinkgroups.g1.sinks = k1 k2  
a1.sinkgroups.g1.processor.type = load_balance  
a1.sinkgroups.g1.processor.selector = round_robin  
a1.sinkgroups.g1.processor.backoff = true    
a1.channels.c1.type = file   
a1.channels.c1.checkpointDir = /tmp/flume/loadcheckpoint  
a1.channels.c1.dataDirs = /tmp/flume/loaddata    
a1.sources.r1.channels = c1  
a1.sources.r1.type = avro  
a1.sources.r1.bind = 0.0.0.0  
a1.sources.r1.port = 41415  

a1.sinks.k1.channel = c1  
a1.sinks.k1.type = avro  
a1.sinks.k1.hostname = host2  
a1.sinks.k1.port = 41414  

a1.sinks.k2.channel = c1  
a1.sinks.k2.type = AVRO  
a1.sinks.k2.hostname = host3  
a1.sinks.k2.port = 41414

host2 机器配置：

a2.channels = c1  
a2.sources = r1  
a2.sinks = k1    
a2.channels.c1.type = FILE  
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint  
a1.channels.c1.dataDirs = /tmp/flume/data    
a2.sources.r1.channels = c1  
a2.sources.r1.type = AVRO  
a2.sources.r1.bind = 0.0.0.0  
a2.sources.r1.port = 41414    
a2.sinks.k1.channel = c1  
a2.sinks.k1.type = file_roll  
a2.sinks.k1.sink.directory = /tmp/load/  
a2.sinks.k1.sink.rollInterval = 0

host3 机器配置：

a2.channels = c1  
a2.sources = r1  
a2.sinks = k1    
a2.channels.c1.type = FILE  
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint  
a1.channels.c1.dataDirs = /tmp/flume/data    
a2.sources.r1.channels = c1  
a2.sources.r1.type = AVRO  
a2.sources.r1.bind = 0.0.0.0  
a2.sources.r1.port = 41414    
a2.sinks.k1.channel = c1  
a2.sinks.k1.type = file_roll  
a2.sinks.k1.sink.directory = /tmp/load/  
a2.sinks.k1.sink.rollInterval = 0

客户端机器暂时使用flume-ng的agent发送，配置如下：

# Define a memory channel called c1 on a1  
a1.channels.c1.type = file   
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint  
a1.channels.c1.dataDirs = /tmp/flume/data      
a1.sources.r1.channels = c1  
a1.sources.r1.type = exec  
a1.sources.r1.command = cat /tmp/linux.log    
a1.sinks.k1.type = avro   
a1.sinks.k1.channel = c1  
a1.sinks.k1.hostname = host1  
a1.sinks.k1.port = 41415    
a1.channels = c1  
a1.sources = r1  
a1.sinks = k1

客户端/tmp/linux.log文件3G左右，发送给host1。

启动：

host2：

bin/flume-ng agent -c conf -f conf/load-sink1.conf -n a2

host3：

bin/flume-ng agent -c conf -f conf/load-sink2.conf -n a2

host1：

bin/flume-ng agent -c conf -f conf/load-balance.conf -n a1

客户端：

bin/flume-ng agent -c conf -f conf/client.conf -n a1

注意：启动方式最好是从下往上启动。即：先启动host2和host3，然后启动host1，最后启动client。

测试过程中，可以随时将host2或host3停止，过一段时间再启动。

这样，就测试了flume-ng的load-balance和failover功能。

转载于:https://www.cnblogs.com/wjsshide/p/5032781.html