Flume-ngfailover以及Loadbalance测试

详情可以查看逛网配置
官网的配置 ： https://flume.apache.org/FlumeUserGuide.html

实现功能:
1.负载均衡
2.故障转移

一.负载均衡
source里的event流经channel，进入sink组，在sink组内部根据负载算法（round_robin、random）选择sink，后续可以选择不同机器上的agent实现负载均衡。

配置如下：

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels=c1
a1.sources.r1.command=tail -F /home/flume/xx.log 

#define sinkgroups
a1.sinkgroups=g1
a1.sinkgroups.g1.sinks=k1 k2
a1.sinkgroups.g1.processor.type=load_balance
a1.sinkgroups.g1.processor.backoff=true # 这个里面是个惩罚机制，会有相关文章进行分析
a1.sinkgroups.g1.processor.selector=round_robin

#define the sink 1
a1.sinks.k1.type=avro
a1.sinks.k1.hostname=192.168.109.112
a1.sinks.k1.port=9900

#define the sink 2
a1.sinks.k2.type=avro
a1.sinks.k2.hostname=192.168.109.113
a1.sinks.k2.port=9900


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel=c1

另外启动112.113上的agent即可
agent1

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 192.168.109.112
a1.sources.r1.port = 9900

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#define the sink 1
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://jiedian1:9000/flume
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.writeFormat=Text
a1.sinks.k1.hdfs.minBlockReplicas=1 --必须配置这个，才能正常的滚动
a1.sinks.k1.hdfs.rollInterval=0            -- 0 不按该规则，间隔时间，滚动成目标文件
a1.sinks.k1.hdfs.rollSize=10000          --临时文件大于该值，则写成正式文件
a1.sinks.k1.hdfs.rollCount=0               -- 0 不按该规则，events数量，滚动成目标文件
a1.sinks.k1.hdfs.idleTimeout=5          --当临时文件在再该值没有数据写入，滚成目标文件
a1.sinks.k1.hdfs.filePrefix=access
a1.sinks.k1.hdfs.fileSuffix=.log
a1.sinks.k1.hdfs.threadsPoolSize=10             


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

agent2

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 192.168.109.113
a1.sources.r1.port = 9900

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#define the sink 1
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://jiedian1:9000/flume
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.writeFormat=Text
a1.sinks.k1.hdfs.minBlockReplicas=1 --必须配置这个，才能正常的滚动
a1.sinks.k1.hdfs.rollInterval=0            -- 0 不按该规则，间隔时间，滚动成目标文件
a1.sinks.k1.hdfs.rollSize=10000          --临时文件大于该值，则写成正式文件
a1.sinks.k1.hdfs.rollCount=0               -- 0 不按该规则，events数量，滚动成目标文件
a1.sinks.k1.hdfs.idleTimeout=5          --当临时文件在再该值没有数据写入，滚成目标文件
a1.sinks.k1.hdfs.filePrefix=access
a1.sinks.k1.hdfs.fileSuffix=.log
a1.sinks.k1.hdfs.threadsPoolSize=10             


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

小结:
1.当两台主机中的一台进程挂掉后，发送端会报错误，但是不影响数据的发送。
2.当挂掉的进程回复后，会继续被接受数据。这里面有个惩罚机制，如果开启的失败的接受端会被等待延迟！
3.当挂掉的进程回复后，发送端会自动添加会发送列表里，报错消失！
4.这个就是一个简单的均衡方式，相当于见了一个sink组，在sink组中的数据被轮询发送！

二.故障转移
Ps:与负载均衡有所区别,上面的是负载均衡，每台机器都会受到数据处理数据，但是有时候业务需求可能需要一台机器为主要一条业务的数据接收器，另一个台在挂掉后才会被启用，这时候这个就有用了。

配置文件所不同:

# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.channels=c1
a1.sources.r1.command=tail -F /home/flume/xx.log 

#在这里与上述有所区别
#priority越高，优先级越高，会优先使用该sink
#Each sink needs to have a different priority. If multiple sinks have
#same priority, only one of them will be used.
a1.sinkgroups=g1  
a1.sinkgroups.g1.sinks=k1 k2  
a1.sinkgroups.g1.processor.type=failover  
a1.sinkgroups.g1.processor.priority.k1=10  
a1.sinkgroups.g1.processor.priority.k2=5  
a1.sinkgroups.g1.processor.maxpenalty=10000

#define the sink 1
a1.sinks.k1.type=avro
a1.sinks.k1.hostname=192.168.109.112
a1.sinks.k1.port=9900

#define the sink 2
a1.sinks.k2.type=avro
a1.sinks.k2.hostname=192.168.109.113
a1.sinks.k2.port=9900


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel=c1