Flume之Log4JAppender接收Log4j日志

简介

Log4JAppender和LoadBalancingLog4jAppender可以将应用服务器的日志通过AvroSource实时的把日志传输到日志服务器，然后在传输到监控系统或者是HDFS中存储，FLume的Log4JAppender和LoadBalancingLog4jAppender必须使用Log4j的异步加载器，否则日志服务器down机，将会导致应用服务器异常，影响线上环境的使用。 应用程序如果要用Log4JAppender就要依赖相关的jar，在应用服务器可以通过如下Maven配置获取到Log4JAppender的jar和log4j的相关jar：

 

<!-- flume log4j appender start -->
<dependency>
    <groupId>org.apache.flume.flume-ng-clients</groupId>
    <artifactId>flume-ng-log4jappender</artifactId>
    <version>1.6.0</version>
</dependency>
<!-- flume log4j appender end -->

<!-- log4j start -->
<dependency>  
    <groupId>log4j</groupId>  
    <artifactId>log4j</artifactId>  
    <version>1.2.17</version>  
</dependency>  
<dependency>  
    <groupId>org.slf4j</groupId>  
    <artifactId>slf4j-api</artifactId>  
    <version>1.7.5</version>  
</dependency>  
<dependency>  
    <groupId>org.slf4j</groupId>  
    <artifactId>slf4j-log4j12</artifactId>  
    <version>1.7.5</version>  
</dependency>
<!-- log4j end -->

 

Log4jAppender和LoadBalancingLog4jAppender有如下区别：

1，LoadBalancingLog4jAppender支持一组AvroSource。
2，LoadBalancingLog4jAppender支持round-robin，random或自定义的负载均衡方式。
3，当Event发送失败的情况下，LoadBalancingLog4jAppender支持backoff策略。

 

Log4JAppender配置文件

 

属性名	默认	描述
Hostname	–	运行在远程Flume Agent中Avro Source的Hostname
Port	–	运行在远程Flume Agent中Avro Source的监听端口
UnsafeMode	false	如果设置为true，那么该Appender在发送Event失败不会抛出异常。
AvroReflectionEnabled	false	使用Avro反射机制序列化Log4j的Event事件。（如果日志是字符串就不需要了。）
AvroSchemaUrl	–	一个可以检索Avro schema的URL

 

LoadBalancingLog4jAppender配置文件

 

属性名	默认	描述
Hosts	–	通过AvroSource监听Event的host:port列表，用空格分隔
Selector	ROUND_ROBIN	选择机制。选择机制包含：ROUND_ROBIN，RANDOM或者是自定义
MaxBackoff	–	MaxBackoff是一个long型值，代表一个节点消费一个event失败的情况下，负载均衡客户端最大的backoff时间（单位:毫秒），默认不启动backoff
UnsafeMode	false	如果设置为true，那么该Appender在发送Event失败不会抛出异常。
AvroReflectionEnabled	false	使用Avro反射机制序列化Log4j的Event事件。（如果日志是字符串就不需要了。）
AvroSchemaUrl	–	一个可以检索Avro schema的URL

示例

应用程序端log4j配置

一，采用Log4JAppender

1，采用properties方式：

 

log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname=10.0.1.75
log4j.appender.flume.Port=5555
log4j.appender.flume.UnsafeMode=true
log4j.appender.flume.layout=org.apache.log4j.PatternLayout
log4j.appender.flume.layout.ConversionPattern=%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n

 

2，采用xml方式：

 

<appender name="flume" class="org.apache.flume.clients.log4jappender.Log4jAppender">
	<param name="Hostname" value="10.0.1.75" />
	<param name="Port" value="5555" />
        <param name="UnsafeMode" value="true" />
	<layout class="org.apache.log4j.PatternLayout">
		<param name="ConversionPattern" value="%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n" />
	</layout>
</appender>

二，采用LoadBalancingLog4jAppender

 

1，采用properties方式：

 

log4j.appender.flume=org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
log4j.appender.flume.Hosts=10.0.1.75:5555 10.0.1.76:5555 10.0.1.77:5555
log4j.appender.flume.UnsafeMode=true
log4j.appender.flume.MaxBackoff=30000
log4j.appender.flume.Selector=RANDOM
log4j.appender.flume.layout=org.apache.log4j.PatternLayout
log4j.appender.flume.layout.ConversionPattern=%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n

 

2，采用xml方式：

<appender name="flume" class="org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender">  
	<param name="Hosts" value="10.0.1.75:5555 10.0.1.76:5555 10.0.1.77:5555" />
	<param name="Selector" value="RANDOM" />
        <param name="UnsafeMode" value="true" />
        <param name="MaxBackoff" value="30000" />
	<layout class="org.apache.log4j.PatternLayout">
		<param name="ConversionPattern" value="%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n" />
	</layout>  
</appender>

 

测试代码：

 

protected static final Logger logger = Logger.getLogger(WriteLog.class);
@Test
public void testLog4jAppender() {
	int i = 1;
	while (true) {
		try {
			logger.error(String.valueOf(new Date().getTime()) + "---第几个：" + i, new NullPointerException("llllll"));
			i++;
			Thread.sleep(2000);
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

 

服务端Flume Agent

 

a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 5555

# Describe the sink
a1.sinks.k1.type = file_roll
a1.sinks.k1.sink.directory = /data/
a1.sinks.k1.sink.rollInterval = 0
a1.sinks.k1.sink.serializer = TEXT
a1.sinks.k1.sink.batchSize = 100

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

如果是采用LoadBalancingLog4jAppender方式，需要部署多台Flume Agent。配置雷同。