Flume日志收集工具

准备工作

1、安装jdk,配置好环境变量。

2、安装hadoop,配置好环境变量

3、详细参照官方文档： http://flume.apache.org/releases/content/1.3.0/FlumeUserGuide.pdf

安装Flume服务端

1、下载Flume1.3 http://www.apache.org/dyn/closer.cgi/flume/1.3.1/apache-flume-1.3.1-bin.tar.gz

2、解压Flume

tar -zxvf apache-flume-1.3.1-bin.tar.gz

3、创建软链接

ln -s /root/tools/apache-flume-1.3.1-bin /usr/local/flume

配置服务端

1、配置flume-env.sh文件中的JAVA_HOME

2、配置flume-conf.properties文件

agent.sources = avrosrc

agent.channels = memoryChannel

agent.sinks = hdfsSink

# For each one of the sources, the type is defined

agent.sources.avrosrc.type = avro

agent.sources.avrosrc.bind = 192.168.75.190

agent.sources.avrosrc.port = 41414

# The channel can be defined as follows.

agent.sources.avrosrc.channels = memoryChannel

# Each channel's type is defined.

agent.channels.memoryChannel.type = memory

agent.channels.memoryChannel.keep-alive = 10

agent.channels.memoryChannel.capacity = 100000

agent.channels.memoryChannel.transactionCapacity =100000

# Each sink's type must be defined

agent.sinks.hdfsSink.type = hdfs

agent.sinks.hdfsSink.channel = memoryChannel

agent.sinks.hdfsSink.hdfs.path = /logdata/%{hostname}_linux/%Y%m%d_date

agent.sinks.hdfsSink.hdfs.filePrefix = %{datacenter}_

agent.sinks.hdfsSink.hdfs.rollInterval = 0

agent.sinks.hdfsSink.hdfs.rollSize = 4000000

agent.sinks.hdfsSink.hdfs.rollCount = 0

agent.sinks.hdfsSink.hdfs.writeFormat = Text

agent.sinks.hdfsSink.hdfs.fileType = DataStream

agent.sinks.hdfsSink.hdfs.batchSize = 10

        

启动Flume服务端

1、启动命令：

/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/flume-conf.properties -n agent&

2、查看日志cat /usr/local/flume/logs/flume.log，如果没有错误，则启动成功。

3、 netstat -an|grep 41414，看端口是否启动成功

FlumeSource端配置安装

安装和Flume服务端一样，不做赘述。

FlumeSource端配置

1、配置flume-env.sh文件中的JAVA_HOME

2、配置flume-conf.properties文件

agent.sources = tailsource

agent.channels = memoryChannel

agent.sinks = remotesink

agent.sources.tailsource.type = exec

agent.sources.tailsource.command = tail -F /tmp/linux2.log

agent.sources.tailsource.channels = memoryChannel

agent.sources.tailsource.interceptors = host_int timestamp_int inter1

agent.sources.tailsource.interceptors.host_int.type = host

agent.sources.tailsource.interceptors.host_int.hostHeader = hostname

agent.sources.tailsource.interceptors.timestamp_int.type = org.apache.flume.interceptor.TimestampInterceptor$Builder

#agent.sources.tailsource-1.interceptors = inter1

agent.sources.tailsource.interceptors.inter1.type = static

agent.sources.tailsource.interceptors.inter1.key = datacenter

agent.sources.tailsource.interceptors.inter1.value = BEIJING

agent.channels.memoryChannel.type = memory

agent.channels.memoryChannel.keep-alive = 10

agent.channels.memoryChannel.capacity = 100000

agent.channels.memoryChannel.transactionCapacity =100000

agent.sinks.remotesink.type = avro

agent.sinks.remotesink.hostname = 192.168.75.190

agent.sinks.remotesink.port = 41414

agent.sinks.remotesink.channel = memoryChannel

启动FlumeSource端

1、创建/tmp/linux2.log测试文件。

2、启动命令：

/usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/flume-conf.properties -n agent&

3、查看日志cat /usr/local/flume/logs/flume.log，如果没有错误，则启动成功。

测试Flume

1、FlumeSource端口执行命令：

for i in {1000..1000000}; do echo "LINUX2  PRESS ************* Flume log rotation $i" >> /tmp/linux2.log; sleep 1; done

2、Hadoop端查看hdfs文件是否有文件，文件是否定时接收新数据。