flume 基础入门

CDH相关的软件下载地址：http://archive.cloudera.com/cdh5/cdh/5/

cdh-5.7.0

生产或者测试环境选择对应CDH版本时，一定要采用尾号是一样的版本

 

 

下载flume：  

 lunx命令：   wget http://archive.cloudera.com/cdh5/cdh/5/ flume-ng-1.6.0-cdh5.7.0.tar.gz

 

配置conf

[root@Machine5 apache-flume-1.6.0-cdh5.7.0-bin]# cd conf/

[root@Machine5 conf]# cp flume-env.sh.template flume-env.sh

[root@Machine5 conf]# vim flume-env.sh

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.


# Enviroment variables can be set here.
export JAVA_HOME=/opt/jdk1.8.0_191


# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
# export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"


# Note that the Flume conf directory is always included in the classpath.
#FLUME_CLASSPATH=""

修改flume-env.sh主要是配置export JAVA_HOME=/opt/jdk1.8.0_191

 

 

参考官网：  http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#avro-source

example.conf

# example.conf: A single-node Flume configuration


# Name the components on this agen
a1.sources = r1
a1.sinks = k1
a1.channels = c1


# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444


# Describe the sink
a1.sinks.k1.type = logger


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
#a1.channels.c1.capacity = 1000
#a1.channels.c1.transactionCapacity = 100


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

 

一个终端：

启动命令：

[root@Machine5 apache-flume-1.6.0-cdh5.7.0-bin]# bin/flume-ng agent --conf conf --conf-file conf/example.conf --name a1 -Dflume.root.logger=INFO,console

 

 

另一个终端：

[root@Machine5 conf]# telnet localhost 44444

 

 

结果显示：

 

日志收集过程（跨机器）：

 

 

exec-memory-avro.conf

# Name the components on this agent
exec-memory-avro.sources = r1
exec-memory-avro.sinks = k1
exec-memory-avro.channels = c1


# Describe/configure the source
exec-memory-avro.sources.r1.type = exec
exec-memory-avro.sources.r1.command = tail -F /root/data.log
exec-memory-avro.sources.tailsource-1.shell = /bin/bash -c



# Describe the sink
exec-memory-avro.sinks.k1.type = avro
exec-memory-avro.sinks.k1.hostname = 127.0.0.1
exec-memory-avro.sinks.k1.port = 4545


# Use a channel which buffers events in memory
exec-memory-avro.channels.c1.type = memory


# Bind the source and sink to the channel
exec-memory-avro.sources.r1.channels = c1
exec-memory-avro.sinks.k1.channel = c1

 

启动命令：

bin/flume-ng agent --conf conf --conf-file conf/exec-memory-avro.conf --name exec-memory-avro -Dflume.root.logger=INFO,console

 

 

avro-memory-logger.conf

avro-memory-logger.sources = r1
avro-memory-logger.sinks = k1
avro-memory-logger.channels = c1

avro-memory-logger.sources.r1.type =avro
avro-memory-logger.sources.r1.bind = 127.0.0.1
avro-memory-logger.sources.r1.port = 4545

avro-memory-logger.sinks.k1.type = logger

avro-memory-logger.channels.c1.type = memory

# Bind the source and sink to the channel
avro-memory-logger.sources.r1.channels = c1
avro-memory-logger.sinks.k1.channel = c1

 

启动命令： 

bin/flume-ng agent --conf conf --conf-file conf/avro-memory-logger.conf --name avro-memory-logger -Dflume.root.logger=INFO,console

 

先启动 avro-memory-logger.conf 在启动 exec-memory-avro.conf

运行结果：

 

启动 avro-memory-logger.conf的终端会显示如下：