flume学习（三）

flume使用实例

实例一

说明

监听客户端的连接然后打印

使用

netcat source---------》监听端口
memory channel  ----》数据通道
logger sink-----------》打印监听到的数据

配置

• 在conf中新建agent目录
  mkdir -p conf/agent
• 在agent目录中新建example-net-mem-log.conf
  touch example-net-mem-log.conf

• 在example-net-mem-log.conf中配置source channel sink

  a1.sources = r1
  a1.channels = c1 
  a1.sinks = k1
  
  a1.sources.r1.type = netcat                              
  a1.sources.r1.bind = dev-hadoop-single.com               
  a1.sources.r1.port = 4444
  
  a1.channels.c1.type                  = memory
      #channel 能够容纳的event数量
  a1.channels.c1.capacity              = 1000
      #channel 一个事务中允许的最大的evnet数量
  a1.channels.c1.transactionCapacity = 100                 
  
  a1.sinks.k1.type = logger
  
  a1.sources.r1.channels = c1                              
  a1.sinks.k1.channel = c1   

  测试

  • 启动flume
    $ flume-ng agent --conf /opt/modules/apache-flume-1.5.0-cdh5.3.6-bin/conf/agent/ --conf-file /opt/modules/apache-flume-1.5.0-cdh5.3.6-bin/conf/agent/example-net-mem-log.conf --name a1 -Dflume.root.logger=INFO,console
  • 使用telnet发送消息
    
$ telnet dev-hadoop-single.com 4444
Trying 192.168.56.101...
Connected to dev-hadoop-single.com.
Escape character is '^]'.
test
OK
message
OK

    flume输出如下
    
16/10/19 18:40:23 INFO sink.LoggerSink: Event: { headers:{} body: 74 65 73 74 0D test. }
16/10/19 18:40:26 INFO sink.LoggerSink: Event: { headers:{} body: 6D 65 73 73 61 67 65 0D message. }


实例二

说明

Nginx + Flume + Hdfs

使用

exec source     1个
memory channel  1个
hdfs sink       1个

配置

flume配置

agent.sources = r1
agent.channels = c1
agent.sinks = k1

##
agent.sources.r1.channels = c1
agent.sinks.k1.channel    = c1

#exec source
agent.sources.r1.type          = exec
agent.sources.r1.command       = tail -F /home/hadoop/access.log
agent.sources.r1.selector.type = replicating


#memory channel
agent.channels.c1.type                = memory
agent.channels.c1.capacity            = 1000
agent.channels.c1.transactionCapacity = 100 
#agent.channels.c1.byteCapacityBufferPercentage = 60
#agent.channels.c1.byteCapacity        = 12800000000
agent.channels.c1.keep-alive          = 60


agent.sinks.k1.type = hdfs
agent.sinks.k1.hdfs.path = hdfs://dev-hadoop-single.com:8020/flume/events-01/%Y-%m-%d
agent.sinks.k1.hdfs.fileType=DataStream
#default:FlumeData
agent.sinks.k1.hdfs.filePrefix = log-spool
agent.sinks.k1.hdfs.fileShuffix = .log
agent.sinks.k1.hdfs.minBlockReplicas = 1 

flume测试

启动flume

$ flume-ng agent --conf /opt/modules/apache-flume-1.5.0-cdh5.3.6-bin/conf/agent/ --conf-file /opt/modules/apache-flume-1.5.0-cdh5.3.6-bin/conf/agent/example-exec-mem-hdfs.conf --name agent -Dflume.root.logger=INFO,console

写入数据
access_log >> access.log
测试结果

16/10/19 19:03:46 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026338.tmp
16/10/19 19:05:36 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026338.tmp
16/10/19 19:05:36 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026338.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026338
16/10/19 19:05:36 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026339.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026339.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026339.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026339
16/10/19 19:05:37 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026340.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026340.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026340.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026340
16/10/19 19:05:37 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026341.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026341.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026341.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026341
16/10/19 19:05:37 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026342.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026342.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026342.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026342
16/10/19 19:05:37 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026343.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026343.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026343.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026343
16/10/19 19:05:37 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026344.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026344.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026344.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026344
16/10/19 19:05:37 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026345.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026345.tmp
16/10/19 19:05:37 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026345.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026345
16/10/19 19:05:38 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026346.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026346.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026346.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026346
16/10/19 19:05:38 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026347.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026347.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026347.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026347
16/10/19 19:05:38 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026348.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026348.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026348.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026348
16/10/19 19:05:38 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026349.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026349.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026349.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026349
16/10/19 19:05:38 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026350.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026350.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026350.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026350
16/10/19 19:05:38 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026351.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026351.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026351.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026351
16/10/19 19:05:38 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026352.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026352.tmp
16/10/19 19:05:38 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026352.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026352
16/10/19 19:05:38 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026353.tmp
16/10/19 19:05:39 INFO hdfs.BucketWriter: Closing hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026353.tmp
16/10/19 19:05:39 INFO hdfs.BucketWriter: Renaming hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026353.tmp to hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026353
16/10/19 19:05:39 INFO hdfs.BucketWriter: Creating hdfs://dev-hadoop-single.com:8020/flume/events-01/2016-10-19/log-spool.1476875026354.tmp