flume 安装

王志龙

墨衍会员 · AI 创作全网分发码龄15年

关注

原创最新推荐文章于 2026-02-07 00:26:34 发布 696 阅读 AI 写同款· GEO 优化 ›

Flume安装配置本任务需要使用root用户完成相关配置，已安装Hadoop及需要配置前置环境，具体要求如下：阅读详情

flume 安装

本文的flume是指新版flume，即flume-ng。

1.下载flume安装包：

flume是Apache的顶级项目，所以可以到Apache的官网下载，我这里下载的是1.5.0.1版本。

链接：http://archive.apache.org/dist/flume/

2.直接解压：

[hadoop@zeroOneMan-nosql ~]$ tar -zxvf apache-flume-1.5.0.1-bin.tar.gz

解压后我嫌它名字长，又把它重命名成flume-1.5.0.1-bin，解压后目录如下：

[hadoop@zeroOneMan-nosql ~]$ ll
total 24764
-rw-r--r-- 1 hadoop hadoop 25319586 Nov 26 10:03 apache-flume-1.5.0.1-bin.tar.gz
drwxrwxr-x 7 hadoop hadoop     4096 Nov 26 10:21 flume-1.5.0.1-bin

3. 配置flume，这里我先用Avro源，因为在我们公司做的实时计算项目也是用这种Avro方式的。

[hadoop@zeroOneMan-nosql ~]$ cd flume-1.5.0.1-bin/conf/

[hadoop@zeroOneMan-nosql conf]$ vi avro.conf 

a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 4141

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
~
~

4.启动flume：

[hadoop@zeroOneMan-nosql flume-1.5.0.1-bin]$ bin/flume-ng agent --conf conf --conf-file conf/avro.conf --name a1 -Dflume.root.logger=INFO,console
Info: Sourcing environment configuration script /home/hadoop/flume-1.5.0.1-bin/conf/flume-env.sh
+ exec /opt/jdk1.7.0_71/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/home/hadoop/flume-1.5.0.1-bin/conf:/home/hadoop/flume-1.5.0.1-bin/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file conf/avro.conf --name a1
2014-11-26 14:56:05,265 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2014-11-26 14:56:05,270 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:conf/avro.conf
2014-11-26 14:56:05,278 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: k1 Agent: a1
2014-11-26 14:56:05,279 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2014-11-26 14:56:05,279 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:k1
2014-11-26 14:56:05,292 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [a1]
2014-11-26 14:56:05,292 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2014-11-26 14:56:05,302 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel c1 type memory
2014-11-26 14:56:05,309 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel c1
2014-11-26 14:56:05,311 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source r1, type avro
2014-11-26 14:56:05,342 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: k1, type: logger
2014-11-26 14:56:05,348 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel c1 connected to [r1, k1]
2014-11-26 14:56:05,356 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: { bindAddress: 0.0.0.0, port: 4141 } }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@31a209df counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
2014-11-26 14:56:05,368 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel c1
2014-11-26 14:56:05,369 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:160)] Waiting for channel: c1 to start. Sleeping for 500 ms
2014-11-26 14:56:05,411 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
2014-11-26 14:56:05,414 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: CHANNEL, name: c1 started
2014-11-26 14:56:05,871 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink k1
2014-11-26 14:56:05,872 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source r1
2014-11-26 14:56:05,875 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:220)] Starting Avro source r1: { bindAddress: 0.0.0.0, port: 4141 }...
2014-11-26 14:56:06,184 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
2014-11-26 14:56:06,184 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)] Component type: SOURCE, name: r1 started
2014-11-26 14:56:06,185 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:245)] Avro source r1 started.

看到最后一行“Avro source r1 started”的日志了吗？呵呵，flume已经成功启动了。

5.测试flume：

先在本地新建个日志文件，往里面写些数据。

另外打开一个新终端，操作如下：

[hadoop@zeroOneMan-nosql ~]$ echo "Hello World" > service.log


[hadoop@zeroOneMan-nosql ~]$ flume-1.5.0.1-bin/bin/flume-ng avro-client -c . -H localhost -p 4141 -F service.log

执行上面这个操作后，回到我们刚才启动的flume的终端，在flume的启动日志下会看到service.log的文件内容。

2014-11-26 15:03:08,131 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 48 65 6C 6C 6F 20 57 6F 72 6C 64                Hello World }

好了，我们已经成功安装flume，并且可以收集外面数据。

收录于