Flume启动与配置文件

本文介绍了Apache Flume的启动步骤,包括进入Flume根目录,运行run.sh和nohup程序。同时,详细讲解了Flume配置文件的修改,如设置不同类型的source、channel和sink,以及监控文件和Kafka Broker的配置。配置文件中定义了多个tailSrc类型的source,用于监听不同路径的日志文件,并将数据发送到Kafka的不同topic上。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Flume启动:

  1、进入到Flume根目录。(cd flume -> cd flume/flume -> ll )可以看到以下内容

 2、启动run.sh程序。( ./run.sh)。

 3、启动nohup程序。(nohup: appending output to ‘nohup.out’)到此Flume启动成功。

Flume配置文件修改如下:

[laps@hn10 flume]$ vi conf/
flume-conf.properties           flume-conf.properties.template  flume-env.sh.template          
.flume-conf.properties.swp      flume-env.ps1.template          log4j.properties               
[laps@hn10 flume]$ vi conf/
flume-conf.properties           flume-conf.properties.template  flume-env.sh.template          
.flume-conf.properties.swp      flume-env.ps1.template          log4j.properties               
[laps@hn10 flume]$ vi conf/flume-conf.properties

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
         
        
# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'

aK.sources = tailSrc tailSrcRain tailSrcAllC tailSrcRainC
aK.channels = memoryChannel memoryChannelRain memoryChannelAllC memoryChannelRainC
aK.sinks =  kafkaSink kafkaSinkRain kafkaSinkAllC kafkaSinkRainC

   
aK.sources.tailSrc.type = exec
#set monitor file commadnd
aK.sources.tailSrc.command =java -jar ./soft/tailer.jar /home/laps/laps_data/prdinfo/lapsAll/%ty%tm%td.log
aK.sources.tailSrc.channels = memoryChannel
aK.sources.tailSrc.restart = true
aK.sources.tailSrc.restartThrottle = 10000
   
aK.channels.memoryChannel.type = memory
aK.channels.memoryChannel.capacity = 500

aK.sinks.kafkaSink.type = org.apache.flume.sink.kafka.KafkaSink
#set topic
aK.sinks.kafkaSink.topic = laps_all
#set broker list
aK.sinks.kafkaSink.brokerList = 100.100.244.192:9092
aK.sinks.kafkaSink.channel = memoryChannel


aK.sources.tailSrcRain.type = exec
#set monitor file commadnd
aK.sources.tailSrcRain.command =java -jar ./soft/tailer.jar /home/laps/laps_data/prdinfo/lapsRain/%ty%tm%td.log
aK.sources.tailSrcRain.channels = memoryChannelRain
aK.sources.tailSrcRain.restart = true
aK.sources.tailSrcRain.restartThrottle = 10000

aK.channels.memoryChannelRain.type = memory
aK.channels.memoryChannelRain.capacity = 500

aK.sinks.kafkaSinkRain.type = org.apache.flume.sink.kafka.KafkaSink
#set topic
aK.sinks.kafkaSinkRain.topic = laps_rain
#set broker list
aK.sinks.kafkaSinkRain.brokerList = 100.100.244.192:9092
aK.sinks.kafkaSinkRain.channel = memoryChannelRain


aK.sources.tailSrcAllC.type = exec
#set monitor file commadnd
aK.sources.tailSrcAllC.command =java -jar ./soft/tailer.jar /home/laps/laps_data/prdinfo/lapsAllC/%ty%tm%td.log
aK.sources.tailSrcAllC.channels = memoryChannelAllC
aK.sources.tailSrcAllC.restart = true
aK.sources.tailSrcAllC.restartThrottle = 10000

aK.channels.memoryChannelAllC.type = memory
aK.channels.memoryChannelAllC.capacity = 500

aK.sinks.kafkaSinkAllC.type = org.apache.flume.sink.kafka.KafkaSink
#set topic
aK.sinks.kafkaSinkAllC.topic = laps_allc
#set broker list
aK.sinks.kafkaSinkAllC.brokerList = 100.100.244.192:9092
aK.sinks.kafkaSinkAllC.channel = memoryChannelAllC


aK.sources.tailSrcRainC.type = exec
#set monitor file commadnd
aK.sources.tailSrcRainC.command =java -jar ./soft/tailer.jar /home/laps/laps_data/prdinfo/lapsRainC/%ty%tm%td.log
aK.sources.tailSrcRainC.channels = memoryChannelRainC
aK.sources.tailSrcRainC.restart = true
aK.sources.tailSrcRainC.restartThrottle = 10000

aK.channels.memoryChannelRainC.type = memory
aK.channels.memoryChannelRainC.capacity = 500

aK.sinks.kafkaSinkRainC.type = org.apache.flume.sink.kafka.KafkaSink
#set topic
aK.sinks.kafkaSinkRainC.topic = laps_rainc
#set broker list
aK.sinks.kafkaSinkRainC.brokerList = 100.100.244.192:9092
aK.sinks.kafkaSinkRainC.channel = memoryChannelRainC


#aK.sinks.loggerSink.channel = memoryChannel
#aK.sinks.loggerSink.type = logger
~                                                                                                                                                                            
~                                                                                                                                                                            
~                                                  





评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值