ubuntu14.+ 下 Flume+kafka+storm的配置

最新推荐文章于 2025-03-18 13:59:48 发布

原创最新推荐文章于 2025-03-18 13:59:48 发布 · 727 阅读

0 ·

CC 4.0 BY-SA版权

实时系统专栏收录该内容

1 篇文章

订阅专栏

本文档详细介绍了如何配置Flume进行日志收集、Kafka作为消息中间件及Storm进行实时数据处理。涵盖了环境搭建、配置及基本操作等关键步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一、配置flume(apache-flume-1.6.0-src.tar.gz)

1.解压

tar -xvf apache-flume-1.6.0-src.tar.gz

2.配置环境变量

vi ~/.bash_profile

export FLUME_HOME=/home/shaka/flume/flume-1.6.0
export PATH=$PATH:$FLUME_HOME/sbin

source ~/.bash_profile

3.启动/关闭flume

1)在conf下编写文件如flume-source-conf.properties

2)启动flume start source

3)关闭flume stop source

二、配置kafka(kafka_2.9.2-0.8.1.1)

1) 根据zookeeper的conf/zoo.cfg

server.1=10.1.73.8:4887:5887

2)配置config下的server.properties

zookeeper.connect=localhost:2181

3)相关操作

    开启：bin/kafka-server-start.sh config/server.properties &
    查看：bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181
    创建：bin/kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic gaoyujie
    删除：bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --topic taoyujie --zookeeper 127.0.0.1:2181
    生产者：bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic-003
    消费者：bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic topic-003
    生产者发送信息，消费者接收
    开两个终端！

三、配置storm

1.配置zookeeper(zookeeper-3.3.6)

1)获取本地ip：ifconfig

2)修改conf/zoo.cfg

      # The number of milliseconds of each tick
          tickTime=2000
      # The number of ticks that the initial
      # synchronization phase can take
          initLimit=10
      # The number of ticks that can pass between
      # sending a request and getting an acknowledgement
          syncLimit=5
      # the directory where the snapshot is stored.
        dataDir=/home/shaka/zookeeper/zookeeper-3.3.6/data
     # the port at which the clients will connect
       clientPort=2181
     # set logs
       dataLogDir=/home/shaka/zookeeper/zookeeper-3.3.6/logs
     # set server
        server.1=10.1.73.8:4887:5887
     #server.2=10.162.219.52:4887:5887
   #server.3=10.163.15.119:4887:5887

   # add by shaka
   # set max client connects
        maxClientCnxns=300

3)启动/关闭

启动服务bin/zkServer.sh start

启动客户端bin/zkCli.sh -server 127.0.0.1:2181

bin/zkServer.sh status

2.配置依赖环境dependence(事先安装gcc-c++)

1)zeromq-4.0.4

①解压

②./configure

③make(root)

④make install(root)

2) jzmq

①解压

②./configure

③make(root)

④make install(root)

3.配置storm(apache-storm-0.9.2-incubating)

1)配置环境变量

vi ~/.bash_profile

export STORM_HOME=/home/shaka/storm/apache-storm-0.9.2-incubating
export PATH=$PATH:$STORM_HOME/bin

source ~/.bash_profile

2)配置文件

vi storm.yaml

        storm.zookeeper.servers:
      - "10.2.73.8"
# - "10.162.219.52"
        #
        nimbus.host: "10.2.73.8"
        storm.local.dir: "/home/shaka/storm/apache-storm-0.9.2-incubating/topology"
   #ui.port: "18080"
   supervisor.slots.ports:
         - 6700
       - 6701
                 - 6702

4.启动

strom nimbus &

strom supervisor &

strom ui &

5.操作

查看ui:http://localhost:8080

正在进行的任务:storm list

停止任务:strom kill word-count

运行任务:storm jar dist/topology-0.0.1.jar starter.WordCountTopology