实验四 综合数据流处理-Storm （单机和集群配置部分）

已于 2024-12-24 21:24:05 修改
阅读量782 收藏 11
点赞数 14
分类专栏： 大数据学习 文章标签： storm 大数据 linux
于 2024-12-23 21:44:33 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/m0_74216612/article/details/144677700
版权

1.前期准备

（1）把docker和docker-compose给下载好

参考：基于docker-compose来搭建zookeeper集群-优快云博客（注意对于这篇文章下面配置zookeeper的内容，可以直接跳过，因为我们只需要看最上面下载docker-compose的部分）

在linux虚拟机上面配置Docker时所踩过的坑_containerd1.6.33镜像源怎么配置-优快云博客

（2）创建好相关目录，并且准备好zookeeper.yml文件，具体代码如下：

mkdir -p /export/server/allstorm/
mkdir -p /export/server/stormdocker/
cd /export/server/
vi zookeeper.yml

配置storm单机使用的目录是stormdocker

配置storm集群使用的目录是allstorm

zookeeper.yml内容

version: '3.0'
services:
 zk_server1:
  image: zookeeper:3.7.1 # 镜像名称
  restart: always # 当发生错误时自动重启
  container_name: zk-node1
  ports: # 端口
   - 2181:2181
  volumes: # 挂载数据卷
   - ./node1/data:/data # 若./node1/data目录不存在，则自动生成
   - ./node1/datalog:/datalog # 若./node1/datalog目录不存在，则自动生成
  environment:
   TZ: Asia/Shanghai
   ZOO_MY_ID: 1 # 节点ID
   ZOO_PORT: 2181 # zookeeper端口号
   ZOO_SERVERS: server.1=172.168.0.1:2888:3888;2181 server.2=172.168.0.2:2888:3888;2181 server.3=172.168.0.3:2888:3888;2181 # zookeeper节点列表
  networks:
   mynet: # 自定义网络名
    ipv4_address: 172.168.0.1

 zk_server2:
  image: zookeeper:3.7.1
  restart: always
  container_name: zk-node2
  ports:
   - 2182:2181
  volumes:
   - ./node2/data:/data
   - ./node2/datalog:/datalog
  environment:
   TZ: Asia/Shanghai
   ZOO_MY_ID: 2
   ZOO_PORT: 2181
   ZOO_SERVERS: server.1=172.168.0.1:2888:3888;2181 server.2=172.168.0.2:2888:3888;2181 server.3=172.168.0.3:2888:3888;2181
  networks:
   mynet:
    ipv4_address: 172.168.0.2

 zk_server3:
  image: zookeeper:3.7.1
  restart: always
  container_name: zk-node3
  ports:
   - 2183:2181
  volumes:
   - ./node3/data:/data
   - ./node3/datalog:/datalog
  environment:
   TZ: Asia/Shanghai
   ZOO_MY_ID: 3
   ZOO_PORT: 2181
   ZOO_SERVERS: server.1=172.168.0.1:2888:3888;2181 server.2=172.168.0.2:2888:3888;2181 server.3=172.168.0.3:2888:3888;2181
  networks:
   mynet:
    ipv4_address: 172.168.0.3

networks:
 mynet: # 自定义网络名
  external: true

2.单机storm配置

准备好相关文件：one_storm.yml,storm.yaml

（1）首先创建文件one_storm.yml，代码如下

cd /export/server/
vi one_storm.yml

one_storm.yml的内容如下

version: '3'
services:
  nimbus:
    image: storm
    restart: always
    command: storm nimbus
    container_name: nimbus
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
    ports:
      - "6627:6627"
    networks:
      - storm-network
      - mynet
  supervisor:
    image: storm
    restart: always
    command: storm supervisor
    container_name: supervisor
    depends_on:
      - nimbus
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
      STORM_NIMBUS_HOST: "nimbus"
    links:
      - nimbus:nimbus
    ports:
      - "6700:6700"
      - "6701:6701"
      - "6702:6702"
      - "6703:6703"
      - "8000:8000"
    networks:
      - storm-network
      - mynet

  ui:
    image: storm
    restart: always
    command: storm ui
    container_name: ui
    depends_on:
      - nimbus
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
      STORM_NIMBUS_HOST: "nimbus"
    links:
      - nimbus:nimbus
    ports:
      - "8080:8080"
    networks:
      - storm-network
      - mynet

networks:
  storm-network:
  mynet:
    external: true

（2）接着创建文件：storm.yaml，代码如下

cd /export/server/stormdocker/
vi storm.yaml

storm.yaml内容如下：

storm.zookeeper.servers:
  - "zk_server1"
  - "zk_server2"
  - "zk_server3"

# Nimbus 种子节点列表,用于集群中的其他守护进程发现 Nimbus
nimbus.seeds:
  - "nimbus"

# Storm 日志目录
storm.log.dir: "/logs"

# Storm 本地存储目录
storm.local.dir: "/data"

# 消息总线服务的主机名
mbus.host: "nimbus"

# Nimbus 服务的 Thrift 端口
nimbus.thrift.port: 6627

（3）启动docker，然后启动zookeeper.yml，然后启动one_storm.yml，再利用docker cp 代码进行相关操作

cd /export/server/
systemctl start docker
docker network create --driver bridge --subnet=172.168.0.0/16 --gateway=172.168.1.1 mynet
docker-compose -f zookeeper.yml up -d
docker-compose -f one_storm.yml up -d
docker cp /export/server/stormdocker/storm.yaml nimbus:/conf/storm.yaml
docker cp /export/server/stormdocker/storm.yaml supervisor:/conf/storm.yaml
docker cp /export/server/stormdocker/storm.yaml ui:/conf/storm.yaml
docker restart nimbus
docker restart ui
docker restart supervisor

（4）最后进入网页 主机号:8080查看

我的主机号是192.168.88.101

192.168.88.101:8080

效果如下（网页要一直点刷新按钮，等一会儿才能够进去）

 （5）退出代码

cd /export/server/
docker-compose -f one_storm.yml down
docker-compose -f zookeeper.yml down

3.storm集群配置

准备好相关文件：many_storm.yml,storm.yaml

（1）首先创建文件many_storm.yml，代码如下

cd /export/server/
vi many_storm.yml

many_storm.yml的内容如下

version: '3'
services:
  nimbus1:
    image: storm
    restart: always
    command: storm nimbus
    container_name: nimbus1
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
    ports:
      - "6627:6627"
    networks:
      - storm-network
      - mynet
    volumes:
      - /export/server/allstorm/storm.yaml:/conf/storm.yaml

  supervisor1:
    image: storm
    restart: always
    command: storm supervisor
    container_name: supervisor1
    depends_on:
      - nimbus1
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
      STORM_NIMBUS_HOST: "nimbus1"
    links:
      - nimbus1:nimbus1
    ports:
      - "6700:6700"
      - "6701:6701"
      - "6702:6702"
      - "6703:6703"
    networks:
      - storm-network
      - mynet
    volumes:
      - /export/server/allstorm/storm.yaml:/conf/storm.yaml

  ui1:
    image: storm
    restart: always
    command: storm ui
    container_name: ui1
    depends_on:
      - nimbus1
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
      STORM_NIMBUS_HOST: "nimbus1"
    links:
      - nimbus1:nimbus1
    ports:
      - "8080:8080"
    networks:
      - storm-network
      - mynet
    volumes:
      - /export/server/allstorm/storm.yaml:/conf/storm.yaml

  nimbus2:
    image: storm
    restart: always
    command: storm nimbus
    container_name: nimbus2
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
      STORM_NIMBUS_HOST: "nimbus1"
    networks:
      - storm-network
      - mynet
    volumes:
      - /export/server/allstorm/storm.yaml:/conf/storm.yaml

  supervisor2:
    image: storm
    restart: always
    command: storm supervisor
    container_name: supervisor2
    depends_on:
      - nimbus2
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
      STORM_NIMBUS_HOST: "nimbus1"
    links:
      - nimbus2:nimbus2
    ports:
      - "6704:6700"
      - "6705:6701"
      - "6706:6702"
      - "6707:6703"
    networks:
      - storm-network
      - mynet
    volumes:
      - /export/server/allstorm/storm.yaml:/conf/storm.yaml

  nimbus3:
    image: storm
    restart: always
    command: storm nimbus
    container_name: nimbus3
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
      STORM_NIMBUS_HOST: "nimbus1"
    networks:
      - storm-network
      - mynet
    volumes:
      - /export/server/allstorm/storm.yaml:/conf/storm.yaml

  supervisor3:
    image: storm
    restart: always
    command: storm supervisor
    container_name: supervisor3
    depends_on:
      - nimbus3
    environment:
      STORM_ZOOKEEPER_SERVERS: "zk_server1:2181,zk_server2:2181,zk_server3:2181"
      STORM_NIMBUS_HOST: "nimbus1"
    links:
      - nimbus3:nimbus3
    ports:
      - "6708:6700"
      - "6709:6701"
      - "6710:6702"
      - "6711:6703"
    networks:
      - storm-network
      - mynet
    volumes:
      - /export/server/allstorm/storm.yaml:/conf/storm.yaml

networks:
  storm-network:
  mynet:
    external: true

（2）接着创建文件：storm.yaml，代码如下

cd /export/server/allstorm/
vi storm.yaml

storm.yaml内容如下：

storm.zookeeper.port: 2181

# Zookeeper 服务器列表
storm.zookeeper.servers:
  - "zk_server1"
  - "zk_server2"
  - "zk_server3"

# Nimbus 种子节点列表,用于集群中的其他守护进程发现 Nimbus
nimbus.seeds:
  - "nimbus1"
  - "nimbus2"
  - "nimbus3"

# Storm 日志目录
storm.log.dir: "/logs"

# Storm 本地存储目录
storm.local.dir: "/data"

# 消息总线服务的主机名
mbus.host: "nimbus1"

# Nimbus 服务的 Thrift 端口
nimbus.thrift.port: 6627

ui:
  host: 0.0.0.0
  port: 8080

supervisor.slots.ports:
  - 6700
  - 6701
  - 6702
  - 6703

（3）启动docker，然后启动zookeeper.yml，然后启动many_storm.yml，

确保网络mynet已经存在

可以通过命令查看

docker network ls

546540d18c2d   mynet                  bridge    local

如果没有就要输入以下代码创建网络mynet

docker network create --driver bridge --subnet=172.168.0.0/16 --gateway=172.168.1.1 mynet

创建好了网络mynet后 

如果docker没有启动的话

systemctl start docker

 启动集群

cd /export/server/
docker-compose -f zookeeper.yml up -d
docker-compose -f many_storm.yml up -d

（4）最后进入网页 主机号:8080查看

我的主机号是192.168.88.101

192.168.88.101:8080

 （5）退出代码

cd /export/server/
docker-compose -f many_storm.yml down
docker-compose -f zookeeper.yml down

4.案例实现（敬请期待......）

好了，到了这里配置基本上就OK了

评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值