storm统计服务开启zookeeper、kafka 、Storm（sasl认证）

置顶

平凡的运维之路

已于 2024-01-18 11:44:38 修改

阅读量1.3k

点赞数 17

文章标签：程序人生

于 2024-01-18 11:40:17 首次发布

本文链接：https://blog.youkuaiyun.com/zhao138969/article/details/135670251

版权

部署storm统计服务开启zookeeper、kafka 、Storm（sasl认证）

当前测试验证结果：
- 单独配置zookeeper 支持acl 设置用户和密码，在storm不修改代码情况下和kafka支持
- 当kafka 开启ACL时，storm 和ccod模块不清楚配置用户和密码。
  - 使用python脚本连接kafka用户和密码是能成功发送消息。
当前部署环境服务版本

服务IP	模块	版本信息
10.130.41.42	zookeeper	zookeeper-3.6.3
10.130.41.43	kafka	kafka_2.11-2.3.1
10.130.41.44	storm	apache-storm-1.2.4

zookeeper部署

部署mongodb_1服务器的zookeeper
传安装部署包和配置文件

[root@mongodb_1 ~]# su - storm
[storm@mongodb_1 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be  storm_node1.tar.gz
[storm@mongodb_1 ~]$ tar xvf storm_node1.tar.gz 
[storm@mongodb_1 ~]$ cd  storm_node1
[storm@mongodb_1 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_1 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

修改zookeeper配置文件,然后在启动

[storm@mongodb_1 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_1 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_1:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_1 conf]$ vim  zk_server_jaas.conf
Server {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="zk_cluster"
    password="zk_cluster_passwd"
    user_admin="Admin@123";
};

#拷贝kafka acl lib
[storm@mongodb_1 zookeeper-3.6.3]$  cd /home/storm/zookeeper-3.6.3
[storm@mongodb_1 zookeeper-3.6.3]$  mkdir zk_sasl_dependencies
[storm@mongodb_1 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_1 libs]$ cp kafka-clients-2.3.1.jar  lz4-java-1.6.0.jar  slf4j-api-1.7.26.jar  snappy-java-1.1.7.3.jar  ~/zookeeper-3.6.3/lib/
[storm@mongodb_1 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_1 ~]$ echo "1" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_1 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_1 bin]$ vim zkEnv.sh  #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
do
    CLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_1 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_1 bin]$ ./zkServer.sh  status 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

部署mongodb_2服务器的zookeeper
上传安装部署包和配置文件

[root@mongodb_2 ~]# su - storm
[storm@mongodb_2 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be storm_node2.tar.gz
[storm@mongodb_2 ~]$ tar xvf storm_node2.tar.gz 
[storm@mongodb_2 ~]$ cd  storm_node2
[storm@mongodb_2 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_2 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

修改zookeeper配置文件,然后在启动

[storm@mongodb_2 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_2 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_2:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_2 conf]$ vim  zk_server_jaas.conf
Server {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="zk_cluster"
    password="zk_cluster_passwd"
    user_admin="Admin@123";
};

#拷贝kafka acl lib
[storm@mongodb_2 zookeeper-3.6.3]$  cd /home/storm/zookeeper-3.6.3
[storm@mongodb_2 zookeeper-3.6.3]$  mkdir zk_sasl_dependencies
[storm@mongodb_2 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_2 libs]$ cp kafka-clients-2.3.1.jar  lz4-java-1.6.0.jar  slf4j-api-1.7.26.jar  snappy-java-1.1.7.3.jar  ~/zookeeper-3.6.3/lib/
[storm@mongodb_2 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_2 ~]$ echo "1" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_2 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_2 bin]$ vim zkEnv.sh  #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3