本文详细记录了Kafka集群从单节点到3节点的搭建过程，并通过测试验证了其高可用性。在关闭不同节点后，观察到消费者和生产者的行为变化，证明了即使控制器节点失效，集群仍能保持运行并重新选举控制器，但当非控制器节点关闭，导致副本不足时，系统的可用性受到影响。此外，还展示了Flink连接Kafka作为数据源的场景，验证了在节点故障时数据消费的连续性。
kafka集群的搭建以及测试高可用(3节点)
单节点kafka测试
-
下载解压kafka包
[root@node1 kafka1]# pwd /usr/local/soft/kafka1 [root@node1 kafka1]# ll total 44 drwxr-xr-x 3 root root 4096 Apr 22 08:08 bin drwxr-xr-x 3 root root 4096 Apr 25 13:22 config drwxr-xr-x 2 root root 4096 Apr 22 08:08 libs -rw-r--r-- 1 root root 14515 Apr 22 08:08 LICENSE drwxr-xr-x 2 root root 4096 Apr 22 08:08 licenses drwxr-xr-x 2 root root 4096 Apr 25 13:20 logs -rw-r--r-- 1 root root 953 Apr 22 08:08 NOTICE drwxr-xr-x 2 root root 4096 Apr 22 08:08 site-docs -
编辑
config/server.properties, 改如下配置,zookeeper配置的是之前搭建好的zk集群,三个节点############################# Server Basics ############################# # The id of the broker. This must be set to a unique integer for each broker. broker.id=1 ############################# Socket Server Settings ############################# # The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 listeners=PLAINTEXT://node1:9092 ############################# Log Basics ############################# # A comma separated list of directories under which to store log files log.dirs=/tmp/kafka-logs1 ############################# Zookeeper ############################# # Zookeeper connection string (see zookeeper docs for details). # This is a comma separated host:port pairs, each corresponding to a zk # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". # You can also append an optional chroot string to the urls to specify the # root directory for all kafka znodes. zookeeper.connect=node1:2181,node1:2182,node1:2183 -
启动kafka
[root@node1 config]# /usr/local/soft/kafka1/bin/kafka-server-start.sh -daemon /usr/local/soft/kafka1/config/server.properties [root@node1 config]# jps 86883 Jps 72179 QuorumPeerMain 552 ZooKeeperMain 80552 QuorumPeerMain 70040 QuorumPeerMain 86776 Kafka -
创建topic
[root@node1 config]# /usr/local/soft/kafka1/bin/kafka-topics.sh --bootstrap-server node1:9092 --create --topic test --partitions 1 Created topic test. -
启动kafka producer,输入数据
[root@node1 config]# /usr/local/soft/kafka1/bin/kafka-console-producer.sh --bootstrap-server node1:9092 --topic test >first message >second message -
在另一个会话窗口,启动kafka consumer
[root@node1 kafka1]# /usr/local/soft/kafka1/bin/kafka-console-consumer.sh --bootstrap-server node1:9092 --topic test --from-beginning first message second message -
关闭kafka进程,可以看到生产者跟消费这都报错
[root@node1 kafka1]# /usr/local/soft/kafka1/bin/kafka-server-stop.sh [root@node1 kafka1]# /usr/local/soft/kafka1/bin/kafka-console-producer.sh --bootstrap-server node1:9092 --topic test >[2021-04-26 02:39:22,885] WARN [Producer clientId=console-producer] Connection to node 1 (node1/172.17.0.2:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2021-04-26 02:39:22,986] WARN [Producer clientId=console-producer] Connection to node 1 (node1/172.17.0.2:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [root@node1 kafka1]# /usr/local/soft/kafka1/bin/kafka-console-consumer.sh --bootstrap-server node1:9092 --topic test [2021-04-26 02:39:22,977] WARN [Consumer clientId=consumer-console-consumer-27395-1, groupId=console-consumer-27395] Connection to node 1 (node1/172.17.0.2:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2021-04-26 02:39:23,077] WARN [Consumer clientId=consumer-console-consumer-27395-1, groupId=console-consumer-27395] Connection to node 1 (node1/172.17.0.2:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) -
现在启动多个kafka
[root@node1 kafka1]# /usr/local/soft/kafka1/bin/kafka-server-start.sh -daemon /usr/local/soft/kafka1/config/server.properties [root@node1 kafka1]# /usr/local/soft/kafka2/bin/kafka-server-start.sh -daemon /usr/local/soft/kafka2/config/server.properties [root@node1 kafka1]# /usr/local/soft/kafka3/bin/kafka-server-start.sh -daemon /usr/local/soft/kafka3/config/server.properties [root@node1 kafka1]# jps 1312 Jps 1216 Kafka 72179 QuorumPeerMain 565 Kafka 80552 QuorumPeerMain 70040 QuorumPeerMain 99610 Kafka #启动的时候做了个记录,知道各个kafka的进程号 #99610 kafka1 #565 kafka2 #1216 kafka3 -
启动kafka producer
[root@node1 kafka1]# /usr/local/soft/kafka1/bin/kafka-console-producer.sh --bootstrap-server node1:9092,node1:9093,node1:9094 --topic test > -
启动kafka consumer,producer生产数据,可以看到consumer也能消费
[root@node1 kafka1]# /usr/local/soft/kafka1/bin/kafka-console-consumer.sh --bootstrap-server node1:9092,node1:9093,node1:9094 --topic test -
查看哪个kafka是controller
# 查看哪个zk是leader [root@node1 ~]# /usr/local/soft/zookeeper1/bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/soft/zookeeper1/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower [root@node1 ~]# /usr/local/soft/zookeeper2/bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/soft/zookeeper2/bin/../conf/zoo.cfg Client port found: 2182. Client address: localhost. Client SSL: false. Mode: leader [root@node1 ~]# /usr/local/soft/zookeeper3/bin/zkServer.sh status ZooKeeper JMX enabled by default Using config: /usr/local/soft/zookeeper3/bin/../conf/zoo.cfg Client port found: 2183. Client address: localhost. Client SSL: false. Mode: follower # 进入zk leader [root@node1 ~]# /usr/local/soft/zookeeper2/bin/zkCli.sh -server node1:2182 # 查看哪个kafka是controller [zk: node1:2183(CONNECTED) 0] get /controller { "version":1,"brokerid":1,"timestamp":"1619577285564"} -
本地项目启动一个kafka消费者,这里使用flink连接kafka做测试
最低0.47元/天 解锁文章
扫一扫
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
