使用docker快速搭建kafka集群

最新推荐文章于 2025-03-11 10:09:36 发布

kk3909

最新推荐文章于 2025-03-11 10:09:36 发布

阅读量2.1k

点赞数

分类专栏：快速部署中间件容器文章标签： kafka zookeeper

本文链接：https://blog.youkuaiyun.com/kk3909/article/details/111937717

版权

容器同时被 2 个专栏收录

16 篇文章

订阅专栏

快速部署中间件

14 篇文章

订阅专栏

环境准备

节点名称	ip地址
node1	192.168.130.20
node2	192.168.130.19
node2	192.168.130.21

安装docker

略

安装zookeeper集群

参考：https://blog.youkuaiyun.com/kk3909/article/details/111937681

安装kafka

环境变量配置参考:https://hub.docker.com/r/bitnami/kafka

创建数据目录

mkdir -p /root/kafka/data && chmod 777 /root/kafka/data
mkdir -p /root/kafka/config

node1

docker rm -f kf-node1

docker run -d --network=host --privileged=true \
--name kf-node1 -p 9092:9092 \
-e KAFKA_BROKER_ID=1 \
-e KAFKA_ZOOKEEPER_CONNECT=192.168.130.20:2181,192.168.130.19:2181,192.168.130.21:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.130.20:9092 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-v /root/kafka/data:/bitnami/kafka/data \
-v /root/kafka/config:/bitnami/kafka/config \
bitnami/kafka:2.6.0

node2

docker rm -f kf-node2

docker run -d --network=host --privileged=true \
--name kf-node2 -p 9092:9092 \
-e KAFKA_BROKER_ID=2 \
-e KAFKA_ZOOKEEPER_CONNECT=192.168.130.20:2181,192.168.130.19:2181,192.168.130.21:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.130.19:9092 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-v /root/kafka/data:/bitnami/kafka/data \
-v /root/kafka/config:/bitnami/kafka/config \
bitnami/kafka:2.6.0

node3

docker rm -f kf-node3

docker run -d --network=host --privileged=true \
--name kf-node3 -p 9092:9092 \
-e KAFKA_BROKER_ID=3 \
-e KAFKA_ZOOKEEPER_CONNECT=192.168.130.20:2181,192.168.130.19:2181,192.168.130.21:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.130.21:9092 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-v /root/kafka/data:/bitnami/kafka/data \
-v /root/kafka/config:/bitnami/kafka/config \
bitnami/kafka:2.6.0

创建和查看topic

进入 node1

docker exec -it kf-node1 sh

创建一个TestTopic主题

/opt/bitnami/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.130.20:2181 --replication-factor 3 --partitions 5 --topic TestTopic

输出信息

Created topic TestTopic.

查看TestTopic主题

/opt/bitnami/kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.130.20:2181 --topic TestTopic

输出信息

Topic: TestTopic	PartitionCount: 5	ReplicationFactor: 3	Configs:
	Topic: TestTopic	Partition: 0	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: TestTopic	Partition: 1	Leader: 1	Replicas: 1,3,2	Isr: 1,3,2
	Topic: TestTopic	Partition: 2	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: TestTopic	Partition: 3	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
	Topic: TestTopic	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3

检查集群状态

进入node2,查看topic是否同步

docker exec -it kf-node2 sh

/opt/bitnami/kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.130.19:2181 --topic TestTopic

Topic: TestTopic	PartitionCount: 5	ReplicationFactor: 3	Configs:
	Topic: TestTopic	Partition: 0	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: TestTopic	Partition: 1	Leader: 1	Replicas: 1,3,2	Isr: 1,3,2
	Topic: TestTopic	Partition: 2	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: TestTopic	Partition: 3	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
	Topic: TestTopic	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3

进入node3,查看topic是否同步

docker exec -it kf-node3 sh

/opt/bitnami/kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.130.21:2181 --topic TestTopic

Topic: TestTopic	PartitionCount: 5	ReplicationFactor: 3	Configs:
	Topic: TestTopic	Partition: 0	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: TestTopic	Partition: 1	Leader: 1	Replicas: 1,3,2	Isr: 1,3,2
	Topic: TestTopic	Partition: 2	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: TestTopic	Partition: 3	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
	Topic: TestTopic	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3

消息发送测试

node1 生产消息

docker exec -it kf-node1 sh

/opt/bitnami/kafka/bin/kafka-console-producer.sh --broker-list 192.168.130.20:9092 --topic TestTopic

输入消息

>hello1
>hello2

node2 消费消息

docker exec -it kf-node2 sh

/opt/bitnami/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.130.19:9092 --topic TestTopic --from-beginning

等待接收消息

node3 消费消息

docker exec -it kf-node3 sh

/opt/bitnami/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.130.21:9092 --topic TestTopic --from-beginning

等待接收消息

安装kafka-manager

zk 用于存储集群等配置信息

docker rm -f kf-manager
docker run -d --name kf-manager -p 9000:9000 -e ZK_HOSTS="192.168.130.20:2181,192.168.130.19:2181,192.168.130.21:2181" -e APPLICATION_SECRET=kf123 sheepkiller/kafka-manager:1.3.1.8