docker-compose 搭建 kafka集群

最新推荐文章于 2025-06-03 10:11:39 发布

转载最新推荐文章于 2025-06-03 10:11:39 发布 · 542 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/reaperhero/articles/10781288.html

文章标签：

#大数据 #运维 #网络

本文详细介绍如何使用Docker和docker-compose快速搭建一个包含两个节点的Kafka集群，包括配置文件、启动命令、主题管理及消息消费等关键操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

搭建集群

docker-compose up -d --scale kafka=2        本机启动一个有两个节点的 Kafka 集群

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092"             # kafka 把9092端口随机映射到主机的端口
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.199.163           #本机ip
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: test:1:1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

输出结果

 chen@macos  ~/docker/kafka01 sudo docker ps                                                                                                                                                   
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
4ede7dc26df7        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   22 minutes ago      Up 22 minutes       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafka01_zookeeper_1
2f7c0a71af03        wurstmeister/kafka       "start-kafka.sh"         22 minutes ago      Up 22 minutes       0.0.0.0:32772->9092/tcp                              kafka01_kafka_2
e0b72d9466e0        wurstmeister/kafka       "start-kafka.sh"         22 minutes ago      Up 22 minutes       0.0.0.0:32771->9092/tcp                              kafka01_kafka_1 

注意：docker-compose up -d --scale kafka=2        # 可以进行扩展或缩容

基础操作

列出主题

容器内部查询主题
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper kafka01_zookeeper_1:2181          #可以通过容器的名字或者IP访问
Topic:test  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: test Partition: 0    Leader: 1002    Replicas: 1002  Isr: 1002
容器外部或内部部查询主题
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh  --describe --zookeeper 192.168.199.163:2181          #这里因为zookeeper映射到了主题的2181端口
Topic:test  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: test Partition: 0    Leader: 1002    Replicas: 1002  Isr: 1002

创建主题

bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh  --create --topic chat --partitions 3 --zookeeper 192.168.199.163:2181 --replication-factor 2
Created topic chat.
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper kafka01_zookeeper_1:2181
Topic:chat  PartitionCount:3    ReplicationFactor:2 Configs:
    Topic: chat Partition: 0    Leader: 1002    Replicas: 1002,1001 Isr: 1002,1001
    Topic: chat Partition: 1    Leader: 1001    Replicas: 1001,1002 Isr: 1001,1002
    Topic: chat Partition: 2    Leader: 1002    Replicas: 1002,1001 Isr: 1002,1001
Topic:test  PartitionCount:1    ReplicationFactor:1 Configs:
    Topic: test Partition: 0    Leader: 1002    Replicas: 1002  Isr: 1002

消息消费

生产者－容器里面
bash-4.4# ./kafka-console-producer.sh --broker-list localhost:9092 --topic chat
>da
>

消费者－容器里面
/opt/kafka_2.12-2.2.0/bin # ./kafka-console-consumer.sh  --bootstrap-server localhost:9092 --topic chat --from-beginning
da

注意：上面要在宿主机上只要将localhost:9092换为kafka暴露在主机上端口即可

转载于:https://www.cnblogs.com/reaperhero/articles/10781288.html