kafka 3.1.1 cluster 三个结点 docker-compose 部署

已于 2022-07-11 12:04:21 修改
阅读量1.4k 收藏
点赞数
分类专栏： Java 文章标签： kafka docker zookeeper
于 2022-06-15 00:59:47 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/ppdouble/article/details/124912331
版权
本文档详细介绍了如何在Docker中部署Kafka集群，包括配置网络、工作目录、Zookeeper服务器、监听器以及完整Compose文件。集群包含三个节点，每个节点配置了两种监听器以区分Broker间通信和主机客户端连接。此外，还展示了如何使用Java API以及通过命令行工具验证客户端的生产消费功能。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

文章目录

ENV

镜像 bitnami/kafka 3.1.1
Docker version 19.03.5, build 633a0ea838
Fedora 29 x86_64 workstation

1 示例安装

首先安装 zookeeper 3.8.0 cluster。
然后 使用镜像 bitnami/kafka:3.1.1 安装 kafka。要能够与 zookeeper 访问，可以设在同一个网段。可以与 zookeeper cluster 的任意一个结点连接 KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181 

[hostuser@host-machine]$ docker run --name kafka-single --network myzkcomposeprj_brzk-kafka -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181 bitnami/kafka:3.1.1
kafka 04:51:24.54 
kafka 04:51:24.54 Welcome to the Bitnami kafka container
kafka 04:51:24.54 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-kafka
kafka 04:51:24.54 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-kafka/issues
kafka 04:51:24.54 
kafka 04:51:24.55 INFO  ==> ** Starting Kafka setup **
kafka 04:51:24.63 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kafka 04:51:24.68 INFO  ==> Initializing Kafka...
kafka 04:51:24.70 INFO  ==> No injected configuration files found, creating default config files
kafka 04:51:24.93 INFO  ==> ** Kafka setup finished! **

kafka 04:51:24.96 INFO  ==> ** Starting Kafka **
[2022-05-22 04:51:26,632] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2022-05-22 04:51:27,389] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2022-05-22 04:51:27,551] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2022-05-22 04:51:27,557] INFO starting (kafka.server.KafkaServer)
[2022-05-22 04:51:27,558] INFO Connecting to zookeeper on zoo1:2181 (kafka.server.KafkaServer)
[2022-05-22 04:51:27,585] INFO [ZooKeeperClient Kafka server] Initializing a new session to zoo1:2181. (kafka.zookeeper.ZooKeeperClient)
[2022-05-22 04:51:27,596] INFO Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.ZooKeeper)
[2022-05-22 04:51:27,596] INFO Client environment:host.name=b5125b3f2678 (org.apache.zookeeper.ZooKeeper)
[2022-05-22 04:51:27,596] INFO Client environment:java.version=11.0.15 (org.apache.zookeeper.ZooKeeper)
[2022-05-22 04:51:27,596] INFO Client environment:java.vendor=BellSoft (org.apache.zookeeper.ZooKeeper)
[2022-05-22 04:51:27,596] INFO Client environment:java.home=/opt/bitnami/java (org.apache.zookeeper.ZooKeeper)
...
...

进入容器

[hostuser@host-machine]$ docker exec -it kafka-single bash
I have no name!@b5125b3f2678:/$ whoami
whoami: cannot find name for user ID 1001

bitnami/kafka:3.1.1 镜像里的 kafka 默认的登录的是 id 为 1001 用户。没有名称，在 /etc/passwd里没有记录。显示登录名为 I have no name!

2 集群部署

Broker 是倾向于物理存储概念。每台服务器作为一个Broker。但是一台机器也可以配置多个Broker。
Topic 是逻辑存储概念，其可以分为 partition，partition 可以根据replication factor在不同的服务器 broker 上冗余，增加系统的可用性。具体地 Topic 组织的是 event，event 是将处理对象抽象为事件。

先看 compose 文件：

2.1.1 网络配置

网络使用 zookeeper cluster 所使用的网络。

networks:
    myzkcomposeprj_brzk-kafka:
        external:
            name: myzkcomposeprj_brzk-kafka

2.1.2 设置工作目录

使用 docker exec --iteractive --tty your_container_name bash 进入容器并打开bash后，将默认进入该目录

    working_dir: /opt/bitnami/kafka

2.1.3 配置 zookeeper 服务器

zookeeper 3.8.0 cluster

      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181

2.1.4 设置 listener

kafka multiple listener

      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=BROKERCLIENT:PLAINTEXT,HOSTCLIENT:PLAINTEXT
      - KAFKA_CFG_LISTENERS=BROKERCLIENT://0.0.0.0:9092,HOSTCLIENT://0.0.0.0:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=BROKERCLIENT://kafka1:9092,HOSTCLIENT://localhost:19093
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=BROKERCLIENT

这里设置了两种 listener，用于 broker 之间监听客户端的 BROKERCLIENT，和用于监听 host 客户端的 HOSTCLIENT。这些 listener 的名字是可以随便起的。advertised 起个对客户端声明的作用，客户端以声名的格式访问 broker 则会被成功监听到。每个容器都使用9093监听host客户端，并通过与host的不同端口进行映射。

Broker 之间连接请求流程为：

  1. initial connect 如 kafka2:9092
  2. kafka2 返回 metadata，比如客户端应以 “kafka2:9092” 的方式访问以进行后续操作
  3. 客户端再次 connect 进行 produce 或者 consume 等操作

Broker 与 host之间连接请求流程为：

  1. host 机器 initial connect 如 localhost:19093，并由dorcker forward 到 kafka1 的9093端口
  2. kafka1 返回 metadata，比如host客户端应以 “localhost:19093” 的方式访问以进行后续操作
  3. 客户端再次 connect 进行 produce 或者 consume 等操作

2.1.5 完整 compose file

services:
  kafka-1:
    image: bitnami/kafka:3.1.1
    hostname: kafka1
    container_name: kafka-cluster-1
    working_dir: /opt/bitnami/kafka
    volumes:
      - /home/data/cluster/kafka/kafka-cluster-1:/bitnami/kafka
    ports:
      - '19093:9093'
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_BROKER_ID=21
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=BROKERCLIENT:PLAINTEXT,HOSTCLIENT:PLAINTEXT
      - KAFKA_CFG_LISTENERS=BROKERCLIENT://0.0.0.0:9092,HOSTCLIENT://0.0.0.0:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=BROKERCLIENT://kafka1:9092,HOSTCLIENT://localhost:19093
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=BROKERCLIENT
    extra_hosts:
      - "kafka2:172.19.0.22"
      - "kafka3:172.19.0.23"
    networks:
        myzkcomposeprj_brzk-kafka:
            ipv4_address: 172.19.0.21

  kafka-2:
    image: bitnami/kafka:3.1.1
    hostname: kafka2
    container_name: kafka-cluster-2
    working_dir: /opt/bitnami/kafka
    volumes:
      - /home/data/cluster/kafka/kafka-cluster-2:/bitnami/kafka
    ports:
      - '29093:9093'
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_BROKER_ID=22
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=BROKERCLIENT:PLAINTEXT,HOSTCLIENT:PLAINTEXT
      - KAFKA_CFG_LISTENERS=BROKERCLIENT://0.0.0.0:9092,HOSTCLIENT://0.0.0.0:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=BROKERCLIENT://kafka2:9092,HOSTCLIENT://localhost:29093
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=BROKERCLIENT
    extra_hosts:
      - "kafka1:172.19.0.21"
      - "kafka3:172.19.0.23"
    networks:
        myzkcomposeprj_brzk-kafka:
            ipv4_address: 172.19.0.22


  kafka-3:
    image: bitnami/kafka:3.1.1
    hostname: kafka3
    container_name: kafka-cluster-3
    working_dir: /opt/bitnami/kafka
    volumes:
      - /home/data/cluster/kafka/kafka-cluster-3:/bitnami/kafka
    ports:
      - '39093:9093'
    environment:
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_BROKER_ID=23
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=BROKERCLIENT:PLAINTEXT,HOSTCLIENT:PLAINTEXT
      - KAFKA_CFG_LISTENERS=BROKERCLIENT://0.0.0.0:9092,HOSTCLIENT://0.0.0.0:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=BROKERCLIENT://kafka3:9092,HOSTCLIENT://localhost:39093
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=BROKERCLIENT
    extra_hosts:
      - "kafka1:172.19.0.21"
      - "kafka2:172.19.0.22"
    networks:
        myzkcomposeprj_brzk-kafka:
            ipv4_address: 172.19.0.23

networks:
    myzkcomposeprj_brzk-kafka:
        external:
            name: myzkcomposeprj_brzk-kafka

3 客户端验证

由于这个镜像的 auto.create.topics.enable=true 所以对于不存在的topic会自动创建。
进入 kafka-cluster-3 用 kafka-console-producer.sh 创建event；在 kafka-cluster-1 接收event；在 host 用 kafka-console-consumer.sh 向 localhost:29093 请求接收event
对于没有创建好的 topic，进行 produce 或者 consume 时可能会出现 LEADER_NOT_AVAILABLE 的警告

[hostuser@host-machine kafka$]docker exec --interactive --tty kafka-cluster-3 bash
I have no name!@kafka3:/opt/bitnami/kafka$ bin/kafka-console-producer.sh --broker-list kafka3:9092 --topic test
>first event
>second event
>

[hostuser@host-machine kafka$]docker exec --interactive --tty kafka-cluster-1 bash
I have no name!@kafka1:/opt/bitnami/kafka$ bin/kafka-console-consumer.sh --bootstrap-server kafka1:9092 --topic test --from-beginning
first event
second event

[hostuser@host-machine kafka$]bin/kafka-console-consumer.sh --bootstrap-server localhost:29093 --topic test --from-beginning
first event
second event

补充
当启动 kafka 后，可以看到 zookeeper cluster 写入了很多 zNode，默认只有 [zookeeper] 一个结点

[zk: 172.19.0.13:2181(CONNECTED) 33] ls /                                                                 
[admin, brokers, cluster, config, consumers, controller, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]

当执行一个不存在的 topic ，在发送消息后才会提示 LEADER_NOT_AVAILABLE 并创建该 topic ，然后 produce 消息

[hostuser@host-machine kafka_2.13-3.1.1]$ bin/kafka-console-producer.sh --broker-list localhost:39093 --topic test
>first event
[2022-06-21 11:17:56,660] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2022-06-21 11:17:56,769] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 5 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>

再看 zookeeper 变化

[zk: 172.19.0.13:2181(CONNECTED) 38] get /brokers/topics/test/partitions/0/state                          
{"controller_epoch":1,"leader":23,"version":1,"leader_epoch":0,"isr":[23]}

4 Java api

code on github

Ref

kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc
kafka-listeners-explained
connect-to-kafka-running-in-docker
docker-kafka-w-python-consumer/52440056#52440056
kafka docker-compose-yml
bitnami-docker-kafka
使用Docker快速搭建Zookeeper和kafka集群 cnblogs
run-producer-consumer
apache kafka quickstart
Docker containerized Kafka services youtube
how-to-create-topics-in-apache-kafka
leader-not-available-kafka-in-console-producer
how-to-fix-the-leader_not_available-error-in-kafka
How to Connect to Apache Kafka running in Docker(multiple scenarios) youtube
running-a-multi-broker-apache-kafka-cluster-on-a-single-node
running-kafka-cluster-with-multiple-brokers-on-your-local-machine
set-up-a-kafka-cluster-in-local-with-multiple-kafka-brokers
github.com/everpeace compose yaml example

评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值