Kraft模式安装Kafka(含常规、容器两种安装方式)、Kafka镜像下载

docker-compose的方式搭建 kafka KRaft 模式集群 如果您觉得有用的话,记得给,写作不易啊^ _ ^。而且听说,实在白嫖的话,那欢迎常来啊!!! 阅读详情
AI摘要

本文详细介绍了Apache Kafka 3.9.0在Kraft模式下的三种部署方式:常规本地安装(含Cluster UUID生成与日志目录初始化)、Docker容器化部署(含镜像拉取、自定义网络及Kraft参数配置),以及Kubernetes集群部署(通过ConfigMap、Deployment和Service编排)。重点突出Kraft模式替代ZooKeeper的核心配置差异与启动流程,适用于单机开发与轻量生产环境。

 一、#创作灵感#

公司使用Kafka的软件项目较多,故写技术笔记巩固知识要点

二、软件环境

- Kafka 3.9.0  官方下载地址:Kafka 3.9.0

- Docker Desktop 4.37 容器图形化工具  官方下载地址:Docker Desktop 4.37

特别说明

- Docker Desktop仅仅是Windows下镜像用,如有能爬楼的服务器可不安装此软件

- 本文以单机模式为例

三、常规方式安装

非容器化Kraft Mode(模式)安装

1、下载Kafka

参见上面的软件环境章节的下载地址

2、修改Kafka配置

# Kraft模式启动

vi $KAFKA_HOME/config/kraft/server.properties

关键设置包括

# 指定此节点是否应该成为控制平面的一部分
process.roles=controller,kafka
 
# KRaft模式下的节点ID,每个节点的ID必须唯一
node.id=1
 
# 监听器地址,用于集群内通信
listeners=PLAINTEXT://localhost:9092
 
# 控制平面监听器地址
controller.listeners=PLAINTEXT://localhost:9090
 
# 用于集群通信的安全协议以及相关的认证和授权配置
inter.broker.listener.name=CONTROLLER
 
# 控制平面的ADVERTISED.LISTENERS,用于其他节点发现
controller.listener.names=CONTROLLER
controller.advertised.listeners=PLAINTEXT://localhost:9090
 
# 用于KRaft模式的元数据复制因子,通常设置为集群中控制器的数量
metadata.max.age=300000
 
# 日志存储配置,指定日志的位置和格式
log.dirs=/tmp/kraft-logs
log.retention.hours=168
 
# 控制平面的选举配置和集群成员配置
control.plane.listener.name=CONTROLLER
control.plane.advertised.listeners=PLAINTEXT://localhost:9090
 
# 集群的初始/bootstrap状态
process.roles=controller
node.id=1
listeners=CONTROLLER://localhost:9090
inter.broker.listener.name=CONTROLLER

3、Kraft模式启动Kafka

Using downloaded files

Generate a Cluster UUID

$ KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

Format Log Directories

$ bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties

Start the Kafka Server

# Start the Kafka broker service
$ bin/kafka-server-start.sh config/kraft/server.properties

四、docker方式安装

容器化Kraft Mode(模式)安装

1、容器启动Kafka

- 下载镜像

下载一体化官方镜像

$ docker pull confluentinc/cp-kafka:7.0.1

windows下载镜像(内网安装)方式见作者其他docker安装其他组件的文章,如:docker安装MySQL8:docker离线安装MySQL、docker在线安装MySQL、MySQL镜像下载、MySQL配置、MySQL命令-优快云博客 

-  创建 Docker 网络

docker network create kafka-network

# docker查看所有网络
docker network ls

- (Kraft模式)使用镜像启动容器

 docker run -d \
  --name=kafka \
  --network=kafka-network \
  -e KAFKA_KRAFT_MODE=true \
  -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
  -v $(pwd)/kafka.properties:/etc/kafka/kafka.properties \
  -e KAFKA_CONFIG_FILE=/etc/kafka/kafka.properties \
  -p 2181:2181 \
  -p 9092:9092 \
  -p 19092:19092 \
  confluentinc/cp-kafka:7.0.1

这条命令的各个部分功能如下:

  • -d: 在后台运行容器。
  • --name=cp-kafka: 为容器命名为的cp-kafka。
  • --network=kafka-network: 将容器连接到之前创建的网络。
  • -e KAFKA_KRAFT_MODE=true: 启用KRaft模式。
  • -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092: 设置监听地址。
  • -v $(pwd)/kafka.properties:/etc/kafka/kafka.properties: 将工作目录下的kafka.properties文件挂载到容器中。
  • -e KAFKA_CONFIG_FILE=/etc/kafka/kafka.properties: 指定使用的Kafka配置文件。
  • confluentinc/cp-kafka:latest: 使用最新的Confluent Kafka镜像。
  • -p 做端口映射

#   ***********  ~~~~~   重点:上面的命令很多版本不好使   ~~~~~   **********

下面的命令针对3.8.1亲测可用,没错Kraft 就是这么简单,启动成功就完事了,就有服务了~

docker run \
 --name=smet-kafka \
 -v /data1/kafka/docker-kafka/log:/kafka-logs \
 -p 19092:9092 \
 -p 19093:9093 \
 -d 1756e53f57e1

按官方的说法,不增加任何配置,也可以直接启动

[docker@GZ***-PM153160 ~]$ docker images
REPOSITORY                                 TAG       IMAGE ID       CREATED        SIZE
apache/kafka                               3.8.1     1756e53f57e1   3 months ago   385MB
[docker@GZ***-PM153160 ~]$ docker run -p 9092:9092 c2b75a96e17e
ZooKeeper JMX enabled by default
Using config: /conf/zoo.cfg
2025-01-27 03:55:46,842 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@177] - Reading configuration from: /conf/zoo.cfg
2025-01-27 03:55:46,845 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@431] - clientPort is not set
2025-01-27 03:55:46,845 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@444] - secureClientPort is not set
2025-01-27 03:55:46,845 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@460] - observerMasterPort is not set
2025-01-27 03:55:46,845 [myid:] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@477] - metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider
2025-01-27 03:55:46,851 [myid:] - ERROR [main:o.a.z.s.q.QuorumPeerConfig@702] - Invalid configuration, only one server specified (ignoring)
2025-01-27 03:55:46,852 [myid:1] - INFO  [main:o.a.z.s.DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2025-01-27 03:55:46,852 [myid:1] - INFO  [main:o.a.z.s.DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2025-01-27 03:55:46,852 [myid:1] - INFO  [main:o.a.z.s.DatadirCleanupManager@101] - Purge task is not scheduled.
2025-01-27 03:55:46,853 [myid:1] - WARN  [main:o.a.z.s.q.QuorumPeerMain@139] - Either no config or no quorum defined in config, running in standalone mode
2025-01-27 03:55:46,854 [myid:1] - INFO  [main:o.a.z.j.ManagedUtil@46] - Log4j 1.2 jmx support not found; jmx disabled.
2025-01-27 03:55:46,854 [myid:1] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@177] - Reading configuration from: /conf/zoo.cfg
2025-01-27 03:55:46,855 [myid:1] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@431] - clientPort is not set
2025-01-27 03:55:46,855 [myid:1] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@444] - secureClientPort is not set
2025-01-27 03:55:46,855 [myid:1] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@460] - observerMasterPort is not set
2025-01-27 03:55:46,855 [myid:1] - INFO  [main:o.a.z.s.q.QuorumPeerConfig@477] - metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider
2025-01-27 03:55:46,855 [myid:1] - ERROR [main:o.a.z.s.q.QuorumPeerConfig@702] - Invalid configuration, only one server specified (ignoring)
2025-01-27 03:55:46,855 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServerMain@123] - Starting server
2025-01-27 03:55:46,865 [myid:1] - INFO  [main:o.a.z.s.ServerMetrics@64] - ServerMetrics initialized with provider org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@4550bb58
2025-01-27 03:55:46,867 [myid:1] - INFO  [main:o.a.z.s.a.DigestAuthenticationProvider@51] - ACL digest algorithm is: SHA1
2025-01-27 03:55:46,867 [myid:1] - INFO  [main:o.a.z.s.a.DigestAuthenticationProvider@65] - zookeeper.DigestAuthenticationProvider.enabled = true
2025-01-27 03:55:46,869 [myid:1] - INFO  [main:o.a.z.s.p.FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false
2025-01-27 03:55:46,877 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] - 
2025-01-27 03:55:46,877 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] -   ______                  _                                          
2025-01-27 03:55:46,877 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] -  |___  /                 | |                                         
2025-01-27 03:55:46,877 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] -     / /    ___     ___   | | __   ___    ___   _ __     ___   _ __   
2025-01-27 03:55:46,877 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] -    / /    / _ \   / _ \  | |/ /  / _ \  / _ \ | '_ \   / _ \ | '__|
2025-01-27 03:55:46,877 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] -   / /__  | (_) | | (_) | |   <  |  __/ |  __/ | |_) | |  __/ | |    
2025-01-27 03:55:46,877 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] -  /_____|  \___/   \___/  |_|\_\  \___|  \___| | .__/   \___| |_|
2025-01-27 03:55:46,877 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] -                                               | |                     
2025-01-27 03:55:46,877 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] -                                               |_|                     
2025-01-27 03:55:46,878 [myid:1] - INFO  [main:o.a.z.ZookeeperBanner@42] - 
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:zookeeper.version=3.9.3-c26634f34490bb0ea7a09cc51e05ede3b4e320ee, built on 2024-10-17 23:21 UTC
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:host.name=501da54a316f
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:java.version=17.0.13
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:java.vendor=Eclipse Adoptium
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:java.home=/opt/java/openjdk
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:java.class.path=/apache-zookeeper-3.9.3-bin/bin/../zookeeper-metrics-providers/zookeeper-prometheus-metrics/target/classes:/apache-zookeeper-3.9.3-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.9.3-bin/bin/../build/classes:/apache-zookeeper-3.9.3-bin/bin/../zookeeper-metrics-providers/zookeeper-prometheus-metrics/target/lib/*.jar:/apache-zookeeper-3.9.3-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.9.3-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/zookeeper-prometheus-metrics-3.9.3.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/zookeeper-jute-3.9.3.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/zookeeper-3.9.3.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/snappy-java-1.1.10.5.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/slf4j-api-1.7.30.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/simpleclient_servlet-0.9.0.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/simpleclient_hotspot-0.9.0.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/simpleclient_common-0.9.0.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/simpleclient-0.9.0.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-transport-native-unix-common-4.1.113.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-transport-native-epoll-4.1.113.Final-linux-x86_64.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-transport-classes-epoll-4.1.113.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-transport-4.1.113.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-tcnative-classes-2.0.66.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-tcnative-boringssl-static-2.0.66.Final-windows-x86_64.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-tcnative-boringssl-static-2.0.66.Final-osx-x86_64.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-tcnative-boringssl-static-2.0.66.Final-osx-aarch_64.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-tcnative-boringssl-static-2.0.66.Final-linux-x86_64.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-tcnative-boringssl-static-2.0.66.Final-linux-aarch_64.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-tcnative-boringssl-static-2.0.66.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-resolver-4.1.113.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-handler-4.1.113.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-common-4.1.113.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-codec-4.1.113.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/netty-buffer-4.1.113.Final.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/metrics-core-4.1.12.1.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/logback-core-1.2.13.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/logback-classic-1.2.13.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jline-2.14.6.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jetty-util-ajax-9.4.56.v20240826.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jetty-util-9.4.56.v20240826.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jetty-servlet-9.4.56.v20240826.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jetty-server-9.4.56.v20240826.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jetty-security-9.4.56.v20240826.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jetty-io-9.4.56.v20240826.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jetty-http-9.4.56.v20240826.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jackson-databind-2.15.2.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jackson-core-2.15.2.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/jackson-annotations-2.15.2.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/commons-io-2.17.0.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/commons-cli-1.5.0.jar:/apache-zookeeper-3.9.3-bin/bin/../lib/audience-annotations-0.12.0.jar:/apache-zookeeper-3.9.3-bin/bin/../zookeeper-*.jar:/apache-zookeeper-3.9.3-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf:
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:java.io.tmpdir=/tmp
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:java.compiler=<NA>
2025-01-27 03:55:46,879 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:os.name=Linux
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:os.arch=amd64
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:os.version=4.19.90-2102.2.0.0062.ctl2.x86_64
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:user.name=zookeeper
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:user.home=/home/zookeeper
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:user.dir=/apache-zookeeper-3.9.3-bin
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:os.memory.free=981MB
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:os.memory.max=1000MB
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.Environment@98] - Server environment:os.memory.total=1000MB
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@149] - zookeeper.enableEagerACLCheck = false
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@162] - zookeeper.digest.enabled = true
2025-01-27 03:55:46,880 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@166] - zookeeper.closeSessionTxn.enabled = true
2025-01-27 03:55:46,881 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@2242] - zookeeper.serializeLastProcessedZxid.enabled = true
2025-01-27 03:55:46,881 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@1585] - zookeeper.flushDelay = 0 ms
2025-01-27 03:55:46,881 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@1594] - zookeeper.maxWriteQueuePollTime = 0 ms
2025-01-27 03:55:46,881 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@1603] - zookeeper.maxBatchSize=1000
2025-01-27 03:55:46,881 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@287] - zookeeper.intBufferStartingSizeBytes = 1024
2025-01-27 03:55:46,882 [myid:1] - INFO  [main:o.a.z.s.BlueThrottle@141] - Weighed connection throttling is disabled
2025-01-27 03:55:46,882 [myid:1] - INFO  [main:o.a.z.s.AuthenticationHelper@66] - zookeeper.enforce.auth.enabled = false
2025-01-27 03:55:46,883 [myid:1] - INFO  [main:o.a.z.s.AuthenticationHelper@67] - zookeeper.enforce.auth.schemes = []
2025-01-27 03:55:46,883 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@1395] - minSessionTimeout set to 4000 ms
2025-01-27 03:55:46,883 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@1404] - maxSessionTimeout set to 40000 ms
2025-01-27 03:55:46,884 [myid:1] - INFO  [main:o.a.z.s.ResponseCache@45] - getData response cache size is initialized with value 400.
2025-01-27 03:55:46,885 [myid:1] - INFO  [main:o.a.z.s.ResponseCache@45] - getChildren response cache size is initialized with value 400.
2025-01-27 03:55:46,885 [myid:1] - INFO  [main:o.a.z.s.u.RequestPathMetricsCollector@109] - zookeeper.pathStats.slotCapacity = 60
2025-01-27 03:55:46,886 [myid:1] - INFO  [main:o.a.z.s.u.RequestPathMetricsCollector@110] - zookeeper.pathStats.slotDuration = 15
2025-01-27 03:55:46,886 [myid:1] - INFO  [main:o.a.z.s.u.RequestPathMetricsCollector@111] - zookeeper.pathStats.maxDepth = 6
2025-01-27 03:55:46,886 [myid:1] - INFO  [main:o.a.z.s.u.RequestPathMetricsCollector@112] - zookeeper.pathStats.initialDelay = 5
2025-01-27 03:55:46,886 [myid:1] - INFO  [main:o.a.z.s.u.RequestPathMetricsCollector@113] - zookeeper.pathStats.delay = 5
2025-01-27 03:55:46,886 [myid:1] - INFO  [main:o.a.z.s.u.RequestPathMetricsCollector@114] - zookeeper.pathStats.enabled = false
2025-01-27 03:55:46,888 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@1622] - The max bytes for all large requests are set to 104857600
2025-01-27 03:55:46,888 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@1636] - The large request threshold is set to -1
2025-01-27 03:55:46,888 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@385] - Created server with tickTime 2000 ms minSessionTimeout 4000 ms maxSessionTimeout 40000 ms clientPortListenBacklog -1 dataLogdir /datalog/version-2 snapdir /data/version-2
2025-01-27 03:55:46,905 [myid:1] - INFO  [main:o.e.j.u.l.Log@170] - Logging initialized @481ms to org.eclipse.jetty.util.log.Slf4jLog
2025-01-27 03:55:46,961 [myid:1] - WARN  [main:o.e.j.s.h.ContextHandler@1662] - o.e.j.s.ServletContextHandler@52af26ee{/,null,STOPPED} contextPath ends with /*
2025-01-27 03:55:46,961 [myid:1] - WARN  [main:o.e.j.s.h.ContextHandler@1673] - Empty contextPath
2025-01-27 03:55:46,978 [myid:1] - INFO  [main:o.e.j.s.Server@375] - jetty-9.4.56.v20240826; built: 2024-08-26T17:15:05.868Z; git: ec6782ff5ead824dabdcf47fa98f90a4aedff401; jvm 17.0.13+11
2025-01-27 03:55:47,001 [myid:1] - INFO  [main:o.e.j.s.s.DefaultSessionIdManager@334] - DefaultSessionIdManager workerName=node0
2025-01-27 03:55:47,001 [myid:1] - INFO  [main:o.e.j.s.s.DefaultSessionIdManager@339] - No SessionScavenger set, using defaults
2025-01-27 03:55:47,002 [myid:1] - INFO  [main:o.e.j.s.s.HouseKeeper@132] - node0 Scavenging every 660000ms
2025-01-27 03:55:47,006 [myid:1] - WARN  [main:o.e.j.s.ConstraintSecurityHandler@759] - ServletContext@o.e.j.s.ServletContextHandler@52af26ee{/,null,STARTING} has uncovered http methods for path: /*
2025-01-27 03:55:47,015 [myid:1] - INFO  [main:o.e.j.s.h.ContextHandler@921] - Started o.e.j.s.ServletContextHandler@52af26ee{/,null,AVAILABLE}
2025-01-27 03:55:47,027 [myid:1] - INFO  [main:o.e.j.s.AbstractConnector@333] - Started ServerConnector@67080771{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2025-01-27 03:55:47,027 [myid:1] - INFO  [main:o.e.j.s.Server@415] - Started @603ms
2025-01-27 03:55:47,027 [myid:1] - INFO  [main:o.a.z.s.a.JettyAdminServer@201] - Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands
2025-01-27 03:55:47,031 [myid:1] - INFO  [main:o.a.z.s.ServerCnxnFactory@172] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2025-01-27 03:55:47,032 [myid:1] - WARN  [main:o.a.z.s.ServerCnxnFactory@337] - maxCnxns is not configured, using default value 0.
2025-01-27 03:55:47,033 [myid:1] - INFO  [main:o.a.z.s.NIOServerCnxnFactory@652] - Configuring NIO connection handler with 10s sessionless connection timeout, 4 selector thread(s), 80 worker threads, and 64 kB direct buffers.
2025-01-27 03:55:47,034 [myid:1] - INFO  [main:o.a.z.s.NIOServerCnxnFactory@660] - binding to port /0.0.0.0:2181
2025-01-27 03:55:47,045 [myid:1] - INFO  [main:o.a.z.s.w.WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2025-01-27 03:55:47,046 [myid:1] - INFO  [main:o.a.z.s.w.WatchManagerFactory@42] - Using org.apache.zookeeper.server.watch.WatchManager as watch manager
2025-01-27 03:55:47,046 [myid:1] - INFO  [main:o.a.z.s.ZKDatabase@134] - zookeeper.snapshotSizeFactor = 0.33
2025-01-27 03:55:47,046 [myid:1] - INFO  [main:o.a.z.s.ZKDatabase@154] - zookeeper.commitLogCount=500
2025-01-27 03:55:47,051 [myid:1] - INFO  [main:o.a.z.s.p.SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED
2025-01-27 03:55:47,051 [myid:1] - INFO  [main:o.a.z.s.p.FileTxnSnapLog@480] - Snapshotting: 0x0 to /data/version-2/snapshot.0
2025-01-27 03:55:47,054 [myid:1] - INFO  [main:o.a.z.s.ZKDatabase@291] - Snapshot loaded in 8 ms, highest zxid is 0x0, digest is 1371985504
2025-01-27 03:55:47,055 [myid:1] - INFO  [main:o.a.z.s.p.FileTxnSnapLog@480] - Snapshotting: 0x0 to /data/version-2/snapshot.0
2025-01-27 03:55:47,055 [myid:1] - INFO  [main:o.a.z.s.ZooKeeperServer@589] - Snapshot taken in 0 ms
2025-01-27 03:55:47,062 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::o.a.z.s.PrepRequestProcessor@138] - PrepRequestProcessor (sid:0) started, reconfigEnabled=false
2025-01-27 03:55:47,062 [myid:1] - INFO  [main:o.a.z.s.RequestThrottler@75] - zookeeper.request_throttler.shutdownTimeout = 10000 ms
2025-01-27 03:55:47,076 [myid:1] - INFO  [main:o.a.z.s.ContainerManager@83] - Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0
2025-01-27 03:55:47,077 [myid:1] - INFO  [main:o.a.z.a.ZKAuditProvider@42] - ZooKeeper audit is disabled.

五、k8s方式安装

k8s环境容器化Kraft Mode(模式)安装

1、容器启动Kafka

 - 创建ConfigMap资源

用于定义Kafka配置

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-config
data:
  config.properties: |
    process.roles=broker
    node.id=1
    listeners=PLAINTEXT://:9092
    inter.broker.listener.name=PLAINTEXT
    ...

- 创建Deployment资源

用于部署Kafka Pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
        - name: kafka
          image: confluentinc/cp-kafka:7.0.1
          ports:
            - containerPort: 9092
          env:
            - name: CONFIG_FILE
              value: "/etc/kafka/conf/config.properties"
            ...
          volumeMounts:
            - name: config-volume
              mountPath: /etc/kafka/conf
      volumes:
        - name: config-volume
          configMap:
            name: kafka-config

- 创建Service资源

用于对外暴露Kafka服务

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
spec:
  selector:
    app: kafka
  ports:
    - protocol: TCP
      port: 9092
      targetPort: 9092

- 应用到你的Kubernetes集群

这些YAML文件定义了Kafka的配置、部署和服务。你需要将它们保存为文件,然后使用kubectl 应用到你的Kubernetes集群

kubectl apply -f configmap.yaml
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

- (或)使用镜像启动容器

# 假设打好的镜像id是fe7e6a65afa5
docker run \
--name=cp-kafka \
-p 2181:2181 \
-p 9092:9092 \
-p 19092:19092 \
-d fe7e6a65afa5

六、要点总结

      - Zookeeper Mode使用的配置是$KFAKA_HOME/config/server.properties

      - Kraft Mode使用的配置是$KFAKA_HOME/config/kraft/server.properties

      - Zookeeper Mode启动见作者的文章:Zookeeper模式安装Kafka

Linux 下使用 Docker-Compose 安装 Kafka 和 Kafka-UI(KRaft 模式) 通过 Kafka-UI 可以可视化查看消息流转情况,maven 依赖版本最好保持和 kafka 一致或高于 kafka 的版本。搭建本地 Kafka 环境,通过 Kafka-UI 可视化管理工具进行消息查看和调试的过程。集成在 ShenYu 插件中的 Kafka 消费者代码如下,用于日志订阅消费。创建的,Kafka 容器内的进程(通常为 UID 1000)无权限写入。是我从容器中搞出来的完整配置,按需修改端口和 ip, 我将端口改成了。Kafka 和 Kafka-UI 的完整部署配置。 阅读详情

相关推荐

docker-compose→kafka镜像制作,kraft集群容器编排一键启动

我开始也是用stars最多的镜像也就是bitnami/kafka,但是我发现这个镜像在做kraft集群的时候有问题,就是它没办法选择配置文件启动,一直用的和zookeeper一起启动的配置文件启动,然后镜像我又没办法控制,所以我干脆就自己写了一个cluster.id可以到容器中去生成,现在先启动,不用管生成好了更新到配置文件,重启容器即可生成命令每个kraft中都有两个配置文件meta.properties和server.properties-d: 后台启动。

u010711867的博客 684

kafka安装包下载慢,国内镜像下载地址

kafka tar安装包快速下载,国内镜像地址

LCGGood的博客 1万+

Kafka 环境搭建和使用之单机模式详细教程

但随着 Kafka 的发展,尤其是从 Kafka 2.8 版本开始,引入了 Kafka Raft 协议(也称为 KRaft 模式),这是一个重大的架构变革,它允许 Kafka 在没有依赖外部 ZooKeeper 的情况下运行。: 在生产环境中,Kafka 和 Zookeeper 会部署在不同的服务器上,形成真正的分布式集群。运行控制台生成器客户端,在主题中写入一些事件。:当消费者群组的成员发生变化(例如消费者加入或离开群组)时,Kafka会触发再均衡过程,重新分配分区给群组中的消费者,确保消息的公平消费。

zp8126的专栏 1440

kafka_2.12-3.2.0安装包快速下载地址分享

kafka_2.12-3.2.0安装包快速下载地址分享,新需求是kafka版本升级,然后确定版本之后,在kafka的官网上,选择了版本之后,就开始下载,但是呢,速度是贼慢哦。得

请叫我大师兄 4461

【云原生技术】国内网页下载kafka的镜像去哪里下载?

• 这几个国内仓库的页面都支持“复制 Pull 命令”和查看镜像层大小,但并不会直接给 .tar 压缩包下载按钮——镜像规范如此设计。 • 只想下载 Kafka 二进制包(非容器)的话,请直接访问 Apache 官网下载 kafka_*.tgz。 • 需要 Docker Compose、KRaft 单机示例或生产集群部署手册,随时告诉我。

weixin_46453070的博客 3005

【云原生】Docker 搭建Kafka服务两种方式实战操作详解

Docker 搭建kafka服务两种方式操作详解

congge_study的博客 3596

保姆级教程Docker部署KRaft模式的Kafka官方镜像

本文以 KRaft 模式为例,介绍了使用 Docker 部署 Kafka 单机或集群的方法,包括拉取镜像、配置方式、单节点部署、集群部署,以及可视化工具的部署。

BigCookies的博客 1万+

kafka:Apache Kafka的镜像

阿帕奇·卡夫卡 有关该项目的详细信息,请参见我们的。 您需要安装 。 我们使用Java 8、11和15构建和测试Apache Kafka。我们将javac和scalac中的release参数设置为8以确保生成的二进制文件与Java 8或更高版本(独立于用于编译的Java版本)兼容。 默认情况下使用Scala 2.13,有关如何使用其他Scala版本或所有受支持的Scala版本的信息,请参见下文。 建造一个罐子并运行它 ./gradlew jar 按照说明进行操作 构建源jar ./gradlew srcJar 构建聚合的Javadoc ./gradlew aggregatedJava

单机Docker部署应用Kraft模式的Kafka集群

以ubuntu系统为基础,封装自己的镜像,使用单台主机模拟三台主机部署kraft模式的Kafka集群。

sky的博客 3809

kafka 3.4.1 镜像,阿里云仓库

kafka 3.4.1 镜像,阿里云仓库

CANoe进阶实战(第十一章)-LIN节点测试调度

《CANoe快速入门实战:LIN节点调度开发》摘要 本文是CANoe系列教程的第十一章,重点讲解如何通过CAPL编程实现LIN报文的灵活调度控制。文章采用"输入-输出-过程"三步骤法,详细介绍了调度表函数linChangeSchedTable()、启动/停止函数linStartScheduler()/linStopDisturbance()等关键API的使用方法。作者创新性地通过系统变量在Test节点间接控制Network节点的调度功能,并提供了完整的节点代码实现方案。最后的Trace窗

weixin_53605885的博客 1025

Docker 安装 kafka (bitnami/kafka:4.0)

这条命令启动了一个基于 Bitnami 提供的 Kafka 4.0 镜像的容器,并在 KRaft 模式(即不依赖 Zookeeper)下同时担任 controller 和 broker。它设置了时区、节点 ID、监听器及广告地址,指定数据和日志存储目录,限制了文件句柄数、内存和 CPU 使用,并在主机上映射了对应端口,保证容器重启策略为“始终重启”。命令的整体概览,然后逐行详解各个参数和环境变量的作用。

m0_51431003的博客 5365

基于物联网技术智慧园林绿化养护管理信息平台

近几年来,随着经济不断的发展,城市面积不断的扩大,城市管理中的园林绿化工作也在这种趋势下不断的发生着深刻改变。管养企业是否按制定的管养计划严格执行?绿化事件整改是否及时到位?绿地面积是否在稳步增长?管养的效果如何?如何客观评价管养企业的管养工作?这些都是我们各级管理者需要面对的问题。   智慧园林绿化养护管理信息平台的开发思路以“考核”为核心,整个信息平台体现出“新、快、高、全、享”五个特点。   智慧园林绿化养护管理信息平台的整个开发理念为:依托五大技术手段(全球卫星定位系统技术、地理信...

Cesium的博客QQ2252224326 2461

docker 部署 Kafka 单机和集群

如果未安装,可以访问 Docker 官网下载并安装 Docker Desktop(Windows 和 Mac)或使用包管理器安装(Linux)。

菜鸡的博客 3996

解决PubMed卡在“正在检查浏览器”页面的完整排查指南

DNS解析是将域名转换为IP地址的基础网络服务,其稳定性和准确性直接影响网站访问。当DNS服务器响应缓慢或返回异常结果时,可能导致客户端与服务器建立连接时出现偏差,从而触发网站的安全验证机制,例如Cloudflare等安全服务提供的浏览器环境检查。这一机制旨在防御恶意流量,但偶尔也会误拦截正常访问。在生物医学信息检索领域,PubMed作为核心文献数据库,其稳定访问对科研工作流至关重要。针对因DNS或本地缓存问题导致的‘正在检查浏览器’页面卡顿,通过刷新DNS缓存、更换为Google Public DNS或C

weixin_29597551的博客 3832

NLPIR 汉语分词系统 (PyNLPIR) 学习手札

1 关于NLPIR最近在做一些NLP相关的任务,其中一个关键的子任务就是如何对汉语进行分词?关于分词的方式,有很多很多,也是很多学习者的练手任务(比如我们当时的期末作业就有这个)。但是,轮子还是要搬来用,自己写的分词只能说是去了解这个方式(了解某个算法),做出来的都是能看不能用,毕竟模型、算法、语料库等一座座大山等着我们攻克。所以,我找了一些资料,看到一个不错的教程,他用了NLPIR,所以我也跟着使

MebiuW的专栏 2万+

Docker for windows 下 Kafka(kraft模式) 安装部署

KAFKA_KRAFT_CLUSTER_ID=iZWRiSqjZAlYwlKEqHFQWI #哪一,三个节点保持一致。# 定义kafka服务端socket监听端口(Docker内部的ip地址和端口)#定义外网访问地址(宿主机ip地址和端口。# 设置broker最大内存,和初始内存。# 标红处修改为自己主机IP。#标红处修改为自己主机IP。# 允许使用Kraft。,标红处修改为自己主机IP。标红处修改为自己的路径。标红处修改为自己的路径。标红处修改为自己的路径。

m0_61913190的博客 2878

如何实现MCGS与S7-200SMART PLC以太网多台通信控制?

第九步:(设备工具箱—设备管理—鼠标依次点击通用TCP/IP父设备—西门子_ Smart200到设备窗口里面去)数据类型:第0位(指PLC地址变量小数点后面的位数,例如我地址是M0.0,那么在数据类型应填写通道的第00位)第十步:(鼠标双击打开通用TCPIP父设备0--[通用TCP/IP父设备]进入通信配置)通道地址:0(指小数点前面起始字节的地址,例如我地址是M0.0,那么应在通道地填写0)。第七步:(选择通用TCP/IP父设备,点击新增,右侧选定设备就会自动添加进去。

u010670857的博客 6720

kubesphere部署单机版kafka4.0.0,使用Kraft模式

参考链接:https://juejin.cn/post/7380421216019365897。value: 'PLAINTEXT://外网地址:30684'

Sunnyingx的博客 1499

GEE生物量预测:利用多源遥感数据和随机森林算法RF构建地上生物量(AGB)预测模型,并实现未来年份的生态变化模拟(大面积生物量反演)

数据融合策略:整合5类遥感产品(AGB、NDVI、LAI、土地覆盖、地形)时序特征构建:采用生长季中值法降低云污染影响机器学习优化分层采样保证数据代表性80棵决策树平衡精度与效率预测可靠性基准期R²达0.76需结合地面实测数据校正。

此星光明博客 419

Oracle 从入门到精通系列(入门篇)1.1-Oracle 数据库发展史

Oracle 从入门到精通系列(入门篇)1.1-Oracle 数据库发展史

LuciferLiu_DBA 2191

Kafka容器化实战:KRaft多监听器与网络拓扑重建指南

Apache Kafka 作为分布式流处理核心,其容器化并非简单打包镜像,而是对通信模型、元数据协调和网络地址协商的系统性重构。理解 Kafka 的双向地址协商机制(listeners 与 advertised.listeners)是解决连接失败、元数据不可达等高频问题的基础;KRaft 模式通过内嵌 Raft 协议替代 ZooKeeper,显著降低容器编排复杂度,成为现代 Kafka 部署的事实标准。技术价值在于实现跨环境一致的可复现部署、CI/CD 快速集成及生产级弹性扩缩。典型应用场景涵盖本地开发调试

aibi9196的博客 740

2024年 Flutter 面试题大全(2024.6.28持续更新中)

2024年 Flutter 面试题大全(持续更新中)

Code Metaverse 6671

手拉手安装Kafka2.13发送和消费消息

手拉手安装Kafka2.13发送和消费消息

weixin_47268883的博客 1570

jdk-8u144-linux-x64.tar.gz.

该资源为搭建Linux系统Java环境用,资源的后缀为tar.gz格式的压缩包

NTscan扫描工具

ntscan是一个扫描器,可以扫描没有密码和弱密码的共享,先到配置文件中填写猜测的用户名和密码,扫描过程中右侧出现的是扫描日志。

容器化部署Kafka的最佳实践:基于KRaft模式的无ZooKeeper方案

可以参考这篇。

srebro.cn | 运维小弟 796

老版QQ原生表情gif动态

这是老版的QQ表情,不过全是动态的的,Windows照片查看器是不会动的,可以右击用浏览器打开查看,再说一遍是老版的,只是分享给宁愿用老版也要动态的人。

本机搭建kafka单机版不需要依赖zookeeper

本机搭建kafka单机版不需要依赖zookeeper

woxiangmanman的博客 4217

OV9734 DATASHEET,完整规格书

OV9734完整规格书,包括电气参数,尺寸规格,寄存器配置表等

上一篇: Zookeeper模式安装Kafka(含常规、容器两种安装方式)
下一篇: GitLab集成GitLab-Runner详细版--及注意事项汇总【最佳实践】
飞火流星02027
博客等级 码龄19年 783粉丝 121原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

飞火流星02027

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值