kafka滚动升级

最新推荐文章于 2025-04-17 15:07:26 发布

原创最新推荐文章于 2025-04-17 15:07:26 发布

· 567 阅读

4 ·

版权

文章标签：

#kafka #分布式

生产环境kafka版本为kafka_2.13-2.8.1.tgz，因存在漏洞需将kafka版本升级至kafka_2.13-3.4.0.tgz以上版本。

IP地址	kafka	zookeeper节点
192.168.10.101	broker0	zookeeper
192.168.10.102	broker1	zookeeper
192.168.10.103	broker2	zookeeper

升级前验证zookeeper与kafka状态是否正常，使用kafka命令查看当前kafka集群的topic和版本号

/opt/hadoop/coer/kafka_2.13-2.8.1/bin/kafka-topics.sh --bootstrap-server 192.168.10.101:9092 --list

/opt/hadoop/coer/kafka_2.13-2.8.1/bin/kafka-topics.sh -version

在第一台broker上安装kafka_2.13-3.4.0版本的kafka，并启动新版本的kafka

cd /opt/hadoop/core
tar -zxvf kafka_2.13-3.4.0.tar.gz
cp kafka_2.13-2.8.1/config/server.properties   kafka_2.13-3.4.0/config

cd kafka_2.13-3.4.0/config

vi server.properties
#添加下面这一行配置，这里的版本号，是你的旧版本，不是将要升级的新版本，即是我目前环境的旧版本2.8.1，如果我这里写成我想要升级的新版本3.4.0，那么就会出现最下面的异常报错。
inter.broker.protocol.version=2.8-IV1

#关闭kafka_2.13-2.8.1进程

cd /opt/hadoop/core/kafka_2.13-2.8.1
bin/kafka-server-stop.sh


#启动kafka_2.13-3.4.0进程
cd /opt/hadoop/core/kafka_2.13-3.4.0
bin/kafka-server-start.sh -daemon config/server.properties

#查看是否能正常访问集群
/opt/hadoop/coer/kafka_2.13-3.4.0/bin/kafka-topics.sh --bootstrap-server --describe --topic topicname

#同时可以查看config/server.properties中的配置，查看brokerid
#访问zookeeper查看kafka是否加入集群
zkCli.sh
ls /kafka/broker/ids

#查看日志，正常日志应该如下所示：
[2024-06-21 15:02:48,940] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions HashSet(__consumer_offsets-8, __consumer_offsets-38, hello_test1-2, __consumer_offsets-26, __consumer_offsets-32, __consumer_offsets-2, __consumer_offsets-14, __consumer_offsets-20, __consumer_offsets-44) (kafka.server.ReplicaFetcherManager)
[2024-06-21 15:02:48,958] INFO [GroupCoordinator 0]: Elected as the group coordinator for partition 2 in epoch 28 (kafka.coordinator.group.GroupCoordinator)
[2024-06-21 15:02:48,958] INFO [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from __consumer_offsets-2 for epoch 28 (kafka.coordinator.group.GroupMetadataManager)
[2024-06-21 15:02:48,958] INFO [GroupCoordinator 0]: Elected as the group coordinator for partition 20 in epoch 28 (kafka.coordinator.group.GroupCoordinator)
[2024-06-21 15:02:48,958] INFO [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from __consumer_offsets-20 for epoch 28 (kafka.coordinator.group.GroupMetadataManager)
[2024-06-21 15:02:48,958] INFO [GroupCoordinator 0]: Elected as the group coordinator for partition 38 in epoch 28 (kafka.coordinator.group.GroupCoordinator)
[2024-06-21 15:02:48,958] INFO [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from __consumer_offsets-38 for epoch 28 (kafka.coordinator.group.GroupMetadataManager)




#可能会出下下面这种异常日志：
java.io.IOException: Connection to 2 was disconnected before the response was read
        at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:99)
        at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:113)
        at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:78)
        at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:309)
        at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:124)
        at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:123)
        at scala.Option.foreach(Option.scala:437)
        at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:123)
        at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:106)
        at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:97)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

逐一对3台kafka进行升级，先停止旧版本的kafka，再启动新版本的kafka，完成滚动升级。