kafka集群扩容后rebalance详细步骤

kafka运维–集群扩容后手动Rebalance topic

1.配置好需要Rebalance 的topic 列表
topicmove.conf
[plain] view plain copy

{"topics": [{"topic": "topic1"},  
             {"topic": "topic2"}],  
 "version":1  
}  

2. 配置好zookeper ip和端口，topic被分配的brokerid 列表，执行下面脚本完成topic rebalance 初始化，然后topic自动扩容
   startRebalance.sh
   [plain] view plain copy

   #!/bin/bash
   zkServer=“zkhost:2181”
   brokerIdList=0,1,2,3,4

   echo “zkConf:” $zkServer
   echo “brokerList:” $brokerIdList
   echo “###########################start generate reblance conf##############################”
   content=bash kafka-reassign-partitions.sh --zookeeper $zkServer --topics-to-move-json-file topicmove.conf --broker-list $brokerIdList --generate
   content=echo $content | awk -F 'Proposed partition reassignment configuration' '{print $2}'
   echo $content
   echo $content > ressgintopic.conf
   echo $content >> ./log/reblance.log
   echo “” >> ./log/reblance.log
   echo “###########################end generate reblance conf##############################”
   #start reblance
   echo “###########################start reblance##############################”
   bash kafka-reassign-partitions.sh --zookeeper $zkServer --reassignment-json-file ressgintopic.conf --execute

3.配置好zookeper ip和端口，执行下面脚本完成Topic Rebalance进度查询。
getRebalanceStatus.sh
[plain] view plain copy

#!/bin/bash  
zkServer="zkhost:2181"  
bash kafka-reassign-partitions.sh --zookeeper $zkServer --reassignment-json-file ressgintopic.conf --verify  

参考： http://kafka.apache.org/documentation#impl_consumerrebalance