kafka replication tools

最新推荐文章于 2020-04-23 14:40:30 发布

原创最新推荐文章于 2020-04-23 14:40:30 发布 · 127 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#kafka #replication

kafka 专栏收录该内容

33 篇文章

订阅专栏

本文介绍了Kafka中使用replicationtool进行副本选举的过程，包括如何指定首选副本，并详细解释了当首选副本不在ISR中时的处理方式。同时，提供了检查所有分区是否已成功迁移到首选副本的方法。

kafka工具

replication tool工作步骤（异步过程，执行完1就结束）：

1.更新zk上的/admin/preferred_replica_election节点，内容为副本的几个位置（不包含已经奔溃的leader）

2.controller中的zk的listener读取topic partition的副本的几个位置

3.controller获取每个topic partition的副本的几个replica中的第一个，作为referred replica；

如果这个preferred replica不是leader且在isr中，则controller发出request到这个preferred replica的broker，通知其作为topic partition的leader

bin/kafka-preferred-replica-election.sh --zookeeper localhost:81821/kafka-zk --path-to-json-file topicPartitionList.json

topicPartitionList.json格式为：

{

"partitions":

[

{"topic": "topic1", "partition": "0"},

{"topic": "topic1", "partition": "1"},

{"topic": "topic1", "partition": "2"},

{"topic": "topic2", "partition": "0"},

{"topic": "topic2", "partition": "1"},

]

}

FAQ

What happens if the preferred replica is not in the ISR?

The controller will fail to move the leadership to the preferred replica if it is not in the ISR. This is to ensure that there is no dataloss. When the replica becomes "in-sync" with the leader, the tool can be run again to move the leader.

则移动失败，避免数据丢失

How to find if all the partitions have been moved to the "preferred replica" after running the tool?

ListTopicCommand is an excellent tool that provides an overview of all the topic partitions in the cluster. For each topic partition, it displays the leader, assigned replicas and current "in-sync" replica set. If the leader and the first replica in the assigned replica set are the same then the Preferred replica leader election" tool succeeded. If not, the tool failed and may have to be run again.

跟isr的第一个replica相同