Kafka 2.x KafkaController 分析 -- 持续更新中

最新推荐文章于 2025-03-27 15:33:54 发布

harry0913

最新推荐文章于 2025-03-27 15:33:54 发布

阅读量137

点赞数

分类专栏： Kafka 文章标签： kafka

本文链接：https://blog.youkuaiyun.com/Dream1248120122/article/details/120607249

版权

Kafka 专栏收录该内容

3 篇文章

订阅专栏

创建

在每个Broker启动时，都会创建一个KafkaController实例，但是只有一个KafkaController会成为Leader，对外提供集群管理服务，选主是基于Zookeeper实现的，即启动时，每个KafkaController会向Zookeeper同一个路径下写入瞬时节点/controller，只有一个会成功，即成为了Leader，其余的成为了Follower。但Leader离线时，其余的Follower重新选主。

ControllerContext初始化

Kafka最重要的概念即为"Topic"和"Partition",这些信息持久化到Zookeeper中，ControllerContext对象保持了Kafka核心的对象信息。

var liveBrokers: Set[Broker] => 读取/brokers/ids 获取集群中所有Broker信息
var liveBrokerEpoch: Map[Int,Long] => 转换为brokerId <-> epoch

[zk] ls /brokers
[ids, topics, seqid]
[zk] ls /brokers/ids
[1]
[zk] get /brokers/ids/1
{"listener_security_protocol_map":{"INTERNAL":"PLAINTEXT","EXTERNAL":"SSL"},"endpoints":["INTERNAL://192.168.10.10:9092","EXTERNAL://172.168.10.10:9093"],"jmx_port":9393,"host":"192.168.10.10","timestamp":"1631534356536","port":9092,"version":4}

var allTopics: Set[String] => 读取/brokers/topics 获取集群中所有Topic信息
val partitionAssignments: mutable.Map.empty[String,mutable.Map[Int,ReplicaAssignment]] => 存放分区副本信息： 0 -> [1] , 1 -> [1], 2-> [1] 每个topic的每个分区副本所在Broker
=> 该信息为创建Topic时，根据人为或是自动策略分配的ReplicationAssignment
```
[zk] ls /brokers
[ids, topics, seqid]

[zk] ls /brokers/topics
....
....


[zk] get /brokers/topics/app_logs
{"version":1,"partitions":{"2":[1],"1":[1],"0":[1]}}
```

val partitionLeadershipInfo: mutable.Map.empty[TopicPartition,LeaderIsrAndControllerEpoch] => 每个topic的每个分区副本的Leader及其ISR列表

[zk] ls /brokers
[ids, topics, seqid]

[zk] get /brokers/topics/app_logs/partitions/0/state
{"controller_epoch":1,"leader":1,"version":1,"leader_epoch":0,"isr":[1]}