kafka 消费者分区rebalance

最新推荐文章于 2025-04-16 15:32:36 发布

xiaoza7

最新推荐文章于 2025-04-16 15:32:36 发布

阅读量441

点赞数

分类专栏： kafka

本文链接：https://blog.youkuaiyun.com/xiaoza7/article/details/97247542

版权

kafka 专栏收录该内容

1 篇文章

订阅专栏

本文深入解析了Kafka中RangeAssignor分配策略的实现细节，包括如何反向获取每个topic订阅的consumerId列表和topic对应的topicPartition数，以及具体的划分逻辑。通过源码分析，阐述了其公平分配topic partitions给消费者的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

比如

RangeAssignor extends AbstractPartitionAssignor

在AbstractPartitionAssignor

public Map<String, Assignment> assign(Cluster metadata, Map<String, Subscription> subscriptions) {
    Set<String> allSubscribedTopics = new HashSet<>();
    for (Map.Entry<String, Subscription> subscriptionEntry : subscriptions.entrySet())
        allSubscribedTopics.addAll(subscriptionEntry.getValue().topics());

    Map<String, Integer> partitionsPerTopic = new HashMap<>();
    for (String topic : allSubscribedTopics) {
        Integer numPartitions = metadata.partitionCountForTopic(topic);
        if (numPartitions != null && numPartitions > 0)
            partitionsPerTopic.put(topic, numPartitions);
        else
            log.debug("Skipping assignment for topic {} since no metadata is available", topic);
    }
    //(topic,partionNum),(consumeruserStr,Subscriptions) 
    Map<String, List<TopicPartition>> rawAssignments = assign(partitionsPerTopic, subscriptions);
   
    // this class maintains no user data, so just wrap the results
    Map<String, Assignment> assignments = new HashMap<>();
    for (Map.Entry<String, List<TopicPartition>> assignmentEntry : rawAssignments.entrySet())
        assignments.put(assignmentEntry.getKey(), new Assignment(assignmentEntry.getValue()));
    return assignments;
}

RangeAssignor具体实现

@Override
public Map<String, List<TopicPartition>> assign(Map<String, Integer> partitionsPerTopic,
                                                Map<String, Subscription> subscriptions) {
    Map<String, List<String>> consumersPerTopic = consumersPerTopic(subscriptions);
    Map<String, List<TopicPartition>> assignment = new HashMap<>();
    for (String memberId : subscriptions.keySet())
        assignment.put(memberId, new ArrayList<>());

    for (Map.Entry<String, List<String>> topicEntry : consumersPerTopic.entrySet()) {
        String topic = topicEntry.getKey();
        List<String> consumersForTopic = topicEntry.getValue();

        Integer numPartitionsForTopic = partitionsPerTopic.get(topic);
        if (numPartitionsForTopic == null)
            continue;

        Collections.sort(consumersForTopic);

        int numPartitionsPerConsumer = numPartitionsForTopic / consumersForTopic.size();
        int consumersWithExtraPartition = numPartitionsForTopic % consumersForTopic.size();

        List<TopicPartition> partitions = AbstractPartitionAssignor.partitions(topic, numPartitionsForTopic);
        for (int i = 0, n = consumersForTopic.size(); i < n; i++) {
            int start = numPartitionsPerConsumer * i + Math.min(i, consumersWithExtraPartition);
            int length = numPartitionsPerConsumer + (i + 1 > consumersWithExtraPartition ? 0 : 1);
            assignment.get(consumersForTopic.get(i)).addAll(partitions.subList(start, start + length));
        }
    }
    return assignment;
}

其中

private Map<String, List<String>> consumersPerTopic(Map<String, Subscription> consumerMetadata) {
    Map<String, List<String>> res = new HashMap<>();
    for (Map.Entry<String, Subscription> subscriptionEntry : consumerMetadata.entrySet()) {
        String consumerId = subscriptionEntry.getKey();
        for (String topic : subscriptionEntry.getValue().topics())
            put(res, topic, consumerId);
    }
    return res;
}

大致意思是反向获取每个topic订阅的consumerId list和topic对应的topicPartition数，然后划分即可