Flink Kafka Patition 分配问题

最新推荐文章于 2025-02-11 18:50:37 发布

原创最新推荐文章于 2025-02-11 18:50:37 发布 · 488 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#编辑器 #flink

Flink在消费多个KafkaTopic时，由于分配算法基于单个Topic的分区数，可能导致数据倾斜，使得部分subtask空闲。问题出在FlinkKafkaConsumerBase的open方法中，KafkaTopicPartitionAssigner的分配策略可能造成重复分配。解决这个问题需要优化分配算法，确保公平分配各个Topic的分区到subtask。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

当Flink消费多个Topic的时候，由于分配算法是针对单个Topic的分区数，因此在对多个Topic进行分配的时候会造成重复分配，就会有几个subtask处于空闲当中，造成subtask数据倾斜

源码位置

FlinkKafkaConsumerBase的open方法

        subscribedPartitionsToStartOffsets = new HashMap<>();
        final List<KafkaTopicPartition> allPartitions = partitionDiscoverer.discoverPartitions();

discoverPartitions

                List<KafkaTopicPartition> newDiscoveredPartitions;
newDiscoveredPartitions =
                            getAllPartitionsForTopics(topicsDescriptor.getFixedTopics());

getAllPartitionsForTopics

    for (String topic : topics) {
                final List<PartitionInfo> kafkaPartitions = kafkaConsumer.partitionsFor(topic);
                System.out.println("getAllPartitionsForTopics=" + kafkaPartitions);

                if (kafkaPartitions == null) {
                    throw new RuntimeException(
                            String.format(
                                    "Could not fetch partitions for %s. Make sure that the topic exists.",
                                    topic));
                }

                for (PartitionInfo partitionInfo : kafkaPartitions) {
                    partitions.add(
                            new KafkaTopicPartition(
                                    partitionInfo.topic(), partitionInfo.partition()));
                }
            }

还在discoverPartitions中

 if (newDiscoveredPartitions == null || newDiscoveredPartitions.isEmpty()) {
                    throw new RuntimeException(
                            "Unable to retrieve any partitions with KafkaTopicsDescriptor: "
                                    + topicsDescriptor);
                } else {
                    Iterator<KafkaTopicPartition> iter = newDiscoveredPartitions.iterator();
                    KafkaTopicPartition nextPartition;
                    while (iter.hasNext()) {
                        nextPartition = iter.next();
                        if (!setAndCheckDiscoveredPartition(nextPartition)) {
                            iter.remove();
                        }
                    }
                }

setAndCheckDiscoveredPartition

    public boolean setAndCheckDiscoveredPartition(KafkaTopicPartition partition) {
        if (isUndiscoveredPartition(partition)) {
            discoveredPartitions.add(partition);

            return KafkaTopicPartitionAssigner.assign(partition, numParallelSubtasks)
                    == indexOfThisSubtask;
        }

        return false;
    }

分配的算法在KafkaTopicPartitionAssigner

    public static int assign(KafkaTopicPartition partition, int numParallelSubtasks) {
        int startIndex =
                ((partition.getTopic().hashCode() * 31) & 0x7FFFFFFF) % numParallelSubtasks;
                
        // here, the assumption is that the id of Kafka partitions are always ascending
        // starting from 0, and therefore can be used directly as the offset clockwise from the
        // start index
        return (startIndex + partition.getPartition()) % numParallelSubtasks;
    }

Flink Kafka Patition 分配问题

相关的类

源码位置