flink 消费 kafka subtask 分区策略

陪你一起捡蛋壳

已于 2024-08-23 19:01:11 修改

阅读量609

点赞数 2

CC 4.0 BY-SA版权

分类专栏： Flink之路文章标签： flink kafka 策略模式

于 2024-08-18 18:38:15 首次发布

本文链接：https://blog.youkuaiyun.com/qq_18218071/article/details/141303303

Flink之路专栏收录该内容

5 篇文章

订阅专栏

在新版本的flink source 采取的是KafkaSourceEnumerator完成分区的分区策略，通过阅读源码发现，真正的分区是下面的代码实现的

 private void addPartitionSplitChangeToPendingAssignments(
            Collection<KafkaPartitionSplit> newPartitionSplits) {
        //kafka source 指定的并行度
        int numReaders = context.currentParallelism();
        for (KafkaPartitionSplit split : newPartitionSplits) {
        	//具体的task与kafka分区分配算法
            int ownerReader = getSplitOwner(split.getTopicPartition(), numReaders);
            //存储着task与分区的对应关系
            pendingPartitionSplitAssignment
                    .computeIfAbsent(ownerReader, r -> new HashSet<>())
                    .add(split);
        }
    }

    static int getSplitOwner(TopicPartition tp, int numReaders) {
    	//按照topic name取 startIndex
        int startIndex = ((tp.topic().hashCode() * 31) & 0x7FFFFFFF) % numReaders;
	    //计算分区与task的对应关系
        return (startIndex + tp.partition()) % numReaders;
    }

举例子说明：
有两个topic:test_topic_partition_one, test_topic_partition_two，每个topic有9个分区，kafka source并行度设置为5

KafkaSource<String> source = KafkaSource.<String>builder()
        .setBootstrapServers("localhost:9092")
        .setProperties(properties)
        .setTopics("test_topic_partition_one", "test_topic_partition_two")
        .setGroupId("my-group")
        .setStartingOffsets(OffsetsInitializer.latest())
        .setValueOnlyDeserializer(new SimpleStringSchema())
        .build();           
env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka Source").setParallelism(5)

根据公式： int startIndex = ((tp.topic().hashCode() * 31) & 0x7FFFFFFF) % numReaders;
第一个topic test_topic_partition_one的 startIndex = 2
所以各个subtask与分区的对应关系为：

subtask 0 ==> test_topic_partition_one-3  test_topic_partition_one-8

subtask 1 ==> test_topic_partition_one-4

subtask 2 ==> test_topic_partition_one-0  test_topic_partition_one-5

subtask 3 ==> test_topic_partition_one-1  test_topic_partition_one-6 

subtask 4 ==> test_topic_partition_one-2  test_topic_partition_one-7

根据公式： int startIndex = ((tp.topic().hashCode() * 31) & 0x7FFFFFFF) % numReaders;
第二个topic test_topic_partition_two的 startIndex = 1
所以各个subtask与分区的对应关系为：

subtask 0 ==> test_topic_partition_two-4

subtask 1 ==> test_topic_partition_two-0  test_topic_partition_two-5

subtask 2 ==> test_topic_partition_two-1  test_topic_partition_two-6

subtask 3 ==> test_topic_partition_two-2  test_topic_partition_two-7 

subtask 4 ==> test_topic_partition_two-3  test_topic_partition_two-8

所以最终flink每个subtask对应的分区是，所以由于topic的流量不同，可能导致数据倾斜影响数据处理的能力。

subtask 0 ==> test_topic_partition_one-3  test_topic_partition_one-8  test_topic_partition_two-4

subtask 1 ==> test_topic_partition_one-4  test_topic_partition_two-0  test_topic_partition_two-5

subtask 2 ==> test_topic_partition_one-0  test_topic_partition_one-5  test_topic_partition_two-1  test_topic_partition_two-6

subtask 3 ==> test_topic_partition_one-1  test_topic_partition_one-6  test_topic_partition_two-2  test_topic_partition_two-7 

subtask 4 ==> test_topic_partition_one-2  test_topic_partition_one-7  test_topic_partition_two-3  test_topic_partition_two-8

对应的日志信息：

2024-08-18 18:39:51 INFO [org.apache.flink.connector.kafka.source.enumerator.KafkaSourceEnumerator  Line:393] Discovered new partitions: [test_topic_partition_one-6, test_topic_partition_one-7, test_topic_partition_one-8, test_topic_partition_two-4, test_topic_partition_two-5, test_topic_partition_two-6, test_topic_partition_one-0, test_topic_partition_two-7, test_topic_partition_one-1, test_topic_partition_two-8, test_topic_partition_one-2, test_topic_partition_one-3, test_topic_partition_one-4, test_topic_partition_one-5, test_topic_partition_two-0, test_topic_partition_two-1, test_topic_partition_two-2, test_topic_partition_two-3]

2024-08-18 18:39:51 INFO [org.apache.flink.connector.kafka.source.enumerator.KafkaSourceEnumerator  Line:353] Assigning splits to readers {0=[[Partition: test_topic_partition_one-3, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_two-4, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_one-8, StartingOffset: -1, StoppingOffset: -9223372036854775808]], 1=[[Partition: test_topic_partition_one-4, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_two-5, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_two-0, StartingOffset: -1, StoppingOffset: -9223372036854775808]], 2=[[Partition: test_topic_partition_two-6, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_one-0, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_two-1, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_one-5, StartingOffset: -1, StoppingOffset: -9223372036854775808]], 3=[[Partition: test_topic_partition_one-1, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_two-7, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_two-2, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_one-6, StartingOffset: -1, StoppingOffset: -9223372036854775808]], 4=[[Partition: test_topic_partition_two-8, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_one-2, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_two-3, StartingOffset: -1, StoppingOffset: -9223372036854775808], [Partition: test_topic_partition_one-7, StartingOffset: -1, StoppingOffset: -9223372036854775808]]}