flink-FlinkFixedPartitioner分区原理

最新推荐文章于 2024-11-15 23:48:40 发布

原创

最新推荐文章于 2024-11-15 23:48:40 发布 · 1.5k 阅读

1 ·

CC 4.0 BY-SA版权

FlinkFixedPartitioner根据Flink sink的subtask ID和Kafka分区数进行取余计算来决定数据流向。例如，当Flink并行度为3，Kafka分区为2时，F0对应P0，F1对应P1，F2对应P0；而当Flink并行度为2，Kafka分区为3时，F0对应P0，F1对应P1。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

FlinkFixedPartitioner源码：

package org.apache.flink.streaming.connectors.kafka.partitioner;

import org.apache.flink.util.Preconditions;

public class FlinkFixedPartitioner<T> extends FlinkKafkaPartitioner<T> {
    private int parallelInstanceId;

    public FlinkFixedPartitioner() {
    }

    public void open(int parallelInstanceId, int parallelInstances) {
        Preconditions.checkArgument(parallelInstanceId >= 0, "Id of this subtask cannot be negative.");
        Preconditions.checkArgument(parallelInstances > 0, "Number of subtasks must be larger than 0.");
        this.parallelInstanceId = parallelInstanceId;
    }

    public int partition(T record, byte[] key, byte[] value, String targetTopic, int[] partitions) {
        Preconditions.checkArgument(partitions != null && partitions.length > 0, "Partitions of the target topic is empty.");
        return partitions[this.parallelInstanceId % partition