kafka其他方式的多分区顺序消费（待补充）_多分区的顺序消费-优快云博客

除单分区顺序消费外，Kafka还可通过以下方式实现消息顺序消费：

1. 基于事务的跨分区顺序消费

原理：Kafka的事务机制允许应用程序在多个分区上原子性地写入消息，从而保证这些消息在多个分区间的顺序性。生产者开启事务后，一系列消息的发送被视为一个原子操作，要么全部成功，要么全部失败。当消费者按顺序消费这些消息时，能保证与生产者事务提交顺序一致。
实现步骤
- 生产者：在Java中，使用KafkaProducer时，首先要配置事务ID，通过ProducerConfig.TRANSACTIONAL_ID_CONFIG属性设置。然后调用initTransactions()初始化事务，使用beginTransaction()开始事务，在发送完一组需要保持顺序的消息后，调用commitTransaction()提交事务。若发送过程中出现错误，调用abortTransaction()回滚事务。示例代码如下：

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class TransactionalProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "my-transactional-id");

        KafkaProducer<String, String> producer = new KafkaProducer<>(props);
        producer.initTransactions();

        try {
            producer.beginTransaction();
            producer.send(new ProducerRecord<>("topic1", "key1", "value1"));
            producer.send(new ProducerRecord<>("topic2", "key2", "value2"));
            producer.commitTransaction();
        } catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
            producer.abortTransaction();
        } finally {
            producer.close();
        }
    }
}

消费者：消费者需确保按顺序读取消息。可通过设置isolation.level为read_committed，这样消费者只会读取已提交事务的消息，从而保证消费顺序与生产者事务提交顺序一致。在Java中，配置ConsumerConfig.ISOLATION_LEVEL_CONFIG属性为read_committed。示例代码如下：

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class TransactionalConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("topic1"));
        consumer.subscribe(Collections.singletonList("topic2"));

        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records) {
                System.out.println("Received message: " + record.value());
            }
        }
    }
}

2. 自定义分区分配与消费逻辑

原理：通过自定义分区分配策略，将具有特定顺序关系的消息分配到不同分区，并确保每个分区由特定的消费者实例或线程处理。例如，在一个电商系统中，可根据店铺ID分配分区，每个店铺的消息按顺序处理。同时，使用单线程消费者或线程安全的消费逻辑，保证每个分区内消息消费顺序。
实现步骤
- 自定义分区分配器：在Java中，可继承AbstractPartitionAssignor类并实现相关方法。例如，根据消息的某个属性（如店铺ID）分配分区。示例代码如下：

import org.apache.kafka.clients.consumer.internals.AbstractPartitionAssignor;
import org.apache.kafka.common.TopicPartition;

import java.util.*;

public class ShopIdPartitionAssignor extends AbstractPartitionAssignor {
    @Override
    public String name() {
        return "shop - id - assignor";
    }

    @Override
    public Map<String, List<TopicPartition>> assign(Map<String, Integer> partitionsPerTopic, Map<String, Subscription> subscriptions) {
        Map<String, List<TopicPartition>> assignment = new HashMap<>();
        for (Map.Entry<String, Subscription> entry : subscriptions.entrySet()) {
            String consumer = entry.getKey();
            List<TopicPartition> partitions = new ArrayList<>();
            for (String topic : entry.getValue().topics()) {
                int numPartitions = partitionsPerTopic.get(topic);
                for (int i = 0; i < numPartitions; i++) {
                    // 假设按店铺ID分配分区，这里简单示例，实际需从消息中获取店铺ID
                    if (i % 2 == 0) {
                        partitions.add(new TopicPartition(topic, i));
                    }
                }
            }
            assignment.put(consumer, partitions);
        }
        return assignment;
    }
}

消费者配置：在消费者端，通过ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG属性指定使用自定义的分区分配器。同时，可使用单线程消费者或在多线程消费时确保每个分区由一个线程处理，保证顺序消费。示例代码如下：

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class CustomAssignorConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "my - group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, ShopIdPartitionAssignor.class.getName());

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("your_topic"));

        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records) {
                System.out.println("Received message: " + record.value());
            }
        }
    }
}