Kafka会不会丢消息？怎么处理的？

最新推荐文章于 2025-03-28 08:09:24 发布

努力的SIR

最新推荐文章于 2025-03-28 08:09:24 发布

阅读量142

点赞数

分类专栏： BIG DATA【更新中】

本文链接：https://blog.youkuaiyun.com/qq_42615032/article/details/117217305

版权

BIG DATA【更新中】专栏收录该内容

7 篇文章

订阅专栏

Kafka存在丢消息的问题，消息丢失会发生在Broker，Producer和Consumer三种。
在这里插入图片描述

在这里插入图片描述

Properties props = new Properties();  
props.put("bootstrap.servers", "localhost:9092");  
props.put("group.id", "test");  
// 自动提交开关  
props.put("enable.auto.commit", "true");  
// 自动提交的时间间隔，此处是1s  
props.put("auto.commit.interval.ms", "1000");  
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");  
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");  
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);  
consumer.subscribe(Arrays.asList("foo", "bar"));  
while (true) {  
        // 调用poll后，1000ms后，消息状态会被改为 committed  
  ConsumerRecords<String, String> records = consumer.poll(100);  
  for (ConsumerRecord<String, String> record : records)  
    insertIntoDB(record); // 将消息入库，时间可能会超过1000ms

在这里插入图片描述

Properties props = new Properties();  
props.put("bootstrap.servers", "localhost:9092");  
props.put("group.id", "test");  
// 关闭自动提交，改为手动提交  
props.put("enable.auto.commit", "false");  
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");  
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");  
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);  
consumer.subscribe(Arrays.asList("foo", "bar"));  
final int minBatchSize = 200;  
List<ConsumerRecord<String, String>> buffer = new ArrayList<>();  
while (true) {  
        // 调用poll后，不会进行auto commit  
  ConsumerRecords<String, String> records = consumer.poll(100);  
  for (ConsumerRecord<String, String> record : records) {  
    buffer.add(record);  
  }  
  if (buffer.size() >= minBatchSize) {  
    insertIntoDb(buffer);  
                // 所有消息消费完毕以后，才进行commit操作  
    consumer.commitSync();  
    buffer.clear();  
  }

在这里插入图片描述

try {  
     while(running) {  
         ConsumerRecords<String, String> records = consumer.poll(Long.MAX_VALUE);  
         for (TopicPartition partition : records.partitions()) {  
             List<ConsumerRecord<String, String>> partitionRecords = records.records(partition);  
             for (ConsumerRecord<String, String> record : partitionRecords) {  
                 System.out.println(record.offset() + ": " + record.value());  
             }  
             long lastOffset = partitionRecords.get(partitionRecords.size() - 1).offset();  
             // 精确控制offset  
             consumer.commitSync(Collections.singletonMap(partition, new OffsetAndMetadata(lastOffset + 1)));  
         }  
     }  
 } finally {  
   consumer.close();  
 }