kafka超时导致的重复消费的问题

最新推荐文章于 2025-06-09 14:12:17 发布

原创

最新推荐文章于 2025-06-09 14:12:17 发布 · 7.6k 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#Kafka #超时 #重复消费

本文探讨了Kafka中由于超时导致的重复消费问题，分析了session.timeout.ms和max.poll.interval.ms的区别与作用。内容涉及Kafka的heartbeat机制、KIP-62的引入以及如何避免消息丢失和消费者超时。建议通过调整max.poll.interval.ms和max.poll.records来控制处理时间和防止重复消费。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

今天看到群友遇到个问题：

问题的表象是超时导致autoCommit失败，从而导致重复消费

错误内容是：

2018-08-21 00:00:00.938 [kudu-0-C-1] WARN o.a.k.c.consumer.internals.ConsumerCoordinator - Auto offset commit failed for group sm: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending