Offset commit with offsets failed

报错现象

2021/05/31-14:17:26.488 [] [JdqConsumerThread_unique_device_identification_mapper_yf_0] ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator- [Consumer clientId=4670fce6, groupId=hisotry1617153935172] Offset commit with offsets {unique_device_identification_mapper_yf-15=OffsetAndMetadata{offset=555905135, leaderEpoch=null, metadata=''}, unique_device_identification_mapper_yf-14=OffsetAndMetadata{offset=554744491, leaderEpoch=null, metadata=''}} failed

org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group.

at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:1093)

at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.doCommitOffsetsAsync(ConsumerCoordinator.java:913)

at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsAsync(ConsumerCoordinator.java:880)

at org.apache.kafka.clients.consumer.KafkaConsumer.commitAsync(KafkaConsumer.java:1584)

at org.apache.kafka.clients.consumer.KafkaConsumer.commitAsync(KafkaConsumer.java:1552)

at org.apache.kafka.clients.consumer.KafkaConsumer.commitAsync(KafkaConsumer.java:1529)

原因

由于取出的一批消息数量太大，consumer在session.timeout.ms时间之内没有消费完成引起kafka rebalance

解决方法

调大sessionTimeout，调小maxPollRecord

session.timeout.ms改为300000,即5分钟(原先配置为10000 即10s)
max.poll.records改为5000(原先配置为10000)

参考文档：https://blog.youkuaiyun.com/a_sunny_a/article/details/112270638