kafka 中到底是怎么通过 __consumer_offsets 存储消费位移的

最新推荐文章于 2023-08-16 22:06:32 发布

原创最新推荐文章于 2023-08-16 22:06:32 发布 · 7.9k 阅读

8 ·

CC 4.0 BY-SA版权

文章标签：

#__consumer_offsets #kafka消费位移 #kafka 位移信息

MQ 专栏收录该内容

4 篇文章

订阅专栏

我们知道老版本的消费位移信息是存储的zookeeper 中的，但是zookeeper 并不适合频繁的写入查询操作，所以在新版本的中消费位移信息存放在了__consumer_offsets内置topic中

那么__consumer_offsets topic 到底是怎么存储我们的消费位移信息呢？

1. 创建topic consumer_offsets_test

bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic consumer_offsets_test --replication-factor 1 --partitions 3

2. 查看topic是否创建成功

bin/kafka-topics.sh --zookeeper localhost:2181 --topic consumer_offsets_test --describe
Topic:consumer_offsets_test   PartitionCount:3   ReplicationFactor:1   Configs:
   Topic: consumer_offsets_test   Partition: 0   Leader: 0   Replicas: 0   Isr: 0
   Topic: consumer_offsets_test   Partition: 1   Leader: 1   Replicas: 1   Isr: 1
   Topic: consumer_offsets_test   Partition: 2   Leader: 2   Replicas: 2   Isr: 2

3. 使用kafka-console-producer.sh脚本生产消息生产10条消息

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic consumer_offsets_test

依次输入10条信息 ^C 退出

4.验证消息是否生产成功

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic consumer_offsets_test --time -1
consumer_offsets_test:0:3
consumer_offsets_test:1:4
consumer_offsets_test:2:4

这里我生产了11条信息，当时kafka-console-producer.sh 多生产了一条

5.创建消费组消费信息

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic consumer_offsets_test --from-beginning --group consumer_offsets_group

消费完信息 ^C 退出

记住刚才创建consumer_offsets_group 消费组ID 稍后会用到

6. 查询consumer_offsets_group 在 __consumer_offsets topic 中存放的位移信息
__consumer_offsets 默认分区50 那么我们的consumer_offsets_group的位移信息是放到那个分区里面了呢？

通过如下公式即可获取：
Math.abs("consumer_offsets_group".hashCode()) % 50

结果放在了11分区上

7.获取对应分区的位移信息
在获取__consumer_offsets 信息之前需要设置consumer.properties中exclude.internal.topics=false
因为__consumer_offsets 属于内置 topic

bin/kafka-console-consumer.sh --topic __consumer_offsets --partition 11 --bootstrap-server localhost:9092 --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config config/consumer.properties --from-beginning

[consumer_offsets_group/*消费组ID*/,consumer_offsets_test/*topic*/,1/*partition*/]::OffsetAndMetadata(offset=4/*提交的消费位移信息*/, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567461031, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,0]::OffsetAndMetadata(offset=3, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567461031, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,2]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567461031, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,1]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567466027, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,0]::OffsetAndMetadata(offset=3, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567466027, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,2]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567466027, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,1]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567468638, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,0]::OffsetAndMetadata(offset=3, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567468638, expireTimestamp=None)
[consumer_offsets_group,consumer_offsets_test,2]::OffsetAndMetadata(offset=4, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1543567468638, expireTimestamp=None)