__consumer_offsets的作用
__consumer_offsets作为kafka的内部topic,用来保存消费组元数据以及对应提交的offset信息,因此它对于消费者来说扮演着举足轻重的地位。每次消费者消费完一批数据需要标记这批数据被消费掉时,需要将消费的偏移量即offset提交掉,这个提交过程其实就是将offset信息写入__consumer_offsets的过程。以下分析完全基于kafka0.10.0.1版本。
__consumer_offset的创建
在刚部署的kafka集群中,我们list所有topic会发现并没有__consumer_offsets,那它是在什么时候创建的呢?我们可以跟下源码:
//KafkaApis
def handle(request: RequestChannel.Request) {
try {
ApiKeys.forId(request.requestId) match {
//...
//服务端收到寻找groupCoordinator的请求并处理
case ApiKeys.GROUP_COORDINATOR => handleGroupCoordinatorRequest(request)
//...
}
} catch {
//
} finally
request.apiLocalCompleteTimeMs = SystemTime.milliseconds
}
def handleGroupCoordinatorRequest(request: RequestChannel.Request) {
//...
val offsetsTopicMetadata = getOrCreateGroupMetadataTopic(request.securityProtocol)
//...
}
private def getOrCreateGroupMetadataTopic(securityProtocol: SecurityProtocol): MetadataResponse.TopicMetadata = {
val topicMetadata = metadataCache.getTopicMetadata(Set(TopicConstants.GROUP_METADATA_TOPIC_NAME), securityProtocol)
topicMetadata.headOption.getOrElse(createGroupMetadataTopic())
}
private def createGroupMetadataTopic