DelayedOperationPurgatory机制(四):DelayProduce之appendMesaages

本文详细解析了Kafka中生产者发送消息至服务端的过程,包括如何通过KafkaApis调用hadleProducerRequest方法,进而使用ReplicaManager.appendMessages方法将消息追加到Log中,以及生成DelayedProduce对象的机制。

生产者把消息发送到服务端时,KafkaApis会调用hadleProducerRequest方法,他会调用ReplicaManager.appendMessages方法将消息追加到Log中,生成相应的DelayedProduce对象并添加到delayedProducePurgatory中处理,下面介绍一下appendMesaages方法

  def appendMessages(timeout: Long,
                     requiredAcks: Short,
                     internalTopicsAllowed: Boolean,
                     messagesPerPartition: Map[TopicPartition, MessageSet],
                     responseCallback: Map[TopicPartition, PartitionResponse] => Unit) {

    if (isValidRequiredAcks(requiredAcks)) {
      val sTime = SystemTime.milliseconds
      // 将消息追加到Log中,同时还会检测delayedFetchPurgatory中相关的key对应的DelayedFetch,满足条件则将其执行完成
      val localProduceResults = appendToLocalLog(internalTopicsAllowed, messagesPerPartition, requiredAcks)
      debug("Produce to local log in %d ms".format(SystemTime.milliseconds - sTime))
      // 对追加的结果进行装换,注意ProducePartitionStateus的参数
      val produceStatus = localProduceResults.map { case (topicPartition, result) =>
        topicPartition ->
                ProducePartitionStatus(
                  result.info.lastOffset + 1, // required offset
                  new PartitionResponse(result.errorCode, result.info.firstOffset, result.info.timestamp)) // response status
      }
      // 下面检测是否生成DelayedProduce,其中一个条件就是检测ProducerRequest中acks的字段是否为-1
      if (delayedRequestRequired(requiredAcks, messagesPerPartition, localProduceResults)) {
        // create delayed produce operation
        val produceMetadata = ProduceMetadata(requiredAcks, produceStatus)
        // 创建DelayedProduce对象
        val delayedProduce = new DelayedProduce(timeout, produceMetadata, this, responseCallback)

        // create a list of (topic, partition) pairs to use as keys for this delayed produce operation
        val producerRequestKeys = messagesPerPartition.keys.map(new TopicPartitionOperationKey(_)).toSeq

        // try to complete the request immediately, otherwise put it into the purgatory
        // this is because while the delayed produce operation is being created, new
        // requests may arrive and hence make this operation completable.
        // 尝试完成DelayProduce,否则将DelayProduce天啊及到DelayProducePurgatory中管理
        delayedProducePurgatory.tryCompleteElseWatch(delayedProduce, producerRequestKeys)

      } else {
        // we can respond immediately
        val produceResponseStatus = produceStatus.mapValues(status => status.responseStatus)
        responseCallback(produceResponseStatus)
      }
    } else {
      // If required.acks is outside accepted range, something is wrong with the client
      // Just return an error and don't handle the request at all
      val responseStatus = messagesPerPartition.map {
        case (topicAndPartition, messageSet) =>
          (topicAndPartition -> new PartitionResponse(Errors.INVALID_REQUIRED_ACKS.code,
                                                      LogAppendInfo.UnknownLogAppendInfo.firstOffset,
                                                      Message.NoTimestamp))
      }
      responseCallback(responseStatus)
    }
  }

 

2025-10-27 17:01:20,950] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2025-10-27 17:01:20,958] INFO [ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper) [2025-10-27 17:01:21,055] INFO Loading logs from log dirs ArrayBuffer(/export/servers/kafka/kafka-logs) (kafka.log.LogManager) [2025-10-27 17:01:21,065] INFO Skipping recovery for all logs in /export/servers/kafka/kafka-logs since clean shutdown file was found (kafka.log.LogManager) [2025-10-27 17:01:21,095] INFO Loaded 0 logs in 40ms. (kafka.log.LogManager) [2025-10-27 17:01:21,096] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager) [2025-10-27 17:01:21,109] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [2025-10-27 17:01:23,477] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas) [2025-10-27 17:01:23,554] INFO Awaiting socket connections on slaver1:9092. (kafka.network.Acceptor) [2025-10-27 17:01:23,758] INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Created data-plane acceptor and processors for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer) [2025-10-27 17:01:23,908] INFO [broker-2-to-controller-send-thread]: Starting (kafka.server.BrokerToControllerRequestThread) [2025-10-27 17:01:23,947] INFO [ExpirationReaper-2-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2025-10-27 17:01:23,964] INFO [ExpirationReaper-2-ElectLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2025-10-27 17:01:23,962] INFO [ExpirationReaper-2-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2025-10-27 17:01:23,962] INFO [ExpirationReaper-2-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2025-10-27 17:01:24,002] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler) [2025-10-27 17:01:24,191] INFO Creating /brokers/ids/2 (is it secure? false) (kafka.zk.KafkaZkClient) [2025-10-27 17:01:24,246] INFO Stat of the created znode at /brokers/ids/2 is: 51539607715,51539607715,1761555684227,1761555684227,1,0,0,216173873305944069,212,0,51539607715 (kafka.zk.KafkaZkClient) [2025-10-27 17:01:24,249] INFO Registered broker 2 at path /brokers/ids/2 with addresses: PLAINTEXT://192.168.10.161:9092, czxid (broker epoch): 51539607715 (kafka.zk.KafkaZkClient) [2025-10-27 17:01:24,561] INFO [ExpirationReaper-2-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2025-10-27 17:01:24,583] INFO [ExpirationReaper-2-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2025-10-27 17:01:24,598] INFO [ExpirationReaper-2-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2025-10-27 17:01:24,668] INFO [GroupCoordinator 2]: Starting up. (kafka.coordinator.group.GroupCoordinator) [2025-10-27 17:01:24,687] INFO [GroupCoordinator 2]: Startup complete. (kafka.coordinator.group.GroupCoordinator) [2025-10-27 17:01:24,923] INFO [ProducerId Manager 2]: Acquired new producerId block (brokerId:2,blockStartProducerId:1000,blockEndProducerId:1999) by writing to Zk with path version 2 (kafka.coordinator.transaction.ProducerIdManager) [2025-10-27 17:01:24,926] INFO [TransactionCoordinator id=2] Starting up. (kafka.coordinator.transaction.TransactionCoordinator) [2025-10-27 17:01:24,936] INFO [TransactionCoordinator id=2] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator) [2025-10-27 17:01:24,950] INFO [Transaction Marker Channel Manager 2]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager) [2025-10-27 17:01:25,272] INFO [ExpirationReaper-2-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) [2025-10-27 17:01:25,388] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread) [2025-10-27 17:01:25,471] INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Starting socket server acceptors and processors (kafka.network.SocketServer) [2025-10-27 17:01:25,557] INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Started data-plane acceptor and processor(s) for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer) [2025-10-27 17:01:25,558] INFO [SocketServer listenerType=ZK_BROKER, nodeId=2] Started socket server acceptors and processors (kafka.network.SocketServer) [2025-10-27 17:01:25,613] INFO Kafka version: 2.8.2 (org.apache.kafka.common.utils.AppInfoParser) [2025-10-27 17:01:25,613] INFO Kafka commitId: 3146c6ff4a24cc24 (org.apache.kafka.common.utils.AppInfoParser) [2025-10-27 17:01:25,613] INFO Kafka startTimeMs: 1761555685559 (org.apache.kafka.common.utils.AppInfoParser) [2025-10-27 17:01:25,618] INFO [KafkaServer id=2] started (kafka.server.KafkaServer) [2025-10-27 17:01:26,019] INFO [broker-2-to-controller-send-thread]: Recorded new controller, from now on will use broker master:9092 (id: 1 rack: null) (kafka.server.BrokerToControllerRequestThread)
最新发布
10-28
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值