Kafka broker 写消息的过程

最新推荐文章于 2025-12-04 14:53:10 发布

原创最新推荐文章于 2025-12-04 14:53:10 发布 · 671 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#kafka #分布式

kafka 专栏收录该内容

12 篇文章

订阅专栏

Producer → Kafka Broker → Replication → Consumer
               |
          Partition chosen (by key or round-robin)
               |
          Message appended to end of log (commit log)

上面的流程是kafka 写操作的大体流程。

kafka 不会特意保留message 在内存中，而是直接写入了disk。
那么消费的时候，如果是最近 produced 的message，即使写入了disk，还是会保留在OS的page cache中。 OS page cache 即使是flush 到了disk，也不会被立刻清除，这个是OS 自己的机制。所以consumer 读最近生产的消息的时候，还是可以从page cache中读取出来的，而不用从disk 中读取。

Kafka Broker Receives the Message
决定partition
Message is Appended to a Partition Log
The selected broker appends the message to the end of the partition log (a sequential write).
Kafka writes messages to disk efficiently using write-ahead logs (WAL).
Replication (for Fault Tolerance)
Kafka waits for acknowledgments based on the producer’s acks setting:
acks=0: Producer doesn’t wait.
acks=1: Wait for leader only.
acks=all: Wait for all replicas to acknowledge.
Message is Made Available to Consumers
Once the acks is fulfilled, it becomes available for consumers.
Consumers read sequentially using offsets.
Kafka keeps messages for a retention period (e.g., 7 days), regardless of whether they’ve been consumed.