note of kafka learning (first part, before replication)

github_22808831

于 2014-11-29 07:11:54 发布

阅读量415

点赞数

CC 4.0 BY-SA版权

文章标签： kafka

本文链接：https://blog.youkuaiyun.com/github_22808831/article/details/41592533

本文探讨了Kafka的设计理念，包括连续磁盘访问优于随机内存访问的情况、对象内存开销高及其对垃圾回收的影响等问题。同时介绍了Kafka通过文件系统的持久化日志来提高性能的方法，以及使用poll模式代替push模式减少资源消耗的技术细节。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

sequential disk access can in some cases be faster than random memory access!

The memory overhead of objects is very high, often doubling the size of the data stored (or worse).
Java garbage collection becomes increasingly fiddly and slow as the in-heap data increases.

All data is immediately written to a persistent log on the filesystem without necessarily flushing to disk. In effect this just means that it is transferred into the kernel's pagecache.

kafka use poll instead of push

To avoid this we have parameters in our pull request that allow the consumer request to block in a "long poll" waiting until data arrives

(Kafka's persistent storage makes me feel like the same as Git ... :-))

So effectively Kafka guarantees at-least-once delivery by default and allows the user to implement at most once delivery by disabling retries on the producer and committing its offset prior to processing a batch of messages. Exactly-once delivery requires co-operation with the destination storage system but Kafka provides the offset which makes implementing this straight-forward.