pulsar学习-message

message 消费和配置手册

结构

约束

默认大小为 5MB
调整大小需要调整 broker 和 bookkeeper 的配置：

• broker.conf : maxMessageSize
• bookkeeper.conf: nettyMaxFrameSizeBytes

ack 机制

ack 的发送在 consumer 消费成功后给broker。

• 确认每一个数据：consumer.acknowledge(msg);
• 确认最后一条成功数据：consumer.acknowledgeCumulative(msg);
• negative ack 机制：用于消费失败重试，可能导致乱序，如果使用，建议超时后才触发
  • Exclusive and Failover subscription types, consumers only negatively acknowledge the last message
  • Shared and Key_Shared subscription types, consumers can negatively acknowledge messages individually.
  • redelivery 超时时间：set a time range during which the client tracks the unacknowledged messages，客户端会计算长时间未成功消费的数据的时间

如果能主动的了解到是否失败，那么主动的去重新拉取消息要比超时等待的策略要好。当然，二者可以结合使用。

If batching is enabled, all messages in one batch are redelivered to the consumer.
In Shared subscription type, messages are acknowledged individually.因为会有多个消费者能够使用同一个 subscription
默认配置中当一个数据被全部的 subscriptions 消费后可以被删除。

重试队列和死信队列

Once the maximum number of retries has been reached, the unconsumed messages are moved to a dead letter topic for manual processing.

重试队列

• 配置默认关闭， 需要配置 enableRetry 来设置consumer
• 默认 maxRedeliverCount 自定义
• 默认topic 名称 ： --RETRY ， 通过retryLetterTopic 方法可自定义
• 消费 ： consumer.reconsumeLater

重试队列和延迟队列区别：Retry letter topic serves failure handling through message redelivery to ensure critical data is not lost, while delayed message delivery is intended to deliver a message with a specified time delay.

死信队列

• 默认topic ： --DLQ
• 使用场景： dead letter topic is enabled in Shared and Key_Shared subscription types.

Dead letter topic serves message redelivery, which is triggered by acknowledgment timeout or negative acknowledgment or retry letter topic.

通信优化

• 压缩消息
• 批量发送：需要 batching 的配置为 true
  • 超过最大数量
  • 超过最大时间
• chunking ：将大消息切割
  • chunkingEnabled = true 表示启用
  • chunking 条件：chunking （chunkingEnabled=true）开启并且消息超过 maxMessageSize
  • chunking 通过 chunked metadata 维护顺序
  • 一个消息的 chunk 会被维护在一个 ledger 中
  • 消费者会缓存 chunk 到 receiver queue直到完整
  • chunk 机制只适用于持久化 topic ：Chunking is only available for persistent topics.
  • chunk 和 batch 机制不能同时使用：Chunking cannot be enabled simultaneously with batching. Before enabling chunking, you need to disable batching.
  • 超时问题：If the consumer fails to receive all chunks of a message within a specified period, it expires incomplete chunks. The default value is 1 minute. For more information about the expireTimeOfIncompleteChunkedMessage parameter, refer to org.apache.pulsar.client.api.

batching 机制

• ack ： a batch is acknowledged when all of its messages are acknowledged by a consumer. 失败会导致 batch 全部重新发送 竟然没有滑动窗口机制～
  • 为了解决发送问题， 2.6.0 引入了 batch index acknowledgment
  • 默认禁用 ：acknowledgmentAtBatchIndexLevelEnabled=false

chunk 机制

• The broker stores all the three chunked messages in the managed ledger and dispatches them to the ordered (exclusive/failover) consumer in the same order.
  
  多producer 下，不同producer 的chunk 会乱序，同一个producer 的会有序
  When multiple producers publish chunked messages into a single topic, the broker stores all the chunked messages coming from different producers in the same managed ledger. The chunked messages in the managed ledger can be interwoven with each other.

持久化机制

• 未完全被消费的消息会持久化保存
• Message retention enables you to store messages that have been acknowledged by a consumer
• Message expiry enables you to set a time to live (TTL) for messages that have not yet been acknowledged
  • Message retention: Keep the data for at least X hours (even if acknowledged)
  • Time-to-live: Discard data after some time (by automatically acknowledging)