kafka Producer详解

最新推荐文章于 2025-06-14 23:59:51 发布

闭门车

最新推荐文章于 2025-06-14 23:59:51 发布

阅读量1.5k

点赞数

CC 4.0 BY-SA版权

分类专栏： NOSQL MQ

本文链接：https://blog.youkuaiyun.com/skymouse2002/article/details/79932912

NOSQL 同时被 2 个专栏收录

20 篇文章

订阅专栏

9 篇文章

订阅专栏

本文深入解析Kafka Producer的工作原理，包括消息发送模式、错误处理机制、性能指标监控及配置优化等核心内容，并提供了自定义分区策略和确保消息顺序性的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Producer

1.发送并忘记这种情况下可能会丢数据。

2.同步发送它会返回一个Future对象，使用get()方法进行等待，就可以知道它是否发送成功。

3.异步发送指定一个回调函数，服务器在进行响应时调用

public Future<RecordMetadata> send(ProducerRecord<K, V> record)；

public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback)；

这两个函数均为发送消息的函数，不同点在于第二个函数多一个回调函数的参数。

    public void close() {
        close(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
    }
    public void close(long timeout, TimeUnit timeUnit) {
        close(timeout, timeUnit, false);
    }

关闭producer的函数，不同之处在于第二个函数可以再指定时间内关闭

    public List<PartitionInfo> partitionsFor(String topic) {
        try {
            return waitOnMetadata(topic, null, maxBlockTimeMs).cluster.partitionsForTopic(topic);
        } catch (InterruptedException e) {
            throw new InterruptException(e);
        }
    }

获得topic的分区信息

    public Map<MetricName, ? extends Metric> metrics() {
        return Collections.unmodifiableMap(this.metrics.metrics());
    }

MetricName [name=io-wait-ratio, group=producer-metrics, description=The fraction of time the I/O thread spent waiting., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@462ee862
MetricName [name=request-size-avg, group=producer-metrics, description=The average size of all requests in the window.., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@48e90ed7
MetricName [name=produce-throttle-time-max, group=producer-metrics, description=The maximum throttle time in ms, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4cd68312
MetricName [name=connection-creation-rate, group=producer-metrics, description=New connections established per second in the window., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@9d404ec
MetricName [name=record-size-avg, group=producer-metrics, description=The average record size, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@3808904c
MetricName [name=connection-close-rate, group=producer-metrics, description=Connections closed per second in the window., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@3897ae2c
MetricName [name=record-queue-time-avg, group=producer-metrics, description=The average time in ms record batches spent in the record accumulator., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@40a1643f
MetricName [name=request-latency-max, group=producer-metrics, description=The maximum request latency in ms, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@69fbfba3
MetricName [name=request-rate, group=producer-metrics, description=The average number of requests sent per second., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@158d9c20
MetricName [name=record-send-rate, group=producer-metrics, description=The average number of records sent per second., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@106eb673
MetricName [name=io-wait-time-ns-avg, group=producer-metrics, description=The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@5ba5a8dd
MetricName [name=connection-count, group=producer-metrics, description=The current number of active connections., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@6db566e9
MetricName [name=outgoing-byte-rate, group=producer-metrics, description=The average number of outgoing bytes sent per second to all servers., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@14a09be3
MetricName [name=select-rate, group=producer-metrics, description=Number of times the I/O layer checked for new I/O to perform per second, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@71186a2a
MetricName [name=record-size-max, group=producer-metrics, description=The maximum record size, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@78dc794
MetricName [name=buffer-exhausted-rate, group=producer-metrics, description=The average per-second number of record sends that are dropped due to buffer exhaustion, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@41c96ba2
MetricName [name=compression-rate-avg, group=producer-metrics, description=The average compression rate of record batches., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@51a68ffb
MetricName [name=io-ratio, group=producer-metrics, description=The fraction of time the I/O thread spent doing I/O, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@635de6df
MetricName [name=count, group=kafka-metrics-count, description=total number of registered metrics, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@a853c47
MetricName [name=request-latency-avg, group=producer-metrics, description=The average request latency in ms, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@9bd2ce
MetricName [name=record-error-rate, group=producer-metrics, description=The average per-second number of record sends that resulted in errors, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@65a74c71
MetricName [name=metadata-age, group=producer-metrics, description=The age in seconds of the current producer metadata being used., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@69403504
MetricName [name=request-size-max, group=producer-metrics, description=The maximum size of any request sent in the window., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4a044b6f
MetricName [name=requests-in-flight, group=producer-metrics, description=The current number of in-flight requests awaiting a response., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@1457fd43
MetricName [name=buffer-available-bytes, group=producer-metrics, description=The total amount of buffer memory that is not being used (either unallocated or in the free list)., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4bccef7c
MetricName [name=io-time-ns-avg, group=producer-metrics, description=The average length of time for I/O per select call in nanoseconds., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@23cc0b14
MetricName [name=buffer-total-bytes, group=producer-metrics, description=The maximum amount of buffer memory the client can use (whether or not it is currently used)., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4b7b6331
MetricName [name=waiting-threads, group=producer-metrics, description=The number of user threads blocked waiting for buffer memory to enqueue their records, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@60ed22f8
MetricName [name=records-per-request-avg, group=producer-metrics, description=The average number of records per request., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4cf68696
MetricName [name=record-queue-time-max, group=producer-metrics, description=The maximum time in ms record batches spent in the record accumulator., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@f21b0b7
MetricName [name=batch-size-avg, group=producer-metrics, description=The average number of bytes sent per partition per-request., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@1f533889
MetricName [name=response-rate, group=producer-metrics, description=Responses received sent per second., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@16202379
MetricName [name=bufferpool-wait-ratio, group=producer-metrics, description=The fraction of time an appender waits for space allocation., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@6e1ac51d
MetricName [name=network-io-rate, group=producer-metrics, description=The average number of network operations (reads or writes) on all connections per second., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@6cbc6461
MetricName [name=incoming-byte-rate, group=producer-metrics, description=Bytes/second read off all sockets, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@6aad8e0f
MetricName [name=batch-size-max, group=producer-metrics, description=The max number of bytes sent per partition per-request., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@7d0143c8
MetricName [name=record-retry-rate, group=producer-metrics, description=The average per-second number of retried record sends, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@458b5358
MetricName [name=produce-throttle-time-avg, group=producer-metrics, description=The average throttle time in ms, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4a9bb8e4

Producer的错误有两种

1.可重试错误这种可以通过重发消息来解决。

2.不可重试错误比如消息太大导致的异常，这时候不会进行任何重试，直接抛出。

获取生产者维护的全套内部指标,上面就是打印出来的全部指标。

问题1：如何指定自定义分区策略或者把消息发到指定分区。

自定义类实现Partitioner接口，Partitioner接口包含configure() Partition() close()三个方法。Partition()为分区方法。

问题2：如何使用其他的序列化方式

问题3：如何保证顺序

如果某些场景要求消息是有序的，那么消息是否写入成功也是很关键的，所以不建议把retries设为0 。可以把Max.in.flight.requests.per.connection 设为1 ，这样在生产者尝试发送第一批悄息时，就不会有其
他的消息发送给broker 。不过这样会严重影响生产者的吞吐量，所以只有在对消息的顺序有严格要求的情况下才能这么做。

retries消息重试次数

Max.in.flight.requests.per.connection 在一个connection中发送多个请求，叫作一个flight,这样可以减少开销，但是如果产生错误，可能会造成数据的发送顺序改变,默认是5 (修改）。