Producer
1.发送并忘记 这种情况下可能会丢数据。
2.同步发送 它会返回一个Future对象,使用get()方法进行等待,就可以知道它是否发送成功。
3.异步发送 指定一个回调函数,服务器在进行响应时调用
public Future<RecordMetadata> send(ProducerRecord<K, V> record);
public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback);
这两个函数均为发送消息的函数,不同点在于第二个函数多一个回调函数的参数。
public void close() {
close(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
}
public void close(long timeout, TimeUnit timeUnit) {
close(timeout, timeUnit, false);
}
关闭producer的函数,不同之处在于第二个函数可以再指定时间内关闭
public List<PartitionInfo> partitionsFor(String topic) {
try {
return waitOnMetadata(topic, null, maxBlockTimeMs).cluster.partitionsForTopic(topic);
} catch (InterruptedException e) {
throw new InterruptException(e);
}
}
获得topic的分区信息
public Map<MetricName, ? extends Metric> metrics() {
return Collections.unmodifiableMap(this.metrics.metrics());
}
MetricName [name=io-wait-ratio, group=producer-metrics, description=The fraction of time the I/O thread spent waiting., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@462ee862
MetricName [name=request-size-avg, group=producer-metrics, description=The average size of all requests in the window.., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@48e90ed7
MetricName [name=produce-throttle-time-max, group=producer-metrics, description=The maximum throttle time in ms, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4cd68312
MetricName [name=connection-creation-rate, group=producer-metrics, description=New connections established per second in the window., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@9d404ec
MetricName [name=record-size-avg, group=producer-metrics, description=The average record size, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@3808904c
MetricName [name=connection-close-rate, group=producer-metrics, description=Connections closed per second in the window., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@3897ae2c
MetricName [name=record-queue-time-avg, group=producer-metrics, description=The average time in ms record batches spent in the record accumulator., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@40a1643f
MetricName [name=request-latency-max, group=producer-metrics, description=The maximum request latency in ms, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@69fbfba3
MetricName [name=request-rate, group=producer-metrics, description=The average number of requests sent per second., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@158d9c20
MetricName [name=record-send-rate, group=producer-metrics, description=The average number of records sent per second., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@106eb673
MetricName [name=io-wait-time-ns-avg, group=producer-metrics, description=The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@5ba5a8dd
MetricName [name=connection-count, group=producer-metrics, description=The current number of active connections., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@6db566e9
MetricName [name=outgoing-byte-rate, group=producer-metrics, description=The average number of outgoing bytes sent per second to all servers., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@14a09be3
MetricName [name=select-rate, group=producer-metrics, description=Number of times the I/O layer checked for new I/O to perform per second, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@71186a2a
MetricName [name=record-size-max, group=producer-metrics, description=The maximum record size, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@78dc794
MetricName [name=buffer-exhausted-rate, group=producer-metrics, description=The average per-second number of record sends that are dropped due to buffer exhaustion, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@41c96ba2
MetricName [name=compression-rate-avg, group=producer-metrics, description=The average compression rate of record batches., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@51a68ffb
MetricName [name=io-ratio, group=producer-metrics, description=The fraction of time the I/O thread spent doing I/O, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@635de6df
MetricName [name=count, group=kafka-metrics-count, description=total number of registered metrics, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@a853c47
MetricName [name=request-latency-avg, group=producer-metrics, description=The average request latency in ms, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@9bd2ce
MetricName [name=record-error-rate, group=producer-metrics, description=The average per-second number of record sends that resulted in errors, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@65a74c71
MetricName [name=metadata-age, group=producer-metrics, description=The age in seconds of the current producer metadata being used., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@69403504
MetricName [name=request-size-max, group=producer-metrics, description=The maximum size of any request sent in the window., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4a044b6f
MetricName [name=requests-in-flight, group=producer-metrics, description=The current number of in-flight requests awaiting a response., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@1457fd43
MetricName [name=buffer-available-bytes, group=producer-metrics, description=The total amount of buffer memory that is not being used (either unallocated or in the free list)., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4bccef7c
MetricName [name=io-time-ns-avg, group=producer-metrics, description=The average length of time for I/O per select call in nanoseconds., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@23cc0b14
MetricName [name=buffer-total-bytes, group=producer-metrics, description=The maximum amount of buffer memory the client can use (whether or not it is currently used)., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4b7b6331
MetricName [name=waiting-threads, group=producer-metrics, description=The number of user threads blocked waiting for buffer memory to enqueue their records, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@60ed22f8
MetricName [name=records-per-request-avg, group=producer-metrics, description=The average number of records per request., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4cf68696
MetricName [name=record-queue-time-max, group=producer-metrics, description=The maximum time in ms record batches spent in the record accumulator., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@f21b0b7
MetricName [name=batch-size-avg, group=producer-metrics, description=The average number of bytes sent per partition per-request., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@1f533889
MetricName [name=response-rate, group=producer-metrics, description=Responses received sent per second., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@16202379
MetricName [name=bufferpool-wait-ratio, group=producer-metrics, description=The fraction of time an appender waits for space allocation., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@6e1ac51d
MetricName [name=network-io-rate, group=producer-metrics, description=The average number of network operations (reads or writes) on all connections per second., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@6cbc6461
MetricName [name=incoming-byte-rate, group=producer-metrics, description=Bytes/second read off all sockets, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@6aad8e0f
MetricName [name=batch-size-max, group=producer-metrics, description=The max number of bytes sent per partition per-request., tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@7d0143c8
MetricName [name=record-retry-rate, group=producer-metrics, description=The average per-second number of retried record sends, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@458b5358
MetricName [name=produce-throttle-time-avg, group=producer-metrics, description=The average throttle time in ms, tags={client-id=MsgProducer}]=org.apache.kafka.common.metrics.KafkaMetric@4a9bb8e4
Producer的错误有两种
1.可重试错误 这种可以通过重发消息来解决。
2.不可重试错误 比如消息太大导致的异常,这时候不会进行任何重试,直接抛出。
获取生产者维护的全套内部指标,上面就是打印出来的全部指标。
问题1:如何指定自定义分区策略 或者把消息发到指定分区。
自定义类实现Partitioner接口,Partitioner接口包含configure() Partition() close()三个方法。Partition()为分区方法。
问题2:如何使用其他的序列化方式
问题3:如何保证顺序
如果某些场景要求消息是有序的,那么消息是否写入成功也是很关键的,所以不建议把retries设为0 。可以把Max.in.flight.requests.per.connection 设为1 ,这样在生产者尝试发送第一批悄息时,就不会有其
他的消息发送给broker 。不过这样会严重影响生产者的吞吐量,所以只有在对消息的顺序有严格要求的情况下才能这么做。
retries消息重试次数
Max.in.flight.requests.per.connection 在一个connection中发送多个请求,叫作一个flight,这样可以减少开销,但是如果产生错误,可能会造成数据的发送顺序改变,默认是5 (修改)。