在Java中使用Kafka

本文详细介绍了Kafka的生产者与消费者的工作原理及其实现。生产者负责将数据发送到指定的主题和分区,而消费者则订阅主题,处理接收到的消息。文章深入探讨了配置参数的作用,如acks、retries等,以及如何使用KafkaProducer和KafkaConsumer接口进行消息的生产和消费。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Producer部分

Producer在实例化后, 对外提供send方法, 用于将数据送到指定的topic和partition; 以及在退出时需要的destroy方法.

接口 KafkaProducer.java

import java.util.List;
import java.util.Properties;

public interface KafkaProducer<D> {

    default void init() {
    }
    default void destroy() {
    }
    boolean send(String topic, D data);
    boolean send(String topic, Integer partition, D data);
    boolean send(String topic, List<D> dataList);
    boolean send(String topic, Integer partition, List<D> dataList);

    /**
     * 默认配置
     */
    default Properties getDefaultProps() {
        Properties props = new Properties();
        props.put("acks", "1");
        props.put("retries", 1);
        props.put("batch.size", 16384);
        props.put("linger.ms", 1);
        props.put("buffer.memory", 32 * 1024 * 1024L);
        return props;
    }
}

参数说明

Properties props = new Properties(); 
props.put("bootstrap.servers", "localhost:9092"); 
// The acks config controls the criteria under which requests are considered complete. The "all" setting we have specified will result in blocking on the full commit of the record, the slowest but most durable setting. 
props.put("acks", "all"); 
// If the request fails, the producer can automatically retry, though since we have specified retries as 0 it won't. Enabling retries also opens up the possibility of duplicates (see the documentation on message delivery semantics for details). 
props.put("retries", 0); 
// The producer maintains buffers of unsent records for each partition. These buffers are of a size specified by the batch.size config. Making this larger can result in more batching, but requires more memory (since we will generally have one of these buffers for each active partition). 
props.put("batch.size", 16384); 
// By default a buffer is available to send immediately even if there is additional unused space in the buffer. However if you want to reduce the number of requests you can set linger.ms to something greater than 0. This will instruct the producer to wait up to that number of milliseconds before sending a request in hope that more records will arrive to fill up the same batch.
props.put("linger.ms", 1); 
// 生产者缓冲大小,当缓冲区耗尽后,额外的发送调用将被阻塞。时间超过max.block.ms将抛出TimeoutException 
props.put("buffer.memory", 33554432); 
// The key.serializer and value.serializer instruct how to turn the key and value objects the user provides with their ProducerRecord into bytes. You can use the included ByteArraySerializer or StringSerializer for simple string or byte types. 
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); 

实现 KafkaProducerImpl.java

import com.google.common.base.Strings;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.List;
import java.util.Map;
import java.util.Properties;

public class KafkaProducerImpl<D> implements KafkaProducer<D> {
    private static final Logger logger = LoggerFactory.getLogger(KafkaProducerImpl.class);
    private final Producer<D, D> producer;

    public KafkaProducerImpl() {
        Properties props = this.getDefaultProps();
        props.put("bootstrap.servers", servers);
        props.put("key.serializer", serializer);
        props.put("value.serializer", serializer);
        producer = new org.apache.kafka.clients.producer.KafkaProducer<>(props);
    }

    @Override
    public void destroy() {
        if (producer != null) {
            producer.close();
        }
    }

    @Override
    public boolean send(String topic, D data) {
        boolean isSuc = true;
        try {
            producer.send(new ProducerRecord<>(topic, data));
        } catch (Exception e) {
            isSuc = false;
            logger.error(String.format("KafkaStringProducer send error.topic:[%s],data:[%s]", topic, data), e);
        }
        return isSuc;
    }

    @Override
    public boolean send(String topic, Integer partition, D data) {
        boolean isSuc = true;
        try {
            producer.send(new ProducerRecord<>(topic, partition, null, data));
        } catch (Exception e) {
            isSuc = false;
            logger.error(String.format("KafkaStringProducer send error.topic:[%s],data:[%s]", topic, data), e);
        }
        return isSuc;
    }

    @Override
    public boolean send(String topic, List<D> dataList) {
        boolean isSuc = true;
        try {
            if (dataList != null) {
                dataList.forEach(item -> producer.send(new ProducerRecord<>(topic, item)));
            }
        } catch (Exception e) {
            isSuc = false;
            logger.error(String.format("KafkaStringProducer send error.topic:[%s],dataList:[%s]", topic, dataList), e);
        }
        return isSuc;
    }

    @Override
    public boolean send(String topic, Integer partition, List<D> dataList) {
        boolean isSuc = true;
        try {
            if (dataList != null) {
                dataList.forEach(item -> producer.send(new ProducerRecord<>(topic, partition, null, item)));
            }
        } catch (Exception e) {
            isSuc = false;
            logger.error(String.format("KafkaStringProducer send error.topic:[%s],partition[%s],dataList:[%s]", topic, partition, dataList), e);
        }
        return isSuc;
    }
}

 

Consumer 部分

Consumer 在实例化后, 负责将ConsumerListener添加到列表, 并订阅指定的topic, 启动一个阻塞的循环, 在收到消息后依次调用ConsumerListener进行处理

接口 KafkaConsumer.java

import java.util.Properties;

public interface KafkaConsumer {

    default void init() {
    }

    default void destroy() {
    }

    void start();

    /**
     * 默认配置
     */
    default Properties getDefaultProps() {
        Properties props = new Properties();
        props.put("enable.auto.commit", "true");
        props.put("auto.commit.interval.ms", "1000");
        props.put("session.timeout.ms", "30000");
        return props;
    }
}  

参数说明

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
// Setting enable.auto.commit means that offsets are committed automatically with a frequency controlled by the config auto.commit.interval.ms.
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
// The deserializer settings specify how to turn bytes into objects. For example, by specifying string deserializers, we are saying that our record's key and value will just be simple strings. 
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
// This consumer is subscribing to the topics foo and bar as part of a group of consumers called test as configured with group.id. 
consumer.subscribe(Arrays.asList("foo", "bar"));
while (true) {
 ConsumerRecords<String, String> records = consumer.poll(100);
 for (ConsumerRecord<String, String> record : records)
     System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}

实现 KafkaConsumerImpl.java

import com.google.common.base.Strings;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.*;

public class KafkaConsumerImpl<K, V> implements KafkaConsumer {
    private static final Logger logger = LoggerFactory.getLogger(KafkaConsumerImpl.class);
    private final List<KafkaConsumerListener<K, V>> consumerListeners = new ArrayList<>();
    private Consumer<K, V> consumer;
    private boolean running = true;

    private final int waitingTimeout = 100;

    public KafkaConsumerImpl(String topic, String groupId, String deserializer) {
        Properties props = this.getDefaultProps();
        props.put("group.id", groupId);
        props.put("bootstrap.servers", servers);
        props.put("key.deserializer", deserializer);
        props.put("value.deserializer", deserializer);
        consumer = new org.apache.kafka.clients.consumer.KafkaConsumer<>(props);
        consumer.subscribe(Arrays.asList(topic));
    }

    public void setConsumerListeners(List<KafkaConsumerListener<K, V>> consumerListeners) {
        synchronized (this) {
            this.consumerListeners.clear();
            if (null != consumerListeners && 0 != consumerListeners.size()) {
                consumerListeners.forEach(this.consumerListeners::add);
            }
        }
    }

    public void addConsumerListener(KafkaConsumerListener<K, V> consumerListener) {
        synchronized (this) {
            if (null != consumerListener && !this.consumerListeners.contains(consumerListener)) {
                this.consumerListeners.add(consumerListener);
            }
        }
    }

    public void removeConsumerListener(KafkaConsumerListener<K, V> consumerListener) {
        synchronized (this) {
            if (null != consumerListener && this.consumerListeners.contains(consumerListener)) {
                this.consumerListeners.remove(consumerListener);
            }
        }
    }

    @Override
    public void init() {
        this.start();
    }

    @Override
    public void destroy() {
        running = false;
    }

    @Override
    public void start() {
        new Thread(() -> {
            while (running) {
                ConsumerRecords<K, V> records = consumer.poll(waitingTimeout);
                for (ConsumerRecord<K, V> record : records) {
                    if (consumerListeners != null) {
                        K key = record.key();
                        if (key == null)
                            consumerListeners.forEach(consumer -> consumer.consume(record.value()));
                        else
                            consumerListeners.forEach(consumer -> consumer.consume(record.key(), record.value()));
                    }
                }
            }
            //should use consumer in different thread, or it will throw ConcurrentModificationException
            if (consumer != null) {
                try {
                    logger.info("start to close consumer.");
                    consumer.close();
                } catch (Exception e) {
                    logger.error("close kafka consumer error.", e);
                }
                consumer = null;
            }
        }).start();
    }
}

接口 KafkaConsumerListener.java

public interface KafkaConsumerListener<K, V> {
    void consume(V value);

    default void consume(K key, V value) {
        consume(value);
    }
}

.

 

### Java使用 Kafka 的教程 #### 1. 添加依赖项 为了在 Java 应用程序中使用 Kafka,首先需要引入 Apache Kafka 客户端库。如果项目基于 Maven 构建,则可以在 `pom.xml` 文件中添加以下依赖项: ```xml <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>3.0.0</version> <!-- 版本号可以根据实际需求调整 --> </dependency> ``` 此操作会自动下载并集成 Kafka 所需的客户端类[^1]。 --- #### 2. 创建生产者 (Producer) 以下是创建 Kafka 生产者的代码示例: ```java import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerRecord; import java.util.Properties; public class KafkaProducerExample { public static void main(String[] args) { Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); // Kafka 集群地址 props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) { ProducerRecord<String, String> record = new ProducerRecord<>("test-topic", "key", "message-value"); producer.send(record); } catch (Exception e) { e.printStackTrace(); } } } ``` 在此代码片段中,设置了 Kafka Broker 地址以及序列化器类型,并通过 `producer.send()` 方法发送消息到指定主题。 --- #### 3. 创建消费者 (Consumer) 下面是一个简单的 Kafka 消费者实现方式: ```java import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.clients.consumer.KafkaConsumer; import java.time.Duration; import java.util.Collections; import java.util.Properties; public class KafkaConsumerExample { public static void main(String[] args) { Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); // Kafka 集群地址 props.put("group.id", "test-group"); // 消费组 ID props.put("enable.auto.commit", "true"); // 自动提交偏移量 props.put("auto.commit.interval.ms", "1000"); // 提交间隔时间 props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) { consumer.subscribe(Collections.singletonList("test-topic")); // 订阅的主题名称 while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100)); records.forEach(record -> { System.out.printf("Partition = %d, Offset = %d, Key = %s, Value = %s%n", record.partition(), record.offset(), record.key(), record.value()); }); } } } } ``` 该代码展示了如何设置消费者的属性、订阅特定主题,并通过轮询机制获取数据记录[^2]。 --- #### 4. 解决常见问题 当遇到连接失败或其他异常情况时,可以尝试以下方法解决问题: - **确认主机名配置** 如果运行环境涉及虚拟机或分布式部署,请确保 Kafka Server 属性文件中的 `host.name` 被正确定义为当前机器的实际 IP 地址[^3]。 - **检查防火墙规则** 确认目标服务器上的端口(默认为 9092)未被阻塞。 - **验证 Zookeeper 运行状态** Kafka 依赖于 Zookeeper 来管理元数据信息;因此,在启动之前应先初始化 Zookeeper 实例。 --- ### 总结 以上介绍了如何利用 Java 编程语言与 Kafka 平台交互的过程,涵盖了从基础依赖导入至高级功能开发的具体实践案例。希望这些资料能够帮助开发者快速掌握相关技能!
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值