Kafka 整合 SpringBoot demo

在windows上部署完kafka后，我又开始整合springboot...

这篇文章建立在kafka部署完的基础上...

如果还没安装运行kafka可以参照《Kafka windows下安装和使用》https://blog.youkuaiyun.com/Alex19961223/article/details/104000790

一、项目架构

二、pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.ylz.demo</groupId>
	<artifactId>kafka-test</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<name>kafka-test</name>
	<description></description>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
		<springboot.version>2.1.0.RELEASE</springboot.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-dependencies</artifactId>
			<version>${springboot.version}</version>
			<type>pom</type>
			<scope>import</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
			<version>${springboot.version}</version>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<version>${springboot.version}</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.kafka</groupId>
			<artifactId>spring-kafka</artifactId>
			<version>2.2.10.RELEASE</version>
		</dependency>
		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<version>1.18.2</version>
		</dependency>
		<dependency>
			<groupId>com.google.code.gson</groupId>
			<artifactId>gson</artifactId>
			<version>2.8.2</version>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>2.3.2</version>
				<configuration>
					<source>1.8</source>
					<target>1.8</target>
					<encoding>UTF-8</encoding>
				</configuration>
			</plugin>
		</plugins>
	</build>
</project>

tips:kafka版本和spring版本有依赖关系，具体参照srping官方文档 https://spring.io/projects/spring-kafka

三、消息实体类 Message.java 

@Data
public class Message {    
    private Long id;           //id
    private String msg;        //消息
    private Date sendTime;     //时间戳
}

四、消息发送类 Sender.java

@Component
@Slf4j
public class Sender {
    @Autowired
    private KafkaTemplate kafkaTemplate;
    private Gson gson = new GsonBuilder().create();

    public void send(){
        Message message = new Message();
        message.setId(System.currentTimeMillis());
        message.setMsg("testJson: " + UUID.randomUUID().toString());
        message.setSendTime(new Date());
        log.info("message={}", gson.toJson(message));
        kafkaTemplate.send("testJson", gson.toJson(message));
    }
}

五、消息接收类 Receiver.java

@Component
@Slf4j
public class Receiver {
    @KafkaListener(topics = "streamingTopic1")
    public void listener(ConsumerRecord record) {
        Optional kafkaMessage = Optional.ofNullable(record.value());
        if (kafkaMessage.isPresent()) {
            Object message = kafkaMessage.get();
            log.info("record--{}", record);
            log.info("message--{}", message);
        }
    }
}

六、springboot 启动类 KafkaTestApplication.java

@SpringBootApplication
@EnableKafka
public class KafkaTestApplication {

    public static void main(String[] args) {
        ConfigurableApplicationContext context = SpringApplication.run(KafkaTestApplication.class, args);
        Sender sender = context.getBean(Sender.class);
        for (int i = 0; i < 3; i++) {
            //调用消息发送类中的消息发送方法
            sender.send();

            try {
                Thread.sleep(3000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}

七、配置文件 application.properties

server.port=10310
#============== kafka ===================
# 指定kafka 代理地址，可以多个
spring.kafka.bootstrap-servers=127.0.0.1:9092

#=============== provider  =======================

spring.kafka.producer.retries=0
# 字节数默认
spring.kafka.producer.batch-size=16384
spring.kafka.producer.buffer-memory=33554432

# 指定消息key和消息体的编解码方式
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

#=============== consumer  =======================
# 指定默认消费者group id
spring.kafka.consumer.group-id=test-consumer-group

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.auto-commit-interval=100

# 指定消息key和消息体的编解码方式
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer

八、开始测试

1.启动zk

2.启动kafka

3.运行demo KafkaTestApplication.java

4.在kafka安装目录下打开命令窗口

5.mock一个消费者 customer， 接收程序发送的测试消息 ；

注意主题topic需要和Sender.java里的 kafkaTemplate.send("testJson", gson.toJson(message)); topic参数保持一致

bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic testJson

 

如图，模拟的消费者成功接收到程序传递的json消息串 

6.mock一个生产者 producer， 发送测试消息 hello kafka ；

注意主题topic需要和Receiver.java里的@KafkaListener(topics = "streamingTopic1") topics参数保持一致

bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic streamingTopic1

如图，程序成功接收到生产者传递的消息 hello spring-kafka

 

参考：

1.《SpringBoot Kafka 整合使用》https://juejin.im/post/5d406a925188255d352ab24e

2. Spring for Apache Kafka https://docs.spring.io/spring-kafka/reference/html/

3. Kafka API https://docs.spring.io/spring-kafka/api/org/springframework/kafka/annotation/KafkaListener.html