Kafka基础知识及示例

最新推荐文章于 2025-02-05 23:35:57 发布

control CV工程师

最新推荐文章于 2025-02-05 23:35:57 发布

阅读量284

点赞数

文章标签： kafka java

本文链接：https://blog.youkuaiyun.com/lixin1997/article/details/132545335

版权

本文介绍了Kafka的基本概念，如阻塞队列和Zookeeper的作用，以及如何在SpringBoot环境中集成和配置Kafka，包括生产者和消费者的创建，以及发送和接收消息的示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

前置知识

在学习Kafka之前，需要先了解什么是阻塞队列和zookeeper，阻塞队列是本次Kafka实现消息缓冲的一种方式，而ZooKeeper用于管理Kafka集群的状态和元数据。

阻塞队列

BlockingQueue

解决线程通信的问题。
阻塞方法：put、take。

生产者消费者模式

生产者：产生数据的线程。
消费者：使用数据的线程。

实现类

ArrayBlockingQueue
LinkedBlockingQueue
PriorityBlockingQueue、SynchronousQueue、DelayQueue等。

zookeeper

ZooKeeper主要服务于分布式系统，可以用ZooKeeper来做：统一配置管理、统一命名服务、分布式锁、集群管理。

Kafka入门

Kafka简介

Kafka是一个分布式的流媒体平台。
应用：消息系统、日志收集、用户行为追踪、流式处理。

Kafka特点

高吞吐量、消息持久化、高可靠性、高扩展性。

Kafka术语

Broker：Kafka的实例
Zookeeper - Topic、Partition、Offset - Leader Replica 、Follower Replica

Spring整合Kafka

1. 引入依赖

此处因为我将Spring-boot版本统一管理，因而不需要写版本号

		<dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>

编辑配置文件

# KafkaProperties
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=test-consumer-group
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.auto-commit-interval=3000

2. 安装、配置Kafka（mac版本）

安装Kafka

上官网下载安装包或者用brew安装
安装后修改两个配置文件的路径zookeeper.properties和server.properties

以下内容需要进入Kafka文件的bin目录

启动Zookeeper

./zookeeper-server-start.sh /Users/lx/kafka_2.12-2.5.1/config/zookeeper.properties

启动Kafka

./kafka-server-start.sh /Users/lx/kafka_2.12-2.5.1/config/server.properties

创建生产者

./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

创建消费者

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

关闭Kafka和zookeeper

./kafka-server-stop.sh
./zookeeper-server-stop.sh

3. 访问Kafka

生产者
kafkaTemplate.send(topic, data);
消费者
@KafkaListener(topics = {“test”})
public void handleMessage(ConsumerRecord record) {}

测试类

public class KafkaTest {
    @Autowired
    KafkaProducer kafkaProducer;
    @Test
    public void testKafka() {
        kafkaProducer.sendMessage("test", "hello");
        kafkaProducer.sendMessage("test", "how are you");
        try {
            Thread.sleep(1000 * 5);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

@Component
class KafkaProducer {
    @Autowired
    KafkaTemplate kafkaTemplate;

    public void sendMessage(String topic, String context) {
        kafkaTemplate.send(topic, context);
    }

}

@Component
class KafkaConsumer {
    @KafkaListener(topics = "test")
    public void handleMessage(ConsumerRecord record) {
        System.out.println(record.value());
    }
}

用Kafka发送系统通知

触发事件

评论后，发布通知
点赞后，发布通知
关注后，发布通知

处理事件

封装事件对象
开发事件的生产者
开发事件的消费者

创建触发事件实体类

public class Event {
    private String topic;
    // 事件触发者ID
    private int userId;
    // 实体类型
    private int entityType;
    // 实体ID
    private int entityId;
    // 实体拥有者Id
    private int entityUserId;
    // 存放数据Map
    Map<String, Object> data = new HashMap<>();

    public String getTopic() {
        return topic;
    }

    public Event setTopic(String topic) {
        this.topic = topic;
        return this;
    }

    public int getUserId() {
        return userId;
    }

    public Event setUserId(int userId) {
        this.userId = userId;
        return this;
    }

    public int getEntityType() {
        return entityType;
    }

    public Event setEntityType(int entityType) {
        this.entityType = entityType;
        return this;
    }

    public int getEntityId() {
        return entityId;
    }

    public Event setEntityId(int entityId) {
        this.entityId = entityId;
        return this;
    }

    public int getEntityUserId() {
        return entityUserId;
    }

    public Event setEntityUserId(int entityUserId) {
        this.entityUserId = entityUserId;
        return this;
    }

    public Map<String, Object> getData() {
        return data;
    }

    public Event setData(String key, Object value) {
        this.data.put(key, value);
        return this;
    }
}

创建事件触发生产者和消费者
生产者

@Component
public class EventProducer{
    @Autowired
    KafkaTemplate kafkaTemplate;

    public void fireEvent(Event event) {
        kafkaTemplate.send(event.getTopic(), JSONObject.toJSONString(event));
    }
}

消费者

@Component
public class EventConsumer implements CommunityConstant {
    private static final Logger logger = LoggerFactory.getLogger(EventConsumer.class);

    @Autowired
    MessageService messageService;

    @KafkaListener(topics = {TOPIC_COMMENT, TOPIC_FOLLOW,TOPIC_LIKE})
    public void handleCommentMessage(ConsumerRecord record) {
        if (record == null || record.value() == null) {
            logger.error("消息的内容为空！");
            return;
        }
        Event event = JSONObject.parseObject(record.value().toString(), Event.class);
        if (event == null) {
            logger.error("消息格式错误！");
            return;
        }
        // 发送消息
        Message message = new Message();
        message.setFromId(SYSTEM_USER_ID);
        message.setConversationId(event.getTopic());
        message.setToId(event.getEntityUserId());
        message.setCreateTime(new Date());

        // 将其他信息放到comment中
        Map<String, Object> content = new HashMap<>();
        content.put("userId", event.getUserId());
        content.put("entityType", event.getEntityType());
        content.put("entityId", event.getEntityId());

        if (!event.getData().isEmpty()) {
            for (Map.Entry<String, Object> entry : event.getData().entrySet()) {
                content.put(entry.getKey(), entry.getValue());
            }
        }

        message.setContent(JSONObject.toJSONString(content));
        messageService.addLetter(message);
    }

}

将生产者分别融入到点赞、评论、关注处
关注

    // 关注
    @RequestMapping(path = "/follow", method = RequestMethod.POST)
    @ResponseBody
    public String follow(int entityType, int entityId) {
        User user = hostHolder.getUser();

        followService.follow(entityType, entityId, user.getId());

        Event event = new Event()
                .setTopic(TOPIC_FOLLOW)
                .setUserId(hostHolder.getUser().getId())
                .setEntityType(entityType)
                .setEntityId(entityId)
                .setEntityUserId(entityId);

        eventProducer.fireEvent(event);
        return CommunityUtil.getJSONString(0, "已关注!");
    }

    @RequestMapping(path = "add/{discussPostId}", method = RequestMethod.POST)
    public String addComment(@PathVariable("discussPostId") int discussPostId, Comment comment) {
        comment.setUserId(hostHolder.getUser().getId());
        comment.setCreateTime(new Date());
        comment.setStatus(0);

        commentService.addComment(comment);

        Event event = new Event()
                .setTopic(TOPIC_COMMENT)
                .setUserId(hostHolder.getUser().getId())
                .setEntityType(comment.getEntityType())
                .setEntityId(comment.getEntityId())
                // 连接到帖子详情页面
                .setData("postId", discussPostId);
        if (comment.getEntityType() == ENTITY_TYPE_POST) {
            DiscussPost target = discussPostService.findDiscussPostById(comment.getEntityId());
            event.setEntityUserId(target.getUserId());
        } else if (comment.getEntityType() == ENTITY_TYPE_COMMENT) {
            Comment target = commentService.findCommentById(comment.getEntityId());
            event.setEntityUserId(target.getUserId());
        }

        eventProducer.fireEvent(event);

        return "redirect:/discuss/detail/" + discussPostId;
    }

    @RequestMapping(path = "/like", method = RequestMethod.POST)
    @ResponseBody
    public String like(int entityType, int entityId, int entityUserId) {
        // 需要拦截器
        // 当前用户
        User user = hostHolder.getUser();

        // 点赞
        likeService.like(user.getId(), entityType, entityId, entityUserId);

        // 点赞数量
        long likeCount = likeService.findEntityLikeCount(entityType, entityId);

        // 点赞状态
        int likeStatus = likeService.findEntityLikeStatus(user.getId(), entityType, entityId);

        // 返回结果
        Map<String, Object> map = new HashMap<>();
        map.put("likeCount", likeCount);
        map.put("likeStatus", likeStatus);

        // 判断如果是取消点赞就不通知
        if (likeStatus == 1) {
            Event event = new Event()
                    .setTopic(TOPIC_LIKE)
                    .setUserId(hostHolder.getUser().getId())
                    .setEntityType(entityType)
                    .setEntityId(entityId)
                    .setEntityUserId(entityUserId)
                    .setData("postId",entityId);
            eventProducer.fireEvent(event);
        }
        return CommunityUtil.getJSONString(0,null,map);
    }

用Kafka显示系统通知

通知列表

显示评论、点赞、关注三种类型的通知

通知详情

分页显示某一类主题所包含的通知

未读消息

在页面头部显示所有的未读消息数量，需要配置拦截器


    @RequestMapping(path = "/notice/list", method = RequestMethod.GET)
    public String getNoticeList(Model model) {
        User user = hostHolder.getUser();

        // 查询评论类通知
        Message message = messageService.findLatestTopic(user.getId(), TOPIC_COMMENT);
        Map<String, Object> messageVO = new HashMap<>();
        if (message != null) {
            messageVO.put("message", message);

            String content = HtmlUtils.htmlUnescape(message.getContent());
            Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);

            messageVO.put("user", userService.findUserById((Integer) data.get("userId")));
            messageVO.put("entityType", data.get("entityType"));
            messageVO.put("entityId", data.get("entityId"));
            messageVO.put("postId", data.get("postId"));

            int count = messageService.findNoticesCount(user.getId(), TOPIC_COMMENT);
            messageVO.put("count", count);

            int unread = messageService.findUnreadNoticesCount(user.getId(), TOPIC_COMMENT);
            messageVO.put("unread", unread);
        }
        model.addAttribute("commentNotice", messageVO);

        // 查询点赞类通知
        message = messageService.findLatestTopic(user.getId(), TOPIC_LIKE);
        messageVO = new HashMap<>();
        if (message != null) {
            messageVO.put("message", message);

            String content = HtmlUtils.htmlUnescape(message.getContent());
            Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);

            messageVO.put("user", userService.findUserById((Integer) data.get("userId")));
            messageVO.put("entityType", data.get("entityType"));
            messageVO.put("entityId", data.get("entityId"));
            messageVO.put("postId", data.get("postId"));

            int count = messageService.findNoticesCount(user.getId(), TOPIC_LIKE);
            messageVO.put("count", count);

            int unread = messageService.findUnreadNoticesCount(user.getId(), TOPIC_LIKE);
            messageVO.put("unread", unread);
        }
        model.addAttribute("likeNotice", messageVO);

        // 查询关注类通知
        message = messageService.findLatestTopic(user.getId(), TOPIC_FOLLOW);
        messageVO = new HashMap<>();
        if (message != null) {
            messageVO.put("message", message);

            String content = HtmlUtils.htmlUnescape(message.getContent());
            Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);

            messageVO.put("user", userService.findUserById((Integer) data.get("userId")));
            messageVO.put("entityType", data.get("entityType"));
            messageVO.put("entityId", data.get("entityId"));

            int count = messageService.findNoticesCount(user.getId(), TOPIC_FOLLOW);
            messageVO.put("count", count);

            int unread = messageService.findUnreadNoticesCount(user.getId(), TOPIC_FOLLOW);
            messageVO.put("unread", unread);
        }
        model.addAttribute("followNotice", messageVO);

        // 查询未读消息数量
        int letterUnreadCount = messageService.findLetterUnreadCount(user.getId(), null);
        model.addAttribute("letterUnreadCount", letterUnreadCount);
        int noticeUnreadCount = messageService.findUnreadNoticesCount(user.getId(), null);
        model.addAttribute("noticeUnreadCount", noticeUnreadCount);

        return "/site/notice";
    }

    @RequestMapping(path = "/notice/detail/{topic}", method = RequestMethod.GET)
    public String getNoticeDetail(@PathVariable("topic") String topic, Page page, Model model) {
        User user = hostHolder.getUser();

        page.setLimit(5);
        page.setPath("/notice/detail/" + topic);
        page.setRows(messageService.findNoticesCount(user.getId(), topic));

        List<Message> noticeList = messageService.findNotices(user.getId(), topic, page.getOffset(), page.getLimit());
        List<Map<String, Object>> noticeVoList = new ArrayList<>();
        if (noticeList != null) {
            for (Message notice : noticeList) {
                Map<String, Object> map = new HashMap<>();
                // 通知
                map.put("notice", notice);
                // 内容
                String content = HtmlUtils.htmlUnescape(notice.getContent());
                Map<String, Object> data = JSONObject.parseObject(content, HashMap.class);
                map.put("user", userService.findUserById((Integer) data.get("userId")));
                map.put("entityType", data.get("entityType"));
                map.put("entityId", data.get("entityId"));
                map.put("postId", data.get("postId"));
                // 通知作者
                map.put("fromUser", userService.findUserById(notice.getFromId()));

                noticeVoList.add(map);
            }
        }
        model.addAttribute("notices", noticeVoList);

        // 设置已读
        List<Integer> ids = getUnReadLetter(noticeList);
        if (!ids.isEmpty()) {
            messageService.readLetter(ids);
        }

        return "/site/notice-detail";
    }