RocketMQ：消息拉取和消息队列负载分布机制

一、前言

        Consumer 消息消费流程比较复杂，包含模块有：消息查找、负载均衡、消息过滤、消息处理、回发确认、消息进度维护等。限于篇幅，本篇主要介绍 Consumer 启动流程及消息拉取实现机制。

        消息消费以组的模式开展，一个消费组内可以包含多个消费者，每个消费组可以订阅多个 Topic，消费组之间有集群模式和广播模式两种消费模式。

• 集群模式：Topic 下的同一条消息只允许被其中一个消费者消费；
• 广播模式：Topic 下的同一条消息将被集群内的所有消费者消费一次；

        消息服务器与消费者之间的消息传送也有两种方式：

• 拉模式：消费端主动发起拉取消息请求；
• 推模式：消息到达消息服务器后，推送给消息消费者；

        注意，RocketMQ 消息推模式的实现是基于拉模式，在拉模式上包装一层，一个拉取任务完成后，开始下一个拉取任务。

        

二、问题

        查看 官方样例 都知道 Consumer 启动时都要指定 Namesrv 地址、订阅的Topic 和 消息监听器。

public class Consumer {
    public static void main(String[] args) throws InterruptedException, MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name");
         
        // Specify name server addresses.
        consumer.setNamesrvAddr("localhost:9876");
        
        // Subscribe one more more topics to consume.
        consumer.subscribe("TopicTest", "*");
        // Register callback to execute on arrival of messages fetched from brokers.
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
                ConsumeConcurrentlyContext context) {
                System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        //Launch the consumer instance.
        consumer.start();
        System.out.printf("Consumer Started.%n");
    }
}

        那么，Consumer 如何启动，怎么从 Broker 拉取消息消费呢？

        那么，集群模式下，多个消费者如何对消息队列进行负载呢？消息队列负载机制遵循一个通用的思想：一个消息队列同一时间只允许被一个消费者消费，一个消费者可以消费多个消息队列。

        

三、消费者启动流程

        消息消费和获取方式都有两种，这里针对日常开发上常用的方式：集群消费、推模式进行分析，一起来进入 DefaultMQPushConsumerImpl#start 方法源码了解一番吧。

org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl代码片段

public synchronized void start() throws MQClientException {
    switch (this.serviceState) {
        case CREATE_JUST:
            // 忽略一些非必要代码...
            
            //1. 构建topic的订阅信息
            this.copySubscription();
            
            //2. 创建MQClientInstance实例、初始化
            this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);
            
            //3 初始化负载均衡实现类
            this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
            this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
          	this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
            this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
            
            this.pullAPIWrapper = new PullAPIWrapper(mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
            this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);
            
            //4. 初始化消息进度
            if (this.defaultMQPushConsumer.getOffsetStore() != null) {
                this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
            } else {
                switch (this.defaultMQPushConsumer.getMessageModel()) {
                    case BROADCASTING:
                        this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                        break;
                    case CLUSTERING:
                        this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                        break;
                    default:
                        break;
                }
                this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
            }
            this.offsetStore.load();
            
            //5. 检查是否顺序消费，创建、启动消费端线程服务
            if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
                this.consumeOrderly = true;
                this.consumeMessageService =
                    new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
            } else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
                this.consumeOrderly = false;
                this.consumeMessageService =
                    new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
            }
            this.consumeMessageService.start();

			//6. 注册Consumer
            boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
            
            //7. 启动MQClientInstance
            mQClientFactory.start();
            this.serviceState = ServiceState.RUNNING;
            break;
    }
   	//订阅变更时，从NameServer更新topic路由信息
    this.updateTopicSubscribeInfoWhenSubscriptionChanged(); 
    this.mQClientFactory.checkClientInBroker();
    this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
    this.mQClientFactory.rebalanceImmediately();
}

        1、构建 Topic 的订阅消息，把 Comsumer 启动前设置的 topic 主题名和过滤条件进行封装，并分配给 RebalanceImpl，为后续负载均衡分配提供遍历条件。

        2、创建 MQClient 实例，并初始化一些服务实例，为后续消息负载分配、消息拉取提供支持。

        3、Consuemr 负载均衡实例，初始化分配策略，由负载均衡服务定期调用，重新分配消费队列。

        4、Consuemr 消费进度存储，广播模式，会在本地客户端维护一份消费进度；集群模式，会从 Broker 获取消费的进度，本地消费成功后，又会向 Broker更新进度。

        5、Comsuemr 消息消费服务，与消息拉取服务解耦，内部独立线程池，对拉取的消息异步消费：成功，向 Broker 更新消费的进度；失败，把消息丢回 Broker，进行延迟消费。

        6、Consuemr 登记列表。

        7、启动 MQClient 实例，启动消息拉取服务、消息负载分配服务，启动Topic路由更新、Comsumer 消费进度持久化等定期实现等。

        下面进行具体的源码分析：

org.apache.rocketmq.client.impl.factory.MQClientInstance代码片段

public void start() throws MQClientException {
    synchronized (this) {
        switch (this.serviceState) {
            case CREATE_JUST:
                // 忽略一些非必要代码...
                
                //启动连接
                this.mQClientAPIImpl.start(); 
                
                //启动定时任务，监听namesrc地址的变更
                this.startScheduledTask(); 
                
                //启动拉取消息服务
                this.pullMessageService.start(); 
                
                //启动负载均衡服务，发送拉取消息请求
                this.rebalanceService.start();
                
                // Start push service
                this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
                this.serviceState = ServiceState.RUNNING;
                break;
        }
    }
}

        在集群模式下，同一个消费组内有多个消息消费者，同一个 Topic 存在多个消费队列，那么消费者如何进行消息队列负载均衡？

        消息队列负载均衡，通常的做法是一个消费队列在同一时间只允许被一个消息消费者消费，一个消息消费者可以同时消费多个消息队列，那么 RocketMQ 是如何实现的呢？

        带着上述的问题，我们开始探讨下 RocketMQ 消息消费机制。从 MQClientInstance 的启动流程中可以看出，RocketMQ 使用一个单独的线程服务 PullMessageService 来负责消息的拉取。

org.apache.rocketmq.client.impl.consumer.PullMessageService#run代码片段

public void run() {
    log.info(this.getServiceName() + " service started");
    while (!this.isStopped()) {
        try {
            PullRequest pullRequest = this.pullRequestQueue.take(); //队列无消息请求，阻塞等待
            this.pullMessage(pullRequest); //发送拉取消息请求
        } catch (InterruptedException ignored) {
        } catch (Exception e) {
            log.error("Pull Message Service Run Method exception", e);
        }
    }
    log.info(this.getServiceName() + " service end");
}

        PullMessageService 消息拉取服务线程，从 pullRequestQueue 队列中获取一个 PullRequest 消息拉取任务，如果 pullRequestQueue 为空，则线程将阻塞，直到有拉取任务被放入，调用 pullMessage 方法进行消息拉取。

        那么，PullRequest 是什么时候被放入队列的呢？

org.apache.rocketmq.client.impl.consumer.PullMessageService代码片段

public void executePullRequestImmediately(final PullRequest pullRequest) {
    try {
        this.pullRequestQueue.put(pullRequest); //拉取请求存入本地队列，由线程处理
    } catch (InterruptedException e) {
        log.error("executePullRequestImmediately pullRequestQueue.put", e);
    }
}

public void executePullRequestLater(final PullRequest pullRequest, final long timeDelay) {
    if (!isStopped()) {
        this.scheduledExecutorService.schedule(new Runnable() {
            @Override
            public void run() {
                PullMessageService.this.executePullRequestImmediately(pullRequest);
            }
        }, timeDelay, TimeUnit.MILLISECONDS);
    } else {
        log.warn("PullMessageServiceScheduledThread has shutdown");
    }
}

        通过跟踪发现，PullRequest 会在两个地方被创建：

        1、RocketMQ 根据 PullRequest 拉取任务执行完一次消息拉取后，又将 PullRequest 对象放入 pullRequestQueue；

org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl代码片段

public void pullMessage(final PullRequest pullRequest) {
    //忽略部分代码...
    
    PullCallback pullCallback = new PullCallback() {
        @Override
        public void onSuccess(PullResult pullResult) {
            if (pullResult != null) {
                //拉取消费消息结果
                pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                    subscriptionData);
                switch (pullResult.getPullStatus()) {
                    case FOUND:
                        long prevRequestOffset = pullRequest.getNextOffset();
                        pullRequest.setNextOffset(pullResult.getNextBeginOffset());
                        long pullRT = System.currentTimeMillis() - beginTimestamp;
                        DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                            pullRequest.getMessageQueue().getTopic(), pullRT);
                        long firstMsgOffset = Long.MAX_VALUE;
                        //判断如果本地消息拉取，没有找到消息列表，则把 pullRequest 重新放入 pullRequestQueue
                        if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
                            DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                        } else {
                            firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();
                 			DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                                pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());

                            boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
                            DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest( 
                                pullResult.getMsgFoundList(),
                                processQueue,
                                pullRequest.getMessageQueue(),
                                dispatchToConsume);
							//根据拉取的时间间隔，是直接放入 pullRequestQueue，还是延迟之后再放入请求队列
                            if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                                DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                                    DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
                            } else {
                                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                            }
                        }
                        //忽略部分的代码...
                        break;
                    //忽略其他状态的代码...
                }
            }
        }

        @Override
        public void onException(Throwable e) {
            DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
        }
    };
    //忽略部分代码...
    try {
        this.pullAPIWrapper.pullKernelImpl( //发送请求
            pullRequest.getMessageQueue(),
            subExpression,
            subscriptionData.getExpressionType(),
            subscriptionData.getSubVersion(),
            pullRequest.getNextOffset(),
            this.defaultMQPushConsumer.getPullBatchSize(),
            sysFlag,
            commitOffsetValue,
            BROKER_SUSPEND_MAX_TIME_MILLIS,
            CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND,
            CommunicationMode.ASYNC,
            pullCallback //请求返回，回调处理
        );
    } catch (Exception e) {
        log.error("pullKernelImpl exception", e);
        this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
    }
}

        2、由 RebalanceService 服务线程轮询已注册消费者订阅 Topic 的信息，在 RebalanceImpl 中创建；

org.apache.rocketmq.client.impl.consumer.RebalanceService

public void run() {
    log.info(this.getServiceName() + " service started");
    while (!this.isStopped()) {
        this.waitForRunning(waitInterval);
        this.mqClientFactory.doRebalance(); //负载均衡
    }
    log.info(this.getServiceName() + " service end");
}

        RebalanceService 线程默认每隔 20s 执行一次 mqClientFactory.doRebalance()方法

org.apache.rocketmq.client.impl.consumer.RebalanceImpl#doRebalance代码片段

public void doRebalance(final boolean isOrder) {
	//获取订阅信息、遍历，在前面DefaultMQPushConsumerImpl执行复制时，已经分配给
    Map<String, SubscriptionData> subTable = this.getSubscriptionInner(); 
    if (subTable != null) {
        for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
            final String topic = entry.getKey();
            try {
                this.rebalanceByTopic(topic, isOrder); //根据Topic订阅消息，进行负载分配
            } catch (Throwable e) {
                if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                    log.warn("rebalanceByTopic Exception", e);
                }
            }
        }
    }
    this.truncateMessageQueueNotMyTopic(); //删除消息队列
}

private void rebalanceByTopic(final String topic, final boolean isOrder) {
    switch (messageModel) {
        case CLUSTERING: {
        	//获取Topic订阅的消费队列，在Comsumer启动期间通过org.apache.rocketmq.client.impl.factory.MQClientInstance#startScheduledTask 定期任务更新
            Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
			
			//发送请求从 Broker 中获取该消费组内当前所有的消费者客户端ID
            List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
            
            if (mqSet != null && cidAll != null) {
                List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
                mqAll.addAll(mqSet);
				//排序，让同一组内看到的视图保持一致，确保同一消费队列不会被多个消费者分配
                Collections.sort(mqAll);
                Collections.sort(cidAll);

				//分配算法，默认使用AllocateMessageQueueAveragely 平均分配算法，
				//例如：8个消费队列，3个消费者，则c1:q1、q2、q3，c2:q4、q5、q6，c3:q7、q8
                AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
                List<MessageQueue> allocateResult = null;
                try {
                    allocateResult = strategy.allocate(
                        this.consumerGroup,
                        this.mQClientFactory.getClientId(),
                        mqAll,
                        cidAll);
                } catch (Throwable e) {
                    return;
                }
                Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
                if (allocateResult != null) {
                    allocateResultSet.addAll(allocateResult);
                }
				
				//更新消费者负载的消息队列缓存表，分派拉取请求
                boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
                if (changed) {
                    this.messageQueueChanged(topic, mqSet, allocateResultSet);
                }
            }
            break;
        }
    }
}

public void dispatchPullRequest(List<PullRequest> pullRequestList) {
    for (PullRequest pullRequest : pullRequestList) {
        this.defaultMQPushConsumerImpl.executePullRequestImmediately(pullRequest);
        log.info("doRebalance, {}, add a new pull request {}", consumerGroup, pullRequest);
    }
}

        至此，拉取消息请求就被放入队列中，由 PullMessageService 服务进行消费。

---

引用

《RocketMQ 技术内幕》