RocketMQ源码解析-PullConsumer取消息(1)

最新推荐文章于 2025-03-09 12:15:40 发布

原创最新推荐文章于 2025-03-09 12:15:40 发布 · 866 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#java #源码 #rocketMQ #中间件

RocketMQ 专栏收录该内容

12 篇文章

订阅专栏

本文详细介绍了RocketMQ中PullConsumer如何通过主动拉取的方式从Broker获取消息的过程。包括了参数验证、订阅主题检查、构造请求头及通过Netty客户端发送请求到Broker等关键步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

PullConsumer取消息需要自己手动调用Consumer的pull方法主动拉取消息。需要的参数有具体的消息队列（调用消费者的fetchSubscibeMessageQueue()可以得到相应topic的所欲消息队列），需要过滤用的tag（可以为空），以及消费队列所在的进度，以及这次取消息的最大量。

调用了DefaultMQPullConsumer的Pull()方法之后会直接调用DefaultMQPullConsumerImpl的pull()方法。最后会调用DefaultMQPullConsumerImpl的pullSyncImpl()来主动拉消息（同步方式）。

private PullResult pullSyncImpl(MessageQueue mq, String subExpression, long offset, int maxNums,
                                boolean block, long timeout) throws MQClientException, RemotingException, MQBrokerException,
        InterruptedException {
    this.makeSureStateOK();

    if (null == mq) {
        throw new MQClientException("mq is null", null);

    }

    if (offset < 0) {
        throw new MQClientException("offset < 0", null);
    }

    if (maxNums <= 0) {
        throw new MQClientException("maxNums <= 0", null);
    }

    this.subscriptionAutomatically(mq.getTopic());

    int sysFlag = PullSysFlag.buildSysFlag(false, block, true, false);

    SubscriptionData subscriptionData;
    try {
        subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPullConsumer.getConsumerGroup(),//
                mq.getTopic(), subExpression);
    } catch (Exception e) {
        throw new MQClientException("parse subscription error", e);
    }

    long timeoutMillis =
            block ? this.defaultMQPullConsumer.getConsumerTimeoutMillisWhenSuspend() : timeout;

    PullResult pullResult = this.pullAPIWrapper.pullKernelImpl(//
            mq, // 1
            subscriptionData.getSubString(), // 2
            0L, // 3
            offset, // 4
            maxNums, // 5
            sysFlag, // 6
            0, // 7
            this.defaultMQPullConsumer.getBrokerSuspendMaxTimeMillis(), // 8
            timeoutMillis, // 9
            CommunicationMode.SYNC, // 10
            null// 11
    );

    return this.pullAPIWrapper.processPullResult(mq, pullResult, subscriptionData);
}

这里可以看到，除了tag之外其他所有参数都是必不可少的，在检查完所有的参数之后，会调用subscriptionAutomatically()方法来检查传入的消息队列的topic是否有相应的订阅topic数据存在。如果不存在，则会尝试重新建立新的topic订阅数据。

private void subscriptionAutomatically(final String topic) {
    if (!this.rebalanceImpl.getSubscriptionInner().containsKey(topic)) {
        try {
            SubscriptionData subscriptionData =
                    FilterAPI.buildSubscriptionData(this.defaultMQPullConsumer.getConsumerGroup(),//
                            topic, SubscriptionData.SUB_ALL);
            this.rebalanceImpl.subscriptionInner.putIfAbsent(topic, subscriptionData);
        } catch (Exception e) {
        }
    }
}

接下来构造这次拉消息操作的标志量sysFlag，并且根据消费者组名，topic，subExpression来构造这次的SubscriptionData。在确认了本次拉取消息的timeout之后，将会调用pullAPIWrapper的pullKerenImpl()来拉取消息。

pullAPIWrapper的pullKerenImpl()来拉取消息。
public PullResult pullKernelImpl(//
        final MessageQueue mq,// 1
        final String subExpression,// 2
        final long subVersion,// 3
        final long offset,// 4
        final int maxNums,// 5
        final int sysFlag,// 6
        final long commitOffset,// 7
        final long brokerSuspendMaxTimeMillis,// 8
        final long timeoutMillis,// 9
        final CommunicationMode communicationMode,// 10
        final PullCallback pullCallback// 11
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
    FindBrokerResult findBrokerResult =
            this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
                this.recalculatePullFromWhichNode(mq), false);
    if (null == findBrokerResult) {
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
        findBrokerResult =
                this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
                    this.recalculatePullFromWhichNode(mq), false);
    }

    if (findBrokerResult != null) {
        int sysFlagInner = sysFlag;

        if (findBrokerResult.isSlave()) {
            sysFlagInner = PullSysFlag.clearCommitOffsetFlag(sysFlagInner);
        }

        PullMessageRequestHeader requestHeader = new PullMessageRequestHeader();
        requestHeader.setConsumerGroup(this.consumerGroup);
        requestHeader.setTopic(mq.getTopic());
        requestHeader.setQueueId(mq.getQueueId());
        requestHeader.setQueueOffset(offset);
        requestHeader.setMaxMsgNums(maxNums);
        requestHeader.setSysFlag(sysFlagInner);
        requestHeader.setCommitOffset(commitOffset);
        requestHeader.setSuspendTimeoutMillis(brokerSuspendMaxTimeMillis);
        requestHeader.setSubscription(subExpression);
        requestHeader.setSubVersion(subVersion);

        String brokerAddr = findBrokerResult.getBrokerAddr();
        if (PullSysFlag.hasClassFilterFlag(sysFlagInner)) {
            brokerAddr = computPullFromWhichFilterServer(mq.getTopic(), brokerAddr);
        }

        PullResult pullResult = this.mQClientFactory.getMQClientAPIImpl().pullMessage(//
            brokerAddr,//
            requestHeader,//
            timeoutMillis,//
            communicationMode,//
            pullCallback);

        return pullResult;
    }

    throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}

在尝试向broker拉取消息的第一步，首选in当然是要获得向哪个broker发送消息的地址，通过消息队列所对应的brokerName，以及brokerId通过客户端的findBrokerAddressInSubscribe()来获取相应的broker的地址。Broker地址只是简单的从客户端所保存的broker地址表中获取相应的broker地址。如果没有找到，则会尝试从地址服务器更新相应topic的地址数据，再尝试找一次。

在根据之前的数据封装完这次请求的requestHeader之后，将会调用MqClientAPIImpl的 pullMessage()方法将这个请求通过netty客户端发送给broker。

public PullResult pullMessage(//
        final String addr,//
        final PullMessageRequestHeader requestHeader,//
        final long timeoutMillis,//
        final CommunicationMode communicationMode,//
        final PullCallback pullCallback//
) throws RemotingException, MQBrokerException, InterruptedException {
    RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.PULL_MESSAGE, requestHeader);

    switch (communicationMode) {
    case ONEWAY:
        assert false;
        return null;
    case ASYNC:
        this.pullMessageAsync(addr, request, timeoutMillis, pullCallback);
        return null;
    case SYNC:
        return this.pullMessageSync(addr, request, timeoutMillis);
    default:
        assert false;
        break;
    }

    return null;
}

这里由于是同步的方式，这里会选择调用相应的方式。

private PullResult pullMessageSync(//
        final String addr,// 1
        final RemotingCommand request,// 2
        final long timeoutMillis// 3
) throws RemotingException, InterruptedException, MQBrokerException {
    RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
    assert response != null;
    return this.processPullResponse(response);
}

这里在同步方式下，会等待netty发送请求后，同步等待结果返回，并通过processPullResponse()对返回的结果进行处理。

private PullResult processPullResponse(final RemotingCommand response) throws MQBrokerException, RemotingCommandException {
    PullStatus pullStatus = PullStatus.NO_NEW_MSG;
    switch (response.getCode()) {
    case ResponseCode.SUCCESS:
        pullStatus = PullStatus.FOUND;
        break;
    case ResponseCode.PULL_NOT_FOUND:
        pullStatus = PullStatus.NO_NEW_MSG;
        break;
    case ResponseCode.PULL_RETRY_IMMEDIATELY:
        pullStatus = PullStatus.NO_MATCHED_MSG;
        break;
    case ResponseCode.PULL_OFFSET_MOVED:
        pullStatus = PullStatus.OFFSET_ILLEGAL;
        break;

    default:
        throw new MQBrokerException(response.getCode(), response.getRemark());
    }

    PullMessageResponseHeader responseHeader =
            (PullMessageResponseHeader) response.decodeCommandCustomHeader(PullMessageResponseHeader.class);

    return new PullResultExt(pullStatus, responseHeader.getNextBeginOffset(), responseHeader.getMinOffset(),
        responseHeader.getMaxOffset(), null, responseHeader.getSuggestWhichBrokerId(), response.getBody());
}

在这里会根据broker返回结果头所包含的信息，以及结果返回码生成结果的抽象类PullResultExt，PullResultExt继承自PullRequest。

private final PullStatus pullStatus;
private final long nextBeginOffset;
private final long minOffset;
private final long maxOffset;
private List<MessageExt> msgFoundList;

以上是PullRequest的成员组成，可以见得，在从broker中返回的结果中，下一次消费的起始偏移量，最大最小偏移量都存在这里，用以对确定下一次从Broker拉取消息时的开始位置。

private final long suggestWhichBrokerId;
private byte[] messageBinary;

而具体的消息信息，则保存在PullResultExt中。

在生成了PullResultExt之后，最后回到了DefaultMQPullConsumerImpl的pullSyncImpl()上，在获取了DefaultMQClient将从Broker那里得到的返回的结果之后包装生成的PullRequestExt之后，调用processPullRequest()方法对拉回来的结果进行处理。

public PullResult processPullResult(final MessageQueue mq, final PullResult pullResult,
        final SubscriptionData subscriptionData) {
    PullResultExt pullResultExt = (PullResultExt) pullResult;

    this.updatePullFromWhichNode(mq, pullResultExt.getSuggestWhichBrokerId());
    if (PullStatus.FOUND == pullResult.getPullStatus()) {
        ByteBuffer byteBuffer = ByteBuffer.wrap(pullResultExt.getMessageBinary());
        List<MessageExt> msgList = MessageDecoder.decodes(byteBuffer);

        List<MessageExt> msgListFilterAgain = msgList;
        if (!subscriptionData.getTagsSet().isEmpty() && !subscriptionData.isClassFilterMode()) {
            msgListFilterAgain = new ArrayList<MessageExt>(msgList.size());
            for (MessageExt msg : msgList) {
                if (msg.getTags() != null) {
                    if (subscriptionData.getTagsSet().contains(msg.getTags())) {
                        msgListFilterAgain.add(msg);
                    }
                }
            }
        }

        if (this.hasHook()) {
            FilterMessageContext filterMessageContext = new FilterMessageContext();
            filterMessageContext.setUnitMode(unitMode);
            filterMessageContext.setMsgList(msgListFilterAgain);
            this.executeHook(filterMessageContext);
        }

        for (MessageExt msg : msgListFilterAgain) {
            MessageAccessor.putProperty(msg, MessageConst.PROPERTY_MIN_OFFSET,
                Long.toString(pullResult.getMinOffset()));
            MessageAccessor.putProperty(msg, MessageConst.PROPERTY_MAX_OFFSET,
                Long.toString(pullResult.getMaxOffset()));
        }

        pullResultExt.setMsgFoundList(msgListFilterAgain);
    }

    pullResultExt.setMessageBinary(null);

    return pullResult;
}

在这个方法中，对消息的具体字节流进行了反序列化，并且根据订阅的tag在这里完成对于消息的过滤。