1. 为什么需要多生产者动态配置?
在实际业务中,我们经常会遇到这样的场景:一个SpringBoot应用需要向多个RocketMQ集群发送消息,或者同一个集群中不同业务模块需要使用独立的生产者实例。这时候如果只用单一生产者配置,就会面临几个痛点:
- 环境隔离问题:开发、测试、生产环境使用不同的NameServer地址,硬编码在代码中需要频繁修改
- 资源竞争:所有业务共用一个生产者实例,某个业务流量激增可能影响其他业务
- 配置僵化:每次新增生产者都需要改代码重启服务
我去年负责的电商平台项目就遇到过这种情况。订单服务和库存服务虽然都用RocketMQ,但订单消息需要保证高优先级处理,而库存更新可以接受一定延迟。如果混用同一个生产者,在双十一大促时订单消息就可能被库存消息阻塞。
2. YML多生产者配置实战
2.1 基础配置模板
在application.yml中配置多个生产者实例(以RocketMQ 4.9.4为例):
rocketmq:
producers:
- producerId: orderProducer # 生产者唯一标识
groupName: order_group # 生产者组名
namesrvAddr: 192.168.1.10:9876
maxMessageSize: 8192 # 8MB
sendMsgTimeout: 5000 # 5秒超时
retryTimes: 3 # 失败重试次数
- producerId: inventoryProducer
groupName: inventory_group
namesrvAddr: 192.168.1.20:9876
vipChannelEnabled: false # 禁用VIP通道
关键参数说明:
- producerId:在代码中获取生产者的唯一键
- vipChannelEnabled:生产环境建议关闭,避免网络策略问题
- maxMessageSize:超过这个值会抛异常,默认4MB
2.2 配置类解析
创建配置映射类时,建议用@ConfigurationProperties校验参数:
@Data
@ConfigurationProperties(prefix = "rocketmq")
public class MqProducerConfig {
@NotEmpty
private List<ProducerConfig> producers;
@Data
public static class ProducerConfig {
@NotBlank
private String producerId;
@NotBlank
private String groupName;
@NotBlank
private String namesrvAddr;
@Min(1024)
private int maxMessageSize = 4096;
@Min(1000)
private int sendMsgTimeout = 3000;
@Min(0)
private int retryTimes = 2;
private boolean vipChannelEnabled = false;
}
}
3. 生产者动态路由策略
3.1 启动时初始化
通过ApplicationListener在应用启动时初始化所有生产者:
@Component
@RequiredArgsConstructor
public class MqProducerInitializer implements ApplicationListener<ApplicationReadyEvent> {
private final MqProducerConfig config;
private static final Map<String, DefaultMQProducer> producerMap = new ConcurrentHashMap<>();
@Override
public void onApplicationEvent(ApplicationReadyEvent event) {
config.getProducers().forEach(conf -> {
DefaultMQProducer producer = new DefaultMQProducer(conf.getGroupName());
producer.setNamesrvAddr(conf.getNamesrvAddr());
producer.setMaxMessageSize(conf.getMaxMessageSize());
producer.setSendMsgTimeout(conf.getSendMsgTimeout());
producer.setRetryTimesWhenSendFailed(conf.getRetryTimes());
producer.setVipChannelEnabled(conf.isVipChannelEnabled());
try {
producer.start();
producerMap.put(conf.getProducerId(), producer);
log.info("RocketMQ producer started: {}", conf.getProducerId());
} catch (MQClientException e) {
log.error("Failed to start producer: " + conf.getProducerId(), e);
throw new RuntimeException(e);
}
});
}
public static DefaultMQProducer getProducer(String producerId) {
return producerMap.get(producerId);
}
}
3.2 业务层使用示例
在Service层根据业务类型选择生产者:
@Service
@RequiredArgsConstructor
public class OrderService {
private final OrderDao orderDao;
public void createOrder(OrderDTO order) {
// 1. 保存订单
Order entity = orderDao.save(convertToEntity(order));
// 2. 发送MQ消息
Message message = new Message(
"ORDER_TOPIC",
"CREATE",
entity.getOrderNo(),
JSON.toJSONBytes(entity)
);
try {
SendResult result = MqProducerInitializer.getProducer("orderProducer")
.send(message);
log.info("订单消息发送成功: {}", result);
} catch (Exception e) {
log.error("订单消息发送失败", e);
// 这里应该触发补偿机制
}
}
}
4. Topic路由与消费者适配
4.1 消费者动态配置
application.yml配置示例:
rocketmq:
consumers:
- groupName: order_consumer
namesrvAddr: 192.168.1.10:9876
topics:
- topicName: ORDER_TOPIC
tagName: "CREATE || PAY"
consumeThreadMin: 8 # 根据CPU核数调整
consumeThreadMax: 16
- groupName: inventory_consumer
namesrvAddr: 192.168.1.20:9876
topics:
- topicName: INVENTORY_TOPIC
tagName: "*"
4.2 消息处理器映射
通过Spring的Bean命名规则实现自动映射:
@Component("mqHandler.ORDER_TOPIC")
@Slf4j
public class OrderMessageHandler implements MessageHandler {
@Override
public void handle(MessageExt message) {
Order order = JSON.parseObject(message.getBody(), Order.class);
log.info("处理订单消息: {}", order.getOrderNo());
// 实际业务逻辑...
}
}
@Component("mqHandler.INVENTORY_TOPIC")
public class InventoryMessageHandler implements MessageHandler {
// 实现略...
}
4.3 消费者启动逻辑
@Override
public void onApplicationEvent(ApplicationReadyEvent event) {
config.getConsumers().forEach(conf -> {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(conf.getGroupName());
// 基础配置...
consumer.registerMessageListener((MessageListenerConcurrently) (msgs, context) -> {
MessageExt msg = msgs.get(0);
String handlerName = "mqHandler." + msg.getTopic();
try {
MessageHandler handler = applicationContext.getBean(handlerName, MessageHandler.class);
handler.handle(msg);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
} catch (Exception e) {
log.error("消息处理失败", e);
return ConsumeConcurrentlyStatus.RECONSUME_LATER;
}
});
// 订阅Topic...
});
}
5. 生产环境注意事项
-
连接池配置:
- 生产者建议设置
clientCallbackExecutorThreads(默认CPU核数) - 消费者线程数计算公式:
核数 * (1 + (io耗时/cpu耗时))
- 生产者建议设置
-
消息轨迹:
producer.setTraceDispatcher(new AsyncTraceDispatcher( conf.getGroupName(), TraceDispatcher.Type.PRODUCE, conf.getNamesrvAddr() )); -
监控指标:
- 通过
DefaultMQProducer.getDefaultMQProducerImpl().getmQClientFactory()获取内部指标 - 关键监控项:发送耗时、失败次数、队列深度
- 通过
-
优雅停机:
@PreDestroy public void destroy() { producerMap.values().forEach(producer -> { producer.shutdown(); log.info("Producer {} shutdown", producer.getProducerGroup()); }); }
6. 性能优化实战案例
在某物流系统中,我们通过以下优化将消息吞吐量提升了3倍:
-
批量发送:
List<Message> messages = orders.stream() .map(order -> new Message(/*...*/)) .collect(Collectors.toList()); producer.send(messages); -
压缩大消息:
message.setBody(CompressUtils.gzip(data)); message.putUserProperty("compress", "gzip"); -
消费者并行度:
consumeThreadMin: 16 consumeThreadMax: 32 pullBatchSize: 32 # 每次拉取数量
踩过的坑:曾经因为没设置maxMessageSize导致大体积物流轨迹消息发送失败,后来统一在接入层做了消息分片。
1827



被折叠的 条评论
为什么被折叠?



