设计场景实际去体验 rabbitmq 重试机制。场景设计-＞消息流转模型设计-＞java代码实现

场景

设想一下场景：
消息发送的客户端，发送消息到交换器，交换器分发消息到消费者，此时我们的业务中的mq的消费者发生异常，消息没有正常消费。此时客户并不知道消息出现问题，我们也不能要求客户重新发送消息给消费者了，此时如何引入重试机制，使得消费者出现异常后，该条异常的消息能够重试n次，重新回到消费者队列，以供消费者重新使用呢？

场景设计

以下是设计的场景：
客户端生产三种格式的图片：png、jpg，svg
两种不同类型的消费者得到照片：
imageConsumer 专门消费 png 、jpg
VectorConsumer 专门消费svg
如何针对此场景设计重试机制，使得消息在发生异常后，能够重新回到消费者关联的队列，并且重试n次过后，该条消息能自动分发到死信队列，代表消息彻底失效呢？

消息流转模型以及设计

以下是场景的消息流转模型：

三个交换器：
x.guideline.work :工作交换器
x.guideline.wait: 缓存交换器
x.guideline.dead: 死信队列交换器

六个队列：
q.guideline.image.work： 针对 png、jpg的工作队列
q.guideline.image.wait： 针对 png、jpg的缓存队列
q.guideline.image.dead：针对 png、jpg的死信队列

q.guideline.vector.work： 针对 svg 的工作队列
q.guideline.vector.wait： 针对svg的缓存队列
q.guideline.vector.dead： 针对svg 的缓存队列

下面是信息流转过程的文字说明：
循环一：
1、随机生成 png、jpg、svg的消息，png、jpg类型进入q.guideline.image.work队列，svg类型进入q.guideline.vector.work
2、设置两个不同类型的消费者imageConsumer、VectorConsumer。imageConsumer监听q.guideline.image.work，VectorConsumer监听q.guideline.vector.work。
3、消费者消费消息的时候，发生异常，此时消息由于是第一次发生异常，设计进入q.guideline.image.wait或q.guideline.vector.wait队列。
4、两个wait队列的消息由于达到了TTL，消息重新回到对应的work队列

以上是第一次循环，下面是第二次循环：

循环二：
1、消息第二次回到work队列，此时消费者去消费消息，有发生异常，我们设计异常发生的阈值为3，超过三次，消息就会被抛弃，进入死信队列，此时还没有达到阈值3，重新走一遍循环

以上是第二次循环，下面是第三次循环：

循环三：
1、消息第三次回到work队列，此时消费者去消费消息，有发生异常，并且达到阈值3，此时消息会被交给q.guideline.image.dead或q.guideline.vector.dead的死信队列，代表消息被遗弃，处理完毕。

消息流转模型java代码实现

实体类

首先我们需要我们的实体类，去定义我们传递的图片类型：

/**
 * @author: 代码丰
 * @Description: 
 * 图片实体类包含： 
 * 1、名字
 * 2、类型（只有三种类型png、jpg、svg）
 * 3、来源
 * 4、大小
 */
@Data
@AllArgsConstructor
@NoArgsConstructor
public class Picture {

    String name;
    String type;
    String source;
    long size;

    @Override
    public String toString() {
        return "Picture{" +
                "name='" + name + '\'' +
                ", type='" + type + '\'' +
                ", source='" + source + '\'' +
                ", size=" + size +
                '}';
    }
}

三个辅助类

其次我们需要辅助类去解析mq传递消息过程中使用的message class的结构（RabbitmqHeader、RabbitmqHeaderXDeath），以及出现异常后，到底是发送给wait缓存队列，还是发送给dead队列的辅助类（DlxProcessingErrorHandler）：

// Message的header部分
public class RabbitmqHeader {

	private static final String KEYWORD_QUEUE_WAIT = "wait";
	private List<RabbitmqHeaderXDeath> xDeaths = new ArrayList<>(2);
	private String xFirstDeathExchange = StringUtils.EMPTY;
	private String xFirstDeathQueue = StringUtils.EMPTY;
	private String xFirstDeathReason = StringUtils.EMPTY;

//解析消息传递过程中，mq使用的Message类
	@SuppressWarnings("unchecked")
	public RabbitmqHeader(Map<String, Object> headers) {
		if (headers != null) {

			Optional.ofNullable(headers.get("x-first-death-exchange")).ifPresent(s -> this.setxFirstDeathExchange(s.toString()));
			Optional.ofNullable(headers.get("x-first-death-queue")).ifPresent(s -> this.setxFirstDeathExchange(s.toString()));
			Optional.ofNullable(headers.get("x-first-death-reason")).ifPresent(s -> this.setxFirstDeathReason(s.toString()));

			List<Map<String, Object>> xDeathHeaders = (List<Map<String, Object>>) headers.get("x-death");

			if (xDeathHeaders != null) {
				for (Map<String, Object> x : xDeathHeaders) {
					RabbitmqHeaderXDeath hdrDeath = new RabbitmqHeaderXDeath();
					Optional.ofNullable(x.get("reason")).ifPresent(s -> hdrDeath.setReason(s.toString()));
					Optional.ofNullable(x.get("count")).ifPresent(s -> hdrDeath.setCount(Integer.parseInt(s.toString())));
					Optional.ofNullable(x.get("exchange")).ifPresent(s -> hdrDeath.setExchange(s.toString()));
					 Optional.ofNullable(x.get("queue")).ifPresent(s -> hdrDeath.setQueue(s.toString()));
					 Optional.ofNullable(x.get("routing-keys")).ifPresent(r -> {
						hdrDeath.setRoutingKeys((List<String>) r);
					});
					 Optional.ofNullable(x.get("time")).ifPresent(d -> hdrDeath.setTime((Date) d));
					xDeaths.add(hdrDeath);
				}
			}
		}
	}

//获取重试的次数
	public int getFailedRetryCount() {
		// get from queue "wait"
		for (RabbitmqHeaderXDeath xDeath : xDeaths) { 
			if (xDeath.getExchange().toLowerCase().endsWith(KEYWORD_QUEUE_WAIT)
					&& xDeath.getQueue().toLowerCase().endsWith(KEYWORD_QUEUE_WAIT)) {
				return xDeath.getCount();
			}
		}

		return 0;
	}

	public List<RabbitmqHeaderXDeath> getxDeaths() {
		return xDeaths;
	}

	public String getxFirstDeathExchange() {
		return xFirstDeathExchange;
	}

	public String getxFirstDeathQueue() {
		return xFirstDeathQueue;
	}

	public String getxFirstDeathReason() {
		return xFirstDeathReason;
	}

	public void setxDeaths(List<RabbitmqHeaderXDeath> xDeaths) {
		this.xDeaths = xDeaths;
	}

	public void setxFirstDeathExchange(String xFirstDeathExchange) {
		this.xFirstDeathExchange = xFirstDeathExchange;
	}

	public void setxFirstDeathQueue(String xFirstDeathQueue) {
		this.xFirstDeathQueue = xFirstDeathQueue;
	}

	public void setxFirstDeathReason(String xFirstDeathReason) {
		this.xFirstDeathReason = xFirstDeathReason;
	}

}

// Message的header部分，内部的一个列表，此时我把它打开，定义为java类，方便解析
public class RabbitmqHeaderXDeath {

	private int count;
	private String exchange;
	private String queue;
	private String reason;
	private List<String> routingKeys;
	private Date time;

	@Override
	public boolean equals(Object obj) {
		if (this == obj) {
			return true;
		}
		if (obj == null) {
			return false;
		}
		if (getClass() != obj.getClass()) {
			return false;
		}
		RabbitmqHeaderXDeath other = (RabbitmqHeaderXDeath) obj;
		if (count != other.count) {
			return false;
		}
		if (exchange == null) {
			if (other.exchange != null) {
				return false;
			}
		} else if (!exchange.equals(other.exchange)) {
			return false;
		}
		if (queue == null) {
			if (other.queue != null) {
				return false;
			}
		} else if (!queue.equals(other.queue)) {
			return false;
		}
		if (reason == null) {
			if (other.reason != null) {
				return false;
			}
		} else if (!reason.equals(other.reason)) {
			return false;
		}
		if (routingKeys == null) {
			if (other.routingKeys != null) {
				return false;
			}
		} else if (!routingKeys.equals(other.routingKeys)) {
			return false;
		}
		if (time == null) {
			if (other.time != null) {
				return false;
			}
		} else if (!time.equals(other.time)) {
			return false;
		}
		return true;
	}

	public int getCount() {
		return count;
	}

	public String getExchange() {
		return exchange;
	}

	public String getQueue() {
		return queue;
	}

	public String getReason() {
		return reason;
	}

	public List<String> getRoutingKeys() {
		return routingKeys;
	}

	public Date getTime() {
		return time;
	}

	@Override
	public int hashCode() {
		final int prime = 19;
		int result = 1;
		result = prime * result + count;
		result = prime * result + ((exchange == null) ? 0 : exchange.hashCode());
		result = prime * result + ((queue == null) ? 0 : queue.hashCode());
		result = prime * result + ((reason == null) ? 0 : reason.hashCode());
		result = prime * result + ((routingKeys == null) ? 0 : routingKeys.hashCode());
		result = prime * result + ((time == null) ? 0 : time.hashCode());
		return result;
	}

	public void setCount(int count) {
		this.count = count;
	}

	public void setExchange(String exchange) {
		this.exchange = exchange;
	}

	public void setQueue(String queue) {
		this.queue = queue;
	}

	public void setReason(String reason) {
		this.reason = reason;
	}

	public void setRoutingKeys(List<String> routingKeys) {
		this.routingKeys = routingKeys;
	}

	public void setTime(Date time) {
		this.time = time;
	}

}

package com.example.rabbitmqlearningproject.retryMechanism.rabbitmq;

import java.io.IOException;
import java.time.LocalDateTime;

import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.amqp.core.Message;
import org.springframework.lang.NonNull;


import com.rabbitmq.client.Channel;

//核心处理异常的类
public class DlxProcessingErrorHandler {

	private static final Logger LOG = LoggerFactory.getLogger(DlxProcessingErrorHandler.class);

	/**
	 * Dead exchange name
	 */
	@NonNull
	private String deadExchangeName;

	private int maxRetryCount = 3;

	/**
	 * Constructor. Will retry for n times (default is 3) and on the next retry will
	 * consider message as dead, put it on dead exchange with given
	 * <code>dlxExchangeName</code> and <code>routingKey</code>
	 * 
	 * @param deadExchangeName dead exchange name. Not a dlx for work queue, but
	 *                         exchange name for really dead message (wont processed
	 *                         antmore).
	 * @throws IllegalArgumentException if <code>dlxExchangeName</code> or
	 *                                  <code>dlxRoutingKey</code> is null or empty.
	 */
	public DlxProcessingErrorHandler(String deadExchangeName) throws IllegalArgumentException {
		super();

		if (StringUtils.isAnyEmpty(deadExchangeName)) {
			throw new IllegalArgumentException("Must define dlx exchange name");
		}

		this.deadExchangeName = deadExchangeName;
	}

	/**
	 * Constructor. Will retry for <code>maxRetryCount</code> times and on the next
	 * retry will consider message as dead, put it on dead exchange with given
	 * <code>dlxExchangeName</code> and <code>routingKey</code>
	 * 
	 * @param deadExchangeName dead exchange name. Not a dlx for work queue, but
	 *                         exchange name for really dead message (wont processed
	 *                         antmore).
	 * @param maxRetryCount    number of retry before message considered as dead (0
	 *                         >= <code> maxRetryCount</code> >= 1000). If set less
	 *                         than 0, will always retry
	 * @throws IllegalArgumentException if <code>dlxExchangeName</code> or
	 *                                  <code>dlxRoutingKey</code> is null or empty.
	 */

	public DlxProcessingErrorHandler(String deadExchangeName, int maxRetryCount) {
		this(deadExchangeName);
		setMaxRetryCount(maxRetryCount);
	}

	public String getDeadExchangeName() {
		return deadExchangeName;
	}

	public int getMaxRetryCount() {
		return maxRetryCount;
	}

	/**
	 * Handle AMQP message consume error. This default implementation will put
	 * message to dead letter exchange for <code>maxRetryCount</code> times, thus
	 * two variables are required when creating this object:
	 * <code>dlxExchangeName</code> and <code>dlxRoutingKey</code>. <br/>
	 * <code>maxRetryCount</code> is 3 by default, but you can set it using
	 * <code>setMaxRetryCount(int)</code>
	 * 
	 * @param message     AMQP message that caused error
	 * @param channel     channel for AMQP message
	 * @param deliveryTag message delivery tag
	 * @return <code>true</code> if error handler works sucessfully,
	 *         <code>false</code> otherwise
	 */
	 //这里就是核心的异常重试的实现，大于阈值-》channel.basicPublish到dead队列，小于阈值-〉消息被拒绝，消息进入wait队列
	 
	public boolean handleErrorProcessingMessage(Message message, Channel channel, long deliveryTag) {
		RabbitmqHeader rabbitMqHeader = new RabbitmqHeader(message.getMessageProperties().getHeaders());

		try {
			if (rabbitMqHeader.getFailedRetryCount() >= maxRetryCount) {
				// publish to dead and ack
				LOG.warn("[DEAD] Error at " + LocalDateTime.now() + " on retry " + rabbitMqHeader.getFailedRetryCount()
						+ " for message " + new String(message.getBody()));

				channel.basicPublish(getDeadExchangeName(), message.getMessageProperties().getReceivedRoutingKey(),
						null, message.getBody());
				channel.basicAck(deliveryTag, false);
			} else {
				LOG.warn("[REQUEUE] Error at " + LocalDateTime.now() + " on retry "
						+ rabbitMqHeader.getFailedRetryCount() + " for message " + new String(message.getBody()));

				channel.basicReject(deliveryTag, false);
			}
			return true;
		} catch (IOException e) {
			LOG.warn("[HANDLER-FAILED] Error at " + LocalDateTime.now() + " on retry "
					+ rabbitMqHeader.getFailedRetryCount() + " for message " + new String(message.getBody()));
		}

		return false;
	}

	public void setMaxRetryCount(int maxRetryCount) throws IllegalArgumentException {
		if (maxRetryCount > 1000) {
			throw new IllegalArgumentException("max retry must between 0-1000");
		}

		this.maxRetryCount = maxRetryCount;
	}

}

生产者

@Service
public class RetryPictureProducer {

    @Autowired
    private RabbitTemplate rabbitTemplate;
    @Autowired
    private ObjectMapper objectMapper;
    public void sendMessageWithException(Picture picture) throws JsonProcessingException {

		//将对象以json格式作为mq传递消息的Message中的payload
        String json = objectMapper.writeValueAsString(picture);
   		rabbitTemplate.convertAndSend("x.guideline.work",picture.getType(),json);

    }
}

两个消费者（一摸一样，只是进入的队列不一样）

//针对jpg png的消费者
@Service
@Slf4j
public class RetryImageConsumer {

	private static final String DEAD_EXCHANGE_NAME = "x.guideline.dead";
	private DlxProcessingErrorHandler dlxProcessingErrorHandler;

	@Autowired
	private ObjectMapper objectMapper;

	public RetryImageConsumer() {
		this.dlxProcessingErrorHandler = new DlxProcessingErrorHandler(DEAD_EXCHANGE_NAME);
	}

	@RabbitListener(queues = "q.guideline.image.work")
	public void listen(Message message, Channel channel, @Header(AmqpHeaders.DELIVERY_TAG) long deliveryTag)
			throws InterruptedException, JsonParseException, JsonMappingException, IOException {
		try {
			Picture p = objectMapper.readValue(message.getBody(), Picture.class);
			// process the image
			if (p.getSize() > 9000) {
				// throw exception, we will use DLX handler for retry mechanism
				throw new IOException("Size too large");
			} 
		} catch (IOException e) {
			log.warn("Error processing message : " + new String(message.getBody()) + " : " + e.getMessage());
			dlxProcessingErrorHandler.handleErrorProcessingMessage(message, channel, deliveryTag);
		}
	}

}

//针对 svg 的消费者
@Service
@Slf4j
public class RetryVectorConsumer {

	private static final String DEAD_EXCHANGE_NAME = "x.guideline.dead";
	private DlxProcessingErrorHandler dlxProcessingErrorHandler;

	@Autowired
	private ObjectMapper objectMapper;

	public RetryVectorConsumer() {
		this.dlxProcessingErrorHandler = new DlxProcessingErrorHandler(DEAD_EXCHANGE_NAME);
	}

	@RabbitListener(queues = "q.guideline.vector.work")
	public void listen(Message message, Channel channel, @Header(AmqpHeaders.DELIVERY_TAG) long deliveryTag)
			throws InterruptedException, JsonParseException, JsonMappingException, IOException {
		try {
			Picture p = objectMapper.readValue(message.getBody(), Picture.class);
			// process the image
			if (p.getSize() > 9000) {
				// throw exception, we will use DLX handler for retry mechanism
				throw new IOException("Size too large");
			}
		} catch (IOException e) {
			log.warn("Error processing message : " + new String(message.getBody()) + " : " + e.getMessage());
			dlxProcessingErrorHandler.handleErrorProcessingMessage(message, channel, deliveryTag);
		}
	}

}

测试类

 @SpringBootApplication
@EnableScheduling
@Slf4j
public class RabbitMqLearningProjectApplication implements CommandLineRunner {


    private List<String> SOURCES = Arrays.asList("mobile", "web");
    private List<String> TYPE = Arrays.asList("jpg", "png", "svg");
        @Autowired
        RetryPictureProducer retryPictureProducer;

    public static void main(String[] args) {
        SpringApplication.run(RabbitMqLearningProjectApplication.class, args);
    }

    @Override
    public void run(String... args) throws Exception {

// 生成10 个 size都大于 9000的图片，使得每一个消息都会进入异常处理的逻辑
        for (int i = 0; i < 10; i++) {
            Picture picture = new Picture();
            picture.setName("retry picture"+i);
            picture.setType(TYPE.get(i % TYPE.size()));
            picture.setSource(SOURCES.get(i % SOURCES.size()));
            picture.setSize(ThreadLocalRandom.current().nextLong(9500,10000));

            retryPictureProducer.sendMessageWithException(picture);
        }



    }
}

最终结果的验证：

消息从work队列第一次来到wait队列，此时传递的Message是以下格式的数据：

( 
(Body:'' 
MessageProperties [headers={}, 
contentType=text/plain, 
contentEncoding=UTF-8, 
contentLength=0, 
receivedDeliveryMode=PERSISTENT, 
priority=0, 
redelivered=false, 
receivedExchange=x.guideline.work, 
receivedRoutingKey=jpg, deliveryTag=1, 
consumerTag=amq.ctag-r6W1JB6nEYrHCWjfS4DZgw, consumerQueue=q.guideline.image.work
    ])

消息经过第一次循环后，重新来到work队列的样子：

(Body:'' 
MessageProperties 
//MessageProperties 之前为空，现在纪录了新的key
//x-first-death-exchange、x-death、x-first-death-reason、x-first-death-queue=
//此时代表消息先经过dlxProcessingErrorHandler.handleErrorProcessingMessage()。
//没有达到阈值，被basickReject后首先rejected了
//然后达到了TTL设置的时间，被expired，头部都记录了消息经过的队列的信息
[headers=
{x-first-death-exchange=x.guideline.work, 
x-death=[
{reason=expired, count=1, exchange=x.guideline.wait, routing-keys=[jpg], time=Sat Nov 04 21: 37: 45 CST 2023, queue=q.guideline.image.wait},
{reason=rejected, count=1, exchange=x.guideline.work, time=Sat Nov 04 21: 37: 15 CST 2023, routing-keys=[jpg], queue=q.guideline.image.work}], 
x-first-death-reason=rejected, 
x-first-death-queue=q.guideline.image.work }, 
// 不重要下面的
contentType=text/plain, contentEncoding=UTF-8, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=x.guideline.work, receivedRoutingKey=jpg, deliveryTag=2, consumerTag=amq.ctag-r6W1JB6nEYrHCWjfS4DZgw, consumerQueue=q.guideline.image.work
        ])

此时头部的Message的结构如下：
可以看到count=1，代表经过了一次循环

消息经过第二次循环后，重新来到work队列的样子：

可以看到count变成2，代表第二次进入work和wait队列

消息经过第三次循环后，重新来到work队列的样子：

可以看到count变成3，代表第三次进入work和wait队列

最终

有任何疑问，可以问我，我可以发全套的代码给你验证，包括postman新建队列，交换器的代码，以及java代码