rabbitmq Introduction two work queues

• 1 "Hello World!"

  The simplest thing that does something

  

  Python |Java

• 2 Work queues

  Distributing tasks among workers

  

  Python |Java

• 3 Publish/Subscribe

  Sending messages to many consumers at once

  

  Python |Java

• 4 Routing

  Receiving messages selectively

  

  Python |Java

• 5 Topics

  Receiving messages based on a pattern

  

  Python |Java

• 6 RPC

  Remote procedure call implementation

  

  Python |Java

Work Queues

(using the Java Client)

Where to get help

If you're having trouble going through this tutorial you can contact us through the discussion list or directly.

In the first tutorial we wrote programs to send and receive messages from a named queue. In this one we'll create a Work Queue that will be used to distribute time-consuming[耗时的] tasks among multiple workers.

The main idea behind Work Queues (aka: Task Queues) is to avoid doing a resource-intensive[资源密集型，大量占用资源的] task immediately and having to wait for it to complete. Instead we schedule[计划] the task to be done later. We encapsulate a task as a message and send it to a queue. A worker process running in the background will pop the tasks and eventually[最终] execute the job. When you run many workers the tasks will be shared between them.

This concept is especially useful in web applications where it's impossible to handle a complex task during a short HTTP request window.

Preparation

In the previous part of this tutorial we sent a message containing"Hello World!". Now we'll be sending strings that stand for complex tasks. We don't have a real-world task, like images to be resized[调整大小] or pdf files to be rendered[提出，归还], so let's fake[伪造] it by just pretending[冒充] we're busy - by using the Thread.sleep() function. We'll take the number of dots[点，圆点] in the string as its complexity[复杂度，复杂性]; every dot will account for[对..有责任，对...做出解释] one second of "work". For example, a fake[伪造] task described by Hello...will take three seconds.

We will slightly[轻微地] modify the Send.java code from our previous example,to allow arbitrary[任意地] messages to be sent from the command line. This program will schedule[安排，计划] tasks to our work queue, so let's name it NewTask.java:

String message = getMessage(argv);

channel.basicPublish("", "hello", null, message.getBytes());
System.out.println(" [x] Sent '" + message + "'");

Some help to get the message from the command line argument:

private static String getMessage(String[] strings){
    if (strings.length < 1)
        return "Hello World!";
    return joinStrings(strings, " ");
}

private static String joinStrings(String[] strings, String delimiter) {
    int length = strings.length;
    if (length == 0) return "";
    StringBuilder words = new StringBuilder(strings[0]);
    for (int i = 1; i < length; i++) {
        words.append(delimiter).append(strings[i]);
    }
    return words.toString();
}

Our old Recv.java script also requires some changes: it needs to fake a second of work for every dot in the message body. It will pop messages from the queue and perform the task, so let's call it Worker.java:

while (true) {
    QueueingConsumer.Delivery delivery = consumer.nextDelivery();
    String message = new String(delivery.getBody());

    System.out.println(" [x] Received '" + message + "'");        
    doWork(message);
    System.out.println(" [x] Done");
}

Our fake task to simulate execution time:

private static void doWork(String task) throws InterruptedException {
    for (char ch: task.toCharArray()) {
        if (ch == '.') Thread.sleep(1000);
    }
}

Compile them as in tutorial[辅导的，个别指导的] one (with the jar files in the working directory):

$ javac -cp rabbitmq-client.jar NewTask.java Worker.java

Round-robin[循环的，循环制] dispatching

One of the advantages of using a Task Queue is the ability to easily parallelise[平行的] work. If we are building up a backlog of work, we can just add more workers and that way, scale[规模，比例] easily.

First, let's try to run two Worker.java scripts at the same time. They will both get messages from the queue, but how exactly[恰好的，正是]? Let's see.

You need three consoles open. Two will run the Worker.javascript. These consoles will be our two consumers - C1 and C2.

shell1$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
 [*] Waiting for messages. To exit press CTRL+C

shell2$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
 [*] Waiting for messages. To exit press CTRL+C

In the third one we'll publish new tasks. Once you've started the consumers you can publish a few messages:

shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask First message.
shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask Second message..
shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask Third message...
shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask Fourth message....
shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask Fifth message.....

Let's see what is delivered to our workers:

shell1$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
 [*] Waiting for messages. To exit press CTRL+C
 [x] Received 'First message.'
 [x] Received 'Third message...'
 [x] Received 'Fifth message.....'

java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
 [*] Waiting for messages. To exit press CTRL+C
 [x] Received 'Second message..'
 [x] Received 'Fourth message....'

By default, RabbitMQ will send each message to the next consumer,in sequence. On average every consumer will get the same number of  messages. This way of distributing messages is called round-robin. Try this out with three or more workers.

Message acknowledgment[消息答复]

Doing a task can take a few seconds. You may wonder[想知道] what happens if one of the consumers starts a long task and dies with it only partly done.With our current code, once RabbitMQ delivers a message to the customer it immediately removes it from memory. In this case, if you kill a worker we will lose the message it was just processing. We'll also lose all the messages that were dispatched to this particular worker but were not yet handled.

But we don't want to lose any tasks. If a worker dies, we'd like the task to be delivered to another worker.

In order to make sure a message is never lost, RabbitMQ supports message acknowledgments. An ack(nowledgement) is sent back from the consumer to tell RabbitMQ that a particular message has been received,processed and that RabbitMQ is free to delete it.

If a consumer dies without sending an ack, RabbitMQ will understand that a message wasn't processed fully and will redeliver it to another consumer. That way you can be sure that no message is lost, even if the workers occasionally[偶然的] die.

There aren't any message timeouts; RabbitMQ will redeliver the message only when the worker connection dies. It's fine even if processing a message takes a very, very long time.

Message acknowledgments are turned on by default. In previous examples we explicitly[明确的] turned them off via the autoAck=trueflag. It's time to remove this flag and send a proper[适当的，本身的] acknowledgment from the worker, once we're done with a task.

QueueingConsumer consumer = new QueueingConsumer(channel);
boolean autoAck = false;
channel.basicConsume("hello", autoAck, consumer);

while (true) {
  QueueingConsumer.Delivery delivery = consumer.nextDelivery();
  //...      
  channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}

Using this code we can be sure that even if you kill a worker using CTRL+C while it was processing a message, nothing will be lost. Soon after the worker dies all unacknowledged messages will be redelivered[再投递].

Forgotten acknowledgment

It's a common mistake to miss the basicAck. It's an easy error,but the consequences[后果，结果，影响] are serious. Messages will be redelivered[再投递] when your client quits (which may look like random redelivery), but RabbitMQ will eat more and more memory as it won't be able to releaseany unacked messages.

In order to debug this kind of mistake you can use rabbitmqctl to print the messages_unacknowledged field:

$ sudo rabbitmqctl list_queues name messages_ready messages_unacknowledged
Listing queues ...
hello    0       0
...done.

Message durability[耐久性，持久性]

We have learned how to make sure that even if the consumer dies, the task isn't lost. But our tasks will still be lost if RabbitMQ server stops.

When RabbitMQ quits or crashes it will forget the queues and messages unless you tell it not to. Two things are required to make sure that messages aren't lost: we need to mark both the queue and messages as durable[耐久的，持久的].

First, we need to make sure that RabbitMQ will never lose our queue. In order to do so, we need to declare it as durable:

boolean durable = true;
channel.queueDeclare("hello", durable, false, false, null);

Although this command is correct by[正确的，改正] itself, it won't work in our present setup. That's because we've already defined a queue called hello which is not durable. RabbitMQ doesn't allow you to redefine an existing queue with different parameters and will return an error to any program that tries to do that. But there is a quick work around - let's declarea queue with different name, for example task_queue:

boolean durable = true;
channel.queueDeclare("task_queue", durable, false, false, null);

This queue Declare change needs to be applied to both the producerand consumer code.

At this point we're sure that the task_queue queue won't be lost even if RabbitMQ restarts. Now we need to mark our messages as persistent- by settingMessageProperties (which implementsBasicProperties) to the value PERSISTENT_TEXT_PLAIN.

import com.rabbitmq.client.MessageProperties;

channel.basicPublish("", "task_queue", 
            MessageProperties.PERSISTENT_TEXT_PLAIN,
            message.getBytes());

Note on message persistence

Marking messages as persistent doesn't fully guarantee[保证，担保] that a message won't be lost. Although it tells RabbitMQ to save the message to disk,there is still a short time window when RabbitMQ has accepted a message and hasn't saved it yet. Also, RabbitMQ doesn't do fsync(2)(文件管理) for every message -- it may be just saved to cache and not really written to the disk. The persistence[坚持，持久性] guarantees[保证，担保] aren't strong, but it's more than enough for our simple task queue. If you need a stronger guarantee[保证，担保] you can wrap[包裹，包装] the publishing code in a transaction.

Fair dispatch[公平调度]

You might have noticed that the dispatching still doesn't work exactly[正常的] as we want. For example in a situation[情况，形式] with two workers, when all odd[奇数的，剩余的] messages are heavy and even messages are light[轻的，容易的], one worker will be constantly[不断的，时长的] busy and the other one will do hardly any work. Well,RabbitMQ doesn't know anything about that and will still dispatch messages evenly.

This happens because RabbitMQ just dispatches a message when the message enters the queue. It doesn't look at the number of unacknowledged[不被承认的] messages for a consumer. It just blindly dispatches every n-th messageto the n-th consumer.

In order to defeat[击败，战胜] that we can use the basicQos method with the prefetchCount =1 setting. This tells RabbitMQ not to give more thanone message to a worker at a time. Or, in other words, don't dispatcha new message to a worker until it has processed and acknowledged theprevious one. Instead, it will dispatch it to the next worker that is not still busy.

int prefetchCount = 1;
channel.basicQos(prefetchCount);

Note about queue size

If all the workers are busy, your queue can fill up[堵塞]. You will want to keep an  eye on that[密切注视], and maybe add more workers, or have some other strategy[战略，策略].

Putting it all together

Final code of our NewTask.java class:

import java.io.IOException;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.MessageProperties;

public class NewTask {

  private static final String TASK_QUEUE_NAME = "task_queue";

  public static void main(String[] argv) 
                      throws java.io.IOException {

    ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();

    channel.queueDeclare(TASK_QUEUE_NAME, true, false, false, null);

    String message = getMessage(argv);

    channel.basicPublish( "", TASK_QUEUE_NAME, 
            MessageProperties.PERSISTENT_TEXT_PLAIN,
            message.getBytes());
    System.out.println(" [x] Sent '" + message + "'");

    channel.close();
    connection.close();
  }      
  //...
}

(NewTask.java source)

   

    import 
    com.rabbitmq.client.ConnectionFactory
    ;
   
   

    import 
    com.rabbitmq.client.Connection
    ;
   
   

    import 
    com.rabbitmq.client.Channel
    ;
   
   

    import 
    com.rabbitmq.client.MessageProperties
    ;
   
   


   
   

    public 
    class 
    NewTask 
    {
   
   

      
   
   

      
    private 
    static 
    final 
    String 
    TASK_QUEUE_NAME 
    = 
    "task_queue"
    ;
   
   


   
   

      
    public 
    static 
    void 
    main
    (
    String
    [] 
    argv
    ) 
    throws 
    Exception 
    {
   
   


   
   

        
    ConnectionFactory 
    factory 
    = 
    new 
    ConnectionFactory
    ();
   
   

        
    factory
    .
    setHost
    (
    "localhost"
    );
   
   

        
    Connection 
    connection 
    = 
    factory
    .
    newConnection
    ();
   
   

        
    Channel 
    channel 
    = 
    connection
    .
    createChannel
    ();
   
   

        
   
   

        
    channel
    .
    queueDeclare
    (
    TASK_QUEUE_NAME
    , 
    true
    , 
    false
    , 
    false
    , 
    null
    );
   
   

        
   
   

        
    String 
    message 
    = 
    getMessage
    (
    argv
    );
   
   

        
   
   

        
    channel
    .
    basicPublish
    ( 
    ""
    , 
    TASK_QUEUE_NAME
    , 
   
   

                    
    MessageProperties
    .
    PERSISTENT_TEXT_PLAIN
    ,
   
   

                    
    message
    .
    getBytes
    ());
   
   

        
    System
    .
    out
    .
    println
    (
    " [x] Sent '" 
    + 
    message 
    + 
    "'"
    );
   
   

        
   
   

        
    channel
    .
    close
    ();
   
   

        
    connection
    .
    close
    ();
   
   

      
    }
   
   

        
   
   

      
    private 
    static 
    String 
    getMessage
    (
    String
    [] 
    strings
    ){
   
   

        
    if 
    (
    strings
    .
    length 
    < 
    1
    )
   
   

          
    return 
    "Hello World!"
    ;
   
   

        
    return 
    joinStrings
    (
    strings
    , 
    " "
    );
   
   

      
    }  
   
   

      
   
   

      
    private 
    static 
    String 
    joinStrings
    (
    String
    [] 
    strings
    , 
    String 
    delimiter
    ) 
    {
   
   

        
    int 
    length 
    = 
    strings
    .
    length
    ;
   
   

        
    if 
    (
    length 
    == 
    0
    ) 
    return 
    ""
    ;
   
   

        
    StringBuilder 
    words 
    = 
    new 
    StringBuilder
    (
    strings
    [
    0
    ]);
   
   

        
    for 
    (
    int 
    i 
    = 
    1
    ; 
    i 
    < 
    length
    ; 
    i
    ++) 
    {
   
   

          
    words
    .
    append
    (
    delimiter
    ).
    append
    (
    strings
    [
    i
    ]);
   
   

        
    }
   
   

        
    return 
    words
    .
    toString
    ();
   
   

      
    }
   
   

    }
   

And our Worker.java:

import java.io.IOException;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.QueueingConsumer;

public class Worker {

  private static final String TASK_QUEUE_NAME = "task_queue";

  public static void main(String[] argv)
                      throws java.io.IOException,
                      java.lang.InterruptedException {

    ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();

    channel.queueDeclare(TASK_QUEUE_NAME, true, false, false, null);
    System.out.println(" [*] Waiting for messages. To exit press CTRL+C");

    channel.basicQos(1);

    QueueingConsumer consumer = new QueueingConsumer(channel);
    channel.basicConsume(TASK_QUEUE_NAME, false, consumer);

    while (true) {
      QueueingConsumer.Delivery delivery = consumer.nextDelivery();
      String message = new String(delivery.getBody());

      System.out.println(" [x] Received '" + message + "'");   
      doWork(message); 
      System.out.println(" [x] Done" );

      channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
    }
  }
  //...
}

(Worker.java source)

import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.QueueingConsumer;
  
public class Worker {

  private static final String TASK_QUEUE_NAME = "task_queue";

  public static void main(String[] argv) throws Exception {

    ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();
    
    channel.queueDeclare(TASK_QUEUE_NAME, true, false, false, null);
    System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
    
    channel.basicQos(1);
    
    QueueingConsumer consumer = new QueueingConsumer(channel);
    channel.basicConsume(TASK_QUEUE_NAME, false, consumer);
    
    while (true) {
      QueueingConsumer.Delivery delivery = consumer.nextDelivery();
      String message = new String(delivery.getBody());
      
      System.out.println(" [x] Received '" + message + "'");
      doWork(message);
      System.out.println(" [x] Done");

      channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
    }
  }
  
  private static void doWork(String task) throws InterruptedException {
    for (char ch: task.toCharArray()) {
      if (ch == '.') Thread.sleep(1000);
    }
  }
}

Using message acknowledgments and prefetchCount you can set up a work queue. The durability[持久性] options let the tasks survive[幸存，存活] even if RabbitMQ is restarted.

For more information on Channel methods and MessageProperties, you can browse thejavadocs online.

Now we can move on to tutorial 3 and learn how to deliver the same message to many consumers.

转载自：http://www.rabbitmq.com/tutorials/tutorial-two-java.html