本文假设读者已经对Storm的基础结构有了全面理解,并知道Nimbus与supervisor在集群之中所扮演的角色。之所以要理解Storm集群的并行机制,是为了能够对数据流中数据地处理顺序有一个深入地理解,这样才能更有信心地使用工具。
首先是需要了解一些与Storm集群并行机制相关地概念:
工作进程(worker process,就是一个JVM进程,通过在supervisor服务器上执行jps命令可以看到一个或多个的worker)的隔离性
一个Storm集群可以运行多个Topologies,但是对于一个worker来说,它只为一个Topology工作,这在一定程度上提供了Topologies之间的隔离性。
执行器(executor)
在工作进程JVM的的内部,可以启动多个线程来一起工作,每个线程就是一个执行器(executor),一个执行器只能执行拓扑中的一个组件,要么是一个Spout,要么是一个Bolt。这里,我们看到一个工作进程内部,某些线程可以执行Spout任务,而另外一些线程可以执行Bolt任务。
任务(tasks)
任务是Storm集群中最细粒度的划分,每个任务要么是一个Spout任务,要么是一个Bolt任务,再结合上面对于执行器的说明,我们可以看到这些任务被分配到一些执行器中去执行。我们在定义拓扑的时候,会定义好每个Spout和每个Bolt任务的数量,一旦定义好之后,是不能在拓扑运行的过程中动态变更相应组件的任务数的。
要理解Storm的并行机制,只需要理解Storm数据流中元组的处理顺序即可。为了更与生产环境接近,使用Maven作为处理包依赖的处理工具,并通过其maven-shade-plugin插件,将需要的jar包依赖共同打包到最终的Storm任务jar包中,而不用对Storm的运行环境做环境变量做处理,这确实带来了非常大的便利。首先,需要通过配置POM文件明确各种包的依赖:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>istorm</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>istorm</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-kafka</artifactId>
<version>0.9.5</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.8.2.1</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
<configuration>
<finalName>withdepends-${artifactId}-${version}</finalName>
</configuration>
</plugin>
</plugins>
</build>
</project>
在刚刚建立好Maven任务之后,需要下载所有的依赖包,这可能需要一些时间。有了这些依赖之后,下一步就要规划工程结构。为了更能够与生产环境接近,我们将Storm的数据源设定为Kafka,Storm和Kafka相关的依赖包已经在上面的POM文件中有所体现了,至于Kafka集群的搭建和使用,在此不做细致的说明。整个工程的结构是这样的:我们通过消费者向Kafka集群的一个TOPIC发送一系列的顺序的数字,1,2,3,4...N,而Storm(KafkaSpout)会读取这些数据,将其交付给SequenceBolt,它负责为序列数字添加上一些线程与任务相关的信息后,将其发送给PrinterBolt,这个PrinterBolt负责将收集到的数据发送给Kafka集群中的一个结果队列,我们能够从终端中读取这个这个队列中的结果,以观察最终的序列被处理的顺序。为了更能说明问题,我们的Topology能够通过参数来控制总的worker数量以及各个组件的tasks的数量,这在下面的代码示例中都会有所体现。
首先,是SequenceBolt的实现代码:
package com.example.istorm;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.Random;
import org.apache.commons.lang.StringUtils;
import com.google.common.collect.ImmutableList;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
public class SequenceBolt extends BaseBasicBolt {
private static final long serialVersionUID = -5735813963113079832L;
private int taskId;
private Random rand;
@Override
public void prepare(Map stormConf, TopologyContext context) {
this.taskId = context.getThisTaskId();
this.rand = new Random();
}
public void execute(Tuple input, BasicOutputCollector collector) {
String data = input.getString(0);
String timestamp = new Date().toString();
List<String> dataList = new ArrayList<String>();
dataList.add("sequenceBoltThreadID:" + Thread.currentThread().getId() + "@" + timestamp + "@sequnceTaskID:" + this.taskId);
dataList.add(data);
// simulate the work latency
try {
Thread.sleep(this.rand.nextInt(47));
} catch (InterruptedException e) {
e.printStackTrace();
}
collector.emit(ImmutableList.of((Object) StringUtils.join(dataList, '\t')));
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("sequence"));
}
public static void main(String[] args) throws InterruptedException{
List<String> dataList = new ArrayList<String>();
dataList.add("100");
dataList.add("200");
System.out.println(StringUtils.join(dataList, '\t'));
System.out.println(ImmutableList.of((Object) StringUtils.join(dataList, '\t')));
System.out.println("Sleep start");
Random rand = new Random();
Thread.sleep(rand.nextInt(1000));
System.out.println("Sleep end");
}
}
这个Bolt的逻辑非常简单,仅仅是在接收到序列数据之后,在其上加上了事件戳、线程信息、任务信息之后,直接提交这个数据。这么简单的逻辑,相信理解起来非常简单。下面是更简单的PrinterBolt,它仅仅负责将过来的数据发送到Kafka的结果队列之中:
package com.example.istorm;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
import org.apache.commons.lang.StringUtils;
import com.example.utils.KafkaUtils;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Tuple;
public class PrinterBolt extends BaseBasicBolt {
private static final long serialVersionUID = -6508964903665304388L;
// Every printer bolt task will hold a kafkaUtils object that will be used to
// post the result to the Kafka servers
private KafkaUtils kafkaUtils = null;
private int taskId;
@Override
public void prepare(Map stormConf, TopologyContext context) {
this.kafkaUtils = new KafkaUtils("resulttopic"); // we will send result to this topic
this.taskId = context.getThisTaskId();
}
@Override
public void cleanup() {
if(this.kafkaUtils != null){
this.kafkaUtils.close();
}
}
public void execute(Tuple input, BasicOutputCollector collector) {
String sequence = input.getStringByField("sequence");
String timestamp = new Date().toString();
List<String> dataList = new ArrayList<String>();
dataList.add("printerBoltThreadID:" + Thread.currentThread().getId() + "@" + timestamp + "@printerTaskID:" + this.taskId);
dataList.add(sequence);
this.kafkaUtils.publishMessage(StringUtils.join(dataList, '\t'));
}
public void declareOutputFields(OutputFieldsDeclarer arg0) {
// we don't emit anything
}
}
在这个程序中的prepare方法中,创建了Kafka的工具类,用来向Kafka的某个TOPIC发送消息,这个工具类的代码也只有简单几行而已:
package com.example.utils;
import java.util.Properties;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
public class KafkaUtils {
private Producer<String, String> producer;
private String topic;
public KafkaUtils(String topic){
Properties props = new Properties();
props.put("metadata.broker.list", "a00:9092,a01:9092,a02:9092");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("partitioner.class", "com.example.utils.SimplePartitioner");
props.put("request.required_acks", "1");
ProducerConfig config = new ProducerConfig(props);
this.producer = new Producer<String, String>(config);
this.topic = topic;
}
public void publishMessage(String msg){
KeyedMessage<String, String> data = new KeyedMessage<String, String>(this.topic, msg, msg);
this.producer.send(data);
}
public void close(){
this.producer.close();
}
public static void main(String[] args){
String topic = "datasrctopic";
KafkaUtils sp = new KafkaUtils(topic);
for(int i=0; i < 100; i++){
sp.publishMessage(Integer.toString(i));
}
sp.close();
}
}
其中的main方法,实现了向Kafka中发送数据的功能,我们以后的测试数据就是通过执行这个方法获取的。为了更能说明问题,我实现了一个Kafka简单的分区函数,其实现也是非常简单的,其代码也是最简短的:
package com.example.utils;
import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;
public class SimplePartitioner implements Partitioner {
public SimplePartitioner(VerifiableProperties props){
}
public int partition(Object key, int a_numPartitions) {
String partitionKey = (String) key;
return partitionKey.hashCode() % a_numPartitions;
}
}
这个代码,我们期望能够将消息分散到Kafka相应TOPIC中不同的分区中。最后,是将整个Topology初始化:
package com.example.istorm;
import backtype.storm.spout.SchemeAsMultiScheme;
import backtype.storm.topology.TopologyBuilder;
import java.util.ArrayList;
import java.util.List;
import backtype.storm.Config;
//import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.generated.AlreadyAliveException;
import backtype.storm.generated.InvalidTopologyException;
import storm.kafka.KafkaSpout;
import storm.kafka.SpoutConfig;
import storm.kafka.StringScheme;
import storm.kafka.ZkHosts;
public class KafkaTopology {
public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException{
// Zookeeper hosts for the Kafka cluster
ZkHosts zkHosts = new ZkHosts("a00:2181,a01:2181,a02:2181");
// Create the KafkaSpout configration
// Second argument is the topic name
// Third argument is the Zookeeper root for Kafka
// Fourth argument is consumer group id
SpoutConfig kafkaConfig = new SpoutConfig(zkHosts, "datasrctopic","/tmp", "id7");
int workerNum = Integer.parseInt(args[0]);
int kafkaSpoutNum = Integer.parseInt(args[1]);
int kafkaSpoutTaskNum = Integer.parseInt(args[2]);
int sequenceBoltNum = Integer.parseInt(args[3]);
int sequenceBoltTaskNum = Integer.parseInt(args[4]);
int printerBoltNum = Integer.parseInt(args[5]);
int printerBoltTaskNum = Integer.parseInt(args[6]);
// final List<String> zkServerAddresses = new ArrayList<String>();
// zkServerAddresses.add("a00");
// zkServerAddresses.add("a01");
// zkServerAddresses.add("a02");
// kafkaConfig.zkServers = zkServerAddresses;
// kafkaConfig.zkPort = 2181;
// Specify that the kafka messages are String
kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
// We want to consume all the first messages in
// the topic every time we run the topology to
// help in debugging. In production, this property should be false
kafkaConfig.forceFromStart = false;
// Now we create the topology
TopologyBuilder builder = new TopologyBuilder();
// Set the kafka spout class
builder.setSpout("KafkaSpout", new KafkaSpout(kafkaConfig), kafkaSpoutNum)
.setNumTasks(kafkaSpoutTaskNum);
// Configure the bolts
builder.setBolt("SequenceBolt", new SequenceBolt(), sequenceBoltNum)
.setNumTasks(sequenceBoltTaskNum)
.globalGrouping("KafkaSpout");
builder.setBolt("PrinterBolt", new PrinterBolt(), printerBoltNum)
.setNumTasks(printerBoltTaskNum)
.globalGrouping("SequenceBolt");
Config conf = new Config();
conf.setNumWorkers(workerNum);
StormSubmitter.submitTopology("KafkaTopology", conf, builder.createTopology());
/*
// Create an instance of LocalCluster class
// for executing in local mode.
LocalCluster cluster = new LocalCluster();
// Submit topology for execution
cluster.submitTopology("KafkaTopology", conf, builder.createTopology());
try{
// Wait for some time before exiting
Thread.sleep(100000);
} catch(Exception exception){
System.out.println("Thread interrupted exception: " + exception);
}
// Kill the KafkaTopology
cluster.killTopology("KafkaTopology");
// Shutdown the storm test cluster
cluster.shutdown();
*/
}
}
这里需要注意的一点是,如果Storm是运行在LocalCluster的模式下,那么我们必须显式指定如下的代码,才能让Storm-Kafka记住上次的偏移量到Zookeeper服务中:
final List<String> zkServerAddresses = new ArrayList<String>();
zkServerAddresses.add("a00");
zkServerAddresses.add("a01");
zkServerAddresses.add("a02");
kafkaConfig.zkServers = zkServerAddresses;
kafkaConfig.zkPort = 2181;
当然,如果是在集群模式下运行,就不需要这几行代码了。在我们的Topology的配置中,我们使用了globalGrouping作为分发数据的方法,这种分发数据流的方法并不会将数据流分区,而是将整个数据流完整地分发到具有最小任务ID地BOLT中,有点儿类似于MapReduce中的Reduce阶段的数据流。
场景1:数据源的顺序确定。
对于数据源的顺序,我们经常会忽视。对于像Kafka这样的数据源,我们就不得不考虑其分区特性对于Storm数据源输入顺序的影响:Kafka在每个分区上,通过顺序偏移量的机制为所有消息提供了强的顺序保证,但是在不同的分区之间并没有这种顺序的保证。为了验证这个想法,我们在Kafka的数据源队列的TOPIC创建时,这样创建:
bin/kafka-topics.sh --create --zookeeper a00:2181,a01:2181,a02:2181 --replication-factor 1 --partitions 10 --topic datasrctopic
数据源TOPIC创建时,指定其有10个分区,我们期待Storm在读取这个TOPIC的数据时,总体上是无序的(虽然每个分区内的消息时有序的)。而最终的结果TOPIC,为了消除多分区对输出顺序的影响,我们仅仅指定其有1个分区:
bin/kafka-topics.sh --create --zookeeper a00:2181,a01:2181,a02:2181 --replication-factor 1 --partitions 1 --topic resulttopic
在有了如上的准备之后,我们就可以提交Topology到Storm集群上了:
bin/storm jar /tmp/withdepends-istorm-0.0.1-SNAPSHOT.jar com.example.istorm.KafkaTopology 1 1 1 1 1 1 1
为了消除其它影响,我们将worker数量以及各个组件的执行器数量和任务数量都设置为1,这样整体的顺序就取决于数据源的顺序了。这个时候,我们去执行一下KafkaUtils中的main函数,我们从结果队列中得到如下的输出:
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 0
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 2
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 1
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 4
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 7
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 5
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 8
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 11
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 3
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 12
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 13
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 15
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 18
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 6
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 17
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 14
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 16
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 9
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 10
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 21
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 19
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 20
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 22
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 24
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 23
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 25
printerBoltThreadID:31@Mon Sep 14 15:31:01 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 28
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:01 CST 2015@sequnceTaskID:3 27
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 26
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 29
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 30
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 31
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 33
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 36
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 35
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 32
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 34
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 38
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 37
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 39
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 40
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 42
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 45
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 44
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 41
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 43
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 47
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 46
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 48
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 50
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 49
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 51
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 52
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 53
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 55
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 54
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 56
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 57
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 58
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 59
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 60
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 61
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 63
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 62
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 65
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 67
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 64
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 66
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 68
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 71
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 69
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 70
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 73
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 75
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 72
printerBoltThreadID:31@Mon Sep 14 15:31:02 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 74
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:02 CST 2015@sequnceTaskID:3 76
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 77
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 79
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 78
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 82
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 81
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 80
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 83
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 84
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 87
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 85
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 86
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 88
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 90
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 89
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 91
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 92
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 93
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 94
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 95
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 97
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 96
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 98
printerBoltThreadID:31@Mon Sep 14 15:31:03 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:31:03 CST 2015@sequnceTaskID:3 99
明显看到,整个输出时无序的。这个时候的无序完全是由于Kafka不能在多个分区上保持消息的有序性造成的,那么如果要让消息有序有两种方法,一种方法就是修改Kafka的分区函数,将其全部消息发向同一个分区;第二种方法,就是在创建TOPIC的时候,指定只有1个分区。当然,作为测试,我们采用第二种方法,在实际应用下,可能要利用第一种方法,采取某种Kafka的分区策略,比如按照用户的uid分区到Kafka的各个分区上,保证每个用户产生的消息是有序的。采用第二种方法就是简单地重建TOPIC:
bin/kafka-topics.sh --delete --zookeeper a00:2181,a01:2181,a02:2181 --topic datasrctopic
bin/kafka-topics.sh --create --zookeeper a00:2181,a01:2181,a02:2181 --replication-factor 1 --partitions 1 --topic datasrctopic
在做好如上地处理之后,我们再执行先前地程序,最终地输出就成为:
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 0
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 1
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 2
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 3
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 4
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 5
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 6
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 7
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 8
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 9
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 10
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 11
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 12
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 13
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 14
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 15
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 16
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 17
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 18
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 19
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 20
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 21
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 22
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 23
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 24
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 25
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 26
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 27
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 28
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 29
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 30
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 31
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 32
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 33
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 34
printerBoltThreadID:31@Mon Sep 14 15:40:42 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 35
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:42 CST 2015@sequnceTaskID:3 36
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 37
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 38
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 39
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 40
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 41
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 42
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 43
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 44
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 45
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 46
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 47
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 48
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 49
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 50
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 51
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 52
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 53
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 54
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 55
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 56
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 57
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 58
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 59
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 60
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 61
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 62
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 63
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 64
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 65
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 66
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 67
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 68
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 69
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 70
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 71
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 72
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 73
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 74
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 75
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 76
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 77
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 78
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 79
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 80
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 81
printerBoltThreadID:31@Mon Sep 14 15:40:43 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 82
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:43 CST 2015@sequnceTaskID:3 83
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 84
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 85
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 86
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 87
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 88
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 89
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 90
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 91
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 92
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 93
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 94
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 95
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 96
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 97
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 98
printerBoltThreadID:31@Mon Sep 14 15:40:44 CST 2015@printerTaskID:2 sequenceBoltThreadID:33@Mon Sep 14 15:40:44 CST 2015@sequnceTaskID:3 99
正如,我们地预期,输出确实变成了顺序输出。这样就提醒我们,在需要顺序处理某些数据地时候,一定不能忽视数据源的顺序,具体到Kafka,就是采用合适的分区策略,利用单个分区的有序特性来达成自己的目的。
场景2:探究Global Grouping的特性。
Global Grouping不会对数据流做分区处理,而是将整个数据流完整发送到具有最小任务ID的BOLT任务上,也就是说,不论一个BOLT开几个任务,只有一个任务会获取数据流,也就是说,在我们的示例中,不论我们开启几个BOLT任务,我们得到也总是有序的输出。为测试,我们的想法,我们将以如下的方式来提交Topology到Storm集群中:
bin/storm jar /tmp/withdepends-istorm-0.0.1-SNAPSHOT.jar com.example.istorm.KafkaTopology 2 2 2 2 2 2 2
这样之后,我们向数据源TOPIC中发送数据,得到的输出仍旧是有序的(为了避免篇幅冗余,不再贴具体输出)。多个worker去读取数据的时候,也是要参照Zookeeper中存储的偏移量的,所以即使是多个worker去读Kafka队列(因为只有一个分区)中的数据,得到的数据也是有序的,并不会在数据读取时发生顺序上的变动。
场景3:探究Shuffle Grouping的特性。
Shuffle Grouping的分发方式是把数据流中的元组随机地在任务之间分发,既然这里提到了随机,那么就有可能出现失序地问题。但是,这里必须十分注意执行器和任务之间地关系。执行器就是一个独立地线程,而任务就是一个Bolt地实例对象,而一个执行器可以执行多个任务。如果每个Bolt不管有多少个任务,但是只有一个执行器来执行它们,那么理论上是不会失序的,那么实际上是怎样的呢?为了验证,我们理论上的想法,可以在启动Topology的时候,将每个Bolt的执行器都设置成1,而任务设置成多个,然后再做测试(不要忘了将Grouping改成Shuffle Grouping,并重新打包):
bin/storm jar /tmp/withdepends-istorm-0.0.1-SNAPSHOT.jar com.example.istorm.KafkaTopology 2 2 2 1 10 1 10
输出如下:
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:13 0
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:21 1
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:15 2
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:17 3
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:14 4
printerBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:19 5
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:22 6
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:16 7
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:18 8
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:09 CST 2015@sequnceTaskID:20 9
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:13 10
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:21 11
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:19 12
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:18 13
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:17 14
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:16 15
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:20 16
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:15 17
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:22 18
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:14 19
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:17 20
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:22 21
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:13 22
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:15 23
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:19 24
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:20 25
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:21 26
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:14 27
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:16 28
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:18 29
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:16 30
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:13 31
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:22 32
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:18 33
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:14 34
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:21 35
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:20 36
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:15 37
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:19 38
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:17 39
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:14 40
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:16 41
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:13 42
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:20 43
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:17 44
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:18 45
printerBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:22 46
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:10 CST 2015@sequnceTaskID:19 47
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 48
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:15 49
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:17 50
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 51
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:16 52
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:22 53
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:13 54
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:15 55
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:20 56
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:19 57
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:18 58
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:14 59
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:22 60
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:16 61
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:19 62
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:18 63
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:17 64
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:13 65
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:15 66
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:20 67
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 68
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:14 69
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:17 70
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:13 71
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:20 72
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:18 73
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:19 74
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 75
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:14 76
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:15 77
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:16 78
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:22 79
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:14 80
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:21 81
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:13 82
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:22 83
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:16 84
printerBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:18 85
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:11 CST 2015@sequnceTaskID:20 86
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:15 87
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:19 88
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:17 89
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:17 90
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:20 91
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:19 92
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:15 93
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:16 94
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:18 95
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:21 96
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:22 97
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:14 98
printerBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:21:12 CST 2015@sequnceTaskID:13 99
可以看到,数据流中的元组被平均分配到了不同的任务中,但是输出依旧是有序的。那么继续我们的思路,如果要让输出失序,只要将其中一个BOLT的执行器设置成2个即可:
bin/storm jar /tmp/withdepends-istorm-0.0.1-SNAPSHOT.jar com.example.istorm.KafkaTopology 2 2 2 2 10 1 10
这里,我们将SequenceBolt的执行器设置成了2个,其对应的输出是:
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 2
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:20 3
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 4
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:16 0
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:14 1
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:19 7
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:15 5
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:13 6
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 8
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:17 9
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 10
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:19 12
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:13 11
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:20 15
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 17
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:16 13
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:15 14
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 18
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:14 16
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:19 20
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 21
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:17 19
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 24
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 26
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:13 22
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:20 29
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:14 23
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:15 25
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:19 30
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:20 32
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:16 27
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 34
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:17 28
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 37
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 38
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:13 31
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:21 40
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:15 33
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:17 35
printerBoltThreadID:40@Mon Sep 14 16:24:37 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:22 42
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:16 36
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:37 CST 2015@sequnceTaskID:18 43
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 47
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 39
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 41
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 49
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 51
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 44
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 45
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 46
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 52
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 48
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 53
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 50
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 54
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 55
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 58
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 56
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 57
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 63
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 64
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 59
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 60
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 65
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 61
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 67
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 62
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 69
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 66
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 68
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 70
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 73
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 71
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 75
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 72
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 76
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 74
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 77
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 78
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 79
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 80
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 82
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 81
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 83
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:10 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 85
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:9 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 84
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:5 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 86
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 88
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 87
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 89
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:7 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:22 92
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:19 95
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:15 90
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:12 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:17 91
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:8 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:20 96
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:18 97
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:11 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:14 93
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:4 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:21 98
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:3 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:16 94
printerBoltThreadID:40@Mon Sep 14 16:24:38 CST 2015@printerTaskID:6 sequenceBoltThreadID:38@Mon Sep 14 16:24:38 CST 2015@sequnceTaskID:13 99
可以看到,在任务流中的元组被随机分配到不同的任务中的同时,其最终的总体顺序也被打乱了,这仅仅是因为我们为SequenceBolt分配了两个执行器。同样,如果我们只为PrinterBolt分配1个以上的执行器,也会造成输出的无序。
总结:希望通过这几个例子,能够有助于更深入理解Storm并行的各个组件,并在业务对流处理有序性有要求的时候能够采取正确的处理方案。类似Kafka这样的输入源,一定要考虑其自身特点,其分区内有序,分区外无序的特性,使得使用个性化的Kafka分区策略是必要的。而要保证Storm处理流程中的有序性,可以使用Global Grouping来完成。
与Kafka分区机制类似的是,Storm的Fields Grouping,比如你可以指定按照用户的uid字段进行Grouping,那么同一个用户的所有的数据都会进入到同一个任务中,这样的话你在一个任务中才能统计到某一个用户的完整信息。
当然,从另一个角度来说,要保证顺序就可能要做对应的性能牺牲,最好能够在业务设计的时候,尽量避免这种一致性的要求,但是如果不能绕过,那也是可以满足需求的,只不过可能要牺牲一些性能来妥协了。