jstorm 数据流分流和合并

最新推荐文章于 2025-06-11 15:38:09 发布

原创最新推荐文章于 2025-06-11 15:38:09 发布 · 2.1k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#storm

本文介绍了使用JStorm进行数据流分发的几种方式，包括发送不同tuple、相同tuple到多个组件以及数据流合并的方法，并详细展示了如何通过配置实现这些功能。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

参考：
https://github.com/alibaba/jstorm/wiki/%E6%95%B0%E6%8D%AE%E6%B5%81%E5%88%86%E6%B5%81%E5%92%8C%E5%90%88%E5%B9%B6

1.分流- 发送不同的tuple

在SplitRecord.java中：SplitRecord将发送两个tuple

collector.emit(SequenceTopologyDef.TRADE_STREAM_ID, new Values(tupleId, trade));
collector.emit(SequenceTopologyDef.CUSTOMER_STREAM_ID, new Values(tupleId, customer));

SequenceTopologyTool.java：TRADE_BOLT_NAME和CUSTOMER_BOLT_NAME接收来自SplitRecord不同的tuple

 builder.setBolt(SequenceTopologyDef.SPLIT_BOLT_NAME, new SplitRecord(), bolt_Parallelism_hint)              .localOrShuffleGrouping(SequenceTopologyDef.SEQUENCE_SPOUT_NAME);

builder.setBolt(SequenceTopologyDef.TRADE_BOLT_NAME, new PairCount(), bolt_Parallelism_hint)
.shuffleGrouping(SequenceTopologyDef.SPLIT_BOLT_NAME, SequenceTopologyDef.TRADE_STREAM_ID);

builder.setBolt(SequenceTopologyDef.CUSTOMER_BOLT_NAME, new PairCount(), bolt_Parallelism_hint)
.shuffleGrouping(SequenceTopologyDef.SPLIT_BOLT_NAME, SequenceTopologyDef.CUSTOMER_STREAM_ID);

定义输出格式：

public void declareOutputFields(OutputFieldsDeclarer declarer) {
  declarer.declareStream(SequenceTopologyDef.TRADE_STREAM_ID, new Fields("ID", "TRADE"));
  declarer.declareStream(SequenceTopologyDef.CUSTOMER_STREAM_ID, new Fields("ID", "CUSTOMER"));
 }

接收数据时判断数据流：

if (input.getSourceStreamId().equals(SequenceTopologyDef.TRADE_STREAM_ID) ) {
            customer = pair;
            customerTuple = input;       
            tradeTuple = tradeMap.get(tupleId);
            if (tradeTuple == null) {
                customerMap.put(tupleId, input);
                return;
            }            
            trade = (Pair) tradeTuple.getValue(1);

        }

2.分流- 发送相同的tuple到多个tuple

SpoutDeclarer spout = builder.setSpout(SequenceTopologyDef.SEQUENCE_SPOUT_NAME,
                new SequenceSpout(), spoutParal);

builder.setBolt(SequenceTopologyDef.TRADE_BOLT_NAME, new PairCount(), 1).shuffleGrouping(
                        SequenceTopologyDef.SEQUENCE_SPOUT_NAME);

builder.setBolt(SequenceTopologyDef.CUSTOMER_BOLT_NAME, new PairCount(), 1)
                        .shuffleGrouping(SequenceTopologyDef.SEQUENCE_SPOUT_NAME);

4.分流- 数据流合并

在下面例子中， MergeRecord 同时接收SequenceTopologyDef.TRADE_BOLT_NAME 和SequenceTopologyDef.CUSTOMER_BOLT_NAME 的数据

          builder.setBolt(SequenceTopologyDef.TRADE_BOLT_NAME, new PairCount(), 1).shuffleGrouping(
                        SequenceTopologyDef.SPLIT_BOLT_NAME, 
                        SequenceTopologyDef.TRADE_STREAM_ID);

                builder.setBolt(SequenceTopologyDef.CUSTOMER_BOLT_NAME, new PairCount(), 1)
                        .shuffleGrouping(SequenceTopologyDef.SPLIT_BOLT_NAME,
                                SequenceTopologyDef.CUSTOMER_STREAM_ID);

                builder.setBolt(SequenceTopologyDef.MERGE_BOLT_NAME, new MergeRecord(), 1)
                        .shuffleGrouping(SequenceTopologyDef.TRADE_BOLT_NAME)
                        .shuffleGrouping(SequenceTopologyDef.CUSTOMER_BOLT_NAME);

接收方
接收方是，区分一下来源component即可识别出数据的来源:

 if (input.getSourceComponent().equals(SequenceTopologyDef.CUSTOMER_BOLT_NAME) ) {
            customer = pair;
            customerTuple = input;

            tradeTuple = tradeMap.get(tupleId);
            if (tradeTuple == null) {
                customerMap.put(tupleId, input);
                return;
            }

            trade = (Pair) tradeTuple.getValue(1);

        } else if (input.getSourceComponent().equals(SequenceTopologyDef.TRADE_BOLT_NAME)) {
            trade = pair;
            tradeTuple = input;

            customerTuple = customerMap.get(tupleId);
            if (customerTuple == null) {
                tradeMap.put(tupleId, input);
                return;
            }

            customer = (Pair) customerTuple.getValue(1);
        }