Storm Trident示例broadcast

最新推荐文章于 2024-10-16 10:15:15 发布

原创最新推荐文章于 2024-10-16 10:15:15 发布 · 231 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#storm #trident #broadcast

storm 专栏收录该内容

13 篇文章

订阅专栏

下代码使用broadcast做repartition, 广播，会把tuples分配到所有的partitions当中, 如果有5个partition，则会把原tuples复制5份，分配到5个partition去

省略部分代码，省略部分可参考：https://blog.youkuaiyun.com/nickta/article/details/79666918

FixedBatchSpout spout = new FixedBatchSpout(new Fields("user", "score"), 3, 
        		new Values("nickt1", 4),
                new Values("nickt2", 7), 
                new Values("nickt3", 8),
                new Values("nickt4", 9), 
                new Values("nickt5", 7),
                new Values("nickt6", 11),
                new Values("nickt7", 5)
                );
        spout.setCycle(false);
        TridentTopology topology = new TridentTopology();
        topology.newStream("spout1", spout)
        		.broadcast()
        		.each(new Fields("user"),new Debug("print:"))
        		.parallelismHint(5);

输出：源tuples从nickt1到nickt7,一共7条，因为parallelismHint为5，所以有5个partition，

7条数据，复制5份，最终输出7*5=35条数据，而且从结果可以看出，每个partition输出的内容是相同的，都是nickt1到nickt7.

<Fri Mar 23 16:05:09 CST 2018[partition4-Thread-142-b-0-executor[37 37]]> DEBUG(print:): [nickt1]
<Fri Mar 23 16:05:09 CST 2018[partition4-Thread-142-b-0-executor[37 37]]> DEBUG(print:): [nickt2]
<Fri Mar 23 16:05:09 CST 2018[partition4-Thread-142-b-0-executor[37 37]]> DEBUG(print:): [nickt3]
<Fri Mar 23 16:05:09 CST 2018[partition0-Thread-106-b-0-executor[33 33]]> DEBUG(print:): [nickt1]
<Fri Mar 23 16:05:09 CST 2018[partition0-Thread-106-b-0-executor[33 33]]> DEBUG(print:): [nickt2]
<Fri Mar 23 16:05:09 CST 2018[partition0-Thread-106-b-0-executor[33 33]]> DEBUG(print:): [nickt3]
<Fri Mar 23 16:05:09 CST 2018[partition0-Thread-106-b-0-executor[33 33]]> DEBUG(print:): [nickt4]
<Fri Mar 23 16:05:09 CST 2018[partition0-Thread-106-b-0-executor[33 33]]> DEBUG(print:): [nickt5]
<Fri Mar 23 16:05:09 CST 2018[partition4-Thread-142-b-0-executor[37 37]]> DEBUG(print:): [nickt4]
<Fri Mar 23 16:05:09 CST 2018[partition0-Thread-106-b-0-executor[33 33]]> DEBUG(print:): [nickt6]
<Fri Mar 23 16:05:09 CST 2018[partition4-Thread-142-b-0-executor[37 37]]> DEBUG(print:): [nickt5]
<Fri Mar 23 16:05:09 CST 2018[partition4-Thread-142-b-0-executor[37 37]]> DEBUG(print:): [nickt6]
<Fri Mar 23 16:05:09 CST 2018[partition2-Thread-99-b-0-executor[35 35]]> DEBUG(print:): [nickt1]
<Fri Mar 23 16:05:09 CST 2018[partition2-Thread-99-b-0-executor[35 35]]> DEBUG(print:): [nickt2]
<Fri Mar 23 16:05:09 CST 2018[partition3-Thread-69-b-0-executor[36 36]]> DEBUG(print:): [nickt1]
<Fri Mar 23 16:05:09 CST 2018[partition3-Thread-69-b-0-executor[36 36]]> DEBUG(print:): [nickt2]
<Fri Mar 23 16:05:09 CST 2018[partition3-Thread-69-b-0-executor[36 36]]> DEBUG(print:): [nickt3]
<Fri Mar 23 16:05:09 CST 2018[partition3-Thread-69-b-0-executor[36 36]]> DEBUG(print:): [nickt4]
<Fri Mar 23 16:05:09 CST 2018[partition3-Thread-69-b-0-executor[36 36]]> DEBUG(print:): [nickt5]
<Fri Mar 23 16:05:09 CST 2018[partition3-Thread-69-b-0-executor[36 36]]> DEBUG(print:): [nickt6]
<Fri Mar 23 16:05:09 CST 2018[partition2-Thread-99-b-0-executor[35 35]]> DEBUG(print:): [nickt3]
<Fri Mar 23 16:05:09 CST 2018[partition2-Thread-99-b-0-executor[35 35]]> DEBUG(print:): [nickt4]
<Fri Mar 23 16:05:09 CST 2018[partition2-Thread-99-b-0-executor[35 35]]> DEBUG(print:): [nickt5]
<Fri Mar 23 16:05:09 CST 2018[partition2-Thread-99-b-0-executor[35 35]]> DEBUG(print:): [nickt6]
<Fri Mar 23 16:05:09 CST 2018[partition1-Thread-65-b-0-executor[34 34]]> DEBUG(print:): [nickt1]
<Fri Mar 23 16:05:09 CST 2018[partition1-Thread-65-b-0-executor[34 34]]> DEBUG(print:): [nickt2]
<Fri Mar 23 16:05:09 CST 2018[partition1-Thread-65-b-0-executor[34 34]]> DEBUG(print:): [nickt3]
<Fri Mar 23 16:05:09 CST 2018[partition1-Thread-65-b-0-executor[34 34]]> DEBUG(print:): [nickt4]
<Fri Mar 23 16:05:09 CST 2018[partition1-Thread-65-b-0-executor[34 34]]> DEBUG(print:): [nickt5]
<Fri Mar 23 16:05:09 CST 2018[partition1-Thread-65-b-0-executor[34 34]]> DEBUG(print:): [nickt6]
<Fri Mar 23 16:05:09 CST 2018[partition4-Thread-142-b-0-executor[37 37]]> DEBUG(print:): [nickt7]
<Fri Mar 23 16:05:09 CST 2018[partition2-Thread-99-b-0-executor[35 35]]> DEBUG(print:): [nickt7]
<Fri Mar 23 16:05:09 CST 2018[partition0-Thread-106-b-0-executor[33 33]]> DEBUG(print:): [nickt7]
<Fri Mar 23 16:05:09 CST 2018[partition3-Thread-69-b-0-executor[36 36]]> DEBUG(print:): [nickt7]
<Fri Mar 23 16:05:09 CST 2018[partition1-Thread-65-b-0-executor[34 34]]> DEBUG(print:): [nickt7]