storm传参,限流

本文探讨了在使用Storm从Kafka读取数据时遇到的并发修改问题,并提供了一种解决方案:通过设置参数来丢弃部分数据。文章还介绍了如何修改Spout和Bolt的构造函数。

storm通过kafka传入数据的时候,有时候会出现并发修改的问题(这很可能是有static变量造成的),界面会出现spout fail的状况,此时可以通过设置参数进行一部分数据的抛弃。
这里写图片描述

builder.setSpout("Data", new SpoutA(args[1]), 1);

builder.setBolt("test", new BoltA(args[2]), 1).shuffleGrouping(
                "Data"); 

args[0]是拓扑名,args[1]是传入spout的数,紧跟着args[1],args[2]是传入bolt的数据,紧跟着args[2]
例如 storm jar test.jar storm.Topology s abc 10
args[0]=s,args[1]=abc,args[2]=10

下面就是修改spout和bolt的构造函数了

SpoutA中作如下修改

public SpoutA(String s) {
        str=s;
    }

BoltA中作如下修改

    public BoltA(String s) {
        x = Integer.parseInt(s);
    }

//TODO 需要通过调用接口,传参为代码下边的时间戳,调用接口和getCaseNumByDimension的接口是同一个 // // SELECT count(1) num, ietype type FROM ods_pre_cases_info_impala where pectime between '${beginTime}' and '${endTime}' and ietype is not null and organcode like '${code}' group by ietype order by num desc InterfaceData interfaceData = new InterfaceData(); CaseStatisticSuperVo caseStatisticSuperVo = new CaseStatisticSuperVo(); /* Date beginDate = DateUtilExclusive.string2Date(beginTime,DateUtilExclusive.F1); Date endDate = DateUtilExclusive.string2Date(endTime,DateUtilExclusive.F1); int days = DateUtilExclusive.daysBetween(beginDate,endDate);*/ String functionUrl = "getCaseNumInClass"; Map<String, String> paramMap = new HashMap(); paramMap.put("beginTime", "2023-01-01 00:00:00"); paramMap.put("endTime", "2023-04-26 23:59:59"); if (StringUtils.isEmpty(region)) { paramMap.put("code", "37%"); } else { paramMap.put("code", region + "%"); } JSONObject paramObject = JSONObject.parseObject(JSONObject.toJSONString(paramMap)); String result = GetScreenResultUtils.getResult(interfaceData, functionUrl, paramObject); List<CaseStatisticVo> caseStatisticVos = JSONObject.parseObject(result, new TypeReference<List<CaseStatisticVo>>() { }); caseStatisticSuperVo.setCaseInClass(caseStatisticVos); for (CaseStatisticVo caseStatisticVo : caseStatisticVos) { String type = caseStatisticVo.getType(); caseStatisticVo.setType(InitMap.caseInClassMap.get(type)); } /* beginTime = DateUtilExclusive.date2String(DateUtilExclusive.dateChangeDay(beginDate,-days),DateUtilExclusive.F1); endTime = DateUtilExclusive.date2String(DateUtilExclusive.dateChangeDay(endDate,-days),DateUtilExclusive.F1);*/ paramMap.put("beginTime", "2022-01-01 00:00:00"); paramMap.put("endTime", "2022-04-26 23:59:59"); paramObject = JSONObject.parseObject(JSONObject.toJSONString(paramMap)); result = GetScreenResultUtils.getResult(interfaceData, functionUrl, paramObject); List<CaseStatisticVo> caseStatisticVos2 = JSONObject.parseObject(result, new TypeReference<List<CaseStatisticVo>>() { }); caseStatisticSuperVo.setCaseInClassLast(caseStatisticVos2); for (CaseStatisticVo caseStatisticVo : caseStatisticVos2) { String type = caseStatisticVo.getType(); caseStatisticVo.setType(InitMap.caseInClassMap.get(type)); } if (caseStatisticVos.size() < InitMap.caseInClassMap.size()) { Set<String> set = InitMap.caseInClassMap.keySet(); for (String s : set) { if (!isInList(InitMap.caseInClassMap.get(s), caseStatisticVos)) { CaseStatisticVo caseStatisticVo1 = new CaseStatisticVo(); caseStatisticVo1.setNum(0l); caseStatisticVo1.setType(InitMap.caseInClassMap.get(s)); caseStatisticVos.add(caseStatisticVo1); } } } if (caseStatisticVos2.size() < InitMap.caseInClassMap.size()) { Set<String> set = InitMap.caseInClassMap.keySet(); for (String s : set) { if (!isInList(InitMap.caseInClassMap.get(s), caseStatisticVos2)) { CaseStatisticVo caseStatisticVo1 = new CaseStatisticVo(); caseStatisticVo1.setNum(0l); caseStatisticVo1.setType(InitMap.caseInClassMap.get(s)); caseStatisticVos2.add(caseStatisticVo1); } } } return caseStatisticSuperVo;
最新发布
12-20
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值