Fastjson关于List.addAll(Collection<E>)方法遇到的问题

本文探讨了在SpringMVC项目中使用FastJSON时遇到的循环引用问题,并提供了解决方案,包括更换JSON处理库为Jackson及如何配置。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

涉及的软件环境:SpringMVC,fastjson

有一个需求,需要将热词信息跟自定义的热词信息(另外一个实体的信息)合并到一个列表里,

获取hotKeyList的代码

List<HotKeys> hotKeyList = hotKeysDao.getAllHotKeys();

获取自定义热词的代码:

 List<AdInfoExt> comprehensivePageAds = adInfoDao.getAdInfoListByRuleCode("comprehensive_search_page", "ad");
        List<HotKeys> cpaKeys = new ArrayList<HotKeys>();
        for(AdInfoExt ad : comprehensivePageAds){
            HotKeys key = new HotKeys();
            key.setHotKey(ad.getAdTittle());
            key.setStats(1);
            key.setUrl(ad.getUrl());
            cpaKeys.add(key);
        }

显然最终都是封装成List<HotKeys>了,然后将hotKeyListcpaKeys合并到一起用了:

cpaKeys.addAll(hotKeyList);

或者:

        for(HotKeys k : hotKeyList){
            cpaKeys.add(k);
        }

都产生了下面的问题:输出到客户端的时候,cpaKeys是正常的,

"cpaKey": [
    {
      "k": "杨振宁遗产分配",
      "sta": 0,
      "u": "https://wap.sogou.com/web/searchList.jsp?keyword=%E6%9D%A8%E6%8C%AF%E5%AE%81%E9%81%97%E4%BA%A7%E5%88%86%E9%85%8D&pid=sogou-mobp-eeea8c180c5dff16&v=5"
    },
    {
      "k": "泉州6车追尾",
      "sta": 0,
      "u": "https://wap.sogou.com/web/searchList.jsp?keyword=%E6%B3%89%E5%B7%9E6%E8%BD%A6%E8%BF%BD%E5%B0%BE&pid=sogou-mobp-eeea8c180c5dff16&v=5"
    },
    {
      "k": "翻车致19人遇难",
      "sta": 1,
      "u": "https://wap.sogou.com/web/searchList.jsp?keyword=%E7%BF%BB%E8%BD%A6%E8%87%B419%E4%BA%BA%E9%81%87%E9%9A%BE&pid=sogou-mobp-eeea8c180c5dff16&v=5"
    }
]

而原来的hotKeyList的数据就变成了以下的模样(它正确的样子应该跟上面的cpaKey一样):

 "hotKey": [
    {
      "$ref": "$.cpaKey[0]"
    },
    {
      "$ref": "$.cpaKey[1]"
    },
    {
      "$ref": "$.cpaKey[2]"
    },
    {
      "$ref": "$.cpaKey[3]"
    },
    {
      "$ref": "$.cpaKey[4]"
    },
    {
      "$ref": "$.cpaKey[5]"
    },
    {
      "$ref": "$.cpaKey[6]"
    },
    {
      "$ref": "$.cpaKey[7]"
    }
 ]

后来将两个list的合并方式改成对象新对象拷贝旧对象信息,将新对象存入列表的方式,就两个都正常了:

        for(HotKeys key : hotKeyList){
            HotKeys k = new HotKeys();
            BeanUtils.copyProperties(key, k);
            cpaKeys.add(k);
        }

这是什么问题?对象引用的问题?

从上面的运行结果可以看出是hotKeyList里面的对象引用发生变化,转而指向cpaKeys中对应的数据,经过addAll()方法处理后hotKeyList只保存了每个对象的引用信息(或者说指针)。hotKeyList并没有另外生成一份数据,看看源码:

public boolean addAll(Collection<? extends E> c) {
        Object[] a = c.toArray();
        int numNew = a.length;
        ensureCapacityInternal(size + numNew);  // Increments modCount
        System.arraycopy(a, 0, elementData, size, numNew);
        size += numNew;
        return numNew != 0;
    }

其中方法System.arraycopy(a, 0, elementData, size, numNew);的描述如下:

Copies an array from the specified source array, beginning at the specified position, to the specified position of the destination array. A subsequence of array components are copied from the source array referenced by src to the destination array referenced by dest. The number of components copied is equal to the length argument. The components at positions srcPos through srcPos+length-1 in the source array are copied into positions destPos through destPos+length-1, respectively, of the destination array.

If the src and dest arguments refer to the same array object, then the copying is performed as if the components at positions srcPos through srcPos+length-1 were first copied to a temporary array with length components and then the contents of the temporary array were copied into positions destPos through destPos+length-1 of the destination array.

第一段话说到”数组组件的子序列从src引用的源数组复制到dest引用的目标数组。”注意“引用的源数组”、“引用的目标数组”的意思。

而通过SpringMVC的@ResponseBody返回JSON的时候,FastJSON并没有将引用转成数据,为了避免日后还会产生这样的问题,最终的解决方案是将FastJSON换为Jackson。配置文件修改如下:

pom.xml 新增jackson的依赖:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.4.2</version>
</dependency>

spring-mvc.xml修改如下:

<!--避免IE执行AJAX时,返回JSON出现下载文件.原来的(fastjson)class="com.alibaba.fastjson.support.spring.FastJsonHttpMessageConverter"-->
    <bean id="mappingJacksonHttpMessageConverter"
        class="org.springframework.http.converter.json.MappingJackson2HttpMessageConverter">
        <property name="supportedMediaTypes">
            <list>
                <value>text/html;charset=UTF-8</value>
                <value>application/json</value>
            </list>
        </property>
        <!-- 使用jackson不需要这个
        <property name="features">
            <list>
                <!// <value>WriteMapNullValue</value> //>
                <value>QuoteFieldNames</value>
            </list>
        </property> -->
    </bean>

    <!-- 修改新增 -->
    <bean id="stringHttpMessageConverter" class="org.springframework.http.converter.StringHttpMessageConverter">
        <property name="supportedMediaTypes">
            <list>
                <value>text/plain;charset=UTF-8</value>
            </list>
        </property>
    </bean>
    <!-- 修改新增 -->
    <bean id="formHttpMessageConverter" class="org.springframework.http.converter.FormHttpMessageConverter" />

    <!-- 启动SpringMVC的注解功能,完成请求和注解POJO的映射 -->
    <bean
        class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
        <property name="messageConverters">
            <list>
                <ref bean="formHttpMessageConverter" />     <!-- 修改新增 -->
                <ref bean="stringHttpMessageConverter" />   <!-- 修改新增 -->
                <ref bean="mappingJacksonHttpMessageConverter" />   <!-- JSON转换器 -->
            </list>
        </property>
    </bean>

还是使用最简单addAll()方法,结果正常。

这次疏忽导致的影响非常大,虽然问题藏得有点深,但关键还是没有去验证对原来数据影响导致,以此作为一次深刻经验教训。

另外,经过集思广益,总结出以上出现$ref的问题的根本是因为对象中存在重复的对象导致,如果仍然使用fastjson方法的话可以这么处理:

FastJson提供了SerializerFeature.DisableCircularReferenceDetect这个序列化选项,用来关闭引用检测。关闭引用检测后,重复引用对象时就不会被$ref代替,但是在循环引用时也会导致StackOverflowError异常。

用法:

 JSON.toJSONString(object, SerializerFeature.DisableCircularReferenceDetect);

得到的是object的json字符串。

我代码能力很差,你解释一下package com.foonsu.efenxiao.biz.handler; import cn.hutool.core.collection.CollectionUtil; import cn.hutool.core.date.DateTime; import cn.hutool.core.date.DateUtil; import cn.hutool.core.text.csv.CsvReadConfig; import cn.hutool.core.text.csv.CsvReader; import cn.hutool.core.text.csv.CsvRow; import cn.hutool.core.util.ObjectUtil; import com.alibaba.excel.EasyExcel; import com.alibaba.fastjson.JSON; import com.alibaba.fastjson.JSONArray; import com.baomidou.mybatisplus.extension.plugins.pagination.Page; import com.foonsu.efenxiao.biz.config.CidReadListener; import com.foonsu.efenxiao.biz.feign.OrderClientWrapper; import com.foonsu.efenxiao.biz.util.OrderUtil; import com.foonsu.efenxiao.order.dto.CidOrder; import com.foonsu.efenxiao.biz.vo.ImportCidAccountVO; import com.foonsu.efenxiao.biz.vo.ProgressVo; import com.foonsu.efenxiao.common.utils.CollectionsUtil; import com.foonsu.efenxiao.order.dto.OrderDTO; import com.foonsu.efenxiao.order.dto.OrderExtendDto; import com.foonsu.efenxiao.order.dto.OrderGoodsDto; import com.foonsu.efenxiao.order.dto.OrderTradeDto; import com.foonsu.efenxiao.order.dubbo.CidClient; import com.foonsu.efenxiao.order.dubbo.OrderClient; import com.foonsu.efenxiao.order.entity.CidAccount; import com.foonsu.efenxiao.order.entity.CidAccountingDetail; import com.foonsu.efenxiao.order.entity.CidPlan; import com.foonsu.efenxiao.order.entity.Order; import com.foonsu.efenxiao.order.vo.CidAccountSearchVO; import com.foonsu.efenxiao.order.vo.CidOrderConsumption; import com.foonsu.efenxiao.order.vo.CidPlanSearchResp; import com.foonsu.efenxiao.order.vo.CidPlanSearchVo; import com.foonsu.efenxiao.order.vo.req.AdvertiserAccount; import com.foonsu.efenxiao.order.vo.req.HistoryPushAccountReq; import com.foonsu.efenxiao.platform.dubbo.GoodsClient; import com.foonsu.efenxiao.platform.entity.CidGoodsPriceHistory; import com.foonsu.efenxiao.platform.entity.PlatformGoods; import com.foonsu.efenxiao.platform.vo.PlatformGoodsSkuVO; import com.foonsu.efenxiao.platform.vo.PlatformGoodsVO; import com.foonsu.efenxiao.search.dubbo.OrderSearchClient; import com.foonsu.efenxiao.search.vo.ordersearch.*; import com.foonsu.efenxiao.webutil.progress.impl.CidImportAccountProgress; import com.foonsu.efenxiao.webutil.progress.impl.CidImportStatementProgress; import com.foonsu.framework.boot.common.utils.MD5Utils; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang.StringUtils; import org.apache.dubbo.config.annotation.DubboReference; import org.springblade.core.tool.api.R; import org.springblade.core.tool.utils.BeanUtil; import org.springblade.core.tool.utils.StringUtil; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.beans.factory.annotation.Value; import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor; import org.springframework.stereotype.Component; import org.springframework.transaction.annotation.Transactional; import org.springframework.web.multipart.MultipartFile; import javax.annotation.Resource; import java.io.IOException; import java.io.InputStreamReader; import java.math.BigDecimal; import java.math.RoundingMode; import java.time.LocalDateTime; import java.time.ZoneId; import java.time.format.DateTimeFormatter; import java.util.*; import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.CountDownLatch; import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Function; import java.util.function.Supplier; import java.util.regex.Matcher; import java.util.regex.Pattern; import java.util.stream.Collectors; @Component @Slf4j public class CidShopBillHandler { private static final Pattern PATTERN = Pattern.compile("\\d{6}-\\d{15}"); @Resource @Qualifier("cidHandleThreadPoolTaskExecutor") ThreadPoolTaskExecutor cidHandleThreadPoolTaskExecutor; @DubboReference private OrderClient orderClient; @DubboReference private CidClient cidClient; @DubboReference private OrderSearchClient orderSearchClient; @DubboReference private GoodsClient goodsClient; @Resource private OrderClientWrapper orderClientWrapper; @Resource private CidImportStatementProgress cidImportStatementProgress; @Resource private CidImportAccountProgress cidImportAccountProgress; @Value("${cid.tenant-id}") private String tenantId; public static final DateTimeFormatter DATETIME_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"); public CidSearchOrderResp searchOrder(SearchOrderReq req, String tenantId) { SearchOrderBackendParam backendParam = new SearchOrderBackendParam(); backendParam.setTenantId(tenantId); EsSearchOrderResp resp = orderSearchClient.searchCidOrder(req, backendParam); CidSearchOrderResp cidSearchOrderResp = new CidSearchOrderResp(); if (!resp.getOrderNos().isEmpty()) { List<String> allOrderNos = resp.getOrderNos().stream().flatMap(Collection::stream).collect(Collectors.toList()); List<Order> ordersData = orderClientWrapper.queryByOrderNos(allOrderNos); List<CidOrderVO> cidOrderVOS = this.convertData(resp, ordersData); cidSearchOrderResp.setData(cidOrderVOS); } cidSearchOrderResp.setTotal(resp.getTotalCount().intValue()); cidSearchOrderResp.setCurrent(req.getPageNo()); cidSearchOrderResp.setSize(req.getPageSize()); cidSearchOrderResp.setPages(cidSearchOrderResp.getTotal() / req.getPageSize() + (cidSearchOrderResp.getTotal() % req.getPageSize() > 0 ? 1 : 0)); return cidSearchOrderResp; } private List<CidOrderVO> convertData(EsSearchOrderResp esSearchOrderResp, List<Order> orders) { List<CidOrderVO> cidOrderVOS = new ArrayList<>(); for (List<String> orderNos : esSearchOrderResp.getOrderNos()) { List<Order> onelineOrders = orders.stream().filter(t -> orderNos.contains(t.getOrderNo())).collect(Collectors.toList()); if (!onelineOrders.isEmpty()) { Map<String, List<Order>> tradeNoOrders = onelineOrders.stream().collect(Collectors.groupingBy(Order::getTradeNo)); for (String tradeNo : tradeNoOrders.keySet()) { List<Order> splitOrders = tradeNoOrders.get(tradeNo); List<OrderDTO> orderDTOS = OrderUtil.convertToDto(splitOrders); // cid默认单商品订单 OrderDTO orderDTO = orderDTOS.get(0); CidOrderVO cidOrderVO = new CidOrderVO(); cidOrderVO.setShopName(orderDTO.getShopName()); cidOrderVO.setJlId(orderDTO.getOrderGoodsDtoList().get(0).getBindJlId()); cidOrderVO.setTradeNo(orderDTO.getTradeNo()); cidOrderVO.setOrderNo(orderDTO.getOrderNo()); cidOrderVO.setPayAmount(orderDTO.getOrderTradeDto().getPayAmount()); CidOrderExtendVO cidOrderExtendVO = JSON.parseObject(orderDTO.getExtendInfo(), CidOrderExtendVO.class); cidOrderVO.setCidOrderExtendVO(cidOrderExtendVO); List<CidGoodsVo> goodsVos = JSON.parseArray(orderDTO.getGoods(), CidGoodsVo.class); cidOrderVO.setGoods(goodsVos); cidOrderVOS.add(cidOrderVO); } } } return cidOrderVOS; } public void compute(String tenantId, List<String> tradeNos) { Map<String, List<CidAccountingDetail>> accountDetailGroupByTradeNo = cidClient.queryAllImportCidTradeNo(tenantId, null, tradeNos) .stream().collect(Collectors.groupingBy(CidAccountingDetail::getTradeNo)); List<String> importCidTradeNos = new ArrayList<>(accountDetailGroupByTradeNo.keySet()); List<List<String>> reqsList = CollectionsUtil.splitListToThreads(importCidTradeNos, 1500, 20); for (List<String> orders : reqsList) { cidHandleThreadPoolTaskExecutor.submit(() -> { for (String order : orders) { this.handleOrder(order, tenantId, accountDetailGroupByTradeNo.get(order)); } }); } } public void compute(List<CidAccountingDetail> accountingDetails) { accountingDetails = accountingDetails.stream().sorted(Comparator.comparing(CidAccountingDetail::getTradeNo)).collect(Collectors.toList()); CollectionsUtil.batchDeal(accountingDetails, 500, accountingDetailList -> { List<CidOrder> cidOrders = new ArrayList<>(); Set<String> tradeNos = accountingDetailList.stream().map(CidAccountingDetail::getTradeNo).collect(Collectors.toSet()); Map<String, List<LocalDateTime>> occurTimeByTradeNos = cidClient.queryOccurentTimeByTradeNos(tradeNos); Map<String, List<CidAccountingDetail>> accountDetailGroupByTradeNo = accountingDetailList.stream().collect(Collectors.groupingBy(CidAccountingDetail::getTradeNo)); List<Order> orders = orderClient.queryOrderByTradeNos(new ArrayList<>(tradeNos)); if (CollectionsUtil.isEmpty(orders)) return; Map<String, Order> ordergroupbyTradeNo = orders.stream().collect(Collectors.toMap(Order::getTradeNo, Function.identity())); Map<String, String> ordersGroupByProductIdMap = orders.stream().collect(Collectors.toMap(Order::getTradeNo, order -> { List<OrderGoodsDto> orderGoodsDtos = JSONArray.parseArray(order.getGoods(), OrderGoodsDto.class); return orderGoodsDtos.get(0).getProductId(); })); List<String> skuIds = orders.stream().map(order -> { List<OrderGoodsDto> orderGoodsDtos = JSONArray.parseArray(order.getGoods(), OrderGoodsDto.class); return orderGoodsDtos.get(0).getSkuId(); }).distinct().collect(Collectors.toList()); Map<String, List<CidGoodsPriceHistory>> cidGoodsPriceGroupBySkuIdMap = goodsClient.selectCidGoodsPriceHistory(tenantId, null, skuIds) .stream().collect(Collectors.groupingBy(CidGoodsPriceHistory::getSkuId)); List<String> productIds = ordersGroupByProductIdMap.values().stream().distinct().collect(Collectors.toList()); Map<String, PlatformGoodsVO> goodsGroupByProductId = goodsClient.selectCidGoods(tenantId, productIds, null) .stream().collect(Collectors.toMap(PlatformGoodsVO::getProductId, Function.identity())); accountDetailGroupByTradeNo.forEach((tradeNo, accountDetails) -> { if (ordergroupbyTradeNo.containsKey(tradeNo)) { Order order = ordergroupbyTradeNo.get(tradeNo); String productId = ordersGroupByProductIdMap.get(tradeNo); if (goodsGroupByProductId.containsKey(productId)) { PlatformGoodsVO platformGoodsVO = goodsGroupByProductId.get(productId); OrderExtendDto orderExtendDto = computeGoodsProfit(platformGoodsVO, order, cidGoodsPriceGroupBySkuIdMap); cidOrders.addAll(covertCidOrders(accountDetails, orderExtendDto, tradeNo, platformGoodsVO.getBindJlId(), occurTimeByTradeNos.get(tradeNo))); } } }); if (cidOrders.isEmpty()) return; orderSearchClient.saveCidOrders(cidOrders); }, cidHandleThreadPoolTaskExecutor); } private CidOrder createEsCidOrder(String tenantId, String bindJlId, String tradeNO) { CidOrder esCidOrder = new CidOrder(); esCidOrder.setAccountId(bindJlId); esCidOrder.setTenantId(tenantId); esCidOrder.setTradeNo(tradeNO); return esCidOrder; } private List<CidOrder> covertCidOrders(List<CidAccountingDetail> accountDetails, OrderExtendDto orderExtendDto, String tradeNO, String bindJlId, List<LocalDateTime> occurTimes) { List<CidOrder> esCidOrders = accountDetails.stream().map(accountingDetail -> { CidOrder cidOrder = createEsCidOrder(tenantId, bindJlId, tradeNO); cidOrder.setId(accountingDetail.getUniqueId()); cidOrder.setOccurrenceTime(Collections.singletonList(LocalDateTime.parse(accountingDetail.getOccurrenceTime(), DATETIME_FORMATTER))); cidOrder.setRefundType(orderExtendDto.getRefundType()); BigDecimal incomeAmount = accountingDetail.getIncomeAmount() == null ? BigDecimal.ZERO : accountingDetail.getIncomeAmount(); BigDecimal expenditureAmount = accountingDetail.getExpenditureAmount() == null ? BigDecimal.ZERO : accountingDetail.getExpenditureAmount(); cidOrder.setOrderProfit(incomeAmount.add(expenditureAmount)); switch (accountingDetail.getAccountingType()) { case "扣款": cidOrder.setDeductionSum(accountingDetail.getExpenditureAmount()); switch (accountingDetail.getServiceType()) { case "0040002": cidOrder.setSalesCompensate(accountingDetail.getExpenditureAmount()); break; case "0040003": cidOrder.setExpressCompensate(accountingDetail.getExpenditureAmount()); break; case "0040004": cidOrder.setDelaySendGoods(accountingDetail.getExpenditureAmount()); break; case "0040005": cidOrder.setFalseSendGoods(accountingDetail.getExpenditureAmount()); break; case "0040006": cidOrder.setOutOfStock(accountingDetail.getExpenditureAmount()); } break; case "技术服务费": cidOrder.setTechnologyFee(accountingDetail.getExpenditureAmount()); break; case "多多进宝": cidOrder.setDdjb(accountingDetail.getExpenditureAmount()); } return cidOrder; }).collect(Collectors.toList()); CidOrder esCidOrder = createEsCidOrder(tenantId, bindJlId, tradeNO); esCidOrder.setOrderProfit(BigDecimal.ZERO.subtract(orderExtendDto.getPurchaseExpressFee())); esCidOrder.setId(tradeNO); esCidOrder.setOccurrenceTime(occurTimes.stream().distinct().collect(Collectors.toList())); esCidOrders.add(esCidOrder); return esCidOrders; } private OrderExtendDto computeGoodsProfit(PlatformGoodsVO platformGoods, Order order, Map<String, List<CidGoodsPriceHistory>> cidGoodsPriceGroupBySkuIdMap) { OrderTradeDto orderTradeDto = JSON.parseObject(order.getTrade(), OrderTradeDto.class); List<OrderGoodsDto> orderGoodsDtos = JSONArray.parseArray(order.getGoods(), OrderGoodsDto.class); OrderGoodsDto orderGoodsDto = orderGoodsDtos.get(0); OrderExtendDto orderExtendDto = new OrderExtendDto(); List<PlatformGoodsSkuVO> skuVOS = platformGoods.getGoodsSkuList().stream().filter(sku -> sku.getSkuId().equals(orderGoodsDto.getSkuId())).collect(Collectors.toList()); if (!skuVOS.isEmpty()) { PlatformGoodsSkuVO platformGoodsSkuVO = skuVOS.get(0); BigDecimal purchasePrice = platformGoodsSkuVO.getPurchaseUnitPrice(); BigDecimal expressFee = platformGoodsSkuVO.getExpressFee(); String skuId = platformGoodsSkuVO.getSkuId(); if (cidGoodsPriceGroupBySkuIdMap.containsKey(skuId)) { List<CidGoodsPriceHistory> cidGoodsPriceHistories = cidGoodsPriceGroupBySkuIdMap.get(skuId); for (CidGoodsPriceHistory cidGoodsPriceHistory : cidGoodsPriceHistories) { if (cidGoodsPriceHistory.getType() == 2 && DateUtil.toLocalDateTime(new Date(orderTradeDto.getSourceCreateDate())).isAfter(cidGoodsPriceHistory.getUpdateTime())) { purchasePrice = cidGoodsPriceHistory.getPrice(); } if (cidGoodsPriceHistory.getType() == 3 && DateUtil.toLocalDateTime(new Date(orderTradeDto.getSourceCreateDate())).isAfter(cidGoodsPriceHistory.getUpdateTime())) { expressFee = cidGoodsPriceHistory.getPrice(); } } } orderExtendDto.setPurchasePrice(purchasePrice == null ? BigDecimal.ZERO : purchasePrice); orderExtendDto.setExpressFee(expressFee == null ? BigDecimal.ZERO : expressFee); } if (orderGoodsDto.getRefundType() != null) { orderExtendDto.setPurchaseExpressFee(orderExtendDto.getExpressFee()); orderExtendDto.setRefundType(orderGoodsDto.getRefundType()); } else { if (orderExtendDto.getPurchasePrice() != null && orderExtendDto.getExpressFee() != null) { orderExtendDto.setPurchaseExpressFee(orderExtendDto.getPurchasePrice().add(orderExtendDto.getExpressFee())); } } return orderExtendDto; } private void handleOrder(String importCidTradeNo, String tenantId, List<CidAccountingDetail> cidAccountingDetails) { if (StringUtil.isBlank(importCidTradeNo)) { return; } List<Order> orders = orderClient.queryOrderByTradeNo(importCidTradeNo); if (orders.isEmpty()) { log.info("对账订单为空"); return; } // 默认都是单商品订单 Order order = orders.get(0); List<CidAccountingDetail> accountingDetails = cidClient.queryAccountingDetailByTradeNo(tenantId, importCidTradeNo); if (accountingDetails.isEmpty()) { log.info("对账明细为空"); return; } OrderTradeDto orderTradeDto = JSON.parseObject(order.getTrade(), OrderTradeDto.class); List<OrderGoodsDto> orderGoodsDtos = JSONArray.parseArray(order.getGoods(), OrderGoodsDto.class); OrderGoodsDto orderGoodsDto = orderGoodsDtos.get(0); List<PlatformGoodsVO> platformGoodsVOS = goodsClient.selectCidGoods(tenantId, null, null); if (platformGoodsVOS.isEmpty()) { return; } PlatformGoodsVO platformGoods = platformGoodsVOS.get(0); OrderExtendDto orderExtendDto = StringUtils.isBlank(order.getExtendInfo()) ? new OrderExtendDto() : JSON.parseObject(order.getExtendInfo(), OrderExtendDto.class); BigDecimal orderNum = BigDecimal.ZERO; BigDecimal deductionSum = BigDecimal.ZERO; BigDecimal technologyFee = BigDecimal.ZERO; BigDecimal ddjb = BigDecimal.ZERO; BigDecimal salesCompensate = BigDecimal.ZERO;// 售后补偿 BigDecimal expressCompensate = BigDecimal.ZERO;// 运费补偿 BigDecimal delaySendGoods = BigDecimal.ZERO;// 延迟发货 BigDecimal falseSendGoods = BigDecimal.ZERO;// 虚假发货 BigDecimal outOfStock = BigDecimal.ZERO;// 缺货 for (CidAccountingDetail accountingDetail : accountingDetails) { orderNum = orderNum.add(accountingDetail.getIncomeAmount()).add(accountingDetail.getExpenditureAmount()); boolean serviceFlag = false; switch (accountingDetail.getAccountingType()) { case "扣款": deductionSum = deductionSum.add(accountingDetail.getExpenditureAmount()); serviceFlag = true; break; case "技术服务费": technologyFee = technologyFee.add(accountingDetail.getExpenditureAmount()); break; case "多多进宝": ddjb = ddjb.add(accountingDetail.getExpenditureAmount()); } if (serviceFlag) { switch (accountingDetail.getServiceType()) { case "0040002": salesCompensate = salesCompensate.add(accountingDetail.getExpenditureAmount()); break; case "0040003": expressCompensate = expressCompensate.add(accountingDetail.getExpenditureAmount()); break; case "0040004": delaySendGoods = delaySendGoods.add(accountingDetail.getExpenditureAmount()); break; case "0040005": falseSendGoods = falseSendGoods.add(accountingDetail.getExpenditureAmount()); break; case "0040006": outOfStock = outOfStock.add(accountingDetail.getExpenditureAmount()); } } } orderExtendDto.setDeductionSum(deductionSum); orderExtendDto.setTechnologyFee(technologyFee); orderExtendDto.setDdjb(ddjb); orderExtendDto.setSalesCompensate(salesCompensate); orderExtendDto.setExpressCompensate(expressCompensate); orderExtendDto.setDelaySendGoods(delaySendGoods); orderExtendDto.setFalseSendGoods(falseSendGoods); orderExtendDto.setOutOfStock(outOfStock); orderExtendDto.setPurchasePrice(BigDecimal.ZERO); orderExtendDto.setExpressFee(BigDecimal.ZERO); List<PlatformGoodsSkuVO> skuVOS = platformGoods.getGoodsSkuList().stream().filter(sku -> sku.getSkuId().equals(orderGoodsDto.getSkuId())).collect(Collectors.toList()); if (!skuVOS.isEmpty()) { PlatformGoodsSkuVO platformGoodsSkuVO = skuVOS.get(0); BigDecimal purchasePrice = platformGoodsSkuVO.getPurchaseUnitPrice(); BigDecimal expressFee = platformGoodsSkuVO.getExpressFee(); List<CidGoodsPriceHistory> cidGoodsPriceHistories = goodsClient.selectCidGoodsPriceHistory(tenantId, null, null); if (!cidGoodsPriceHistories.isEmpty()) { for (CidGoodsPriceHistory cidGoodsPriceHistory : cidGoodsPriceHistories) { if (cidGoodsPriceHistory.getType() == 2 && DateUtil.toLocalDateTime(new Date(orderTradeDto.getSourceCreateDate())).isAfter(cidGoodsPriceHistory.getUpdateTime())) { purchasePrice = cidGoodsPriceHistory.getPrice(); } if (cidGoodsPriceHistory.getType() == 3 && DateUtil.toLocalDateTime(new Date(orderTradeDto.getSourceCreateDate())).isAfter(cidGoodsPriceHistory.getUpdateTime())) { expressFee = cidGoodsPriceHistory.getPrice(); } } } orderExtendDto.setPurchasePrice(purchasePrice == null ? BigDecimal.ZERO : purchasePrice); orderExtendDto.setExpressFee(expressFee == null ? BigDecimal.ZERO : expressFee); } if (orderGoodsDto.getRefundType() != null) { orderExtendDto.setPurchaseExpressFee(orderExtendDto.getExpressFee()); orderExtendDto.setRefundType(orderGoodsDto.getRefundType()); } else { if (orderExtendDto.getPurchasePrice() != null && orderExtendDto.getExpressFee() != null) { orderExtendDto.setPurchaseExpressFee(orderExtendDto.getPurchasePrice().add(orderExtendDto.getExpressFee())); } } orderExtendDto.setOrderSum(orderNum); orderExtendDto.setOrderProfit(orderExtendDto.getOrderSum().subtract(orderExtendDto.getPurchaseExpressFee())); orderGoodsDtos.get(0).setBindJlId(platformGoods.getBindJlId()); order.setGoods(JSON.toJSONString(orderGoodsDtos)); order.setExtendInfo(JSON.toJSONString(orderExtendDto)); orderClient.updateCidOrder(order); orderSearchClient.saveCidOrders(orderExtendDto, tenantId, accountingDetails, platformGoods.getBindJlId(), importCidTradeNo); } public void importFile(MultipartFile file, String tenantId, String taskId) { ProgressVo progressVo = new ProgressVo(); R<ProgressVo> result = R.data(progressVo); try { progressVo.setTotalCount(0); progressVo.setAllHasDone(false); cidImportStatementProgress.saveProgress(taskId, result); CsvReader csvReader = new CsvReader(new InputStreamReader(file.getInputStream(), "GBK"), new CsvReadConfig()); List<CsvRow> rows = csvReader.read().getRows(); int size = rows.size(); log.info("文件行数:{}", size); if (size == 0) { log.info("文件为空"); progressVo.setTotalCount(0); progressVo.setAllHasDone(true); cidImportStatementProgress.saveProgress(taskId, result); return; } this.handlerAccountingDetail(rows, tenantId, taskId); } catch (IOException e) { log.error("导入失败", e); throw new RuntimeException(e); } } private void handlerAccountingDetail(List<CsvRow> rows, String tenantId, String taskId) { LocalDateTime importFileTime = LocalDateTime.now(); log.info("执行处理:{}", rows.size()); AtomicInteger atomicInteger = new AtomicInteger(0); List<CidAccountingDetail> accountingDetails = new ArrayList<>(); try { for (CsvRow row : rows) { try { // log.info("row:{}", row); if (row.size() != 7) { continue; } if (StringUtil.isBlank(row.get(1))) { continue; } if ("商户订单号".equals(row.get(0))) { continue; } atomicInteger.getAndIncrement(); CidAccountingDetail accountingDetail = new CidAccountingDetail(); accountingDetail.setTenantId(tenantId); String tradeNo = row.get(0); if (StringUtil.isBlank(tradeNo)) { Matcher matcher = PATTERN.matcher(row.get(5)); if (matcher.find()) { tradeNo = matcher.group(); } else { continue; } } accountingDetail.setTradeNo(tradeNo); accountingDetail.setOccurrenceTime(row.get(1)); accountingDetail.setIncomeAmount(new BigDecimal(row.get(2))); accountingDetail.setExpenditureAmount(new BigDecimal(row.get(3))); accountingDetail.setAccountingType(row.get(4)); accountingDetail.setRemark(row.get(5)); String[] service = row.get(6).split("\\|"); accountingDetail.setServiceType(service[0]); accountingDetail.setServiceDesc(service[1]); accountingDetail.setCreateTime(importFileTime); String uniqueId = MD5Utils.encryptMD5(accountingDetail.getTradeNo() + accountingDetail.getOccurrenceTime() + accountingDetail.getAccountingType()); accountingDetail.setUniqueId(uniqueId); accountingDetails.add(accountingDetail); } catch (Exception e) { throw new RuntimeException(e); } } if (accountingDetails.isEmpty()) return; CountDownLatch countDownLatch = new CountDownLatch(accountingDetails.size() / 200 + 1); for (int j = 0; j < accountingDetails.size(); j += 200) { List<CidAccountingDetail> cidAccountList = accountingDetails.subList(j, Math.min(j + 200, accountingDetails.size())); cidHandleThreadPoolTaskExecutor.submit(() -> { cidClient.batchSaveAccountingDetail(cidAccountList); countDownLatch.countDown(); }); } countDownLatch.await(); compute(accountingDetails); } catch (Exception e) { throw new RuntimeException(e); } } public void importAccount(MultipartFile file, String tenantId, String taskId) { LocalDateTime importTime = LocalDateTime.now(); ProgressVo progressVo = new ProgressVo(); R<ProgressVo> result = R.data(progressVo); try { progressVo.setTotalCount(0); progressVo.setAllHasDone(false); cidImportAccountProgress.saveProgress(taskId, result); CidReadListener cidReadListener = new CidReadListener(); cidReadListener.setTenantId(tenantId); EasyExcel.read(file.getInputStream(), ImportCidAccountVO.class, cidReadListener).sheet().doRead(); int size = cidReadListener.getCachedDataList().size(); log.info("账户文件行数:{}", size); if (size == 0) { progressVo.setTotalCount(0); progressVo.setAllHasDone(true); cidImportAccountProgress.saveProgress(taskId, result); log.info("文件为空"); return; } int threadHandlerNum = size / 5 + (size % 5 > 0 ? 1 : 0); int fromIdx = 0; for (int i = 0; i < 5; i++) { List<CidAccount> cidAccounts = cidReadListener.getCachedDataList().subList(fromIdx, size - fromIdx >= threadHandlerNum ? fromIdx + threadHandlerNum : size); cidHandleThreadPoolTaskExecutor.submit(() -> { for (int j = 0; j < cidAccounts.size(); j += 100) { List<CidAccount> cidAccountList = cidAccounts.subList(j, Math.min(j + 100, cidAccounts.size())); cidClient.batchSaveCidAccount(cidAccountList); } log.info("处理账户完成:{}", cidAccounts.size()); return "任务完成"; }); fromIdx += cidAccounts.size(); } try { Thread.sleep(5000); } catch (InterruptedException e) { throw new RuntimeException(e); } log.info("处理账户完成, 调用生成报表:{}", importTime); this.generatePlan(tenantId, taskId, importTime, size); } catch (IOException e) { log.error("导入报表失败:{}", e.getMessage(), e); throw new RuntimeException(e); } } public void generatePlan(String tenantId, String taskId, LocalDateTime importTime, int insertCidAccountCount) { log.info("生成cid报表开始"); CidAccountSearchVO cidAccountSearchVO = new CidAccountSearchVO(); cidAccountSearchVO.setTenantId(tenantId); cidAccountSearchVO.setImportTime(importTime); Long size = 0L; for (int i = 1; i <= 10; i++) { size = cidClient.queryCidAccountCount(cidAccountSearchVO); log.info("计算cidAccount中,查询数据:{}", size); if (insertCidAccountCount == size) { break; } try { Thread.sleep(1000); } catch (InterruptedException e) { throw new RuntimeException(e); } } if (size > 0L) { log.info("查询到{}条数据", size); AtomicInteger num = new AtomicInteger(0); ProgressVo progressVo = new ProgressVo(); progressVo.setTotalCount(size.intValue()); R<ProgressVo> result = R.data(progressVo); progressVo.setSuccessCount(num.get()); cidImportAccountProgress.saveProgress(taskId, result); Long finalSize = size; int pages = 1; int pageNo = 1; do { cidAccountSearchVO.setPageNo(pageNo); Page<CidAccount> cidAccountPage = cidClient.queryCidAccountPage(cidAccountSearchVO); pages = (int) cidAccountPage.getPages(); cidHandleThreadPoolTaskExecutor.submit(() -> { log.info("处理数量:{}", cidAccountPage.getRecords().size()); this.handleGroupPlan(cidAccountPage.getRecords(), taskId, finalSize.intValue(), num); }); pageNo++; } while (pageNo <= pages); } log.info("生成cid报表完成"); } private void handleGroupPlan(List<CidAccount> cidAccounts, String taskId, int accountListSize, AtomicInteger num) { ProgressVo progressVo = new ProgressVo(); progressVo.setTotalCount(accountListSize); R<ProgressVo> result = R.data(progressVo); for (CidAccount cidAccount : cidAccounts) { List<PlatformGoods> platformGoodsList = goodsClient.queryGoodsByAccountId(cidAccount.getTenantId(), cidAccount.getAccountId()); this.handlePlan(() -> { CidPlan cidPlan = new CidPlan(); cidPlan.setTenantId(cidAccount.getTenantId()); cidPlan.setAccountName(cidAccount.getAccount()); cidPlan.setChannel("巨量引擎"); cidPlan.setJlId(cidAccount.getAccountId()); cidPlan.setTotalConsumption(cidAccount.getConsume()); return cidPlan; }, platformGoodsList, null); num.getAndIncrement(); progressVo.setSuccessCount(num.get()); cidImportAccountProgress.saveProgress(taskId, result); } } private void handlePlan(Supplier<CidPlan> cidPlanSupplier, List<PlatformGoods> platformGoodsList, CidPlanSearchVo cidPlanSearchVo) { CidPlan cidPlan = cidPlanSupplier.get(); CidPlan result = orderSearchClient.statisticsCidOrder(cidPlan, DateUtil.toLocalDateTime(cidPlanSearchVo.getStartTime()), DateUtil.toLocalDateTime(cidPlanSearchVo.getEndTime())); BeanUtil.copyProperties(result, cidPlan); if (CollectionUtil.isNotEmpty(platformGoodsList)) { log.info("计划:{},没有找到商品", cidPlan.getJlId()); PlatformGoods platformGoods = platformGoodsList.get(0); cidPlan.setSalePrice(platformGoods.getSaleUnitPrice()); cidPlan.setPurchasePrice(platformGoods.getPurchaseUnitPrice()); cidPlan.setExpressFee(platformGoods.getExpressFee()); } } private List<CidOrderVO> filterOrderByDate(CidPlanSearchVo cidPlanSearchVo, List<CidOrderVO> orders) { if (cidPlanSearchVo == null) return orders; List<String> tradeNos = cidClient.selectOrderByTradeNoAndTime( orders.stream().map(CidOrderVO::getTradeNo).collect(Collectors.toList()), cidPlanSearchVo); //orderS过滤掉不包含tradeNo的订单 return orders.stream().filter(cidOrderVO -> tradeNos.contains(cidOrderVO.getTradeNo())).collect(Collectors.toList()); } private void handleOrder(List<CidOrderVO> data, Map<String, BigDecimal> param, Map<String, BigDecimal> deductionDetail, Map<String, Integer> orderParam) { int count = (int) data.stream().filter(cidOrderVO -> cidOrderVO.getCidOrderExtendVO().getRefundType() != null).count(); orderParam.put("refundOrderNum", orderParam.get("refundOrderNum") + count); orderParam.put("orderNum", orderParam.get("orderNum") + data.size()); for (CidOrderVO cidOrderVO : data) { if (cidOrderVO.getCidOrderExtendVO().getDeductionSum() != null) { param.put("deductionSum", param.get("deductionSum").add(cidOrderVO.getCidOrderExtendVO().getDeductionSum())); } if (cidOrderVO.getCidOrderExtendVO().getTechnologyFee() != null) { param.put("technologyFee", param.get("technologyFee").add(cidOrderVO.getCidOrderExtendVO().getTechnologyFee())); } if (cidOrderVO.getCidOrderExtendVO().getDdjb() != null) { param.put("ddjb", param.get("ddjb").add(cidOrderVO.getCidOrderExtendVO().getDdjb())); } if (cidOrderVO.getCidOrderExtendVO().getOrderProfit() != null) { param.put("profit", param.get("profit").add(cidOrderVO.getCidOrderExtendVO().getOrderProfit())); } if (cidOrderVO.getCidOrderExtendVO().getSalesCompensate() != null) { deductionDetail.put("salesCompensate", deductionDetail.get("salesCompensate").add(cidOrderVO.getCidOrderExtendVO().getSalesCompensate())); } if (cidOrderVO.getCidOrderExtendVO().getExpressCompensate() != null) { deductionDetail.put("expressCompensate", deductionDetail.get("expressCompensate").add(cidOrderVO.getCidOrderExtendVO().getExpressCompensate())); } if (cidOrderVO.getCidOrderExtendVO().getDelaySendGoods() != null) { deductionDetail.put("delaySendGoods", deductionDetail.get("delaySendGoods").add(cidOrderVO.getCidOrderExtendVO().getDelaySendGoods())); } if (cidOrderVO.getCidOrderExtendVO().getFalseSendGoods() != null) { deductionDetail.put("falseSendGoods", deductionDetail.get("falseSendGoods").add(cidOrderVO.getCidOrderExtendVO().getFalseSendGoods())); } if (cidOrderVO.getCidOrderExtendVO().getOutOfStock() != null) { deductionDetail.put("outOfStock", deductionDetail.get("outOfStock").add(cidOrderVO.getCidOrderExtendVO().getOutOfStock())); } } } public CidPlanSearchResp searchPlan(CidPlanSearchVo cidPlanSearchVo) { CidPlanSearchResp cidPlanSearchResp = new CidPlanSearchResp(); Page<CidPlan> cidPlanPage = cidClient.queryCidPlanPage(cidPlanSearchVo); List<CidPlan> cidPlans = cidPlanPage.getRecords(); if (ObjectUtil.isAllNotEmpty(cidPlanSearchVo.getStartTime(), cidPlanSearchVo.getEndTime())) { realTimeCompute(cidPlans, cidPlanSearchVo); } BigDecimal currentProfit = cidPlans.stream().map(CidPlan::getProfit).reduce(BigDecimal.ZERO, BigDecimal::add).setScale(2, RoundingMode.HALF_UP); cidPlanSearchResp.setCurrentProfit(currentProfit); cidPlanSearchResp.setData(cidPlanPage); CidPlan result = orderSearchClient.statisticsCidOrder(new CidPlan(), DateUtil.toLocalDateTime(cidPlanSearchVo.getStartTime()), DateUtil.toLocalDateTime(cidPlanSearchVo.getEndTime())); cidPlanSearchResp.setTotalProfit(result.getProfit()); return cidPlanSearchResp; } private void realTimeCompute(List<CidPlan> cidPlans, CidPlanSearchVo cidPlanSearchVo) { if (CollectionsUtil.isEmpty(cidPlans)) return; computeCidPlan(cidPlans, cidPlanSearchVo, cidPlans.stream().map(CidPlan::getJlId).collect(Collectors.toList())); } private void computeCidPlan(List<CidPlan> cidPlans, CidPlanSearchVo cidPlanSearchVo, List<String> accountIds) { Map<String, List<PlatformGoods>> goodsMapGroupByBindJlId = goodsClient.queryGoodsByAccountIds(cidPlanSearchVo.getTenantId(), accountIds) .stream().collect(Collectors.groupingBy(PlatformGoods::getBindJlId)); CountDownLatch countDownLatch = new CountDownLatch(cidPlans.size()); cidPlans.forEach(cidPlan -> cidHandleThreadPoolTaskExecutor.execute(() -> { this.handlePlan(() -> cidPlan, goodsMapGroupByBindJlId.get(cidPlan.getJlId()), cidPlanSearchVo); countDownLatch.countDown(); })); try { countDownLatch.await(); } catch (InterruptedException e) { log.error("realTimeCompute error: {}", e.getMessage()); } } @Transactional(rollbackFor = Exception.class) public Boolean pushAccount(List<AdvertiserAccount> data) { List<String> accountIds = data.stream().map(AdvertiserAccount::getAdvertiserId).collect(Collectors.toList()); Map<String, List<PlatformGoods>> goodsMapGroupByBindJlId = goodsClient.queryGoodsByAccountIds(tenantId, accountIds) .stream().collect(Collectors.groupingBy(PlatformGoods::getBindJlId)); Map<String, CidPlan> cidPlanMap = cidClient.queryCidPlans(tenantId, accountIds).stream().collect(Collectors.toMap(CidPlan::getJlId, Function.identity())); CopyOnWriteArrayList<CidPlan> cidPlansList = new CopyOnWriteArrayList<>(); CountDownLatch count = new CountDownLatch(accountIds.size()); List<CidAccount> cidAccounts = data.stream().map(account -> { CidAccount cidAccount = account.mapToCidAccount(new HashMap<>(), tenantId); cidHandleThreadPoolTaskExecutor.execute(() -> { String accountId = cidAccount.getAccountId(); this.handlePlan(() -> dealHistoryPlan(cidAccount, cidPlanMap, cidPlansList), goodsMapGroupByBindJlId.get(accountId), null); count.countDown(); }); return cidAccount; }).collect(Collectors.toList()); cidClient.batchSaveCidPlan(cidPlansList); orderSearchClient.saveCidConsumption(convertEsCidOrderConsumption(data, new Date())); return cidClient.batchSaveCidAccount(cidAccounts); } private CidPlan dealHistoryPlan(CidAccount cidAccount, Map<String, CidPlan> cidPlanMap, CopyOnWriteArrayList<CidPlan> cidPlansList) { String accountId = cidAccount.getAccountId(); List<CidPlan.HistoryTotalConsumption> historyTotalConsumptions = cidPlanMap.containsKey(accountId) && CollectionsUtil.isNotEmpty(cidPlanMap.get(accountId).getHistoryTotalConsumptions()) ? cidPlanMap.get(accountId).getHistoryTotalConsumptions() : new ArrayList<>(); String today = DateUtil.formatDate(new Date()); BigDecimal consume = cidAccount.getConsume(); CidPlan.HistoryTotalConsumption todayData = new CidPlan.HistoryTotalConsumption(DateUtil.parseDate(today), consume); if (historyTotalConsumptions.isEmpty() || cidPlanMap.get(accountId).getHistoryTotalConsumptions().stream() .noneMatch(historyTotalConsumption -> today.equals(DateUtil.formatDate(historyTotalConsumption.getDate())))) { historyTotalConsumptions.add(todayData); } CidPlan cidPlan = new CidPlan(); cidPlan.setTenantId(cidAccount.getTenantId()); cidPlan.setAccountName(cidAccount.getAccount()); cidPlan.setChannel("巨量引擎"); cidPlan.setJlId(accountId); cidPlan.setTotalConsumption(consume); cidPlan.setHistoryTotalConsumption(JSON.toJSONString(historyTotalConsumptions)); cidPlansList.add(cidPlan); return cidPlan; } @Transactional(rollbackFor = Exception.class) public void pushHistoryData(HistoryPushAccountReq historyPushAccountReq) { String day = historyPushAccountReq.getDate(); DateTime time = DateUtil.parseDate(day); List<AdvertiserAccount> advertiserAccounts = historyPushAccountReq.getData(); List<String> accountIds = advertiserAccounts.stream().map(AdvertiserAccount::getAdvertiserId).collect(Collectors.toList()); Map<String, BigDecimal> historyDataMap = advertiserAccounts.stream().collect(Collectors.toMap(AdvertiserAccount::getAdvertiserId, AdvertiserAccount::getStatCost)); List<CidPlan> cidPlans = cidClient.queryCidPlans(tenantId, accountIds); cidPlans.forEach(cidPlan -> { BigDecimal consume = historyDataMap.get(cidPlan.getJlId()); CidPlan.HistoryTotalConsumption todayData = new CidPlan.HistoryTotalConsumption(time, consume); List<CidPlan.HistoryTotalConsumption> historyTotalConsumptions = cidPlan.getHistoryTotalConsumptions(); if (CollectionsUtil.isNotEmpty(historyTotalConsumptions)) { if (historyTotalConsumptions.stream() .noneMatch(consumption -> day.equals(DateUtil.formatDate(consumption.getDate())))) { historyTotalConsumptions.add(todayData); historyTotalConsumptions.sort(Comparator.comparing(CidPlan.HistoryTotalConsumption::getDate)); cidPlan.setHistoryTotalConsumption(JSON.toJSONString(historyTotalConsumptions)); } } else { cidPlan.setHistoryTotalConsumption(JSON.toJSONString(Collections.singleton(todayData))); } }); if (!cidPlans.isEmpty()) { cidClient.batchSaveCidPlan(cidPlans); orderSearchClient.saveCidConsumption(convertEsCidOrderConsumption(advertiserAccounts, time)); } } private List<CidOrderConsumption> convertEsCidOrderConsumption(List<AdvertiserAccount> advertiserAccounts, Date time) { return advertiserAccounts.stream().map(advertiserAccount -> new CidOrderConsumption(advertiserAccount, time.toInstant() .atZone(ZoneId.systemDefault()) .toLocalDate())).collect(Collectors.toList()); } public void syncData(String accountId) { CollectionsUtil.batchDealWholeData(pageNo -> { Page<CidPlan> cidPlanPage = cidClient.queryCidPlanPage(new CidPlanSearchVo(pageNo, 1000, tenantId, accountId)); List<CidPlan> cidPlans = cidPlanPage.getRecords(); log.info("cidPlans:{}", JSON.toJSONString(cidPlans)); List<CidOrderConsumption> cidOrderConsumptions = cidPlans.stream().flatMap(cidPlan -> cidPlan.getHistoryTotalConsumptions().stream() .filter(historyTotalConsumption -> BigDecimal.ZERO.compareTo(historyTotalConsumption.getTotalConsumption()) < 0) .map(historyTotalConsumption -> new CidOrderConsumption(cidPlan.getJlId(), historyTotalConsumption)) ).collect(Collectors.toList()); log.info("cidOrderConsumptions:{}", JSON.toJSONString(cidOrderConsumptions)); if (CollectionUtil.isNotEmpty(cidOrderConsumptions)) { orderSearchClient.saveCidConsumption(cidOrderConsumptions); } return cidPlans; }); } } 并且告诉我每一行为什么这样写
最新发布
07-01
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <!-- 继承父项目 --> <parent> <groupId>com.xixi</groupId> <artifactId>dataBase</artifactId> <version>0.0.1-SNAPSHOT</version> <relativePath>../pom.xml</relativePath> <!-- 父项目 pom 相对路径 --> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>flink-job</artifactId> <!-- 子模块 artifactId --> <name>flink-job</name> <description>Flink Job Module</description> <!-- 子模块依赖(继承父项目版本) --> <dependencies> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-base</artifactId> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients</artifactId> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> </dependency> <dependency> <groupId>com.alibaba.fastjson2</groupId> <artifactId>fastjson2</artifactId> </dependency> <dependency> <groupId>com.alibaba.fastjson2</groupId> <artifactId>fastjson2-extension</artifactId> </dependency> <dependency> <groupId>com.alibaba.fastjson2</groupId> <artifactId>fastjson2-extension-spring6</artifactId> </dependency> <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> </dependency> <dependency> <groupId>com.xixi</groupId> <artifactId>common</artifactId> <!-- 与 common 模块的 artifactId 一致 --> </dependency> </dependencies> <build> <plugins> <!-- 编译插件(继承父项目配置) --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>17</source> <target>17</target> </configuration> </plugin> <!-- 可选:Spring Boot 插件(若需要本地运行 Flink 作业) --> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <excludes> <exclude> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </exclude> </excludes> </configuration> </plugin> </plugins> </build> </project><?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>3.5.0</version> <relativePath/> <!-- lookup parent from repository --> </parent> <packaging>pom</packaging> <modules> <!-- 子模块名称,对应目录名(如 flink-job) --> <module>flink-job</module> <module>common</module> <!-- 若有其他子模块(如 canal、service),也需在此声明 --> </modules> <groupId>com.xixi</groupId> <artifactId>dataBase</artifactId> <version>0.0.1-SNAPSHOT</version> <name>dataBase</name> <description>dataBase</description> <properties> <java.version>17</java.version> <flink.version>2.0.0</flink.version> <fastjson.version>2.0.49</fastjson.version> </properties> <dependencyManagement> <dependencies> <!-- Flink --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-base</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>4.0.0-2.0</version> </dependency> <!-- JSON 解析器和生成器 --> <dependency> <groupId>com.alibaba.fastjson2</groupId> <artifactId>fastjson2</artifactId> <version>${fastjson.version}</version> </dependency> <dependency> <groupId>com.alibaba.fastjson2</groupId> <artifactId>fastjson2-extension</artifactId> <version>${fastjson.version}</version> </dependency> <dependency> <groupId>com.alibaba.fastjson2</groupId> <artifactId>fastjson2-extension-spring6</artifactId> <version>${fastjson.version}</version> </dependency> <!-- kafka --> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>3.9.1</version> </dependency> <!-- mysql --> <dependency> <groupId>com.mysql</groupId> <artifactId>mysql-connector-j</artifactId> <version>8.2.0</version> <scope>runtime</scope> </dependency> <!-- MyBatis-Plus 核心 --> <dependency> <groupId>com.baomidou</groupId> <artifactId>mybatis-plus-spring-boot3-starter</artifactId> <version>3.5.11</version> </dependency> <!-- Spring Kafka --> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency> <!-- Lombok(简化实体类) --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.26</version> <optional>true</optional> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> <version>2.16.0</version> <!-- 使用最新版本,但注意与Flink版本的兼容性 --> </dependency> </dependencies> </dependencyManagement> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>${java.version}</source> <target>${java.version}</target> </configuration> </plugin> </plugins> </build> <repositories> <repository> <id>central</id> <url>https://repo.maven.apache.org/maven2</url> </repository> </repositories> </project> 我的pom有什么问题嘛,为什么我的flinkjob读不上依赖呢
06-14
<think>我们正在解决一个关于FlinkJob模块依赖配置的问题,特别是pom文件中的配置问题导致无法正确读取依赖。根据提供的引用,特别是引用[5]中提到了一个类似问题:Scala类没有被打进jar包,原因是maven默认不打包Scala代码,需要添加scala-maven-plugin插件。用户的问题:检查pom文件配置,解决FlinkJob模块无法正确读取依赖的问题。我们需要分析pom文件中可能存在的依赖配置问题,并提供解决方案。常见问题包括:1.依赖项未正确声明或版本冲突。2.构建插件配置不正确,导致打包时缺失必要的类(如Scala类)。3.依赖作用域(scope)设置不当。4.多模块项目中模块依赖未正确配置。考虑到引用[5]中提到的错误(jar包中缺少Scala类),我们应重点检查打包插件配置。解决方案步骤:1.确保所有必需的依赖项都在pom文件的<dependencies>部分正确声明,并注意作用域(scope)。2.如果使用了Scala,确保使用了合适的scala-maven-plugin插件来编译和打包Scala代码。3.检查是否配置了maven-assembly-plugin或maven-shade-plugin来生成包含依赖的fatjar(如果项目需要)。4.检查Flink相关依赖的版本是否一致。下面提供一个示例pom文件配置,重点展示如何配置scala-maven-plugin和maven-assembly-plugin(或maven-shade-plugin)来构建包含所有依赖的jar包。由于用户问题中提到了“FlinkJob模块”,我们假设这是一个多模块项目,所以还要检查模块之间的依赖关系。下面是一个可能的pom文件(针对FlinkJob模块)的配置示例,注意以下关键部分:步骤1:添加Scala语言支持(如果使用Scala)步骤2:添加必要的依赖(Flink核心依赖等)步骤3:配置打包插件(如scala-maven-plugin和maven-shade-plugin)假设我们使用maven-shade-plugin来创建包含所有依赖的fatjar。</think> 根据您描述的Flink Job模块依赖配置问题,结合引用信息分析,以下是排查步骤和解决方案: ### 一、常见POM配置问题分析 1. **Scala依赖打包缺失**(引用[5]问题) Maven默认不打包Scala类,需显式配置插件: ```xml <build> <plugins> <!-- Scala编译插件 --> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>3.2.0</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> </execution> </executions> </plugin> </plugins> </build> ``` 2. **Hadoop类路径缺失**(引用[4]问题) 缺少Hadoop依赖会导致文件系统访问异常: ```xml <dependencies> <!-- 必需Hadoop依赖 --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>${hadoop.version}</version> <!-- 需与Flink版本匹配 --> <scope>provided</scope> </dependency> </dependencies> ``` 3. **Flink核心依赖范围错误** 避免将核心依赖设为`provided`导致运行时缺失: ```xml <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> <version>${flink.version}</version> <!-- 移除scope或设为compile --> </dependency> ``` ### 二、推荐解决方案 1. **完整POM配置示例**: ```xml <project> <properties> <flink.version>1.15.2</flink.version> <scala.version>2.12</scala.version> </properties> <dependencies> <!-- Flink核心依赖 --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> <version>${flink.version}</version> </dependency> <!-- Hadoop集成 --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-hadoop-compatibility</artifactId> <version>${flink.version}</version> </dependency> </dependencies> <build> <plugins> <!-- Scala编译 --> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>3.2.0</version> <executions> <execution> <goals><goal>compile</goal></goals> </execution> </executions> </plugin> <!-- 生成包含依赖的Fat JAR --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.3.0</version> <executions> <execution> <phase>package</phase> <goals><goal>shade</goal></goals> <configuration> <transformers> <transformer implementation="..." /> </transformers> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> ``` ### 三、验证方法 1. **检查生成的JAR包**: ```bash # 解压JAR查看类文件 jar tvf target/your-job.jar | grep YourMainClass ``` 2. **本地测试依赖完整性**: ```java // 添加测试代码 public static void main(String[] args) { FileSystem.get(new Configuration()); // 验证HDFS依赖 } ``` ### 四、延伸问题解决 - **依赖冲突**:用`mvn dependency:tree`排查版本冲突 - **类加载问题**:在`flink-conf.yaml`中添加`classloader: parent-first`(引用[2]YARN配置相关)[^1]
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值