整理收集几种多线程分段处理list集合的方法

原创于 2021-06-25 16:17:41 发布 · 1.8k 阅读

7 ·

CC 4.0 BY-SA版权

文章标签：

#多进程 #多线程

多线程专栏收录该内容

10 篇文章

订阅专栏

本文介绍Java中利用线程池处理大量数据的四种方法，包括分段处理List集合、多线程并行处理及结合Excel数据导入的场景优化。通过不同案例分析，展示如何有效提升程序运行效率。

方法一

来源：https://blog.youkuaiyun.com/jenny8080/article/details/

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
 
public class App {
	public static final String module = App.class.getName();
 
	public static void main(String[] args) {
 
		try {
			List<String> list = new ArrayList<>();
			for (int i = 0; i < 100; i++) {
				list.add(i + ",");
			}
 
			System.out.println(new App().list2Str(list, 5));
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
	public String list2Str(List<String> list, final int nThreads)
			throws Exception {
		if (list == null || list.isEmpty()) {
			return null;
		}
		StringBuffer ret = new StringBuffer();
 
		int size = list.size();
		ExecutorService executorService = Executors.newFixedThreadPool(nThreads);
		List<Future<String>> futures = new ArrayList<Future<String>>(nThreads);
		for (int i = 0; i < nThreads; i++) {
			final List<String> subList = list.subList(size / nThreads * i, size/ nThreads * (i + 1));
			Callable<String> task = new Callable<String>() {
				@Override
				public String call() throws Exception {
					StringBuffer sb = new StringBuffer();
					for (String str : subList) {
						sb.append(str);
					}
					return sb.toString();
				}
			};
			futures.add(executorService.submit(task));
		}
 
		for (Future<String> future : futures) {
			ret.append(future.get());
		}
		executorService.shutdown();
 
		return ret.toString();
	}<span id="transmark"></span>
 
}

方法二

来源：https://blog.youkuaiyun.com/sinat_38364990/article/details/81170003

ava用线程池处理集合问题

循环集合每多少条数据开启一个集合，此处每十万数据开启一个线程

public void testStr() {
        List<BaseEntity> list = new ArrayList<>();
        for (int i = 0; i < 2000000; i++) {
            BaseEntity entity = new BaseEntity();
            entity.setId("这是**一个**测试" + i);
            list.add(entity);
        }

        long start = System.currentTimeMillis();
        check = new ChineseCheck();
        ExecutorService executor = Executors.newFixedThreadPool(5);
        int size = list.size();
        if (size > 100000) {
            int batch = size % 100000 == 0 ? size / 100000 : size / 100000 + 1;
            for (int j=0; j<batch; j++) {
                int end = (j+1)*100000;
                if (end > size) {
                    end = size;
                }
                List<BaseEntity> subList = list.subList(j*100000, end);
                TestCallable callable = new TestCallable(subList, check);
                executor.execute(callable);
            }
        }
        executor.shutdown();

        while (true) {
            if (executor.isTerminated()) {
                break;
            }
        }
        long date = System.currentTimeMillis() - start;
        System.out.println("======" + date + "======");
    }

用时：1361 ms 

两百万条数据做校验，每一条数据开一个线程

List<BaseEntity> list = new ArrayList<>();
        for (int i = 0; i < 2000000; i++) {
            BaseEntity entity = new BaseEntity();
            entity.setId("这是**一个**测试" + i);
            list.add(entity);
        }

        long start = System.currentTimeMillis();
        check = new ChineseCheck();
        ExecutorService executor = Executors.newFixedThreadPool(5);
        for (BaseEntity entity: list) {
            TestCallable callable = new TestCallable(entity, check);
            executor.execute(callable);
        }
        executor.shutdown();

        while (true) {
            if (executor.isTerminated()) {
                break;
            }
        }
        long date = System.currentTimeMillis() - start;
        System.out.println("======" + date + "======");

用时：29875 ms
以上两种情况对比，当使用线程池开启多线程的时候，每一个线程中校验多条数据，此时效率会高
可以按照这批数据的处理次数来创建线程池，规定线程池最大线程数，然后不大于这个线程数的时候可以按照处理次数来创建线程
此处，多个线程同用一个单例处理数据和多线程用不同的实例对象处理数据效果相同
即，此处循环list时，是否每次都new一个check

方法三：结合导入

来源：https://blog.youkuaiyun.com/qq_19734597/article/details/85037038

获取导入excel文件中的数据，存入list中进行多线程处理，设定每个线程处理多少条数据，进行设定值动态启动线程数量数据，根据线程处理
数量拆分动态list数据进行相关业务操作
package com.ig.bi.game.controller;
 
import org.apache.log4j.Logger;
import org.apache.poi.hssf.usermodel.HSSFRow;
import org.apache.poi.hssf.usermodel.HSSFSheet;
import org.apache.poi.hssf.usermodel.HSSFWorkbook;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
import javax.servlet.http.HttpServletRequest;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.ThreadPoolExecutor;
 
/**
 * Created by admin 
 */
@RestController
@RequestMapping("/importData")
public class HpImportController extends BaseController{
    private static final Logger logger = Logger.getLogger(HpImportController.class);
 
    @Reference(interfaceClass = SidCustManageService.class,version = Constants.SERVICE_VERSION)
    private SidCustManageService sidCustManageService;
 
 
 
    @RequestMapping(value = "importDate")
    @ResponseBody
    public Response getOrdertDate(HttpServletRequest request,MultipartFile importData){
 
 
        //创建Excel工作薄
        try {
            HSSFWorkbook hwb = new HSSFWorkbook(importData.getInputStream());
            //得到第一个工作表
            HSSFSheet sheet = hwb.getSheetAt(0);
            HSSFRow row = null;
 
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
            List<SidCustPersonContactInfo> list = new ArrayList<SidCustPersonContactInfo>();
 
            //遍历该表格中所有的工作表，i表示工作表的数量 getNumberOfSheets表示工作表的总数
            for(int i = 0; i < hwb.getNumberOfSheets(); i++) {
                sheet = hwb.getSheetAt(i);
 
                //遍历该行所有的行,j表示行数 getPhysicalNumberOfRows行的总数
                for(int j = 1; j < sheet.getPhysicalNumberOfRows(); j++) {
                    row = sheet.getRow(j);
                    String loginName = ExcelUtils.getStringValue(row, 0);
                    String agentName = ExcelUtils.getStringValue(row, 1);
                    String pwd = "123123";
                    String  amount = ExcelUtils.getStringValue(row, 3);
                    String cust_lev = ExcelUtils.getStringValue(row,4);
                    String name = ExcelUtils.getStringValue(row, 5);
                    String tel = ExcelUtils.getStringValue(row, 6);
                    String qq = ExcelUtils.getStringValue(row, 7);
                    String birday = ExcelUtils.getStringValue(row, 8);
                    String email = ExcelUtils.getStringValue(row, 9);
                    String createTp = ExcelUtils.getStringValue(row, 10);
                    String bankAccountName = ExcelUtils.getStringValue(row, 11);
                    String bankNo = ExcelUtils.getStringValue(row, 12);
                    String bankName = ExcelUtils.getStringValue(row, 13);
                    String bankType = ExcelUtils.getStringValue(row, 14);
 
 
                    SidCustPersonContactInfo custCpci = new SidCustPersonContactInfo();
                    custCpci.setCustCode(loginName);
                    custCpci.setAmount(Double.valueOf(amount));
                    custCpci.setPasswd(pwd);
                    custCpci.setBankName(bankName);
                    custCpci.setBankNo(bankNo);
                    custCpci.setBankAccountName(bankAccountName);
                    if(!birday.equals("")){
                        custCpci.setBirthDate(sdf.parse(birday));
                    }
                    custCpci.setFirstName(name);
                    custCpci.setNickName(name);
                    custCpci.setQq(qq);
                    custCpci.setBankType(bankType);
                    custCpci.setEmailAddress(email);
                    custCpci.setMobileNumber(tel);
                    custCpci.setHallId(Long.valueOf(5));
                    list.add(custCpci);
 
                }
            }
 
 
            int count = 50;
            int listSize = list.size();
            int RunSize = (listSize / count)+1;
 
            ThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(RunSize);
            CountDownLatch countDownLatch = new CountDownLatch(RunSize);
            List<SidCustPersonContactInfo> newList = null ;
            for (int i = 0; i < RunSize; i++) {
                if((i+1)==RunSize){
                    int startIndex = (i*count);;
                    int endIndex = list.size();
                    newList =list.subList(startIndex,endIndex);
                }else{
                    int startIndex = i*count;;
                    int endIndex = (i+1)*count;
                    newList =list.subList(startIndex,endIndex);
                }
 
                ExecupteHp hpRunnable = new ExecupteHp(newList);
                executor.execute(hpRunnable);
            }
 
            countDownLatch.await();
            executor.shutdown();
 
 
        }catch (Exception e){
        logger.info("导入数据开始.");
        e.printStackTrace();
    }
        return Response.ok(GlobalEnum.GLOBAL_ENUM_SUCCESS_CODE.getValue() , SidCustEnum.SID_CRST_ERROR_SID_SUCCESS_07.getValue());
 
    }
 
 
 
    class ExecupteHp implements Runnable{
        private List <SidCustPersonContactInfo> list;
        public ExecupteHp (List <SidCustPersonContactInfo> list){
            this.list  = list ;
        }
 
        @Override
        public void run() {
            if(null!=list){
                for (SidCustPersonContactInfo scpc : list){
                       System.out.print("更新会员联系方式" +scpc.getFirstName() + "---"+scpc.getCustCode() );
                    SidCustContactInfo custContactInfo  = new SidCustContactInfo();
                    custContactInfo.setCustCode(scpc.getCustCode());
                    custContactInfo.setEmailAddress(scpc.getEmailAddress());
                    custContactInfo.setMobileNumber(scpc.getMobileNumber());
                    custContactInfo.setFirstName(scpc.getFirstName());
                    custContactInfo.setNicName(scpc.getNickName());
                    custContactInfo.setQq(scpc.getQq());
                    custContactInfoService.updateByContactByCustCode(custContactInfo);
 
                }
            }
 
        }
    }
 
   
 
}

方法四：

来源：https://blog.youkuaiyun.com/timmyroy/article/details/84920854

解决方案：
1.List集合分段

2.动态创建线程池newFixedThreadPool

3.多线程处理数据

 

代码如下：

 public static void main(String[] args) throws Exception {

// 开始时间

long start = System.currentTimeMillis();

// 模拟数据

List List<String> list = new ArrayList<String>();

for (int i = 1; i <= 3000; i++){

list.add(i + "");

}

// 每500条数据开启一条线程

int threadSize = 500;

// 总数据条数

int dataSize = list.size();

// 线程数

int threadNum = dataSize / threadSize + 1;

// 定义标记,过滤threadNum为整数

boolean special = dataSize % threadSize == 0;

// 创建一个线程池

ExecutorService exec = Executors.newFixedThreadPool(threadNum);

// 定义一个任务集合

List<Callable<Integer>> tasks = new ArrayList<Callable<Integer>>();

Callable<Integer> task = null;

List<String> cutList = null;

// 确定每条线程的数据

for (int i = 0; i < threadNum; i++) {

if (i == threadNum - 1) {

if (special) {

break;

}

cutList = list.subList(threadSize * i, dataSize);

} else {

cutList = list.subList(threadSize * i, threadSize * (i + 1));

}

// System.out.println("第" + (i + 1) + "组：" + cutList.toString());

final List<String> listStr = cutList;

task = new Callable<Integer>() {

@Override public

Integer call() throws Exception {

System.out.println(Thread.currentThread().getName() + "线程：" + listStr);

return 1;

}

};

// 这里提交的任务容器列表和返回的Future列表存在顺序对应的关系

tasks.add(task);

}

List<Future<Integer>> results = exec.invokeAll(tasks);

for (Future<Integer> future : results) {

System.out.println(future.get());

}

// 关闭线程池

exec.shutdown();

System.out.println("线程任务执行结束");

System.err.println("执行任务消耗了 ：" + (System.currentTimeMillis() - start) + "毫秒");

}