SpringBatch中使用分区及多线程性能比较

1. 表结构说明：借据表（测试之前批量写入了200万数据）；借据备份表1（使用多线程处理）；借据备份表2（使用分区处理）

CREATE TABLE `t_batch_duebill` (
  `duebillno` char(32) NOT NULL COMMENT '借据号',
  `balance` decimal(16,2) NOT NULL DEFAULT '0.00' COMMENT '剩余本金',
  `payoff_flag` char(3) NOT NULL DEFAULT 'WJQ' COMMENT '结清标志：WJQ 未结清; YJQ 已结清',
  `interest` decimal(16,2) NOT NULL DEFAULT '0.00' COMMENT '当前利息',
  `data_index` int(11) DEFAULT NULL COMMENT '当前处理的下标',
  PRIMARY KEY (`duebillno`),
  UNIQUE KEY `data_index_unique` (`data_index`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='测试-借据表';

CREATE TABLE `t_batch_duebill_bak1` (
  `duebillno` char(32) NOT NULL COMMENT '借据号',
  PRIMARY KEY (`duebillno`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='借据表备份表1';

CREATE TABLE `t_batch_duebill_bak2` (
  `duebillno` char(32) NOT NULL COMMENT '借据号',
  PRIMARY KEY (`duebillno`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='借据表备份表2';

1. 测试说明：
   将借据表中的数据写入到2个备份表中：
   考虑到线程安全问题，写入的备份表都只有一个借据编号主键，如果使用非安全的读（如JdbcCursorItemReader ）将可能会造成主键冲突，所以2种方案都选用了JdbcPagingItemReader 线程安全的读分页（每页2000条）写入频率为10000条一提交
   第一种方案使用多线程读写：直接从借据表t_batch_duebill中读取数据
   第二种方案使用分区读写：
   1、首先需要把借据表t_batch_duebill的data_index字段从1开始更新（这步不在程序中体现，手动通过SQL处理）
   2、查询出借据表t_batch_duebill中总记录数，根据传入的分区数，计算每个分区需要处理多少数据（总数/分区数）
2. 测试结果：
   第一种方案跟第二种方案测试结果如下：
   1、[77293]ms（读取[2000000]条，写入[2000000]条, 跳过[0]数据）
   2、[81835]ms（读取[2000000]条，写入[2000000]条, 跳过[0]数据）
   3、通过多次反复测试结果来看，这2种方案时间差距不大，而且如果线程池初始线程为50个的话，都用这个线程池，第二种方案使用10个分区的话第一种方案稍微快点（毕竟第一种方案是直接使用全部50个线程，而第二种方案中每个分区使用一个线程处理，只使用了10个线程）
   4、还忽略了一个问题，就是第二种方案中第一个步骤需要将data_index字段从1开始标号，这个时间如果用SQL处理的话大概是10-30秒，如果用存储过程游标实现，时间达到了惊人的4分10秒（可能是我的SQL水平太弱）
3. 代码
   1、配置文件（使用spring boot-1.5.4） application.yml 启动文件

management:
  health:
    zookeeper:
      enabled: true
server:
  port: 8080
  context-path: /

security:
  basic:
    enabled: false
logging:
  file: partition.log
  level: info
  path: /Users/xulong/logs/partition
spring:
  session:
    store-type: none
  cloud:
    zookeeper:
      enabled: false
  batch:
    initializer:
      enabled: false
    job:
      enabled: false
  druid:
    username: root
    password: root
    min-idle: 5
    max-wait: 6000
    max-active: 200
    initial-size: 5
    test-on-borrow: false
    test-while-idle: true
    test-on-return: false
    filters: stat,wall,logback
    pool-prepared-statements: true
    min-evictable-idle-time-millis: 300000
    time-between-eviction-runs-millis: 60000
    max-pool-prepared-statement-per-connection-size: 20
    connection-properties: druid.stat.mergeSql=true;druid.stat.slowSqlMillis=5000
    db-url: jdbc:mysql://localhost:3306/batch?useUnicode=true&characterEncoding=utf-8&useSSL=true

pom.xml 文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.csii.loan.batch</groupId>
    <artifactId>partition</artifactId>
    <version>1.0.0</version>
    <packaging>jar</packaging>

    <name>partition</name>
    <description>分区测试</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.4.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <!--<spring-cloud.version>Dalston.SR1</spring-cloud.version>-->
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-batch</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-cache</artifactId>
        </dependency>
        <!--<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-zookeeper-config</artifactId>
        </dependency>-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-integration</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jdbc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-security</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.session</groupId>
            <artifactId>spring-session</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>druid</artifactId>
            <version>1.0.29</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.30</version>
        </dependency>
    </dependencies>

    <!--<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>-->

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

classpath:batch/applicationContext.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:batch="http://www.springframework.org/schema/batch"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd


        http://www.springframework.org/schema/batch
        http://www.springframework.org/schema/batch/spring-batch.xsd">