spring Batch实现数据库大数据量读写

查询数据库最大行数限制探究 查询数据库最大行数限制探究 1. 问题背景 最近,在对项目中的机构部门表的数据访问层代码进行单元测试时,甚是神奇地发现了这个指标——查询数据库最大行数限制。 首先,使用查询总行数语句查出该表的总行数为:10768。 SELECT count(1) FROM department_define; 单元测试: @Test public void test() { List<DepartmentDefinePO> departmentDefinePOList = baseDataMapper.li 阅读详情
spring Batch实现数据库大数据量读写

1. data-source-context.xml


<?xml version="1.0" encoding="UTF-8"?>  
<beans xmlns="http://www.springframework.org/schema/beans"  
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"  
    xmlns:tx="http://www.springframework.org/schema/tx"  
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd  
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd  
    http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.0.xsd">  
  
    <!-- 1) USE ANNOTATIONS TO IDENTIFY AND WIRE SPRING BEANS. -->  
    <context:component-scan base-package="net.etongbao.vasp.ac" />  
      
    <!-- 2) DATASOURCE, TRANSACTION MANAGER AND JDBC TEMPLATE -->  
   
    <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"  
        destroy-method="close" abstract="false" scope="singleton">  
        <!-- oracle.jdbc.driver.oracleDriver -->  
        <property name="driverClass" value="oracle.jdbc.OracleDriver" />  
        <property name="jdbcUrl" value="jdbc:oracle:thin:@192.168.1.23:1521:orcl01" />  
        <property name="user" value="USR_DEV01" />  
        <property name="password" value="2AF0829C" />  
        <property name="checkoutTimeout" value="30000" />  
        <property name="maxIdleTime" value="120" />  
        <property name="maxPoolSize" value="100" />  
        <property name="minPoolSize" value="2" />  
        <property name="initialPoolSize" value="2" />  
        <property name="maxStatements" value="0" />  
        <property name="maxStatementsPerConnection" value="0" />  
        <property name="idleConnectionTestPeriod" value="30" />     
    </bean>  
    <bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">  
        <property name="dataSource" ref="dataSource" />  
    </bean>  
  
    <bean id="transactionManager"  
        class="org.springframework.jdbc.datasource.DataSourceTransactionManager">  
        <property name="dataSource" ref="dataSource" />  
    </bean>  
      
    <tx:annotation-driven transaction-manager="transactionManager" />  
</beans>  


2. quart-zcontext.xml commit-interval="10000"每次批量数据的条数,数值越大效率越高,

可在此处添加事物处理,每次回滚数就是commit-interval数


<?xml version="1.0" encoding="UTF-8"?>  
<beans xmlns="http://www.springframework.org/schema/beans"  
    xmlns:batch="http://www.springframework.org/schema/batch"  
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"  
    xmlns:tx="http://www.springframework.org/schema/tx"  
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd  
    http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd  
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd  
    http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.0.xsd">  
  
    <import resource="data-source-context.xml"/>  
      
    <!--   JOB REPOSITORY - WE USE IN-MEMORY REPOSITORY FOR OUR EXAMPLE -->  
    <bean id="jobRepository"  
        class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">  
        <property name="transactionManager" ref="transactionManager" />  
    </bean>  
      
    <!-- batch config -->  
    <bean id="jobLauncher"  
        class="org.springframework.batch.core.launch.support.SimpleJobLauncher">  
        <property name="jobRepository" ref="jobRepository" />  
    </bean>  
  
    <!--  FINALLY OUR JOB DEFINITION. THIS IS A 1 STEP JOB -->  
    <batch:job id="ledgerJob">  
        <batch:listeners>  
            <batch:listener ref="appJobExecutionListener" />  
        </batch:listeners>  
      <batch:step id="step1"> 
        <batch:tasklet transaction-manager="transactionManager">  
        <batch:tasklet>  
          <batch:listeners>  
              <batch:listener ref="itemFailureLoggerListener" />  
          </batch:listeners>  
          <batch:chunk reader="ledgerReader" writer="ledgerWriter"  commit-interval="10000" /> <!-- 1万条进行一次commit -->  
        </batch:tasklet>
        </batch:tasklet>  
     </batch:step>  
</batch:job>  
   
<!--  READER -->  
<bean id="ledgerReader"  class="org.springframework.batch.item.database.JdbcCursorItemReader">  
    <property name="dataSource" ref="dataSource" />  
    <property name="sql" value="select * from ledger" />   
    <property name="rowMapper" ref="ledgerRowMapper" />  
</bean>  
   
<!-- Spring Batch Job同一个job instance,成功执行后是不允许重新执行的【失败后是否允许重跑,可通过配置Job的restartable参数来控制,默认是true】,如果需要重新执行,可以变通处理,  
    添加一个JobParameters构建类,以当前时间作为参数,保证其他参数相同的情况下却是不同的job instance -->  
<bean id="jobParameterBulider" class="org.springframework.batch.core.JobParametersBuilder" />  
  
<!-- 定时任务 开始 -->    
<bean id="ledgerJobDetail" class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">    
    <property name="targetObject">    
        <!-- 定时执行的类 -->    
        <ref bean="quartzLedgerJob" />    
    </property>    
    <property name="targetMethod">    
        <!-- 定时执行的类方法 -->    
        <value>execute</value>    
    </property>    
</bean>    
   
<bean id="ledgerCronTrigger" class="org.springframework.scheduling.quartz.CronTriggerBean" >    
    <!-- 这里不可以直接在属性jobDetail中引用taskJob,因为他要求的是一个jobDetail类型的对象,所以我们得通过MethodInvokingJobDetailFactoryBean来转一下 -->    
    <property name="jobDetail" >    
        <ref bean="ledgerJobDetail" />    
    </property>    
    <!--在每天下午18点到下午18:59期间的每1分钟触发  -->    
    <!--在每天上午10点40分准时触发  -->    
    <property name="cronExpression" >    
        <!-- <value>0 * 15 * * ?</value> -->  
        <value>0 45 10 * * ? * </value>   
    </property>    
         
</bean>    
       
<!-- 触发器工厂,将所有的定时任务都注入工厂-->    
<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">    
    <!-- 添加触发器 -->    
    <property name="triggers">    
        <list>    
            <!-- 将上面定义的测试定时任务注入(可以定义多个定时任务,同时注入)-->    
            <ref local="ledgerCronTrigger" />    
        </list>    
    </property>    
</bean>    
<!-- 定时任务 结束 -->    
  /beans>    


3.定时调度job类 QuartzLedgerJob.java 

package net.etongbao.vasp.ac.quartz;


import java.util.Date;  
import org.slf4j.Logger;  
import org.slf4j.LoggerFactory;  
import org.springframework.batch.core.Job;  
import org.springframework.batch.core.JobParametersBuilder;  
import org.springframework.batch.core.launch.JobLauncher;  
import org.springframework.beans.factory.annotation.Autowired;  
import org.springframework.stereotype.Component;  
import org.springframework.util.StopWatch;  
  
/** 
 * 定时调度类 
 * @author Fu Wei 
 * 
 */  
  
@Component("quartzLedgerJob")  
public class QuartzLedgerJob {  
  
    private static final Logger LOG = LoggerFactory.getLogger(QuartzLedgerJob.class);  
  
    @Autowired  
    private JobLauncher jobLauncher;  
  
    @Autowired  
    private Job ledgerJob;  
  
    @Autowired  
    JobParametersBuilder jobParameterBulider;  
  
    private static long counter = 0l;  
      
    /** 
     * 执行业务方法 
     * @throws Exception 
     */  
    public void execute() throws Exception {  
        LOG.debug("start...");  
        StopWatch sw = new StopWatch();  
        sw.start();  
        /* 
         * Spring Batch Job同一个job instance,成功执行后是不允许重新执行的【失败后是否允许重跑, 
         * 可通过配置Job的restartable参数来控制,默认是true】,如果需要重新执行,可以变通处理, 
         * 添加一个JobParameters构建类,以当前时间作为参数,保证其他参数相同的情况下却是不同的job instance 
         */  
        jobParameterBulider.addDate("date", new Date());  
        jobLauncher.run(ledgerJob, jobParameterBulider.toJobParameters());  
        sw.stop();  
        LOG.debug("Time elapsed:{},Execute quartz ledgerJob:{}", sw.prettyPrint(), ++counter);  
    }  
}  

4.程序启动类 StartQuartz.java 

package net.etongbao.vasp.ac.quartz;


import java.io.FileNotFoundException;  
  
import org.springframework.context.support.ClassPathXmlApplicationContext;  
  
/** 
 * 启动定时调度 
 * @author Fu Wei 
 * 
 */  
public class StartQuartz {  
  
    public static void main(String[] args) throws FileNotFoundException {  
  
        new ClassPathXmlApplicationContext("/net/etongbao/vasp/ac/resources/quartz-context.xml");  
    }  
}  


5. pojo类  Ledger.java


package net.etongbao.vasp.ac.pojo;  
  
import java.io.Serializable;  
import java.util.Date;  
  
public class Ledger implements Serializable {  
  
    private int id;  
    private Date receiptDate;  
    private String memberName;  
    private String checkNumber;  
    private Date checkDate;  
    private String paymentType;  
    private double depositAmount;  
    private double paymentAmount;  
    private String comments;  
  
    public Ledger() {  
        super();  
    }  
  
    public Ledger(int id, Date receiptDate, String memberName, String checkNumber, Date checkDate, String paymentType,  
            double depositAmount, double paymentAmount, String comments) {  
        super();  
        this.id = id;  
        this.receiptDate = receiptDate;  
        this.memberName = memberName;  
        this.checkNumber = checkNumber;  
        this.checkDate = checkDate;  
        this.paymentType = paymentType;  
        this.depositAmount = depositAmount;  
        this.paymentAmount = paymentAmount;  
        this.comments = comments;  
    }  
    public int getId() {  
        return id;  
    }  
  
    public void setId(int id) {  
        this.id = id;  
    }  
  
    public Date getReceiptDate() {  
        return receiptDate;  
    }  
  
    public void setReceiptDate(Date receiptDate) {  
        this.receiptDate = receiptDate;  
    }  
  
    public String getMemberName() {  
        return memberName;  
    }  
  
    public void setMemberName(String memberName) {  
        this.memberName = memberName;  
    }  
  
    public String getCheckNumber() {  
        return checkNumber;  
    }  
  
    public void setCheckNumber(String checkNumber) {  
        this.checkNumber = checkNumber;  
    }  
  
    public Date getCheckDate() {  
        return checkDate;  
    }  
  
    public void setCheckDate(Date checkDate) {  
        this.checkDate = checkDate;  
    }  
    public String getPaymentType() {  
        return paymentType;  
    }  
  
    public void setPaymentType(String paymentType) {  
        this.paymentType = paymentType;  
    }  
  
    public double getDepositAmount() {  
        return depositAmount;  
    }  
  
    public void setDepositAmount(double depositAmount) {  
        this.depositAmount = depositAmount;  
    }  
  
    public double getPaymentAmount() {  
        return paymentAmount;  
    }  
  
    public void setPaymentAmount(double paymentAmount) {  
        this.paymentAmount = paymentAmount;  
    }  
  
    public String getComments() {  
        return comments;  
    }  
  
    public void setComments(String comments) {  
        this.comments = comments;  
    }  
}  


6. LedgerDaoImpl.java

    package net.etongbao.vasp.ac.dao.impl;


import java.sql.PreparedStatement;  
import java.sql.SQLException;  
import net.etongbao.vasp.ac.dao.LedgerDao;  
import net.etongbao.vasp.ac.pojo.Ledger;  
import org.springframework.beans.factory.annotation.Autowired;  
import org.springframework.jdbc.core.JdbcTemplate;  
import org.springframework.jdbc.core.PreparedStatementSetter;  
import org.springframework.stereotype.Repository;  
  
/** 
 * ledger数据操作类 
 *  
 * @author Fu Wei 
 *  
 */  
  
@Repository  
public class LedgerDaoImpl implements LedgerDao {  
  
    private static final String SAVE_SQL = "insert into ledger_temp (rcv_dt, mbr_nm, chk_nbr, chk_dt, pymt_typ, dpst_amt, pymt_amt, comments) values(?,?,?,?,?,?,?,?)";  
  
    @Autowired  
    private JdbcTemplate jdbcTemplate;  
  
    @Override  
    public void save(final Ledger item) {  
        jdbcTemplate.update(SAVE_SQL, new PreparedStatementSetter() {  
            public void setValues(PreparedStatement stmt) throws SQLException {  
                stmt.setDate(1, new java.sql.Date(item.getReceiptDate().getTime()));  
                stmt.setString(2, item.getMemberName());  
                stmt.setString(3, item.getCheckNumber());  
                stmt.setDate(4, new java.sql.Date(item.getCheckDate().getTime()));  
                stmt.setString(5, item.getPaymentType());  
                stmt.setDouble(6, item.getDepositAmount());  
                stmt.setDouble(7, item.getPaymentAmount()); 
                stmt.setString(8, item.getComments());  
            }  
        });  
    }   
}  

7. 接口 LedgerDao .java


package net.etongbao.vasp.ac.dao;  
  
import net.etongbao.vasp.ac.pojo.Ledger;  
  
public interface LedgerDao {  
    public void save(final Ledger item) ;  
}  

 8. JdbcTemplete 需要的LedgerRowMapper.java

     package net.etongbao.vasp.ac.batch.writer;


import java.sql.ResultSet;  
import java.sql.SQLException;  
  
import net.etongbao.vasp.ac.pojo.Ledger;  
  
import org.springframework.jdbc.core.RowMapper;  
import org.springframework.stereotype.Component;  
   
/** 
 * ledger行的映射类 
 * @author Administrator 
 * 
 */  
@Component("ledgerRowMapper")  
public class LedgerRowMapper implements RowMapper {  
    public Object mapRow(ResultSet rs, int rowNum) throws SQLException {  
        Ledger ledger = new Ledger();  
        ledger.setId(rs.getInt("id"));  
        ledger.setReceiptDate(rs.getDate("rcv_dt"));  
        ledger.setMemberName(rs.getString("mbr_nm"));  
        ledger.setCheckNumber(rs.getString("chk_nbr"));  
        ledger.setCheckDate(rs.getDate("chk_dt"));  
        ledger.setPaymentType(rs.getString("pymt_typ"));  
        ledger.setDepositAmount(rs.getDouble("dpst_amt"));  
        ledger.setPaymentAmount(rs.getDouble("pymt_amt"));  
        ledger.setComments(rs.getString("comments"));  
        return ledger;  
    }  
}  


9.关键类LedgerWriter.java,写入数据,负责数据的添加 


package net.etongbao.vasp.ac.batch.writer;  
  
import java.util.List;  
  
import net.etongbao.vasp.ac.dao.LedgerDao;  
import net.etongbao.vasp.ac.pojo.Ledger;  
  
import org.springframework.batch.item.ItemWriter;  
import org.springframework.beans.factory.annotation.Autowired;  
import org.springframework.stereotype.Component;  
  
/** 
 * ledger写入数据 
 *  
 * @author Fu Wei 
 *  
 */  
@Component("ledgerWriter")  
public class LedgerWriter implements ItemWriter<Ledger> {  
  
    @Autowired  
    private LedgerDao ledgerDao;  
  
    /** 
     * 写入数据 
     *  
     * @param ledgers 
     */  
    public void write(List<? extends Ledger> ledgers) throws Exception {  
        for (Ledger ledger : ledgers) {  
            ledgerDao.save(ledger);  
        }  
    }    
}  


classPath:<?xml version="1.0" encoding="UTF-8"?>


<classpath>  
    <classpathentry kind="src" path="src"/>  
    <classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/jrockit-jdk1.6.0_24-R28.1.3-4.0.1"/>  
    <classpathentry kind="lib" path="lib/aopalliance-1.0.jar"/>  
    <classpathentry kind="lib" path="lib/c3p0-0.9.1.2.jar"/>  
    <classpathentry kind="lib" path="lib/commons-collections-3.2.1.jar"/>  
    <classpathentry kind="lib" path="lib/commons-lang-2.3.jar"/>  
    <classpathentry kind="lib" path="lib/commons-logging-1.1.1.jar"/>  
    <classpathentry kind="lib" path="lib/etb-log4j-1.2.16.jar"/>  
    <classpathentry kind="lib" path="lib/etb-slf4j-api-1.5.8.jar"/>  
    <classpathentry kind="lib" path="lib/etb-slf4j-log4j12-1.5.8.jar"/>  
    <classpathentry kind="lib" path="lib/ojdbc6.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.aop-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.asm-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.aspects-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.beans-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.context-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.context.support-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.core-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.expression-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.instrument-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.instrument.tomcat-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.jdbc-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.jms-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.orm-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.oxm-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.test-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/org.springframework.transaction-3.0.5.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/quartz-all-1.6.5.jar"/>  
    <classpathentry kind="lib" path="lib/spring-batch-core-2.1.6.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/spring-batch-infrastructure-2.1.6.RELEASE.jar"/>  
    <classpathentry kind="lib" path="lib/spring-batch-test-2.1.6.RELEASE.jar"/>  
    <classpathentry kind="con" path="org.eclipse.jdt.junit.JUNIT_CONTAINER/4"/>  
    <classpathentry kind="output" path="bin"/>  
</classpath>  

 总结: 测试数据8万多条,响应时间3分多钟。

 

关键在于quartz-context.xml中


<bean id="ledgerReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">

  <property name="dataSource" ref="dataSource" />

  <property name="sql" value="select * from ledger" /> 

  <property name="rowMapper" ref="ledgerRowMapper" />

</bean>

 

负责读取数据在程序执行时一次性抓取全部数据后在批量的交给LedgerWriter进行写操作。当然也可以使用分页读取JdbcPagingItemReader,但要分页数量与写入数量要大写相

同,还可以对分页出来的数据进行添加悲观锁LedgerWriter.java 负责写入数据,每次写入1000条。








Spring Batch 专题系列(七):Spring Batch 与数据库集成 在上一篇文章中,我们学习了 Spring Batch 的并行处理机制(多线程 Step、分区、并行 Job)和性能优化技巧,掌握了如何处理海量数据。在批处理任务中,数据库操作是最常见的场景之一,例如从源表读取数据,经过转换后写入目标表(ETL)、批量更新记录或生成报表。本文详细讲解了 Spring Batch 与数据库的集成,包括 JDBC、JPA 和 MyBatis 的读写方式,以及事务管理和性能优化实践。A:JDBC 适合高性能 ETL,JPA 适合复杂实体关系,MyBatis 介于两者,灵活性高。 阅读详情

相关推荐

Spring Batch从入门到精通的知识介绍

Spring Batch 是一个轻量级的、全面的批处理框架,用于开发企业级的批处理应用程序。它构建在经典的 Java EE 核心技术之上,同时利用了 Spring Framework 提供的灵活性和可扩展性。

技术探秘者的博客 1503

Spring Batch之读数据库——JdbcCursorItemReader(三十五)

一、JdbcCursorItemReader结构及关键属性 JdbcCursorItemReader将数据库记录转换为Java对象时主要有两步工作:首先根据PreparedStatement从数据库中获取结果集ResultSet;然后使用RowMapper将结果集ResultSet转换为Java对象。 二、关键接口、类和关键属性 1.JdbcCursorItemReader关键接口、类 关键类 说明 DataSource 提供读取数据库的数据源信息 PreparedState

qq_21370419的博客 4912

企业微信群机器人配置全攻略:从零到消息推送的完整流程

本文详细介绍了企业微信群机器人的配置与使用全流程,从零开始指导如何创建机器人、获取Webhook地址,到通过HTTP请求发送文本、Markdown、图片及模板卡片等多种消息类型。内容涵盖核心的接口配置、安全实践及实战应用场景,帮助开发与运维团队快速构建自动化消息推送中枢,提升协作效率。

weixin_29231725的博客 849

大数据批处理框架Spring Batch

如今微服务架构讨论的如火如荼。但在企业架构里除了大量的OLTP交易外,还存在海量的批处理交易。在诸如银行的金融机构中,每天有3-4万笔的批处理作业需要处理。针对OLTP,业界有大量的开源框架、优秀的架构设计给予支撑;但批处理领域的框架确凤毛麟角。是时候和我们一起来了解下批处理的世界哪些优秀的框架和设计了,今天我将以Spring Batch为例,和大家一起探秘批处理的世界。 初识批处理

java、c++、机器学习方向King 1988

C语言 在合并编译 ( 多.c文件 ) 情况下 实现 linked list ( 链表 ) 教程 . 更新ing

写在前面  依旧是参考借鉴了 浙江大学 温老师 的视频教程,很不错,在中国大学MOOC上,强力推荐!  虽说有借鉴,但是能保证的是每一行代码都是我自己经过理解后手敲出来的,所以才来做了这部分的教程。  最后,带 * 字符是我个人觉得可以添加的功能,按理来讲是可以不看的,而且大概率是可以有更好的改进的。  PS: 文章还在更新,如果文字蓝了(有超链接),说明写完了,如果没蓝,说明我在开军舰.jpg 源代码:  这里→源代码. 第一步:前置工作  先说一下每一篇都会出现 main 函数的,stage1 只是给没

weixin_47287635的博客 527

告别SquareLine!用免费的NXP GUI-Guider为海思3516AV300开发LVGL界面(附完整移植流程)

本文详细介绍了如何利用免费的NXP GUI-Guider工具为海思Hi3516AV300开发LVGL界面,包括环境配置、代码移植和性能优化等全流程实战指南。特别针对二维码组件缺失等常见问题提供解决方案,帮助开发者零成本构建高效嵌入式GUI系统。

weixin_28292761的博客 417

spring batch mysql_Spring批处理 - JdbcCursorItemReader使用大型MySQL表抛出OutOfMemoryError

我正在编写一个使用Spring Batch处理来自MySQL数据库表的7,637,064行的程序。我已经用较小的表获得成功,但当JdbcCursorItemReader试图打开游标时,此表中的大量行会导致OutOfMemoryError异常。Spring批处理 - JdbcCursorItemReader使用大型MySQL表抛出OutOfMemoryError我可以通过抛出一个更大的Xmx来解决这...

weixin_26820427的博客 620

Windows 10/11下R与RStudio最新版安装避坑指南(含镜像加速配置)

本文提供了Windows 10/11系统下R与RStudio最新版的详细安装指南,包含镜像加速配置和常见问题解决方案。通过分步教程帮助用户避开安装陷阱,优化开发环境设置,特别适合国内用户快速搭建高效的数据分析平台。

weixin_29065959的博客 1008

Spring Batch(6)——数据库批数据读写

因此当数据写入与整个Step绑定为事物时还需要实现一个控制过程是:在写入数据的过程中出现异常时要擦除本次事物已经写入的数据,这样才能和整个Step的状态保持一致。,因为数据库的各种链接池本身就保证了链接->写入->释放的高效执行,也不存在向队尾添加数据的问题。Spring Batch为不同类型的文件的写入提供了多个实现类,但并没有为数据库的写入提供任何实现类,而是交由开发者自己去实现接口。因此,对于数据库的写入操作只要按照常规的批量数据写入的方式即可,开发者使用任何工具都可以完成这个过程。

justlpf的专栏 2206

pytorch转onnx转paddlepaddle

1.pt转onnx 可查看我之前的文章:https://blog.youkuaiyun.com/oyjwin/article/details/117778583?spm=1001.2014.3001.5501

oyjwin的博客 1462

基于STM32、HAL库的TPL0501数字电位器驱动程序设计

@param value 输出值(0-255)* @param hdev TPL0501句柄。* @param hdev TPL0501句柄。* @brief 获取TPL0501当前设置值。* @param hdev TPL0501句柄。* @brief 获取TPL0501当前设置值。* @brief 设置TPL0501输出值。* @brief 设置TPL0501输出值。* @return 当前值(0-255)* @brief 初始化TPL0501。* @brief 初始化TPL0501。

corlin6688的博客 1027

Java后台框架篇--spring Batch实现数据库大数据量读写

1. data-source-context.xml   Xml代码   xml version="1.0" encoding="UTF-8"?>   beans xmlns="http://www.springframework.org/schema/beans"       xmlns:xsi="http://www.w3.org/2001/XMLSchema

非淡泊无以明志,非宁静无以致远 2409

让 PyCharm 代码“发光”:高亮插件与特效全攻略

PyCharm 作为一款强大的 Python IDE,支持多种插件让代码“发光”——无论是动态输入特效、彩虹括号,还是霓虹主题,都能让你的代码脱颖而出!进入 File → Settings → Editor → Color Scheme → Rainbow Brackets。进入 File → Settings → Appearance & Behavior → Material Theme。进入 File → Settings → Editor → Color Scheme → General。

qq_43580271的博客 3840

Spring Batch 批处理框架

一、 介绍Spring Batch能够支持简单的、复杂的和大数据量的批处理作业,是一个批处理应用框架,不是调度框架,但需要和调度框架(Quartz, Tivoli, Control-M, Cron等)合作来构建完成批处理任务。整体架构如下: 二、 基本概念 Job: 作业, 由多个 Step 组成,封装整个批处理操作, 是 Batch 操作的基础单元; Job Instance: 作业实例,每次作

不影响全局 3777

基于Simulink的伺服电机位置闭环控制仿真

本文带你从零构建了伺服电机位置闭环控制系统​ 的完整Simulink仿真模型,实现了:✅ 搭建PMSM伺服电机、逆变器、机械臂负载的主电路与编码器反馈模型;✅ 用MATLAB Function实现三环控制(位置环PID、坐标变换、电流环PI);✅ 验证阶跃定位(稳态误差0.002°、响应0.08s)与正弦跟踪(误差0.008°)的高性能;✅ 对比开环控制,明确位置闭环在精度、响应、抗扰上的优势;✅ 理解“三环嵌套+反馈校正”的核心思想(精密定位的工程实现)。核心收获。

amy_mhd的博客 174

RU DOS版、EFI版32bit/64bit BIOS常用工具

DOS版、EFI版的RU工具,可用来查看IO、MEM、PCI、SMBIOS等资源。亲测在EFI SHELL下可用。

Spring Batch 大数据量处理实战:从入门到精通

别叫我大神,叫我 Alex 就好。处理百万级数据不用愁,Spring Batch 让批处理变得优雅而高效。

alex_goden的博客 4244

假人有限元模型

假人模型,车辆碰撞假人有限元模型,适合做乘员安全的研究者利用

关于 Spring Batch 的详细解析及其同类框架的对比分析,以及如何自己设计一个java批处理框架(类似spring batch)的步骤

关于 Spring Batch 的详细解析及其同类框架的对比分析

爱的叹息的专栏 2030

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

温度PID控制PLC程序

温度PID稳定控制程序,适用于欧姆龙PLC,对其他PLC也具有很大参考价值

深入解析Spring Batch:企业级批处理框架的技术之旅

Spring Batch是一个开源的、轻量级的批处理框架,它基于Spring框架构建,继承了Spring的诸多优点,如依赖注入、面向切面编程等。Spring Batch旨在简化批处理应用程序的开发,提供了一套丰富的功能来支持事务管理、作业调度、异常处理、日志记录等。Spring Batch是一个完善的批处理框架,旨在帮助企业建立健壮、高效的批处理应用。它是Spring的一个子项目,使用Java语言并基于Spring框架为基础开发,使得已经使用Spring框架的开发者或者企业更容易访问和利用企业服务。

码到三十五 2万+

Spring Batch 批处理框架优化实践,效率嘎嘎高!

一、Spring Batch简介1 框架概述2 核心概念和组件二、批处理优化实践4.1 使用Spring Boot Actuator进行监控4.2 使用管理控制台来监控3.1 批处理启动前校验3.2 读写校验2.1 多进程处理2.2 多线程处理1.1 分页处理数据1.2 使用读写缓存1.3 行级别写操作1 减少读写次数2 并发处理任务3 提高数据校验准确性4 监控批处理任务三、实践示例3.1 修改...

Java知音 1335

Spring Batch零基础入门:第一个批处理程序

Spring Batch是Spring生态中的一个轻量级批处理框架,主要用于处理大量数据的读取、处理和写入。它的核心思想是将批处理任务分解为一个个独立的步骤,每个步骤由读取、处理和写入三个部分组成。

CyanWave34的博客 1043
上一篇: Java中的日志——Java.util.logging、log4j、commons-logging
下一篇: spring batch教程 之 配置并运行Job
回头等终点
博客等级 码龄10年 29粉丝 17原创
评论 2
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值