JDBC Batch Insert OutOfMemoryError

最新推荐文章于 2025-04-07 09:29:38 发布

原创最新推荐文章于 2025-04-07 09:29:38 发布 · 316 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#java #数据库

JAVA 专栏收录该内容

89 篇文章

订阅专栏

本文介绍了一种使用JDBC批处理向MySQL数据库中批量插入大量记录的方法，并解决了在此过程中遇到的内存溢出问题。通过调整批处理大小并定期执行批处理任务，有效地避免了内存占用过高的情况。

I have written a method insert() in which I am trying to use JDBC Batch for inserting half a million records into a MySQL database:

public void insert(int nameListId, String[] names) {
        String sql = "INSERT INTO name_list_subscribers (name_list_id, name, date_added)"+" VALUES (?, ?, NOW())";
        Connection conn = null;
        PreparedStatement ps = null;

        try{
            conn = getConnection();
            ps = conn.prepareStatement(sql);

            for(String s : names ){
                ps.setInt(1, nameListId); 
                ps.setString(2, s);
                ps.addBatch();
            }

            ps.executeBatch();

        }catch(SQLException e){
            throw new RuntimeException(e);
        }finally{
            closeDbResources(ps, null, conn);
        }
    }

But whenever I try to run this method, I get the following error:

java.lang.OutOfMemoryError: Java heap space
    com.mysql.jdbc.ServerPreparedStatement$BatchedBindValues.<init>(ServerPreparedStatement.java:72)
    com.mysql.jdbc.ServerPreparedStatement.addBatch(ServerPreparedStatement.java:330)
    org.apache.commons.dbcp.DelegatingPreparedStatement.addBatch(DelegatingPreparedStatement.java:171)

If I replace ps.addBatch() with ps.executeUpdate() and remove ps.executeBatch(), it works fine, though it takes some time. Please let me know if you know if using Batch is appropriate in this situation, and if it is, then why does it give OurOfMemoryError?

[color=green]It is out of memory because it hold all the transaction in memory and only send it over to the database when you call executeBatch.

If you don't need it to be atomic and would like the get better performance, you can keep a counter and call executeBatch every n number of records.[/color]

public void insert(int nameListId, String[] names) {
        String sql = "INSERT INTO name_list_subscribers (name_list_id, name, date_added)"+" VALUES (?, ?, NOW())";
        Connection conn = null;
        PreparedStatement ps = null;

        try{
            conn = getConnection();
            ps = conn.prepareStatement(sql);
            int icount = 0;
            for(String s : names ){
                ps.setInt(1, nameListId); 
                ps.setString(2, s);
                ps.addBatch();
                if(icount%1000 == 0){
                  ps.executeBatch();
                  ps.clearBatch();
                }

                icount++;
            }

            ps.executeBatch();

        }catch(SQLException e){
            throw new RuntimeException(e);
        }finally{
            closeDbResources(ps, null, conn);
        }
    }