java.util.zip.Deflater引发的内存溢出

本文详细解析了使用Ant的ZipOutputStream进行大量数据打包时遇到的内存溢出问题。通过深入研究,发现未正确关闭Deflater对象导致的内存泄漏是根本原因。文章提供了正确的资源释放方法,并强调了在使用高版本Ant及调用finish方法的重要性。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

1、问题:

这几天在使用ant的打包工具zip类时,发现打包大量数据时出现OutOfMemory:

 

Caused by: java.lang.OutOfMemoryError

        at java.util.zip.Deflater.init(Native Method)

        at java.util.zip.Deflater.<init>(Deflater.java:124)

        at org.apache.tools.zip.ZipOutputStream.<init>(ZipOutputStream.java:213)

 

2、求证过程:

2.1  外事不决问google

内存溢出总是这么让人头疼,看别人代码更是一种让人想死的活,多种手段求索无果发火.后看到报错地方@Deflater,述使用google查询下Deflater的相关介绍,查到如下网址:http://www-01.ibm.com/support/docview.wss?uid=swg21227106

豁然开朗,IBM好同志啊,文中介绍了Deflater不正确使用可能导致crash问题并介绍了正确使用方式,部分内容摘录如下:

 

If a deflater object is not explicitly closed and ended, a native memory leak occurs. This leak can result in OutOfMemoryError's and/or Java�6�4 Virtual Machine (JVM) crashes.

Cause

Not explicitly closing and ending a deflater object (such as, DeflaterOutputStream, GZIPOutputStream or ZipOutputStream) causes a native memory leak. This native leak can result in OutOfMemoryError's in the WebSphere Application Server logs as well as JVM hangs and crashes. The following error in the WebSphere Application Server SystemOut.log or SystemErr.log files is a potential indicator of a native issue caused by this type of deflater leak

       正确使用方式:

try {
    Deflater zip = new Deflater(level);
    DeflaterOutputStream defStream = new DeflaterOutputStream(out);
}  
finally {
    defStream.close();
    zip.end();
}

在使用完Deflater之后一定要调用该类的end方法进行释放!!      

  看到这里为什么内存溢出似乎已经有了个光明正大的解释,但是问题是我并不是直接使用Deflater类做压缩的。。。

2.2 求解apache

查看堆栈确定是使用org.apache.tools.zip.ZipOutputStream时报出来的错误,这个包来源于ant.jar,下载ant的源码找到该包中Deflater使用的地方,如下:

          public void finish()  throws IOException
  {
    closeEntry();
    this.cdOffset = this.written;
    for (Iterator i = this.entries.iterator(); i.hasNext(); ) {
      writeCentralFileHeader((ZipEntry)i.next());
    }
    this.cdLength = (this.written - this.cdOffset);
    writeCentralDirectoryEnd();
    this.offsets.clear();
    this.entries.clear();
    this.def.end();
  }

        原来是在finish方法中进行Deflater资源的释放~~,一般在使用流的时候只需要close掉流即可,看来使用ZipOutputStream还需要调用finish方法进行资源的释放才行。经修改代码加在finish方法的调用后解决。

3、总结

Deflater使用完后一定得要end;ant的打包工具类ZipOutputStream除了close掉外还得使用finish来释放资源;尽量使用高版本(我使用的是1.8)的ant打包,有些低版本ant中的ZipOutputStream即使你finish了照样没用,因为低版本的这个方法里也米有end掉Deflater,结果仍然OOM。

@ApiOperation("批量下载图像及label文件成压缩包接口") @PostMapping("/downloadZip") public DeferredResult<ResponseEntity<StreamingResponseBody>> downloadZip(@RequestBody List<Long> ids) { DeferredResult<ResponseEntity<StreamingResponseBody>> deferredResult = new DeferredResult<>(3600000L); CompletableFuture.runAsync(() -> { try { // 提前校验数据 List<CpImageInfo> cpImageInfos = cpImageMapper.selectBatchIds(ids); if (cpImageInfos.isEmpty()) { throw new RuntimeException("未找到样本信息"); } String zipName = new SimpleDateFormat("yyyyMMddHHmmss").format(new Date()) + ".zip"; HttpHeaders headers = new HttpHeaders(); headers.setContentType(MediaType.APPLICATION_OCTET_STREAM); headers.setContentDisposition(ContentDisposition.builder("attachment") .filename(URLEncoder.encode(zipName, "UTF-8")).build()); // 使用匿名内部类 StreamingResponseBody stream = new StreamingResponseBody() { @Override public void writeTo(OutputStream outputStream) throws IOException { try { fileImageService.downloadZip(ids, outputStream); } catch (Exception e) { log.error("文件流生成异常", e); throw new IOException("压缩过程发生错误"); } } }; deferredResult.setResult(ResponseEntity.ok() .headers(headers) .body(stream)); } catch (Exception e) { deferredResult.setErrorResult( ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR) .body("下载失败: " + e.getMessage())); } }); deferredResult.onTimeout(() -> deferredResult.setErrorResult( ResponseEntity.status(HttpStatus.REQUEST_TIMEOUT) .body("下载超时"))); deferredResult.onError(ex -> deferredResult.setErrorResult( ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR) .body("系统错误: " + ex.getMessage()))); return deferredResult; } @Override public void downloadZip(List<Long> ids, OutputStream outputStream) throws IOException { List<CpImageInfo> cpImageInfos = cpImageMapper.selectBatchIds(ids); if (cpImageInfos.isEmpty()) { throw new RuntimeException("未找到样本信息"); } // 使用非缓冲流提高大文件性能 try (ZipOutputStream zos = new ZipOutputStream(outputStream)) { zos.setLevel(Deflater.DEFAULT_COMPRESSION); // 平衡速度和压缩率 String rootDir = new SimpleDateFormat("yyyyMMddHHmmss").format(new Date()) + "/"; int i = 0; for (CpImageInfo imageInfo : cpImageInfos) { if (Thread.currentThread().isInterrupted()) { throw new IOException("下载被中断"); } i++; String safeImageName = sanitizeFilename(imageInfo.getImageName()); String name = removeParentheses(safeImageName); String sampleDir = rootDir + name + "(" + i + ")/"; // 处理图像文件 List<CpImageBasicInfo> imageFiles = cpImageBasicMapper.selectByUniqueValue(imageInfo.getUniqueValue()); for (CpImageBasicInfo file : imageFiles) { addFileToZip(zos, sampleDir + "img/" + file.getImageName(), file.getImagePath()); } // 处理标签文件 List<CpLabelBasicInfo> labelFiles = cpLabelBasicMapper.selectByUniqueValue(imageInfo.getUniqueValue()); for (CpLabelBasicInfo file : labelFiles) { addFileToZip(zos, sampleDir + "gt/" + file.getLabelFileName(), file.getLabelFilePath()); } // 定期刷新输出流 zos.flush(); outputStream.flush(); } } } // 改进的文件添加方法 private void addFileToZip(ZipOutputStream zos, String entryName, String filePath) throws IOException { File file = new File(filePath); if (!file.exists()) { log.warn("文件不存在: {}", filePath); return; } try (FileInputStream fis = new FileInputStream(file)) { ZipEntry zipEntry = new ZipEntry(entryName); zos.putNextEntry(zipEntry); byte[] buffer = new byte[65536]; // 64KB 缓冲区 int len; while ((len = fis.read(buffer)) > 0) { // 检查是否中断 if (Thread.currentThread().isInterrupted()) { throw new IOException("下载被中断"); } zos.write(buffer, 0, len); } zos.closeEntry(); } } // 文件名消毒处理 private String sanitizeFilename(String filename) { return filename.replaceAll("[\\\\/:*?\"<>|]", "_"); } public String removeParentheses(String filename) { if (filename == null) { return null; } String result = filename; int prevLength; // 循环替换直到没有括号内容 do { prevLength = result.length(); result = result.replaceAll("\\(.*?\\)", ""); } while (result.length() < prevLength); return result; } 2025-06-06 10:56:59.176 ERROR 22432 --- [ MvcAsync1] c.a.e.c.sample.FileImageController : 文件流生成异常 org.apache.catalina.connector.ClientAbortException: java.io.IOException: 远程主机强迫关闭了一个现有的连接。 at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:783) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:688) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:388) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:366) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253) ~[na:1.8.0_102] at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211) ~[na:1.8.0_102] at java.util.zip.ZipOutputStream.write(ZipOutputStream.java:331) ~[na:1.8.0_102] at com.aircas.evaluation.service.impl.sample.FileImageInfoServiceImpl.addFileToZip(FileImageInfoServiceImpl.java:876) ~[classes/:na] at com.aircas.evaluation.service.impl.sample.FileImageInfoServiceImpl.downloadZip(FileImageInfoServiceImpl.java:841) ~[classes/:na] at com.aircas.evaluation.service.impl.sample.FileImageInfoServiceImpl$$FastClassBySpringCGLIB$$50be5633.invoke(<generated>) ~[classes/:na] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.20.jar:5.3.20] at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) ~[spring-aop-5.3.20.jar:5.3.20] at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) ~[spring-aop-5.3.20.jar:5.3.20] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704) ~[spring-aop-5.3.20.jar:5.3.20] at com.aircas.evaluation.service.impl.sample.FileImageInfoServiceImpl$$EnhancerBySpringCGLIB$$5e579eb.downloadZip(<generated>) ~[classes/:na] at com.aircas.evaluation.controller.sample.FileImageController$1.writeTo(FileImageController.java:315) ~[classes/:na] at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:111) [spring-webmvc-5.3.20.jar:5.3.20] at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:98) [spring-webmvc-5.3.20.jar:5.3.20] at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing$4(WebAsyncManager.java:337) [spring-web-5.3.20.jar:5.3.20] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102] at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) ~[na:1.8.0_102] at java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:1.8.0_102] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102] Suppressed: org.apache.catalina.connector.ClientAbortException: java.io.IOException: The current thread was interrupted at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:783) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:688) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:388) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:366) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253) ~[na:1.8.0_102] at java.util.zip.ZipOutputStream.closeEntry(ZipOutputStream.java:255) ~[na:1.8.0_102] at java.util.zip.ZipOutputStream.finish(ZipOutputStream.java:360) ~[na:1.8.0_102] at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:238) ~[na:1.8.0_102] at java.util.zip.ZipOutputStream.close(ZipOutputStream.java:377) ~[na:1.8.0_102] at com.aircas.evaluation.service.impl.sample.FileImageInfoServiceImpl.downloadZip(FileImageInfoServiceImpl.java:854) ~[classes/:na] ... 14 common frames omitted Caused by: java.io.IOException: The current thread was interrupted at org.apache.tomcat.util.net.NioChannel.checkInterruptStatus(NioChannel.java:236) at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1384) at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:773) at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:593) at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:537) at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:547) at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:110) at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:194) at org.apache.coyote.Response.doWrite(Response.java:615) at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:340) ... 25 common frames omitted Caused by: java.io.IOException: 远程主机强迫关闭了一个现有的连接。 at sun.nio.ch.SocketDispatcher.write0(Native Method) ~[na:1.8.0_102] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51) ~[na:1.8.0_102] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_102] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_102] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_102] at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:135) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1384) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:773) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:593) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:537) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:547) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:112) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:194) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.coyote.Response.doWrite(Response.java:615) ~[tomcat-embed-core-9.0.63.jar:9.0.63] at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:340) ~[tomcat-embed-core-9.0.63.jar:9.0.63] ... 24 common frames omitted
06-07
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值