大文件上传与下载

原创 于 2019-05-23 11:54:53 发布 · 1.1k 阅读 · 2 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。

本文介绍了在SpringCloud后端环境下，针对大文件上传失败的问题，提出了分片上传的解决方案，并讨论了前端如何实现。同时，针对文件下载，特别是大文件下载导致的界面缓存和长时间下载问题，提出了利用WebUploader组件和模拟直接访问后端服务器的跨域下载策略，以及分片下载来防止内存溢出。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

开发环境

后端:springCloud

前端:dva+react

问题

1、大文件上传失败

springCloud可以设置文件上传的大小(不同的版本设置参数可能不同)

如下：

  1. spring.http.multipart.enabled=true

  2. spring.http.multipart.max-file-size=20MB

  3. spring.http.multipart.max-request-size=1000MB

实际中上传的文件可能会很大，前台文件上传设置也不会根据后台设置进行控制，有可能需要上传1G多的文件

这时如果上传1G多的文件，则会受后端限制，从而上传失败

所以需要对文件进行适当的分片

普通文件上传

前端代码：

function uploadFile(data,url){
    let obj = {
        method:'post',
        url:url,
        data:data,
        headers:{
            'Content-Type':'multipart/form-data',
        }
    }
    retrun Axios.create().request(obj).then(res => {
        return res;
    }).catch(err => {
        console.log(err);
        return ''
    })
}

后端代码:

public static void upload(InputStream input,String url){
        OutputStream os = null;
        try{
            os = new FileOutputStream(url);
            byte[] b = new byte[1024];
            int length = 0;
            while ((length = input.read(b)) > 0){
                os.write(b,0,length);
            }
        }catch (Exception e){
            e.printStackTrace();
        }finally {
            try{
                os.close();
            }catch (IOException e){
                e.printStackTrace();
            }
            try{
                input.close();
            }catch (IOException e){
                e.printStackTrace();
            }
        }
    }

大文件上传

参考:https://blog.youkuaiyun.com/u014150463/article/details/74044467(包括断点续传，秒传，本人只是用了大文件分片上传，以下代码做了部分修改)

前端代码

使用的是WebUploader组件(https://fex.baidu.com/webuploader/doc/index.html)

import React from 'react';
import { connect } from 'dva';
import { Button,message,Modal} from 'antd';
import WebUploader from 'webuploader'
import styles from './webuploader.less'
import $ from 'jquery'
// 实例化

class ModalContent extends React.Component {


  componentDidMount(){
    debugger
    var $btn = $('#ctlBtn');
    var chunkSize = 5 * 1024 * 1024;
    var uploader = WebUploader.create({
      pick: {
        id: '#picker',
        label:'请选择文件',
        title:''
      },
      formData: {
        uid: 0,
        md5: '',
        chunkSize: chunkSize
      },
      chunked: true,
      chunkSize: chunkSize, // 字节 5M分块
      threads: 1,
      server: 'http://localhost:9090/index/fileUpload',
      auto: false,
      // 禁掉全局的拖拽功能。这样不会出现图片拖进页面的时候，把图片打开。
      disableGlobalDnd: true,
      fileNumLimit: 1024,//验证文件总数量, 超出则不允许加入队列
      fileSizeLimit: 1024 * 1024 * 1024,    // 验证文件总大小是否超出限制, 超出则不允许加入队列
      fileSingleSizeLimit: 1024 * 1024 * 1024    // 验证单个文件大小是否超出限制, 超出则不允许加入队列
    });

//当某个文件的分块在发送前触发，主要用来询问是否要添加附带参数，大文件在开起分片上传的前提下此事件可能会触发多次。
    uploader.onUploadBeforeSend = function (obj, data) {
      var file = obj.file;
      data.uid = file.uid;
    };
    uploader.on('uploadError', function (file) {
      $('#' + file.id).find('p.state').text('上传出错');
    });
// 文件上传
    $btn.on('click', function () {
      uploader.upload();
    });
  }
  render() {
    return (
      <div  className="wu-example">
        <div id="thelist" className="uploader-list"></div>
        <div className="btns">
          <div id="picker"/>
          {/*<Button id='ctlBtn' className="btn btn-default">开始上传</Button>*/}
        </div>
      </div>
    );
  }
}
//将需要的state的项注入到与此视图数据相关的组件props上
function mapStateToProps({ example }) {
  return { example }
}
export default connect(mapStateToProps)(ModalContent);

[class ^= webuploader-container] {
	position: relative;
}
[class ^= webuploader-element-invisible] {
	position: absolute !important;
	clip: rect(1px 1px 1px 1px); /* IE6, IE7 */
    clip: rect(1px,1px,1px,1px);
}
[class ^= webuploader-pick] {
	position: relative;
	display: inline-block;
	cursor: pointer;
	background: #00b7ee;
	padding: 10px 15px;
	color: #fff;
	text-align: center;
	border-radius: 3px;
	overflow: hidden;
}
[class ^= webuploader-pick-hover] {
	background: #00a2d4;
}

[class ^= webuploader-pick-disable] {
	opacity: 0.6;
	pointer-events:none;
}

后端代码

package win.pangniu.learn.controller;

import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.io.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;
import win.pangniu.learn.param.MultipartFileParam;
import win.pangniu.learn.service.StorageService;
import win.pangniu.learn.utils.Constants;
import win.pangniu.learn.vo.ResultStatus;
import win.pangniu.learn.vo.ResultVo;

import javax.servlet.http.HttpServletRequest;
import java.io.File;
import java.io.IOException;
import java.util.LinkedList;
import java.util.List;

/**
 * 默认控制层
 * Created by wenwen on 2017/4/11.
 * version 1.0
 */
@Controller
@RequestMapping(value = "/index")
public class IndexController {

    private Logger logger = LoggerFactory.getLogger(IndexController.class);

    @Autowired
    private StringRedisTemplate stringRedisTemplate;

    @Autowired
    private StorageService storageService;

    /**
     * 秒传判断，断点判断
     *
     * @return
     */
//    @RequestMapping(value = "checkFileMd5", method = RequestMethod.POST)
//    @ResponseBody
//    public Object checkFileMd5(String md5) throws IOException {
//        Object processingObj = stringRedisTemplate.opsForHash().get(Constants.FILE_UPLOAD_STATUS, md5);//根据MD5值和上传进度判断文件是否已上传
//        if (processingObj == null) {
//            return new ResultVo(ResultStatus.NO_HAVE);//没上传
//        }
//        String processingStr = processingObj.toString();
//        boolean processing = Boolean.parseBoolean(processingStr);
//        String value = stringRedisTemplate.opsForValue().get(Constants.FILE_MD5_KEY + md5);
//        if (processing) {
//            return new ResultVo(ResultStatus.IS_HAVE, value);//已上传
//        } else {
//            File confFile = new File(value);
//            byte[] completeList = FileUtils.readFileToByteArray(confFile);
//            List<String> missChunkList = new LinkedList<>();
//            for (int i = 0; i < completeList.length; i++) {
//                if (completeList[i] != Byte.MAX_VALUE) {
//                    missChunkList.add(i + "");
//                }
//            }
//            return new ResultVo<>(ResultStatus.ING_HAVE, missChunkList);//上传中
//        }
//    }

    /**
     * 上传文件
     *
     * @param param
     * @param request
     * @return
     * @throws Exception
     */
    @RequestMapping(value = "/fileUpload", method = RequestMethod.POST)
    @ResponseBody
    public ResponseEntity fileUpload(MultipartFileParam param, HttpServletRequest request) {
        boolean isMultipart = ServletFileUpload.isMultipartContent(request);
        if (isMultipart) {
            logger.info("上传文件start。");
            try {
                // 方法1
                //storageService.uploadFileRandomAccessFile(param);
                // 方法2 这个更快点
                storageService.uploadFileByMappedByteBuffer(param);
            } catch (IOException e) {
                e.printStackTrace();
                logger.error("文件上传失败。{}", param.toString());
            }
            logger.info("上传文件end。");
        }
        return ResponseEntity.ok().body("上传成功。");
    }


}

package win.pangniu.learn.param;


import org.springframework.web.multipart.MultipartFile;

/**
 * Created by wenwen on 2017/4/16.
 * version 1.0
 */
public class MultipartFileParam {

    // 用户id
    private String uid;
    //任务ID
    private String id;
    //总分片数量
    private int chunks;
    //当前为第几块分片
    private int chunk;
    //当前分片大小
    private long size = 0L;
    //文件名
    private String name;
    //分片对象
    private MultipartFile file;
    // MD5
    private String md5;

    public String getUid() {
        return uid;
    }

    public void setUid(String uid) {
        this.uid = uid;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public int getChunks() {
        return chunks;
    }

    public void setChunks(int chunks) {
        this.chunks = chunks;
    }

    public int getChunk() {
        return chunk;
    }

    public void setChunk(int chunk) {
        this.chunk = chunk;
    }

    public long getSize() {
        return size;
    }

    public void setSize(long size) {
        this.size = size;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public MultipartFile getFile() {
        return file;
    }

    public void setFile(MultipartFile file) {
        this.file = file;
    }

    public String getMd5() {
        return md5;
    }

    public void setMd5(String md5) {
        this.md5 = md5;
    }

    @Override
    public String toString() {
        return "MultipartFileParam{" +
                "uid='" + uid + '\'' +
                ", id='" + id + '\'' +
                ", chunks=" + chunks +
                ", chunk=" + chunk +
                ", size=" + size +
                ", name='" + name + '\'' +
                ", file=" + file +
                ", md5='" + md5 + '\'' +
                '}';
    }
}

package win.pangniu.learn.service;

import win.pangniu.learn.param.MultipartFileParam;

import java.io.IOException;

/**
 * 存储操作的service
 * Created by 超文 on 2017/5/2.
 */
public interface StorageService {

    /**
     * 删除全部数据
     */
    void deleteAll();

    /**
     * 初始化方法
     */
    void init();

    /**
     * 上传文件方法1
     *
     * @param param
     * @throws IOException
     */
    void uploadFileRandomAccessFile(MultipartFileParam param) throws IOException;

    /**
     * 上传文件方法2
     * 处理文件分块，基于MappedByteBuffer来实现文件的保存
     *
     * @param param
     * @throws IOException
     */
    void uploadFileByMappedByteBuffer(MultipartFileParam param) throws IOException;

}
package win.pangniu.learn.service.impl;

import org.apache.commons.io.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import org.springframework.util.FileSystemUtils;
import win.pangniu.learn.param.MultipartFileParam;
import win.pangniu.learn.service.StorageService;
import win.pangniu.learn.utils.Constants;
import win.pangniu.learn.utils.FileMD5Util;

import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.file.FileAlreadyExistsException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

/**
 * Created by 超文 on 2017/5/2.
 */
@Service
public class StorageServiceImpl implements StorageService {

    private final Logger logger = LoggerFactory.getLogger(StorageServiceImpl.class);
    // 保存文件的根目录
    private Path rootPath;

//    @Autowired
//    private StringRedisTemplate stringRedisTemplate;

    //这个必须与前端设定的值一致
    @Value("${breakpoint.upload.chunkSize}")
    private long CHUNK_SIZE;

    @Value("${breakpoint.upload.dir}")
    private String finalDirPath;

    @Autowired
    public StorageServiceImpl(@Value("${breakpoint.upload.dir}") String location) {
        this.rootPath = Paths.get(location);
    }

    @Override
    public void deleteAll() {
        logger.info("开发初始化清理数据，start");
        FileSystemUtils.deleteRecursively(rootPath.toFile());
//        stringRedisTemplate.delete(Constants.FILE_UPLOAD_STATUS);
//        stringRedisTemplate.delete(Constants.FILE_MD5_KEY);
        logger.info("开发初始化清理数据，end");
    }

    @Override
    public void init() {
        try {
            Files.createDirectory(rootPath);
        } catch (FileAlreadyExistsException e) {
            logger.error("文件夹已经存在了，不用再创建。");
        } catch (IOException e) {
            logger.error("初始化root文件夹失败。", e);
        }
    }

    @Override
    public void uploadFileRandomAccessFile(MultipartFileParam param) throws IOException {
        String fileName = param.getName();
        String tempDirPath = finalDirPath + param.getMd5();
        String tempFileName = fileName + "_tmp";
        File tmpDir = new File(tempDirPath);
        File tmpFile = new File(tempDirPath, tempFileName);
        if (!tmpDir.exists()) {
            tmpDir.mkdirs();
        }

        RandomAccessFile accessTmpFile = new RandomAccessFile(tmpFile, "rw");
        long offset = CHUNK_SIZE * param.getChunk();
        //定位到该分片的偏移量
        accessTmpFile.seek(offset);
        //写入该分片数据
        accessTmpFile.write(param.getFile().getBytes());
        // 释放
        accessTmpFile.close();

        boolean isOk = checkAndSetUploadProgress(param, tempDirPath);
        if (isOk) {
            boolean flag = renameFile(tmpFile, fileName);
            System.out.println("upload complete !!" + flag + " name=" + fileName);
        }
    }

    @Override
    public void uploadFileByMappedByteBuffer(MultipartFileParam param) throws IOException {
        String fileName = param.getName();
        String uploadDirPath = finalDirPath + param.getMd5();
        String tempFileName = fileName + "_tmp";
        File tmpDir = new File(uploadDirPath);
        File tmpFile = new File(uploadDirPath, tempFileName);
        if (!tmpDir.exists()) {
            tmpDir.mkdirs();
        }

        RandomAccessFile tempRaf = new RandomAccessFile(tmpFile, "rw");
        FileChannel fileChannel = tempRaf.getChannel();

        //写入该分片数据
        long offset = CHUNK_SIZE * param.getChunk();
        byte[] fileData = param.getFile().getBytes();
        MappedByteBuffer mappedByteBuffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, offset, fileData.length);
        mappedByteBuffer.put(fileData);
        // 释放
        FileMD5Util.freedMappedByteBuffer(mappedByteBuffer);
        fileChannel.close();

        boolean isOk = checkAndSetUploadProgress(param, uploadDirPath);
        if (isOk) {
            boolean flag = renameFile(tmpFile, fileName);
            System.out.println("upload complete !!" + flag + " name=" + fileName);
        }
    }

    /**
     * 检查并修改文件上传进度
     *
     * @param param
     * @param uploadDirPath
     * @return
     * @throws IOException
     */
    private boolean checkAndSetUploadProgress(MultipartFileParam param, String uploadDirPath) throws IOException {
        String fileName = param.getName();
        File confFile = new File(uploadDirPath, fileName + ".conf");
        RandomAccessFile accessConfFile = new RandomAccessFile(confFile, "rw");
        //把该分段标记为 true 表示完成
        System.out.println("set part " + param.getChunk() + " complete");
        accessConfFile.setLength(param.getChunks());//总分片数
        accessConfFile.seek(param.getChunk());//当前分片数
        accessConfFile.write(Byte.MAX_VALUE);

        //completeList 检查是否全部完成,如果数组里是否全部都是(全部分片都成功上传)
        byte[] completeList = FileUtils.readFileToByteArray(confFile);
        byte isComplete = Byte.MAX_VALUE;
        for (int i = 0; i < completeList.length && isComplete == Byte.MAX_VALUE; i++) {
            //与运算, 如果有部分没有完成则 isComplete 不是 Byte.MAX_VALUE
            isComplete = (byte) (isComplete & completeList[i]);
            System.out.println("check part " + i + " complete?:" + completeList[i]);
        }

        accessConfFile.close();
        if (isComplete == Byte.MAX_VALUE) {
            confFile.delete();//删除辅助文件
            return true;
        } else {
            return false;
        }
    }

    /**
     * 文件重命名
     *
     * @param toBeRenamed   将要修改名字的文件
     * @param toFileNewName 新的名字
     * @return
     */
    public boolean renameFile(File toBeRenamed, String toFileNewName) {
        //检查要重命名的文件是否存在，是否是文件
        if (!toBeRenamed.exists() || toBeRenamed.isDirectory()) {
            logger.info("File does not exist: " + toBeRenamed.getName());
            return false;
        }
        String p = toBeRenamed.getParent();
        File newFile = new File(p + File.separatorChar + toFileNewName);
        //修改文件名
        return toBeRenamed.renameTo(newFile);
    }

    public static void main(String[] args) {
        try{
            // create a new RandomAccessFile with filename test
            RandomAccessFile raf = new RandomAccessFile("d:/test.txt", "rw");
            // write something in the file
            raf.setLength(30);
            raf.seek(10);
            raf.writeUTF("12345689");
            raf.seek(18);
            raf.writeUTF("AAAA");
            // set the file pointer at 0 position
            // print the string
//            raf.seek(10);
            System.out.println("" + raf.readUTF());
            // print current length
            System.out.println("" + raf.length());
            // set the file length to 30
            // print the new length
            System.out.println("" + raf.length());
        }catch (Exception e){
            e.printStackTrace();
        }
    }

}

文件下载

普通文件下载

前端代码

//下载
function onDownLoadFile ({ key,fileName }){

  let obj = {
      method: 'post',
      url: config.fileApi.downloadFile + `?key=${key}?fileName=${fileName}`,
      responseType: "blob",
      headers: {
        //'Authorization': '121212',
      }
   }
    return Axios.create().request(obj).then(res=> {
      if (res.status == "200") {
        let evt = document.createEvent("MouseEvents");
        let a = document.createElement('a');
        a.href = URL.createObjectURL(res.data);
        a.download = fileName;
        evt.initEvent("click", true, true);     //注册单机事件
        a.dispatchEvent(evt);                    //元素绑定事件
      } else {
        message.error('请求失败');
      }
    }).catch(err => {
      let parseError = JSON.parse(JSON.stringify(err));
    })
}

注意: 这种下载方式发现了两个问题，

1、文件会先在界面缓存，完成后才会开始往本地下载文件，所以，用户点击下载后，界面可能没有反应

2、文件过大，导致下载时间过长，导致下载终端，后端异常:connect reset by peer(尝试了几种方式未解决)

后改为如下方式

由于前后端分离，所以直接访问会有跨域问题

其中添加a.target = '_Blank'可以模拟直接访问后端服务器下载，避免跨域问题，点击下载，可直接写入本地

function download({fileName,url}){
    let evt = document.createEvent("MouseEvent");
    let a = document.createElement("a");
    a.href = url;
    a.target = '_Blank';
    a.download = fileName;
    evt.initEvent("click",true,true);
    a.dispatchEvent(evt);
}

 

后端代码

@RequestMapping("/downLoadFile")
@ResponseBody
public void downLoadFile(HttpServletRequest request, HttpServletResponse response) {
    String name = request.getParameter("file");
    String path = "/file" + File.separator + name;
 
    File imageFile = new File(path);
    if (!imageFile.exists()) {
        return;
    }
 
    //下载的文件携带这个名称
    response.setHeader("Content-Disposition", "attachment;filename=" + name);
    //文件下载类型--二进制文件
    response.setContentType("application/octet-stream");
 
    try {
        FileInputStream fis = new FileInputStream(path);
        byte[] content = new byte[fis.available()];
        fis.read(content);
        fis.close();
 
        ServletOutputStream sos = response.getOutputStream();
        sos.write(content);
 
        sos.flush();
        sos.close();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

本段代码来自网络，中间部分也可写成循环，但本人经过测试大文件下载会内存溢出:

后改为分片下载(代码在内网，所以截图，请见谅)

开始下载大文件没问题，但是下载几KB的小文件有问题，发现少了flush操作

评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值