大文件断点上传/下载总结
1. 背景与需求
在现代Web应用中，处理大文件的上传和下载是一个常见的需求。传统的文件上传和下载方式在处理大文件时可能会遇到性能瓶颈，如网络中断、服务器资源占用过高等问题。为了解决这些问题，断点上传和下载技术应运而生。断点上传和下载允许用户在网络中断或暂停后继续上传或下载文件，而不需要重新开始整个过程，从而提高了用户体验和系统稳定性。
2. 断点上传
2.1 实现原理
断点上传的核心思想是将大文件分割成多个小块（chunk），逐个上传这些小块，并在服务器端将这些小块合并成完整的文件。如果上传过程中出现中断，用户可以从中断的地方继续上传，而不需要重新上传整个文件。
2.2 关键技术点
- 文件分块：将大文件分割成固定大小的块，通常为几MB到几十MB。
- 分块上传：逐个上传这些块，并记录每个块的上传状态。
- 断点续传：在网络中断或用户暂停上传后，能够从中断的地方继续上传。
- 合并文件：所有块上传完成后，服务器将这些块合并成一个完整的文件。
2.3 实现细节
-
前端实现：
- 使用
File.slice方法将文件分割成块。 - 使用
FormData对象将每个块上传到服务器。 - 使用
AbortController实现暂停和恢复上传功能。 - 通过进度条实时显示上传进度。
- 使用
-
后端实现：
- 接收并保存每个上传的块。
- 在所有块上传完成后，将这些块合并成一个完整的文件。
- 提供接口用于查询文件是否已上传完成。
3. 断点下载
3.1 实现原理
断点下载的原理与断点上传类似，但方向相反。客户端从服务器下载文件时，将文件分割成多个小块，逐个下载这些小块，并在客户端将这些小块合并成完整的文件。如果下载过程中出现中断，用户可以从中断的地方继续下载。
3.2 关键技术点
- 文件分块：将大文件分割成固定大小的块，通常为几MB到几十MB。
- 分块下载：逐个下载这些块，并记录每个块的下载状态。
- 断点续传：在网络中断或用户暂停下载后，能够从中断的地方继续下载。
- 合并文件：所有块下载完成后，客户端将这些块合并成一个完整的文件。
3.3 实现细节
-
前端实现：
- 使用
fetchAPI进行分块下载，并设置Range请求头指定下载范围。 - 使用
AbortController实现暂停和恢复下载功能。 - 通过进度条实时显示下载进度。
- 下载完成后，将所有块合并成一个完整的文件并提供下载链接。
- 使用
-
后端实现：
- 支持
Range请求头，允许客户端指定下载范围。 - 提供文件的元数据（如文件大小），以便客户端计算下载进度。
- 支持
4. 总结
断点上传和下载技术在大文件处理中具有显著的优势，能够有效应对网络不稳定、服务器资源占用等问题。通过将大文件分割成小块，逐个处理这些小块，并在中断后能够从中断的地方继续处理，断点上传和下载技术极大地提高了文件传输的可靠性和用户体验。
在实现过程中，前端和后端需要紧密配合，确保文件分块、上传/下载、合并等操作的顺利进行。前端通过File.slice和fetch API实现文件的分块和下载，后端则通过支持Range请求头和文件合并操作来实现断点续传功能。
通过合理的设计和实现，断点上传和下载技术可以广泛应用于各种需要处理大文件的场景，如视频网站、云存储服务、在线办公系统等。
application.properties
file.upload.path=/tmp/upload/
package org.example.demoupload.utils;
import org.apache.commons.io.FileUtils;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.util.StringUtils;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.*;
import java.nio.file.Files;
import java.util.Objects;
@RestController
@RequestMapping("/api/file")
@CrossOrigin
public class FileController {
private static final String UPLOAD_DIR = "uploads";
private static final int BUFFER_SIZE = 8192; // 8KB buffer
public FileController() {
File dir = new File(UPLOAD_DIR);
if (!dir.exists()) {
dir.mkdirs();
}
}
@PostMapping("/upload")
public ResponseEntity<String> uploadFile(
@RequestParam("file") MultipartFile file,
@RequestParam("chunk") Integer chunk,
@RequestParam("chunks") Integer chunks,
@RequestParam("filename") String filename) throws IOException {
File targetFile = new File(UPLOAD_DIR + File.separator + filename);
File tempFile = new File(UPLOAD_DIR + File.separator + filename + ".tmp");
try (RandomAccessFile raf = new RandomAccessFile(tempFile, "rw")) {
long chunkSize = file.getSize();
raf.seek(chunk * chunkSize);
raf.write(file.getBytes());
}
if (chunk == chunks - 1) {
if (targetFile.exists()) {
FileUtils.deleteQuietly(targetFile);
}
if (!tempFile.renameTo(targetFile)) {
FileUtils.copyFile(tempFile, targetFile);
FileUtils.deleteQuietly(tempFile);
}
return ResponseEntity.ok("File uploaded successfully");
}
return ResponseEntity.ok("Chunk uploaded successfully");
}
@RequestMapping("/download/{filename}")
public void downloadFile(
@PathVariable String filename,
@RequestParam(required = false) Boolean inline,
HttpServletRequest request,
HttpServletResponse response) throws IOException {
File file = new File(UPLOAD_DIR + File.separator + filename);
if (!file.exists()) {
response.sendError(HttpServletResponse.SC_NOT_FOUND);
return;
}
// 获取文件的MIME类型
String mimeType = Files.probeContentType(file.toPath());
if (mimeType == null) {
mimeType = "application/octet-stream";
}
// 获取断点续传的Range信息
long fileLength = file.length();
long startByte = 0;
long endByte = fileLength - 1;
String range = request.getHeader("Range");
if (range != null && range.startsWith("bytes=")) {
String[] ranges = range.substring("bytes=".length()).split("-");
try {
startByte = Long.parseLong(ranges[0]);
if (ranges.length > 1) {
endByte = Long.parseLong(ranges[1]);
}
} catch (NumberFormatException e) {
response.setStatus(HttpServletResponse.SC_BAD_REQUEST);
return;
}
}
// 计算要传输的字节数
long contentLength = endByte - startByte + 1;
// 设置响应头
response.setContentType(mimeType);
response.setHeader("Accept-Ranges", "bytes");
response.setHeader("Content-Length", String.valueOf(contentLength));
// 如果是断点续传，设置206状态码
if (range != null) {
response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
response.setHeader("Content-Range", "bytes " + startByte + "-" + endByte + "/" + fileLength);
} else {
response.setStatus(HttpServletResponse.SC_OK);
}
// 设置文件下载方式（inline在浏览器中打开，attachment下载到本地）
String disposition = inline != null && inline ? "inline" : "attachment";
response.setHeader("Content-Disposition", disposition + ";filename=" + encodeFilename(filename, request));
// 使用带缓冲的流来传输文件
try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
BufferedOutputStream bos = new BufferedOutputStream(response.getOutputStream())) {
// 跳过已经下载的部分
bis.skip(startByte);
// 使用缓冲区读写文件
byte[] buffer = new byte[BUFFER_SIZE];
long remainingBytes = contentLength;
int bytesRead;
while (remainingBytes > 0 && (bytesRead = bis.read(buffer, 0, (int) Math.min(buffer.length, remainingBytes))) != -1) {
bos.write(buffer, 0, bytesRead);
remainingBytes -= bytesRead;
bos.flush(); // 实时刷新输出流
}
} catch (IOException e) {
// 客户端中断下载是正常的，不需要记录为错误
if (!e.getMessage().contains("Broken pipe") &&
!e.getMessage().contains("Connection reset by peer")) {
throw e;
}
}
}
/**
* 处理文件名编码，支持中文等特殊字符
*/
private String encodeFilename(String filename, HttpServletRequest request) {
String userAgent = request.getHeader("User-Agent");
try {
if (userAgent != null && userAgent.contains("MSIE")) {
// IE浏览器
filename = java.net.URLEncoder.encode(filename, "UTF-8");
} else if (userAgent != null && userAgent.contains("Mozilla")) {
// 火狐、谷歌等浏览器
filename = new String(filename.getBytes("UTF-8"), "ISO-8859-1");
}
} catch (UnsupportedEncodingException e) {
// 编码失败时使用原文件名
}
return filename;
}
@GetMapping("/verify")
public ResponseEntity<Boolean> verifyFileExist(@RequestParam("filename") String filename) {
File file = new File(UPLOAD_DIR + File.separator + filename);
return ResponseEntity.ok(file.exists());
}
}
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>文件上传下载系统</title>
<script src="https://cdn.jsdelivr.net/npm/vue@2.6.14/dist/vue.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/element-ui@2.15.13/lib/theme-chalk/index.css">
<script src="https://cdn.jsdelivr.net/npm/element-ui@2.15.13/lib/index.js"></script>
<style>
.upload-container {
width: 800px;
margin: 20px auto;
padding: 20px;
box-shadow: 0 0 10px rgba(0,0,0,0.1);
}
.progress-container {
margin-top: 20px;
}
.control-buttons {
margin-top: 10px;
}
.progress-with-size {
display: flex;
align-items: center;
gap: 10px;
}
.file-size {
white-space: nowrap;
min-width: 120px;
}
.download-section {
margin-top: 20px;
padding-top: 20px;
border-top: 1px solid #eee;
}
</style>
</head>
<body>
<div id="app">
<div class="upload-container">
<h2>文件上传下载系统</h2>
<!-- 上传区域 -->
<el-upload
ref="upload"
action="#"
:auto-upload="false"
:show-file-list="false"
:on-change="handleFileChange">
<el-button slot="trigger" type="primary">选择文件</el-button>
</el-upload>
<!-- 上传进度显示 -->
<div v-if="currentFile" class="progress-container">
<div>当前文件：{{ currentFile.name }}</div>
<div class="progress-with-size">
<el-progress :percentage="uploadProgress"></el-progress>
<span class="file-size">{{ formatFileSize(currentFile.size) }}</span>
</div>
<div class="control-buttons">
<el-button
type="primary"
@click="startUpload"
:disabled="uploading">
{{ uploadProgress === 100 ? '上传完成' : '开始上传' }}
</el-button>
<el-button
type="warning"
@click="pauseUpload"
v-if="uploading && !isPaused">
暂停上传
</el-button>
<el-button
type="success"
@click="resumeUpload"
v-if="isPaused">
继续上传
</el-button>
</div>
</div>
<!-- 下载区域 -->
<div v-if="uploadedFile" class="download-section">
<h3>最近上传文件</h3>
<div class="file-info">
<span>{{ uploadedFile.name }}</span>
<span class="file-size">{{ formatFileSize(uploadedFile.size) }}</span>
<template v-if="!isDownloading">
<el-button
type="primary"
size="small"
@click="downloadFile(uploadedFile)">
下载
</el-button>
</template>
<template v-else>
<div class="progress-with-size">
<el-button
type="warning"
size="small"
v-if="!isPausingDownload"
@click="pauseDownload">
暂停
</el-button>
<el-button
type="success"
size="small"
v-else
@click="resumeDownload">
继续
</el-button>
<el-progress
:percentage="downloadProgress"
:format="formatProgress"
style="width: 120px;">
</el-progress>
</div>
</template>
</div>
</div>
</div>
</div>
<script>
new Vue({
el: '#app',
data: {
// 上传相关
currentFile: null,
chunkSize: 2 * 1024 * 1024, // 2MB per chunk
chunks: [],
currentChunk: 0,
uploadProgress: 0,
uploading: false,
isPaused: false,
uploadController: null,
isUploading: false,
// 下载相关
uploadedFile: null,
downloadProgress: 0,
isDownloading: false,
isPausingDownload: false,
downloadController: null,
downloadedSize: 0,
totalSize: 0,
downloadChunks: [], // 存储已下载的块
currentDownloadFilename: null,
baseUrl: 'http://localhost:8080/api/file'
},
methods: {
formatFileSize(bytes) {
if (!bytes) return '0 B';
const k = 1024;
const sizes = ['B', 'KB', 'MB', 'GB', 'TB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return `${(bytes / Math.pow(k, i)).toFixed(2)} ${sizes[i]}`;
},
formatProgress(percentage) {
if (this.totalSize) {
return `${percentage}% (${this.formatFileSize(this.downloadedSize)}/${this.formatFileSize(this.totalSize)})`;
}
return `${percentage}%`;
},
handleFileChange(file) {
this.currentFile = file.raw;
this.chunks = [];
this.currentChunk = 0;
this.uploadProgress = 0;
this.isPaused = false;
this.prepareUpload();
},
prepareUpload() {
const chunks = Math.ceil(this.currentFile.size / this.chunkSize);
this.chunks = Array.from({ length: chunks }, (_, index) => {
const start = index * this.chunkSize;
const end = Math.min(start + this.chunkSize, this.currentFile.size);
return this.currentFile.slice(start, end);
});
},
async startUpload() {
if (!this.currentFile) return;
this.uploading = true;
this.isPaused = false;
this.isUploading = true;
try {
await this.uploadChunks();
} catch (error) {
console.error('Upload failed:', error);
this.$message.error('上传失败：' + error.message);
}
},
pauseUpload() {
this.isPaused = true;
this.isUploading = false;
if (this.uploadController) {
this.uploadController.abort();
this.uploadController = null;
}
},
async resumeUpload() {
this.isPaused = false;
this.isUploading = true;
await this.uploadChunks();
},
async uploadChunks() {
while (this.currentChunk < this.chunks.length && !this.isPaused) {
try {
this.uploadController = new AbortController();
const formData = new FormData();
formData.append('file', this.chunks[this.currentChunk]);
formData.append('chunk', this.currentChunk);
formData.append('chunks', this.chunks.length);
formData.append('filename', this.currentFile.name);
const response = await fetch(`${this.baseUrl}/upload`, {
method: 'POST',
body: formData,
signal: this.uploadController.signal
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
this.currentChunk++;
this.uploadProgress = Math.round((this.currentChunk / this.chunks.length) * 100);
if (this.currentChunk === this.chunks.length) {
this.uploading = false;
this.isUploading = false;
this.uploadedFile = {
name: this.currentFile.name,
size: this.currentFile.size
};
this.$message.success('上传完成！');
}
} catch (error) {
if (error.name === 'AbortError') {
console.log('Upload paused');
return;
}
this.uploading = false;
this.isUploading = false;
throw error;
}
}
},
// 下载相关方法
async downloadFile(file) {
if (this.isDownloading) {
this.$message.warning('已有下载任务正在进行中');
return;
}
try {
this.isDownloading = true;
this.downloadProgress = 0;
this.downloadedSize = 0;
this.isPausingDownload = false;
this.downloadChunks = [];
this.currentDownloadFilename = file.name;
// 获取文件大小
const headResponse = await fetch(`${this.baseUrl}/download/${file.name}`, {
method: 'HEAD'
});
if (!headResponse.ok) {
throw new Error(`Failed to get file size: ${headResponse.status}`);
}
this.totalSize = parseInt(headResponse.headers.get('Content-Length') || 0);
await this.startChunkedDownload();
} catch (error) {
console.error('Download error:', error);
this.$message.error('下载失败：' + error.message);
this.resetDownload();
}
},
resetDownload() {
this.isDownloading = false;
this.isPausingDownload = false;
this.downloadController = null;
this.downloadProgress = 0;
this.downloadedSize = 0;
this.downloadChunks = [];
this.currentDownloadFilename = null;
},
async startChunkedDownload() {
const chunkSize = 1024 * 1024; // 1MB per chunk
try {
while (this.downloadedSize < this.totalSize && !this.isPausingDownload) {
const end = Math.min(this.downloadedSize + chunkSize - 1, this.totalSize - 1);
this.downloadController = new AbortController();
const response = await fetch(`${this.baseUrl}/download/${this.currentDownloadFilename}`, {
headers: {
Range: `bytes=${this.downloadedSize}-${end}`
},
signal: this.downloadController.signal
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const blob = await response.blob();
this.downloadChunks.push(blob);
this.downloadedSize = end + 1;
this.downloadProgress = Math.round((this.downloadedSize / this.totalSize) * 100);
}
if (!this.isPausingDownload) {
await this.completeDownload();
}
} catch (error) {
if (error.name === 'AbortError') {
console.log('Download paused');
} else {
throw error;
}
}
},
async completeDownload() {
const blob = new Blob(this.downloadChunks, { type: 'application/octet-stream' });
const url = window.URL.createObjectURL(blob);
const link = document.createElement('a');
link.href = url;
link.download = this.currentDownloadFilename;
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
window.URL.revokeObjectURL(url);
this.resetDownload();
this.$message.success('下载完成！');
},
pauseDownload() {
if (this.downloadController) {
this.isPausingDownload = true;
this.downloadController.abort();
this.downloadController = null;
this.$message.info('已暂停下载');
}
},
async resumeDownload() {
if (this.downloadedSize < this.totalSize) {
this.isPausingDownload = false;
await this.startChunkedDownload();
}
}
}
});
</script>
</body>
</html>
扫一扫
209
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
