java使用minio上传大文件

最新推荐文章于 2025-11-02 14:54:29 发布

原创最新推荐文章于 2025-11-02 14:54:29 发布 · 1.5k 阅读

11 ·

CC 4.0 BY-SA版权

文章标签：

#java #开发语言

该文章已生成可运行项目，

简介

当上传大文件（如超过100MB）至 MinIO 对象存储时，推荐使用分片上传（Multipart Upload）方案。该方案具有以下优势：

支持断点续传
降低单次内存占用
提升网络传输稳定性
允许并行上传分片

前置条件

运行中的 MinIO 服务（本地或远程）
Java 8+ 开发环境
Maven/Gradle 项目

添加依赖

Maven 配置

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>8.5.7</version> 
</dependency>

Gradle 配置

implementation 'io.minio:minio:8.5.7'

初始化 MinIO 客户端

import io.minio.MinioClient;

public class MinioUploader {
    private static final String ENDPOINT = "http://localhost:9000";
    private static final String ACCESS_KEY = "your-access-key";
    private static final String SECRET_KEY = "your-secret-key";
    private static final String BUCKET_NAME = "uploads";

    private static MinioClient minioClient;

    public static void init() throws Exception {
        minioClient = MinioClient.builder()
                .endpoint(ENDPOINT)
                .credentials(ACCESS_KEY, SECRET_KEY)
                .build();
        
        // 确保存储桶存在
        if (!minioClient.bucketExists(BucketExistsArgs.builder().bucket(BUCKET_NAME).build())) {
            minioClient.makeBucket(MakeBucketArgs.builder().bucket(BUCKET_NAME).build());
        }
    }
}

分片上传流程

5.1 初始化分片上传

String objectName = "large-file.zip";
String uploadId = minioClient.createMultipartUpload(
        BUCKET_NAME,
        null,
        objectName,
        null,
        null
).result().uploadId();

5.2 分片文件上传

File file = new File("/path/to/large-file.zip");
long chunkSize = 50 * 1024 * 1024; // 50MB分片
int partNumber = 1;
Map<Integer, String> etags = new HashMap<>();

try (FileInputStream fis = new FileInputStream(file)) {
    byte[] buffer = new byte[(int) chunkSize];
    int bytesRead;
    
    while ((bytesRead = fis.read(buffer)) != -1) {
        ByteArrayInputStream partStream = new ByteArrayInputStream(buffer, 0, bytesRead);
        
        UploadPartResponse response = minioClient.uploadPart(
                BUCKET_NAME,
                null,
                objectName,
                uploadId,
                partNumber,
                partStream,
                bytesRead,
                null
        );
        
        etags.put(partNumber, response.etag());
        partNumber++;
    }
}

5.3 合并分片

List<Part> parts = new ArrayList<>();
for (int i = 1; i < partNumber; i++) {
    parts.add(new Part(i, etags.get(i)));
}

minioClient.completeMultipartUpload(
        BUCKET_NAME,
        null,
        objectName,
        uploadId,
        parts.toArray(new Part[0]),
        null,
        null
);

完整代码示例

public class MinioMultipartUploader {
    // 初始化代码...

    public static void uploadLargeFile(String filePath) throws Exception {
        init();
        File file = new File(filePath);
        String objectName = file.getName();
        
        // 初始化上传
        String uploadId = initializeMultipartUpload(objectName);
        
        // 分片上传
        Map<Integer, String> etags = uploadParts(objectName, uploadId, file);
        
        // 合并分片
        completeUpload(objectName, uploadId, etags);
    }

    private static String initializeMultipartUpload(String objectName) throws Exception {
        return minioClient.createMultipartUpload(
                BUCKET_NAME,
                null,
                objectName,
                null,
                null
        ).result().uploadId();
    }

    private static Map<Integer, String> uploadParts(String objectName, String uploadId, File file) throws Exception {
        // 分片上传实现...
    }

    private static void completeUpload(String objectName, String uploadId, Map<Integer, String> etags) throws Exception {
        // 合并实现...
    }
}

注意事项

分片大小：每个分片最小5MB（最后分片除外）
临时文件管理：建议使用临时目录存储分片
异常处理：
- 捕获所有 MinIO 异常
- 实现上传进度记录
- 添加重试机制

网络优化：

MinioClient.builder()
    .endpoint(endpoint)
    .credentials(accessKey, secretKey)
    .httpClient(HttpClient.newBuilder()
            .connectTimeout(Duration.ofMinutes(5))
            .build())
    .build();