突破上传限制：Bottle.py实现TUS协议大文件分片上传方案

突破上传限制：Bottle.py实现TUS协议大文件分片上传方案

【免费下载链接】bottle bottle.py is a fast and simple micro-framework for python web-applications. 项目地址: https://gitcode.com/gh_mirrors/bo/bottle

引言：大文件上传的技术痛点与解决方案

你是否遇到过浏览器上传2GB以上文件时进度条停滞的问题？是否因网络不稳定导致几小时的上传前功尽弃？传统的multipart/form-data上传方式在处理大文件时面临三大核心痛点：内存溢出风险、断点续传缺失、上传状态不可控。本文将详细介绍如何基于Bottle.py轻量级框架，实现TUS协议（HTTP-based Resumable File Upload Protocol）的分片上传方案，彻底解决这些问题。

读完本文你将获得：

• 理解TUS协议核心规范与断点续传原理
• 掌握Bottle.py中Request对象与FileUpload类的高级用法
• 实现支持暂停/继续、断点续传的分片上传系统
• 代码级优化大文件处理性能的实战技巧

TUS协议核心规范与工作原理

TUS（Temporary Upload Storage）协议是一个基于HTTP的可恢复文件上传开放标准，目前最新版本为1.0.0。其核心设计目标是解决大文件上传场景中的可靠性问题，主要通过以下机制实现：

核心协议字段

头部字段	作用	示例值
Tus-Resumable	声明协议版本	1.0.0
Upload-Length	文件总大小（字节）	4529840576
Upload-Offset	当前上传偏移量	104857600
Upload-ID	唯一上传标识	a1b2c3d4-e5f6-4a5b-9c8d-7e6f5a4b3c2d
Content-Type	分片数据类型	application/offset+octet-stream

协议交互流程

Bottle.py实现TUS协议的技术准备

环境依赖与项目结构

Bottle.py作为单文件微框架，本身已包含处理HTTP请求和文件上传的核心组件。实现TUS协议需要以下准备：

bottle_tus/
├── app.py               # 主应用入口
├── tus_handler.py       # TUS协议处理器
├── storage/             # 存储模块
│   ├── __init__.py
│   ├── local_storage.py # 本地文件存储实现
│   └── redis_state.py   # Redis状态管理
├── config.py            # 配置参数
└── tests/               # 单元测试
    ├── test_tus.py
    └── test_storage.py

关键依赖安装

pip install bottle redis python-multipart

核心实现：Bottle.py TUS服务器

1. 初始化Bottle应用与路由配置

from bottle import Bottle, request, response, abort, hook
import json
from tus_handler import TusHandler
from storage.local_storage import LocalStorage
from storage.redis_state import RedisStateManager

app = Bottle()
state_manager = RedisStateManager(host='localhost', port=6379, db=0)
storage = LocalStorage(base_path='/data/uploads', state_manager=state_manager)
tus_handler = TusHandler(storage=storage)

# TUS协议路由
app.route('/uploads', method='OPTIONS', callback=tus_handler.handle_options)
app.route('/uploads', method='POST', callback=tus_handler.handle_create)
app.route('/uploads/<upload_id>', method='PATCH', callback=tus_handler.handle_patch)
app.route('/uploads/<upload_id>', method='HEAD', callback=tus_handler.handle_head)
app.route('/uploads/<upload_id>', method='DELETE', callback=tus_handler.handle_delete)

# CORS支持
@app.hook('after_request')
def enable_cors():
    response.headers['Access-Control-Allow-Origin'] = '*'
    response.headers['Access-Control-Allow-Methods'] = 'OPTIONS, POST, PATCH, HEAD, DELETE'
    response.headers['Access-Control-Allow-Headers'] = 'Tus-Resumable, Upload-Length, Upload-Offset, Content-Type, Upload-ID'

2. TUS协议处理器实现

核心的TusHandler类需要实现协议定义的所有方法，下面重点介绍几个关键处理函数：

import uuid
import os
from bottle import request, response, abort

class TusHandler:
    def __init__(self, storage):
        self.storage = storage
        self.supported_versions = "1.0.0"
        self.max_chunk_size = 5 * 1024 * 1024  # 5MB默认分片大小

    def handle_options(self):
        """处理OPTIONS请求，声明TUS支持"""
        response.headers['Tus-Resumable'] = self.supported_versions
        response.headers['Tus-Version'] = self.supported_versions
        response.headers['Tus-Max-Size'] = str(5 * 1024 * 1024 * 1024)  # 5GB
        response.status = 204
        return

    def handle_create(self):
        """创建新的上传会话"""
        if request.headers.get('Tus-Resumable') != self.supported_versions:
            abort(412, "Unsupported Tus version")
            
        try:
            upload_length = int(request.headers.get('Upload-Length', 0))
            if upload_length <= 0:
                abort(400, "Invalid Upload-Length")
                
            upload_id = str(uuid.uuid4())
            self.storage.create_upload(upload_id, upload_length)
            
            response.status = 201
            response.headers['Tus-Resumable'] = self.supported_versions
            response.headers['Upload-ID'] = upload_id
            response.headers['Location'] = f"/uploads/{upload_id}"
            return
        except Exception as e:
            abort(400, str(e))

    def handle_patch(self, upload_id):
        """处理分片上传数据"""
        if request.headers.get('Tus-Resumable') != self.supported_versions:
            abort(412, "Unsupported Tus version")
            
        try:
            upload_offset = int(request.headers.get('Upload-Offset', 0))
            content_length = int(request.headers.get('Content-Length', 0))
            
            if content_length <= 0 or content_length > self.max_chunk_size:
                abort(413, f"Chunk size exceeds maximum {self.max_chunk_size} bytes")
                
            # 获取分片数据
            chunk_data = request.body.read(content_length)
            
            # 存储分片数据并更新偏移量
            new_offset = self.storage.append_chunk(
                upload_id, 
                chunk_data, 
                upload_offset
            )
            
            response.status = 204
            response.headers['Tus-Resumable'] = self.supported_versions
            response.headers['Upload-Offset'] = str(new_offset)
            return
        except Exception as e:
            abort(400, str(e))

3. 存储系统实现

Bottle.py的FileUpload类提供了基础的文件处理能力，我们需要扩展它来支持TUS协议的分片存储需求：

import os
import json
import redis
from bottle import abort

class LocalStorage:
    def __init__(self, base_path, state_manager):
        self.base_path = base_path
        self.state_manager = state_manager
        os.makedirs(base_path, exist_ok=True)
        
    def get_upload_path(self, upload_id):
        """获取上传文件的存储路径"""
        return os.path.join(self.base_path, upload_id)
    
    def create_upload(self, upload_id, total_length):
        """创建新的上传记录"""
        upload_path = self.get_upload_path(upload_id)
        
        # 初始化空文件
        with open(upload_path, 'wb') as f:
            pass
            
        # 存储上传元数据
        self.state_manager.set_state(
            upload_id, 
            {
                'total_length': total_length,
                'offset': 0,
                'created_at': time.time(),
                'status': 'in_progress'
            }
        )
        
    def append_chunk(self, upload_id, chunk_data, offset):
        """追加分片数据到文件"""
        upload_path = self.get_upload_path(upload_id)
        state = self.state_manager.get_state(upload_id)
        
        if not state:
            abort(404, "Upload not found")
            
        if offset != state['offset']:
            abort(409, f"Offset mismatch. Expected {state['offset']}")
            
        # 写入分片数据
        with open(upload_path, 'r+b') as f:
            f.seek(offset)
            f.write(chunk_data)
            
        # 更新状态
        new_offset = offset + len(chunk_data)
        state['offset'] = new_offset
        
        # 检查是否上传完成
        if new_offset >= state['total_length']:
            state['status'] = 'completed'
            state['completed_at'] = time.time()
            
        self.state_manager.set_state(upload_id, state)
        return new_offset

4. Redis状态管理

使用Redis存储上传状态信息，支持分布式部署和快速状态查询：

import redis
import json

class RedisStateManager:
    def __init__(self, host='localhost', port=6379, db=0, prefix='tus:'):
        self.redis = redis.Redis(host=host, port=port, db=db)
        self.prefix = prefix
        
    def _get_key(self, upload_id):
        return f"{self.prefix}{upload_id}"
        
    def set_state(self, upload_id, state):
        key = self._get_key(upload_id)
        self.redis.setex(key, 86400 * 7, json.dumps(state))  # 7天过期
        
    def get_state(self, upload_id):
        key = self._get_key(upload_id)
        data = self.redis.get(key)
        return json.loads(data) if data else None
        
    def delete_state(self, upload_id):
        key = self._get_key(upload_id)
        self.redis.delete(key)
        
    def list_uploads(self, status=None):
        keys = self.redis.keys(f"{self.prefix}*")
        uploads = []
        
        for key in keys:
            state = json.loads(self.redis.get(key))
            if not status or state.get('status') == status:
                upload_id = key.decode().replace(self.prefix, '')
                uploads.append({**state, 'upload_id': upload_id})
                
        return uploads

高级功能实现与性能优化

断点续传与幂等性保障

为确保上传过程的可靠性，需要处理网络中断、客户端崩溃等异常情况：

def handle_patch(self, upload_id):
    # ... 现有代码 ...
    
    # 添加幂等性检查
    current_state = self.state_manager.get_state(upload_id)
    if not current_state:
        abort(404, "Upload not found")
        
    # 检查偏移量是否匹配
    if offset != current_state['offset']:
        # 向客户端返回当前服务器偏移量
        response.headers['Upload-Offset'] = str(current_state['offset'])
        abort(409, f"Offset mismatch. Server has {current_state['offset']}")
        
    # ... 写入数据代码 ...

分片合并与校验

上传完成后需要对分片进行合并和校验，确保文件完整性：

def complete_upload(self, upload_id):
    """完成上传并验证文件完整性"""
    upload_path = self.get_upload_path(upload_id)
    state = self.state_manager.get_state(upload_id)
    
    if state['status'] == 'completed':
        # 可选：计算文件哈希进行校验
        file_hash = self._calculate_file_hash(upload_path)
        
        # 重命名临时文件
        final_path = os.path.join(self.base_path, f"{upload_id}.complete")
        os.rename(upload_path, final_path)
        
        return {
            'status': 'verified',
            'file_path': final_path,
            'file_hash': file_hash
        }
        
    return {'status': state['status']}
    
def _calculate_file_hash(self, file_path):
    """计算文件SHA-256哈希"""
    import hashlib
    hash_sha256 = hashlib.sha256()
    
    with open(file_path, 'rb') as f:
        for chunk in iter(lambda: f.read(4096), b''):
            hash_sha256.update(chunk)
            
    return hash_sha256.hexdigest()

并发控制与资源限制

为防止服务器资源耗尽，需要实现并发控制和上传限制：

def handle_patch(self, upload_id):
    # ... 现有代码 ...
    
    # 检查服务器负载
    current_load = self._get_server_load()
    if current_load > 0.8:  # 负载超过80%
        abort(503, "Server busy. Please try again later")
        
    # 限制每个客户端的并发上传数
    client_ip = request.remote_addr
    client_uploads = self.state_manager.count_client_uploads(client_ip)
    
    if client_uploads > 5:  # 每个客户端最多5个并发上传
        abort(429, "Too many concurrent uploads")
        
    # ... 处理上传代码 ...

客户端实现示例

以下是一个简单的JavaScript客户端实现，用于测试TUS协议上传：

class TusClient {
    constructor(endpoint, file) {
        this.endpoint = endpoint;
        this.file = file;
        this.uploadId = null;
        this.offset = 0;
        this.chunkSize = 5 * 1024 * 1024; // 5MB分片
    }
    
    async createUpload() {
        const response = await fetch(this.endpoint, {
            method: 'POST',
            headers: {
                'Tus-Resumable': '1.0.0',
                'Upload-Length': this.file.size.toString()
            }
        });
        
        if (response.status !== 201) {
            throw new Error(`Create upload failed: ${response.status}`);
        }
        
        this.uploadId = response.headers.get('Upload-ID');
        this.uploadUrl = response.headers.get('Location');
        return this.uploadId;
    }
    
    async uploadChunk() {
        const start = this.offset;
        const end = Math.min(start + this.chunkSize, this.file.size);
        const chunk = this.file.slice(start, end);
        
        const response = await fetch(this.uploadUrl, {
            method: 'PATCH',
            headers: {
                'Tus-Resumable': '1.0.0',
                'Upload-Offset': start.toString(),
                'Content-Type': 'application/offset+octet-stream'
            },
            body: chunk
        });
        
        if (response.status !== 204) {
            throw new Error(`Upload chunk failed: ${response.status}`);
        }
        
        this.offset = parseInt(response.headers.get('Upload-Offset'));
        return {
            offset: this.offset,
            completed: this.offset >= this.file.size
        };
    }
    
    async upload() {
        if (!this.uploadId) {
            await this.createUpload();
        }
        
        while (this.offset < this.file.size) {
            const result = await this.uploadChunk();
            console.log(`Uploaded ${this.offset}/${this.file.size} bytes`);
            
            if (result.completed) {
                console.log('Upload completed!');
                return true;
            }
        }
    }
}

// 使用示例
document.getElementById('file-input').addEventListener('change', function(e) {
    const file = e.target.files[0];
    if (file) {
        const client = new TusClient('/uploads', file);
        client.upload().catch(console.error);
    }
});

部署与扩展建议

生产环境配置

# 生产环境配置
app.config.update({
    'server': 'gunicorn',
    'host': '0.0.0.0',
    'port': 8080,
    'workers': 4,
    'worker_class': 'gevent',
    'max_requests': 1000,
    'timeout': 300  # 长连接超时设置
})

# 配置日志
import logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[logging.FileHandler('tus_server.log'), logging.StreamHandler()]
)

水平扩展方案

对于大规模部署，可采用以下扩展策略：

总结与最佳实践

本文详细介绍了如何基于Bottle.py实现TUS协议的大文件分片上传系统。关键要点包括：

1. 协议实现：完整实现了TUS协议的核心方法（OPTIONS、POST、PATCH、HEAD）
2. 存储设计：采用文件系统存储分片数据，Redis存储上传状态
3. 可靠性保障：实现了断点续传、偏移量校验、幂等性处理
4. 性能优化：支持并发上传、分片校验、资源限制

最佳实践建议

• 分片大小选择：根据网络环境选择合适的分片大小，一般建议5-10MB
• 状态管理：定期清理过期的未完成上传，释放存储空间
• 监控告警：实现上传进度监控和失败告警机制
• 安全措施：添加身份验证、权限控制和流量限制
• 客户端适配：提供多种语言的客户端SDK，方便集成

【免费下载链接】bottle bottle.py is a fast and simple micro-framework for python web-applications. 项目地址: https://gitcode.com/gh_mirrors/bo/bottle