libcurl实现多线程文件下载(Win32, C++)

原创 已于 2024-03-29 15:05:22 修改 · 917 阅读 · 1 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#windows #Win32 #C++ #libcurl

于 2024-03-29 11:34:36 首次发布
博客提及了CCurlDownload.h、CCurlDownload.cpp、CCurlDownloadMgr.h、CCurlDownloadMgr.cpp等文件，结合标签可知，这些文件可能是基于C++和libcurl库在Windows系统下开发的下载模块相关文件。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

CCurlDownload.h

#pragma once
#include <string>
#include <vector>
#include <atomic>
#include <functional>
#include <map>
#include "curl/include/curl/curl.h"
#include <windows.h>

#ifdef _UNICODE
using _tstring = std::wstring;
#else
using _tstring = std::string;
#endif

//请求头信息
using CURL_HEADER_INFO = std::map<std::string, std::string>;

class CCurlDownload;
//单个文件下载
class CCurlDownload
{
public:
    CCurlDownload();

    CCurlDownload(const CCurlDownload& obj);

    CCurlDownload(const std::string& strUrl, const std::string& strPath, int nConcurrency = 6);

    ~CCurlDownload();

    //获取内容数据大小
    size_t GetContentLength(const std::string& strUrl, CURL_HEADER_INFO& headerInfo);
    size_t GetContentLength(const std::string& strUrl);

    //设置下载信息
    void SetDownloadInfo(const std::string& strUrl, const std::string& strPath, int nConcurrency = 6);

    // 同步下载文件
    bool SyncDownload(std::function<void(long long dlNow, long long dlTotal, double lfProgress)> cb = nullptr);

    // 异步下载文件
    void AsyncDownload(std::function<void(long long dlNow, long long dlTotal, double lfProgress)> cb = nullptr);

    // 取消下载
    void CancelDownload();

    // 等待下载结束
    void WaitForDownload() const;

    // 是否下载中
    bool IsDownloading() const;

    // 获取上次下载结果
    bool GetLastDownloadResult() const;

    long long GetDownloadingSize() const;

    long long GetDownloadingTotal() const;

private:

    typedef struct _PackagePart PackagePart;

    bool _IsSupportAcceptRanges(const CURL_HEADER_INFO& headerInfo);
         
    bool _InitCurlInfo(PackagePart *pPackageInfo, const std::string& strUrl);

    // 分段多线程下载文件
    bool _DownloadByParts(long long nTotalSize, long long packageSize, int nConcurrency);

    // 下载前清空文件
    void _TruncateFile(const std::string& strPath);

    // 进行数据传输
    bool _TransferData(PackagePart* pPackageInfo);

    // 下载前备份文件
    bool _BackupFile(const std::string& strPath);

    // 头部写入回调
    static size_t _HeaderFunc(char* buffer, size_t size, size_t nitems, void* pUserData);

    // 文件写入回调
    static size_t _WriteFunc(void* pData, size_t size, size_t nmemb, void* pUserData);

    // 下载进度回调
    static int _ProgressFunc(void* pUserData, curl_off_t dlTotal, curl_off_t dlNow, curl_off_t ulTotal, curl_off_t ulNow);

    // 文件写入
    size_t _WriteFile(const void* pBuffer, DWORD size, PackagePart* pPackageInfo);

    // 下载进度
    int _ProgressProc(PackagePart* pPackageInfo, long long dlTotal, long long dlNow);

    // 休眠
    void _SleepMillisecond(int millisecond) const;

private:
    std::string m_strUrl;                           //下载链接地址
    std::string m_strPath;                          //文件存放地址
    std::vector<PackagePart*> m_packageParts;       //分包下载信息
    std::function<void(long long dlNow, long long dlTotal, double lfProgress)> m_cbProgress;     //下载进度回调
    std::atomic<int> m_threadCount;                 //下载线程数量
    int m_nConcurrency;                             //下载线程数
    bool m_bCancel;                                 //是否取消
    bool m_bDownloading;                            //是否下载中
    bool m_bDownloadSuccess;                        //是否下载成功
    long long m_lastDownload;                       //下载大小
    long long m_lastTotal;                          //总共大小
};

CCurlDownload.cpp

#include "CCurlDownload.h"
#include "ThreadPool.h"
#include <tchar.h>
#include <future>
#include <thread>

#define MINIMUM_PACKAGE_SIZE        (16384) // 最小包大小
#define BACKUP_MAX_CONUT            (0)     // 最多备份数量

#pragma comment(lib, "curl/lib/libcurl.lib")

typedef struct _PackagePart
{
    bool bSuccess;          //下载是否成功
    CCurlDownload* pThis;   //对象指针
    int partId;             //分段ID
    HANDLE hFile;           //文件句柄
    long long beginPos;     //起始偏移
    long long endPos;       //结束偏移
    long long downloadSize; //已下载大小
    long long totalSize;    //包数据总大小
    void* curl;             //

    _PackagePart() : hFile(INVALID_HANDLE_VALUE)
    {
        curl = NULL;
        partId = 0;
        beginPos = 0;
        endPos = 0;
        downloadSize = 0;
        totalSize = 0;
        bSuccess = false;
        pThis = nullptr;
    }
}PackagePart;

CCurlDownload::CCurlDownload()
    : m_threadCount(0)
{
    m_nConcurrency = 1;
    m_bCancel = false;
    m_bDownloading = false;
    m_lastDownload = 0;
    m_lastTotal = 0;
    m_bDownloadSuccess = false;
}

CCurlDownload::CCurlDownload(const CCurlDownload& obj)
    : m_strUrl(obj.m_strUrl), m_strPath(obj.m_strPath), m_threadCount(0)
{
    m_bCancel = false;
    m_bDownloading = false;
    m_lastDownload = 0;
    m_lastTotal = 0;
    m_bDownloadSuccess = false;
    m_nConcurrency = obj.m_nConcurrency;
}

CCurlDownload::CCurlDownload(const std::string& strUrl, const std::string& strPath, int nConcurrency)
    : m_strUrl(strUrl), m_strPath(strPath), m_threadCount(0)
{
    m_bCancel = false;
    m_bDownloading = false;
    m_lastDownload = 0;
    m_lastTotal = 0;
    m_bDownloadSuccess = false;
    m_nConcurrency = nConcurrency <= 0 ? 1 : nConcurrency;
}

CCurlDownload::~CCurlDownload()
{
    (void)CancelDownload();
}

void CCurlDownload::_SleepMillisecond(int millisecond) const
{
    const int span = 20;
    if (millisecond < span)
    {
        std::this_thread::sleep_for(std::chrono::milliseconds(millisecond));
        return;
    }

    clock_t tmBegin = clock();
    while (true)
    {
        if (clock() - tmBegin > millisecond)
        {
            break;
        }

        if (m_bCancel)
        {
            break;
        }

        std::this_thread::sleep_for(std::chrono::milliseconds(span));
    }
}

size_t CCurlDownload::GetContentLength(const std::string& strUrl, CURL_HEADER_INFO& headerInfo)
{
    CURL* handle = curl_easy_init();
    if (NULL == handle)
    {
        return (size_t)(0);
    }

    headerInfo.clear();
    
    curl_easy_setopt(handle, CURLOPT_URL, strUrl.c_str());
    curl_easy_setopt(handle, CURLOPT_HEADER, 1L);    // 需要将标头传递到数据流 
    
    // 设置头部写入回调
    curl_easy_setopt(handle, CURLOPT_HEADERFUNCTION, _HeaderFunc);
    curl_easy_setopt(handle, CURLOPT_HEADERDATA, (void*)&headerInfo);

    curl_easy_setopt(handle, CURLOPT_NOBODY, 1L);    // 不需要body  
    curl_easy_setopt(handle, CURLOPT_SSL_VERIFYPEER, 0L);
    curl_easy_setopt(handle, CURLOPT_SSL_VERIFYHOST, 0L);
    curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1L);
    curl_easy_setopt(handle, CURLOPT_VERBOSE, 0L);

    curl_off_t downloadFileLenth = 0;
    CURLcode curlCode = CURLE_OK;

    // 重试几次, 可能会出现超时CURLE_OPERATION_TIMEDOUT
    for (int i = 0; i < 10; i++)
    {
        curlCode = curl_easy_perform(handle);
        if (CURLE_OK == curlCode)
        {
            curlCode = curl_easy_getinfo(handle, CURLINFO_CONTENT_LENGTH_DOWNLOAD_T, &downloadFileLenth);
            long response_code = 0;
            curlCode = curl_easy_getinfo(handle, CURLINFO_RESPONSE_CODE, &response_code);
            if (response_code < 200 || response_code >= 300)
            {
                downloadFileLenth = 0;
            }

            break;
        }

        std::this_thread::sleep_for(std::chrono::milliseconds(50));
    }

    (void)curl_easy_cleanup(handle);

    return (size_t)downloadFileLenth;
}

size_t CCurlDownload::GetContentLength(const std::string& strUrl)
{
    CURL_HEADER_INFO headerInfo;
    return GetContentLength(strUrl, headerInfo);
}

void CCurlDownload::SetDownloadInfo(const std::string& strUrl, const std::string& strPath, int nConcurrency)
{
    (void)WaitForDownload();

    m_bCancel = false;
    m_strUrl = strUrl;
    m_strPath = strPath;
    m_bDownloading = false;
    m_lastDownload = 0;
    m_lastTotal = 0;

    if (nConcurrency <= 0)
    {
        nConcurrency = 1;
    }

    m_nConcurrency = nConcurrency;
}

bool CCurlDownload::_IsSupportAcceptRanges(const CURL_HEADER_INFO& headerInfo)
{
    auto itKey = headerInfo.find("Accept-Ranges");
    if (itKey != headerInfo.end())
    {
        if (itKey->second == "none")
        {
            return false;
        }

        // 检查是否支持分段下载(或断点续传)
        if (itKey->second == "bytes")
        {
            return true;
        }
    }

    return false;
}

bool CCurlDownload::_InitCurlInfo(PackagePart* pPackageInfo, const std::string& strUrl)
{
    if (nullptr == pPackageInfo)
    {
        return false;
    }

    CURL* curl = curl_easy_init();
    if (NULL == curl)
    {
        return false;
    }

    pPackageInfo->curl = curl;

    curl_easy_setopt(curl, CURLOPT_URL, strUrl.c_str());                            // 设置链接URL
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, _WriteFunc);                      // 设置文件写入回调
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, static_cast<void*>(pPackageInfo));    // 设置文件写入回调参数
    curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);                                 // 启用下载进度回调
    curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, _ProgressFunc);                // 设置下载进度回调
    curl_easy_setopt(curl, CURLOPT_XFERINFODATA, static_cast<void*>(pPackageInfo)); // 设置下载进度回调参数
    curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L);                                   // 指示 libcurl 不要使用任何信号/警报处理程序，即使在使用超时时时也是如此
    curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1L);                            // 以 字节/秒 为单位的低速度限制
    curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, 1L);                             // 低速限制时间段
    curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 60);                             // 设置连接超时秒数
    curl_easy_setopt(curl, CURLOPT_TIMEOUT, 60000);                                 // 设置允许传输完成的最长时间
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);                             // 忽略证书
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);                             // 设置我们是否应该从 ssl 中的对等证书中验证公用名
    curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);                             // 设置重定向
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 0L);                                    // 设置是否打印 libcurl 执行过程

    // 设置请求数据范围
    char range[MAX_PATH] = { 0 };

    // 未知大小不可断点续传, 失败就只能从头开始
    if ((size_t)(-1) == pPackageInfo->endPos)
    {
        pPackageInfo->beginPos = 0;
        curl_easy_setopt(curl, CURLOPT_RANGE, 0L);
        ::SetFilePointer(pPackageInfo->hFile, 0, 0, FILE_BEGIN);
        ::SetEndOfFile(pPackageInfo->hFile);
    }
    else
    {
        snprintf(range, sizeof(range), "%lld-%lld", pPackageInfo->beginPos, pPackageInfo->endPos);
        curl_easy_setopt(curl, CURLOPT_RANGE, range);
    }

    return true;
}

void CCurlDownload::_TruncateFile(const std::string& strPath)
{
    HANDLE hFile = INVALID_HANDLE_VALUE;
    hFile = ::CreateFileA(strPath.c_str(),
        GENERIC_READ | GENERIC_WRITE, FILE_SHARE_WRITE,
        NULL,
        TRUNCATE_EXISTING,
        FILE_ATTRIBUTE_ARCHIVE,
        NULL);

    if (INVALID_HANDLE_VALUE != hFile)
    {
        ::CloseHandle(hFile);
    }
}

bool CCurlDownload::_BackupFile(const std::string& strPath)
{
    char szBuf[MAX_PATH] = { 0 };
    std::string strBakFile;
    bool isBackupOk = false;

    if (0 == BACKUP_MAX_CONUT)
    {
        return true;
    }

    for (int i = 1; i <= BACKUP_MAX_CONUT; i++)
    {
        sprintf_s(szBuf, _countof(szBuf), "%02d", i);
        strBakFile = strPath + "." + szBuf + ".bak";
        if (::MoveFileA(strPath.c_str(), strBakFile.c_str()))
        {
            isBackupOk = true;
            break;
        }

        if (ERROR_FILE_NOT_FOUND == ::GetLastError())
        {
            isBackupOk = true;
            break;
        }
    }

    //备份失败, 则删除最早一个备份文件
    if (!isBackupOk)
    {
        std::string strOld;
        std::string strNew;

        sprintf_s(szBuf, _countof(szBuf), "%02d", 1);
        strOld = strPath + "." + szBuf + ".bak";

        // 无法删除
        if (!::DeleteFileA(strOld.c_str()))
        {
            return false;
        }

        for (int i = 1; i < BACKUP_MAX_CONUT; i++)
        {
            sprintf_s(szBuf, _countof(szBuf), "%02d", i);
            strNew = strPath + "." + szBuf + ".bak";;
            sprintf_s(szBuf, _countof(szBuf), "%02d", i + 1);
            strOld = strPath + "." + szBuf + ".bak";;
            if (!::MoveFileA(strOld.c_str(), strNew.c_str()))
            {
                break;
            }
        }

        sprintf_s(szBuf, _countof(szBuf), "%02d", BACKUP_MAX_CONUT);
        strBakFile = strPath + "." + szBuf + ".bak";
        if (::MoveFileA(strPath.c_str(), strBakFile.c_str()))
        {
            isBackupOk = true;
        }
    }

    return isBackupOk;
}

bool CCurlDownload::_DownloadByParts(long long nTotalSize, long long packageSize, int nConcurrency)
{
    std::string strTmpFile = m_strPath + ".tmp";
    bool bDownloadFinish = true;

    do
    {
        ThreadPool pool(nConcurrency);
        std::string strBakFile;

        // 备份文件
        if (!_BackupFile(m_strPath))
        {
            bDownloadFinish = false;
            break;
        }

        // 删除文件, 防止下载后改名冲突
        if (!::DeleteFileA(m_strPath.c_str()))
        {
            if (ERROR_FILE_NOT_FOUND != ::GetLastError())
            {
                bDownloadFinish = false;
                break;
            }
        }

        // 清空一下临时文件(如果存在的话)
        (void)_TruncateFile(strTmpFile);

        // 分包下载
        for (int i = 0; i < nConcurrency; i++)
        {
            HANDLE hFile = INVALID_HANDLE_VALUE;
            hFile = ::CreateFileA(strTmpFile.c_str(),
                GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE,
                NULL,
                OPEN_ALWAYS,
                FILE_ATTRIBUTE_ARCHIVE,
                NULL);

            if (INVALID_HANDLE_VALUE == hFile)
            {
                m_bCancel = true;
                break;
            }

            PackagePart* pPackageInfo = new (std::nothrow) PackagePart();
            if (NULL == pPackageInfo)
            {
                m_bCancel = true;
                break;
            }

            pPackageInfo->partId = i;
            pPackageInfo->hFile = hFile;
            pPackageInfo->pThis = this;

            if (1 == nConcurrency)
            {
                pPackageInfo->beginPos = 0;
                pPackageInfo->endPos = nTotalSize - 1;
                pPackageInfo->totalSize = nTotalSize;

                if ((size_t)(-1) == nTotalSize)
                {
                    pPackageInfo->endPos = nTotalSize;
                }
            }
            else
            {
                pPackageInfo->beginPos = i * packageSize;
                pPackageInfo->endPos = (i < m_nConcurrency - 1) ? ((i + 1) * packageSize - 1) : (nTotalSize - 1);
                pPackageInfo->totalSize = (pPackageInfo->endPos - pPackageInfo->beginPos) + 1;
            }

            LARGE_INTEGER liDistanceToMove = { 0 };
            liDistanceToMove.QuadPart = pPackageInfo->beginPos;

            // 设置文件写入指针位置
            ::SetFilePointerEx(pPackageInfo->hFile, liDistanceToMove, NULL, FILE_BEGIN);

            m_packageParts.push_back(pPackageInfo);

            // 工作函数
            auto workFunc = [this, pPackageInfo]() -> void {
                _TransferData(pPackageInfo);
                m_threadCount--;
            };

            try
            {
                pool.enqueue(workFunc);
            }
            catch (...)
            {
                _tprintf(_T("download excetion"));
            }

            m_threadCount++;
        }
    } while (false);

    // 等待下载线程结束
    while (m_threadCount)
    {
        _SleepMillisecond(100);
    }

    // 检查分段下载结果, 存在一个失败则认为全部是失败
    for (auto& item : m_packageParts)
    {
        if (!item->bSuccess)
        {
            bDownloadFinish = false;
        }
        delete item;
    }

    m_packageParts.clear();

    m_bDownloadSuccess = bDownloadFinish;
    m_bCancel = false;

    // 下载完成则修改文件名
    if (bDownloadFinish)
    {
        ::MoveFileA(strTmpFile.c_str(), m_strPath.c_str());
    }

    return bDownloadFinish;
}

bool CCurlDownload::_TransferData(PackagePart* pPackageInfo)
{
    // 重试30次
    for (int i = 0; i < 30 && !m_bCancel; i++)
    {
        if (!_InitCurlInfo(pPackageInfo, m_strUrl))
        {
            break;
        }

        CURLcode res = curl_easy_perform(pPackageInfo->curl);

        // 主动取消则不再尝试
        if (CURLE_ABORTED_BY_CALLBACK == res)
        {
            (void)curl_easy_cleanup(pPackageInfo->curl);
            break;
        }

        if (res == CURLE_OK)
        {
            long response_code = 0;
            curl_easy_getinfo(pPackageInfo->curl, CURLINFO_RESPONSE_CODE, &response_code);

            if (404 == response_code)
            {
                _tprintf(_T("curl_easy_getinfo failed rescode: %ld"), response_code);
                break;
            }
            else if (response_code < 200 || response_code > 300)
            {
                _tprintf(_T("curl_easy_getinfo failed rescode: %ld"), response_code);
            }
            else
            {
                (void)curl_easy_cleanup(pPackageInfo->curl);
                pPackageInfo->bSuccess = true;
                break;
            }
        }
        else
        {
            _tprintf(_T("curl_easy_perform failed res: %d: %S"), res, curl_easy_strerror(res));
        }

        (void)curl_easy_cleanup(pPackageInfo->curl);
        _SleepMillisecond(500);
        _tprintf(_T("retry: %d"), i + 1);
    }

    ::CloseHandle(pPackageInfo->hFile);

    return true;
}

bool CCurlDownload::SyncDownload(std::function<void(long long dlNow, long long dlTotal, double lfProgress)> cb)
{
    long long nPagePackageSize = 0;
    bool isSuccess = false;

    // 等待下载线程结束
    (void)WaitForDownload();

    CURL_HEADER_INFO headerInfo;

    // 获取内容大小
    m_lastTotal = GetContentLength(m_strUrl, headerInfo);

    // 返回0表示失败
    if (((size_t)0) == m_lastTotal)
    {
        return false;
    }

    int nConcurrency = m_nConcurrency;

    //不支持断点续传则仅单线程下载
    if (!_IsSupportAcceptRanges(headerInfo))
    {
        nConcurrency = 1;
    }

    //文件大小未知则仅单线程下载
    if ((size_t)(-1) == m_lastTotal)
    {
        nConcurrency = 1;
        nPagePackageSize = (size_t)(-1);
    }
    else
    {
        if (m_lastTotal <= MINIMUM_PACKAGE_SIZE)
        {
            nConcurrency = 1;
        }

        // 获取分包大小
        nPagePackageSize = m_lastTotal / nConcurrency;
    }

    m_cbProgress = cb;

    m_bDownloading = true;
    isSuccess = _DownloadByParts(m_lastTotal, nPagePackageSize, nConcurrency);
    m_bDownloading = false;

    return isSuccess;
}

// 异步下载文件
void CCurlDownload::AsyncDownload(std::function<void(long long dlNow, long long dlTotal, double lfProgress)> cb)
{
    // 先等待下载状态结束
    WaitForDownload();

    std::promise<bool> promiseReady;
    std::future<bool> futureReady(promiseReady.get_future());

    std::thread([this, cb](std::promise<bool>& fReady) -> void {

        size_t nPagePackageSize = 0;
        bool isSuccess = false;

        // 等待下载线程结束
        (void)WaitForDownload();

        // 获取内容大小
        CURL_HEADER_INFO headerInfo;
        m_lastTotal = GetContentLength(m_strUrl, headerInfo);
        if (((size_t)-1) == m_lastTotal)
        {
            fReady.set_value(true);
            return;
        }

        int nConcurrency = m_nConcurrency;
        //不支持断点续传则仅单线程下载
        if (!_IsSupportAcceptRanges(headerInfo))
        {
            nConcurrency = 1;
        }

        //文件大小未知则仅单线程下载
        if ((size_t)(-1) == m_lastTotal)
        {
            nConcurrency = 1;
            nPagePackageSize = (size_t)(-1);
        }
        else
        {
            if (m_lastTotal <= MINIMUM_PACKAGE_SIZE)
            {
                nConcurrency = 1;
            }

            // 获取分包大小
            nPagePackageSize = (size_t)(m_lastTotal / nConcurrency);
        }

        m_cbProgress = cb;

        m_bDownloading = true;
        fReady.set_value(true);
        isSuccess = _DownloadByParts(m_lastTotal, nPagePackageSize, nConcurrency);
        m_bDownloading = false;

        }, ref(promiseReady)).detach();

        futureReady.get();
}

bool CCurlDownload::IsDownloading() const
{
    return m_bDownloading;
}

bool CCurlDownload::GetLastDownloadResult() const
{
    return m_bDownloadSuccess;
}

void CCurlDownload::CancelDownload()
{
    m_bCancel = true;
    (void)WaitForDownload();
}

// 等待到空闲状态
void CCurlDownload::WaitForDownload() const
{
    // 等待下载线程结束
    while (m_bDownloading)
    {
        _SleepMillisecond(100);
    }
}

long long CCurlDownload::GetDownloadingSize() const
{
    return m_lastDownload;
}

long long CCurlDownload::GetDownloadingTotal() const
{
    return m_lastTotal;
}

size_t CCurlDownload::_HeaderFunc(char* buffer, size_t size, size_t nitems, void* pUserData)
{
    size_t realSize = size * nitems;

    std::map<std::string, std::string>* pHeaderContent = (std::map<std::string, std::string>*)pUserData;

    std::string strContent = buffer;

    size_t nPos = strContent.find(':');

    if (std::string::npos != nPos)
    {
        std::string strKey = strContent.substr(0, nPos);

        std::string strValue = strContent.substr(nPos + 2);
        size_t nReturnPos = strValue.rfind('\r');
        if (std::string::npos != nReturnPos)
        {
            strValue.resize(nReturnPos);
        }

        pHeaderContent->insert(std::make_pair(strKey, strValue));
    }

    return realSize;
}

size_t CCurlDownload::_WriteFunc(void* pBuffer, size_t size, size_t nmemb, void* pUserData)
{
    size_t realSize = size * nmemb;
    if (NULL == pBuffer || NULL == pUserData)
    {
        return realSize;
    }

    PackagePart* pPackageInfo = static_cast<PackagePart*>(pUserData);
    CCurlDownload* pThis = pPackageInfo->pThis;

    return pThis->_WriteFile(pBuffer, (DWORD)realSize, pPackageInfo);
}

size_t CCurlDownload::_WriteFile(const void* pBuffer, DWORD size, PackagePart* pPackageInfo)
{
    DWORD dwWritten = 0;
    ::WriteFile(pPackageInfo->hFile, pBuffer, (DWORD)size, &dwWritten, NULL);
    pPackageInfo->downloadSize += dwWritten;
    pPackageInfo->beginPos += dwWritten;
    return dwWritten;
}

int CCurlDownload::_ProgressFunc(void* pUserData, curl_off_t dlTotal, curl_off_t dlNow, curl_off_t ulTotal, curl_off_t ulNow)
{
    UNREFERENCED_PARAMETER(ulTotal);
    UNREFERENCED_PARAMETER(ulNow);

    if (NULL == pUserData || 0 == dlTotal)
    {
        return CURLE_OK;
    }

    PackagePart* pPackageInfo = static_cast<PackagePart*>(pUserData);
    CCurlDownload* pThis = pPackageInfo->pThis;

    return pThis->_ProgressProc(pPackageInfo, dlTotal, dlNow);
}

int CCurlDownload::_ProgressProc(PackagePart* pPackageInfo, long long dlTotal, long long dlNow)
{
    UNREFERENCED_PARAMETER(pPackageInfo);
    UNREFERENCED_PARAMETER(dlTotal);
    UNREFERENCED_PARAMETER(dlNow);

    long long download = 0;

    // 统计已下载数据和总文件大小
    for (auto& item : m_packageParts)
    {
        download += item->downloadSize;
    }

    // 下载进度回调
    if (m_lastTotal > 0 && m_lastDownload != download && m_cbProgress)
    {
        m_cbProgress(download, m_lastTotal, (double)download / (double)m_lastTotal);
    }

    m_lastDownload = download;

    if (m_bCancel)
    {
        return -1;
    }

    return CURLE_OK;
}

CCurlDownloadMgr.h

#pragma once

#include "CCurlDownload.h"

//多文件下载管理
class CCurlDownloadMgr
{
public:

    CCurlDownloadMgr();

    ~CCurlDownloadMgr();

    void PushBack(const CCurlDownload& task);

    void Clear();

    bool StartDownload(std::function<void(long long dlNow, long long dlTotal, double lfProgress)> cb = nullptr);

    void CancelDownload();

private:

    // 休眠
    void SleepMillisecond(int millisecond) const;

private:
    std::vector<CCurlDownload> m_downloadTasks;         //下载任务队列
    std::function<void(long long dlNow, long long dlTotal, double lfProgress)> m_cbProgress;     //下载进度回调
    bool m_IsCancel;                                    //是否取消
};

CCurlDownloadMgr.cpp

#include "CCurlDownloadMgr.h"
#include <time.h>
#include <algorithm>
#include <thread>

CCurlDownloadMgr::CCurlDownloadMgr()
{
    m_IsCancel = false;
}

CCurlDownloadMgr::~CCurlDownloadMgr()
{
    CancelDownload();
}

void CCurlDownloadMgr::PushBack(const CCurlDownload& task)
{
    m_downloadTasks.push_back(task);
}

void CCurlDownloadMgr::Clear()
{
    m_downloadTasks.clear();
}

bool CCurlDownloadMgr::StartDownload(std::function<void(long long dlNow, long long dlTotal, double lfProgress)> cb)
{
    for (auto& item : m_downloadTasks)
    {
        item.AsyncDownload();
    }

    while (true)
    {
        //检查是否有线程还在下载
        bool isDownloading = std::any_of(m_downloadTasks.begin(), m_downloadTasks.end(), [&isDownloading](const CCurlDownload& item)
            {
                return item.IsDownloading();
            });

        //统计下载任务下载进度
        long long nDownload = 0;
        long long nTotal = 0;

        for (const auto& item : m_downloadTasks)
        {
            nDownload += item.GetDownloadingSize();
            nTotal += item.GetDownloadingTotal();
        }

        if (cb && nTotal > 0)
        {
            cb(nDownload, nTotal, (double)nDownload / (double)nTotal);
        }

        //非下载状态及时退出
        if (!isDownloading)
        {
            break;
        }

        SleepMillisecond(100);
    }

    //检查是否每个任务都下载完成
    bool isDownloadFinish = std::all_of(m_downloadTasks.begin(), m_downloadTasks.end(), [](const CCurlDownload& item)
        {
            return item.GetLastDownloadResult();
        });

    return isDownloadFinish;
}

void CCurlDownloadMgr::CancelDownload()
{
    m_IsCancel = true;
    for (auto& item : m_downloadTasks)
    {
        item.CancelDownload();
    }
}

void CCurlDownloadMgr::SleepMillisecond(int millisecond) const
{
    const int span = 20;
    if (millisecond < span)
    {
        std::this_thread::sleep_for(std::chrono::milliseconds(millisecond));
        return;
    }

    clock_t tmBegin = clock();
    while (true)
    {
        if (clock() - tmBegin > millisecond)
        {
            break;
        }

        if (m_IsCancel)
        {
            break;
        }

        std::this_thread::sleep_for(std::chrono::milliseconds(span));
    }
}

// CurlDownload.cpp : 此文件包含 "main" 函数。程序执行将在此处开始并结束。
//

#include <iostream>
#include <tchar.h>
#include "CCurlDownloadMgr.h"
#include "ThreadPool.h"
#include "CCurlRequest.h"
#include "CStrUtils.h"
#include "CTimeUtils.h"

int main()
{
    curl_global_init(CURL_GLOBAL_ALL);


    CCurlDownloadMgr mgr;
    CCurlDownload task;

    task.SetDownloadInfo("https://mirrors.sdu.edu.cn/software/Windows/WePE/WePE32_V2.2.exe", "WePE32_V2.2.exe", 6);
    mgr.PushBack(task);

    mgr.StartDownload([](long long dlNow, long long dlTotal, double lfProgress) {
        _tprintf(_T("%.2lf%% %lld/%lld\n"), lfProgress* 100, dlNow, dlTotal);
        });

    curl_global_cleanup();

    return 0;
}

【C++开源库使用】使用libcurl开源库发送url请求（http/https请求）去下载用户头像文件（附完整源码）
dvlinker的技术专栏
07-07 8421
本文详细讲述如何使用libcurl开源库发送url请求（http请求）去下载用户头像文件，并给出了完整实现源码。
libcurl库 libcurl-7.19.3-win32-ssl-msvc
06-23
7. **多线程支持**：在多线程环境中，libcurl可以同时处理多个请求，提升并发性能。 8. **代理支持**：libcurl可以配置通过HTTP、SOCKS等类型的代理服务器进行网络通信，适应不同的网络环境。 9. **认证机制**：...
libcurl多线程下载
07-14
基于libcurl/pthreads的多线程下载器，跨平台，支持暂停、还原、中止，断点续传，支持HTTP,HTTPS协议链接，http://blog.youkuaiyun.com/u012532305/article/details/46709495也是这篇文章的源码
简单的基于libcurl和c++11 thread多线程的多线程下载程序
lhw330472787的博客
01-04 935
简单的基于libcurl和c++11 thread多线程的多线程下载程序
怎么判断文件是否支持多线程下载
最新发布
ai
05-25 741
怎么判断文件是否支持多线程下载 怎么判断那个文件存储的服务器必须支持HTTP Range请求 怎么判断那个文件是否支持多线程下载 即使服务器存储的是一个压缩包，也可以多线程分段下载吗 压缩包文件放在服务器，能实现多线程下载吗
libcurl下载文件
weixin_33795093的博客
04-22 701
一、初始化 CURL *pHandler = curl_easy_init();   二、设置请求参数； 调用curl_easy_setopt方法，设置选项 curl_easy_setopt(pHandler , CURLOPT_WRITEFUNCTION, WriteData); curl_easy_setopt(pHandler , CURLOPT_WRITEDATA, pFile...
多线程使用libcurl的坑
热门推荐
技术千山
04-14 1万+
用libcurl一段时间遇到莫名其妙的程序崩溃的情况，开会觉得是线程栈溢出导致的段错误，专门增加了线程栈的大小貌似无效。线程也是分离的。用valgrind定位到问题可能出现在curl的调用上。 排查的时候也发现了libcurl一些额外的坑，现做个总结笔记。 1.、 线程使用libcurl访问时，设置了超时时间，而libcurl库不会为这个超时信号做任何处理，信号产生而没有信号句柄处理，可能导
给予c++的多线程下载
07-14
该程序默认下载的是QQ下载完成后会默认自动打开 就因为这一点360有时会报毒 源码已经提供 可以查阅是否属于病毒 代码中的url地址和保存文件路径是可以自己修改 其中的线程数也可以自己修改
libcurl使用多线程下载大文件源码示例！
雪峰流云
09-14 1万+
使用libcurl多线程下载的大文件的基本思想： 首选打开文件，将文件等分为指定的片段，使用http range下载，一个线程下载一个片段，当线程下载片段时，它们将数据写到打开文件的指定位置，类似BT文件下载的方式（这样片段下载完成后不用再合并），当所有的子线程下载完成后，这个大文件也就随之下载完成了。 要注意的问题 基本上，每个线程都应该有自己的easy handle用于数据通
C++ 用libcurl库进行http通讯网络编程 - 绿色冰点 - 博客园1
08-03
使用libcurl进行基本的HTTP GET和POST操作、下载HTML网页、保存文件、进度条显示和断点续传，都是libcurl常见的应用场景，可以参考官方文档或示例代码实现。 总结，libcurl为C++开发者提供了一个强大且灵活的网络...
libcurl SDK Win32
12-04
**libcurl SDK Win32** 是一个专门为在Windows 32位环境中运行的C++程序设计的库，它提供了强大的网络通信功能，尤其适合进行HTTP、FTP、SMTP等多种协议的客户端开发。libcurl是一个开源项目，它使得开发者能够轻松...
C/C++多线程下载模块
03-01
这是工作中用curl库进行文件下载的一个引擎，支持多任务，断点续传。
可以多线程下载文件的c++客户端
11-25
可以多线程下载文件c++的客户端软件，可以切割文件并传输
libcurl下载大文件
06-26
http://blog.youkuaiyun.com/infoworld/article/details/46646933 使用libcurl下载大文件简单例子.
vc++使用libcurl下载文件，支持断点续传，支持多文件下载
08-09
vc++使用libcurl下载文件，支持断点续传，支持多文件下载；代码经测试，可以编译运行；使用url解析文件名；谢谢支持！
C/C++多线程文件传输v2.2完整源码(断点续传 网络消息收发 高效率 界面与后台线程交互)
05-08
2.2完整源码版,包括工程文件保证可以编译通过 多线程socket文件传输/支持断点续传/收发消息/点对多点 使用阻塞方式的socket,使用多线程,有较高的性能. 在局域网中测试达到极限速度. 支持断点续传. 服务端可同时接收多个文件. 传输文件的同时可以发送网络消息. 如有问题,请与我联系： hongxing777@gmail.com
libcurl 上传文件
10-20
在 Visual C++ 6 (VC6) 环境下，我们可以利用 libcurl 实现文件上传到 Web 服务器的功能。下面将详细介绍如何使用 libcurl 在 VC6 中实现文件上传。 1. **安装和配置 libcurl** 首先，你需要从官方站点下载 ...
多线程下载文件
天浩
09-22 186
[code="java"]package mutiDownload; import java.io.IOException; import java.io.InputStream; import java.io.RandomAccessFile; import java.net.HttpURLConnection; import java.net.MalformedURLExcept...
libcurl实现多线程下载器
farrellzhang的专栏
07-01 4975
libcurl官网(http://curl.haxx.se/)是一个很强大网络功能的库，支持当前DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMTP, SMTPS, Telnet and TFTP. curl supp
评论 4
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值