移动conda虚拟环境的安装目录

最新推荐文章于 2025-10-14 14:16:23 发布

原创最新推荐文章于 2025-10-14 14:16:23 发布 · 712 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#conda #linux #运维

部署运行你感兴趣的模型镜像

方法 1：重新创建环境（推荐）

(1) 导出环境配置（生成 environment.yml）：

conda activate old_env  # 激活原环境
conda env export > environment.yml  # 导出配置

(llmtuner) :~$ conda env export > environment.yml                                   
(llmtuner) :~$ tail -f environment.yml                                        
    - websockets==15.0.1
    - wget==3.2
    - wheel==0.45.1
    - wrapt==1.17.2
    - xformers==0.0.30
    - xgrammar==0.1.19
    - xxhash==3.5.0
    - yarl==1.20.0
    - zipp==3.23.0
prefix: /home/anaconda3/envs/llmtuner

vim 打开 environment.yml

 1 name: llmtuner
  2 channels:
  3   - defaults
  4 dependencies:
  5   - _libgcc_mutex=0.1=main
  6   - _openmp_mutex=5.1=1_gnu
  7   - bzip2=1.0.8=h5eee18b_6
  8   - ca-certificates=2025.2.25=h06a4308_0
  9   - ld_impl_linux-64=2.40=h12ee557_0
 10   - libffi=3.4.4=h6a678d5_1
 11   - libgcc-ng=11.2.0=h1234567_1
 12   - libgomp=11.2.0=h1234567_1
 13   - libstdcxx-ng=11.2.0=h1234567_1
 14   - libuuid=1.41.5=h5eee18b_0
 15   - ncurses=6.4=h6a678d5_0
 16   - openssl=3.0.16=h5eee18b_0
 17   - python=3.10.16=he870216_1
 18   - readline=8.2=h5eee18b_0
 19   - sqlite=3.45.3=h5eee18b_0
 20   - tk=8.6.14=h39e8969_0
 21   - xz=5.6.4=h5eee18b_1
 22   - zlib=1.2.13=h5eee18b_1
 23   - pip:
 24     - accelerate==1.7.0
 25     - aiohttp==3.9.5
 26     - aiosignal==1.3.2
 27     - airportsdata==20250622
 28     - annotated-types==0.7.0
 29     - anyio==4.9.0

(2) 添加国内源

在您的 environment.yml 文件中指定使用清华源：

name: old_env
channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/
dependencies:

(3) 在新目录创建相同环境：

mkdir -p ./anaconda3/envs/llmtuner

conda env create --prefix /data2/anaconda3/envs/llmtuner -f environment.yml

此时会出现告警

Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies.  Conda may not use the correct pip to install your packages, and they may end up in the wrong place.  Please add an explicit pip dependency.  I'm adding one for you, but still nagging you.

警告信息是关于 pip 依赖的问题，这不会影响环境创建过程，但为了更好的实践，可以手动修改 environment.yml 文件来消除这些警告。

解决步骤
1. 修改 environment.yml 文件

在您的 environment.yml 文件中添加 pip 作为显式依赖项：

name: old_env
channels:
  - defaults
dependencies:
  - python=3.x.x  # 保持原版本
  - pip           # 添加这一行
  - ...           # 其他依赖项
  - pip:
    - package1    # pip安装的包
    - package2

(4) 解决 pip 下载超时和中断问题

conda env create --prefix /data2/anaconda3/envs/llmtuner -f environment.yml

出现下面问题

Collecting lark==1.2.2 (from -r /data2/condaenv.frhhj0_2.requirements.txt (line 67))
  Using cached lark-1.2.2-py3-none-any.whl.metadata (1.8 kB)
Collecting llama-cpp-python==0.3.9 (from -r /data2/condaenv.frhhj0_2.requirements.txt (line 68))
  Downloading llama_cpp_python-0.3.9.tar.gz (67.9 MB)
                                              0.3/67.9 MB ? eta -:--:--

Pip subprocess error:
  WARNING: Connection timed out while downloading.
error: incomplete-download

× Download failed because not enough bytes were received (262 kB/67.9 MB)
╰─> URL: https://files.pythonhosted.org/packages/de/6d/4a20e676bdf7d9d3523be3a081bf327af958f9bdfe2a564f5cf485faeaec/llama_cpp_python-0.3.9.tar.gz

note: This is an issue with network connectivity, not pip.
hint: Consider using --resume-retries to enable download resumption.

failed

CondaEnvException: Pip failed

遇到的问题是由于网络连接不稳定导致 llama-cpp-python 大文件下载中断。以下是几种解决方案：

方法1：使用清华源并启用断点续传
# 激活环境
conda activate /data2/anaconda3/envs/llmtuner
# 使用清华源并启用断点续传
pip install --index-url https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn --retries 10 --timeout 1000 llama-cpp-python==0.3.9

方法2：分步安装依赖项
先安装小文件依赖项：
pip install --index-url https://pypi.tuna.tsinghua.edu.cn/simple lark==1.2.2
单独安装大文件：
pip install --index-url https://pypi.tuna.tsinghua.edu.cn/simple --retries 10 --timeout 1000 llama-cpp-python==0.3.9

方法3：使用预编译的wheel文件
从清华源下载预编译的wheel文件：
wget https://pypi.tuna.tsinghua.edu.cn/packages/de/6d/4a20e676bdf7d9d3523be3a081bf327af958f9bdfe2a564f5cf485faeaec/llama_cpp_python-0.3.9.tar.gz
本地安装：
pip install llama_cpp_python-0.3.9.tar.gz

方法4：修改conda环境创建方式
修改您的environment.yml文件，将pip安装的包分开：
name: llmtuner
channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
dependencies:
  - python=3.x
  - pip
  - lark=1.2.2
  # 其他conda包...
  - pip:
    - llama-cpp-python==0.3.9 --index-url https://pypi.tuna.tsinghua.edu.cn/simple --trust

(5) 继续安装中断后的库

1. 从现有环境重新生成 requirements.txt

# 激活原环境（如果知道环境名）
conda activate 原环境名

# 或者使用完整路径激活
conda activate /原/环境/路径

# 生成新的requirements文件
pip freeze > new_requirements.txt

cp new_requirements.txt /data2
pip install -r new_requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

安装完成，显示：

Requirement already satisfied: setuptools in ./anaconda3/envs/llmtuner/lib/python3.10/site-packages (from swanlab==0.6.3->-r new_requirements.txt (line 171)) (78.1.1)
Installing collected packages: wget, wcwidth, sentencepiece, pytz, py-cpuinfo, pure_eval, ptyprocess, nvidia-ml-py, nvidia-cusparselt-cu12, mpmath, fastrlock, blake3, zipp, xxhash, wrapt, websockets, uvloop, urllib3, tzdata, typing-inspection, triton, traitlets, tqdm, tornado, tomli, sympy, sniffio, six, simplejson, shellingham, scipy, safetensors, rpds-py, regex, pyzmq, PyYAML, python-multipart, python-json-logger, python-dotenv, pyparsing, pynvml, Pygments, pydantic_core, pycountry, pyarrow, psutil, protobuf, propcache, prompt_toolkit, prometheus_client, prettytable, pluggy, platformdirs, pillow, pexpect, partial-json-parser, parso, packaging, opentelemetry-semantic-conventions-ai, opencv-python-headless, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, ninja, networkx, nest-asyncio, multidict, msgspec, msgpack, mdurl, llvmlite, llguidance, lark, kiwisolver, jmespath, jiter, interegular, iniconfig, idna, httptools, hf-xet, h11, grpcio, fsspec, frozenlist, fonttools, filelock, executing, exceptiongroup, einops, dnspython, distro, dill, decorator, debugpy, cycler, cupy-cuda12x, contourpy, cloudpickle, click, charset-normalizer, certifi, cachetools, attrs, async-timeout, asttokens, astor, annotated-types, airportsdata, yarl, uvicorn, swankit, stack-data, requests, referencing, python-dateutil, pytest, pyecharts, pydantic, opentelemetry-proto, nvidia-cusparse-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, numba, multiprocess, matplotlib-inline, markdown-it-py, jupyter_core, jedi, importlib_metadata, httpcore, googleapis-common-protos, gguf, email_validator, depyf, comm, anyio, aiosignal, watchfiles, tiktoken, starlette, rich, pandas, opentelemetry-exporter-otlp-proto-common, opentelemetry-api, nvidia-cusolver-cu12, matplotlib, lm-format-enforcer, jupyter_client, jsonschema-specifications, ipython, huggingface-hub, httpx, botocore, aiohttp, typer, torch, tokenizers, seaborn, s3transfer, rich-toolkit, prometheus-fastapi-instrumentator, opentelemetry-semantic-conventions, openai, ollama, jsonschema, ipykernel, fastapi, xformers, transformers, torchvision, torchaudio, ray, outlines_core, opentelemetry-sdk, mistral_common, fastapi-cli, datasets, boto3, accelerate, xgrammar, swanlab, peft, outlines, opentelemetry-exporter-otlp-proto-http, opentelemetry-exporter-otlp-proto-grpc, compressed-tensors, opentelemetry-exporter-otlp, vllm
Successfully installed PyYAML-6.0.2 Pygments-2.19.1 accelerate-1.7.0 aiohttp-3.9.5 aiosignal-1.3.2 airportsdata-20250622 annotated-types-0.7.0 anyio-4.9.0 astor-0.8.1 asttokens-3.0.0 async-timeout-4.0.3 attrs-25.3.0 blake3-1.0.5 boto3-1.38.36 botocore-1.38.36 cachetools-6.1.0 certifi-2025.1.31 charset-normalizer-3.4.1 click-8.1.8 cloudpickle-3.1.1 comm-0.2.2 compressed-tensors-0.10.1 contourpy-1.3.2 cupy-cuda12x-13.4.1 cycler-0.12.1 datasets-3.6.0 debugpy-1.8.14 decorator-5.2.1 depyf-0.18.0 dill-0.3.8 distro-1.9.0 dnspython-2.7.0 einops-0.8.1 email_validator-2.2.0 exceptiongroup-1.2.2 executing-2.2.0 fastapi-0.115.14 fastapi-cli-0.0.7 fastrlock-0.8.3 filelock-3.18.0 fonttools-4.57.0 frozenlist-1.6.0 fsspec-2025.3.0 gguf-0.16.2 googleapis-common-protos-1.70.0 grpcio-1.73.1 h11-0.14.0 hf-xet-1.1.3 httpcore-1.0.8 httptools-0.6.4 httpx-0.28.1 huggingface-hub-0.32.5 idna-3.10 importlib_metadata-8.7.0 iniconfig-2.1.0 interegular-0.3.3 ipykernel-6.29.5 ipython-8.37.0 jedi-0.19.2 jiter-0.9.0 jmespath-1.0.1 jsonschema-4.24.0 jsonschema-specifications-2025.4.1 jupyter_client-8.6.3 jupyter_core-5.8.1 kiwisolver-1.4.8 lark-1.2.2 llguidance-0.7.30 llvmlite-0.44.0 lm-format-enforcer-0.10.11 markdown-it-py-3.0.0 matplotlib-3.10.1 matplotlib-inline-0.1.7 mdurl-0.1.2 mistral_common-1.6.2 mpmath-1.3.0 msgpack-1.1.1 msgspec-0.19.0 multidict-6.4.3 multiprocess-0.70.16 nest-asyncio-1.6.0 networkx-3.4.2 ninja-1.11.1.4 numba-0.61.2 nvidia-cublas-cu12-12.6.4.1 nvidia-cuda-cupti-cu12-12.6.80 nvidia-cuda-nvrtc-cu12-12.6.77 nvidia-cuda-runtime-cu12-12.6.77 nvidia-cudnn-cu12-9.5.1.17 nvidia-cufft-cu12-11.3.0.4 nvidia-cufile-cu12-1.11.1.6 nvidia-curand-cu12-10.3.7.77 nvidia-cusolver-cu12-11.7.1.2 nvidia-cusparse-cu12-12.5.4.2 nvidia-cusparselt-cu12-0.6.3 nvidia-ml-py-12.575.51 nvidia-nccl-cu12-2.26.2 nvidia-nvjitlink-cu12-12.6.85 nvidia-nvtx-cu12-12.6.77 ollama-0.4.8 openai-1.55.3 opencv-python-headless-4.11.0.86 opentelemetry-api-1.34.1 opentelemetry-exporter-otlp-1.34.1 opentelemetry-exporter-otlp-proto-common-1.34.1 opentelemetry-exporter-otlp-proto-grpc-1.34.1 opentelemetry-exporter-otlp-proto-http-1.34.1 opentelemetry-proto-1.34.1 opentelemetry-sdk-1.34.1 opentelemetry-semantic-conventions-0.55b1 opentelemetry-semantic-conventions-ai-0.4.9 outlines-0.1.11 outlines_core-0.1.26 packaging-25.0 pandas-2.2.3 parso-0.8.4 partial-json-parser-0.2.1.1.post6 peft-0.15.2 pexpect-4.9.0 pillow-11.2.1 platformdirs-4.3.8 pluggy-1.5.0 prettytable-3.16.0 prometheus-fastapi-instrumentator-7.1.0 prometheus_client-0.20.0 prompt_toolkit-3.0.51 propcache-0.3.1 protobuf-5.29.5 psutil-7.0.0 ptyprocess-0.7.0 pure_eval-0.2.3 py-cpuinfo-9.0.0 pyarrow-20.0.0 pycountry-24.6.1 pydantic-2.11.3 pydantic_core-2.33.1 pyecharts-2.0.8 pynvml-12.0.0 pyparsing-3.2.3 pytest-8.3.5 python-dateutil-2.9.0.post0 python-dotenv-1.1.1 python-json-logger-3.3.0 python-multipart-0.0.20 pytz-2025.2 pyzmq-27.0.0 ray-2.47.1 referencing-0.36.2 regex-2024.11.6 requests-2.32.3 rich-14.0.0 rich-toolkit-0.14.7 rpds-py-0.25.1 s3transfer-0.13.0 safetensors-0.5.3 scipy-1.15.3 seaborn-0.13.2 sentencepiece-0.2.0 shellingham-1.5.4 simplejson-3.20.1 six-1.17.0 sniffio-1.3.1 stack-data-0.6.3 starlette-0.46.2 swankit-0.2.3 swanlab-0.6.3 sympy-1.14.0 tiktoken-0.9.0 tokenizers-0.21.2 tomli-2.2.1 torch-2.7.0 torchaudio-2.7.0 torchvision-0.22.0 tornado-6.5.1 tqdm-4.67.1 traitlets-5.14.3 transformers-4.53.0 triton-3.3.0 typer-0.15.2 typing-inspection-0.4.0 tzdata-2025.2 urllib3-2.4.0 uvicorn-0.35.0 uvloop-0.21.0 vllm-0.9.1 watchfiles-1.1.0 wcwidth-0.2.13 websockets-15.0.1 wget-3.2 wrapt-1.17.2 xformers-0.0.30 xgrammar-0.1.19 xxhash-3.5.0 yarl-1.20.0 zipp-3.23.0
(llmtuner) @cvm-tc-001:/data2/$

(6) 新的环境没有名字

$ conda env list                                                                                                                                                                        
# conda environments:
#
                      *  /data2/anaconda3/envs/llmtuner
base                     /home/anaconda3
llmtuner                 /home/anaconda3/envs/llmtuner

Conda 环境列表显示有两个环境路径指向同一个环境 llmtuner，但其中一个没有名字（只有路径）。这是可以修复的。以下是解决方案：

1. 给无名的环境添加名称

conda config --append envs_dirs /data2/anaconda3/envs
conda config --append envs_dirs /home/anaconda3/envs

然后重新列出环境查看：
conda env list

2. 修复重复环境问题

建议删除重复的环境路径：

# 先停用当前环境
conda deactivate

# 删除重复的环境路径（保留一个即可）
conda env remove -p /home/anaconda3/envs/llmtuner

$ conda env list                                              
# conda environments:
#
llmtuner                 /data2/anaconda3/envs/llmtuner
base                  *  /home/anaconda3

更换环境前：

$ df -h 
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            19G  154M   19G   1% /run
/dev/vda2        99G   66G   29G  70% /

更换环境后：

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            19G  186M   19G   1% /run
/dev/vda2        99G   59G   37G  62% /

方法 2：直接移动文件夹（需修复路径）

适用于同系统内迁移，但需手动修复路径引用。步骤：

停用环境：
conda deactivate

移动文件夹：
mv /old/path/to/env /new/path/to/env  # Linux/MacOS
move C:\old\path\to\env C:\new\path\to\env  # Windows

修复路径引用：
修改 Conda 的环境索引文件（environments.txt）：
# 文件路径通常为：~/.conda/environments.txt 或 %CONDA_ROOT%\envs\.txt
# 将旧路径替换为新路径
sed -i 's|/old/path|/new/path|g' ~/.conda/environments.txt  # Linux/MacOS

如果环境激活失败，尝试重新注册：
conda config --append envs_dirs /new/path/to/env

测试激活：
conda activate /new/path/to/env

注意事项：
硬编码路径问题：某些包（如 PyTorch、CUDA）可能在安装时写死了绝对路径，移动后需重新安装这些包。

符号链接：如果原环境包含符号链接，移动后需手动修复。

方法 3：使用 conda-pack（适合大环境迁移）

通过打包环境避免重新下载包。

安装 conda-pack：
conda install conda-pack

打包原环境：
conda pack -n old_env -o old_env.tar.gz
# 或指定路径
conda pack -p /old/path/to/env -o old_env.tar.gz

在新位置解压：
mkdir -p /new/path/to/env
tar -xzf old_env.tar.gz -C /new/path/to/env

激活环境：
conda activate /new/path/to/env


验证迁移成功
检查 Python 和关键包是否正常：
conda list
python -c "import numpy; print(numpy.__file__)"  # 检查包路径

总结

| 方法               | 适用场景                          | 注意事项                     |
|--------------------|----------------------------------|----------------------------|
| 重新创建环境        | 跨平台或需要彻底清理时          | 耗时较长                   |
| 直接移动文件夹      | 同系统快速迁移                  | 需修复路径和符号链接        |
| conda-pack         | 大环境或网络受限时              | 需额外安装工具             |