Install Rouge and Pyrouge for python 2.7 (conda virtual env) on Linux 64

这篇博客详细记录了在Linux上使用conda创建Python 2.7虚拟环境,并安装问题重重的Rouge和Pyrouge的过程,包括从废弃网站获取ROUGE-1.5.5,设置路径,安装插件,处理Wordnet异常和修复测试代码错误等步骤。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Who to complain to about this terrible package???? Its official website was down, its bugs and exceptions are famous, and its python wrapper code has errors...But everyone  use it for summarization tasks. Not cool at all.

忍不住吐槽,Rouge,一个连官网都关闭了的,纯靠遗产继承的,超级史诗级烂尾工程,还那么多人用.....

pyrouge安装了之后报错居然改代码搞定的,你特么没弄好就别发布行不行?


Install Rouge and Pyrouge


Step 1: Create a new conda virtual env

conda create -n py27env python=2.7 pip

source activate py27env

# use " source deactivate" to deactivate the env


Step 2: Install pyrouge by conda from 3rd party

conda install -c auto pyrouge


Step 3: Download ROUGE-1.5.5 from its closed website by using a time machine (ask Einstein or go to Github), then copy ROUGE-1.5.5 to ~/rouge_path

cp RELEASE-1.5.5 ~/rouge/


Step 4: Set ROUGE path for pyrouge

pyrouge_set_rouge_path ~/rouge/RELEASE-1.5.5


Step 5: Install plugins for ROUGE-1.5.5

sudo cpan App::cpanminus

sudo cpanm XML::DOM


Step 6: Deal with Wordnet exceptions for ROUGE-1.5.5 (f**k)

cd ~/rouge/RELEASE-1.5.5/data/WordNet-2.0-Exceptions/

./buildExeptionDB.pl . exc WordNet-2.0.exc.db

cd ../

ln -s WordNet-2.0-Exceptions/WordNet-2.0.exc.db WordNet-2.0.exc.db


Step 7: Deal with test code errors for pyrouge (f**k AGAIN!)

In Rouge155_test.py file, you should modify two lines (refhttps://stackoverflow.com/a/41382391):

vi ~/.anaconda3/envs/py27env/lib/python2.7/site-packages/pyrouge/tests/Rouge155_test.py

modify

"pyrouge_evaluate_plain_text_files.py -m {} -s {} -sfp "

to

"pyrouge_evaluate_plain_text_files -m {} -s {} -sfp "

And, modify

"pyrouge_write_config_file.py -m {m} -s {s} " 

to

"pyrouge_write_config_file -m {m} -s {s} " 


Step 8: Run test until "OK" appears.

>python -m pyrouge.test

Ran 10 tests in 10.583s


OK

Done. 

root@eais-bj815mou6mdew33nhqiv-0:/mnt/workspace# pip install "modelscope[nlp]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html Looking in indexes: https://mirrors.cloud.aliyuncs.com/pypi/simple Looking in links: https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html Requirement already satisfied: modelscope[nlp] in /usr/local/lib/python3.11/site-packages (1.27.0) Requirement already satisfied: requests>=2.25 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (2.32.3) Requirement already satisfied: setuptools in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (65.5.1) Requirement already satisfied: tqdm>=4.64.0 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (4.67.1) Requirement already satisfied: urllib3>=1.26 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (2.3.0) Requirement already satisfied: addict in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (2.4.0) Requirement already satisfied: attrs in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (25.3.0) Requirement already satisfied: datasets<=3.2.0,>=3.0.0 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (3.2.0) Requirement already satisfied: einops in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (0.8.1) Requirement already satisfied: Pillow in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (11.1.0) Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (2.9.0.post0) Requirement already satisfied: scipy in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (1.12.0) Requirement already satisfied: simplejson>=3.3.0 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (3.20.1) Requirement already satisfied: sortedcontainers>=1.5.9 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (2.4.0) Requirement already satisfied: transformers in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (4.52.4) Requirement already satisfied: boto3 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (1.38.31) Requirement already satisfied: embeddings in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (0.0.8) Requirement already satisfied: filelock in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (3.17.0) Requirement already satisfied: ftfy in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (6.3.1) Requirement already satisfied: jieba>=0.42.1 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (0.42.1) Requirement already satisfied: matplotlib in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (3.10.3) Requirement already satisfied: megatron-util in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (1.3.2) Requirement already satisfied: nltk in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (3.9.1) Requirement already satisfied: pandas in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (2.3.0) Requirement already satisfied: protobuf<3.21.0,>=3.19.0 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (3.20.3) Requirement already satisfied: pythainlp in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (5.1.2) Requirement already satisfied: pyvi in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (0.1.1) Requirement already satisfied: regex in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (2024.11.6) Requirement already satisfied: rouge in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (1.0.1) Requirement already satisfied: sacremoses>=0.0.41 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (0.1.1) Requirement already satisfied: scikit-learn in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (1.7.0) Requirement already satisfied: sentencepiece in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (0.2.0) Requirement already satisfied: seqeval in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (1.2.2) Requirement already satisfied: spacy<=3.7.0,>=2.3.5 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (3.7.0) Requirement already satisfied: stanza in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (1.10.1) Requirement already satisfied: subword-nmt>=0.3.8 in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (0.3.8) Requirement already satisfied: termcolor in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (2.5.0) Requirement already satisfied: tokenizers in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (0.21.1) Requirement already satisfied: zhconv in /usr/local/lib/python3.11/site-packages (from modelscope[nlp]) (1.4.3) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.11/site-packages (from datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (1.26.4) Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.11/site-packages (from datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (20.0.0) Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.11/site-packages (from datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (0.3.8) Requirement already satisfied: xxhash in /usr/local/lib/python3.11/site-packages (from datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (3.5.0) Requirement already satisfied: multiprocess<0.70.17 in /usr/local/lib/python3.11/site-packages (from datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (0.70.16) Requirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /usr/local/lib/python3.11/site-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (2024.9.0) Requirement already satisfied: aiohttp in /usr/local/lib/python3.11/site-packages (from datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (3.12.9) Requirement already satisfied: huggingface-hub>=0.23.0 in /usr/local/lib/python3.11/site-packages (from datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (0.32.6) Requirement already satisfied: packaging in /usr/local/lib/python3.11/site-packages (from datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (24.2) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.11/site-packages (from datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (6.0.2) Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (3.0.12) Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (1.0.5) Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (1.0.13) Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (2.0.11) Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (3.0.10) Requirement already satisfied: thinc<8.3.0,>=8.1.8 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (8.2.5) Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (1.1.3) Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (2.5.1) Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (2.0.10) Requirement already satisfied: weasel<0.4.0,>=0.1.0 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (0.3.4) DEPRECATION: Wheel filename 'modelscope-1.6.1.flextrain-py3-none-any.whl' is not correctly normalised. Future versions of pip will raise the following error: Invalid wheel filename (invalid version): 'modelscope-1.6.1.flextrain-py3-none-any' pip 25.3 will enforce this behaviour change. A possible replacement is to rename the wheel to use a correctly normalised name (this may require updating the version in the project metadata). Discussion can be found at https://github.com/pypa/pip/issues/12938 DEPRECATION: Wheel filename 'fairseq-maas-py3-none-any.whl' is not correctly normalised. Future versions of pip will raise the following error: Invalid wheel filename (invalid version): 'fairseq-maas-py3-none-any' pip 25.3 will enforce this behaviour change. A possible replacement is to rename the wheel to use a correctly normalised name (this may require updating the version in the project metadata). Discussion can be found at https://github.com/pypa/pip/issues/12938 DEPRECATION: Wheel filename 'ofa-0.0.2_3.6-py3-none-any.whl' is not correctly normalised. Future versions of pip will raise the following error: Invalid wheel filename (invalid version): 'ofa-0.0.2_3.6-py3-none-any' pip 25.3 will enforce this behaviour change. A possible replacement is to rename the wheel to use a correctly normalised name (this may require updating the version in the project metadata). Discussion can be found at https://github.com/pypa/pip/issues/12938 Collecting typer<0.10.0,>=0.3.0 (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) Downloading https://mirrors.cloud.aliyuncs.com/pypi/packages/62/39/82c9d3e10979851847361d922a373bdfef4091020da7f893acfaf07c0225/typer-0.9.4-py3-none-any.whl (45 kB) Requirement already satisfied: pathy>=0.10.0 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (0.11.0) Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (6.4.0) Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (2.11.5) Requirement already satisfied: jinja2 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (3.1.6) Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.11/site-packages (from spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (3.5.0) Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.11/site-packages (from langcodes<4.0.0,>=3.2.0->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (1.3.0) Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (0.7.0) Requirement already satisfied: pydantic-core==2.33.2 in /usr/local/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (2.33.2) Requirement already satisfied: typing-extensions>=4.12.2 in /usr/local/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (4.12.2) Requirement already satisfied: typing-inspection>=0.4.0 in /usr/local/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (0.4.1) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/site-packages (from requests>=2.25->modelscope[nlp]) (3.4.1) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/site-packages (from requests>=2.25->modelscope[nlp]) (3.10) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/site-packages (from requests>=2.25->modelscope[nlp]) (2025.1.31) Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.11/site-packages (from thinc<8.3.0,>=8.1.8->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (0.7.11) Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.11/site-packages (from thinc<8.3.0,>=8.1.8->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (0.1.5) Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.11/site-packages (from typer<0.10.0,>=0.3.0->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (8.1.8) Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in /usr/local/lib/python3.11/site-packages (from weasel<0.4.0,>=0.1.0->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (0.16.0) Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (2.6.1) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (1.3.2) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (1.6.2) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (6.4.4) Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (0.3.1) Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.11/site-packages (from aiohttp->datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (1.20.0) Requirement already satisfied: hf-xet<2.0.0,>=1.1.2 in /usr/local/lib/python3.11/site-packages (from huggingface-hub>=0.23.0->datasets<=3.2.0,>=3.0.0->modelscope[nlp]) (1.1.3) Requirement already satisfied: marisa-trie>=1.1.0 in /usr/local/lib/python3.11/site-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (1.2.1) Requirement already satisfied: pathlib-abc==0.1.1 in /usr/local/lib/python3.11/site-packages (from pathy>=0.10.0->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (0.1.1) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/site-packages (from python-dateutil>=2.1->modelscope[nlp]) (1.17.0) Requirement already satisfied: joblib in /usr/local/lib/python3.11/site-packages (from sacremoses>=0.0.41->modelscope[nlp]) (1.4.2) Requirement already satisfied: mock in /usr/local/lib/python3.11/site-packages (from subword-nmt>=0.3.8->modelscope[nlp]) (5.2.0) Requirement already satisfied: safetensors>=0.4.3 in /usr/local/lib/python3.11/site-packages (from transformers->modelscope[nlp]) (0.5.3) Requirement already satisfied: botocore<1.39.0,>=1.38.31 in /usr/local/lib/python3.11/site-packages (from boto3->modelscope[nlp]) (1.38.31) Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.11/site-packages (from boto3->modelscope[nlp]) (0.10.0) Requirement already satisfied: s3transfer<0.14.0,>=0.13.0 in /usr/local/lib/python3.11/site-packages (from boto3->modelscope[nlp]) (0.13.0) Requirement already satisfied: wcwidth in /usr/local/lib/python3.11/site-packages (from ftfy->modelscope[nlp]) (0.2.13) Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/site-packages (from jinja2->spacy<=3.7.0,>=2.3.5->modelscope[nlp]) (3.0.2) Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.11/site-packages (from matplotlib->modelscope[nlp]) (1.3.2) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.11/site-packages (from matplotlib->modelscope[nlp]) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.11/site-packages (from matplotlib->modelscope[nlp]) (4.58.1) Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.11/site-packages (from matplotlib->modelscope[nlp]) (1.4.8) Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.11/site-packages (from matplotlib->modelscope[nlp]) (3.2.3) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/site-packages (from pandas->modelscope[nlp]) (2025.2) Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/site-packages (from pandas->modelscope[nlp]) (2025.2) Requirement already satisfied: sklearn-crfsuite in /usr/local/lib/python3.11/site-packages (from pyvi->modelscope[nlp]) (0.5.0) Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.11/site-packages (from scikit-learn->modelscope[nlp]) (3.6.0) Requirement already satisfied: python-crfsuite>=0.9.7 in /usr/local/lib/python3.11/site-packages (from sklearn-crfsuite->pyvi->modelscope[nlp]) (0.9.11) Requirement already satisfied: tabulate>=0.4.2 in /usr/local/lib/python3.11/site-packages (from sklearn-crfsuite->pyvi->modelscope[nlp]) (0.9.0) Requirement already satisfied: emoji in /usr/local/lib/python3.11/site-packages (from stanza->modelscope[nlp]) (2.14.1) Requirement already satisfied: networkx in /usr/local/lib/python3.11/site-packages (from stanza->modelscope[nlp]) (3.4.2) Requirement already satisfied: torch>=1.3.0 in /usr/local/lib/python3.11/site-packages (from stanza->modelscope[nlp]) (2.3.1) Requirement already satisfied: sympy in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (1.13.3) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (12.1.105) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (12.1.105) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (12.1.105) Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (8.9.2.26) Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (12.1.3.1) Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (11.0.2.54) Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (10.3.2.106) Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (11.4.5.107) Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (12.1.0.106) Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (2.20.5) Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (12.1.105) Requirement already satisfied: triton==2.3.1 in /usr/local/lib/python3.11/site-packages (from torch>=1.3.0->stanza->modelscope[nlp]) (2.3.1) Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.3.0->stanza->modelscope[nlp]) (12.8.93) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.11/site-packages (from sympy->torch>=1.3.0->stanza->modelscope[nlp]) (1.3.0) WARNING: Error parsing dependencies of pytorch-lightning: .* suffix can only be used with `==` or `!=` operators torch (>=1.9.*) ~~~~~~^ Installing collected packages: typer Attempting uninstall: typer Found existing installation: typer 0.16.0 Uninstalling typer-0.16.0: Successfully uninstalled typer-0.16.0 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. gradio 5.33.1 requires typer<1.0,>=0.12; sys_platform != "emscripten", but you have typer 0.9.4 which is incompatible. Successfully installed typer-0.9.4 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
最新发布
07-07
一、目的1. 加速训练过程2. 适应大规模数据3. 资源利用率高4. 提升训练速度5. 增大系统容量6. 提高系统可用性7. 加速模型迭代二、 LLaMA-Factory1.安装2. LLaMA-Factory 校验三、 训练引擎1.DDP2. DeepSpeed3.FSDP四、WebUI五. 参数配置1. 模型2. 数据3. 训练参数4. 多卡参数1. ZeRO-12. ZeRO-23. ZeRO-3六、训练七、推理八、XTuner一、目的分布式训练是一种在多个计算节点上共同完成机器学习模型训练任务的过程,它可以充分利用多台计算机的资源,提高训练效率和模型准确性。分布式训练的主要优势包括:1. 加速训练过程通过并行计算,分布式训练大幅缩短了训练时间,提高了训练效率。提高模型准确性:利用更多的计算资源和数据样本进行训练,减少了过拟合风险,提高了模型的泛化能力和准确性。2. 适应大规模数据分布式训练能够处理传统单机训练难以应对的大规模数据集。3. 资源利用率高有效利用了计算资源,避免了单机训练时的资源闲置和浪费。4. 提升训练速度通过并行计算,分布式训练能够显著缩短模型训练的时间,尤其是在处理大规模数据集和复杂模型时效果更为明显。5. 增大系统容量随着业务量的增长,单机性能已无法满足需求。分布式训练通过多台计算设备的协同工作,能够应对更大规模的应用场景。6. 提高系统可用性分布式架构能够消除单点故障,提高系统的整体可用性。即使某个计算设备出现故障,也不会影响整个训练任务的进行。7. 加速模型迭代在快速迭代的机器学习项目中,分布式训练能够更快地完成模型训练,从而加速模型迭代和优化过程。总的来说,分布式训练在深度学习领域提高训练效率和加快模型收敛的重要手段 。二、 LLaMA-Factory1.安装在安装 LLaMA-Factory 之前,请确保您安装了下列依赖:运行以下指令以安装 LLaMA-Factory 及其依赖:git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.gitcd LLaMA-Factorypip install -e ".[torch,metrics]"123如果出现环境冲突,请尝试使用 pip install --no-deps -e . 解决2. LLaMA-Factory 校验完成安装后,可以通过使用 llamafactory-cli version 来快速校验安装是否成功如果看到类似下面的界面,就说明安装成功了。 Successfully uninstalled requests-2.31.0 Attempting uninstall: anyio Found existing installation: anyio 4.4.0 Uninstalling anyio-4.4.0: Successfully uninstalled anyio-4.4.0Successfully installed accelerate-1.2.1 aiofiles-23.2.1 aiohappyeyeballs-2.4.6 aiohttp-3.11.12 aiosignal-1.3.2 annotated-types-0.7.0 anyio-4.8.0 audioread-3.0.1 av-14.1.0 click-8.1.8 datasets-3.2.0 dill-0.3.8 docstring-parser-0.16 einops-0.8.1 fastapi-0.115.8 ffmpy-0.5.0 fire-0.7.0 frozenlist-1.5.0 gradio-5.12.0 gradio-client-1.5.4 huggingface-hub-0.28.1 jieba-0.42.1 joblib-1.4.2 lazy-loader-0.4 librosa-0.10.2.post1 llamafactory-0.9.2.dev0 llvmlite-0.44.0 markdown-it-py-3.0.0 mdurl-0.1.2 msgpack-1.1.0 multidict-6.1.0 multiprocess-0.70.16 nltk-3.9.1 numba-0.61.0 orjson-3.10.15 pandas-2.2.3 peft-0.12.0 pooch-1.8.2 propcache-0.2.1 pyarrow-19.0.0 pydantic-2.10.6 pydantic-core-2.27.2 pydub-0.25.1 python-multipart-0.0.20 pytz-2025.1 regex-2024.11.6 requests-2.32.3 rich-13.9.4 rouge-chinese-1.0.3 ruff-0.9.6 safehttpx-0.1.6 safetensors-0.5.2 scikit-learn-1.6.1 scipy-1.15.1 semantic-version-2.10.0 sentencepiece-0.2.0 shellingham-1.5.4 shtab-1.7.1 soundfile-0.13.1 soxr-0.5.0.post1 sse-starlette-2.2.1 starlette-0.45.3 termcolor-2.5.0 threadpoolctl-3.5.0 tiktoken-0.9.0 tokenizers-0.21.0 tomlkit-0.13.2 tqdm-4.67.1 transformers-4.48.3 trl-0.9.6 typer-0.15.1 typing-extensions-4.12.2 tyro-0.8.14 tzdata-2025.1 uvicorn-0.34.0 websockets-14.2 xxhash-3.5.0 yarl-1.18.3WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venvroot@autodl-container-c2d74383d9-db8bb7c4:~/autodl-tmp/LLaMA-Factory# llamafactory-cli version----------------------------------------------------------| Welcome to LLaMA Factory, version 0.9.2.dev0 || || Project page: https://github.com/hiyouga/LLaMA-Factory |----------------------------------------------------------root@autodl-container-c2d74383d9-db8bb7c4:~/autodl-tmp/LLaMA-Factory# 1234567891011121314三、 训练引擎LLaMA-Factory 支持单机多卡和多机多卡分布式训练。同时也支持 DDP , DeepSpeed 和 FSDP 三种分布式引擎。1.DDPDDP (DistributedDataParallel) 通过实现模型并行和数据并行实现训练加速。 使用 DDP 的程序需要生成多个进程并且为每个进程创建一个 DDP 实例,他们之间通过 torch.distributed 库同步。2. DeepSpeedDeepSpeed 是微软开发的分布式训练引擎,并提供ZeRO(Zero Redundancy Optimizer)、offload、Sparse Attention、1 bit Adam、流水线并行等优化技术。 您可以根据任务需求与设备选择使用。3.FSDP通过全切片数据并行技术(Fully Sharded Data Parallel)来处理更多更大的模型。在 DDP 中,每张 GPU 都各自保留了一份完整的模型参数和优化器参数。而 FSDP 切分了模型参数、梯度与优化器参数,使得每张 GPU 只保留这些参数的一部分。 除了并行技术之外,FSDP 还支持将模型参数卸载至CPU,从而进一步降低显存需求。由于deepseek分布式训练加速,采用混合精度(fp16/fp32)和ZeRO优化,减少显存占用,从而加速训练。所以本文采用DeepSpeed 是训练引擎。四、WebUILLaMA-Factory 支持通过 WebUI 零代码微调大语言模型。 在完成 安装 后,您可以通过以下指令进入 WebUI:llamafactory-cli webui1WebUI 主要分为四个界面:训练、评估与预测、对话、导出。当运行上面命令后,打开如下界面在开始训练模型之前,需要指定的参数有:模型名称及路径训练阶段微调方法训练数据集学习率、训练轮数等训练参数微调参数等其他参数输出目录及配置路径
03-20
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值