Qwen3 4B + LLaMA Factory: 让大模型高效学习新闻领域分类

我爱学大模型

于 2025-06-04 14:59:20 发布

阅读量579

点赞数 13

CC 4.0 BY-SA版权

文章标签：自然语言处理 AI大模型人工智能 AI qwen3 LLaMA Factory 大模型微调

本文链接：https://blog.youkuaiyun.com/python1234_/article/details/148427230

使用LLaMA-Factory微调Qwen3-4B-instruct模型

LLaMA Factory 是一款开源低代码大模型微调框架，集成了业界广泛使用的微调技术，支持通过 Web UI 界面零代码微调大模型，目前已经成为开源社区内最受欢迎的微调框架之一，GitHub 星标超过 4.5 万。

Qwen3 是阿里云新一代 Qwen系列模型：

\1. 支持场景更广泛：推理模式和非推理模式的有效融合，可在对话中根据用户要求，实现两种模式的任意切换
\2. 推理性能更高效：新增了两个Qwen3-235B-A22B，Qwen3-30B-A3B的MOE模型
\3. 细分通用能力增强：创意写作、角色扮演、多轮对话、指令遵循能力均有明显提升
\4. 工具调用更准确：强化了agent能力，实现了更好的工具调用，更好的MCP支持
\5. 多语言支持更全面：多语言专项增强，可以支持100+语言&方言

本教程以Qwen最新开源的 Qwen3-4B-Instruct 模型为例，介绍如何使用 PAI 平台及 LLaMA Factory 训练框架微调得到新闻标题分类器：给定新闻的类别范围，通过自然语言触发新闻标题分类的功能，并以特定的格式进行返回。

具体的要求为：

给定新闻种类，根据新闻标题对新闻进行分类。
输入为新闻标题，输出为新闻所属的种类，不需要额外解释，但保留思维链的内容；在输入的新闻标题前加上“新闻分类：”等类似文本来标明询问目的。

运行环境要求

GPU 显存：大于等于 12 GB

1. 安装LLaMA Factory

首先，拉取LLaMA-Factory项目到DSW实例。

!git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
正克隆到 'LLaMA-Factory'...
remote: Enumerating objects: 357, done.
remote: Counting objects: 100% (357/357), done.
remote: Compressing objects: 100% (287/287), done.
remote: Total 357 (delta 94), reused 166 (delta 55), pack-reused 0 (from 0)
接收对象中: 100% (357/357), 9.71 MiB | 1.20 MiB/s, 完成.
处理 delta 中: 100% (94/94), 完成.

接着，我们安装LLaMA-Factory依赖环境。

%cd LLaMA-Factory
!pip install -e ".[metrics]"
/mnt/workspace/demos/easy_dataset_llama_factory/LLaMA-Factory
/usr/local/lib/python3.10/site-packages/IPython/core/magics/osm.py:417: UserWarning: This is now an optional IPython functionality, setting dhist requires you to install the `pickleshare` library.
  self.shell.db['dhist'] = compress_dhist(dhist)[-100:]
Looking in indexes: https://mirrors.aliyun.com/pypi/simple/
Obtaining file:///mnt/workspace/demos/easy_dataset_llama_factory/LLaMA-Factory
  Installing build dependencies ... ?25ldone
?25h  Checking if build backend supports build_editable ... ?25ldone
?25h  Getting requirements to build editable ... ?25ldone
?25h  Preparing editable metadata (pyproject.toml) ... ?25ldone
?25hCollecting transformers!=4.46.*,!=4.47.*,!=4.48.0,<=4.51.3,>=4.45.0 (from llamafactory==0.9.3.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/a9/b6/5257d04ae327b44db31f15cce39e6020cc986333c715660b1315a9724d82/transformers-4.51.3-py3-none-any.whl (10.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.4/10.4 MB 1.3 MB/s eta 0:00:0000:0100:01
?25hRequirement already satisfied: datasets<=3.5.0,>=2.16.0 in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (3.0.1)
Requirement already satisfied: accelerate<=1.6.0,>=0.34.0 in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (1.2.1)
Requirement already satisfied: peft<=0.15.1,>=0.14.0 in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.14.0)
Collecting trl<=0.9.6,>=0.8.6 (from llamafactory==0.9.3.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/a5/c3/6565c2c376a829f99da20d39c2912405195ec1fa6aae068dc45c46793e72/trl-0.9.6-py3-none-any.whl (245 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 245.8/245.8 kB 5.7 MB/s eta 0:00:00a 0:00:01
?25hRequirement already satisfied: tokenizers<=0.21.1,>=0.19.0 in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.21.0)
Requirement already satisfied: gradio<=5.25.0,>=4.38.0 in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (5.9.1)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (1.15.0)
Requirement already satisfied: einops in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.8.0)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.2.0)
Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.7.0)
Requirement already satisfied: protobuf in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (5.29.2)
Requirement already satisfied: uvicorn in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.34.0)
Requirement already satisfied: fastapi in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.115.6)
Requirement already satisfied: sse-starlette in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (2.2.1)
Requirement already satisfied: matplotlib>=3.7.0 in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (3.10.0)
Requirement already satisfied: fire in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.7.0)
Requirement already satisfied: omegaconf in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (2.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (24.2)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (6.0.2)
Requirement already satisfied: numpy<2.0.0 in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (1.26.4)
Requirement already satisfied: pydantic<=2.10.6 in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (2.10.4)
Requirement already satisfied: pandas>=2.0.0 in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (2.2.3)
Requirement already satisfied: av in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (14.0.1)
Requirement already satisfied: librosa in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.10.2.post1)
Collecting tyro<0.9.0 (from llamafactory==0.9.3.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/60/ec/e34d546cfd9c5b906d1d534bb75557be9f2b179609d60bb9e97ec07e8ead/tyro-0.8.14-py3-none-any.whl (109 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.8/109.8 kB 10.5 MB/s eta 0:00:00
?25hRequirement already satisfied: nltk in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (3.9.1)
Requirement already satisfied: jieba in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (0.42.1)
Requirement already satisfied: rouge-chinese in /usr/local/lib/python3.10/site-packages (from llamafactory==0.9.3.dev0) (1.0.3)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/site-packages (from accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (6.1.1)
Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/site-packages (from accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (2.5.1)
Requirement already satisfied: huggingface-hub>=0.21.0 in /usr/local/lib/python3.10/site-packages (from accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (0.25.2)
Requirement already satisfied: safetensors>=0.4.3 in /usr/local/lib/python3.10/site-packages (from accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (0.5.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/site-packages (from datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (3.16.1)
Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/site-packages (from datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (18.1.0)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/site-packages (from datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (0.3.8)
Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.10/site-packages (from datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (2.32.3)
Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.10/site-packages (from datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (4.67.1)
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/site-packages (from datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (3.5.0)
Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/site-packages (from datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (0.70.16)
Requirement already satisfied: fsspec<=2024.6.1,>=2023.1.0 in /usr/local/lib/python3.10/site-packages (from fsspec[http]<=2024.6.1,>=2023.1.0->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (2024.6.1)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/site-packages (from datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (3.11.11)
Requirement already satisfied: aiofiles<24.0,>=22.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (23.2.1)
Requirement already satisfied: anyio<5.0,>=3.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (4.8.0)
Requirement already satisfied: ffmpy in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (0.5.0)
Requirement already satisfied: gradio-client==1.5.2 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (1.5.2)
Requirement already satisfied: httpx>=0.24.1 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (0.28.1)
Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (3.1.5)
Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (2.1.5)
Requirement already satisfied: orjson~=3.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (3.10.13)
Requirement already satisfied: pillow<12.0,>=8.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (10.4.0)
Requirement already satisfied: pydub in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (0.25.1)
Requirement already satisfied: python-multipart>=0.0.18 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (0.0.20)
Requirement already satisfied: ruff>=0.2.2 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (0.8.6)
Requirement already satisfied: safehttpx<0.2.0,>=0.1.6 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (0.1.6)
Requirement already satisfied: semantic-version~=2.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (2.10.0)
Requirement already satisfied: starlette<1.0,>=0.40.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (0.41.3)
Requirement already satisfied: tomlkit<0.14.0,>=0.12.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (0.13.2)
Requirement already satisfied: typer<1.0,>=0.12 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (0.15.1)
Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/site-packages (from gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (4.12.2)
Requirement already satisfied: websockets<15.0,>=10.0 in /usr/local/lib/python3.10/site-packages (from gradio-client==1.5.2->gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (14.1)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/site-packages (from matplotlib>=3.7.0->llamafactory==0.9.3.dev0) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/site-packages (from matplotlib>=3.7.0->llamafactory==0.9.3.dev0) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/site-packages (from matplotlib>=3.7.0->llamafactory==0.9.3.dev0) (4.55.3)
Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.10/site-packages (from matplotlib>=3.7.0->llamafactory==0.9.3.dev0) (1.4.8)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/site-packages (from matplotlib>=3.7.0->llamafactory==0.9.3.dev0) (3.2.1)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/site-packages (from matplotlib>=3.7.0->llamafactory==0.9.3.dev0) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/site-packages (from pandas>=2.0.0->llamafactory==0.9.3.dev0) (2024.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/site-packages (from pandas>=2.0.0->llamafactory==0.9.3.dev0) (2024.2)
Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/site-packages (from pydantic<=2.10.6->llamafactory==0.9.3.dev0) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.2 in /usr/local/lib/python3.10/site-packages (from pydantic<=2.10.6->llamafactory==0.9.3.dev0) (2.27.2)
Collecting huggingface-hub>=0.21.0 (from accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/93/27/1fb384a841e9661faad1c31cbfa62864f59632e876df5d795234da51c395/huggingface_hub-0.30.2-py3-none-any.whl (481 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 481.4/481.4 kB 16.7 MB/s eta 0:00:00
?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/site-packages (from transformers!=4.46.*,!=4.47.*,!=4.48.0,<=4.51.3,>=4.45.0->llamafactory==0.9.3.dev0) (2024.11.6)
Requirement already satisfied: docstring-parser>=0.16 in /usr/local/lib/python3.10/site-packages (from tyro<0.9.0->llamafactory==0.9.3.dev0) (0.16)
Requirement already satisfied: rich>=11.1.0 in /usr/local/lib/python3.10/site-packages (from tyro<0.9.0->llamafactory==0.9.3.dev0) (13.9.4)
Requirement already satisfied: shtab>=1.5.6 in /usr/local/lib/python3.10/site-packages (from tyro<0.9.0->llamafactory==0.9.3.dev0) (1.7.1)
Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/site-packages (from uvicorn->llamafactory==0.9.3.dev0) (8.1.8)
Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/site-packages (from uvicorn->llamafactory==0.9.3.dev0) (0.14.0)
Requirement already satisfied: termcolor in /usr/local/lib/python3.10/site-packages (from fire->llamafactory==0.9.3.dev0) (2.5.0)
Requirement already satisfied: audioread>=2.1.9 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (3.0.1)
Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (1.6.0)
Requirement already satisfied: joblib>=0.14 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (1.4.2)
Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (5.1.1)
Requirement already satisfied: numba>=0.51.0 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (0.60.0)
Requirement already satisfied: soundfile>=0.12.1 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (0.13.0)
Requirement already satisfied: pooch>=1.1 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (1.8.2)
Requirement already satisfied: soxr>=0.3.2 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (0.5.0.post1)
Requirement already satisfied: lazy-loader>=0.1 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (0.4)
Requirement already satisfied: msgpack>=1.0 in /usr/local/lib/python3.10/site-packages (from librosa->llamafactory==0.9.3.dev0) (1.1.0)
Requirement already satisfied: antlr4-python3-runtime==4.9.* in /usr/local/lib/python3.10/site-packages (from omegaconf->llamafactory==0.9.3.dev0) (4.9.3)
Requirement already satisfied: six in /usr/local/lib/python3.10/site-packages (from rouge-chinese->llamafactory==0.9.3.dev0) (1.17.0)
Requirement already satisfied: exceptiongroup>=1.0.2 in /usr/local/lib/python3.10/site-packages (from anyio<5.0,>=3.0->gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (1.2.2)
Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/site-packages (from anyio<5.0,>=3.0->gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (3.10)
Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/site-packages (from anyio<5.0,>=3.0->gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (1.3.1)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (2.4.4)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/site-packages (from aiohttp->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (1.3.2)
Requirement already satisfied: async-timeout<6.0,>=4.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (4.0.3)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (24.3.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/site-packages (from aiohttp->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (1.5.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/site-packages (from aiohttp->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (6.1.0)
Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (0.2.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.10/site-packages (from aiohttp->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (1.18.3)
Requirement already satisfied: certifi in /usr/local/lib/python3.10/site-packages (from httpx>=0.24.1->gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (2024.12.14)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/site-packages (from httpx>=0.24.1->gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (1.0.7)
Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /usr/local/lib/python3.10/site-packages (from numba>=0.51.0->librosa->llamafactory==0.9.3.dev0) (0.43.0)
Requirement already satisfied: platformdirs>=2.5.0 in /usr/local/lib/python3.10/site-packages (from pooch>=1.1->librosa->llamafactory==0.9.3.dev0) (4.3.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/site-packages (from requests>=2.32.2->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (3.4.1)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/site-packages (from requests>=2.32.2->datasets<=3.5.0,>=2.16.0->llamafactory==0.9.3.dev0) (2.3.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/site-packages (from rich>=11.1.0->tyro<0.9.0->llamafactory==0.9.3.dev0) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/site-packages (from rich>=11.1.0->tyro<0.9.0->llamafactory==0.9.3.dev0) (2.19.0)
Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.10/site-packages (from scikit-learn>=0.20.0->librosa->llamafactory==0.9.3.dev0) (3.5.0)
Requirement already satisfied: cffi>=1.0 in /usr/local/lib/python3.10/site-packages (from soundfile>=0.12.1->librosa->llamafactory==0.9.3.dev0) (1.17.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (3.4.2)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (12.4.127)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (12.4.127)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (12.4.127)
Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (9.1.0.70)
Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (12.4.5.8)
Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (11.2.1.3)
Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (10.3.5.147)
Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (11.6.1.9)
Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (12.3.1.170)
Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (2.21.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (12.4.127)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (12.4.127)
Requirement already satisfied: triton==3.1.0 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (3.1.0)
Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/site-packages (from torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.10.0->accelerate<=1.6.0,>=0.34.0->llamafactory==0.9.3.dev0) (1.3.0)
Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/site-packages (from typer<1.0,>=0.12->gradio<=5.25.0,>=4.38.0->llamafactory==0.9.3.dev0) (1.5.4)
Requirement already satisfied: pycparser in /usr/local/lib/python3.10/site-packages (from cffi>=1.0->soundfile>=0.12.1->librosa->llamafactory==0.9.3.dev0) (2.22)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich>=11.1.0->tyro<0.9.0->llamafactory==0.9.3.dev0) (0.1.2)
Checking if build backend supports build_editable ... ?25ldone
?25hBuilding wheels for collected packages: llamafactory
  Building editable for llamafactory (pyproject.toml) ... ?25ldone
?25h  Created wheel for llamafactory: filename=llamafactory-0.9.3.dev0-0.editable-py3-none-any.whl size=26719 sha256=ba1e9d3091f158e1b169bd8a0e6d90624d6889424fdaa0ec085cc79434286ab4
  Stored in directory: /tmp/pip-ephem-wheel-cache-u1f0ad2j/wheels/14/a8/89/bbbed1f6592dfa3886162ada484c9bb1ca9feeefd1576a8262
Successfully built llamafactory
Installing collected packages: huggingface-hub, tyro, transformers, trl, llamafactory
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface-hub 0.25.2
    Uninstalling huggingface-hub-0.25.2:
      Successfully uninstalled huggingface-hub-0.25.2
  Attempting uninstall: tyro
    Found existing installation: tyro 0.9.5
    Uninstalling tyro-0.9.5:
      Successfully uninstalled tyro-0.9.5
  Attempting uninstall: transformers
    Found existing installation: transformers 4.47.1
    Uninstalling transformers-4.47.1:
      Successfully uninstalled transformers-4.47.1
  Attempting uninstall: trl
    Found existing installation: trl 0.11.4
    Uninstalling trl-0.11.4:
      Successfully uninstalled trl-0.11.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
xtuner 0.1.23 requires lagent>=0.1.2, which is not installed.
xtuner 0.1.23 requires mmengine>=0.10.3, which is not installed.
xtuner 0.1.23 requires scikit-image, which is not installed.
lmdeploy 0.6.4 requires outlines<0.1.0, but you have outlines 0.1.11 which is incompatible.
lmdeploy 0.6.4 requires peft<=0.11.1, but you have peft 0.14.0 which is incompatible.
lmdeploy 0.6.4 requires torch<=2.4.0,>=2.0.0, but you have torch 2.5.1 which is incompatible.
lmdeploy 0.6.4 requires torchvision<=0.19.0,>=0.15.0, but you have torchvision 0.20.1 which is incompatible.
lmdeploy 0.6.4 requires triton<=3.0.0,>=2.2.0; sys_platform == "linux", but you have triton 3.1.0 which is incompatible.
ms-swift 3.0.2 requires transformers<4.49,>=4.33, but you have transformers 4.51.3 which is incompatible.
ms-swift 3.0.2 requires trl<0.12,>=0.11, but you have trl 0.9.6 which is incompatible.
Successfully installed huggingface-hub-0.30.2 llamafactory-0.9.3.dev0 transformers-4.51.3 trl-0.9.6 tyro-0.8.14
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.3.2 -> 25.1
[notice] To update, run: pip install --upgrade pip

发现出现环境冲突，使用 pip install --no-deps -e . 解决

!pip install --no-deps -e .

运行如下命令，如果显示llamafactory-cli的版本，则表示安装成功。

!llamafactory-cli version
[2025-04-29 13:33:02,860] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
df: /root/.triton/autotune: 没有那个文件或目录
----------------------------------------------------------
| Welcome to LLaMA Factory, version 0.9.3.dev0           |
|                                                        |
| Project page: https://github.com/hiyouga/LLaMA-Factory |
----------------------------------------------------------

2. 下载数据集

LLaMA-Factory项目内置了丰富的数据集，放在了data目录下。您也可以准备自定义数据集，将数据处理为框架特定的格式，放在data下，并且修改dataset_info.json文件。本教程准备了一份sharegpt格式的对话数据集，运行下述命令下载数据。

!wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/llama_factory/data_news_300.zip
!unzip data_news_300.zip
--2025-04-29 13:55:55--  https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/llama_factory/data_news_300.zip
正在解析主机 atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com (atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com)... 47.101.88.43
正在连接 atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com (atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com)|47.101.88.43|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 151242 (148K) [application/zip]
正在保存至: ‘data_news_300.zip’

data_news_300.zip   100%[===================>] 147.70K  --.-KB/s    用时 0.02s   

2025-04-29 13:55:55 (7.04 MB/s) - 已保存 ‘data_news_300.zip’ [151242/151242])

Archive:  data_news_300.zip
  inflating: data_news_300/dataset_info.json  
  inflating: data_news_300/eval.json  
  inflating: data_news_300/train.json

打印此刻数据集的路径，待会训练要用

!cd data_news_300/ && pwd
/mnt/workspace/demos/easy_dataset_llama_factory/LLaMA-Factory/data_news_300

数据集分为训练集和验证集两个文件，其中训练集包含300组对话样本，是根据新闻分类器的要求准备的：

根据新闻标题对新闻进行分类，新闻种类包括：政治、经济、科技、娱乐、体育、教育、健康、国际、国内、社会。
输入为新闻标题，输出为新闻所属的种类，不需要额外解释，但保留思维链的内容；在输入的新闻标题前加上“新闻分类：”等类似标签来标明询问目的。

数据样例如下：

[
    {
        "conversations": [
            {
                "from": "human",
                "value": "新闻分类：给力！中国两次出手，美到手的订单黄了，该国从此只认中国制造！"
            },
            {
                "from": "gpt",
                "value": "<think>\n首先，我需要分析给出的新闻标题：“给力！中国两次出手，美到手的订单黄了，该国从此只认中国制造！”\n\n接下来，根据标题内容进行分类。标题中提到了中国两次行动导致美国订单出现问题，并且其他国家开始依赖中国制造，说明这涉及国家之间的经济合作和社会影响。\n\n结合新闻种类，考虑到涉及国际贸易和经济合作，最合适的分类是“经济”。所以，这条新闻应该归类到“经济”类别中。\n</think>\n\n经济"
            }
        ]
    },
    {
        "conversations": [
            {
                "from": "human",
                "value": "新闻分类：澳大利亚旅游"
            },
            {
                "from": "gpt",
                "value": "<think>\n好吧，我现在要分析这条新闻标题“澳大利亚旅游”属于哪个类别。首先，标题提到的是澳大利亚旅游，所以它和旅游相关。接下来，我需要看看新闻种类里有没有相关的类别。新闻种类包括政治、经济、科技、娱乐、体育、教育、健康、国际、国内、社会，以及旅游。\n\n根据知识储备，旅游本身就属于其中一个类别，可能属于社会或娱乐，但通常旅游更具体，可能单独作为一个类别。不过在这份分类中，并没有独自一个旅游的类别，可能需要看看有没有更合适的。比如，如果列举的是旅游，可能属于社会，因为旅游涉及到人们的生活方式和社会活动。\n\n再想一下，可能“旅游”不太明显属于哪一个大类，可能需要归类到社会中。或者，在给定的选项中，可能没有明确的旅游类别，所以可能需要根据其他因素来判断。假设在预定义的分类中，有旅游作为独立类别的话，那就是旅游；如果没有，就放到社会里。\n\n不过根据用户提供的分类，旅游并没有单独作为一个类，所以可能得归类到社会。或者，可能更接近于“国际”或“国内”，因为澳大利亚可能分为国际或国内旅游。但在这份列表里，国际和国内是两个类别，所以“澳大利亚旅游”作为国内的，可能更准确。\n\n再仔细分析，用户提供的新闻类型是国际、国内，所以如果是国内，应该是国内旅游；如果是跨国，比如其他国家的旅游，那就是国际。所以“澳大利亚旅游”属于国内，因为澳大利亚是中国的国内。所以最终归类到“国内”。\n</think>\n\n国内"
            }
        ]
    }
]

3. 启动微调任务

3.1.启动LLaMA Board

确认 LLaMA Factory 安装完成后，运行以下指令启动 LLaMA Board。这里用到的环境变量解释如下：

USE_MODELSCOPE_HUB设为1，表示模型来源是ModelScope。使用HuggingFace模型可能会有网络问题。
CUDA_VISIBLE_DEVICES：指定使用的显卡序号，默认全部使用

!export USE_MODELSCOPE_HUB=1 CUDA_VISIBLE_DEVICES=0 && llamafactory-cli webui

点击返回的URL地址，进入Web UI页面。 t-a-1

3.2.选择Qwen3-4B-Instruct模型

进入Web UI界面后，选择模型为Qwen3-4B-Instruct模型，模型路径可填写本地绝对路径，不填则从互联网下载 t-a-2

3.3.选择微调数据集

将数据路径改为使用我们上述打印的数据路径，选择train、eval数据。 t-a-3

3.4.配置微调参数

为了让模型更好地学习数据知识，将学习率改为 1e-4，训练轮数提高到 8 轮。批处理大小和梯度累计则根据设备显存大小调整，在显存允许的情况下提高批处理大小有助于加速训练，一般保持批处理大小×梯度累积×显卡数量等于 32 即可。 t-a-4

3.5.其他参数配置

点击其他参数设置，将保存间隔设置为 50，保存更多的检查点，有助于观察模型效果随训练轮数的变化。 t-a-5

3.6.LoRA参数设置

点击 LoRA 参数设置，将 LoRA 秩设置为 16，并把 LoRA 缩放系数设置为 32。 t-a-6

3.7.开始训练

点击开始按钮，等待模型下载，一段时间后应能观察到训练过程的损失曲线。 t-a-7

3.8.训练完成

等待模型训练完毕，视显卡性能，训练时间可能在 20-60 分钟不等。 t-a-8

4. 验证微调效果

4.1. 选择检查点路径为刚才的输出目录，打开 Chat 页面，点击加载模型 p-1

4.2. 在页面底部的对话框输入想要和模型对话的内容，点击「提交」即可发送消息

发送后模型会逐字生成回答，从回答中可以发现模型学习到了数据集中的内容，能够识别出用户是想对后面的标题进行新闻分类，且满足要求的输出格式，直接输出新闻分类，没有额外解释。

把前缀“新闻分类”改成类似的表达去询问，发现仍能满足要求。

清空历史后（避免上下文的影响），问模型其他的问题，发现其仍能正常回答，没有过拟合。

如何学习大模型技术，享受AI红利？

面对AI大模型开发领域的复杂与深入，精准学习显得尤为重要。一份系统的技术路线图，详尽的全套学习资料，不仅能够帮助开发者清晰地了解从入门到精通所需掌握的知识点，还能提供一条高效、有序的学习路径。

无论是初学者，还是希望在某一细分领域深入发展的资深开发者，这样的学习路线图都能够起到事半功倍的效果。它不仅能够节省大量时间，避免无效学习，更能帮助开发者建立系统的知识体系，为职业生涯的长远发展奠定坚实的基础。

这份完整版的大模型 AI 学习资料已经上传优快云，朋友们如果需要可以微信扫描下方优快云官方认证二维码免费领取【保证100%免费】

大模型知识脑图

为了成为更好的 AI大模型开发者，这里为大家提供了总的路线图。它的用处就在于，你可以按照上面的知识点去找对应的学习资源，保证自己学得较为全面。

read-normal-img

经典书籍阅读

阅读AI大模型经典书籍可以帮助读者提高技术水平，开拓视野，掌握核心技术，提高解决问题的能力，同时也可以借鉴他人的经验。对于想要深入学习AI大模型开发的读者来说，阅读经典书籍是非常有必要的。

AI大模型时代的转行选择，新风口能否抵御35岁中年危机？_语言模型_02

实战案例

光学理论是没用的，要学会跟着一起敲，要动手实操，才能将自己的所学运用到实际当中去，这时候可以搞点实战案例来学习。

AI大模型时代的转行选择，新风口能否抵御35岁中年危机？_语言模型_03

面试资料

我们学习AI大模型必然是想找到高薪的工作，下面这些面试题都是总结当前最新、最热、最高频的面试题，并且每道题都有详细的答案，面试前刷完这套面试题资料，小小offer，不在话下

AI大模型时代的转行选择，新风口能否抵御35岁中年危机？_ai_04

640套AI大模型报告合集

这套包含640份报告的合集，涵盖了AI大模型的理论研究、技术实现、行业应用等多个方面。无论您是科研人员、工程师，还是对AI大模型感兴趣的爱好者，这套报告合集都将为您提供宝贵的信息和启示。

AI大模型时代的转行选择，新风口能否抵御35岁中年危机？_ai_05

结语

大模型作为新时代的风口，确实为那些希望转行或寻求职业突破的人提供了广阔的舞台。然而，是否选择进入这一领域还需综合考虑自身的兴趣、特长以及长远规划。通过构建基础知识体系、参与实际项目、拓展软技能、关注跨学科融合以及建立广泛的社交网络，你可以在这个充满机遇的新领域中迅速站稳脚跟。

这份完整版的大模型 AI 学习资料已经上传优快云，朋友们如果需要可以微信扫描下方优快云官方认证二维码免费领取【保证100%免费】