【大模型】解决报错ValueError: Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.

运行Qwen1.5-7B-Chat大模型时报错：

2024-04-29 17:50:54,192 - INFO - intel_extension_for_pytorch auto imported
Traceback (most recent call last):
  File "/root/xx/llm/ipex/test-xpu.py", line 16, in <module>
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/xx/miniconda3/envs/llm-ipex/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 699, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.

环境

• ubuntu 22.04
• python 3.11

错误信息

2024-04-29 17:50:54,192 - INFO - intel_extension_for_pytorch auto imported
Traceback (most recent call last):
  File "/root/xx/llm/ipex/test-xpu.py", line 16, in <module>
    tokenizer = A