一文搞懂Transformer应用：文本分类、QA、NER全流程代码解析

最新推荐文章于 2025-09-21 15:36:42 发布

原创最新推荐文章于 2025-09-21 15:36:42 发布 · 921 阅读

13 ·

CC 4.0 BY-SA版权

文章标签：

#transformer #深度学习 #程序员 #ai #人工智能 #转行 #大模型

本文将通过代码实战带你快速掌握NLP三大核心任务，使用Hugging Face Transformer库实现工业级AI应用开发。

一、环境准备

pip install transformers datasets torch tensorboard

二、文本分类实战（情感分析）

1. 数据加载与预处理

from datasets import load_datasetfrom transformers import AutoTokenizerdataset = load_dataset("imdb")tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")def tokenize_function(examples):    return tokenizer(examples["text"], padding="max_length", truncation=True)tokenized_datasets = dataset.map(tokenize_function, batched=True)

2. 模型训练

from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainermodel = AutoModelForSequenceClassification.from_pretrained(    "distilbert-base-uncased", num_labels=2)training_args = TrainingArguments(    output_dir="./results",    evaluation_strategy="epoch",    learning_rate=2e-5,    per_device_train_batch_size=16,    num_train_epochs=3,)trainer = Trainer(    model=model,    args=training_args,    train_dataset=tokenized_datasets["train"],    eval_dataset=tokenized_datasets["test"],)trainer.train()

3. 推理预测

from transformers import pipelineclassifier = pipeline("text-classification", model=model, tokenizer=tokenizer)result = classifier("This movie was absolutely fantastic!")print(result)  # [{'label': 'POSITIVE', 'score': 0.999}]

在这里插入图片描述

三、问答系统实战（SQuAD数据集）

1. 加载问答数据集

dataset = load_dataset("squad")tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")def preprocess_function(examples):    questions = [q.strip() for q in examples["question"]]    inputs = tokenizer(        questions,        examples["context"],        max_length=384,        truncation="only_second",        return_offsets_mapping=True,        padding="max_length",    )    return inputstokenized_squad = dataset.map(preprocess_function, batched=True)

2. 训练问答模型

from transformers import AutoModelForQuestionAnsweringmodel = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-uncased")training_args = TrainingArguments(    output_dir="./qa_results",    evaluation_strategy="epoch",    learning_rate=3e-5,    per_device_train_batch_size=12,    num_train_epochs=2,)trainer = Trainer(    model=model,    args=training_args,    train_dataset=tokenized_squad["train"],    eval_dataset=tokenized_squad["validation"],)trainer.train()

3. 执行问答

question = "What does NLP stand for?"context = "Natural Language Processing (NLP) is a subfield of artificial intelligence."qa_pipeline = pipeline("question-answering", model=model, tokenizer=tokenizer)result = qa_pipeline(question=question, context=context)print(result)# {'score': 0.982, 'start': 0, 'end': 24, 'answer': 'Natural Language Processing'}

四、命名实体识别实战（CoNLL-2003）

1. 数据预处理

dataset = load_dataset("conll2003")tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")label_list = dataset["train"].features["ner_tags"].feature.namesdef tokenize_and_align_labels(examples):    tokenized_inputs = tokenizer(        examples["tokens"],         truncation=True,        is_split_into_words=True    )        labels = []    for i, label in enumerate(examples["ner_tags"]):        word_ids = tokenized_inputs.word_ids(batch_index=i)        previous_word_idx = None        label_ids = []        for word_idx in word_ids:            if word_idx is None:                label_ids.append(-100)            elif word_idx != previous_word_idx:                label_ids.append(label[word_idx])            else:                label_ids.append(-100)            previous_word_idx = word_idx        labels.append(label_ids)        tokenized_inputs["labels"] = labels    return tokenized_inputstokenized_dataset = dataset.map(tokenize_and_align_labels, batched=True)

2. 训练NER模型

from transformers import AutoModelForTokenClassificationmodel = AutoModelForTokenClassification.from_pretrained(    "bert-base-cased",     num_labels=len(label_list)    training_args = TrainingArguments(    output_dir="./ner_results",    evaluation_strategy="epoch",    learning_rate=2e-5,    per_device_train_batch_size=16,    num_train_epochs=3,)trainer = Trainer(    model=model,    args=training_args,    train_dataset=tokenized_dataset["train"],    eval_dataset=tokenized_dataset["validation"],)trainer.train()

3. 实体识别推理

from transformers import pipelinener_pipeline = pipeline("ner", model=model, tokenizer=tokenizer)sample_text = "Apple was founded by Steve Jobs in Cupertino, California."entities = ner_pipeline(sample_text)for entity in entities:    print(f"{entity['word']} -> {label_list[entity['entity'][-1]]}")    # Apple -> B-ORG# Steve Jobs -> B-PER# Cupertino -> B-LOC# California -> B-LOC