【人工智能项目】Bert实现问答预测系统

关注

原创最新推荐文章于 2026-05-08 09:40:18 发布 2.3k 阅读 AI 写同款· GEO 优化 ›

BERT 模型微调实战：使用Transformers微调BERT模型实现问答和文本分类任务 BERT（Bidirectional Encoder Representations from Transformers）是一种预训练的自然语言处理模型，由Google于2018年发布。BERT模型的核心是Transformer编码器，它可以在大规模语料库上进行无监督预训练，然后通过微调在各种NLP任务上进行微调。BERT模型是一种双向的深度学习模型，它可以同时考虑上下文中的所有单词，从而更好地理解句子的含义。BERT模型已被证明在多项NLP任务上取得了最佳结果，包括问答、文本分类、命名实体识别等。阅读详情

【人工智能项目】Bert实现问答预测系统

在这里插入图片描述

DrQA 是一个开放域的问答系统。向 DrQA 系统输入一段文本，然后提一个答案能在该文本中找到的问题，那么 DrQA 就能给出这个问题的答案。

本次任务就是用Bert实现一个问答系统，输入Q1问题，Q2文本，标签为是否能在Q2中找到Q1的答案。

那么话不多说，走起！！！
在这里插入图片描述

环境

本次环境是在google colab进行的，所用的gpu情况具体如下所示：

!nvidia-smi

在这里插入图片描述

导入所需的模块

#! -*- coding:utf-8 -*-
import re, os, json, codecs, gc
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import os
import time
import datetime

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.utils import class_weight as cw

from keras import Sequential

from keras.models import Model

from keras.layers import LSTM,Activation,Dense,Dropout,Input,Embedding,BatchNormalization,Add,concatenate,Flatten
from keras.layers import Conv1D,Conv2D,Convolution1D,MaxPool1D,SeparableConv1D,SpatialDropout1D,GlobalAvgPool1D,GlobalMaxPool1D,GlobalMaxPooling1D
from keras.layers.pooling import _GlobalPooling1D
from keras.layers import MaxPooling2D,GlobalMaxPooling2D,GlobalAveragePooling2D

from keras.optimizers import RMSprop,Adam

from keras.preprocessing.text import Tokenizer
from keras.preprocessing import sequence

from keras.utils import to_categorical

from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint
from keras.callbacks import ReduceLROnPlateau

%matplotlib inline

import warnings
warnings.filterwarnings("ignore")

读取文件

train_dataset_path = "./train.tsv"
test_dataset_path = "./test.tsv"

train_df = pd.read_csv(train_dataset_path, sep='\t',names=["Q","Q1","Q2","label"])
test_df = pd.read_csv(test_dataset_path, sep='\t',names=["Q","Q1","Q2","label"])

train_df

在这里插入图片描述

test_df

在这里插入图片描述

去除无标签的数据

train_df.dropna(axis=0, how='any', inplace=True)

train_df

在这里插入图片描述

清洗文本数据

对文本中的一些特殊符号进行数据清洗工作。

import re
def filter(text):
    text = re.sub("[A-Za-z0-9\!\=\？\%\[\]\,\（\）\>\<:&lt;\/#\. -----\_]", "", text)
    text = text.replace('图片', '')
    text = text.replace('\xa0', '') # 删除nbsp
    # new
    r1 =  "\\【.*?】+|\\《.*?》+|\\#.*?#+|[.!/_,$&%^*()<>+""'?@|:~{}#]+|[——！\\\，。=？、：“”‘’￥……（）《》【】]"
    cleanr = re.compile('<.*?>')
    text = re.sub(cleanr, ' ', text)        #去除html标签
    text = re.sub(r1,'',text)
    text = text.strip()
    return text

def clean_text(data):
    data['text'] = data['Q1'].apply(lambda x: filter(x))
    data['text'] = data['Q2'].apply(lambda x: filter(x))

    return data

train = clean_text(train_df)
test = clean_text(test_df)

在这里插入图片描述

数据预处理

对Q1和Q2这两列数据进行合并，以方便后续处理工作。

train_df["ocr"] = train_df["Q1"] + "  " + train_df["Q2"]
test_df["ocr"] = test_df["Q1"] + "  " + test_df["Q2"]

在这里插入图片描述

标签统计

sns.countplot(train_df["label"])
plt.xlabel("Label")
plt.title("News sentiment analysis")

在这里插入图片描述

sns.countplot(test_df["label"])
plt.xlabel("Label")
plt.title("News sentiment analysis")

在这里插入图片描述

最终训练集

train_df.shape

(21527, 6)

在这里插入图片描述

test_df

在这里插入图片描述

安装bert

!pip install keras_bert

Looking in indexes: http://mirrors.aliyun.com/pypi/simple/
Requirement already satisfied: keras_bert in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (0.81.0)
Requirement already satisfied: numpy in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from keras_bert) (1.18.1)
Requirement already satisfied: Keras in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from keras_bert) (2.3.1)
Requirement already satisfied: keras-transformer>=0.30.0 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from keras_bert) (0.33.0)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from Keras->keras_bert) (1.1.0)
Requirement already satisfied: scipy>=0.14 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from Keras->keras_bert) (1.4.1)
Requirement already satisfied: h5py in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from Keras->keras_bert) (2.8.0)
Requirement already satisfied: keras-applications>=1.0.6 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from Keras->keras_bert) (1.0.8)
Requirement already satisfied: pyyaml in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from Keras->keras_bert) (5.3)
Requirement already satisfied: six>=1.9.0 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from Keras->keras_bert) (1.14.0)
Requirement already satisfied: keras-multi-head>=0.22.0 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from keras-transformer>=0.30.0->keras_bert) (0.22.0)
Requirement already satisfied: keras-layer-normalization>=0.12.0 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from keras-transformer>=0.30.0->keras_bert) (0.14.0)
Requirement already satisfied: keras-embed-sim>=0.7.0 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from keras-transformer>=0.30.0->keras_bert) (0.7.0)
Requirement already satisfied: keras-pos-embd>=0.10.0 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from keras-transformer>=0.30.0->keras_bert) (0.11.0)
Requirement already satisfied: keras-position-wise-feed-forward>=0.5.0 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from keras-transformer>=0.30.0->keras_bert) (0.6.0)
Requirement already satisfied: keras-self-attention==0.41.0 in /root/miniconda3/envs/myconda/lib/python3.7/site-packages (from keras-multi-head>=0.22.0->keras-transformer>=0.30.0->keras_bert) (0.41.0)

下载本次所需的bert预训练好的模型

!wget -c https://storage.googleapis.com/chineseglue/pretrain_models/roeberta_zh_L-24_H-1024_A-16.zip

–2020-05-06 13:01:24-- https://storage.googleapis.com/chineseglue/pretrain_models/roeberta_zh_L-24_H-1024_A-16.zip
Resolving storage.googleapis.com (storage.googleapis.com)… 172.253.120.128, 2a00:1450:400c:c00::80
Connecting to storage.googleapis.com (storage.googleapis.com)|172.253.120.128|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 1208484809 (1.1G) [application/zip]
Saving to: ‘roeberta_zh_L-24_H-1024_A-16.zip’
roeberta_zh_L-24_H- 100%[===================>] 1.12G 82.3MB/s in 15s
2020-05-06 13:01:40 (76.2 MB/s) - ‘roeberta_zh_L-24_H-1024_A-16.zip’ saved [1208484809/1208484809]

!unzip roeberta_zh_L-24_H-1024_A-16.zip

Archive: roeberta_zh_L-24_H-1024_A-16.zip
inflating: bert_config_large.json
inflating: checkpoint
inflating: roberta_zh_large_model.ckpt.data-00000-of-00001
inflating: roberta_zh_large_model.ckpt.index
inflating: roberta_zh_large_model.ckpt.meta
inflating: vocab.txt

训练

#! -*- coding:utf-8 -*-
import re, os, json, codecs, gc
import numpy as np
import pandas as pd
from random import choice
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import KFold
from keras_bert import load_trained_model_from_checkpoint, Tokenizer


from keras.layers import *
from keras.callbacks import *
from keras.models import Model
import keras.backend as K
from keras.optimizers import Adam

maxlen = 128
config_path = './bert_config_large.json'
# checkpoint_path = '/export/home/liuyuzhong/kaggle/bert/chinese_L-12_H-768_A-12/bert_model.ckpt'
checkpoint_path = './roberta_zh_large_model.ckpt'
dict_path = './vocab.txt'

token_dict = {}
with codecs.open(dict_path, 'r', 'utf8') as reader:
    for line in reader:
        token = line.strip()
        token_dict[token] = len(token_dict)

class OurTokenizer(Tokenizer):
    def _tokenize(self, text):
        R = []
        for c in text:
            if c in self._token_dict:
                R.append(c)
            elif self._is_space(c):
                R.append('[unused1]') # space类用未经训练的[unused1]表示
            else:
                R.append('[UNK]') # 剩余的字符是[UNK]
        return R

tokenizer = OurTokenizer(token_dict)

def seq_padding(X, padding=0):
    L = [len(x) for x in X]
    ML = max(L)
    return np.array([
        np.concatenate([x, [padding] * (ML - len(x))]) if len(x) < ML else x for x in X
    ])

class data_generator:
    def __init__(self, data, batch_size=4, shuffle=True):
        self.data = data
        self.batch_size = batch_size
        self.shuffle = shuffle
        self.steps = len(self.data) // self.batch_size
        if len(self.data) % self.batch_size != 0:
            self.steps += 1
    def __len__(self):
        return self.steps
    def __iter__(self):
        while True:
            idxs = list(range(len(self.data)))
            
            if self.shuffle:
                np.random.shuffle(idxs)
            
            X1, X2, Y = [], [], []
            for i in idxs:
                d = self.data[i]
                text = d[0][:maxlen]
                x1, x2 = tokenizer.encode(first=text)
                y = d[1]
                X1.append(x1)
                X2.append(x2)
                Y.append([y])
                if len(X1) == self.batch_size or i == idxs[-1]:
                    X1 = seq_padding(X1)
                    X2 = seq_padding(X2)
                    Y = seq_padding(Y)
                    yield [X1, X2], Y[:, 0, :]
                    [X1, X2, Y] = [], [], []

from keras.metrics import top_k_categorical_accuracy
def acc_top5(y_true, y_pred):
    return top_k_categorical_accuracy(y_true, y_pred, k=5)
                    
def build_bert(nclass):
    bert_model = load_trained_model_from_checkpoint(config_path, checkpoint_path, seq_len=None)

    for l in bert_model.layers:
        l.trainable = True

    x1_in = Input(shape=(None,))
    x2_in = Input(shape=(None,))

    x = bert_model([x1_in, x2_in])
    x = Lambda(lambda x: x[:, 0])(x)
    p = Dense(nclass, activation='softmax')(x)

    model = Model([x1_in, x2_in], p)
    model.compile(loss='categorical_crossentropy', 
                  optimizer=Adam(1e-5),
                  metrics=['accuracy', acc_top5])
    print(model.summary())
    return model

def run_cv(nfold, data, data_label, data_test):
    kf = KFold(n_splits=nfold, shuffle=True, random_state=520).split(data)
    # 改
    train_model_pred = np.zeros((len(data), 2))
    test_model_pred = np.zeros((len(data_test), 2))
        
    k = 0
    for i, (train_fold, test_fold) in enumerate(kf):
        X_train, X_valid, = data[train_fold, :], data[test_fold, :]
        # 改
        model = build_bert(2)
        early_stopping = EarlyStopping(monitor='val_acc', patience=3)
        plateau = ReduceLROnPlateau(monitor="val_acc", verbose=1, mode='max', factor=0.5, patience=2)
        checkpoint = ModelCheckpoint('./bert_dump/' + str(i) + '.hdf5', monitor='val_acc', 
                                         verbose=2, save_best_only=True, mode='max',save_weights_only=True)
        
        train_D = data_generator(X_train, shuffle=True)
        valid_D = data_generator(X_valid, shuffle=True)
        test_D = data_generator(data_test, shuffle=False)
        
        history = model.fit_generator(
            train_D.__iter__(),
            steps_per_epoch=len(train_D),
            epochs=3,
            validation_data=valid_D.__iter__(),
            validation_steps=len(valid_D),
            callbacks=[early_stopping, plateau, checkpoint],
        )
        
        # return model
        train_model_pred[test_fold, :] =  model.predict_generator(valid_D.__iter__(), steps=len(valid_D),verbose=1)
        test_model_pred += model.predict_generator(test_D.__iter__(), steps=len(test_D),verbose=1)
        
        del model; gc.collect()
        K.clear_session()
        
        # break
        if k == 0:
            break
        
    return train_model_pred, test_model_pred,history

train_model_pred, test_model_pred,history = run_cv(2, DATA_LIST, None, DATA_LIST_TEST)

在这里插入图片描述

绘制loss和acc曲线

# 绘制训练过程中的 loss 和 acc 变化曲线
import matplotlib.pyplot as plt
%matplotlib inline

def history_plot(history_fit):
    plt.figure(figsize=(12,6))
    
    # summarize history for accuracy
    plt.subplot(121)
    plt.plot(history_fit.history["accuracy"])
    plt.plot(history_fit.history["val_accuracy"])
    plt.title("model accuracy")
    plt.ylabel("accuracy")
    plt.xlabel("epoch")
    plt.legend(["train", "valid"], loc="upper left")
    
    # summarize history for loss
    plt.subplot(122)
    plt.plot(history_fit.history["loss"])
    plt.plot(history_fit.history["val_loss"])
    plt.title("model loss")
    plt.ylabel("loss")
    plt.xlabel("epoch")
    plt.legend(["train", "test"], loc="upper left")
    
    plt.show()

history_plot(history)

预测代码

test_pred = [np.argmax(x) for x in test_model_pred]
test_df['labels'] = test_pred
test_df['labels']
from sklearn.metrics import classification_report
print(classification_report(test_df["label"], test_df["labels"]))