本文介绍了一个基于IMDb电影评论的情感分析项目，利用深度学习技术进行二分类预测，包括数据预处理、模型构建、训练及评估过程。
概述
对imdb中的电影评论进行分类，是一个二分类的问题，这是一种重要且广泛适用的机器学习问题。
数据
imdb的数据包含50000条电影评论。拥有25000条训练数据以及25000条评估数据，有着相同数量的正面与负面评论。
下载imdb数据
imdb中的数据已经被预处理好，为整数序列，每个整数代表着一个特定单词。可用imdb的词典进行翻译。（https://s3.amazonaws.com/text-datasets/imdb.npz）如果不能科学上网，可以在https://pan.baidu.com/s/1pNDbE3VMdYJiiXyaN2roaw 提取码：0wnn下载
读取数据
import tensorflow as tf
from tensorflow import keras
import numpy as np
mdb = keras.datasets.imdb
(train_data, train_labels), (test_data, test_labels) = imdb.load_data('/home/kesci/input/idmb2286/imdb.npz',num_words=10000)
将load_data中的路径改为imdb.npz所在的路径，num_words=15000保留出现频率最高的前10000个词。丢弃罕见单词以保持数据的可管理。
了解数据
在处理数据前，我们需要先了解数据，经过数据的预处理后，每个例子都是整数序列，以整数来表示电影的单词。每个整数对应词典的一个单词。用0和1来确定label的种类。
print("Training entries: {}, labels: {}".format(len(train_data), len(train_labels)))
Training entries: 25000, labels: 25000
我们可以看下第一条评论
print(train_data[0])
[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 10311, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 12118, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]
每条数据的单词数量不同，然而神经网络的输入要求长度必须相同。我们将在下面解决这个问题。
len(train_data[0]),len(train_data[1])
(218,219)
将整数转换为文本
# A dictionary mapping words to an integer index
word_index = imdb.get_word_index()
# The first indices are reserved
word_index = {k:(v+3) for k,v in word_index.items()}
word_index["<PAD>"] = 0
word_index["<START>"] = 1
word_index["<UNK>"] = 2 # unknown
word_index["<UNUSED>"] = 3
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
def decode_review(text):
return ' '.join([reverse_word_index.get(i, '?') for i in text])
我们可以用decode_review将整数序列转换为文本
decode_review(train_data[20])
"<START> shown in australia as <UNK> this incredibly bad movie is so bad that you become <UNK> and have to watch it to the end just to see if it could get any worse and it does the storyline is so predictable it seems written by a high school <UNK> class the sets are pathetic but marginally better than the <UNK> and the acting is wooden br br the infant <UNK> seems to have been stolen from the props <UNK> of <UNK> <UNK> there didn't seem to be a single original idea in the whole movie br br i found this movie to be so bad that i laughed most of the way through br br malcolm mcdowell should hang his head in shame he obviously needed the money"
准备数据
必须在输入神经网络前转换为张量。
- 可转换为独热向量
- 或填充数组，使他们具有相同长度，然后创建一个num_example*max_length的整型张量。可以将此作为神经网络的第一层。
在次我们使用第二种方法
train_data = keras.preprocessing.sequence.pad_sequences(train_data,
value=word_index["<PAD>"],
padding='post',
maxlen=256)
test_data = keras.preprocessing.sequence.pad_sequences(test_data,
value=word_index["<PAD>"],
padding='post',
maxlen=256)
将数据集转换为256维，不足的从后面补齐（补零）。
len(train_data[0]), len(train_data[1])
（256，256）
看下处理后的数据
print(train_data[0])
[ 1 14 22 16 43 530 973 1622 1385 65 458 4468
66 3941 4 173 36 256 5 25 100 43 838 112
50 670 2 9 35 480 284 5 150 4 172 112
167 2 336 385 39 4 172 4536 1111 17 546 38
13 447 4 192 50 16 6 147 2025 19 14 22
4 1920 4613 469 4 22 71 87 12 16 43 530
38 76 15 13 1247 4 22 17 515 17 12 16
626 18 2 5 62 386 12 8 316 8 106 5
4 2223 5244 16 480 66 3785 33 4 130 12 16
38 619 5 25 124 51 36 135 48 25 1415 33
6 22 12 215 28 77 52 5 14 407 16 82
10311 8 4 107 117 5952 15 256 4 2 7 3766
5 723 36 71 43 530 476 26 400 317 46 7
4 12118 1029 13 104 88 4 381 15 297 98 32
2071 56 26 141 6 194 7486 18 4 226 22 21
134 476 26 480 5 144 30 5535 18 51 36 28
224 92 25 104 4 226 65 16 38 1334 88 12
16 283 5 16 4472 113 103 32 15 16 5345 19
178 32 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0]
创建模型
input shape is the vocabulary count used for the movie reviews (10,000 words)
vocab_size = 10000
model = keras.Sequential()
model.add(keras.layers.Embedding(vocab_size, 16))
model.add(keras.layers.GlobalAveragePooling1D())
#model.add(keras.layers.GlobalMaxPooling1D())
model.add(keras.layers.Dense(16, activation=tf.nn.relu))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(1, activation=tf.nn.sigmoid))
model.summary()
损失函数
模型需要一个损失函数和一个用于训练的优化器。 由于这是二元分类问题和概率模型输出（具有S形激活的单个单元层），我们将使用binary_crossentropy损失函数。
model.compile(optimizer=tf.train.AdamOptimizer(),
loss='binary_crossentropy',
metrics=['accuracy'])
建立验证集
10000以前为验证集，一万以后为训练集
x_val = train_data[:10000]
partial_x_train = train_data[10000:]
y_val = train_labels[:10000]
partial_y_train = train_labels[10000:]
训练模型
#每一Epochs都进行F1计算
import numpy as np
from keras.callbacks import Callback
from keras.engine.training import Model
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score
class Metrics(Callback):
def on_train_begin(self, logs={}):
self.val_f1s = []
self.val_recalls = []
self.val_precisions = []
def on_epoch_end(self, epoch, logs={}):
val_predict = (np.asarray(self.model.predict(self.validation_data[0]))).round()
val_targ = self.validation_data[1]
_val_f1 = f1_score(val_targ, val_predict,average='weighted')
_val_recall = recall_score(val_targ, val_predict,average='weighted')
_val_precision = precision_score(val_targ, val_predict,average='weighted')
self.val_f1s.append(_val_f1)
self.val_recalls.append(_val_recall)
self.val_precisions.append(_val_precision)
print( ' — val_f1: %f — val_precision: %f — val_recall %f' %(_val_f1, _val_precision, _val_recall))
return
metrics = Metrics()
from keras.callbacks import EarlyStopping
earlystopping=keras.callbacks.EarlyStopping(monitor='val_acc', patience=8, verbose=0, mode='max')
history = model.fit(partial_x_train,
partial_y_train,
epochs=90,
batch_size=512,
validation_data=(x_val, y_val),
callbacks=[metrics,earlystopping],
verbose=1)
测试模型
results = model.evaluate(test_data, test_labels)
print(results)
25000/25000 [==============================] - 2s 61us/step
[0.31110355438232423, 0.87736]
我们可以看到损失函数为0.31，准确率为0.87.
扫一扫
784
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
