【自然语言处理二】电影评论文本分类

本文介绍利用LSTM模型，实现评论文本分类的过程。这是一个二元（binary）或者二分类问题，一种重要且应用广泛的机器学习问题。

1.预处理

	imdb = keras.datasets.imdb
    (train_data, train_labels), (test_data,
                                 test_labels) = imdb.load_data(num_words=10000)

# 一个映射单词到整数索引的词典
    word_index = imdb.get_word_index()

# 保留第一个索引
    word_index = {k: (v+3) for k, v in word_index.items()}
    word_index["<PAD>"] = 0
    word_index["<START>"] = 1
    word_index["<UNK>"] = 2  # unknown
    word_index["<UNUSED>"] = 3

    reverse_word_index = dict([(value, key)
                              for (key, value) in word_index.items()])
    train_data = keras.preprocessing.sequence.pad_sequences(
        train_data, value=word_index["<PAD>"], padding='post', maxlen=word_num)
    test_data = keras.preprocessing.sequence.pad_sequences(
        test_data, value=word_index["<PAD>"], padding='post', maxlen=word_num)

    x_val = train_data[:10000]
    partial_x_train = train_data[10000:]

    y_val = train_labels[:10000]
    partial_y_train = train_labels[10000:]

2.构建模型

	epochs = 40
    model = Sequential()
    model.add(Embedding(vocab, embedding_dim))
    model.add(LSTM(state_dim, return_sequences=False))
    model.add(keras.layers.Dense(16, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))

    model.summary()
    model.compile(optimizer='adam',
                  loss='binary_crossentropy', metrics=['acc'])

3.训练及测试

history = model.fit(partial_x_train, partial_y_train, epochs=epochs,
                        batch_size=batch_size, validation_data=(x_val, y_val), verbose=1)