基于LSTM的垃圾短信分类器实现教程-优快云博客

本文链接：https://blog.youkuaiyun.com/gitblog_00864/article/details/148758426

基于LSTM的垃圾短信分类器实现教程

pythoncode-tutorials The Python Code Tutorials 项目地址: https://gitcode.com/gh_mirrors/py/pythoncode-tutorials

前言

在当今数字化时代，短信分类技术对于过滤垃圾信息、保护用户隐私具有重要意义。本教程将详细介绍如何使用Python和TensorFlow构建一个基于LSTM的垃圾短信分类器，采用自然语言处理技术对短信进行分类。

环境准备

首先需要安装必要的Python库：

import tensorflow as tf
import numpy as np
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard
from sklearn.model_selection import train_test_split
from tensorflow.keras.layers import Embedding, LSTM, Dropout, Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.metrics import Recall, Precision

参数配置

在开始构建模型前，我们需要设置一些关键参数：

SEQUENCE_LENGTH = 100  # 每条短信的最大单词数
EMBEDDING_SIZE = 100   # 使用100维的GloVe词向量
TEST_SIZE = 0.25       # 测试集比例
BATCH_SIZE = 64        # 批处理大小
EPOCHS = 10            # 训练轮数

# 标签映射
label2int = {"ham": 0, "spam": 1}
int2label = {0: "ham", 1: "spam"}

数据加载与预处理

1. 加载数据集

我们使用经典的SMS垃圾短信数据集，包含正常短信(ham)和垃圾短信(spam)：

def load_data():
    texts, labels = [], []
    with open("data/SMSSpamCollection") as f:
        for line in f:
            split = line.split()
            labels.append(split[0].strip())
            texts.append(' '.join(split[1:]).strip())
    return texts, labels

2. 文本向量化

使用Tokenizer将文本转换为数字序列：

tokenizer = Tokenizer()
tokenizer.fit_on_texts(X)
# 保存tokenizer供后续使用
pickle.dump(tokenizer, open("results/tokenizer.pickle", "wb"))

X = tokenizer.texts_to_sequences(X)

3. 序列填充

为了保持输入长度一致，对序列进行填充：

X = pad_sequences(X, maxlen=SEQUENCE_LENGTH)

4. 标签编码

将文本标签转换为one-hot编码：

y = [label2int[label] for label in y]
y = to_categorical(y)

5. 数据集划分

将数据划分为训练集和测试集：

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=TEST_SIZE, random_state=7)

词嵌入处理

使用预训练的GloVe词向量：

def get_embedding_vectors(tokenizer, dim=100):
    embedding_index = {}
    with open(f"data/glove.6B.{dim}d.txt", encoding='utf8') as f:
        for line in tqdm.tqdm(f, "Reading GloVe"):
            values = line.split()
            word = values[0]
            vectors = np.asarray(values[1:], dtype='float32')
            embedding_index[word] = vectors
    
    word_index = tokenizer.word_index
    embedding_matrix = np.zeros((len(word_index)+1, dim))
    for word, i in word_index.items():
        embedding_vector = embedding_index.get(word)
        if embedding_vector is not None:
            embedding_matrix[i] = embedding_vector
    return embedding_matrix

模型构建

构建LSTM分类模型：

def get_model(tokenizer, lstm_units):
    embedding_matrix = get_embedding_vectors(tokenizer)
    model = Sequential()
    model.add(Embedding(len(tokenizer.word_index)+1,
              EMBEDDING_SIZE,
              weights=[embedding_matrix],
              trainable=False,
              input_length=SEQUENCE_LENGTH))

    model.add(LSTM(lstm_units, recurrent_dropout=0.2))
    model.add(Dropout(0.3))
    model.add(Dense(2, activation="softmax"))
    
    model.compile(optimizer="rmsprop", loss="categorical_crossentropy",
                metrics=["accuracy", Precision(), Recall()])
    model.summary()
    return model

模型训练

配置回调函数并开始训练：

model = get_model(tokenizer=tokenizer, lstm_units=128)

# 配置模型检查点和TensorBoard回调
model_checkpoint = ModelCheckpoint("results/spam_classifier_{val_loss:.2f}.h5", 
                                 save_best_only=True, verbose=1)
tensorboard = TensorBoard(f"logs/spam_classifier_{time.time()}")

# 开始训练
model.fit(X_train, y_train, validation_data=(X_test, y_test),
          batch_size=BATCH_SIZE, epochs=EPOCHS,
          callbacks=[tensorboard, model_checkpoint],
          verbose=1)

模型评估

评估模型性能：

result = model.evaluate(X_test, y_test)
loss = result[0]
accuracy = result[1]
precision = result[2]
recall = result[3]

print(f"[+] Accuracy: {accuracy*100:.2f}%")
print(f"[+] Precision: {precision*100:.2f}%")
print(f"[+] Recall: {recall*100:.2f}%")

预测功能

实现预测函数对新短信进行分类：

def get_predictions(text):
    sequence = tokenizer.texts_to_sequences([text])
    sequence = pad_sequences(sequence, maxlen=SEQUENCE_LENGTH)
    prediction = model.predict(sequence)[0]
    return int2label[np.argmax(prediction)]

测试示例：

text = "You won a prize of 1,000$, click here to claim!"
print(get_predictions(text))  # 输出: spam

text = "Hi man, I was wondering if we can meet tomorrow"
print(get_predictions(text))  # 输出: ham