pytorch实现的TextCNN（Dataset, DataLoader的使用）

最新推荐文章于 2025-04-25 00:14:14 发布

SinGaln

最新推荐文章于 2025-04-25 00:14:14 发布

阅读量1.2k

点赞数

分类专栏：深度学习（deep learning）文章标签：深度学习 python pytorch

本文链接：https://blog.youkuaiyun.com/qq_27492735/article/details/110039109

版权

本文介绍了如何使用PyTorch构建TextCNN模型，并探讨了Dataset和DataLoader在训练过程中的应用，旨在提供深度学习文本分类的实践示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

主要是Dataset, DataLoader的使用

（1）数据处理，生成Batch和向量化词表

import torch
import numpy as np
from tqdm import tqdm
from torch.utils.data import Dataset, DataLoader

tokenizer = lambda x: [y for y in x]
UNK, PAD = '<UNK>', '<PAD>'  # 未知字，padding符号


def build_vocab(path, max_size, freq_min):
    vocab_dic = {
   }
    with open(path, "r", encoding="utf-8") as f:
        for sentences in tqdm(f):
            sentence = sentences.strip()
            if not sentence:
                continue
            content = sentence.split("\t")[0]
            for word in tokenizer(content):
                vocab_dic[word] = vocab_dic.get(word, 0) + 1
        vocab_list = sorted([_ for _ in vocab_dic.items() if _[1] >= freq_min], key=lambda x: x[1], reverse=True)[
                     :max_size]
        vocab_dic = {
   word_count[0]: idx for idx, word_count in enumerate(vocab_list)}
        vocab_dic.update({
   UNK: len(vocab_dic), PAD: len(vocab_dic) + 1})
    return vocab_dic


def load_data(path, padding_size=32):
    contents = []
    vocab = build_vocab(path, max_size=10000, freq_min=1)
    with open(path, "r", encoding="utf-8") as f:
        for sentences in tqdm(f):
            sentence = sentences.strip()
            if not sentence:
                continue
            text, label = sentence.split("\t")
            word_line = []
            token = tokenizer(text)
            seq_len = len(token)
            if padding_size:
                if len(token) < padding_size:
                    token.extend([PAD] * (padding_size - len(token)))

最低0.47元/天解锁文章