循环神经网络中等篇

循环神经网络（RNN）中等篇：进阶模型与实战技巧

一、RNN核心改进方案

1.1 常见变体架构对比

模型类型	核心改进	适用场景
LSTM	门控机制解决梯度消失	长序列依赖任务
GRU	简化门控结构	资源受限的实时场景
双向RNN	双向信息捕捉	上下文敏感任务

1.2 LSTM结构解析

LSTM单元包含三个门控：

class LSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.input_gate = nn.Linear(input_size + hidden_size, hidden_size)
        self.forget_gate = nn.Linear(input_size + hidden_size, hidden_size)
        self.output_gate = nn.Linear(input_size + hidden_size, hidden_size)
        self.cell_gate = nn.Linear(input_size + hidden_size, hidden_size)
    
    def forward(self, x, hc):
        h, c = hc
        combined = torch.cat((x, h), dim=1)
        i = torch.sigmoid(self.input_gate(combined))
        f = torch.sigmoid(self.forget_gate(combined))
        o = torch.sigmoid(self.output_gate(combined))
        g = torch.tanh(self.cell_gate(combined))
        c_new = f * c + i * g
        h_new = o * torch.tanh(c_new)
        return (h_new, c_new)

二、梯度问题解决方案

2.1 梯度裁剪实战

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(epochs):
    for inputs, targets in dataloader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = loss_fn(outputs, targets)
        loss.backward()
        # 梯度裁剪
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        optimizer.step()

2.2 参数初始化技巧

# LSTM权重初始化
for name, param in model.named_parameters():
    if 'weight_ih' in name:
        nn.init.xavier_uniform_(param)
    elif 'weight_hh' in name:
        nn.init.orthogonal_(param)
    elif 'bias' in name:
        nn.init.zeros_(param)

三、序列处理实战技巧

3.1 变长序列处理

from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence

# 填充序列
padded_seq = pad_sequence(sequences, batch_first=True)
# 创建长度张量
lengths = torch.tensor([len(seq) for seq in sequences])
# 打包序列
packed_input = pack_padded_sequence(padded_seq, lengths, 
                                   batch_first=True, 
                                   enforce_sorted=False)

3.2 注意力机制集成

class Attention(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.attn = nn.Linear(hidden_size * 2, hidden_size)
        self.v = nn.Linear(hidden_size, 1, bias=False)

    def forward(self, hidden, encoder_outputs):
        seq_len = encoder_outputs.size(0)
        hidden = hidden.repeat(seq_len, 1, 1)
        energy = torch.tanh(self.attn(torch.cat((hidden, encoder_outputs), dim=2))
        attention = self.v(energy).squeeze(2)
        return torch.softmax(attention, dim=1)

四、典型应用案例

4.1 股票预测模型架构

输入层 -> 双向LSTM -> 注意力层 -> 全连接层 -> 输出

4.2 文本生成示例

model = nn.LSTM(input_size=embed_dim, 
               hidden_size=256, 
               num_layers=3,
               dropout=0.2)

# 温度采样生成
def generate_text(seed, temperature=0.8):
    hidden = None
    for _ in range(100):
        output, hidden = model(seed, hidden)
        prob = torch.softmax(output / temperature, dim=1)
        next_token = torch.multinomial(prob, 1)
        seed = next_token
    return generated_text

五、常见问题FAQ

Q：LSTM和GRU如何选择？
A：GRU参数更少适合小数据集，LSTM理论表达能力更强

Q：如何处理超长序列？
A：可采用截断BPTT、增大隐层维度、结合注意力机制

Q：注意力机制在RNN中的作用？
A：增强关键时间步关注，缓解长距离依赖问题

Q：如何提高训练效率？
A：使用CuDNN加速、开启JIT编译、调整batch_size

---

相关标签： #深度学习 #循环神经网络 #LSTM #PyTorch #时间序列预测