24、深度学习中的RNN、LSTM与联邦学习

深度学习中的RNN、LSTM与联邦学习

1. RNN反向传播示例

为了直观地观察梯度消失和梯度爆炸的现象，我们可以构建一个简单的示例。以下代码展示了使用sigmoid和relu激活函数的循环反向传播过程：

import numpy as np

(sigmoid, relu) = (lambda x: 1 / (1 + np.exp(-x)), lambda x: (x > 0).astype(float) * x)
weights = np.array([[1, 4], [4, 1]])
activation = sigmoid(np.array([1, 0.01]))

print("Sigmoid Activations")
activations = list()
for iter in range(10):
    activation = sigmoid(activation.dot(weights))
    activations.append(activation)
    print(activation)

print("\nSigmoid Gradients")
gradient = np.ones_like(activation)
for activation in reversed(activations):
    gradient = (activation * (1 - activation) * gradient)
    gradient = gradient.dot(weights.transpose())
    print(gradient)

print("Relu Activation