Python和PyTorch对比实现批标准化 Batch Normalization 函数在测试或推理过程中的算法

摘要

本文使用Python和PyTorch对比实现批标准化 Batch Normalization 函数在测试或推理过程中的算法.

相关

原理及详细解释, 请参考文章 :

Batch Normalization的测试或推理过程及样本参数更新方法.

系列文章索引 :
https://blog.youkuaiyun.com/oBrightLamp/article/details/85067981

正文

1. Batch Normalization 类

文件目录 : vanilla_nn/batch_normalization.py

import numpy as np


class BatchNorm1d:
    def __init__(self, train=True, momentum=0.1, eps=1e-5):
        self.train = train
        self.momentum = momentum
        self.eps = eps

        self.weight = None
        self.bias = None

        self.std = None
        self.dw = None
        self.db = None

        self.sqrt = None
        self.std = None

        self.running_mean = None
        self.running_var = None

    def __call__(self, x):
        if self.train is True:
            mean = np.mean(x, axis=0, keepdims=True)
            var = np.var(x, axis=0, keepdims=True)
            sqrt = np.sqrt(var + self.eps)
            std = (x - mean) / sqrt
            self.sqrt = sqrt
            self.std = std

            if self.running_mean is None:
                self.running_mean = np.zeros_lik