【机器学习笔记】正则化

原创已于 2023-09-15 17:34:33 修改 · 380 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#机器学习 #笔记 #人工智能

于 2023-09-15 17:34:18 首次发布

人工智能专栏收录该内容

8 篇文章

订阅专栏

博客介绍了正则化用于解决过拟合问题的方法，通过限制权重削减其对模型的影响。详细阐述了线性回归和逻辑回归的正则化代价函数，分析了参数λ对权重分布的影响，并给出代码。还说明了通过梯度下降求解最优参数组合，对比了正则化前后公式的差异。

1.简介

正则化用于解决过拟合问题是一个非常好的方法！！通过限制某个项对应的权重来削减其对模型的影响，同时又保留该项不至于删掉特征值导致模型不够完整

2. 正则化代价函数

2.1 线性回归

其代价函数经过正则化后如下所示
$J(w⃗,b)=12m∑i=0m−1(fw⃗,b(x(i))−y(i))2+λ2m∑j=0n−1wj2J(\vec{w},b) = \frac{1}{2m} \sum\limits_{i = 0}^{m-1} (f_{\vec{w},b}(\mathbf{x}^{(i)}) - y^{(i)})^2 + \frac{\lambda}{2m} \sum_{j=0}^{n-1} w_j^2$ 其中 $λ\lambda$ 是一个很小的数，通过它来约束 $w_j$ ,当然读者也可以对 $b$ 也进行正则化，但是一般都只正则化 $w_j$

这里的λ当取值较大时，所限制的w更趋向于正态分布；而当取值较小时，所限制的w更趋向平均分布

以上正则化代价函数的最终代码如下

def compute_cost_linear_reg(X, y, w, b, lambda_ = 1):
    """
    Computes the cost over all examples
    Args:
      X (ndarray (m,n): Data, m examples with n features
      y (ndarray (m,)): target values
      w (ndarray (n,)): model parameters  
      b (scalar)      : model parameter
      lambda_ (scalar): Controls amount of regularization
    Returns:
      total_cost (scalar):  cost 
    """

    m  = X.shape[0]
    n  = len(w)
    cost = 0.
    #左半部分
    for i in range(m):
        f_wb_i = np.dot(X[i], w) + b                                   #(n,)(n,)=scalar, see np.dot
        cost = cost + (f_wb_i - y[i])**2                               #scalar             
    cost = cost / (2 * m)                                              #scalar  
 	#右半部分
    reg_cost = 0
    for j in range(n):
        reg_cost += (w[j]**2)                                          #scalar
    reg_cost = (lambda_/(2*m)) * reg_cost                              #scalar
    
    total_cost = cost + reg_cost                                       #scalar
    return total_cost                                                  #scalar

2.2 逻辑回归

其代价函数经过正则化后如下所示
$J(w⃗,b)=1m∑i=0m−1[−y(i)log⁡(fw⃗,b(x(i)))−(1−y(i))log⁡(1−fw⃗,b(x(i)))]+λ2m∑j=0n−1wj2J(\vec{w},b) = \frac{1}{m} \sum_{i=0}^{m-1} \left[ -y^{(i)} \log\left(f_{\vec{w},b}\left( \mathbf{x}^{(i)} \right) \right) - \left( 1 - y^{(i)}\right) \log \left( 1 - f_{\vec{w},b}\left( \mathbf{x}^{(i)} \right) \right) \right] \\ + \frac{\lambda}{2m} \sum_{j=0}^{n-1} w_j^2$ 其中 $λ\lambda$ 的解释同上，注意逻辑回归用的loss function与线性回归不同噢
以上正则化代价函数的最终代码如下

def compute_cost_logistic_reg(X, y, w, b, lambda_ = 1):
    """
    Computes the cost over all examples
    Args:
    Args:
      X (ndarray (m,n): Data, m examples with n features
      y (ndarray (m,)): target values
      w (ndarray (n,)): model parameters  
      b (scalar)      : model parameter
      lambda_ (scalar): Controls amount of regularization
    Returns:
      total_cost (scalar):  cost 
    """

    m,n  = X.shape
    cost = 0.
    #上左式
    for i in range(m):
        z_i = np.dot(X[i], w) + b                                      #(n,)(n,)=scalar, see np.dot
        f_wb_i = sigmoid(z_i)                                          #scalar
        cost +=  -y[i]*np.log(f_wb_i) - (1-y[i])*np.log(1-f_wb_i)      #scalar
             
    cost = cost/m                                                      #scalar
 	#上右式
    reg_cost = 0
    for j in range(n):
        reg_cost += (w[j]**2)                                          #scalar
    reg_cost = (lambda_/(2*m)) * reg_cost                              #scalar
    
    total_cost = cost + reg_cost                                       #scalar
    return total_cost                                                  #scalar

3. 梯度下降

有了代价函数之后我们要通过梯度下降来求解最优的参数组合，那其公式跟之前没用正则化的时候有什么不同呢🧐 大体上跟之前的样式不变还是，如下
$b=b−α∂J(w⃗,b)∂b}\begin{align*} &\text{repeat until convergence:} \; \lbrace \\ & \; \; \;w_j = w_j - \alpha \frac{\partial J(\vec{w},b)}{\partial w_j} \; & \text{for j $\in$ [0,n-1] } \\ & \; \; \; \; \;b = b - \alpha \frac{\partial J(\vec{w},b)}{\partial b} \\ &\rbrace \end{align*}$ 而求偏导部分就有所不同，由于加入了正则化多了一项再求导过程中出现 $2w_j$ ，然后与前面的常数项相乘化简后结果如下
$∂J(w⃗,b)∂wj=1m∑i=0m−1(fw⃗,b(x(i))−y(i))xj(i)+λmwj∂J(w⃗,b)∂b=1m∑i=0m−1(fw⃗,b(x(i))−y(i))\begin{align*} \frac{\partial J(\vec{w},b)}{\partial w_j} &= \frac{1}{m} \sum\limits_{i = 0}^{m-1} (f_{\vec{w},b}(\mathbf{x}^{(i)}) - y^{(i)})x_{j}^{(i)} + \frac{\lambda}{m} w_j \\ \frac{\partial J(\vec{w},b)}{\partial b} &= \frac{1}{m} \sum\limits_{i = 0}^{m-1} (f_{\vec{w},b}(\mathbf{x}^{(i)}) - y^{(i)}) \end{align*}$
值得注意的是对于线性回归和逻辑回归，它们的求导公式是一样的，区别就在于函数 $f(w⃗,b)(x(i))f_{(\vec w,b)}(x^{(i)})$ 不同

线性回归是 $f(w⃗,b)(x(i))=w⃗x⃗+bf_{(\vec w,b)}(x^{(i)}) = \vec w \vec x + b$ 其对应的梯度下降计算代码如下

def compute_gradient_linear_reg(X, y, w, b, lambda_): 
    """
    Computes the gradient for linear regression 
    Args:
      X (ndarray (m,n): Data, m examples with n features
      y (ndarray (m,)): target values
      w (ndarray (n,)): model parameters  
      b (scalar)      : model parameter
      lambda_ (scalar): Controls amount of regularization
      
    Returns:
      dj_dw (ndarray (n,)): The gradient of the cost w.r.t. the parameters w. 
      dj_db (scalar):       The gradient of the cost w.r.t. the parameter b. 
    """
    m,n = X.shape           #(number of examples, number of features)
    dj_dw = np.zeros((n,))
    dj_db = 0.

    for i in range(m):    
    	#函数f                         
        err = (np.dot(X[i], w) + b) - y[i]                 
        for j in range(n):                         
            dj_dw[j] = dj_dw[j] + err * X[i, j]               
        dj_db = dj_db + err                        
    dj_dw = dj_dw / m                                
    dj_db = dj_db / m   
    
    for j in range(n):
        dj_dw[j] = dj_dw[j] + (lambda_/m) * w[j]

    return dj_db, dj_dw

逻辑回归是 $f(w⃗,b)(x(i))=g(z)=g(w⃗x⃗+b)=11+ew⃗x⃗+bf_{(\vec w,b)}(x^{(i)}) = g(z) = g(\vec w \vec x + b) = \frac{1}{1+e^{\vec w \vec x + b}}$ 其对应的梯度下降计算代码如下

def compute_gradient_logistic_reg(X, y, w, b, lambda_): 
    """
    Computes the gradient for linear regression 
 
    Args:
      X (ndarray (m,n): Data, m examples with n features
      y (ndarray (m,)): target values
      w (ndarray (n,)): model parameters  
      b (scalar)      : model parameter
      lambda_ (scalar): Controls amount of regularization
    Returns
      dj_dw (ndarray Shape (n,)): The gradient of the cost w.r.t. the parameters w. 
      dj_db (scalar)            : The gradient of the cost w.r.t. the parameter b. 
    """
    m,n = X.shape
    dj_dw = np.zeros((n,))                            #(n,)
    dj_db = 0.0                                       #scalar

    for i in range(m):
    	#函数f 
        f_wb_i = sigmoid(np.dot(X[i],w) + b)          #(n,)(n,)=scalar
        err_i  = f_wb_i  - y[i]                       #scalar
        for j in range(n):
            dj_dw[j] = dj_dw[j] + err_i * X[i,j]      #scalar
        dj_db = dj_db + err_i
    dj_dw = dj_dw/m                                   #(n,)
    dj_db = dj_db/m                                   #scalar

    for j in range(n):
        dj_dw[j] = dj_dw[j] + (lambda_/m) * w[j]

    return dj_db, dj_dw