牛客题解 | 使用梯度下降实现Lasso回归

题目

题目链接

Lasso回归是一种线性回归模型，其目标函数为：

$$\underset{w}{\min }\frac{1}{2n}\sum _{i=1}^n(y_i-w^T{x}_i{)}^2+\lambda \Vert w{\Vert }_1$$
其中，$w$是模型参数，$x_i$是输入特征，$y_i$是输出标签，$\lambda$是正则化参数。
本算法的关键在于对权重进行L1正则化，即在每次迭代中对权重进行L1范数惩罚。

梯度下降的公式为：

$$w_{t+1}=w_t-\eta \nabla f(w_t)$$
其中，$w_t$是第$t$次迭代时的权重，$\eta$是学习率，$\nabla f(w_t)$是目标函数在$w_t$处的梯度。而f(w_t)正是Lasso回归的目标函数。其梯度为：
$$\nabla f(w_t)=\frac{1}{n}\sum _{i=1}^n(y_i-w_t^T{x}_i)x_i+\lambda \text{sign}(w_t)$$

标准代码如下

def l1_regularization_gradient_descent(X: np.array, y: np.array, alpha: float = 0.1, learning_rate: float = 0.01, max_iter: int = 1000, tol: float = 1e-4) -> tuple:
    n_samples, n_features = X.shape
    # Zero out weights and bias
    weights = np.zeros(n_features)
    bias = 0
    
    for iteration in range(max_iter):
        # Predict values
        y_pred = np.dot(X, weights) + bias
        # Calculate error
        error = y_pred - y
        # Gradient for weights with L1 penalty
        grad_w = (1 / n_samples) * np.dot(X.T, error) + alpha * np.sign(weights)
        # Gradient for bias (no penalty for bias)
        grad_b = (1 / n_samples) * np.sum(error)
        
        # Update weights and bias
        weights -= learning_rate * grad_w
        bias -= learning_rate * grad_b
        
        # Check for convergence
        if np.linalg.norm(grad_w, ord=1) < tol:
            break
    return [round(w, 3) for w in weights], round(bias, 3)