L1正则化、L2正则化的多角度分析和概率角度的解释

L1正则化、L2正则化的公式如下

$$\begin{gathered} \underset{w}{\min }{L}_1(w)=\underset{w}{\min }f(w)+\frac{\lambda }{n}\sum _{i=1}^n|w_i| \\ \underset{w}{\min }{L}_2(w)=\underset{w}{\min }f(w)+\frac{\lambda }{2n}\sum _{i=1}^n{w}_i^2 \end{gathered}$$

从优化问题的视角来看

$$\begin{gathered} \underset{x}{\min }L(w)<=>\underset{w}{\min }f(w) \\ s.t.\sum _{i=1}^n|w_i|<C \end{gathered}$$

L1正则的限制条件，在坐标轴上显示则是一个正方形，与坐标轴的交点分别是(0,C),(C,0),(0,-C),(-C,0)

L2正则的限制条件，在坐标轴上显示则是一个圆，与坐标轴的交点分别是(0,C),(C,0),(0,-C),(-C,0)

从梯度视角来看

∂ L 1 ( w ) ∂ w i = ∂ f ( w ) ∂ w i + λ n s i g n ( w i ) w i ′ = w i − η ∂ L 1 ( w ) ∂ w i w i ′ = w i − η ∂ f ( w ) ∂ w i − η λ n s i g n ( w i ) \frac{\partial L_1(w)}{\partial w_i}=\frac{\partial f(w)}{\partial w_i}+\frac{\lambda}{n}sign(w_i)\\ w_i^{'}=w_i - \eta \frac{\partial L_1(w)}{\partial w_i}\\ w_i^{'}=w_i - \eta \frac{\partial f(w)}{\partial w_i} - \eta \frac{\lambda}{n}sign(w_i) ∂wi​∂L1​(w)​=∂wi​∂f(w)​+nλ​sign(wi​)wi′​=wi​−η∂wi​∂L