CS 229 notes Supervised Learning

最新推荐文章于 2024-05-30 09:52:37 发布

转载最新推荐文章于 2024-05-30 09:52:37 发布 · 73 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/EtoDemerzel/p/7881434.html

文章标签：

#人工智能

本文探讨了监督学习中正规方程的证明过程，并回顾了最小二乘法的相关概念。文章首先介绍了所需的线性代数预备知识，接着详细推导了正规方程，并通过矩阵微分进一步解释了其原理。

CS 229 notes Supervised Learning

标签（空格分隔）：监督学习线性代数

Forword

the proof of Normal equation and, before that, some linear algebra equations, which will be used in the proof.

The normal equation

Linear algebra preparation

For two matrices $A$ and $B$ such that $AB$ is square, $trAB\ = \ trBA$ .

Proof:

Some properties:

some facts of matrix derivative:
$\nabla_AtrAB=B^T...................................................................(1)$

Proof:

$\nabla_{A^T}f(A) = (\nabla_Af(A))^T...........................................................(2)$
$\nabla_AtrABA^TC = CAB+C^TAB^T..................................................(3)$

Proof 1:

Proof 2:

$\nabla_A|A| = |A|(A^{-1})^T.............................................................(4)$

Proof: $(\nabla_A |A|)_{pq} = C_{pq} = A^*_{qp} = (A^*)^T_{pq} = |A|(A^{-1})_{pq}$
( $C$ refers to the cofactor)

Least squares revisited

$X = \begin{bmatrix}-(x^{(1)})^T-\\-(x^{(2)})^T-\\.\\.\\.\\-(x^{(m)})^T-\end{bmatrix}$ (if we don’t include the intercept term)

$\vec y = \begin{bmatrix}y^{(1)}\\y^{(2)}\\.\\.\\.\\y^{(m)}\end{bmatrix}$

since $h_\theta(x^{(i)} = (x^{(i)})^T\theta$ ,

Thus,
$\frac{1}{2}(X\theta-\vec{y})^T(X\theta-\vec{y}) =
\frac{1}{2}\displaystyle{\sum{i=1}^{m}(h\theta(x^{(i)}) -y^{(i)})^2} = J(\theta) $.

Combine Equations $(2),(3)$ ：
$\nabla_{A^T}trABA^TC = B^TA^TC^T+BA^TC..............................................(5)$

Hence

$\nabla_\theta J(\theta) = \frac{1}{2}\nabla_\theta(X\theta-\vec{y})^T(X\theta-\vec{y})\\ = \frac{1}{2}\nabla_\theta(\theta^TX^TX\theta-\theta^TX^T\vec{y}-\vec{y}X\theta -({\vec{y}})^T\vec{y})$