Part Ⅱ

Part Ⅱ

2.1 The words list

• linear regression 线性回归
• gradient descent 梯度下降
• learning rate 学习率
• stochastic gradient descent 随机梯度下降法
• Matrix derivatives 矩阵求导

2.2 Gradient descent concept

Gradient descent is kind of the iterative method, which can solve the least squares problems(linear or nonlinear).Gradient descent is one of the most commonly used method for solving the model parameters in machine learning algorithms, also named unconstrained optimization problems. Another method is the least squares method. We can get the minimize of loss functions and model parameters by gradient descent to solve step by step when we solve the minimize of the loss functions. In contrast, we use the gradient rise method to iterate if we need the maximum of the loss functions. In the machine learning, developing two methods based on the most essential gradient descent, stochastic gradient descent method and batch gradient descent method respectively.

The fundamental formula:

${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )$.

The $J(\theta )$ is cost function defined:

$J(\theta )=\frac{1}{2}(h_{\theta }(x)-y{)}^2$.

because $h(\theta )={\Sigma }_{i=1}^n{\theta }_i{x}_i$

so after simplification we get ${\theta }_j:={\theta }_j+\alpha (y^{(i)}-h_{\theta }(x^{(i)}))x_j^{(i)}$.

Repeat until convergence{

${\theta }_j:={\theta }_j+\alpha {\Sigma }_{i=1}^m(y^{(i)}-h_{\theta (x^{(i)})}{x}_j^{(i)}$. >for every j

}

This algorithm is called batch gradient descent .

Loop{

for i = 1 to m{
${\theta }_j:={\theta }_j+\alpha (y^{(i)}-h_{\theta }(x^{(i)}))x_j^{(i)}$.

}
}

This algorithm is called stochastic gradient descent method.

2.3 The normal equation of the least square method

the training example’s input values in its rows:

$X=\left[\begin{array}{c}—(x^{(1)}{)}^T—\\ —(x^{(2)}{)}^T—\\ ⋮\\ —(x^{(m)}{)}^T—\end{array}\right]$.

Also,let $\vec{y}$ be the m-dimensional vector containing all the target values from the training set:

$\vec{y}=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ ⋮\\ y^{(m)}\end{array}\right]$.

Now,since $h_{\theta }(x^{(i)})=(x^{(i)}{)}^T\theta$,we can easily verify that

$X\theta -\vec{y}=\left[\begin{array}{c}(x^{(1)}{)}^T\theta \\ ⋮\\ (x^{(m)}{)}^T\theta \end{array}\right]-\left[\begin{array}{c}{y}^{(1)}\\ ⋮\\ y^{(m)}\end{array}\right]$.

$=\left[\begin{array}{c}{h}_{\theta }(x^{(1)})-y^{(1)}\\ ⋮\\ h_{\theta }(x^{(m)})-y^{(m)}\end{array}\right]$.

Thus, using the fact that for a vector $z$, we have that $z^{T}z={\Sigma }_i{z}_i^2:$

$\frac{1}{2}(X\theta -\vec{y}{)}^T(X\theta -\vec{y})=\frac{1}{2}{\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2$.

$=J(\theta )$.

After simplification,we can the normal equation:

$X^{T}X\theta =X^T\vec{y}$.

$\theta =(X^{T}X{)}^{(-1)}{X}^T\vec{y}$.

PS:The least square method have essential difference with the gradient descent method. The least square method is to solve the global optimal solution and the gradient descent is to solve the local optimal solution.

2.4 summarize

The main content of this section is the difference between the gradient descent method and the least square method.For the gradient descent method,main methods are batch gradient descent and stochastic gradient descent. In difference places, these two methods have different performances.Meanwhile,we need notice the essence between the least square and the gradient descent.