An introduction to Linear Regression

An Introduction to Linear Regression

Intent

supervised learning:
- regression (continious)
- classfication (discrete)

$h(\theta)=\displaystyle\sum_{i=0}^{n}\theta_iX_i=\theta^{T}X$

For historical reasons, this function h is called a hypothesis.

$\theta^{'}s \quad\text{is the parameters}$

$(x^{(i)},y^{(i)})\qquad(training \quad example)$

$\{(x^{(i)},y^{(i)}); i=1,...,m\}\qquad(training \quad set)$

要让$h(\theta)$接近于trainset中的$y^{(i)}$

cost function: $J(\theta) =\frac{1}{2}\displaystyle\sum_{i=1}^{m}(h_{\theta}(x^{(i)} - y^{(i)})^2$

为了让train出的model能stable，需要在bias和variance中tradeoff。regularized regression 会使用L1(lasso)和L2(ridge)作为penalty。其中L1相当于feature selection(parameter变为0)。而L2主要是减少variance。cross validation和bootstrap也是常用的trade bias和variance的方法。

LMS

gradient descent

$\theta_j = \theta_j - \alpha\frac{\partial }{\partial \theta_j}J(\theta)$
$\alpha$ is learning rate

∂∂θj