机器学习简单回顾：1 linear regression

比较久之前看的机器学习了，因为做的深度学习方向，所以回顾一下知识点

Finding the model parameter is a specific condition of Maximum Likelihood Estimation which is known as MLE:
And overfitting is the general property of MLE. By using a bayesian method, we can easily avoid overfitting. (indeed, effective number of parameters will be changed while the size of datasets changes)

$$\begin{gathered} . \\ . \end{gathered}$$

MLE:
$$y^{(i)}={\theta }^T{x}^{(i)}+{ϵ}^{(i)}$$

assume that $ϵ$ is i.i.d.
we got the likelihood as below
$$p(y^{(i)}|x^{(i)};\theta )=\frac{1}{\sqrt{2\pi }\sigma }exp\left(-\frac{(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2}{2{\sigma }^2}\right)$$
maximize the likelihood
$$\begin{array}{rl}argmaxL(\theta )& =argmaxp(\vec{y}|X;\theta )\\ & =argmax\prod _{i=1}^n\frac{1}{\sqrt{2\pi }\sigma }exp\left(-\frac{(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2}{2{\sigma }^2}\right)\end{array}$$
actually log $L(\theta )$ is more fit for calculating:
$$\begin{array}{rl}l(\theta )& =logL(\theta )=log\prod _{i=1}^n\frac{1}{\sqrt{2\pi }\sigma }exp\left(-\frac{(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2}{2{\sigma }^2}\right)\\ & =\sum _{i=1}^{n}log\frac{1}{\sqrt{2\pi }\sigma }exp\left(-\frac{(y^{(i)}-{\theta }^T{x}^{(i)}{)}^2}{2{\sigma }^2}\right)\\ & =nlog\frac{1}{\sqrt{2\pi }\sigma }-\frac{1}{{\sigma }^2}\cdot \frac{1}{2}\sum _{i=1}^n((y^{(i)}-{\theta }^T{x}^{(i)}{)}^2\end{array}$$

substitute for newton method could be

sklearn linear_model
Parameters

fit_interceptbool, optional, default True
Whether to calculate the intercept for this model. If set to False, no intercept will be used in calculations (i.e. data is expected to be centered).

normalizebool, optional, default False
This parameter is ignored when fit_intercept is set to False. If True, the regressors X will be normalized before regression by subtracting the mean and dividing by the l2-norm. If you wish to standardize, please use sklearn.preprocessing.StandardScaler before calling fit on an estimator with normalize=False.

copy_Xbool, optional, default True
If True, X will be copied; else, it may be overwritten.

n_jobsint or None, optional (default=None)
The number of jobs to use for the computation. This will only provide speedup for n_targets > 1 and sufficient large problems. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary for more details.

Attributes

coef_array of shape (n_features, ) or (n_targets, n_features)
Estimated coefficients for the linear regression problem. If multiple targets are passed during the fit (y 2D), this is a 2D array of shape (n_targets, n_features), while if only one target is passed, this is a 1D array of length n_features.

rank_int
Rank of matrix X. Only available when X is dense.

singular_array of shape (min(X, y),)
Singular values of X. Only available when X is dense.

intercept_float or array of shape of (n_targets,)
Independent term in the linear model. Set to 0.0 if fit_intercept = False.