最小二乘估计，矩阵方法求解

D={ (x1,y1),(x2,y2),...(xN,yN)}xi∈Rp(xi是一个列向量),yi∈R,i=1,2,...,NX=[x1x2⋯xN]T=[x1Tx2T⋮xNT]=[x11x12⋯x1Px21x22⋯x2P⋮⋮⋱⋮xN1xN2⋯xNP]N×PY=[y1Ty2T⋮yNT]N×1我们拟合的直线的方程：f(W)=WTx+b为了方便计算，我们将b转变成w0x0,x0等于1，w0等于bf(W)=WTxW=[w1Tw2T⋮wPT]P×1最小二乘法估计：L(W)=∑i=1N∣∣WTxi−yi∣∣2=∑i=1N(WTxi−yi)2=[WTx1−y1WTx2−y2⋯WTxN−yN][WTx1−y1WTx2−y2⋮WTxN−yN]=([WTx1WTx2⋯WTxN]−[y1y2⋯yN])[WTx1−y1WTx2−y2⋮WTxN−yN]=(WT[x1x2⋯xN]−[y1y2⋯yN])[WTx1−y1WTx2−y2⋮WTxN−yN]=(WTXT−YT)[WTx1−y1WTx2−y2⋮WTxN−yN]=(WTXT−YT)(XW−Y)=WTXTXW−WTXTY−YTXW+YTY=WTXTXW−2WTXTY+YTY W^=argminL(W)∂L(W)∂W=2XTXW−2XTY=0∴XTXW=XTYW=(XTX)−1XTY \begin{aligned} D&=\{(x_1,y_1),(x_2,y_2),...(x_N,y_N)\}\\ x_i&\in \mathbb{R}^p(x_i是一个列向量),y_i\in \mathbb{R},i=1,2,...,N\\ X&=\left[ \begin{matrix} x_1 & x_2 & \cdots & x_N \\ \end{matrix} \right]^T\\ &=\left[ \begin{matrix} x_1^T \\ x_2^T \\ \vdots\\ x_N^T \\ \end{matrix} \right]\\ &=\left[ \begin{matrix} x_{11} & x_{12} & \cdots & x_{1P} \\ x_{21} & x_{22} & \cdots & x_{2P} \\ \vdots & \vdots & \ddots & \vdots \\ x_{N1} & x_{N2} & \cdots & x_{NP} \\ \end{matrix} \right]_{N\times P}\\ Y&= \left[ \begin{matrix} y_1^T \\ y_2^T \\ \vdots\\ y_N^T \\ \end{matrix} \right]_{N\times 1}\\ 我们&拟合的直线的方程：\\ f(W)&=W^Tx+b\\ 为了&方便计算，我们将b转变成w_0x_0,x_0等于1，w_0等于b\\ f(W)&=W^Tx\\ W&=\left[ \begin{matrix} w_1^T \\ w_2^T \\ \vdots\\ w_P^T \\ \end{matrix} \right]_{P\times 1}\\ &最小二乘法估计：\\ L(W)&=\sum_{i=1}^N||W^Tx_i-y_i||^2\\ &=\sum_{i=1}^N(W^Tx_i-y_i)^2\\ &=\left[ \begin{matrix} W^Tx_1-y_1 & W^Tx_2-y_2 & \cdots & W^Tx_N-y_N \\ \end{matrix} \right] \left[ \begin{matrix} W^Tx_1-y_1 \\ W^Tx_2-y_2 \\ \vdots\\ W^Tx_N-y_N \\ \end{matrix} \right]\\ &=(\left[ \begin{matrix} W^Tx_1 & W^Tx_2 & \cdots & W^Tx_N \\ \end{matrix} \right]-\left[ \begin{matrix} y_1 & y_2 & \cdots & y_N \\ \end{matrix} \right])\left[ \begin{matrix} W^Tx_1-y_1 \\ W^Tx_2-y_2 \\ \vdots\\ W^Tx_N-y_N \\ \end{matrix} \right]\\ &=(W^T\left[ \begin{matrix} x_1 & x_2 & \cdots & x_N \\ \end{matrix} \right]-\left[ \begin{matrix} y_1 & y_2 & \cdots & y_N \\ \end{matrix} \right])\left[ \begin{matrix} W^Tx_1-y_1 \\ W^Tx_2-y_2 \\ \vdots\\ W^Tx_N-y_N \\ \end{matrix} \right]\\ &=(W^TX^T-Y^T)\left[ \begin{matrix} W^Tx_1-y_1 \\ W^Tx_2-y_2 \\ \vdots\\ W^Tx_N-y_N \\ \end{matrix} \right]\\ &=(W^TX^T-Y^T)(XW-Y)\\ &=W^TX^TXW-W^TX^TY-Y^TXW+Y^TY\\ &=W^TX^TXW-2W^TX^TY+Y^TY\\ ~\\ \hat{W}&=argmin L(W)\\ \frac{\partial L(W)}{\partial W}&=2X^TXW-2X^TY=0\\ \therefore X^TXW&=X^TY\\ W&=(X^TX)^{-1}X^TY \end{aligned} Dxi​XY我们f(W)为了f(W)WL(W) W^∂W∂L(W)​∴XTXWW​={ (x1​,y1​),(x2​,y2​),...(xN​,yN​)}∈Rp(xi​是一个列向量),yi​∈R,i=1,2,...,N=[x1​​x2​​⋯​xN​​]T=⎣⎢⎢⎢⎡​x1T​x2T​⋮xNT​​⎦⎥⎥⎥⎤​=⎣