最小二乘估计理论

模型

不需输入输出为随机过程，求最优权向量使得输出估计结果的样本均方误差最小
b^⃗H=w⃗HAH{b^H=[d^(M)d^(M+1)⋯d^(N)]AH=[u(M)u(M+1)⋯u(N)]=[u(M)u(M+1)⋯u(N)u(M−1)u(M)⋯u(N−1)⋮⋮⋱⋮u(1)u(2)⋯u(N−M+1)]w=[w0w1⋯wM−1]Hmin{J}{e=b−b^J=eHe \begin {align} &\vec{\hat{b}}^H=\vec{w}^HA^H \\ &\left\{\begin{aligned} \hat{b}^H&=\begin{bmatrix} \hat{d}(M)&\hat{d}(M+1)&\cdots&\hat{d}(N) \end{bmatrix} \\ A^H&=\begin{bmatrix} u(M)&u(M+1)&\cdots&u(N)\end{bmatrix}\\ &=\begin{bmatrix} u(M)&u(M+1)&\cdots&u(N)\\ u(M-1)&u(M)&\cdots&u(N-1)\\ \vdots&\vdots&\ddots&\vdots\\ u(1)&u(2)&\cdots&u(N-M+1)\\ \end{bmatrix}\\ w&=\begin{bmatrix} w_0 & w_1 &\cdots &w_{M-1} \end{bmatrix} ^H\\ \end{aligned} \right.\\ &min\{J\}\\ &\left\{\begin{aligned} &e=b-\hat{b} \\ &J=e^He \end{aligned} \right. \end {align} ​b^H=wHAH⎩⎨⎧​b^HAHw​=[d^(M)​d^(M+1)​⋯​d^(N)​]=[u(M)​u(M+1)​⋯​u(N)​]=​u(M)u(M−1)⋮u(1)​u(M+1)u(M)⋮u(2)​⋯⋯⋱⋯​u(N)u(N−1)⋮u(N−M+1)​​=[w0​​w1​​⋯​wM−1​​]H​min{J}{​e=b−b^J=eHe​​​

方法

最优权向量满足确定性正则方程：
$$A^{H}A\hat{w}=A^{H}b$$

算法

基于SVD的LS算法

LS最优权重仅与数据矩阵$A$的奇异值和特征向量有关，故求得数据矩阵的奇异值和特征向量即可求得最有权重。
$$\begin{array}{r}\{\begin{array}{rl}& \hat{w}=\sum _{i=1}^K(\frac{{\overrightarrow{x_i}}^H\vec{\theta }}{{\sigma }_i^2})\overrightarrow{x_i}\\ & \vec{\theta }=A^{H}b\end{array}\end{array}$$
关于$A^{H}A$是否奇异分类讨论：
$$\begin{array}{r}\{\begin{array}{rl}非奇异K=M:& \hat{w}=\sum _{i=1}^M(\frac{{\overrightarrow{x_i}}^H\vec{\theta }}{{\sigma }_i^2})\overrightarrow{x_i}\\ 奇异K<M:& \hat{w}=\sum _{i=1}^K(\frac{{\overrightarrow{x_i}}^H\vec{\theta }}{{\sigma }_i^2})\overrightarrow{x_i}(加入最小范数条件使得有唯一解)\end{array}\end{array}$$