PCA向量空间推导

$设行向量x_1,x_2,\dots \dots ,x_n\in R^n。$
$$\begin{gathered} 拉格朗日乘子法求解下述优化问题： \\ \{\begin{array}{l}minimize{\sum }_{i=1}^N||x_i-{\sum }_{k=1}^m<x_i,w_k>w_k|{|}^2\\ subjectto<w_k,w_l>={\delta }_{i,j}\end{array} \\ 其中m<n,<\cdot ,\cdot >为标准内积。 \end{gathered}$$
$$w_i\in R^n是标准正交的列向量，m<n,希望只保留w_i方向上的信息，即把n维信息压缩成m维，且信息偏差尽可能小。$$

$解：记\bar{x}={\sum }_{k=1}^m<x_i,w_k>w_k,W_{n,m}=[w_1,w_2,\dots \dots ，w_n],则{\bar{x}}_i=x_{i}W{W}^T$
$$\begin{gathered} 目标函数\sum _{i=1}^N||{\bar{x}}_i-x_i|{|}^2=\sum _{i=1}^m{\bar{x}}_i{\bar{x}}_i^T-2\sum _{i=1}^m{\bar{x}}_i{x}_i^T+\sum _{i=1}^m{x}_i{x}_i^T \\ 其中\sum _{i=1}^m{x}_i{x}_i^T无变量，考虑\sum _{i=1}^m{\bar{x}}_i{\bar{x}}_i^T-2\sum _{i=1}^m{\bar{x}}_i{x}_i^T \\ \sum _{i=1}^m{\bar{x}}_i^T{\bar{x}}_i中，标量{\bar{x}}_i{\bar{x}}_i^T因为{\bar{x}}_i=x_{i}W{W}^T, \\ (x_{i}W{W}^T)\ast (x_{i}W{W}^T{)}^T=x_{i}W{W}^{T}W{W}^T{x}_i^T=x_{i}W(W^{T}W)W^T{x}_i^T=x_{i}W(I_{m,m})W^T{x}_i^T \\ \sum _{i=1}^m{\bar{x}}_i{x}_i^T中，{\bar{x}}_i{x}_i^T=x_{i}W{W}^T{x}_i^T \\ 所以目标函数=-\sum _{i=1}^n{x}_{i}W{W}^T{x}_i^T \end{gathered}$$
$$\begin{gathered} 对于矩阵乘法的迹：tr(AB)=\sum _i\sum _j{A}_{i,j}{B}_{j,i}或tr(A^{T}B)=\sum _i\sum _j{A}_{i,j}{B}_{i,j} \\ 所以，A=B=(X^{T}W),原目标函数=-tr(W^{T}X{X}^{T}W) \end{gathered}$$
L(W，λ)=−tr(WTXXTW+λ(WTW−I))∵{tr(XAXT)}′=X(A+AT)∴  L′(W，λ)W=2∗(−XXTW+λ(W−I))=0XXTW=λW,即W为XXT的特征向量组成的矩阵 L(W，\lambda)=-tr(W^TXX^TW+\lambda (W^TW-I))\\ \because \{tr(XAX^T)\}'=X(A+A^T)\\ \therefore \ \ L'(W，\lambda)_W=2*(-XX^TW+\lambda (W-I))=0\\ XX^TW=\lambda W,即W为XX^T的特征向量组成的矩阵 L(W，λ)=−tr(WTXXTW+λ(WTW−I))∵{tr(XAXT)}′=X(A+AT)∴  L′(W，λ)W​=2∗(−XXTW+λ(W−I))=0XXTW=λW,即W为XXT的特征向量组成的矩阵

---

基于最大投影方差：

$$\begin{gathered} \sum _i{x}_i=0,E(y_i)=E(x_i\ast w)=0 \\ {y}_i=x_{i}w,即为上述情况m=1（W=w）的情形 \end{gathered}$$