Chapter 7 (Symmetric Matrices and Quadratic Forms): The Singular Value Decomposition (奇异值分解, SVD)

• As we know, not all matrices can be factored as $A=PD{P}^{-1}$ with $D$ diagonal. However, a special factorization (Singular Value Decomposition) $A=QD{P}^{-1}$ is possible for any $m\times n$ matrix $A$!

奇异值

奇异值的定义

• Let $A$ be an $m\times n$ matrix. Then $A^{T}A$ is symmetric and can be orthogonally diagonalized. Let { v 1 , . . . , v n } \{\boldsymbol v_1,...,\boldsymbol v_n\} {v1​,...,vn​} be an orthonormal basis for ${\mathbb{R}}^n$ consisting of eigenvectors of $A^{T}A$, and let { λ 1 , . . . , λ n } \{\lambda_1,...,\lambda_n\} {λ1​,...,λn​} be the associated eigenvalues of $A^{T}A$. Then, for $1\le i\le n$,
  So the eigenvalues of $A^{T}A$ are all nonnegative. By renumbering, if necessary, we may assume that the eigenvalues are arranged so that
  
• The singular values of $A$ are the square roots of the eigenvalues of $A^{T}A$, denoted by ${\sigma }_1,...,{\sigma }_n$, and they are arranged in decreasing order. By equation (2), the singular values of $A$ are the lengths of the vectors $A{\mathbf{v}}_1,...,A{\mathbf{v}}_n$.
  • The first singular value ${\sigma }_1$ of an $m\times n$ matrix $A$ is the maximum of $\Vert Ax\Vert$ over all unit vectors. This maximum value is attained at a unit eigenvector ${\mathbf{v}}_1$ of $A^{T}A$ corresponding to the greatest eigenvalue ${\lambda }_1$ of $A^{T}A$. The second singular value is the maximum of $\Vert Ax\Vert$ over all unit vectors orthogonal to ${\mathbf{v}}_1$.

---

EXERCISE

How are the singular values of $A$ and $A^T$ related?

SOLUTION

• $A^T=(U\Sigma V^T{)}^T=V{\Sigma }^T{U}^T$. This is an SVD of $A^T$ because $V$ and $U$ are orthogonal matrices and ${\Sigma }^T$ is an $n\times m$ “diagonal” matrix. Since $\Sigma$ and ${\Sigma }^T$ have the same nonzero diagonal entries, $A$ and $A^T$ have the same nonzero singular values.

非零奇异值

PROOF

• For $i\ne j$ ,
  Thus { A v 1 , . . . , A v n } \{A\boldsymbol v_1,...,A\boldsymbol v_n\} {Av1​,...,Avn​} is an orthogonal set.
• Furthermore, since the lengths of the vectors $A{\mathbf{v}}_1,...,A{\mathbf{v}}_n$ are the singular values of $A$, and since there are $r$ nonzero singular values, $A{\mathbf{v}}_i\ne 0$ if and only if $1\le i\le r$. So $A{\mathbf{v}}_1,...,A{\mathbf{v}}_r$ are linearly independent vectors, and they are in $ColA$.
• Finally, for any $\mathbf{y}=A\mathbf{x}$ in $ColA$, we can write $\mathbf{x}=c_1{\mathbf{v}}_1+...+c_n{\mathbf{v}}_n$, and
  Thus $\mathbf{y}$ is in S p a n { A v 1 , . . . , A v r } Span\{A\boldsymbol v_1,...,A\boldsymbol v_r\} Span{Av1​,...,Avr​}, which shows that { A v 1 , . . . , A v r } \{A\boldsymbol v_1,...,A\boldsymbol v_r\} {Av1​,...,Avr​} is an (orthogonal) basis for $ColA$. Hence $rankA=dimColA=r$. ($r$ 包括了重复的奇异值)

The Singular Value Decomposition (SVD)

奇异值分解

• The decomposition of $A$ involves an $m\times n$ “diagonal” matrix $\Sigma$ of the form
  where $D$ is an $r\times r$ diagonal matrix ($r\le m;r\le n$)

---

• The matrices $U$ and $V$ are not uniquely determined by $A$. The columns of $U$ in such a decomposition are called left singular vectors of $A$, and the columns of $V$ are called right singular vectors of $A$.
• 注意：常将奇异值按降序排列以确保 $\Sigma$ 的唯一性．
• 当 $A$ 为正定矩阵时，奇异值分解与特征值分解结果相同
  • 对 $A$ 进行特征值分解可得 $A=PD{P}^T$，其中 $P$ 的每一列 ${\mathbf{p}}_i$ 均为 $A$ 的一个特征向量且它们互相正交。易证 $A$ 的特征向量 ${\mathbf{p}}_i$ 也为 $A^{T}A$ 的特征向量且对应的 $A$ 的特征值的平方也为 $A^{T}A$ 的特征值，因此找到了 $A^{T}A$ 的一组特征向量基 { p 1 , . . . , p n } \{\boldsymbol p_1,...,\boldsymbol p_n\} {p1​,...,pn​} 且对应的特征向量为 { λ 1 2 , . . , λ n 2 } \{\lambda_1^2,..,\lambda_n^2\} {λ12​,..,λn2​}，因此 ${\mathbf{v}}_i={\mathbf{p}}_i$ 且 ${\sigma }_i=\sqrt{{\lambda }_i^2}={\lambda }_i$，因此有 $V=P,\Sigma =D$，进而可以推出 $U=P$

PROOF

• Let ${\lambda }_i$ and ${\mathbf{v}}_i$ be as in Theorem 9, so that { A v 1 , . . . , A v r } \{A\boldsymbol v_1,...,A\boldsymbol v_r\} {Av1​,...,Avr​} is an orthogonal basis for $ColA$. Normalize each $A{\mathbf{v}}_i$ to obtain an orthonormal basis { u 1 , . . . , u r } \{\boldsymbol u_1,...,\boldsymbol u_r\} {u1​,...,ur​}, where
  and
  
• Now extend { u 1 , . . . , u r } \{\boldsymbol u_1,...,\boldsymbol u_r\} {u1​,...,ur​} to an orthonormal basis { u 1 , . . . , u m } \{\boldsymbol u_1,...,\boldsymbol u_m\} {u1​,...,um​} of ${\mathbb{R}}^m$, and let
  By construction, $U$ and $V$ are orthogonal matrices. Also, from (4),
  
• Let $D$ be the diagonal matrix with diagonal entries ${\sigma }_1,...,{\sigma }_r$ , and let $\Sigma$ be as in (3) above. Then
  Since $V$ is an orthogonal matrix,
  

---

EXAMPLE 4

Find a singular value decomposition of

SOLUTION

• Step 1. Find an orthogonal diagonalization of $A^{T}A$. The eigenvalues of $A^{T}A$ are 18 and 0, with corresponding unit eigenvectors
  
• Step 2. Set up $V$ and $\Sigma$.
  
• Step 3. Construct $U$. To construct $U$, first construct $A{\mathbf{v}}_1$ and $A{\mathbf{v}}_2$:
  The only column found for $U$ so far is
  The other columns of $U$ are found by extending the set { u 1 } \{\boldsymbol u_1\} {u1​} to an orthonormal basis for ${\mathbb{R}}^3$. In this case, we need two orthogonal unit vectors ${\mathbf{u}}_2$ and ${\mathbf{u}}_3$ that are orthogonal to ${\mathbf{u}}_1$. Each vector must satisfy ${\mathbf{u}}_1^T\mathbf{x}=0$, which is equivalent to the equation $x_1-2x_2+2x_3=0$. A basis for the solution set of this equation is
  Apply the Gram–Schmidt process (with normalizations) to { w 1 , w 2 } \{\boldsymbol w_1,\boldsymbol w_2\} {w1​,w2​}, and obtain
  
  • Another way to find ${\mathbf{u}}_2$ and ${\mathbf{u}}_3$ is to realize that ${\mathbf{u}}_1$ form an orthonormal basis for $ColA$. The remaining ${\mathbf{u}}_2$ and ${\mathbf{u}}_3$ must be a basis for $(ColA{)}^{\perp }=Nul{A}^T$.

一些性质

• 由之前的讨论可知，假设 $A$ 有 $r$ 个非零奇异值，那么 { A v 1 , . . . , A v r } \{Av_1,...,Av_r\} {Av1​,...,Avr​} 为 $ColA$ 的一组正交基。由 $u_i=\frac{A{v}_1}{{\sigma }_i}$ 可知，$A$ 的 $r$ 个左奇异向量 $u_1,...,u_r$ 构成了 $ColA$ 的一组标准正交基；由此可知，$A$ 的 $m-r$ 个左奇异向量 $u_{r+1},...,u_m$ 构成了 $Null{A}^T$ 的一组标准正交基
• 由于 $A^T=V{\Sigma }^T{U}^T$，同理可知，$A$ 的 $r$ 个右奇异向量 $v_1,...,v_r$ 构成了 $Col{A}^T$ 的一组标准正交基；$A$ 的 $n-r$ 个右奇异向量 $v_{r+1},...,v_n$ 构成了 $NullA$ 的一组标准正交基

几何解释

• 从线性变换的角度理解奇异值分解，$m\times n$ 矩阵 $A$ 表示从 $n$ 维空间 ${\mathbb{R}}^n$ 到 $m$ 维空间 ${\mathbb{R}}^m$ 的一个线性变换，
  $$T:x\to Ax$$
• 由奇异值分解可知，线性变换可以分解为三个简单的变换: 一个坐标系的旋转或反射变换、一个坐标轴的缩放变换、另一个坐标系的旋转或反射变换
  

正交矩阵对应的正交变换不改变向量长度，也不改变向量内积结果，因此不改变向量的正交性。也就是说，一组正交基在经过正交变换后仍然是一组正交基，且基的长度不变，因此正交变换可以看作坐标系的旋转或反射变换 (To learn more: Orthogonal Transformations)

紧奇异值分解与截断奇异值分解

• 定理 10 给出的奇异值分解
  $$A=U\Sigma V^T$$又称为矩阵的完全奇异值分解 (full singular value decomposition)。实际常用的是奇异值分解的紧凑形式和截断形式。紧奇异值分解是与原始矩阵等秩的奇异值分解，截断奇异值分解是比原始矩阵低秩的奇异值分解

---

紧奇异值分解

证明
$$\begin{array}{rl}A& =U\Sigma V^T\\ & =\left[\begin{array}{ccc}{u}_1& ...& u_m\end{array}\right]\left[\begin{array}{cc}{\Sigma }_r& 0\\ 0& 0\end{array}\right]\left[\begin{array}{c}{v}_1^T\\ ...\\ v_n^T\end{array}\right]\\ & =\left[\begin{array}{cccccc}{\sigma }_1{u}_1& ...& {\sigma }_r{u}_r& 0& ...& 0\end{array}\right]\left[\begin{array}{c}{v}_1^T\\ ...\\ v_n^T\end{array}\right]\\ & =\sum _{i=1}^r{\sigma }_i{u}_i{v}_i^T\\ & =U_r{\Sigma }_r{V}_r^T\end{array}$$

---

截断奇异值分解

• 在矩阵的奇异值分解中，只取最大的 $k$ 个奇异值 ($k<r$, $r$ 为矩阵的秩) 对应的部分，就得到矩阵的截断奇异值分解。实际应用中提到矩阵的奇异值分解时，通常指截断奇异值分解

奇异值分解与矩阵近似

• 奇异值分解是在平方损失 (弗罗贝尼乌斯范数) 意义下对矩阵的最优近似，即数据压缩。紧奇异值分解对应着无损压缩，截断奇异值分解对应着有损压缩

弗罗贝尼乌斯范数 (Frobenius norm)

• 矩阵的弗罗贝尼乌斯范数是向量的 $L_2$ 范数的直接推广，对应着机器学习中的平方损失函数

---

证明

• 一般地，若 $Q$ 是 $m$ 阶正交矩阵，则有
  因为
  
• 若 $P$ 是 $n$ 阶正交矩阵，则由 $||A|{|}_F=||A^T|{|}_F$ 可知，
  $$||A{P}^T|{|}_F=||P{A}^T|{|}_F=||A^T|{|}_F=||A|{|}_F$$
• 故
  $$\Vert A{\Vert }_F={\Vert U\Sigma V^{\mathrm{T}}\Vert }_F=\Vert \Sigma {\Vert }_F=({\sigma }_1^2+{\sigma }_2^2+...+{\sigma }_n^2{)}^{\frac{1}{2}}$$

矩阵的最优近似

---

• 上述定理说明，奇异值分解是在平方损失 (弗罗贝尼乌斯范数) 意义下对矩阵的最优近似，即数据压缩。紧奇异值分解对应着无损压缩，截断奇异值分解对应着有损压缩
• 将 $A$ 进行谱分解可得
  $$A=\sum _{i=1}^n{\sigma }_i{u}_i{v}_i^T=\sum _{i=1}^r{\sigma }_i{u}_i{v}_i^T$$一般地，设 $A$ 的截断奇异值分解为 $$A_k=\sum _{i=1}^k{\sigma }_i{u}_i{v}_i^T$$则 $A_k$ 的秩为 $k$，并且 $A_k$ 是秩为 $k$ 的矩阵中在弗罗贝尼乌斯范数意义下 $A$ 的最优近似矩阵；由于通常奇异值 ${\sigma }_i$ 递减很快，所以 $k$ 取很小值时，$A_k$ 也可以对 $A$ 有很好的近似

证明

• 设 $X\in \mathcal{M}$ 为满足 $||A-X|{|}_F={\min }_{S\in \mathcal{M}}||A-S|{|}_F$ 的一个矩阵，因此有
  下面证明
  即可
• 设 $X$ 的奇异值分解为 $Q\Omega P^T$，其中
  若令矩阵 $B=Q^{T}AP$，则 $A=QB{P}^T$。由此得到
  用 $\Omega$ 的分块方法对 $B$ 分块
  可得
  现证 $B_{12}=0$, $B_{21}=0$。用反证法。若 $B_{12}\ne 0$，令
  则 $Y\in \mathcal{M}$，且
  这与 $X$ 的定义式矛盾，证明了 $B_{12}=0$。同样可证 $B_{21}=0$。于是
  再证 $B_{11}={\Omega }_k$。为此令
  则 $Z\in \mathcal{M}$，且
  因此
  $$||B_{11}-{\Omega }_k|{|}_F^2=0$$即 $B_{11}={\Omega }_k$。最后看 $B_{22}$。若 $(m-k)\times (n-k)$ 子矩阵 $B_{22}$ 有奇异值分解 $U_1\Lambda V_1^{\mathrm{T}}$，则
  下面证明 $\Lambda$ 的对角线元素为 $A$ 的奇异值。为此，令
  其中 $I_k$ 为 $k$ 阶单位矩阵，则
  $$\begin{array}{rl}{U}_2^T{Q}^{T}AP{V}_2& =U_2^{T}B{V}_2\\ & =\left[\begin{array}{cc}{I}_k& 0\\ 0& U_1^T\end{array}\right]\left[\begin{array}{cc}{\Omega }_k& 0\\ 0& U_1\Lambda V_1^{\mathrm{T}}\end{array}\right]\left[\begin{array}{cc}{I}_k& 0\\ 0& V_1\end{array}\right]\\ & =\left[\begin{array}{cc}{\Omega }_k& 0\\ 0& \Lambda \end{array}\right]\end{array}$$因此
  由此可知 $\Lambda$ 的对角线元素为 $A$ 的奇异值。故有
  于是证明了
  

Applications of the Singular Value Decomposition

• The next few exercises show some interesting facts.

EXERCISE 19

$A$ is an $m\times n$ matrix with a singular value decomposition $A=U\Sigma V^T$ , where $U$ is an $m\times m$ orthogonal matrix, $\Sigma$ is an $m\times n$ “diagonal” matrix with $r$ positive entries and no negative entries, and $V$ is an $n\times n$ orthogonal matrix. Show that the columns of $V$ are eigenvectors of $A^{T}A$, the columns of $U$ are eigenvectors of $A{A}^T$ , and the diagonal entries of $\Sigma$ are the singular values of $A$.

SOLUTION

• [Hint: Use the SVD to compute $A^{T}A$ and $A{A}^T$ .]
  

EXERCISE 25

Let $T:{\mathbb{R}}^n\mapsto {\mathbb{R}}^m$ be a linear transformation. Describe how to find a basis $\mathcal{B}$ for ${\mathbb{R}}^n$ and a basis $\mathcal{C}$ for ${\mathbb{R}}^m$ such that the matrix for $T$ relative to $\mathcal{B}$ and $\mathcal{C}$ is an $m\times n$ “diagonal” matrix.

SOLUTION

• Consider the SVD for the standard matrix of $T$, say, $A=U\sum V^T$. Let B = { v 1 , … , v n } \mathcal B = \{\boldsymbol v_1, …, \boldsymbol v_n\} B={v1​,…,vn​} and C = { u 1 , … , u m } C = \{\boldsymbol u_1, …, \boldsymbol u_m\} C={u1​,…,um​} be bases constructed from the columns of $V$ and $U$, respectively. Observe that, since the columns of $V$ are orthonormal, $V^T{\mathbf{v}}_j={\mathbf{e}}_j$, where ${\mathbf{e}}_j$ is the $j$th column of the $n\times n$ identity matrix. To find the matrix of $T$ relative to $\mathcal{B}$ and $\mathcal{C}$, compute
  So $[T({\mathbf{v}}_j){]}_{\mathcal{C}}={\sigma }_j{\mathbf{e}}_j$. The discussion at the beginning of Section 5.4 shows that the “diagonal” matrix $\Sigma$ is the matrix of $T$ relative to $\mathcal{B}$ and $\mathcal{C}$.

Polar Decomposition (极分解)

• Prove that any $n\times n$ matrix $A$ admits a polar decomposition of the form $A=PQ$, where $P$ is an $n\times n$ positive semidefinite matrix with the same rank as $A$ and where $Q$ is an $n\times n$ orthogonal matrix.

Proof

• [Hint: Use a singular value decomposition, $A=U\Sigma V^T$ , and observe that $A=(U\Sigma U^T)(U{V}^T)$ and $U\Sigma U^T$ is a symmetric matrix.]

估计矩阵的秩

Check Theorem 9

The Condition Number (条件数)

• Most numerical calculations involving an equation $A\mathbf{x}=\mathbf{b}$ are as reliable as possible when the SVD of $A$ is used. The two orthogonal matrices $U$ and $V$ do not affect lengths of vectors or angles between vectors. Any possible instabilities in numerical calculations are identified in $\sum$. If the singular values of $A$ are extremely large or small, roundoff errors are almost inevitable, but an error analysis is aided by knowing the entries in $\sum$ and $V$ .
• If $A$ is an invertible $n\times n$ matrix, then the ratio ${\sigma }_1={\sigma }_n$ of the largest and smallest singular values gives the condition number of $A$. Exercises 41–43 in Section 2.3 showed how the condition number affects the sensitivity of a solution of $A\mathbf{x}=\mathbf{b}$ to changes (or errors) in the entries of A. (Actually, a “condition number” of $A$ can be computed in several ways, but the definition given here is widely used for studying $A\mathbf{x}=\mathbf{b}$.)

Bases for Fundamental Subspaces

• Given an SVD for an $m\times n$ matrix $A$, let ${\mathbf{u}}_1,...,{\mathbf{u}}_m$ be the left singular vectors,${\mathbf{v}}_1,...,{\mathbf{v}}_n$ the right singular vectors, and ${\sigma }_1,...,{\sigma }_n$ the singular values, and let $r$ be the rank of $A$. By Theorem 9,
  { u 1 , . . . , u r }      ( 5 ) \{\boldsymbol u_1,...,\boldsymbol u_r\}\ \ \ \ (5) {u1​,...,ur​}    (5) is an orthonormal basis for $ColA$.
• Recall that $(ColA{)}^{\perp }=Nul{A}^T$ . Hence
  { u r + 1 , . . . , u m }      ( 6 ) \{\boldsymbol u_{r+1},...,\boldsymbol u_m\}\ \ \ \ (6) {ur+1​,...,um​}    (6)is an orthonormal basis for $Nul{A}^T$ .
• Since $\Vert A{\mathbf{v}}_i\Vert ={\sigma }_i$ for $1\le i\le n$, and ${\sigma }_i$ is 0 if and only if $i>r$, the vectors ${\mathbf{v}}_{r+1},...,{\mathbf{v}}_n$ span a subspace of $NulA$ of dimension $n-r$. By the Rank Theorem, $dimNulA=n-rankA=n-r$. It follows that
  { v r + 1 , . . . , v n }      ( 7 ) \{\boldsymbol v_{r+1},...,\boldsymbol v_n\}\ \ \ \ (7) {vr+1​,...,vn​}    (7)is an orthonormal basis for $NulA$.
• $(NulA{)}^{\perp }=Col{A}^T=RowA$. Hence, from $(7)$,
  { v 1 , . . . , v r }      ( 8 ) \{\boldsymbol v_1,...,\boldsymbol v_r\}\ \ \ \ (8) {v1​,...,vr​}    (8)is an orthonormal basis for $RowA$.

---

• Figure 4 summarizes $(5)–(8)$, but shows the orthogonal basis { σ 1 u 1 , . . . , σ r u r } \{\sigma_1\boldsymbol u_1,...,\sigma_r\boldsymbol u_r\} {σ1​u1​,...,σr​ur​} for $ColA$ instead of the normalized basis, to remind you that $A{\mathbf{v}}_i={\sigma }_i{\mathbf{u}}_i$ for $1\le i\le r$.
  

---

• The four fundamental subspaces and the concept of singular values provide the final statements of the Invertible Matrix Theorem.

Reduced SVD and the Pseudoinverse of $A$ (奇异值分解的简化和 $A$ 的伪逆)

• When $\Sigma$ contains rows or columns of zeros, a more compact decomposition of $A$ is possible. Using the notation established above, let $r=rankA$, and partition $U$ and $V$ into submatrices whose first blocks contain $r$ columns:
  Then $U_r$ is $m\times r$ and $V_r$ is $n\times r$. (To simplify notation, we consider $U_{m-r}$ or $V_{n-r}$ even though one of them may have no columns.) Then partitioned matrix multiplication shows that
  
• This factorization of $A$ is called a reduced singular value decomposition of $A$. Since the diagonal entries in $D$ are nonzero, $D$ is invertible. The following matrix is called the pseudoinverse (伪逆) (also, the Moore–Penrose inverse (穆尔-彭罗斯逆)) of $A$:
  

---

• The next Supplementary exercises explore some of the properties of the reduced singular value decomposition and the pseudoinverse.

Supplementary EXERCISE 12

• Verify the properties of $A^{+}$:
  a. For each $\mathbf{y}$ in ${\mathbb{R}}^m$, $A{A}^{+}\mathbf{y}$ is the orthogonal projection of $\mathbf{y}$ onto $ColA$.
  b. For each $\mathbf{x}$ in ${\mathbb{R}}^n$, $A^{+}A\mathbf{x}$ is the orthogonal projection of $\mathbf{x}$ onto $RowA$.
  c. $A{A}^{+}A=A$ and $A^{+}A{A}^{+}=A^{+}$.

Supplementary EXERCISE 13
Suppose the equation $A\mathbf{x}=\mathbf{b}$ is consistent, and let ${\mathbf{x}}^{+}=A^{+}\mathbf{b}$. By Exercise 23 in Section 6.3, there is exactly one vector $\mathbf{p}$ in $RowA$ such that $A\mathbf{p}=\mathbf{b}$. The following steps prove that ${\mathbf{x}}^{+}=\mathbf{p}$ and ${\mathbf{x}}^{+}$ is the minimum length solution of $A\mathbf{x}=\mathbf{b}$.
a. Show that ${\mathbf{x}}^{+}$ is in $RowA$.
b. Show that ${\mathbf{x}}^{+}$ is a solution of $A\mathbf{x}=\mathbf{b}$.
c. Show that if $\mathbf{u}$ is any solution of $A\mathbf{x}=\mathbf{b}$, then $\Vert {\mathbf{x}}^{+}\Vert \le \Vert \mathbf{u}\Vert$, with equality only if $\mathbf{u}={\mathbf{x}}^{+}$.
SOLUTION
a. ${\mathbf{x}}^{+}=V_r{D}^{-1}{U}_r^T$. Since the columns of $V_r$ form an orthonormal basis for $RowA$, ${\mathbf{x}}^{+}$ is a linear combination of the $RowA$'s orthonormal basis. Thus ${\mathbf{x}}^{+}$ is in $RowA$.
b. $A{\mathbf{x}}^{+}=A{A}^{+}\mathbf{b}=A{A}^{+}A\mathbf{x}=A\mathbf{x}=\mathbf{b}$
c. ${\mathbf{x}}^{+}$ is the orthogonal projection of $\mathbf{u}$ onto $RowA$. …

Supplementary EXERCISE 14
Given any $\mathbf{b}$ in ${\mathbb{R}}^m$, adapt Exercise 13 to show that $A^{+}\mathbf{b}$ is the least-squares solution of minimum length.
SOLUTION
[Hint: Consider the equation $A\mathbf{x}=\hat{\mathbf{b}}$, where $\hat{\mathbf{b}}$ is the orthogonal projection of $\mathbf{b}$ onto $ColA$.]

---

EXAMPLE 8 (Least-Squares Solution)
Given the equation $A\mathbf{x}=\mathbf{b}$, use the pseudoinverse of $A$ to define

Then,

$U_r{U}_r^T\mathbf{b}$ is the orthogonal projection $\hat{\mathbf{b}}$ of $\mathbf{b}$ onto $ColA$. Thus $\hat{\mathbf{x}}$ is a least-squares solution of $A\mathbf{x}=\mathbf{b}$. In fact, this $\hat{\mathbf{x}}$ has the smallest length among all least-squares solutions of $A\mathbf{x}=\mathbf{b}$. See Supplementary Exercise 14.

---

---

Ref

• 《统计学习方法》
• $Linear$ $algebra$ $and$ $its$ $applications$