线性代数 · SVD | 令人困扰的精度 2

注：本文为 “线性代数 · SVD” 相关英文引文，机翻未校。
如有内容异常，请看原文。

csdn 篇幅字数限制，分为两篇，此为第 2 篇。

• 线性代数 · SVD | 令人困扰的精度 1-优快云博客
  https://blog.youkuaiyun.com/u013669912/article/details/152056616

---

Properties

性质

Singular value decomposition has lots of useful properties, some of which we’ll prove here. First, note that taking the transpose of a singular value decomposition $M=U\Sigma V^T$ gives another singular value decomposition
奇异值分解具有许多实用性质，以下将证明其中一部分。首先，对奇异值分解 $M=U\Sigma V^T$ 取转置，可得到另一个奇异值分解

$$M^T=V{\Sigma }^T{U}^T$$

showing that $M^T$ has the same singular values as $M$, but with the left and right singular vectors swapped. This can be proven more conceptually as follows.
这表明 $M^T$ 与 $M$ 具有相同的奇异值，但左、右奇异向量互换。我们也可以从更概念化的角度证明这一点，如下所示。

Key lemma #2: Write $B(u,v)=⟨u,Mv⟩=⟨M^{T}u,v⟩$. Then for every $1\le i\le r$, the left and right singular vectors $u_i$, $v_i$ maximize the value of $B(u,v)$ subject to the constraint that $\Vert u\Vert =\Vert v\Vert =1$, $u$ is orthogonal to $u_j$ for all $j\le i-1$, and $v$ is orthogonal to $v_j$ for all $j\le i-1$. This maximum value is ${\sigma }_i$.
关键引理2：定义 $B(u,v)=⟨u,Mv⟩=⟨M^{T}u,v⟩$。则对任意 $1\le i\le r$，左奇异向量 $u_i$ 和右奇异向量 $v_i$ 满足：在约束条件（$\Vert u\Vert =\Vert v\Vert =1$、$u$ 与所有 $j\le i-1$ 的 $u_j$ 正交、$v$ 与所有 $j\le i-1$ 的 $v_j$ 正交）下，$B(u,v)$ 取得最大值，且该最大值为 ${\sigma }_i$。

Proof. At the maximum value of $B(u,v)$ subject to the above constraints, if we fix $v$ then $B(\cdot ,v)$ takes its maximum value at $u$. But for fixed $v$, $B(u,v)=⟨u,Mv⟩$ uniquely takes its maximum value when $u$ is proportional to $Mv$ (if $Mv\ne 0$), hence must in fact be equal to $\frac{Mv}{\Vert Mv\Vert }$; moreover, this is always possible thanks to key lemma #1. So we are in fact maximizing
证明：在上述约束条件下，当 $B(u,v)$ 取得最大值时，若固定 $v$，则 $B(\cdot ,v)$ 在 $u$ 处取得最大值。对固定的 $v$，$B(u,v)=⟨u,Mv⟩$ 的最大值唯一地在 $u$ 与 $Mv$ 成比例（若 $Mv\ne 0$）时取得，因此 $u$ 必等于 $\frac{Mv}{\Vert Mv\Vert }$；且借助关键引理1，这种选择始终可行。因此，我们实际上是在上述约束条件下最大化

$$⟨\frac{Mv}{\Vert Mv\Vert },Mv⟩=\Vert Mv\Vert$$

subject to the above constraints, and we already know the solution is given by $v=v_i$.
而我们已知该最大化问题的解为 $v=v_i$，即证得所需结论。

Left-right symmetry: Let ${\sigma }_i$, $u_i$, $v_i$ be the singular values, left singular vectors, and right singular vectors of $M$ as above. Then ${\sigma }_i$, $v_i$, $u_i$ are the singular values, left singular vectors, and right singular vectors of $M^T$. In particular, $M^T{u}_i={\sigma }_i{v}_i$.
左右对称性：设 ${\sigma }_i$、$u_i$、$v_i$ 分别为上述 $M$ 的奇异值、左奇异向量、右奇异向量，则 ${\sigma }_i$、$v_i$、$u_i$ 分别为 $M^T$ 的奇异值、左奇异向量、右奇异向量。特别地，有 $M^T{u}_i={\sigma }_i{v}_i$。

Proof. Apply key lemma #2 to $M^T$, and note that $B(u,v)$ is the same for $M$ and $M^T$, just with the roles of $u$ and $v$ switched.
证明：对 $M^T$ 应用关键引理2，注意到 $M$ 和 $M^T$ 对应的 $B(u,v)$ 完全相同，仅需交换 $u$ 和 $v$ 的角色，即证得所需结论。

Singular = eigen: The left singular vectors $u_i$ are the eigenvectors of $M{M}^T$ corresponding to its nonzero eigenvalues, which are ${\sigma }_i^2$ for $1\le i\le r$. The right singular vectors $v_i$ are the eigenvectors of $M^{T}M$ corresponding to its nonzero eigenvalues, which are also ${\sigma }_i^2$ for $1\le i\le r$.
奇异向量与特征向量的关系：左奇异向量 $u_i$ 是矩阵 $M{M}^T$ 对应于非零特征值的特征向量，这些非零特征值为 $1\le i\le r$ 对应的 ${\sigma }_i^2$；右奇异向量 $v_i$ 是矩阵 $M^{T}M$ 对应于非零特征值的特征向量，这些非零特征值同样为 $1\le i\le r$ 对应的 ${\sigma }_i^2$。

Proof. We now know that $M{v}_i={\sigma }_i{u}_i$ and that $M^T{u}_i={\sigma }_i{v}_i$, hence
证明：由前文可知 $M{v}_i={\sigma }_i{u}_i$ 且 $M^T{u}_i={\sigma }_i{v}_i$，因此

$$M^{T}M{v}_i=M^T({\sigma }_i{u}_i)={\sigma }_i^2{v}_i$$

and
同时有

$$M{M}^T{u}_i=M({\sigma }_i{v}_i)={\sigma }_i^2{u}_i.$$

Hence $v_i$, $u_i$ are orthonormal eigenvectors of $M^{T}M$, $M{M}^T$ respectively. Moreover, these matrices have rank at most (in fact exactly) $r$, so this exhausts all eigenvectors corresponding to nonzero eigenvalues.
因此，$v_i$ 和 $u_i$ 分别是 $M^{T}M$ 和 $M{M}^T$ 的标准正交特征向量。此外，这两个矩阵的秩至多为 $r$（实际上恰好为 $r$），因此上述特征向量已穷尽所有对应于非零特征值的特征向量。

This gives an alternative route to understanding singular value decomposition which comes from writing $\Vert Mv{\Vert }^2$ as
这为理解奇异值分解提供了另一种途径：将 $\Vert Mv{\Vert }^2$ 表示为

$$\Vert Mv{\Vert }^2=⟨Mv,Mv⟩=⟨v,M^{T}Mv⟩$$

and then applying the spectral theorem (https://en.wikipedia.org/wiki/Spectral_theorem) to $M^{T}M$ to diagonalize it. But I think it’s worth knowing that there’s a route to singular value decomposition which is independent of the spectral theorem.
然后对 $M^{T}M$ 应用谱定理（https://en.wikipedia.org/wiki/Spectral_theorem）将其对角化。但值得注意的是，存在一种不依赖于谱定理的奇异值分解构造方法（即前文所述方法）。

In addition to the above algebraic characterization of singular values, the singular values also admit the following variational characterization.
除了上述奇异值的代数刻画外，奇异值还具有如下变分刻画。

Variational characterizations of singular values (Courant-Fischer): We have
奇异值的变分刻画（柯朗-费希尔定理）：

$${\sigma }_k=\underset{V\subseteq {\mathbb{R}}^m,\dim V=k}{\max }\underset{v\in V,\Vert v\Vert =1}{\min }\Vert Mv\Vert$$

以及

$${\sigma }_{k+1}=\underset{V\subseteq {\mathbb{R}}^m,\dim V=m-k}{\min }\underset{v\in V,\Vert v\Vert =1}{\max }\Vert Mv\Vert$$

Proof. For the first characterization, any $k$-dimensional subspace $V$ intersects $span(v_k,\dots ,v_m)$ nontrivially, hence contains a unit vector of the form
证明：对于第一个刻画，任意 $k$ 维子空间 $V$ 与 $span(v_k,\dots ,v_m)$ 的交集非空，因此 $V$ 中必包含形如

$$v=\sum _{i=k}^m{c}_i{v}_i,\Vert v\Vert =\sum _{i=k}^m{c}_i^2=1.$$

We compute that
计算可得

$$Mv=\sum _{i=k}^m{c}_i{\sigma }_i{u}_i$$

and hence that
因此

$$\Vert Mv{\Vert }^2=\sum _{i=k}^m{c}_i^2{\sigma }_i^2\le {\sigma }_k^2.$$

We conclude that every $V$ contains a $v$ such that $\Vert Mv\Vert \le {\sigma }_k$, hence ${\min }_{v\in V,\Vert v\Vert =1}\Vert Mv\Vert \le {\sigma }_k$. Equality is obtained when $V=span(v_1,\dots ,v_k)$.
由此可知，每个子空间 $V$ 中都存在向量 $v$ 使得 $\Vert Mv\Vert \le {\sigma }_k$，因此 ${\min }_{v\in V,\Vert v\Vert =1}\Vert Mv\Vert \le {\sigma }_k$。当 $V=span(v_1,\dots ,v_k)$ 时，等号成立。

The second characterization is very similar. Any $m-k$-dimensional subspace $V$ intersects $span(v_1,\dots ,v_{k+1})$ nontrivially, hence contains a unit vector of the form
第二个刻画的证明非常类似。任意 $m-k$ 维子空间 $V$ 与 $span(v_1,\dots ,v_{k+1})$ 的交集非空，因此 $V$ 中必包含形如

$$v=\sum _{i=1}^{k+1}{c}_i{v}_i,\Vert v\Vert =\sum _{i=1}^{k+1}{c}_i^2=1.$$

$$v=\sum _{i=1}^{k+1}{c}_i{v}_i$$ 的单位向量，其中 $\Vert v\Vert ={\sum }_{i=1}^{k+1}{c}_i^2=1$

$$Mv=\sum _{i=1}^{k+1}{c}_i{\sigma }_i{u}_i$$

$$\Vert Mv{\Vert }^2=\sum _{i=1}^{k+1}{c}_i^2{\sigma }_i^2\ge {\sigma }_{k+1}^2.$$

We conclude that every $V$ contains a vector $v$ such that $\Vert Mv\Vert \ge {\sigma }_{k+1}$, hence ${\max }_{v\in V,\Vert v\Vert =1}\Vert Mv\Vert \ge {\sigma }_{k+1}$. Equality is obtained when $V=span(v_{k+1},\dots ,v_m)$.
由此可知，每个子空间 $V$ 中都存在向量 $v$ 使得 $\Vert Mv\Vert \ge {\sigma }_{k+1}$，因此 ${\max }_{v\in V,\Vert v\Vert =1}\Vert Mv\Vert \ge {\sigma }_{k+1}$。当 $V=span(v_{k+1},\dots ,v_m)$ 时，等号成立。

The second variational characterization above can be used to prove the following important theorem.
上述第二个变分刻画可用于证明下述重要定理。

Low rank approximation (Eckart-Young): If $M=U\Sigma V^T$ is the SVD of $M$, let $M_k=U{\Sigma }_k{V}^T$ where ${\Sigma }_k$ has diagonal entries ${\sigma }_1,\dots ,{\sigma }_k$ and all other entries zero. Then $M_k$ is the closest matrix to $M$ in operator norm with rank at most $k$; that is, $M_k$ minimizes $\Vert M-X\Vert$ subject to the constraint that $rank(X)\le k$. This minimum value is ${\sigma }_{k+1}$.
低秩逼近（埃卡特-杨定理）：设 $M=U\Sigma V^T$ 是 $M$ 的奇异值分解，定义 $M_k=U{\Sigma }_k{V}^T$，其中 ${\Sigma }_k$ 的对角元为 ${\sigma }_1,\dots ,{\sigma }_k$，其余元素均为零。则 $M_k$ 是算子范数意义下与 $M$ 最接近的秩至多为 $k$ 的矩阵；即 $M_k$ 在约束条件 $rank(X)\le k$ 下最小化 $\Vert M-X\Vert$，且该最小值为 ${\sigma }_{k+1}$。

Proof. Suppose $X$ is a matrix of rank at most $k$. Let $W=ker(X)$ be the nullspace of $X$, which by hypothesis has dimension at least $m-k$. By the second variational characterization above, this means that $W$ contains a vector $w$ such that $\Vert Mw\Vert \ge {\sigma }_{k+1}$, and since $Xw=0$ this gives
证明：设 $X$ 是秩至多为 $k$ 的矩阵，令 $W=ker(X)$（即 $X$ 的零空间），由假设可知 $\dim W\ge m-k$。根据上述第二个变分刻画，$W$ 中必存在向量 $w$ 使得 $\Vert Mw\Vert \ge {\sigma }_{k+1}$；又因为 $Xw=0$，因此有
$$\Vert (M-X)w\Vert =\Vert Mw\Vert \ge {\sigma }_{k+1}$$

and hence that $\Vert M-X\Vert \ge {\sigma }_{k+1}$. Equality is obtained when $X=M_k$ as defined above.
进而可得 $\Vert M-X\Vert \ge {\sigma }_{k+1}$。当 $X=M_k$（如上述定义）时，等号成立。

The variational characterizations can also be used to prove the following inequality relating the singular values of two matrices and of their sum, which can be thought of as a quantitative refinement of the observation that the rank of a sum $M+N$ of two matrices is at most the sum of their ranks.
变分刻画还可用于证明下述不等式——该不等式将两个矩阵及其和的奇异值联系起来，可视为对“两个矩阵之和 $M+N$ 的秩至多为两矩阵秩之和”这一结论的定量细化。

Additive perturbation (Weyl): Let $M$, $N$ be $n\times m$ matrices with singular values ${\sigma }_i(M)$, ${\sigma }_i(N)$. Then
加性扰动（外尔不等式）：设 $M$、$N$ 均为 $n\times m$ 矩阵，其奇异值分别为 ${\sigma }_i(M)$、${\sigma }_i(N)$，则

$${\sigma }_{k+\ell +1}(M+N)\le {\sigma }_{k+1}(M)+{\sigma }_{\ell +1}(N).$$

Proof. We want to bound ${\sigma }_{k+\ell +1}(M+N)$ in terms of the singular values of $M$ and $N$. By the second variational characterization, we have
证明：我们希望用 $M$ 和 $N$ 的奇异值来估计 ${\sigma }_{k+\ell +1}(M+N)$。根据第二个变分刻画，有

$${\sigma }_{k+\ell +1}(M+N)=\underset{V\subseteq {\mathbb{R}}^m,\dim V=m-k-\ell }{\min }\underset{v\in V,\Vert v\Vert =1}{\max }\Vert (M+N)v\Vert .$$

To give an upper bound on a minimum value of a function, we just need to give an upper bound on some value that it takes. Let $V_M$ and $V_N$ be the subspaces of ${\mathbb{R}}^m$ of dimensions $m-k$, $m-\ell$ respectively which achieve the minimum values of ${\max }_{v\in V_M,\Vert v\Vert =1}\Vert Mv\Vert$ and ${\max }_{v\in V_N,\Vert v\Vert =1}\Vert Nv\Vert$ respectively, and let $W=V_M\cap V_N$ be their intersection. This intersection has dimension at least $m-k-\ell$, and by construction
要对一个函数的最小值给出上界，只需对该函数的某个取值给出上界即可。设 $V_M$ 和 $V_N$ 分别是 ${\mathbb{R}}^m$ 中维数为 $m-k$ 和 $m-\ell$ 的子空间，且分别使得 ${\max }_{v\in V_M,\Vert v\Vert =1}\Vert Mv\Vert$ 和 ${\max }_{v\in V_N,\Vert v\Vert =1}\Vert Nv\Vert$ 取得最小值。令 $W=V_M\cap V_N$（即两子空间的交集），则 $\dim W\ge m-k-\ell$；根据构造，有

$$\underset{v\in W,\Vert v\Vert =1}{\max }\Vert Mv+Nv\Vert \le \underset{v\in W,\Vert v\Vert =1}{\max }\Vert Mv\Vert +\underset{v\in W,\Vert v\Vert =1}{\max }\Vert Nv\Vert \le {\sigma }_{k+1}(M)+{\sigma }_{\ell +1}(N).$$

Since $W$ has dimension at least $m-k-\ell$, the above is an upper bound on the value of ${\max }_{v\in V,\Vert v\Vert =1}\Vert (M+N)v\Vert$ for any $(m-k-\ell )$-dimensional subspace $V\subseteq W$, from which the conclusion follows.
由于 $W$ 的维数至少为 $m-k-\ell$，上述不等式给出了对任意 $(m-k-\ell )$ 维子空间 $V\subseteq W$ 而言，${\max }_{v\in V,\Vert v\Vert =1}\Vert (M+N)v\Vert$ 的上界，由此可推出所需结论。

The slightly curious off-by-one indexing in the above inequality can be understood as follows: if ${\sigma }_{k+1}(M)$ and ${\sigma }_{\ell +1}(N)$ are both very small, this means that $M$ and $N$ are close to matrices of rank at most $k$ and $\ell$ respectively, and hence $M+N$ is close to a matrix of rank at most $k+\ell$, hence ${\sigma }_{k+\ell +1}(M+N)$ also ought to be small.
上述不等式中略显特别的“错位1”下标可解释如下：若 ${\sigma }_{k+1}(M)$ 和 ${\sigma }_{\ell +1}(N)$ 都非常小，则意味着 $M$ 接近秩至多为 $k$ 的矩阵，$N$ 接近秩至多为 $\ell$ 的矩阵，因此 $M+N$ 接近秩至多为 $k+\ell$ 的矩阵，进而 ${\sigma }_{k+\ell +1}(M+N)$ 也应当很小。

Setting $\ell =0$ in the additive perturbation inequality we deduce the following corollary.
在加性扰动不等式中令 $\ell =0$，可推出如下推论。

Singular values are Lipschitz: The singular values, as functions on matrices, are uniformly Lipschitz with respect to the operator norm with Lipschitz constant 1: that is,
奇异值的 Lipschitz 连续性：奇异值作为定义在矩阵上的函数，关于算子范数是一致 Lipschitz 连续的，且 Lipschitz 常数为 1，即

$$\left|{\sigma }_k(M)-{\sigma }_k(N)\right|\le \Vert M-N\Vert .$$

Proof. Apply additive perturbation twice with $\ell =0$, first to get
证明：令 $\ell =0$，两次应用加性扰动不等式。第一次应用可得

$${\sigma }_k(M)\le {\sigma }_k(N)+{\sigma }_1(M-N)$$

(remembering that ${\sigma }_1$ is the operator norm), and second to get
（注意 ${\sigma }_1$ 即为算子范数）；第二次应用可得

$${\sigma }_k(N)\le {\sigma }_k(M)+{\sigma }_1(N-M)$$

(remembering that 注意 ${\sigma }_1(N-M)={\sigma }_1(M-N)$).

Combining these two inequalities gives $\left|{\sigma }_k(M)-{\sigma }_k(N)\right|\le \Vert M-N\Vert$.
将这两个不等式结合，即可得到 $\left|{\sigma }_k(M)-{\sigma }_k(N)\right|\le \Vert M-N\Vert$。

This is very much not the case with eigenvalues: a small perturbation of a square matrix can have a large effect on its eigenvalues. This is explained e.g. in this blog post by Terence Tao, and is related to pseudospectra.
这一性质与特征值形成鲜明对比：方阵的微小扰动可能导致其特征值发生巨大变化。陶哲轩（Terence Tao）在其博客文章中对此进行了阐述，该现象与伪谱相关。

Setting ${\sigma }_{\ell +1}(N)=0$ (or equivalently $rank(N)\le \ell$) in the additive perturbation inequality, we deduce the following corollary.
在加性扰动不等式中令 ${\sigma }_{\ell +1}(N)=0$（等价于 $rank(N)\le \ell$），可推出如下推论。

Interlacing: Suppose $M$, $N$ are matrices such that $rank(M-N)\le \ell$. Then
交错性：设 $M$、$N$ 为矩阵，且满足 $rank(M-N)\le \ell$，则

$${\sigma }_{k+\ell }(M)\le {\sigma }_k(N)\le {\sigma }_{k-\ell }(M).$$

(Here, we take ${\sigma }_i(A)=0$ if $i>rank(A)$ and ${\sigma }_i(A)=\infty$ if $i<1$ for any matrix $A$.)
（对于任意矩阵 $A$，此处约定：若 $i>rank(A)$，则 ${\sigma }_i(A)=0$；若 $i<1$，则 ${\sigma }_i(A)=\infty$。）

Proof. Apply additive perturbation twice, first to get
证明：两次应用加性扰动不等式。第一次应用可得

$${\sigma }_{k+\ell }(M)={\sigma }_{(k)+(\ell )+1-1}(M)\le {\sigma }_k(N)+{\sigma }_{\ell +1}(M-N)={\sigma }_k(N)$$

$${\sigma }_{k+\ell }(M)\le {\sigma }_k(N)+{\sigma }_{\ell +1}(M-N)={\sigma }_k(N)$$

(since 由于 ${\sigma }_{\ell +1}(M-N)=0$ because 故 $rank(M-N)\le \ell$), and second to get 第二次应用可得

$${\sigma }_k(N)\le {\sigma }_{k-\ell }(M)+{\sigma }_{\ell +1}(N-M)={\sigma }_{k-\ell }(M)$$

(since ${\sigma }_{\ell +1}(N-M)={\sigma }_{\ell +1}(M-N)=0$).

This completes the proof.
至此证明完毕。

Interlacing gives us some control over what happens to the singular values under a low-rank perturbation (as opposed to a low-norm perturbation; a low-rank perturbation may have arbitrarily high norm, and vice versa). For example, we learn that if all of the singular values of $M$ are clumped together, then a rank-$\ell$ perturbation will keep most of the singular values clumped together, except possibly for either the $\ell$ largest or $\ell$ smallest singular values. We can’t expect any control over these, since in the worst case a rank-$\ell$ perturbation can make the $\ell$ largest singular values arbitrarily large, or make the $\ell$ smallest singular values arbitrarily small.
交错性使我们能够控制低秩扰动下奇异值的变化（与低范数扰动不同：低秩扰动的范数可能任意大，反之亦然）。例如，若 $M$ 的所有奇异值都聚集在一起，则秩为 $\ell$ 的扰动会使大部分奇异值仍保持聚集状态，仅可能影响最大的 $\ell$ 个或最小的 $\ell$ 个奇异值。我们无法对这部分奇异值的变化进行控制，因为在最坏情况下，秩为 $\ell$ 的扰动可使最大的 $\ell$ 个奇异值变得任意大，或使最小的 $\ell$ 个奇异值变得任意小。

A particular special case of a low-rank perturbation is deleting a small number of rows or columns (note that a row or column which is entirely zero does not affect the singular values, so deleting a row or column is equivalent to setting all of its entries to zero), in which case the upper bound above can be tightened.
低秩扰动的一个特殊情形是删除少量行或列（注意：全零行或全零列不影响奇异值，因此删除一行或一列等价于将该行或列的所有元素设为零），此时上述上界可进一步收紧。

Cauchy interlacing: Suppose $M$ is a matrix and $N$ is obtained from $M$ by deleting at most $\ell$ rows. Then
柯西交错性：设 $M$ 为矩阵，$N$ 是由 $M$ 删除至多 $\ell$ 行后得到的矩阵，则

$${\sigma }_k(M)\ge {\sigma }_k(N)\ge {\sigma }_{k+\ell }(M)$$

Proof. The lower bound follows from interlacing (since deleting $\ell$ rows is a rank-$\ell$ perturbation). The upper bound follows from the observation that we have $\Vert Nv\Vert \le \Vert Mv\Vert$ for all $v$, then applying either variational characterization of the singular values.
证明：下界可由交错性推出（删除 $\ell$ 行属于秩为 $\ell$ 的扰动）。上界可由如下观察推出：对所有向量 $v$，有 $\Vert Nv\Vert \le \Vert Mv\Vert$；再应用奇异值的任意一个变分刻画，即可得到上界。

Cauchy interlacing also applies to deleting columns, or combinations of rows and columns, because the singular values are unchanged by transposition. In particular, we learn that if $N$ is obtained from $M$ by deleting either a single row or a single column, then the singular values of $N$ interlace with the singular values of $M$, hence the name.
柯西交错性同样适用于删除列或同时删除行和列的情形，因为转置不改变矩阵的奇异值。特别地，若 $N$ 是由 $M$ 删除一行或一列得到的矩阵，则 $N$ 的奇异值与 $M$ 的奇异值满足交错关系，“交错性”由此得名。

In particular, if all of the singular values of $M$ are clumped together then so are those of $N$, with no exceptions. Taking the contrapositive, if the singular values of $N$ are spread out, then the singular values of $M$ must be as well.
特别地，若 $M$ 的所有奇异值都聚集在一起，则 $N$ 的所有奇异值也必然聚集在一起，无一例外。取其逆否命题：若 $N$ 的奇异值分散，则 $M$ 的奇异值也必然分散。

Three special cases

三种特殊情形

Three special cases of the general singular value decomposition $M=U\Sigma V^T$ are worth pointing out.
一般形式的奇异值分解 $M=U\Sigma V^T$ 有三种特殊情形值得关注。

First, if $M$ has orthogonal columns, or equivalently if $M^{T}M$ is diagonal, then the singular values ${\sigma }_i$ are the lengths of its columns, we can take the right singular vectors to be the standard basis vectors $v_i=e_i$ and we can take the left singular vectors to be the unit rescalings of its columns. This means that we can take $V=I$ to be the identity matrix, and in general suggests that $\Vert I-V\Vert$ is a measure of the extent to which the columns of $M$ fail to be orthogonal (with the caveat that $V$ is not unique and so in general we would want to look at the $V$ closest to $I$).
第一种情形：若 $M$ 的列向量正交（等价于 $M^{T}M$ 为对角矩阵），则奇异值 ${\sigma }_i$ 等于各列向量的长度；右奇异向量可取标准基向量 $v_i=e_i$；左奇异向量可取各列向量单位化后的向量。这意味着我们可令 $V=I$（单位矩阵），且通常而言，$\Vert I-V\Vert$ 可用于衡量 $M$ 的列向量偏离正交性的程度（需注意：$V$ 并非唯一，因此通常需选取与 $I$ 最接近的 $V$）。

Second, if $M$ has orthogonal rows, or equivalently if $M{M}^T$ is diagonal, then the singular values ${\sigma }_i$ are the lengths of its rows, we can take the left singular vectors to be the standard basis vectors $u_i=e_i$, and we can take the right singular vectors to be the unit rescalings of its rows. This means that we can take $U=I$ to be the identity matrix, and in general suggests that $\Vert I-U\Vert$ is a measure of the extent to which the rows of $M$ fail to be orthogonal (with the same caveat as above).
第二种情形：若 $M$ 的行向量正交（等价于 $M{M}^T$ 为对角矩阵），则奇异值 ${\sigma }_i$ 等于各行向量的长度；左奇异向量可取标准基向量 $u_i=e_i$；右奇异向量可取各行向量单位化后的向量。这意味着我们可令 $U=I$（单位矩阵），且通常而言，$\Vert I-U\Vert$ 可用于衡量 $M$ 的行向量偏离正交性的程度（需注意：与上述情形相同，$U$ 并非唯一）。

Finally, if $M$ is square and an orthogonal matrix, so that $M^{T}M=M{M}^T=I$, then the singular values ${\sigma }_i$ are all equal to 1, and an arbitrary choice of either the left or the right singular vectors uniquely determines the other. This means that we can take $\Sigma =I$ to be the identity matrix, and in general suggests that $\Vert I-\Sigma \Vert$ is a measure of the extent to which $M$ fails to be orthogonal. In fact it is possible to show that the closest orthogonal matrix to $M=U\Sigma V^T$ is given by $U{V}^T$, or in other words by replacing all of the singular values of $M$ with 1, so

第三种情形：若 $M$ 是方阵且为正交矩阵（即满足 $M^{T}M=M{M}^T=I$），则所有奇异值 ${\sigma }_i$ 均等于 1；且任意选定左奇异向量或右奇异向量后，另一组奇异向量会被唯一确定。这意味着我们可令 $\Sigma =I$（单位矩阵），且通常而言，$\Vert I-\Sigma \Vert$ 可用于衡量 $M$ 偏离正交性的程度。事实上可证明：与 $M=U\Sigma V^T$ 最接近的正交矩阵为 $U{V}^T$，换言之，是将 $M$ 的所有奇异值替换为 1 后得到的矩阵，因此

$$\Vert I-\Sigma \Vert =\underset{i}{\max }\left|1-{\sigma }_i\right|$$

is precisely the distance from $M$ to the nearest orthogonal matrix. This fact can be used to solve the orthogonal Procrustes problem.
$\Vert I-\Sigma \Vert$ 恰好等于 $M$ 到最近正交矩阵的距离。这一事实可用于求解正交普罗克拉斯提斯问题。

In general, we should expect that the SVD of a matrix $M$ is relevant to answering any question about $M$ whose answer is invariant under left and right multiplication by orthogonal matrices. This includes, for example, the question of low-rank approximations to $M$ with respect to operator norm we answered above, since both rank and operator norm are invariant.
一般而言，对于任何关于矩阵 $M$ 的问题，若其答案在 $M$ 左右两侧均乘以正交矩阵后保持不变，则 $M$ 的奇异值分解（SVD）必然与该问题的解答相关。例如，我们前文讨论的“算子范数意义下 $M$ 的低秩逼近”问题就属于此类——因为矩阵的秩和算子范数均具有正交不变性。

Posted in math.SP | 4 Comments

4 Responses

Ramsay

on March 18, 2017 at 11:17 am | Reply

Btw. In the proof of interlacing, I don’t see the first displayed equality: ${\sigma }_k(N)+{\sigma }_{\ell +1}(M-N)={\sigma }_{k+\ell }(N)$. What am I missing?
顺便提一句：在交错性的证明中，我无法理解第一个显式等式 ${\sigma }_k(N)+{\sigma }_{\ell +1}(M-N)={\sigma }_{k+\ell }(N)$，我哪里理解错了？

Qiaochu Yuan

on March 18, 2017 at 2:14 pm | Reply

Oops, that’s a typo; it should just be ${\sigma }_k(N)$ on the RHS.
哦，那是一个笔误；等式右边应该只有 ${\sigma }_k(N)$。

on March 18, 2017 at 11:14 am | Reply

Thanks for that concise and clear introduction to the SVD. I do not understand why it is often not even touched in a first class in linear algebra. It seems to me that it would make sense to introduce it even before the spectral theorem.
感谢你对奇异值分解（SVD）简洁清晰的介绍。我不明白为什么线性代数入门课程通常甚至不会提及它——在我看来，甚至应该在谱定理之前介绍奇异值分解才合理。

Regarding “weighted projections”: up to a scale factor (i.e., a single weight ${\sigma }_1$), you can view a linear transformation $T:X\to Y$ between Euclidean spaces as an orthogonal projection. Specifically, if $X$ and $Y$ are subspaces of ${\mathbb{R}}^N$ of dimension $n$ and $m$ respectively, and $P_{Y|X}$ is the orthogonal projection onto $Y$ restricted to $X$, then the singular values of $P_{Y|X}$ are the cosines of the principal angles. If $N\ge m+n$, then these singular values can take any value in $[0,1]$. So we see that any $T:X\to Y$ can be represented as ${\sigma }_1(T)P_{Y|X}$ for an appropriate choice of $X,Y$ as subspaces of ${\mathbb{R}}^{m+n-1}$.
关于“加权投影”：在相差一个比例因子（即单个权重 ${\sigma }_1$）的意义下，欧几里得空间之间的线性变换 $T:X\to Y$ 可视为正交投影。具体而言，若 $X$ 和 $Y$ 分别是 ${\mathbb{R}}^N$ 中维数为 $n$ 和 $m$ 的子空间，且 $P_{Y|X}$ 是“到 $Y$ 的正交投影”在 $X$ 上的限制，则 $P_{Y|X}$ 的奇异值等于主角度的余弦值。若 $N\ge m+n$，则这些奇异值可取 $[0,1]$ 中的任意值。因此可知，对 ${\mathbb{R}}^{m+n-1}$ 中适当选取的子空间 $X,Y$，任意线性变换 $T:X\to Y$ 均可表示为 ${\sigma }_1(T)P_{Y|X}$。

Also, regarding the best orthogonal transformation to represent $M$, it is worth pointing out that you are talking about the orthogonal factor in the polar decomposition, which is an immediate consequence of the SVD. We can always represent our matrix $M$ as a composition of an orthogonal matrix and a positive semidefinite matrix: $M=\Phi R=R^{\prime }\Phi$ where $\Phi =U{V}^T$ and $R=V\Sigma V^T$ and $R^{\prime }=U\Sigma U^T$.
此外，关于“表示 $M$ 的最佳正交变换”，值得指出的是：你所讨论的是极分解中的正交因子，而极分解是奇异值分解（SVD）的直接推论。我们总可将矩阵 $M$ 表示为一个正交矩阵与一个半正定矩阵的乘积：$M=\Phi R=R^{\prime }\Phi$，其中 $\Phi =U{V}^T$，$R=V\Sigma V^T$，$R^{\prime }=U\Sigma U^T$。

Ammar Husain

on March 15, 2017 at 2:57 am | Reply

Thinking of the QR algorithm and the Toda lattice, you gain when you swap and refactor repeatedly as dressing transformations. Haven’t thought about what if anything reasonable happens when you permute the factors in a square SVD.
考虑到 QR 算法和托达（Toda）格子，当你将“交换因子”和“重新分解”作为修饰变换（dressing transformations）反复进行时，会得到一些有用的结果。但我尚未思考过：对 square SVD（方阵的奇异值分解）中的因子进行置换时，是否会产生任何有意义的结果。

---

via:

• Singular value decomposition | Annoying Precision
  https://qchu.wordpress.com/2017/03/13/singular-value-decomposition/