MIT 线性代数

vedio 9

Suppose A m by n while m < n, then there are nonzero solutions to AX = 0.
Reason: There will be free variables. (rank < n)
When $v_1,…, v_n$ are columns of A, they are independent if nullspace of A is only {zero vector}. (rank = n)
They are dependent if AC = 0 for some nonzero C.
Vectors $v_1,...,v_l$ span a space means: The space consists all combinations of those vectors.
Basis for a space is a sequence of vectors $v_1,..., v_d$ with 2 properties
1 They are independent
2 They span the space
Every basis for the space has the same number of vectors.

rank(A) = number of pivot columns = dimention of C(A)
dim N(A) = number of free variables = n - r

vedio 10

4 foundermental sub-spaces:
columnspace: C(A) in $R^m$, to be exactly, it is r, and r < m
nullspace: N(A) in $R^n$, it is (n - r)
rowspace = all colums of row = all columns of $A^T$: $C(A^T)$ in $R^n$
nullspace of $A^T$: $N(A^T)$ in $R^m$= left null space of A

left null space 名字由来:
$A^Ty = 0 \Rightarrow (A^Ty)^T = 0 \Rightarrow y^TA = 0$

Ex:
A = [1, 1, 1, 1]
dim rowspace of A = 1
dim N(A) = n - r = 3
dim C(A) = 1
dim N($A^T$) = 0

C(A)	N(A)
pivot columns	special columns
r	n - r

行变换以后 same row space, different column space

vedio 11

矩阵空间$R^{3X3}$是9维，basis有9个，分别如下：
$\begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ \end{bmatrix}$
$\begin{bmatrix} 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ \end{bmatrix}$
$\begin{bmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ \end{bmatrix}$
……
$\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix}$

S是三维对称矩阵，U是上三角矩阵(似乎基的数量等于维数)
dim S = 6, dim U = 6
dim(S $\cap$ U) = 3
dim(S + U) = 9 (S + U can span $R^3$)

Graph = {nodes, edges}

vedio 14

number	name	dim
1	row space	r
2	null space	n - r
3	column space	r
4	null space of $A^T$	m - r

1 is orthogonal to 2, 3 is orthogonal to 4

if x is orthogonal to y, $x^Ty = 0$, then $x^Tx + y^Ty = (x + y)^T(x + y)$
Proof:
$x^Tx + y^Ty = x^Tx + x^Ty + y^Tx + y^Ty$
$\Leftrightarrow 0 = x^Ty + y^Tx = 2x^Ty$
$\Leftrightarrow 0 = x^Ty$

Subspace S is orthogonal to Subspace T means: every vector in S is orthogonal to every vector in T.

Nullspace(n - r) and rowspace(r) are orthogonal Complements(补) in $R^n.$

Nullspace contains all vectors $\bot$ rowspace.

Ax = b(m > n), when there is no solution: $A^TA\hat x = A^Tb$

$N(A^TA) = N(A)$
$rank\ of\ A^TA = rank\ of\ A$
$A^TA\ is\ invertible\ exactly\ if\ A\ has\ independent\ columns.$

vedio 15

有关投影，b在a上的投影为p，p = ax，e = b - p，
a 和e正交，因此$a^Te = 0$
$\Leftrightarrow a^T(b - p) = 0$
$\Leftrightarrow a^T(b - ax) = 0$
$\Leftrightarrow a^Tax =a^Tb$
$\Leftrightarrow x = \frac {a^Tb}{a^Ta}$
$p = ax = a\frac {a^Tb}{a^Ta} = \frac{aa^T}{a^Ta} b$

project matric(投影矩阵为) $\mathbb P = \frac{aa^T}{a^Ta}$

$\mathbb P^T = \mathbb P$ 从上式就可以看出来
$\mathbb P^2 = \mathbb P$ 投影两次和一次的结果是一样的

为什么要投影，why project?
Because Ax = b may have no solution, then solve $A\hat x =\mathbb P$ instead

二维情况下:
$\mathbb P = A\hat x ，\ find \ \hat x$
key： $e = b - A\hat x$ is perpendicular to plane

$a_1^T(b - A\hat x) = 0$
$a_2^T(b - A\hat x) = 0$

$\left[ \begin{matrix} a_1^T \\ a_2^T \\ \end{matrix} \right] (b - A\hat x) = \left[ \begin{matrix} 0 \\ 0 \\ \end{matrix} \right]$
$\Leftrightarrow A^T(b - A\hat x) = 0$
$\Leftrightarrow A^TA\hat x = A^Tb$
$\Leftrightarrow \hat x = \frac {A^Tb}{A^TA}$
$p = A\hat x = A(A^TA)^{-1}A^Tb$
投影矩阵$\mathbb P = A(A^TA)^{-1}A^T$
$\mathbb P^T = \mathbb P,\ \mathbb P^2 = \mathbb P$

vedio 16

$\mathbb P = A(A^TA)^{-1}A^T$
If b in column space $\mathbb Pb = b$
If b $\bot$ column space $\mathbb Pb = 0$

数学上的解释:
Column space 与 Nullspace of $A^T$ 正交，

如果b in column space, $b = Ax,$
$\mathbb Pb = A(A^TA)^{-1}A^Tb = A(A^TA)^{-1}A^TAx = Ax = b$

如果b in Null space of $A^T, A^Tb = 0 \Rightarrow \mathbb Pb = 0$

对于任意向量b，在C(A)上的投影为p，在N(A)上的投影为e， p + e = b
其中$p = \mathbb Pb, e = (I - \mathbb P)b$
也很好理解，$e = b - p = b - \mathbb Pb = (I - \mathbb P)b$

线性规划的本质就是 Minimize $||Ax - b||^2 = e^2$
当我们没办法求出Ax = b时，只能求$A\hat x = p$, p是b在C(A)上的投影
$A\hat x = p =\mathbb Pb = A(A^TA)^{-1}A^Tb$
$\Leftrightarrow A^TA\hat x = A^Tb$

$e = b - p = b - A\hat x$

If A has independent Columns space, then $A^TA$ is invertible.