线性代数笔记（网易公开课）

Linear Algebra Handnote(1)

• If $L$ is lower triangular with 1’s on the diagonal, so is $L^{-1}$

• Elimination = Facotization: $A=LU$

• $A^T$ is the matrix that makes these two inner products equal for every $x$ and $y$:
  

  $$(Ax)^Ty=x^T(A^Ty)$$
  Inner product of $Ax$ with $y$ = Inner product of $x$ with $A^Ty$

• DEFINITION: The space $R^n$ consists of all column vectors $v$ with $n$ components

• DEFINITION: A subspace of a vector space is a set of vectors (including 0) that satisfies two requirements: (1) $v+w$ is in the subspace, (2) $cv$ is in the subspace

• The colomn space consists of all linear combinations of the columns. The combinations are all possible vectors $Ax$. They fill the column space $C(A)$

  The system $Ax=b$ is solvable if and only if $b$ is in the column space of $A$

• The nullspace of A consists of all solutions to $Ax=0$. These vectors $x$ are in $R^n$. The nullspace containing all solutions of $Ax=0$ is denoted by $N(A)$

• the nullspace is a subspace of $R^n$, the column space is a subspace of $R^m$
• the nullspace consists of all combinations of the special solutions

• Nullspace(plane) perpendicular to row space(line)

• $Ax=0$ has $r$ pivots and $n-r$ free variables: $n$ columns minus $r$ pivot columns. The nullspace matrix $N$ (contains all special solutions) contains the $n-r$ special solutions. Then $AN=0$

• $Ax=0$ has $r$ independent equations so it has $n-r$ independent solutions.

• $x_{particular}$: the particular solution solves $Ax_p=b$

• $x_{nullspace}$: the $n-r$ special solutions solve $Ax_n=0$

• Complete solution: one $x_p$, many $x_n$: $x=x_p+x_n$

• The four possibilities for linear equations depend on the rank $r$:

  • $r=m$, and $r=n$: Square ane invertible, $Ax=b$ has 1 solution

  • $r=m$, and $r<n$: Short and wide, $Ax=b$ has $\infty$ solutions
  • $r<m$, and $r=n$: Tall and thin, $Ax=b$ has 0 or 1 solutions
  • $r<m$, and $r<n$: Not full rank, $Ax=b$ has 0 or $\infty$ solutions

  • Independent vections (no extra vectors)

  • Spanning a space (enough vectors to produce the rest)
  • Basis for a space (not too many or too few)

  • Dimension of a space (the number of vectors in a basis)

  • Any set of $n$ vectors in $R^m$ must be linearly dependent if $n>m$

  • The columns spans the column space. The rows span the row space

    • The column space / row space of a matrix is the subspace of $R^m$/$R^n$ spanned by the columns/rows.

  • A basis for a vector space is a sequence of vectors with two properties: linear independent and span the space.

    • The basis is not unique. But the combination that produces the vector is unique.
    • The columns of a $n \times n$ invertible matrix are a basis for $R^n$.
    • The pivot columns of A are a basis for its column space.

  • DEFINITION: The dimension of a space is the number of vectors in every basis.

  The space $Z$ that contains only the zero vector. The dimension of this space is zero. The empty set (containing no vectors) is a basis for Z. We can never allow the zero vector into a basis, because then linear independence is lost.

  Four Fundamental Subspaces
  1. The row space $C(A^T)$, a subspace of $R^n$
  2. The column space $C(A)$, a subspace of $R^m$
  3. The nullspace is $N(A)$, a subspace of $R^n$
  4. The left nullspace $N(A^T)$, a subspace of $R^m$

  1. $A$ has the same row space as $R$. Same dimension $r$ and same basis.
  2. The column space of $A$ has dimension $r$. The number of independent columns equals the number of independent rows.
  3. $A$ has the same nullspace as $R$. Same dimension $n-r$ and same basis.
  4. The left nullspace of $A$ (the nullspace of $A^T$ has dimension $m-r$.

  Fundamental Theorem of Linear Algebra, Part 1

  • The column space and row space both have dimension $r$.

  • The nullspaces have dimensions $n-r$ and $m-r$.

  • Every rank one matrix has the special form $A=uv^T=column \times row.$

  • The nullspace $N(A)$ and the row space $C(A^T)$ are orthogonal subspaces of $R^n$.

  • DEFINITION: The orthogonal complement of a subspace $V$ contains every vector that is perpendicular to $V$.

  Fundamental Theorem of Linear Algebra, Part 2
  * $N(A)$ is the orthogonal complement of the row space $C(A^T)$ (in $R^n$)
  * $N(A^T)$ is the orthogonal complement of the column space $C(A)$ (in $R^m$)

  Projection Onto a Line
  * The projection matrix $P=\frac{aa^T}{a^Ta}$ onto the line through $a$
  * The projection $p=\bar{x}a=\frac{a^Tb}{a^Ta}a$

  Projection Onto a Subspace

  Problem: Find the combination $p=\bar{x_1}a_1+\cdots+\bar{x_n}a_n$ closest to a given vector $b$. The $n$ vectors $a_1, \cdots, a_n$ in $R_m$ span the column space of $A$. Thus the problem is to find the particular combination $p=A\bar{x}$(the projection) that is closest to $b$. When $n=1$, the best choice is $\frac{a^Tb}{a^Ta}$

  • $A^T(b-A\bar{x})=0$, or $A^TA\bar{x}=A^Tb$

  • The symmetric matrix $A^TA$ is $n\times n$. It is inverible if the $a$’s are independent.

  • The solution is $\bar{x}=(A^TA)^{-1}A^Tb$

  • The projection of $b$ onto the subspace $p=A\bar{x}=A(A^TA)^{-1}A^Tb$
  • The projection matrix $P=A(A^TA)^{-1}A^T$

  * $A^TA$ is invertible if and only if $A$ has linearly independent columns*

  Least Squares Approximations

  • When $Ax=b$ has no solution, multiply by $A^T$ and solve $A^TA\bar{x}=A^Tb$

    • The least squares solution $\bar{x}$ minimizes $E=||Ax-b||^2$. This is the sum of squares of the errors in the $m$ equations ($m>n$)

    • The best $\bar{x}$ comes from the normal equations $A^TA\bar{x}=A^Tb$

    Orthogonal Bases and Gram-Schmidt

    • orthonormal vectors

      • A matrix with orthonormal columns is assigned the special letter $Q$. The matrix $Q$ is easy to work with because $Q^TQ=I$
      • When $Q$ is square, $Q^TQ=I$ means that $Q^T=Q^{-1}$: transpose = inverse.
      • If the columns are only orthogonal (not unit vectors), dot products give a diagonal matrix (not the identity matrix)

    • Every permutation matrix is an orthogonal matrix.

    • If $Q$ has orthonormal columns ($Q^TQ=I$), it leaves lengths unchanged

    • Orthogonal is good

    • Use Gram-Schmidt for the Factorization $A=QR$

    $\left[ {\begin{array}{*{20}{c}} {\rm{a}}&b&c \end{array}} \right] = \left[ {\begin{array}{*{20}{c}} {{q_1}}&{{q_2}}&{{q_3}} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {q_1^Ta}&{q_1^Tb}&{q_1^Tc}\\ {}&{q_2^Tb}&{q_2^Tc}\\ {}&{}&{q_3^Tc} \end{array}} \right]$

    • (Gram-Schmidt) From independent vectors $a_1, \cdots, a_n$, Gram-Schmidt constructs orthonormal vectors $q_1, \cdots, q_n$. The matrces with these columns satisfy $A=QR$. Then $R=Q^TA$ is upper triangular because later $q$’s are orthogonal to earlier $a$’s.

    • Least squares: $R^TR\bar{x}=R^TQ^Tb$ or $R\bar{x}=Q^Tb$ or $\bar{x}=R^{-1}Q^Tb$

    Determinants

    • The determinant is zero when the matrix has no inverse
    • The product of the pivots is the determinant
    • The determinant changes sign when two rows (or two columns) are exchanged
    • Determinants give $A^{-1}$ and $A^{-1}b$ (this formulat is called Cramer’s Rule)
    • When the edge of a box are the rows of $A$, the volume is $|det A|$
    • For $n$ special numbers $\lambda$, called eigenvalues, the determinants of $A-\lambda I$ is zero.

    The properties of the determinant

    1. The determinant of the $n \times n$ identity matrix is 1.
    2. The determinant changes sign when two rows are exchanged
    3. The determinant is a linear function of each row separately (all other rows stay fixed!)
    4. If two rows of $A$ are equal, then $det A = 0$
    5. Subtracting a multiple of one row from another row leave $det A$ unchanged.
       
       • $\left| {\begin{array}{*{20}{c}} {\rm{a}}&b\ {c - la}&{d - lb} \end{array}} \right| = \left| {\begin{array}{*{20}{c}} a&b\\ c&d \end{array}} \right|$
    6. A matrix with a row of zeros has $det A=0$
    7. If $A$ is triangular then $det A=a_{11}a_{22}\cdots a_{nn}=product of diagonal entries$
    8. If $A$ is singular then $det A=0$. If $A$ is invertible then $det A \neq 0$
       
       • Elimination goes from $A$ to $U$.
       • $det A = +-det U = +-(product of the pivots)$
    9. The determinant of $AB$ is $det A times det B$
    10. The transpose $A^T$ has the same determinant as $A$

    Every rule of the rows can apply to columns*

    Cramer’s Rule

    • If $det A$ is not zero, $Ax=b$ is solved by determinants:
      
      • $x_1=\frac{det B_1}{det A}, x_2=\frac{det B_2}{det A}, \cdots, x_n=\frac{det B_n}{det A}$
      • The matrix $B_j$ has the $j$th column of $A$ replaced by the vector $b$

    Cross Product

    • $||u \times v|| = ||u||||v|||\sin \theta|$

    • $|u \cdot v| = ||u||||v|||\cos \theta|$

    • The length of $u \times v$ equals the area of the parallelogram with sides $u$ and $v$

    • It points by the right hand rule (points along your right thumb when the fingers curl from $u$ to $v$

    Eigenvalues and Eigenvectors

    • The basic equation is $Ax=\lambda x$, The number $\lambda$ is an eigenvalue of $A$

      • When $A$ is squared, the eigenvectors stay the same. The eigenvalues are squared.

    • The projection matrix has eigenvalues $\lambda = 1$ and $\lambda = 0$

      • $P$ is singular, so $\lambda=0$ is an eigenvalue
      • Each column of $P$ adds to 1, so $\lambda=1$ is an eigenvalue
      • $P$ is symmetric, so its eigenvectors are perpendicular
    • Permutations have all $|\lambda|=1$

    • The reflection matrix has eigenvalues 1 and -1

    • Solve the eigenvalue problem for an $n \times n$ matrix

      • Compute the determinant of $A-\lambda I$. It is a polynomial in $\lambda$ of degree $n$
      • Find the roots of this polynomial
      • For each eigenvalue $\lambda$, solve $(A-\lambda I)x=0$ to find an eigenvector $x$

    • Bad news: elimination does not preserve the $\lambda$’s

    • Good news: the product of eigenvalues equals the determinant, the sum of the eigenvalues equals the sum of the diagonal entries (trace)

    Diagonalizing a Matrix

    • Suppose the $n \times n$ matrix $A$ has $n$ linearly independent eigenvectors $x_1, \cdots, x_n$. Put them into the columns of an eigenvector matrix $S$. Then $S^{-1}AS$ is the eigenvalue matrix $\Lambda$:

      • ${S^{ - 1}}AS = \Lambda = \left[ {\begin{array}{*{20}{c}} {{\lambda _1}}&{}&{}\ {}& \ddots &{}\ {}&{}&{{\lambda _n}} \end{array}} \right]$

    • There is no connection between invertibility and diagonalizability:

      • Invertibility is concerned with the eigenvalues ($\lambda=0$ or $\lambda \neq 0$)
      • Diagonalizability is concerned with the eigenvectors (too few or enough for $S$)

    Applications to differential equations

    • One equation $\frac{du}{dt}=\lambda u$ has the solution $u(t)=Ce^{\lambda t}$

    • $n$ equations $\frac{d\textbf{u}}{dt}=A\textbf{u}$ starting from the vector $\textbf{u}(0)$ at $t=0$

    • Solve linear constant coefficient equations by exponentials $e^{\lambda t}\textbf{x}$, when $A\textbf{x}=\lambda \textbf{x}$

    Symmetric Matrices

    • A symmetric matrix has only real eigenvalues.

    • The eigenvectors can be chosen orthonormal.

    • (Spectral Theorem) Every symmetric matrix haas the facorization $A=Q\Lambda Q^T$ with real eigenvalues in $\Lambda$ and orthonormal eigenvectors in $S=Q$:

      • Symmetric diagonalization: $A=Q\Lambda Q^{-1}=Q \Lambda Q^T$ with $Q^{-1}=Q^T$

    • （Orthogonal Eigenvectors） Eigenvectors of a real symmetric matrix (when they correspond to different $\lambda$’s) are always perpendicular.

    • product of pivots = determinant = product of eigenvalues

    • Eigenvalues VS. Pivots

    • For symmetric matrices the pivots and the eigenvalues have the same signs:

      • The number of positive eigenvalues of $A=A^T$ equals the number of positive pivots.

    • All symmetric matrices are diagonalizable

    Positive Definite Matrices

    • Symmetric matrices that have positive eigenvalues

    • 2 $\times$ 2 matrices

      • The eigenvalues of $A$ are positive if and only if $a>0$ and $ac-b^2>0$.

    • $x^TAx$ is positive for all nonzero vectors $x$

      • If $A$ and $B$ are symmetric positive definite, so is $A+B$

    • When a symmetric matrix has one of these five properties, it has them all:

      • All $n$ pivots are positive
      • All $n$ upper left determinants are positive
      • All $n$ eigenvalues are positive
      • $x^TAx$ is positive except at $x=0$. This is the energy-based definition
      • $A$ equals $R^TR$ for a matrix $R$ with independent columns

    Positive Semidefinite Matrices

    Similar Matrices

    • DEFINITION: Let $M$ be any invertible matrix. Then $B=M^{-1}AM$ is similar to $A$

    • (No change in $\lambda$’s) Similar matrices $A$ and $M^{-1}AM$ have the same eigenvalues. If $x$ is an eigenvector of $A$, then $M^{-1}x$ is an eigenvector of $B$. But two matrices can have the same repeated $\lambda$, and fail to be similar.

    Jordan Form

    • What is “Jordan Form”?
      
      • For every $A$, we want to choose $M$ so that $M^{-1}AM$ is nearly diagonal as possible

    • $J^T$ is the similar to $J$, the matrix $M$ that produces the similarity happens to be the reverse identity

    • (Jordan form) If $A$ has $s$ independent eigenvectors, it is similar to a matrix $J$ that has $s$ Jordan blocks on its diagonal: Some matrix $M$ puts $A$ into Jordan form.

      • Jordan block: The eigenvalue is on the diagonal with $1$’s just above it. Each block in $J$ has one eigenvalue $\lambda_i$, one eigenvector. and 1’s above the diagonal

    ${{\rm{M}}^{ - 1}}AM = \left[ {\begin{array}{*{20}{c}} {{J_1}}&{}&{}\ {}& \ddots &{}\ {}&{}&{{J_s}} \end{array}} \right] = J$

    ${J_i} = \left[ {\begin{array}{*{20}{c}} {{\lambda _i}}&1&{}&{}\ {}& \ddots &1&{}\ {}&{}& \ddots &1\ {}&{}&{}&{{\lambda _i}} \end{array}} \right]$

    • $A$ is similar to $B$ if they share the same Jordan form $J$ – not otherwise

    Singular Value Decomposition (SVD)

    • Two sets of singular vectors, $u$’s and $v$’s. The $u$’s are eigenvectors of $AA^T$ and the $v$’s are eigenvectors of $A^TA$.

    • The singular vectos $v_1, \cdots, v_r$ are in the row space of $A$. The outputs $u_1, \cdots, u_r$ are in the column space of $A$. The singular values $\sigma_1, \cdots, \sigma_r$ are all positive numbers, the equatinos $Av_i=\sigma_i u_i$ tell us:

    $A[\begin{array}{*{20}{c}} {{v_1}}& \cdots &{{v_r}} \end{array}] = \left[ {\begin{array}{*{20}{c}} {{u_1}}& \cdots &{{u_r}} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{\sigma _1}}&{}&{}\\ {}& \ddots &{}\\ {}&{}&{{\sigma _r}} \end{array}} \right]$

    • We need $n-r$ more $v$’s and $m-r$ more $u$’s, from the nullspace $N(A)$ and the left nullspace $N(A^T)$. They can be orthonormal bases for those two nullspaces. Include all the $v$’s and $u$’s in $V$ and $U$, so these matrices become square.

    $A[\begin{array}{*{20}{c}} {{v_1}}& \cdots &{{v_r}} \end{array} \cdots {v_n}] = \left[ {\begin{array}{*{20}{c}} {{u_1}}& \cdots &{{u_r} \cdots u{}_m} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{\sigma _1}}&{}&{}&{}\\ {}& \ddots &{}&{}\\ {}&{}&{{\sigma _r}}&{}\\ {}&{}&{}&{} \end{array}} \right]$

    $V$ is now a square orthogonal matrix, with $V^{-1}=V^T$. So $AV=U\Sigma$ can become $A=U\Sigma V^T$. This is the Singular Value Decomposition:
    

    $$A=U \Sigma V^T=u_1 \sigma_1 v_1^T+\cdots +u_r \sigma_r v_r^T$$

    The orthonormal columns of $U$ and $V$ are eigenvectors of $AA^T$ and $A^TA$