Machine Learning - Assignment 1

$\begin{array}{l}\text{ Machine Learning }\\ \\ \text{ Assignment 1 (Linear Algebra) }\\ \\ \text{Instructor: Beilun Wang }\text{Name:Daiyang Luan ID:61518421}\\ \end{array}$
​

Problem 1

​
Let two vectors $a=(1,2,3{)}^{\mathrm{T}}$ and $b=(-8,1,2{)}^{\mathrm{T}}$.Answer the following equations:
​
(1) Compute the ${\ell }_2$ norm of $a$ and $b$
​
(2) Calculate the Euclidean distance between $a$ and $b$
​
(3) Are $a$ and $b$ orthogonal?
​
Solution:

(1)The ${\ell }_2$ norm of $a$ is $\sqrt{14}$ and the ${\ell }_2$ norm of $b$ is $\sqrt{69}$.

(2)The Euclidean distance between $a$ and $b$ is $\sqrt{83}$.

(3)As $a^{\mathrm{T}}b=1\times (-8)+2\times 1+3\times 2=0$, $a$ and $b$ is orthogonal.
​

Problem 2

​
Suppose $A=\left[\begin{array}{ccc}1& -3& 3\\ 3& -5& 3\\ 6& -6& 4\end{array}\right]$, answer the following questions:
​
(1) Calculate $A^{-1}$ and $\det (A)$.
​
(2) The Rank of $A$ is?
​
(3) The trace of $A$ is?
​
(4) Calculate $A+A^T$
​
(5) Is $A$ an orthogonal matrix? State your reason.
​
(6) Calculate all the eigenvalue $\lambda$ and corresponding eigenvectors of $A$.
​
(7) Diagonalize the matrix $A$.
​
(8) Calculate the ${\ell }_{2,1}$ norm $\Vert A{\Vert }_{2,1}$ and the Frobenius norm (i.e. ${\ell }_2$ norm) $\Vert A{\Vert }_F$
​
(9) Calculate the nuclear norm $\Vert A{\Vert }_{\ast }$ and the spectral norm $\Vert A{\Vert }_2$

Solution:
(1)$\left[\begin{array}{cc}A& I\end{array}\right]=\left[\begin{array}{cccccc}1& -3& 3& 1& 0& 0\\ 3& -5& 3& 0& 1& 0\\ 6& -6& 4& 0& 0& 1\end{array}\right]\stackrel{row}{⟶}\left[\begin{array}{cccccc}1& 0& 0& -1/8& -3/8& 3/8\\ 0& 1& 0& 3/8& -7/8& 3/8\\ 0& 0& 1& 3/4& -3/4& 1/4\end{array}\right]=\left[\begin{array}{cc}I& A^{-1}\end{array}\right]$
Hence,$A^{-1}=\left[\begin{array}{ccc}-1/8& -3/8& 3/8\\ 3/8& -7/8& 3/8\\ 3/4& -3/4& 1/4\end{array}\right]$
$det(A)=\left|\begin{array}{ccc}1& -3& 3\\ 3& -5& 3\\ 6& -6& 4\end{array}\right|=\left|\begin{array}{ccc}1& -3& 3\\ 0& 4& -6\\ 0& 0& 4\end{array}\right|=16$

(2)As $det(A)\ne 0$, $A$ is a full-rank matrix. Thus, the rank of $A$ is $3$.

(3)$tr(A)=1+(-5)+4=0$. That is, the trace of $A$ is $0$.

(4)$A+A^T=\left[\begin{array}{ccc}1& -3& 3\\ 3& -5& 3\\ 6& -6& 4\end{array}\right]+\left[\begin{array}{ccc}1& 3& 6\\ -3& -5& -6\\ 3& 3& 4\end{array}\right]=\left[\begin{array}{ccc}2& 0& 9\\ 0& -10& -3\\ 9& -3& 8\end{array}\right]$

(5)$A^{T}A=\left[\begin{array}{ccc}46& -54& 36\\ -54& 70& -48\\ 36& -48& 34\end{array}\right]\ne I$, so $A$ is not an orthogonal matrix.

(6)The characteristic determinant of $A$ is $\left|\begin{array}{ccc}\lambda -1& 3& -3\\ -3& \lambda +5& -3\\ -6& 6& \lambda -4\end{array}\right|=(\lambda +2{)}^2(\lambda -4).$ Thus, all the eigenvalues of $A$ are ${\lambda }_1={\lambda }_2=-2,{\lambda }_3=4.$ Let $A{\alpha }_i={\lambda }_i{\alpha }_i,i=1,2,3$. Then we have ${\alpha }_1=\left[\begin{array}{c}1\\ 1\\ 0\end{array}\right],{\alpha }_2=\left[\begin{array}{c}0\\ 1\\ 1\end{array}\right],{\alpha }_3=\left[\begin{array}{c}1\\ 1\\ 2\end{array}\right]$. ${\alpha }_i(i=1,2,3)$ are the corresponding eigenvectors.

(7)The diagonal matrix corresponding to matrix $A$ is $\left[\begin{array}{ccc}-2& 0& 0\\ 0& -2& 0\\ 0& 0& 4\end{array}\right]$

(8)In order to calculate the ${\ell }_{2,1}$ norm $\Vert A{\Vert }_{2,1}$, we first calculate the 2-norm of each row:$\sqrt{19},\sqrt{43},2\sqrt{22}$. Thus, $\Vert A{\Vert }_{2,1}=\sqrt{19}+\sqrt{43}+2\sqrt{22}$.
$\Vert A{\Vert }_F={\left(\sum _{i=1}^m\sum _{j=1}^n(a_{ij}{)}^2\right)}^{\frac{1}{2}}=\sqrt{1+9+9+9+25+9+36+36+16}=\sqrt{150}.$

(9)The nuclear norm $\Vert A{\Vert }_{\ast }$ is defined as the sum of all the singular values of matrix $A$. As is calculated above, $A^{T}A=\left[\begin{array}{ccc}46& -54& 36\\ -54& 70& -48\\ 36& -48& 34\end{array}\right]$. Supposing the eigenvalues of $A^{T}A$ are ${\lambda }_i,i=1,2,3$, we have $|\lambda I-A|=0$.
That is,
$$\left|\begin{array}{lcc}\lambda -46& 54& -36\\ 54& \lambda -70& 48\\ -36& 48& \lambda -34\end{array}\right|=0$$
Hence, we have $${\lambda }^3-150{\lambda }^2+648\lambda -256=0$$
The solution of the equation is:
$${\lambda }_1=4$$$${\lambda }_2=73+9\sqrt{65}$$$${\lambda }_3=73-9\sqrt{65}$$
Thus, $\Vert A{\Vert }_{\ast }=2+\sqrt{73+9\sqrt{65}}+\sqrt{73-9\sqrt{65}}\approx 14.727922061357859$.
$\Vert A{\Vert }_2=\sqrt{max(A^{T}A})=\sqrt{73+9\sqrt{65}}\approx 12.064838156174618$

Problem 3​

Please give some proper steps to show how you get the answer. Let $x={\left(x_1,x_2,x_3\right)}^T$ and
$$\{\begin{array}{l}2x_1+2x_2+3x_3=1\\ x_1-x_2=-1\\ -x_1+2x_2+x_3=2\end{array}$$
Answer the following questions:
​
(1) Solve the linear equations
​
(2) Write it into matrix form(i.e. $Ax=b$ ) and we will use the same $A$ and $b$ in the following questions.
​
(3) The Rank of $A$ is?
​
(4) Calculate $A^{-1}$ and $\det (A)$
​
(5) Use (4) to solve the linear equations
​
(6) Calculate the inner product and outer product of $x$ and $b$.(i.e. $⟨x,b⟩$ and $x\otimes b$ )
​
(7) Calculate the ${\ell }_1,{\ell }_2$ and ${\ell }_{\infty }$ norm of $b$
​
(8) Suppose $y=\left(y_1,y_2,y_3\right),$ calculate $y^{T}Ay,{\nabla }_y{y}^{T}Ay$
​
(9) We add one linear equation $-x_1+2x_2+x_3=2$ into linear equations above. Write it into matrix form(i.e. $A_{1}x=b)$
​
(10) The rank of $A_1$ is?
​
(11) Could these linear equations $A_{1}x=b$ be solved? State reasons.
​
​Solution:
(1)Solving the linear equations, we have: $x_1=-1,x_2=0,x_3=1$.

(2)The linear equation can be written into matrix form $Ax=b$ where
$A=\left[\begin{array}{lcc}2& 2& 3\\ 1& -1& 0\\ -1& 2& 1\end{array}\right]$
and
$b=\left[\begin{array}{l}1\\ -1\\ 2\end{array}\right]$

(3)The rank of $A$ is 3.

(4)$A^{-1}=\left[\begin{array}{lcc}1& -4& -3\\ 1& -5& -3\\ -1& 6& 4\end{array}\right]$
$det(A)=-1.$

(5)$$x=A^{-1}b=\left[\begin{array}{lcc}1& -4& -3\\ 1& -5& -3\\ -1& 6& 4\end{array}\right]\left[\begin{array}{l}1\\ -1\\ 2\end{array}\right]=\left[\begin{array}{l}-1\\ 0\\ 1\end{array}\right]$$
That is, $x_1=-1,x_2=0,x_3=1$, which is consistent with the result of question1.

(6)$<x,b>=1,x⨂b={\left[\begin{array}{lcc}1& 3& 1\end{array}\right]}^T$

(7)The ${\ell }_1$ norm of $b$ is $\Vert b{\Vert }_1=1+1+2=4$.
The ${\ell }_2$ norm of $b$ is $\Vert b{\Vert }_2=\sqrt{1+1+4}=\sqrt{6}$.
The ${\ell }_{\infty }$ norm of $b$ is $\Vert b{\Vert }_{\infty }=max(1,1,2)=2$.

(8)$$y^{T}Ay=\left[\begin{array}{lcc}{y}_1& y_2& y_3\end{array}\right]\left[\begin{array}{lcc}2& 2& 3\\ 1& -1& 0\\ -1& 2& 1\end{array}\right]\left[\begin{array}{l}{y}_1\\ y_2\\ y_3\end{array}\right]=2y_1^2-y_2^2+y_3^2+3y_1{y}_2+2y_2{y}_3+2y_1{y}_3$$
$${\nabla }_y{y}^{T}Ay=\left[\begin{array}{l}4y_1+3y_2+2y_3\\ 3y_1-2y_2+2y_3\\ 2y_1+2y_2+2y_3\end{array}\right]$$

(9)The new linear equation can be written into matrix form $A_{1}x=b_1$ where
$A_1=\left[\begin{array}{lcc}2& 2& 3\\ 1& -1& 0\\ -1& 2& 1\\ -1& 2& 1\end{array}\right]$
and
$b_1=\left[\begin{array}{l}1\\ -1\\ 2\\ 2\end{array}\right]$

(10)The rank of $A_1$ is 3.

(11)Yes.
The number of variables is the same as the rank of the new matrix $A_1$ and thus there is no more than one solution to the non homogeneous linear equations. Moreover, after diagonalizing the matrix $A$, we can see that after deleting the row whose elements are all zero, determinant of the new matrix is not zero. This indicates that a solution exists for these linear equations.