About Matrix Decompositions

Many complex matrix operations cannot be solved efficiently or with stability using the limited precision of computers. Matrix decompositions are methods that reduce a matrix into constituent parts that make it easier to calculate more complex matrix operations. Matrix decomposition methods, also called matrix factorization methods, are a foundation of linear algebra in computers, even for basic operations such as solving systems of linear equations, calculating the inverse, and calculating the determinant of a matrix. In this tutorial, you will discover matrix decompositions and how to calculate them in Python. After completing this tutorial, you will know:

•  What a matrix decomposition is and why these types of operations are important.
• How to calculate an LU and QR matrix decompositions in Python.
• How to calculate a Cholesky matrix decomposition in Python.

1.1 Tutorial Overview

This tutorial is divided into 3 parts, they are:

1. What is a Matrix Decomposition
2. LU Decomposition
3. QR Decomposition
4. Cholesky Decomposition

1.2 What is a Matrix Decomposition

A matrix decomposition is a way of reducing a matrix into its constituent parts. It is an approach that can simplify more complex matrix operations that can be performed on the decomposed matrix rather than on the original matrix itself. A common analogy for matrix decomposition is the factoring of numbers, such as the factoring of 10 into 2 × 5. For this reason, matrix decomposition is also called matrix factorization. Like factoring real values, there are many ways to decompose a matrix, hence there are a range of different matrix decomposition techniques. Two simple and widely used matrix decomposition methods are the LU matrix decomposition and the QR matrix decomposition. Next, we will take a closer look at each of these methods.

1.3 LU Decomposition

 

# LU decompositions
from numpy import array
from scipy.linalg import lu
# Define a square matrix
A = array([
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
])
print(A)

#factorize
P,L,U = lu(A)
print(P)
print(L)
print(U)
# reconstruct
B = P.dot(L).dot(U)
print(B)

 Running the example first prints the defined 3 × 3 matrix, then the P, L, and U components of the decomposition, then finally the original matrix is reconstructed.

 1.4 QR Decomposition

# Example of calculating an QR decomposition
# QR decomposition
from numpy import array
from numpy.linalg import qr
# define rectangular matrix
A = array([
    [1, 2],
    [3, 4],
    [5, 6]
])
print(A)
# factorize
Q,R = qr(A,'complete')
print(Q)
print(R)

# reconstruct 
B = Q.dot(R)
print(B)

Running the example first prints the defined 3 × 2 matrix, then the Q and R elements, then finally the reconstructed matrix that matches what we started with.

1.5 Cholesky Decomposition

The Cholesky decomposition is for square symmetric matrices where all values are greater than zero, so-called positive definite matrices. For our interests in machine learning, we will focus on the Cholesky decomposition for real-valued matrices and ignore the cases when working with complex numbers. The decomposition is defined as follows:

                                        

Or without the dot notation:

                                        

Where A is the matrix being decomposed, L is the lower triangular matrix and L T is the transpose of L. The decompose can also be written as the product of the upper triangular matrix, for example:

                                     

Where U is the upper triangular matrix. The Cholesky decomposition is used for solving linear least squares for linear regression, as well as simulation and optimization methods. When decomposing symmetric matrices, the Cholesky decomposition is nearly twice as efficient as the LU decomposition and should be preferred in these cases.

The Cholesky decomposition can be implemented in NumPy by calling the cholesky() function. The function only returns L as we can easily access the L transpose as needed. The example below defines a 3×3 symmetric and positive definite matrix and calculates the Cholesky decomposition, then the original matrix is reconstructed.

# Cholesky decomposition
from numpy import array
from numpy.linalg import cholesky

# define  sysmmetrical matrix
A = array([
    [2, 1, 1],
    [1, 2, 1],
    [1, 1, 2]
])
print(A)
# factorize
L = cholesky(A)
print(L)

# reconstruct
B = L.dot(L.T)
print(B)

1.6 Extwensions

This section lists some ideas for extending the tutorial that you may wish to explore.

• Write a summary of matrix decomposition to explain the principle to other students.
• Create one example using each operation with your own small array data.
• Search machine learning papers and find  1 example of each operation being used.

If you explore any of these extensions, I'd love to know.

1.7 Summary

In this tutorial, you discovered matrix decompositions and how to calculate them in Python. Specifically, you learned:

• What a matrix decomposition is and why these types of operations are important.
• How to calculate an LU and QR matrix decompositions in Python.
• How to calculate a Cholesky matrix decomposition in Python.