Linear Algebra Review

Matlab is a vector / matrix language, which means it thinks and stores information in either a scalar, vector or matrix. Matlab is designed to execute mathematical operations by entering them just as you would write them when solving math by hand. Reviewing how matrices interact is a valuable way to gain insight into how Matlab works, and how it can work for you.

Matrix Addition & Scalar Multiplication

Suppose the matrices A and B exist such that

\[\begin{split}\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix} \mathbf{B} = \begin{bmatrix} b_{11} & b_{12} & \cdots & b_{1n} \\ b_{21} & b_{22} & \cdots & b_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ b_{m1} & b_{m2} & \cdots & b_{mn} \end{bmatrix}\end{split}\]

Let the matrices A and B be of equal size (m x n), then the sum of A and B is the sum of the corresponding elements of A and B

\[\begin{split}\mathbf{A+B} = \begin{bmatrix} a_{11}+b_{11} & a_{12}+b_{12} & \cdots & a_{1n}+b_{1n} \\ a_{21}+b_{21} & a_{22}+b_{22} & \cdots & a_{2n}+b_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1}+b_{m1} & a_{m2}+b_{m2} & \cdots & a_{mn}+b_{mn} \end{bmatrix}\end{split}\]

The product of any scalar k with the matrix A is equal to \(k \cdot A\). The resulting matrix is obtained by multiplying each element of A by k

\[\begin{split}\mathbf{k \cdot A} = \begin{bmatrix} ka_{11} & ka_{12} & \cdots & ka_{1n} \\ ka_{21} & ka_{22} & \cdots & ka_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ ka_{m1} & ka_{m2} & \cdots & ka_{mn} \end{bmatrix}\end{split}\]

Consider the scalar \(k=-1\), then

\[-A=(-1)A\]\[A-B = A+(-B)\]

Theorems

Let the matrices A,B,C be of equal size, then for any scalars k, k’

  • \(A + B = B + A\)
  • \(A + 0 = 0 + A = A\)
  • \(A + (-A) = (-A) + A = 0\)
  • \((A + B) + C = A + (B + C)\)
  • \(1 \cdot A = A\)
  • \((k + k')A = kA + k'A\)
  • \((kk')A = k(k'A)\)
  • \(k(A + B) = kA + kB\)

Matrix Multiplication

Suppose the matrices A and B exist such that

\[\begin{split}\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1p} \\ a_{21} & a_{22} & \cdots & a_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mp} \end{bmatrix} \mathbf{B} = \begin{bmatrix} b_{11} & b_{12} & \cdots & b_{1n} \\ b_{21} & b_{22} & \cdots & b_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ b_{p1} & b_{p2} & \cdots & b_{pm} \end{bmatrix}\end{split}\]

Let A be an (m x p) matrix, and let B be an (p x n) matrix. If the number of columns of A are equal to the number of rows of B, then the product AB is an (m x n) matrix whose ij-element is equal to the dot product of the i-th row of A with the j-th column of B

\[\begin{split}\begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1p} \\ a_{21} & a_{22} & \cdots & a_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mp} \end{bmatrix} \begin{bmatrix} b_{11} & b_{12} & \cdots & b_{1n} \\ b_{21} & b_{22} & \cdots & b_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ b_{p1} & b_{p2} & \cdots & b_{pm} \end{bmatrix} = \begin{bmatrix} c_{11} & c_{12} & \cdots & c_{1n} \\ c_{21} & c_{22} & \cdots & c_{2n} \\ \vdots & \vdots & c_{ij} & \vdots \\ c_{m1} & c_{m2} & \cdots & c_{mn} \end{bmatrix}\end{split}\]\[c_{ij} = a_{i1}b_{1j} + a_{i2}b_{2j} + \cdots + a_{ip}b_{pj} = \sum_{k=1}^{p}a_{ik}b_{kj}\]

If the number of columns of A do not equal the number of rows of the B, then the product AB is not defined

Matrix Power

Let A be an n x n matrix, then

  • \(A^{0} = I\)
  • \(AA=A^{2}\)
  • \(AAA = A^{2}A = A^{3}\)
  • \(A^{n}A = A^{n+1}\)

Theorems

Let A,B,C be matrices whose products are defined and let k be any scalar, then

  • \(AB \neq BA\)
  • \((AB)C = A(BC)\)
  • \(A(B+C)=AB+AC\)
  • \((B+C)A = BA + CA\)
  • \(k(AB) = (kA)B = A(kB)\)

Matrix Transpose

Let A be an (m x n) matrix of the form

\[\begin{split}\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix}\end{split}\]

The transpose of A, or \(A^{T}\), is obtained by any of the following:

  • Writing the columns of A as rows
  • Writing the rows of A as columns
  • Reflecting A over the main diagonal

This results is an (n x m) matrix such that \(A=[a_{ij}]\) and \(A^{T}=[a_{ji}]\)

\[\begin{split}\mathbf{A^{T}} = \begin{bmatrix} a_{11} & a_{21} & \cdots & a_{m1} \\ a_{12} & a_{22} & \cdots & a_{m2} \\ \vdots & \vdots & \ddots & \vdots \\ a_{1n} & a_{2n} & \cdots & a_{mn} \end{bmatrix}\end{split}\]

Theorems

Let A and B be matrices where the sums and products are defined, and let k be any scalar, then

  • \((A+B)^{T} = A^{T} + B^{T}\)
  • \((kA)^{T} = kA^{T}\)
  • \((A^{T})^{T} = A\)
  • \((AB)^{T} = B^{T}A^{T}\)

Useful Properties of Square Matrices

Identity Matrix

An identity matrix is an (n x n) matrix with 1’s on the diagonal and 0’s everywhere else.

Let A be an n x n matrix such that \([a_{ij}] = 1\) if \(i=j\), and \([a_{ij}] = 0\) if \(i \neq j\) , then A is an identity matrix I

\[\begin{split}\mathbf{A} = \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 1 \end{bmatrix} = \mathbf{I}\end{split}\]

For any (n x n) matrix B

\[IB = BI = B\]

For any scalar k, the matrix kI is called the scalar matrix. Multiplying any (n x n) matrix B by any scalar matrix has the same effect as multiplying B by a scalar

\[(kI)B = k(IB) = kB\]

Diagonal Matrix

  • The diagonal elements of an (n x n) matrix D are the elements \([d_{ij}]\) such that \(i=j\).
  • D is said to be diagonal if all of the off diagonal elements are zero, or \([d_{ij}]=0\) when \(i \neq j\).
  • The identity matrix is a particular case of diagonal matrix

Trace

The trace of a matrix A is the sum of the diagonal elements of A

Let A and B be (n x n) matrices, then:

  • \(tr(A) = a_{11} + a_{22} + \cdots + a_{nn}\)
  • \(tr(A+B) = tr(A) + tr(B)\)
  • \(tr(kA) = k*tr(A)\)
  • \(tr(A^{T}) = tr(A)\)
  • \(tr(AB) = tr(BA)\)

Triangular Matrix

  • A triangular matrix, sometimes called upper triangular, is an (n x n) matrix whose elements below the main diagonal are all zero

  • A lower triangular matrix is an (n x n) matrix whose elements above the main diagonal are all zero

Example of an upper triangular matrix:

\[\begin{split}\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ 0 & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & a_{nn} \end{bmatrix}\end{split}\]

Example of a lower triangular matrix:

\[\begin{split}\mathbf{A} = \begin{bmatrix} a_{11} & 0 & \cdots & 0 \\ a_{21} & a_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nn} \end{bmatrix}\end{split}\]

Matrix Inverse

  • An (n x n) matrix A is said to be invertible or nonsingular if there exists a matrix B such that
\[AB = BA = I\]
  • A matrix B that has this property is said to be the inverse of A, and is denoted \(A^{-1}\)

  • The inverse of a matrix is often used to solve the equation \(Ax=b\) by multiplying both sides of the equation with \(A^{-1}\)
\[\begin{split}Ax &= b \\ A^{-1}Ax &= A^{-1}b \\ x &= A^{-1}b\end{split}\]
  • A matrix is invertible only if A is nonsingular, or equivalently, if the determinant of A does not equal zero

Steps for finding \(A^{-1}\)

Step 1 Form the (n x 2n) matrix \([A | I]\)

Step 2 Use elementary row operations to transform \([A | I]\) into \([I|B]\)

Step 3 Now, \(A^{-1} = B\)

Matrix Determinant

  • A system of equations has a unique solution if and only if the determinant of its coefficient matrix does not equal zero

  • If the determinant is equal to zero, then the system either has no solution or infinite solutions

First Order Determinant

The determinant of a scalar matrix A

\[\begin{split}\det(A) &= \begin{vmatrix} a_{11} \end{vmatrix} = a_{11}\end{split}\]

Second Order Determinant

The determinant of a (2 x 2) matrix A

\[\begin{split}\det(A) = \begin{vmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{vmatrix} = a_{11}a_{22} - a_{21}a_{12}\end{split}\]

Third Order Determinant

The determinant of a (3 x 3) matrix A

\[\begin{split}\det(A) &= \begin{vmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{vmatrix} \\ \\ &= a_{11} \begin{vmatrix} a_{22} & a_{23} \\ a_{32} & a_{33} \end{vmatrix} - a_{12} \begin{vmatrix} a_{21} & a_{23} \\ a_{31} & a_{33} \end{vmatrix} + a_{13} \begin{vmatrix} a_{21} & a_{22} \\ a_{31} & a_{32} \end{vmatrix} \\ \\ &= a_{11}(a_{22}a_{23} - a_{23}a_{32}) -a_{12}(a_{21}a_{33} - a_{23}a_{31}) +a_{13}(a_{21}a_{32} - a_{22}a_{31})\end{split}\]

Eigenvalues & Eigenvectors

For any (n x n) matrix A, a scalar \(\lambda\) is called an eigenvalue of A if there exists a nonzero vector \(\upsilon\) such that

\[A \upsilon = \lambda \upsilon\]

Any vector \(\upsilon\) that satisfies this relationship is called an eigenvector of A belonging to the eigenvalue \(\lambda\)

Finding Eigenvalues & Eigenvectors

We need to find all scalars \(\lambda\) such that the equation \(A \upsilon = \lambda \upsilon\) has a nonzero solution \(\upsilon\), which is equivalent to solving:

\[\begin{split}A \upsilon &= \lambda \upsilon \\ A \upsilon - \lambda \upsilon &= \theta \\ (A - \lambda I)\upsilon &= \theta\end{split}\]

The solution then has two parts:

  1. Find all scalars \(\lambda\) such that \(A-\lambda I\) is singular. This is equivalent to solving the characteristic polynomial of the \(\det(A-\lambda I)\)

  1. Given a scalar \(\lambda\) such that \(A-\lambda I\) is singular, find all nonzero vectors \(\upsilon\) such that \((A - \lambda I)\upsilon = \theta\). After you have solved for the eigenvalues, plug them in and solve for all values \(\upsilon\)